.\" Automatically generated by Pod::Man 4.10 (Pod::Simple 3.35) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. \*(C+ will .\" give a nicer C++. Capital omega is used to do unbreakable dashes and .\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff, .\" nothing in troff, for use with C<>. .tr \(*W- .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' . ds C` . ds C' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is >0, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .\" .\" Avoid warning from groff about undefined register 'F'. .de IX .. .nr rF 0 .if \n(.g .if rF .nr rF 1 .if (\n(rF:(\n(.g==0)) \{\ . if \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . if !\nF==2 \{\ . nr % 0 . nr F 2 . \} . \} .\} .rr rF .\" .\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2). .\" Fear. Run. Save yourself. No user-serviceable parts. . \" fudge factors for nroff and troff .if n \{\ . ds #H 0 . ds #V .8m . ds #F .3m . ds #[ \f1 . ds #] \fP .\} .if t \{\ . ds #H ((1u-(\\\\n(.fu%2u))*.13m) . ds #V .6m . ds #F 0 . ds #[ \& . ds #] \& .\} . \" simple accents for nroff and troff .if n \{\ . ds ' \& . ds ` \& . ds ^ \& . ds , \& . ds ~ ~ . ds / .\} .if t \{\ . ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u" . ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u' . ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u' . ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u' . ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u' . ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u' .\} . \" troff and (daisy-wheel) nroff accents .ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V' .ds 8 \h'\*(#H'\(*b\h'-\*(#H' .ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#] .ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H' .ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u' .ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#] .ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#] .ds ae a\h'-(\w'a'u*4/10)'e .ds Ae A\h'-(\w'A'u*4/10)'E . \" corrections for vroff .if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u' .if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u' . \" for low resolution devices (crt and lpr) .if \n(.H>23 .if \n(.V>19 \ \{\ . ds : e . ds 8 ss . ds o a . ds d- d\h'-1'\(ga . ds D- D\h'-1'\(hy . ds th \o'bp' . ds Th \o'LP' . ds ae ae . ds Ae AE .\} .rm #[ #] #H #V #F C .\" ======================================================================== .\" .IX Title "Catmandu::Introduction 3pm" .TH Catmandu::Introduction 3pm "2019-01-29" "perl v5.28.1" "User Contributed Perl Documentation" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh .SH "NAME" Catmandu::Introduction \- An Introduction to Catmandu data processing toolkit .SH "Introduction" .IX Header "Introduction" Catmandu is a data processing toolkit developed as part of the LibreCat project. Catmandu provides the command line client catmandu and a suite of tools to ease the import, storage, retrieval, export and transformation of data. For instance, to transform a \s-1CSV\s0 file into \s-1JSON\s0 use the command: .PP .Vb 1 \& $ catmandu convert JSON to CSV < data.json .Ve .PP Or, to store a \s-1YAML\s0 file into an ElasticSearch database type (requires Catmandu::ElasticSearch): .PP .Vb 1 \& $ catmandu import YAML to ElasticSearch \-\-index_name demo < test.yml .Ve .PP To export all the data from an Solr search engine into \s-1JSON\s0 type (requires Catmandu::Solr): .PP .Vb 1 \& $ catmandu export Solr \-\-url http://localhost:8983/solr to JSON .Ve .PP With Catmandu one can import OAI-PMH records in your application (requires Catmandu::OAI): .PP .Vb 1 \& $ catmandu convert OAI \-\-url http://biblio.ugent.be/oai \-\-set allFtxt .Ve .PP and export records into formats such as \s-1JSON, YAML, CSV, XLS, RDF\s0 and many more. .PP Catmandu also provides the small scripting language \*(L"Fix\*(R" to manipulate data, extract parts of your dataset and transform records. For instance, rename fields with the move_field Fix command: .PP .Vb 1 \& $ catmandu convert JSON \-\-fix \*(Aqmove_field(title,my_title)\*(Aq < data.json .Ve .PP In the example above, we renamed all the \f(CW\*(C`title\*(C'\fR fields in the dataset into the \f(CW\*(C`my_title\*(C'\fR field. .PP One can also work on deeply nested data. E.g. create a deeply nested data structure with the \f(CW\*(C`move_field\*(C'\fR Fix command: .PP .Vb 1 \& $ catmandu convert JSON \-\-fix \*(Aqmove_field(title,my.deeply.nested.title)\*(Aq < data.json .Ve .PP In this example we moved the field \f(CW\*(C`title\*(C'\fR into the field \f(CW\*(C`my\*(C'\fR, which contains a (sub)field \f(CW\*(C`deeply\*(C'\fR, which contains a (sub)field \f(CW\*(C`nested\*(C'\fR. .PP Catmandu was originally created by librarians for librarians. We process a lot of metadata especially library metadata in formats such as \s-1MARC, MAB2\s0 and \s-1MODS.\s0 With the following command we can extract data from a marc record and to store it into the title field (requires Catmandu::MARC): .PP .Vb 1 \& $ catmandu convert MARC \-\-fix \*(Aqmarc_map(245,title)\*(Aq < data.mrc .Ve .PP Or, in case only the 245a subfield is needed write: .PP .Vb 1 \& $ catmandu convert MARC \-\-fix \*(Aqmarc_map(245a,title)\*(Aq < data.mrc .Ve .PP When processing data a lot of Fix commands could be required. It wouldn't be very practical to type them all on the command line. By creating a Fix script which contains all the fix commands complicated data transformations can be created. For instance, if the file \f(CW\*(C`myfixes.txt\*(C'\fR contains: .PP .Vb 5 \& marc_map(245a,title) \& marc_map(100a,author.$append) \& marc_map(700a,author.$append) \& marc_map(020a,isbn) \& replace_all(isbn,\*(Aq[^0\-9\-]+\*(Aq,\*(Aq\*(Aq) .Ve .PP then they can be executed on a \s-1MARC\s0 file using this command: .PP .Vb 1 \& $ catmandu convert MARC \-\-fix myfixes.txt < data.mrc .Ve .PP Fixes can also be turned into executable scripts by adding a bash 'shebang' line at the top. E.g. to harvest records from an \s-1OAI\s0 repository write this fix file: .PP .Vb 4 \& #!/usr/bin/env catmandu run \& do importer(OAI,url:"http://lib.ugent.be/oai") \& add_to_exporter(.,JSON) \& end .Ve .PP Run this (on Linux) by setting the executable bit: .PP .Vb 2 \& $ chmod 755 myfix.fix \& $ ./myfix.fix .Ve .PP To experiment with the Fix language you can also run the catmandu Fix interpreter in an interactive mode: .PP .Vb 8 \& $ catmandu run \& Catmandu 0.95 interactive mode \& Type: \eh for the command history \& fix > add_field(hello,world) \& \-\-\- \& hello: world \& ... \& fix > .Ve .PP Catmandu contains many powerful fixes. Visit to get an overview what is possible. .SH "Documentation" .IX Header "Documentation" For more information read our documentation pages and blog for a complete introduction and update into all Catmandu features. .PP In the winter of 2014 an Advent calendar tutorial was created to provide a day by day introduction into the \s-1UNIX\s0 command line and Catmandu: .PP .PP If you need extra training, our developers regulary host workshops at library conferences and events: .SH "Installation" .IX Header "Installation" There are several ways to get a working version of Catmandu on your computer. .PP For a quick and demo installation visit our blog where a VirtualBox image is available containing all the Catmandu modules, including ElasticSearch and MongoDB. A similarly easy method is using Docker : At librecat/catmandu a Docker image is provided with version tags for each release, \f(CW\*(C`latest\*(C'\fR for the latest release and \f(CW\*(C`dev\*(C'\fR for the latest development version. The image can be tried online in a Jupyter notebook via binder . .PP On our website we provide installation instructions for: .IP "Debian" 4 .IX Item "Debian" .PD 0 .IP "Ubuntu Server" 4 .IX Item "Ubuntu Server" .IP "CentOS" 4 .IX Item "CentOS" .IP "openSUSE" 4 .IX Item "openSUSE" .IP "OpenBSD" 4 .IX Item "OpenBSD" .IP "Windows" 4 .IX Item "Windows" .PD .SH "Open Source" .IX Header "Open Source" Catmandu software published at https://github.com/LibreCat/Catmandu is free software without warranty, liabilities or support; you can redistribute it and/or modify it under the terms of the \s-1GNU\s0 General Public License as published by the Free Software Foundation; either version 2 or any later version. Every contributor is free to state her/his copyright. .SH "Developers & Support" .IX Header "Developers & Support" Catmandu has a very active international developer community. We welcome all feedback, bug reports and feature enhancement. .PP Join our mailing list to receive more information: \f(CW\*(C`librecat\-dev@librecat.org\*(C'\fR .PP Are a developer and want to contribute to the project? Feel free to submit pull requests or create new Catmandu packages! .SH "Kudos" .IX Header "Kudos" Catmandu is created in a cooperation with many developers world wide. Without them this project isn't possible. We would like to thank our core maintainer: Nicolas Steenlant and all contributors: Christian Pietsch, Dave Sherohman, Friedrich Summann, Jakob Voss, Johann Rolschewski, Jorgen Eriksson, Magnus Enger, Maria Hedberg, Mathias Loesch, Najko Jahn, Nicolas Franck, Patrick Hochstenbach, Petra Kohorst, Snorri Briem, Upasana Shukla and Vitali Peil. .SH "SEE ALSO" .IX Header "SEE ALSO" Catmandu .PP