.\" Automatically generated by Pod::Man 4.07 (Pod::Simple 3.32) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. \*(C+ will .\" give a nicer C++. Capital omega is used to do unbreakable dashes and .\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff, .\" nothing in troff, for use with C<>. .tr \(*W- .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' . ds C` . ds C' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is >0, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .\" .\" Avoid warning from groff about undefined register 'F'. .de IX .. .if !\nF .nr F 0 .if \nF>0 \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . if !\nF==2 \{\ . nr % 0 . nr F 2 . \} .\} .\" .\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2). .\" Fear. Run. Save yourself. No user-serviceable parts. . \" fudge factors for nroff and troff .if n \{\ . ds #H 0 . ds #V .8m . ds #F .3m . ds #[ \f1 . ds #] \fP .\} .if t \{\ . ds #H ((1u-(\\\\n(.fu%2u))*.13m) . ds #V .6m . ds #F 0 . ds #[ \& . ds #] \& .\} . \" simple accents for nroff and troff .if n \{\ . ds ' \& . ds ` \& . ds ^ \& . ds , \& . ds ~ ~ . ds / .\} .if t \{\ . ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u" . ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u' . ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u' . ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u' . ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u' . ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u' .\} . \" troff and (daisy-wheel) nroff accents .ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V' .ds 8 \h'\*(#H'\(*b\h'-\*(#H' .ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#] .ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H' .ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u' .ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#] .ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#] .ds ae a\h'-(\w'a'u*4/10)'e .ds Ae A\h'-(\w'A'u*4/10)'E . \" corrections for vroff .if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u' .if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u' . \" for low resolution devices (crt and lpr) .if \n(.H>23 .if \n(.V>19 \ \{\ . ds : e . ds 8 ss . ds o a . ds d- d\h'-1'\(ga . ds D- D\h'-1'\(hy . ds th \o'bp' . ds Th \o'LP' . ds ae ae . ds Ae AE .\} .rm #[ #] #H #V #F C .\" ======================================================================== .\" .IX Title "Catmandu::Introduction 3pm" .TH Catmandu::Introduction 3pm "2017-01-04" "perl v5.24.1" "User Contributed Perl Documentation" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh .SH "NAME" Catmandu::Introduction \- An Introduction to Catmandu data processing toolkit .SH "Introduction" .IX Header "Introduction" Catmandu is a data processing toolkit developed as part of the LibreCat project. Catmandu provides the command line client catmandu and a suite of tools to ease the import, storage, retrieval, export and transformation of data. For instance, to transform a \s-1CSV\s0 file into \s-1JSON\s0 use the command: .PP .Vb 1 \& $ catmandu convert JSON to CSV < data.json .Ve .PP Or, to store a \s-1YAML\s0 file into an ElasticSearch database type (requires Catmandu::ElasticSearch): .PP .Vb 1 \& $ catmandu import YAML to ElasticSearch \-\-index_name demo < test.yml .Ve .PP To export all the data from an Solr search engine into \s-1JSON\s0 type (requires Catmandu::Solr): .PP .Vb 1 \& $ catmandu export Solr \-\-url http://localhost:8983/solr to JSON .Ve .PP With Catmandu one can import OAI-PMH records in your application (requires Catmandu::OAI): .PP .Vb 1 \& $ catmandu convert OAI \-\-url http://biblio.ugent.be/oai \-\-set allFtxt .Ve .PP and export records into formats such as \s-1JSON, YAML, CSV, XLS, RDF\s0 and many more. .PP Catmandu also provides a small scripting language to manipulate data, extract parts of your dataset and transform records. For instance, rename fields with the 'move_field' Fix command: .PP .Vb 1 \& $ catmandu convert JSON \-\-fix \*(Aqmove_field(title,my_title)\*(Aq < data.json .Ve .PP In the example above, we renamed all the 'title' fields in the dataset into the 'my_title' field. .PP One can also work on deeply nested data. E.g. create a deeply nested data structure with the \&'move_field' Fix command: .PP .Vb 1 \& $ catmandu convert JSON \-\-fix \*(Aqmove_field(title,my.deeply.nested.title)\*(Aq < data.json .Ve .PP In this example we moved the field 'title' into the field 'my', which contains a (sub)field 'deeply', which contains a (sub)field 'nested'. .PP Catmandu was originally created by librarians for librarians. We process a lot of metadata especially library metadata in formats such as \s-1MARC, MAB2\s0 and \s-1MODS.\s0 With the following command we can extract data from a marc record and to store it into the title field (requires Catmandu::MARC): .PP .Vb 1 \& $ catmandu convert MARC \-\-fix \*(Aqmarc_map(245,title)\*(Aq < data.mrc .Ve .PP Or, in case only the 245a subfield is needed write: .PP .Vb 1 \& $ catmandu convert MARC \-\-fix \*(Aqmarc_map(245a,title)\*(Aq < data.mrc .Ve .PP When processing data a lot of Fix commands could be required. It wouldn't be very practical to type them all on the command line. By creating a Fix script which contains all the fix commands complicated data transformations can be created. For instance, if the file \f(CW\*(C`myfixes.txt\*(C'\fR contains: .PP .Vb 5 \& marc_map(245a,title) \& marc_map(100a,author.$append) \& marc_map(700a,author.$append) \& marc_map(020a,isbn) \& replace_all(isbn,\*(Aq[^0\-9\-]+\*(Aq,\*(Aq\*(Aq) .Ve .PP then they can be executed on a \s-1MARC\s0 file using this command: .PP .Vb 1 \& $ catmandu convert MARC \-\-fix myfixes.txt < data.mrc .Ve .PP Fixes can also be turned into executable scripts by adding a bash 'shebang' line at the top. E.g. to harvest records from an \s-1OAI\s0 repository write this fix file: .PP .Vb 4 \& #!/usr/bin/env catmandu run \& do importer(OAI,url:"http://lib.ugent.be/oai") \& add_to_exporter(.,JSON) \& end .Ve .PP Run this (on Linux) by setting the executable bit: .PP .Vb 2 \& $ chmod 755 myfix.fix \& $ ./myfix.fix .Ve .PP To experiment with the Fix language you can also run the catmandu Fix interpreter in an interactive mode: .PP .Vb 8 \& $ catmandu run \& Catmandu 0.95 interactive mode \& Type: \eh for the command history \& fix > add_field(hello,world) \& \-\-\- \& hello: world \& ... \& fix > .Ve .PP Catmandu contains many powerful fixes. Visit to get an overview what is possible. .SH "Documentation" .IX Header "Documentation" For more information read our documentation pages and blog for a complete introduction and update into all Catmandu features. .PP In the winter of 2014 an Advent calendar tutorial was created to provide a day by day introduction into the \s-1UNIX\s0 command line and Catmandu: .PP .PP If you need extra training, our developers regulary host workshops at library conferences and events: .SH "Installation" .IX Header "Installation" There are several ways to get a working version of Catmandu on your computer. For a quick and demo installation visit our blog where a VirtualBox image is available containing all the Catmandu modules, including ElasticSearch and MongoDB. .PP On our website we provide installation instructions for: .IP "Debian" 4 .IX Item "Debian" .PD 0 .IP "Ubuntu Server" 4 .IX Item "Ubuntu Server" .IP "CentOS" 4 .IX Item "CentOS" .IP "openSUSE" 4 .IX Item "openSUSE" .IP "OpenBSD" 4 .IX Item "OpenBSD" .IP "Windows" 4 .IX Item "Windows" .PD .PP and even a generic installation using Docker : . .SH "Open Source" .IX Header "Open Source" Catmandu software published at https://github.com/LibreCat/Catmandu is free software without warranty, liabilities or support; you can redistribute it and/or modify it under the terms of the \s-1GNU\s0 General Public License as published by the Free Software Foundation; either version 2 or any later version. Every contributor is free to state her/his copyright. .SH "Developers & Support" .IX Header "Developers & Support" Catmandu has a very active international developer community. We welcome all feedback, bug reports and feature enhancement. .PP Join our mailing list to receive more information: \f(CW\*(C`librecat\-dev@librecat.org\*(C'\fR .PP Are a developer and want to contribute to the project? Feel free to submit pull requests or create new Catmandu packages! .SH "Kudos" .IX Header "Kudos" Catmandu is created in a cooperation with many developers world wide. Without them this project isn't possible. We would like to thank our core maintainer: Nicolas Steenlant and all contributors: Christian Pietsch , Dave Sherohman , Friedrich Summann , Jakob Voss , Johann Rolschewski , Jorgen Eriksson , Magnus Enger , Maria Hedberg , Mathias Loesch , Najko Jahn , Nicolas Franck , Patrick Hochstenbach , Petra Kohorst , Snorri Briem , Upasana Shukla and Vitali Peil .SH "SEE ALSO" .IX Header "SEE ALSO" Catmandu .PP