.\" Automatically generated by Pod::Man 4.10 (Pod::Simple 3.35) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. \*(C+ will .\" give a nicer C++. Capital omega is used to do unbreakable dashes and .\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff, .\" nothing in troff, for use with C<>. .tr \(*W- .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' . ds C` . ds C' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is >0, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .\" .\" Avoid warning from groff about undefined register 'F'. .de IX .. .nr rF 0 .if \n(.g .if rF .nr rF 1 .if (\n(rF:(\n(.g==0)) \{\ . if \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . if !\nF==2 \{\ . nr % 0 . nr F 2 . \} . \} .\} .rr rF .\" ======================================================================== .\" .IX Title "Search::Elasticsearch::Client::2_0::Bulk 3pm" .TH Search::Elasticsearch::Client::2_0::Bulk 3pm "2018-12-07" "perl v5.28.1" "User Contributed Perl Documentation" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh .SH "NAME" Search::Elasticsearch::Client::2_0::Bulk \- A helper module for the Bulk API and for reindexing .SH "VERSION" .IX Header "VERSION" version 5.02 .SH "SYNOPSIS" .IX Header "SYNOPSIS" .Vb 1 \& use Search::Elasticsearch; \& \& my $es = Search::Elasticsearch\->new; \& my $bulk = $es\->bulk_helper( \& index => \*(Aqmy_index\*(Aq, \& type => \*(Aqmy_type\*(Aq \& ); \& \& # Index docs: \& $bulk\->index({ id => 1, source => { foo => \*(Aqbar\*(Aq }}); \& $bulk\->add_action( index => { id => 1, source => { foo=> \*(Aqbar\*(Aq }}); \& \& # Create docs: \& $bulk\->create({ id => 1, source => { foo => \*(Aqbar\*(Aq }}); \& $bulk\->add_action( create => { id => 1, source => { foo=> \*(Aqbar\*(Aq }}); \& $bulk\->create_docs({ foo => \*(Aqbar\*(Aq }) \& \& # Delete docs: \& $bulk\->delete({ id => 1}); \& $bulk\->add_action( delete => { id => 1 }); \& $bulk\->delete_ids(1,2,3) \& \& # Update docs: \& $bulk\->update({ id => 1, script => \*(Aq...\*(Aq }); \& $bulk\->add_action( update => { id => 1, script => \*(Aq...\*(Aq }); \& \& # Manual flush \& $bulk\->flush; \& \& # Reindex docs: \& my $bulk = $es\->bulk_helper( \& index => \*(Aqnew_index\*(Aq, \& verbose => 1 \& ); \& \& $bulk\->reindex( source => { index => \*(Aqold_index\*(Aq }); .Ve .SH "DESCRIPTION" .IX Header "DESCRIPTION" This module provides a wrapper for the \*(L"\fBbulk()\fR\*(R" in Search::Elasticsearch::Client::2_0::Direct method which makes it easier to run multiple create, index, update or delete actions in a single request. It also provides a simple interface for reindexing documents. .PP The Search::Elasticsearch::Client::2_0::Bulk module acts as a queue, buffering up actions until it reaches a maximum count of actions, or a maximum size of \s-1JSON\s0 request body, at which point it issues a \f(CW\*(C`bulk()\*(C'\fR request. .PP Once you have finished adding actions, call \*(L"\fBflush()\fR\*(R" to force the final \&\f(CW\*(C`bulk()\*(C'\fR request on the items left in the queue. .PP This class does Search::Elasticsearch::Client::2_0::Role::Bulk and Search::Elasticsearch::Role::Is_Sync. .SH "CREATING A NEW INSTANCE" .IX Header "CREATING A NEW INSTANCE" .ie n .SS """new()""" .el .SS "\f(CWnew()\fP" .IX Subsection "new()" .Vb 1 \& my $bulk = $es\->bulk_helper( \& \& index => \*(Aqdefault_index\*(Aq, # optional \& type => \*(Aqdefault_type\*(Aq, # optional \& %other_bulk_params # optional \& \& max_count => 1_000, # optional \& max_size => 1_000_000, # optional \& max_time => 5, # optional \& \& verbose => 0 | 1, # optional \& \& on_success => sub {...}, # optional \& on_error => sub {...}, # optional \& on_conflict => sub {...}, # optional \& \& \& ); .Ve .PP The \f(CW\*(C`new()\*(C'\fR method returns a new \f(CW$bulk\fR object. You must pass your Search::Elasticsearch client as the \f(CW\*(C`es\*(C'\fR argument. .PP The \f(CW\*(C`index\*(C'\fR and \f(CW\*(C`type\*(C'\fR parameters provide default values for \&\f(CW\*(C`index\*(C'\fR and \f(CW\*(C`type\*(C'\fR, which can be overridden in each action. You can also pass any other values which are accepted by the \fBbulk()\fR method. .PP See \*(L"\fBflush()\fR\*(R" for more information about the other parameters. .SH "FLUSHING THE BUFFER" .IX Header "FLUSHING THE BUFFER" .ie n .SS """flush()""" .el .SS "\f(CWflush()\fP" .IX Subsection "flush()" .Vb 1 \& $result = $bulk\->flush; .Ve .PP The \f(CW\*(C`flush()\*(C'\fR method sends all buffered actions to Elasticsearch using a \fBbulk()\fR request. .SS "Auto-flushing" .IX Subsection "Auto-flushing" An automatic \*(L"\fBflush()\fR\*(R" is triggered whenever the \f(CW\*(C`max_count\*(C'\fR, \f(CW\*(C`max_size\*(C'\fR, or \f(CW\*(C`max_time\*(C'\fR threshold is breached. This causes all actions in the buffer to be sent to Elasticsearch. .IP "\(bu" 4 \&\f(CW\*(C`max_count\*(C'\fR .Sp The maximum number of actions to allow before triggering a \*(L"\fBflush()\fR\*(R". This can be disabled by setting \f(CW\*(C`max_count\*(C'\fR to \f(CW0\fR. Defaults to \&\f(CW\*(C`1,000\*(C'\fR. .IP "\(bu" 4 \&\f(CW\*(C`max_size\*(C'\fR .Sp The maximum size of \s-1JSON\s0 request body to allow before triggering a \&\*(L"\fBflush()\fR\*(R". This can be disabled by setting \f(CW\*(C`max_size\*(C'\fR to \f(CW0\fR. Defaults to \f(CW\*(C`1_000,000\*(C'\fR bytes. .IP "\(bu" 4 \&\f(CW\*(C`max_time\*(C'\fR .Sp The maximum number of seconds to wait before triggering a flush. Defaults to \f(CW0\fR seconds, which means that it is disabled. \fBNote:\fR This timeout is only triggered when new items are added to the queue, not in the background. .SS "Errors when flushing" .IX Subsection "Errors when flushing" There are two types of error which can be thrown when \*(L"\fBflush()\fR\*(R" is called, either manually or automatically. .IP "\(bu" 4 Temporary Elasticsearch errors .Sp A \f(CW\*(C`Cxn\*(C'\fR error like a \f(CW\*(C`NoNodes\*(C'\fR error which indicates that your cluster is down. These errors do not clear the buffer, as they can be retried later on. .IP "\(bu" 4 Action errors .Sp Individual actions may fail. For instance, a \f(CW\*(C`create\*(C'\fR action will fail if a document with the same \f(CW\*(C`index\*(C'\fR, \f(CW\*(C`type\*(C'\fR and \f(CW\*(C`id\*(C'\fR already exists. These action errors are reported via callbacks. .SS "Using callbacks" .IX Subsection "Using callbacks" By default, any \fIAction errors\fR (see above) cause warnings to be written to \f(CW\*(C`STDERR\*(C'\fR. However, you can use the \f(CW\*(C`on_error\*(C'\fR, \f(CW\*(C`on_conflict\*(C'\fR and \f(CW\*(C`on_success\*(C'\fR callbacks for more fine-grained control. .PP All callbacks receive the following arguments: .ie n .IP "$action" 4 .el .IP "\f(CW$action\fR" 4 .IX Item "$action" The name of the action, ie \f(CW\*(C`index\*(C'\fR, \f(CW\*(C`create\*(C'\fR, \f(CW\*(C`update\*(C'\fR or \f(CW\*(C`delete\*(C'\fR. .ie n .IP "$response" 4 .el .IP "\f(CW$response\fR" 4 .IX Item "$response" The response that Elasticsearch returned for this action. .ie n .IP "$i" 4 .el .IP "\f(CW$i\fR" 4 .IX Item "$i" The index of the action, ie the first action in the flush request will have \f(CW$i\fR set to \f(CW0\fR, the second will have \f(CW$i\fR set to \f(CW1\fR etc. .PP \fI\f(CI\*(C`on_success\*(C'\fI\fR .IX Subsection "on_success" .PP .Vb 6 \& my $bulk = $es\->bulk_helper( \& on_success => sub { \& my ($action,$response,$i) = @_; \& # do something \& }, \& ); .Ve .PP The \f(CW\*(C`on_success\*(C'\fR callback is called for every action that has a successful response. .PP \fI\f(CI\*(C`on_conflict\*(C'\fI\fR .IX Subsection "on_conflict" .PP .Vb 6 \& my $bulk = $es\->bulk_helper( \& on_conflict => sub { \& my ($action,$response,$i,$version) = @_; \& # do something \& }, \& ); .Ve .PP The \f(CW\*(C`on_conflict\*(C'\fR callback is called for actions that have triggered a \f(CW\*(C`Conflict\*(C'\fR error, eg trying to \f(CW\*(C`create\*(C'\fR a document which already exists. The \f(CW$version\fR argument will contain the version number of the document currently stored in Elasticsearch (if found). .PP \fI\f(CI\*(C`on_error\*(C'\fI\fR .IX Subsection "on_error" .PP .Vb 6 \& my $bulk = $es\->bulk_helper( \& on_error => sub { \& my ($action,$response,$i) = @_; \& # do something \& }, \& ); .Ve .PP The \f(CW\*(C`on_error\*(C'\fR callback is called for any error (unless the \f(CW\*(C`on_conflict\*(C'\fR) callback has already been called). .SS "Disabling callbacks and autoflush" .IX Subsection "Disabling callbacks and autoflush" If you want to be in control of flushing, and you just want to receive the raw response that Elasticsearch sends instead of using callbacks, then you can do so as follows: .PP .Vb 5 \& my $bulk = $es\->bulk_helper( \& max_count => 0, \& max_size => 0, \& on_error => undef \& ); \& \& $bulk\->add_actions(....); \& $response = $bulk\->flush; .Ve .SH "CREATE, INDEX, UPDATE, DELETE" .IX Header "CREATE, INDEX, UPDATE, DELETE" .ie n .SS """add_action()""" .el .SS "\f(CWadd_action()\fP" .IX Subsection "add_action()" .Vb 6 \& $bulk\->add_action( \& create => { ...params... }, \& index => { ...params... }, \& update => { ...params... }, \& delete => { ...params... } \& ); .Ve .PP The \f(CW\*(C`add_action()\*(C'\fR method allows you to add multiple \f(CW\*(C`create\*(C'\fR, \f(CW\*(C`index\*(C'\fR, \&\f(CW\*(C`update\*(C'\fR and \f(CW\*(C`delete\*(C'\fR actions to the queue. The first value is the action type, and the second value is the parameters that describe that action. See the individual helper methods below for details. .PP \&\fBNote:\fR Parameters like \f(CW\*(C`index\*(C'\fR or \f(CW\*(C`type\*(C'\fR can be specified as \f(CW\*(C`index\*(C'\fR or as \&\f(CW\*(C`_index\*(C'\fR, so the following two lines are equivalent: .PP .Vb 2 \& index => { index => \*(Aqindex\*(Aq, type => \*(Aqtype\*(Aq, id => 1, source => {...}}, \& index => { _index => \*(Aqindex\*(Aq, _type => \*(Aqtype\*(Aq, _id => 1, _source => {...}}, .Ve .PP \&\fBNote:\fR The \f(CW\*(C`index\*(C'\fR and \f(CW\*(C`type\*(C'\fR parameters can be specified in the params for any action, but if not specified, will default to the \f(CW\*(C`index\*(C'\fR and \f(CW\*(C`type\*(C'\fR values specified in \*(L"\fBnew()\fR\*(R". These are required parameters: they must be specified either in \*(L"\fBnew()\fR\*(R" or in every action. .ie n .SS """create()""" .el .SS "\f(CWcreate()\fP" .IX Subsection "create()" .Vb 5 \& $bulk\->create( \& { index => \*(Aqcustom_index\*(Aq, source => { doc body }}, \& { type => \*(Aqcustom_type\*(Aq, id => 1, source => { doc body }}, \& ... \& ); .Ve .PP The \f(CW\*(C`create()\*(C'\fR helper method allows you to add multiple \f(CW\*(C`create\*(C'\fR actions. It accepts the same parameters as \*(L"\fBcreate()\fR\*(R" in Search::Elasticsearch::Client::2_0::Direct except that the document body should be passed as the \f(CW\*(C`source\*(C'\fR or \f(CW\*(C`_source\*(C'\fR parameter, instead of as \f(CW\*(C`body\*(C'\fR. .ie n .SS """create_docs()""" .el .SS "\f(CWcreate_docs()\fP" .IX Subsection "create_docs()" .Vb 5 \& $bulk\->create_docs( \& { doc body }, \& { doc body }, \& ... \& ); .Ve .PP The \f(CW\*(C`create_docs()\*(C'\fR helper is a shorter form of \*(L"\fBcreate()\fR\*(R" which can be used when you are using the default \f(CW\*(C`index\*(C'\fR and \f(CW\*(C`type\*(C'\fR as set in \*(L"\fBnew()\fR\*(R" and you are not specifying a custom \f(CW\*(C`id\*(C'\fR per document. In this case, you can just pass the individual document bodies. .ie n .SS """index()""" .el .SS "\f(CWindex()\fP" .IX Subsection "index()" .Vb 5 \& $bulk\->index( \& { index => \*(Aqcustom_index\*(Aq, source => { doc body }}, \& { type => \*(Aqcustom_type\*(Aq, id => 1, source => { doc body }}, \& ... \& ); .Ve .PP The \f(CW\*(C`index()\*(C'\fR helper method allows you to add multiple \f(CW\*(C`index\*(C'\fR actions. It accepts the same parameters as \*(L"\fBindex()\fR\*(R" in Search::Elasticsearch::Client::2_0::Direct except that the document body should be passed as the \f(CW\*(C`source\*(C'\fR or \f(CW\*(C`_source\*(C'\fR parameter, instead of as \f(CW\*(C`body\*(C'\fR. .ie n .SS """delete()""" .el .SS "\f(CWdelete()\fP" .IX Subsection "delete()" .Vb 5 \& $bulk\->delete( \& { index => \*(Aqcustom_index\*(Aq, id => 1}, \& { type => \*(Aqcustom_type\*(Aq, id => 2}, \& ... \& ); .Ve .PP The \f(CW\*(C`delete()\*(C'\fR helper method allows you to add multiple \f(CW\*(C`delete\*(C'\fR actions. It accepts the same parameters as \*(L"\fBdelete()\fR\*(R" in Search::Elasticsearch::Client::2_0::Direct. .ie n .SS """delete_ids()""" .el .SS "\f(CWdelete_ids()\fP" .IX Subsection "delete_ids()" .Vb 1 \& $bulk\->delete_ids(1,2,3...) .Ve .PP The \f(CW\*(C`delete_ids()\*(C'\fR helper method can be used when all of the documents you want to delete have the default \f(CW\*(C`index\*(C'\fR and \f(CW\*(C`type\*(C'\fR as set in \*(L"\fBnew()\fR\*(R". In this case, all you have to do is to pass in a list of IDs. .ie n .SS """update()""" .el .SS "\f(CWupdate()\fP" .IX Subsection "update()" .Vb 12 \& $bulk\->update( \& { id => 1, \& doc => { partial doc }, \& doc_as_upsert => 1 \& }, \& { id => 2, \& lang => \*(Aqmvel\*(Aq, \& script => { script } \& upsert => { upsert doc } \& }, \& ... \& ); .Ve .PP The \f(CW\*(C`update()\*(C'\fR helper method allows you to add multiple \f(CW\*(C`update\*(C'\fR actions. It accepts the same parameters as \*(L"\fBupdate()\fR\*(R" in Search::Elasticsearch::Client::2_0::Direct. An update can either use a \fIpartial doc\fR which gets merged with an existing doc (example 1 above), or can use a \f(CW\*(C`script\*(C'\fR to update an existing doc (example 2 above). More information on \f(CW\*(C`script\*(C'\fR can be found here: \&\*(L"\fBupdate()\fR\*(R" in Search::Elasticsearch::Client::2_0::Direct. .SH "REINDEXING DOCUMENTS" .IX Header "REINDEXING DOCUMENTS" A common use case for bulk indexing is to reindex a whole index when changing the type mappings or analysis chain. This typically combines bulk indexing with scrolled searches: the scrolled search pulls all of the data from the source index, and the bulk indexer indexes the data into the new index. .ie n .SS """reindex()""" .el .SS "\f(CWreindex()\fP" .IX Subsection "reindex()" .Vb 5 \& $bulk\->reindex( \& source => $source, # required \& transform => \e&transform, # optional \& version_type => \*(Aqexternal|internal\*(Aq, # optional \& ); .Ve .PP The \f(CW\*(C`reindex()\*(C'\fR method requires a \f(CW$source\fR parameter, which provides the source for the documents which are to be reindexed. .SS "Reindexing from another index" .IX Subsection "Reindexing from another index" If the \f(CW\*(C`source\*(C'\fR argument is a \s-1HASH\s0 ref, then the hash is passed to \&\*(L"\fBnew()\fR\*(R" in Search::Elasticsearch::Client::2_0::Scroll to create a new scrolled search. .PP .Vb 4 \& my $bulk = $es\->bulk_helper( \& index => \*(Aqnew_index\*(Aq, \& verbose => 1 \& ); \& \& $bulk\->reindex( \& source => { \& index => \*(Aqold_index\*(Aq, \& size => 500, # default \& search_type => \*(Aqscan\*(Aq # default \& } \& ); .Ve .PP If a default \f(CW\*(C`index\*(C'\fR or \f(CW\*(C`type\*(C'\fR has been specified in the call to \&\*(L"\fBnew()\fR\*(R", then it will replace the \f(CW\*(C`index\*(C'\fR and \f(CW\*(C`type\*(C'\fR values for the docs returned from the scrolled search. In the example above, all docs will be retrieved from \f(CW"old_index"\fR and will be bulk indexed into \f(CW"new_index"\fR. .SS "Reindexing from a generic source" .IX Subsection "Reindexing from a generic source" The \f(CW\*(C`source\*(C'\fR parameter also accepts a coderef or an anonymous sub, which should return one or more new documents every time it is executed. This allows you to pass any iterator, wrapped in an anonymous sub: .PP .Vb 1 \& my $iter = get_iterator_from_somewhere(); \& \& $bulk\->reindex( \& source => sub { $iter\->next } \& ); .Ve .SS "Transforming docs on the fly" .IX Subsection "Transforming docs on the fly" The \f(CW\*(C`transform\*(C'\fR parameter allows you to change documents on the fly, using a callback. The callback receives the document as the only argument, and should return the updated document, or \f(CW\*(C`undef\*(C'\fR if the document should not be indexed: .PP .Vb 4 \& $bulk\->reindex( \& source => { index => \*(Aqold_index\*(Aq }, \& transform => sub { \& my $doc = shift; \& \& # don\*(Aqt index doc marked as valid:false \& return undef unless $doc\->{_source}{valid}; \& \& # convert $tag to @tags \& $doc\->{_source}{tags} = [ delete $doc\->{_source}{tag}]; \& return $doc \& } \& ); .Ve .SS "Reindexing from another cluster" .IX Subsection "Reindexing from another cluster" By default, \*(L"\fBreindex()\fR\*(R" expects the source and destination indices to be in the same cluster. To pull data from one cluster and index it into another, you can use two separate \f(CW$es\fR objects: .PP .Vb 2 \& $es_target = Search::Elasticsearch\->new( nodes => \*(Aqlocalhost:9200\*(Aq ); \& $es_source = Search::Elasticsearch\->new( nodes => \*(Aqsearch1:9200\*(Aq ); \& \& my $bulk = $es_targert\->bulk_helper( \& verbose => 1 \& ) \& \-> reindex( \& source => { \& es => $es_source, \& index => \*(Aqmy_index\*(Aq \& } \& ); .Ve .SS "Parents and routing" .IX Subsection "Parents and routing" If you are using parent-child relationships or custom \f(CW\*(C`routing\*(C'\fR values, and you want to preserve these when you reindex your documents, then you will need to request these values specifically, as follows: .PP .Vb 6 \& $bulk\->reindex( \& source => { \& index => \*(Aqold_index\*(Aq, \& fields => [\*(Aq_source\*(Aq,\*(Aq_parent\*(Aq,\*(Aq_routing\*(Aq] \& } \& ); .Ve .SS "Working with version numbers" .IX Subsection "Working with version numbers" Every document in Elasticsearch has a current \f(CW\*(C`version\*(C'\fR number, which is used for optimistic concurrency control , that is, to ensure that you don't overwrite changes that have been made by another process. .PP All \s-1CRUD\s0 operations accept a \f(CW\*(C`version\*(C'\fR parameter and a \f(CW\*(C`version_type\*(C'\fR parameter which tells Elasticsearch that the change should only be made if the current document corresponds to these parameters. The \&\f(CW\*(C`version_type\*(C'\fR parameter can have the following values: .IP "\(bu" 4 \&\f(CW\*(C`internal\*(C'\fR .Sp Use Elasticsearch version numbers. Documents are only changed if the document in Elasticsearch has the \fBsame\fR \f(CW\*(C`version\*(C'\fR number that is specified in the \s-1CRUD\s0 operation. After the change, the new version number is \f(CW\*(C`version+1\*(C'\fR. .IP "\(bu" 4 \&\f(CW\*(C`external\*(C'\fR .Sp Use an external versioning system, such as timestamps or version numbers from an external database. Documents are only changed if the document in Elasticsearch has a \fBlower\fR \f(CW\*(C`version\*(C'\fR number than the one specified in the \s-1CRUD\s0 operation. After the change, the new version number is \f(CW\*(C`version\*(C'\fR. .PP If you would like to reindex documents from one index to another, preserving the \f(CW\*(C`version\*(C'\fR numbers from the original index, then you need the following: .PP .Vb 7 \& $bulk\->reindex( \& source => { \& index => \*(Aqold_index\*(Aq, \& version => 1, # retrieve version numbers in search \& }, \& version_type => \*(Aqexternal\*(Aq # use these "external" version numbers \& ); .Ve .SH "AUTHOR" .IX Header "AUTHOR" Clinton Gormley .SH "COPYRIGHT AND LICENSE" .IX Header "COPYRIGHT AND LICENSE" This software is Copyright (c) 2017 by Elasticsearch \s-1BV.\s0 .PP This is free software, licensed under: .PP .Vb 1 \& The Apache License, Version 2.0, January 2004 .Ve