.\" Automatically generated by Pod::Man 4.14 (Pod::Simple 3.40) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. \*(C+ will .\" give a nicer C++. Capital omega is used to do unbreakable dashes and .\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff, .\" nothing in troff, for use with C<>. .tr \(*W- .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' . ds C` . ds C' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is >0, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .\" .\" Avoid warning from groff about undefined register 'F'. .de IX .. .nr rF 0 .if \n(.g .if rF .nr rF 1 .if (\n(rF:(\n(.g==0)) \{\ . if \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . if !\nF==2 \{\ . nr % 0 . nr F 2 . \} . \} .\} .rr rF .\" ======================================================================== .\" .IX Title "Data::Stream::Bulk 3pm" .TH Data::Stream::Bulk 3pm "2021-01-01" "perl v5.32.0" "User Contributed Perl Documentation" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh .SH "NAME" Data::Stream::Bulk \- N at a time iteration API .SH "VERSION" .IX Header "VERSION" version 0.11 .SH "SYNOPSIS" .IX Header "SYNOPSIS" .Vb 2 \& # get a bulk stream from somewere \& my $s = Data::Stream::Bulk::Foo\->new( ... ); \& \& # can be used like this: \& until ( $s\->is_done ) { \& foreach my $item ( $s\->items ) { \& process($item); \& } \& } \& \& # or like this: \& while( my $block = $s\->next ) { \& foreach my $item ( @$block ) { \& process($item); \& } \& } .Ve .SH "DESCRIPTION" .IX Header "DESCRIPTION" This module tries to find middle ground between one at a time and all at once processing of data sets. .PP The purpose of this module is to avoid the overhead of implementing an iterative api when this isn't necessary, without breaking forward compatibility in case that becomes necessary later on. .PP The \s-1API\s0 optimizes for when a data set typically fits in memory and is returned as an array, but the consumer cannot assume that the data set is bounded. .PP The \s-1API\s0 is destructive in order to minimize the chance that resultsets are leaked due to improper usage. .SH "API" .IX Header "API" .SS "Required Methods" .IX Subsection "Required Methods" The \s-1API\s0 requires two methods to be implemented: .IP "is_done" 4 .IX Item "is_done" Should return true if the stream is exhausted. .Sp As long as this method returns a false value (not done) \f(CW\*(C`next\*(C'\fR could potentially return another block. .IP "next" 4 .IX Item "next" Returns the next block. .Sp Note that \f(CW\*(C`next\*(C'\fR is not guaranteed to return an array reference, even if \&\f(CW\*(C`is_done\*(C'\fR returned false prior to calling it. .SS "Convenience Methods" .IX Subsection "Convenience Methods" .IP "items" 4 .IX Item "items" This method calls \f(CW\*(C`next\*(C'\fR and dereferences the result if there are pending items. .IP "all" 4 .IX Item "all" Force evaluation of the entire resultset. .Sp Note that for large data sets this might cause swap thrashing of various other undesired effects. Use with caution. .ie n .IP "cat @streams" 4 .el .IP "cat \f(CW@streams\fR" 4 .IX Item "cat @streams" Concatenates this stream with \f(CW@streams\fR, returning a single stream. .ie n .IP "list_cat @tail" 4 .el .IP "list_cat \f(CW@tail\fR" 4 .IX Item "list_cat @tail" Returns a possibly cleaned up list of streams. .Sp Used by \f(CW\*(C`cat\*(C'\fR. .Sp Overridden by Data::Stream::Bulk::Array, Data::Stream::Bulk::Cat and Data::Stream::Bulk::Nil to implement some simple short circuiting. .ie n .IP "filter $filter" 4 .el .IP "filter \f(CW$filter\fR" 4 .IX Item "filter $filter" Applies a per-block block filter to the stream. .Sp Returns a possibly new stream with the filtering layered. .Sp \&\f(CW$filter\fR is invoked once per block and should return an array reference to the filtered block. .ie n .IP "chunk $chunk_size" 4 .el .IP "chunk \f(CW$chunk_size\fR" 4 .IX Item "chunk $chunk_size" Chunks the input stream so that each block returned by \f(CW\*(C`next\*(C'\fR will have at least \f(CW$chunk_size\fR items. .IP "loaded" 4 .IX Item "loaded" Should be overridden to return true if all the items are already realized (e.g. in the case of Data::Stream::Bulk::Array). .Sp Returns false by default. .Sp When true calling \f(CW\*(C`all\*(C'\fR is supposed to be safe (memory usage should be in the same order of magnitude as stream's own usage). .Sp This is typically useful when tranforming an array is easier than transorming a stream (e.g. optional duplicate filtering). .SH "CLASSES" .IX Header "CLASSES" .IP "Data::Stream::Bulk::Array" 4 .IX Item "Data::Stream::Bulk::Array" This class is not a stream at all, but just one block. When the data set easily fits in memory this class can be used, while retaining forward compatibility with larger data sets. .IP "Data::Stream::Bulk::Callback" 4 .IX Item "Data::Stream::Bulk::Callback" Callback driven iteration. .IP "Data::Stream::Bulk::Chunked" 4 .IX Item "Data::Stream::Bulk::Chunked" Wrapper to return larger blocks from an existing stream. .IP "Data::Stream::Bulk::DBI" 4 .IX Item "Data::Stream::Bulk::DBI" Bulk fetching of data from \s-1DBI\s0 statement handles. .IP "Data::Stream::Bulk::DBIC" 4 .IX Item "Data::Stream::Bulk::DBIC" DBIx::Class::ResultSet iteration. .IP "Data::Stream::Bulk::FileHandle" 4 .IX Item "Data::Stream::Bulk::FileHandle" Iterates over lines in a text file. .IP "Data::Stream::Bulk::Nil" 4 .IX Item "Data::Stream::Bulk::Nil" An empty result set. .IP "Data::Stream::Bulk::Cat" 4 .IX Item "Data::Stream::Bulk::Cat" A concatenation of several streams. .IP "Data::Stream::Bulk::Filter" 4 .IX Item "Data::Stream::Bulk::Filter" A filter wrapping a stream. .SH "SEE ALSO" .IX Header "SEE ALSO" HOP::Stream, Iterator, Class::Iterator etc for one by one iteration .PP \&\s-1DBI\s0, DBIx::Class::ResultSet .PP POE::Filter .PP Data::Page .PP Parallel::Iterator .PP , \s-1LISP,\s0 and all that other kool aid .SH "TODO" .IX Header "TODO" .IP "Sorted streams" 4 .IX Item "Sorted streams" Add a hint for sorted streams (like \f(CW\*(C`loaded\*(C'\fR but as an attribute in the base role). .Sp Introduce a \f(CW\*(C`merge\*(C'\fR operation for merging of sorted streams. .Sp Optimize \f(CW\*(C`unique\*(C'\fR to make use of sorting hints for constant space uniquing. .IP "More utility functions" 4 .IX Item "More utility functions" To assist in proccessing and creating streams. .IP "Coercion tables" 4 .IX Item "Coercion tables" Moose::Util::TypeConstraints .SH "VERSION CONTROL" .IX Header "VERSION CONTROL" This module is maintained using git. You can get the latest version from . .SH "AUTHOR" .IX Header "AUTHOR" Yuval Kogman .SH "COPYRIGHT AND LICENSE" .IX Header "COPYRIGHT AND LICENSE" This software is copyright (c) 2012 by Yuval Kogman. .PP This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.