.\" Automatically generated by Pod::Man 4.14 (Pod::Simple 3.43) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. \*(C+ will .\" give a nicer C++. Capital omega is used to do unbreakable dashes and .\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff, .\" nothing in troff, for use with C<>. .tr \(*W- .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' . ds C` . ds C' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is >0, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .\" .\" Avoid warning from groff about undefined register 'F'. .de IX .. .nr rF 0 .if \n(.g .if rF .nr rF 1 .if (\n(rF:(\n(.g==0)) \{\ . if \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . if !\nF==2 \{\ . nr % 0 . nr F 2 . \} . \} .\} .rr rF .\" ======================================================================== .\" .IX Title "DATAFLOW 1p" .TH DATAFLOW 1p "2023-06-17" "perl v5.36.0" "User Contributed Perl Documentation" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh .SH "NAME" PDL::Dataflow \-\- description of the dataflow implementation and philosophy .SH "SYNOPSIS" .IX Header "SYNOPSIS" .Vb 5 \& pdl> $x = zeroes(10); \& pdl> $y = $x\->slice("2:4:2"); \& pdl> $y ++; \& pdl> print $x; \& [0 0 1 0 1 0 0 0 0 0] .Ve .SH "DESCRIPTION" .IX Header "DESCRIPTION" As of 2.079, this is now a description of the current implementation, together with some design thoughts from its original author, Tuomas Lukka. .PP Two-directional dataflow (which implements \f(CW\*(C`\->slice()\*(C'\fR etc.) is fully functional, as shown in the \s-1SYNOPSIS.\s0 One-way is implemented, but with restrictions. .SH "TWO-WAY" .IX Header "TWO-WAY" Just about any function which returns some subset of the values in some ndarray will make a binding. \f(CW$y\fR has become effectively a window to some sub-elements of \f(CW$x\fR. You can also define your own routines that do different types of subsets. If you don't want \f(CW$y\fR to be a window to \f(CW$x\fR, you must do .PP .Vb 1 \& $y = $x\->slice("some parts")\->sever; .Ve .PP The \f(CW\*(C`sever\*(C'\fR destroys the \f(CW\*(C`slice\*(C'\fR transform, thereby turning off all dataflow between the two ndarrays. .SS "Type conversions" .IX Subsection "Type conversions" This works, thanks to a two-way flowing transform that implements type-conversions, particularly for supplied outputs of the \*(L"wrong\*(R" type for the given transform: .PP .Vb 7 \& pdl> $a_bad = pdl double, \*(Aq[1 BAD 3]\*(Aq; \& pdl> $b_float = zeroes float, 3; \& pdl> $a_bad\->assgn($b_float); # could be written as $b_float .= $a_bad \& pdl> p $b_float\->badflag; \& 1 \& pdl> p $b_float; \& [1 BAD 3] .Ve .SH "ONE-WAY" .IX Header "ONE-WAY" You need to explicitly turn on one-way dataflow on an ndarray to activate it for non-flowing operations, so .PP .Vb 8 \& pdl> $x = pdl 2,3,4; \& pdl> $x\->doflow; \& pdl> $y = $x * 2; \& pdl> print $y; \& [4 6 8] \& pdl> $x\->set(0,5); \& pdl> print $y; \& [10 6 8] .Ve .PP It is not possible to turn on backwards dataflow (such as is used by \&\f(CW\*(C`slice\*(C'\fR\-type operations), because there is no general way for \s-1PDL\s0 (or maths, in fact) to know how to reverse most operations \- consider \&\f(CW\*(C`$z = $x * $y\*(C'\fR, then adding one to \f(CW$z\fR. .PP Consider the following code: .PP .Vb 7 \& $u = sequence(3,3); $u\->doflow; \& $v = ones(3,3); $v\->doflow; \& $w = $u + $v; $w\->doflow; # must turn on for each \& $y = $w + 1; $y\->doflow; \& $x = $w\->diagonal(0,1); \& $x += 50; \& $z = $w + 2; .Ve .PP What do \f(CW$y\fR and \f(CW$z\fR contain now? .PP .Vb 12 \& pdl> p $y \& [ \& [52 3 4] \& [ 5 56 7] \& [ 8 9 60] \& ] \& pdl> p $z \& [ \& [53 4 5] \& [ 6 57 8] \& [ 9 10 61] \& ] .Ve .PP What about when \f(CW$u\fR is changed and a recalculation is triggered? A problem arises, in that \s-1PDL\s0 currently (as of 2.079) disallows (see \fIpdlapi.c\fR), for normal transforms, output ndarrays with flow, or output ndarrays with any parent with dataflow. So \f(CW\*(C`$u++\*(C'\fR throws an exception. But it is currently possible to use \f(CW\*(C`set\*(C'\fR, which is a sort of micro-transform that calls (in the C \s-1API\s0) \f(CW\*(C`PDL.set\*(C'\fR to mutate the data, then \&\f(CW\*(C`PDL.changed\*(C'\fR to trigger flow updates: .PP .Vb 7 \& pdl> $u\->set(1,1,90) \& pdl> p $y \& [ \& [ 2 3 4] \& [ 5 92 7] \& [ 8 9 10] \& ] .Ve .PP You'll notice that while the setting of \f(CW\*(C`1,1\*(C'\fR (the middle) of \f(CW$u\fR updated \&\f(CW$y\fR, the changes to \f(CW$y\fR that resulted from adding 50 to the diagonal (via \f(CW$x\fR, and two-way flow) got lost. This is one-way flow. .SH "LAZY EVALUATION" .IX Header "LAZY EVALUATION" In one-way flow context like the above, with: .PP .Vb 1 \& pdl> $y = $x * 2; .Ve .PP nothing will have been calculated at this point. Even the memory for the contents of \f(CW$y\fR has not been allocated. Only the command .PP .Vb 1 \& pdl> print $y .Ve .PP will actually cause \f(CW$y\fR to be calculated. This is important to bear in mind when doing performance measurements and benchmarks as well as when tracking errors. .PP There is an explanation for this behaviour: it may save cycles but more importantly, imagine the following: .PP .Vb 6 \& pdl> $x = pdl 2,3,4; $x\->doflow; \& pdl> $y = pdl 5,6,7; $y\->doflow; \& pdl> $c = $x + $y; \& pdl> $x\->setdims([4]); \& pdl> $y\->setdims([4]); \& pdl> print $c; .Ve .PP Now, if \f(CW$c\fR were evaluated between the two resizes, an error condition of incompatible sizes would occur. .PP What happens in the current version is that resizing \f(CW$x\fR raises a flag in \f(CW$c:\fR \*(L"\s-1PDL_PARENTDIMSCHANGED\*(R"\s0 and \f(CW$y\fR just raises the same flag again. When \f(CW$c\fR is next evaluated, the flags are checked and it is found that a recalculation is needed. .PP Of course, lazy evaluation can sometimes make debugging more painful because errors may occur somewhere where you'd not expect them. .SH "FAMILIES" .IX Header "FAMILIES" This is one of the more intricate concepts of dataflow. In order to make dataflow work like you'd expect, a rather strange concept must be introduced: families. Let us make a diagram of the one-way flow example \- it uses a hypergraph because the transforms (with \f(CW\*(C`+\*(C'\fR) are connectors between ndarrays (with \f(CW\*(C`*\*(C'\fR): .PP .Vb 10 \& u* *v \& \e / \& +(plus) \& | \& 1* *w \& \e /|\e \& \e / | \e \& (plus)+ | +(diagonal) \& | | | \& y* | *x \& | \& | *1 \& |/ \& +(plus) \& | \& z* .Ve .PP This is what \s-1PDL\s0 actually has in memory after the first three lines. When \f(CW$x\fR is changed, \f(CW$w\fR changes due to \f(CW\*(C`diagonal\*(C'\fR being a two-way operation. .PP If you want flow from \f(CW$w\fR, you opt in using \f(CW\*(C`$w\->doflow\*(C'\fR (as shown in this scenario). If you didn't, then don't enable it. If you have it but want to stop it, call \f(CW\*(C`$ndarray\->sever\*(C'\fR. That will destroy the ndarray's \f(CW\*(C`trans_parent\*(C'\fR (here, a node marked with \f(CW\*(C`+\*(C'\fR), and as you can visually tell, will stop changes flowing thereafter. If you want to leave the flow operating, but get a copy of the ndarray at that point, use \f(CW\*(C`$ndarray\->copy\*(C'\fR \- it will have the same data at that moment, but have no flow relationships. .SH "EVENTS" .IX Header "EVENTS" There is the start of a mechanism to bind events onto changed data, intended to allow this to work: .PP .Vb 10 \& pdl> $x = pdl 2,3,4 \& pdl> $y = $x + 1; \& pdl> $c = $y * 2; \& pdl> $c\->bind( sub { print "A now: $x, C now: $c\en" } ) \& pdl> PDL::dowhenidle(); \& A now: [2,3,4], C now: [6 8 10] \& pdl> $x\->set(0,1); \& pdl> $x\->set(1,1); \& pdl> PDL::dowhenidle(); \& A now: [1,1,4], C now: [4 4 10] .Ve .PP This hooks into \s-1PDL\s0's \f(CW\*(C`magic\*(C'\fR which resembles Perl's, but does not currently operate. .PP There would be many kinds of uses for this feature: self-updating charts, for instance. It is not yet fully clear whether it would be most useful to queue up changes (useful for doing asynchronously, e.g. when idle), or to activate things immediately. .PP In the 2022 era of both GPUs and multiple cores, it is a pity that Perl's dominant model remains single-threaded on \s-1CPU,\s0 but \s-1PDL\s0 can use multi-cores for \s-1CPU\s0 processing (albeit controlled in a single-threaded style) \- see PDL::ParallelCPU. It is planned that \s-1PDL\s0 will gain the ability to use GPUs, and there might be a way to hook that up albeit probably with an event loop to \*(L"subscribe\*(R" to \s-1GPU\s0 events. .SH "TRANSFORMATIONS" .IX Header "TRANSFORMATIONS" \&\s-1PDL\s0 implements nearly everything (except for \s-1XS\s0 oddities like \&\f(CW\*(C`set\*(C'\fR) using transforms which connect ndarrays. This includes data transformations like addition, \*(L"slicing\*(R" to access/operate on subsets, and data-type conversions (which have two-way dataflow, see \&\*(L"Type conversions\*(R"). .PP This does not currently include a resizing transformation, and \f(CW\*(C`setdims\*(C'\fR mutates its input. This is intended to change. .SH "AUTHOR" .IX Header "AUTHOR" Copyright(C) 1997 Tuomas J. Lukka (lukka@fas.harvard.edu). Same terms as the rest of \s-1PDL.\s0