.\" Automatically generated by Pod::Man 4.14 (Pod::Simple 3.40) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. \*(C+ will .\" give a nicer C++. Capital omega is used to do unbreakable dashes and .\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff, .\" nothing in troff, for use with C<>. .tr \(*W- .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' . ds C` . ds C' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is >0, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .\" .\" Avoid warning from groff about undefined register 'F'. .de IX .. .nr rF 0 .if \n(.g .if rF .nr rF 1 .if (\n(rF:(\n(.g==0)) \{\ . if \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . if !\nF==2 \{\ . nr % 0 . nr F 2 . \} . \} .\} .rr rF .\" ======================================================================== .\" .IX Title "Message::Passing::ZeroMQ 3pm" .TH Message::Passing::ZeroMQ 3pm "2021-12-22" "perl v5.32.1" "User Contributed Perl Documentation" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh .SH "NAME" Message::Passing::ZeroMQ \- input and output messages to ZeroMQ. .SH "SYNOPSIS" .IX Header "SYNOPSIS" .Vb 3 \& # Terminal 1: \& $ message\-passing \-\-input STDIN \-\-output ZeroMQ \-\-output_options \*(Aq{"connect":"tcp://127.0.0.1:5552"}\*(Aq \& {"data":{"some":"data"},"@metadata":"value"} \& \& # Terminal 2: \& $ message\-passing \-\-output STDOUT \-\-input ZeroMQ \-\-input_options \*(Aq{"socket_bind":"tcp://*:5552"}\*(Aq \& {"data":{"some":"data"},"@metadata":"value"} .Ve .SH "DESCRIPTION" .IX Header "DESCRIPTION" A ZeroMQ transport for Message::Passing. .PP Designed for use as a log transport and aggregation mechanism for perl applications, allowing you to aggregate structured and non-structured log messages across the network in a non-blocking manor. .PP Clients (I.e. users of the Message::Passing::Output::ZeroMQ class) connect to a server (I.e. a user of the Message::Passing::Input::ZeroMQ class) via ZeroMQ's pub/sub sockets. These are setup to be lossy and non-blocking, meaning that if the log-receiver process is down or slow, then the application will queue a small (and configurable) amount of logs on it's side, and after that log messages will be dropped. .PP Whilst throwing away log messages isn't a good thing to do, or something that you want to happen regularly, in many (especially web application) contexts, network logging being a single point of failure is not acceptable from a reliability and graceful degradation standpoint. .PP The application grinding to a halt as a non-essential centralised resource is unavailable (e.g. the log aggregation server) is significantly less acceptable than the loss of non-essential logging data. .SH "HOW TO USE" .IX Header "HOW TO USE" In your application emitting messages, you can either use Message::Passing::Output::ZeroMQ directly, or you can use it via Log::Dispatch::Message::Passing. .PP .Vb 4 \& use Log::Dispatch; \& use Log::Dispatch::Message::Passing; \& use Message::Passing::Output::ZeroMQ; \& use Message::Passing::Filter::Encode::JSON; \& \& my $log = Log::Dispatch\->new; \& \& $log\->add(Log::Dispatch::Message::Passing\->new( \& name => \*(Aqmyapp_aggregate_log\*(Aq, \& min_level => \*(Aqdebug\*(Aq, \& output => Message::Passing::Filter::Encode::JSON\->new( \& output_to => Message::Passing::Output::ZeroMQ\->new( \& connect => \*(Aqtcp://192.168.0.1:5558\*(Aq, \& ) \& ), \& )); \& \& $log\->warn($_) for qw/ foo bar baz /; .Ve .PP On your log aggregation server, just run the message-passing utility: .PP .Vb 2 \& message\-passing \-\-input ZeroMQ \-\-input_options \*(Aq{"socket_bind":"tcp://*:5222"}\*(Aq \e \& \-\-output File \-\-output_options \*(Aq{"filename":"/tmp/my_test.log"}\*(Aq .Ve .SH "SOCKET TYPES" .IX Header "SOCKET TYPES" ZeroMQ supports multiple socket types, the only ones used in Message::Passing::ZeroMQ are: .SS "\s-1PUB/SUB\s0" .IX Subsection "PUB/SUB" Used for general message distribution \- you can have either multiple producers (\s-1PUB\s0) which connect to one consumer (\s-1SUB\s0), or multiple consumers (\s-1SUB\s0) which connect to one producer (\s-1PUB\s0). .PP All consumers will get a copy of every message. .PP In Message::Passing terms, Message::Passing::Input::ZeroMQ is for \s-1SUB\s0 sockets, and Message::Passing::Output::ZeroMQ is for \s-1PUB\s0 sockets. .SS "\s-1PUSH/PULL\s0" .IX Subsection "PUSH/PULL" Used for message distribution. A sever (\s-1PUSH\s0) distributes messages between a number of connecting clients (\s-1PULL\s0) .PP In Message::Passing terms, Message::Passing::Input::ZeroMQ is for \s-1PULL\s0 sockets, and Message::Passing::Output::ZeroMQ is for \s-1PUSH\s0 sockets. .SH "CONNECTION DIRECTION" .IX Header "CONNECTION DIRECTION" Note that in ZeroMQ, the connection direction and the direction of message flow can be entirely opposite. I.e. a client can connect to a server and send messages to it, or receive messages from it (depending on the direction of the socket types). .SH "CONNECTION ATTRIBUTES" .IX Header "CONNECTION ATTRIBUTES" Both Message::Passing::Input::ZeroMQ and Message::Passing::Output::ZeroMQ support either binding a server or connecting to a remote host, due to the fact that ZeroMQ connections can be in any direction, as noted above. .PP Therefore, each input or output should have one (but not both!) of the following attributes: .SS "connect" .IX Subsection "connect" Connects to a remote server, e.g. \f(CW\*(C`tcp://192.168.0.1:5222\*(C'\fR .SS "socket_bind" .IX Subsection "socket_bind" Binds a server and waits for connections from clients, e.g. \f(CW\*(C`tcp://*:5222\*(C'\fR .SS "socket_type" .IX Subsection "socket_type" This defaults to \f(CW\*(C`SUB\*(C'\fR for Message::Passing::Input::ZeroMQ and \f(CW\*(C`PUB\*(C'\fR for Message::Passing::Output::ZeroMQ, however you can override it to \f(CW\*(C`PUSH\*(C'\fR/\f(CW\*(C`PULL\*(C'\fR as appropriate for your use case if desired. .SH "MORE COMPLEX EXAMPLES" .IX Header "MORE COMPLEX EXAMPLES" With this in mind, we can easily create a system which aggregates messages from multiple publishers, and passes them out (in a round-robin fashion) to a pool of workers. .PP .Vb 3 \& # The message distributor: \& message\-passing \-\-input ZeroMQ \-\-input_options \*(Aq{"socket_bind":"tcp://*:5222"}\*(Aq \e \& \-\-output ZeroMQ \-\-output_options \*(Aq{"socket_bind":"tcp://*:5223","socket_type":"PUSH"}\*(Aq \& \& # Workers \& { \& package MyApp::MessageWorker; \& use Moo; \& \& with \*(AqMessage::Passing::Role::Filter\*(Aq; \& \& sub filter { \& my ($self, $message) = @_; \& # .... process the message in any way you want here \& return undef; # Do not output the message.. \& } \& } \& \& message\-passing \-\-input ZeroMQ \-\-input_options \*(Aq{"connect":"tcp://127.0.0.1:5223","socket_type":"PULL"}\*(Aq \& \-\-filter \*(Aq+MyApp::MessageWorker\*(Aq \& \-\-output STDOUT .Ve .PP You log messages into the distributor as per the above simple example, and you can run multiple worker processes.. .PP Less trivial setups could/would emit messages on error, or maybe re-emit the incoming message after transforming it in some way. .SH "SEE ALSO" .IX Header "SEE ALSO" For more detailed information about ZeroMQ and how it works, please consult the ZeroMQ guide and the other links below: .IP "Message::Passing::Output::ZeroMQ" 4 .IX Item "Message::Passing::Output::ZeroMQ" .PD 0 .IP "Message::Passing::Input::ZeroMQ" 4 .IX Item "Message::Passing::Input::ZeroMQ" .IP "Message::Passing" 4 .IX Item "Message::Passing" .IP "\s-1ZMQ::FFI\s0" 4 .IX Item "ZMQ::FFI" .IP "" 4 .IX Item "" .IP "" 4 .IX Item "" .PD .SH "AUTHOR" .IX Header "AUTHOR" Tomas (t0m) Doran .SH "SPONSORSHIP" .IX Header "SPONSORSHIP" This module exists due to the wonderful people at Suretec Systems Ltd. who sponsored its development for its VoIP division called SureVoIP for use with the SureVoIP \s-1API\s0 \- .SH "COPYRIGHT" .IX Header "COPYRIGHT" Copyright Suretec Systems 2012. .SH "LICENSE" .IX Header "LICENSE" \&\s-1GNU\s0 Affero General Public License, Version 3 .PP If you feel this is too restrictive to be able to use this software, please talk to us as we'd be willing to consider re-licensing under less restrictive terms.