.TH "allpairs_master" 1 "" "CCTools 7.1.2 FINAL" "Cooperative Computing Tools" .SH NAME .LP \fBallpairs_master\fP - executes All-Pairs workflow in parallel on distributed systems .SH SYNOPSIS .LP \FC\fBallparis_master [options] \fP\FT .SH DESCRIPTION .LP \fBallpairs_master\fP computes the Cartesian product of two sets (\fB\fP and \fB\fP), generating a matrix where each cell M[i,j] contains the output of the function F (\fB\fP) on objects A[i] (an item in \fB\fP) and B[j] (an item in \fB\fP). The resulting matrix is displayed on the standard output, one comparison result per line along with the associated X and Y indices. .PP \fBallpairs_master\fP uses the Work Queue system to distribute tasks among processors. Each processor utilizes the \fBallpairs_multicore(1)\fP program to execute the tasks in parallel if multiple cores are present. After starting \fBallpairs_master\fP, you must start a number of \fBwork_queue_worker(1)\fP processes on remote machines. The workers will then connect back to the master process and begin executing tasks. .SH OPTIONS .LP .LP .TP \fB-p\fP, \fB-\-port\fP=\fI\fP . The port that the master will be listening on. .TP \fB-e\fP, \fB-\-extra-args\fP=\fI\fP . Extra arguments to pass to the comparison function. .TP \fB-f\fP, \fB-\-input-file\fP=\fI\fP . Extra input file needed by the comparison function. (may be given multiple times) .TP \fB-o\fP, \fB-\-debug-file\fP=\fI\fP . Write debugging output to this file. By default, debugging is sent to stderr (":stderr"). You may specify logs be sent to stdout (":stdout"), to the system syslog (":syslog"), or to the systemd journal (":journal"). .TP \fB-O\fP, \fB-\---output-file\fP=\fI\fP . Write task output to this file (default to standard output) .TP \fB-t\fP, \fB-\-estimated-time\fP=\fI\fP . Estimated time to run one comparison. (default chosen at runtime) .TP \fB-x\fP, \fB-\-width\fP=\fI\fP . Width of one work unit, in items to compare. (default chosen at runtime) .TP \fB-y\fP, \fB-\-height\fP=\fI\fP . Height of one work unit, in items to compare. (default chosen at runtime) .TP \fB-N\fP, \fB-\-project-name\fP=\fI\fP . Report the master information to a catalog server with the project name - .TP \fB-P\fP, \fB-\-priority\fP=\fI\fP . Priority. Higher the value, higher the priority. .TP \fB-d\fP, \fB-\-debug\fP=\fI\fP . Enable debugging for this subsystem. (Try -d all to start.) .TP .B \ -v, --version . Show program version. .TP .BI \ -h, --help \ <> . Display this message. .TP \fB-Z\fP, \fB-\-port-file\fP=\fI\fP . Select port at random and write it to this file. (default is disabled) .TP .BI \ --work-queue-preferred-connection \ . Indicate preferred connection. Chose one of by_ip or by_hostname. (default is by_ip) .SH EXIT STATUS .LP On success, returns zero. On failure, returns non-zero. .SH EXAMPLES .LP Let's suppose you have a whole lot of files that you want to compare all to each other, named \FCa\FT, \FCb\FT, \FCc\FT, and so on. Suppose that you also have a program named \fBcompareit\fP that when invoked as \FCcompareit a b\FT will compare files \FCa\FT and \FCb\FT and produce some output summarizing the difference between the two, like this: .fam C .nf .nh .IP "" 8 a b are 45 percent similar .fi .hy .fam .P To use the allpairs framework, create a file called \FCset.list\FT that lists each of your files, one per line: .fam C .nf .nh .IP "" 8 a b c ... .fi .hy .fam .P Because \fBallpairs_master\fP utilizes \fBallpairs_multicore(1)\fP, so please make sure \fBallpairs_multicore(1)\fP is in your PATH before you proceed.To run a All-Pairs workflow sequentially, start a single \fBwork_queue_worker(1)\fP process in the background. Then, invoke \fBallpairs_master\fP. .fam C .nf .nh .IP "" 8 % work_queue_worker localhost 9123 & % allpairs_master set.list set.list compareit .fi .hy .fam .P The framework will carry out all possible comparisons of the objects, and print the results one by one (note that the first two columns are X and Y indices in the resulting matrix): .fam C .nf .nh .IP "" 8 1 1 a a are 100 percent similar 1 2 a b are 45 percent similar 1 3 a c are 37 percent similar ... .fi .hy .fam .P To speed up the process, run more \fBwork_queue_worker(1)\fP processes on other machines, or use \fBcondor_submit_workers(1)\fP or \fBsge_submit_workers(1)\fP to start hundreds of workers in your local batch system. .PP The following is an example of adding more workers to execute a All-Pairs workflow. Suppose your \fBallpairs_master\fP is running on a machine named barney.nd.edu. If you have access to login to other machines, you could simply start worker processes on each one, like this: .fam C .nf .nh .IP "" 8 % work_queue_worker barney.nd.edu 9123 .fi .hy .fam .P If you have access to a batch system like Condor, you can submit multiple workers at once: .fam C .nf .nh .IP "" 8 % condor_submit_workers barney.nd.edu 9123 10 Submitting job(s).......... Logging submit event(s).......... 10 job(s) submitted to cluster 298. .fi .hy .fam .P .SH COPYRIGHT .LP The Cooperative Computing Tools are Copyright (C) 2005-2019 The University of Notre Dame. This software is distributed under the GNU General Public License. See the file COPYING for details. .SH SEE ALSO .LP .IP \(bu 4 The Cooperative Computing Tools (\fB"http://ccl.cse.nd.edu/software/manuals"\fP) .IP \(bu 4 All-Pairs User Manual (\fB"http://ccl.cse.nd.edu/software/manuals/allpairs.html"\fP) .IP \(bu 4 Work Queue User Manual (\fB"http://ccl.cse.nd.edu/software/manuals/workqueue.html"\fP) .IP \(bu 4 \fBwork_queue_worker(1)\fP .IP \(bu 4 \fBcondor_submit_workers(1)\fP .IP \(bu 4 \fBsge_submit_workers(1)\fP .IP \(bu 4 \fBallpairs_multicore(1)\fP