.TH "work_queue_factory" 1 "" "CCTools 7.1.2 FINAL" "Cooperative Computing Tools"

.SH NAME
.LP
\fBwork_queue_factory\fP - maintain a pool of Work Queue workers on a batch system.

.SH SYNOPSIS
.LP
\FC\fBwork_queue_factory -M <project-name> -T <batch-type> [options]\fP\FT

.SH DESCRIPTION
.LP
\fBwork_queue_factory\fP submits and maintains a number
of \fBwork_queue_worker(1)\fP processes on various batch systems, such as
Condor and SGE.  All the workers managed by a \fBwork_queue_factory\fP process
will be directed to work for a specific master, or any set of masters matching
a given project name.  \fBwork_queue_factory\fP will automatically determine
the correct number of workers to have running, based on criteria set on
the command line.  The decision on how many workers to run is reconsidered
once per minute.
.PP
By default, \fBwork_queue_factory\fP will run as many workers as the
indicated masters have tasks ready to run.  If there are multiple
masters, then enough workers will be started to satisfy their collective needs.
For example, if there are two masters with the same project name, each with
10 tasks to run, then \fBwork_queue_factory\fP will start a total of 20 workers.
.PP
If the number of needed workers increases, \fBwork_queue_factory\fP will submit
more workers to meet the desired need.  However, it will not run more than
a fixed maximum number of workers, given by the -W option.
.PP
If the need for workers drops, \fBwork_queue_factory\fP does not remove them immediately,
but waits to them to exit on their own.  (This happens when the worker has been idle
for a certain time.)  A minimum number of workers will be maintained, given
by the -w option.
.PP
If given the -c option, then \fBwork_queue_factory\fP will consider the capacity
reported by each master.  The capacity is the estimated number of workers
that the master thinks it can handle, based on the task execution and data
transfer times currently observed at the master.  With the -c option on,
\fBwork_queue_factory\fP will consider the master's capacity to be the maximum
number of workers to run.
.PP
If \fBwork_queue_factory\fP receives a terminating signal, it will attempt to
remove all running workers before exiting.

.SH OPTIONS
.LP

.SS Batch Options
.LP
.LP
.TP
\fB-M\fP, \fB-\-master-name\fP=\fI<project>\fP
.
Project name of masters to serve, can be a regular expression.
.TP
\fB-F\fP, \fB-\-foremen-name\fP=\fI<project>\fP
.
Foremen to serve, can be a regular expression.
.TP
.BI \ --catalog \ <catalog>
.
Catalog server to query for masters (default: catalog.cse.nd.edu,backup-catalog.cse.nd.edu:9097).
.TP
\fB-T\fP, \fB-\-batch-type\fP=\fI<type>\fP
.
Batch system type (required). One of: local, wq, condor, sge, torque, mesos, k8s, moab, slurm, chirp, amazon, lambda, dryrun, amazon-batch
.TP
\fB-B\fP, \fB-\-batch-options\fP=\fI<options>\fP
.
Add these options to all batch submit files.
.TP
\fB-P\fP, \fB-\-password\fP=\fI<file>\fP
.
Password file for workers to authenticate to master.
.TP
\fB-C\fP, \fB-\-config-file\fP=\fI<file>\fP
.
Use the configuration file <file>.
.TP
\fB-w\fP, \fB-\-min-workers\fP=\fI<workers>\fP
.
Minimum workers running.  (default=5)
.TP
\fB-W\fP, \fB-\-max-workers\fP=\fI<workers>\fP
.
Maximum workers running.  (default=100)
.TP
.BI \ --workers-per-cycle \ <workers>
.
Maximum number of new workers per 30 seconds.  ( less than 1 disables limit, default=5)
.TP
.BI \ --tasks-per-worker \ <workers>
.
Average tasks per worker (default=one task per core).
.TP
\fB-t\fP, \fB-\-timeout\fP=\fI<time>\fP
.
Workers abort after this amount of idle time (default=300).
.TP
.BI \ --env \ <variable=value>
.
Environment variable that should be added to the worker (May be specified multiple times).
.TP
\fB-E\fP, \fB-\-extra-options\fP=\fI<options>\fP
.
Extra options that should be added to the worker.
.TP
.BI \ --cores \ <n>
.
Set the number of cores requested per worker.
.TP
.BI \ --gpus \ <n>
.
Set the number of GPUs requested per worker.
.TP
.BI \ --memory \ <mb>
.
Set the amount of memory (in MB) requested per worker.
.TP
.BI \ --disk \ <mb>
.
Set the amount of disk (in MB) requested per worker.
.TP
.B \ --autosize
.
Automatically size a worker to an available slot (Condor, Mesos, and Kubernetes).
.TP
.B \ --condor-requirements
.
Set requirements for the workers as Condor jobs. May be specified several times with expresions and-ed together (Condor only).
.TP
.BI \ --factory-timeout \ <n>
.
Exit after no master has been seen in <n> seconds.
.TP
\fB-S\fP, \fB-\-scratch-dir\fP=\fI<file>\fP
.
Use this scratch dir for temporary files (default is /tmp/wq-pool-$uid).
.TP
.B \ -c, --capacity
.
Use worker capacity reported by masters.
.TP
\fB-d\fP, \fB-\-debug\fP=\fI<subsystem>\fP
.
Enable debugging for this subsystem.
.TP
.B \ --amazon-config
.
Specify Amazon config file (for use with -T amazon).
.TP
.B \ --wrapper
.
Wrap factory with this command prefix.
.TP
.B \ --wrapper-input
.
Add this input file needed by the wrapper.
.TP
.BI \ --mesos-master \ <hostname>
.
Specify the host name to mesos master node (for use with -T mesos).
.TP
.BI \ --mesos-path \ <filepath>
.
Specify path to mesos python library (for use with -T mesos).
.TP
.BI \ --mesos-preload \ <library>
.
Specify the linking libraries for running mesos (for use with -T mesos).
.TP
.B \ --k8s-image
.
Specify the container image for using Kubernetes (for use with -T k8s).
.TP
.B \ --k8s-worker-image
.
Specify the container image that contains work_queue_worker availabe for using Kubernetes (for use with -T k8s).
.TP
\fB-o\fP, \fB-\-debug-file\fP=\fI<file>\fP
.
Send debugging to this file (can also be :stderr, :stdout, :syslog, or :journal).
.TP
\fB-O\fP, \fB-\-debug-file-size\fP=\fI<mb>\fP
.
Specify the size of the debug file (must use with -o option).
.TP
.BI \ --worker-binary \ <file>
.
Specify the binary to use for the worker (relative or hard path). It should accept the same arguments as the default work_queue_worker.
.TP
.BI \ --runos \ <img>
.
Will make a best attempt to ensure the worker will execute in the specified OS environment, regardless of the underlying OS.
.TP
.B \ --run-factory-as-master
.
Force factory to run itself as a work queue master.
.TP
.B \ -v, --version
.
Show the version string.
.TP
.B \ -h, --help
.
Show this screen.


.SH EXIT STATUS
.LP
On success, returns zero. On failure, returns non-zero.

.SH EXAMPLES
.LP

Suppose you have a Work Queue master with a project name of "barney".
To maintain workers for barney, do this:

.fam C
.nf
.nh
.IP "" 8
work_queue_factory -T condor -M barney
.fi
.hy
.fam
.P

To maintain a maximum of 100 workers on an SGE batch system, do this:

.fam C
.nf
.nh
.IP "" 8
work_queue_factory -T sge -M barney -W 100
.fi
.hy
.fam
.P

To start workers such that the workers exit after 5 minutes (300s) of idleness:

.fam C
.nf
.nh
.IP "" 8
work_queue_factory -T condor -M barney -t 300
.fi
.hy
.fam
.P

If you want to start workers that match any project that begins
with barney, use a regular expression:

.fam C
.nf
.nh
.IP "" 8
work_queue_factory -T condor -M barney.\* -t 300
.fi
.hy
.fam
.P

If running on condor, you may manually specify condor requirements:

.fam C
.nf
.nh
.IP "" 8
work_queue_factory -T condor -M barney --condor-requirements 'MachineGroup == "disc"' --condor-requirements 'has_matlab == true'
.fi
.hy
.fam
.P

Repeated uses of \FCcondor-requirements\FT are and-ed together. The previous example will produce a statement equivalent to:

\FCrequirements = ((MachineGroup == "disc") && (has_matlab == true))\FT

Use the configuration file \fBmy_conf\fP:

.fam C
.nf
.nh
.IP "" 8
work_queue_factory -Cmy_conf
.fi
.hy
.fam
.P

\fBmy_conf\fP should be a proper JSON document, as:
.fam C
.nf
.nh
.IP "" 8
{
        "master-name": "my_master.*",
        "max-workers": 100,
        "min-workers": 0
}
.fi
.hy
.fam
.P

Valid configuration fields are:

.fam C
.nf
.nh
.IP "" 8
master-name
foremen-name
min-workers
max-workers
workers-per-cycle
task-per-worker
timeout
worker-extra-options
condor-requirements
cores
memory
disk
.fi
.hy
.fam
.P

.SH KNOWN BUGS
.LP

The capacity measurement currently assumes single-core tasks running on single-core
workers, and behaves unexpectedly with multi-core tasks or multi-core workers.

.SH COPYRIGHT
.LP
The Cooperative Computing Tools are Copyright (C) 2005-2019 The University of Notre Dame.  This software is distributed under the GNU General Public License.  See the file COPYING for details.

.SH SEE ALSO
.LP


.IP \(bu 4
\fBCooperative Computing Tools Documentation\fP
.IP \(bu 4
\fBWork Queue User Manual\fP
.IP \(bu 4
\fBwork_queue_worker(1)\fP \fBwork_queue_status(1)\fP \fBwork_queue_factory(1)\fP \fBcondor_submit_workers(1)\fP \fBsge_submit_workers(1)\fP \fBtorque_submit_workers(1)\fP