NAME¶

weaver - workflow engine for executing distributed workflows

SYNOPSIS¶

weaver [options] <weaverfile>

DESCRIPTION¶

Weaver is a high level interface to makeflow. A weaver input file is written in python, with the definition of functions to be applied on sets of files. weaver interprets this input file and generates a workflow specification that can be executed by makeflow. This allows an straightforward implementation of different workflow execution patterns, such as MapReduce, and AllPairs.

				      /--------				    +-+ Python |
				    | ---+----/
+---------------------------------+ |     | Generate DAG
|	      Weaver		  +-+     v
+---------------------------------+   /-------|	     Makeflow		  +---+  DAG  |
+--------+-----------+-----+------+   -------/
| Condor | WorkQueue | SGE | Unix +-+     | Dispatch Jobs
+--------+-----------+-----+------+ |     v
				    | /-------				    +-+ Jobs  |
				      -------/
    

OPTIONS¶

By default, running weaver on a <weaverfile> generates an input file for makeflow, <Makeflow>, and a directory, <_Stash>, in which intermediate files are stored.

General options:

 -h

Give help information.

 -W

Stop on warnings.

 -g

Include debugging symbols in DAG.

 -I

Do not automatically import built-ins.

 -N

Do not normalize paths.

 -b <options>

Set batch job options (cpu, memory, disk, batch, local, collect).

 -d <subsystem>

Enable debugging for subsystem.

 -o <log_path>

Set log path (default: stderr).

 -O <directory>

Set stash output directory (default <_Stash>).

Optimization Options:

 -a

Automatically nest abstractions.

 -t <group_size>

Inline tasks based on group size.

Engine Options:

 -x

Execute DAG using workflow engine after compiling.

 -e <arguments>

Set arguments to workflow engine when executing.

 -w <wrapper>

Set workflow engine wrapper.

EXIT STATUS¶

On success, returns zero. On failure, returns non-zero.

EXAMPLES¶

Weaver expresses common workflow patterns succinctly. For example, with only the following three lines of code we can express a map pattern, in which we convert some images to the jpeg format:

convert = ParseFunction('convert {IN} {OUT}')
dataset = Glob('/usr/share/pixmaps/*.xpm')
jpgs    = Map(convert, dataset, '{basename_woext}.jpg')
    

Please refer to cctools/doc/weaver_examples for further information.

The Cooperative Computing Tools are Copyright (C) 2022 The University of Notre Dame. This software is distributed under the GNU General Public License. See the file COPYING for details.