.\" Man page generated from reStructuredText.
.
.
.nr rst2man-indent-level 0
.
.de1 rstReportMargin
\\$1 \\n[an-margin]
level \\n[rst2man-indent-level]
level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
-
\\n[rst2man-indent0]
\\n[rst2man-indent1]
\\n[rst2man-indent2]
..
.de1 INDENT
.\" .rstReportMargin pre:
. RS \\$1
. nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin]
. nr rst2man-indent-level +1
.\" .rstReportMargin post:
..
.de UNINDENT
. RE
.\" indent \\n[an-margin]
.\" old: \\n[rst2man-indent\\n[rst2man-indent-level]]
.nr rst2man-indent-level -1
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "CONDOR_DAGMAN" "1" "Apr 01, 2024" "" "HTCondor Manual"
.SH NAME
condor_dagman \- HTCondor Manual
.sp
meta scheduler of the jobs submitted as the nodes of a DAG or DAGs
  
.SH SYNOPSIS
.sp
\fBcondor_dagman\fP \fI\-f\fP \fI\-t\fP \fI\-l .\fP \fB\-help\fP
.sp
\fBcondor_dagman\fP \fB\-version\fP
.sp
\fBcondor_dagman\fP \fI\-f\fP \fI\-l .\fP \fB\-csdversion\fP \fIversion_string\fP
[\fB\-debug\fP \fIlevel\fP] [\fB\-dryrun\fP] [\fB\-maxidle\fP \fInumberOfProcs\fP]
[\fB\-maxjobs\fP \fInumberOfJobs\fP] [\fB\-maxpre\fP \fINumberOfPreScripts\fP]
[\fB\-maxpost\fP \fINumberOfPostScripts\fP] [\fB\-maxhold\fP \fINumberOfHoldScripts\fP]
[\fB\-usedagdir\fP ] \fB\-lockfile\fP \fIfilename\fP [\fB\-waitfordebug\fP ]
[\fB\-autorescue\fP \fI0|1\fP] [\fB\-dorescuefrom\fP \fInumber\fP]
[\fB\-load_save\fP \fIfilename\fP] [\fB\-allowversionmismatch\fP ]
[\fB\-DumpRescue\fP ] [\fB\-verbose\fP ] [\fB\-force\fP ]
[\fB\-notification\fP \fIvalue\fP] [\fB\-suppress_notification\fP ]
[\fB\-dont_suppress_notification\fP ] [\fB\-dagman\fP \fIDagmanExecutable\fP]
[\fB\-outfile_dir\fP \fIdirectory\fP] [\fB\-update_submit\fP ]
[\fB\-import_env\fP ] [\fB\-include_env\fP \fIVariables\fP]
[\fB\-insert_env\fP \fIKey=Value\fP] [\fB\-priority\fP \fInumber\fP]
[\fB\-DontAlwaysRunPost\fP ] [\fB\-AlwaysRunPost\fP ]
[\fB\-DoRecovery\fP ] [\fB\-dot\fP] \fB\-dag\fP \fIdag_file\fP
[\fB\-dag\fP \fIdag_file_2\fP ... \fB\-dag\fP \fIdag_file_n\fP ]
.SH DESCRIPTION
.sp
\fIcondor_dagman\fP is a meta scheduler for the HTCondor jobs within a DAG
(directed acyclic graph) (or multiple DAGs). In typical usage, a
submitter of jobs that are organized into a DAG submits the DAG using
\fIcondor_submit_dag\fP\&. \fIcondor_submit_dag\fP does error checking on
aspects of the DAG and then submits \fIcondor_dagman\fP as an HTCondor job.
\fIcondor_dagman\fP uses log files to coordinate the further submission of
the jobs within the DAG.
.sp
All command line arguments to the \fIDaemonCore\fP library functions work
for \fIcondor_dagman\fP\&. When invoked from the command line,
\fIcondor_dagman\fP requires the arguments \fI\-f \-l .\fP to appear first on the
command line, to be processed by \fIDaemonCore\fP\&. The \fBcsdversion\fP must
also be specified; at start up, \fIcondor_dagman\fP checks for a version
mismatch with the \fIcondor_submit_dag\fP version in this argument. The
\fI\-t\fP argument must also be present for the \fB\-help\fP option, such that
output is sent to the terminal.
.sp
Arguments to \fIcondor_dagman\fP are either automatically set by
\fIcondor_submit_dag\fP or they are specified as command\-line arguments to
\fIcondor_submit_dag\fP and passed on to \fIcondor_dagman\fP\&. The method by
which the arguments are set is given in their description below.
.sp
\fIcondor_dagman\fP can run multiple, independent DAGs. This is done by
specifying multiple \fB\-dag\fP \fIa\fP rguments. Pass multiple DAG input
files as command\-line arguments to \fIcondor_submit_dag\fP\&.
.sp
Debugging output may be obtained by using the \fB\-debug\fP \fIlevel\fP
option. Level values and what they produce is described as
.INDENT 0.0
.IP \(bu 2
level = 0; never produce output, except for usage info
.IP \(bu 2
level = 1; very quiet, output severe errors
.IP \(bu 2
level = 2; normal output, errors and warnings
.IP \(bu 2
level = 3; output errors, as well as all warnings
.IP \(bu 2
level = 4; internal debugging output
.IP \(bu 2
level = 5; internal debugging output; outer loop debugging
.IP \(bu 2
level = 6; internal debugging output; inner loop debugging; output
DAG input file lines as they are parsed
.IP \(bu 2
level = 7; internal debugging output; rarely used; output DAG input
file lines as they are parsed
.UNINDENT
.SH OPTIONS
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.TP
\fB\-help\fP
Display usage information and exit.
.TP
\fB\-version\fP
Display version information and exit.
.TP
\fB\-csdversion\fP \fIVersionString\fP
Sets the version of \fIcondor_submit_dag\fP command used to submit
the DAGMan workflow. Used to help identify version mismatching.
.TP
\fB\-debug\fP \fIlevel\fP
An integer level of debugging output. \fIlevel\fP is an integer, with
values of 0\-7 inclusive, where 7 is the most verbose output. This
command\-line option to \fIcondor_submit_dag\fP is passed to
\fIcondor_dagman\fP or defaults to the value 3.
.TP
\fB\-dryrun\fP
Inform DAGMan to do a dry run. Where the DAG is ran but node jobs
are not actually submitted.
.TP
\fB\-maxidle\fP \fINumberOfProcs\fP
Sets the maximum number of idle procs allowed before
\fIcondor_dagman\fP stops submitting more node jobs. If this option is
omitted then the number of idle procs is limited by the configuration
variable \fI\%DAGMAN_MAX_JOBS_IDLE\fP which defaults to 1000.
To disable this limit, set \fINumberOfProcs\fP to 0. The \fINumberOfProcs\fP
can be exceeded if a nodes job has a queue command with more than
one proc to queue. i.e. \fBqueue 500\fP will submit all procs even
if \fINumberOfProcs\fP is \fB250\fP\&. In this case DAGMan will wait for
for the number of idle procs to fall below 250 before submitting
more jobs to the \fBcondor_schedd\fP\&.
.TP
\fB\-maxjobs\fP \fINumberOfClusters\fP
Sets the maximum number of clusters within the DAG that will be
submitted to HTCondor at one time. Each cluster is associated with
one node job no matter how many individual procs are in the cluster.
\fINumberOfClusters\fP is a non\-negative integer. If this option is
omitted then the number of clusters is limited by the configuration
variable \fI\%DAGMAN_MAX_JOBS_SUBMITTED\fP which defaults to 0 (unlimited).
.TP
\fB\-maxpre\fP \fINumberOfPreScripts\fP
Sets the maximum number of PRE scripts within the DAG that may be
running at one time. \fINumberOfPreScripts\fP is a non\-negative integer.
If this option is omitted, the number of PRE scripts is limited by
the configuration variable \fI\%DAGMAN_MAX_PRE_SCRIPTS\fP
which defaults to 20.
.TP
\fB\-maxpost\fP \fINumberOfPostScripts\fP
Sets the maximum number of POST scripts within the DAG that may be
running at one time. \fINumberOfPostScripts\fP is a non\-negative
integer. If this option is omitted, the number of POST scripts is
limited by the configuration variable \fI\%DAGMAN_MAX_POST_SCRIPTS\fP
which defaults to 20.
.TP
\fB\-maxhold\fP \fINumberOfHoldScripts\fP
Sets the maximum number of HOLD scripts within the DAG that may be
running at one time. \fINumberOfHoldscripts\fP is a non\-negative integer.
If this option is omitted, the number of HOLD scripts is limited by
the configuration variable \fI\%DAGMAN_MAX_HOLD_SCRIPTS\fP, which
defaults to 0 (unlimited).
.TP
\fB\-usedagdir\fP
This optional argument causes \fIcondor_dagman\fP to run each specified
DAG as if the directory containing that DAG file was the current
working directory. This option is most useful when running multiple
DAGs in a single \fIcondor_dagman\fP\&.
.TP
\fB\-lockfile\fP \fIfilename\fP
Names the file created and used as a lock file. The lock file
prevents execution of two of the same DAG, as defined by a DAG input
file. A default lock file ending with the suffix \fB\&.dag.lock\fP is
passed to \fIcondor_dagman\fP by \fIcondor_submit_dag\fP\&.
.TP
\fB\-waitfordebug\fP
This optional argument causes \fIcondor_dagman\fP to wait at startup
until someone attaches to the process with a debugger and sets the
wait_for_debug variable in main_init() to false.
.TP
\fB\-autorescue\fP \fI0|1\fP
Whether to automatically run the newest rescue DAG for the given DAG
file, if one exists (0 = \fBfalse\fP, 1 = \fBtrue\fP).
.TP
\fB\-dorescuefrom\fP \fInumber\fP
Forces \fIcondor_dagman\fP to run the specified rescue DAG number for
the given DAG. A value of 0 is the same as not specifying this
option. Specifying a nonexistent rescue DAG is a fatal error.
.TP
\fB\-load_save\fP \fIfilename\fP
Specify a file with saved DAG progress to re\-run the DAG from. If
given a path DAGMan will attempt to read that file following that
path. Otherwise, DAGMan will check for the file in the DAG\(aqs
\fBsave_files\fP sub\-directory.
.TP
\fB\-allowversionmismatch\fP
This optional argument causes \fIcondor_dagman\fP to allow a version
mismatch between \fIcondor_dagman\fP itself and the \fB\&.condor.sub\fP
file produced by \fIcondor_submit_dag\fP (or, in other words, between
\fIcondor_submit_dag\fP and \fIcondor_dagman\fP). WARNING! This option
should be used only if absolutely necessary. Allowing version
mismatches can cause subtle problems when running DAGs.
.TP
\fB\-DumpRescue\fP
This optional argument causes \fIcondor_dagman\fP to immediately dump a
Rescue DAG and then exit, as opposed to actually running the DAG.
This feature is mainly intended for testing. The Rescue DAG file is
produced whether or not there are parse errors reading the original
DAG input file. The name of the file differs if there was a parse
error.
.TP
\fB\-verbose\fP
(This argument is included only to be passed to
\fIcondor_submit_dag\fP if lazy submit file generation is used for
nested DAGs.) Cause \fIcondor_submit_dag\fP to give verbose error
messages.
.TP
\fB\-force\fP
(This argument is included only to be passed to
\fIcondor_submit_dag\fP if lazy submit file generation is used for
nested DAGs.) Require \fIcondor_submit_dag\fP to overwrite the files
that it produces, if the files already exist. Note that
\fBdagman.out\fP will be appended to, not overwritten. If new\-style
rescue DAG mode is in effect, and any new\-style rescue DAGs exist,
the \fB\-force\fP flag will cause them to be renamed, and the original
DAG will be run. If old\-style rescue DAG mode is in effect, any
existing old\-style rescue DAGs will be deleted, and the original DAG
will be run. See the HTCondor manual section on Rescue DAGs for more
information.
.TP
\fB\-notification\fP \fIvalue\fP
This argument is only included to be passed to \fIcondor_submit_dag\fP
if lazy submit file generation is used for nested DAGs. Sets the
e\-mail notification for DAGMan itself. This information will be used
within the HTCondor submit description file for DAGMan. This file is
produced by \fIcondor_submit_dag\fP\&. The \fBnotification\fP option is
described in the \fIcondor_submit\fP manual page.
.TP
\fB\-suppress_notification\fP
Causes jobs submitted by \fIcondor_dagman\fP to not send email
notification for events. The same effect can be achieved by setting
the configuration variable \fI\%DAGMAN_SUPPRESS_NOTIFICATION\fP to
\fBTrue\fP\&. This command line option is independent of the \fB\-notification\fP
command line option, which controls notification for the \fIcondor_dagman\fP
job itself. This flag is generally superfluous, as
\fI\%DAGMAN_SUPPRESS_NOTIFICATION\fP defaults to \fBTrue\fP\&.
.TP
\fB\-dont_suppress_notification\fP
Causes jobs submitted by \fIcondor_dagman\fP to defer to content within
the submit description file when deciding to send email notification
for events. The same effect can be achieved by setting the
configuration variable \fI\%DAGMAN_SUPPRESS_NOTIFICATION\fP to
\fBFalse\fP\&. This command line flag is independent of the \fB\-notification\fP
command line option, which controls notification for the \fIcondor_dagman\fP
job itself. If both \fB\-dont_suppress_notification\fP and
\fB\-suppress_notification\fP are specified within the same command
line, the last argument is used.
.TP
\fB\-dagman\fP \fIDagmanExecutable\fP
(This argument is included only to be passed to
\fIcondor_submit_dag\fP if lazy submit file generation is used for
nested DAGs.) Allows the specification of an alternate
\fIcondor_dagman\fP executable to be used instead of the one found in
the user\(aqs path. This must be a fully qualified path.
.TP
\fB\-outfile_dir\fP \fIdirectory\fP
(This argument is included only to be passed to
\fIcondor_submit_dag\fP if lazy submit file generation is used for
nested DAGs.) Specifies the directory in which the \fB\&.dagman.out\fP
file will be written. The \fIdirectory\fP may be specified relative to
the current working directory as \fIcondor_submit_dag\fP is executed,
or specified with an absolute path. Without this option, the
\fB\&.dagman.out\fP file is placed in the same directory as the first
DAG input file listed on the command line.
.TP
\fB\-update_submit\fP
(This argument is included only to be passed to
\fIcondor_submit_dag\fP if lazy submit file generation is used for
nested DAGs.) This optional argument causes an existing
\fB\&.condor.sub\fP file to not be treated as an error; rather, the
\fB\&.condor.sub\fP file will be overwritten, but the existing values of
\fB\-maxjobs\fP, \fB\-maxidle\fP, \fB\-maxpre\fP, and \fB\-maxpost\fP will be
preserved.
.TP
\fB\-import_env\fP
(This argument is included only to be passed to
\fIcondor_submit_dag\fP if lazy submit file generation is used for
nested DAGs.) This optional argument causes \fIcondor_submit_dag\fP to
import the current environment into the \fBenvironment\fP command of
the \fB\&.condor.sub\fP file it generates.
.TP
\fB\-include_env\fP \fIVariables\fP
This optional argument takes a comma separated list of enviroment
variables to add to \fB\&.condor.sub\fP \fBgetenv\fP environment filter
which causes found matching environment variables to be added to
the DAGMan manager jobs \fBenvironment\fP\&.
.TP
\fB\-insert_env\fP \fIKey=Value\fP
This optional argument takes a delimited string of \fIKey=Value\fP pairs
to explicitly set into the \fB\&.condor.sub\fP files \fBenvironment\fP macro.
The base delimiter is a semicolon that can be overriden by setting
the first character in the string to a valid delimiting character.
If multiple \fB\-insert_env\fP flags contain the same \fIKey\fP then the last
occurances \fIValue\fP will be set in the DAGMan jobs \fBenvironment\fP\&.
.TP
\fB\-priority\fP \fInumber\fP
Sets the minimum job priority of node jobs submitted and running
under this \fIcondor_dagman\fP job.
.TP
\fB\-DontAlwaysRunPost\fP
This option causes \fIcondor_dagman\fP to not run the POST script of a
node if the PRE script fails.
.TP
\fB\-AlwaysRunPost\fP
This option causes \fIcondor_dagman\fP to always run the POST script of
a node, even if the PRE script fails.
.TP
\fB\-DoRecovery\fP
Causes \fIcondor_dagman\fP to start in recovery mode. This means that
it reads the relevant job user log(s) and catches up to the given
DAG\(aqs previous state before submitting any new jobs.
.TP
\fB\-dot\fP
Run \fIcondor_dagman\fP up until the point when a \fBDOT\fP file is
produced.
.TP
\fB\-dag\fP \fIfilename\fP
\fIfilename\fP is the name of the DAG input file that is set as an
argument to \fIcondor_submit_dag\fP, and passed to \fIcondor_dagman\fP\&.
.UNINDENT
.UNINDENT
.UNINDENT
.SH EXIT STATUS
.sp
\fIcondor_dagman\fP will exit with a status value of 0 (zero) upon success,
and it will exit with the value 1 (one) upon failure.
.SH EXAMPLES
.sp
\fIcondor_dagman\fP is normally not run directly, but submitted as an
HTCondor job by running condor_submit_dag. See the
\fI\%condor_submit_dag\fP manual page for examples.
.SH AUTHOR
HTCondor Team
.SH COPYRIGHT
1990-2024, Center for High Throughput Computing, Computer Sciences Department, University of Wisconsin-Madison, Madison, WI, US. Licensed under the Apache License, Version 2.0.
.\" Generated by docutils manpage writer.
.