.\" Man page generated from reStructuredText.
.
.
.nr rst2man-indent-level 0
.
.de1 rstReportMargin
\\$1 \\n[an-margin]
level \\n[rst2man-indent-level]
level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
-
\\n[rst2man-indent0]
\\n[rst2man-indent1]
\\n[rst2man-indent2]
..
.de1 INDENT
.\" .rstReportMargin pre:
. RS \\$1
. nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin]
. nr rst2man-indent-level +1
.\" .rstReportMargin post:
..
.de UNINDENT
. RE
.\" indent \\n[an-margin]
.\" old: \\n[rst2man-indent\\n[rst2man-indent-level]]
.nr rst2man-indent-level -1
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "CONDOR_SUBMIT_DAG" "1" "Apr 01, 2024" "" "HTCondor Manual"
.SH NAME
condor_submit_dag \- HTCondor Manual
.sp
Manage and queue jobs within a specified DAG for execution on remote
machines
  
.SH SYNOPSIS
.sp
\fBcondor_submit_dag\fP [\fB\-help | \-version\fP ]
.sp
\fBcondor_submit_dag\fP [\fB\-no_submit\fP ] [\fB\-verbose\fP ]
[\fB\-force\fP ] [\fB\-dagman\fP \fIDagmanExecutable\fP]
[\fB\-maxidle\fP \fINumberOfProcs\fP] [\fB\-maxjobs\fP \fINumberOfClusters\fP]
[\fB\-maxpre\fP \fINumberOfPreScripts\fP] [\fB\-maxpost\fP \fINumberOfPostScripts\fP]
[\fB\-notification\fP \fIvalue\fP] [\fB\-r\fP \fIschedd_name\fP]
[\fB\-debug\fP \fIlevel\fP] [\fB\-usedagdir\fP ]
[\fB\-outfile_dir\fP \fIdirectory\fP] [\fB\-config\fP \fIConfigFileName\fP]
[\fB\-insert_sub_file\fP \fIFileName\fP] [\fB\-append\fP \fICommand\fP]
[\fB\-batch\-name\fP \fIbatch_name\fP] [\fB\-autorescue\fP \fI0|1\fP]
[\fB\-dorescuefrom\fP \fInumber\fP] [\fB\-load_save\fP \fIfilename\fP]
[\fB\-allowversionmismatch\fP ]
[\fB\-no_recurse\fP ] [\fB\-do_recurse\fP ] [\fB\-update_submit\fP ]
[\fB\-import_env\fP ] [\fB\-include_env\fP \fIVariables\fP] [\fB\-insert_env\fP \fIKey=Value\fP]
[\fB\-DumpRescue\fP ] [\fB\-valgrind\fP ] [\fB\-DontAlwaysRunPost\fP ] [\fB\-AlwaysRunPost\fP ]
[\fB\-priority\fP \fInumber\fP]
[\fB\-schedd\-daemon\-ad\-file\fP \fIFileName\fP]
[\fB\-schedd\-address\-file\fP \fIFileName\fP] [\fB\-suppress_notification\fP ]
[\fB\-dont_suppress_notification\fP ] [\fB\-DoRecovery\fP ]
\fIDAGInputFile1\fP [\fIDAGInputFile2 ... DAGInputFileN\fP ]
.SH DESCRIPTION
.sp
\fIcondor_submit_dag\fP is the program for submitting a DAG (directed
acyclic graph) of jobs for execution under HTCondor. The program
enforces the job dependencies defined in one or more \fIDAGInputFile\fPs.
Each \fIDAGInputFile\fP contains commands to direct the submission of jobs
implied by the nodes of a DAG to HTCondor. Extensive documentation is in
the HTCondor User Manual section on DAGMan.
 
 
.sp
Some options may be specified on the command line or in the
configuration or in a node job\(aqs submit description file. Precedence is
given to command line options or configuration over settings from a
submit description file. An example is e\-mail notifications. When
configuration variable  \fI\%DAGMAN_SUPPRESS_NOTIFICATION\fP is its default value of
\fBTrue\fP, and a node job\(aqs submit description file contains
.INDENT 0.0
.INDENT 3.5
.sp
.EX
notification = Complete
.EE
.UNINDENT
.UNINDENT
.sp
e\-mail will not be sent upon completion, as the value of
\fI\%DAGMAN_SUPPRESS_NOTIFICATION\fP is enforced.
.SH OPTIONS
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.TP
\fB\-help\fP
Display usage information and exit.
.TP
\fB\-version\fP
Display version information and exit.
.TP
\fB\-no_submit\fP
Produce the HTCondor submit description file for DAGMan, but do not
submit DAGMan as an HTCondor job.
.TP
\fB\-verbose\fP
Cause \fIcondor_submit_dag\fP to give verbose error messages.
.TP
\fB\-force\fP
Require \fIcondor_submit_dag\fP to overwrite the files that it
produces, if the files already exist. Note that \fBdagman.out\fP will
be appended to, not overwritten. If rescue files exist then
DAGMan will run the original DAG and rename the rescue files.
Any old\-style rescue files will be deleted.
.TP
\fB\-dagman\fP \fIDagmanExecutable\fP
Allows the specification of an alternate \fIcondor_dagman\fP executable
to be used instead of the one found in the user\(aqs path. This must be
a fully qualified path.
.TP
\fB\-maxidle\fP \fINumberOfProcs\fP
Sets the maximum number of idle procs allowed before
\fIcondor_dagman\fP stops submitting more node jobs. If this option is
omitted then the number of idle procs is limited by the configuration
variable \fI\%DAGMAN_MAX_JOBS_IDLE\fP which defaults to 1000.
To disable this limit, set \fINumberOfProcs\fP to 0. The \fINumberOfProcs\fP
can be exceeded if a nodes job has a queue command with more than
one proc to queue. i.e. \fBqueue 500\fP will submit all procs even
if \fINumberOfProcs\fP is \fB250\fP\&. In this case DAGMan will wait for
for the number of idle procs to fall below 250 before submitting
more jobs to the \fBcondor_schedd\fP\&.
.TP
\fB\-maxjobs\fP \fINumberOfClusters\fP
Sets the maximum number of clusters within the DAG that will be
submitted to HTCondor at one time. Each cluster is associated with
one node job no matter how many individual procs are in the cluster.
\fINumberOfClusters\fP is a non\-negative integer. If this option is
omitted then the number of clusters is limited by the configuration
variable \fI\%DAGMAN_MAX_JOBS_SUBMITTED\fP which defaults to 0 (unlimited).
.TP
\fB\-maxpre\fP \fINumberOfPreScripts\fP
Sets the maximum number of PRE scripts within the DAG that may be
running at one time. \fINumberOfPreScripts\fP is a non\-negative integer.
If this option is omitted, the number of PRE scripts is limited by
the configuration variable \fI\%DAGMAN_MAX_PRE_SCRIPTS\fP
which defaults to 20.
.TP
\fB\-maxpost\fP \fINumberOfPostScripts\fP
Sets the maximum number of POST scripts within the DAG that may be
running at one time. \fINumberOfPostScripts\fP is a non\-negative
integer. If this option is omitted, the number of POST scripts is
limited by the configuration variable \fI\%DAGMAN_MAX_POST_SCRIPTS\fP
which defaults to 20.
.TP
\fB\-notification\fP \fIvalue\fP
Sets the e\-mail notification for DAGMan itself. This information
will be used within the HTCondor submit description file for DAGMan.
This file is produced by \fIcondor_submit_dag\fP\&. See the description
of \fBnotification\fP  
within \fIcondor_submit\fP manual page for a specification of \fIvalue\fP\&.
.TP
\fB\-r\fP \fIschedd_name\fP
Submit \fIcondor_dagman\fP to a \fIcondor_schedd\fP on a remote machine.
It is assumed that any necessary files will be present on the
remote machine via some method like a shared filesystem between the
local and remote machines. The user also requires the correct
permissions to submit remotely similarly to \fIcondor_submit\fP\(aqs
\fB\-remote\fP option. If other options are desired, including
transfer of other input files, consider using the \fB\-no_submit\fP
option and modifying the resulting submit file for specific needs
before using \fIcondor_submit\fP on the prouduced DAGMan job submit file.
.TP
\fB\-debug\fP \fIlevel\fP
Passes the the \fIlevel\fP of debugging output desired to
\fIcondor_dagman\fP\&. \fIlevel\fP is an integer, with values of 0\-7
inclusive, where 7 is the most verbose output. See the
\fIcondor_dagman\fP manual page for detailed descriptions of these
values. If not specified, no \fB\-debug\fP \fIV\fPalue is passed to
\fIcondor_dagman\fP\&.
.TP
\fB\-usedagdir\fP
This optional argument causes \fIcondor_dagman\fP to run each specified
DAG as if \fIcondor_submit_dag\fP had been run in the directory
containing that DAG file. This option is most useful when running
multiple DAGs in a single \fIcondor_dagman\fP\&. Note that the
\fB\-usedagdir\fP flag must not be used when running an old\-style
Rescue DAG.
.TP
\fB\-outfile_dir\fP \fIdirectory\fP
Specifies the directory in which the \fB\&.dagman.out\fP file will be
written. The \fIdirectory\fP may be specified relative to the current
working directory as \fIcondor_submit_dag\fP is executed, or specified
with an absolute path. Without this option, the \fB\&.dagman.out\fP file
is placed in the same directory as the first DAG input file listed
on the command line.
.TP
\fB\-config\fP \fIConfigFileName\fP
Specifies a configuration file to be used for this DAGMan run. This
configuration will apply to all DAGs submitted in via DAGMan. Note
that only one custom configuration file can be specified for a DAGMan
workflow which will cause a failure if used in conjuntion with a
DAG using the \fBCONFIG\fP command.
.TP
\fB\-insert_sub_file\fP \fIFileName\fP
Specifies a file to insert into the \fB\&.condor.sub\fP file created by
\fIcondor_submit_dag\fP\&. The specified file must contain only legal
submit file commands. Only one file can be inserted. The specified
file will override the file set by the configuration variable
\fI\%DAGMAN_INSERT_SUB_FILE\fP\&. The specified file is inserted
into the \fB\&.condor.sub\fP file before the queue command and
any commands specified with the \fB\-append\fP option.
.TP
\fB\-append\fP \fICommand\fP
Specifies a command to append to the \fB\&.condor.sub\fP file created by
\fIcondor_submit_dag\fP\&. The specified command is appended to the
\fB\&.condor.sub\fP file immediately before the queue command and after
any commands added via \fB\-insert_sub_file\fP or \fI\%DAGMAN_INSERT_SUB_FILE\fP\&.
Multiple commands are specified by using the \fB\-append\fP option
multiple times. Commands with spaces in them must be enclosed in
double quotes.
.TP
\fB\-batch\-name\fP \fIbatch_name\fP
Set the batch name for this DAG/workflow. The batch name is
displayed by \fIcondor_q\fP\&. If omitted DAGMan will set the batch
name to \fBDagFile+ClusterId\fP where \fIDagFile\fP is the name of
the primary DAG submitted DAGMan and \fIClusterId\fP is the DAGMan
proper jobs \fBClusterId\fP\&. The batch name is set in all jobs
submitted by DAGMan and propagated down into sub\-DAGs. Note:
set the batch name to \(aq \(aq (space) to avoid overriding batch
names specified in node job submit files.
.TP
\fB\-autorescue\fP \fI0|1\fP
Whether to automatically run the newest rescue DAG for the given DAG
file, if one exists (0 = \fBfalse\fP, 1 = \fBtrue\fP).
.TP
\fB\-dorescuefrom\fP \fInumber\fP
Forces \fIcondor_dagman\fP to run the specified rescue DAG number for
the given DAG. A value of 0 is the same as not specifying this
option. Specifying a non\-existent rescue DAG is a fatal error.
.TP
\fB\-load_save\fP \fIfilename\fP
Specify a file with saved DAG progress to re\-run the DAG from. If
given a path DAGMan will attempt to read that file following that
path. Otherwise, DAGMan will check for the file in the DAG\(aqs
\fBsave_files\fP sub\-directory.
.TP
\fB\-allowversionmismatch\fP
This optional argument causes \fIcondor_dagman\fP to allow a version
mismatch between \fIcondor_dagman\fP itself and the \fB\&.condor.sub\fP
file produced by \fIcondor_submit_dag\fP (or, in other words, between
\fIcondor_submit_dag\fP and \fIcondor_dagman\fP). WARNING! This option
should be used only if absolutely necessary. Allowing version
mismatches can cause subtle problems when running DAGs.
.TP
\fB\-no_recurse\fP
This optional argument causes \fIcondor_submit_dag\fP to not run
itself recursively on nested DAGs (this is now the default; this
flag has been kept mainly for backwards compatibility).
.TP
\fB\-do_recurse\fP
This optional argument causes \fIcondor_submit_dag\fP to run itself
recursively on nested DAGs to pre\-produce their \fB\&.condor.sub\fP
files. DAG nodes specified with the \fBSUBDAG EXTERNAL\fP keyword
or with submit file names ending in \fB\&.condor.sub\fP are considered
nested DAGs. This flag is useful when the configuration variable
\fI\%DAGMAN_GENERATE_SUBDAG_SUBMITS\fP is \fBFalse\fP (Not default).
.TP
\fB\-update_submit\fP
This optional argument causes an existing \fB\&.condor.sub\fP file to
not be treated as an error; rather, the \fB\&.condor.sub\fP file will be
overwritten, but the existing values of \fB\-maxjobs\fP, \fB\-maxidle\fP,
\fB\-maxpre\fP, and \fB\-maxpost\fP will be preserved.
.TP
\fB\-import_env\fP
This optional argument causes \fIcondor_submit_dag\fP to import the
current environment into the \fBenvironment\fP command of the
\fB\&.condor.sub\fP file it generates.
.TP
\fB\-include_env\fP \fIVariables\fP
This optional argument takes a comma separated list of enviroment
variables to add to \fB\&.condor.sub\fP \fBgetenv\fP environment filter
which causes found matching environment variables to be added to
the DAGMan manager jobs \fBenvironment\fP\&.
.TP
\fB\-insert_env\fP \fIKey=Value\fP
This optional argument takes a delimited string of \fIKey=Value\fP pairs
to explicitly set into the \fB\&.condor.sub\fP files \fBenvironment\fP macro.
The base delimiter is a semicolon that can be overriden by setting
the first character in the string to a valid delimiting character.
If multiple \fB\-insert_env\fP flags contain the same \fIKey\fP then the last
occurances \fIValue\fP will be set in the DAGMan jobs \fBenvironment\fP\&.
.TP
\fB\-DumpRescue\fP
This optional argument tells \fIcondor_dagman\fP to immediately dump a
rescue DAG and then exit, as opposed to actually running the DAG.
This feature is mainly intended for testing. The Rescue DAG file is
produced whether or not there are parse errors reading the original
DAG input file. The name of the file differs if there was a parse
error.
.TP
\fB\-valgrind\fP
This optional argument causes the submit description file generated
for the submission of \fIcondor_dagman\fP to be modified. The
executable becomes \fIvalgrind\fP run on \fIcondor_dagman\fP, with a
specific set of arguments intended for testing \fIcondor_dagman\fP\&.
Note that this argument is intended for testing purposes only. Using
the \fB\-valgrind\fP option without the necessary \fIvalgrind\fP software
installed will cause the DAG to fail. If the DAG does run, it will
run much more slowly than usual.
.TP
\fB\-DontAlwaysRunPost\fP
This option causes the submit description file generated for the
submission of \fIcondor_dagman\fP to be modified. It causes
\fIcondor_dagman\fP to not run the POST script of a node if the PRE
script fails.
.TP
\fB\-AlwaysRunPost\fP
This option causes the submit description file generated for the
submission of \fIcondor_dagman\fP to be modified. It causes
\fIcondor_dagman\fP to always run the POST script of a node, even if
the PRE script fails.
.TP
\fB\-priority\fP \fInumber\fP
Sets the minimum job priority of node jobs submitted and running
under the \fIcondor_dagman\fP job submitted by this
\fIcondor_submit_dag\fP command.
.TP
\fB\-schedd\-daemon\-ad\-file\fP \fIFileName\fP
Specifies a full path to a daemon ad file dropped by a
\fIcondor_schedd\fP\&. Therefore this allows submission to a specific
scheduler if several are available without repeatedly querying the
\fIcondor_collector\fP\&. The value for this argument defaults to the
configuration attribute \fI\%SCHEDD_DAEMON_AD_FILE\fP\&.
.TP
\fB\-schedd\-address\-file\fP \fIFileName\fP
Specifies a full path to an address file dropped by a
\fIcondor_schedd\fP\&. Therefore this allows submission to a specific
scheduler if several are available without repeatedly querying the
\fIcondor_collector\fP\&. The value for this argument defaults to the
configuration attribute \fI\%SCHEDD_ADDRESS_FILE\fP\&.
.TP
\fB\-suppress_notification\fP
Causes jobs submitted by \fIcondor_dagman\fP to not send email
notification for events. The same effect can be achieved by setting
configuration variable \fI\%DAGMAN_SUPPRESS_NOTIFICATION\fP to \fBTrue\fP\&. This
command line option is independent of the \fB\-notification\fP command
line option, which controls notification for the \fIcondor_dagman\fP
job itself.
.TP
\fB\-dont_suppress_notification\fP
Causes jobs submitted by \fIcondor_dagman\fP to defer to content within
the submit description file when deciding to send email notification
for events. The same effect can be achieved by setting configuration
variable \fI\%DAGMAN_SUPPRESS_NOTIFICATION\fP to \fBFalse\fP\&. This
command line flag is independent of the \fB\-notification\fP command
line option, which controls notification for the \fIcondor_dagman\fP
job itself. If both \fB\-dont_suppress_notification\fP and
\fB\-suppress_notification\fP are specified with the same command
line, the last argument is used.
.TP
\fB\-DoRecovery\fP
Causes \fIcondor_dagman\fP to start in recovery mode. This means that
DAGMan reads the relevant \fB\&.nodes.log\fP file to restore its previous
state of node completions and failures to continue running.
.UNINDENT
.UNINDENT
.UNINDENT
.SH EXIT STATUS
.sp
\fIcondor_submit_dag\fP will exit with a status value of 0 (zero) upon
success, and it will exit with the value 1 (one) upon failure.
.SH EXAMPLES
.sp
To run a single DAG:
.INDENT 0.0
.INDENT 3.5
.sp
.EX
$ condor_submit_dag diamond.dag
.EE
.UNINDENT
.UNINDENT
.sp
To run a DAG when it has already been run and the output files exist:
.INDENT 0.0
.INDENT 3.5
.sp
.EX
$ condor_submit_dag \-force diamond.dag
.EE
.UNINDENT
.UNINDENT
.sp
To run a DAG, limiting the number of idle node jobs in the DAG to a
maximum of five:
.INDENT 0.0
.INDENT 3.5
.sp
.EX
$ condor_submit_dag \-maxidle 5 diamond.dag
.EE
.UNINDENT
.UNINDENT
.sp
To run a DAG, limiting the number of concurrent PRE scripts to 10 and
the number of concurrent POST scripts to five:
.INDENT 0.0
.INDENT 3.5
.sp
.EX
$ condor_submit_dag \-maxpre 10 \-maxpost 5 diamond.dag
.EE
.UNINDENT
.UNINDENT
.sp
To run two DAGs, each of which is set up to run in its own directory:
.INDENT 0.0
.INDENT 3.5
.sp
.EX
$ condor_submit_dag \-usedagdir dag1/diamond1.dag dag2/diamond2.dag
.EE
.UNINDENT
.UNINDENT
.SH AUTHOR
HTCondor Team
.SH COPYRIGHT
1990-2024, Center for High Throughput Computing, Computer Sciences Department, University of Wisconsin-Madison, Madison, WI, US. Licensed under the Apache License, Version 2.0.
.\" Generated by docutils manpage writer.
.