.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "CONDOR_SUBMIT" "1" "Apr 01, 2024" "" "HTCondor Manual" .SH NAME condor_submit \- HTCondor Manual .sp Queue jobs for execution under HTCondor .SH SYNOPSIS .sp \fBcondor_submit\fP [\fB\-terse\fP ] [\fB\-verbose\fP ] [\fB\-unused\fP ] [\fB\-file\fP \fIsubmit_file\fP] [\fB\-name\fP \fIschedd_name\fP] [\fB\-remote\fP \fIschedd_name\fP] [\fB\-addr\fP \fI\fP] [\fB\-pool\fP \fIpool_name\fP] [\fB\-disable\fP ] [\fB\-password\fP \fIpassphrase\fP] [\fB\-debug\fP ] [\fB\-append\fP \fIcommand\fP \fB\&...\fP][\fB\-batch\-name\fP \fIbatch_name\fP] [\fB\-spool\fP ] [\fB\-dump\fP \fIfilename\fP] [\fB\-interactive\fP ] [\fB\-factory\fP ] [\fB\-allow\-crlf\-script\fP ] [\fB\-dry\-run\fP ] [\fB\-maxjobs\fP \fInumber\-of\-jobs\fP] [\fB\-single\-cluster\fP ] [\fB=\fP ] [\fIsubmit description file\fP ] [\fB\-queue\fP \fIqueue_arguments\fP] .SH DESCRIPTION .sp \fIcondor_submit\fP is the program for submitting jobs for execution under HTCondor. \fIcondor_submit\fP requires one or more submit description commands to direct the queuing of jobs. These commands may come from a file, standard input, the command line, or from some combination of these. One submit description may contain specifications for the queuing of many HTCondor jobs at once. A single invocation of \fIcondor_submit\fP may cause one or more clusters. A cluster is a set of jobs specified in the submit description between \fI\%queue\fP commands for which the executable is not changed. It is advantageous to submit multiple jobs as a single cluster because the schedd uses much less memory to hold the jobs. .sp Multiple clusters may be specified within a single submit description. Each cluster must specify a single executable. .sp The job ClassAd attribute \fBClusterId\fP identifies a cluster. .sp The \fIsubmit description file\fP argument is the path and file name of the submit description file. If this optional argument is the dash character (\fB\-\fP), then the commands are taken from standard input. If \fB\-\fP is specified for the \fIsubmit description file\fP, \fB\-verbose\fP is implied; this can be overridden by specifying \fB\-terse\fP\&. .sp If no \fIsubmit description file\fP argument is given, and no \fI\-queue\fP argument is given, commands are taken automatically from standard input. .sp Note that submission of jobs from a Windows machine requires a stashed password to allow HTCondor to impersonate the user submitting the job. To stash a password, use the \fIcondor_store_cred\fP command. See the manual page for details. .sp For lengthy lines within the submit description file, the backslash (\e) is a line continuation character. Placing the backslash at the end of a line causes the current line\(aqs command to be continued with the next line of the file. Submit description files may contain comments. A comment is any line beginning with a pound character (#). .SH OPTIONS .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP \fB\-terse\fP Terse output \- display JobId ranges only. .TP \fB\-verbose\fP Verbose output \- display the created job ClassAd .TP \fB\-unused\fP As a default, causes no warnings to be issued about user\-defined macros not being used within the submit description file. The meaning reverses (toggles) when the configuration variable \fI\%WARN_ON_UNUSED_SUBMIT_FILE_MACROS\fP is set to the non default value of \fBFalse\fP\&. Printing the warnings can help identify spelling errors of submit description file commands. The warnings are sent to stderr. .TP \fB\-file\fP \fIsubmit_file\fP Use \fIsubmit_file\fP as the submit description file. This is equivalent to providing \fIsubmit_file\fP as an argument without the preceding \fI\-file\fP\&. .TP \fB\-name\fP \fIschedd_name\fP Submit to the specified \fIcondor_schedd\fP\&. Use this option to submit to a \fIcondor_schedd\fP other than the default local one. \fIschedd_name\fP is the value of the \fBName\fP ClassAd attribute on the machine where the \fIcondor_schedd\fP daemon runs. .TP \fB\-remote\fP \fIschedd_name\fP Submit to the specified \fIcondor_schedd\fP, spooling all required input files over the network connection. \fIschedd_name\fP is the value of the \fBName\fP ClassAd attribute on the machine where the \fIcondor_schedd\fP daemon runs. This option is equivalent to using both \fB\-name\fP and \fB\-spool\fP\&. .TP \fB\-addr\fP \fI\fP Submit to the \fIcondor_schedd\fP at the IP address and port given by the sinful string argument \fI\fP\&. .TP \fB\-pool\fP \fIpool_name\fP Look in the specified pool for the \fIcondor_schedd\fP to submit to. This option is used with \fB\-name\fP or \fB\-remote\fP\&. .TP \fB\-disable\fP Disable file permission checks when submitting a job for read permissions on all input files, such as those defined by commands \fI\%input\fP and \fI\%transfer_input_files\fP, as well as write permission to output files, such as a log file defined by \fI\%log\fP and output files defined with \fI\%output\fP or \fI\%transfer_output_files\fP\&. .TP \fB\-debug\fP Cause debugging information to be sent to \fBstderr\fP, based on the value of the configuration variable \fI\%TOOL_DEBUG\fP\&. .TP \fB\-append\fP \fIcommand\fP Augment the commands in the submit description file with the given \fIcommand\fP\&. This command will be considered to immediately precede the \fBqueue\fP command within the submit description file, and come after all other previous commands. If the \fIcommand\fP specifies a \fBqueue\fP command, as in the example .sp \fBcondor_submit mysubmitfile \-append \(dqqueue input in A, B, C\(dq\fP .sp then the entire \fB\-append\fP command line option and its arguments are converted to .sp \fBcondor_submit mysubmitfile \-queue input in A, B, C\fP .sp The submit description file is not modified. Multiple commands are specified by using the \fB\-append\fP option multiple times. Each new command is given in a separate \fB\-append\fP option. Commands with spaces in them will need to be enclosed in double quote marks. .TP \fB\-batch\-name\fP \fIbatch_name\fP Set the batch name for this submit. The batch name is displayed by \fIcondor_q\fP \fB\-batch\fP\&. It is intended for use by users to give meaningful names to their jobs and to influence how \fIcondor_q\fP groups jobs for display. Use of this argument takes precedence over a batch name specified in the submit description file itself. .TP \fB\-spool\fP Spool all required input files, job event log, and proxy over the connection to the \fIcondor_schedd\fP\&. After submission, modify local copies of the files without affecting your jobs. Any output files for completed jobs need to be retrieved with \fIcondor_transfer_data\fP\&. .TP \fB\-dump\fP \fIfilename\fP Sends all ClassAds to the specified file, instead of to the \fIcondor_schedd\fP\&. .TP \fB\-interactive\fP Indicates that the user wants to run an interactive shell on an execute machine in the pool. This is equivalent to creating a submit description file of a vanilla universe sleep job, and then running \fIcondor_ssh_to_job\fP by hand. Without any additional arguments, \fIcondor_submit\fP with the \-interactive flag creates a dummy vanilla universe job that sleeps, submits it to the local scheduler, waits for the job to run, and then launches \fIcondor_ssh_to_job\fP to run a shell. If the user would like to run the shell on a machine that matches a particular \fI\%requirements\fP expression, the submit description file is specified, and it will contain the expression. Note that all policy expressions specified in the submit description file are honored, but any \fI\%executable\fP or \fI\%universe\fP commands are overwritten to be sleep and vanilla. The job ClassAd attribute \fBInteractiveJob\fP is set to \fBTrue\fP to identify interactive jobs for \fIcondor_startd\fP policy usage. .TP \fB\-factory\fP Sends all of the jobs as a late materialization job factory. A job factory consists of a single cluster classad and a digest containing the submit commands necessary to describe the differences between jobs. If the \fBQueue\fP statement has itemdata, then the itemdata will be sent. Using this option is equivalent to using the \fI\%max_materialize\fP submit command. .TP \fB\-allow\-crlf\-script\fP Changes the check for an invalid line ending on the executable script\(aqs \fB#!\fP line from an ERROR to a WARNING. The \fB#!\fP line will be ignored by Windows, so it won\(aqt matter if it is invalid; but Unix and Linux will not run a script that has a Windows/DOS line ending on the first line of the script. So \fIcondor_submit\fP will not allow such a script to be submitted as the job\(aqs executable unless this option is supplied. .TP \fB\-dry\-run\fP \fIfile\fP Parse the submit description file, sending the resulting job ClassAd to the file given by \fIfile\fP, but do not submit the job(s). This permits observation of the job specification, and it facilitates debugging the submit description file contents. If \fIfile\fP is \fB\-\fP, the output is written to \fBstdout\fP\&. .TP \fB\-maxjobs\fP \fInumber\-of\-jobs\fP If the total number of jobs specified by the submit description file is more than the integer value given by \fInumber\-of\-jobs\fP, then no jobs are submitted for execution and an error message is generated. A 0 or negative value for the \fInumber\-of\-jobs\fP causes no limit to be imposed. .TP \fB\-single\-cluster\fP If the jobs specified by the submit description file causes more than a single cluster value to be assigned, then no jobs are submitted for execution and an error message is generated. .TP \fB=\fP Defines a submit command or submit variable with a value, and parses it as if it was placed at the beginning of the submit description file. The submit description file is not changed. To correctly parse the \fIcondor_submit\fP command line, this option must be specified without white space characters before and after the equals sign (\fB=\fP), or the entire option must be surrounded by double quote marks. .TP \fB\-queue\fP \fIqueue_arguments\fP A command line specification of how many jobs to queue, which is only permitted if the submit description file does not have a \fBqueue\fP command. The \fIqueue_arguments\fP are the same as may be within a submit description file. The parsing of the \fIqueue_arguments\fP finishes at the end of the line or when a dash character (\fB\-\fP) is encountered. Therefore, its best placement within the command line will be at the end of the command line. .sp On a Unix command line, the shell expands file globs before parsing occurs. .UNINDENT .UNINDENT .UNINDENT .SH SUBMIT DESCRIPTION FILE COMMANDS .sp .sp Note: more information on submitting HTCondor jobs can be found here: \fI\%Submitting a Job\fP\&. .sp As of version 8.5.6, the \fIcondor_submit\fP language supports multi\-line values in commands. The syntax is the same as the configuration language (see more details here: admin\-manual/introduction\-to\-configuration:multi\-line values). .sp Each submit description file describes one or more clusters of jobs to be placed in the HTCondor execution pool. All jobs in a cluster must share the same executable, but they may have different input and output files, and different program arguments. The submit description file is generally the last command\-line argument to \fIcondor_submit\fP\&. If the submit description file argument is omitted, \fIcondor_submit\fP will read the submit description from standard input. .sp The submit description file must contain at least one \fIexecutable\fP command and at least one \fIqueue\fP command. All of the other commands have default actions. .sp \fBNote that a submit file that contains more than one executable command will produce multiple clusters when submitted. This is not generally recommended, and is not allowed for submit files that are run as DAG node jobs by condor_dagman.\fP .sp The commands which can appear in the submit description file are numerous. They are listed here in alphabetical order by category. .sp BASIC COMMANDS .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B arguments = List of arguments to be supplied to the executable as part of the command line. .sp In the \fBjava\fP universe, the first argument must be the name of the class containing \fBmain\fP\&. .sp There are two permissible formats for specifying arguments, identified as the old syntax and the new syntax. The old syntax supports white space characters within arguments only in special circumstances; when used, the command line arguments are represented in the job ClassAd attribute \fBArgs\fP\&. The new syntax supports uniform quoting of white space characters within arguments; when used, the command line arguments are represented in the job ClassAd attribute \fBArguments\fP\&. .sp \fBOld Syntax\fP .sp In the old syntax, individual command line arguments are delimited (separated) by space characters. To allow a double quote mark in an argument, it is escaped with a backslash; that is, the two character sequence \e\(dq becomes a single double quote mark within an argument. .sp Further interpretation of the argument string differs depending on the operating system. On Windows, the entire argument string is passed verbatim (other than the backslash in front of double quote marks) to the Windows application. Most Windows applications will allow spaces within an argument value by surrounding the argument with double quotes marks. In all other cases, there is no further interpretation of the arguments. .sp Example: .INDENT 7.0 .INDENT 3.5 .sp .EX arguments = one \e\(dqtwo\e\(dq \(aqthree\(aq .EE .UNINDENT .UNINDENT .sp Produces in Unix vanilla universe: .INDENT 7.0 .INDENT 3.5 .sp .EX argument 1: one argument 2: \(dqtwo\(dq argument 3: \(aqthree\(aq .EE .UNINDENT .UNINDENT .sp \fBNew Syntax\fP .sp Here are the rules for using the new syntax: .INDENT 7.0 .IP 1. 3 The entire string representing the command line arguments is surrounded by double quote marks. This permits the white space characters of spaces and tabs to potentially be embedded within a single argument. Putting the double quote mark within the arguments is accomplished by escaping it with another double quote mark. .IP 2. 3 The white space characters of spaces or tabs delimit arguments. .IP 3. 3 To embed white space characters of spaces or tabs within a single argument, surround the entire argument with single quote marks. .IP 4. 3 To insert a literal single quote mark, escape it within an argument already delimited by single quote marks by adding another single quote mark. .UNINDENT .sp Example: .INDENT 7.0 .INDENT 3.5 .sp .EX arguments = \(dq3 simple arguments\(dq .EE .UNINDENT .UNINDENT .sp Produces: .INDENT 7.0 .INDENT 3.5 .sp .EX argument 1: 3 argument 2: simple argument 3: arguments .EE .UNINDENT .UNINDENT .sp Another example: .INDENT 7.0 .INDENT 3.5 .sp .EX arguments = \(dqone \(aqtwo with spaces\(aq 3\(dq .EE .UNINDENT .UNINDENT .sp Produces: .INDENT 7.0 .INDENT 3.5 .sp .EX argument 1: one argument 2: two with spaces argument 3: 3 .EE .UNINDENT .UNINDENT .sp And yet another example: .INDENT 7.0 .INDENT 3.5 .sp .EX arguments = \(dqone \(dq\(dqtwo\(dq\(dq \(aqspacey \(aq\(aqquoted\(aq\(aq argument\(aq\(dq .EE .UNINDENT .UNINDENT .sp Produces: .INDENT 7.0 .INDENT 3.5 .sp .EX argument 1: one argument 2: \(dqtwo\(dq argument 3: spacey \(aqquoted\(aq argument .EE .UNINDENT .UNINDENT .sp Notice that in the new syntax, the backslash has no special meaning. This is for the convenience of Windows users. .sp .TP .B environment = List of environment variables. .sp There are two different formats for specifying the environment variables: the old format and the new format. The old format is retained for backward\-compatibility. It suffers from a platform\-dependent syntax and the inability to insert some special characters into the environment. .sp The new syntax for specifying environment values: .INDENT 7.0 .IP 1. 3 Put double quote marks around the entire argument string. This distinguishes the new syntax from the old. The old syntax does not have double quote marks around it. Any literal double quote marks within the string must be escaped by repeating the double quote mark. .IP 2. 3 Each environment entry has the form .INDENT 3.0 .INDENT 3.5 .sp .EX = .EE .UNINDENT .UNINDENT .IP 3. 3 Use white space (space or tab characters) to separate environment entries. .IP 4. 3 To put any white space in an environment entry, surround the space and as much of the surrounding entry as desired with single quote marks. .IP 5. 3 To insert a literal single quote mark, repeat the single quote mark anywhere inside of a section surrounded by single quote marks. .UNINDENT .sp Example: .INDENT 7.0 .INDENT 3.5 .sp .EX environment = \(dqone=1 two=\(dq\(dq2\(dq\(dq three=\(aqspacey \(aq\(aqquoted\(aq\(aq value\(aq\(dq .EE .UNINDENT .UNINDENT .sp Produces the following environment entries: .INDENT 7.0 .INDENT 3.5 .sp .EX one=1 two=\(dq2\(dq three=spacey \(aqquoted\(aq value .EE .UNINDENT .UNINDENT .sp Under the old syntax, there are no double quote marks surrounding the environment specification. Each environment entry remains of the form .INDENT 7.0 .INDENT 3.5 .sp .EX = .EE .UNINDENT .UNINDENT .sp Under Unix, list multiple environment entries by separating them with a semicolon (;). Under Windows, separate multiple entries with a vertical bar (|). There is no way to insert a literal semicolon under Unix or a literal vertical bar under Windows. Note that spaces are accepted, but rarely desired, characters within parameter names and values, because they are treated as literal characters, not separators or ignored white space. Place spaces within the parameter list only if required. .sp A Unix example: .INDENT 7.0 .INDENT 3.5 .sp .EX environment = one=1;two=2;three=\(dqquotes have no \(aqspecial\(aq meaning\(dq .EE .UNINDENT .UNINDENT .sp This produces the following: .INDENT 7.0 .INDENT 3.5 .sp .EX one=1 two=2 three=\(dqquotes have no \(aqspecial\(aq meaning\(dq .EE .UNINDENT .UNINDENT .sp If the environment is set with the \fBenvironment\fP command and \fBgetenv\fP is also set, values specified with \fBenvironment\fP override values in the submitter\(aqs environment (regardless of the order of the \fBenvironment\fP and \fBgetenv\fP commands). .TP .B error = A path and file name used by HTCondor to capture any error messages the program would normally write to the screen (that is, this file becomes \fBstderr\fP). A path is given with respect to the file system of the machine on which the job is submitted. The file is written (by the job) in the remote scratch directory of the machine where the job is executed. When the job exits, the resulting file is transferred back to the machine where the job was submitted, and the path is utilized for file placement. If you specify a relative path, the final path will be relative to the job\(aqs initial working directory, and HTCondor will create directories as necessary to transfer the file. If not specified, the default value of \fB/dev/null\fP is used for submission to a Unix machine. If not specified, error messages are ignored for submission to a Windows machine. More than one job should not use the same error file, since this will cause one job to overwrite the errors of another. If HTCondor detects that the error and output files for a job are the same, it will run the job such that the output and error data is merged. .TP .B executable = An optional path and a required file name of the executable file for this job cluster. Only one \fBexecutable\fP command within a submit description file is guaranteed to work properly. More than one often works. .sp If no path or a relative path is used, then the executable file is presumed to be relative to the current working directory of the user as the \fIcondor_submit\fP command is issued. .TP .B batch_name = Set the batch name for this submit. The batch name is displayed by \fIcondor_q\fP \fB\-batch\fP\&. It is intended for use by users to give meaningful names to their jobs and to influence how \fIcondor_q\fP groups jobs for display. This value in a submit file can be overridden by specifying the \fB\-batch\-name\fP argument on the \fIcondor_submit\fP command line. .TP .B getenv = < | True | False> If \fBgetenv\fP is set to \fBTrue\fP, then \fIcondor_submit\fP will copy all of the user\(aqs current shell environment variables at the time of job submission into the job ClassAd. The job will therefore execute with the same set of environment variables that the user had at submit time. Defaults to \fBFalse\fP\&. A wholesale import of the user\(aqs environment is very likely to lead to problems executing the job on a remote machine unless there is a shared file system for users\(aq home directories between the access point and execute machine. So rather than setting getenv to \fBTrue\fP, it is much better to set it to a list of environment variables to import. .sp Matchlist is a comma, semicolon or space separated list of environment variable names and name patterns that match or reject names. Matchlist members are matched case\-insensitively to each name in the environment and those that match are imported. Matchlist members can contain \fB*\fP as wildcard character which matches anything at that position. Members can have two \fB*\fP characters if one of them is at the end. Members can be prefixed with \fB!\fP to force a matching environment variable to not be imported. The order of members in the Matchlist has no effect on the result. \fBgetenv = true\fP is equivalent to \fBgetenv = *\fP .sp Prior to HTCondor 8.9.7 \fBgetenv\fP allows only \fBTrue\fP or \fBFalse\fP as values. .sp Examples: .INDENT 7.0 .INDENT 3.5 .sp .EX # import everything except PATH and INCLUDE (also path, include and other case\-variants) getenv = !PATH, !INCLUDE # import everything with CUDA in the name getenv = *cuda* # Import every environment variable that starts with P or Q, except PATH getenv = !path, P*, Q* .EE .UNINDENT .UNINDENT .sp If the environment is set with the \fBenvironment\fP command and \fBgetenv\fP is also set, values specified with \fBenvironment\fP override values in the submitter\(aqs environment (regardless of the order of the \fBenvironment\fP and \fBgetenv\fP commands). .TP .B input = HTCondor assumes that its jobs are long\-running, and that the user will not wait at the terminal for their completion. Because of this, the standard files which normally access the terminal, (\fBstdin\fP, \fBstdout\fP, and \fBstderr\fP), must refer to files. Thus, the file name specified with \fBinput\fP should contain any keyboard input the program requires (that is, this file becomes \fBstdin\fP). A path is given with respect to the file system of the machine on which the job is submitted. The file is transferred before execution to the remote scratch directory of the machine where the job is executed. If not specified, the default value of \fB/dev/null\fP is used for submission to a Unix machine. If not specified, input is ignored for submission to a Windows machine. .sp Note that this command does not refer to the command\-line arguments of the program. The command\-line arguments are specified by the \fBarguments\fP command. .TP .B log = Use \fBlog\fP to specify a file name where HTCondor will write a log file of what is happening with this job cluster, called a job event log. For example, HTCondor will place a log entry into this file when and where the job begins running, when it transfers files, if the job is evicted, and when the job completes. Most users find specifying a \fBlog\fP file to be handy; its use is recommended. If no \fBlog\fP entry is specified, HTCondor does not create a log for this cluster. If a relative path is specified, it is relative to the current working directory as the job is submitted or the directory specified by submit command \fBinitialdir\fP on the access point. .sp .TP .B notification = Owners of HTCondor jobs are notified by e\-mail when certain events occur. If defined by \fIAlways\fP or \fIComplete\fP, the owner will be notified when the job terminates. If defined by \fIError\fP, the owner will only be notified if the job terminates abnormally, (as defined by \fBJobSuccessExitCode\fP, if defined) or if the job is placed on hold because of a failure, and not by user request. If defined by \fINever\fP (the default), the owner will not receive e\-mail, regardless to what happens to the job. The HTCondor User\(aqs manual documents statistics included in the e\-mail. .TP .B notify_user = Used to specify the e\-mail address to use when HTCondor sends e\-mail about a job. If not specified, HTCondor defaults to using the e\-mail address defined by .INDENT 7.0 .INDENT 3.5 .sp .EX job\-owner@UID_DOMAIN .EE .UNINDENT .UNINDENT .sp where the configuration variable \fI\%UID_DOMAIN\fP is specified by the HTCondor site administrator. If \fI\%UID_DOMAIN\fP has not been specified, HTCondor sends the e\-mail to: .INDENT 7.0 .INDENT 3.5 .sp .EX job\-owner@submit\-machine\-name .EE .UNINDENT .UNINDENT .TP .B output = The \fBoutput\fP file captures any information the program would ordinarily write to the screen (that is, this file becomes \fBstdout\fP). A path is given with respect to the file system of the machine on which the job is submitted. The file is written (by the job) in the remote scratch directory of the machine where the job is executed. When the job exits, the resulting file is transferred back to the machine where the job was submitted, and the path is utilized for file placement. If you specify a relative path, the final path will be relative to the job\(aqs initial working directory, and HTCondor will create directories as necessary to transfer the file. If not specified, the default value of \fB/dev/null\fP is used for submission to a Unix machine. If not specified, output is ignored for submission to a Windows machine. Multiple jobs should not use the same output file, since this will cause one job to overwrite the output of another. If HTCondor detects that the error and output files for a job are the same, it will run the job such that the output and error data is merged. .sp Note that if a program explicitly opens and writes to a file, that file should not be specified as the \fBoutput\fP file. .TP .B priority = An HTCondor job priority can be any integer, with 0 being the default. Jobs with higher numerical priority will run before jobs with lower numerical priority. Note that this priority is on a per user basis. One user with many jobs may use this command to order his/her own jobs, and this will have no effect on whether or not these jobs will run ahead of another user\(aqs jobs. .sp Note that the priority setting in an HTCondor submit file will be overridden by \fIcondor_dagman\fP if the submit file is used for a node in a DAG, and the priority of the node within the DAG is non\-zero (see automated\-workflows/dagman\-priorities:Setting Priorities for Nodes for more details). .TP queue [\fB\fP ] Places zero or more copies of the job into the HTCondor queue. .TP .B queue [\fB\fP ] [\fB\fP ] \fBin\fP [\fBslice\fP ] \fB\fP Places zero or more copies of the job in the queue based on items in a \fB\fP .TP .B queue [\fB\fP ] [\fB\fP ] \fBmatching\fP [\fBfiles | dirs\fP ] [\fBslice\fP ] \fB\fP] Places zero or more copies of the job in the queue based on files that match a \fB\fP .TP .B queue [\fB\fP ] [\fB\fP ] \fBfrom\fP [\fBslice\fP ] \fB | \fP] Places zero or more copies of the job in the queue based on lines from the submit file or from \fB\fP .sp The optional argument \fI\fP specifies how many times to repeat the job submission for a given set of arguments. It may be an integer or an expression that evaluates to an integer, and it defaults to 1. All but the first form of this command are various ways of specifying a list of items. When these forms are used \fI\fP jobs will be queued for each item in the list. The \fIin\fP, \fImatching\fP and \fIfrom\fP keyword indicates how the list will be specified. .INDENT 7.0 .IP \(bu 2 \fIin\fP The list of items is an explicit comma and/or space separated \fB\fP\&. If the \fB\fP begins with an open paren, and the close paren is not on the same line as the open, then the list continues until a line that begins with a close paren is read from the submit file. .IP \(bu 2 \fImatching\fP Each item in the \fB\fP will be matched against the names of files and directories relative to the current directory, the set of matching names is the resulting list of items. .INDENT 2.0 .IP \(bu 2 \fIfiles\fP Only filenames will matched. .IP \(bu 2 \fIdirs\fP Only directory names will be matched. .UNINDENT .IP \(bu 2 \fIfrom\fP \fB | \fP Each line from \fB\fP or \fB\fP is a single item, this allows for multiple variables to be set for each item. Lines from \fB\fP or \fB\fP will be split on comma and/or space until there are values for each of the variables specified in \fB\fP\&. The last variable will contain the remainder of the line. When the \fB\fP form is used, the list continues until the first line that begins with a close paren, and lines beginning with pound sign (\(aq#\(aq) will be skipped. When using the \fB\fP form, if the \fB\fP ends with |, then it will be executed as a script whatever the script writes to \fBstdout\fP will be the list of items. .UNINDENT .sp The optional argument \fI\fP or \fI\fP is the name or names of of variables that will be set to the value of the current item when queuing the job. If no \fI\fP is specified the variable ITEM will be used. Leading and trailing whitespace be trimmed. The optional argument \fI\fP is a python style slice selecting only some of the items in the list of items. Negative step values are not supported. .sp A submit file may contain more than one \fBqueue\fP statement, and if desired, any commands may be placed between subsequent \fBqueue\fP commands, such as new \fBinput\fP, \fBoutput\fP, \fBerror\fP, \fBinitialdir\fP, or \fBarguments\fP commands. This is handy when submitting multiple runs into one cluster with one submit description file. .TP .B universe = Specifies which HTCondor universe to use when running this job. The HTCondor universe specifies an HTCondor execution environment. .sp The \fBvanilla\fP universe is the default (except where the configuration variable \fI\%DEFAULT_UNIVERSE\fP defines it otherwise). .sp The \fBscheduler\fP universe is for a job that is to run on the machine where the job is submitted. This universe is intended for a job that acts as a metascheduler and will not be preempted. .sp The \fBlocal\fP universe is for a job that is to run on the machine where the job is submitted. This universe runs the job immediately and will not preempt the job. .sp The \fBgrid\fP universe forwards the job to an external job management system. Further specification of the \fBgrid\fP universe is done with the \fBgrid_resource\fP command. .sp The \fBjava\fP universe is for programs written to the Java Virtual Machine. .sp The \fBvm\fP universe facilitates the execution of a virtual machine. .sp The \fBparallel\fP universe is for parallel jobs (e.g. MPI) that require multiple machines in order to run. .sp The \fBdocker\fP universe runs a docker container as an HTCondor job. .TP .B max_materialize = Submit jobs as a late materialization factory and instruct the \fIcondor_schedd\fP to keep the given number of jobs materialized. Use this option to reduce the load on the \fIcondor_schedd\fP when submitting a large number of jobs. The limit can be an expression but it must evaluate to a constant at submit time. A limit less than 1 will be treated as unlimited. The \fIcondor_schedd\fP can be configured to have a materialization limit as well, the lower of the two limits will be used. (see users\-manual/submitting\-a\-job:submitting lots of jobs for more details). .TP .B max_idle = Submit jobs as a late materialization factory and instruct the \fIcondor_schedd\fP to keep the given number of non\-running jobs materialized. Use this option to reduce the load on the \fIcondor_schedd\fP when submitting a large number of jobs. The limit may be an expression but it must evaluate to a constant at submit time. Jobs in the Held state are considered to be Idle for this limit. A limit of less than 1 will prevent jobs from being materialized although the factory will still be submitted to the \fIcondor_schedd\fP\&. (see users\-manual/submitting\-a\-job:submitting lots of jobs for more details). .UNINDENT .UNINDENT .UNINDENT .sp COMMANDS FOR MATCHMAKING .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B rank = A ClassAd Floating\-Point expression that states how to rank machines which have already met the requirements expression. Essentially, rank expresses preference. A higher numeric value equals better rank. HTCondor will give the job the machine with the highest rank. For example, .INDENT 7.0 .INDENT 3.5 .sp .EX request_memory = max({60, Target.TotalSlotMemory}) rank = Memory .EE .UNINDENT .UNINDENT .sp asks HTCondor to find all available machines with more than 60 megabytes of memory and give to the job the machine with the most amount of memory. The HTCondor User\(aqs Manual contains complete information on the syntax and available attributes that can be used in the ClassAd expression. .TP .B request_cpus = A requested number of CPUs (cores). If not specified, the number requested will be 1. If specified, the expression .INDENT 7.0 .INDENT 3.5 .sp .EX && (RequestCpus <= Target.Cpus) .EE .UNINDENT .UNINDENT .sp is appended to the \fBrequirements\fP expression for the job. .sp For pools that enable dynamic \fIcondor_startd\fP provisioning, specifies the minimum number of CPUs requested for this job, resulting in a dynamic slot being created with this many cores. .TP .B request_disk = The requested amount of disk space in KiB requested for this job. If not specified, it will be set to the job ClassAd attribute \fBDiskUsage\fP\&. The expression .INDENT 7.0 .INDENT 3.5 .sp .EX && (RequestDisk <= Target.Disk) .EE .UNINDENT .UNINDENT .sp is appended to the \fBrequirements\fP expression for the job. .sp For pools that enable dynamic \fIcondor_startd\fP provisioning, a dynamic slot will be created with at least this much disk space. .sp Characters may be appended to a numerical value to indicate units. \fBK\fP or \fBKB\fP indicates KiB, 2\s-2\u10\d\s0 numbers of bytes. \fBM\fP or \fBMB\fP indicates MiB, 2\s-2\u20\d\s0 numbers of bytes. \fBG\fP or \fBGB\fP indicates GiB, 2\s-2\u30\d\s0 numbers of bytes. \fBT\fP or \fBTB\fP indicates TiB, 2\s-2\u40\d\s0 numbers of bytes. .TP .B request_gpus = A requested number of GPUs. If not specified, no GPUs will be requested. If specified and \fBrequire_gpus\fP is not also specified, the expression .INDENT 7.0 .INDENT 3.5 .sp .EX && (Target.GPUs >= RequestGPUs) .EE .UNINDENT .UNINDENT .sp is appended to the \fBrequirements\fP expression for the job. .sp For pools that enable dynamic \fIcondor_startd\fP provisioning, specifies the minimum number of GPUs requested for this job, resulting in a dynamic slot being created with this many GPUs. .TP .B require_gpus = A constraint on the properties of GPUs when used with a non\-zero \fBrequest_gpus\fP value. If not specified, no constraint on GPUs will be added to the job. If specified and \fBrequest_gpus\fP is non\-zero, the expression .INDENT 7.0 .INDENT 3.5 .sp .EX && (countMatches(MY.RequireGPUs, TARGET.AvailableGPUs) >= RequestGPUs) .EE .UNINDENT .UNINDENT .sp is appended to the \fBrequirements\fP expression for the job. This expression cannot be evaluated by HTCondor prior to version 9.8.0. A warning to this will effect will be printed when \fIcondor_submit\fP detects this condition. .sp For pools that enable dynamic \fIcondor_startd\fP provisioning and are at least version 9.8.0, the constraint will be tested against the properties of AvailableGPUs and only those that match will be assigned to the dynamic slot. .TP .B request_memory = The required amount of memory in MiB that this job needs to avoid excessive swapping. If not specified and the submit command \fBvm_memory\fP is specified, then the value specified for \fBvm_memory\fP defines \fBrequest_memory\fP , If neither \fBrequest_memory\fP nor \fBvm_memory\fP is specified, the value is set by the configuration variable \fI\%JOB_DEFAULT_REQUESTMEMORY\fP The actual amount of memory used by a job is represented by the job ClassAd attribute \fBMemoryUsage\fP\&. .sp For pools that enable dynamic \fIcondor_startd\fP provisioning, a dynamic slot will be created with at least this much RAM. .sp The expression .INDENT 7.0 .INDENT 3.5 .sp .EX && (RequestMemory <= Target.Memory) .EE .UNINDENT .UNINDENT .sp is appended to the \fBrequirements\fP expression for the job. .sp Characters may be appended to a numerical value to indicate units. \fBK\fP or \fBKB\fP indicates KiB, 2\s-2\u10\d\s0 numbers of bytes. \fBM\fP or \fBMB\fP indicates MiB, 2\s-2\u20\d\s0 numbers of bytes. \fBG\fP or \fBGB\fP indicates GiB, 2\s-2\u30\d\s0 numbers of bytes. \fBT\fP or \fBTB\fP indicates TiB, 2\s-2\u40\d\s0 numbers of bytes. .sp request_GPUs .TP .B request_ = The required amount of the custom machine resource identified by \fB\fP that this job needs. The custom machine resource is defined in the machine\(aqs configuration. Machines that have available GPUs will define \fB\fP to be \fBGPUs\fP\&. \fB\fP must be at least two characters, and must not begin with \fB_\fP\&. If \fB\fP is either \fBCpu\fP or \fBGpu\fP a warning will be printed since these are common typos. .TP .B cuda_version = The version of the CUDA runtime, if any, used or required by this job, specified as \fB.\fP (for example, \fB9.1\fP). If the minor version number is zero, you may specify only the major version number. A single version number of 1000 or higher is assumed to be the integer\-coded version number (\fBmajor * 1000 + (minor % 100)\fP). .sp This does \fInot\fP arrange for the CUDA runtime to be present, only for the job to run on a machine whose driver supports the specified version. .TP .B requirements = The requirements command is a boolean ClassAd expression which uses C\-like operators. In order for any job in this cluster to run on a given machine, this requirements expression must evaluate to true on the given machine. .sp For scheduler and local universe jobs, the requirements expression is evaluated against the \fBScheduler\fP ClassAd which represents the the \fIcondor_schedd\fP daemon running on the access point, rather than a remote machine. Like all commands in the submit description file, if multiple requirements commands are present, all but the last one are ignored. By default, \fIcondor_submit\fP appends the following clauses to the requirements expression: .INDENT 7.0 .IP 1. 3 Arch and OpSys are set equal to the Arch and OpSys of the submit machine. In other words: unless you request otherwise, HTCondor will give your job machines with the same architecture and operating system version as the machine running \fIcondor_submit\fP\&. .IP 2. 3 Cpus >= RequestCpus, if the job ClassAd attribute \fBRequestCpus\fP is defined. .IP 3. 3 Disk >= RequestDisk, if the job ClassAd attribute \fBRequestDisk\fP is defined. Otherwise, Disk >= DiskUsage is appended to the requirements. The \fBDiskUsage\fP attribute is initialized to the size of the executable plus the size of any files specified in a \fBtransfer_input_files\fP command. It exists to ensure there is enough disk space on the target machine for HTCondor to copy over both the executable and needed input files. The \fBDiskUsage\fP attribute represents the maximum amount of total disk space required by the job in kilobytes. HTCondor automatically updates the \fBDiskUsage\fP attribute approximately every 20 minutes while the job runs with the amount of space being used by the job on the execute machine. .IP 4. 3 Memory >= RequestMemory, if the job ClassAd attribute \fBRequestMemory\fP is defined. .IP 5. 3 If Universe is set to Vanilla, FileSystemDomain is set equal to the access point\(aqs FileSystemDomain. .UNINDENT .sp View the requirements of a job which has already been submitted (along with everything else about the job ClassAd) with the command \fIcondor_q \-l\fP; see the command reference for \fI\%condor_q\fP\&. Also, see the HTCondor Users Manual for complete information on the syntax and available attributes that can be used in the ClassAd expression. .UNINDENT .UNINDENT .UNINDENT .sp FILE TRANSFER COMMANDS .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .INDENT 3.5 .UNINDENT .UNINDENT .INDENT 0.0 .TP .B dont_encrypt_input_files = < file1,file2,file... > A comma and/or space separated list of input files that are not to be network encrypted when transferred with the file transfer mechanism. Specification of files in this manner overrides configuration that would use encryption. Each input file must also be in the list given by \fBtransfer_input_files\fP\&. When a path to an input file or directory is specified, this specifies the path to the file on the submit side. A single wild card character (\fB*\fP) may be used in each file name. .sp .TP .B dont_encrypt_output_files = < file1,file2,file... > A comma and/or space separated list of output files that are not to be network encrypted when transferred back with the file transfer mechanism. Specification of files in this manner overrides configuration that would use encryption. The output file(s) must also either be in the list given by \fBtransfer_output_files\fP or be discovered and to be transferred back with the file transfer mechanism. When a path to an output file or directory is specified, this specifies the path to the file on the execute side. A single wild card character (\fB*\fP) may be used in each file name. .TP .B encrypt_execute_directory = Defaults to \fBFalse\fP\&. If set to \fBTrue\fP, HTCondor will encrypt the contents of the remote scratch directory of the machine where the job is executed. This encryption is transparent to the job itself, but ensures that files left behind on the local disk of the execute machine, perhaps due to a system crash, will remain private. In addition, \fIcondor_submit\fP will append to the job\(aqs \fBrequirements\fP expression .INDENT 7.0 .INDENT 3.5 .sp .EX && (TARGET.HasEncryptExecuteDirectory) .EE .UNINDENT .UNINDENT .sp to ensure the job is matched to a machine that is capable of encrypting the contents of the execute directory. This support is limited to Windows platforms that use the NTFS file system and Linux platforms with the \fIecryptfs\-utils\fP package installed. .sp .TP .B encrypt_input_files = < file1,file2,file... > A comma and/or space separated list of input files that are to be network encrypted when transferred with the file transfer mechanism. Specification of files in this manner overrides configuration that would not use encryption. Each input file must also be in the list given by \fBtransfer_input_files\fP\&. When a path to an input file or directory is specified, this specifies the path to the file on the submit side. A single wild card character (\fB*\fP) may be used in each file name. The method of encryption utilized will be as agreed upon in security negotiation; if that negotiation failed, then the file transfer mechanism must also fail for files to be network encrypted. .sp .TP .B encrypt_output_files = < file1,file2,file... > A comma and/or space separated list of output files that are to be network encrypted when transferred back with the file transfer mechanism. Specification of files in this manner overrides configuration that would not use encryption. The output file(s) must also either be in the list given by \fBtransfer_output_files\fP or be discovered and to be transferred back with the file transfer mechanism. When a path to an output file or directory is specified, this specifies the path to the file on the execute side. A single wild card character (\fB*\fP) may be used in each file name. The method of encryption utilized will be as agreed upon in security negotiation; if that negotiation failed, then the file transfer mechanism must also fail for files to be network encrypted. .TP .B erase_output_and_error_on_restart = < true | false> If false, and \fBwhen_to_transfer_output\fP is \fBON_EXIT_OR_EVICT\fP, HTCondor will append to the output and error logs rather than erase (truncate) them when the job restarts. .TP .B max_transfer_input_mb = This integer expression specifies the maximum allowed total size in MiB of the input files that are transferred for a job. This expression does not apply to grid universe or files transferred via file transfer plug\-ins. The expression may refer to attributes of the job. The special value \-1 indicates no limit. If not defined, the value set by configuration variable \fI\%MAX_TRANSFER_INPUT_MB\fP is used. If the observed size of all input files at submit time is larger than the limit, the job will be immediately placed on hold with a \fBHoldReasonCode\fP value of 32. If the job passes this initial test, but the size of the input files increases or the limit decreases so that the limit is violated, the job will be placed on hold at the time when the file transfer is attempted. .TP .B max_transfer_output_mb = This integer expression specifies the maximum allowed total size in MiB of the output files that are transferred for a job. This expression does not apply to grid universe or files transferred via file transfer plug\-ins. The expression may refer to attributes of the job. The special value \-1 indicates no limit. If not set, the value set by configuration variable \fI\%MAX_TRANSFER_OUTPUT_MB\fP is used. If the total size of the job\(aqs output files to be transferred is larger than the limit, the job will be placed on hold with a \fBHoldReasonCode\fP value of 33. The output will be transferred up to the point when the limit is hit, so some files may be fully transferred, some partially, and some not at all. .sp .TP .B output_destination = When present, defines a URL that specifies both a plug\-in and a destination for the transfer of the entire output sandbox or a subset of output files as specified by the submit command \fBtransfer_output_files\fP\&. The plug\-in does the transfer of files, and no files are sent back to the access point. The HTCondor Administrator\(aqs manual has full details. .TP .B should_transfer_files = The \fBshould_transfer_files\fP setting is used to define if HTCondor should transfer files to and from the remote machine where the job runs. The file transfer mechanism is used to run jobs on machines which do not have a shared file system with the submit machine. \fBshould_transfer_files\fP equal to \fIYES\fP will cause HTCondor to always transfer files for the job. \fINO\fP disables HTCondor\(aqs file transfer mechanism. \fIIF_NEEDED\fP will not transfer files for the job if it is matched with a resource in the same \fBFileSystemDomain\fP as the access point (and therefore, on a machine with the same shared file system). If the job is matched with a remote resource in a different \fBFileSystemDomain\fP, HTCondor will transfer the necessary files. .sp For more information about this and other settings related to transferring files, see the HTCondor User\(aqs manual section on the file transfer mechanism. .sp Note that \fBshould_transfer_files\fP is not supported for jobs submitted to the grid universe. .TP .B skip_filechecks = When \fBTrue\fP, file permission checks for the submitted job are disabled. When \fBFalse\fP, file permissions are checked; this is the behavior when this command is not present in the submit description file. File permissions are checked for read permissions on all input files, such as those defined by commands \fBinput\fP and \fBtransfer_input_files\fP, and for write permission to output files, such as a log file defined by \fBlog\fP and output files defined with \fBoutput\fP or \fBtransfer_output_files\fP\&. .TP .B stream_error = If \fBTrue\fP, then \fBstderr\fP is streamed back to the machine from which the job was submitted. If \fBFalse\fP, \fBstderr\fP is stored locally and transferred back when the job completes. This command is ignored if the job ClassAd attribute \fBTransferErr\fP is \fBFalse\fP\&. The default value is \fBFalse\fP\&. This command must be used in conjunction with \fBerror\fP, otherwise \fBstderr\fP will sent to \fB/dev/null\fP on Unix machines and ignored on Windows machines. .TP .B stream_input = If \fBTrue\fP, then \fBstdin\fP is streamed from the machine on which the job was submitted. The default value is \fBFalse\fP\&. The command is only relevant for jobs submitted to the vanilla or java universes, and it is ignored by the grid universe. This command must be used in conjunction with \fBinput\fP, otherwise \fBstdin\fP will be \fB/dev/null\fP on Unix machines and ignored on Windows machines. .TP .B stream_output = If \fBTrue\fP, then \fBstdout\fP is streamed back to the machine from which the job was submitted. If \fBFalse\fP, \fBstdout\fP is stored locally and transferred back when the job completes. This command is ignored if the job ClassAd attribute \fBTransferOut\fP is \fBFalse\fP\&. The default value is \fBFalse\fP\&. This command must be used in conjunction with \fBoutput\fP, otherwise \fBstdout\fP will sent to \fB/dev/null\fP on Unix machines and ignored on Windows machines. .TP .B transfer_executable = This command is applicable to jobs submitted to the grid and vanilla universes. If \fBtransfer_executable\fP is set to \fBFalse\fP, then HTCondor looks for the executable on the remote machine, and does not transfer the executable over. This is useful for an already pre\-staged executable; HTCondor behaves more like rsh. The default value is \fBTrue\fP\&. .TP .B transfer_input_files = < file1,file2,file... > A comma\-delimited list of all the files and directories to be transferred into the working directory for the job, before the job is started. By default, the file specified in the \fBexecutable\fP command and any file specified in the \fBinput\fP command (for example, \fBstdin\fP) are transferred. .sp When a path to an input file or directory is specified, this specifies the path to the file on the submit side. The file is placed in the job\(aqs temporary scratch directory on the execute side, and it is named using the base name of the original path. For example, \fB/path/to/input_file\fP becomes \fBinput_file\fP in the job\(aqs scratch directory. .sp When a directory is specified, the behavior depends on whether there is a trailing path separator character. When a directory is specified with a trailing path separator, it is as if each of the items within the directory were listed in the transfer list. Therefore, the contents are transferred, but the directory itself is not. When there is no trailing path separator, the directory itself is transferred with all of its contents inside it. On platforms such as Windows where the path separator is not a forward slash (/), a trailing forward slash is treated as equivalent to a trailing path separator. An example of an input directory specified with a trailing forward slash is \fBinput_data/\fP\&. .sp For grid universe jobs other than HTCondor\-C, the transfer of directories is not currently supported. .sp Symbolic links to files are transferred as the files they point to. Transfer of symbolic links to directories is not currently supported. .sp For vanilla and vm universe jobs only, a file may be specified by giving a URL, instead of a file name. The implementation for URL transfers requires both configuration and available plug\-in. .sp If you have a plugin which handles \fBhttps://\fP URLs (and HTCondor ships with one enabled), HTCondor supports pre\-signing S3 URLs. This allows you to specify S3 URLs for this command, for \fBtransfer_output_remaps\fP, and for \fBoutput_destination\fP\&. By pre\-signing the URLs on the submit node, HTCondor avoids transferring your S3 credentials to the execute node. You must specify \fBaws_access_key_id_file\fP and \fBaws_secret_access_key_file\fP; you may specify \fBaws_region\fP, if necessary; see below. To use the S3 service provided by AWS, use S3 URLs of the following forms: .INDENT 7.0 .INDENT 3.5 .sp .EX # For older buckets that aren\(aqt region\-specific. s3:/// # For newer, region\-specific buckets. s3://.s3..amazonaws.com/ .EE .UNINDENT .UNINDENT .sp To use other S3 services, where \fB\fP must contain a \fB\&.\fP: .INDENT 7.0 .INDENT 3.5 .sp .EX s3:/// # If necessary aws_region = .EE .UNINDENT .UNINDENT .sp You may specify the corresponding access key ID and secret access key with \fBs3_access_key_id_file\fP and \fBs3_secret_access_key_file\fP if you prefer (which may reduce confusion, if you\(aqre not using AWS). .sp If you must access S3 using temporary credentials, you may specify the temporary credentials using \fBaws_access_key_id_file\fP and \fBaws_secret_access_key_file\fP for the files containing the corresponding temporary token, and \fB+EC2SessionToken\fP for the file containing the session token. .sp Temporary credentials have a limited lifetime. If you are using S3 only to download input files, the job must start before the credentials expire. If you are using S3 to upload output files, the job must finish before the credentials expire. HTCondor does not know when the credentials will expire; if they do so before they are needed, file transfer will fail. .sp HTCondor does not presently support transferring entire buckets or directories from S3. .sp HTCondor supports Google Cloud Storage URLs \-\- \fBgs://\fP \-\- via Google\(aqs \(dqinteroperability\(dq API. You may specify \fBgs://\fP URLs as if they were \fBs3://\fP URLs, and they work the same way. You may specify the corresponding access key ID and secret access key with \fBgs_access_key_id_file\fP and \fBgs_secret_access_key_file\fP if you prefer (which may reduce confusion). .sp Note that (at present), you may not provide more than one set of credentials for \fBs3://\fP or \fBgs://\fP file transfer; this implies that all such URLs download from or upload to the same service. .TP .B transfer_output_files = < file1,file2,file... > This command forms an explicit list of output files and directories to be transferred back from the temporary working directory on the execute machine to the access point. If there are multiple files, they must be delimited with commas. Setting \fBtransfer_output_files\fP to the empty string (\(dq\(dq) means that no files are to be transferred. .sp For HTCondor\-C jobs and all other non\-grid universe jobs, if \fBtransfer_output_files\fP is not specified, HTCondor will automatically transfer back all files in the job\(aqs temporary working directory which have been modified or created by the job. Subdirectories are not scanned for output, so if output from subdirectories is desired, the output list must be explicitly specified. For grid universe jobs other than HTCondor\-C, desired output files must also be explicitly listed. Another reason to explicitly list output files is for a job that creates many files, and the user wants only a subset transferred back. .sp For grid universe jobs other than with grid type \fBcondor\fP, to have files other than standard output and standard error transferred from the execute machine back to the access point, do use \fBtransfer_output_files\fP, listing all files to be transferred. These files are found on the execute machine in the working directory of the job. .sp When a path to an output file or directory is specified, it specifies the path to the file on the execute side. As a destination on the submit side, the file is placed in the job\(aqs initial working directory, and it is named using the base name of the original path. For example, \fBpath/to/output_file\fP becomes \fBoutput_file\fP in the job\(aqs initial working directory. The name and path of the file that is written on the submit side may be modified by using \fBtransfer_output_remaps\fP\&. Note that this remap function only works with files but not with directories. .sp When a directory is specified, the behavior depends on whether there is a trailing path separator character. When a directory is specified with a trailing path separator, it is as if each of the items within the directory were listed in the transfer list. Therefore, the contents are transferred, but the directory itself is not. When there is no trailing path separator, the directory itself is transferred with all of its contents inside it. On platforms such as Windows where the path separator is not a forward slash (/), a trailing forward slash is treated as equivalent to a trailing path separator. An example of an input directory specified with a trailing forward slash is \fBinput_data/\fP\&. .sp For grid universe jobs other than HTCondor\-C, the transfer of directories is not currently supported. .sp Symbolic links to files are transferred as the files they point to. Transfer of symbolic links to directories is not currently supported. .TP .B transfer_checkpoint_files = < file1,file2,file3... > If present, this command defines the list of files and/or directories which constitute the job\(aqs checkpoint. When the job successfully checkpoints \-\- see \fBcheckpoint_exit_code\fP \-\- these files will be transferred to the submit node\(aqs spool. .sp If this command is absent, the output is transferred instead. .sp If no files or directories are specified, nothing will be transferred. This is generally not useful. .sp The list is interpreted like \fBtransfer_output_files\fP, but there is no corresponding \fBremaps\fP command. .sp .TP .B checkpoint_destination = When present, defines a URL that specifies both a plug\-in and a destination for the transfer of the entire checkpoint of a job. The plug\-in does the transfer of files, and no files are sent back to the access point. .TP .B preserve_relative_paths = < True | False > For vanilla and Docker \-universe jobs (and others that use the shadow), this command modifies the behavior of the file transfer commands. When set to true, the destination for an entry that is a relative path in a file transfer list becomes its relative path, not its basename. For example, \fBinput_data/b\fP (and its contents, if it is a directory) will be transferred to \fBinput_data/b\fP, not \fBb\fP\&. This applies to the input, output, and checkpoint lists. .sp Trailing slashes are ignored when \fBpreserve_relative_paths\fP is set. .TP .B transfer_output_remaps = < \(dq name = newname ; name2 = newname2 ... \(dq> This specifies the name (and optionally path) to use when downloading output files from the completed job. Normally, output files are transferred back to the initial working directory with the same name they had in the execution directory. This gives you the option to save them with a different path or name. If you specify a relative path, the final path will be relative to the job\(aqs initial working directory, and HTCondor will create directories as necessary to transfer the file. .sp \fIname\fP describes an output file name produced by your job, and \fInewname\fP describes the file name it should be downloaded to. Multiple remaps can be specified by separating each with a semicolon. If you wish to remap file names that contain equals signs or semicolons, these special characters may be escaped with a backslash. You cannot specify directories to be remapped. .sp Note that whether an output file is transferred is controlled by \fBtransfer_output_files\fP\&. Listing a file in \fBtransfer_output_remaps\fP is not sufficient to cause it to be transferred. .TP .B transfer_plugins = < tag=plugin ; tag2,tag3=plugin2 ... > Specifies the file transfer plugins (see \fI\%Third Party/Delegated file, credential and checkpoint transfer\fP) that should be transferred along with the input files prior to invoking file transfer plugins for files specified in \fItransfer_input_files\fP\&. \fItag\fP should be a URL prefix that is used in \fItransfer_input_files\fP, and \fIplugin\fP is the path to a file transfer plugin that will handle that type of URL transfer. .TP .B when_to_transfer_output = < ON_EXIT | ON_EXIT_OR_EVICT | ON_SUCCESS > Setting \fBwhen_to_transfer_output\fP to \fBON_EXIT\fP will cause HTCondor to transfer the job\(aqs output files back to the submitting machine when the job completes (exits on its own). If a job is evicted and started again, the subsequent execution will start with only the executable and input files in the scratch directory sandbox. If \fBtransfer_output_files\fP is not set, HTCondor considers all new files in the sandbox\(aqs top\-level directory to be the output; subdirectories and their contents will not be transferred. .sp Setting \fBwhen_to_transfer_output\fP to \fBON_EXIT_OR_EVICT\fP will cause HTCondor to transfer the job\(aqs output files when the job completes (exits on its own) and when the job is evicted. When the job is evicted, HTCondor will transfer the output files to a temporary directory on the submit node (determined by the \fI\%SPOOL\fP configuration variable). When the job restarts, these files will be transferred instead of the input files. If \fBtransfer_output_files\fP is not set, HTCondor considers all files in the sandbox\(aqs top\-level directory to be the output; subdirectories and their contents will not be transferred. .sp Setting \fBwhen_to_transfer_output\fP to \fBON_SUCCESS\fP will cause HTCondor to transfer the job\(aqs output files when the job completes successfully. Success is defined by the \fBsuccess_exit_code\fP command, which must be set, even if the successful value is the default \fB0\fP\&. If \fBtransfer_output_files\fP is not set, HTCondor considers all new files in the sandbox\(aqs top\-level directory to be the output; subdirectories and their contents will not be transferred. .sp In all three cases, the job will go on hold if \fBtransfer_output_files\fP specifies a file which does not exist at transfer time. .TP .B aws_access_key_id_file, s3_access_key_id_file One of these commands is required if you specify an \fBs3://\fP URL; they specify the file containing the access key ID (and only the access key ID) used to pre\-sign the URLs. Use only one. .TP .B aws_secret_access_key_file, s3_secret_access_key_file One of these commands is required if you specify an \fBs3://\fP URL; they specify the file containing the secret access key (and only the secret access key) used to pre\-sign the URLs. Use only one. .TP .B aws_region Optional if you specify an S3 URL (and ignored otherwise), this command specifies the region to use if one is not specified in the URL. .TP .B gs_access_key_id_file Required if you specify a \fBgs://\fP URLs, this command specifies the file containing the access key ID (and only the access key ID) used to pre\-sign the URLs. .TP .B gs_secret_access_key_file Required if you specify a \fBgs://\fP URLs, this command specifies the file containing the secret access key (and only the secret access key) used to pre\-sign the URLs. .UNINDENT .UNINDENT .UNINDENT .sp POLICY COMMANDS .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B allowed_execute_duration = The longest time for which a job may be executing. Jobs which exceed this duration will go on hold. This time does not include file\-transfer time. Jobs which self\-checkpoint have this long to write out each checkpoint. .sp This attribute is intended to help minimize the time wasted by jobs which may erroneously run forever. .TP .B allowed_job_duration = The longest time for which a job may continuously be in the running state. Jobs which exceed this duration will go on hold. Exiting the running state resets the job duration used by this command. .sp This command is intended to help minimize the time wasted by jobs which may erroneously run forever. .TP .B max_retries = The maximum number of retries allowed for this job (must be non\-negative). If the job fails (does not exit with the \fBsuccess_exit_code\fP exit code) it will be retried up to \fBmax_retries\fP times (unless retries are ceased because of the \fBretry_until\fP command). If \fBmax_retries\fP is not defined, and either \fBretry_until\fP or \fBsuccess_exit_code\fP is, the value of \fI\%DEFAULT_JOB_MAX_RETRIES\fP will be used for the maximum number of retries. .sp The combination of the \fBmax_retries\fP, \fBretry_until\fP, and \fBsuccess_exit_code\fP commands causes an appropriate \fBOnExitRemove\fP expression to be automatically generated. If retry command(s) and \fBon_exit_remove\fP are both defined, the \fBOnExitRemove\fP expression will be generated by OR\(aqing the expression specified in \fBOnExitRemove\fP and the expression generated by the retry commands. .TP .B retry_until = An integer value or boolean expression that prevents further retries from taking place, even if \fBmax_retries\fP have not been exhausted. If \fBretry_until\fP is an integer, the job exiting with that exit code will cause retries to cease. If \fBretry_until\fP is a ClassAd expression, the expression evaluating to \fBTrue\fP will cause retries to cease. For example, if you only want to retry exit codes 17, 34, and 81: .INDENT 7.0 .INDENT 3.5 .sp .EX max_retries = 5 retry_until = !member( ExitCode, {17, 34, 81} ) .EE .UNINDENT .UNINDENT .TP .B success_exit_code = The exit code that is considered successful for this job. Defaults to 0 if not defined. .sp \fBNote\fP: non\-zero values of success_exit_code should generally not be used for DAG node jobs, unless \fBwhen_to_transfer_output\fP is set to \fBON_SUCCESS\fP in order to avoid failed jobs going on hold. .sp At the present time, \fIcondor_dagman\fP does not take into account the value of \fBsuccess_exit_code\fP\&. This means that, if \fBsuccess_exit_code\fP is set to a non\-zero value, \fIcondor_dagman\fP will consider the job failed when it actually succeeds. For single\-proc DAG node jobs, this can be overcome by using a POST script that takes into account the value of \fBsuccess_exit_code\fP (although this is not recommended). For multi\-proc DAG node jobs, there is currently no way to overcome this limitation. .TP .B checkpoint_exit_code = The exit code which indicates that the executable has exited after successfully taking a checkpoint. The checkpoint will transferred and the executable restarted. See users\-manual/self\-checkpointing\-applications:Self\-Checkpointing Applications for details. .TP .B hold = If \fBhold\fP is set to \fBTrue\fP, then the submitted job will be placed into the Hold state. Jobs in the Hold state will not run until released by \fIcondor_release\fP\&. Defaults to \fBFalse\fP\&. .TP .B keep_claim_idle = An integer number of seconds that a job requests the \fIcondor_schedd\fP to wait before releasing its claim after the job exits or after the job is removed. .sp The process by which the \fIcondor_schedd\fP claims a \fIcondor_startd\fP is somewhat time\-consuming. To amortize this cost, the \fIcondor_schedd\fP tries to reuse claims to run subsequent jobs, after a job using a claim is done. However, it can only do this if there is an idle job in the queue at the moment the previous job completes. Sometimes, and especially for the node jobs when using DAGMan, there is a subsequent job about to be submitted, but it has not yet arrived in the queue when the previous job completes. As a result, the \fIcondor_schedd\fP releases the claim, and the next job must wait an entire negotiation cycle to start. When this submit command is defined with a non\-negative integer, when the job exits, the \fIcondor_schedd\fP tries as usual to reuse the claim. If it cannot, instead of releasing the claim, the \fIcondor_schedd\fP keeps the claim until either the number of seconds given as a parameter, or a new job which matches that claim arrives, whichever comes first. The \fIcondor_startd\fP in question will remain in the Claimed/Idle state, and the original job will be \(dqcharged\(dq (in terms of priority) for the time in this state. .TP .B leave_in_queue = When the ClassAd Expression evaluates to \fBTrue\fP, the job is not removed from the queue upon completion. This allows the user of a remotely spooled job to retrieve output files in cases where HTCondor would have removed them as part of the cleanup associated with completion. The job will only exit the queue once it has been marked for removal (via \fIcondor_rm\fP, for example) and the \fBleave_in_queue\fP, expression has become \fBFalse\fP\&. \fBleave_in_queue\fP defaults to \fBFalse\fP\&. .sp As an example, if the job is to be removed once the output is retrieved with \fIcondor_transfer_data\fP, then use .INDENT 7.0 .INDENT 3.5 .sp .EX leave_in_queue = (JobStatus == 4) && ((StageOutFinish =?= UNDEFINED) ||\e (StageOutFinish == 0)) .EE .UNINDENT .UNINDENT .TP .B next_job_start_delay = This expression specifies the number of seconds to delay after starting up this job before the next job is started. The maximum allowed delay is specified by the HTCondor configuration variable \fI\%MAX_NEXT_JOB_START_DELAY\fP, which defaults to 10 minutes. This command does not apply to \fBscheduler\fP or \fBlocal\fP universe jobs. .sp This command has been historically used to implement a form of job start throttling from the job submitter\(aqs perspective. It was effective for the case of multiple job submission where the transfer of extremely large input data sets to the execute machine caused machine performance to suffer. This command is no longer useful, as throttling should be accomplished through configuration of the \fIcondor_schedd\fP daemon. .TP .B on_exit_hold = The ClassAd expression is checked when the job exits, and if \fBTrue\fP, places the job into the Hold state. If \fBFalse\fP (the default value when not defined), then nothing happens and the \fBon_exit_remove\fP expression is checked to determine if that needs to be applied. .sp For example: Suppose a job is known to run for a minimum of an hour. If the job exits after less than an hour, the job should be placed on hold and an e\-mail notification sent, instead of being allowed to leave the queue. .INDENT 7.0 .INDENT 3.5 .sp .EX on_exit_hold = (time() \- JobStartDate) < (60 * $(MINUTE)) .EE .UNINDENT .UNINDENT .sp This expression places the job on hold if it exits for any reason before running for an hour. An e\-mail will be sent to the user explaining that the job was placed on hold because this expression became \fBTrue\fP\&. .sp \fBperiodic_*\fP expressions take precedence over \fBon_exit_*\fP expressions, and \fB*_hold\fP expressions take precedence over a \fB*_remove\fP expressions. .sp Only job ClassAd attributes will be defined for use by this ClassAd expression. This expression is available for the vanilla, java, parallel, grid, local and scheduler universes. .TP .B on_exit_hold_reason = When the job is placed on hold due to the \fBon_exit_hold\fP expression becoming \fBTrue\fP, this expression is evaluated to set the value of \fBHoldReason\fP in the job ClassAd. If this expression is \fBUNDEFINED\fP or produces an empty or invalid string, a default description is used. .TP .B on_exit_hold_subcode = When the job is placed on hold due to the \fBon_exit_hold\fP expression becoming \fBTrue\fP, this expression is evaluated to set the value of \fBHoldReasonSubCode\fP in the job ClassAd. The default subcode is 0. The \fBHoldReasonCode\fP will be set to 3, which indicates that the job went on hold due to a job policy expression. .TP .B on_exit_remove = The ClassAd expression is checked when the job exits, and if \fBTrue\fP (the default value when undefined), then it allows the job to leave the queue normally. If \fBFalse\fP, then the job is placed back into the Idle state. If the user job runs under the vanilla universe, then the job restarts from the beginning. .sp For example, suppose a job occasionally segfaults, but chances are that the job will finish successfully if the job is run again with the same data. The \fBon_exit_remove\fP expression can cause the job to run again with the following command. Assume that the signal identifier for the segmentation fault is 11 on the platform where the job will be running. .INDENT 7.0 .INDENT 3.5 .sp .EX on_exit_remove = (ExitBySignal == False) || (ExitSignal != 11) .EE .UNINDENT .UNINDENT .sp This expression lets the job leave the queue if the job was not killed by a signal or if it was killed by a signal other than 11, representing segmentation fault in this example. So, if the exited due to signal 11, it will stay in the job queue. In any other case of the job exiting, the job will leave the queue as it normally would have done. .sp As another example, if the job should only leave the queue if it exited on its own with status 0, this \fBon_exit_remove\fP expression works well: .INDENT 7.0 .INDENT 3.5 .sp .EX on_exit_remove = (ExitBySignal == False) && (ExitCode == 0) .EE .UNINDENT .UNINDENT .sp If the job was killed by a signal or exited with a non\-zero exit status, HTCondor would leave the job in the queue to run again. .sp \fBperiodic_*\fP expressions take precedence over \fBon_exit_*\fP expressions, and \fB*_hold\fP expressions take precedence over a \fB*_remove\fP expressions. .sp Only job ClassAd attributes will be defined for use by this ClassAd expression. .TP .B periodic_hold = This expression is checked periodically when the job is not in the Held state. If it becomes \fBTrue\fP, the job will be placed on hold. If unspecified, the default value is \fBFalse\fP\&. .sp \fBperiodic_*\fP expressions take precedence over \fBon_exit_*\fP expressions, and \fB*_hold\fP expressions take precedence over a \fB*_remove\fP expressions. .sp Only job ClassAd attributes will be defined for use by this ClassAd expression. Note that, by default, this expression is only checked once every 60 seconds. The period of these evaluations can be adjusted by setting the \fI\%PERIODIC_EXPR_INTERVAL\fP, \fI\%MAX_PERIODIC_EXPR_INTERVAL\fP, and \fI\%PERIODIC_EXPR_TIMESLICE\fP configuration macros. .TP .B periodic_hold_reason = When the job is placed on hold due to the \fBperiodic_hold\fP expression becoming \fBTrue\fP, this expression is evaluated to set the value of \fBHoldReason\fP in the job ClassAd. If this expression is \fBUNDEFINED\fP or produces an empty or invalid string, a default description is used. .TP .B periodic_hold_subcode = When the job is placed on hold due to the \fBperiodic_hold\fP expression becoming true, this expression is evaluated to set the value of \fBHoldReasonSubCode\fP in the job ClassAd. The default subcode is 0. The \fBHoldReasonCode\fP will be set to 3, which indicates that the job went on hold due to a job policy expression. .TP .B periodic_release = This expression is checked periodically when the job is in the Held state. If the expression becomes \fBTrue\fP, the job will be released. If the job was held via \fIcondor_hold\fP (i.e. \fBHoldReasonCode\fP is \fB1\fP), then this expression is ignored. .sp Only job ClassAd attributes will be defined for use by this ClassAd expression. Note that, by default, this expression is only checked once every 60 seconds. The period of these evaluations can be adjusted by setting the \fI\%PERIODIC_EXPR_INTERVAL\fP, \fI\%MAX_PERIODIC_EXPR_INTERVAL\fP, and \fI\%PERIODIC_EXPR_TIMESLICE\fP configuration macros. .TP .B periodic_remove = This expression is checked periodically. If it becomes \fBTrue\fP, the job is removed from the queue. If unspecified, the default value is \fBFalse\fP\&. .sp See the Examples section of this manual page for an example of a \fBperiodic_remove\fP expression. .sp \fBperiodic_*\fP expressions take precedence over \fBon_exit_*\fP expressions, and \fB*_hold\fP expressions take precedence over a \fB*_remove\fP expressions. So, the \fBperiodic_remove\fP expression takes precedent over the \fBon_exit_remove\fP expression, if the two describe conflicting actions. .sp Only job ClassAd attributes will be defined for use by this ClassAd expression. Note that, by default, this expression is only checked once every 60 seconds. The period of these evaluations can be adjusted by setting the \fI\%PERIODIC_EXPR_INTERVAL\fP, \fI\%MAX_PERIODIC_EXPR_INTERVAL\fP, and \fI\%PERIODIC_EXPR_TIMESLICE\fP configuration macros. .sp .TP .B periodic_vacate = This expression is checked periodically for running jobs. If it becomes \fBTrue\fP, job is evicted from the machine it is running on, and return to the queue, in an Idle state. If unspecified, the default value is \fBFalse\fP\&. .sp Only job ClassAd attributes will be defined for use by this ClassAd expression. Note that, by default, this expression is only checked once every 60 seconds. The period of these evaluations can be adjusted by setting the \fI\%PERIODIC_EXPR_INTERVAL\fP, \fI\%MAX_PERIODIC_EXPR_INTERVAL\fP, and \fI\%PERIODIC_EXPR_TIMESLICE\fP configuration macros. .UNINDENT .UNINDENT .UNINDENT .sp COMMANDS FOR THE GRID .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B arc_application = For grid universe jobs of type \fBarc\fP, provides additional XML attributes under the \fB\fP section of the ARC ADL job description which are not covered by regular submit description file parameters. .TP .B arc_resources = For grid universe jobs of type \fBarc\fP, provides additional XML attributes under the \fB\fP section of the ARC ADL job description which are not covered by regular submit description file parameters. .TP .B arc_rte = < rte1 option,rte2 > For grid universe jobs of type \fBarc\fP, provides a list of Runtime Environment names that the job requires on the ARC system. The list is comma\-delimited. If a Runtime Environment name supports options, those can be provided after the name, separated by spaces. Runtime Environment names are defined by the ARC server. .TP .B azure_admin_key = For grid type \fBazure\fP jobs, specifies the path and file name of a file that contains an SSH public key. This key can be used to log into the administrator account of the instance via SSH. .TP .B azure_admin_username = For grid type \fBazure\fP jobs, specifies the name of an administrator account to be created in the instance. This account can be logged into via SSH. .TP .B azure_auth_file = For grid type \fBazure\fP jobs, specifies a path and file name of the authorization file that grants permission for HTCondor to use the Azure account. If it\(aqs not defined, then HTCondor will attempt to use the default credentials of the Azure CLI tools. .TP .B azure_image = For grid type \fBazure\fP jobs, identifies the disk image to be used for the boot disk of the instance. This image must already be registered within Azure. .TP .B azure_location = For grid type \fBazure\fP jobs, identifies the location within Azure where the instance should be run. As an example, one current location is \fBcentralus\fP\&. .TP .B azure_size = For grid type \fBazure\fP jobs, the hardware configuration that the virtual machine instance is to run on. .TP .B batch_extra_submit_args = Used for \fBbatch\fP grid universe jobs. Specifies additional command\-line arguments to be given to the target batch system\(aqs job submission command. .TP .B batch_project = Used for \fBbatch\fP grid universe jobs. Specifies the name of the PBS/LSF/SGE/SLURM project, account, or allocation that should be charged for the resources used by the job. .TP .B batch_queue = Used for \fBbatch\fP grid universe jobs. Specifies the name of the PBS/LSF/SGE/SLURM job queue into which the job should be submitted. If not specified, the default queue is used. For a multi\-cluster SLURM configuration, which cluster to use can be specified by supplying the name after an \fB@\fP symbol. For example, to submit a job to the \fBdebug\fP queue on cluster \fBfoo\fP, you would use the value \fBdebug@foo\fP\&. .TP .B batch_runtime = Used for \fBbatch\fP grid universe jobs. Specifies a limit in seconds on the execution time of the job. This limit is enforced by the PBS/LSF/SGE/SLURM scheduler. .TP .B cloud_label_names = For grid type \fBgce\fP jobs, specifies the case of tag names that will be associated with the running instance. This is only necessary if a tag name case matters. By default the list will be automatically generated. .TP .B cloud_label_ = For grid type \fBgce\fP jobs, specifies a label and value to be associated with the running instance. The label name will be lower\-cased; use \fBcloud_label_names\fP to change the case. .TP .B delegate_job_GSI_credentials_lifetime = Specifies the maximum number of seconds for which delegated proxies should be valid. The default behavior when this command is not specified is determined by the configuration variable \fI\%DELEGATE_JOB_GSI_CREDENTIALS_LIFETIME\fP, which defaults to one day. A value of 0 indicates that the delegated proxy should be valid for as long as allowed by the credential used to create the proxy. This setting currently only applies to proxies delegated for non\-grid jobs and for HTCondor\-C jobs. This variable has no effect if the configuration variable \fI\%DELEGATE_JOB_GSI_CREDENTIALS\fP is \fBFalse\fP, because in that case the job proxy is copied rather than delegated. .TP .B ec2_access_key_id = For grid type \fBec2\fP jobs, identifies the file containing the access key. .TP .B ec2_ami_id = For grid type \fBec2\fP jobs, identifies the machine image. Services compatible with the EC2 Query API may refer to these with abbreviations other than \fBAMI\fP, for example \fBEMI\fP is valid for Eucalyptus. .TP .B ec2_availability_zone = For grid type \fBec2\fP jobs, specifies the Availability Zone that the instance should be run in. This command is optional, unless \fBec2_ebs_volumes\fP is set. As an example, one current zone is \fBus\-east\-1b\fP\&. .TP .B ec2_block_device_mapping = :,:, ... For grid type \fBec2\fP jobs, specifies the block device to kernel device mapping. This command is optional. .TP .B ec2_ebs_volumes = :,:,... For grid type \fBec2\fP jobs, optionally specifies a list of Elastic Block Store (EBS) volumes to be made available to the instance and the device names they should have in the instance. .TP .B ec2_elastic_ip = For grid type \fBec2\fP jobs, and optional specification of an Elastic IP address that should be assigned to this instance. .TP .B ec2_iam_profile_arn = For grid type \fBec2\fP jobs, an Amazon Resource Name (ARN) identifying which Identity and Access Management (IAM) (instance) profile to associate with the instance. .TP .B ec2_iam_profile_name = For grid type \fBec2\fP jobs, a name identifying which Identity and Access Management (IAM) (instance) profile to associate with the instance. .TP .B ec2_instance_type = For grid type \fBec2\fP jobs, identifies the instance type. Different services may offer different instance types, so no default value is set. .TP .B ec2_keypair = For grid type \fBec2\fP jobs, specifies the name of an SSH key\-pair that is already registered with the EC2 service. The associated private key can be used to \fIssh\fP into the virtual machine once it is running. .TP .B ec2_keypair_file = For grid type \fBec2\fP jobs, specifies the complete path and file name of a file into which HTCondor will write an SSH key for use with ec2 jobs. The key can be used to \fIssh\fP into the virtual machine once it is running. If \fBec2_keypair\fP is specified for a job, \fBec2_keypair_file\fP is ignored. .TP .B ec2_parameter_names = ParameterName1, ParameterName2, ... For grid type \fBec2\fP jobs, a space or comma separated list of the names of additional parameters to pass when instantiating an instance. .TP .B ec2_parameter_ = For grid type \fBec2\fP jobs, specifies the value for the correspondingly named (instance instantiation) parameter. \fB\fP is the parameter name specified in the submit command \fBec2_parameter_names\fP, but with any periods replaced by underscores. .TP .B ec2_secret_access_key = For grid type \fBec2\fP jobs, specifies the path and file name containing the secret access key. .TP .B ec2_security_groups = group1, group2, ... For grid type \fBec2\fP jobs, defines the list of EC2 security groups which should be associated with the job. .TP .B ec2_security_ids = id1, id2, ... For grid type \fBec2\fP jobs, defines the list of EC2 security group IDs which should be associated with the job. .TP .B ec2_spot_price = For grid type \fBec2\fP jobs, specifies the spot instance bid, which is the most that the job submitter is willing to pay per hour to run this job. .TP .B ec2_tag_names = For grid type \fBec2\fP jobs, specifies the case of tag names that will be associated with the running instance. This is only necessary if a tag name case matters. By default the list will be automatically generated. .TP .B ec2_tag_ = For grid type \fBec2\fP jobs, specifies a tag to be associated with the running instance. The tag name will be lower\-cased; use \fBec2_tag_names\fP to change the case. .TP .B WantNameTag = For grid type \fBec2\fP jobs, a job may request that its \(aqname\(aq tag be (not) set by HTCondor. If the job does not otherwise specify any tags, not setting its name tag will eliminate a call by the EC2 GAHP, improving performance. .TP .B ec2_user_data = For grid type \fBec2\fP jobs, provides a block of data that can be accessed by the virtual machine. If both \fBec2_user_data\fP and \fBec2_user_data_file\fP are specified for a job, the two blocks of data are concatenated, with the data from this \fBec2_user_data\fP submit command occurring first. .TP .B ec2_user_data_file = For grid type \fBec2\fP jobs, specifies a path and file name whose contents can be accessed by the virtual machine. If both \fBec2_user_data\fP and \fBec2_user_data_file\fP are specified for a job, the two blocks of data are concatenated, with the data from that \fBec2_user_data\fP submit command occurring first. .TP .B ec2_vpc_ip = For grid type \fBec2\fP jobs, that are part of a Virtual Private Cloud (VPC), an optional specification of the IP address that this instance should have within the VPC. .TP .B ec2_vpc_subnet = For grid type \fBec2\fP jobs, an optional specification of the Virtual Private Cloud (VPC) that this instance should be a part of. .TP .B gce_account = For grid type \fBgce\fP jobs, specifies the Google cloud services account to use. If this submit command isn\(aqt specified, then a random account from the authorization file given by \fBgce_auth_file\fP will be used. .TP .B gce_auth_file = For grid type \fBgce\fP jobs, specifies a path and file name of the authorization file that grants permission for HTCondor to use the Google account. If this command is not specified, then the default file of the Google command\-line tools will be used. .TP .B gce_image = For grid type \fBgce\fP jobs, the identifier of the virtual machine image representing the HTCondor job to be run. This virtual machine image must already be register with GCE and reside in Google\(aqs Cloud Storage service. .TP .B gce_json_file = For grid type \fBgce\fP jobs, specifies the path and file name of a file that contains JSON elements that should be added to the instance description submitted to the GCE service. .TP .B gce_machine_type = For grid type \fBgce\fP jobs, the long form of the URL that describes the machine configuration that the virtual machine instance is to run on. .TP .B gce_metadata = For grid type \fBgce\fP jobs, a comma separated list of name and value pairs that define metadata for a virtual machine instance that is an HTCondor job. .TP .B gce_metadata_file = For grid type \fBgce\fP jobs, specifies a path and file name of the file that contains metadata for a virtual machine instance that is an HTCondor job. Within the file, each name and value pair is on its own line; so, the pairs are separated by the newline character. .TP .B gce_preemptible = For grid type \fBgce\fP jobs, specifies whether the virtual machine instance should be preemptible. The default is for the instance to not be preemptible. .TP .B grid_resource = For each \fBgrid\-type\-string\fP value, there are further type\-specific values that must specified. This submit description file command allows each to be given in a space\-separated list. Allowable \fBgrid\-type\-string\fP values are \fBarc\fP, \fBazure\fP, \fBbatch\fP, \fBcondor\fP, \fBec2\fP, and \fBgce\fP\&. The HTCondor manual chapter on Grid Computing details the variety of grid types. .sp For a \fBgrid\-type\-string\fP of \fBbatch\fP, the single parameter is the name of the local batch system, and will be one of \fBpbs\fP, \fBlsf\fP, \fBslurm\fP, or \fBsge\fP\&. .sp For a \fBgrid\-type\-string\fP of \fBcondor\fP, the first parameter is the name of the remote \fIcondor_schedd\fP daemon. The second parameter is the name of the pool to which the remote \fIcondor_schedd\fP daemon belongs. .sp For a \fBgrid\-type\-string\fP of \fBec2\fP, one additional parameter specifies the EC2 URL. .sp For a \fBgrid\-type\-string\fP of \fBarc\fP, the single parameter is the name of the ARC resource to be used. .TP .B transfer_error = For jobs submitted to the grid universe only. If \fBTrue\fP, then the error output (from \fBstderr\fP) from the job is transferred from the remote machine back to the access point. The name of the file after transfer is given by the \fBerror\fP command. If \fBFalse\fP, no transfer takes place (from the remote machine to access point), and the name of the file is given by the \fBerror\fP command. The default value is \fBTrue\fP\&. .TP .B transfer_input = For jobs submitted to the grid universe only. If \fBTrue\fP, then the job input (\fBstdin\fP) is transferred from the machine where the job was submitted to the remote machine. The name of the file that is transferred is given by the \fBinput\fP command. If \fBFalse\fP, then the job\(aqs input is taken from a pre\-staged file on the remote machine, and the name of the file is given by the \fBinput\fP command. The default value is \fBTrue\fP\&. .sp For transferring files other than \fBstdin\fP, see \fBtransfer_input_files\fP\&. .TP .B transfer_output = For jobs submitted to the grid universe only. If \fBTrue\fP, then the output (from \fBstdout\fP) from the job is transferred from the remote machine back to the access point. The name of the file after transfer is given by the \fBoutput\fP command. If \fBFalse\fP, no transfer takes place (from the remote machine to access point), and the name of the file is given by the \fBoutput\fP command. The default value is \fBTrue\fP\&. .sp For transferring files other than \fBstdout\fP, see \fBtransfer_output_files\fP, .TP .B use_x509userproxy = Set this command to \fBTrue\fP to indicate that the job requires an X.509 user proxy. If \fBx509userproxy\fP is set, then that file is used for the proxy. Otherwise, the proxy is looked for in the standard locations. If \fBx509userproxy\fP is set or if the job is a grid universe job of grid type \fBarc\fP, then the value of \fBuse_x509userproxy\fP is forced to \fBTrue\fP\&. Defaults to \fBFalse\fP\&. .TP .B x509userproxy = Used to override the default path name for X.509 user certificates. The default location for X.509 proxies is the \fB/tmp\fP directory, which is generally a local file system. Setting this value would allow HTCondor to access the proxy in a shared file system (for example, AFS). HTCondor will use the proxy specified in the submit description file first. If nothing is specified in the submit description file, it will use the environment variable X509_USER_PROXY. If that variable is not present, it will search in the default location. Note that proxies are only valid for a limited time. Condor_submit will not submit a job with an expired proxy, it will return an error. Also, if the configuration parameter CRED_MIN_TIME_LEFT is set to some number of seconds, and if the proxy will expire before that many seconds, condor_submit will also refuse to submit the job. That is, if CRED_MIN_TIME_LEFT is set to 60, condor_submit will refuse to submit a job whose proxy will expire 60 seconds from the time of submission. .sp \fBx509userproxy\fP is relevant when the \fBuniverse\fP is \fBvanilla\fP, or when the \fBuniverse\fP is \fBgrid\fP and the type of grid system is one of \fBcondor\fP, or \fBarc\fP\&. Defining a value causes the proxy to be delegated to the execute machine. Further, VOMS attributes defined in the proxy will appear in the job ClassAd. .TP .B use_scitokens = Set this command to \fBTrue\fP to indicate that the job requires a scitoken. If \fBscitokens_file\fP is set, then that file is used for the scitoken filename. Otherwise, the the scitoken filename is looked for in the \fBBEARER_TOKEN_FILE\fP environment variable. If \fBscitokens_file\fP is set then the value of \fBuse_scitokens\fP defaults to \fBTrue\fP\&. If the filename is not defined in on one of these two places, then \fIcondor_submit\fP will fail with an error message. Set this command to \fBAuto\fP to indicate that the job will use a scitoken if \fBscitokens_file\fP or the \fBBEARER_TOKEN_FILE\fP environment variable is set, but it will not be an error if no file is specified. .TP .B scitokens_file = Used to set the path to the file containing the scitoken that the job needs, or to override the path to the scitoken contained in the \fBBEARER_TOKEN_FILE\fP environment variable. .UNINDENT .UNINDENT .UNINDENT .sp COMMANDS FOR PARALLEL, JAVA, and SCHEDULER UNIVERSES .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B hold_kill_sig = For the scheduler universe only, \fBsignal\-number\fP is the signal delivered to the job when the job is put on hold with \fIcondor_hold\fP\&. \fBsignal\-number\fP may be either the platform\-specific name or value of the signal. If this command is not present, the value of \fBkill_sig\fP is used. .TP .B jar_files = Specifies a list of additional JAR files to include when using the Java universe. JAR files will be transferred along with the executable and automatically added to the classpath. .TP .B java_vm_args = Specifies a list of additional arguments to the Java VM itself, When HTCondor runs the Java program, these are the arguments that go before the class name. This can be used to set VM\-specific arguments like stack size, garbage\-collector arguments and initial property values. .TP .B machine_count = For the parallel universe, a single value (\fImax\fP) is required. It is neither a maximum or minimum, but the number of machines to be dedicated toward running the job. .TP .B remove_kill_sig = For the scheduler universe only, \fBsignal\-number\fP is the signal delivered to the job when the job is removed with \fIcondor_rm\fP\&. \fBsignal\-number\fP may be either the platform\-specific name or value of the signal. This example shows it both ways for a Linux signal: .INDENT 7.0 .INDENT 3.5 .sp .EX remove_kill_sig = SIGUSR1 remove_kill_sig = 10 .EE .UNINDENT .UNINDENT .sp If this command is not present, the value of \fBkill_sig\fP is used. .UNINDENT .UNINDENT .UNINDENT .sp COMMANDS FOR THE VM UNIVERSE .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B vm_disk = file1:device1:permission1, file2:device2:permission2:format2, ... A list of comma separated disk files. Each disk file is specified by 4 colon separated fields. The first field is the path and file name of the disk file. The second field specifies the device. The third field specifies permissions, and the optional fourth field specifies the image format. If a disk file will be transferred by HTCondor, then the first field should just be the simple file name (no path information). .sp An example that specifies two disk files: .INDENT 7.0 .INDENT 3.5 .sp .EX vm_disk = /myxen/diskfile.img:sda1:w,/myxen/swap.img:sda2:w .EE .UNINDENT .UNINDENT .TP .B vm_checkpoint = A boolean value specifying whether or not to take checkpoints. If not specified, the default value is \fBFalse\fP\&. In the current implementation, setting both \fBvm_checkpoint\fP and \fBvm_networking\fP to \fBTrue\fP does not yet work in all cases. Networking cannot be used if a vm universe job uses a checkpoint in order to continue execution after migration to another machine. .TP .B vm_macaddr = Defines that MAC address that the virtual machine\(aqs network interface should have, in the standard format of six groups of two hexadecimal digits separated by colons. .TP .B vm_memory = The amount of memory in MBytes that a vm universe job requires. .TP .B vm_networking = Specifies whether to use networking or not. In the current implementation, setting both \fBvm_checkpoint\fP and \fBvm_networking\fP to \fBTrue\fP does not yet work in all cases. Networking cannot be used if a vm universe job uses a checkpoint in order to continue execution after migration to another machine. .TP .B vm_networking_type = When \fBvm_networking\fP is \fBTrue\fP, this definition augments the job\(aqs requirements to match only machines with the specified networking. If not specified, then either networking type matches. .TP .B vm_no_output_vm = When \fBTrue\fP, prevents HTCondor from transferring output files back to the machine from which the vm universe job was submitted. If not specified, the default value is \fBFalse\fP\&. .TP .B vm_type = Specifies the underlying virtual machine software that this job expects. .TP .B xen_initrd = When \fBxen_kernel\fP gives a file name for the kernel image to use, this optional command may specify a path to a ramdisk (\fBinitrd\fP) image file. If the image file will be transferred by HTCondor, then the value should just be the simple file name (no path information). .TP .B xen_kernel = A value of \fBincluded\fP specifies that the kernel is included in the disk file. If not one of these values, then the value is a path and file name of the kernel to be used. If a kernel file will be transferred by HTCondor, then the value should just be the simple file name (no path information). .TP .B xen_kernel_params = A string that is appended to the Xen kernel command line. .TP .B xen_root = A string that is appended to the Xen kernel command line to specify the root device. This string is required when \fBxen_kernel\fP gives a path to a kernel. Omission for this required case results in an error message during submission. .UNINDENT .UNINDENT .UNINDENT .sp COMMANDS FOR THE DOCKER UNIVERSE .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B docker_image = < image\-name > Defines the name of the Docker image that is the basis for the docker container. .TP .B docker_network_type = < host | none | custom_admin_defined_value> If docker_network_type is set to the string host, then the job is run using the host\(aqs network. If docker_network_type is set to the string none, then the job is run with no network. If this is not set, each job gets a private network interface. Some administrators may define site specific docker networks on a given worker node. When this is the case, additional values may be valid here. .TP .B docker_pull_policy = < always > if docker_pull_policy is set to \fIalways\fP, when a docker universe job starts on a worker node, the option \(dq\-\-pull always\(dq will be passed to the docker run command. This only impacts worker nodes which already have a locally cached version of the image. With this option, docker will always check with the repo to see if the cached version is out of date. This requires more network connectivity, and may cause docker hub to throttle future pull requests. It is generally recommened to never mutate docker image tag name, and avoid needing this option. .TP .B container_service_names = [, ]* A string\- or comma\- separated list of \fIservice name\fPs. Each \fIservice\-name\fP must have a corresponding \fB_container_port\fP command specifying a port number (an integer from 0 to 65535). HTCondor will ask Docker to forward from a host port to the specified port inside the container. When Docker has done so, HTCondor will add an attribute to the job ad for each service, \fBHostPort\fP, which contains the port number on the host forwarding to the corresponding service. .UNINDENT .UNINDENT .UNINDENT .sp COMMANDS FOR THE CONTAINER UNIVERSE .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B container_image = < image\-name > Defines the name of the container image. Can be a singularity .sif file, a singularity exploded directory, or a path to an image in a docker style repository .TP .B container_target_dir = < path\-to\-directory\-inside\-container > Defines the working directory of the job inside the container. Will be mapped to the scratch directory on the worker node. .UNINDENT .UNINDENT .UNINDENT .sp ADVANCED COMMANDS .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B accounting_group = Causes jobs to negotiate under the given accounting group. This value is advertised in the job ClassAd as \fBAcctGroup\fP\&. The HTCondor Administrator\(aqs manual contains more information about accounting groups. .TP .B accounting_group_user = Sets the name associated with this job to be used for resource usage accounting purposes, such as computation of fair\-share priority and reporting via \fBcondor_userprio\fP\&. If not set, defaults to the value of the job ClassAd attribute \fBUser\fP\&. This value is advertised in the job ClassAd as \fBAcctGroupUser\fP\&. .TP .B concurrency_limits = A list of resources that this job needs. The resources are presumed to have concurrency limits placed upon them, thereby limiting the number of concurrent jobs in execution which need the named resource. Commas and space characters delimit the items in the list. Each item in the list is a string that identifies the limit, or it is a ClassAd expression that evaluates to a string, and it is evaluated in the context of machine ClassAd being considered as a match. Each item in the list also may specify a numerical value identifying the integer number of resources required for the job. The syntax follows the resource name by a colon character (:) and the numerical value. Details on concurrency limits are in the HTCondor Administrator\(aqs manual. .TP .B concurrency_limits_expr = A ClassAd expression that represents the list of resources that this job needs after evaluation. The ClassAd expression may specify machine ClassAd attributes that are evaluated against a matched machine. After evaluation, the list sets \fBconcurrency_limits\fP\&. .TP .B copy_to_spool = If \fBcopy_to_spool\fP is \fBTrue\fP, then \fIcondor_submit\fP copies the executable to the local spool directory before running it on a remote host. As copying can be quite time consuming and unnecessary, the default value is \fBFalse\fP for all job universes. When \fBFalse\fP, \fIcondor_submit\fP does not copy the executable to a local spool directory. .TP .B coresize = Should the user\(aqs program abort and produce a core file, \fBcoresize\fP specifies the maximum size in bytes of the core file which the user wishes to keep. If \fBcoresize\fP is not specified in the command file, this is set to 0 (meaning no core will be generated). .TP .B cron_day_of_month = The set of days of the month for which a deferral time applies. The HTCondor User\(aqs manual section on Time Scheduling for Job Execution has further details. .TP .B cron_day_of_week = The set of days of the week for which a deferral time applies. The HTCondor User\(aqs manual section on Time Scheduling for Job Execution has further details. .TP .B cron_hour = The set of hours of the day for which a deferral time applies. The HTCondor User\(aqs manual section on Time Scheduling for Job Execution has further details. .TP .B cron_minute = The set of minutes within an hour for which a deferral time applies. The HTCondor User\(aqs manual section on Time Scheduling for Job Execution has further details. .TP .B cron_month = The set of months within a year for which a deferral time applies. The HTCondor User\(aqs manual section on Time Scheduling for Job Execution has further details. .TP .B cron_prep_time = Analogous to \fBdeferral_prep_time\fP\&. The number of seconds prior to a job\(aqs deferral time that the job may be matched and sent to an execution machine. .TP .B cron_window = Analogous to the submit command \fBdeferral_window\fP\&. It allows cron jobs that miss their deferral time to begin execution. .sp The HTCondor User\(aqs manual section on Time Scheduling for Job Execution has further details. .TP .B dagman_log = DAGMan inserts this command to specify an event log that it watches to maintain the state of the DAG. If the \fBlog\fP command is not specified in the submit file, DAGMan uses the \fBlog\fP command to specify the event log. .TP .B deferral_prep_time = The number of seconds prior to a job\(aqs deferral time that the job may be matched and sent to an execution machine. .sp The HTCondor User\(aqs manual section on Time Scheduling for Job Execution has further details. .TP .B deferral_time = Allows a job to specify the time at which its execution is to begin, instead of beginning execution as soon as it arrives at the execution machine. The deferral time is an expression that evaluates to a Unix Epoch timestamp (the number of seconds elapsed since 00:00:00 on January 1, 1970, Coordinated Universal Time). Deferral time is evaluated with respect to the execution machine. This option delays the start of execution, but not the matching and claiming of a machine for the job. If the job is not available and ready to begin execution at the deferral time, it has missed its deferral time. A job that misses its deferral time will be put on hold in the queue. .sp The HTCondor User\(aqs manual section on Time Scheduling for Job Execution has further details. .sp Due to implementation details, a deferral time may not be used for scheduler universe jobs. .TP .B deferral_window = The deferral window is used in conjunction with the \fBdeferral_time\fP command to allow jobs that miss their deferral time to begin execution. .sp The HTCondor User\(aqs manual section on Time Scheduling for Job Execution has further details. .TP .B description = A string that sets the value of the job ClassAd attribute \fBJobDescription\fP\&. When set, tools which display the executable such as \fIcondor_q\fP will instead use this string. .TP .B email_attributes = A comma\-separated list of attributes from the job ClassAd. These attributes and their values will be included in the e\-mail notification of job completion. .TP .B image_size = Advice to HTCondor specifying the maximum virtual image size to which the job will grow during its execution. HTCondor will then execute the job only on machines which have enough resources, (such as virtual memory), to support executing the job. If not specified, HTCondor will automatically make a (reasonably accurate) estimate about the job\(aqs size and adjust this estimate as the program runs. If specified and underestimated, the job may crash due to the inability to acquire more address space; for example, if malloc() fails. If the image size is overestimated, HTCondor may have difficulty finding machines which have the required resources. \fIsize\fP is specified in KiB. For example, for an image size of 8 MiB, \fIsize\fP should be 8000. .TP .B initialdir = Used to give jobs a directory with respect to file input and output. Also provides a directory (on the machine from which the job is submitted) for the job event log, when a full path is not specified. .sp For vanilla universe jobs where there is a shared file system, it is the current working directory on the machine where the job is executed. .sp For vanilla or grid universe jobs where file transfer mechanisms are utilized (there is not a shared file system), it is the directory on the machine from which the job is submitted where the input files come from, and where the job\(aqs output files go to. .sp For scheduler universe jobs, it is the directory on the machine from which the job is submitted where the job runs; the current working directory for file input and output with respect to relative path names. .sp Note that the path to the executable is not relative to \fBinitialdir\fP if it is a relative path, it is relative to the directory in which the \fIcondor_submit\fP command is run. .TP .B job_ad_information_attrs = A comma\-separated list of job ClassAd attribute names. The named attributes and their values are written to the job event log whenever any event is being written to the log. This implements the same thing as the configuration variable \fBEVENT_LOG_INFORMATION_ATTRS\fP (see the admin\-manual/configuration\-macros:daemon logging configuration file entries page), but it applies to the job event log, instead of the system event log. .TP .B job_lease_duration = For vanilla, parallel, VM, and java universe jobs only, the duration in seconds of a job lease. The default value is 2,400, or forty minutes. If a job lease is not desired, the value can be explicitly set to 0 to disable the job lease semantics. The value can also be a ClassAd expression that evaluates to an integer. The HTCondor User\(aqs manual section on Special Environment Considerations has further details. .TP .B job_machine_attrs = A comma and/or space separated list of machine attribute names that should be recorded in the job ClassAd in addition to the ones specified by the \fIcondor_schedd\fP daemon\(aqs system configuration variable \fI\%SYSTEM_JOB_MACHINE_ATTRS\fP\&. When there are multiple run attempts, history of machine attributes from previous run attempts may be kept. The number of run attempts to store may be extended beyond the system\-specified history length by using the submit file command .TP .B job_machine_attrs_history_length A machine attribute named \fBX\fP will be inserted into the job ClassAd as an attribute named \fBMachineAttrX0\fP\&. The previous value of this attribute will be named \fBMachineAttrX1\fP, the previous to that will be named \fBMachineAttrX2\fP, and so on, up to the specified history length. A history of length 1 means that only \fBMachineAttrX0\fP will be recorded. The value recorded in the job ClassAd is the evaluation of the machine attribute in the context of the job ClassAd when the \fIcondor_schedd\fP daemon initiates the start up of the job. If the evaluation results in an \fBUndefined\fP or \fBError\fP result, the value recorded in the job ad will be \fBUndefined\fP or \fBError\fP, respectively. .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B want_graceful_removal = If \fBtrue\fP, this job will be given a chance to shut down cleanly when removed. The job will be given as much time as the administrator of the execute resource allows, which may be none. The default is \fBfalse\fP\&. For details, see the configuration setting \fI\%GRACEFULLY_REMOVE_JOBS\fP\&. .TP .B kill_sig = When HTCondor needs to kick a job off of a machine, it will send the job the signal specified by \fBsignal\-number\fP, which needs to be an integer which represents a valid signal on the execution machine. The default value is SIGTERM, which is the standard way to terminate a program in Unix. .TP .B kill_sig_timeout = This submit command should no longer be used. \fBjob_max_vacate_time\fP instead. If \fBjob_max_vacate_time\fP is not defined, this defines the number of seconds that HTCondor should wait following the sending of the kill signal defined by \fBkill_sig\fP and forcibly killing the job. The actual amount of time between sending the signal and forcibly killing the job is the smallest of this value and the configuration variable \fI\%KILLING_TIMEOUT\fP , as defined on the execute machine. .TP .B load_profile = When \fBTrue\fP, loads the account profile of the dedicated run account for Windows jobs. May not be used with \fBrun_as_owner\fP\&. .TP .B log_xml = If \fBlog_xml\fP is \fBTrue\fP, then the job event log file will be written in ClassAd XML. If not specified, XML is not used. Note that the file is an XML fragment; it is missing the file header and footer. Do not mix XML and non\-XML within a single file. If multiple jobs write to a single job event log file, ensure that all of the jobs specify this option in the same way. .TP .B match_list_length = Defaults to the value zero (0). When \fBmatch_list_length\fP is defined with an integer value greater than zero (0), attributes are inserted into the job ClassAd. The maximum number of attributes defined is given by the integer value. The job ClassAds introduced are given as .INDENT 7.0 .INDENT 3.5 .sp .EX LastMatchName0 = \(dqmost\-recent\-Name\(dq LastMatchName1 = \(dqnext\-most\-recent\-Name\(dq .EE .UNINDENT .UNINDENT .sp The value for each introduced ClassAd is given by the value of the \fBName\fP attribute from the machine ClassAd of a previous execution (match). As a job is matched, the definitions for these attributes will roll, with LastMatchName1 becoming LastMatchName2, LastMatchName0 becoming LastMatchName1, and LastMatchName0 being set by the most recent value of the \fBName\fP attribute. .sp An intended use of these job attributes is in the requirements expression. The requirements can allow a job to prefer a match with either the same or a different resource than a previous match. .TP .B job_max_vacate_time = An integer\-valued expression (in seconds) that may be used to adjust the time given to an evicted job for gracefully shutting down. If the job\(aqs setting is less than the machine\(aqs, the job\(aqs is used. If the job\(aqs setting is larger than the machine\(aqs, the result depends on whether the job has any excess retirement time. If the job has more retirement time left than the machine\(aqs max vacate time setting, then retirement time will be converted into vacating time, up to the amount requested by the job. .sp Setting this expression does not affect the job\(aqs resource requirements or preferences. For a job to only run on a machine with a minimum \fI\%MachineMaxVacateTime\fP, or to preferentially run on such machines, explicitly specify this in the requirements and/or rank expressions. .TP .B manifest = For vanilla and Docker \-universe jobs (and others that use the shadow), specifies if HTCondor (the starter) should produce a \(dqmanifest\(dq, which is directory containing three files: the list of files and directories at the top level of the sandbox when file transfer in completes (\fBin\fP), the same when file transfer out begins (\fBout\fP), and a dump of the environment set for the job (\fBenvironment\fP). .sp This feature is not presently available for Windows. .TP .B manifest_dir = For vanilla and Docker \-universe jobs (and others that use the shadow), specifies the directory in which to record the manifest. Specifying this enables the creation of a manifest. By default, the manifest directory is named \fB__manifest\fP, to avoid conflicts. .sp This feature is not presently available for Windows. .TP .B max_job_retirement_time = An integer\-valued expression (in seconds) that does nothing unless the machine that runs the job has been configured to provide retirement time. Retirement time is a grace period given to a job to finish when a resource claim is about to be preempted. The default behavior in many cases is to take as much retirement time as the machine offers, so this command will rarely appear in a submit description file. .sp When a resource claim is to be preempted, this expression in the submit file specifies the maximum run time of the job (in seconds, since the job started). This expression has no effect, if it is greater than the maximum retirement time provided by the machine policy. If the resource claim is not preempted, this expression and the machine retirement policy are irrelevant. If the resource claim is preempted the job will be allowed to run until the retirement time expires, at which point it is hard\-killed. The job will be soft\-killed when it is getting close to the end of retirement in order to give it time to gracefully shut down. The amount of lead\-time for soft\-killing is determined by the maximum vacating time granted to the job. .sp Any jobs running with \fI\%nice_user\fP priority have a default \fBmax_job_retirement_time\fP of 0, so no retirement time is utilized by default. In all other cases, no default value is provided, so the maximum amount of retirement time is utilized by default. .sp Setting this expression does not affect the job\(aqs resource requirements or preferences. For a job to only run on a machine with a minimum \fBMaxJobRetirementTime\fP, or to preferentially run on such machines, explicitly specify this in the requirements and/or rank expressions. .TP .B nice_user = Normally, when a machine becomes available to HTCondor, HTCondor decides which job to run based upon user and job priorities. Setting \fBnice_user\fP equal to \fBTrue\fP tells HTCondor not to use your regular user priority, but that this job should have last priority among all users and all jobs. So jobs submitted in this fashion run only on machines which no other non\-nice_user job wants \- a true bottom\-feeder job! This is very handy if a user has some jobs they wish to run, but do not wish to use resources that could instead be used to run other people\(aqs HTCondor jobs. Jobs submitted in this fashion have an accounting group. The accounting group is configurable by setting \fI\%NICE_USER_ACCOUNTING_GROUP_NAME\fP which defaults to \fBnice\-user\fP The default value is \fBFalse\fP\&. .TP .B noop_job = When this boolean expression is \fBTrue\fP, the job is immediately removed from the queue, and HTCondor makes no attempt at running the job. The log file for the job will show a job submitted event and a job terminated event, along with an exit code of 0, unless the user specifies a different signal or exit code. .TP .B noop_job_exit_code = When \fBnoop_job\fP is in the submit description file and evaluates to \fBTrue\fP, this command allows the job to specify the return value as shown in the job\(aqs log file job terminated event. If not specified, the job will show as having terminated with status 0. This overrides any value specified with \fBnoop_job_exit_signal\fP\&. .TP .B noop_job_exit_signal = When \fBnoop_job\fP is in the submit description file and evaluates to \fBTrue\fP, this command allows the job to specify the signal number that the job\(aqs log event will show the job having terminated with. .TP .B remote_initialdir = The path specifies the directory in which the job is to be executed on the remote machine. .TP .B rendezvousdir = Used to specify the shared file system directory to be used for file system authentication when submitting to a remote scheduler. Should be a path to a preexisting directory. .TP .B run_as_owner = A boolean value that causes the job to be run under the login of the submitter, if supported by the joint configuration of the submit and execute machines. On Unix platforms, this defaults to \fBTrue\fP, and on Windows platforms, it defaults to \fBFalse\fP\&. May not be used with \fBload_profile\fP\&. See the HTCondor manual Platform\-Specific Information chapter for administrative details on configuring Windows to support this option. .TP .B stack_size = This command applies only to Linux platforms. An integer number of bytes, representing the amount of stack space to be allocated for the job. This value replaces the default allocation of stack space, which is unlimited in size. .TP .B submit_event_notes = A string that is appended to the submit event in the job\(aqs log file. For DAGMan jobs, the string \fBDAG Node:\fP and the node\(aqs name is automatically defined for \fBsubmit_event_notes\fP, causing the logged submit event to identify the DAG node job submitted. .TP .B ulog_execute_attrs = A comma\-separated list of machine ClassAd attribute names. The named attributes and their values are written as part of the execution event in the job event log. .TP .B use_oauth_services = A comma\-separated list of credential\-providing service names for which the job should be provided credentials for the job execution environment. The credential service providers must be configured by the pool admin. .TP .B _oauth_permissions[_] = A string containing the scope(s) that should be requested for the credential named [_], where is optionally provided to differentiate between multiple credentials from the same credential service provider. .TP .B _oauth_resource[_] = A string containing the resource (or \(dqaudience\(dq) that should be requested for the credential named [_], where is optionally provided to differentiate between multiple credentials from the same credential service provider. .TP .B MY. = or + = A macro that begins with MY. or a line that begins with a \(aq+\(aq (plus) character instructs \fIcondor_submit\fP to insert the given \fIattribute\fP (without + or MY.) into the job ClassAd with the given \fIvalue\fP\&. The macro can be referenced in other submit statements by using \fB$(MY.)\fP\&. A + is converted to MY. when the file is read. .sp Note that setting an job attribute in this way should not be used in place of one of the specific commands listed above. Often, the command name does not directly correspond to an attribute name; furthermore, many submit commands result in actions more complex than simply setting an attribute or attributes. See \fI\%Job ClassAd Attributes\fP for a list of HTCondor job attributes. .UNINDENT .UNINDENT .UNINDENT .sp MACROS AND COMMENTS .sp In addition to commands, the submit description file can contain macros and comments. .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B Macros Parameterless macros in the form of \fB$(macro_name:default initial value)\fP may be used anywhere in HTCondor submit description files to provide textual substitution at submit time. Macros can be defined by lines in the form of .INDENT 7.0 .INDENT 3.5 .sp .EX = .EE .UNINDENT .UNINDENT .sp Several pre\-defined macros are supplied by the submit description file parser. The \fB$(Cluster)\fP or \fB$(ClusterId)\fP macro supplies the value of the \fBClusterId\fP job ClassAd attribute, and the \fB$(Process)\fP or \fB$(ProcId)\fP macro supplies the value of the \fBProcId\fP job ClassAd attribute. The \fB$(JobId)\fP macro supplies the full job id. It is equivalent to \fB$(ClusterId).$(ProcId)\fP\&. These macros are intended to aid in the specification of input/output files, arguments, etc., for clusters with lots of jobs, and/or could be used to supply an HTCondor process with its own cluster and process numbers on the command line. .sp The \fB$(Node)\fP macro is defined for parallel universe jobs, and is especially relevant for MPI applications. It is a unique value assigned for the duration of the job that essentially identifies the machine (slot) on which a program is executing. Values assigned start at 0 and increase monotonically. The values are assigned as the parallel job is about to start. .sp Recursive definition of macros is permitted. An example of a construction that works is the following: .INDENT 7.0 .INDENT 3.5 .sp .EX foo = bar foo = snap $(foo) .EE .UNINDENT .UNINDENT .sp As a result, \fBfoo = snap bar\fP\&. .sp Note that both left\- and right\- recursion works, so .INDENT 7.0 .INDENT 3.5 .sp .EX foo = bar foo = $(foo) snap .EE .UNINDENT .UNINDENT .sp has as its result \fBfoo = bar snap\fP\&. .sp The construction .INDENT 7.0 .INDENT 3.5 .sp .EX foo = $(foo) bar .EE .UNINDENT .UNINDENT .sp by itself will not work, as it does not have an initial base case. Mutually recursive constructions such as: .INDENT 7.0 .INDENT 3.5 .sp .EX B = bar C = $(B) B = $(C) boo .EE .UNINDENT .UNINDENT .sp will not work, and will fill memory with expansions. .sp A default value may be specified, for use if the macro has no definition. Consider the example .INDENT 7.0 .INDENT 3.5 .sp .EX D = $(E:24) .EE .UNINDENT .UNINDENT .sp Where \fBE\fP is not defined within the submit description file, the default value 24 is used, resulting in .INDENT 7.0 .INDENT 3.5 .sp .EX D = 24 .EE .UNINDENT .UNINDENT .sp This is useful for creating submit templates where values can be passed on the \fIcondor_submit\fP command line, but that have a default value as well. In the above example, if you give a value for E on the command line like this .INDENT 7.0 .INDENT 3.5 .sp .EX condor_submit E=99 .EE .UNINDENT .UNINDENT .sp The value of 99 is used for E, resulting in .INDENT 7.0 .INDENT 3.5 .sp .EX D = 99 .EE .UNINDENT .UNINDENT .sp .sp To use the dollar sign character ($) as a literal, without macro expansion, use .INDENT 7.0 .INDENT 3.5 .sp .EX $(DOLLAR) .EE .UNINDENT .UNINDENT .sp In addition to the normal macro, there is also a special kind of macro called a substitution macro that allows the substitution of a machine ClassAd attribute value defined on the resource machine itself (gotten after a match to the machine has been made) into specific commands within the submit description file. The substitution macro is of the form: .INDENT 7.0 .INDENT 3.5 .sp .EX $$(attribute) .EE .UNINDENT .UNINDENT .sp As this form of the substitution macro is only evaluated within the context of the machine ClassAd, use of a scope resolution prefix \fBTARGET.\fP or \fBMY.\fP is not allowed. .sp A common use of this form of the substitution macro is for the heterogeneous submission of an executable: .INDENT 7.0 .INDENT 3.5 .sp .EX executable = povray.$$(OpSys).$$(Arch) .EE .UNINDENT .UNINDENT .sp Values for the \fBOpSys\fP and \fBArch\fP attributes are substituted at match time for any given resource. This example allows HTCondor to automatically choose the correct executable for the matched machine. .sp An extension to the syntax of the substitution macro provides an alternative string to use if the machine attribute within the substitution macro is undefined. The syntax appears as: .INDENT 7.0 .INDENT 3.5 .sp .EX $$(attribute:string_if_attribute_undefined) .EE .UNINDENT .UNINDENT .sp An example using this extended syntax provides a path name to a required input file. Since the file can be placed in different locations on different machines, the file\(aqs path name is given as an argument to the program. .INDENT 7.0 .INDENT 3.5 .sp .EX arguments = $$(input_file_path:/usr/foo) .EE .UNINDENT .UNINDENT .sp On the machine, if the attribute \fBinput_file_path\fP is not defined, then the path \fB/usr/foo\fP is used instead. .sp As a special case that only works within the submit file \fIenvironment\fP command, the string $$(CondorScratchDir) is expanded to the value of the job\(aqs scratch directory. This does not work for scheduler universe or grid universe jobs. .sp For example, to set PYTHONPATH to a subdirectory of the job scratch dir, one could set .INDENT 7.0 .INDENT 3.5 .sp .EX environment = PYTHONPATH=$$(CondorScratchDir)/some/directory .EE .UNINDENT .UNINDENT .sp A further extension to the syntax of the substitution macro allows the evaluation of a ClassAd expression to define the value. In this form, the expression may refer to machine attributes by prefacing them with the \fBTARGET.\fP scope resolution prefix. To place a ClassAd expression into the substitution macro, square brackets are added to delimit the expression. The syntax appears as: .INDENT 7.0 .INDENT 3.5 .sp .EX $$([ClassAd expression]) .EE .UNINDENT .UNINDENT .sp An example of a job that uses this syntax may be one that wants to know how much memory it can use. The application cannot detect this itself, as it would potentially use all of the memory on a multi\-slot machine. So the job determines the memory per slot, reducing it by 10% to account for miscellaneous overhead, and passes this as a command line argument to the application. In the submit description file will be .INDENT 7.0 .INDENT 3.5 .sp .EX arguments = \-\-memory $$([TARGET.Memory * 0.9]) .EE .UNINDENT .UNINDENT .sp .sp To insert two dollar sign characters ($$) as literals into a ClassAd string, use .INDENT 7.0 .INDENT 3.5 .sp .EX $$(DOLLARDOLLAR) .EE .UNINDENT .UNINDENT .sp .sp The environment macro, $ENV, allows the evaluation of an environment variable to be used in setting a submit description file command. The syntax used is .INDENT 7.0 .INDENT 3.5 .sp .EX $ENV(variable) .EE .UNINDENT .UNINDENT .sp An example submit description file command that uses this functionality evaluates the submitter\(aqs home directory in order to set the path and file name of a log file: .INDENT 7.0 .INDENT 3.5 .sp .EX log = $ENV(HOME)/jobs/logfile .EE .UNINDENT .UNINDENT .sp The environment variable is evaluated when the submit description file is processed. .sp The $RANDOM_CHOICE macro allows a random choice to be made from a given list of parameters at submission time. For an expression, if some randomness needs to be generated, the macro may appear as .INDENT 7.0 .INDENT 3.5 .sp .EX $RANDOM_CHOICE(0,1,2,3,4,5,6) .EE .UNINDENT .UNINDENT .sp When evaluated, one of the parameters values will be chosen. .TP .B Comments Blank lines and lines beginning with a pound sign (\(aq#\(aq) character are ignored by the submit description file parser. .UNINDENT .UNINDENT .UNINDENT .SH SUBMIT VARIABLES .sp .sp While processing the \fI\%queue\fP command in a submit file or from the command line, \fIcondor_submit\fP will set the values of several automatic submit variables so that they can be referred to by statements in the submit file. With the exception of Cluster and Process, if these variables are set by the submit file, they will not be modified during \fI\%queue\fP processing. .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B ClusterId Set to the integer value that the \fBClusterId\fP attribute that the job ClassAd will have when the job is submitted. All jobs in a single submit will normally have the same value for the \fBClusterId\fP\&. If the \fB\-dry\-run\fP argument is specified, The value will be 1. .TP .B Cluster Alternate name for the ClusterId submit variable. Before HTCondor version 8.4 this was the only name. .TP .B ProcId Set to the integer value that the \fBProcId\fP attribute of the job ClassAd will have when the job is submitted. The value will start at 0 and increment by 1 for each job submitted. .TP .B Process Alternate name for the ProcId submit variable. Before HTCondor version 8.4 this was the only name. .TP .B JobId Set to \fB$(ClusterId).$(ProcId)\fP so that it will expand to the full id of the job. .TP .B Node For parallel universes, set to the value #pArAlLeLnOdE# or #MpInOdE# depending on the parallel universe type For other universes it is set to nothing. .TP .B Step Set to the step value as it varies from 0 to N\-1 where N is the number provided on the \fI\%queue\fP argument. This variable changes at the same rate as ProcId when it changes at all. For submit files that don\(aqt make use of the queue number option, Step will always be 0. For submit files that don\(aqt make use of any of the foreach options, Step and ProcId will always be the same. .TP .B ItemIndex Set to the index within the item list being processed by the various queue foreach options. For submit files that don\(aqt make use of any queue foreach list, ItemIndex will always be 0 For submit files that make use of a slice to select only some items in a foreach list, ItemIndex will only be set to selected values. .TP .B Row Alternate name for ItemIndex. .TP .B Item when a queue foreach option is used and no variable list is supplied, this variable will be set to the value of the current item. .UNINDENT .UNINDENT .UNINDENT .sp \fBThe automatic variables below are set before parsing the submit file, and will not vary during processing unless the submit file itself sets them.\fP .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B ARCH Set to the CPU architecture of the machine running \fIcondor_submit\fP\&. The value will be the same as the automatic configuration variable of the same name. .TP .B OPSYS Set to the name of the operating system on the machine running \fIcondor_submit\fP\&. The value will be the same as the automatic configuration variable of the same name. .TP .B OPSYSANDVER Set to the name and major version of the operating system on the machine running \fIcondor_submit\fP\&. The value will be the same as the automatic configuration variable of the same name. .TP .B OPSYSMAJORVER Set to the major version of the operating system on the machine running \fIcondor_submit\fP\&. The value will be the same as the automatic configuration variable of the same name. .TP .B OPSYSVER Set to the version of the operating system on the machine running \fIcondor_submit\fP\&. The value will be the same as the automatic configuration variable of the same name. .TP .B SPOOL Set to the full path of the HTCondor spool directory. The value will be the same as the automatic configuration variable of the same name. .TP .B IsLinux Set to true if the operating system of the machine running \fIcondor_submit\fP is a Linux variant. Set to false otherwise. .TP .B IsWindows Set to true if the operating system of the machine running \fIcondor_submit\fP is a Microsoft Windows variant. Set to false otherwise. .TP .B SUBMIT_FILE Set to the full pathname of the submit file being processed by \fIcondor_submit\fP\&. If submit statements are read from standard input, it is set to nothing. .TP .B SUBMIT_TIME Set to the unix timestamp of the current time when the job is submitted. .TP .B YEAR Set to the 4 digit year when the job is submitted. .TP .B MONTH Set to the 2 digit month when the job is submitted. .TP .B DAY Set to the 2 digit day when the job is submitted. .UNINDENT .UNINDENT .UNINDENT .SH EXIT STATUS .sp \fIcondor_submit\fP will exit with a status value of 0 (zero) upon success, and a non\-zero value upon failure. .SH EXAMPLES .INDENT 0.0 .IP \(bu 2 Submit Description File Example 1: This example queues three jobs for execution by HTCondor. The first will be given command line arguments of \fI15\fP and \fI2000\fP, and it will write its standard output to \fBfoo.out1\fP\&. The second will be given command line arguments of \fI30\fP and \fI2000\fP, and it will write its standard output to \fBfoo.out2\fP\&. Similarly the third will have arguments of \fI45\fP and \fI6000\fP, and it will use \fBfoo.out3\fP for its standard output. Standard error output (if any) from all three programs will appear in \fBfoo.error\fP\&. .INDENT 2.0 .INDENT 3.5 .sp .EX #################### # # submit description file # Example 1: queuing multiple jobs with differing # command line arguments and output files. # #################### Executable = foo Universe = vanilla Arguments = 15 2000 Output = foo.out0 Error = foo.err0 Queue Arguments = 30 2000 Output = foo.out1 Error = foo.err1 Queue Arguments = 45 6000 Output = foo.out2 Error = foo.err2 Queue .EE .UNINDENT .UNINDENT .sp Or you can get the same results as the above submit file by using a list of arguments with the Queue statement .INDENT 2.0 .INDENT 3.5 .sp .EX #################### # # submit description file # Example 1b: queuing multiple jobs with differing # command line arguments and output files, alternate syntax # #################### Executable = foo Universe = vanilla # generate different output and error filenames for each process Output = foo.out$(Process) Error = foo.err$(Process) Queue Arguments From ( 15 2000 30 2000 45 6000 ) .EE .UNINDENT .UNINDENT .IP \(bu 2 Submit Description File Example 2: This submit description file example queues 150 runs of program \fIfoo\fP which must have been compiled and linked for an Intel x86 processor running RHEL 3. HTCondor will not attempt to run the processes on machines which have less than 32 Megabytes of physical memory, and it will run them on machines which have at least 64 Megabytes, if such machines are available. Stdin, stdout, and stderr will refer to \fBin.0\fP, \fBout.0\fP, and \fBerr.0\fP for the first run of this program (process 0). Stdin, stdout, and stderr will refer to \fBin.1\fP, \fBout.1\fP, and \fBerr.1\fP for process 1, and so forth. A log file containing entries about where and when HTCondor runs, transfers file, if it\(aqs evicted, and when it terminates, among other things, the various processes in this cluster will be written into file \fBfoo.log\fP\&. .INDENT 2.0 .INDENT 3.5 .sp .EX #################### # # Example 2: Show off some fancy features including # use of pre\-defined macros and logging. # #################### Executable = foo Universe = vanilla Requirements = OpSys == \(dqLINUX\(dq && Arch ==\(dqINTEL\(dq Rank = Memory >= 64 Request_Memory = 32 Mb Image_Size = 28 Mb Error = err.$(Process) Input = in.$(Process) Output = out.$(Process) Log = foo.log Queue 150 .EE .UNINDENT .UNINDENT .IP \(bu 2 Submit Description File Example 3: This example targets the \fI/bin/sleep\fP program to run only on a platform running a RHEL 6 operating system. The example presumes that the pool contains machines running more than one version of Linux, and this job needs the particular operating system to run correctly. .INDENT 2.0 .INDENT 3.5 .sp .EX #################### # # Example 3: Run on a RedHat 6 machine # #################### Universe = vanilla Executable = /bin/sleep Arguments = 30 Requirements = (OpSysAndVer == \(dqRedHat6\(dq) Error = err.$(Process) Input = in.$(Process) Output = out.$(Process) Log = sleep.log Queue .EE .UNINDENT .UNINDENT .IP \(bu 2 Command Line example: The following command uses the \fB\-append\fP option to add two commands before the job(s) is queued. A log file and an error log file are specified. The submit description file is unchanged. .INDENT 2.0 .INDENT 3.5 .sp .EX $ condor_submit \-a \(dqlog = out.log\(dq \-a \(dqerror = error.log\(dq mysubmitfile .EE .UNINDENT .UNINDENT .sp Note that each of the added commands is contained within quote marks because there are space characters within the command. .IP \(bu 2 \fBperiodic_remove\fP example: A job should be removed from the queue, if the total suspension time of the job is more than half of the run time of the job. .sp Including the command .INDENT 2.0 .INDENT 3.5 .sp .EX periodic_remove = CumulativeSuspensionTime > ((RemoteWallClockTime \- CumulativeSuspensionTime) / 2.0) .EE .UNINDENT .UNINDENT .sp in the submit description file causes this to happen. .UNINDENT .SH GENERAL REMARKS .INDENT 0.0 .IP \(bu 2 For security reasons, HTCondor will refuse to run any jobs submitted by user root (UID = 0) or by a user whose default group is group wheel (GID = 0). Jobs submitted by user root or a user with a default group of wheel will appear to sit forever in the queue in an idle state. .IP \(bu 2 All path names specified in the submit description file must be less than 256 characters in length, and command line arguments must be less than 4096 characters in length; otherwise, \fIcondor_submit\fP gives a warning message but the jobs will not execute properly. .IP \(bu 2 Somewhat understandably, behavior gets bizarre if the user makes the mistake of requesting multiple HTCondor jobs to write to the same file, and/or if the user alters any files that need to be accessed by an HTCondor job which is still in the queue. For example, the compressing of data or output files before an HTCondor job has completed is a common mistake. .UNINDENT .SH SEE ALSO .sp HTCondor User Manual .SH AUTHOR HTCondor Team .SH COPYRIGHT 1990-2024, Center for High Throughput Computing, Computer Sciences Department, University of Wisconsin-Madison, Madison, WI, US. Licensed under the Apache License, Version 2.0. .\" Generated by docutils manpage writer. .