.TH "slurm.conf" "5" "Slurm Configuration File" "December 2018" "Slurm Configuration File" .SH "NAME" slurm.conf \- Slurm configuration file .SH "DESCRIPTION" \fBslurm.conf\fP is an ASCII file which describes general Slurm configuration information, the nodes to be managed, information about how those nodes are grouped into partitions, and various scheduling parameters associated with those partitions. This file should be consistent across all nodes in the cluster. .LP The file location can be modified at system build time using the DEFAULT_SLURM_CONF parameter or at execution time by setting the SLURM_CONF environment variable. The Slurm daemons also allow you to override both the built\-in and environment\-provided location using the "\-f" option on the command line. .LP The contents of the file are case insensitive except for the names of nodes and partitions. Any text following a "#" in the configuration file is treated as a comment through the end of that line. Changes to the configuration file take effect upon restart of Slurm daemons, daemon receipt of the SIGHUP signal, or execution of the command "scontrol reconfigure" unless otherwise noted. .LP If a line begins with the word "Include" followed by whitespace and then a file name, that file will be included inline with the current configuration file. For large or complex systems, multiple configuration files may prove easier to manage and enable reuse of some files (See INCLUDE MODIFIERS for more details). .LP Note on file permissions: .LP The \fIslurm.conf\fR file must be readable by all users of Slurm, since it is used by many of the Slurm commands. Other files that are defined in the \fIslurm.conf\fR file, such as log files and job accounting files, may need to be created/owned by the user "SlurmUser" to be successfully accessed. Use the "chown" and "chmod" commands to set the ownership and permissions appropriately. See the section \fBFILE AND DIRECTORY PERMISSIONS\fR for information about the various files and directories used by Slurm. .SH "PARAMETERS" .LP The overall configuration parameters available include: .TP \fBAccountingStorageBackupHost\fR The name of the backup machine hosting the accounting storage database. If used with the accounting_storage/slurmdbd plugin, this is where the backup slurmdbd would be running. Only used with systems using SlurmDBD, ignored otherwise. .TP \fBAccountingStorageEnforce\fR This controls what level of association\-based enforcement to impose on job submissions. Valid options are any combination of \fIassociations\fR, \fIlimits\fR, \fInojobs\fR, \fInosteps\fR, \fIqos\fR, \fIsafe\fR, and \fIwckeys\fR, or \fIall\fR for all things (expect nojobs and nosteps, they must be requested as well). If limits, qos, or wckeys are set, associations will automatically be set. If wckeys is set, TrackWCKey will automatically be set. If safe is set, limits and associations will automatically be set. If nojobs is set nosteps will automatically be set. By enforcing Associations no new job is allowed to run unless a corresponding association exists in the system. If limits are enforced users can be limited by association to whatever job size or run time limits are defined. If nojobs is set Slurm will not account for any jobs or steps on the system, like wise if nosteps is set Slurm will not account for any steps ran limits will still be enforced. If safe is enforced a job will only be launched against an association or qos that has a GrpCPUMins limit set if the job will be able to run to completion. Without this option set, jobs will be launched as long as their usage hasn't reached the cpu-minutes limit which can lead to jobs being launched but then killed when the limit is reached. With qos and/or wckeys enforced jobs will not be scheduled unless a valid qos and/or workload characterization key is specified. When \fBAccountingStorageEnforce\fR is changed, a restart of the slurmctld daemon is required (not just a "scontrol reconfig"). .TP \fBAccountingStorageHost\fR The name of the machine hosting the accounting storage database. Only used with systems using SlurmDBD, ignored otherwise. Also see \fBDefaultStorageHost\fR. .TP \fBAccountingStorageLoc\fR The fully qualified file name where accounting records are written when the \fBAccountingStorageType\fR is "accounting_storage/filetxt". Also see \fBDefaultStorageLoc\fR. .TP \fBAccountingStoragePass\fR The password used to gain access to the database to store the accounting data. Only used for database type storage plugins, ignored otherwise. In the case of Slurm DBD (Database Daemon) with MUNGE authentication this can be configured to use a MUNGE daemon specifically configured to provide authentication between clusters while the default MUNGE daemon provides authentication within a cluster. In that case, \fBAccountingStoragePass\fR should specify the named port to be used for communications with the alternate MUNGE daemon (e.g. "/var/run/munge/global.socket.2"). The default value is NULL. Also see \fBDefaultStoragePass\fR. .TP \fBAccountingStoragePort\fR The listening port of the accounting storage database server. Only used for database type storage plugins, ignored otherwise. Also see \fBDefaultStoragePort\fR. .TP \fBAccountingStorageTRES\fR Comma separated list of resources you wish to track on the cluster. These are the resources requested by the sbatch/srun job when it is submitted. Currently this consists of any GRES, BB (burst buffer) or license along with CPU, Memory, Node, Energy, FS/[Disk|Lustre], IC/OFED, Pages, and VMem. By default Billing, CPU, Energy, Memory, Node, FS/Disk, Pages and VMem are tracked. AccountingStorageTRES=gres/craynetwork,license/iop1 will track billing, cpu, energy, memory, nodes, FS/disk along with a gres called craynetwork as well as a license called iop1. Whenever these resources are used on the cluster they are recorded. The TRES are automatically set up in the database on the start of the slurmctld. .TP \fBAccountingStorageType\fR The accounting storage mechanism type. Acceptable values at present include "accounting_storage/filetxt", "accounting_storage/none" and "accounting_storage/slurmdbd". The "accounting_storage/filetxt" value indicates that accounting records will be written to the file specified by the \fBAccountingStorageLoc\fR parameter. The "accounting_storage/slurmdbd" value indicates that accounting records will be written to the Slurm DBD, which manages an underlying MySQL database. See "man slurmdbd" for more information. The default value is "accounting_storage/none" and indicates that account records are not maintained. Note: The filetxt plugin records only a limited subset of accounting information and will prevent some sacct options from proper operation. Also see \fBDefaultStorageType\fR. .TP \fBAccountingStorageUser\fR The user account for accessing the accounting storage database. Only used for database type storage plugins, ignored otherwise. Also see \fBDefaultStorageUser\fR. .TP \fBAccountingStoreJobComment\fR If set to "YES" then include the job's comment field in the job complete message sent to the Accounting Storage database. The default is "YES". Note the AdminComment and SystemComment are always recorded in the database. .TP \fBAcctGatherNodeFreq\fR The AcctGather plugins sampling interval for node accounting. For AcctGather plugin values of none, this parameter is ignored. For all other values this parameter is the number of seconds between node accounting samples. For the acct_gather_energy/rapl plugin, set a value less than 300 because the counters may overflow beyond this rate. The default value is zero. This value disables accounting sampling for nodes. Note: The accounting sampling interval for jobs is determined by the value of \fBJobAcctGatherFrequency\fR. .TP \fBAcctGatherEnergyType\fR Identifies the plugin to be used for energy consumption accounting. The jobacct_gather plugin and slurmd daemon call this plugin to collect energy consumption data for jobs and nodes. The collection of energy consumption data takes place on the node level, hence only in case of exclusive job allocation the energy consumption measurements will reflect the job's real consumption. In case of node sharing between jobs the reported consumed energy per job (through sstat or sacct) will not reflect the real energy consumed by the jobs. Configurable values at present are: .RS .TP 20 \fBacct_gather_energy/none\fR No energy consumption data is collected. .TP \fBacct_gather_energy/ipmi\fR Energy consumption data is collected from the Baseboard Management Controller (BMC) using the Intelligent Platform Management Interface (IPMI). .TP \fBacct_gather_energy/rapl\fR Energy consumption data is collected from hardware sensors using the Running Average Power Limit (RAPL) mechanism. Note that enabling RAPL may require the execution of the command "sudo modprobe msr". .RE .TP \fBAcctGatherInfinibandType\fR Identifies the plugin to be used for infiniband network traffic accounting. The jobacct_gather plugin and slurmd daemon call this plugin to collect network traffic data for jobs and nodes. The collection of network traffic data takes place on the node level, hence only in case of exclusive job allocation the collected values will reflect the job's real traffic. In case of node sharing between jobs the reported network traffic per job (through sstat or sacct) will not reflect the real network traffic by the jobs. Configurable values at present are: .RS .TP 20 \fBacct_gather_infiniband/none\fR No infiniband network data are collected. .TP \fBacct_gather_infiniband/ofed\fR Infiniband network traffic data are collected from the hardware monitoring counters of Infiniband devices through the OFED library. In order to account for per job network traffic, add the "ic/ofed" TRES to \fIAccountingStorageTRES\fR. .RE .TP \fBAcctGatherFilesystemType\fR Identifies the plugin to be used for filesystem traffic accounting. The jobacct_gather plugin and slurmd daemon call this plugin to collect filesystem traffic data for jobs and nodes. The collection of filesystem traffic data takes place on the node level, hence only in case of exclusive job allocation the collected values will reflect the job's real traffic. In case of node sharing between jobs the reported filesystem traffic per job (through sstat or sacct) will not reflect the real filesystem traffic by the jobs. Configurable values at present are: .RS .TP 20 \fBacct_gather_filesystem/none\fR No filesystem data are collected. .TP \fBacct_gather_filesystem/lustre\fR Lustre filesystem traffic data are collected from the counters found in /proc/fs/lustre/. In order to account for per job lustre traffic, add the "fs/lustre" TRES to \fIAccountingStorageTRES\fR. .RE .TP \fBAcctGatherProfileType\fR Identifies the plugin to be used for detailed job profiling. The jobacct_gather plugin and slurmd daemon call this plugin to collect detailed data such as I/O counts, memory usage, or energy consumption for jobs and nodes. There are interfaces in this plugin to collect data as step start and completion, task start and completion, and at the account gather frequency. The data collected at the node level is related to jobs only in case of exclusive job allocation. Configurable values at present are: .RS .TP 20 \fBacct_gather_profile/none\fR No profile data is collected. .TP \fBacct_gather_profile/hdf5\fR This enables the HDF5 plugin. The directory where the profile files are stored and which values are collected are configured in the acct_gather.conf file. .TP \fBacct_gather_profile/influxdb\fR This enables the influxdb plugin. The influxdb instance host, port, database, retention policy and which values are collected are configured in the acct_gather.conf file. .RE .TP \fBAllowSpecResourcesUsage\fR If set to 1, Slurm allows individual jobs to override node's configured CoreSpecCount value. For a job to take advantage of this feature, a command line option of \-\-core\-spec must be specified. The default value for this option is 1 for Cray systems and 0 for other system types. .TP \fBAuthInfo\fR Additional information to be used for authentication of communications between the Slurm daemons (slurmctld and slurmd) and the Slurm clients. The interpretation of this option is specific to the configured \fBAuthType\fR. Multiple options may be specified in a comma delimited list. If not specified, the default authentication information will be used. .RS .TP 14 \fBcred_expire\fR Default job step credential lifetime, in seconds (e.g. "cred_expire=1200"). It must be sufficiently long enough to load user environment, run prolog, deal with the slurmd getting paged out of memory, etc. This also controls how long a requeued job must wait before starting again. The default value is 120 seconds. .TP \fBsocket\fR Path name to a MUNGE daemon socket to use (e.g. "socket=/var/run/munge/munge.socket.2"). The default value is "/var/run/munge/munge.socket.2". Used by \fIauth/munge\fR and \fIcrypto/munge\fR. .TP \fBttl\fR Credential lifetime, in seconds (e.g. "ttl=300"). The default value is dependent upon the MUNGE installation, but is typically 300 seconds. .RE .TP \fBAuthType\fR The authentication method for communications between Slurm components. Acceptable values at present include "auth/munge" and "auth/none". The default value is "auth/munge". "auth/none" includes the UID in each communication, but it is not verified. This may be fine for testing purposes, but \fBdo not use "auth/none" if you desire any security\fR. "auth/munge" indicates that MUNGE is to be used. (See "https://dun.github.io/munge/" for more information). All Slurm daemons and commands must be terminated prior to changing the value of \fBAuthType\fR and later restarted. .TP \fBBackupAddr\fR Defunct option, see \fBSlurmctldHost\fR. .TP \fBBackupController\fR Defunct option, see \fBSlurmctldHost\fR. The backup controller recovers state information from the \fBStateSaveLocation\fR directory, which must be readable and writable from both the primary and backup controllers. While not essential, it is recommended that you specify a backup controller. See the \fBRELOCATING CONTROLLERS\fR section if you change this. .TP \fBBatchStartTimeout\fR The maximum time (in seconds) that a batch job is permitted for launching before being considered missing and releasing the allocation. The default value is 10 (seconds). Larger values may be required if more time is required to execute the \fBProlog\fR, load user environment variables (for Moab spawned jobs), or if the slurmd daemon gets paged from memory. .br .br \fBNote\fR: The test for a job being successfully launched is only performed when the Slurm daemon on the compute node registers state with the slurmctld daemon on the head node, which happens fairly rarely. Therefore a job will not necessarily be terminated if its start time exceeds \fBBatchStartTimeout\fR. This configuration parameter is also applied to launch tasks and avoid aborting \fBsrun\fR commands due to long running \fBProlog\fR scripts. .TP \fBBurstBufferType\fR The plugin used to manage burst buffers. Acceptable values at present include "burst_buffer/none". More information later... .TP \fBCheckpointType\fR The system\-initiated checkpoint method to be used for user jobs. The slurmctld daemon must be restarted for a change in \fBCheckpointType\fR to take effect. Supported values presently include: .RS .TP 18 \fBcheckpoint/blcr\fR Berkeley Lab Checkpoint Restart (BLCR). NOTE: If a file is found at sbin/scch (relative to the Slurm installation location), it will be executed upon completion of the checkpoint. This can be a script used for managing the checkpoint files. NOTE: Slurm's BLCR logic only supports batch jobs. .TP \fBcheckpoint/none\fR no checkpoint support (default) .TP \fBcheckpoint/ompi\fR OpenMPI (version 1.3 or higher) .RE .TP \fBClusterName\fR The name by which this Slurm managed cluster is known in the accounting database. This is needed distinguish accounting records when multiple clusters report to the same database. Because of limitations in some databases, any upper case letters in the name will be silently mapped to lower case. In order to avoid confusion, it is recommended that the name be lower case. .TP \fBCommunicationParameters\fR Comma separated options identifying communication options. .RS .TP 15 \fBCheckGhalQuiesce\fR Used specifically on a Cray using an Aries Ghal interconnect. This will check to see if the system is quiescing when sending a message, and if so, we wait until it is done before sending. .TP \fBNoCtldInAddrAny\fR Used to directly bind to the address of what the node resolves to running the slurmctld instead of binding messages to any address on the node, which is the default. .TP \fBNoInAddrAny\fR Used to directly bind to the address of what the node resolves to instead of binding messages to any address on the node which is the default. This option is for all daemons/clients except for the slurmctld. .RE .TP \fBCompleteWait\fR The time, in seconds, given for a job to remain in COMPLETING state before any additional jobs are scheduled. If set to zero, pending jobs will be started as soon as possible. Since a COMPLETING job's resources are released for use by other jobs as soon as the \fBEpilog\fR completes on each individual node, this can result in very fragmented resource allocations. To provide jobs with the minimum response time, a value of zero is recommended (no waiting). To minimize fragmentation of resources, a value equal to \fBKillWait\fR plus two is recommended. In that case, setting \fBKillWait\fR to a small value may be beneficial. The default value of \fBCompleteWait\fR is zero seconds. The value may not exceed 65533. .TP \fBControlAddr\fR Defunct option, see \fBSlurmctldHost\fR. .TP \fBControlMachine\fR Defunct option, see \fBSlurmctldHost\fR. .TP \fBCoreSpecPlugin\fR Identifies the plugins to be used for enforcement of core specialization. The slurmd daemon must be restarted for a change in CoreSpecPlugin to take effect. Acceptable values at present include: .RS .TP 20 \fBcore_spec/cray\fR used only for Cray systems .TP \fBcore_spec/none\fR used for all other system types .RE .TP \fBCpuFreqDef\fR Default CPU frequency value or frequency governor to use when running a job step if it has not been explicitly set with the \-\-cpu\-freq option. Acceptable values at present include a numeric value (frequency in kilohertz) or one of the following governors: .RS .TP 14 \fBConservative\fR attempts to use the Conservative CPU governor .TP \fBOnDemand\fR attempts to use the OnDemand CPU governor .TP \fBPerformance\fR attempts to use the Performance CPU governor .TP \fBPowerSave\fR attempts to use the PowerSave CPU governor .RE There is no default value. If unset, no attempt to set the governor is made if the \-\-cpu\-freq option has not been set. .TP \fBCpuFreqGovernors\fR List of CPU frequency governors allowed to be set with the salloc, sbatch, or srun option \-\-cpu\-freq. Acceptable values at present include: .RS .TP 14 \fBConservative\fR attempts to use the Conservative CPU governor .TP \fBOnDemand\fR attempts to use the OnDemand CPU governor (a default value) .TP \fBPerformance\fR attempts to use the Performance CPU governor (a default value) .TP \fBPowerSave\fR attempts to use the PowerSave CPU governor .TP \fBUserSpace\fR attempts to use the UserSpace CPU governor (a default value) .RE The default is OnDemand, Performance and UserSpace. .TP \fBCryptoType\fR The cryptographic signature tool to be used in the creation of job step credentials. The slurmctld daemon must be restarted for a change in \fBCryptoType\fR to take effect. Acceptable values at present include "crypto/munge". The default value is "crypto/munge" and is the recommended. .TP \fBDebugFlags\fR Defines specific subsystems which should provide more detailed event logging. Multiple subsystems can be specified with comma separators. Most DebugFlags will result in verbose logging for the identified subsystems and could impact performance. Valid subsystems available today (with more to come) include: .RS .TP 17 \fBBackfill\fR Backfill scheduler details .TP \fBBackfillMap\fR Backfill scheduler to log a very verbose map of reserved resources through time. Combine with \fBBackfill\fR for a verbose and complete view of the backfill scheduler's work. .TP \fBBurstBuffer\fR Burst Buffer plugin .TP \fBCPU_Bind\fR CPU binding details for jobs and steps .TP \fBCpuFrequency\fR Cpu frequency details for jobs and steps using the \-\-cpu\-freq option. .TP \fBElasticsearch\fR Elasticsearch debug info .TP \fBEnergy\fR AcctGatherEnergy debug info .TP \fBExtSensors\fR External Sensors debug info .TP \fBFederation\fR Federation scheduling debug info .TP \fBFrontEnd\fR Front end node details .TP \fBGres\fR Generic resource details .TP \fBHeteroJobs\fR Heterogeneous job details .TP \fBGang\fR Gang scheduling details .TP \fBJobContainer\fR Job container plugin details .TP \fBLicense\fR License management details .TP \fBNodeFeatures\fR Node Features plugin debug info .TP \fBNO_CONF_HASH\fR Do not log when the slurm.conf files differs between Slurm daemons .TP \fBPower\fR Power management plugin .TP \fBPriority\fR Job prioritization .TP \fBProfile\fR AcctGatherProfile plugins details .TP \fBProtocol\fR Communication protocol details .TP \fBReservation\fR Advanced reservations .TP \fBSelectType\fR Resource selection plugin .TP \fBSteps\fR Slurmctld resource allocation for job steps .TP \fBSwitch\fR Switch plugin .TP \fBTimeCray\fR Timing of Cray APIs .TP \fBTraceJobs\fR Trace jobs in slurmctld. It will print detailed job information including state, job ids and allocated nodes counter. .TP \fBTriggers\fR Slurmctld triggers .RE .TP \fBDefMemPerCPU\fR Default real memory size available per allocated CPU in megabytes. Used to avoid over\-subscribing memory and causing paging. \fBDefMemPerCPU\fR would generally be used if individual processors are allocated to jobs (\fBSelectType=select/cons_res\fR). The default value is 0 (unlimited). Also see \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR. \fBDefMemPerCPU\fR and \fBDefMemPerNode\fR are mutually exclusive. .TP \fBDefMemPerNode\fR Default real memory size available per allocated node in megabytes. Used to avoid over\-subscribing memory and causing paging. \fBDefMemPerNode\fR would generally be used if whole nodes are allocated to jobs (\fBSelectType=select/linear\fR) and resources are over\-subscribed (\fBOverSubscribe=yes\fR or \fBOverSubscribe=force\fR). The default value is 0 (unlimited). Also see \fBDefMemPerCPU\fR and \fBMaxMemPerNode\fR. \fBDefMemPerCPU\fR and \fBDefMemPerNode\fR are mutually exclusive. .TP \fBDefaultStorageHost\fR The default name of the machine hosting the accounting storage and job completion databases. Only used for database type storage plugins and when the \fBAccountingStorageHost\fR and \fBJobCompHost\fR have not been defined. .TP \fBDefaultStorageLoc\fR The fully qualified file name where accounting records and/or job completion records are written when the \fBDefaultStorageType\fR is "filetxt". Also see \fBAccountingStorageLoc\fR and \fBJobCompLoc\fR. .TP \fBDefaultStoragePass\fR The password used to gain access to the database to store the accounting and job completion data. Only used for database type storage plugins, ignored otherwise. Also see \fBAccountingStoragePass\fR and \fBJobCompPass\fR. .TP \fBDefaultStoragePort\fR The listening port of the accounting storage and/or job completion database server. Only used for database type storage plugins, ignored otherwise. Also see \fBAccountingStoragePort\fR and \fBJobCompPort\fR. .TP \fBDefaultStorageType\fR The accounting and job completion storage mechanism type. Acceptable values at present include "filetxt", "mysql" and "none". The value "filetxt" indicates that records will be written to a file. The value "mysql" indicates that accounting records will be written to a MySQL or MariaDB database. The default value is "none", which means that records are not maintained. Also see \fBAccountingStorageType\fR and \fBJobCompType\fR. .TP \fBDefaultStorageUser\fR The user account for accessing the accounting storage and/or job completion database. Only used for database type storage plugins, ignored otherwise. Also see \fBAccountingStorageUser\fR and \fBJobCompUser\fR. .TP \fBDisableRootJobs\fR If set to "YES" then user root will be prevented from running any jobs. The default value is "NO", meaning user root will be able to execute jobs. \fBDisableRootJobs\fR may also be set by partition. .TP \fBEioTimeout\fR The number of seconds srun waits for slurmstepd to close the TCP/IP connection used to relay data between the user application and srun when the user application terminates. The default value is 60 seconds. May not exceed 65533. .TP \fBEnforcePartLimits\fR If set to "ALL" then jobs which exceed a partition's size and/or time limits will be rejected at submission time. If job is submitted to multiple partitions, the job must satisfy the limits on all the requested partitions. If set to "NO" then the job will be accepted and remain queued until the partition limits are altered(Time and Node Limits). If set to "ANY" or "YES" a job must satisfy any of the requested partitions to be submitted. The default value is "NO". NOTE: If set, then a job's QOS can not be used to exceed partition limits. NOTE: The partition limits being considered are it's configured MaxMemPerCPU, MaxMemPerNode, MinNodes, MaxNodes, MaxTime, AllocNodes, AllowAccounts, AllowGroups, AllowQOS, and QOS usage threshold. .TP \fBEpilog\fR Fully qualified pathname of a script to execute as user root on every node when a user's job completes (e.g. "/usr/local/slurm/epilog"). A glob pattern (See \fBglob\fR (7)) may also be used to run more than one epilog script (e.g. "/etc/slurm/epilog.d/*"). The Epilog script or scripts may be used to purge files, disable user login, etc. By default there is no epilog. See \fBProlog and Epilog Scripts\fR for more information. .TP \fBEpilogMsgTime\fR The number of microseconds that the slurmctld daemon requires to process an epilog completion message from the slurmd daemons. This parameter can be used to prevent a burst of epilog completion messages from being sent at the same time which should help prevent lost messages and improve throughput for large jobs. The default value is 2000 microseconds. For a 1000 node job, this spreads the epilog completion messages out over two seconds. .TP \fBEpilogSlurmctld\fR Fully qualified pathname of a program for the slurmctld to execute upon termination of a job allocation (e.g. "/usr/local/slurm/epilog_controller"). The program executes as SlurmUser, which gives it permission to drain nodes and requeue the job if a failure occurs (See scontrol(1)). Exactly what the program does and how it accomplishes this is completely at the discretion of the system administrator. Information about the job being initiated, it's allocated nodes, etc. are passed to the program using environment variables. See \fBProlog and Epilog Scripts\fR for more information. .TP \fBExtSensorsFreq\fR The external sensors plugin sampling interval. If \fBExtSensorsType=ext_sensors/none\fR, this parameter is ignored. For all other values of \fBExtSensorsType\fR, this parameter is the number of seconds between external sensors samples for hardware components (nodes, switches, etc.) The default value is zero. This value disables external sensors sampling. Note: This parameter does not affect external sensors data collection for jobs/steps. .TP \fBExtSensorsType\fR Identifies the plugin to be used for external sensors data collection. Slurmctld calls this plugin to collect external sensors data for jobs/steps and hardware components. In case of node sharing between jobs the reported values per job/step (through sstat or sacct) may not be accurate. See also "man ext_sensors.conf". Configurable values at present are: .RS .TP 20 \fBext_sensors/none\fR No external sensors data is collected. .TP \fBext_sensors/rrd\fR External sensors data is collected from the RRD database. .RE .TP \fBFairShareDampeningFactor\fR Dampen the effect of exceeding a user or group's fair share of allocated resources. Higher values will provides greater ability to differentiate between exceeding the fair share at high levels (e.g. a value of 1 results in almost no difference between overconsumption by a factor of 10 and 100, while a value of 5 will result in a significant difference in priority). The default value is 1. .TP \fBFastSchedule\fR Controls how a node's configuration specifications in slurm.conf are used. If the number of node configuration entries in the configuration file is significantly lower than the number of nodes, setting FastSchedule to 1 will permit much faster scheduling decisions to be made. (The scheduler can just check the values in a few configuration records instead of possibly thousands of node records.) Note that on systems with hyper\-threading, the processor count reported by the node will be twice the actual processor count. Consider which value you want to be used for scheduling purposes. .RS .TP 5 \fB0\fR Base scheduling decisions upon the actual configuration of each individual node except that the node's processor count in Slurm's configuration must match the actual hardware configuration if \fBPreemptMode=suspend,gang\fR or \fBSelectType=select/cons_res\fR are configured (both of those plugins maintain resource allocation information using bitmaps for the cores in the system and must remain static, while the node's memory and disk space can be established later). .TP \fB1\fR (default) Consider the configuration of each node to be that specified in the slurm.conf configuration file and any node with less than the configured resources will be set to DRAIN. .TP \fB2\fR Consider the configuration of each node to be that specified in the slurm.conf configuration file and any node with less than the configured resources will \fBnot\fR be set DRAIN. This option is generally only useful for testing purposes. .RE .TP \fBFederationParameters\fR Used to define federation options. Multiple options may be comma separated. .RS .TP \fBfed_display\fR If set, then the client status commands (e.g. squeue, sinfo, sprio, etc.) will display information in a federated view by default. This option is functionally equivalent to using the \-\-federation options on each command. Use the client's \-\-local option to override the federated view and get a local view of the given cluster. .RE .TP \fBFirstJobId\fR The job id to be used for the first submitted to Slurm without a specific requested value. Job id values generated will incremented by 1 for each subsequent job. This may be used to provide a meta\-scheduler with a job id space which is disjoint from the interactive jobs. The default value is 1. Also see \fBMaxJobId\fR .TP \fBGetEnvTimeout\fR Used for Moab scheduled jobs only. Controls how long job should wait in seconds for loading the user's environment before attempting to load it from a cache file. Applies when the srun or sbatch \fI\-\-get\-user\-env\fR option is used. If set to 0 then always load the user's environment from the cache file. The default value is 2 seconds. .TP \fBGresTypes\fR A comma delimited list of generic resources to be managed. These generic resources may have an associated plugin available to provide additional functionality. No generic resources are managed by default. Ensure this parameter is consistent across all nodes in the cluster for proper operation. The slurmctld daemon must be restarted for changes to this parameter to become effective. .TP \fBGroupUpdateForce\fR If set to a non\-zero value, then information about which users are members of groups allowed to use a partition will be updated periodically, even when there have been no changes to the /etc/group file. If set to zero, group member information will be updated only after the /etc/group file is updated. The default value is 1. Also see the \fBGroupUpdateTime\fR parameter. .TP \fBGroupUpdateTime\fR Controls how frequently information about which users are members of groups allowed to use a partition will be updated, and how long user group membership lists will be cached. The time interval is given in seconds with a default value of 600 seconds. A value of zero will prevent periodic updating of group membership information. Also see the \fBGroupUpdateForce\fR parameter. .TP \fBHealthCheckInterval\fR The interval in seconds between executions of \fBHealthCheckProgram\fR. The default value is zero, which disables execution. .TP \fBHealthCheckNodeState\fR Identify what node states should execute the \fBHealthCheckProgram\fR. Multiple state values may be specified with a comma separator. The default value is ANY to execute on nodes in any state. .RS .TP 12 \fBALLOC\fR Run on nodes in the ALLOC state (all CPUs allocated). .TP \fBANY\fR Run on nodes in any state. .TP \fBCYCLE\fR Rather than running the health check program on all nodes at the same time, cycle through running on all compute nodes through the course of the \fBHealthCheckInterval\fR. May be combined with the various node state options. .TP \fBIDLE\fR Run on nodes in the IDLE state. .TP \fBMIXED\fR Run on nodes in the MIXED state (some CPUs idle and other CPUs allocated). .RE .TP \fBHealthCheckProgram\fR Fully qualified pathname of a script to execute as user root periodically on all compute nodes that are \fBnot\fR in the NOT_RESPONDING state. This program may be used to verify the node is fully operational and DRAIN the node or send email if a problem is detected. Any action to be taken must be explicitly performed by the program (e.g. execute "scontrol update NodeName=foo State=drain Reason=tmp_file_system_full" to drain a node). The execution interval is controlled using the \fBHealthCheckInterval\fR parameter. Note that the \fBHealthCheckProgram\fR will be executed at the same time on all nodes to minimize its impact upon parallel programs. This program is will be killed if it does not terminate normally within 60 seconds. This program will also be executed when the slurmd daemon is first started. By default, no program will be executed. .TP \fBInactiveLimit\fR The interval, in seconds, after which a non\-responsive job allocation command (e.g. \fBsrun\fR or \fBsalloc\fR) will result in the job being terminated. If the node on which the command is executed fails or the command abnormally terminates, this will terminate its job allocation. This option has no effect upon batch jobs. When setting a value, take into consideration that a debugger using \fBsrun\fR to launch an application may leave the \fBsrun\fR command in a stopped state for extended periods of time. This limit is ignored for jobs running in partitions with the \fBRootOnly\fR flag set (the scheduler running as root will be responsible for the job). The default value is unlimited (zero) and may not exceed 65533 seconds. .TP \fBJobAcctGatherType\fR The job accounting mechanism type. Acceptable values at present include "jobacct_gather/linux" (for Linux systems) and is the recommended one, "jobacct_gather/cgroup" and "jobacct_gather/none" (no accounting data collected). The default value is "jobacct_gather/none". "jobacct_gather/cgroup" is a plugin for the Linux operating system that uses cgroups to collect accounting statistics. The plugin collects the following statistics: From the cgroup memory subsystem: memory.usage_in_bytes (reported as 'pages') and rss from memory.stat (reported as 'rss'). From the cgroup cpuacct subsystem: user cpu time and system cpu time. No value is provided by cgroups for virtual memory size ('vsize'). In order to use the \fBsstat\fR tool "jobacct_gather/linux", or "jobacct_gather/cgroup" must be configured. .br \fBNOTE:\fR Changing this configuration parameter changes the contents of the messages between Slurm daemons. Any previously running job steps are managed by a slurmstepd daemon that will persist through the lifetime of that job step and not change it's communication protocol. Only change this configuration parameter when there are no running job steps. .TP \fBJobAcctGatherFrequency\fR The job accounting and profiling sampling intervals. The supported format is follows: .RS .TP 12 \fBJobAcctGatherFrequency=\fR\fI\fR\fB=\fR\fI\fR where \fI\fR=\fI\fR specifies the task sampling interval for the jobacct_gather plugin or a sampling interval for a profiling type by the acct_gather_profile plugin. Multiple, comma-separated \fI\fR=\fI\fR intervals may be specified. Supported datatypes are as follows: .RS .TP \fBtask=\fI\fR where \fI\fR is the task sampling interval in seconds for the jobacct_gather plugins and for task profiling by the acct_gather_profile plugin. .TP \fBenergy=\fI\fR where \fI\fR is the sampling interval in seconds for energy profiling using the acct_gather_energy plugin .TP \fBnetwork=\fI\fR where \fI\fR is the sampling interval in seconds for infiniband profiling using the acct_gather_infiniband plugin. .TP \fBfilesystem=\fI\fR where \fI\fR is the sampling interval in seconds for filesystem profiling using the acct_gather_filesystem plugin. .TP .RE .RE The default value for task sampling interval is 30 seconds. The default value for all other intervals is 0. An interval of 0 disables sampling of the specified type. If the task sampling interval is 0, accounting information is collected only at job termination (reducing Slurm interference with the job). .br .br Smaller (non\-zero) values have a greater impact upon job performance, but a value of 30 seconds is not likely to be noticeable for applications having less than 10,000 tasks. .br .br Users can independently override each interval on a per job basis using the \fB\-\-acctg\-freq\fR option when submitting the job. .RE .TP \fBJobAcctGatherParams\fR Arbitrary parameters for the job account gather plugin Acceptable values at present include: .RS .TP 20 \fBNoShared\fR Exclude shared memory from accounting. .TP \fBUsePss\fR Use PSS value instead of RSS to calculate real usage of memory. The PSS value will be saved as RSS. .TP \fBOverMemoryKill\fR Kill steps that are being detected to use more memory than requested, every time accounting information is gathered by JobAcctGather plugin. This parameter will not kill a job directly, but only the step. See \fBMemLimitEnforce\fB for that purpose. This parameter should be used with caution as if jobs exceeds its memory allocation it may affect other processes and/or machine health. NOTE: It is recommended to limit memory by enabling task/cgroup in TaskPlugin and making use of ConstrainRAMSpace=yes cgroup.conf instead of using this JobAcctGather mechanism for memory enforcement, since the former has a lower resolution (JobAcctGatherFreq) and OOMs could happen at some point. .RE .TP \fBJobCheckpointDir\fR Specifies the default directory for storing or reading job checkpoint information. The data stored here is only a few thousand bytes per job and includes information needed to resubmit the job request, not job's memory image. The directory must be readable and writable by \fBSlurmUser\fR, but not writable by regular users. The job memory images may be in a different location as specified by \fB\-\-checkpoint\-dir\fR option at job submit time or scontrol's \fBImageDir\fR option. .TP \fBJobCompHost\fR The name of the machine hosting the job completion database. Only used for database type storage plugins, ignored otherwise. Also see \fBDefaultStorageHost\fR. .TP \fBJobCompLoc\fR The fully qualified file name where job completion records are written when the \fBJobCompType\fR is "jobcomp/filetxt" or the database where job completion records are stored when the \fBJobCompType\fR is a database, or an url with format http://yourelasticserver:port when \fBJobCompType\fR is "jobcomp/elasticsearch". NOTE: when you specify a URL for Elasticsearch, Slurm will remove any trailing slashes "/" from the configured URL and append "/slurm/jobcomp", which are the Elasticsearch index name (slurm) and mapping (jobcomp). NOTE: More information is available at the Slurm web site ( https://slurm.schedmd.com/elasticsearch.html ). Also see \fBDefaultStorageLoc\fR. .TP \fBJobCompPass\fR The password used to gain access to the database to store the job completion data. Only used for database type storage plugins, ignored otherwise. Also see \fBDefaultStoragePass\fR. .TP \fBJobCompPort\fR The listening port of the job completion database server. Only used for database type storage plugins, ignored otherwise. Also see \fBDefaultStoragePort\fR. .TP \fBJobCompType\fR The job completion logging mechanism type. Acceptable values at present include "jobcomp/none", "jobcomp/elasticsearch", "jobcomp/filetxt", "jobcomp/mysql" and "jobcomp/script". The default value is "jobcomp/none", which means that upon job completion the record of the job is purged from the system. If using the accounting infrastructure this plugin may not be of interest since the information here is redundant. The value "jobcomp/elasticsearch" indicates that a record of the job should be written to an Elasticsearch server specified by the \fBJobCompLoc\fR parameter. NOTE: More information is available at the Slurm web site ( https://slurm.schedmd.com/elasticsearch.html ). The value "jobcomp/filetxt" indicates that a record of the job should be written to a text file specified by the \fBJobCompLoc\fR parameter. The value "jobcomp/mysql" indicates that a record of the job should be written to a MySQL or MariaDB database specified by the \fBJobCompLoc\fR parameter. The value "jobcomp/script" indicates that a script specified by the \fBJobCompLoc\fR parameter is to be executed with environment variables indicating the job information. .TP \fBJobCompUser\fR The user account for accessing the job completion database. Only used for database type storage plugins, ignored otherwise. Also see \fBDefaultStorageUser\fR. .TP \fBJobContainerType\fR Identifies the plugin to be used for job tracking. The slurmd daemon must be restarted for a change in JobContainerType to take effect. NOTE: The \fBJobContainerType\fR applies to a job allocation, while \fBProctrackType\fR applies to job steps. Acceptable values at present include: .RS .TP 20 \fBjob_container/cncu\fR used only for Cray systems (CNCU = Compute Node Clean Up) .TP \fBjob_container/none\fR used for all other system types .RE .TP \fBJobCredentialPrivateKey\fR Fully qualified pathname of a file containing a private key used for authentication by Slurm daemons. This parameter is ignored if \fBCryptoType=crypto/munge\fR. .TP \fBJobCredentialPublicCertificate\fR Fully qualified pathname of a file containing a public key used for authentication by Slurm daemons. This parameter is ignored if \fBCryptoType=crypto/munge\fR. .TP \fBJobFileAppend\fR This option controls what to do if a job's output or error file exist when the job is started. If \fBJobFileAppend\fR is set to a value of 1, then append to the existing file. By default, any existing file is truncated. .TP \fBJobRequeue\fR This option controls the default ability for batch jobs to be requeued. Jobs may be requeued explicitly by a system administrator, after node failure, or upon preemption by a higher priority job. If \fBJobRequeue\fR is set to a value of 1, then batch job may be requeued unless explicitly disabled by the user. If \fBJobRequeue\fR is set to a value of 0, then batch job will not be requeued unless explicitly enabled by the user. Use the \fBsbatch\fR \fI\-\-no\-requeue\fR or \fI\-\-requeue\fR option to change the default behavior for individual jobs. The default value is 1. .TP \fBJobSubmitPlugins\fR A comma delimited list of job submission plugins to be used. The specified plugins will be executed in the order listed. These are intended to be site\-specific plugins which can be used to set default job parameters and/or logging events. Sample plugins available in the distribution include "all_partitions", "defaults", "logging", "lua", and "partition". For examples of use, see the Slurm code in "src/plugins/job_submit" and "contribs/lua/job_submit*.lua" then modify the code to satisfy your needs. Slurm can be configured to use multiple job_submit plugins if desired, however the lua plugin will only execute one lua script named "job_submit.lua" located in the default script directory (typically the subdirectory "etc" of the installation directory). No job submission plugins are used by default. .TP \fBKeepAliveTime\fR Specifies how long sockets communications used between the srun command and its slurmstepd process are kept alive after disconnect. Longer values can be used to improve reliability of communications in the event of network failures. The default value leaves the system default value. The value may not exceed 65533. .TP \fBKillOnBadExit\fR If set to 1, a step will be terminated immediately if any task is crashed or aborted, as indicated by a non-zero exit code. With the default value of 0, if one of the processes is crashed or aborted the other processes will continue to run while the crashed or aborted process waits. The user can override this configuration parameter by using srun's \fB\-K\fR, \fB\-\-kill\-on\-bad\-exit\fR. .TP \fBKillWait\fR The interval, in seconds, given to a job's processes between the SIGTERM and SIGKILL signals upon reaching its time limit. If the job fails to terminate gracefully in the interval specified, it will be forcibly terminated. The default value is 30 seconds. The value may not exceed 65533. .TP \fBNodeFeaturesPlugins\fR Identifies the plugins to be used for support of node features which can change through time. For example, a node which might be booted with various BIOS setting. This is supported through the use of a node's active_features and available_features information. Acceptable values at present include: .RS .TP 20 \fBnode_features/knl_cray\fR used only for Intel Knights Landing processors (KNL) on Cray systems .TP 20 \fBnode_features/knl_generic\fR used for Intel Knights Landing processors (KNL) on a generic Linux system .RE .TP \fBLaunchParameters\fR Identifies options to the job launch plugin. Acceptable values include: .RS .TP 24 \fBbatch_step_set_cpu_freq\fR Set the cpu frequency for the batch step from given \-\-cpu\-freq, or slurm.conf CpuFreqDef, option. By default only steps started with srun will utilize the cpu freq setting options. NOTE: If you are using srun to launch your steps inside a batch script (advised) this option will create a situation where you may have multiple agents setting the cpu_freq as the batch step usually runs on the same resources one or more steps the sruns in the script will create. .TP 24 \fBcray_net_exclusive\fR Allow jobs on a Cray Native cluster exclusive access to network resources. This should only be set on clusters providing exclusive access to each node to a single job at once, and not using parallel steps within the job, otherwise resources on the node can be oversubscribed. .TP 24 \fBlustre_no_flush\fR If set on a Cray Native cluster, then do not flush the Lustre cache on job step completion. This setting will only take effect after reconfiguring, and will only take effect for newly launched jobs. .TP 24 \fBmem_sort\fR Sort NUMA memory at step start. User can override this default with SLURM_MEM_BIND environment variable or \-\-mem\-bind=nosort command line option. .TP \fBsend_gids\fR Lookup and send the user_name and extended gids for a job within the slurmctld, rather than individual on each node as part of each task launch. Should avoid issues around name service scalability when launching jobs involving many nodes. .TP 24 \fBslurmstepd_memlock\fR Lock the slurmstepd process's current memory in RAM. .TP \fBslurmstepd_memlock_all\fR Lock the slurmstepd process's current and future memory in RAM. .TP \fBtest_exec\fR Have srun verify existence of the executable program along with user execute permission on the node where srun was called before attempting to launch it on nodes in the step. .RE .TP \fBLaunchType\fR Identifies the mechanism to be used to launch application tasks. Acceptable values include: .RS .TP \fBlaunch/slurm\fR The default value. .RE .TP \fBLicenses\fR Specification of licenses (or other resources available on all nodes of the cluster) which can be allocated to jobs. License names can optionally be followed by a colon and count with a default count of one. Multiple license names should be comma separated (e.g. "Licenses=foo:4,bar"). Note that Slurm prevents jobs from being scheduled if their required license specification is not available. Slurm does not prevent jobs from using licenses that are not explicitly listed in the job submission specification. .TP \fBLogTimeFormat\fR Format of the timestamp in slurmctld and slurmd log files. Accepted values are "iso8601", "iso8601_ms", "rfc5424", "rfc5424_ms", "clock", "short" and "thread_id". The values ending in "_ms" differ from the ones without in that fractional seconds with millisecond precision are printed. The default value is "iso8601_ms". The "rfc5424" formats are the same as the "iso8601" formats except that the timezone value is also shown. The "clock" format shows a timestamp in microseconds retrieved with the C standard clock() function. The "short" format is a short date and time format. The "thread_id" format shows the timestamp in the C standard ctime() function form without the year but including the microseconds, the daemon's process ID and the current thread name and ID. .TP \fBMailDomain\fR Domain name to qualify usernames if email address is not explicitly given with the "--mail-user" option. If unset, the local MTA will need to qualify local address itself. .TP \fBMailProg\fR Fully qualified pathname to the program used to send email per user request. The default value is "/bin/mail" (or "/usr/bin/mail" if "/bin/mail" does not exist but "/usr/bin/mail" does exist). .TP \fBMaxArraySize\fR The maximum job array size. The maximum job array task index value will be one less than MaxArraySize to allow for an index value of zero. Configure MaxArraySize to 0 in order to disable job array use. The value may not exceed 4000001. The value of \fBMaxJobCount\fR should be much larger than \fBMaxArraySize\fR. The default value is 1001. .TP \fBMaxJobCount\fR The maximum number of jobs Slurm can have in its active database at one time. Set the values of \fBMaxJobCount\fR and \fBMinJobAge\fR to ensure the slurmctld daemon does not exhaust its memory or other resources. Once this limit is reached, requests to submit additional jobs will fail. The default value is 10000 jobs. NOTE: Each task of a job array counts as one job even though they will not occupy separate job records until modified or initiated. Performance can suffer with more than a few hundred thousand jobs. Setting per MaxSubmitJobs per user is generally valuable to prevent a single user from filling the system with jobs. This is accomplished using Slurm's database and configuring enforcement of resource limits. This value may not be reset via "scontrol reconfig". It only takes effect upon restart of the slurmctld daemon. .TP \fBMaxJobId\fR The maximum job id to be used for jobs submitted to Slurm without a specific requested value. Job ids are unsigned 32bit integers with the first 26 bits reserved for local job ids and the remaining 6 bits reserved for a cluster id to identify a federated job's origin. The maximun allowed local job id is 67,108,863 (0x3FFFFFF). The default value is 67,043,328 (0x03ff0000). \fBMaxJobId\fR only applies to the local job id and not the federated job id. Job id values generated will be incremented by 1 for each subsequent job. Once \fBMaxJobId\fR is reached, the next job will be assigned \fBFirstJobId\fR. Federated jobs will always have a job ID of 67,108,865 or higher. Also see \fBFirstJobId\fR. .TP \fBMaxMemPerCPU\fR Maximum real memory size available per allocated CPU in megabytes. Used to avoid over\-subscribing memory and causing paging. \fBMaxMemPerCPU\fR would generally be used if individual processors are allocated to jobs (\fBSelectType=select/cons_res\fR). The default value is 0 (unlimited). Also see \fBDefMemPerCPU\fR and \fBMaxMemPerNode\fR. \fBMaxMemPerCPU\fR and \fBMaxMemPerNode\fR are mutually exclusive. NOTE: If a job specifies a memory per CPU limit that exceeds this system limit, that job's count of CPUs per task will automatically be increased. This may result in the job failing due to CPU count limits. .TP \fBMaxMemPerNode\fR Maximum real memory size available per allocated node in megabytes. Used to avoid over\-subscribing memory and causing paging. \fBMaxMemPerNode\fR would generally be used if whole nodes are allocated to jobs (\fBSelectType=select/linear\fR) and resources are over\-subscribed (\fBOverSubscribe=yes\fR or \fBOverSubscribe=force\fR). The default value is 0 (unlimited). Also see \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR. \fBMaxMemPerCPU\fR and \fBMaxMemPerNode\fR are mutually exclusive. .TP \fBMaxStepCount\fR The maximum number of steps that any job can initiate. This parameter is intended to limit the effect of bad batch scripts. The default value is 40000 steps. .TP \fBMaxTasksPerNode\fR Maximum number of tasks Slurm will allow a job step to spawn on a single node. The default \fBMaxTasksPerNode\fR is 512. May not exceed 65533. .TP \fBMCSParameters\fR MCS = Multi-Category Security MCS Plugin Parameters. The supported parameters are specific to the \fBMCSPlugin\fR. Changes to this value take effect when the Slurm daemons are reconfigured. More information about MCS is available here . .TP \fBMCSPlugin\fR MCS = Multi-Category Security : associate a security label to jobs and ensure that nodes can only be shared among jobs using the same security label. Acceptable values include: .RS .TP 12 \fBmcs/none\fR is the default value. No security label associated with jobs, no particular security restriction when sharing nodes among jobs. .TP \fBmcs/account\fR only users with the same account can share the nodes (requires enabling of accounting). .TP \fBmcs/group\fR only users with the same group can share the nodes. .TP \fBmcs/user\fR a node cannot be shared with other users. .RE .TP \fBMemLimitEnforce\fR If set to yes then Slurm will terminate the job if it exceeds the value requested using the \-\-mem\-per\-cpu option of salloc/sbatch/srun. This is useful in combination with \fBJobAcctGaterParams=OverMemoryKill\fB. Used when jobs need to specify \-\-mem\-per\-cpu for scheduling and they should be terminated if they exceed the estimated value. The default value is 'no', which disables this enforcing mechanism. NOTE: It is recommended to limit memory by enabling task/cgroup in TaskPlugin and making use of ConstrainRAMSpace=yes cgroup.conf instead of using this JobAcctGather mechanism for memory enforcement, since the former has a lower resolution (JobAcctGatherFreq) and OOMs could happen at some point. .RE .TP \fBMessageTimeout\fR Time permitted for a round\-trip communication to complete in seconds. Default value is 10 seconds. For systems with shared nodes, the slurmd daemon could be paged out and necessitate higher values. .TP \fBMinJobAge\fR The minimum age of a completed job before its record is purged from Slurm's active database. Set the values of \fBMaxJobCount\fR and to ensure the slurmctld daemon does not exhaust its memory or other resources. The default value is 300 seconds. A value of zero prevents any job record purging. In order to eliminate some possible race conditions, the minimum non\-zero value for \fBMinJobAge\fR recommended is 2. .TP \fBMpiDefault\fR Identifies the default type of MPI to be used. Srun may override this configuration parameter in any case. Currently supported versions include: \fBopenmpi\fR, \fBpmi2\fR, \fBpmix\fR, and \fBnone\fR (default, which works for many other versions of MPI). More information about MPI use is available here . .TP \fBMpiParams\fR MPI parameters. Used to identify ports used by older versions of OpenMPI and native Cray systems. The input format is "ports=12000\-12999" to identify a range of communication ports to be used. NOTE: This is not needed for modern versions of OpenMPI, taking it out can cause a small boost in scheduling performance. NOTE: This is require for Cray's PMI. .TP \fBMsgAggregationParams\fR Message aggregation parameters. Message aggregation is an optional feature that may improve system performance by reducing the number of separate messages passed between nodes. The feature works by routing messages through one or more message collector nodes between their source and destination nodes. At each collector node, messages with the same destination received during a defined message collection window are packaged into a single composite message. When the window expires, the composite message is sent to the next collector node on the route to its destination. The route between each source and destination node is provided by the Route plugin. When a composite message is received at its destination node, the original messages are extracted and processed as if they had been sent directly. .br .br Currently, the only message types supported by message aggregation are the node registration, batch script completion, step completion, and epilog complete messages. .br .br The format for this parameter is as follows: .RE .RS .TP 12 \fBMsgAggregationParams=\fR\fI