pe_name¶

slots¶

user_lists¶

xuser_lists¶

Each user contained in at least one of the user_lists access lists has access to the parallel environment. If the user_lists parameter is set to NONE (the default) any user has access if not explicitly excluded via the xuser_lists parameter.

Each user contained in at least one of the xuser_lists access lists is not allowed to access the parallel environment. If the xuser_lists parameter is set to NONE (the default) any user has access.

If a user is contained both in an access list in xuser_lists and user_lists the user is denied access to the parallel environment.

start_proc_args¶

stop_proc_args¶

The startup procedure is invoked by sge_shepherd(8) on the master node of the job prior to executing the job script. Its purpose is to setup the parallel environment according to its needs. The shutdown procedure is invoked by sge_shepherd(8) after the job script has finished. Its purpose is to stop the parallel environment and to remove it from all participating systems. The standard output of the procedure is redirected to the file REQUEST.poJID in the job's working directory (see qsub(1)), with REQUEST being the name of the job as displayed by qstat(1), and JID being the job's identification number. Likewise, the standard error output is redirected to REQUEST.peJID. If the -e or -o options are given on job submission, the PE error and standard output is merged into the paths specified.

The following special variables, expanded at runtime, can be used (besides any other strings which have to be interpreted by the start and stop procedures) to constitute a command line:

$pe_hostfile

The pathname of a file containing a detailed description of the layout of the parallel environment to be setup by the start-up procedure. Each line of the file refers to a host on which parallel processes are to be run. The first entry of each line denotes the hostname, the second entry the number of parallel processes to be run on the host, the third entry the name of the queue. The entries are separated by spaces. If -binding pe is specified on job submission, the fourth column is the core binding specification as colon-separated socket-core pairs, like "0,0:0,1", meaning the first core on the first socket and the second core on the first socket can be used for binding. Otherwise it will be "UNDEFINED". With the obsolete queue processors specification the fourth entry could be a multi-processor configuration (or "<NULL>").

$host

The name of the host on which the startup or stop procedures are run.

$ja_task_id

The array job task index (0 if not an array job).

$job_owner

The user name of the job owner.

$job_id

Grid Engine's unique job identification number.

$job_name

The name of the job.

$pe

The name of the parallel environment in use.

$pe_slots

Number of slots granted for the job.

$processors

The processors string as contained in the queue configuration (see queue_conf(5)) of the master queue (the queue in which the startup and stop procedures are run).

$queue

The cluster queue of the master queue instance.

$sge_cell

The SGE_CELL environment variable (useful for locating files).

$sge_root

The SGE_ROOT environment variable (useful for locating files).

$stdin_path

The standard input path.

$stderr_path

The standard error path.

$stdout_path

The standard output path.

$merge_stderr

$fs_stdin_host

$fs_stdin_path

$fs_stdin_tmp_path

$fs_stdin_file_staging

$fs_stdout_host

$fs_stdout_path

$fs_stdout_tmp_path

$fs_stdout_file_staging

$fs_stderr_host

$fs_stderr_path

$fs_stderr_tmp_path

$fs_stderr_file_staging

The start and stop commands are run with the same environment setting as that of the job to be started afterwards (see qsub(1)).

allocation_rule¶

The current version of the scheduler only understands the following allocation rules:

int

An integer, fixing the number of processes per host. If it is 1, all processes have to reside on different hosts. If the special name $pe_slots is used, the full range of processes as specified with the qsub(1) -pe switch has to be allocated on a single host (no matter what value belonging to the range is finally chosen for the job to be allocated).

$fill_up

Starting from the best suitable host/queue, all available slots are allocated. Further hosts and queues are "filled up" as long as a job still requires slots for parallel tasks.

$round_robin

From all suitable hosts, a single slot is allocated until all tasks requested by the parallel job are dispatched. If more tasks are requested than suitable hosts are found, allocation starts again from the first host. The allocation scheme walks through suitable hosts in a most-suitable-first order.

control_slaves¶

• resource limits are enforced for all tasks, even on slave hosts;
• resource consumption is properly accounted on all hosts;
• proper control of tasks, with no need to write a customized terminate method to ensure that whole job is finished on qdel and that tasks are properly reaped in the case of abnormal job termination;
• all tasks are started with the appropriate nice value which was configured as priority in the queue configuration;
• propagation of the job environment to slave hosts, e.g. so that they write into the appropriate per-job temporary directory specified by TMPDIR, which is created on each host and properly cleaned up.

To gain control over the slave tasks of a parallel application, a sophisticated PE interface is required, which works closely together with Grid Engine facilities, typically interpreting the Grid Engine hostfile and starting remote tasks with qrsh(1) and its -inherit option. See, for instance, the $SGE_ROOT/mpi directory and the howto pages ⟨URL: http://arc.liv.ac.uk/SGE/howto/#Tight%20Integration%20of%20Parallel%20Libraries ⟩.

Please set the control_slaves parameter to false for all other PE interfaces.

job_is_first_task¶

This should be TRUE for the common modern MPI implementations with tight integration. Consider if the allocation rule is $fill_up, and a job is allocated only a single slot on the master host; then one of the MPI processes actually runs in that slot, and should be accounted as such, so the job is the first task.

If wallclock accounting is used (execd_params ACCT_RESERVED_USAGE and/or SHARETREE_RESERVED_USAGE Is TRUE) and control_slaves is set to FALSE, the job_is_first_task parameter influences the accounting for the job: A value of TRUE means that accounting for CPU and requested memory gets multiplied by the number of slots requested with the -pe switch. FALSE means the accounting information gets multiplied by number of slots + 1. Otherwise, the only significant effect of the parameter is on the display of the job.

urgency_slots¶

The assumed slot allocation has a meaning when determining the resource-request-based priority contribution for numeric resources as described in sge_priority(5) and is displayed when qstat(1) is run without -g t option.

The following methods are supported:

int

The specified integer number is directly used as prospective slot amount.

min

The slot range minimum is used as prospective slot amount. If no lower bound is specified with the range, 1 is assumed.

max

The slot range maximum is used as prospective slot amount. If no upper bound is specified with the range, the absolute maximum possible due to the PE's slots setting is assumed.

avg

The average of all numbers occurring within the job's PE range request is assumed.

accounting_summary¶

The accounting_summary parameter can be set to TRUE or FALSE. A value of TRUE indicates that only a single accounting record is written to the accounting(5) file, containing the accounting summary of the whole job, including all slave tasks, while a value of FALSE indicates an individual accounting(5) record is written for every slave task, as well as for the master task.

Note: When running tightly integrated jobs with SHARETREE_RESERVED_USAGE set, and accounting_summary enabled in the parallel environment, reserved usage will only be reported by the master task of the parallel job. No per-parallel task usage records will be sent from execd to qmaster, which can significantly reduce load on the qmaster when running large, tightly integrated parallel jobs. However, this removes the only post-hoc information about which hosts a job used.

qsort_args library qsort-function [arg1 ...]¶