NAME¶
jsv_is_param, jsv_get_param, jsv_del_param, jsv_sub_is_param, jsv_sub_get_param,
jsv_sub_add_param, jsv_sub_del_param, jsv_is_env, jsv_get_env, jsv_add_env,
jsv_mod_env, jsv_del_env, jsv_accept, jsv_correct, jsv_reject,
jsv_reject_wait, jsv_show_params, jsv_show_envs, jsv_log_info,
jsv_log_warning, jsv_log_error, jsv_main, jsv_logging_enabled - Grid Engine
Job Submission Verifier Scripting Interface
SYNOPSIS¶
jsv_clear_params();
jsv_is_param(param_name);
jsv_get_param(param_name);
jsv_set_param(param_name, param_value);
jsv_del_param(param_name);
jsv_sub_is_param(param_name, variable_name);
jsv_sub_get_param(param_name, variable_name);
jsv_sub_add_param(param_name, variable_name, variable_value);
jsv_sub_del_param(param_name, variable_name);
jsv_clear_envs();
jsv_is_env(variable_name);
jsv_get_env(variable_name);
jsv_add_env(variable_name, variable_value);
jsv_mod_env(variable_name, variable_value);
jsv_del_env(variable_name);
jsv_accept(message);
jsv_correct(message);
jsv_reject(message);
jsv_reject_wait(message);
jsv_show_params();
jsv_show_envs();
jsv_log_info(message);
jsv_log_warning(message);
jsv_log_error(message);
jsv_main();
jsv_send_env();
jsv_on_start();
jsv_on_verify();
DESCRIPTION¶
The functions documented here implement the server side of the JSV protocol as
described in (where "server" applies to both client- and server-side
JSVs). These functions are available in Bourne shell (preferably using for
greater efficiency), TCL, Perl, or Python scripts after sourcing/including the
files jsv_inlcude.sh, jsv_include.tcl, JSV.pm, or JSV.py. The files and
corresponding JSV script templates are located in the directory
$SGE_ROOT/util/resources/jsv. There is also a Java implementation which
has a different structure, with Javadoc documentation normally in
$SGE_ROOT/doc/javadocs/jjsv, and example files
SimpleJsv.java
and
jjsv.sh in
$SGE_ROOT/util/resources/jsv.
Note that Bourne shell server JSVs are discouraged because any problems with
unintended expansion of job parameters represent a security hazard (with the
qmaster running as the sgeadmin user). Also they may be relatively slow, and
constitute a bottleneck in high-throughput clusters. However, shell JSVs may
be faster using even compared with a generally faster shell, since fewer
external commands are invoked.
In the descriptions of routines here, a calling sequence like
function(arg1, arg2)
should be interpreted for Bourne shell and Tcl scripts as
function arg1 arg2
jsv_clear_params()¶
This function clears all received job parameters that were stored during the
last job verification process.
jsv_clear_envs()¶
This function clears all received job environment variables that were stored
during the last job verification process.
jsv_show_params()¶
A call of this function reports all known job parameters to the counterpart of
this script (client or master daemon thread). These parameters will be
reported as info messages and appear either in the stdout stream of the client
or in the message file of the master process.
jsv_show_envs()¶
This function reports all known job environment variables to the counterpart of
this script (client or master daemon thread). They will be reported as info
messages and appear in the stdout stream of the client or in the message file
of the master process.
jsv_is_param(param_name)¶
This function returns whether or not a specific job parameter is available for
the job which is currently being verified. Either the string
true or
false will be returned. The availability/absence of a job parameter
does not mean that the corresponding command line switch was used/not used.
The values allowed for
param_name are listed below. Find additional
information in describing the availability and value format. Job parameters
written in capital letters are pseudo parameters. A detailed description for
them can be found in Note that
The following parameters directly reflect arguments of the same name supplied to
the submission command (
qsub etc.) or corresponding values specified
with
qmon:
A,
a,
ar,
b,
ckpt,
dl,
e,
h,
hold_jid,
hold_jid_ad,
i,
j,
js,
M,
m,
masterq,
N,
notify,
o,
P,
p,
R,
r,
S,
shell,
tc,
w
Other parameters are related to the submission command arguments as follows:
- ac
- The the job context. The outcome of the evaluation of all -ac,
-dc, and -sc options is passed as a parameter with the name
ac, whose value is a comma-separated list of variable/value
pairs;
- binding_strategy, binding_type, binding_amount, binding_step, binding_socket, binding_core, binding_exp_n, binding_exp_socketid, binding_exp_coreid
- The values passed to the -binding parameter are passed as multiple
parameters to JSV instances. binding_strategy represents the
strategy to be used and is one of: linear, striding or
explicit. binding_type is the instance that should do the
binding, one of: env, set or pe.
binding_socket and binding_core are socket/core values
whereas binding_step is the step size (used only for striding
binding). The length of the socket/core value list of the explicit binding
is reported as binding_exp_n. The id part of
binding_exp_socketid and binding_exp_coreid
will be replaced by the position of the socket/core pair within the
explicit binding list (0 <= id <
binding_exp_n). The first socket/core pair of the explicit
binding will be reported with the parameter names
binding_exp_socket0 and binding_exp_core0. Values that do
not apply for the specified binding will not be reported to JSV. E.g.
binding_step will only be reported for the striding binding and all
binding_exp_... values will only be passed if explicit binding was
specified.
- c_interval
- Checkpoint interval, specified as a numeric value with -c;
- c_occasion
- Checkpoint "occasion_specifier" -c. (n,
s, m, or x) specified with -c;
- cwd
- Working directory, possibly specified with -cwd or -wd;
- display
- Reflects the -display submit argument and also sets job environment
variable DISPLAY to the same value;
- l_hard
- -l or -hard followed by -l;
- l_soft
- -soft followed by -l;
- pe_name, pe_min, pe_max
- The PE name and range limits specified with -pe;
- q_hard
- -q, or -hard followed by -q;
- q_soft
- -soft followed by -q.
See for explanation of the following pseudo parameters:
CLIENT,
CMDNAME,
CMDARGS,
CMDARGi, CONTEXT,
GROUP,
JOB_ID,
USER,
VERSION.
jsv_get_param(param_name)¶
This function returns the value of a specific job parameter
param_name.
This value is only available if the function
jsv_is_param() returns
true. Otherwise an empty string is returned.
Find a list of allowed parameter names in the section for the function
jsv_is_param().
jsv_set_param(param_name, param_value)¶
This function changes the job parameter
param_name to the value
param_value.
If
param_value is an empty string then the corresponding job parameter
will be deleted, similarly to the function
jsv_del_param(). As a
result, the job parameter is not available, as if the corresponding command
line switch was not specified during job submission.
For boolean parameters that only accept the values
yes or
no it is
not allowed to pass an empty string as
param_value.
Also for the parameters
c and
m it is not allowed to use empty
strings. Details can be found in
jsv_del_param(param_name)¶
This function deletes the job parameter
param_name.
Find a list of allowed parameter names in the section for the function
jsv_is_param().
jsv_sub_is_param(param_name, variable_name)¶
Some job parameters are lists that can contain multiple variables with an
optional value.
This function returns
true if a job's parameters contain the list-valued
param_name, with
variable_name in the list; otherwise it returns
false.
false might also indicate that the parameter list itself
is not available. Use the function
jsv_is_param() to check if the
parameter list is not available.
The following parameters are list parameters. The second column describes the
corresponding variable names to be used. The third column contains a dash (-)
if there is no value (
variable_value) allowed with the function
jsv_sub_add_param() or
jsv_sub_get_param() will return always an
empty string. A question mark (?) shows that the value is optional.
param_name |
description of variable_name |
variable_value |
|
ac |
job context variable name |
|
hold_jid |
job identifier |
- |
hold_jid_id |
array job identifier |
- |
l_hard |
complex attribute name |
? |
l_soft |
complex attribute name |
? |
M |
mail address |
- |
masterq |
cluster queue name or |
- |
|
queue instance name |
|
q_hard |
cluster queue name or |
- |
|
queue instance name |
|
q_soft |
cluster queue name or |
- |
|
queue instance name |
|
jsv_sub_get_param(param_name, variable_name)¶
Some job parameters are lists that can contain multiple variables with an
optional value.
This function returns the value of a variable
variable_name in the
parameter list
param_name. For sub list elements that have no value an
empty string will be returned.
Find a list of allowed parameter names (
param_name) and variable names (
variable_name) in the section for the function
jsv_sub_is_param().
jsv_sub_add_param(param_name, variable_name, variable_value)¶
Some job parameters are lists that can contain multiple variables with an
optional value.
This function either adds a new variable with a new value or it modifies the
value if the variable is already in the list.
variable_value is
optional, and if it is not supplied the variable has no value.
Find a list of allowed parameter names (
param_name) and variable names (
variable_name) in the section for the function
jsv_sub_is_param().
jsv_sub_del_param(param_name, variable_name)¶
Some job parameters are lists which can contain multiple variables with an
optional value.
This function deletes a variable
variable_name and, if available, the
corresponding value. If
variable_name is not available in the job
parameter then the command will be ignored.
Find a list of allowed parameter names (
param_name) and variable names (
variable_name ) in the section for the function
jsv_sub_is_param().
jsv_is_env(variable_name)¶
If the function returns
true, then the job environment variable with the
name
variable_name exists in the job currently being verified, and
jsv_get_env() can be used to retrieve the value of that variable. If
the function returns
false, then the job environment variable does not
exist.
jsv_get_env(variable_name)¶
This function returns the value of a job environment variable
variable_name.
This variable has to be passed with the
qsub command line switch
-v or
-V, and passing of environment variable data to JSV
scripts has to be enabled. Environment variable data are passed when the
function
jsv_send_env() is called in the callback function
jsv_on_start().
If the variable does not exist, or if environment variable information is not
available, then an empty string will be returned.
jsv_add_env(variable_name, variable_value)¶
This function adds an additional environment variable to the set of variables
that will exported to the job when it is started. As a result the
variable_name and
variable_value become available, as if
-v Or
-V was specified during job submission.
variable_value is optional. If an empty string is passed, then the
variable is defined without a value.
If
variable_name already exists in the set of job environment variables,
the corresponding value will be replaced by
variable_value, as if the
function
jsv_mod_env() was used. If an empty string is passed then the
old value will be deleted.
To delete an environment variable, the function
jsv_del_env() has to be
used.
jsv_mod_env(variable_name, variable_value)¶
This function modifies an existing environment variable that is in the set of
variables which will exported to the job when it is started. As a result, the
variable_name and
variable_value will be available as if
-v Or
-V was specified during job submission.
variable_value is optional. If an empty string is passed, then the
variable is defined without a value.
If
variable_name does not already exist in the set of job environment
variables, then the corresponding name and value will be added as if the
function
jsv_add_env() was used.
To delete a environment variable, use the function
jsv_del_env().
jsv_del_env(variable_name)¶
This function removes job environment variable
variable_name from the set
of variables that will be exported to the job when it is started.
If
variable_name does not already exist in the set of job environment
variables then the command is ignored.
To change the value of a variable use the function
jsv_mod_env(); to add
a new value, call the function
jsv_add_env().
jsv_accept(message)¶
This function can only be used in
jsv_on_verify(). After it has been
called, the function
jsv_on_verify() has to return immediately.
A call to this function indicates that the job that is currently being verified
should be accepted as it was initially provided. All job modifications that
might have been applied in
jsv_on_verify() before this function was
called, are then ignored.
Instead of calling
jsv_accept() in
jsv_on_verify(), the functions
jsv_correct(),
jsv_reject() or
jsv_reject_wait() can be
called, but only one of these functions can be used at a time.
jsv_correct(message)¶
This function can only be used in
jsv_on_verify(). After it has been
called, the function
jsv_on_verify() has to return immediately.
A call to this function indicates that the job that is currently being verified
has to be modified before it can be accepted. All job parameter modifications
that were previously applied will be committed and the job will be accepted.
"Accept" in that case means that the job will either be passed to
the next JSV instance for modification or that it is passed to that component
in the master daemon that adds it to the master data store when the last JSV
instance has verified the job.
Instead of calling
jsv_correct() in
jsv_on_verify(), the functions
jsv_accept(),
jsv_reject() or
jsv_reject_wait() can be
called, but only one of these functions can be used.
jsv_reject(message)¶
This function can only be used in
jsv_on_verify(). After it has been
called the function
jsv_on_verify() has to return immediately.
The job that is currently being verified will be rejected.
message will
be passed to the client application that tried to submit the job. Commandline
clients like
qsub will print that message to stdout to inform the user
that the submission has failed.
jsv_reject_wait() should be called if the user may try to submit the job
again.
jsv_reject_wait() indicates that the verification process might
be successful in the future.
Instead of calling
jsv_reject() in
jsv_on_verify(), the functions
jsv_accept(),
jsv_correct() or
jsv_reject_wait() can be
called, but only one of these functions can be used.
jsv_reject_wait(message)¶
This function can only be used in
jsv_on_verify(). After it has been
called the function
jsv_on_verify() has to return immediately.
The job which is currently verified will be rejected.
message will be
passed to the client application, that tries to submit the job. Commandline
clients like
qsub will print that message to stdout to inform the user
that the submission has failed.
This function should be called if the user who tries to submit the job might
have a chance to submit the job later.
jsv_reject indicates that the
verified job will also be rejected in future.
Instead of calling
jsv_reject_wait() in
jsv_on_verify() the
functions
jsv_accept(),
jsv_correct() or
jsv_reject() can
be called, but only one of these functions can be used.
jsv_log_info(message)¶
This function sends an info
message to the client or master daemon
instance that started the JSV script.
For client JSVs, this means that the command line client will get the
information and print it to the stdout stream. Server JSVs will print that
message as an info message to the master daemon message file.
If
message is missing then an empty line will be printed.
jsv_log_warning(message)¶
This function sends a warning
message to the client or master daemon
instance that started the JSV script.
For client JSVs, this means that the command line client will get the
information and print it to the stdout stream. Server JSVs will print that
message as a warning message to the master daemon message file.
If
message is missing then an empty line will be printed.
jsv_log_error(message)¶
This function sends an error
message to the client or master daemon
instance that started the JSV script.
For client JSVs, this means that the command line client will get the
information and print it to the stdout stream. Server JSVs will print that
message as an error message to the master daemon message file.
If
message is missing then an empty line will be printed.
jsv_send_env()¶
This function can only be used in
jsv_on_start(). If it is used there,
then the job environment information will be available in
jsv_on_verify() for the next job that is scheduled to be verified.
This function must be called for the functions
jsv_show_envs(),
jsv_is_env(),
jsv_get_env(),
jsv_add_env() and
jsv_mod_env() to behave correctly.
Job environments might become very big (10kB and more). This will slow down the
executing component (submit client or master daemon thread). For this reason,
job environment information is not passed to JSV scripts by default.
Please note also that the data in the job environment can't be verified by Grid
Engine and might therefore contain values which could be misinterpreted in the
script environment and cause security issues.
jsv_main()¶
This function has to be called in the main function in JSV scripts. It
implements the JSV protocol and performs the communication with client and
server components which might start JSV scripts.
This function does not return immediately. It returns only when the
"QUIT" command is sent by the client or server component.
During the communication with client and server components, this function
triggers two callback functions for each job that should be verified. First
jsv_on_start() and later on
jsv_on_verify().
jsv_on_start() can be used to initialize certain things that might be
needed for the verification process.
jsv_on_verify() does the
verification process itself.
The function
jsv_send_env() can be called in
jsv_on_start() so
that the job environment is available in
jsv_on_verify().
The following functions can only be used in
jsv_on_verify(). Simple job
parameters can be accessed/modified with:
jsv_is_param,
jsv_get_param,
jsv_set_param and
jsv_del_param.
List based job parameters can be accessed with:
jsv_sub_is_param,
jsv_sub_get_param,
jsv_sub_add_param and
jsv_sub_del_param.
If the environment was requested with
jsv_send_env() in
jsv_on_start() then the environment can be accessed/modified with the
following commands:
jsv_is_env,
jsv_get_env,
jsv_add_env,
jsv_mod_env and
jsv_del_env.
Jobs can be accepted/rejected with the following:
jsv_accept,
jsv_correct,
jsv_reject and
jsv_reject_wait.
The following functions send messages to the calling component of a JSV that
will either appear on the stdout stream of the client or in the master message
file. This is especially useful when new JSV scripts should be tested:
jsv_show_params , jsv_show_envs,
jsv_log_info , jsv_log_warning and
jsv_log_error .
jsv_on_start()¶
This is a callback function that has to be defined by the creator of a JSV
script. It is called for every job a short time before the verification
process of a job starts.
Within this function
jsv_send_env can be called to request job
environment information for the next job scheduled to be verified.
After this function returns
jsv_on_verify() will be called. This function
does the verification process itself.
jsv_on_verify()¶
This is a callback function that has to be defined by the creator of a JSV
script. It is called for every job, and when it returns the job will either be
accepted or rejected. Find implementation examples in the directory
$SGE_ROOT/util/resources/jsv.
The logic of this function completely depends on its creator. The creator has
only to take care that one of the functions
jsv_accept(),
jsv_reject(),
jsv_reject_wait() or
jsv_correct() is
called before the function returns.
jsv_logging_enabled¶
Setting this variable to
true produces logging output tracing the JSV
protocol, sent to a file of the form
/tmp/jsv_$$.log. In the case of
shell JSVs, it may be set in the environment of the job submission to effect
logging without modifying the script.
EXAMPLES¶
Find in the table below the returned values for the "*is*" and
"*get*" functions when following job is submitted:
qsub -l mem=1G,mem2=200M -l a=lx-amd64 ...
function call |
returned value |
_ |
_ |
jsv_is_param(l_hard) |
"true" |
jsv_get_param(l_hard) |
"mem=1G,mem2=200M,a=lx-amd64" |
jsv_sub_is_param(l_hard,mem) |
"true" |
jsv_sub_get_param(l_hard,mem) |
"1G" |
jsv_sub_is_param(l_hard,mem3) |
"false" |
jsv_sub_get_param(l_hard,mem3) |
"" |
jsv_sub_is_param(l_hard,a) |
"true" |
jsv_sub_get_param(l_hard,a) |
"lx-amd64" |
jsv_sub_is_param(l_hard,arch) |
"false" |
jsv_sub_get_param(l_hard,arch) |
"" |
FILES¶
Include files:
$SGE_ROOT/util/resources/jsv/jsv_include.sh
$SGE_ROOT/util/resources/jsv/jsv_include.tcl
$SGE_ROOT/util/resources/jsv/JSV.pm
Example files:
$SGE_ROOT/util/resources/jsv/jsv.sh
$SGE_ROOT/util/resources/jsv/jsv.tcl
$SGE_ROOT/util/resources/jsv/jsv.pl
$SGE_ROOT/util/resources/jsv/jjsv.sh
$SGE_ROOT/util/resources/jsv/SimpleJsv.java
Debugging log file:
/tmp/jsv_$$.log
BUGS¶
Complex names seen by the script are not canonicalized, i.e. if the name and
shortcut vary, it is necessary to consider both.
SEE ALSO¶
COPYRIGHT¶
See for a full statement of rights and permissions.