.TH slurmrestd "8" "Slurm REST Daemon" "July 2023" "Slurm REST Daemon"

.SH "NAME"
slurmrestd \- Interface to Slurm via REST API.
.SH "SYNOPSIS"
\fBslurmrestd\fR [\fIOPTIONS\fR...] <\fI[host]:port\fR|\fIunix:/path/to/socket\fR>...
.SH "DESCRIPTION"
\fBslurmrestd\fR is REST API interface for Slurm. It can be used in two modes:

.PP
\fBInetd Mode\fR: slurmrestd will read and write to STDIO. If STDIN is a socket
file descriptor, then slurmrestd will detect this and use relevant
functionality. If a controlling TTY is detected, interactive mode will
automatically activate to provide additional logging information. This mode is
designed to work with piped input, inetd, xinetd or systemd socket activation.

.PP
\fBListen Mode\fR: slurmrestd will open a listening socket on each requested
\fIhost\fR:\fIport\fR pair or UNIX socket.

.SH "OPTIONS"

.TP
\fB[host]:port\fR
Hostname and port to listen against. \fIhost\fR may be an IPv4/IPv6 IP or a
resolvable hostname. Hostnames are only looked up at startup and do not change
for the life of the process. \fIhost\fR is optional; if not provided, slurmrestd
will listen on all network interfaces.
.IP

.TP
\fBunix:/path/to/socket\fR
Listen on local UNIX socket. Must have permission to create socket in
filesystem.
.IP

.TP
\fB\-a <authentication plugins>\fR
Comma\-delimited list of authentication plugins to load.
Default behavior is to load all REST authentication plugins found at load time.
.RS
.TP
\fBlist\fR
Display a list of the possible plugins to load.
.IP

.TP
\fBrest_auth/local\fR
Allows authentication via UNIX sockets when \fBauth/munge\fR is active.
.br
\fBNOTE\fR: slurmrestd and client processes must run under the same UID or the
client requests will be rejected.
.IP

.TP
\fBrest_auth/jwt\fR
Allows authentication via TCP and UNIX sockets when \fBAuthAltTypes=auth/jwt\fR
is active. User must specify the following HTTP cookies with each request:
.RS
.TP
\fBX-SLURM-USER-NAME\fR:<user name>
.IP
.TP
\fBX-SLURM-USER-TOKEN\fR:<JSON Web Token>
.RE
.IP
\fBNOTE\fR: Tokens are usually generated via calling "\fBscontrol token\fR".
.RE
.IP

.TP
\fB\-d <data_parser plugins to load>\fR
Comma\-delimited list of data_parser plugins.
Set to "list" to dump a list of the possible plugins to load.
Defaults: all builtin supported data_parser plugins.
.IP

.TP
\fB\-f <file>\fR
Read Slurm configuration from the specified file. See \fBNOTES\fR below.
.IP

.TP
\fB\-g <group id>\fR
Change group id (and drop supplemental groups) before processing client
request. This should be a unique group with no write access or special
permissions. Do not set this to the group belonging to to SlurmUser or
root or the daemon won't start with the default settings.
.IP

.TP
\fB\-h\fR
Help; print a brief summary of command options.
.IP

.TP
\fB\-\-max\-connections <count>\fR
Set the maximum number of connections to process at any one time. This is
independent of the number of connections that can connect to slurmrestd at any
one time. The kernel allows any number of connections to be pending for
processing at any one time when SYN cookies are active.
.RS
.TP
\fBCaution\fR:
Each connection could cause one RPC to the controller daemons, leading to
potential overloading of the controller. Each connection can also hold memory
for the duration of the life of the connection. Having too many connections
processing at once could use considerably more memory. Process limits
(\fBulimit\fR(3)) may require adjustment when this value is increased.
.TP
Default: 124
.RE
.IP

.TP
\fB\-s <OpenAPI plugins to load>\fR
Comma\-delimited list of OpenAPI plugins.
Set to "list" to dump a list of the possible plugins to load.
Defaults: all builtin supported OpenAPI plugins.
.IP

.TP
\fB\-t <THREAD COUNT>\fR
Specify number of threads to use to process client connections.
Ignored in inetd mode. Default: 20
.IP

.TP
\fB\-u <user id>\fR
Change user id before processing client request. This should be a unique group
with no write access or special permissions. Do not set this user to SlurmUser
or root or the daemon won't start with the default settings.
.IP

.TP
\fB\-v\fR
Verbose operation. Multiple \fBv\fR's can be specified, with each '\fBv\fR'
beyond the first increasing verbosity, up to 6 times (i.e. \-vvvvvv).
Higher verbosity levels will have significant performance impact.
.IP

.TP
\fB\-V\fR
Print version information and exit.
.IP

.SH "ENVIRONMENT VARIABLES"
The following environment variables can be used to override settings
compiled into slurmctld.

.TP
\fBSLURM_CONF\fR
The location of the Slurm configuration file.
.IP

.TP
\fBSLURM_DEBUG_FLAGS\fR
Specify debug flags for slurmrestd to use. See DebugFlags in the
\fBslurm.conf\fR(5) man page for a full list of flags. The environment
variable takes precedence over the setting in the slurm.conf.
.IP

.TP
\fBSLURMRESTD_JSON\fR
Control JSON serialization:
.IP
.RS
.TP
\fBcompact\fR
Output JSON as compact as possible.
.IP

.TP
\fBpretty\fR
Output JSON in pretty format to make it more readable.
.IP
.RE

.TP
\fBSLURM_JWT\fR
This variable must be set to use JWT token authentication.
.IP

.TP
\fBSLURMRESTD_AUTH_TYPES\fR
Set allowed authentication types. See \fB\-a\fR
.IP

.TP
\fBSLURMRESTD_DEBUG\fR
Set debug level explicitly. Valid values are 0\-9, or the same string values as
the debug options such as SlurmctldDebug in slurm.conf(5).
Ignored if \fB\-v\fR passed as argument during invocation.
.IP

.TP
\fBSLURMRESTD_DATA_PARSER_PLUGINS\fR
Comma\-delimited list of data_parser plugins to load. See \fB\-d\fR
.IP

.TP
\fBSLURMRESTD_LISTEN\fR
Comma\-delimited list of host:port pairs or unix sockets to listen on.
.IP

.TP
\fBSLURMRESTD_MAX_CONNECTIONS\fR
Set the maximum number of connections to process at any one time. See
\fB\-\-max\-connections\fR
.IP

.TP
\fBSLURMRESTD_OPENAPI_PLUGINS\fR
Comma\-delimited list of OpenAPI plugins to load. See \fB\-s\fR
.IP

.TP
\fBSLURMRESTD_SECURITY\fR
Control slurmrestd security functionality using the following comma\-delimited
values:
.IP
.RS
.TP
\fBbecome_user\fR
Allows \fBslurmrestd\fR to be run as root in order to become the requesting
user for all requests. When combined with \fBrest_auth/local\fB, when a user
connects via a named UNIX socket, \fBslurmrestd\fR will setuid()/setgid() into
that user/group and then complete all requests as the given user. This mode is
only intended for inet mode as the user change is permanent for the life of the
process. This mode is incompatible with \fBrest_auth/jwt\fR and it is suggested
to start \fBslurmrestd\fR with "-a \fBrest_auth/local\fR" arguments.
.IP

.TP
\fBdisable_unshare_files\fR
Disables unsharing file descriptors with parent process.
.IP

.TP
\fBdisable_unshare_sysv\fR
Disables unsharing the SYSV namespace.
.IP

.TP
\fBdisable_user_check\fR
Disables check that slurmrestd is not running as root or SlurmUser, or with the
root or SlurmUser's primary group.
.RE
.IP

.TP
\fBSLURMRESTD_YAML\fR
Control YAML serialization:
.IP
.RS
.TP
\fBcompact\fR
Output YAML as compact as possible.
.IP

.TP
\fBpretty\fR
Output YAML in pretty format to make it more readable.
.RE
.IP

.SH "SIGNALS"

.TP 6
\fBSIGINT\fR
\fBslurmrestd\fR will shutdown cleanly.
.IP

.TP
\fBSIGPIPE\fR
This signal is explicitly ignored.
.IP

.SH "NOTES"
\fBSPANK\fR and \fBclifilter\fR plugins are not supported in \fBslurmrestd\fR
due to their lack of thread safety. Active \fBSPANK\fR plugins and
\fBJobSubmitPlugins\fR in \fBslurmctld\fR are independent of slurmrestd and can
be used to enforce site policy on job submissions.

.SH "EXAMPLES"

.LP
Start \fBslurmrestd\fR with a UNIX socket in listen mode:
.IP
.nf
$ export SLURMRESTD=/var/spool/slurm/restd/rest
$ slurmrestd -s dbv0.0.39,v0.0.39 unix:$SLURMRESTD
.fi

.LP
Verify connectivity with the controller with a ping, with \fBslurmrestd\fR
running in listen mode:
.IP
.nf
$ curl --unix-socket "${SLURMRESTD}" 'http://localhost:8080/slurm/v0.0.39/ping'
{
  "meta": {
    "plugin": {
      "type": "openapi\/v0.0.39",
      "name": "Slurm OpenAPI v0.0.39",
      "data_parser": "v0.0.39"
    },
    "client": {
      "source": "\/tmp\/slurmrestd\/restd->fd:8"
    },
    "Slurm": {
      "version": {
        "major": 23,
        "micro": 3,
        "minor": 2
      },
      "release": "23.02.3"
    }
  },
  "errors": [
  ],
  "warnings": [
  ],
  "pings": [
    {
      "hostname": "kitt",
      "pinged": "UP",
      "latency": 606,
      "mode": "primary"
    }
  ]
}
.fi

.LP
Query the status of a node with \fBslurmrestd\fR running in INETD mode:
.IP
.nf
$ echo -e "GET http://localhost:8080/slurm/v0.0.39/node/node01 HTTP/1.1\\r\\n" | slurmrestd
slurmrestd: operations_router: [fd:0->/dev/pts/1] GET /slurm/v0.0.39/node/node01
slurmrestd: rest_auth/local: slurm_rest_auth_p_authenticate: [fd:0->/dev/pts/1] accepted connection from user: user1[1001]
HTTP/1.1 200 OK
Content-Length: 3777
Content-Type: application/json

{
  "meta": {
    "plugin": {
      "type": "openapi\/v0.0.39",
      "name": "Slurm OpenAPI v0.0.39",
      "data_parser": "v0.0.39"
    },
    "client": {
      "source": "fd:0->\/dev\/pts\/1"
    },
    "Slurm": {
      "version": {
        "major": 23,
        "micro": 3,
        "minor": 2
      },
      "release": "23.02.3"
    }
  },
  "errors": [
  ],
  "warnings": [
  ],
  "nodes": [
    {
      "architecture": "x86_64",
      "burstbuffer_network_address": "",
      "boards": 1,
      "boot_time": 1688652669,
      "cluster_name": "",
      "cores": 12,
      "specialized_cores": 0,
      "cpu_binding": 0,
      "cpu_load": {
        "set": true,
        "infinite": false,
        "number": 17
      },
      "free_mem": {
        "set": true,
        "infinite": false,
        "number": 187
      },
      "cpus": 24,
      "effective_cpus": 24,
      "specialized_cpus": "",
      "energy": {
        "average_watts": 0,
        "base_consumed_energy": 0,
        "consumed_energy": 0,
        "current_watts": 0,
        "previous_consumed_energy": 0,
        "last_collected": 0
      },
      "external_sensors": {
        "consumed_energy": {
          "set": false,
          "infinite": false,
          "number": 0
        },
        "temperature": {
          "set": false,
          "infinite": false,
          "number": 0
        },
        "energy_update_time": 0,
        "current_watts": 0
      },
      "extra": "",
      "power": {
        "maximum_watts": {
          "set": false,
          "infinite": false,
          "number": 0
        },
        "current_watts": 0,
        "total_energy": 0,
        "new_maximum_watts": 0,
        "peak_watts": 0,
        "lowest_watts": 0,
        "new_job_time": 0,
        "state": 0,
        "time_start_day": 0
      },
      "features": [
        "rhel7",
        "rhel8",
        "rhel76",
        "rhel79",
        "rhel85",
        "rack1"
      ],
      "active_features": [
        "rhel7",
        "rack1"
      ],
      "gres": "cpu:24,gpu:tesla:4(S:0),test:8",
      "gres_drained": "N\/A",
      "gres_used": "cpu:0,gpu:tesla:0(IDX:N\/A),tesla:0,test:0",
      "last_busy": 1688671269,
      "mcs_label": "",
      "specialized_memory": 0,
      "name": "node01",
      "next_state_after_reboot": [
        "INVALID",
        "PERFCTRS",
        "RESERVED",
        "UNDRAIN",
        "CLOUD",
        "RESUME",
        "DRAIN",
        "COMPLETING",
        "NOT_RESPONDING",
        "POWERED_DOWN",
        "FAIL",
        "POWERING_UP",
        "MAINTENANCE",
        "REBOOT_REQUESTED",
        "REBOOT_CANCELED",
        "POWERING_DOWN",
        "DYNAMIC_FUTURE",
        "REBOOT_ISSUED",
        "PLANNED",
        "INVALID_REG",
        "POWER_DOWN",
        "POWER_UP",
        "POWER_DRAIN",
        "DYNAMIC_NORM"
      ],
      "address": "kitt",
      "hostname": "kitt",
      "state": [
        "IDLE"
      ],
      "operating_system": "Linux 5.15.0-76-generic #83-Ubuntu SMP Thu Jun 15 19:16:32 UTC 2023",
      "owner": "",
      "partitions": [
        "debug"
      ],
      "port": 18001,
      "real_memory": 15678,
      "comment": "",
      "reason": "",
      "reason_changed_at": 0,
      "reason_set_by_user": "",
      "resume_after": {
        "set": true,
        "infinite": false,
        "number": 0
      },
      "reservation": "",
      "alloc_memory": 0,
      "alloc_cpus": 0,
      "alloc_idle_cpus": 24,
      "tres_used": "",
      "tres_weighted": 0.0,
      "slurmd_start_time": 1688671266,
      "sockets": 1,
      "threads": 2,
      "temporary_disk": 0,
      "weight": 1,
      "tres": "cpu=24,mem=15678M,billing=39,gres\/gpu=4,gres\/gpu:tesla=4,gres\/test=8",
      "version": "23.02.3"
    }
  ]
}
.fi

.LP
Submit a job to \fBslurmrestd\fR with it running in listen mode:
.IP
.nf
$ cat example_job.json
{"script": "#!/bin/bash\\nsleep 30",
  "job": {
    "name": "ExampleJob",
    "account": "sub1",
    "hold": false,
    "environment": {
      "PATH": "/bin"
    },
    "tasks": 12,
    "memory_per_cpu": 100,
    "time_limit": 240
  }
}

$ curl -H "Content-Type: application/json" -d @example_job.json --unix-socket "${SLURMRESTD}" 'http://localhost:8080/slurm/v0.0.39/job/submit'
{
  "meta": {
    "plugin": {
      "type": "openapi\/v0.0.39",
      "name": "Slurm OpenAPI v0.0.39",
      "data_parser": "v0.0.39"
    },
    "client": {
      "source": "\/tmp\/slurmrestd\/restd->fd:8"
    },
    "Slurm": {
      "version": {
        "major": 23,
        "micro": 3,
        "minor": 2
      },
      "release": "23.02.3"
    }
  },
  "errors": [
  ],
  "warnings": [
    {
      "description": "Expected OpenAPI type=array (Slurm type=list) but got OpenAPI type=object (Slurm type=dictionary)",
      "source": "#\/job\/environment\/"
    },
    {
      "description": "Expected OpenAPI type=object (Slurm type=dictionary) but got OpenAPI type=integer (Slurm type=int 64bits)",
      "source": "#\/job\/time_limit\/"
    },
    {
      "description": "Expected OpenAPI type=object (Slurm type=dictionary) but got OpenAPI type=integer (Slurm type=int 64bits)",
      "source": "#\/job\/memory_per_cpu\/"
    }
  ],
  "result": {
    "job_id": 8990,
    "step_id": "batch",
    "error_code": 0,
    "error": "No error",
    "job_submit_user_msg": ""
  },
  "job_id": 8990,
  "step_id": "batch",
  "job_submit_user_msg": ""
}
.fi

.SH "COPYING"
Copyright (C) 2019\-2022 SchedMD LLC.
.LP
This file is part of Slurm, a resource management program.
For details, see <https://slurm.schedmd.com/>.
.LP
Slurm is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free
Software Foundation; either version 2 of the License, or (at your option)
any later version.
.LP
Slurm is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
details.

.SH "SEE ALSO"
\fBslurm.conf\fR(5), \fBslurmctld\fR(8), \fBslurmdbd\fR(8)