.\" Man page generated from reStructuredText.
.
.TH "CH-RUN" "1" "2018-12-14 19:09 Coordinated Universal Time" "" "Charliecloud"
.SH NAME
ch-run \- Run a command in a Charliecloud container
.
.nr rst2man-indent-level 0
.
.de1 rstReportMargin
\\$1 \\n[an-margin]
level \\n[rst2man-indent-level]
level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
-
\\n[rst2man-indent0]
\\n[rst2man-indent1]
\\n[rst2man-indent2]
..
.de1 INDENT
.\" .rstReportMargin pre:
. RS \\$1
. nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin]
. nr rst2man-indent-level +1
.\" .rstReportMargin post:
..
.de UNINDENT
. RE
.\" indent \\n[an-margin]
.\" old: \\n[rst2man-indent\\n[rst2man-indent-level]]
.nr rst2man-indent-level -1
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.SH SYNOPSIS
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
$ ch\-run [OPTION...] NEWROOT CMD [ARG...]
.ft P
.fi
.UNINDENT
.UNINDENT
.SH DESCRIPTION
.sp
Run command \fBCMD\fP in a Charliecloud container using the flattened and
unpacked image directory located at \fBNEWROOT\fP\&.
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.TP
.B \fB\-b\fP, \fB\-\-bind=SRC[:DST]\fP
mount \fBSRC\fP at guest \fBDST\fP (default \fB/mnt/0\fP,
\fB/mnt/1\fP, etc.)
.TP
.B \fB\-c\fP, \fB\-\-cd=DIR\fP
initial working directory in container
.TP
.B \fB\-g\fP, \fB\-\-gid=GID\fP
run as group \fBGID\fP within container
.TP
.B \fB\-j\fP, \fB\-\-join\fP
use the same container (namespaces) as peer \fBch\-run\fP invocations
.TP
.B \fB\-\-join\-pid=PID\fP
join the namespaces of an existing process
.TP
.B \fB\-\-join\-ct=N\fP
number of \fBch\-run\fP peers (implies \fB\-\-join\fP; default: see below)
.TP
.B \fB\-\-join\-tag=TAG\fP
label for \fBch\-run\fP peer group (implies \fB\-\-join\fP; default: see
below)
.TP
.B \fB\-\-no\-home\fP
do not bind\-mount your home directory (by default, your home directory is
mounted at \fB/home/$USER\fP in the container)
.TP
.B \fB\-t\fP, \fB\-\-private\-tmp\fP
use container\-private \fB/tmp\fP (by default, \fB/tmp\fP is shared with
the host)
.TP
.B \fB\-u\fP, \fB\-\-uid=UID\fP
run as user \fBUID\fP within container
.TP
.B \fB\-v\fP, \fB\-\-verbose\fP
be more verbose (debug if repeated)
.TP
.B \fB\-w\fP, \fB\-\-write\fP
mount image read\-write (by default, the image is mounted read\-only)
.TP
.B \fB\-?\fP, \fB\-\-help\fP
print help and exit
.TP
.B \fB\-\-usage\fP
print a short usage message and exit
.TP
.B \fB\-V\fP, \fB\-\-version\fP
print version and exit
.UNINDENT
.UNINDENT
.UNINDENT
.SH HOST FILES AND DIRECTORIES AVAILABLE IN CONTAINER VIA BIND MOUNTS
.sp
In addition to any directories specified by the user with \fB\-\-bind\fP,
\fBch\-run\fP has standard host files and directories that are bind\-mounted
in as well.
.sp
The following host files and directories are bind\-mounted at the same location
in the container. These cannot be disabled.
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
\fB/dev\fP
.IP \(bu 2
\fB/etc/passwd\fP
.IP \(bu 2
\fB/etc/group\fP
.IP \(bu 2
\fB/etc/hosts\fP
.IP \(bu 2
\fB/etc/resolv.conf\fP
.IP \(bu 2
\fB/proc\fP
.IP \(bu 2
\fB/sys\fP
.UNINDENT
.UNINDENT
.UNINDENT
.sp
Three additional bind mounts can be disabled by the user:
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
Your home directory (i.e., \fB$HOME\fP) is mounted at guest
\fB/home/$USER\fP by default. This is accomplished by mounting a new
\fBtmpfs\fP at \fB/home\fP, which hides any image content under that
path. If \fB\-\-no\-home\fP is specified, neither of these things happens
and the image’s \fB/home\fP is exposed unaltered.
.IP \(bu 2
\fB/tmp\fP is shared with the host by default. If \fB\-\-private\-tmp\fP
is specified, a new \fBtmpfs\fP is mounted on the guest’s \fB/tmp\fP
instead.
.IP \(bu 2
If file \fB/usr/bin/ch\-ssh\fP is present in the image, it is
over\-mounted with the \fBch\-ssh\fP binary in the same directory as
\fBch\-run\fP\&.
.UNINDENT
.UNINDENT
.UNINDENT
.SH MULTIPLE PROCESSES IN THE SAME CONTAINER WITH --JOIN
.sp
By default, different \fBch\-run\fP invocations use different user and mount
namespaces (i.e., different containers). While this has no impact on sharing
most resources between invocations, there are a few important exceptions.
These include:
.INDENT 0.0
.IP 1. 3
\fBptrace(2)\fP, used by debuggers and related tools. One can attach a
debugger to processes in descendant namespaces, but not sibling namespaces.
The practical effect of this is that (without \fB\-\-join\fP), you can’t
run a command with \fBch\-run\fP and then attach to it with a debugger
also run with \fBch\-run\fP\&.
.IP 2. 3
\fICross\-memory attach\fP (CMA) is used by cooperating processes to communicate
by simply reading and writing one another’s memory. This is also not
permitted between sibling namespaces. This affects various MPI
implementations that use CMA to pass messages between ranks on the same
node, because it’s faster than traditional shared memory.
.UNINDENT
.sp
\fB\-\-join\fP is designed to address this by placing related \fBch\-run\fP
commands (the “peer group”) in the same container. This is done by one of the
peers creating the namespaces with \fBunshare(2)\fP and the others joining
with \fBsetns(2)\fP\&.
.sp
To do so, we need to know the number of peers and a name for the group. These
are specified by additional arguments that can (hopefully) be left at default
values in most cases:
.INDENT 0.0
.IP \(bu 2
\fB\-\-join\-ct\fP sets the number of peers. The default is the value of the
first of the following environment variables that is defined:
\fBOMPI_COMM_WORLD_LOCAL_SIZE\fP, \fBSLURM_STEP_TASKS_PER_NODE\fP,
\fBSLURM_CPUS_ON_NODE\fP\&.
.IP \(bu 2
\fB\-\-join\-tag\fP sets the tag that names the peer group. The default is
environment variable \fBSLURM_STEP_ID\fP, if defined; otherwise, the PID
of \fBch\-run\fP’s parent. Tags can be re\-used for peer groups that start
at different times, i.e., once all peer \fBch\-run\fP have replaced
themselves with the user command, the tag can be re\-used.
.UNINDENT
.sp
Caveats:
.INDENT 0.0
.IP \(bu 2
One cannot currently add peers after the fact, for example, if one decides
to start a debugger after the fact. (This is only required for code with
bugs and is thus an unusual use case.)
.IP \(bu 2
\fBch\-run\fP instances race. The winner of this race sets up the
namespaces, and the other peers use the winner to find the namespaces to
join. Therefore, if the user command of the winner exits, any remaining
peers will not be able to join the namespaces, even if they are still
active. There is currently no general way to specify which \fBch\-run\fP
should be the winner.
.IP \(bu 2
If \fB\-\-join\-ct\fP is too high, the winning \fBch\-run\fP’s user command
exits before all peers join, or \fBch\-run\fP itself crashes, IPC resources
such as semaphores and shared memory segments will be leaked. These appear
as files in \fB/dev/shm/\fP and can be removed with \fBrm(1)\fP\&.
.IP \(bu 2
Many of the arguments given to the race losers, such as the image path and
\fB\-\-bind\fP, will be ignored in favor of what was given to the winner.
.UNINDENT
.SH ENVIRONMENT VARIABLES
.sp
\fBch\-run\fP generally tries to leave environment variables unchanged, but
in some cases, guests can be significantly broken unless environment variables
are tweaked. This section lists those changes.
.INDENT 0.0
.IP \(bu 2
\fB$HOME\fP: If the path to your home directory is not \fB/home/$USER\fP
on the host, then an inherited \fB$HOME\fP will be incorrect inside the
guest. This confuses some software, such as Spack.
.sp
Thus, we change \fB$HOME\fP to \fB/home/$USER\fP, unless
\fB\-\-no\-home\fP is specified, in which case it is left unchanged.
.IP \(bu 2
\fB$PATH\fP: Newer Linux distributions replace some root\-level
directories, such as \fB/bin\fP, with symlinks to their counterparts in
\fB/usr\fP\&.
.sp
Some of these distributions (e.g., Fedora 24) have also dropped \fB/bin\fP
from the default \fB$PATH\fP\&. This is a problem when the guest OS does
\fInot\fP have a merged \fB/usr\fP (e.g., Debian 8 “Jessie”). Thus, we add
\fB/bin\fP to \fB$PATH\fP if it’s not already present.
.sp
Further reading:
.INDENT 2.0
.INDENT 3.5
.INDENT 0.0
.IP \(bu 2
\fI\%The case for the /usr Merge\fP
.IP \(bu 2
\fI\%Fedora\fP
.IP \(bu 2
\fI\%Debian\fP
.UNINDENT
.UNINDENT
.UNINDENT
.UNINDENT
.SH EXAMPLES
.sp
Run the command \fBecho hello\fP inside a Charliecloud container using the
unpacked image at \fB/data/foo\fP:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
$ ch\-run /data/foo \-\- echo hello
hello
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Run an MPI job that can use CMA to communicate:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
$ srun ch\-run \-\-join /data/foo \-\- bar
.ft P
.fi
.UNINDENT
.UNINDENT
.SH REPORTING BUGS
.sp
If Charliecloud was obtained from your Linux distribution, use your
distribution’s bug reporting procedures.
.sp
Otherwise, report bugs to: <\fI\%https://github.com/hpc/charliecloud/issues\fP>
.SH SEE ALSO
.sp
charliecloud(1)
.sp
Full documentation at: <\fI\%https://hpc.github.io/charliecloud\fP>
.SH AUTHOR
Reid Priedhorsky, Tim Randles, and others
.SH COPYRIGHT
2014–2018, Los Alamos National Security, LLC
.\" Generated by docutils manpage writer.
.