'\" t
.\" Title: boothd
.\" Author: [see the "AUTHOR" section]
.\" Generator: DocBook XSL Stylesheets v1.79.1
.\" Date: 10/12/2016
.\" Manual: \ \&
.\" Source: \ \&
.\" Language: English
.\"
.TH "BOOTHD" "8" "10/12/2016" "\ \&" "\ \&"
.\" -----------------------------------------------------------------
.\" * Define some portability stuff
.\" -----------------------------------------------------------------
.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.\" http://bugs.debian.org/507673
.\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html
.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.ie \n(.g .ds Aq \(aq
.el .ds Aq '
.\" -----------------------------------------------------------------
.\" * set default formatting
.\" -----------------------------------------------------------------
.\" disable hyphenation
.nh
.\" disable justification (adjust text to left margin only)
.ad l
.\" -----------------------------------------------------------------
.\" * MAIN CONTENT STARTS HERE *
.\" -----------------------------------------------------------------
.SH "NAME"
boothd \- The Booth Cluster Ticket Manager\&.
.SH "SYNOPSIS"
.sp
\fBboothd\fR \fIdaemon\fR [\-SD] [\-c \fIconfig\fR] [\-l \fIlockfile\fR]
.sp
\fBbooth\fR \fIlist\fR [\-s \fIsite\fR] [\-c \fIconfig\fR]
.sp
\fBbooth\fR \fIgrant\fR [\-s \fIsite\fR] [\-c \fIconfig\fR] [\-FCw] \fIticket\fR
.sp
\fBbooth\fR \fIrevoke\fR [\-s \fIsite\fR] [\-c \fIconfig\fR] [\-w] \fIticket\fR
.sp
\fBbooth\fR \fIpeers\fR [\-s \fIsite\fR] [\-c \fIconfig\fR]
.sp
\fBbooth\fR \fIstatus\fR [\-D] [\-c \fIconfig\fR]
.SH "DESCRIPTION"
.sp
Booth manages tickets which authorizes one of the cluster sites located in geographically dispersed distances to run certain resources\&. It is designed to be extend Pacemaker to support geographically distributed clustering\&.
.sp
It is based on the RAFT protocol, see eg\&. https://ramcloud\&.stanford\&.edu/wiki/download/attachments/11370504/raft\&.pdf for details\&.
.SH "SHORT EXAMPLES"
.sp
.if n \{\
.RS 4
.\}
.nf
# boothd daemon \-D
# booth list
# booth grant ticket\-nfs
# booth revoke ticket\-nfs
.fi
.if n \{\
.RE
.\}
.SH "OPTIONS"
.PP
\fB\-c\fR \fIconfigfile\fR
.RS 4
Configuration to use\&.
.sp
Can be a full path to a configuration file, or a short name; in the latter case, the directory
\fI/etc/booth\fR
and suffix
\fI\&.conf\fR
are added\&. Per default
\fIbooth\fR
is used, which results in the path
\fI/etc/booth/booth\&.conf\fR\&.
.sp
The configuration name also determines the name of the PID file \- for the defaults,
\fI/var/run/booth/booth\&.pid\fR\&.
.RE
.PP
\fB\-s\fR
.RS 4
Site address or name\&.
.sp
.if n \{\
.RS 4
.\}
.nf
The special value \*(Aqother\*(Aq can be used to specify the other
site\&. Obviously, in that case, the booth configuration must
have exactly two sites defined\&.
.fi
.if n \{\
.RE
.\}
.RE
.PP
\fB\-F\fR
.RS 4
\fIimmediate grant\fR: Don\(cqt wait for unreachable sites to relinquish the ticket\&. See the
\fIBooth ticket management\fR
section below for more details\&.
.sp
.if n \{\
.RS 4
.\}
.nf
This option may be DANGEROUS\&. It makes booth grant the ticket
even though it cannot ascertain that unreachable sites don\*(Aqt
hold the same ticket\&. It is up to the user to make sure that
unreachable sites don\*(Aqt have this ticket as granted\&.
.fi
.if n \{\
.RE
.\}
.RE
.PP
\fB\-w\fR
.RS 4
\fIwait for the request outcome\fR: The client waits for the final outcome of grant or revoke request\&.
.RE
.PP
\fB\-C\fR
.RS 4
\fIwait for ticket commit to CIB\fR: The client waits for the ticket commit to CIB (only for grant requests)\&. If one or more sites are unreachable, this takes the ticket expire time (plus, if defined, the
\fIacquire\-after\fR
time)\&.
.RE
.PP
\fB\-h\fR, \fB\-\-help\fR
.RS 4
Give a short usage output\&.
.RE
.PP
\fB\-\-version\fR
.RS 4
Report version information\&.
.RE
.PP
\fB\-S\fR
.RS 4
\fIsystemd\fR
mode: don\(cqt fork\&. This is like
\fI\-D\fR
but without the debug output\&.
.RE
.PP
\fB\-D\fR
.RS 4
Debug output/don\(cqt daemonize\&. Increases the debug output level; booth daemon remains in the foreground\&.
.RE
.PP
\fB\-l\fR \fIlockfile\fR
.RS 4
Use another lock file\&. By default, the lock file name is inferred from the configuration file name\&. Normally not needed\&.
.RE
.SH "COMMANDS"
.sp
Whether the binary is called as \fIboothd\fR or \fIbooth\fR doesn\(cqt matter; the first argument determines the mode of operation\&.
.PP
\fB\fIdaemon\fR\fR
.RS 4
Tells
\fIboothd\fR
to serve a site\&. The locally configured interfaces are searched for an IP address that is defined in the configuration\&. booth then runs in either /arbitrator/ or /site/ mode\&.
.RE
.PP
\fB\fIclient\fR\fR
.RS 4
Booth clients can list the ticket information (see also
\fIcrm_ticket \-L\fR), and revoke or grant tickets to a site\&.
.sp
The grant and, under certain circumstances, revoke operations may take a while to return a definite operation\(cqs outcome\&. The client will wait up to the network timeout value (by default 5 seconds) for the result\&. Unless the
\fI\-w\fR
option was set, in which case the client waits indefinitely\&.
.sp
In this mode the configuration file is searched for an IP address that is locally reachable, ie\&. matches a configured subnet\&. This allows to run the client commands on another node in the same cluster, as long as the config file and the service IP is locally reachable\&.
.sp
For instance, if the booth service IP is 192\&.168\&.55\&.200, and the local node has 192\&.168\&.55\&.15 configured on one of its network interfaces, it knows which site it belongs to\&.
.sp
Use
\fI\-s\fR
to direct client to connect to a different site\&.
.RE
.PP
\fB\fIstatus\fR\fR
.RS 4
\fIboothd\fR
looks for the (locked) PID file and the UDP socket, prints some output to stdout (for use in shell scripts) and returns an OCF\-compatible return code\&. With
\fI\-D\fR, a human\-readable message is printed to STDERR as well\&.
.RE
.PP
\fB\fIpeers\fR\fR
.RS 4
List the other
\fIboothd\fR
servers we know about\&.
.sp
In addition to the type, name (IP address), and the last time the server was heard from, network statistics are also printed\&. The statistics are split into two rows, the first one consists of counters for the sent packets and the second one for the received packets\&. The first counter is the total number of packets and descriptions of the other counters follows:
.RE
.PP
\fIresends\fR
.RS 4
Packets which had to be resent because the recipient didn\(cqt acknowledge a message\&. This usually means that either the message or the acknowledgement got lost\&. The number of resends usually reflect the network reliability\&.
.RE
.PP
\fIerror\fR
.RS 4
Packets which either couldn\(cqt be sent, got truncated, or were badly formed\&. Should be zero\&.
.RE
.PP
\fIinvalid\fR
.RS 4
These packets contain either invalid or non\-existing ticket name or refer to a non\-existing ticket leader\&. Should be zero\&.
.RE
.PP
\fIauthfail\fR
.RS 4
Packets which couldn\(cqt be authenticated\&. Should be zero\&.
.RE
.SH "CONFIGURATION FILE"
.sp
The configuration file must be identical on all sites and arbitrators\&.
.sp
A minimal file may look like this:
.sp
.if n \{\
.RS 4
.\}
.nf
site="192\&.168\&.201\&.100"
site="192\&.168\&.202\&.100"
arbitrator="192\&.168\&.203\&.100"
ticket="ticket\-db8"
.fi
.if n \{\
.RE
.\}
.sp
Comments start with a hash\-sign (\fI\*(Aq#\fR\*(Aq)\&. Whitespace at the start and end of the line, and around the \fI\*(Aq=\fR\*(Aq, are ignored\&.
.sp
The following key/value pairs are defined:
.PP
\fB\fIport\fR\fR
.RS 4
The UDP/TCP port to use\&. Default is
\fI9929\fR\&.
.RE
.PP
\fB\fItransport\fR\fR
.RS 4
The transport protocol to use for Raft exchanges\&. Currently only UDP is supported\&.
.sp
Clients use TCP to communicate with a daemon; Booth will always bind and listen to both UDP and TCP ports\&.
.RE
.PP
\fB\fIauthfile\fR\fR
.RS 4
File containing the authentication key\&. The key can be either binary or text\&. If the latter, then both leading and trailing white space, including new lines, is ignored\&. This key is a shared secret and used to authenticate both clients and servers\&. The key must be between 8 and 64 characters long and be readable only by the file owner\&.
.RE
.PP
\fB\fImaxtimeskew\fR\fR
.RS 4
As protection against replay attacks, packets contain generation timestamps\&. Such a timestamp is not allowed to be too old\&. Just how old can be specified with this parameter\&. The value is in seconds and the default is 600 (10 minutes)\&. If clocks vary more than this default between sites and nodes (which is definitely something you should fix) then set this parameter to a higher value\&. The time skew test is performed only in concert with authentication\&.
.RE
.PP
\fB\fIsite\fR\fR
.RS 4
Defines a site Raft member with the given IP\&. Sites can acquire tickets\&. The sites\*(Aq IP should be managed by the cluster\&.
.RE
.PP
\fB\fIarbitrator\fR\fR
.RS 4
Defines an arbitrator Raft member with the given IP\&. Arbitrators help reach consensus in elections and cannot hold tickets\&.
.RE
.sp
Booth needs at least three members for normal operation\&. Odd number of members provides more redundancy\&.
.PP
\fB\fIsite\-user\fR\fR, \fB\fIsite\-group\fR\fR, \fB\fIarbitrator\-user\fR\fR, \fB\fIarbitrator\-group\fR\fR
.RS 4
These define the credentials
\fIboothd\fR
will be running with\&.
.sp
On a (Pacemaker) site the booth process will have to call
\fIcrm_ticket\fR, so the default is to use
\fIhacluster\fR:\*(Aqhaclient\*(Aq; for an arbitrator this user and group might not exists, so there we default to
\fInobody\fR:\*(Aqnobody\*(Aq\&.
.RE
.PP
\fB\fIticket\fR\fR
.RS 4
Registers a ticket\&. Multiple tickets can be handled by single Booth instance\&.
.sp
Use the special ticket name
\fI\fIdefaults\fR\fR
to modify the defaults\&. The
\fI\fIdefaults\fR\fR
stanza must precede all the other ticket specifications\&.
.RE
.sp
All times are in seconds\&.
.PP
\fB\fIexpire\fR\fR
.RS 4
The lease time for a ticket\&. After that time the ticket can be acquired by another site if the ticket holder is not reachable\&.
.sp
The default is
\fI600\fR\&.
.RE
.PP
\fB\fIacquire\-after\fR\fR
.RS 4
Once a ticket is lost, wait this time in addition before acquiring the ticket\&.
.sp
This is to allow for the site that lost the ticket to relinquish the resources, by either stopping them or fencing a node\&.
.sp
A typical delay might be 60 seconds, but ultimately it depends on the protected resources and the fencing configuration\&.
.sp
The default is
\fI0\fR\&.
.RE
.PP
\fB\fIrenewal\-freq\fR\fR
.RS 4
Set the ticket renewal frequency period\&.
.sp
If the network reliability is often reduced over prolonged periods, it is advisable to try to renew more often\&.
.sp
Before every renewal, if defined, the command specified in
\fIbefore\-acquire\-handler\fR
is run\&. In that case the
\fIrenewal\-freq\fR
parameter is effectively also the local cluster monitoring interval\&.
.RE
.PP
\fB\fItimeout\fR\fR
.RS 4
After that time
\fIbooth\fR
will re\-send packets if there was an insufficient number of replies\&. This should be long enough to allow packets to reach other members\&.
.sp
The default is
\fI5\fR\&.
.RE
.PP
\fB\fIretries\fR\fR
.RS 4
Defines how many times to retry sending packets before giving up waiting for acks from other members\&.
.sp
Default is
\fI10\fR\&. Values lower than 3 are illegal\&.
.sp
Ticket renewals should allow for this number of retries\&. Hence, the total retry time must be shorter than the renewal time (either half the expire time or
\fB\fIrenewal\-freq\fR\fR):
.sp
.if n \{\
.RS 4
.\}
.nf
timeout*(retries+1) < renewal
.fi
.if n \{\
.RE
.\}
.RE
.PP
\fB\fIweights\fR\fR
.RS 4
A comma\-separated list of integers that define the weight of individual Raft members, in the same order as the
\fIsite\fR
and
\fIarbitrator\fR
lines\&.
.sp
Default is
\fI0\fR
for all; this means that the order in the configuration file defines priority for conflicting requests\&.
.RE
.PP
\fB\fIbefore\-acquire\-handler\fR\fR
.RS 4
If set, this command will be called before
\fIboothd\fR
tries to acquire or renew a ticket\&. On exit code other than 0,
\fIboothd\fR
relinquishes the ticket\&.
.sp
Thus it is possible to ensure whether the services and its dependencies protected by the ticket are in good shape at this site\&. For instance, if a service in the dependency\-chain has a failcount of
\fIINFINITY\fR
on all available nodes, the service will be unable to run\&. In that case, it is of no use to claim the ticket\&.
.sp
See below for details about booth specific environment variables\&. The distributed
\fIservice\-runnable\fR
script is an example which may be used to test whether a pacemaker resource can be started\&.
.RE
.PP
\fB\fIattr\-prereq\fR\fR
.RS 4
Sites can have GEO attributes managed with the
\fIgeostore(8)\fR
program\&. Attributes are within ticket\(cqs scope and may be tested by
\fIboothd\fR
for additional control of ticket failover (automatic) or ticket acquire (manual)\&.
.sp
Attributes are typically used to convey extra information about resources, for instance database replication status\&. The attributes are commonly updated by resource agents\&.
.sp
Attribute values are referenced in expressions and may be tested for equality with the
\fIeq\fR
binary operator or inequality with the
\fIne\fR
operator\&. The usage is as follows:
.sp
.if n \{\
.RS 4
.\}
.nf
attr\-prereq =
.fi
.if n \{\
.RE
.\}
.sp
.if n \{\
.RS 4
.\}
.nf
: "auto" | "manual"
: attribute name
: "eq" | "ne"
: attribute value
.fi
.if n \{\
.RE
.\}
.sp
The two grant types are
\fIauto\fR
for ticket failover and
\fImanual\fR
for grants using the booth client\&. Only in case the expression evaluates to true can the ticket be granted\&.
.sp
It is not clear whether the
\fImanual\fR
grant type has any practical use because, obviously, this operation is anyway controlled by a human\&.
.sp
Note that there can be no guarantee on whether an attribute value is up to date, i\&.e\&. if it actually reflects the current state\&.
.RE
.sp
One example of a booth configuration file:
.sp
.if n \{\
.RS 4
.\}
.nf
transport = udp
port = 9930
# D\-85774
site="192\&.168\&.201\&.100"
# D\-90409
site="::ffff:192\&.168\&.202\&.100"
# A\-1120
arbitrator="192\&.168\&.203\&.100"
ticket="ticket\-db8"
expire = 600
acquire\-after = 60
timeout = 10
retries = 5
renewal\-freq = 60
before\-acquire\-handler = /usr/share/booth/service\-runnable db8
attr\-prereq = auto repl_state eq ACTIVE
.fi
.if n \{\
.RE
.\}
.SH "BOOTH TICKET MANAGEMENT"
.sp
The booth cluster guarantees that every ticket is owned by only one site at the time\&.
.sp
Tickets must be initially granted with the \fIbooth client grant\fR command\&. Once it gets granted, the ticket is managed by the booth cluster\&. Hence, only granted tickets are managed by \fIbooth\fR\&.
.sp
If the ticket gets lost, i\&.e\&. that the other members of the booth cluster do not hear from the ticket owner in a sufficiently long time, one of the remaining sites will acquire the ticket\&. This is what is called \fIticket failover\fR\&.
.sp
If the remaining members cannot form a majority, then the ticket cannot fail over\&.
.sp
A ticket may be revoked at any time with the \fIbooth client revoke\fR command\&. For revoke to succeed, the site holding the ticket must be reachable\&.
.sp
Once the ticket is administratively revoked, it is not managed by the booth cluster anymore\&. For the booth cluster to start managing the ticket again, it must be again granted to a site\&.
.sp
The grant operation, in case not all sites are reachable, may get delayed for the ticket expire time (and, if defined, the \fIacquire\-after\fR time)\&. The reason is that the other booth members may not know if the ticket is currently granted at the unreachable site\&.
.sp
This delay may be disabled with the \fI\-F\fR option\&. In that case, it is up to the administrator to make sure that the unreachable site is not holding the ticket\&.
.sp
When the ticket is managed by \fIbooth\fR, it is dangerous to modify it manually using either crm_ticket command or crm site ticket\&. Neither of these tools is aware of \fIbooth\fR and, consequently, \fIbooth\fR itself may not be aware of any ticket status changes\&. A notable exception is setting the ticket to standby which is typically done before a planned failover\&.
.SH "NOTES"
.sp
Tickets are not meant to be moved around quickly, the default \fIexpire\fR time is 600 seconds (10 minutes)\&.
.sp
\fIbooth\fR works with both IPv4 and IPv6 addresses\&.
.sp
\fIbooth\fR renews a ticket before it expires, to account for possible transmission delays\&. The renewal time, unless explicitly set, is set to half the \fIexpire\fR time\&.
.SH "HANDLERS"
.sp
Currently, there\(cqs only one external handler defined (see the \fIbefore\-acquire\-handler\fR configuration item above)\&.
.sp
The following environment variables are exported to the handler:
.PP
*\fIBOOTH_TICKET\fR
.RS 4
The ticket name, as given in the configuration file\&. (See
\fIticket\fR
item above\&.)
.RE
.PP
*\fIBOOTH_LOCAL\fR
.RS 4
The local site name, as defined in
\fIsite\fR\&.
.RE
.PP
*\fIBOOTH_CONF_PATH\fR
.RS 4
The path to the active configuration file\&.
.RE
.PP
*\fIBOOTH_CONF_NAME\fR
.RS 4
The configuration name, as used by the
\fI\-c\fR
commandline argument\&.
.RE
.PP
*\fIBOOTH_TICKET_EXPIRES\fR
.RS 4
When the ticket expires (in seconds since 1\&.1\&.1970), or
\fI0\fR\&.
.RE
.sp
The handler is invoked with positional arguments specified after it\&.
.SH "FILES"
.PP
\fB\fI/etc/booth/booth\&.conf\fR\fR
.RS 4
The default configuration file name\&. See also the
\fI\-c\fR
argument\&.
.RE
.PP
\fB\fI/etc/booth/authkey\fR\fR
.RS 4
There is no default, but this is a typical location for the shared secret (authentication key)\&.
.RE
.PP
\fB\fI/var/run/booth/\fR\fR
.RS 4
Directory that holds PID/lock files\&. See also the
\fIstatus\fR
command\&.
.RE
.SH "RAFT IMPLEMENTATION"
.sp
In essence, every ticket corresponds to a separate Raft cluster\&.
.sp
A ticket is granted to the Raft \fILeader\fR which then owns (or keeps) the ticket\&.
.SH "ARBITRATOR MANAGEMENT"
.sp
The booth daemon for an arbitrator which typically doesn\(cqt run the cluster stack, may be started through systemd or with \fI/etc/init\&.d/booth\-arbitrator\fR, depending on which init system the platform supports\&.
.sp
The SysV init script starts a booth arbitrator for every configuration file found in \fI/etc/booth\fR\&.
.sp
Platforms running systemd can enable and start every configuration separately using \fIsystemctl\fR:
.sp
.if n \{\
.RS 4
.\}
.nf
# systemctl enable booth@
# systemctl start booth@
.fi
.if n \{\
.RE
.\}
.sp
\fIsystemctl\fR requires the configuration name, even for the default name \fIbooth\fR\&.
.SH "EXIT STATUS"
.PP
\fB0\fR
.RS 4
Success\&. For the
\fIstatus\fR
command: Daemon running\&.
.RE
.PP
\fB1\fR (PCMK_OCF_UNKNOWN_ERROR)
.RS 4
General error code\&.
.RE
.PP
\fB7\fR (PCMK_OCF_NOT_RUNNING)
.RS 4
No daemon process for that configuration active\&.
.RE
.SH "BUGS"
.sp
Booth is tested regularly\&. See the README\-testing file for more information\&.
.sp
Please report any bugs either at GitHub: https://github\&.com/ClusterLabs/booth/issues
.sp
Or, if you prefer bugzilla, at openSUSE bugzilla (component "High Availability"): https://bugzilla\&.opensuse\&.org/enter_bug\&.cgi?product=openSUSE%20Factory
.SH "AUTHOR"
.sp
\fIboothd\fR was originally written (mostly) by Jiaju Zhang\&.
.sp
In 2013 and 2014 Philipp Marek took over maintainership\&.
.sp
Since April 2014 it has been mainly developed by Dejan Muhamedagic\&.
.sp
Many people contributed (see the AUTHORS file)\&.
.SH "RESOURCES"
.sp
GitHub: https://github\&.com/ClusterLabs/booth
.sp
Documentation: http://doc\&.opensuse\&.org/products/draft/SLE\-HA/SLE\-ha\-guide_sd_draft/cha\&.ha\&.geo\&.html
.SH "COPYING"
.sp
Copyright \(co 2011 Jiaju Zhang
.sp
Copyright \(co 2013\-2014 Philipp Marek
.sp
Copyright \(co 2014 Dejan Muhamedagic
.sp
Free use of this software is granted under the terms of the GNU General Public License (GPL)\&.