.\" Man page generated from reStructuredText. . .TH "CIRCUS" "1" "November 30, 2015" "0.12.1" "Circus" .SH NAME circus \- Circus Documentation . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. [image] .sp Circus is a Python program which can be used to monitor and control processes and sockets. .sp Circus can be driven via a command\-line interface, a web interface or programmatically through its python API. .sp To install it and try its features check out the examples, or read the rest of this page for a quick introduction. .SH RUNNING A CIRCUS DAEMON .sp Circus provides a command\-line script call \fBcircusd\fP that can be used to manage processes organized in one or more watchers\&. .sp Circus\(aq command\-line tool is configurable using an ini\-style configuration file. .sp Here\(aqs a very minimal example: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [watcher:program] cmd = python myprogram.py numprocesses = 5 [watcher:anotherprogram] cmd = another_program numprocesses = 2 .ft P .fi .UNINDENT .UNINDENT .sp The file is then passed to \fIcircusd\fP: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusd example.ini .ft P .fi .UNINDENT .UNINDENT .sp Besides processes, Circus can also bind sockets. Since every process managed by Circus is a child of the main Circus daemon, that means any program that\(aqs controlled by Circus can use those sockets. .sp Running a socket is as simple as adding a \fIsocket\fP section in the config file: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [socket:mysocket] host = localhost port = 8080 .ft P .fi .UNINDENT .UNINDENT .sp To learn more about sockets, see sockets\&. .sp To understand why it\(aqs a killer feature, read whycircussockets\&. .SH CONTROLLING CIRCUS .sp Circus provides two command\-line tools to manage your running daemon: .INDENT 0.0 .IP \(bu 2 \fIcircusctl\fP, a management console you can use to perform actions such as adding or removing workers .IP \(bu 2 \fIcircus\-top\fP, a top\-like console you can use to display the memory and cpu usage of your running Circus. .UNINDENT .sp To learn more about these, see cli .sp Circus also offers a web dashboard that can connect to a running Circus daemon and let you monitor and interact with it. .sp To learn more about this feature, see circushttpd .SS What now ? .sp If you are a developer and want to leverage Circus in your own project, write plugins or hooks, go to fordevs\&. .sp If you are an ops and want to manage your processes using Circus, go to forops\&. .SS Contributions and Feedback .sp More on contributing: contribs\&. .sp Useful Links: .INDENT 0.0 .IP \(bu 2 There\(aqs a mailing\-list for any feedback or question: \fI\%http://tech.groups.yahoo.com/group/circus\-dev/\fP .IP \(bu 2 The repository and issue tracker are on GitHub : \fI\%https://github.com/circus\-tent/circus\fP .IP \(bu 2 Join us on the IRC : Freenode, channel \fB#circus\-tent\fP .UNINDENT .SS Documentation index .SS Installing Circus .sp Circus is a Python package which is published on PyPI \- the Python Package Index. .sp The simplest way to install it is to use pip, a tool for installing and managing Python packages: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ pip install circus .ft P .fi .UNINDENT .UNINDENT .sp Or download the \fI\%archive on PyPI\fP, extract and install it manually with: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ python setup.py install .ft P .fi .UNINDENT .UNINDENT .sp If you want to try out Circus, see the examples\&. .sp If you are using debian or any debian based distribution, you also can use the ppa to install circus, it\(aqs at \fI\%https://launchpad.net/~roman\-imankulov/+archive/circus\fP .SS zc.buildout .sp We provide a \fI\%zc.buildout\fP configuration, you can use it by simply running the bootstrap script, then calling buildout: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ python bootstrap.py $ bin/buildout .ft P .fi .UNINDENT .UNINDENT .SS More on Requirements .sp Circus works with: .INDENT 0.0 .IP \(bu 2 Python 2.6, 2.7, 3.2 or 3.3 .IP \(bu 2 .INDENT 2.0 .TP .B zeromq >= 2.1.10 .INDENT 7.0 .IP \(bu 2 The version of zeromq supported is ultimately determined by what version of \fI\%pyzmq\fP is installed by pip during circus installation. .IP \(bu 2 Their current release supports 2.x (limited), 3.x, and 4.x ZeroMQ versions. .IP \(bu 2 \fBNote\fP: If you are using PyPy instead of CPython, make sure to read their installation docs as ZeroMQ version support is not the same on PyPy. .UNINDENT .UNINDENT .UNINDENT .sp When you install circus, the latest versions of the Python dependencies will be pulled out for you. .sp You can also install them manually using the pip\-requirements.txt file we provide: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ pip install \-r pip\-requirements.txt .ft P .fi .UNINDENT .UNINDENT .sp If you want to run the Web console you will need to install \fBcircus\-web\fP: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ pip install circus\-web .ft P .fi .UNINDENT .UNINDENT .SS Tutorial .SS Step\-by\-step tutorial .sp The \fI\%examples directory\fP in the Circus repository contains many examples to get you started, but here\(aqs a full tutorial that gives you an overview of the features. .sp We\(aqre going to supervise a WSGI application. .SS Installation .sp Circus is tested on Mac OS X and Linux with the latest Python 2.6, 2.7, 3.2 and 3.3. To run a full Circus, you will also need \fBlibzmq\fP, \fBlibevent\fP & \fBvirtualenv\fP\&. .sp On Debian\-based systems: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ sudo apt\-get install libzmq\-dev libevent\-dev python\-dev python\-virtualenv .ft P .fi .UNINDENT .UNINDENT .sp Create a virtualenv and install \fIcircus\fP, \fIcircus\-web\fP and \fIchaussette\fP in it .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ virtualenv /tmp/circus $ cd /tmp/circus $ bin/pip install circus $ bin/pip install circus\-web $ bin/pip install chaussette .ft P .fi .UNINDENT .UNINDENT .sp Once this is done, you\(aqll find a plethora of commands in the local bin dir. .SS Usage .sp \fIChaussette\fP comes with a default Hello world app, try to run it: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ bin/chaussette .ft P .fi .UNINDENT .UNINDENT .sp You should be able to visit \fI\%http://localhost:8080\fP and see \fIhello world\fP\&. .sp Stop Chaussette and add a circus.ini file in the directory containing: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [circus] statsd = 1 httpd = 1 [watcher:webapp] cmd = bin/chaussette \-\-fd $(circus.sockets.web) numprocesses = 3 use_sockets = True [socket:web] host = 127.0.0.1 port = 9999 .ft P .fi .UNINDENT .UNINDENT .sp This config file tells Circus to bind a socket on port \fI9999\fP and run 3 chaussettes workers against it. It also activates the Circus web dashboard and the statistics module. .sp Save it & run it using \fBcircusd\fP: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ bin/circusd \-\-daemon circus.ini .ft P .fi .UNINDENT .UNINDENT .sp Now visit \fI\%http://127.0.0.1:9999\fP, you should see the hello world app. The difference now is that the socket is managed by Circus and there are several web workers that are accepting connections against it. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 The load balancing is operated by the operating system so you\(aqre getting the same speed as any other pre\-fork web server like Apache or NGinx. Circus does not interfer with the data that goes through. .UNINDENT .UNINDENT .sp You can also visit \fI\%http://localhost:8080/\fP and enjoy the Circus web dashboard. .SS Interaction .sp Let\(aqs use the circusctl shell while the system is running: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ bin/circusctl circusctl 0.7.1 circusd\-stats: active circushttpd: active webapp: active (circusctl) .ft P .fi .UNINDENT .UNINDENT .sp You get into an interactive shell. Type \fBhelp\fP to get all commands: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C (circusctl) help Documented commands (type help ): ======================================== add get list numprocesses quit rm start stop decr globaloptions listen numwatchers reload set stats dstats incr listsockets options restart signal status Undocumented commands: ====================== EOF help .ft P .fi .UNINDENT .UNINDENT .sp Let\(aqs try basic things. Let\(aqs list the web workers processes and add a new one: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C (circusctl) list webapp 13712,13713,13714 (circusctl) incr webapp 4 (circusctl) list webapp 13712,13713,13714,13973 .ft P .fi .UNINDENT .UNINDENT .sp Congrats, you\(aqve interacted with your Circus! Get off the shell with Ctrl+D and now run circus\-top: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ bin/circus\-top .ft P .fi .UNINDENT .UNINDENT .sp This is a top\-like command to watch all your processes\(aq memory and CPU usage in real time. .sp Hit Ctrl+C and now let\(aqs quit Circus completely via circus\-ctl: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ bin/circusctl quit ok .ft P .fi .UNINDENT .UNINDENT .SS Next steps .sp You can plug your own WSGI application instead of Chaussette\(aqs hello world simply by pointing the application callable. .sp Chaussette also comes with many backends like Gevent or Meinheld. .sp Read \fI\%https://chaussette.readthedocs.org/\fP for all options. .SS Why should I use Circus instead of X ? .INDENT 0.0 .IP 1. 3 \fBCircus simplifies your web stack process management\fP .sp Circus knows how to manage processes \fIand\fP sockets, so you don\(aqt have to delegate web workers management to a WGSI server. .sp See whycircussockets .IP 2. 3 \fBCircus provides pub/sub and poll notifications via ZeroMQ\fP .UNINDENT .INDENT 0.0 .INDENT 3.5 Circus has a pub/sub channel you can subscribe to. This channel receives all events happening in Circus. For example, you can be notified when a process is flapping, or build a client that triggers a warning when some processes are eating all the CPU or RAM. .sp These events are sent via a ZeroMQ channel, which makes it different from the stdin stream Supervisord uses: .INDENT 0.0 .IP \(bu 2 Circus sends events in a fire\-and\-forget fashion, so there\(aqs no need to manually loop through \fIall\fP listeners and maintain their states. .IP \(bu 2 Subscribers can be located on a remote host. .UNINDENT .sp Circus also provides ways to get status updates via one\-time polls on a req/rep channel. This means you can get your information without having to subscribe to a stream. The cli command provided by Circus uses this channel. .sp See examples\&. .UNINDENT .UNINDENT .INDENT 0.0 .IP 3. 3 \fBCircus is (Python) developer friendly\fP .UNINDENT .INDENT 0.0 .INDENT 3.5 While Circus can be driven entirely by a config file and the \fIcircusctl\fP / \fIcircusd\fP commands, it is easy to reuse all or part of the system to build your own custom process watcher in Python. .sp Every layer of the system is isolated, so you can reuse independently: .INDENT 0.0 .IP \(bu 2 the process wrapper (\fBProcess\fP) .IP \(bu 2 the processes manager (\fBWatcher\fP) .IP \(bu 2 the global manager that runs several processes managers (\fBArbiter\fP) .IP \(bu 2 and so on… .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .IP 4. 3 \fBCircus scales\fP .UNINDENT .INDENT 0.0 .INDENT 3.5 One of the use cases of Circus is to manage thousands of processes without adding overhead \-\- we\(aqre dedicated to focusing on this. .UNINDENT .UNINDENT .SS Coming from Supervisor .sp Supervisor is a very popular solution in the Python world and we\(aqre often asked how Circus compares with it. .sp If you are coming from \fI\%Supervisor\fP, this page tries to give an overview of how the tools differ. .SS Differences overview .sp Supervisor & Circus have the same goals \- they both manage processes and provide a command\-line script — respectively \fBsupervisord\fP and \fBcircusd\fP — that reads a configuration file, forks new processes and keep them alive. .sp Circus has an extra feature: the ability to bind sockets and let the processes it manages use them. This "pre\-fork" model is used by many web servers out there, like \fI\%Apache\fP or \fI\%Unicorn\fP\&. Having this option in Circus can simplify a web app stack: all processes and sockets are managed by a single tool. .sp Both projects provide a way to control a running daemon via another script. respectively \fBsupervisorctl\fP and \fBcircusctl\fP\&. They also both have events and a way to subscribe to them. The main difference is the underlying technology: Supervisor uses XML\-RPC for interacting with the daemon, while Circus uses ZeroMQ. .sp Circus & Supervisor both have a web interface to display what\(aqs going on. Circus\(aq is more advanced because you can follow in real time what\(aqs going on and interact with the daemon. It uses web sockets and is developed in a separate project (\fI\%circus\-web\fP\&.) .sp There are many other subtle differences in the core design, we might list here one day… In the meantime, you can learn more about circus internals in design\&. .SS Configuration .sp Both systems use an ini\-like file as a configuration. .INDENT 0.0 .IP \(bu 2 \fI\%Supervisor documentation\fP .IP \(bu 2 \fI\%Circus documentation\fP .UNINDENT .sp Here\(aqs a small example of running an application with Supervisor. In this case, the application will be started and restarted in case it crashes .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [program:example] command=npm start directory=/home/www/my\-server/ user=www\-data autostart=true autorestart=true redirect_stderr=True .ft P .fi .UNINDENT .UNINDENT .sp In Circus, the same configuration is done by: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [watcher:example] cmd=npm start working_dir=/home/www/my\-server/ user=www\-data stderr_stream.class=StdoutStream .ft P .fi .UNINDENT .UNINDENT .sp Notice that the stderr redirection is slightly different in Circus. The tool does not have a \fBtail\fP feature like in Supervisor, but will let you hook any piece of code to deal with the incoming stream. You can create your own stream hook (as a Class) and do whatever you want with the incoming stream. Circus provides some built\-in stream classes like \fBStdoutStream\fP, \fBFileStream\fP, \fBWatchedFileStream\fP, or \fBTimedRotatingFileStream\fP\&. .SS Circus for Ops .sp \fBWARNING:\fP .INDENT 0.0 .INDENT 3.5 By default, Circus doesn\(aqt secure its messages when sending information through ZeroMQ. Before running Circus in a production environment, make sure to read the Security page. .UNINDENT .UNINDENT .sp The first step to manage a Circus daemon is to write its configuration file. See configuration\&. If you are deploying a web stack, have a look at sockets\&. .sp Circus can be deployed using Python 2.6, 2.7, 3.2 or 3.3 \- most deployments out there are done in 2.7. To learn how to deploy Circus, check out deployment\&. .sp To manage a Circus daemon, you should get familiar with the list of commands you can use in \fBcircusctl\fP\&. Notice that you can have the same help online when you run \fBcircusctl\fP as a shell. .sp We also provide \fBcircus\-top\fP, see cli and a nice web dashboard. see circushttpd\&. .sp Last, to get the most out of Circus, make sure to check out how to use plugins and hooks. See plugins and hooks\&. .SS Ops documentation index .SS Configuration .sp Circus can be configured using an ini\-style configuration file. .sp Example: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [circus] check_delay = 5 endpoint = tcp://127.0.0.1:5555 pubsub_endpoint = tcp://127.0.0.1:5556 include = \e*.more.config.ini umask = 002 [watcher:myprogram] cmd = python args = \-u myprogram.py $(circus.wid) $(CIRCUS.ENV.VAR) warmup_delay = 0 numprocesses = 5 # hook hooks.before_start = my.hooks.control_redis # will push in test.log the stream every 300 ms stdout_stream.class = FileStream stdout_stream.filename = test.log # optionally rotate the log file when it reaches 1 gb # and save 5 copied of rotated files stdout_stream.max_bytes = 1073741824 stdout_stream.backup_count = 5 [env:myprogram] PATH = $PATH:/bin CAKE = lie [plugin:statsd] use = circus.plugins.statsd.StatsdEmitter host = localhost port = 8125 sample_rate = 1.0 application_name = example [socket:web] host = localhost port = 8080 .ft P .fi .UNINDENT .UNINDENT .SS circus \- single section .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B \fBendpoint\fP The ZMQ socket used to manage Circus via \fBcircusctl\fP\&. (default: \fItcp://127.0.0.1:5555\fP) .TP .B \fBendpoint_owner\fP If set to a system username and the endpoint is an ipc socket like \fIipc://var/run/circusd.sock\fP, then ownership of the socket file will be changed to that user at startup. For more details, see security\&. (default: None) .TP .B \fBpubsub_endpoint\fP The ZMQ PUB/SUB socket receiving publications of events. (default: \fItcp://127.0.0.1:5556\fP) .TP .B \fBpapa_endpoint\fP If using papa, you can specify the endpoint, such as \fIipc://var/run/circusd.sock\fP\&. (default: \fItcp://127.0.0.1:20202\fP) .TP .B \fBstatsd\fP If set to True, Circus runs the circusd\-stats daemon. (default: False) .TP .B \fBstats_endpoint\fP The ZMQ PUB/SUB socket receiving publications of stats. (default: \fItcp://127.0.0.1:5557\fP) .TP .B \fBstatsd_close_outputs\fP If True sends the circusd\-stats stdout/stderr to /dev/null. (default: False) .TP .B \fBcheck_delay\fP The polling interval in seconds for the ZMQ socket. (default: 5) .TP .B \fBinclude\fP List of config files to include. You can use wildcards (\fI*\fP) to include particular schemes for your files. The paths are absolute or relative to the config file. (default: None) .TP .B \fBinclude_dir\fP List of config directories. All files matching \fI*.ini\fP under each directory will be included. The paths are absolute or relative to the config file. (default: None) .TP .B \fBstream_backend\fP Defines the type of backend to use for the streaming. Possible values are \fBthread\fP or \fBgevent\fP\&. (default: thread) .TP .B \fBwarmup_delay\fP The interval in seconds between two watchers start. Must be an int. (default: 0) .TP .B \fBhttpd\fP If set to True, Circus runs the circushttpd daemon. (default: False) .TP .B \fBhttpd_host\fP The host ran by the circushttpd daemon. (default: localhost) .TP .B \fBhttpd_port\fP The port ran by the circushttpd daemon. (default: 8080) .TP .B \fBhttpd_close_outputs\fP If True, sends the circushttpd stdout/stderr to /dev/null. (default: False) .TP .B \fBdebug\fP If set to True, all Circus stout/stderr daemons are redirected to circusd stdout/stderr (default: False) .TP .B \fBdebug_gc\fP If set to True, circusd outputs additional log info from the garbage collector. This can be useful in tracking down memory leaks. (default: False) .TP .B \fBpidfile\fP The file that must be used to keep the daemon pid. .TP .B \fBumask\fP Value for umask. If not set, circusd will not attempt to modify umask. .TP .B \fBloglevel\fP The loglevel that we want to see (default: INFO) .TP .B \fBlogoutput\fP The logoutput file where we want to log (default: \fB\-\fP to log on stdout). You can log to a remote syslog by using the following syntax: \fBsyslog://host:port?facility\fP where host is your syslog server, port is optional and facility is the syslog facility to use. If you wish to log to a local syslog you can use \fBsyslog:///path/to/syslog/socket?facility\fP instead. .TP .B \fBloggerconfig\fP A path to an INI, JSON or YAML file to configure standard Python logging for the Arbiter. The special value "default" uses the builtin logging configuration based on the optional loglevel and logoutput options. .sp \fBExample YAML Configuration File\fP .UNINDENT .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C version: 1 disable_existing_loggers: false formatters: simple: format: \(aq%(asctime)s \- %(name)s \- [%(levelname)s] %(message)s\(aq handlers: logfile: class: logging.FileHandler filename: logoutput.txt level: DEBUG formatter: simple loggers: circus: level: DEBUG handlers: [logfile] propagate: no root: level: DEBUG handlers: [logfile] .ft P .fi .UNINDENT .UNINDENT .UNINDENT .UNINDENT .SS watcher:NAME \- as many sections as you want .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B \fBNAME\fP The name of the watcher. This name is used in \fBcircusctl\fP .TP .B \fBcmd\fP The executable program to run. .TP .B \fBargs\fP Command\-line arguments to pass to the program. You can use the python format syntax here to build the parameters. Environment variables are available, as well as the worker id and the environment variables that you passed, if any, with the "env" parameter. See \fI\%Formatting the commands and arguments with dynamic variables\fP for more information on this. .TP .B \fBshell\fP If True, the processes are run in the shell (default: False) .TP .B \fBshell_args\fP Command\-line arguments to pass to the shell command when \fBshell\fP is True. Works only for *nix system (default: None) .TP .B \fBworking_dir\fP The working dir for the processes (default: None) .TP .B \fBuid\fP The user id or name the command should run with. (The current uid is the default). .TP .B \fBgid\fP The group id or name the command should run with. (The current gid is the default). .TP .B \fBcopy_env\fP If set to true, the local environment variables will be copied and passed to the workers when spawning them. (Default: False) .TP .B \fBcopy_path\fP If set to true, \fBsys.path\fP is passed in the subprocess environ using \fIPYTHONPATH\fP\&. \fBcopy_env\fP has to be true. (Default: False) .TP .B \fBwarmup_delay\fP The delay (in seconds) between running processes. .TP .B \fBautostart\fP If set to false, the watcher will not be started automatically when the arbiter starts. The watcher can be started explicitly (example: \fIcircusctrl start myprogram\fP). (Default: True) .TP .B \fBnumprocesses\fP The number of processes to run for this watcher. .TP .B \fBrlimit_LIMIT\fP Set resource limit LIMIT for the watched processes. The config name should match the RLIMIT_* constants (not case sensitive) listed in the \fI\%Python resource module reference\fP\&. For example, the config line \(aqrlimit_nofile = 500\(aq sets the maximum number of open files to 500. To set a limit value to RLIM_INFINITY, do not set a value, like this config line: \(aqrlimit_nofile = \(aq. .TP .B \fBstderr_stream.class\fP A fully qualified Python class name that will be instanciated, and will receive the \fBstderr\fP stream of all processes in its \fB__call__()\fP method. .sp Circus provides some stream classes you can use without prefix: .INDENT 7.0 .IP \(bu 2 \fBFileStream\fP: writes in a file and can do automatic log rotation .IP \(bu 2 \fBWatchedFileStream\fP: writes in a file and relies on external log rotation .IP \(bu 2 \fBTimedRotatingFileStream\fP: writes in a file and can do rotate at certain timed intervals. .IP \(bu 2 \fBQueueStream\fP: write in a memory Queue .IP \(bu 2 \fBStdoutStream\fP: writes in the stdout .IP \(bu 2 \fBFancyStdoutStream\fP: writes colored output with time prefixes in the stdout .UNINDENT .TP .B \fBstderr_stream.*\fP All options starting with \fIstderr_stream.\fP other than \fIclass\fP will be passed the constructor when creating an instance of the class defined in \fBstderr_stream.class\fP\&. .TP .B \fBstdout_stream.class\fP A fully qualified Python class name that will be instanciated, and will receive the \fBstdout\fP stream of all processes in its \fB__call__()\fP method. .sp Circus provides some stream classes you can use without prefix: .INDENT 7.0 .IP \(bu 2 \fBFileStream\fP: writes in a file and can do automatic log rotation .IP \(bu 2 \fBWatchedFileStream\fP: writes in a file and relies on external log rotation .IP \(bu 2 \fBTimedRotatingFileStream\fP: writes in a file and can do rotate at certain timed intervals. .IP \(bu 2 \fBQueueStream\fP: write in a memory Queue .IP \(bu 2 \fBStdoutStream\fP: writes in the stdout .IP \(bu 2 \fBFancyStdoutStream\fP: writes colored output with time prefixes in the stdout .UNINDENT .TP .B \fBstdout_stream.*\fP All options starting with \fIstdout_stream.\fP other than \fIclass\fP will be passed the constructor when creating an instance of the class defined in \fBstdout_stream.class\fP\&. .TP .B \fBclose_child_stdout\fP If set to True, the stdout stream of each process will be sent to /dev/null after the fork. Defaults to False. .TP .B \fBclose_child_stderr\fP If set to True, the stderr stream of each process will be sent to /dev/null after the fork. Defaults to False. .TP .B \fBsend_hup\fP If True, a process reload will be done by sending the SIGHUP signal. Defaults to False. .TP .B \fBstop_signal\fP The signal to send when stopping the process. Can be specified as a number or a signal name. Signal names are case\-insensitive and can include \(aqSIG\(aq or not. So valid examples include \fIquit\fP, \fIINT\fP, \fISIGTERM\fP and \fI3\fP\&. Defaults to SIGTERM. .TP .B \fBstop_children\fP When sending the \fIstop_signal\fP, send it to the children as well. Defaults to False. .TP .B \fBmax_retry\fP The number of times we attempt to start a process, before we abandon and stop the whole watcher. Defaults to 5. Set to \-1 to disable max_retry and retry indefinitely. .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B \fBgraceful_timeout\fP The number of seconds to wait for a process to terminate gracefully before killing it. .sp When stopping a process, we first send it a \fIstop_signal\fP\&. A worker may catch this signal to perform clean up operations before exiting. If the worker is still active after graceful_timeout seconds, we send it a SIGKILL signal. It is not possible to catch SIGKILL signals so the worker will stop. .sp Defaults to 30s. .TP .B \fBpriority\fP Integer that defines a priority for the watcher. When the Arbiter do some operations on all watchers, it will sort them with this field, from the bigger number to the smallest. Defaults to 0. .TP .B \fBsingleton\fP If set to True, this watcher will have at the most one process. Defaults to False. .TP .B \fBuse_sockets\fP If set to True, this watcher will be able to access defined sockets via their file descriptors. If False, all parent fds are closed when the child process is forked. Defaults to False. .TP .B \fBmax_age\fP If set then the process will be restarted sometime after max_age seconds. This is useful when processes deal with pool of connectors: restarting processes improves the load balancing. Defaults to being disabled. .TP .B \fBmax_age_variance\fP If max_age is set then the process will live between max_age and max_age + random(0, max_age_variance) seconds. This avoids restarting all processes for a watcher at once. Defaults to 30 seconds. .TP .B \fBon_demand\fP If set to True, the processes will be started only after the first connection to one of the configured sockets (see below). If a restart is needed, it will be only triggered at the next socket event. .TP .B \fBhooks.*\fP Available hooks: \fBbefore_start\fP, \fBafter_start\fP, \fBbefore_spawn\fP, \fBafter_spawn\fP, \fBbefore_stop\fP, \fBafter_stop\fP, \fBbefore_signal\fP, \fBafter_signal\fP, \fBextended_stats\fP .sp Define callback functions that hook into the watcher startup/shutdown process. .sp If the hook returns \fBFalse\fP and if the hook is one of \fBbefore_start\fP, \fBbefore_spawn\fP, \fBafter_start\fP or \fBafter_spawn\fP, the startup will be aborted. .sp If the hook is \fBbefore_signal\fP and returns \fBFalse\fP, then the corresponding signal will not be sent (except SIGKILL which is always sent) .sp Notice that a hook that fails during the stopping process will not abort it. .sp The callback definition can be followed by a boolean flag separated by a comma. When the flag is set to \fBtrue\fP, any error occuring in the hook will be ignored. If set to \fBfalse\fP (the default), the hook will return \fBFalse\fP\&. .sp More on hooks\&. .TP .B \fBvirtualenv\fP When provided, points to the root of a Virtualenv directory. The watcher will scan the local \fBsite\-packages\fP and loads its content into the execution environment. Must be used with \fBcopy_env\fP set to True. Defaults to None. .TP .B \fBvirtualenv_py_ver\fP Specifies the python version of the virtualenv (e.g "3.3"). It\(aqs usefull if circus run with another python version (e.g "2.7") The watcher will scan the local \fBsite\-packages\fP of the specified python version and load its content into the execution environment. Must be used with \fBvirtualenv\fP\&. Defaults to None. .TP .B \fBrespawn\fP If set to False, the processes handled by a watcher will not be respawned automatically. The processes can be manually respawned with the \fIstart\fP command. (default: True) .TP .B \fBuse_papa\fP Set to true to use the papa\&. .UNINDENT .UNINDENT .UNINDENT .SS socket:NAME \- as many sections as you want .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B \fBhost\fP The host of the socket. Defaults to \(aqlocalhost\(aq .TP .B \fBport\fP The port. Defaults to 8080. .TP .B \fBfamily\fP The socket family. Can be \(aqAF_UNIX\(aq, \(aqAF_INET\(aq or \(aqAF_INET6\(aq. Defaults to \(aqAF_INET\(aq. .TP .B \fBtype\fP The socket type. Can be \(aqSOCK_STREAM\(aq, \(aqSOCK_DGRAM\(aq, \(aqSOCK_RAW\(aq, \(aqSOCK_RDM\(aq or \(aqSOCK_SEQPACKET\(aq. Defaults to \(aqSOCK_STREAM\(aq. .TP .B \fBinterface\fP When provided a network interface name like \(aqeth0\(aq, binds the socket to that particular device so that only packets received from that particular interface are processed by the socket. This can be used for example to limit which device to bind when binding on IN_ADDR_ANY (0.0.0.0) or IN_ADDR_BROADCAST (255.255.255.255). Note that this only works for some socket types, particularly AF_INET sockets. .TP .B \fBpath\fP When provided a path to a file that will be used as a unix socket file. If a path is provided, \fBfamily\fP is forced to AF_UNIX and \fBhost\fP and \fBport\fP are ignored. .TP .B \fBumask\fP When provided, sets the umask that will be used to create an AF_UNIX socket. For example, \fIumask=000\fP will produce a socket with permission \fI777\fP\&. .TP .B \fBreplace\fP When creating Unix sockets (\(aqAF_UNIX\(aq), an existing file may indicate a problem so the default is to fail. Specify \fITrue\fP to simply remove the old file if you are sure that the socket is managed only by Circus. .TP .B \fBso_reuseport\fP If set to True and SO_REUSEPORT is available on target platform, circus will create and bind new SO_REUSEPORT socket(s) for every worker it starts which is a user of this socket(s). .TP .B \fBuse_papa\fP Set to true to use the papa\&. .UNINDENT .UNINDENT .UNINDENT .sp Once a socket is created, the \fI${circus.sockets.NAME}\fP string can be used in the command (\fIcmd\fP or \fIargs\fP) of a watcher. Circus will replace it by the FD value. The watcher must also have \fIuse_sockets\fP set to \fITrue\fP otherwise the socket will have been closed and you will get errors when the watcher tries to use it. .sp Example: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [watcher:webworker] cmd = chaussette \-\-fd $(circus.sockets.webapp) chaussette.util.bench_app use_sockets = True [socket:webapp] host = 127.0.0.1 port = 8888 .ft P .fi .UNINDENT .UNINDENT .SS plugin:NAME \- as many sections as you want .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B \fBuse\fP The fully qualified name that points to the plugin class. .TP .B \fBanything else\fP Every other key found in the section is passed to the plugin constructor in the \fBconfig\fP mapping. .sp You can use all the watcher options, since a plugin is started like a watcher. .UNINDENT .UNINDENT .UNINDENT .sp Circus comes with a few pre\-shipped plugins but you can also extend them easily by developing your own\&. .SS env or env[:WATCHERS] \- as many sections as you want .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B \fBanything\fP The name of an environment variable to assign value to. bash style environment substitutions are supported. for example, append /bin to \fIPATH\fP \(aqPATH = $PATH:/bin\(aq .UNINDENT .UNINDENT .UNINDENT .sp Section responsible for delivering environment variable to run processes. .sp Example: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [watcher:worker1] cmd = ping 127.0.0.1 [watcher:worker2] cmd = ping 127.0.0.1 [env] CAKE = lie .ft P .fi .UNINDENT .UNINDENT .sp The variable \fICAKE\fP will propagated to all watchers defined in config file. .sp WATCHERS can be a comma separated list of watcher sections to apply this environment to. if multiple env sections match a watcher, they will be combine in the order they appear in the configuration file. later entries will take precedence. .sp Example: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [watcher:worker1] cmd = ping 127.0.0.1 [watcher:worker2] cmd = ping 127.0.0.1 [env:worker1,worker2] PATH = /bin [env:worker1] PATH = $PATH [env:worker2] CAKE = lie .ft P .fi .UNINDENT .UNINDENT .sp \fIworker1\fP will be run with PATH = $PATH (expanded from the environment circusd was run in) \fIworker2\fP will be run with PATH = /bin and CAKE = lie .sp It\(aqs possible to use wildcards as well. .sp Example: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [watcher:worker1] cmd = ping 127.0.0.1 [watcher:worker2] cmd = ping 127.0.0.1 [env:worker*] PATH = /bin .ft P .fi .UNINDENT .UNINDENT .sp Both \fIworker1\fP and \fIworker2\fP will be run with PATH = /bin .SS Using environment variables .sp When writing your configuration file, you can use environment variables defined in the \fIenv\fP section or in \fIos.environ\fP itself. .sp You just have to use the \fIcircus.env.\fP prefix. .sp Example: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [watcher:worker1] cmd = $(circus.env.shell) [watcher:worker2] baz = $(circus.env.user) bar = $(circus.env.yeah) sup = $(circus.env.oh) [socket:socket1] port = $(circus.env.port) [plugin:plugin1] use = some.path parameter1 = $(circus.env.plugin_param) [env] yeah = boo [env:worker2] oh = ok .ft P .fi .UNINDENT .UNINDENT .sp If a variable is defined in several places, the most specialized value has precedence: a variable defined in \fIenv:XXX\fP will override a variable defined in \fIenv\fP, which will override a variable defined in \fIos.environ\fP\&. .sp environment substitutions can be used in any section of the configuration in any section variable. .SS Formatting the commands and arguments with dynamic variables .sp As you may have seen, it is possible to pass some information that are computed dynamically when running the processes. Among other things, you can get the worker id (WID) and all the options that are passed to the \fBProcess\fP\&. Additionally, it is possible to access the options passed to the \fBWatcher\fP which instanciated the process. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 The worker id is different from the process id. It\(aqs a unique value, starting at 1, which is only unique for the watcher. .UNINDENT .UNINDENT .sp For instance, if you want to access some variables that are contained in the environment, you would need to do it with a setting like this: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C cmd = "make\-me\-a\-coffee \-\-sugar $(CIRCUS.ENV.SUGAR_AMOUNT)" .ft P .fi .UNINDENT .UNINDENT .sp This works with both \fIcmd\fP and \fIargs\fP\&. .sp \fBImportant\fP: .INDENT 0.0 .IP \(bu 2 All variables are prefixed with \fIcircus.\fP .IP \(bu 2 The replacement is case insensitive. .UNINDENT .SS Stream configuration .sp Simple stream class like \fIQueueStream\fP and \fIStdoutStream\fP don\(aqt have specific attributes but some other stream class may have some: .SS FileStream .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B \fBfilename\fP The file path where log will be written. .TP .B \fBtime_format\fP The strftime format that will be used to prefix each time with a timestamp. By default they will be not prefixed. .sp i.e: %Y\-%m\-%d %H:%M:%S .TP .B \fBmax_bytes\fP The max size of the log file before a new file is started. If not provided, the file is not rolled over. .TP .B \fBbackup_count\fP The number of log files that will be kept By default backup_count is null. .UNINDENT .UNINDENT .UNINDENT .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Rollover occurs whenever the current log file is nearly max_bytes in length. If backup_count is >= 1, the system will successively create new files with the same pathname as the base file, but with extensions ".1", ".2" etc. appended to it. For example, with a backup_count of 5 and a base file name of "app.log", you would get "app.log", "app.log.1", "app.log.2", ... through to "app.log.5". The file being written to is always "app.log" \- when it gets filled up, it is closed and renamed to "app.log.1", and if files "app.log.1", "app.log.2" etc. exist, then they are renamed to "app.log.2", "app.log.3" etc. respectively. .UNINDENT .UNINDENT .sp Example: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [watcher:myprogram] cmd = python \-m myapp.server stdout_stream.class = FileStream stdout_stream.filename = test.log stdout_stream.time_format = %Y\-%m\-%d %H:%M:%S stdout_stream.max_bytes = 1073741824 stdout_stream.backup_count = 5 .ft P .fi .UNINDENT .UNINDENT .SS WatchedFileStream .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B \fBfilename\fP The file path where log will be written. .TP .B \fBtime_format\fP The strftime format that will be used to prefix each time with a timestamp. By default they will be not prefixed. .sp i.e: %Y\-%m\-%d %H:%M:%S .UNINDENT .UNINDENT .UNINDENT .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 WatchedFileStream relies on an external log rotation tool to ensure that log files don\(aqt become too big. The output file will be monitored and if it is ever deleted or moved by the external log rotation tool, then the output file handle will be automatically reloaded. .UNINDENT .UNINDENT .sp Example: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [watcher:myprogram] cmd = python \-m myapp.server stdout_stream.class = WatchedFileStream stdout_stream.filename = test.log stdout_stream.time_format = %Y\-%m\-%d %H:%M:%S .ft P .fi .UNINDENT .UNINDENT .SS TimedRotatingFileStream .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B \fBfilename\fP The file path where log will be written. .TP .B \fBbackup_count\fP The number of log files that will be kept By default backup_count is null. .TP .B \fBtime_format\fP The strftime format that will be used to prefix each time with a timestamp. By default they will be not prefixed. .sp i.e: %Y\-%m\-%d %H:%M:%S .TP .B \fBrotate_when\fP The type of interval. The list of possible values is below. Note that they are not case sensitive. .TS center; |l|l|. _ T{ Value T} T{ Type of interval T} _ T{ \(aqS\(aq T} T{ Seconds T} _ T{ \(aqM\(aq T} T{ Minutes T} _ T{ \(aqH\(aq T} T{ Hours T} _ T{ \(aqD\(aq T} T{ Days T} _ T{ \(aqW0\(aq\-\(aqW6\(aq T} T{ Weekday (0=Monday) T} _ T{ \(aqmidnight\(aq T} T{ Roll over at midnight T} _ .TE .TP .B \fBrotate_interval\fP The rollover interval. .UNINDENT .UNINDENT .UNINDENT .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 TimedRotatingFileStream rotates logfiles at certain timed intervals. Rollover interval is determined by a combination of rotate_when and rotate_interval. .UNINDENT .UNINDENT .sp Example: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [watcher:myprogram] cmd = python \-m myapp.server stdout_stream.class = TimedRotatingFileStream stdout_stream.filename = test.log stdout_stream.time_format = %Y\-%m\-%d %H:%M:%S stdout_stream.utc = True stdout_stream.rotate_when = H stdout_stream.rotate_interval = 1 .ft P .fi .UNINDENT .UNINDENT .SS FancyStdoutStream .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B \fBcolor\fP .INDENT 7.0 .TP .B The name of an ascii color: .INDENT 7.0 .IP \(bu 2 red .IP \(bu 2 green .IP \(bu 2 yellow .IP \(bu 2 blue .IP \(bu 2 magenta .IP \(bu 2 cyan .IP \(bu 2 white .UNINDENT .UNINDENT .TP .B \fBtime_format\fP The strftime format that each line will be prefixed with. .sp Default to: %Y\-%m\-%d %H:%M:%S .UNINDENT .UNINDENT .UNINDENT .sp Example: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [watcher:myprogram] cmd = python \-m myapp.server stdout_stream.class = FancyStdoutStream stdout_stream.color = green stdout_stream.time_format = %Y/%m/%d | %H:%M:%S .ft P .fi .UNINDENT .UNINDENT .SS Commands .sp At the epicenter of circus lives the command systems. \fIcircusctl\fP is just a zeromq client, and if needed you can drive programmaticaly the Circus system by writing your own zmq client. .sp All messages are JSON mappings. .sp For each command below, we provide a usage example with circusctl but also the input / output zmq messages. .SS circus\-ctl commands .INDENT 0.0 .IP \(bu 2 \fBadd\fP: \fBcommands/add\fP .IP \(bu 2 \fBdecr\fP: \fBcommands/decr\fP .IP \(bu 2 \fBdstats\fP: \fBcommands/dstats\fP .IP \(bu 2 \fBget\fP: \fBcommands/get\fP .IP \(bu 2 \fBglobaloptions\fP: \fBcommands/globaloptions\fP .IP \(bu 2 \fBincr\fP: \fBcommands/incr\fP .IP \(bu 2 \fBipython\fP: \fBcommands/ipython\fP .IP \(bu 2 \fBlist\fP: \fBcommands/list\fP .IP \(bu 2 \fBlisten\fP: \fBcommands/listen\fP .IP \(bu 2 \fBlistsockets\fP: \fBcommands/listsockets\fP .IP \(bu 2 \fBnumprocesses\fP: \fBcommands/numprocesses\fP .IP \(bu 2 \fBnumwatchers\fP: \fBcommands/numwatchers\fP .IP \(bu 2 \fBoptions\fP: \fBcommands/options\fP .IP \(bu 2 \fBquit\fP: \fBcommands/quit\fP .IP \(bu 2 \fBreload\fP: \fBcommands/reload\fP .IP \(bu 2 \fBreloadconfig\fP: \fBcommands/reloadconfig\fP .IP \(bu 2 \fBrestart\fP: \fBcommands/restart\fP .IP \(bu 2 \fBrm\fP: \fBcommands/rm\fP .IP \(bu 2 \fBset\fP: \fBcommands/set\fP .IP \(bu 2 \fBsignal\fP: \fBcommands/signal\fP .IP \(bu 2 \fBstart\fP: \fBcommands/start\fP .IP \(bu 2 \fBstats\fP: \fBcommands/stats\fP .IP \(bu 2 \fBstatus\fP: \fBcommands/status\fP .IP \(bu 2 \fBstop\fP: \fBcommands/stop\fP .UNINDENT .SS Add a watcher .sp This command add a watcher dynamically to a arbiter. .SS ZMQ Message .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "add", "properties": { "cmd": "/path/to/commandline \-\-option" "name": "nameofwatcher" "args": [], "options": {}, "start": false } } .ft P .fi .UNINDENT .UNINDENT .sp A message contains 2 properties: .INDENT 0.0 .IP \(bu 2 cmd: Full command line to execute in a process .IP \(bu 2 args: array, arguments passed to the command (optional) .IP \(bu 2 name: name of watcher .IP \(bu 2 options: options of a watcher .IP \(bu 2 start: start the watcher after the creation .UNINDENT .sp The response return a status "ok". .SS Command line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl add [\-\-start] .ft P .fi .UNINDENT .UNINDENT .SS Options .INDENT 0.0 .IP \(bu 2 : name of the watcher to create .IP \(bu 2 : full command line to execute in a process .IP \(bu 2 \-\-start: start the watcher immediately .UNINDENT .SS Decrement the number of processes in a watcher .sp This comment decrement the number of processes in a watcher by , 1 being the default. .SS ZMQ Message .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "decr", "propeties": { "name": "" "nb": "waiting": False } } .ft P .fi .UNINDENT .UNINDENT .sp The response return the number of processes in the \(aqnumprocesses\(ga property: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "status": "ok", "numprocesses": , "time", "timestamp" } .ft P .fi .UNINDENT .UNINDENT .SS Command line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl decr [] [\-\-waiting] .ft P .fi .UNINDENT .UNINDENT .SS Options .INDENT 0.0 .IP \(bu 2 : name of the watcher .IP \(bu 2 : the number of processes to remove. .UNINDENT .SS Get circusd stats .sp You can get at any time some statistics about circusd with the dstat command. .SS ZMQ Message .sp To get the circusd stats, simply run: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "dstats" } .ft P .fi .UNINDENT .UNINDENT .sp The response returns a mapping the property "infos" containing some process informations: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "info": { "children": [], "cmdline": "python", "cpu": 0.1, "ctime": "0:00.41", "mem": 0.1, "mem_info1": "3M", "mem_info2": "2G", "nice": 0, "pid": 47864, "username": "root" }, "status": "ok", "time": 1332265655.897085 } .ft P .fi .UNINDENT .UNINDENT .SS Command Line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl dstats .ft P .fi .UNINDENT .UNINDENT .SS Get the value of specific watcher options .sp This command can be used to query the current value of one or more watcher options. .SS ZMQ Message .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "get", "properties": { "keys": ["key1, "key2"] "name": "nameofwatcher" } } .ft P .fi .UNINDENT .UNINDENT .sp A request message contains two properties: .INDENT 0.0 .IP \(bu 2 keys: list, The option keys for which you want to get the values .IP \(bu 2 name: name of watcher .UNINDENT .sp The response object has a property \fBoptions\fP which is a dictionary of option names and values. .sp eg: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "status": "ok", "options": { "graceful_timeout": 300, "send_hup": True, }, time\(aq: 1332202594.754644 } .ft P .fi .UNINDENT .UNINDENT .SS Command line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl get .ft P .fi .UNINDENT .UNINDENT .SS Get the arbiter options .sp This command return the arbiter options .SS ZMQ Message .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "globaloptions", "properties": { "key1": "val1", .. } } .ft P .fi .UNINDENT .UNINDENT .sp A message contains 2 properties: .INDENT 0.0 .IP \(bu 2 keys: list, The option keys for which you want to get the values .UNINDENT .sp The response return an object with a property "options" containing the list of key/value returned by circus. .sp eg: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "status": "ok", "options": { "check_delay": 1, ... }, time\(aq: 1332202594.754644 } .ft P .fi .UNINDENT .UNINDENT .SS Command line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl globaloptions .ft P .fi .UNINDENT .UNINDENT .SS Options .sp Options Keys are: .INDENT 0.0 .IP \(bu 2 endpoint: the controller ZMQ endpoint .IP \(bu 2 pubsub_endpoint: the pubsub endpoint .IP \(bu 2 check_delay: the delay between two controller points .IP \(bu 2 multicast_endpoint: the multicast endpoint for circusd cluster auto\-discovery .UNINDENT .SS Increment the number of processes in a watcher .sp This comment increment the number of processes in a watcher by , 1 being the default .SS ZMQ Message .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "incr", "properties": { "name": "", "nb": , "waiting": False } } .ft P .fi .UNINDENT .UNINDENT .sp The response return the number of processes in the \(aqnumprocesses\(ga property: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "status": "ok", "numprocesses": , "time", "timestamp" } .ft P .fi .UNINDENT .UNINDENT .SS Command line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl incr [] [\-\-waiting] .ft P .fi .UNINDENT .UNINDENT .SS Options .INDENT 0.0 .IP \(bu 2 : name of the watcher. .IP \(bu 2 : the number of processes to add. .UNINDENT .SS Create shell into circusd process .sp This command is only useful if you have the ipython package installed. .SS Command Line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl ipython .ft P .fi .UNINDENT .UNINDENT .SS Get list of watchers or processes in a watcher .SS ZMQ Message .sp To get the list of all the watchers: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "list", } .ft P .fi .UNINDENT .UNINDENT .sp To get the list of active processes in a watcher: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "list", "properties": { "name": "nameofwatcher", } } .ft P .fi .UNINDENT .UNINDENT .sp The response return the list asked. the mapping returned can either be \(aqwatchers\(aq or \(aqpids\(aq depending the request. .SS Command line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl list [] .ft P .fi .UNINDENT .UNINDENT .SS Subscribe to a watcher event .SS ZMQ .sp At any moment you can subscribe to a circus event. Circus provides a PUB/SUB feed on which any clients can subscribe. The subscriber endpoint URI is set in the circus.ini configuration file. .sp Events are pubsub topics: .INDENT 0.0 .IP \(bu 2 \fIwatcher..reap\fP: when a process is reaped .IP \(bu 2 \fIwatcher..spawn\fP: when a process is spawned .IP \(bu 2 \fIwatcher..kill\fP: when a process is killed .IP \(bu 2 \fIwatcher..updated\fP: when watcher configuration is updated .IP \(bu 2 \fIwatcher..stop\fP: when a watcher is stopped .IP \(bu 2 \fIwatcher..start\fP: when a watcher is started .UNINDENT .sp All events messages are in a json struct. .SS Command line .sp The client has been updated to provide a simple way to listen on the events: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C circusctl listen [, ...] .ft P .fi .UNINDENT .UNINDENT .SS Example of result: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl listen tcp://127.0.0.1:5556 watcher.refuge.spawn: {u\(aqprocess_id\(aq: 6, u\(aqprocess_pid\(aq: 72976, u\(aqtime\(aq: 1331681080.985104} watcher.refuge.spawn: {u\(aqprocess_id\(aq: 7, u\(aqprocess_pid\(aq: 72995, u\(aqtime\(aq: 1331681086.208542} watcher.refuge.spawn: {u\(aqprocess_id\(aq: 8, u\(aqprocess_pid\(aq: 73014, u\(aqtime\(aq: 1331681091.427005} .ft P .fi .UNINDENT .UNINDENT .SS Get the list of sockets .SS ZMQ Message .sp To get the list of sockets: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "listsockets", } .ft P .fi .UNINDENT .UNINDENT .sp The response return a list of json mappings with keys for fd, name, host and port. .SS Command line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl listsockets .ft P .fi .UNINDENT .UNINDENT .SS Get the number of processes .sp Get the number of processes in a watcher or in a arbiter .SS ZMQ Message .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "numprocesses", "propeties": { "name": "" } } .ft P .fi .UNINDENT .UNINDENT .sp The response return the number of processes in the \(aqnumprocesses\(ga property: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "status": "ok", "numprocesses": , "time", "timestamp" } .ft P .fi .UNINDENT .UNINDENT .sp If the property name isn\(aqt specified, the sum of all processes managed is returned. .SS Command line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl numprocesses [] .ft P .fi .UNINDENT .UNINDENT .SS Options .INDENT 0.0 .IP \(bu 2 : name of the watcher .UNINDENT .SS Get the number of watchers .sp Get the number of watchers in a arbiter .SS ZMQ Message .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "numwatchers", } .ft P .fi .UNINDENT .UNINDENT .sp The response return the number of watchers in the \(aqnumwatchers\(ga property: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "status": "ok", "numwatchers": , "time", "timestamp" } .ft P .fi .UNINDENT .UNINDENT .SS Command line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl numwatchers .ft P .fi .UNINDENT .UNINDENT .SS Get the value of all options for a watcher .sp This command returns all option values for a given watcher. .SS ZMQ Message .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "options", "properties": { "name": "nameofwatcher", } } .ft P .fi .UNINDENT .UNINDENT .sp A message contains 1 property: .INDENT 0.0 .IP \(bu 2 name: name of watcher .UNINDENT .sp The response object has a property \fBoptions\fP which is a dictionary of option names and values. .sp eg: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "status": "ok", "options": { "graceful_timeout": 300, "send_hup": True, ... }, time\(aq: 1332202594.754644 } .ft P .fi .UNINDENT .UNINDENT .SS Command line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl options .ft P .fi .UNINDENT .UNINDENT .SS Options .INDENT 0.0 .IP \(bu 2 : name of the watcher .UNINDENT .sp Options Keys are: .INDENT 0.0 .IP \(bu 2 numprocesses: integer, number of processes .IP \(bu 2 warmup_delay: integer or number, delay to wait between process spawning in seconds .IP \(bu 2 working_dir: string, directory where the process will be executed .IP \(bu 2 uid: string or integer, user ID used to launch the process .IP \(bu 2 gid: string or integer, group ID used to launch the process .IP \(bu 2 send_hup: boolean, if TRU the signal HUP will be used on reload .IP \(bu 2 shell: boolean, will run the command in the shell environment if true .IP \(bu 2 cmd: string, The command line used to launch the process .IP \(bu 2 env: object, define the environnement in which the process will be launch .IP \(bu 2 retry_in: integer or number, time in seconds we wait before we retry to launch the process if the maximum number of attempts has been reach. .IP \(bu 2 max_retry: integer, The maximum of retries loops .IP \(bu 2 graceful_timeout: integer or number, time we wait before we definitely kill a process. .IP \(bu 2 priority: used to sort watchers in the arbiter .IP \(bu 2 singleton: if True, a singleton watcher. .IP \(bu 2 max_age: time a process can live before being restarted .IP \(bu 2 max_age_variance: variable additional time to live, avoids stampeding herd. .UNINDENT .SS Quit the arbiter immediately .sp When the arbiter receive this command, the arbiter exit. .SS ZMQ Message .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "quit", "waiting": False } .ft P .fi .UNINDENT .UNINDENT .sp The response return the status "ok". .sp If \fBwaiting\fP is False (default), the call will return immediately after calling \fBstop_signal\fP on each process. .sp If \fBwaiting\fP is True, the call will return only when the stop process is completely ended. Because of the graceful_timeout option, it can take some time. .SS Command line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl quit [\-\-waiting] .ft P .fi .UNINDENT .UNINDENT .SS Reload the arbiter or a watcher .sp This command reloads all the process in a watcher or all watchers. This will happen in one of 3 ways: .INDENT 0.0 .IP \(bu 2 If graceful is false, a simple restart occurs. .IP \(bu 2 If \fIsend_hup\fP is true for the watcher, a HUP signal is sent to each process. .IP \(bu 2 .INDENT 2.0 .TP .B Otherwise: .INDENT 7.0 .IP \(bu 2 If sequential is false, the arbiter will attempt to spawn \fInumprocesses\fP new processes. If the new processes are spawned successfully, the result is that all of the old processes are stopped, since by default the oldest processes are stopped when the actual number of processes for a watcher is greater than \fInumprocesses\fP\&. .IP \(bu 2 If sequential is true, the arbiter will restart each process in a sequential way (with a \fIwarmup_delay\fP pause between each step) .UNINDENT .UNINDENT .UNINDENT .SS ZMQ Message .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "reload", "properties": { "name": \(aq", "graceful": true, "sequential": false, "waiting": False } } .ft P .fi .UNINDENT .UNINDENT .sp The response return the status "ok". If the property graceful is set to true the processes will be exited gracefully. .sp If the property name is present, then the reload will be applied to the watcher. .SS Command line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl reload [] [\-\-terminate] [\-\-waiting] [\-\-sequential] .ft P .fi .UNINDENT .UNINDENT .SS Options .INDENT 0.0 .IP \(bu 2 : name of the watcher .IP \(bu 2 \-\-terminate; quit the node immediately .UNINDENT .SS Reload the configuration file .sp This command reloads the configuration file, so changes in the configuration file will be reflected in the configuration of circus. .SS ZMQ Message .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "reloadconfig", "waiting": False } .ft P .fi .UNINDENT .UNINDENT .sp The response return the status "ok". If the property graceful is set to true the processes will be exited gracefully. .SS Command line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl reloadconfig [\-\-waiting] .ft P .fi .UNINDENT .UNINDENT .SS Restart the arbiter or a watcher .sp This command restart all the process in a watcher or all watchers. This funtion simply stop a watcher then restart it. .SS ZMQ Message .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "restart", "properties": { "name": "", "waiting": False, "match": "[simple|glob|regex]" } } .ft P .fi .UNINDENT .UNINDENT .sp The response return the status "ok". .sp If the property name is present, then the reload will be applied to the watcher. .sp If \fBwaiting\fP is False (default), the call will return immediately after calling \fIstop_signal\fP on each process. .sp If \fBwaiting\fP is True, the call will return only when the restart process is completely ended. Because of the graceful_timeout option, it can take some time. .sp The \fBmatch\fP parameter can have the value \fBsimple\fP for string compare, \fBglob\fP for wildcard matching (default) or \fBregex\fP for regex matching. .SS Command line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl restart [name] [\-\-waiting] [\-\-match=simple|glob|regex] .ft P .fi .UNINDENT .UNINDENT .SS Options .INDENT 0.0 .IP \(bu 2 : name or pattern of the watcher(s) .IP \(bu 2 : watcher match method .UNINDENT .SS Remove a watcher .sp This command removes a watcher dynamically from the arbiter. The watchers are gracefully stopped by default. .SS ZMQ Message .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "rm", "properties": { "name": "", "nostop": False, "waiting": False } } .ft P .fi .UNINDENT .UNINDENT .sp The response return a status "ok". .sp If \fBnostop\fP is True (default: False), the processes for the watcher will not be stopped \- instead the watcher will just be forgotten by circus and the watcher processes will be responsible for stopping themselves. If \fBnostop\fP is not specified or is False, then the watcher processes will be stopped gracefully. .sp If \fBwaiting\fP is False (default), the call will return immediately after starting to remove and stop the corresponding watcher. .sp If \fBwaiting\fP is True, the call will return only when the remove and stop process is completely ended. Because of the graceful_timeout option, it can take some time. .SS Command line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl rm [\-\-waiting] [\-\-nostop] .ft P .fi .UNINDENT .UNINDENT .SS Options .INDENT 0.0 .IP \(bu 2 : name of the watcher to remove .IP \(bu 2 nostop: do not stop the watcher processes, just remove the watcher .UNINDENT .SS Set a watcher option .SS ZMQ Message .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "set", "properties": { "name": "nameofwatcher", "options": { "key1": "val1", .. } "waiting": False } } .ft P .fi .UNINDENT .UNINDENT .sp The response return the status "ok". See the command Options for a list of key to set. .SS Command line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl set \-\-waiting .ft P .fi .UNINDENT .UNINDENT .SS Send a signal .sp This command allows you to send a signal to all processes in a watcher, a specific process in a watcher or its children. .SS ZMQ Message .sp To send a signal to all the processes for a watcher: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "signal", "property": { "name": , "signum": } .ft P .fi .UNINDENT .UNINDENT .sp To send a signal to a process: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "signal", "property": { "name": , "pid": , "signum": } .ft P .fi .UNINDENT .UNINDENT .sp An optional property "children" can be used to send the signal to all the children rather than the process itself: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "signal", "property": { "name": , "pid": , "signum": , "children": True } .ft P .fi .UNINDENT .UNINDENT .sp To send a signal to a process child: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "signal", "property": { "name": , "pid": , "signum": , "child_pid": , } .ft P .fi .UNINDENT .UNINDENT .sp It is also possible to send a signal to all the children of the watcher: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "signal", "property": { "name": , "signum": , "children": True } .ft P .fi .UNINDENT .UNINDENT .sp Lastly, you can send a signal to the process \fIand\fP its children, with the \fIrecursive\fP option: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "signal", "property": { "name": , "signum": , "recursive": True } .ft P .fi .UNINDENT .UNINDENT .SS Command line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl signal [] [\-\-children] [\-\-recursive] .ft P .fi .UNINDENT .UNINDENT .SS Options: .INDENT 0.0 .IP \(bu 2 : the name of the watcher .IP \(bu 2 : integer, the process id. .IP \(bu 2 : the signal number (or name) to send. .IP \(bu 2 : the pid of a child, if any .IP \(bu 2 : boolean, send the signal to all the children .IP \(bu 2 : boolean, send the signal to the process and its children .UNINDENT .SS Start the arbiter or a watcher .sp This command starts all the processes in a watcher or all watchers. .SS ZMQ Message .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "start", "properties": { "name": \(aq", "waiting": False, "match": "[simple|glob|regex]" } } .ft P .fi .UNINDENT .UNINDENT .sp The response return the status "ok". .sp If the property name is present, the watcher will be started. .sp If \fBwaiting\fP is False (default), the call will return immediately after calling \fIstart\fP on each process. .sp If \fBwaiting\fP is True, the call will return only when the start process is completely ended. Because of the graceful_timeout option, it can take some time. .sp The \fBmatch\fP parameter can have the value \fBsimple\fP for string compare, \fBglob\fP for wildcard matching (default) or \fBregex\fP for regex matching. .SS Command line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl restart [name] [\-\-waiting] [\-\-match=simple|glob|regex] .ft P .fi .UNINDENT .UNINDENT .SS Options .INDENT 0.0 .IP \(bu 2 : name or pattern of the watcher(s) .IP \(bu 2 : watcher match method .UNINDENT .SS Get process infos .sp You can get at any time some statistics about your processes with the stat command. .SS ZMQ Message .sp To get stats for all watchers: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "stats" } .ft P .fi .UNINDENT .UNINDENT .sp To get stats for a watcher: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "stats", "properties": { "name": } } .ft P .fi .UNINDENT .UNINDENT .sp To get stats for a process: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "stats", "properties": { "name": , "process": } } .ft P .fi .UNINDENT .UNINDENT .sp Stats can be extended with the extended_stats hook but extended stats need to be requested: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "stats", "properties": { "name": , "process": , "extended": True } } .ft P .fi .UNINDENT .UNINDENT .sp The response retun an object per process with the property "info" containing some process informations: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "info": { "children": [], "cmdline": "python", "cpu": 0.1, "ctime": "0:00.41", "mem": 0.1, "mem_info1": "3M", "mem_info2": "2G", "nice": 0, "pid": 47864, "username": "root" }, "process": 5, "status": "ok", "time": 1332265655.897085 } .ft P .fi .UNINDENT .UNINDENT .SS Command Line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl stats [\-\-extended] [] [] .ft P .fi .UNINDENT .UNINDENT .SS Get the status of a watcher or all watchers .sp This command start get the status of a watcher or all watchers. .SS ZMQ Message .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "status", "properties": { "name": \(aq", } } .ft P .fi .UNINDENT .UNINDENT .sp The response return the status "active" or "stopped" or the status / watchers. .SS Command line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl status [] .ft P .fi .UNINDENT .UNINDENT .SS Options .INDENT 0.0 .IP \(bu 2 : name of the watcher .UNINDENT .SS Example .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl status dummy active $ circusctl status dummy: active dummy2: active refuge: active .ft P .fi .UNINDENT .UNINDENT .SS Stop watchers .sp This command stops a given watcher or all watchers. .SS ZMQ Message .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C { "command": "stop", "properties": { "name": "", "waiting": False, "match": "[simple|glob|regex]" } } .ft P .fi .UNINDENT .UNINDENT .sp The response returns the status "ok". .sp If the \fBname\fP property is present, then the stop will be applied to the watcher corresponding to that name. Otherwise, all watchers will get stopped. .sp If \fBwaiting\fP is False (default), the call will return immediatly after calling \fIstop_signal\fP on each process. .sp If \fBwaiting\fP is True, the call will return only when the stop process is completly ended. Because of the graceful_timeout option, it can take some time. .sp The \fBmatch\fP parameter can have the value \fBsimple\fP for string compare, \fBglob\fP for wildcard matching (default) or \fBregex\fP for regex matching. .SS Command line .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl stop [name] [\-\-waiting] [\-\-match=simple|glob|regex] .ft P .fi .UNINDENT .UNINDENT .SS Options .INDENT 0.0 .IP \(bu 2 : name or pattern of the watcher(s) .IP \(bu 2 : watcher match method .UNINDENT .SS CLI tools .SS circus\-top .sp \fIcircus\-top\fP is a top\-like console you can run to watch live your running Circus system. It will display the CPU, Memory usage and socket hits if you have some. .sp Example of output: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- circusd\-stats PID CPU (%) MEMORY (%) 14252 0.8 0.4 0.8 (avg) 0.4 (sum) dummy PID CPU (%) MEMORY (%) 14257 78.6 0.1 14256 76.6 0.1 14258 74.3 0.1 14260 71.4 0.1 14259 70.7 0.1 74.32 (avg) 0.5 (sum) \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- .ft P .fi .UNINDENT .UNINDENT .sp \fIcircus\-top\fP is a read\-only console. If you want to interact with the system, use \fIcircusctl\fP\&. .SS circusctl .sp \fIcircusctl\fP can be used to run any command listed in commands . For example, you can get a list of all the watchers, you can do .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusctl list .ft P .fi .UNINDENT .UNINDENT .sp Besides supporting a handful of options you can also specify the endpoint \fIcircusctl\fP should use using the \fBCIRCUSCTL_ENDPOINT\fP environment variable. .SS The Web Console .sp Circus comes with a Web Console that can be used to manage the system. .sp The Web Console lets you: .INDENT 0.0 .IP \(bu 2 Connect to any running Circus system .IP \(bu 2 Watch the processes CPU and Memory usage in real\-time .IP \(bu 2 Add or kill processes .IP \(bu 2 Add new watchers .UNINDENT .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 The real\-time CPU & Memory usage feature uses the stats socket. If you want to activate it, make sure the Circus system you\(aqll connect to has the stats enpoint enabled in its configuration: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [circus] statsd = True .ft P .fi .UNINDENT .UNINDENT .sp By default, this option is not activated. .UNINDENT .UNINDENT .sp The web console is its own package, you need to install: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ pip install circus\-web .ft P .fi .UNINDENT .UNINDENT .sp To enable the console, add a few options in the Circus ini file: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [circus] httpd = True httpd_host = localhost httpd_port = 8080 .ft P .fi .UNINDENT .UNINDENT .sp \fIhttpd_host\fP and \fIhttpd_port\fP are optional, and default to \fIlocalhost\fP and \fI8080\fP\&. .sp If you want to run the web app on its own, just run the \fBcircushttpd\fP script: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circushttpd Bottle server starting up... Listening on http://localhost:8080/ Hit Ctrl\-C to quit. .ft P .fi .UNINDENT .UNINDENT .sp By default the script will run the Web Console on port 8080, but the \-\-port option can be used to change it. .SS Using the console .sp Once the script is running, you can open a browser and visit \fIhttp://localhost:8080\fP\&. You should get this screen: [image] .sp The Web Console is ready to be connected to a Circus system, given its \fBendpoint\fP\&. By default the endpoint is \fItcp://127.0.0.1:5555\fP\&. .sp Once you hit \fIConnect\fP, the web application will connect to the Circus system. .sp With the Web Console logged in, you should get a list of watchers, and a real\-time status of the two Circus processes (circusd and circusd\-stats). \fI\%\fP .sp You can click on the status of each watcher to toggle it from \fBActive\fP (green) to \fBInactive\fP (red). This change is effective immediatly and let you start & stop watchers. .sp If you click on the watcher name, you will get a web page for that particular watcher, with its processes: \fI\%\fP .sp On this screen, you can add or remove processes, and kill existing ones. .sp Last but not least, you can add a brand new watcher by clicking on the \fIAdd Watcher\fP link in the left menu: \fI\%\fP.SS Running behind Nginx .sp Nginx can act as a proxy and security layer in front of circus\-web. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 To receive real\-time status updates and graphs in circus\-web, you must provide a Nginx proxy solution that has websocket support .UNINDENT .UNINDENT .SS Nginx >= 1.3.13 .sp As of Nginx>=1.3.13 websocket support is built\-in, so there is no need to combine Nginx with Varnish or HAProxy. An example Nginx config with websocket support: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C upstream circusweb_server { server 127.0.0.1:8080; } server { listen 80; server_name _; location / { proxy_pass http://circusweb_server; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header Host $host; proxy_set_header X\-Real\-IP $remote_addr; proxy_set_header X\-Forwarded\-For $proxy_add_x_forwarded_for; proxy_set_header X\-Forwarded\-Proto http; proxy_redirect off; } location ~/media/\e*(.png|.jpg|.css|.js|.ico)$ { alias /path_to_site\-packages/circusweb/media/; } } .ft P .fi .UNINDENT .UNINDENT .SS Nginx < 1.3.13 .sp Nginx versions < 1.3.13 do not have websocket support built\-in. .sp To provide websocket support for circus\-web when using Nginx < 1.3.13, you can combine Nginx with Varnish or HAProxy. That is, Nginx in front of circus\-web, with Varnish or HAProxy in front of Nginx. .sp The example below shows the combined Nginix and Varnish configuration required to proxy circus\-web and provide websocket support. .sp \fBNginx configuration:\fP .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C upstream circusweb_server { server 127.0.0.1:8080; } server { listen 8001; server_name _; location / { proxy_set_header X\-Forwarded\-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_redirect off; proxy_pass http://circusweb_server; } location ~/media/\e*(.png|.jpg|.css|.js|.ico)$ { alias /path_to_site\-packages/circusweb/media/; } } .ft P .fi .UNINDENT .UNINDENT .sp If you want more Nginx configuration options, see \fI\%http://wiki.nginx.org/HttpProxyModule\fP\&. .sp \fBVarnish configuration:\fP .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C backend default { .host = "127.0.0.1"; .port = "8001"; } backend socket { .host = "127.0.0.1"; .port = "8080"; .connect_timeout = 1s; .first_byte_timeout = 2s; .between_bytes_timeout = 60s; } sub vcl_pipe { if (req.http.upgrade) { set bereq.http.upgrade = req.http.upgrade; } } sub vcl_recv { if (req.http.Upgrade ~ "(?i)websocket") { set req.backend = socket; return (pipe); } } .ft P .fi .UNINDENT .UNINDENT .sp In the Varnish configuration example above two backends are defined. One serving the web console and one serving the socket connections. Web console requests are bound to port 8001. The Nginx \(aqserver\(aq directive should be configured to listen on port 8001. .sp Websocket connections are upgraded and piped directly to the circushttpd process listening on port 8080 by Varnish. i.e. bypassing the Nginx proxy. .SS Ubuntu .sp Since the version 13.10 (\fISaucy\fP), Ubuntu includes Nginx with websocket support in its own repositories. For older versions, you can install Nginx>=1.3.13 from the official Nginx stable PPA, as so: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C sudo apt\-get install python\-software\-properties sudo add\-apt\-repository ppa:nginx/stable sudo apt\-get update sudo apt\-get install nginx nginx \-v .ft P .fi .UNINDENT .UNINDENT .SS Password\-protect circushttpd .sp As explained in the Security page, running \fIcircushttpd\fP is pretty unsafe. We don\(aqt provide any security in Circus itself, but you can protect your console at the NGinx level, by using \fI\%http://wiki.nginx.org/HttpAuthBasicModule\fP .sp Example: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C location / { proxy_set_header X\-Forwarded\-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_set_header X\-Forwarded\-Host: $http_host; proxy_set_header X\-Forwarded\-Proto: $scheme; proxy_redirect off; proxy_pass http://127.0.0.1:8080; auth_basic "Restricted"; auth_basic_user_file /path/to/htpasswd; } .ft P .fi .UNINDENT .UNINDENT .sp The \fBhtpasswd\fP file contains users and their passwords, and a password prompt will pop when you access the console. .sp You can use Apache\(aqs htpasswd script to edit it, or the Python script they provide at: \fI\%http://trac.edgewall.org/browser/trunk/contrib/htpasswd.py\fP .sp However, there\(aqs no native support for the combined use of HTTP Authentication and WebSockets (the server will throw HTTP 401 error codes). A workaround is to disable such authentication for the socket.io server. .sp Example (needs to be added before the previous rule): .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C location /socket.io { proxy_set_header X\-Forwarded\-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_set_header X\-Forwarded\-Host: $http_host; proxy_set_header X\-Forwarded\-Proto: $scheme; proxy_redirect off; proxy_pass http://127.0.0.1:8080; } .ft P .fi .UNINDENT .UNINDENT .sp Of course that\(aqs just one way to protect your web console, you could use many other techniques. .SS Extending the web console .sp We picked \fIbottle\fP to build the webconsole, mainly because it\(aqs a really tiny framework that doesn\(aqt do much. By having a look at the code of the web console, you\(aqll eventually find out that it\(aqs really simple to understand. .sp Here is how it\(aqs split: .INDENT 0.0 .IP \(bu 2 The \fIcircushttpd.py\fP file contains the "views" definitions and some code to handle the socket connection (via socketio). .IP \(bu 2 the \fIcontroller.py\fP contains a single class which is in charge of doing the communication with the circus controller. It allows to have a nicer high level API when defining the web server. .UNINDENT .sp If you want to add a feature in the web console you can reuse the code that\(aqs existing. A few tools are at your disposal to ease the process: .INDENT 0.0 .IP \(bu 2 There is a \fIrender_template\fP function, which takes the named arguments you pass to it and pass them to the template renderer and return the resulting HTML. It also passes some additional variables, such as the session, the circus version and the client if defined. .IP \(bu 2 If you want to run commands and doa redirection depending the result of it, you can use the \fIrun_command\fP function, which takes a callable as a first argument, a message in case of success and a redirection url. .UNINDENT .sp The \fBStatsNamespace\fP class is responsible for managing the websocket communication on the server side. Its documentation should help you to understand what it does. .SS Working with sockets .sp Circus can bind network sockets and manage them as it does for processes. .sp The main idea is that a child process that\(aqs created by Circus to run one of the watcher\(aqs command can inherit from all the opened file descriptors. .sp That\(aqs how Apache or Unicorn works, and many other tools out there. .SS Goal .sp The goal of having sockets managed by Circus is to be able to manage network applications in Circus exactly like other applications. .sp For example, if you use Circus with \fI\%Chaussette\fP \-\- a WGSI server, you can get a very fast web server running and manage \fI"Web Workers"\fP in Circus as you would do for any other process. .sp Splitting the socket managment from the network application itself offers a lot of opportunities to scale and manage your stack. .SS Design .sp The gist of the feature is done by binding the socket and start listening to it in \fBcircusd\fP: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C import socket sock = socket.socket(FAMILY, TYPE) sock.bind((HOST, PORT)) sock.listen(BACKLOG) fd = sock.fileno() .ft P .fi .UNINDENT .UNINDENT .sp Circus then keeps track of all the opened fds, and let the processes it runs as children have access to them if they want. .sp If you create a small Python network script that you intend to run in Circus, it could look like this: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C import socket import sys fd = int(sys.argv[1]) # getting the FD from circus sock = socket.fromfd(fd, FAMILY, TYPE) # dealing with one request at a time while True: conn, addr = sock.accept() request = conn.recv(1024) .. do something .. conn.sendall(response) conn.close() .ft P .fi .UNINDENT .UNINDENT .sp Then Circus could run like this: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [circus] check_delay = 5 endpoint = tcp://127.0.0.1:5555 pubsub_endpoint = tcp://127.0.0.1:5556 stats_endpoint = tcp://127.0.0.1:5557 [watcher:dummy] cmd = mycoolscript $(circus.sockets.foo) use_sockets = True warmup_delay = 0 numprocesses = 5 [socket:foo] host = 127.0.0.1 port = 8888 .ft P .fi .UNINDENT .UNINDENT .sp \fI$(circus.sockets.foo)\fP will be replaced by the FD value once the socket is created and bound on the 8888 \fIport\fP\&. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Starting at Circus 0.8 there\(aqs an alternate syntax to avoid some conflicts with some config parsers. You can write: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C ((circus.sockets.foo)) .ft P .fi .UNINDENT .UNINDENT .UNINDENT .UNINDENT .SS Real\-world example .sp \fI\%Chaussette\fP is the perfect Circus companion if you want to run your WSGI application. .sp Once it\(aqs installed, running 5 \fBmeinheld\fP workers can be done by creating a socket and calling the \fBchaussette\fP command in a worker, like this: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [circus] endpoint = tcp://127.0.0.1:5555 pubsub_endpoint = tcp://127.0.0.1:5556 stats_endpoint = tcp://127.0.0.1:5557 [watcher:web] cmd = chaussette \-\-fd $(circus.sockets.web) \-\-backend meinheld mycool.app use_sockets = True numprocesses = 5 [socket:web] host = 0.0.0.0 port = 8000 .ft P .fi .UNINDENT .UNINDENT .sp We did not publish benchmarks yet, but a Web cluster managed by Circus with a Gevent or Meinheld backend is as fast as any pre\-fork WSGI server out there. .SS Using built\-in plugins .sp Circus comes with a few built\-in plugins. This section presents these plugins and their configuration options. .SS Statsd .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B \fBuse\fP set to \(aqcircus.plugins.statsd.StatsdEmitter\(aq .TP .B \fBapplication_name\fP the name used to identify the bucket prefix to emit the stats to (it will be prefixed with \fBcircus.\fP and suffixed with \fB\&.watcher\fP) .TP .B \fBhost\fP the host to post the statds data to .TP .B \fBport\fP the port the statsd daemon listens on .TP .B \fBsample_rate\fP if you prefer a different sample rate than 1, you can set it here .UNINDENT .UNINDENT .UNINDENT .SS FullStats .INDENT 0.0 .INDENT 3.5 An extension on the Statsd plugin that is also publishing the process stats. As such it has the same configuration options as Statsd and the following. .INDENT 0.0 .TP .B \fBuse\fP set to \fBcircus.plugins.statsd.FullStats\fP .TP .B \fBloop_rate\fP the frequency the plugin should ask for the stats in seconds. Default: 60. .UNINDENT .UNINDENT .UNINDENT .SS RedisObserver .INDENT 0.0 .INDENT 3.5 This services observers a redis process for you, publishes the information to statsd and offers to restart the watcher when it doesn\(aqt react in a given timeout. This plugin requires \fI\%redis\-py\fP to run. .sp It has the same configuration as statsd and adds the following: .INDENT 0.0 .TP .B \fBuse\fP set to \fBcircus.plugins.redis_observer.RedisObserver\fP .TP .B \fBloop_rate\fP the frequency the plugin should ask for the stats in seconds. Default: 60. .TP .B \fBredis_url\fP the database to check for as a redis url. Default: "redis://localhost:6379/0" .TP .B \fBtimeout\fP the timeout in seconds the request can take before it is considered down. Defaults to 5. .TP .B \fBrestart_on_timeout\fP the name of the process to restart when the request timed out. No restart triggered when not given. Default: None. .UNINDENT .UNINDENT .UNINDENT .SS HttpObserver .INDENT 0.0 .INDENT 3.5 This services observers a http process for you by pinging a certain website regularly. Similar to the redis observer it offers to restart the watcher on an error. It requires \fI\%tornado\fP to run. .sp It has the same configuration as statsd and adds the following: .INDENT 0.0 .TP .B \fBuse\fP set to \fBcircus.plugins.http_observer.HttpObserver\fP .TP .B \fBloop_rate\fP the frequency the plugin should ask for the stats in seconds. Default: 60. .TP .B \fBcheck_url\fP the url to check for. Default: \fBhttp://localhost/\fP .TP .B \fBtimeout\fP the timeout in seconds the request can take before it is considered down. Defaults to 10. .TP .B \fBrestart_on_error\fP the name of the process to restart when the request timed out or returned any other kind of error. No restart triggered when not given. Default: None. .UNINDENT .UNINDENT .UNINDENT .SS ResourceWatcher .INDENT 0.0 .INDENT 3.5 This services watches the resources of the given process and triggers a restart when they exceed certain limitations too often in a row. .sp It has the same configuration as statsd and adds the following: .INDENT 0.0 .TP .B \fBuse\fP set to \fBcircus.plugins.resource_watcher.ResourceWatcher\fP .TP .B \fBloop_rate\fP the frequency the plugin should ask for the stats in seconds. Default: 60. .TP .B \fBwatcher\fP the watcher this resource watcher should be looking after. (previously called \fBservice\fP but \fBservice\fP is now deprecated) .TP .B \fBmax_cpu\fP The maximum cpu one process is allowed to consume (in %). Default: 90 .TP .B \fBmin_cpu\fP The minimum cpu one process should consume (in %). Default: None (no minimum) You can set the min_cpu to 0 (zero), in this case if one process consume exactly 0% cpu, it will trigger an exceeded limit. .TP .B \fBmax_mem\fP The amount of memory one process of this watcher is allowed to consume. Default: 90. If no unit is specified, the value is in %. Example: 50 If a unit is specified, the value is in bytes. Supported units are B, K, M, G, T, P, E, Z, Y. Example: 250M .TP .B \fBmin_mem\fP The minimum memory one process of this watcher should consume. Default: None (no minimum). If no unit is specified, the value is in %. Example: 50 If a unit is specified, the value is in bytes. Supported units are B, K, M, G, T, P, E, Z, Y. Example: 250M .TP .B \fBhealth_threshold\fP The health is the average of cpu and memory (in %) the watchers processes are allowed to consume (in %). Default: 75 .TP .B \fBmax_count\fP How often these limits (each one is counted separately) are allowed to be exceeded before a restart will be triggered. Default: 3 .UNINDENT .UNINDENT .UNINDENT .sp Example: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [circus] ; ... [watcher:program] cmd = sleep 120 [plugin:myplugin] use = circus.plugins.resource_watcher.ResourceWatcher watcher = program min_cpu = 10 max_cpu = 70 min_mem = 0 max_mem = 20 .ft P .fi .UNINDENT .UNINDENT .SS Watchdog .INDENT 0.0 .INDENT 3.5 Plugin that binds an udp socket and wait for watchdog messages. For "watchdoged" processes, the watchdog will kill them if they don\(aqt send a heartbeat in a certain period of time materialized by loop_rate * max_count. (circus will automatically restart the missing processes in the watcher) .sp Each monitored process should send udp message at least at the loop_rate. The udp message format is a line of text, decoded using \fBmsg_regex\fP parameter. The heartbeat message MUST at least contain the pid of the process sending the message. .sp The list of monitored watchers are determined by the parameter \fBwatchers_regex\fP in the configuration. .sp Configuration parameters: .INDENT 0.0 .TP .B \fBuse\fP set to \fBcircus.plugins.watchdog.WatchDog\fP .TP .B \fBloop_rate\fP watchdog loop rate in seconds. At each loop, WatchDog will looks for "dead" processes. .TP .B \fBwatchers_regex\fP regex for matching watcher names that should be monitored by the watchdog (default: \fB\&.*\fP all watchers are monitored) .TP .B \fBmsg_regex\fP regex for decoding the received heartbeat message in udp (default: \fB^(?P.*);(?P.*)$\fP) the default format is a simple text message: \fBpid;timestamp\fP .TP .B \fBmax_count\fP max number of passed loop without receiving any heartbeat before restarting process (default: 3) .TP .B \fBip\fP ip the watchdog will bind on (default: 127.0.0.1) .TP .B \fBport\fP port the watchdog will bind on (default: 1664) .UNINDENT .UNINDENT .UNINDENT .SS Flapping .INDENT 0.0 .INDENT 3.5 When a worker restarts too often, we say that it is \fIflapping\fP\&. This plugin keeps track of worker restarts and stops the corresponding watcher in case it is flapping. This plugin may be used to automatically stop workers that get constantly restarted because they\(aqre not working properly. .INDENT 0.0 .TP .B \fBuse\fP set to \fBcircus.plugins.flapping.Flapping\fP .TP .B \fBattempts\fP the number of times a process can restart, within \fBwindow\fP seconds, before we consider it flapping (default: 2) .TP .B \fBwindow\fP the time window in seconds to test for flapping. If the process restarts more than \fBattempts\fP times within this time window, we consider it a flapping process. (default: 1) .TP .B \fBretry_in\fP time in seconds to wait until we try to start again a process that has been flapping. (default: 7) .TP .B \fBmax_retry\fP the number of times we attempt to start a process that has been flapping, before we abandon and stop the whole watcher. (default: 5) Set to \-1 to disable max_retry and retry indefinitely. .TP .B \fBactive\fP define if the plugin is active or not (default: True). If the global flag is set to False, the plugin is not started. .UNINDENT .UNINDENT .UNINDENT .sp Options can be overriden in the watcher section using a \fBflapping.\fP prefix. For instance, here is how you would configure a specific \fBmax_retry\fP value for nginx: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [watcher:nginx] cmd = /path/to/nginx flapping.max_retry = 2 [watcher:myscript] cmd = ./my_script.py ; ... other watchers [plugin:flapping] use = circus.plugins.flapping.Flapping max_retry = 5 .ft P .fi .UNINDENT .UNINDENT .SS CommandReloader .INDENT 0.0 .INDENT 3.5 This plugin will restart watchers when their command file is modified. It works by checking the modification time and the path of the file pointed by the \fBcmd\fP option every \fBloop_rate\fP seconds. This may be useful while developing worker processes or even for hot code upgrade in production. .INDENT 0.0 .TP .B \fBuse\fP set to \fBcircus.plugins.command_reloader.CommandReloader\fP .TP .B \fBloop_rate\fP the frequency the plugin should check for modification in seconds. Default: 1. .UNINDENT .UNINDENT .UNINDENT .SS Deployment .sp Although the Circus daemon can be managed with the circusd command, it\(aqs easier to have it start on boot. If your system supports Upstart, you can create this Upstart script in /etc/init/circus.conf. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C start on filesystem and net\-device\-up IFACE=lo stop on runlevel [016] respawn exec /usr/local/bin/circusd /etc/circus/circusd.ini .ft P .fi .UNINDENT .UNINDENT .sp This assumes that circusd.ini is located at /etc/circus/circusd.ini. After rebooting, you can control circusd with the service command: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C # service circus start/stop/restart .ft P .fi .UNINDENT .UNINDENT .sp If your system supports systemd, you can create this systemd unit file under /etc/systemd/system/circus.service. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [Unit] Description=Circus process manager After=syslog.target network.target nss\-lookup.target [Service] Type=simple ExecReload=/usr/bin/circusctl reload ExecStart=/usr/bin/circusd /etc/circus/circus.ini Restart=always RestartSec=5 [Install] WantedBy=default.target .ft P .fi .UNINDENT .UNINDENT .sp A reboot isn\(aqt required if you run the daemon\-reload command below: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C # systemctl \-\-system daemon\-reload .ft P .fi .UNINDENT .UNINDENT .sp Then circus can be managed via: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C # systemctl start/stop/status/reload circus .ft P .fi .UNINDENT .UNINDENT .SS Recipes .sp This section will contain recipes to deploy Circus. Until then you can look at Pete\(aqs \fI\%Puppet recipe\fP or at Remy\(aqs \fI\%Chef recipe\fP .SS Papa Process Kernel .sp One problem common to process managers is that you cannot restart the process manager without restarting all of the processes it manages. This makes it difficult to deploy a new version of Circus or new versions of any of the libraries on which it depends. .sp If you are on a Unix\-type system, Circus can use the Papa process kernel. When used, Papa will create a long\-lived daemon that will serve as the host for any processes and sockets you create with it. If circus is shutdown, Papa will maintain everything it is hosting. .SS Setup .sp Start by installing the \fIpapa\fP and \fIsetproctitle\fP modules: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C pip install papa pip install setproctitle .ft P .fi .UNINDENT .UNINDENT .sp The \fIsetproctitle\fP module is optional. It will be used if present to rename the Papa daemon for \fItop\fP and \fIps\fP to something like "papa daemon from circusd". If you do not install the \fIsetproctitle\fP module, that title will be the command line of the process that launched it. Very confusing. .sp Once Papa is installed, add \fIuse_papa=true\fP to your critical processes and sockets. Generally you want to house all of the processes of your stack in Papa, and none of the Circus support processes such as the flapping and stats plugins. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [circus] loglevel = info [watcher:nginx] cmd = /usr/local/nginx/sbin/nginx \-p /Users/scottmax/Source/service\-framework/Common/conf/nginx \-c /Users/scottmax/Source/service\-framework/Common/conf/nginx/nginx.conf warmup_delay = 3 graceful_timeout = 10 max_retry = 5 singleton = true send_hup = true stop_signal = QUIT stdout_stream.class = FileStream stdout_stream.filename = /var/logs/web\-server.log stdout_stream.max_bytes = 10000000 stdout_stream.backup_count = 10 stderr_stream.class = FileStream stderr_stream.filename = /var/logs/web\-server\-error.log stderr_stream.max_bytes = 1000000 stderr_stream.backup_count = 10 active = true use_papa = true [watcher:logger] cmd = /my_service/env/bin/python logger.py run working_dir = /my_service graceful_timeout = 10 singleton = true stop_signal = INT stdout_stream.class = FileStream stdout_stream.filename = /var/logs/logger.log stdout_stream.max_bytes = 10000000 stdout_stream.backup_count = 10 stderr_stream.class = FileStream stderr_stream.filename = /var/logs/logger.log stderr_stream.max_bytes = 1000000 stderr_stream.backup_count = 10 priority = 50 use_papa = true [watcher:web_app] cmd = /my_service/env/bin/uwsgi \-\-ini uwsgi\-live.ini \-\-socket fd://$(circus.sockets.web) \-\-stats 127.0.0.1:809$(circus.wid) working_dir = /my_service/web_app graceful_timeout=10 stop_signal = QUIT use_sockets = True stdout_stream.class = FileStream stdout_stream.filename = /var/logs/web_app.log stdout_stream.max_bytes = 10000000 stdout_stream.backup_count = 10 stderr_stream.class = FileStream stderr_stream.filename = /var/logs/web_app.log stderr_stream.max_bytes = 1000000 stderr_stream.backup_count = 10 hooks.after_spawn = examples.uwsgi_lossless_reload.children_started hooks.before_signal = examples.uwsgi_lossless_reload.clean_stop hooks.extended_stats = examples.uwsgi_lossless_reload.extended_stats priority = 40 use_papa = true [socket:web] path = /my_service/sock/uwsgi use_papa = true [plugin:flapping] use = circus.plugins.flapping.Flapping window = 10 priority = 1000 .ft P .fi .UNINDENT .UNINDENT .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 If the Papa processes use any sockets, those sockets must also use papa. .UNINDENT .UNINDENT .SS Design Goal .sp Papa is designed to be very minimalist in features and requirements. It does: .INDENT 0.0 .IP \(bu 2 Start and stop sockets .IP \(bu 2 Provide a key/value store .IP \(bu 2 Start processes and return stdout, stderr and the exit code .UNINDENT .sp It does not: .INDENT 0.0 .IP \(bu 2 Restart processes .IP \(bu 2 Provide a way to stop processes .IP \(bu 2 Provide any information about processes other than whether or not they are still running .UNINDENT .sp Papa requires no third\-party libraries so it can run on just the standard Python library. It can make use of the \fIsetproctitle\fP package but that is only used for making the title prettier for \fIps\fP and \fItop\fP and is not essential. .sp The functionality has been kept to a minimum so that you should never need to restart the Papa daemon. As much of the functionality has been pushed to the client library as possible. That way you should be able to deploy a new copy of Papa for new client features without needing to restart the Papa daemon. Papa is meant to be a pillar of stability in a changing sea of 3rd party libraries. .SS Operation .sp Most things remain unchanged whether you use Papa or not. You can still start and stop processes. You can still get status and stats for processes. The main thing that changes is that when you do \fIcircusctl quit\fP, all of the Papa processes are left running. When you start \fBcircusd\fP back up, those processes are recovered. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 When processes are recovered, \fIbefore_start\fP and \fIbefore_spawn\fP hooks are skipped. .UNINDENT .UNINDENT .SS Logging .sp While Circus is shut down, Papa will store up to 2M of output per process. Then it will start dumping the oldest data. When you restart Circus, this cached output will be quickly retrieved and sent to the output streams. Papa requires that receipt of output be acknowledged, so you should not lose any output during a shutdown. .sp Not only that, but Papa saves the timestamp of the output. Circus has been enhanced to take advantage of timestamp data if present. So if you are writing the output to log files or somewhere, your timestamps should all be correct. .SS Problems .sp If you use the \fIincr\fP or \fIdecr\fP command to change the process count for a watcher, this will be reset to the level specified in the INI file when \fBcircusd\fP is restarted. .sp Also, I have experienced problems with the combination of \fIcopy_env\fP and \fIvirtualenv\fP\&. You may note that the INI sample above circumvents this issue with explicit paths. .SS Telnet Interface .sp Papa has a basic command\-line interface that you can access through telnet: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C telnet localhost 20202 help .ft P .fi .UNINDENT .UNINDENT .SS Circus for developers .SS Using Circus as a library .sp Circus provides high\-level classes and functions that will let you manage processes in your own applications. .sp For example, if you want to run four processes forever, you could write: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C from circus import get_arbiter myprogram = {"cmd": "python myprogram.py", "numprocesses": 4} arbiter = get_arbiter([myprogram]) try: arbiter.start() finally: arbiter.stop() .ft P .fi .UNINDENT .UNINDENT .sp This snippet will run four instances of \fImyprogram\fP and watch them for you, restarting them if they die unexpectedly. .sp To learn more about this, see library .SS Extending Circus .sp It\(aqs easy to extend Circus to create a more complex system, by listening to all the \fBcircusd\fP events via its pub/sub channel, and driving it via commands. .sp That\(aqs how the flapping feature works for instance: it listens to all the processes dying, measures how often it happens, and stops the incriminated watchers after too many restarts attempts. .sp Circus comes with a plugin system to help you write such extensions, and a few built\-in plugins you can reuse. See plugins\&. .sp You can also have a more subtile startup and shutdown behavior by using the \fBhooks\fP system that will let you run arbitrary code before and after some processes are started or stopped. See hooks\&. .sp Last but not least, you can also add new commands. See addingcmds\&. .SS Developers Documentation Index .SS Circus Library .sp The Circus package is composed of a high\-level \fBget_arbiter()\fP function and many classes. In most cases, using the high\-level function should be enough, as it creates everything that is needed for Circus to run. .sp You can subclass Circus\(aq classes if you need more granularity than what is offered by the configuration. .SS The get_arbiter function .sp \fBget_arbiter()\fP is just a convenience on top of the various circus classes. It creates a arbiter (class \fBArbiter\fP) instance with the provided options, which in turn runs a single \fBWatcher\fP with a single \fBProcess\fP\&. .INDENT 0.0 .TP .B circus.get_arbiter() .UNINDENT .sp Example: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C from circus import get_arbiter arbiter = get_arbiter([{"cmd": "myprogram", "numprocesses": 3}]) try: arbiter.start() finally: arbiter.stop() .ft P .fi .UNINDENT .UNINDENT .SS Classes .sp Circus provides a series of classes you can use to implement your own process manager: .INDENT 0.0 .IP \(bu 2 \fBProcess\fP: wraps a running process and provides a few helpers on top of it. .IP \(bu 2 \fBWatcher\fP: run several instances of \fBProcess\fP against the same command. Manage the death and life of processes. .IP \(bu 2 \fBArbiter\fP: manages several \fBWatcher\fP\&. .UNINDENT .INDENT 0.0 .TP .B class circus.process.Process(name, wid, cmd, args=None, working_dir=None, shell=False, uid=None, gid=None, env=None, rlimits=None, executable=None, use_fds=False, watcher=None, spawn=True, pipe_stdout=True, pipe_stderr=True, close_child_stdout=False, close_child_stderr=False) Wraps a process. .sp Options: .INDENT 7.0 .IP \(bu 2 \fBwid\fP: the process unique identifier. This value will be used to replace the \fI$WID\fP string in the command line if present. .IP \(bu 2 \fBcmd\fP: the command to run. May contain any of the variables available that are being passed to this class. They will be replaced using the python format syntax. .IP \(bu 2 \fBargs\fP: the arguments for the command to run. Can be a list or a string. If \fBargs\fP is a string, it\(aqs splitted using \fBshlex.split()\fP\&. Defaults to None. .IP \(bu 2 \fBexecutable\fP: When executable is given, the first item in the args sequence obtained from \fBcmd\fP is still treated by most programs as the command name, which can then be different from the actual executable name. It becomes the display name for the executing program in utilities such as \fBps\fP\&. .IP \(bu 2 \fBworking_dir\fP: the working directory to run the command in. If not provided, will default to the current working directory. .IP \(bu 2 \fBshell\fP: if \fITrue\fP, will run the command in the shell environment. \fIFalse\fP by default. \fBwarning: this is a security hazard\fP\&. .IP \(bu 2 \fBuid\fP: if given, is the user id or name the command should run with. The current uid is the default. .IP \(bu 2 \fBgid\fP: if given, is the group id or name the command should run with. The current gid is the default. .IP \(bu 2 \fBenv\fP: a mapping containing the environment variables the command will run with. Optional. .IP \(bu 2 \fBrlimits\fP: a mapping containing rlimit names and values that will be set before the command runs. .IP \(bu 2 \fBuse_fds\fP: if True, will not close the fds in the subprocess. Must be be set to True on Windows if stdout or stderr are redirected. default: False. .IP \(bu 2 \fBpipe_stdout\fP: if True, will open a PIPE on stdout. default: True. .IP \(bu 2 \fBpipe_stderr\fP: if True, will open a PIPE on stderr. default: True. .IP \(bu 2 \fBclose_child_stdout\fP: If True, redirects the child process\(aq stdout to /dev/null after the fork. default: False. .IP \(bu 2 \fBclose_child_stderr\fP: If True, redirects the child process\(aq stdout to /dev/null after the fork. default: False. .UNINDENT .INDENT 7.0 .TP .B age() Return the age of the process in seconds. .UNINDENT .INDENT 7.0 .TP .B children() Return a list of children pids. .UNINDENT .INDENT 7.0 .TP .B info() Return process info. .sp The info returned is a mapping with these keys: .INDENT 7.0 .IP \(bu 2 \fBmem_info1\fP: Resident Set Size Memory in bytes (RSS) .IP \(bu 2 \fBmem_info2\fP: Virtual Memory Size in bytes (VMS). .IP \(bu 2 \fBcpu\fP: % of cpu usage. .IP \(bu 2 \fBmem\fP: % of memory usage. .IP \(bu 2 \fBctime\fP: process CPU (user + system) time in seconds. .IP \(bu 2 \fBpid\fP: process id. .IP \(bu 2 \fBusername\fP: user name that owns the process. .IP \(bu 2 \fBnice\fP: process niceness (between \-20 and 20) .IP \(bu 2 \fBcmdline\fP: the command line the process was run with. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B is_child(pid) Return True is the given \fIpid\fP is a child of that process. .UNINDENT .INDENT 7.0 .TP .B pid Return the \fIpid\fP .UNINDENT .INDENT 7.0 .TP .B send_signal(*args, **kw) Sends a signal \fBsig\fP to the process. .UNINDENT .INDENT 7.0 .TP .B send_signal_child(*args, **kw) Send signal \fIsignum\fP to child \fIpid\fP\&. .UNINDENT .INDENT 7.0 .TP .B send_signal_children(*args, **kw) Send signal \fIsignum\fP to all children. .UNINDENT .INDENT 7.0 .TP .B status Return the process status as a constant .INDENT 7.0 .IP \(bu 2 RUNNING .IP \(bu 2 DEAD_OR_ZOMBIE .IP \(bu 2 UNEXISTING .IP \(bu 2 OTHER .UNINDENT .UNINDENT .INDENT 7.0 .TP .B stderr Return the \fIstdout\fP stream .UNINDENT .INDENT 7.0 .TP .B stdout Return the \fIstdout\fP stream .UNINDENT .INDENT 7.0 .TP .B stop(*args, **kw) Stop the process and close stdout/stderr .sp If the corresponding process is still here (normally it\(aqs already killed by the watcher), a SIGTERM is sent, then a SIGKILL after 1 second. .sp The shutdown process (SIGTERM then SIGKILL) is normally taken by the watcher. So if the process is still there here, it\(aqs a kind of bad behavior because the graceful timeout won\(aqt be respected here. .UNINDENT .UNINDENT .sp Example: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C >>> from circus.process import Process >>> process = Process(\(aqTop\(aq, \(aqtop\(aq, shell=True) >>> process.age() 3.0107998847961426 >>> process.info() \(aqTop: 6812 N/A tarek Zombie N/A N/A N/A N/A N/A\(aq >>> process.status 1 >>> process.stop() >>> process.status 2 >>> process.info() \(aqNo such process (stopped?)\(aq .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class circus.watcher.Watcher(name, cmd, args=None, numprocesses=1, warmup_delay=0.0, working_dir=None, shell=False, shell_args=None, uid=None, max_retry=5, gid=None, send_hup=False, stop_signal=15, stop_children=False, env=None, graceful_timeout=30.0, prereload_fn=None, rlimits=None, executable=None, stdout_stream=None, stderr_stream=None, priority=0, loop=None, singleton=False, use_sockets=False, copy_env=False, copy_path=False, max_age=0, max_age_variance=30, hooks=None, respawn=True, autostart=True, on_demand=False, virtualenv=None, close_child_stdout=False, close_child_stderr=False, virtualenv_py_ver=None, use_papa=False, **options) Class managing a list of processes for a given command. .sp Options: .INDENT 7.0 .IP \(bu 2 \fBname\fP: name given to the watcher. Used to uniquely identify it. .IP \(bu 2 \fBcmd\fP: the command to run. May contain \fI$WID\fP, which will be replaced by \fBwid\fP\&. .IP \(bu 2 \fBargs\fP: the arguments for the command to run. Can be a list or a string. If \fBargs\fP is a string, it\(aqs splitted using \fBshlex.split()\fP\&. Defaults to None. .IP \(bu 2 \fBnumprocesses\fP: Number of processes to run. .IP \(bu 2 \fBworking_dir\fP: the working directory to run the command in. If not provided, will default to the current working directory. .IP \(bu 2 \fBshell\fP: if \fITrue\fP, will run the command in the shell environment. \fIFalse\fP by default. \fBwarning: this is a security hazard\fP\&. .IP \(bu 2 \fBuid\fP: if given, is the user id or name the command should run with. The current uid is the default. .IP \(bu 2 \fBgid\fP: if given, is the group id or name the command should run with. The current gid is the default. .IP \(bu 2 \fBsend_hup\fP: if True, a process reload will be done by sending the SIGHUP signal. Defaults to False. .IP \(bu 2 \fBstop_signal\fP: the signal to send when stopping the process. Defaults to SIGTERM. .IP \(bu 2 \fBstop_children\fP: send the \fBstop_signal\fP to the children too. Defaults to False. .IP \(bu 2 \fBenv\fP: a mapping containing the environment variables the command will run with. Optional. .IP \(bu 2 \fBrlimits\fP: a mapping containing rlimit names and values that will be set before the command runs. .IP \(bu 2 \fBstdout_stream\fP: a mapping that defines the stream for the process stdout. Defaults to None. .sp Optional. When provided, \fIstdout_stream\fP is a mapping containing up to three keys: .INDENT 2.0 .IP \(bu 2 \fBclass\fP: the stream class. Defaults to \fIcircus.stream.FileStream\fP .IP \(bu 2 \fBfilename\fP: the filename, if using a FileStream .IP \(bu 2 \fBmax_bytes\fP: maximum file size, after which a new output file is opened. defaults to 0 which means no maximum size (only applicable with FileStream). .IP \(bu 2 \fBbackup_count\fP: how many backups to retain when rotating files according to the max_bytes parameter. defaults to 0 which means no backups are made (only applicable with FileStream) .UNINDENT .sp This mapping will be used to create a stream callable of the specified class. Each entry received by the callable is a mapping containing: .INDENT 2.0 .IP \(bu 2 \fBpid\fP \- the process pid .IP \(bu 2 \fBname\fP \- the stream name (\fIstderr\fP or \fIstdout\fP) .IP \(bu 2 \fBdata\fP \- the data .UNINDENT .sp This is not supported on Windows. .IP \(bu 2 \fBstderr_stream\fP: a mapping that defines the stream for the process stderr. Defaults to None. .sp Optional. When provided, \fIstderr_stream\fP is a mapping containing up to three keys: \- \fBclass\fP: the stream class. Defaults to \fIcircus.stream.FileStream\fP \- \fBfilename\fP: the filename, if using a FileStream \- \fBmax_bytes\fP: maximum file size, after which a new output file is .INDENT 2.0 .INDENT 3.5 opened. defaults to 0 which means no maximum size (only applicable with FileStream) .UNINDENT .UNINDENT .INDENT 2.0 .IP \(bu 2 \fBbackup_count\fP: how many backups to retain when rotating files according to the max_bytes parameter. defaults to 0 which means no backups are made (only applicable with FileStream). .UNINDENT .sp This mapping will be used to create a stream callable of the specified class. .sp Each entry received by the callable is a mapping containing: .INDENT 2.0 .IP \(bu 2 \fBpid\fP \- the process pid .IP \(bu 2 \fBname\fP \- the stream name (\fIstderr\fP or \fIstdout\fP) .IP \(bu 2 \fBdata\fP \- the data .UNINDENT .sp This is not supported on Windows. .IP \(bu 2 \fBpriority\fP \-\- integer that defines a priority for the watcher. When the Arbiter do some operations on all watchers, it will sort them with this field, from the bigger number to the smallest. (default: 0) .IP \(bu 2 \fBsingleton\fP \-\- If True, this watcher has a single process. (default:False) .IP \(bu 2 \fBuse_sockets\fP \-\- If True, the processes will inherit the file descriptors, thus can reuse the sockets opened by circusd. (default: False) .IP \(bu 2 \fBon_demand\fP \-\- If True, the processes will be started only at the first connection to the socket (default: False) .IP \(bu 2 \fBcopy_env\fP \-\- If True, the environment in which circus is running run will be reproduced for the workers. This defaults to True on Windows as you cannot run any executable without the \fBSYSTEMROOT\fP variable. (default: False) .IP \(bu 2 \fBcopy_path\fP \-\- If True, circusd \fIsys.path\fP is sent to the process through \fIPYTHONPATH\fP\&. You must activate \fBcopy_env\fP for \fBcopy_path\fP to work. (default: False) .IP \(bu 2 \fBmax_age\fP: If set after around max_age seconds, the process is replaced with a new one. (default: 0, Disabled) .IP \(bu 2 \fBmax_age_variance\fP: The maximum number of seconds that can be added to max_age. This extra value is to avoid restarting all processes at the same time. A process will live between max_age and max_age + max_age_variance seconds. .IP \(bu 2 \fBhooks\fP: callback functions for hooking into the watcher startup and shutdown process. \fBhooks\fP is a dict where each key is the hook name and each value is a 2\-tuple with the name of the callable or the callabled itself and a boolean flag indicating if an exception occuring in the hook should not be ignored. Possible values for the hook name: \fIbefore_start\fP, \fIafter_start\fP, \fIbefore_spawn\fP, \fIafter_spawn\fP, \fIbefore_stop\fP, \fIafter_stop\fP\&., \fIbefore_signal\fP, \fIafter_signal\fP or \fIextended_stats\fP\&. .IP \(bu 2 \fBoptions\fP \-\- extra options for the worker. All options found in the configuration file for instance, are passed in this mapping \-\- this can be used by plugins for watcher\-specific options. .IP \(bu 2 \fBrespawn\fP \-\- If set to False, the processes handled by a watcher will not be respawned automatically. (default: True) .IP \(bu 2 \fBvirtualenv\fP \-\- The root directory of a virtualenv. If provided, the watcher will load the environment for its execution. (default: None) .IP \(bu 2 \fBclose_child_stdout\fP: If True, closes the stdout after the fork. default: False. .IP \(bu 2 \fBclose_child_stderr\fP: If True, closes the stderr after the fork. default: False. .IP \(bu 2 \fBuse_papa\fP: If True, use the papa process kernel for this process. default: False. .UNINDENT .INDENT 7.0 .TP .B kill_process(*args, **kwargs) Kill process (stop_signal, graceful_timeout then SIGKILL) .UNINDENT .INDENT 7.0 .TP .B kill_processes(*args, **kwargs) Kill all processes (stop_signal, graceful_timeout then SIGKILL) .UNINDENT .INDENT 7.0 .TP .B manage_processes(*args, **kwargs) Manage processes. .UNINDENT .INDENT 7.0 .TP .B notify_event(topic, msg) Publish a message on the event publisher channel .UNINDENT .INDENT 7.0 .TP .B reap_and_manage_processes(*args, **kwargs) Reap & manage processes. .UNINDENT .INDENT 7.0 .TP .B reap_processes(*args, **kw) Reap all the processes for this watcher. .UNINDENT .INDENT 7.0 .TP .B send_signal_child(*args, **kw) Send signal to a child. .UNINDENT .INDENT 7.0 .TP .B spawn_process(recovery_wid=None) Spawn process. .sp Return True if ok, False if the watcher must be stopped .UNINDENT .INDENT 7.0 .TP .B spawn_processes(*args, **kwargs) Spawn processes. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class circus.arbiter.Arbiter(watchers, endpoint, pubsub_endpoint, check_delay=1.0, prereload_fn=None, context=None, loop=None, statsd=False, stats_endpoint=None, statsd_close_outputs=False, multicast_endpoint=None, plugins=None, sockets=None, warmup_delay=0, httpd=False, httpd_host=\(aqlocalhost\(aq, httpd_port=8080, httpd_close_outputs=False, debug=False, debug_gc=False, ssh_server=None, proc_name=\(aqcircusd\(aq, pidfile=None, loglevel=None, logoutput=None, loggerconfig=None, fqdn_prefix=None, umask=None, endpoint_owner=None, papa_endpoint=None) Class used to control a list of watchers. .sp Options: .INDENT 7.0 .IP \(bu 2 \fBwatchers\fP \-\- a list of Watcher objects .IP \(bu 2 \fBendpoint\fP \-\- the controller ZMQ endpoint .IP \(bu 2 \fBpubsub_endpoint\fP \-\- the pubsub endpoint .IP \(bu 2 \fBstatsd\fP \-\- If True, a circusd\-stats process is run (default: False) .IP \(bu 2 \fBstats_endpoint\fP \-\- the stats endpoint. .IP \(bu 2 \fBstatsd_close_outputs\fP \-\- if True sends the circusd\-stats stdout/stderr to /dev/null (default: False) .IP \(bu 2 \fBmulticast_endpoint\fP \-\- the multicast endpoint for circusd cluster auto\-discovery (default: udp://237.219.251.97:12027) Multicast addr should be between 224.0.0.0 to 239.255.255.255 and the same for the all cluster. .IP \(bu 2 \fBcheck_delay\fP \-\- the delay between two controller points (default: 1 s) .IP \(bu 2 \fBprereload_fn\fP \-\- callable that will be executed on each reload (default: None) .IP \(bu 2 \fBcontext\fP \-\- if provided, the zmq context to reuse. (default: None) .IP \(bu 2 .INDENT 2.0 .TP .B \fBloop\fP: if provided, a \fBzmq.eventloop.ioloop.IOLoop\fP instance to reuse. (default: None) .UNINDENT .IP \(bu 2 \fBplugins\fP \-\- a list of plugins. Each item is a mapping with: .INDENT 2.0 .INDENT 3.5 .INDENT 0.0 .IP \(bu 2 \fBuse\fP \-\- Fully qualified name that points to the plugin class .IP \(bu 2 every other value is passed to the plugin in the \fBconfig\fP option .UNINDENT .UNINDENT .UNINDENT .IP \(bu 2 \fBsockets\fP \-\- a mapping of sockets. Each key is the socket name, and each value a \fBCircusSocket\fP class. (default: None) .IP \(bu 2 \fBwarmup_delay\fP \-\- a delay in seconds between two watchers startup. (default: 0) .IP \(bu 2 \fBhttpd\fP \-\- If True, a circushttpd process is run (default: False) .IP \(bu 2 \fBhttpd_host\fP \-\- the circushttpd host (default: localhost) .IP \(bu 2 \fBhttpd_port\fP \-\- the circushttpd port (default: 8080) .IP \(bu 2 \fBhttpd_close_outputs\fP \-\- if True, sends circushttpd stdout/stderr to /dev/null. (default: False) .IP \(bu 2 \fBdebug\fP \-\- if True, adds a lot of debug info in the stdout (default: False) .IP \(bu 2 \fBdebug_gc\fP \-\- if True, does gc.set_debug(gc.DEBUG_LEAK) (default: False) to circusd to analyze problems (default: False) .IP \(bu 2 \fBproc_name\fP \-\- the arbiter process name .IP \(bu 2 .INDENT 2.0 .TP .B \fBfqdn_prefix\fP \-\- a prefix for the unique identifier of the circus instance on the cluster. .UNINDENT .IP \(bu 2 \fBendpoint_owner\fP \-\- unix user to chown the endpoint to if using ipc. .IP \(bu 2 \fBpapa_endpoint\fP \-\- the papa process kernel endpoint .UNINDENT .INDENT 7.0 .TP .B add_watcher(*args, **kwargs) Adds a watcher. .sp Options: .INDENT 7.0 .IP \(bu 2 \fBname\fP: name of the watcher to add .IP \(bu 2 \fBcmd\fP: command to run. .IP \(bu 2 all other options defined in the Watcher constructor. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_watcher(name) Return the watcher \fIname\fP\&. .UNINDENT .INDENT 7.0 .TP .B numprocesses() Return the number of processes running across all watchers. .UNINDENT .INDENT 7.0 .TP .B numwatchers() Return the number of watchers. .UNINDENT .INDENT 7.0 .TP .B reload(*args, **kwargs) Reloads everything. .sp Run the \fBprereload_fn()\fP callable if any, then gracefuly reload all watchers. .UNINDENT .INDENT 7.0 .TP .B start(*args, **kwargs) Starts all the watchers. .sp If the ioloop has been provided during __init__() call, starts all watchers as a standard coroutine .sp If the ioloop hasn\(aqt been provided during __init__() call (default), starts all watchers and the eventloop (and blocks here). In this mode the method MUST NOT yield anything because it\(aqs called as a standard method. .INDENT 7.0 .TP .B Parameters \fBcb\fP \-\- Callback called after all the watchers have been started, when the loop hasn\(aqt been provided. .UNINDENT .UNINDENT .UNINDENT .SS Writing plugins .sp Circus comes with a plugin system which lets you interact with \fBcircusd\fP\&. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 We might add circusd\-stats support to plugins later on. .UNINDENT .UNINDENT .sp A Plugin is composed of two parts: .INDENT 0.0 .IP \(bu 2 a ZMQ subscriber to all events published by \fBcircusd\fP .IP \(bu 2 a ZMQ client to send commands to \fBcircusd\fP .UNINDENT .sp Each plugin is run as a separate process under a custom watcher. .sp A few examples of some plugins you could create with this system: .INDENT 0.0 .IP \(bu 2 a notification system that sends e\-mail alerts when a watcher is flapping .IP \(bu 2 a logger .IP \(bu 2 a tool that adds or removes processes depending on the load .IP \(bu 2 etc. .UNINDENT .sp Circus itself comes with a few built\-in plugins\&. .SS The CircusPlugin class .sp Circus provides a base class to help you implement plugins: \fI\%circus.plugins.CircusPlugin\fP .INDENT 0.0 .TP .B class circus.plugins.CircusPlugin(endpoint, pubsub_endpoint, check_delay, ssh_server=None, **config) Base class to write plugins. .sp Options: .INDENT 7.0 .IP \(bu 2 \fBcontext\fP \-\- the ZMQ context to use .IP \(bu 2 \fBendpoint\fP \-\- the circusd ZMQ endpoint .IP \(bu 2 \fBpubsub_endpoint\fP \-\- the circusd ZMQ pub/sub endpoint .IP \(bu 2 \fBcheck_delay\fP \-\- the configured check delay .IP \(bu 2 \fBconfig\fP \-\- free config mapping .UNINDENT .INDENT 7.0 .TP .B call(command, **props) Sends the command to \fBcircusd\fP .sp Options: .INDENT 7.0 .IP \(bu 2 \fBcommand\fP \-\- the command to call .IP \(bu 2 \fBprops\fP \-\- keyword arguments to add to the call .UNINDENT .sp Returns the JSON mapping sent back by \fBcircusd\fP .UNINDENT .INDENT 7.0 .TP .B cast(command, **props) Fire\-and\-forget a command to \fBcircusd\fP .sp Options: .INDENT 7.0 .IP \(bu 2 \fBcommand\fP \-\- the command to call .IP \(bu 2 \fBprops\fP \-\- keyword arguments to add to the call .UNINDENT .UNINDENT .INDENT 7.0 .TP .B handle_init() Called right before a plugin is started \- in the thread context. .UNINDENT .INDENT 7.0 .TP .B handle_recv(data) Receives every event published by \fBcircusd\fP .sp Options: .INDENT 7.0 .IP \(bu 2 \fBdata\fP \-\- a tuple containing the topic and the message. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B handle_stop() Called right before the plugin is stopped by Circus. .UNINDENT .UNINDENT .sp When initialized by Circus, this class creates its own event loop that receives all \fBcircusd\fP events and pass them to \fBhandle_recv()\fP\&. The data received is a tuple containing the topic and the data itself. .sp \fBhandle_recv()\fP \fBmust\fP be implemented by the plugin. .sp The \fBcall()\fP and \fBcast()\fP methods can be used to interact with \fBcircusd\fP if you are building a Plugin that actively interacts with the daemon. .sp \fBhandle_init()\fP and \fBhandle_stop()\fP are just convenience methods you can use to initialize and clean up your code. \fBhandle_init()\fP is called within the thread that just started. \fBhandle_stop()\fP is called in the main thread just before the thread is stopped and joined. .SS Writing a plugin .sp Let\(aqs write a plugin that logs in a file every event happening in \fBcircusd\fP\&. It takes one argument which is the filename. .sp The plugin may look like this: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C from circus.plugins import CircusPlugin class Logger(CircusPlugin): name = \(aqlogger\(aq def __init__(self, *args, **config): super(Logger, self).__init__(*args, **config) self.filename = config.get(\(aqfilename\(aq) self.file = None def handle_init(self): self.file = open(self.filename, \(aqa+\(aq, buffering=1) def handle_stop(self): self.file.close() def handle_recv(self, data): watcher_name, action, msg = self.split_data(data) msg_dict = self.load_message(msg) self.file.write(\(aq%s %s::%r\en\(aq % (action, watcher_name, msg_dict)) .ft P .fi .UNINDENT .UNINDENT .sp That\(aqs it ! This class can be saved in any package/module, as long as it can be seen by Python. .sp For example, \fBLogger\fP may be found in a \fIplugins\fP module within a \fImyproject\fP package. .SS Async requests .sp In case you want to make any asynchronous operations (like a Tornado call or using periodicCall) make sure you are using the right loop. The loop you always want to be using is self.loop as it gets set up by the base class. The default loop often isn\(aqt the same and therefore code might not get executed as expected. .SS Trying a plugin .sp You can run a plugin through the command line with the \fBcircus\-plugin\fP command, by specifying the plugin fully qualified name: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circus\-plugin \-\-endpoint tcp://127.0.0.1:5555 \-\-pubsub tcp://127.0.0.1:5556 \-\-config filename:circus\-events.log myproject.plugins.Logger [INFO] Loading the plugin... [INFO] Endpoint: \(aqtcp://127.0.0.1:5555\(aq [INFO] Pub/sub: \(aqtcp://127.0.0.1:5556\(aq [INFO] Starting .ft P .fi .UNINDENT .UNINDENT .sp Another way to run a plugin is to let Circus handle its initialization. This is done by adding a \fB[plugin:NAME]\fP section in the configuration file, where \fINAME\fP is a unique name for your plugin: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [plugin:logger] use = myproject.plugins.Logger filename = /var/myproject/circus.log .ft P .fi .UNINDENT .UNINDENT .sp \fBuse\fP is mandatory and points to the fully qualified name of the plugin. .sp When Circus starts, it creates a watcher with one process that runs the pointed class, and pass any other variable contained in the section to the plugin constructor via the \fBconfig\fP mapping. .sp You can also programmatically add plugins when you create a \fBcircus.arbiter.Arbiter\fP class or use \fBcircus.get_arbiter()\fP, see library\&. .SS Performances .sp Since every plugin is loaded in its own process, it should not impact the overall performances of the system as long as the work done by the plugin is not doing too many calls to the \fBcircusd\fP process. .SS Hooks .sp Circus provides hooks that can be used to trigger actions upon watcher events. Available hooks are: .INDENT 0.0 .IP \(bu 2 \fBbefore_start\fP: called before the watcher is started. If the hook returns \fBFalse\fP the startup is aborted. .IP \(bu 2 \fBafter_start\fP: called after the watcher is started. If the hook returns \fBFalse\fP the watcher is immediately stopped and the startup is aborted. .IP \(bu 2 \fBbefore_spawn\fP: called before the watcher spawns a new process. If the hook returns \fBFalse\fP the watcher is immediately stopped and the startup is aborted. .IP \(bu 2 \fBafter_spawn\fP: called after the watcher spawns a new process. If the hook returns \fBFalse\fP the watcher is immediately stopped and the startup is aborted. .IP \(bu 2 \fBbefore_stop\fP: called before the watcher is stopped. The hook result is ignored. .IP \(bu 2 \fBafter_stop\fP: called after the watcher is stopped. The hook result is ignored. .IP \(bu 2 \fBbefore_signal\fP: called before a signal is sent to a watcher\(aqs process. If the hook returns \fBFalse\fP the signal is not sent (except SIGKILL which is always sent) .IP \(bu 2 \fBafter_signal\fP: called after a signal is sent to a watcher\(aqs process. .IP \(bu 2 \fBextended_stats\fP: called when stats are requested with extended=True. Used for adding process\-specific stats to the regular stats output. .UNINDENT .SS Example .sp A typical use case is to control that all the conditions are met for a process to start. Let\(aqs say you have a watcher that runs \fIRedis\fP and a watcher that runs a Python script that works with \fIRedis\fP\&. With Circus you can order the startup by using the \fBpriority\fP option: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [watcher:queue\-worker] cmd = python \-u worker.py priority = 1 [watcher:redis] cmd = redis\-server priority = 2 .ft P .fi .UNINDENT .UNINDENT .sp With this setup, Circus will start \fIRedis\fP first and then it will start the queue worker. But Circus does not really control that \fIRedis\fP is up and running. It just starts the process it was asked to start. What we miss here is a way to control that \fIRedis\fP is started and fully functional. A function that controls this could be: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C import redis import time def check_redis(*args, **kw): time.sleep(.5) # give it a chance to start r = redis.StrictRedis(host=\(aqlocalhost\(aq, port=6379, db=0) r.set(\(aqfoo\(aq, \(aqbar\(aq) return r.get(\(aqfoo\(aq) == \(aqbar\(aq .ft P .fi .UNINDENT .UNINDENT .sp This function can be plugged into Circus as an \fBbefore_start\fP hook: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [watcher:queue\-worker] cmd = python \-u worker.py hooks.before_start = mycoolapp.myplugins.check_redis priority = 1 [watcher:redis] cmd = redis\-server priority = 2 .ft P .fi .UNINDENT .UNINDENT .sp Once Circus has started the \fBredis\fP watcher, it will start the \fBqueue\-worker\fP watcher, since it follows the \fBpriority\fP ordering. Just before starting the second watcher, it will run the \fBcheck_redis\fP function, and in case it returns \fBFalse\fP will abort the watcher starting process. .SS Hook signature .sp A hook must follow this signature: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C def hook(watcher, arbiter, hook_name, **kwargs): ... # If you don\(aqt return True, the hook can change # the behavior of circus (depending on the hook) return True .ft P .fi .UNINDENT .UNINDENT .sp Where \fBwatcher\fP is the \fBWatcher\fP class instance, \fBarbiter\fP the \fBArbiter\fP one, \fBhook_name\fP the hook name and \fBkwargs\fP some additional optional parameters (depending on the hook type). .sp The \fBafter_spawn\fP hook adds the pid parameters: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C def after_spawn(watcher, arbiter, hook_name, pid, **kwargs): ... # If you don\(aqt return True, circus will kill the process return True .ft P .fi .UNINDENT .UNINDENT .sp Where \fBpid\fP is the PID of the corresponding process. .sp Likewise, \fBbefore_signal\fP and \fBafter_signal\fP hooks add pid and signum: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C def before_signal_hook(watcher, arbiter, hook_name, pid, signum, **kwargs): ... # If you don\(aqt return True, circus won\(aqt send the signum signal # (SIGKILL is always sent) return True .ft P .fi .UNINDENT .UNINDENT .sp Where \fBpid\fP is the PID of the corresponding process and \fBsignum\fP is the corresponding signal. .sp You can ignore those but being able to use the watcher and/or arbiter data and methods can be useful in some hooks. .sp Note that hooks are called with named arguments. So use the hook signature without changing argument names. .sp The \fBextended_stats\fP hook has its own additional parameters in \fBkwargs\fP: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C def extended_stats_hook(watcher, arbiter, hook_name, pid, stats, **kwargs): ... .ft P .fi .UNINDENT .UNINDENT .sp Where \fBpid\fP is the PID of the corresponding process and \fBstats\fP the regular stats to be returned. Add your own stats into \fBstats\fP\&. An example is in examples/uwsgi_lossless_reload.py. .sp As a last example, here is a super hook which can deal with all kind of signals: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C def super_hook(watcher, arbiter, hook_name, **kwargs): pid = None signum = None if hook_name in (\(aqbefore_signal\(aq, \(aqafter_signal\(aq): pid = kwargs[\(aqpid\(aq] signum = kwargs[\(aqsignum\(aq] ... return True .ft P .fi .UNINDENT .UNINDENT .SS Hook events .sp Everytime a hook is run, its result is notified as an event in Circus. .sp There are two events related to hooks: .INDENT 0.0 .IP \(bu 2 \fBhook_success\fP: a hook was successfully called. The event keys are \fBname\fP the name if the event, and \fBtime\fP: the date of the events. .IP \(bu 2 \fBhook_failure\fP: a hook has failed. The event keys are \fBname\fP the name if the event, \fBtime\fP: the date of the events and \fBerror\fP: the exception that occurred in the event, if any. .UNINDENT .SS Adding new commands .sp We tried to make adding new commands as simple as possible. .sp You need to do three things: .INDENT 0.0 .IP 1. 3 create a \fByour_command.py\fP file under \fBcircus/commands/\fP\&. .IP 2. 3 Implement a single class in there, with predefined methods .IP 3. 3 Add the new command in \fBcircus/commands/__init__.py\fP\&. .UNINDENT .sp Let\(aqs say we want to add a command which returns the number of watchers currently in use, we would do something like this (extensively commented to allow you to follow more easily): .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C from circus.commands.base import Command from circus.exc import ArgumentError, MessageError class NumWatchers(Command): """It is a good practice to describe what the class does here. Have a look at other commands to see how we are used to format this text. It will be automatically included in the documentation, so don\(aqt be affraid of being exhaustive, that\(aqs what it is made for. """ # all the commands inherit from \(gacircus.commands.base.Command\(ga # you need to specify a name so we find back the command somehow name = "numwatchers" # Set waiting to True or False to define your default behavior # \- If waiting is True, the command is run synchronously, and the client may get # back results. # \- If waiting is False, the command is run asynchronously on the server and the client immediately # gets back an \(aqok\(aq response # # By default, commands are set to waiting = False waiting = True # options options = [(\(aq\(aq, \(aqoptname\(aq, default_value, \(aqdescription\(aq)] properties = [\(aqfoo\(aq, \(aqbar\(aq] # properties list the command arguments that are mandatory. If they are # not provided, then an error will be thrown def execute(self, arbiter, props): # the execute method is the core of the command: put here all the # logic of the command and return a dict containing the values you # want to return, if any return {"numwatchers": arbiter.numwatchers()} def console_msg(self, msg): # msg is what is returned by the execute method. # this method is used to format the response for a console (it is # used for instance by circusctl to print its messages) return "a string that will be displayed" def message(self, *args, **opts): # message handles console input. # this method is used to map console arguments to the command # options. (its is used for instance when calling the command via # circusctl) # NotImplementedError will be thrown if the function is missing numArgs = 1 if not len(args) == numArgs: raise ArgumentError(\(aqInvalid number of arguments.\(aq) else: opts[\(aqoptname\(aq] = args[0] return self.make_message(**opts) def validate(self, props): # this method is used to validate that the arguments passed to the # command are correct. An ArgumentError should be thrown in case # there is an error in the passed arguments (for instance if they # do not match together. # In case there is a problem wrt their content, a MessageError # should be thrown. This method can modify the content of the props # dict, it will be passed to execute afterwards. .ft P .fi .UNINDENT .UNINDENT .SS Use cases examples .sp This chapter presents a few use cases, to give you an idea on how to use Circus in your environment. .SS Running a WSGI application .sp Running a WSGI application with Circus is quite interesting because you can watch & manage your \fIweb workers\fP using \fIcircus\-top\fP, \fIcircusctl\fP or the Web interface. .sp This is made possible by using Circus sockets. See whycircussockets\&. .sp Let\(aqs take an example with a minimal \fI\%Pyramid\fP application: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C from pyramid.config import Configurator from pyramid.response import Response def hello_world(request): return Response(\(aqHello %(name)s!\(aq % request.matchdict) config = Configurator() config.add_route(\(aqhello\(aq, \(aq/hello/{name}\(aq) config.add_view(hello_world, route_name=\(aqhello\(aq) application = config.make_wsgi_app() .ft P .fi .UNINDENT .UNINDENT .sp Save this script into an \fBapp.py\fP file, then install those projects: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ pip install Pyramid $ pip install chaussette .ft P .fi .UNINDENT .UNINDENT .sp Next, make sure you can run your Pyramid application using the \fBchaussette\fP console script: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ chaussette app.application Application is Serving on localhost:8080 Using as a backend .ft P .fi .UNINDENT .UNINDENT .sp And check that you can reach it by visiting \fBhttp://localhost:8080/hello/tarek\fP .sp Now that your application is up and running, let\(aqs create a Circus configuration file: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [circus] check_delay = 5 endpoint = tcp://127.0.0.1:5555 pubsub_endpoint = tcp://127.0.0.1:5556 stats_endpoint = tcp://127.0.0.1:5557 [watcher:webworker] cmd = chaussette \-\-fd $(circus.sockets.webapp) app.application use_sockets = True numprocesses = 3 [socket:webapp] host = 127.0.0.1 port = 8080 .ft P .fi .UNINDENT .UNINDENT .sp This file tells Circus to bind a socket on port \fI8080\fP and run \fIchaussette\fP workers on that socket \-\- by passing its fd. .sp Save it to \fIserver.ini\fP and try to run it using \fBcircusd\fP .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circusd server.ini [INFO] Starting master on pid 8971 [INFO] sockets started [INFO] circusd\-stats started [INFO] webapp started [INFO] Arbiter now waiting for commands .ft P .fi .UNINDENT .UNINDENT .sp Make sure you still get the app on \fBhttp://localhost:8080/hello/tarek\fP\&. .sp Congrats ! you have a WSGI application running 3 workers. .sp You can run the circushttpd or the cli, and enjoy Circus management. .SS Running a Django application .sp Running a Django application is done exactly like running a WSGI application. Use the \fIPYTHONPATH\fP to import the directory the project is in, the directory that contains the directory that has settings.py in it (with Django 1.4+ this directory has manage.py in it) : .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [socket:dwebapp] host = 127.0.0.1 port = 8080 [watcher:dwebworker] cmd = chaussette \-\-fd $(circus.sockets.dwebapp) dproject.wsgi.application use_sockets = True numprocesses = 2 [env:dwebworker] PYTHONPATH = /path/to/parent\-of\-dproject .ft P .fi .UNINDENT .UNINDENT .sp If you need to pass the \fIDJANGO_SETTINGS_MODULE\fP for a backend worker for example, you can pass that also though the \fIenv\fP configation option: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [watcher:dbackend] cmd = /path/to/script.py numprocesses=3 [env:dbackend] PYTHONPATH = /path/to/parent\-of\-dproject DJANGO_SETTINGS_MODULE=dproject.settings .ft P .fi .UNINDENT .UNINDENT .sp See \fI\%http://chaussette.readthedocs.org\fP for more about chaussette. .SS Design decisions .SS Overall architecture [image] .sp Circus is composed of a main process called \fBcircusd\fP which takes care of running all the processes. Each process managed by Circus is a child process of \fBcircusd\fP\&. .sp Processes are organized in groups called \fBwatchers\fP\&. A \fBwatcher\fP is basically a command \fBcircusd\fP runs on your system, and for each command you can configure how many processes you want to run. .sp The concept of \fIwatcher\fP is useful when you want to manage all the processes running the same command \-\- like restart them, etc. .sp \fBcircusd\fP binds two ZeroMQ sockets: .INDENT 0.0 .IP \(bu 2 \fBREQ/REP\fP \-\- a socket used to control \fBcircusd\fP using json\-based \fIcommands\fP\&. .IP \(bu 2 \fBPUB/SUB\fP \-\- a socket where \fBcircusd\fP publishes events, like when a process is started or stopped. .UNINDENT .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Despite its name, ZeroMQ is not a queue management system. Think of it as an inter\-process communication (IPC) library. .UNINDENT .UNINDENT .sp Another process called \fBcircusd\-stats\fP is run by \fBcircusd\fP when the option is activated. \fBcircusd\-stats\fP\(aqs job is to publish CPU/Memory usage statistics in a dedicated \fBPUB/SUB\fP channel. .sp This specialized channel is used by \fBcircus\-top\fP and \fBcircus\-httpd\fP to display a live stream of the activity. .sp \fBcircus\-top\fP is a console script that mimics \fBtop\fP to display all the CPU and Memory usage of the processes managed by Circus. .sp \fBcircus\-httpd\fP is the web managment interface that will let you interact with Circus. It displays a live stream using web sockets and the \fBcircusd\-stats\fP channel, but also let you interact with \fBcircusd\fP via its \fBREQ/REP\fP channel. .sp Last but not least, \fBcircusctl\fP is a command\-line tool that let you drive \fBcircusd\fP via its \fBREQ/REP\fP channel. .sp You can also have plugins that subscribe to \fBcircusd\fP\(aqs \fBPUB/SUB\fP channel and let you send commands to the \fBREQ/REP\fP channel like \fBcircusctl\fP would. .SS Security .sp Circus is built on the top of the ZeroMQ library and comes with no security at all in its protocols. However, you can run a Circus system on a server and set up an SSH tunnel to access it from another machine. .sp This section explains what Circus does on your system when you run it, and ends up describing how to use an SSH tunnel. .sp You can also read \fI\%http://www.zeromq.org/area:faq#toc5\fP .SS TCP ports .sp By default, Circus opens the following TCP ports on the local host: .INDENT 0.0 .IP \(bu 2 \fB5555\fP \-\- the port used to control circus via \fBcircusctl\fP .IP \(bu 2 \fB5556\fP \-\- the port used for the Publisher/Subscriber channel. .IP \(bu 2 \fB5557\fP \-\- the port used for the statistics channel \-\- if activated. .IP \(bu 2 \fB8080\fP \-\- the port used by the Web UI \-\- if activated. .UNINDENT .sp These ports allow client apps to interact with your Circus system, and depending on how your infrastructure is organized, you may want to protect these ports via firewalls \fBor\fP configure Circus to run using \fBIPC\fP ports. .sp Here\(aqs an example of running Circus using only IPC entry points: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [circus] check_delay = 5 endpoint = ipc:///var/circus/endpoint pubsub_endpoint = ipc:///var/circus/pubsub stats_endpoint = ipc:///var/circus/stats .ft P .fi .UNINDENT .UNINDENT .sp When Configured using IPC, the commands must be run from the same box, but no one can access them from outside, unlike using TCP. The commands must also be run as a user that has write access to the ipc socket paths. You can modify the owner of the \fBendpoint\fP using the \fBendpoint_owner\fP config option. This allows you to run circusd as the root user, but allow non\-root processes to send commands to \fBcircusd\fP\&. Note that when using \fBendpoint_owner\fP, in order to prevent non\-root processes from being able to start arbitrary processes that run with greater privileges, the add command will enforce that new Watchers must run as the \fBendpoint_owner\fP user. Watcher definitions in the local config files will not be restricted this way. .sp Of course, if you activate the Web UI, the \fB8080\fP port will still be open. .SS circushttpd .sp When you run \fBcircushttpd\fP manually, or when you use the \fBhttpd\fP option in the ini file like this: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [circus] check_delay = 5 endpoint = ipc:///var/circus/endpoint pubsub_endpoint = ipc:///var/circus/pubsub stats_endpoint = ipc:///var/circus/stats httpd = 1 .ft P .fi .UNINDENT .UNINDENT .sp The web application will run on port \fI8080\fP and will let anyone accessing the web page manage the \fBcircusd\fP daemon. .sp That includes creating new watchers that can run any command on your system ! .sp \fBDo not make it publicly available\fP .sp If you want to protect the access to the web panel, you can serve it behind Nginx or Apache or any proxy\-capable web server, that can take care of the security. .SS User and Group Permissions .sp By default, all processes started with Circus will be running with the same user and group as \fBcircusd\fP\&. Depending on the privileges the user has on the system, you may not have access to all the features Circus provides. .sp For instance, some statistics features on a running processes require extended privileges. Typically, if the CPU usage numbers you get using the \fBstats\fP command are \fIN/A\fP, it means your user can\(aqt access the proc files. This will be the case by default under Mac OS X. .sp You may run \fBcircusd\fP as root to fix this, and set the \fBuid\fP and \fBgid\fP values for each watcher to get all the features. .sp But beware that running \fBcircusd\fP as root exposes you to potential privilege escalation bugs. While we\(aqre doing our best to avoid any bugs, running as root and facing a bug that performs unwanted actions on your system may be dangerous. .sp The best way to prevent this is to make sure that the system running Circus is completely isolated (like a VM) \fBor\fP to run the whole system under a controlled user. .SS SSH tunneling .sp Clients can connect to a \fBcircusd\fP instance by creating an SSH tunnel. To do so, pass the command line option \fB\-\-ssh\fP followed by \fBuser@address\fP, where \fBuser\fP is the user on the remote server and \fBaddress\fP is the server\(aqs address as seen by the client. The SSH protocol will require credentials to complete the login. .sp If \fBcircusd\fP as seen by the SSH server is not at the default endpoint address \fBlocalhost:5555\fP then specify the \fBcircusd\fP address using the option \fB\-\-endpoint\fP .SS Secured setup example .sp Setting up a secured Circus server can be done by: .INDENT 0.0 .IP \(bu 2 Running an SSH Server .IP \(bu 2 Running Apache or Nginx on the \fI80\fP port, and doing a reverse\-proxy on the \fI8080\fP port. .IP \(bu 2 Blocking the \fI8080\fP port from outside access. .IP \(bu 2 Running all ZMQ Circusd ports using IPC files instead of TCP ports, and tunneling all calls via SSH. .UNINDENT [image] .SS Contributing to Circus .sp Circus has been started at Mozilla but its goal is not to stay only there. We\(aqre trying to build a tool that\(aqs useful for others, and easily extensible. .sp We really are open to any contributions, in the form of code, documentation, discussions, feature proposal etc. .sp You can start a topic in our mailing list : \fI\%http://tech.groups.yahoo.com/group/circus\-dev/\fP .sp Or add an issue in our \fI\%bug tracker\fP .SS Fixing typos and enhancing the documentation .sp It\(aqs totally possible that your eyes are bleeding while reading this half\-english half\-french documentation, don\(aqt hesitate to contribute any rephrasing / enhancement on the form in the documentation. You probably don\(aqt even need to understand how Circus works under the hood to do that. .SS Adding new features .sp New features are of course very much appreciated. If you have the need and the time to work on new features, adding them to Circus shouldn\(aqt be that complicated. We tried very hard to have a clean and understandable API, hope it serves the purpose. .sp You will need to add documentation and tests alongside with the code of the new feature. Otherwise we\(aqll not be able to accept the patch. .SS How to submit your changes .sp We\(aqre using git as a DVCS. The best way to propose changes is to create a branch on your side (via \fIgit checkout \-b branchname\fP) and commit your changes there. Once you have something ready for prime\-time, issue a pull request against this branch. .sp We are following this model to allow to have low coupling between the features you are proposing. For instance, we can accept one pull request while still being in discussion for another one. .sp Before proposing your changes, double check that they are not breaking anything! You can use the \fItox\fP command to ensure this, it will run the testsuite under the different supported python versions. .sp Please use : \fI\%http://issue2pr.herokuapp.com/\fP to reference a commit to an existing circus issue, if any. .SS Avoiding merge commits .sp Avoiding merge commits allows to have a clean and readable history. To do so, instead of doing "git pull" and letting git handling the merges for you, using git pull \-\-rebase will put your changes after the changes that are commited in the branch, or when working on master. .sp That is, for us core developers, it\(aqs not possible anymore to use the handy github green button on pull requests if developers didn\(aqt rebased their work themselves or if we wait too much time between the request and the actual merge. Instead, the flow looks like this: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C git remote add name repo\-url git fetch name git checkout feature\-branch git rebase master # check that everything is working properly and then merge on master git checkout master git merge feature\-branch .ft P .fi .UNINDENT .UNINDENT .SS Discussing .sp If you find yourself in need of any help while looking at the code of Circus, you can go and find us on irc at \fI\%#circus\-tent on irc.freenode.org\fP (or if you don\(aqt have any IRC client, use \fI\%the webchat\fP) .sp You can also start a thread in our mailing list \- \fI\%http://tech.groups.yahoo.com/group/circus\-dev\fP .SS Frequently Asked Questions .sp Here is a list of frequently asked questions about Circus: .SS How does Circus stack compare to a classical stack? .sp In a classical WSGI stack, you have a server like Gunicorn that serves on a port or an unix socket and is usually deployed behind a web server like Nginx: [image] .sp Clients call Nginx, which reverse proxies all the calls to Gunicorn. .sp If you want to make sure the Gunicorn process stays up and running, you have to use a program like Supervisord or upstart. .sp Gunicorn in turn watches for its processes ("workers"). .sp In other words you are using two levels of process managment. One that you manage and control (supervisord), and a second one that you have to manage in a different UI, with a different philosophy and less control over what\(aqs going on (the wsgi server\(aqs one) .sp This is true for Gunicorn and most multi\-processes WSGI servers out there I know about. uWsgi is a bit different as it offers plethoras of options. .sp But if you want to add a Redis server in your stack, you \fIwill\fP end up with managing your stack processes in two different places. .sp Circus\(aq approach on this is to manage processes \fIand\fP sockets. .sp A Circus stack can look like this: [image] .sp So, like Gunicorn, Circus is able to bind a socket that will be proxied by Nginx. Circus don\(aqt deal with the requests but simply binds the socket. It\(aqs then up to a web worker process to accept connections on the socket and do the work. .sp It provides equivalent features than Supervisord but will also let you manage all processes at the same level, wether they are web workers or Redis or whatever. Adding a new web worker is done exactly like adding a new Redis process. .SS Benches .sp We did a few benches to compare Circus & Chaussette with Gunicorn. To summarize, Circus is not adding any overhead and you can pick up many different backends for your web workers. .sp See: .INDENT 0.0 .IP \(bu 2 \fI\%http://blog.ziade.org/2012/06/28/wgsi\-web\-servers\-bench\fP .IP \(bu 2 \fI\%http://blog.ziade.org/2012/07/03/wsgi\-web\-servers\-bench\-part\-2\fP .UNINDENT .SS How to troubleshoot Circus? .sp By default, \fIcircusd\fP keeps its logging to \fIstdout\fP rather sparse. This lack of output can make things hard to troubleshoot when processes seem to be having trouble starting. .sp To increase the logging \fIcircusd\fP provides, try increasing the log level. To see the available log levels just use the \fI\-\-help\fP flag. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ circus \-\-log\-level debug test.ini .ft P .fi .UNINDENT .UNINDENT .sp One word of warning. If a process is flapping and the debug log level is turned on, you will see messages for each start attempt. It might be helpful to configure the app that is flapping to use a \fIwarmup_delay\fP to slow down the messages to a manageable pace. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [watcher:webapp] cmd = python \-m myapp.wsgi warmup_delay = 5 .ft P .fi .UNINDENT .UNINDENT .sp By default, \fIstdout\fP and \fIstderr\fP are captured by the \fIcircusd\fP process. If you are testing your config and want to see the output in line with the circusd output, you can configure your watcher to use the \fIStdoutStream\fP class. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C [watcher:webapp] cmd = python \-m myapp.wsgi stdout_stream.class = StdoutStream stderr_stream.class = StdoutStream .ft P .fi .UNINDENT .UNINDENT .sp If your application is producing a traceback or error when it is trying to start up you should be able to see it in the output. .SS Changelog history .SS 0.12.1 \- 2015\-08\-05 .INDENT 0.0 .IP \(bu 2 Fix error when restarting a watcher with an output stream \- #913 .IP \(bu 2 Minor doc tweaks .UNINDENT .SS 0.12 \- 2015\-06\-02 .sp This release brings Python 3.4, Tornado 4 and Windows support, among several exciting features and fixes. .sp The Windows support is still experimental, and does not handle streams. .sp Major changes: .INDENT 0.0 .IP \(bu 2 Compatibility with Python 3.4 \- #768 .IP \(bu 2 Experimental Windows support \- #788 .IP \(bu 2 Compatibility with Tornado 4 \- #872 .IP \(bu 2 Revamped Debian packaging \- #896 \- #903 .IP \(bu 2 Add support for Papa process kernel \- #850 .IP \(bu 2 Add globing and regex matching for starting, stopping and restarting watchers \- #829 \- #902 .UNINDENT .sp More changes: .INDENT 0.0 .IP \(bu 2 Optimization of the shutdown \- #784 \- #842 .IP \(bu 2 Add possibility to specify virtualenv version for the watchers \- #805 .IP \(bu 2 Add \-\-nostop option to the rmwatcher command \- #777 .IP \(bu 2 Add a callback to Arbiter.start \- #840 .IP \(bu 2 Fix reloading watchers with uppercase letters \- #823 .IP \(bu 2 Remove leaking socket in stats daemon \- #843 .IP \(bu 2 Fix multicast on SunOS \- #876 .IP \(bu 2 Close output streams when stopping a watcher \- #885 .IP \(bu 2 Fix signal sending to grandchildren with \-\-recursive \- #888 .UNINDENT .SS 0.11.1 \- 2014\-05\-22 .INDENT 0.0 .IP \(bu 2 Fixed a regression that broke Circus on 2.6 \- #782 .UNINDENT .SS 0.11 \- 2014\-05\-21 .sp This release is not introducing a lot of features, and focused on making Circus more robust & stable. .sp Major changes/fixes: .INDENT 0.0 .IP \(bu 2 Make sure we cannot execute two conflictings commands on the arbiter simultanously. .IP \(bu 2 we have 2 new streams class: TimedRotatingFileStream, WatchedFileStream .IP \(bu 2 we have one new hook: after_spawn hook .IP \(bu 2 CircusPlugin is easier to use .IP \(bu 2 fix autostart=False watchers during start (regression) .UNINDENT .sp More changes: .INDENT 0.0 .IP \(bu 2 circus messages can be routed to syslog now \- #748 .IP \(bu 2 endpoint_owner option added so we can define which user owns ipc socket files created by circus. .IP \(bu 2 Started Windows support (just circusctl for now) .IP \(bu 2 fixed a lot of leaks in the tests .IP \(bu 2 Allow case sensitive environment variables .IP \(bu 2 The resource plugin now accepts absolute memory values \- #609 .IP \(bu 2 Add support to the add command for the \(aqsingleton\(aq option \- #767 .IP \(bu 2 Allow sending arbitrary signals to child procs via resource watcher \- #756 .IP \(bu 2 Allow INI/JSON/YAML configuration for logging .IP \(bu 2 Make sure we\(aqre compatible with psutil 2.x \fIand\fP 3.x .IP \(bu 2 Added more metrics to the statsd provider \- #698 .IP \(bu 2 Fixed multicast discovery \- #731 .IP \(bu 2 Make start, restart and reload more uniform \- #673 .IP \(bu 2 Correctly initialize all use groups \- #635 .IP \(bu 2 improved tests stability .IP \(bu 2 many, many more things.... .UNINDENT .SS 0.10 \- 2013\-11\-04 .sp Major changes: .INDENT 0.0 .IP \(bu 2 Now Python 3.2 & 3.3 compatible \- #586 .IP \(bu 2 Moved the core to a fully async model \- #569 .IP \(bu 2 Improved documentation \- #622 .UNINDENT .sp More changes: .INDENT 0.0 .IP \(bu 2 Added stop_signal & stop_children \- #594 .IP \(bu 2 Make sure the watchdog plugin closes the sockets \- #588 .IP \(bu 2 Switched to ZMQ JSON parser .IP \(bu 2 IN not supported on all platforms \- #573 .IP \(bu 2 Allow global environment substitutions in any config section \- #560 .IP \(bu 2 Allow dashes in sections names \- #546 .IP \(bu 2 Now variables are expanded everywhere in the config \- #554 .IP \(bu 2 Added the CommandReloader plugin .IP \(bu 2 Added before_signal & after_signal hooks .IP \(bu 2 Allow flapping plugin to retry indefinitely .IP \(bu 2 Don\(aqt respawn procs when the watcher is stopping \- #529 \- #536 .IP \(bu 2 Added a unique id for each client message \- #517 .IP \(bu 2 worker ids are now "slots" \- .IP \(bu 2 Fixed the graceful shutdown behavior \- #515 .IP \(bu 2 Make sure we can add watchers even if the arbiter is not started \- #503 .IP \(bu 2 Make sure make sure we pop expired process \- #510 .IP \(bu 2 Make sure the set command can set several hooks .IP \(bu 2 Correctly support ipv6 sockets \- #507 .IP \(bu 2 Allow custom options for stdout_stream and stderr_stream \- #495 .IP \(bu 2 Added time_format for FileStream \- #493 .IP \(bu 2 Added new socket config option to bind to a specific interface by name .UNINDENT .SS 0.9.3 \- 2013\-09\-04 .INDENT 0.0 .IP \(bu 2 Make sure we can add watchers even if the arbiter is not started .IP \(bu 2 Make sure we pop expired process .IP \(bu 2 Make sure the set command can set one or several hooks .IP \(bu 2 Correctly support ipv6 sockets and improvments of CircusSockets .IP \(bu 2 Give path default value to prevent UnboundLocalError .IP \(bu 2 Added a test for multicast_endpoint existence in Controller initialization .IP \(bu 2 Not converting every string of digits to ints anymore .IP \(bu 2 Add tests .IP \(bu 2 No need for special cases when converting stdout_stream options .IP \(bu 2 also accept umask as an argument for consistency .IP \(bu 2 Allow custom options for stdout_stream and stderr_stream. .IP \(bu 2 Add new socket config option to bind to a specific interface by name .IP \(bu 2 Add time_format for FileStream + tests .IP \(bu 2 Update circus.upstart .UNINDENT .SS 0.9.2 \- 2013\-07\-17 .INDENT 0.0 .IP \(bu 2 When a PYTHONPATH is defined in a config file, it\(aqs loaded in sys.path so hooks can be located there \- #477, #481 .IP \(bu 2 Use a single argument for add_callback so it works with PyZMQ < 13.1.x \- see #478 .UNINDENT .SS 0.9 \- 2013\-07\-16 .INDENT 0.0 .IP \(bu 2 added [env] sections wildcards .IP \(bu 2 added global [env] secrtion .IP \(bu 2 fixed hidden exception when circus\-web is not installed \- #424 .IP \(bu 2 make sure incr/decr commands really us the nb option \- #421 .IP \(bu 2 Fix watcher virtualenv site\-packages not in PYTHONPATH .IP \(bu 2 make sure we dont try to remove more processes than 0 \- #429 .IP \(bu 2 updated bootstrap.py \- #436 .IP \(bu 2 fixed multiplatform separator in pythonpath virtualenv watcher .IP \(bu 2 refactored socket close function .IP \(bu 2 Ensure env sections are applied to all watchers \- #437 .IP \(bu 2 added the reloadconfig command .IP \(bu 2 added circus.green and removed gevent from the core \- #441, #452 .IP \(bu 2 silenced spurious stdout & warnings in the tests \- #438 .IP \(bu 2 $(circus.env.*) can be used for all options in the config now .IP \(bu 2 added a before_spawn hook .IP \(bu 2 correct the path of circusd in systemd service file \- #450 .IP \(bu 2 make sure we can change hooks and set streams via CLI \- #455 .IP \(bu 2 improved doc .IP \(bu 2 added a spawn_count stat in watcher .IP \(bu 2 added min_cpu and min_mem parameters in ResourceWatcher plugin .IP \(bu 2 added the FQDN information to the arbiter. .UNINDENT .SS 0.8.1 \- 2013\-05\-28 .INDENT 0.0 .IP \(bu 2 circusd\-stats was choking on unix sockets \- #415 .IP \(bu 2 circusd\-stats & circushttpd child processes stdout/stderr are now left open by default. Python <= 2.7.5 would choke in the logging module in case the 2/3 fds were closed \- #415 .IP \(bu 2 Now redirecting to /dev/null in the child process instead of closing. #417 .UNINDENT .SS 0.8 \- 2013\-05\-24 .INDENT 0.0 .IP \(bu 2 Integrated log handlers into zmq io loop. .IP \(bu 2 Make redirector restartable and subsequently more robust. .IP \(bu 2 Uses zmq.green.eventloop when gevent is detected .IP \(bu 2 Added support for CIRCUSCTL_ENDPOINT environment variable to circusctl \- #396 .IP \(bu 2 util: fix bug in to_uid function \- #397 .IP \(bu 2 Remove handler on ioloop error \- #398. .IP \(bu 2 Improved test coverage .IP \(bu 2 Deprecated the \(aqservice\(aq option for the ResourceWatcher plugin \- #404 .IP \(bu 2 removed psutil.error usage .IP \(bu 2 Added UDP discovery in circusd \- #407 .IP \(bu 2 Now allowing globs at arbitrary directory levels \- #388 .IP \(bu 2 Added the \(aqstatd\(aq configuration option \- #408 .IP \(bu 2 Add pidfile, logoutput and loglevel option to circus configuration file \- #379 .IP \(bu 2 Added a tutorial in the docs. .IP \(bu 2 make sure we\(aqre merging all sections when using include \- #414 .IP \(bu 2 added pipe_stdout, pipe_stderr, close_child_stderr & close_child_stdout options to the Process class .IP \(bu 2 added close_child_stderr & close_child_stdout options to the watcher .UNINDENT .SS 0.7.1 \- 2013\-05\-02 .INDENT 0.0 .IP \(bu 2 Fixed the respawn option \- #382 .IP \(bu 2 Make sure we use an int for the timeout \- #380 .IP \(bu 2 display the unix sockets as well \- #381 .IP \(bu 2 Make sure it works with the latest pyzmq .IP \(bu 2 introduced a second syntax for the fd notation .UNINDENT .SS 0.7 \- 2013\-04\-08 .INDENT 0.0 .IP \(bu 2 Fix get_arbiter example to use a dict for the watchers argument. #304 .IP \(bu 2 Add some troubleshooting documentation #323 .IP \(bu 2 Add python buildout support .IP \(bu 2 Removed the gevent and the thread redirectors. now using the ioloop \- fixes #346. Relates #340 .IP \(bu 2 circus.web is now its own project .IP \(bu 2 removed the pyzmq patching .IP \(bu 2 Allow the watcher to be configured but not started #283 .IP \(bu 2 Add an option to load a virtualenv site dir .IP \(bu 2 added on_demand watchers .IP \(bu 2 added doc about nginx+websockets #371 .IP \(bu 2 now properly parsing the options list of each command #369 .IP \(bu 2 Fixed circusd\-stats events handling #372 .IP \(bu 2 fixed the overflow issue in circus\-top #378 .IP \(bu 2 many more things... .UNINDENT .SS 0.6 \- 2012\-12\-18 .INDENT 0.0 .IP \(bu 2 Patching protocols name for sockets \- #248 .IP \(bu 2 Don\(aqt autoscale graphs. #240 .IP \(bu 2 circusctl: add per command help, from docstrings #217 .IP \(bu 2 Added workers hooks .IP \(bu 2 Added Debian package \- #227 .IP \(bu 2 Added Redis, HTTP Observer, Full stats & Resource plugins .IP \(bu 2 Now processes can have titles .IP \(bu 2 Added autocompletion .IP \(bu 2 Added process/watcher age in the webui .IP \(bu 2 Added SSH tunnel support .IP \(bu 2 Now using pyzmq.green .IP \(bu 2 Added upstart script & Varnish doc .IP \(bu 2 Added environment variables & sections .IP \(bu 2 Added unix sockets support .IP \(bu 2 Added the \fIrespawn\fP option to have single\-run watchers .IP \(bu 2 Now using tox in the tests .IP \(bu 2 Allow socket substitution in args .IP \(bu 2 New doc theme .IP \(bu 2 New rotation options for streams: max_bytes/backup_count .UNINDENT .SS 0.5.2 \- 2012\-07\-26 .INDENT 0.0 .IP \(bu 2 now patching the thread module from the stdlib to avoid some Python bugs \- #203 .IP \(bu 2 better looking circusctl help screen .IP \(bu 2 uses pustil get_nice() when available (nice was deprecated) \- #208 .IP \(bu 2 added max_age support \- #221 .IP \(bu 2 only call listen() on SOCK_STREAM or SOCK_SEQPACKET sockets .IP \(bu 2 make sure the controller empties the plugins list in update_watchers() \- #220 .IP \(bu 2 added \-\-log\-level and \-\-log\-output to circushttpd .IP \(bu 2 fix the process killing via the web UI \- #219 .IP \(bu 2 now circus is zc.buildout compatible for scripts. .IP \(bu 2 cleanup the websocket when the client disconnect \- #225 .IP \(bu 2 fixed the default value for the endpoint \- #199 .IP \(bu 2 splitted circushttpd in logical modules .UNINDENT .SS 0.5.1 \- 2012\-07\-11 .INDENT 0.0 .IP \(bu 2 Fixed a bunch of typos in the documentation .IP \(bu 2 Added the debug option .IP \(bu 2 Package web\-requirements.txt properly .IP \(bu 2 Added a errno error code in the messages \- fixes #111 .UNINDENT .SS 0.5 \- 2012\-07\-06 .INDENT 0.0 .IP \(bu 2 added socket support .IP \(bu 2 added a listsocket command .IP \(bu 2 sockets have stats too ! .IP \(bu 2 fixed a lot of small bugs .IP \(bu 2 removed the wid \- now using pid everywhere .IP \(bu 2 faster tests .IP \(bu 2 changed the variables syntax .IP \(bu 2 use pyzmq\(aqs ioloop in more places .IP \(bu 2 now using iowait for all select() calls .IP \(bu 2 incr/decr commands now have an nbprocess parameter .IP \(bu 2 Add a reproduce_env option to watchers .IP \(bu 2 Add a new UNEXISTING status to the processes .IP \(bu 2 Added the global \fIhttpd\fP option to run circushttpd as a watcher .UNINDENT .SS 0.4 \- 2012\-06\-12 .INDENT 0.0 .IP \(bu 2 Added a plugin system .IP \(bu 2 Added a "singleton" option for watchers .IP \(bu 2 Fixed circus\-top screen flickering .IP \(bu 2 Removed threads from circus.stats in favor of zmq periodic callbacks .IP \(bu 2 Enhanced the documentation .IP \(bu 2 Circus client now have a send_message api .IP \(bu 2 The flapping feature is now a plugin .IP \(bu 2 Every command line tool have a \-\-version option .IP \(bu 2 Added a statsd plugin (sends the events from circus to statsd) .IP \(bu 2 The web UI now uses websockets (via socketio) to get the stats .IP \(bu 2 The web UI now uses sessions for "flash messages" in the web ui .UNINDENT .SS 0.3.4 \- 2012\-05\-30 .INDENT 0.0 .IP \(bu 2 Fixed a race condition that prevented the controller to cleanly reap finished processes. .IP \(bu 2 Now check_flapping can be controlled in the configuration. And activated/deactivated per watcher. .UNINDENT .SS 0.3.3 \- 2012\-05\-29 .INDENT 0.0 .IP \(bu 2 Fixed the regression on the uid handling .UNINDENT .SS 0.3.2 \- 2012\-05\-24 .INDENT 0.0 .IP \(bu 2 allows optional args property to add_watcher command. .IP \(bu 2 added circushttpd, circus\-top and circusd\-stats .IP \(bu 2 allowing Arbiter.add_watcher() to set all Watcher option .IP \(bu 2 make sure the redirectors are re\-created on restarts .UNINDENT .SS 0.3.1 \- 2012\-04\-18 .INDENT 0.0 .IP \(bu 2 fix: make sure watcher\(aq defaults aren\(aqt overrided .IP \(bu 2 added a StdoutStream class. .UNINDENT .SS 0.3 \- 2012\-04\-18 .INDENT 0.0 .IP \(bu 2 added the streaming feature .IP \(bu 2 now displaying coverage in the Sphinx doc .IP \(bu 2 fixed the way the processes are killed (no more SIGQUIT) .IP \(bu 2 the configuration has been factored out .IP \(bu 2 setproctitle support .UNINDENT .SS 0.2 \- 2012\-04\-04 .INDENT 0.0 .IP \(bu 2 Removed the \fIshow\fP name. replaced by \fIwatcher\fP\&. .IP \(bu 2 Added support for setting process \fBrlimit\fP\&. .IP \(bu 2 Added support for include dirs in the config file. .IP \(bu 2 Fixed a couple of leaking file descriptors. .IP \(bu 2 Fixed a core dump in the flapping .IP \(bu 2 Doc improvments .IP \(bu 2 Make sure circusd errors properly when another circusd is running on the same socket. .IP \(bu 2 get_arbiter now accepts several watchers. .IP \(bu 2 Fixed the cmd vs args vs executable in the process init. .IP \(bu 2 Fixed \-\-start on circusctl add .UNINDENT .SS 0.1 \- 2012\-03\-20 .INDENT 0.0 .IP \(bu 2 initial release .UNINDENT .SS man pages .SS circusd man page .SS Synopsis .sp circusd [options] [config] .SS Description .sp circusd is the main process of the Circus architecture. It takes care of running all the processes. Each process managed by Circus is a child process of \fBcircusd\fP\&. .SS Arguments .INDENT 0.0 .TP .B config configuration file .UNINDENT .SS Options .INDENT 0.0 .TP .B \-h, \-\-help Show the help message and exit .TP .B \-\-log\-level \fILEVEL\fP Specify the log level. \fILEVEL\fP can be \fIinfo\fP, \fIdebug\fP, \fIcritical\fP, \fIwarning\fP or \fIerror\fP\&. .TP .B \-\-log\-output \fILOGOUTPUT\fP The location where the logs will be written. The default behavior is to write to stdout (you can force it by passing \(aq\-\(aq to this option). Takes a filename otherwise. .TP .B \-\-logger\-config \fILOGGERCONFIG\fP The location where a standard Python logger configuration INI, JSON or YAML file can be found. This can be used to override the default logging configuration for the arbiter. .TP .B \-\-daemon Start circusd in the background. .TP .B \-\-pidfile \fIPIDFILE\fP The location of the PID file. .TP .B \-\-version Displays Circus version and exits. .UNINDENT .SS See also .sp \fIcircus\fP (1), \fIcircusctl\fP (1), \fIcircusd\-stats\fP (1), \fIcircus\-plugin\fP (1), \fIcircus\-top\fP (1). .sp Full Documentation is available at \fI\%http://circus.readthedocs.org\fP .SS circusctl man page .SS Synopsis .sp circusctl [options] command [args] .SS Description .sp circusctl is front end to control the Circus daemon. It is designed to help the administrator control the functionning of the Circud \fBcircusd\fP daemon. .SS Commands .INDENT 0.0 .TP .B add Add a watcher .TP .B decr Decrement the number of processes in a watcher .TP .B dstats Get circusd stats .TP .B get Get the value of specific watcher options .TP .B globaloptions Get the arbiter options .TP .B incr Increment the number of processes in a watcher .TP .B ipython Create shell into circusd process .TP .B list Get list of watchers or processes in a watcher .TP .B listen Subscribe to a watcher event .TP .B listsockets Get the list of sockets .TP .B numprocesses Get the number of processes .TP .B numwatchers Get the number of watchers .TP .B options Get the value of all options for a watcher .TP .B quit Quit the arbiter immediately .TP .B reload Reload the arbiter or a watcher .TP .B reloadconfig Reload the configuration file .TP .B restart Restart the arbiter or a watcher .TP .B rm Remove a watcher .TP .B set Set a watcher option .TP .B signal Send a signal .TP .B start Start the arbiter or a watcher .TP .B stats Get process infos .TP .B status Get the status of a watcher or all watchers .TP .B stop Stop watchers .UNINDENT .SS Options .INDENT 0.0 .TP .B \-\-endpoint \fIENDPOINT\fP connection endpoint .TP .B \-h, \-\-help Show the help message and exit .TP .B \-\-json output to JSON .TP .B \-\-prettify prettify output .TP .B \-\-ssh \fISSH\fP SSH Server in the format \fBuser@host:port\fP .TP .B \-\-ssh_keyfile \fISSH_KEYFILE\fP path to the keyfile to authorise the user .TP .B \-\-timeout \fITIMEOUT\fP connection timeout .TP .B \-\-version Displays Circus version and exits. .UNINDENT .SS See Also .sp \fIcircus\fP (1), \fIcircusd\fP (1), \fIcircusd\-stats\fP (1), \fIcircus\-plugin\fP (1), \fIcircus\-top\fP (1). .sp Full Documentation is available at \fI\%http://circus.readthedocs.org\fP .SS circus\-plugin man page .SS Synopsis .sp circus\-plugin [options] [plugin] .SS Description .sp circus\-plugin allows to launch a plugin from a running Circus daemon. .SS Arguments .INDENT 0.0 .TP .B plugin Fully qualified name of the plugin class. .UNINDENT .SS Options .INDENT 0.0 .TP .B \-\-endpoint \fIENDPOINT\fP Connection endpoint. .TP .B \-\-pubsub \fIPUBSUB\fP The circusd ZeroMQ pub/sub socket to connect to. .TP .B \-\-config \fICONFIG\fP The plugin configuration file. .TP .B \-\-check\-delay \fICHECK_DELAY\fP Check delay. .TP .B \-\-log\-level \fILEVEL\fP Specify the log level. \fILEVEL\fP can be \fIinfo\fP, \fIdebug\fP, \fIcritical\fP, \fIwarning\fP or \fIerror\fP\&. .TP .B \-\-log\-output \fILOGOUTPUT\fP The location where the logs will be written. The default behavior is to write to stdout (you can force it by passing \(aq\-\(aq to this option). Takes a filename otherwise. .TP .B \-\-ssh \fISSH\fP SSH Server in the format \fBuser@host:port\fP\&. .TP .B \-h, \-\-help Show the help message and exit. .TP .B \-\-version Displays Circus version and exits. .UNINDENT .SS See also .sp \fIcircus\fP (1), \fIcircusd\fP (1), \fIcircusctl\fP (1), \fIcircusd\-stats\fP (1), \fIcircus\-top\fP (1). .sp Full Documentation is available at \fI\%http://circus.readthedocs.org\fP .SS circus\-top man page .SS Synopsis .sp circus\-top [options] .SS Description .sp circus\-top is a \fItop\fP\-like command to display the Circus daemon and processes managed by circus. .SS Options .INDENT 0.0 .TP .B \-\-endpoint \fIENDPOINT\fP Connection endpoint. .TP .B \-\-ssh \fISSH\fP SSH Server in the format \fBuser@host:port\fP\&. .TP .B \-\-process\-timeout \fIPROCESS_TIMEOUT\fP After this delay of inactivity, a process will be removed. .TP .B \-h, \-\-help Show the help message and exit. .TP .B \-\-version Displays Circus version and exits. .UNINDENT .SS See also .sp \fIcircus\fP (1), \fIcircusctl\fP (1), \fIcircusd\fP (1), \fIcircusd\-stats\fP (1), \fIcircus\-plugin\fP (1). .sp Full Documentation is available at \fI\%http://circus.readthedocs.org\fP .SS circusd\-stats man page .SS Synopsis .sp circusd\-stats [options] .SS Description .sp circusd\-stats runs the stats aggregator for Circus. .SS Options .INDENT 0.0 .TP .B \-\-endpoint \fIENDPOINT\fP Connection endpoint. .TP .B \-\-pubsub \fIPUBSUB\fP The circusd ZeroMQ pub/sub socket to connect to. .TP .B \-\-statspoint \fISTATSPOINT\fP The ZeroMQ pub/sub socket to send data to. .TP .B \-\-log\-level \fILEVEL\fP Specify the log level. \fILEVEL\fP can be \fIinfo\fP, \fIdebug\fP, \fIcritical\fP, \fIwarning\fP or \fIerror\fP\&. .TP .B \-\-log\-output \fILOGOUTPUT\fP The location where the logs will be written. The default behavior is to write to stdout (you can force it by passing \(aq\-\(aq to this option). Takes a filename otherwise. .TP .B \-\-ssh \fISSH\fP SSH Server in the format \fBuser@host:port\fP\&. .TP .B \-h, \-\-help Show the help message and exit. .TP .B \-\-version Displays Circus version and exits. .UNINDENT .SS See also .sp \fIcircus\fP (1), \fIcircusd\fP (1), \fIcircusctl\fP (1), \fIcircus\-plugin\fP (1), \fIcircus\-top\fP (1). .sp Full Documentation is available at \fI\%http://circus.readthedocs.org\fP .SS Glossary: Circus\-specific terms .INDENT 0.0 .TP .B arbiter The \fIarbiter\fP is responsible for managing all the watchers within circus, ensuring all processes run correctly. .TP .B controller A \fIcontroller\fP contains the set of actions that can be performed on the arbiter. .TP .B flapping The \fIflapping detection\fP subscribes to events and detects when some processes are constantly restarting. .TP .B pub/sub Circus has a \fIpubsub\fP that receives events from the watchers and dispatches them to all subscribers. .TP .B remote controller The \fIremote controller\fP allows you to communicate with the controller via ZMQ to control Circus. .TP .B watcher, watchers A \fIwatcher\fP is the program you tell Circus to run. A single Circus instance can run one or more watchers. .TP .B worker, workers, process, processes A \fIprocess\fP is an independent OS process instance of your program. A single watcher can run one or more processes. We also call them workers. .UNINDENT .SS Copyright .sp Circus was initiated by Tarek Ziade and is licenced under APLv2 .sp Benoit Chesneau was an early contributor and did many things, like most of the circus.commands work. .SS Licence .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C Copyright 2012 \- Mozilla Foundation Copyright 2012 \- Benoit Chesneau Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE\-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. .ft P .fi .UNINDENT .UNINDENT .SS Contributors .sp See the full list at \fI\%https://github.com/circus\-tent/circus/blob/master/CONTRIBUTORS.txt\fP .SH AUTHOR Mozilla Foundation, Benoit Chesneau .\" Generated by docutils manpage writer. .