NAME¶
include_server.py - conservative approximation of include dependencies for C/C++
SYNOPSIS¶
include_server --port
INCLUDE_SERVER_PORT [OPTIONS]
DESCRIPTION¶
include_server.py starts an include server process. This process answers queries
from
distcc(1) clients about what files to include in C/C++
compilations. The include_server.py command itself terminates as soon as the
include server has been spawned.
The INCLUDE_SERVER_PORT argument is the name of a socket used for all
communication between distcc clients and the include server. The
distcc-pump(1) command is responsible for creating the socket location,
for passing it to this script, and for passing it to all distcc clients via
the environment variable named INCLUDE_SERVER_PORT.
The protocol used by the include server uses distcc's RPC implementation. Each
distcc request consists of (1) the current directory and (2) the list of
arguments of the compilation command.
If the include server is able to process the request, then it answers the distcc
client by sending a list of filepaths. The filepaths are those of the
compressed source and header files found to be necessary for compilation
through include analysis. The list also comprises symbolic links and even
dummy files needed for the compilation server to construct an accurate replica
of the parts of the filesystem needed for compilation. In this way, a needed
header file like /path/foo.h is compressed, renamed, and stored in a temporary
location, such as /dev/shm/tmpiAvfGv.include_server-9368-1/path/foo.h.lzo. The
distcc client will pass these files on to a compilation server, where they
will be uncompressed and mounted temporarily.
If the include server is not able to process the request, then it returns the
empty list to the distcc client.
There are two kinds of failures that relate to the include server. The include
server may fail to compute the includes or fail in other ways, see section
INCLUDE SERVER SYMPTOMS. Also, the compilation on the remove server may
fail due to inadequacy of the calculated include closure, but then succeed
when locally retried, see section
DISTCC DISCREPANCY SYMPTOMS.
OPTION SUMMARY¶
The following options are understood by include_server.py.
- -dPAT, --debug_pattern=PAT
- Bit vector for turning on warnings and debugging
1 = warnings
2 = trace some functions
other powers of two: see include_server/basics.py.
- -e, --email
- Send email to 'distcc-pump-errors' or if defined, the value of enviroment
variable DISTCC_EMAILLOG_WHOM_TO_BLAME, when include server gets in
trouble. The default is to not send email.
- --email_bound NUMBER
- Maximal number of emails to send (in addition to a final email). Default:
3.
- --no-email
- Do not send email. This is the default.
- --path_observation_re=RE
- Issue warning message whenever a filename is resolved to a realpath that
is matched by RE, which is a regular expression in Python syntax. This is
useful for finding out where files included actually come from. Use
RE="" to find them all. Note: warnings must be enabled with at
least -d1.
- --pid_file FILEPATH
- The pid of the include server is written to file FILEPATH. This allows a
script such a distcc-pump to tear down the include server.
- -s, --statistics
- Print information to stdout about include analysis.
- --stat_reset_triggers=LIST
- Flush stat caches when the timestamp of any filepath in LIST changes or
the filepath comes in or out of existence. LIST is a colon separated
string of filepaths, possibly containing simple globs (as allowed by
Python's glob module). Print a warning whenever such a change happens (if
warnings are enabled). This option allows limited exceptions to
distcc-pump's normal assumption that source files are not modified during
the build.
- -t, --time
- Print elapsed, user, and system time to stderr.
- --unsafe_absolute_includes
- Do preprocessing on the compilation server even if includes of absolute
filepaths are encountered. Normally the include-server will fall back on
local preprocessing if it detects any absolute includes. Thus, this flag
is useful for preventing such fallbacks when the absolute includes are a
false alarm, either because the absolute include is discarded during
preprocessing or because the absolutely included file exists on the
compilation servers.
More precisely, with --unsafe_absolute_includes absolute includes are
ignored for the purposes of gathering the include closure. Using this
option may lead to incorrect results because (1) the header may actually
be included on the compilation server and it may not be the same as on the
client, (2) the include directives of the header are not further analyzed.
The option is useful for compiling code that has such hardcoded absolute
locations of header files inside conditional directives (e.g.
"#ifdef") that render the includes irrelevant. More precisely,
these includes must be eliminated during preprocessing for the actual
configuration. Then the question of existence of the header file is moot
and the remote compilation is sound. This is often the case if such
includes are meant for unusual configurations different from the actual
configuration.
- -v, --verify
- Verify that files in CPP closure are contained in closure calculated by
include processor.
- -w, --write_include_closure
- Write a .d_approx file which lists all the included files calculated by
the include server; with -x, additionally write the included files as
calculated by CPP to a .d_exact file.
- -x, --exact_analysis
- Use CPP instead, do not omit system headers files.
INCLUDE SERVER SYMPTOMS AND ISSUES¶
The most likely messages and warnings to come from the include processor are
listed below.
- Preprocessing locally. Include server not covering: Couldn't determine
default system include directories
- To determine the default system header directories, the include server
runs the compiler once for each language needed during its session. This
message indicates that the compiler specified to distcc is not present on
the client.
- Preprocessing locally. Include server not covering: Bailing out because
include server spent more than ...s user time handling request
- In uncommon situations, the include server fails to analyze very
complicated macro expressions. The distcc client will use plain distcc
mode.
- Warning: Filepath must be relative but isn't
- The include server does not accept absolute filepaths, such as
/usr/include/stdio.h, in include directives, because there is no guarantee
that this header on the compilation server machine will be the same as
that on the client. The include server gives up analyzing the include
closure. The distcc client cannot use pump-mode.
To overcome this problem in a not always reliable way, set the environment
variable INCLUDE_SERVER_ARGS='--unsafe_absolute_includes' when invoking
the distcc-pump script to pass the --unsafe_absolute_includes option to
the include server.
- Warning: Absolute filepath ... was IGNORED
- The --unsafe_absolute_includes is in use. This situation happens under the
same circumstances as when "Filepath must be relative but isn't"
is issued, but in this case the include will provide an answer to the
distcc client.
- Warning: Path '/PATH/FILE' changed/came into existence/no longer
exists
- These warnings are issued when using stat reset triggers. Because
/PATH/FILE changed, the include server clears its caches; the new version
of the file (or the lack of it) renders the include analysis invalid. This
message can usually be ignored; it does signify a somewhat precarious use
of files by the build system. It is recommended to fix the build system so
that files are not rewritten.
- Warning: For translation unit ..., lookup of file ... resolved to ...
whose realpath is ...
- This warning occurs with --path_observation_re when a new realpath
matching a source or header file is observed.
DISTCC DISCREPANCY SYMPTOMS¶
The interactions between the build system, distcc, and the include server is
somewhat complex. When a distcc commands receives a failing compilation from
the remote server it retries the compilation locally. This section discusses
the causes of discrepancies between remote and local compilation. These are
flagged by the demotion message:
__________Warning: ... pump-mode compilation(s) failed on server, but
succeeded locally.
__________Distcc-pump was demoted to plain mode. See the Distcc
Discrepancy Symptoms section in the include_server(1) man page.
The distcc-pump script issues this message at the end of the build. This means
that for at least one distcc invocation a local compilation succeeded after
the remote compilation failed. Each distcc invocation for which such a
discrepancy occurred in turn also issues a message such as:
Warning: remote compilation of '...' failed, retried locally and got a
different result.
The demotion makes subsequent distcc invocations use plain distcc mode. Thus
preprocessing will take place on the local machine for the remainder of the
build. This technique prevents very slow builds where all compilations end up
on the local machine after failing remotely.
Of course, if the local compilations fails after the remote failures, then the
distcc invocation exits with the non-zero status of the local compilation. The
error messages printed are also those of the local compilation.
The fallback behavior for distcc-pump mode to local compilation can be disabled
by setting the environment variable DISTCC_FALLBACK to 1, which makes the
distcc command fail as soon as the remote compilation has failed. This setting
is very useful for debugging why the remote compilation went wrong, because
now the output from the server will be printed.
Next we discuss the possible causes of discrepancies.
- The user changed a source or header file during the build.
- This yields inconsistent results of course.
- A source or header file changed during the build.
- The build system rewrites a file. For Linux kernel 2.6, this happens for
'include/linux/compile.h' and 'include/asm/asm-offsets.h'. This condition
is fixed by letting the include server know that it must reset its caches
when a stat of any of the files changes. Practically, this is done by
gathering the files in a colon-separated list and then setting the
INCLUDE_SERVER_ARGS command as in:
include_server_args="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"
- A header file is potentially included, but does not exist, and is then
later included.
-
This occurs when some header foo.h includes another header file trick.h, but
the trick.h file has not yet been generated and the inclusion is actually
ignored because of preprocessing directives. The include server will probe
for the existence of trick.h, because it overapproximates all possible
ways directives actually evaluate. The file trick.h is determined not to
exist. If it is later generated, and then really included, then the
include server will falsely believe that the file still does not exist.
The solution to this problem is to make the build system generate trick.h
before the first time any header file is included that makes a syntactic
reference to trick.h
- The include server was started with
--unsafe_absolute_includes.
- This is a problem if there are header files locally that do not exist
remotely and that are actually used. Such includes are often protected by
conditional directives that evaluate so that are actually used on only
specific and often uncommon platforms. If you are not compiling for such a
platform, then it may be correct to use --unsafe_absolute_include.
- The include server has calculated the wrong includes.
- We do not know of such a situation.
EXIT CODES¶
The exit code of include_server.py is usually 0. That the include server has
been started properly is communicated through the existence of the pid_file.
ENVIRONMENT VARIABLES¶
DISTCC_EMAILLOG_WHOM_TO_BLAME The email address to use for include server
automated emails. The default is 'distcc-pump-errors' (which is an email
address that probably will not exist in your domain).
Additionally, the invocation of the compiler may use additional environment
variables.
BUGS¶
If you think you have found a distcc bug, please see the file
reporting-bugs.txt in the documentation directory for information on
how to report it.
In distcc-pump mode, the include server is unable to handle certain very
complicated computed includes as found in parts of the Boost library. The
include server will time out and distcc will revert to plain mode.
Other known bugs may be documented on
http://code.google.com/p/distcc/
AUTHOR¶
The include server was written by Nils Klarlund, with assistance from Fergus
Henderson, Manos Renieris, and Craig Silverstein. Please report bugs to
<distcc@lists.samba.org>.
LICENCE¶
You are free to use distcc. distcc (including this manual) may be copied,
modified or distributed only under the terms of the GNU General Public Licence
version 2 or later. distcc comes with absolutely no warrany. A copy of the GPL
is included in the file COPYING.
SEE ALSO¶
distcc(1),
distccd(1),
include_server(1), and
gcc(1).
http://code.google.com/p/distcc/ http://ccache.samba.org/