.TH ttb 3erl "observer 2.14.0.1" "Ericsson AB" "Erlang Module Definition"
.SH NAME
ttb \- A base for building trace tools for distributed systems.
.SH DESCRIPTION
.LP
The Trace Tool Builder, \fIttb\fR\&, is a base for building trace tools for distributed systems\&.
.LP
When using \fIttb\fR\&, do not use module \fIdbg\fR\& in application Runtime_Tools in parallel\&.
.SH EXPORTS
.LP
.B
start_trace(Nodes, Patterns, FlagSpec, Opts) -> Result
.br
.RS
.LP
Types:

.RS 3
Result = see p/2
.br
Nodes = see tracer/2
.br
Patterns = [tuple()]
.br
FlagSpec = {Procs, Flags}
.br
Proc = see p/2
.br
Flags = see p/2
.br
Opts = see tracer/2
.br
.RE
.RE
.RS
.LP
This function is a shortcut allowing to start a trace with one command\&. Each tuple in \fIPatterns\fR\& is converted to a list, which in turn is passed to \fIttb:tpl/2,3,4\fR\&\&.
.LP
The call:
.LP
.nf

> ttb:start_trace([Node, OtherNode], [{mod, foo, []}, {mod, bar, 2}], {all, call}, [{file, File}, {handler,{fun myhandler/4, S}}])\&.
.fi
.LP
is equivalent to:
.LP
.nf

> ttb:start_trace([Node, OtherNode], [{file, File}, {handler,{fun myhandler/4, S}}]), ttb:tpl(mod, foo, []), ttb:tpl(mod, bar, 2, []), ttb:p(all, call)\&.
.fi
.RE

.LP
.B
tracer() -> Result
.br
.RS
.LP
Equivalent to \fItracer(node())\fR\&\&.
.RE

.LP
.B
tracer(Shortcut) -> Result
.br
.RS
.LP
Types:

.RS 3
Shortcut = shell | dbg
.br
.RE
.RE
.RS
.LP
Handy shortcuts for common tracing settings\&.
.LP
\fIshell\fR\& is equivalent to \fItracer(node(),[{file, {local, "ttb"}}, shell])\fR\&\&.
.LP
\fIdbg\fR\& is equivalent to \fItracer(node(),[{shell, only}])\fR\&\&.
.RE

.LP
.B
tracer(Nodes) -> Result
.br
.RS
.LP
Equivalent to \fItracer(Nodes,[])\fR\&\&.
.RE

.LP
.B
tracer(Nodes,Opts) -> Result
.br
.RS
.LP
Types:

.RS 3
Result = {ok, ActivatedNodes} | {error,Reason}
.br
Nodes = atom() | [atom()] | all | existing | new
.br
Opts = Opt | [Opt]
.br
Opt = {file,Client} | {handler, FormatHandler} | {process_info,PI} | shell | {shell, ShellSpec} | {timer, TimerSpec} | {overload_check, {MSec, Module, Function}} | {flush, MSec} | resume | {resume, FetchTimeout} | {queue_size, QueueSize}
.br
TimerSpec = MSec | {MSec, StopOpts}
.br
MSec = FetchTimeout = integer()
.br
Module = Function = atom() 
.br
StopOpts = see stop/2
.br
Client = File | {local, File}
.br
File = Filename | Wrap
.br
Filename = string()
.br
Wrap = {wrap,Filename} | {wrap,Filename,Size,Count}
.br
FormatHandler = See format/2
.br
PI = true | false 
.br
ShellSpec = true | false | only
.br
QueueSize = non_neg_integer()
.br
.RE
.RE
.RS
.LP
Starts a file trace port on all specified nodes and points the system tracer for sequential tracing to the same port\&.
.LP
\fIOptions:\fR\&
.RS 2
.TP 2
.B
\fIFilename\fR\&:
The specified \fIFilename\fR\& is prefixed with the node name\&. Default \fIFilename\fR\& is \fIttb\fR\&\&.
.TP 2
.B
\fIFile={wrap,Filename,Size,Count}\fR\&:
Can be used if the size of the trace logs must be limited\&. Default values are \fISize=128*1024\fR\& and \fICount=8\fR\&\&.
.TP 2
.B
\fIClient\fR\&:
When tracing diskless nodes, \fIttb\fR\& must be started from an external "trace control node" with disk access, and \fIClient\fR\& must be \fI{local, File}\fR\&\&. All trace information is then sent to the trace control node where it is written to file\&.
.TP 2
.B
\fIqueue_size\fR\&:
When tracing to shell or \fI{local,File}\fR\&, an ip trace driver is used internally\&. The ip trace driver has a queue of maximum \fIQueueSize\fR\& messages waiting to be delivered\&. If the driver cannot deliver messages as fast as they are produced, the queue size might be exceeded and messages are dropped\&. This parameter is optional, and is only useful if many \fI{drop,N}\fR\& trace messages are received by the trace handler\&. It has no meaning if shell or \fI{local,File}\fR\& is not used\&. See dbg:trace_port/2 for more information about the ip trace driver\&.
.TP 2
.B
\fIprocess_info\fR\&:
Indicates if process information is to be collected\&. If \fIPI = true\fR\& (which is default), each process identifier \fIPid\fR\& is replaced by a tuple \fI{Pid,ProcessInfo,Node}\fR\&, where \fIProcessInfo\fR\& is the registered process name, its globally registered name, or its initial function\&. To turn off this functionality, set \fIPI = false\fR\&\&.
.TP 2
.B
\fI{shell, ShellSpec}\fR\&:
Indicates that trace messages are to be printed on the console as they are received by the tracing process\&. This implies trace client \fI{local, File}\fR\&\&. If \fIShellSpec\fR\& is \fIonly\fR\& (instead of \fItrue\fR\&), no trace logs are stored\&.
.TP 2
.B
\fIshell\fR\&:
Shortcut for \fI{shell, true}\fR\&\&.
.TP 2
.B
\fItimer\fR\&:
Indicates that the trace is to be automatically stopped after \fIMSec\fR\& milliseconds\&. \fIStopOpts\fR\& are passed to command \fIttb:stop/2\fR\& if specified (default is \fI[]\fR\&)\&. Notice that the timing is approximate, as delays related to network communication are always present\&. The timer starts after \fIttb:p/2\fR\& is issued, so you can set up your trace patterns before\&.
.TP 2
.B
\fIoverload_check\fR\&:
Allows to enable overload checking on the nodes under trace\&. \fIModule:Function(check)\fR\& is performed each \fIMSec\fR\& millisecond\&. If the check returns \fItrue\fR\&, the tracing is disabled on a specified node\&.
.RS 2
.LP
\fIModule:Function\fR\& must be able to handle at least three atoms: \fIinit\fR\&, \fIcheck\fR\&, and \fIstop\fR\&\&. \fIinit\fR\& and \fIstop\fR\& allows you to initialize and clean up the check environment\&.
.RE

.RS 2
.LP
When a node gets overloaded, it is not possible to issue \fIttb:p/2\fR\& or any command from the \fIttb:tp/2,3,4\fR\& family, as it would lead to inconsistent tracing state (different trace specifications on different nodes)\&.
.RE

.TP 2
.B
\fIflush\fR\&:
Periodically flushes all file trace port clients (see \fIdbg:flush_trace_port/1\fR\&)\&. When enabled, the buffers are freed each \fIMSec\fR\& millisecond\&. This option is not allowed with \fI{file, {local, File}}\fR\& tracing\&.
.TP 2
.B
\fI{resume, FetchTimeout}\fR\&:
Enables the autoresume feature\&. When enabled, remote nodes try to reconnect to the controlling node if they are restarted\&. The feature requires application Runtime_Tools to be started (so it has to be present in the \fI\&.boot\fR\& scripts if the traced nodes run with embedded Erlang)\&. If this is not possible, resume can be performed manually by starting \fIRuntime_Tools\fR\& remotely using \fIrpc:call/4\fR\&\&.
.RS 2
.LP
\fIttb\fR\& tries to fetch all logs from a reconnecting node before reinitializing the trace\&. This must finish within \fIFetchTimeout\fR\& milliseconds or is aborted\&.
.RE

.RS 2
.LP
By default, autostart information is stored in a file named \fIttb_autostart\&.bin\fR\& on each node\&. If this is not desired (for example, on diskless nodes), a custom module handling autostart information storage and retrieval can be provided by specifying environment variable \fIttb_autostart_module\fR\& for the application Runtime_Tools\&. The module must respond to the following API:
.RE

.RS 2
.TP 2
.B
\fIwrite_config(Data) -> ok\fR\&:
Stores the provided data for further retrieval\&. It is important to realize that the data storage used must not be affected by the node crash\&.
.TP 2
.B
\fIread_config() -> {ok, Data} | {error, Error}\fR\&:
Retrieves configuration stored with \fIwrite_config(Data)\fR\&\&.
.TP 2
.B
\fIdelete_config() -> ok\fR\&:
Deletes configuration stored with \fIwrite_config(Data)\fR\&\&. Notice that after this call any subsequent calls to \fIread_config\fR\& must return \fI{error, Error}\fR\&\&.
.RE
.RS 2
.LP
\fIresume\fR\& implies the default \fIFetchTimeout\fR\&, which is 10 seconds
.RE

.RE
.RE

.LP
.B
p(Item,Flags) -> Return
.br
.RS
.LP
Types:

.RS 3
Return = {ok,[{Item,MatchDesc}]}
.br
Items = Item | [Item]
.br
Item = pid() | port() | RegName | {global,GlobalRegName} | all | processes | ports | existing | existing_processes | existing_ports | new | new_processes | new_ports
.br
RegName = atom()
.br
GlobalRegName = term()
.br
Flags = Flag | [Flag]
.br
.RE
.RE
.RS
.LP
Sets the specified trace flags on the specified processes or ports\&. Flag \fItimestamp\fR\& is always turned on\&.
.LP
See the Reference Manual for module \fIdbg\fR\& for the possible trace flags\&. Parameter \fIMatchDesc\fR\& is the same as returned from \fIdbg:p/2\fR\&\&.
.LP
Processes can be specified as registered names, globally registered names, or process identifiers\&. Ports can be specified as registered names or port identifiers\&. If a registered name is specified, the flags are set on processes/ports with this name on all active nodes\&.
.LP
Issuing this command starts the timer for this trace if option \fItimer\fR\& is specified with \fItracer/2\fR\&\&.
.RE

.LP
.B
tp(Module [, Function [, Arity]], MatchSpec)
.br
.B
tp({Module, Function , Arity}, MatchSpec)
.br
.B
tpl(Module [, Function [, Arity]], MatchSpec)
.br
.B
tpl({Module, Function , Arity}, MatchSpec)
.br
.B
ctp()
.br
.B
ctp(Module [, Function [, Arity]])
.br
.B
ctp({Module, Function, Arity})
.br
.B
ctpl()
.br
.B
ctpl(Module [, Function [, Arity]])
.br
.B
ctpl({Module, Function, Arity})
.br
.B
ctpg()
.br
.B
ctpg(Module [, Function [, Arity]])
.br
.B
ctpg({Module, Function, Arity})
.br
.B
tpe(Event,MatchSpec)
.br
.B
ctpe(Event)
.br
.RS
.LP
These functions are to be used with trace flag \fIcall\fR\&, \fIsend\fR\&, and \fI\&'receive\&'\fR\& for setting and clearing trace patterns\&.
.LP
When trace flag \fIcall\fR\& is set on a process, function calls are traced on that process if a trace pattern is set for the called function\&.
.LP
The \fIsend\fR\& and \fI\&'receive\&'\fR\& flags enable tracing of all messages sent and received by the process/port\&. Trace patterns set with \fItpe\fR\& may limit traced messages based on the message content, the sender, and/or the receiver\&.
.LP
Trace patterns specify how to trace a function or a message by using match specifications\&. Match specifications are described in the \fIERTS User\&'s Guide\fR\&\&.
.LP
These functions are equivalent to the corresponding functions in module dbg, but all calls are stored in the history\&. The history buffer makes it easy to create configuration files; the same trace environment can be set up many times, for example, to compare two test runs\&. It also reduces the amount of typing when using \fIttb\fR\& from the Erlang shell\&.
.RS 2
.TP 2
.B
\fItp\fR\&:
Sets trace patterns on global function calls\&.
.TP 2
.B
\fItpl\fR\&:
Sets trace patterns on local and global function calls\&.
.TP 2
.B
\fItpe\fR\&:
Sets trace patterns on messages\&.
.TP 2
.B
\fIctp\fR\&:
Clears trace patterns on local and global function calls\&.
.TP 2
.B
\fIctpl\fR\&:
Clears trace patterns on local function calls\&.
.TP 2
.B
\fIctpg\fR\&:
Clears trace patterns on global function calls\&.
.TP 2
.B
\fIctpe\fR\&:
Clears trace patterns on messages\&.
.RE
.LP
With \fItp\fR\& and \fItpl\fR\&, one of the match specification shortcuts can be used (for example, \fIttb:tp(foo_module, caller)\fR\&)\&.
.LP
The shortcuts are as follows:
.RS 2
.TP 2
*
\fIreturn\fR\& - for \fI[{\&'_\&',[],[{return_trace}]}]\fR\& (report the return value from a traced function)
.LP
.TP 2
*
\fIcaller\fR\& - for \fI[{\&'_\&',[],[{message,{caller}}]}]\fR\& (report the calling function)
.LP
.TP 2
*
\fI{codestr, Str}\fR\& - for \fIdbg:fun2ms/1\fR\& arguments passed as strings (example: \fI"fun(_) -> return_trace() end"\fR\&) 
.LP
.RE

.RE

.LP
.B
list_history() -> History
.br
.RS
.LP
Types:

.RS 3
History = [{N,Func,Args}]
.br
.RE
.RE
.RS
.LP
All calls to \fIttb\fR\& is stored in the history\&. This function returns the current content of the history\&. Any entry can be reexecuted with \fIrun_history/1\fR\& or stored in a configuration file with \fIwrite_config/2,3\fR\&\&.
.RE

.LP
.B
run_history(N) -> ok | {error, Reason}
.br
.RS
.LP
Types:

.RS 3
N = integer() | [integer()]
.br
.RE
.RE
.RS
.LP
Executes the specified entry or entries from the history list\&. To list history, use \fIlist_history/0\fR\&\&.
.RE

.LP
.B
write_config(ConfigFile,Config)
.br
.RS
.LP
Equivalent to \fIwrite_config(ConfigFile,Config,[])\fR\&\&.
.RE

.LP
.B
write_config(ConfigFile,Config,Opts) -> ok | {error,Reason}
.br
.RS
.LP
Types:

.RS 3
ConfigFile = string()
.br
Config = all | [integer()] | [{Mod,Func,Args}]
.br
Mod = atom()
.br
Func = atom()
.br
Args = [term()]
.br
Opts = Opt | [Opt]
.br
Opt = append
.br
.RE
.RE
.RS
.LP
Creates or extends a configuration file, which can be used for restoring a specific configuration later\&.
.LP
The contents of the configuration file can either be fetched from the history or specified directly as a list of \fI{Mod,Func,Args}\fR\&\&.
.LP
If the complete history is to be stored in the configuration file, \fIConfig\fR\& must be \fIall\fR\&\&. If only a selected number of entries from the history are to be stored, \fIConfig\fR\& must be a list of integers pointing out the entries to be stored\&.
.LP
If \fIOpts\fR\& is not specified or if it is \fI[]\fR\&, \fIConfigFile\fR\& is deleted and a new file is created\&. If \fIOpts = [append]\fR\&, \fIConfigFile\fR\& is not deleted\&. The new information is appended at the end of the file\&.
.RE

.LP
.B
run_config(ConfigFile) -> ok | {error,Reason}
.br
.RS
.LP
Types:

.RS 3
ConfigFile = string()
.br
.RE
.RE
.RS
.LP
Executes all entries in the specified configuration file\&. Notice that the history of the last trace is always available in file \fIttb_last_config\fR\&\&.
.RE

.LP
.B
run_config(ConfigFile,NumList) -> ok | {error,Reason}
.br
.RS
.LP
Types:

.RS 3
ConfigFile = string()
.br
NumList = [integer()]
.br
.RE
.RE
.RS
.LP
Executes selected entries from the specified configuration file\&. \fINumList\fR\& is a list of integers pointing out the entries to be executed\&.
.LP
To list the contents of a configuration file, use \fIlist_config/1\fR\&\&.
.LP
Notice that the history of the last trace is always available in file \fIttb_last_config\fR\&\&.
.RE

.LP
.B
list_config(ConfigFile) -> Config | {error,Reason}
.br
.RS
.LP
Types:

.RS 3
ConfigFile = string()
.br
Config = [{N,Func,Args}]
.br
.RE
.RE
.RS
.LP
Lists all entries in the specified configuration file\&.
.RE

.LP
.B
write_trace_info(Key,Info) -> ok
.br
.RS
.LP
Types:

.RS 3
Key = term()
.br
Info = Data | fun() -> Data
.br
Data = term()
.br
.RE
.RE
.RS
.LP
File \fI\&.ti\fR\& contains \fI{Key,ValueList}\fR\& tuples\&. This function adds \fIData\fR\& to the \fIValueList\fR\& associated with \fIKey\fR\&\&. All information written with this function is included in the call to the format handler\&.
.RE

.LP
.B
seq_trigger_ms() -> MatchSpec
.br
.RS
.LP
Equivalent to \fIseq_trigger_ms(all)\fR\&\&.
.RE

.LP
.B
seq_trigger_ms(Flags) -> MatchSpec
.br
.RS
.LP
Types:

.RS 3
MatchSpec = match_spec()
.br
Flags = all | SeqTraceFlag | [SeqTraceFlag]
.br
SeqTraceFlag = atom()
.br
.RE
.RE
.RS
.LP
A match specification can turn on or off sequential tracing\&. This function returns a match specification, which turns on sequential tracing with the specified \fIFlags\fR\&\&.
.LP
This match specification can be specified as the last argument to \fItp\fR\& or \fItpl\fR\&\&. The activated \fIItem\fR\& then becomes a \fItrigger\fR\& for sequential tracing\&. This means that if the item is called on a process with trace flag \fIcall\fR\& set, the process is "contaminated" with token \fIseq_trace\fR\&\&.
.LP
If \fIFlags = all\fR\&, all possible flags are set\&.
.LP
The possible values for \fISeqTraceFlag\fR\& are available in \fIseq_trace\fR\&\&.
.LP
For a description of the \fImatch_spec()\fR\& syntax, see section \fIMatch Specifications in Erlang\fR\& in ERTS, which explains the general match specification "language"\&.
.LP

.RS -4
.B
Note:
.RE
The \fIsystem tracer\fR\& for sequential tracing is automatically initiated by \fIttb\fR\& when a trace port is started with \fIttb:tracer/0,1,2\fR\&\&.

.LP
An example of how to use function \fIseq_trigger_ms/0,1\fR\& follows:
.LP
.nf

(tiger@durin)5> ttb:tracer()\&.
{ok,[tiger@durin]}
(tiger@durin)6> ttb:p(all,call)\&.
{ok,{[all],[call]}}
(tiger@durin)7> ttb:tp(mod,func,ttb:seq_trigger_ms())\&.
{ok,[{matched,1},{saved,1}]}
(tiger@durin)8>
.fi
.LP
Whenever \fImod:func(\&.\&.\&.)\fR\& is called after this, token \fIseq_trace\fR\& is set on the executing process\&.
.RE

.LP
.B
stop()
.br
.RS
.LP
Equivalent to \fIstop([])\fR\&\&.
.RE

.LP
.B
stop(Opts) -> stopped | {stopped, Dir}
.br
.RS
.LP
Types:

.RS 3
Opts = Opt | [Opt]
.br
Opt = nofetch | {fetch_dir, Dir} | format | {format, FormatOpts} | return_fetch_dir
.br
Dir = string()
.br
FormatOpts = see format/2
.br
.RE
.RE
.RS
.LP
Stops tracing on all nodes\&. Logs and trace information files are sent to the trace control node and stored in a directory named \fIttb_upload_FileName-Timestamp\fR\&, where \fIFilename\fR\& is the one provided with \fI{file, File}\fR\& during trace setup and \fITimestamp\fR\& is of the form \fIyyyymmdd-hhmmss\fR\&\&. Even logs from nodes on the same machine as the trace control node are moved to this directory\&. The history list is saved to a file named \fIttb_last_config\fR\& for further reference (as it is no longer accessible through history and configuration management functions, like \fIttb:list_history/0\fR\&)\&.
.LP
\fIOptions:\fR\&
.RS 2
.TP 2
.B
\fInofetch\fR\&:
Indicates that trace logs are not to be collected after tracing is stopped\&.
.TP 2
.B
\fI{fetch, Dir}\fR\&:
Allows specification of the directory to fetch the data to\&. If the directory already exists, an error is thrown\&.
.TP 2
.B
\fIformat\fR\&:
Indicates the trace logs to be formatted after tracing is stopped\&. All logs in the fetch directory are merged\&.
.TP 2
.B
\fIreturn_fetch_dir\fR\&:
Indicates the return value to be \fI{stopped, Dir}\fR\& and not just \fIstopped\fR\&\&. This implies \fIfetch\fR\&\&.
.RE
.RE

.LP
.B
get_et_handler()
.br
.RS
.LP
Returns the \fIet\fR\& handler, which can be used with \fIformat/2\fR\& or \fItracer/2\fR\&\&.
.LP
Example: \fIttb:format(Dir, [{handler, ttb:get_et_handler()}])\fR\&\&.
.RE

.LP
.B
format(File)
.br
.RS
.LP
Equivalent to \fIformat(File,[])\fR\&\&.
.RE

.LP
.B
format(File,Options) -> ok | {error, Reason}
.br
.RS
.LP
Types:

.RS 3
File = string() | [string()]
.br
.RS 2
This can be the name of a binary log, a list of such logs, or the name of a directory containing one or more binary logs\&.
.RE
Options = Opt | [Opt]
.br
Opt = {out,Out} | {handler,FormatHandler} | disable_sort
.br
Out = standard_io | string()
.br
FormatHandler = {Function, InitialState}
.br
Function = fun(Fd,Trace,TraceInfo,State) -> State
.br
Fd = standard_io | FileDescriptor
.br
.RS 2
File descriptor of the destination file \fIOut\fR\&\&.
.RE
Trace = tuple()
.br
.RS 2
The trace message\&. For details, see the Reference Manual for module \fIerlang\fR\&\&.
.RE
TraceInfo = [{Key,ValueList}]
.br
.RS 2
Includes the keys \fIflags\fR\&, \fIclient\fR\&, and \fInode\fR\&\&. If \fIhandler\fR\& is specified as option to the tracer function, this is also included\&. Also, all information written with function \fIwrite_trace_info/2\fR\& is included\&.
.RE
.RE
.RE
.RS
.LP
Reads the specified binary trace log(s)\&. The logs are processed in the order of their time stamps as long as option \fIdisable_sort\fR\& is not specified\&.
.LP
If \fIFormatHandler = {Function,InitialState}\fR\&, \fIFunction\fR\& is called for each trace message\&.
.LP
If \fIFormatHandler = get_et_handler()\fR\&, \fIet_viewer\fR\& in application ET is used for presenting the trace log graphically\&. \fIttb\fR\& provides a few different filters that can be selected from menu \fIFilters and scaling\fR\& in the \fIet_viewer\fR\&\&.
.LP
If \fIFormatHandler\fR\& is not specified, a default handler is used presenting each trace message as a text line\&.
.LP
The state returned from each call of \fIFunction\fR\& is passed to the next call, even if the next call is to format a message from another log file\&.
.LP
If \fIOut\fR\& is specified, \fIFormatHandler\fR\& gets the file descriptor to \fIOut\fR\& as the first parameter\&.
.LP
\fIOut\fR\& is ignored if the \fIet\fR\& format handler is used\&.
.LP
Wrap logs can be formatted one by one or all at once\&. To format one of the wrap logs in a set, specify the exact file name\&. To format the whole set of wrap logs, specify the name with \fI*\fR\& instead of the wrap count\&. For examples, see the \fIUser\&'s Guide\fR\&\&.
.RE