'\" t
'\"macro stdmacro
.\"
.\" Copyright (c) 2014-2018 Red Hat.
.\" 
.\" This program is free software; you can redistribute it and/or modify it
.\" under the terms of the GNU General Public License as published by the
.\" Free Software Foundation; either version 2 of the License, or (at your
.\" option) any later version.
.\" 
.\" This program is distributed in the hope that it will be useful, but
.\" WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
.\" or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
.\" for more details.
.\" 
.TH LOGARCHIVE 5 "" "Performance Co-Pilot"
.SH NAME
\f3LOGARCHIVE\f1 \- performance metrics archive format
.SH DESCRIPTION
Performance Co-Pilot (PCP) archives store historical values
about arbitrary metrics recorded from a single host.
Archives are machine independent and self-contained \- all
metadata required for off-line or off-site analysis is stored.
.PP
The format is stable in order to allow long-term historical
storage and processing by PMAPI client tools.
.PP
Archives may be read by most PCP client tools, using the 
.IR "\-a/\-\-archive NAME"
option, or dumped raw by
.BR pmdumplog (1).
Archives are created primarily by 
.BR pmlogger (1),
however they can also be created using the
.BR LOGIMPORT (3)
programming interface.
.PP
Archives may be merged, analyzed, modified and subsampled using
.BR pmlogreduce (1),
.BR pmlogsummary (1),
.BR pmlogrewrite (1)
and
.BR pmlogextract (1).
In addition, PCP archives may be examined in sets or grouped
together into "archive folios", which are created and managed
by the
.BR mkaf (1)
and
.BR pmafm (1)
tools.
.PP
Archives consist of several physical files that share a common
arbitrary prefix, e.g.
.IR myarchive .
.TP
\f2myarchive\f1.0, \f2myarchive\f1.1, ...
One or more data volumes containing the metric values and any
error codes encountered during metric sampling.
Typically the largest of the files and may grow very rapidly,
depending on the
.B pmlogger
sampling interval(s) being used.
.TP
.IR myarchive .meta
Information for PMAPI functions such as
.BR pmLookupDesc (3),
.BR pmLookupLabels (3)
and
.BR pmLookupInDom (3).
The metadata file may grow sporadically as logged metrics,
instance domains and labels vary over time.
.TP
.IR myarchive .index
A temporal index, mapping timestamps to offsets in the other files.
.SH COMMON FEATURES
All three types of files have a similar record-based structure, a
convention of network-byte-order (big-endian) encoding, and 32-bit
fields for tagging/padding for those records.
Strings are stored as 8-bit characters without assuming a specific
encoding, so normally ASCII.
See also the
.BR __pmLog* 
types in
.IR libpcp.h .
.SS RECORD FRAMING
.PP
The volume and meta files are divided into self-identifying records.
.TS
box,center;
c | c | c
c | c | l.
Offset	Length	Name
_
0	4	N, length of record, in bytes, including this field
4	N-8	record payload, usually starting with a 32-bit tag
N-4	4	N, length of record (again)
.TE

.SS ARCHIVE LOG LABEL
All three types of files begin with a "log label" header, which
identifies the host name, the time interval covered, and a time zone.
.TS
box,center;
c | c | c
c | c | l.
Offset	Length	Name
_
0	4	tag, PM_LOG_MAGIC | PM_LOG_VERS02=0x50052602
4	4	pid of pmlogger process that wrote file
8	4	log start time, seconds part (past UNIX epoch)
12	4	log start time, microseconds part
16	4	current log volume number (or \-1=.meta, \-2=.index)
20	64	name of collection host
80	40	time zone string ($TZ environment variable)
.TE

.PP
All fields, except for the current log volume number field, match for
all archive-related files produced by a single run of the tool.
.SH ARCHIVE VOLUME (.0, .1, ...) RECORDS
.SS pmResult
After the archive log label record, an archive volume file contains
metric values corresponding to the 
.IR pmResult
set of one 
.IR pmFetch
operation, which is almost identical to the form on disk.
The record size may vary according to number of PMIDs being fetched,
the number of instances for their domains.
File size is limited to 2GiB, due to storage of 32-bit offsets within
the temporal index.
.TS
box,center;
c | c | c
c | c | l.
Offset	Length	Name
_
0	4	timestamp, seconds part (past UNIX epoch)
4	4	timestamp, microseconds part
8	4	number of PMIDs with data following
12	M	pmValueSet #0
12+M	N	pmValueSet #1
12+M+N	...	...
NOP	X	pmValueBlock #0
NOP+X	Y	pmValueBlock #1
NOP+X+Y	...	...
.TE

.PP
Records with a number-of-PMIDs equal to zero are "mark records", and
represent interruptions, missing data, or time discontinuities in
logging.
.SS pmValueSet
This subrecord represents the measurements for one metric.
.TS
box,center;
c | c | c
c | c | l.
Offset	Length	Name
_
0	4	PMID
4	4	number of values
8	4	storage mode, PM_VAL_INSITU=0 or PM_VAL_DPTR=1
12	M	pmValue #0
12+M	N	pmValue #1
12+M+N	...	...
.TE

.PP
The metric-description metadata for PMIDs is found in the .meta files.
These entries are not timestamped, so the metadata is assumed to be
unchanging throughout the archiving session.
.SS pmValue
This subrecord represents one measurement for one instance of the metric.
It is a variant type, depending on the parent pmValueSet's value-format
field.  This allows small numbers to be encoded compactly, but retain
flexibility for larger or variable-length data to be stored later in the
pmResult record.
.TS
box,center;
c | c | c
c | c | l.
Offset	Length	Name
_
0	4	number in instance-domain (or PM_IN_NULL=-1)
4	4	value (INSITU) \fIor\fR
		offset in pmResult to our pmValueBlock (DPTR)
.TE

.PP
The instance-domain metadata for PMIDs is found in the .meta files.
Since the numeric mappings may change during the lifetime of the
logging session, it is important to match up the timestamp of the
measurement record with the corresponding instance-domain record.
That is, the instance-domain corresponding to a measurement at time T
are the records with largest timestamps T' <= T.
.SS pmValueBlock
Instances of this subrecord are placed at the end of the 
.IR pmValueSet ,
after all the 
.IR pmValue
subrecords.
If (and only if) needed, they are padded at the end to the
next-higher 32-bit boundary.
.TS
box,center;
c | c | c
c | c | l.
Offset	Length	Name
_
0	1	value type (same as \fIpmDesc.type\fR)
1	3	4 + N, the length of the subrecord
4	N	bytes that make up the raw value
4+N	0-3	padding (not included in the 4+N length field)
.TE

.PP
Note that for
.IR PM_TYPE_STRING ,
the length includes an explicit NULL terminator byte.
For
.IR PM_TYPE_EVENT ,
the value bytestring is further structured.
.\" .SS pmEventArray
.SH METADATA FILE (.meta) RECORDS
After the archive log label record, the metadata file contains
interleaved metric-description and timestamped instance-domain
descriptors.
File size is limited to 2GiB, due to storage of 32-bit offsets
within the temporal index.
Unlike the data volumes, these records are not forced to 32-bit
alignment.
See also
.IR libpcp/logmeta.c .
.SS pmDesc 
Instances of this record represent the metric description, giving a
name, type, instance-domain identifier, and a set of names to each
PMID used in the archive volume.
.TS
box,center;
c | c | c
c | c | l.
Offset	Length	Name
_
0	4	tag, TYPE_DESC=1
4	4	PMID
8	4	type (PM_TYPE_*)
12	4	instance domain number
16	4	semantics of value (PM_SEM_*)
20	4	units: bit-packed pmUnits
4	4	number of alternative names for this PMID
28	4	N: number of bytes in this name
32	N	bytes of the name, no NULL terminator nor padding
32+N	4	N2: number of bytes in next name 
36+N	N2	bytes of the name, no NULL terminator nor padding
\...	...	...
.TE

.PP
.SS pmLogIndom
Instances of this record represent the number-string mapping
table of an instance domain.
The instance domain number will have already been mentioned
in a prior
.IR pmDesc
record.
As new instances may appear over a long archiving run these
records are timestamped, and must be searched when decoding 
.IR pmResult
records from the archive data volumes.
Instance names may be reused between instance numbers, so an
offset-based string table is used that facilitates sharing.
.TS
box,center;
c | c | c
c | c | l.
Offset	Length	Name
_
0	4	tag, TYPE_INDOM=2
4	4	timestamp, seconds part (past UNIX epoch)
8	4	timestamp, microseconds part
12	4	instance domain number
16	4	N: number of instances in domain, normally >0
20	4	first instance number
24	4	second instance number (if appropriate)
\...	...	...
20+4*N	4	first offset into string table (see below)
20+4*N+4	4	second offset into string table (etc.)
\...	...	...
20+8*N	M	base of string table, containing
		packed, NULL-terminated instance names
.TE

.PP
Records of this form \fIreplace\fR the existing instance-domain: prior
records are not searched for resolving instance numbers in measurements
after this timestamp.
.SS pmLogLabelSet
Instances of this record represent sets of name:value pairs
associated with labels of the context, instance domains and
individual performance metrics \- refer to
.BR pmLookupLabels (3)
for further details.
.PP
Any instance domain number will have already been mentioned
in a prior
.IR pmDesc
record.
As new labels can appear during an archiving session, these
records are timestamped and must be searched when decoding 
.IR pmResult
records from the archive data volumes.
.TS
box,center;
c | c | c
c | c | l.
Offset	Length	Name
_
0	4	tag, TYPE_LABEL=3
4	4	timestamp, seconds part (past UNIX epoch)
8	4	timestamp, microseconds part
12	4	label type (PM_LABEL_* type macros.)
16	4	numeric identifier - domain, PMID, etc
		or PM_IN_NULL=-1 for context labels
20	4	N: number of label sets in this record,
		usually 1 except in the case of instances
24	4	offset to the start of the JSONB labels string
28	L1	first labelset array entry (see below)
\...	...	...
28+L1	LN	N-th labelset array entry (see below)
\...	...	...
28+L1+...LN	M	concatenated JSONB strings for all labelsets
.TE

.PP
Records of this form \fIreplace\fR the existing labels for a given
type: prior records are not searched for resolving that class of
label in measurements after this timestamp.
.PP
The individual labelset array entries are variable length, depending
on the number of labels present within that set.
These entries contain the instance identifiers (in the case of type
PM_LABEL_INSTANCES labels), lengths and offsets of each label name
and value, and also any flags set for each label.
.TS
box,center;
c | c | c
c | c | l.
Offset	Length	Name
_
0	4	instance identifier (or PM_IN_NULL=-1)
4	4	length of JSONB label string
8	4	N: number of labels in this labelset
12	2	first label name offset
14	1	first label name length
15	1	first label flags (e.g. optionality)
16	2	first label value offset
18	2	first label value length
20	2	second label name offset (if appropriate)
\...	...	...
.TE

.SS pmLogText
This record stores help text associated with a metric or an
instance domain \- as provided by
.BR pmLookupText (3)
and
.BR pmLookupInDomText (3).
.PP
The metric identifier and instance domain number will have
already been mentioned in a prior
.IR pmDesc
record.

.TS
box,center;
c | c | c
c | c | l.
Offset	Length	Name
_
0	4	tag, TYPE_TEXT=4
4	4	text and identifier type (PM_TEXT_* macros.)
8	4	numeric identifier - PMID or instance domain
12	M	help text string, arbitrary text
.TE

.SH INDEX FILE (.index) RECORDS
After the archive log label record, the temporal index file contains a
plainly concatenated, unframed group of tuples, which relate timestamps
to 32-bit seek offsets in the volume and meta files.
These records are fixed-size, fixed-format, and are \fInot\fR enclosed
in the standard length/payload/length wrapper: they take up the entire
remainder of the .index file.
See also
.IR libpcp/logutil.c .
.TS
box,center;
c | c | c
c | c | l.
Offset	Length	Name
_
0	4	event time, seconds part (past UNIX epoch)
4	4	event time, microseconds part
8	4	archive volume number (0...N)
12	4	byte offset in .meta file of pmDesc or pmLogIndom
16	4	byte offset in archive volume file of pmResult
.TE

.PP
Since the temporal index is optional, and exists only to speed up
time-based random access to metrics and their metadata, the index
records are emitted only intermittently.
An archive reader program should not presume any particular rate of
data flow into the index.
However, common events that may trigger a new temporal-index record
include changes in instance-domains, switching over to a new archive
volume, and starting or stopping logging.
One reliable invariant however is that, for each index entry, there
are to be no meta or archive-volume records with a timestamp after
that in the index, but physically before the byte-offset in the index.
.SH FILES
Several PCP tools create archives in standard locations:
.PP
.PD 0
.TP 10
.B $HOME/.pcp/pmlogger
default location for the interactive chart recording mode in
.BR pmchart (1)
.TP 10
.B $PCP_LOG_DIR/pmlogger
default location for
.BR pmlogger_daily (1)
and
.BR pmlogger_check (1)
scripts
.TP 10
.B $PCP_LOG_DIR/pmmgr
default location for the PCP daemon manager
.BR pmmgr (1)
.PD
.SH SEE ALSO
.BR PCPIntro (1),
.BR PMAPI (3),
.BR pmLookupDesc (3),
.BR pmLookupInDom (3),
.BR pmLookupInDomText (3),
.BR pmLookupLabels (3),
.BR pmLookupText (3),
.BR mkaf (1),
.BR pmafm (1),
.BR pmchart (1),
.BR pmdumplog (1),
.BR pmlogger (1),
.BR pmlogger_check (1),
.BR pmlogger_daily (1),
.BR pmlogreduce (1),
.BR pmlogrewrite (1),
.BR pmlogsummary (1),
.BR pmmgr (1),
.BR pcp.conf (5),
and
.BR pcp.env (5).