'\"macro stdmacro
.\"
.\" Copyright (c) 2016 Red Hat.  All Rights Reserved.
.\" Copyright (c) 2000 Silicon Graphics, Inc.  All Rights Reserved.
.\"
.\" This program is free software; you can redistribute it and/or modify it
.\" under the terms of the GNU General Public License as published by the
.\" Free Software Foundation; either version 2 of the License, or (at your
.\" option) any later version.
.\"
.\" This program is distributed in the hope that it will be useful, but
.\" WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
.\" or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
.\" for more details.
.\"
.\"
.TH PMLOGEXTRACT 1 "PCP" "Performance Co-Pilot"
.SH NAME
\f3pmlogextract\f1 \- reduce, extract, concatenate
and merge Performance Co-Pilot archives
.SH SYNOPSIS
\f3pmlogextract\f1
[\f3\-dfmwxz?\f1]
[\f3\-c\f1 \f2configfile\f1]
[\f3\-S\f1 \f2starttime\f1]
[\f3\-s\f1 \f2samples\f1]
[\f3\-T\f1 \f2endtime\f1]
[\f3\-V\f1 \f2version\f1]
[\f3\-v\f1 \f2volsamples\f1]
[\f3\-Z\f1 \f2timezone\f1]
\f2input\f1 [...] \f2output\f1
.SH DESCRIPTION
.B pmlogextract
reads one or more Performance Co-Pilot (PCP) archives
identified by
.I input
and creates a merged and/or reduced PCP archive in
.IR output .
Each
.I input
argument is either a name or
a comma-separated list of names, and each name
is the name of one file from an archive or
the base name of an archive or the name of a directory containing
one or more archives.
The nature of merging is controlled by the number of
.I input
archives, while the nature of data reduction is controlled by
the command line arguments.
The
.I input
arguments must be archives created by
.BR pmlogger (1)
with performance data collected from the
.B same
host, but usually over different time periods and possibly (although
not usually) with different performance metrics being logged.
.PP
If only one
.I input
is specified, then the default behavior simply copies the
.I input
PCP archive (with possible conversion to a newer
version of the archive format, see
.B \-V
below), into the
.I output
PCP archive.
When two or more PCP archives are specified as
.IR input ,
the archives are merged (or concatenated) and written to
.IR output .
.PP
In the
.I output
archive a
.B <mark>
record may be inserted at a time
just past the end of each of the
.I input
archive to indicate
a possible temporal discontinuity between the end of one
.I input
archive and the start of the next
.I input
archive.
See the
.B "MARK RECORDS"
section below for more information.
There is no
.B <mark>
record after the end of the
.B last
(in temporal order) of the records from the
.I input
archive(s).
.SH OPTIONS
The available command line options are:
.TP 5
\fB\-c\fR \fIconfig\fR, \fB\-\-config\fR=\fIconfig\fR
Extract only the metrics specified in
.I config
from the
.I input
PCP archive(s).
The
.I config
syntax accepted by
.B pmlogextract
is explained in more detail in the
.B CONFIGURATION FILE SYNTAX
section.
.TP
\fB\-d\fR, \fB\-\-desperate\fR
Desperate mode.
Normally if a fatal error occurs, all trace of
the partially written PCP archive
.I output
is removed.
With the
.B \-d
option, the
.I output
archive is not removed.
.TP
\fB\-f\fR, \fB\-\-first\fR
For most common uses, all of the
.I input
archives will have been collected in the same timezone.
But if this is not the case, then
.B pmlogextract
must choose one of the timezones from the
.I input
archives to be
used as the timezone for the
.I output
archive.
The default is to use the timezone from the
.B last
.I input
archive.
The
.B \-f
option forces the timezone from the
.B first
.I input
archive to be used.
.TP
\fB\-m\fR, \fB\-\-mark\fR
As described in the
.B "MARK RECORDS"
section below, sometimes it is possible to safely omit
.B <mark>
records from the
.I output
archive.
If the
.B \-m
option is specified, then the
.B epilogue
and
.B prologue
test is skipped and a
.B <mark>
record will always be inserted at the end of each
.I input
archive (except the last).
This is the original behaviour for
.BR pmlogextract .
.TP
\fB\-S\fR \fIstarttime\fR, \fB\-\-start\fR=\fIstarttime\fR
Define the start of a time window to restrict the records processed;
refer to
.BR PCPIntro (1).
See also the
.B \-w
option.
.TP
\fB\-s\fR \fIsamples\fR, \fB\-\-samples\fR=\fIsamples\fR
The argument
.I samples
defines the number of samples (or records) to be written to
.IR output .
If
.I samples
is 0 or
.B -s
is not specified,
.B pmlogextract
will continue until the end of all the
.I input
archives
or until the end of the time window as specified by
.BR -T ,
whichever comes first.
The
.B -s
option will override the
.B -T
option if it occurs sooner.
.TP
\fB\-T\fR \fIendtime\fR, \fB\-\-finish\fR=\fIendtime\fR
Define the end of a time window to restrict the records processed;
refer to
.BR PCPIntro (1).
See also the
.B \-w
option.
.TP
\fB\-V\fR \fIversion\fR, \fB\-\-outputversion\fR=\fIversion\fR
Each PCP archive has a version for the physical record format,
currently 2 or 3.
By default, the
.I output
archive is created with a version equal to the
.B maximum
of the version of the
.I input
archives.
The
.B \-V
option may be used to explicitly force the version for
.IR output ,
provided
.I version
is no smaller than the archive version that would have been
chosen by the default rule.
.RS
.PP
For example, specifying
.B \-V\ 3
may be used to produce a version 3
.I output
archive from
.I input
archives that could be a mixture of version 2 and/or version 3.
.RE
.TP
\fB\-v\fR \fIvolsamples\fR
The
.I output
archive is potentially a multi-volume data set, and the
.B \-v
option causes
.B pmlogextract
to start a new volume after
.I volsamples
records have been written to the archive.
.RS
.PP
Independent of any
.B \-v
option, each volume of an archive is limited to no more than
2^31 bytes, so
.B pmlogextract
will automatically create a new volume for the archive before
this limit is reached.
.RE
.TP
\fB\-w\fR
Where
.B \-S
and
.B \-T
specify a time window within the same day, the
.B \-w
flag will cause the data within the time window to be extracted,
for
.B every
day in the archive.
For example, the options
.B \-w \-S "@11:00" \-T "@15:00"
specify that
.B pmlogextract
should include archive records only for the periods from 11am
to 3pm on each day.
When
.B \-w
is used, the
.I output
archive will contain
.B <mark>
records to indicate the temporal
discontinuity between the end of one time window and the start of
the next.
.TP
\fB\-x\fR
It is expected that the metadata
(name, PMID, type, semantics and units)
for each metric
will be consistent across all of the
.I input
PCP archive(s) in which that metric appears.
In rare cases, e.g. in development, in QA and when a PMDA is upgraded, this may not be the case
and
.B pmlogextract
will report the issue and abort without creating the
.I output
archive.
This is done so the problem can be fixed with
.BR pmlogrewrite (1)
before retrying the merge.
In unattended or QA environments it may be preferable to force the
merge and omit the metrics with the mismatched metadata.
The
.B \-x
option does this.
.TP
\fB\-Z\fR \fItimezone\fR, \fB\-\-timezone\fR=\fItimezone\fR
Use
.I timezone
when displaying the date and time in diagnostics.
.I Timezone
is in the format of the environment variable
.B TZ
as described in
.BR environ (7).
The default is to initially use the timezone of the local host.
.TP
\fB\-z\fR, \fB\-\-hostzone\fR
Use the local timezone of the host from the
.I input
archive(s) when displaying the date and time in diagnostics.
The default is to initially use the timezone of the local host.
.TP
\fB\-?\fR, \fB\-\-help\fR
Display usage message and exit.
.SH CONFIGURATION FILE SYNTAX
The
.I configfile
contains metrics of interest \- only those metrics (or instances)
mentioned explicitly or implicitly in the configuration file will be
included in the
.I output
archive.
Each specification must begin on a new line, and may span multiple lines
in the configuration file.
Instances
may also be specified, but they are optional.
The format for each specification is
.sp
.nf
        metric
or
        metric \fB[\fP instance ... \fB]\fP
.fi
.sp
where
.I metric
may be a leaf or a non-leaf name of a metric in the Performance Metrics
Name Space (PMNS, see
.BR PMNS (5)).
If a
.I metric
refers to a non-leaf node in the PMNS,
.B pmlogextract
will recursively descend the PMNS and include all metrics
corresponding to descendent leaf nodes.
.PP
Instances are
optional and are specified as a list space (or comma) separated of
.I instance
identifiers, with the list enclosed by square brackets.
Each
.I instance
identifier may be
a number or a string (enclosed in single or double quotes).
.I instance
identifiers that are numbers are assumed to be internal
instance identifiers, else the string values are assumed to be
external instance identifiers; see
.BR pmGetInDom (3)
for more information.
If no instances are given, then
.B all
instances of the associated metric(s) will be extracted.
.PP
Any additional white space is ignored and
comments may be added with a `#' prefix.
.SH CONFIGURATION FILE EXAMPLE
This is an example of a valid
.IR configfile :
.PP
        #
        # config file for pmlogextract
        #

        kernel.all.cpu
        kernel.percpu.cpu.sys ["cpu0","cpu1"]
        disk.dev ["dks0d1"]
.SH MARK RECORDS
When more than one
.I input
archive contributes performance data to the
.I output
archive, then
.B <mark>
records may be inserted to indicate a possible
temporal discontinuity in the performance data.
.PP
A
.B <mark>
record contains a timestamp and no performance data and
is used to indicate that there is a time period
in the PCP archive where we do not know the values of
.B any
performance metrics, because there was no
.BR pmlogger (1)
collecting performance data during this period.
Since these periods are
often associated with the restart of a service or
.BR pmcd (1)
or a system reboot, there may be considerable doubt as to the continuity of
performance data across this time period.
.PP
Most current archives are created with a
.B prologue
record at the beginning and an
.B epilogue
record at the end.
These records identify the state of
.BR pmcd (1)
at the time, and may be used by
.B pmlogextract
to determine that there is no discontinuity between the end of
one archive and the next output record, and as a consequence the
.B <mark>
record can safely be omitted from the
.I output
archive.
.PP
The rationale behind
.B <mark>
records may be demonstrated with an example.
Consider one
.I input
archive that starts at 00:10 and ends at 09:15 on the
same day, and another
.I input
archive that starts at 09:20 on the
same day and ends at 00:10 the following morning.
This would be a very common case for archives managed and rotated by
.BR pmlogger_check (1)
and
.BR pmlogger_daily (1).
.PP
The
.I output
archive created by
.B pmlogextract
would contain:
.ta 12n
.br
00:10.000\ \ \ \ first record from first input archive
.br
\&...
.br
09:15.000\ \ \ \ last record from first input archive
.br
09:15.001\ \ \ \ <mark> record
.br
09:20.000\ \ \ \ first record from second input archive
.br
\&...
.br
01:10.000\ \ \ \ last record from second input archive
.PP
The time period where the performance data is missing starts just after
09:15 and ends just before 09:20.
When the
.I output
archive is processed with any of the PCP reporting
tools, the
.B <mark>
record is used to indicate a period of missing data.
For example using the
.I output
archive above, imagine one was reporting the average
I/O rate at 30 minute intervals
aligned on the hour and half-hour.
The I/O count metric is a counter, so the
average I/O rate requires two valid values from
consecutive sample times.
There would be
values for all the intervals ending at 09:00,
then no values at 09:30 because of the
.B <mark>
record, then no values at 10:00 because the ``prior'' value at 09:30 is not
available, then the rate would be reported again at 10:30 and continue
every 30 minutes until the last reported value at 01:00.
.PP
The presence of
.B <mark>
records in a PCP archive can be established
using
.BR pmdumplog (1)
where a timestamp and the annotation
.B <mark>
is used to indicate a
.B <mark>
record.
.SH METADATA CHECKS
When more than one
.I input
archive is specified,
.B pmlogextract
performs a number of checks to ensure the metadata is consistent for
metrics appearing in more than one of the
.I input
archives.
These checks include:
.IP * 2n
metric data type is the same
.PD 0
.IP * 2n
metric semantics are the same
.IP * 2n
metric units are the same
.IP * 2n
metric is either always singular or always has the same instance domain
.IP * 2n
metrics with the same name have the same PMID
.IP * 2n
metrics with the same PMID have the same name
.PD
.PP
If any of these checks fail,
.B pmlogextract
reports the details and aborts without creating the
.I output
archive.
.PP
To address these semantic issues, use
.BR pmlogrewrite (1)
to translate the
.I input
archives into equivalent archives with consistent metadata before using
.BR pmlogextract .
.PP
Refer to the
.B \-x
and
.B \-d
command line options above for alternatives to the
default handling of errors during metadata checks.
.SH CAVEATS
The
.B prologue
metrics (pmcd.pmlogger.archive, pmcd.pmlogger.host,
and pmcd.pmlogger.port), which are automatically recorded by
.B pmlogger
at the start of the archive, may not be present in the archive output by
.BR pmlogextract .
These metrics are only relevant while the archive is being created,
and have no significance once recording has finished.
.SH DIAGNOSTICS
All error conditions detected by
.B pmlogextract
are reported on
.I stderr
with textual (if sometimes terse) explanation.
.PP
If one of the
.I input
archives contains no archive records then
an ``empty archive''
warning is issued and that archive is skipped.
.PP
Should one of the
.I input
archive(s) be corrupted (this can happen
if the
.B pmlogger
instance writing the archive suddenly dies), then
.B pmlogextract
will detect and report the position of the corruption in the file,
and any subsequent information from that archive will not be processed.
.PP
If any error is detected,
.B pmlogextract
will exit with a non-zero status.
.SH FILES
For each of the
.I input
and
.I output
archive, several physical files are used.
.TP 5
\f2archive\f3.meta
metadata (metric descriptions, instance domains, etc.) for the archive
.TP
\f2archive\f3.0
initial volume of metrics values (subsequent volumes have suffixes
.BR 1 ,
.BR 2 ,
\&...) \- for
.I input
these files may have been previously compressed with
.BR bzip2 (1)
or
.BR gzip (1)
and thus may have an additional
.B .bz2
or
.B .gz
suffix.
.TP
\f2archive\f3.index
temporal index to support rapid random access to the other files in the
archive.
.SH PCP ENVIRONMENT
Environment variables with the prefix \fBPCP_\fP are used to parameterize
the file and directory names used by PCP.
On each installation, the
file \fI/etc/pcp.conf\fP contains the local values for these variables.
The \fB$PCP_CONF\fP variable may be used to specify an alternative
configuration file, as described in \fBpcp.conf\fP(5).
.PP
For environment variables affecting PCP tools, see \fBpmGetOptions\fP(3).
.SH SEE ALSO
.BR PCPIntro (1),
.BR pmdumplog (1),
.BR pmlc (1),
.BR pmlogger (1),
.BR pmlogreduce (1),
.BR pmlogrewrite (1),
.BR pcp.conf (5),
.BR pcp.env (5)
and
.BR PMNS (5).