NAME¶
libarchive_internals
—
description of libarchive internal interfaces
OVERVIEW¶
The
libarchive
library provides a flexible
interface for reading and writing streaming archive files such as tar and
cpio. Internally, it follows a modular layered design that should make it easy
to add new archive and compression formats.
GENERAL ARCHITECTURE¶
Externally, libarchive exposes most operations through an opaque, object-style
interface. The
archive_entry(3) objects store
information about a single filesystem object. The rest of the library provides
facilities to write
archive_entry(3) objects to
archive files, read them from archive files, and write them to disk. (There
are plans to add a facility to read
archive_entry(3) objects from disk as well.)
The read and write APIs each have four layers: a public API layer, a format
layer that understands the archive file format, a compression layer, and an
I/O layer. The I/O layer is completely exposed to clients who can replace it
entirely with their own functions.
In order to provide as much consistency as possible for clients, some public
functions are virtualized. Eventually, it should be possible for clients to
open an archive or disk writer, and then use a single set of code to select
and write entries, regardless of the target.
READ ARCHITECTURE¶
From the outside, clients use the
archive_read(3)
API to manipulate an
archive
object to read
entries and bodies from an archive stream. Internally, the
archive
object is cast to an
archive_read
object, which holds all
read-specific data. The API has four layers: The lowest layer is the I/O
layer. This layer can be overridden by clients, but most clients use the
packaged I/O callbacks provided, for example, by
archive_read_open_memory(3), and
archive_read_open_fd(3). The compression layer
calls the I/O layer to read bytes and decompresses them for the format layer.
The format layer unpacks a stream of uncompressed bytes and creates
archive_entry
objects from the incoming
data. The API layer tracks overall state (for example, it prevents clients
from reading data before reading a header) and invokes the format and
compression layer operations through registered function pointers. In
particular, the API layer drives the format-detection process: When opening
the archive, it reads an initial block of data and offers it to each
registered compression handler. The one with the highest bid is initialized
with the first block. Similarly, the format handlers are polled to see which
handler is the best for each archive. (Prior to 2.4.0, the format bidders were
invoked for each entry, but this design hindered error recovery.)
I/O Layer and Client Callbacks¶
The read API goes to some lengths to be nice to clients. As a result, there are
few restrictions on the behavior of the client callbacks.
The client read callback is expected to provide a block of data on each call. A
zero-length return does indicate end of file, but otherwise blocks may be as
small as one byte or as large as the entire file. In particular, blocks may be
of different sizes.
The client skip callback returns the number of bytes actually skipped, which may
be much smaller than the skip requested. The only requirement is that the skip
not be larger. In particular, clients are allowed to return zero for any skip
that they don't want to handle. The skip callback must never be invoked with a
negative value.
Keep in mind that not all clients are reading from disk: clients reading from
networks may provide different-sized blocks on every request and cannot skip
at all; advanced clients may use
mmap(2) to read
the entire file into memory at once and return the entire file to libarchive
as a single block; other clients may begin asynchronous I/O operations for the
next block on each request.
Decompresssion Layer¶
The decompression layer not only handles decompression, it also buffers data so
that the format handlers see a much nicer I/O model. The decompression API is
a two stage peek/consume model. A read_ahead request specifies a minimum read
amount; the decompression layer must provide a pointer to at least that much
data. If more data is immediately available, it should return more: the format
layer handles bulk data reads by asking for a minimum of one byte and then
copying as much data as is available.
A subsequent call to the
consume
() function
advances the read pointer. Note that data returned from a
read_ahead
() call is guaranteed to remain
in place until the next call to
read_ahead
(). Intervening calls to
consume
() should not cause the data to
move.
Skip requests must always be handled exactly. Decompression handlers that cannot
seek forward should not register a skip handler; the API layer fills in a
generic skip handler that reads and discards data.
A decompression handler has a specific lifecycle:
- Registration/Configuration
- When the client invokes the public support function, the decompression
handler invokes the internal
__archive_read_register_compression
()
function to provide bid and initialization functions. This function
returns NULL
on error or else a pointer
to a struct decompressor_t
. This
structure contains a void * config slot
that can be used for storing any customization information.
- Bid
- The bid function is invoked with a pointer and size of a block of data.
The decompressor can access its config data through the
decompressor element of the
archive_read
object. The bid function
is otherwise stateless. In particular, it must not perform any I/O
operations.
The value returned by the bid function indicates its suitability for
handling this data stream. A bid of zero will ensure that this
decompressor is never invoked. Return zero if magic number checks fail.
Otherwise, your initial implementation should return the number of bits
actually checked. For example, if you verify two full bytes and three bits
of another byte, bid 19. Note that the initial block may be very short; be
careful to only inspect the data you are given. (The current decompressors
require two bytes for correct bidding.)
- Initialize
- The winning bidder will have its init function called. This function
should initialize the remaining slots of the
struct decompressor_t object pointed to
by the decompressor element of the
archive_read object. In particular, it
should allocate any working data it needs in the
data slot of that structure. The init
function is called with the block of data that was used for tasting. At
this point, the decompressor is responsible for all I/O requests to the
client callbacks. The decompressor is free to read more data as and when
necessary.
- Satisfy I/O requests
- The format handler will invoke the
read_ahead,
consume, and
skip functions as needed.
- Finish
- The finish method is called only once when the archive is closed. It
should release anything stored in the
data and
config slots of the
decompressor object. It should not invoke
the client close callback.
The read formats have a similar lifecycle to the decompression handlers:
- Registration
- Allocate your private data and initialize your pointers.
- Bid
- Formats bid by invoking the
read_ahead
() decompression method but
not calling the consume
() method. This
allows each bidder to look ahead in the input stream. Bidders should not
look further ahead than necessary, as long look aheads put pressure on the
decompression layer to buffer lots of data. Most formats only require a
few hundred bytes of look ahead; look aheads of a few kilobytes are
reasonable. (The ISO9660 reader sometimes looks ahead by 48k, which should
be considered an upper limit.)
- Read header
- The header read is usually the most complex part of any format. There are
a few strategies worth mentioning: For formats such as tar or cpio,
reading and parsing the header is straightforward since headers alternate
with data. For formats that store all header data at the beginning of the
file, the first header read request may have to read all headers into
memory and store that data, sorted by the location of the file data.
Subsequent header read requests will skip forward to the beginning of the
file data and return the corresponding header.
- Read Data
- The read data interface supports sparse files; this requires that each
call return a block of data specifying the file offset and size. This may
require you to carefully track the location so that you can return
accurate file offsets for each read. Remember that the decompressor will
return as much data as it has. Generally, you will want to request one
byte, examine the return value to see how much data is available, and
possibly trim that to the amount you can use. You should invoke consume
for each block just before you return it.
- Skip All Data
- The skip data call should skip over all file data and trailing padding.
This is called automatically by the API layer just before each header
read. It is also called in response to the client calling the public
data_skip
() function.
- Cleanup
- On cleanup, the format should release all of its allocated memory.
API Layer¶
XXX to do XXX
WRITE ARCHITECTURE¶
The write API has a similar set of four layers: an API layer, a format layer, a
compression layer, and an I/O layer. The registration here is much simpler
because only one format and one compression can be registered at a time.
I/O Layer and Client Callbacks¶
XXX To be written XXX
Compression Layer¶
XXX To be written XXX
XXX To be written XXX
API Layer¶
XXX To be written XXX
WRITE_DISK ARCHITECTURE¶
The write_disk API is intended to look just like the write API to clients. Since
it does not handle multiple formats or compression, it is not layered
internally.
GENERAL SERVICES¶
The
archive_read
,
archive_write
, and
archive_write_disk
objects all contain an
initial
archive
object which provides
common support for a set of standard services. (Recall that ANSI/ISO C90
guarantees that you can cast freely between a pointer to a structure and a
pointer to the first element of that structure.) The
archive
object has a magic value that
indicates which API this object is associated with, slots for storing error
information, and function pointers for virtualized API functions.
MISCELLANEOUS NOTES¶
Connecting existing archiving libraries into libarchive is generally quite
difficult. In particular, many existing libraries strongly assume that you are
reading from a file; they seek forwards and backwards as necessary to locate
various pieces of information. In contrast, libarchive never seeks backwards
in its input, which sometimes requires very different approaches.
For example, libarchive's ISO9660 support operates very differently from most
ISO9660 readers. The libarchive support utilizes a work-queue design that
keeps a list of known entries sorted by their location in the input. Whenever
libarchive's ISO9660 implementation is asked for the next header, checks this
list to find the next item on the disk. Directories are parsed when they are
encountered and new items are added to the list. This design relies heavily on
the ISO9660 image being optimized so that directories always occur earlier on
the disk than the files they describe.
Depending on the specific format, such approaches may not be possible. The ZIP
format specification, for example, allows archivers to store key information
only at the end of the file. In theory, it is possible to create ZIP archives
that cannot be read without seeking. Fortunately, such archives are very rare,
and libarchive can read most ZIP archives, though it cannot always extract as
much information as a dedicated ZIP program.
SEE ALSO¶
archive(3),
archive_entry(3),
archive_read(3),
archive_write(3),
archive_write_disk(3)
HISTORY¶
The
libarchive
library first appeared in
FreeBSD 5.3.
AUTHORS¶
The
libarchive
library was written by
Tim Kientzle
⟨kientzle@acm.org⟩.