.\" Automatically generated by Pandoc 2.9.2.1
.\"
.TH "" "" "2023-05-26" "PMDK - " "PMDK Programmer's Manual"
.hy
.\" SPDX-License-Identifier: BSD-3-Clause
.\" Copyright 2019-2023, Intel Corporation
.SH NAME
.PP
\f[B]libpmem2\f[R] - persistent memory support library
.RS
.PP
NOTE: Support for Windows and FreeBSD deprecated since PMDK 1.13.0
release and will be removed in the PMDK 1.14.0 release.
.RE
.SH SYNOPSIS
.IP
.nf
\f[C]
#include <libpmem2.h>
cc ... -lpmem2
\f[R]
.fi
.SH DESCRIPTION
.PP
\f[B]libpmem2\f[R] provides low-level \f[I]persistent memory\f[R] (pmem)
support for applications using direct access storage (DAX), which is
storage that supports load/store access without paging blocks from a
block storage device.
Some types of \f[I]non-volatile memory DIMMs\f[R] (NVDIMMs) provide this
type of byte addressable access to storage.
A \f[I]persistent memory aware file system\f[R] is typically used to
expose the direct access to applications.
Memory mapping a file from this type of file system results in the
load/store, non-paged access to pmem.
.PP
This library is for applications that use persistent memory directly,
without the help of any library-supplied transactions or memory
allocation.
Higher-level libraries that \f[I]currently\f[R] build on
\f[B]libpmem\f[R] (previous variation of libpmem2) are available and are
recommended for most applications, see:
.IP \[bu] 2
\f[B]libpmemobj\f[R](7), a general use persistent memory API, providing
memory allocation and transactional operations on variable-sized
objects.
.IP \[bu] 2
\f[B]libpmemblk\f[R](7), providing pmem-resident arrays of fixed-sized
blocks with atomic updates.
.IP \[bu] 2
\f[B]libpmemlog\f[R](7), providing a pmem-resident log file.
.PP
The \f[B]libpmem2\f[R] library provides a comprehensive set of functions
for robust use of Persistent Memory.
It relies on three core concepts: \f[I]struct pmem2_src source\f[R],
\f[I]struct pmem2_config config\f[R] and \f[I]struct pmem2_map map\f[R]:
.IP \[bu] 2
\f[I]source\f[R] - an object describing the data source for mapping.
The data source can be a file descriptor, a file handle, or an anonymous
mapping.
APIs dedicated for creating \f[I]source\f[R] are:
\f[B]pmem2_source_from_fd\f[R](3),
\f[B]pmem2_source_from_handle\f[R](3),
\f[B]pmem2_source_from_anon\f[R](3).
.IP \[bu] 2
\f[I]config\f[R] - an object containing parameters that are used to
create a mapping from a \f[I]source\f[R].
The configuration structure must always be provided to create a mapping,
but the only required parameter to set in the \f[I]config\f[R] is
\f[I]granularity\f[R].
The granularity should by set using dedicated \f[B]libpmem2\f[R]
function \f[B]pmem2_config_set_required_store_granularity\f[R](3) which
defines a maximum permitted granularity requested by the user.
For more information about the granularity concept read
\f[B]GRANULARITY\f[R] section below.
.PP
In addition to the granularity setting, libpmem2 provides multiple
optional functions to configure target mapping, e.g.,
\f[B]pmem2_config_set_length\f[R](3) to set length which will be used
for mapping, or \f[B]pmem2_config_set_offset\f[R](3) which will be used
to map the contents from the specified location of the source,
\f[B]pmem2_config_set_sharing\f[R](3) which defines the behavior and
visibility of writes to the mapping\[cq]s pages.
.IP \[bu] 2
\f[I]map\f[R] - an object created by \f[B]pmem2_map_new\f[R](3) using
\f[I]source\f[R] and \f[I]config\f[R] as an input parameters.
The map structure can be then used to directly operate on the created
mapping through the use of its associated set of functions:
\f[B]pmem2_map_get_address\f[R](3), \f[B]pmem2_map_get_size\f[R](3),
\f[B]pmem2_map_get_store_granularity\f[R](3) - for getting address, size
and effective mapping granularity.
.PP
In addition to the basic functionality of managing the virtual address
mapping, \f[B]libpmem2\f[R] also provides optimized functions for
modifying the mapped data.
This includes data flushing as well as memory copying.
.PP
To get proper function for data flushing use:
\f[B]pmem2_get_flush_fn\f[R](3), \f[B]pmem2_get_persist_fn\f[R](3) or
\f[B]pmem2_get_drain_fn\f[R](3).
To get proper function for copying to persistent memory, use
\f[I]map\f[R] getters: \f[B]pmem2_get_memcpy_fn\f[R](3),
\f[B]pmem2_get_memset_fn\f[R](3), \f[B]pmem2_get_memmove_fn\f[R](3).
.PP
The \f[B]libpmem2\f[R] API also provides support for the badblock and
unsafe shutdown state handling.
.PP
To read or clear badblocks, the following functions are provided:
\f[B]pmem2_badblock_context_new\f[R](3),
\f[B]pmem2_badblock_context_delete\f[R](3),
\f[B]pmem2_badblock_next\f[R](3) and \f[B]pmem2_badblock_clear\f[R](3).
.PP
To handle unsafe shutdown in the application, the following functions
are provided: \f[B]pmem2_source_device_id\f[R](3),
\f[B]pmem2_source_device_usc\f[R](3).
More detailed information about unsafe shutdown detection and unsafe
shutdown count can be found in the \f[B]libpmem2_unsafe_shutdown\f[R](7)
man page.
.SH GRANULARITY
.PP
The \f[B]libpmem2\f[R] library introduces the concept of granularity
through which you may easily distinguish between different levels of
storage performance capabilities available to the application as related
to \f[I]power-fail protected domain\f[R].
The way data reaches this protected domain differs based on the platform
and storage device capabilities.
.PP
Traditional block storage devices (SSD, HDD) must use system API calls
such as \f[C]msync()\f[R], \f[C]fsync()\f[R] on Linux, or
\f[C]FlushFileBuffers()\f[R],\f[C]FlushViewOfFile()\f[R] on Windows to
write data reliably.
Invoking these functions flushes the data to the medium with page
granularity.
In the \f[B]libpmem2\f[R] library, this type of flushing behavior is
called \f[B]PMEM2_GRANULARITY_PAGE\f[R].
.PP
In systems with persistent memory support, a \f[I]power-fail protected
domain\f[R] may cover different sets of resources: either the memory
controller or the memory controller and CPU caches.
For this reason, \f[B]libpmem2\f[R] distinguishes two types of
granularity for persistent memory:
\f[B]PMEM2_GRANULARITY_CACHE_LINE\f[R] and
\f[B]PMEM2_GRANULARITY_BYTE\f[R].
.PP
If the \f[I]power-fail protected domain\f[R] covers only the memory
controller, the CPU appropriate cache lines must be flushed for the data
to be considered persistent.
This granularity type is called \f[B]PMEM2_GRANULARITY_CACHE_LINE\f[R].
Depending on the architecture, there are different types of machine
instructions for flushing \f[I]cache lines\f[R] (e.g., \f[I]CLWB\f[R],
\f[I]CLFLUSHOPT\f[R], \f[I]CLFLUSH\f[R] for Intel x86_64 architecture).
Usually, to ensure the ordering of stores, such instructions must be
followed by a barrier (e.g., \f[I]SFENCE\f[R]).
.PP
The third type of granularity \f[B]PMEM2_GRANULARITY_BYTE\f[R] applies
to platforms where \f[I]power-fail protected domain\f[R] covers both the
memory controller and CPU caches.
In such cases, cache flush instructions are no longer needed, and the
platform itself guarantees the persistence of data.
But barriers might still be required for ordering.
.PP
The library declares these granularity level in
\f[I]pmem2_granularity\f[R] enum, which the application must set in
\f[I]pmem2_config\f[R] to the appropriate level for a mapping to
succeed.
The software should set this config parameter to a value that most
accurately represents the target hardware characteristics and the
storage patterns of the application.
For example, a database storage engine that operates on large logical
pages that reside either on SSDs or PMEM should set this value to
\f[B]PMEM2_GRANULARITY_PAGE\f[R].
The library will create mappings where the new map granularity is lower
or equal to the requested one.
For example, a mapping with \f[B]PMEM2_GRANULARITY_CACHE_LINE\f[R] can
be created for the required granularity
\f[B]PMEM2_GRANULARITY_PAGE\f[R], but not vice versa.
.SH CAVEATS
.PP
\f[B]libpmem2\f[R] relies on the library destructor being called from
the main thread.
For this reason, all functions that might trigger destruction
(e.g.\ dlclose(3)) should be called in the main thread.
Otherwise some of the resources associated with that thread might not be
cleaned up properly.
.SH ENVIRONMENT
.PP
\f[B]libpmem2\f[R] can change its default behavior based on the
following environment variables.
These are primarily intended for testing and are generally not required.
.IP \[bu] 2
\f[B]PMEM2_FORCE_GRANULARITY\f[R]=\f[I]val\f[R]
.PP
Setting this environment variable to \f[I]val\f[R] forces
\f[B]libpmem2\f[R] to use persist method specific for forced granularity
and skip granularity autodetecting mechanism.
The concept of the granularity is described in \f[I]GRANULARITY\f[R]
section above.
This variable is intended for use during library testing.
.PP
The \f[I]val\f[R] argument accepts following text values:
.IP \[bu] 2
\f[B]BYTE\f[R] - force byte granularity.
.IP \[bu] 2
\f[B]CACHE_LINE\f[R] - force cache line granularity.
.IP \[bu] 2
\f[B]PAGE\f[R] - force page granularity.
.PP
Granularity values listed above are case-insensitive.
.RS
.PP
NOTE: The value of \f[B]PMEM2_FORCE_GRANULARITY\f[R] is not queried (and
cached) at library initialization time, but read during each
\f[B]pmem2_map_new\f[R](3) call.
.RE
.PP
This means that \f[B]PMEM2_FORCE_GRANULARITY\f[R] may still be set or
modified by the program until the first attempt to map a file.
.IP \[bu] 2
\f[B]PMEM_NO_CLWB\f[R]=1
.PP
Setting this environment variable to 1 forces \f[B]libpmem2\f[R] to
never issue the \f[B]CLWB\f[R] instruction on Intel hardware, falling
back to other cache flush instructions on that hardware instead
(\f[B]CLFLUSHOPT\f[R] or \f[B]CLFLUSH\f[R]).
Without this setting, \f[B]libpmem2\f[R] will always use the
\f[B]CLWB\f[R] instruction for flushing processor caches on platforms
that support this instruction.
This variable is intended for use during library testing, but may be
required for some rare cases when using \f[B]CLWB\f[R] has a negative
impact on performance.
.IP \[bu] 2
\f[B]PMEM_NO_CLFLUSHOPT\f[R]=1
.PP
Setting this environment variable to 1 forces \f[B]libpmem2\f[R] to
never issue the \f[B]CLFLUSHOPT\f[R] instruction on Intel hardware,
falling back to the \f[B]CLFLUSH\f[R] instructions instead.
Without this environment variable, \f[B]libpmem2\f[R] will always use
the \f[B]CLFLUSHOPT\f[R] instruction for flushing processor caches on
platforms that support the instruction, but where \f[B]CLWB\f[R] is not
available.
This variable is intended for use during library testing.
.IP \[bu] 2
\f[B]PMEM_NO_MOVNT\f[R]=1
.PP
Setting this environment variable to 1 forces \f[B]libpmem2\f[R] to
never use the \f[I]non-temporal\f[R] move instructions on Intel
hardware.
Without this environment variable, \f[B]libpmem2\f[R] will use the
non-temporal instructions for copying larger ranges to persistent memory
on platforms that support these instructions.
This variable is intended for use during library testing.
.IP \[bu] 2
\f[B]PMEM_MOVNT_THRESHOLD\f[R]=\f[I]val\f[R]
.PP
This environment variable allows overriding the minimum length of the
\f[I]pmem2_memmove_fn\f[R] operations, for which \f[B]libpmem2\f[R] uses
\f[I]non-temporal\f[R] move instructions.
Setting this environment variable to 0 forces \f[B]libpmem2\f[R] to
always use the \f[I]non-temporal\f[R] move instructions if available.
It has no effect if \f[B]PMEM_NO_MOVNT\f[R] is set to 1.
This variable is intended for use during library testing.
.SH DEBUGGING
.PP
Two versions of \f[B]libpmem2\f[R] are typically available on a
development system.
The normal version, accessed when a program is linked using the
\f[B]-lpmem2\f[R] option, is optimized for performance.
That version skips checks that impact performance and never logs any
trace information or performs any run-time assertions.
.PP
A second version of \f[B]libpmem2\f[R], accessed when a program uses the
libraries under \f[B]/usr/lib/pmdk_debug\f[R], contains run-time
assertions and trace points.
The typical way to access the debug version is to set the environment
variable \f[B]LD_LIBRARY_PATH\f[R] to \f[B]/usr/lib/pmdk_debug\f[R] or
\f[B]/usr/lib64/pmdk_debug\f[R], as appropriate.
Debugging output is controlled using the following environment
variables.
These variables have no effect on the non-debug version of the library.
.RS
.PP
NOTE: On Debian/Ubuntu systems, this extra debug version of the library
is shipped in the respective \f[B]-debug\f[R] Debian package and placed
in the \f[B]/usr/lib/$ARCH/pmdk_dbg/\f[R] directory.
.RE
.IP \[bu] 2
\f[B]PMEM2_LOG_LEVEL\f[R]
.PP
The value of \f[B]PMEM2_LOG_LEVEL\f[R] enables trace points in the debug
version of the library, as follows:
.IP \[bu] 2
\f[B]0\f[R] - This is the default level when \f[B]PMEM2_LOG_LEVEL\f[R]
is not set.
No log messages are emitted at this level.
.IP \[bu] 2
\f[B]1\f[R] - Additional details on any errors detected are logged, in
addition to returning the \f[I]errno\f[R]-based errors as usual.
The same information may be retrieved using \f[B]pmem2_errormsg\f[R]().
.IP \[bu] 2
\f[B]2\f[R] - A trace of basic operations is logged.
.IP \[bu] 2
\f[B]3\f[R] - Enables a very verbose amount of function call tracing in
the library.
.IP \[bu] 2
\f[B]4\f[R] - Enables voluminous and fairly obscure tracing information
that is likely only useful to the \f[B]libpmem2\f[R] developers.
.PP
Unless \f[B]PMEM2_LOG_FILE\f[R] is set, debugging output is written to
\f[I]stderr\f[R].
.IP \[bu] 2
\f[B]PMEM2_LOG_FILE\f[R]
.PP
Specifies the name of a file where all logging information should be
written.
If the last character in the name is \[lq]-\[rq], the \f[I]PID\f[R] of
the current process will be appended to the file name when the log file
is created.
If \f[B]PMEM2_LOG_FILE\f[R] is not set, output is written to
\f[I]stderr\f[R].
.SH EXAMPLE
.PP
The following example uses \f[B]libpmem2\f[R] to flush changes made to
raw, memory-mapped persistent memory.
.RS
.PP
WARNING: There is nothing transactional about the \f[I]persist\f[R] from
\f[B]pmem2_get_persist_fn\f[R](3) call in this example.
Interrupting the program may result in a partial write to pmem.
Use a transactional library such as \f[B]libpmemobj\f[R](7) to avoid
torn updates.
.RE
.PP
The basic example can be found in the repository under path
src/examples/libpmem2/basic/basic.c (https://github.com/pmem/pmdk/blob/master/src/examples/libpmem2/basic/basic.c).
It is described in detail here (https://pmem.io/pmdk/libpmem2/).
.SH ACKNOWLEDGEMENTS
.PP
\f[B]libpmem2\f[R] builds on the persistent memory programming model
recommended by the SNIA NVM Programming Technical Work Group:
<https://snia.org/nvmp>
.SH SEE ALSO
.PP
\f[B]FlushFileBuffers\f[R](), \f[B]fsync\f[R](2), \f[B]msync\f[R](2),
\f[B]pmem2_config_set_length\f[R](3),
\f[B]pmem2_config_set_offset\f[R](3),
\f[B]pmem2_config_set_required_store_granularity\f[R](3),
\f[B]pmem2_config_set_sharing\f[R](3),\f[B]pmem2_get_drain_fn\f[R](3),
\f[B]pmem2_get_flush_fn\f[R](3), \f[B]pmem2_get_memcpy_fn\f[R](3),
\f[B]pmem2_get_memmove_fn\f[R](3), \f[B]pmem2_get_memset_fn\f[R](3),
\f[B]pmem2_get_persist_fn\f[R](3),\f[B]pmem2_map_get_store_granularity\f[R](3),
\f[B]pmem2_map_new\f[R](3), \f[B]pmem2_source_from_anon\f[R](3),
\f[B]pmem2_source_from_fd\f[R](3),
\f[B]pmem2_source_from_handle\f[R](3),
\f[B]libpmem2_unsafe_shutdown\f[R](7), \f[B]libpmemblk\f[R](7),
\f[B]libpmemlog\f[R](7), \f[B]libpmemobj\f[R](7) and
\f[B]<https://pmem.io>\f[R]