'\" t
.\"     Title: daxctl-reconfigure-device
.\"    Author: [see the "AUTHOR(S)" section]
.\" Generator: Asciidoctor 2.0.12
.\"      Date: 2022-04-08
.\"    Manual: daxctl Manual
.\"    Source: daxctl
.\"  Language: English
.\"
.TH "DAXCTL\-RECONFIGURE\-DEVICE" "1" "2022-04-08" "daxctl" "daxctl Manual"
.ie \n(.g .ds Aq \(aq
.el       .ds Aq '
.ss \n[.ss] 0
.nh
.ad l
.de URL
\fI\\$2\fP <\\$1>\\$3
..
.als MTO URL
.if \n[.g] \{\
.  mso www.tmac
.  am URL
.    ad l
.  .
.  am MTO
.    ad l
.  .
.  LINKSTYLE blue R < >
.\}
.SH "NAME"
daxctl\-reconfigure\-device \- Reconfigure a dax device into a different mode
.SH "SYNOPSIS"
.sp
.sp
.nf
\fIdaxctl reconfigure\-device\fP <dax0.0> [<dax1.0>...<daxY.Z>] [<options>]
.fi
.br
.SH "DESCRIPTION"
.sp
Reconfigure the operational mode of a dax device. This can be used to convert
a regular \fIdevdax\fP mode device to the \fIsystem\-ram\fP mode which arranges for the
dax range to be hot\-plugged into the system as regular memory.
.if n .sp
.RS 4
.it 1 an-trap
.nr an-no-space-flag 1
.nr an-break-flag 1
.br
.ps +1
.B Note
.ps -1
.br
.sp
This is a destructive operation. Any data on the dax device \fBwill\fP be
lost.
.sp .5v
.RE
.if n .sp
.RS 4
.it 1 an-trap
.nr an-no-space-flag 1
.nr an-break-flag 1
.br
.ps +1
.B Note
.ps -1
.br
.sp
Device reconfiguration depends on the dax\-bus device model. See
daxctl\-migrate\-device\-model(1) for more information. If dax\-class is
in use (via the dax_pmem_compat driver), the reconfiguration will fail with an
error such as the following:
.sp .5v
.RE
.sp
.if n .RS 4
.nf
.fam C
# daxctl reconfigure\-device \-\-mode=system\-ram \-\-region=0 all
libdaxctl: daxctl_dev_disable: dax3.0: error: device model is dax\-class
dax3.0: disable failed: Operation not supported
error reconfiguring devices: Operation not supported
reconfigured 0 devices
.fam
.fi
.if n .RE
.sp
\fIdaxctl\-reconfigure\-device\fP nominally expects that it will online new memory
blocks as \fImovable\fP, so that kernel data doesn\(cqt make it into this memory.
However, there are other potential agents that may be configured to
automatically online new hot\-plugged memory as it appears. Most notably,
these are the \fI/sys/devices/system/memory/auto_online_blocks\fP configuration,
or system udev rules. If such an agent races to online memory sections, daxctl
checks if the blocks were onlined as \fImovable\fP memory. If this was not the
case, and the memory blocks are found to be in a different zone, then a
warning is displayed. If it is desired that a different agent control the
onlining of memory blocks, and the associated memory zone, then it is
recommended to use the \-\-no\-online option described below. This will abridge
the device reconfiguration operation to just hotplugging the memory, and
refrain from then onlining it.
.sp
In case daxctl detects that there is a kernel policy to auto\-online blocks
(via /sys/devices/system/memory/auto_online_blocks), then reconfiguring to
system\-ram will result in a failure. This can be overridden with \fI\-\-force\fP.
.SH "THEORY OF OPERATION"
.sp
The kernel device\-dax subsystem surfaces character devices
that provide DAX\-access (direct mappings sans page\-cache buffering) to a
given memory region. The devices are named /dev/daxX.Y where X is a
region\-id and Y is an instance\-id within that region. There are 2
mechanisms that trigger device\-dax instances to appear:
.sp
.RS 4
.ie n \{\
\h'-04' 1.\h'+01'\c
.\}
.el \{\
.  sp -1
.  IP " 1." 4.2
.\}
Persistent Memory (PMEM) namespace configured in "devdax" mode. See
"ndctl create\-namspace \-\-help" and
.URL "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/dax/Kconfig" "CONFIG_DEV_DAX_PMEM" "."
In this case the device\-dax instance is statically sized to its host
memory region which is bounded to the physical address range of the host
namespace.
.RE
.sp
.RS 4
.ie n \{\
\h'-04' 2.\h'+01'\c
.\}
.el \{\
.  sp -1
.  IP " 2." 4.2
.\}
Soft Reserved memory enumerated by platform firmware. On EFI systems
this is communicated via the so called EFI_MEMORY_SP "Special Purpose"
attribute. See
.URL "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/dax/Kconfig" "CONFIG_DEV_DAX_HMEM" "."
In this case the device\-dax instance(s) associated with the given memory
region can be resized and divided into multiple devices.
.RE
.sp
In the Soft Reservation case the expectation for EFI + ACPI based
platforms is that in addition to the EFI_MEMORY_SP attribute the
firmware also creates distinct ACPI proximity domains for any address
range that has different performance characteristics than default
"System RAM". So, the SRAT will define the proximity domain, the SLIT
communicates relative distance to other proximity domains, and the HMAT
is populated with nominal read/write latency and read/write bandwidth
data. That HMAT data is emitted to the kernel log on bootup, and also
exported to sysfs. See
.URL "https://www.kernel.org/doc/html/latest/admin\-guide/mm/numaperf.html" "NUMAPERF" ","
for the runtime representation of CPU to Memory node performance
details.
.sp
Outside of the NUMA performance details linked above the other method to
detect the presence of "Soft Reserved" memory is to dump /proc/iomem and
look for "Soft Reserved" ranges. If the kernel was not built with
CONFIG_EFI_SOFTRESERVE, predates the introduction of
CONFIG_EFI_SOFTRESERVE (v5.5), or was booted with the efi=nosoftreserve
command line then device\-dax will not attach and the expectation is that
the memory shows up as a memory\-only NUMA node. Otherwise the memory
shows up as a device\-dax instance and DAXCTL(1) can be used to
optionally partition it and assign the memory back to the kernel as
"System RAM", or the device can be mapped directly as the back end of a
userspace memory allocator like \c
.URL "https://pmem.io/vmem/libvmem/" "LIBVMEM" "."
.SH "EXAMPLES"
.sp
.RS 4
.ie n \{\
\h'-04'\(bu\h'+03'\c
.\}
.el \{\
.  sp -1
.  IP \(bu 2.3
.\}
Reconfigure dax0.0 to system\-ram mode, don\(cqt online the memory
.RE
.sp
.if n .RS 4
.nf
.fam C
# daxctl reconfigure\-device \-\-mode=system\-ram \-\-no\-online dax0.0
[
  {
    "chardev":"dax0.0",
    "size":16777216000,
    "target_node":2,
    "mode":"system\-ram"
  }
]
.fam
.fi
.if n .RE
.sp
.RS 4
.ie n \{\
\h'-04'\(bu\h'+03'\c
.\}
.el \{\
.  sp -1
.  IP \(bu 2.3
.\}
Reconfigure dax0.0 to devdax mode, attempt to offline the memory
.RE
.sp
.if n .RS 4
.nf
.fam C
# daxctl reconfigure\-device \-\-human \-\-mode=devdax \-\-force dax0.0
{
  "chardev":"dax0.0",
  "size":"15.63 GiB (16.78 GB)",
  "target_node":2,
  "mode":"devdax"
}
.fam
.fi
.if n .RE
.sp
.RS 4
.ie n \{\
\h'-04'\(bu\h'+03'\c
.\}
.el \{\
.  sp -1
.  IP \(bu 2.3
.\}
Reconfigure all dax devices on region0 to system\-ram mode
.RE
.sp
.if n .RS 4
.nf
.fam C
# daxctl reconfigure\-device \-\-mode=system\-ram \-\-region=0 all
[
  {
    "chardev":"dax0.0",
    "size":16777216000,
    "target_node":2,
    "mode":"system\-ram"
  },
  {
    "chardev":"dax0.1",
    "size":16777216000,
    "target_node":3,
    "mode":"system\-ram"
  }
]
.fam
.fi
.if n .RE
.sp
.RS 4
.ie n \{\
\h'-04'\(bu\h'+03'\c
.\}
.el \{\
.  sp -1
.  IP \(bu 2.3
.\}
Run a process called \fIsome\-service\fP using numactl to restrict its cpu
nodes to \fI0\fP and \fI1\fP, and  memory allocations to node 2 (determined using
daxctl_dev_get_target_node() or \fIdaxctl list\fP)
.RE
.sp
.if n .RS 4
.nf
.fam C
# daxctl reconfigure\-device \-\-mode=system\-ram dax0.0
[
  {
    "chardev":"dax0.0",
    "size":16777216000,
    "target_node":2,
    "mode":"system\-ram"
  }
]

# numactl \-\-cpunodebind=0\-1 \-\-membind=2 \-\- some\-service \-\-opt1 \-\-opt2
.fam
.fi
.if n .RE
.sp
.RS 4
.ie n \{\
\h'-04'\(bu\h'+03'\c
.\}
.el \{\
.  sp -1
.  IP \(bu 2.3
.\}
Change the size of a dax device
.RE
.sp
.if n .RS 4
.nf
.fam C
# daxctl reconfigure\-device dax0.1 \-s 16G
reconfigured 1 device
# daxctl reconfigure\-device dax0.1 \-s 0
reconfigured 1 device
.fam
.fi
.if n .RE
.SH "OPTIONS"
.sp
\-r, \-\-region=
.RS 4
Restrict the operation to devices belonging to the specified region(s).
A device\-dax region is a contiguous range of memory that hosts one or
more /dev/daxX.Y devices, where X is the region id and Y is the device
instance id.
.RE
.sp
\-s, \-\-size=
.RS 4
For regions that support dax device creation, change the device size
in bytes. This option supports the suffixes "k" or "K" for KiB, "m" or
"M" for MiB, "g" or "G" for GiB and "t" or "T" for TiB.
.sp
.if n .RS 4
.nf
.fam C
The size must be a multiple of the region alignment.
.fam
.fi
.if n .RE
.sp
.if n .RS 4
.nf
.fam C
This option is mutually exclusive with \-m or \-\-mode.
.fam
.fi
.if n .RE
.RE
.sp
\-a, \-\-align
.RS 4
Applications that want to establish dax memory mappings with
page table entries greater than system base page size (4K on
x86) need a device that is sufficiently aligned. This defaults
to 2M. Note that "devdax" mode enforces all mappings to be
aligned to this value, i.e. it fails unaligned mapping attempts.
.sp
.if n .RS 4
.nf
.fam C
This option is mutually exclusive with \-m or \-\-mode.
.fam
.fi
.if n .RE
.RE
.sp
\-m, \-\-mode=
.RS 4
Specify the mode to which the dax device(s) should be reconfigured.
.sp
.RS 4
.ie n \{\
\h'-04'\(bu\h'+03'\c
.\}
.el \{\
.  sp -1
.  IP \(bu 2.3
.\}
"system\-ram": hotplug the device into system memory.
.RE
.sp
.RS 4
.ie n \{\
\h'-04'\(bu\h'+03'\c
.\}
.el \{\
.  sp -1
.  IP \(bu 2.3
.\}
"devdax": switch to the normal "device dax" mode. This requires the
kernel to support hot\-unplugging \fIkmem\fP based memory. If this is not
available, a reboot is the only way to switch back to \fIdevdax\fP mode.
.RE
.RE
.sp
\-N, \-\-no\-online
.RS 4
By default, memory sections provided by system\-ram devices will be
brought online automatically and immediately with the \fIonline_movable\fP
policy. Use this option to disable the automatic onlining behavior.
.RE
.sp
\-C, \-\-check\-config
.RS 4
Get reconfiguration parameters from the global daxctl config file.
This is typically used when daxctl\-reconfigure\-device is called from
a systemd\-udevd device unit file. The reconfiguration proceeds only
if the match parameters in a \fIreconfigure\-device\fP section of the
config match the dax device specified on the command line. See the
\fIPERSISTENT RECONFIGURATION\fP section for more details.
.RE
.sp
\-\-no\-movable
.RS 4
\fI\-\-movable\fP is the default. This can be overridden to online new
memory such that it is not \fImovable\fP. This allows any allocation
to potentially be served from this memory. This may preclude subsequent
removal. With the \fI\-\-movable\fP behavior (which is default), kernel
allocations will not consider this memory, and it will be reserved
for application use.
.RE
.sp
\-f, \-\-force
.RS 4
.sp
.RS 4
.ie n \{\
\h'-04'\(bu\h'+03'\c
.\}
.el \{\
.  sp -1
.  IP \(bu 2.3
.\}
When converting from "system\-ram" mode to "devdax", it is expected
that all the memory sections are first made offline. By default,
daxctl won\(cqt touch online memory. However with this option, attempt
to offline the memory on the NUMA node associated with the dax device
before converting it back to "devdax" mode.
.RE
.sp
.RS 4
.ie n \{\
\h'-04'\(bu\h'+03'\c
.\}
.el \{\
.  sp -1
.  IP \(bu 2.3
.\}
Additionally, if a kernel policy to auto\-online blocks is detected,
reconfiguration to system\-ram fails. With this option, the failure can
be overridden to allow reconfiguration regardless of kernel policy.
Doing this may result in a successful reconfiguration, but it may
not be possible to subsequently offline the memory without a reboot.
.RE
.RE
.sp
\-u, \-\-human
.RS 4
By default the command will output machine\-friendly raw\-integer
data. Instead, with this flag, numbers representing storage size
will be formatted as human readable strings with units, other
fields are converted to hexadecimal strings.
.RE
.sp
\-v, \-\-verbose
.RS 4
Emit more debug messages
.RE
.SH "PERSISTENT RECONFIGURATION"
.sp
The \fImode\fP of a daxctl device is not persistent across reboots by default. This
is because the device itself does not hold any metadata that hints at what mode
it was set to, or is intended to be used. The default mode for such a device
on boot is \fIdevdax\fP.
.sp
The administrator may set policy such that certain dax devices are always
reconfigured into a target configuration every boot. This is accomplished via a
daxctl config file.
.sp
The config file may have multiple sections influencing different aspects of
daxctl operation. The section of interest for persistent reconfiguration is
\fIreconfigure\-device\fP. The format of this is as follows:
.sp
.if n .RS 4
.nf
.fam C
[reconfigure\-device <unique_subsection_name>]
nvdimm.uuid = <NVDIMM namespace uuid>
mode = <desired reconfiguration mode> (default: system\-ram)
online = <true|false> (default: true)
movable = <true|false> (default: true)
.fam
.fi
.if n .RE
.sp
Here is an example of a config snippet for managing three devdax namespaces,
one is left in devdax mode, the second is changed to system\-ram mode with
default options (online, movable), and the third is set to system\-ram mode,
the memory is onlined, but not movable.
.sp
Note that the \fIsubsection name\fP can be arbitrary, and is only used to
identify a specific config section. It does not have to match the \fIdevice
name\fP (e.g. \fIdax0.0\fP etc).
.sp
.if n .RS 4
.nf
.fam C
[reconfigure\-device dax0]
nvdimm.uuid = ed93e918\-e165\-49d8\-921d\-383d7b9660c5
mode = devdax

[reconfigure\-device dax1]
nvdimm.uuid = f36d02ff\-1d9f\-4fb9\-a5b9\-8ceb10a00fe3
mode = system\-ram

[reconfigure\-device dax2]
nvdimm.uuid = f36d02ff\-1d9f\-4fb9\-a5b9\-8ceb10a00fe3
mode = system\-ram
online = true
movable = false
.fam
.fi
.if n .RE
.sp
The following example can be used to create a devdax mode namespace, and
simultaneously add the newly created namespace to the config file for
system\-ram conversion.
.sp
.if n .RS 4
.nf
.fam C
ndctl create\-namespace \-\-mode=devdax | \(rs
        jq \-r "\(rs"[reconfigure\-device $(uuidgen)]\(rs", \(rs"nvdimm.uuid = \(rs(.uuid)\(rs", \(rs"mode = system\-ram\(rs"" >> $config_path
.fam
.fi
.if n .RE
.sp
The default location for daxctl config files is under /etc/daxctl.conf.d/,
and any file with a \fI.conf\fP suffix at this location is considered. It is
acceptable to have multiple files containing ini\-style config sections,
but the {section, subsection} tuple must be unique across all config files
under /etc/daxctl.conf.d/.
.SH "COPYRIGHT"
.sp
Copyright \(co 2016 \- 2020, Intel Corporation. License GPLv2: GNU GPL
version 2 \c
.URL "http://gnu.org/licenses/gpl.html" "" "."
This is free software:
you are free to change and redistribute it.  There is NO WARRANTY, to
the extent permitted by law.
.SH "SEE ALSO"
.sp
daxctl\-list(1),daxctl\-migrate\-device\-model[1]