.TH "fi_verbs" "7" "2017\-07\-24" "Libfabric Programmer\[aq]s Manual" "\@VERSION\@"
.SH NAME
.PP
fi_verbs \- The Verbs Fabric Provider
.SH OVERVIEW
.PP
The verbs provider enables applications using OFI to be run over any
verbs hardware (Infiniband, iWarp, etc).
It uses the Linux Verbs API for network transport and provides a
translation of OFI calls to appropriate verbs API calls.
It uses librdmacm for communication management and libibverbs for other
control and data transfer operations.
.SH SUPPORTED FEATURES
.PP
The verbs provider supports a subset of OFI features.
.SS Endpoint types
.PP
FI_EP_MSG, FI_EP_RDM (Experimental support FI_MSG, FI_TAGGED, FI_RMA
interfaces).
.SS Endpoint capabilities and features
.SS MSG endpoints
.PP
FI_MSG, FI_RMA, FI_ATOMIC and shared receive contexts.
.SS RDM endpoints
.PP
FI_MSG, FI_TAGGED, FI_RMA
.SS Modes
.PP
Verbs provider requires applications to support the following modes:
.IP \[bu] 2
FI_LOCAL_MR for all applications.
.IP \[bu] 2
FI_RX_CQ_DATA for applications that want to use RMA.
Applications must take responsibility of posting receives for any
incoming CQ data.
.IP \[bu] 2
FI_CONTEXT for applications making uses of the experimental FI_EP_RDM
capability.
.SS Addressing Formats
.PP
Supported addressing formats include FI_SOCKADDR, FI_SOCKADDR_IN,
FI_SOCKADDR_IN6, FI_SOCKADDR_IB
.SS Progress
.PP
Verbs provider supports FI_PROGRESS_AUTO: Asynchronous operations make
forward progress automatically.
.SS Operation flags
.PP
Verbs provider supports FI_INJECT, FI_COMPLETION, FI_REMOTE_CQ_DATA,
FI_TRANSMIT_COMPLETE.
.SS Msg Ordering
.PP
Verbs provider support the following message ordering:
.IP \[bu] 2
Read after Read
.IP \[bu] 2
Read after Write
.IP \[bu] 2
Read after Send
.IP \[bu] 2
Write after Write
.IP \[bu] 2
Write after Send
.IP \[bu] 2
Send after Write
.IP \[bu] 2
Send after Send
.PP
and the following completion ordering:
.IP \[bu] 2
TX contexts: FI_ORDER_STRICT
.IP \[bu] 2
RX contexts: FI_ORDER_DATA
.SS Fork
.PP
Verbs provider supports the fork system call by default.
See the limitations section for restrictions.
It can be turned off by setting the FI_FORK_UNSAFE environment variable
to "yes".
This can improve the performance of memory registrations but it also
makes the use of fork unsafe.
.SH LIMITATIONS
.SS Memory Regions
.PP
Only FI_MR_BASIC mode is supported.
Adding regions via s/g list is supported only up to s/g list size of 1.
No support for binding memory regions to a counter.
.SS Wait objects
.PP
Only FI_WAIT_FD wait object is supported only for FI_EP_MSG endpoint
type.
Wait sets are not supported.
.SS Resource Management
.PP
Application has to make sure CQs are not overrun as this cannot be
detected by the provider.
.SS Unsupported Features
.PP
The following features are not supported in verbs provider:
.SS Unsupported Endpoint types
.PP
FI_EP_DGRAM
.SS Unsupported Capabilities
.PP
FI_NAMED_RX_CTX, FI_DIRECTED_RECV, FI_TRIGGER, FI_MULTI_RECV,
FI_RMA_EVENT
.SS Other unsupported features
.PP
Scalable endpoints, FABRIC_DIRECT
.SS Unsupported features specific to MSG endpoints
.IP \[bu] 2
Counters, FI_SOURCE, FI_TAGGED, FI_PEEK, FI_CLAIM, fi_cancel,
fi_ep_alias, shared TX context, cq_readfrom operations.
.IP \[bu] 2
Completion flags are not reported if a request posted to an endpoint
completes in error.
.SS Unsupported features specific to RDM endpoints
.PP
The RDM support for verbs have the following limitations:
.IP \[bu] 2
Supports iovs of only size 1.
.IP \[bu] 2
Wait objects are not supported.
.IP \[bu] 2
Not thread safe.
.SS Fork
.PP
The support for fork in the provider has the following limitations:
.IP \[bu] 2
Fabric resources like endpoint, CQ, EQ, etc.
should not be used in the forked process.
.IP \[bu] 2
The memory registered using fi_mr_reg has to be page aligned since
ibv_reg_mr marks the entire page that a memory region belongs to as not
to be re\-mapped when the process is forked (MADV_DONTFORK).
.SH RUNTIME PARAMETERS
.PP
The verbs provider checks for the following environment variables.
.SS Variables specific to MSG endpoints
.PP
\f[I]FI_VERBS_TX_SIZE\f[] : Default maximum tx context size (default:
384)
.PP
\f[I]FI_VERBS_RX_SIZE\f[] : Default maximum rx context size (default:
384)
.PP
\f[I]FI_VERBS_TX_IOV_LIMIT\f[] : Default maximum tx iov_limit (default:
4)
.PP
\f[I]FI_VERBS_RX_IOV_LIMIT\f[] : Default maximum rx iov_limit (default:
4)
.SS Variables specific to RDM endpoints
.PP
\f[I]FI_VERBS_IFACE\f[] : The prefix or the full name of the network
interface associated with the IB device (default: ib)
.PP
\f[I]FI_VERBS_RDM_BUFFER_NUM\f[] : The number of pre\-registered buffers
for buffered operations between the endpoints, must be a power of 2
(default: 8).
.PP
\f[I]FI_VERBS_RDM_BUFFER_SIZE\f[] : The maximum size of a buffered
operation (bytes) (default: platform specific).
.PP
\f[I]FI_VERBS_RDM_RNDV_SEG_SIZE\f[] : The segment size for zero copy
protocols (bytes)(default: 1073741824).
.PP
\f[I]FI_VERBS_RDM_CQREAD_BUNCH_SIZE\f[] : The number of entries to be
read from the verbs completion queue at a time (default: 8).
.PP
\f[I]FI_VERBS_RDM_THREAD_TIMEOUT\f[] : The wake up timeout of the helper
thread (usec) (default: 100).
.PP
\f[I]FI_VERBS_RDM_EAGER_SEND_OPCODE\f[] : The operation code that will
be used for eager messaging.
Only IBV_WR_SEND and IBV_WR_RDMA_WRITE_WITH_IMM are supported.
The last one is not applicable for iWarp.
(default: IBV_WR_SEND)
.SS Environment variables notes
.PP
The fi_info utility would give the up\-to\-date information on
environment variables: fi_info \-p verbs \-e
.SH Troubleshooting / Known issues
.PP
When running an app over verbs provider with valgrind, there may be
reports of memory leak in functions from dependent libraries (e.g.
libibverbs, librdmacm).
These leaks are safe to ignore.
.SH SEE ALSO
.PP
\f[C]fabric\f[](7), \f[C]fi_provider\f[](7),
.SH AUTHORS
OpenFabrics.