'\" p
.\" -*- nroff -*-
.TH "ovn-architecture" 7 "OVN Architecture" "Open vSwitch 2\[char46]11\[char46]2" "Open vSwitch Manual"
.fp 5 L CR              \\" Make fixed-width font available as \\fL.
.de TQ
.  br
.  ns
.  TP "\\$1"
..
.de ST
.  PP
.  RS -0.15in
.  I "\\$1"
.  RE
..
.PP
.SH "NAME"
.PP
.PP
ovn-architecture \- Open Virtual Network architecture
.SH "DESCRIPTION"
.PP
.PP
OVN, the Open Virtual Network, is a system to support virtual network abstraction\[char46] OVN complements the existing capabilities of OVS to add native support for virtual network abstractions, such as virtual L2 and L3 overlays and security groups\[char46] Services such as DHCP are also desirable features\[char46] Just like OVS, OVN\(cqs design goal is to have a production-quality implementation that can operate at significant scale\[char46]
.PP
.PP
An OVN deployment consists of several components:
.RS
.IP \(bu
A \fICloud Management System\fR (\fICMS\fR), which is OVN\(cqs ultimate client (via its users and administrators)\[char46] OVN integration requires installing a CMS-specific plugin and related software (see below)\[char46] OVN initially targets OpenStack as CMS\[char46]
.IP
We generally speak of ``the\(cq\(cq CMS, but one can imagine scenarios in which multiple CMSes manage different parts of an OVN deployment\[char46]
.IP \(bu
An OVN Database physical or virtual node (or, eventually, cluster) installed in a central location\[char46]
.IP \(bu
One or more (usually many) \fIhypervisors\fR\[char46] Hypervisors must run Open vSwitch and implement the interface described in \fBIntegrationGuide\[char46]rst\fR in the OVS source tree\[char46] Any hypervisor platform supported by Open vSwitch is acceptable\[char46]
.IP \(bu
Zero or more \fIgateways\fR\[char46] A gateway extends a tunnel-based logical network into a physical network by bidirectionally forwarding packets between tunnels and a physical Ethernet port\[char46] This allows non-virtualized machines to participate in logical networks\[char46] A gateway may be a physical host, a virtual machine, or an ASIC-based hardware switch that supports the \fBvtep\fR(5) schema\[char46]
.IP
Hypervisors and gateways are together called \fItransport node\fR or \fIchassis\fR\[char46]
.RE
.PP
.PP
The diagram below shows how the major components of OVN and related software interact\[char46] Starting at the top of the diagram, we have:
.RS
.IP \(bu
The Cloud Management System, as defined above\[char46]
.IP \(bu
The \fIOVN/CMS Plugin\fR is the component of the CMS that interfaces to OVN\[char46] In OpenStack, this is a Neutron plugin\[char46] The plugin\(cqs main purpose is to translate the CMS\(cqs notion of logical network configuration, stored in the CMS\(cqs configuration database in a CMS-specific format, into an intermediate representation understood by OVN\[char46]
.IP
This component is necessarily CMS-specific, so a new plugin needs to be developed for each CMS that is integrated with OVN\[char46] All of the components below this one in the diagram are CMS-independent\[char46]
.IP \(bu
The \fIOVN Northbound Database\fR receives the intermediate representation of logical network configuration passed down by the OVN/CMS Plugin\[char46] The database schema is meant to be ``impedance matched\(cq\(cq with the concepts used in a CMS, so that it directly supports notions of logical switches, routers, ACLs, and so on\[char46] See \fBovn\-nb\fR(5) for details\[char46]
.IP
The OVN Northbound Database has only two clients: the OVN/CMS Plugin above it and \fBovn\-northd\fR below it\[char46]
.IP \(bu
\fBovn\-northd\fR(8) connects to the OVN Northbound Database above it and the OVN Southbound Database below it\[char46] It translates the logical network configuration in terms of conventional network concepts, taken from the OVN Northbound Database, into logical datapath flows in the OVN Southbound Database below it\[char46]
.IP \(bu
The \fIOVN Southbound Database\fR is the center of the system\[char46] Its clients are \fBovn\-northd\fR(8) above it and \fBovn\-controller\fR(8) on every transport node below it\[char46]
.IP
The OVN Southbound Database contains three kinds of data: \fIPhysical
Network\fR (PN) tables that specify how to reach hypervisor and other nodes, \fILogical Network\fR (LN) tables that describe the logical network in terms of ``logical datapath flows,\(cq\(cq and \fIBinding\fR tables that link logical network components\(cq locations to the physical network\[char46] The hypervisors populate the PN and Port_Binding tables, whereas \fBovn\-northd\fR(8) populates the LN tables\[char46]
.IP
OVN Southbound Database performance must scale with the number of transport nodes\[char46] This will likely require some work on \fBovsdb\-server\fR(1) as we encounter bottlenecks\[char46] Clustering for availability may be needed\[char46]
.RE
.PP
.PP
The remaining components are replicated onto each hypervisor:
.RS
.IP \(bu
\fBovn\-controller\fR(8) is OVN\(cqs agent on each hypervisor and software gateway\[char46] Northbound, it connects to the OVN Southbound Database to learn about OVN configuration and status and to populate the PN table and the \fBChassis\fR column in \fBBinding\fR table with the hypervisor\(cqs status\[char46] Southbound, it connects to \fBovs\-vswitchd\fR(8) as an OpenFlow controller, for control over network traffic, and to the local \fBovsdb\-server\fR(1) to allow it to monitor and control Open vSwitch configuration\[char46]
.IP \(bu
\fBovs\-vswitchd\fR(8) and \fBovsdb\-server\fR(1) are conventional components of Open vSwitch\[char46]
.RE
.PP
.nf
\fL
.br
\fL                                  CMS
.br
\fL                                   |
.br
\fL                                   |
.br
\fL                       +\-\-\-\-\-\-\-\-\-\-\-|\-\-\-\-\-\-\-\-\-\-\-+
.br
\fL                       |           |           |
.br
\fL                       |     OVN/CMS Plugin    |
.br
\fL                       |           |           |
.br
\fL                       |           |           |
.br
\fL                       |   OVN Northbound DB   |
.br
\fL                       |           |           |
.br
\fL                       |           |           |
.br
\fL                       |       ovn\-northd      |
.br
\fL                       |           |           |
.br
\fL                       +\-\-\-\-\-\-\-\-\-\-\-|\-\-\-\-\-\-\-\-\-\-\-+
.br
\fL                                   |
.br
\fL                                   |
.br
\fL                         +\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+
.br
\fL                         | OVN Southbound DB |
.br
\fL                         +\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+
.br
\fL                                   |
.br
\fL                                   |
.br
\fL                +\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+
.br
\fL                |                  |                  |
.br
\fL  HV 1          |                  |    HV n          |
.br
\fL+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-|\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+  \[char46]  +\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-|\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+
.br
\fL|               |               |  \[char46]  |               |               |
.br
\fL|        ovn\-controller         |  \[char46]  |        ovn\-controller         |
.br
\fL|         |          |          |  \[char46]  |         |          |          |
.br
\fL|         |          |          |     |         |          |          |
.br
\fL|  ovs\-vswitchd   ovsdb\-server  |     |  ovs\-vswitchd   ovsdb\-server  |
.br
\fL|                               |     |                               |
.br
\fL+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+     +\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+
.br
\fL  \fR
.fi
.SS "Information Flow in OVN"
.PP
.PP
Configuration data in OVN flows from north to south\[char46] The CMS, through its OVN/CMS plugin, passes the logical network configuration to \fBovn\-northd\fR via the northbound database\[char46] In turn, \fBovn\-northd\fR compiles the configuration into a lower-level form and passes it to all of the chassis via the southbound database\[char46]
.PP
.PP
Status information in OVN flows from south to north\[char46] OVN currently provides only a few forms of status information\[char46] First, \fBovn\-northd\fR populates the \fBup\fR column in the northbound \fBLogical_Switch_Port\fR table: if a logical port\(cqs \fBchassis\fR column in the southbound \fBPort_Binding\fR table is nonempty, it sets \fBup\fR to \fBtrue\fR, otherwise to \fBfalse\fR\[char46] This allows the CMS to detect when a VM\(cqs networking has come up\[char46]
.PP
.PP
Second, OVN provides feedback to the CMS on the realization of its configuration, that is, whether the configuration provided by the CMS has taken effect\[char46] This feature requires the CMS to participate in a sequence number protocol, which works the following way:
.RS
.IP 1. .4in
When the CMS updates the configuration in the northbound database, as part of the same transaction, it increments the value of the \fBnb_cfg\fR column in the \fBNB_Global\fR table\[char46] (This is only necessary if the CMS wants to know when the configuration has been realized\[char46])
.IP 2. .4in
When \fBovn\-northd\fR updates the southbound database based on a given snapshot of the northbound database, it copies \fBnb_cfg\fR from northbound \fBNB_Global\fR into the southbound database \fBSB_Global\fR table, as part of the same transaction\[char46] (Thus, an observer monitoring both databases can determine when the southbound database is caught up with the northbound\[char46])
.IP 3. .4in
After \fBovn\-northd\fR receives confirmation from the southbound database server that its changes have committed, it updates \fBsb_cfg\fR in the northbound \fBNB_Global\fR table to the \fBnb_cfg\fR version that was pushed down\[char46] (Thus, the CMS or another observer can determine when the southbound database is caught up without a connection to the southbound database\[char46])
.IP 4. .4in
The \fBovn\-controller\fR process on each chassis receives the updated southbound database, with the updated \fBnb_cfg\fR\[char46] This process in turn updates the physical flows installed in the chassis\(cqs Open vSwitch instances\[char46] When it receives confirmation from Open vSwitch that the physical flows have been updated, it updates \fBnb_cfg\fR in its own \fBChassis\fR record in the southbound database\[char46]
.IP 5. .4in
\fBovn\-northd\fR monitors the \fBnb_cfg\fR column in all of the \fBChassis\fR records in the southbound database\[char46] It keeps track of the minimum value among all the records and copies it into the \fBhv_cfg\fR column in the northbound \fBNB_Global\fR table\[char46] (Thus, the CMS or another observer can determine when all of the hypervisors have caught up to the northbound configuration\[char46])
.RE
.SS "Chassis Setup"
.PP
.PP
Each chassis in an OVN deployment must be configured with an Open vSwitch bridge dedicated for OVN\(cqs use, called the \fIintegration bridge\fR\[char46] System startup scripts may create this bridge prior to starting \fBovn\-controller\fR if desired\[char46] If this bridge does not exist when ovn-controller starts, it will be created automatically with the default configuration suggested below\[char46] The ports on the integration bridge include:
.RS
.IP \(bu
On any chassis, tunnel ports that OVN uses to maintain logical network connectivity\[char46] \fBovn\-controller\fR adds, updates, and removes these tunnel ports\[char46]
.IP \(bu
On a hypervisor, any VIFs that are to be attached to logical networks\[char46] The hypervisor itself, or the integration between Open vSwitch and the hypervisor (described in \fBIntegrationGuide\[char46]rst\fR) takes care of this\[char46] (This is not part of OVN or new to OVN; this is pre-existing integration work that has already been done on hypervisors that support OVS\[char46])
.IP \(bu
On a gateway, the physical port used for logical network connectivity\[char46] System startup scripts add this port to the bridge prior to starting \fBovn\-controller\fR\[char46] This can be a patch port to another bridge, instead of a physical port, in more sophisticated setups\[char46]
.RE
.PP
.PP
Other ports should not be attached to the integration bridge\[char46] In particular, physical ports attached to the underlay network (as opposed to gateway ports, which are physical ports attached to logical networks) must not be attached to the integration bridge\[char46] Underlay physical ports should instead be attached to a separate Open vSwitch bridge (they need not be attached to any bridge at all, in fact)\[char46]
.PP
.PP
The integration bridge should be configured as described below\[char46] The effect of each of these settings is documented in \fBovs\-vswitchd\[char46]conf\[char46]db\fR(5):
.RS
.TP
\fBfail\-mode=secure\fR
Avoids switching packets between isolated logical networks before \fBovn\-controller\fR starts up\[char46] See \fBController Failure
Settings\fR in \fBovs\-vsctl\fR(8) for more information\[char46]
.TP
\fBother\-config:disable\-in\-band=true\fR
Suppresses in-band control flows for the integration bridge\[char46] It would be unusual for such flows to show up anyway, because OVN uses a local controller (over a Unix domain socket) instead of a remote controller\[char46] It\(cqs possible, however, for some other bridge in the same system to have an in-band remote controller, and in that case this suppresses the flows that in-band control would ordinarily set up\[char46] Refer to the documentation for more information\[char46]
.RE
.PP
.PP
The customary name for the integration bridge is \fBbr\-int\fR, but another name may be used\[char46]
.SS "Logical Networks"
.PP
.PP
A \fIlogical network\fR implements the same concepts as physical networks, but they are insulated from the physical network with tunnels or other encapsulations\[char46] This allows logical networks to have separate IP and other address spaces that overlap, without conflicting, with those used for physical networks\[char46] Logical network topologies can be arranged without regard for the topologies of the physical networks on which they run\[char46]
.PP
.PP
Logical network concepts in OVN include:
.RS
.IP \(bu
\fILogical switches\fR, the logical version of Ethernet switches\[char46]
.IP \(bu
\fILogical routers\fR, the logical version of IP routers\[char46] Logical switches and routers can be connected into sophisticated topologies\[char46]
.IP \(bu
\fILogical datapaths\fR are the logical version of an OpenFlow switch\[char46] Logical switches and routers are both implemented as logical datapaths\[char46]
.IP \(bu
\fILogical ports\fR represent the points of connectivity in and out of logical switches and logical routers\[char46] Some common types of logical ports are:
.RS
.IP \(bu
Logical ports representing VIFs\[char46]
.IP \(bu
\fILocalnet ports\fR represent the points of connectivity between logical switches and the physical network\[char46] They are implemented as OVS patch ports between the integration bridge and the separate Open vSwitch bridge that underlay physical ports attach to\[char46]
.IP \(bu
\fILogical patch ports\fR represent the points of connectivity between logical switches and logical routers, and in some cases between peer logical routers\[char46] There is a pair of logical patch ports at each such point of connectivity, one on each side\[char46]
.IP \(bu
\fILocalport ports\fR represent the points of local connectivity between logical switches and VIFs\[char46] These ports are present in every chassis (not bound to any particular one) and traffic from them will never go through a tunnel\[char46] A \fBlocalport\fR is expected to only generate traffic destined for a local destination, typically in response to a request it received\[char46] One use case is how OpenStack Neutron uses a \fBlocalport\fR port for serving metadata to VM\(cqs residing on every hypervisor\[char46] A metadata proxy process is attached to this port on every host and all VM\(cqs within the same network will reach it at the same IP/MAC address without any traffic being sent over a tunnel\[char46] Further details can be seen at https://docs\[char46]openstack\[char46]org/developer/networking-ovn/design/metadata_api\[char46]html\[char46]
.RE
.RE
.SS "Life Cycle of a VIF"
.PP
.PP
Tables and their schemas presented in isolation are difficult to understand\[char46] Here\(cqs an example\[char46]
.PP
.PP
A VIF on a hypervisor is a virtual network interface attached either to a VM or a container running directly on that hypervisor (This is different from the interface of a container running inside a VM)\[char46]
.PP
.PP
The steps in this example refer often to details of the OVN and OVN Northbound database schemas\[char46] Please see \fBovn\-sb\fR(5) and \fBovn\-nb\fR(5), respectively, for the full story on these databases\[char46]
.RS
.IP 1. .4in
A VIF\(cqs life cycle begins when a CMS administrator creates a new VIF using the CMS user interface or API and adds it to a switch (one implemented by OVN as a logical switch)\[char46] The CMS updates its own configuration\[char46] This includes associating unique, persistent identifier \fIvif-id\fR and Ethernet address \fImac\fR with the VIF\[char46]
.IP 2. .4in
The CMS plugin updates the OVN Northbound database to include the new VIF, by adding a row to the \fBLogical_Switch_Port\fR table\[char46] In the new row, \fBname\fR is \fIvif-id\fR, \fBmac\fR is \fImac\fR, \fBswitch\fR points to the OVN logical switch\(cqs Logical_Switch record, and other columns are initialized appropriately\[char46]
.IP 3. .4in
\fBovn\-northd\fR receives the OVN Northbound database update\[char46] In turn, it makes the corresponding updates to the OVN Southbound database, by adding rows to the OVN Southbound database \fBLogical_Flow\fR table to reflect the new port, e\[char46]g\[char46] add a flow to recognize that packets destined to the new port\(cqs MAC address should be delivered to it, and update the flow that delivers broadcast and multicast packets to include the new port\[char46] It also creates a record in the \fBBinding\fR table and populates all its columns except the column that identifies the \fBchassis\fR\[char46]
.IP 4. .4in
On every hypervisor, \fBovn\-controller\fR receives the \fBLogical_Flow\fR table updates that \fBovn\-northd\fR made in the previous step\[char46] As long as the VM that owns the VIF is powered off, \fBovn\-controller\fR cannot do much; it cannot, for example, arrange to send packets to or receive packets from the VIF, because the VIF does not actually exist anywhere\[char46]
.IP 5. .4in
Eventually, a user powers on the VM that owns the VIF\[char46] On the hypervisor where the VM is powered on, the integration between the hypervisor and Open vSwitch (described in \fBIntegrationGuide\[char46]rst\fR) adds the VIF to the OVN integration bridge and stores \fIvif-id\fR in \fBexternal_ids\fR:\fBiface\-id\fR to indicate that the interface is an instantiation of the new VIF\[char46] (None of this code is new in OVN; this is pre-existing integration work that has already been done on hypervisors that support OVS\[char46])
.IP 6. .4in
On the hypervisor where the VM is powered on, \fBovn\-controller\fR notices \fBexternal_ids\fR:\fBiface\-id\fR in the new Interface\[char46] In response, in the OVN Southbound DB, it updates the \fBBinding\fR table\(cqs \fBchassis\fR column for the row that links the logical port from \fBexternal_ids\fR:\fB
iface\-id\fR to the hypervisor\[char46] Afterward, \fBovn\-controller\fR updates the local hypervisor\(cqs OpenFlow tables so that packets to and from the VIF are properly handled\[char46]
.IP 7. .4in
Some CMS systems, including OpenStack, fully start a VM only when its networking is ready\[char46] To support this, \fBovn\-northd\fR notices the \fBchassis\fR column updated for the row in \fBBinding\fR table and pushes this upward by updating the \fBup\fR column in the OVN Northbound database\(cqs \fBLogical_Switch_Port\fR table to indicate that the VIF is now up\[char46] The CMS, if it uses this feature, can then react by allowing the VM\(cqs execution to proceed\[char46]
.IP 8. .4in
On every hypervisor but the one where the VIF resides, \fBovn\-controller\fR notices the completely populated row in the \fBBinding\fR table\[char46] This provides \fBovn\-controller\fR the physical location of the logical port, so each instance updates the OpenFlow tables of its switch (based on logical datapath flows in the OVN DB \fBLogical_Flow\fR table) so that packets to and from the VIF can be properly handled via tunnels\[char46]
.IP 9. .4in
Eventually, a user powers off the VM that owns the VIF\[char46] On the hypervisor where the VM was powered off, the VIF is deleted from the OVN integration bridge\[char46]
.IP 10. .4in
On the hypervisor where the VM was powered off, \fBovn\-controller\fR notices that the VIF was deleted\[char46] In response, it removes the \fBChassis\fR column content in the \fBBinding\fR table for the logical port\[char46]
.IP 11. .4in
On every hypervisor, \fBovn\-controller\fR notices the empty \fBChassis\fR column in the \fBBinding\fR table\(cqs row for the logical port\[char46] This means that \fBovn\-controller\fR no longer knows the physical location of the logical port, so each instance updates its OpenFlow table to reflect that\[char46]
.IP 12. .4in
Eventually, when the VIF (or its entire VM) is no longer needed by anyone, an administrator deletes the VIF using the CMS user interface or API\[char46] The CMS updates its own configuration\[char46]
.IP 13. .4in
The CMS plugin removes the VIF from the OVN Northbound database, by deleting its row in the \fBLogical_Switch_Port\fR table\[char46]
.IP 14. .4in
\fBovn\-northd\fR receives the OVN Northbound update and in turn updates the OVN Southbound database accordingly, by removing or updating the rows from the OVN Southbound database \fBLogical_Flow\fR table and \fBBinding\fR table that were related to the now-destroyed VIF\[char46]
.IP 15. .4in
On every hypervisor, \fBovn\-controller\fR receives the \fBLogical_Flow\fR table updates that \fBovn\-northd\fR made in the previous step\[char46] \fBovn\-controller\fR updates OpenFlow tables to reflect the update, although there may not be much to do, since the VIF had already become unreachable when it was removed from the \fBBinding\fR table in a previous step\[char46]
.RE
.SS "Life Cycle of a Container Interface Inside a VM"
.PP
.PP
OVN provides virtual network abstractions by converting information written in OVN_NB database to OpenFlow flows in each hypervisor\[char46] Secure virtual networking for multi-tenants can only be provided if OVN controller is the only entity that can modify flows in Open vSwitch\[char46] When the Open vSwitch integration bridge resides in the hypervisor, it is a fair assumption to make that tenant workloads running inside VMs cannot make any changes to Open vSwitch flows\[char46]
.PP
.PP
If the infrastructure provider trusts the applications inside the containers not to break out and modify the Open vSwitch flows, then containers can be run in hypervisors\[char46] This is also the case when containers are run inside the VMs and Open vSwitch integration bridge with flows added by OVN controller resides in the same VM\[char46] For both the above cases, the workflow is the same as explained with an example in the previous section (\(dqLife Cycle of a VIF\(dq)\[char46]
.PP
.PP
This section talks about the life cycle of a container interface (CIF) when containers are created in the VMs and the Open vSwitch integration bridge resides inside the hypervisor\[char46] In this case, even if a container application breaks out, other tenants are not affected because the containers running inside the VMs cannot modify the flows in the Open vSwitch integration bridge\[char46]
.PP
.PP
When multiple containers are created inside a VM, there are multiple CIFs associated with them\[char46] The network traffic associated with these CIFs need to reach the Open vSwitch integration bridge running in the hypervisor for OVN to support virtual network abstractions\[char46] OVN should also be able to distinguish network traffic coming from different CIFs\[char46] There are two ways to distinguish network traffic of CIFs\[char46]
.PP
.PP
One way is to provide one VIF for every CIF (1:1 model)\[char46] This means that there could be a lot of network devices in the hypervisor\[char46] This would slow down OVS because of all the additional CPU cycles needed for the management of all the VIFs\[char46] It would also mean that the entity creating the containers in a VM should also be able to create the corresponding VIFs in the hypervisor\[char46]
.PP
.PP
The second way is to provide a single VIF for all the CIFs (1:many model)\[char46] OVN could then distinguish network traffic coming from different CIFs via a tag written in every packet\[char46] OVN uses this mechanism and uses VLAN as the tagging mechanism\[char46]
.RS
.IP 1. .4in
A CIF\(cqs life cycle begins when a container is spawned inside a VM by the either the same CMS that created the VM or a tenant that owns that VM or even a container Orchestration System that is different than the CMS that initially created the VM\[char46] Whoever the entity is, it will need to know the \fIvif-id\fR that is associated with the network interface of the VM through which the container interface\(cqs network traffic is expected to go through\[char46] The entity that creates the container interface will also need to choose an unused VLAN inside that VM\[char46]
.IP 2. .4in
The container spawning entity (either directly or through the CMS that manages the underlying infrastructure) updates the OVN Northbound database to include the new CIF, by adding a row to the \fBLogical_Switch_Port\fR table\[char46] In the new row, \fBname\fR is any unique identifier, \fBparent_name\fR is the \fIvif-id\fR of the VM through which the CIF\(cqs network traffic is expected to go through and the \fBtag\fR is the VLAN tag that identifies the network traffic of that CIF\[char46]
.IP 3. .4in
\fBovn\-northd\fR receives the OVN Northbound database update\[char46] In turn, it makes the corresponding updates to the OVN Southbound database, by adding rows to the OVN Southbound database\(cqs \fBLogical_Flow\fR table to reflect the new port and also by creating a new row in the \fBBinding\fR table and populating all its columns except the column that identifies the \fBchassis\fR\[char46]
.IP 4. .4in
On every hypervisor, \fBovn\-controller\fR subscribes to the changes in the \fBBinding\fR table\[char46] When a new row is created by \fBovn\-northd\fR that includes a value in \fBparent_port\fR column of \fBBinding\fR table, the \fBovn\-controller\fR in the hypervisor whose OVN integration bridge has that same value in \fIvif-id\fR in \fBexternal_ids\fR:\fBiface\-id\fR updates the local hypervisor\(cqs OpenFlow tables so that packets to and from the VIF with the particular VLAN \fBtag\fR are properly handled\[char46] Afterward it updates the \fBchassis\fR column of the \fBBinding\fR to reflect the physical location\[char46]
.IP 5. .4in
One can only start the application inside the container after the underlying network is ready\[char46] To support this, \fBovn\-northd\fR notices the updated \fBchassis\fR column in \fBBinding\fR table and updates the \fBup\fR column in the OVN Northbound database\(cqs \fBLogical_Switch_Port\fR table to indicate that the CIF is now up\[char46] The entity responsible to start the container application queries this value and starts the application\[char46]
.IP 6. .4in
Eventually the entity that created and started the container, stops it\[char46] The entity, through the CMS (or directly) deletes its row in the \fBLogical_Switch_Port\fR table\[char46]
.IP 7. .4in
\fBovn\-northd\fR receives the OVN Northbound update and in turn updates the OVN Southbound database accordingly, by removing or updating the rows from the OVN Southbound database \fBLogical_Flow\fR table that were related to the now-destroyed CIF\[char46] It also deletes the row in the \fBBinding\fR table for that CIF\[char46]
.IP 8. .4in
On every hypervisor, \fBovn\-controller\fR receives the \fBLogical_Flow\fR table updates that \fBovn\-northd\fR made in the previous step\[char46] \fBovn\-controller\fR updates OpenFlow tables to reflect the update\[char46]
.RE
.SS "Architectural Physical Life Cycle of a Packet"
.PP
.PP
This section describes how a packet travels from one virtual machine or container to another through OVN\[char46] This description focuses on the physical treatment of a packet; for a description of the logical life cycle of a packet, please refer to the \fBLogical_Flow\fR table in \fBovn\-sb\fR(5)\[char46]
.PP
.PP
This section mentions several data and metadata fields, for clarity summarized here:
.RS
.TP
tunnel key
When OVN encapsulates a packet in Geneve or another tunnel, it attaches extra data to it to allow the receiving OVN instance to process it correctly\[char46] This takes different forms depending on the particular encapsulation, but in each case we refer to it here as the ``tunnel key\[char46]\(cq\(cq See \fBTunnel Encapsulations\fR, below, for details\[char46]
.TP
logical datapath field
A field that denotes the logical datapath through which a packet is being processed\[char46] OVN uses the field that OpenFlow 1\[char46]1+ simply (and confusingly) calls ``metadata\(cq\(cq to store the logical datapath\[char46] (This field is passed across tunnels as part of the tunnel key\[char46])
.TP
logical input port field
A field that denotes the logical port from which the packet entered the logical datapath\[char46] OVN stores this in Open vSwitch extension register number 14\[char46]
.IP
Geneve and STT tunnels pass this field as part of the tunnel key\[char46] Although VXLAN tunnels do not explicitly carry a logical input port, OVN only uses VXLAN to communicate with gateways that from OVN\(cqs perspective consist of only a single logical port, so that OVN can set the logical input port field to this one on ingress to the OVN logical pipeline\[char46]
.TP
logical output port field
A field that denotes the logical port from which the packet will leave the logical datapath\[char46] This is initialized to 0 at the beginning of the logical ingress pipeline\[char46] OVN stores this in Open vSwitch extension register number 15\[char46]
.IP
Geneve and STT tunnels pass this field as part of the tunnel key\[char46] VXLAN tunnels do not transmit the logical output port field\[char46] Since VXLAN tunnels do not carry a logical output port field in the tunnel key, when a packet is received from VXLAN tunnel by an OVN hypervisor, the packet is resubmitted to table 8 to determine the output port(s); when the packet reaches table 32, these packets are resubmitted to table 33 for local delivery by checking a MLF_RCV_FROM_VXLAN flag, which is set when the packet arrives from a VXLAN tunnel\[char46]
.TP
conntrack zone field for logical ports
A field that denotes the connection tracking zone for logical ports\[char46] The value only has local significance and is not meaningful between chassis\[char46] This is initialized to 0 at the beginning of the logical ingress pipeline\[char46] OVN stores this in Open vSwitch extension register number 13\[char46]
.TP
conntrack zone fields for routers
Fields that denote the connection tracking zones for routers\[char46] These values only have local significance and are not meaningful between chassis\[char46] OVN stores the zone information for DNATting in Open vSwitch extension register number 11 and zone information for SNATing in Open vSwitch extension register number 12\[char46]
.TP
logical flow flags
The logical flags are intended to handle keeping context between tables in order to decide which rules in subsequent tables are matched\[char46] These values only have local significance and are not meaningful between chassis\[char46] OVN stores the logical flags in Open vSwitch extension register number 10\[char46]
.TP
VLAN ID
The VLAN ID is used as an interface between OVN and containers nested inside a VM (see \fBLife Cycle of a container interface inside a
VM\fR, above, for more information)\[char46]
.RE
.PP
.PP
Initially, a VM or container on the ingress hypervisor sends a packet on a port attached to the OVN integration bridge\[char46] Then:
.RS
.IP 1. .4in
OpenFlow table 0 performs physical-to-logical translation\[char46] It matches the packet\(cqs ingress port\[char46] Its actions annotate the packet with logical metadata, by setting the logical datapath field to identify the logical datapath that the packet is traversing and the logical input port field to identify the ingress port\[char46] Then it resubmits to table 8 to enter the logical ingress pipeline\[char46]
.IP
Packets that originate from a container nested within a VM are treated in a slightly different way\[char46] The originating container can be distinguished based on the VIF-specific VLAN ID, so the physical-to-logical translation flows additionally match on VLAN ID and the actions strip the VLAN header\[char46] Following this step, OVN treats packets from containers just like any other packets\[char46]
.IP
Table 0 also processes packets that arrive from other chassis\[char46] It distinguishes them from other packets by ingress port, which is a tunnel\[char46] As with packets just entering the OVN pipeline, the actions annotate these packets with logical datapath and logical ingress port metadata\[char46] In addition, the actions set the logical output port field, which is available because in OVN tunneling occurs after the logical output port is known\[char46] These three pieces of information are obtained from the tunnel encapsulation metadata (see \fBTunnel
Encapsulations\fR for encoding details)\[char46] Then the actions resubmit to table 33 to enter the logical egress pipeline\[char46]
.IP 2. .4in
OpenFlow tables 8 through 31 execute the logical ingress pipeline from the \fBLogical_Flow\fR table in the OVN Southbound database\[char46] These tables are expressed entirely in terms of logical concepts like logical ports and logical datapaths\[char46] A big part of \fBovn\-controller\fR\(cqs job is to translate them into equivalent OpenFlow (in particular it translates the table numbers: \fBLogical_Flow\fR tables 0 through 23 become OpenFlow tables 8 through 31)\[char46]
.IP
Each logical flow maps to one or more OpenFlow flows\[char46] An actual packet ordinarily matches only one of these, although in some cases it can match more than one of these flows (which is not a problem because all of them have the same actions)\[char46] \fBovn\-controller\fR uses the first 32 bits of the logical flow\(cqs UUID as the cookie for its OpenFlow flow or flows\[char46] (This is not necessarily unique, since the first 32 bits of a logical flow\(cqs UUID is not necessarily unique\[char46])
.IP
Some logical flows can map to the Open vSwitch ``conjunctive match\(cq\(cq extension (see \fBovs\-fields\fR(7))\[char46] Flows with a \fBconjunction\fR action use an OpenFlow cookie of 0, because they can correspond to multiple logical flows\[char46] The OpenFlow flow for a conjunctive match includes a match on \fBconj_id\fR\[char46]
.IP
Some logical flows may not be represented in the OpenFlow tables on a given hypervisor, if they could not be used on that hypervisor\[char46] For example, if no VIF in a logical switch resides on a given hypervisor, and the logical switch is not otherwise reachable on that hypervisor (e\[char46]g\[char46] over a series of hops through logical switches and routers starting from a VIF on the hypervisor), then the logical flow may not be represented there\[char46]
.IP
Most OVN actions have fairly obvious implementations in OpenFlow (with OVS extensions), e\[char46]g\[char46] \fBnext;\fR is implemented as \fBresubmit\fR, \fB\fIfield\fB =
\fIconstant\fB;\fR as \fBset_field\fR\[char46] A few are worth describing in more detail:
.RS
.TP
\fBoutput:\fR
Implemented by resubmitting the packet to table 32\[char46] If the pipeline executes more than one \fBoutput\fR action, then each one is separately resubmitted to table 32\[char46] This can be used to send multiple copies of the packet to multiple ports\[char46] (If the packet was not modified between the \fBoutput\fR actions, and some of the copies are destined to the same hypervisor, then using a logical multicast output port would save bandwidth between hypervisors\[char46])
.TP
\fBget_arp(\fIP\fB, \fIA\fB);\fR
.TQ .5in
\fBget_nd(\fIP\fB, \fIA\fB);\fR
Implemented by storing arguments into OpenFlow fields, then resubmitting to table 66, which \fBovn\-controller\fR populates with flows generated from the \fBMAC_Binding\fR table in the OVN Southbound database\[char46] If there is a match in table 66, then its actions store the bound MAC in the Ethernet destination address field\[char46]
.IP
(The OpenFlow actions save and restore the OpenFlow fields used for the arguments, so that the OVN actions do not have to be aware of this temporary use\[char46])
.TP
\fBput_arp(\fIP\fB, \fIA\fB, \fIE\fB);\fR
.TQ .5in
\fBput_nd(\fIP\fB, \fIA\fB, \fIE\fB);\fR
Implemented by storing the arguments into OpenFlow fields, then outputting a packet to \fBovn\-controller\fR, which updates the \fBMAC_Binding\fR table\[char46]
.IP
(The OpenFlow actions save and restore the OpenFlow fields used for the arguments, so that the OVN actions do not have to be aware of this temporary use\[char46])
.RE
.IP 3. .4in
OpenFlow tables 32 through 47 implement the \fBoutput\fR action in the logical ingress pipeline\[char46] Specifically, table 32 handles packets to remote hypervisors, table 33 handles packets to the local hypervisor, and table 34 checks whether packets whose logical ingress and egress port are the same should be discarded\[char46]
.IP
Logical patch ports are a special case\[char46] Logical patch ports do not have a physical location and effectively reside on every hypervisor\[char46] Thus, flow table 33, for output to ports on the local hypervisor, naturally implements output to unicast logical patch ports too\[char46] However, applying the same logic to a logical patch port that is part of a logical multicast group yields packet duplication, because each hypervisor that contains a logical port in the multicast group will also output the packet to the logical patch port\[char46] Thus, multicast groups implement output to logical patch ports in table 32\[char46]
.IP
Each flow in table 32 matches on a logical output port for unicast or multicast logical ports that include a logical port on a remote hypervisor\[char46] Each flow\(cqs actions implement sending a packet to the port it matches\[char46] For unicast logical output ports on remote hypervisors, the actions set the tunnel key to the correct value, then send the packet on the tunnel port to the correct hypervisor\[char46] (When the remote hypervisor receives the packet, table 0 there will recognize it as a tunneled packet and pass it along to table 33\[char46]) For multicast logical output ports, the actions send one copy of the packet to each remote hypervisor, in the same way as for unicast destinations\[char46] If a multicast group includes a logical port or ports on the local hypervisor, then its actions also resubmit to table 33\[char46] Table 32 also includes:
.RS
.IP \(bu
A higher-priority rule to match packets received from VXLAN tunnels, based on flag MLF_RCV_FROM_VXLAN, and resubmit these packets to table 33 for local delivery\[char46] Packets received from VXLAN tunnels reach here because of a lack of logical output port field in the tunnel key and thus these packets needed to be submitted to table 8 to determine the output port\[char46]
.IP \(bu
A higher-priority rule to match packets received from ports of type \fBlocalport\fR, based on the logical input port, and resubmit these packets to table 33 for local delivery\[char46] Ports of type \fBlocalport\fR exist on every hypervisor and by definition their traffic should never go out through a tunnel\[char46]
.IP \(bu
A higher-priority rule to match packets that have the MLF_LOCAL_ONLY logical flow flag set, and whose destination is a multicast address\[char46] This flag indicates that the packet should not be delivered to remote hypervisors, even if the multicast destination includes ports on remote hypervisors\[char46] This flag is used when \fBovn\-controller\fR is the originator of the multicast packet\[char46] Since each \fBovn\-controller\fR instance is originating these packets, the packets only need to be delivered to local ports\[char46]
.IP \(bu
A fallback flow that resubmits to table 33 if there is no other match\[char46]
.RE
.IP
Flows in table 33 resemble those in table 32 but for logical ports that reside locally rather than remotely\[char46] For unicast logical output ports on the local hypervisor, the actions just resubmit to table 34\[char46] For multicast output ports that include one or more logical ports on the local hypervisor, for each such logical port \fIP\fR, the actions change the logical output port to \fIP\fR, then resubmit to table 34\[char46]
.IP
A special case is that when a localnet port exists on the datapath, remote port is connected by switching to the localnet port\[char46] In this case, instead of adding a flow in table 32 to reach the remote port, a flow is added in table 33 to switch the logical outport to the localnet port, and resubmit to table 33 as if it were unicasted to a logical port on the local hypervisor\[char46]
.IP
Table 34 matches and drops packets for which the logical input and output ports are the same and the MLF_ALLOW_LOOPBACK flag is not set\[char46] It resubmits other packets to table 40\[char46]
.IP 4. .4in
OpenFlow tables 40 through 63 execute the logical egress pipeline from the \fBLogical_Flow\fR table in the OVN Southbound database\[char46] The egress pipeline can perform a final stage of validation before packet delivery\[char46] Eventually, it may execute an \fBoutput\fR action, which \fBovn\-controller\fR implements by resubmitting to table 64\[char46] A packet for which the pipeline never executes \fBoutput\fR is effectively dropped (although it may have been transmitted through a tunnel across a physical network)\[char46]
.IP
The egress pipeline cannot change the logical output port or cause further tunneling\[char46]
.IP 5. .4in
Table 64 bypasses OpenFlow loopback when MLF_ALLOW_LOOPBACK is set\[char46] Logical loopback was handled in table 34, but OpenFlow by default also prevents loopback to the OpenFlow ingress port\[char46] Thus, when MLF_ALLOW_LOOPBACK is set, OpenFlow table 64 saves the OpenFlow ingress port, sets it to zero, resubmits to table 65 for logical-to-physical transformation, and then restores the OpenFlow ingress port, effectively disabling OpenFlow loopback prevents\[char46] When MLF_ALLOW_LOOPBACK is unset, table 64 flow simply resubmits to table 65\[char46]
.IP 6. .4in
OpenFlow table 65 performs logical-to-physical translation, the opposite of table 0\[char46] It matches the packet\(cqs logical egress port\[char46] Its actions output the packet to the port attached to the OVN integration bridge that represents that logical port\[char46] If the logical egress port is a container nested with a VM, then before sending the packet the actions push on a VLAN header with an appropriate VLAN ID\[char46]
.RE
.SS "Logical Routers and Logical Patch Ports"
.PP
.PP
Typically logical routers and logical patch ports do not have a physical location and effectively reside on every hypervisor\[char46] This is the case for logical patch ports between logical routers and logical switches behind those logical routers, to which VMs (and VIFs) attach\[char46]
.PP
.PP
Consider a packet sent from one virtual machine or container to another VM or container that resides on a different subnet\[char46] The packet will traverse tables 0 to 65 as described in the previous section \fBArchitectural Physical Life Cycle of a Packet\fR, using the logical datapath representing the logical switch that the sender is attached to\[char46] At table 32, the packet will use the fallback flow that resubmits locally to table 33 on the same hypervisor\[char46] In this case, all of the processing from table 0 to table 65 occurs on the hypervisor where the sender resides\[char46]
.PP
.PP
When the packet reaches table 65, the logical egress port is a logical patch port\[char46] The implementation in table 65 differs depending on the OVS version, although the observed behavior is meant to be the same:
.RS
.IP \(bu
In OVS versions 2\[char46]6 and earlier, table 65 outputs to an OVS patch port that represents the logical patch port\[char46] The packet re-enters the OpenFlow flow table from the OVS patch port\(cqs peer in table 0, which identifies the logical datapath and logical input port based on the OVS patch port\(cqs OpenFlow port number\[char46]
.IP \(bu
In OVS versions 2\[char46]7 and later, the packet is cloned and resubmitted directly to the first OpenFlow flow table in the ingress pipeline, setting the logical ingress port to the peer logical patch port, and using the peer logical patch port\(cqs logical datapath (that represents the logical router)\[char46]
.RE
.PP
.PP
The packet re-enters the ingress pipeline in order to traverse tables 8 to 65 again, this time using the logical datapath representing the logical router\[char46] The processing continues as described in the previous section \fBArchitectural Physical Life Cycle of a Packet\fR\[char46] When the packet reachs table 65, the logical egress port will once again be a logical patch port\[char46] In the same manner as described above, this logical patch port will cause the packet to be resubmitted to OpenFlow tables 8 to 65, this time using the logical datapath representing the logical switch that the destination VM or container is attached to\[char46]
.PP
.PP
The packet traverses tables 8 to 65 a third and final time\[char46] If the destination VM or container resides on a remote hypervisor, then table 32 will send the packet on a tunnel port from the sender\(cqs hypervisor to the remote hypervisor\[char46] Finally table 65 will output the packet directly to the destination VM or container\[char46]
.PP
.PP
The following sections describe two exceptions, where logical routers and/or logical patch ports are associated with a physical location\[char46]
.ST "Gateway Routers"
.PP
.PP
A \fIgateway router\fR is a logical router that is bound to a physical location\[char46] This includes all of the logical patch ports of the logical router, as well as all of the peer logical patch ports on logical switches\[char46] In the OVN Southbound database, the \fBPort_Binding\fR entries for these logical patch ports use the type \fBl3gateway\fR rather than \fBpatch\fR, in order to distinguish that these logical patch ports are bound to a chassis\[char46]
.PP
.PP
When a hypervisor processes a packet on a logical datapath representing a logical switch, and the logical egress port is a \fBl3gateway\fR port representing connectivity to a gateway router, the packet will match a flow in table 32 that sends the packet on a tunnel port to the chassis where the gateway router resides\[char46] This processing in table 32 is done in the same manner as for VIFs\[char46]
.PP
.PP
Gateway routers are typically used in between distributed logical routers and physical networks\[char46] The distributed logical router and the logical switches behind it, to which VMs and containers attach, effectively reside on each hypervisor\[char46] The distributed router and the gateway router are connected by another logical switch, sometimes referred to as a \fBjoin\fR logical switch\[char46] On the other side, the gateway router connects to another logical switch that has a localnet port connecting to the physical network\[char46]
.PP
.PP
When using gateway routers, DNAT and SNAT rules are associated with the gateway router, which provides a central location that can handle one-to-many SNAT (aka IP masquerading)\[char46]
.ST "Distributed Gateway Ports"
.PP
.PP
\fIDistributed gateway ports\fR are logical router patch ports that directly connect distributed logical routers to logical switches with localnet ports\[char46]
.PP
.PP
The primary design goal of distributed gateway ports is to allow as much traffic as possible to be handled locally on the hypervisor where a VM or container resides\[char46] Whenever possible, packets from the VM or container to the outside world should be processed completely on that VM\(cqs or container\(cqs hypervisor, eventually traversing a localnet port instance on that hypervisor to the physical network\[char46] Whenever possible, packets from the outside world to a VM or container should be directed through the physical network directly to the VM\(cqs or container\(cqs hypervisor, where the packet will enter the integration bridge through a localnet port\[char46]
.PP
.PP
In order to allow for the distributed processing of packets described in the paragraph above, distributed gateway ports need to be logical patch ports that effectively reside on every hypervisor, rather than \fBl3gateway\fR ports that are bound to a particular chassis\[char46] However, the flows associated with distributed gateway ports often need to be associated with physical locations, for the following reasons:
.RS
.IP \(bu
The physical network that the localnet port is attached to typically uses L2 learning\[char46] Any Ethernet address used over the distributed gateway port must be restricted to a single physical location so that upstream L2 learning is not confused\[char46] Traffic sent out the distributed gateway port towards the localnet port with a specific Ethernet address must be sent out one specific instance of the distributed gateway port on one specific chassis\[char46] Traffic received from the localnet port (or from a VIF on the same logical switch as the localnet port) with a specific Ethernet address must be directed to the logical switch\(cqs patch port instance on that specific chassis\[char46]
.IP
Due to the implications of L2 learning, the Ethernet address and IP address of the distributed gateway port need to be restricted to a single physical location\[char46] For this reason, the user must specify one chassis associated with the distributed gateway port\[char46] Note that traffic traversing the distributed gateway port using other Ethernet addresses and IP addresses (e\[char46]g\[char46] one-to-one NAT) is not restricted to this chassis\[char46]
.IP
Replies to ARP and ND requests must be restricted to a single physical location, where the Ethernet address in the reply resides\[char46] This includes ARP and ND replies for the IP address of the distributed gateway port, which are restricted to the chassis that the user associated with the distributed gateway port\[char46]
.IP \(bu
In order to support one-to-many SNAT (aka IP masquerading), where multiple logical IP addresses spread across multiple chassis are mapped to a single external IP address, it will be necessary to handle some of the logical router processing on a specific chassis in a centralized manner\[char46] Since the SNAT external IP address is typically the distributed gateway port IP address, and for simplicity, the same chassis associated with the distributed gateway port is used\[char46]
.RE
.PP
.PP
The details of flow restrictions to specific chassis are described in the \fBovn\-northd\fR documentation\[char46]
.PP
.PP
While most of the physical location dependent aspects of distributed gateway ports can be handled by restricting some flows to specific chassis, one additional mechanism is required\[char46] When a packet leaves the ingress pipeline and the logical egress port is the distributed gateway port, one of two different sets of actions is required at table 32:
.RS
.IP \(bu
If the packet can be handled locally on the sender\(cqs hypervisor (e\[char46]g\[char46] one-to-one NAT traffic), then the packet should just be resubmitted locally to table 33, in the normal manner for distributed logical patch ports\[char46]
.IP \(bu
However, if the packet needs to be handled on the chassis associated with the distributed gateway port (e\[char46]g\[char46] one-to-many SNAT traffic or non-NAT traffic), then table 32 must send the packet on a tunnel port to that chassis\[char46]
.RE
.PP
.PP
In order to trigger the second set of actions, the \fBchassisredirect\fR type of southbound \fBPort_Binding\fR has been added\[char46] Setting the logical egress port to the type \fBchassisredirect\fR logical port is simply a way to indicate that although the packet is destined for the distributed gateway port, it needs to be redirected to a different chassis\[char46] At table 32, packets with this logical egress port are sent to a specific chassis, in the same way that table 32 directs packets whose logical egress port is a VIF or a type \fBl3gateway\fR port to different chassis\[char46] Once the packet arrives at that chassis, table 33 resets the logical egress port to the value representing the distributed gateway port\[char46] For each distributed gateway port, there is one type \fBchassisredirect\fR port, in addition to the distributed logical patch port representing the distributed gateway port\[char46]
.ST "High Availability for Distributed Gateway Ports"
.PP
.PP
OVN allows you to specify a prioritized list of chassis for a distributed gateway port\[char46] This is done by associating multiple \fBGateway_Chassis\fR rows with a \fBLogical_Router_Port\fR in the \fBOVN_Northbound\fR database\[char46]
.PP
.PP
When multiple chassis have been specified for a gateway, all chassis that may send packets to that gateway will enable BFD on tunnels to all configured gateway chassis\[char46] The current master chassis for the gateway is the highest priority gateway chassis that is currently viewed as active based on BFD status\[char46]
.PP
.PP
For more information on L3 gateway high availability, please refer to http://docs\[char46]openvswitch\[char46]org/en/latest/topics/high-availability\[char46]
.SS "Multiple localnet logical switches connected to a Logical Router"
.PP
.PP
It is possible to have multiple logical switches each with a localnet port (representing physical networks) connected to a logical router, in which one localnet logical switch may provide the external connectivity via a distributed gateway port and rest of the localnet logical switches use VLAN tagging in the physical network\[char46] It is expected that \fBovn\-bridge\-mappings\fR is configured appropriately on the chassis for all these localnet networks\[char46]
.ST "East West routing"
.PP
.PP
East-West routing between these localnet VLAN tagged logical switches work almost the same way as normal logical switches\[char46] When the VM sends such a packet, then:
.RS
.IP 1. .4in
It first enters the ingress pipeline, and then egress pipeline of the source localnet logical switch datapath\[char46] It then enters the ingress pipeline of the logical router datapath via the logical router port in the source chassis\[char46]
.IP 2. .4in
Routing decision is taken\[char46]
.IP 3. .4in
From the router datapath, packet enters the ingress pipeline and then egress pipeline of the destination localnet logical switch datapath and goes out of the integration bridge to the provider bridge ( belonging to the destination logical switch) via the localnet port\[char46]
.IP 4. .4in
The destination chassis receives the packet via the localnet port and sends it to the integration bridge\[char46] The packet enters the ingress pipeline and then egress pipeline of the destination localnet logical switch and finally gets delivered to the destination VM port\[char46]
.RE
.ST "External traffic"
.PP
.PP
The following happens when a VM sends an external traffic (which requires NATting) and the chassis hosting the VM doesn\(cqt have a distributed gateway port\[char46]
.RS
.IP 1. .4in
The packet first enters the ingress pipeline, and then egress pipeline of the source localnet logical switch datapath\[char46] It then enters the ingress pipeline of the logical router datapath via the logical router port in the source chassis\[char46]
.IP 2. .4in
Routing decision is taken\[char46] Since the gateway router or the distributed gateway port doesn\(cqt reside in the source chassis, the traffic is redirected to the gateway chassis via the tunnel port\[char46]
.IP 3. .4in
The gateway chassis receives the packet via the tunnel port and the packet enters the egress pipeline of the logical router datapath\[char46] NAT rules are applied here\[char46] The packet then enters the ingress pipeline and then egress pipeline of the localnet logical switch datapath which provides external connectivity and finally goes out via the localnet port of the logical switch which provides external connectivity\[char46]
.RE
.PP
.PP
Although this works, the VM traffic is tunnelled when sent from the compute chassis to the gateway chassis\[char46] In order for it to work properly, the MTU of the localnet logical switches must be lowered to account for the tunnel encapsulation\[char46]
.SS "
Centralized routing for localnet VLAN tagged logical switches connected
to a Logical Router
"
.PP
.PP
To overcome the tunnel encapsulation problem described in the previous section, \fBOVN\fR supports the option of enabling centralized routing for localnet VLAN tagged logical switches\[char46] CMS can configure the option \fBoptions:reside-on-redirect-chassis\fR to \fBtrue\fR for each \fBLogical_Router_Port\fR which connects to the localnet VLAN tagged logical switches\[char46] This causes the gateway chassis (hosting the distributed gateway port) to handle all the routing for these networks, making it centralized\[char46] It will reply to the ARP requests for the logical router port IPs\[char46]
.PP
.PP
If the logical router doesn\(cqt have a distributed gateway port connecting to the localnet logical switch which provides external connectivity, then this option is ignored by \fBOVN\fR\[char46]
.PP
.PP
The following happens when a VM sends an east-west traffic which needs to be routed:
.RS
.IP 1. .4in
The packet first enters the ingress pipeline, and then egress pipeline of the source localnet logical switch datapath and is sent out via the localnet port of the source localnet logical switch (instead of sending it to router pipeline)\[char46]
.IP 2. .4in
The gateway chassis receives the packet via the localnet port of the source localnet logical switch and sends it to the integration bridge\[char46] The packet then enters the ingress pipeline, and then egress pipeline of the source localnet logical switch datapath and enters the ingress pipeline of the logical router datapath\[char46]
.IP 3. .4in
Routing decision is taken\[char46]
.IP 4. .4in
From the router datapath, packet enters the ingress pipeline and then egress pipeline of the destination localnet logical switch datapath\[char46] It then goes out of the integration bridge to the provider bridge ( belonging to the destination logical switch) via the localnet port\[char46]
.IP 5. .4in
The destination chassis receives the packet via the localnet port and sends it to the integration bridge\[char46] The packet enters the ingress pipeline and then egress pipeline of the destination localnet logical switch and finally delivered to the destination VM port\[char46]
.RE
.PP
.PP
The following happens when a VM sends an external traffic which requires NATting:
.RS
.IP 1. .4in
The packet first enters the ingress pipeline, and then egress pipeline of the source localnet logical switch datapath and is sent out via the localnet port of the source localnet logical switch (instead of sending it to router pipeline)\[char46]
.IP 2. .4in
The gateway chassis receives the packet via the localnet port of the source localnet logical switch and sends it to the integration bridge\[char46] The packet then enters the ingress pipeline, and then egress pipeline of the source localnet logical switch datapath and enters the ingress pipeline of the logical router datapath\[char46]
.IP 3. .4in
Routing decision is taken and NAT rules are applied\[char46]
.IP 4. .4in
From the router datapath, packet enters the ingress pipeline and then egress pipeline of the localnet logical switch datapath which provides external connectivity\[char46] It then goes out of the integration bridge to the provider bridge (belonging to the logical switch which provides external connectivity) via the localnet port\[char46]
.RE
.PP
.PP
The following happens for the reverse external traffic\[char46]
.RS
.IP 1. .4in
The gateway chassis receives the packet from the localnet port of the logical switch which provides external connectivity\[char46] The packet then enters the ingress pipeline and then egress pipeline of the localnet logical switch (which provides external connectivity)\[char46] The packet then enters the ingress pipeline of the logical router datapath\[char46]
.IP 2. .4in
The ingress pipeline of the logical router datapath applies the unNATting rules\[char46] The packet then enters the ingress pipeline and then egress pipeline of the source localnet logical switch\[char46] Since the source VM doesn\(cqt reside in the gateway chassis, the packet is sent out via the localnet port of the source logical switch\[char46]
.IP 3. .4in
The source chassis receives the packet via the localnet port and sends it to the integration bridge\[char46] The packet enters the ingress pipeline and then egress pipeline of the source localnet logical switch and finally gets delivered to the source VM port\[char46]
.RE
.SS "Life Cycle of a VTEP gateway"
.PP
.PP
A gateway is a chassis that forwards traffic between the OVN-managed part of a logical network and a physical VLAN, extending a tunnel-based logical network into a physical network\[char46]
.PP
.PP
The steps below refer often to details of the OVN and VTEP database schemas\[char46] Please see \fBovn\-sb\fR(5), \fBovn\-nb\fR(5) and \fBvtep\fR(5), respectively, for the full story on these databases\[char46]
.RS
.IP 1. .4in
A VTEP gateway\(cqs life cycle begins with the administrator registering the VTEP gateway as a \fBPhysical_Switch\fR table entry in the \fBVTEP\fR database\[char46] The \fBovn\-controller\-vtep\fR connected to this VTEP database, will recognize the new VTEP gateway and create a new \fBChassis\fR table entry for it in the \fBOVN_Southbound\fR database\[char46]
.IP 2. .4in
The administrator can then create a new \fBLogical_Switch\fR table entry, and bind a particular vlan on a VTEP gateway\(cqs port to any VTEP logical switch\[char46] Once a VTEP logical switch is bound to a VTEP gateway, the \fBovn\-controller\-vtep\fR will detect it and add its name to the \fIvtep_logical_switches\fR column of the \fBChassis\fR table in the \fB
OVN_Southbound\fR database\[char46] Note, the \fItunnel_key\fR column of VTEP logical switch is not filled at creation\[char46] The \fBovn\-controller\-vtep\fR will set the column when the correponding vtep logical switch is bound to an OVN logical network\[char46]
.IP 3. .4in
Now, the administrator can use the CMS to add a VTEP logical switch to the OVN logical network\[char46] To do that, the CMS must first create a new \fBLogical_Switch_Port\fR table entry in the \fB
OVN_Northbound\fR database\[char46] Then, the \fItype\fR column of this entry must be set to \(dqvtep\(dq\[char46] Next, the \fI
vtep-logical-switch\fR and \fIvtep-physical-switch\fR keys in the \fIoptions\fR column must also be specified, since multiple VTEP gateways can attach to the same VTEP logical switch\[char46]
.IP 4. .4in
The newly created logical port in the \fBOVN_Northbound\fR database and its configuration will be passed down to the \fB
OVN_Southbound\fR database as a new \fBPort_Binding\fR table entry\[char46] The \fBovn\-controller\-vtep\fR will recognize the change and bind the logical port to the corresponding VTEP gateway chassis\[char46] Configuration of binding the same VTEP logical switch to a different OVN logical networks is not allowed and a warning will be generated in the log\[char46]
.IP 5. .4in
Beside binding to the VTEP gateway chassis, the \fB
ovn\-controller\-vtep\fR will update the \fItunnel_key\fR column of the VTEP logical switch to the corresponding \fB
Datapath_Binding\fR table entry\(cqs \fItunnel_key\fR for the bound OVN logical network\[char46]
.IP 6. .4in
Next, the \fBovn\-controller\-vtep\fR will keep reacting to the configuration change in the \fBPort_Binding\fR in the \fBOVN_Northbound\fR database, and updating the \fBUcast_Macs_Remote\fR table in the \fBVTEP\fR database\[char46] This allows the VTEP gateway to understand where to forward the unicast traffic coming from the extended external network\[char46]
.IP 7. .4in
Eventually, the VTEP gateway\(cqs life cycle ends when the administrator unregisters the VTEP gateway from the \fBVTEP\fR database\[char46] The \fBovn\-controller\-vtep\fR will recognize the event and remove all related configurations (\fBChassis\fR table entry and port bindings) in the \fBOVN_Southbound\fR database\[char46]
.IP 8. .4in
When the \fBovn\-controller\-vtep\fR is terminated, all related configurations in the \fBOVN_Southbound\fR database and the \fBVTEP\fR database will be cleaned, including \fBChassis\fR table entries for all registered VTEP gateways and their port bindings, and all \fBUcast_Macs_Remote\fR table entries and the \fBLogical_Switch\fR tunnel keys\[char46]
.RE
.SH "SECURITY"
.SS "Role\-Based Access Controls for the Soutbound DB"
.PP
.PP
In order to provide additional security against the possibility of an OVN chassis becoming compromised in such a way as to allow rogue software to make arbitrary modifications to the southbound database state and thus disrupt the OVN network, role-based access controls (see \fBovsdb\-server(1)\fR for additional details) are provided for the southbound database\[char46]
.PP
.PP
The implementation of role-based access controls (RBAC) requires the addition of two tables to an OVSDB schema: the \fBRBAC_Role\fR table, which is indexed by role name and maps the the names of the various tables that may be modifiable for a given role to individual rows in a permissions table containing detailed permission information for that role, and the permission table itself which consists of rows containing the following information:
.RS
.TP
\fBTable Name\fR
The name of the associated table\[char46] This column exists primarily as an aid for humans reading the contents of this table\[char46]
.TP
\fBAuth Criteria\fR
A set of strings containing the names of columns (or column:key pairs for columns containing string:string maps)\[char46] The contents of at least one of the columns or column:key values in a row to be modified, inserted, or deleted must be equal to the ID of the client attempting to act on the row in order for the authorization check to pass\[char46] If the authorization criteria is empty, authorization checking is disabled and all clients for the role will be treated as authorized\[char46]
.TP
\fBInsert/Delete\fR
Row insertion/deletion permission; boolean value indicating whether insertion and deletion of rows is allowed for the associated table\[char46] If true, insertion and deletion of rows is allowed for authorized clients\[char46]
.TP
\fBUpdatable Columns\fR
A set of strings containing the names of columns or column:key pairs that may be updated or mutated by authorized clients\[char46] Modifications to columns within a row are only permitted when the authorization check for the client passes and all columns to be modified are included in this set of modifiable columns\[char46]
.RE
.PP
.PP
RBAC configuration for the OVN southbound database is maintained by ovn-northd\[char46] With RBAC enabled, modifications are only permitted for the \fBChassis\fR, \fBEncap\fR, \fBPort_Binding\fR, and \fBMAC_Binding\fR tables, and are resstricted as follows:
.RS
.TP
\fBChassis\fR
\fBAuthorization\fR: client ID must match the chassis name\[char46]
.IP
\fBInsert/Delete\fR: authorized row insertion and deletion are permitted\[char46]
.IP
\fBUpdate\fR: The columns \fBnb_cfg\fR, \fBexternal_ids\fR, \fBencaps\fR, and \fBvtep_logical_switches\fR may be modified when authorized\[char46]
.TP
\fBEncap\fR
\fBAuthorization\fR: client ID must match the chassis name\[char46]
.IP
\fBInsert/Delete\fR: row insertion and row deletion are permitted\[char46]
.IP
\fBUpdate\fR: The columns \fBtype\fR, \fBoptions\fR, and \fBip\fR can be modified\[char46]
.TP
\fBPort_Binding\fR
\fBAuthorization\fR: disabled (all clients are considered authorized\[char46] A future enhancement may add columns (or keys to \fBexternal_ids\fR) in order to control which chassis are allowed to bind each port\[char46]
.IP
\fBInsert/Delete\fR: row insertion/deletion are not permitted (ovn-northd maintains rows in this table\[char46]
.IP
\fBUpdate\fR: Only modifications to the \fBchassis\fR column are permitted\[char46]
.TP
\fBMAC_Binding\fR
\fBAuthorization\fR: disabled (all clients are considered to be authorized)\[char46]
.IP
\fBInsert/Delete\fR: row insertion/deletion are permitted\[char46]
.IP
\fBUpdate\fR: The columns \fBlogical_port\fR, \fBip\fR, \fBmac\fR, and \fBdatapath\fR may be modified by ovn-controller\[char46]
.RE
.PP
.PP
Enabling RBAC for ovn-controller connections to the southbound database requires the following steps:
.RS
.IP 1. .4in
Creating SSL certificates for each chassis with the certificate CN field set to the chassis name (e\[char46]g\[char46] for a chassis with \fBexternal\-ids:system\-id=chassis\-1\fR, via the command \(dq\fBovs\-pki \-u req+sign chassis\-1 switch\fR\(dq)\[char46]
.IP 2. .4in
Configuring each ovn-controller to use SSL when connecting to the southbound database (e\[char46]g\[char46] via \(dq\fBovs\-vsctl set open \[char46]
external\-ids:ovn\-remote=ssl:x\[char46]x\[char46]x\[char46]x:6642\fR\(dq)\[char46]
.IP 3. .4in
Configuring a southbound database SSL remote with \(dqovn-controller\(dq role (e\[char46]g\[char46] via \(dq\fBovn\-sbctl set\-connection role=ovn\-controller
pssl:6642\fR\(dq)\[char46]
.RE
.SS "Encrypt Tunnel Traffic with IPsec"
.PP
.PP
OVN tunnel traffic goes through physical routers and switches\[char46] These physical devices could be untrusted (devices in public network) or might be compromised\[char46] Enabling encryption to the tunnel traffic can prevent the traffic data from being monitored and manipulated\[char46]
.PP
.PP
The tunnel traffic is encrypted with IPsec\[char46] The CMS sets the \fBipsec\fR column in the northbound \fBNB_Global\fR table to enable or disable IPsec encrytion\[char46] If \fBipsec\fR is true, all OVN tunnels will be encrypted\[char46] If \fBipsec\fR is false, no OVN tunnels will be encrypted\[char46]
.PP
.PP
When CMS updates the \fBipsec\fR column in the northbound \fBNB_Global\fR table, \fBovn\-northd\fR copies the value to the \fBipsec\fR column in the southbound \fBSB_Global\fR table\[char46] \fBovn\-controller\fR in each chassis monitors the southbound database and sets the options of the OVS tunnel interface accordingly\[char46] OVS tunnel interface options are monitored by the \fBovs\-monitor\-ipsec\fR daemon which configures IKE daemon to set up IPsec connections\[char46]
.PP
.PP
Chassis authenticates each other by using certificate\[char46] The authentication succeeds if the other end in tunnel presents a certificate signed by a trusted CA and the common name (CN) matches the expected chassis name\[char46] The SSL certificates used in role-based access controls (RBAC) can be used in IPsec\[char46] Or use \fBovs\-pki\fR to create different certificates\[char46] The certificate is required to be x\[char46]509 version 3, and with CN field and subjectAltName field being set to the chassis name\[char46]
.PP
.PP
The CA certificate, chassis certificate and private key are required to be installed in each chassis before enabling IPsec\[char46] Please see \fBovs\-vswitchd\[char46]conf\[char46]db\fR(5) for setting up CA based IPsec authentication\[char46]
.SH "DESIGN DECISIONS"
.SS "Tunnel Encapsulations"
.PP
.PP
OVN annotates logical network packets that it sends from one hypervisor to another with the following three pieces of metadata, which are encoded in an encapsulation-specific fashion:
.RS
.IP \(bu
24-bit logical datapath identifier, from the \fBtunnel_key\fR column in the OVN Southbound \fBDatapath_Binding\fR table\[char46]
.IP \(bu
15-bit logical ingress port identifier\[char46] ID 0 is reserved for internal use within OVN\[char46] IDs 1 through 32767, inclusive, may be assigned to logical ports (see the \fBtunnel_key\fR column in the OVN Southbound \fBPort_Binding\fR table)\[char46]
.IP \(bu
16-bit logical egress port identifier\[char46] IDs 0 through 32767 have the same meaning as for logical ingress ports\[char46] IDs 32768 through 65535, inclusive, may be assigned to logical multicast groups (see the \fBtunnel_key\fR column in the OVN Southbound \fBMulticast_Group\fR table)\[char46]
.RE
.PP
.PP
For hypervisor-to-hypervisor traffic, OVN supports only Geneve and STT encapsulations, for the following reasons:
.RS
.IP \(bu
Only STT and Geneve support the large amounts of metadata (over 32 bits per packet) that OVN uses (as described above)\[char46]
.IP \(bu
STT and Geneve use randomized UDP or TCP source ports that allows efficient distribution among multiple paths in environments that use ECMP in their underlay\[char46]
.IP \(bu
NICs are available to offload STT and Geneve encapsulation and decapsulation\[char46]
.RE
.PP
.PP
Due to its flexibility, the preferred encapsulation between hypervisors is Geneve\[char46] For Geneve encapsulation, OVN transmits the logical datapath identifier in the Geneve VNI\[char46] OVN transmits the logical ingress and logical egress ports in a TLV with class 0x0102, type 0x80, and a 32-bit value encoded as follows, from MSB to LSB:
.PP
.\" check if in troff mode (TTY)
.if t \{
.PS
boxht = .2
textht = 1/6
fillval = .2
[
B0: box "rsv" width .25
B1: box "ingress port" width .75
B2: box "egress port" width .75
"1" at B0.n above
"0" at B0.s below
"15" at B1.n above
"" at B1.s below
"16" at B2.n above
"" at B2.s below
line <-> invis "" above from B0.nw + (0,textht) to B2.ne + (0,textht)
]
.PE
\}
.\" check if in nroff mode:
.if n \{
.nf
\fL  1       15          16
.br
\fL+---+------------+-----------+
.br
\fL|rsv|ingress port|egress port|
.br
\fL+---+------------+-----------+
.br
\fL  0
.fi
\}
.PP
.PP
Environments whose NICs lack Geneve offload may prefer STT encapsulation for performance reasons\[char46] For STT encapsulation, OVN encodes all three pieces of logical metadata in the STT 64-bit tunnel ID as follows, from MSB to LSB:
.PP
.\" check if in troff mode (TTY)
.if t \{
.PS
boxht = .2
textht = 1/6
fillval = .2
[
B0: box "reserved" width .5
B1: box "ingress port" width .75
B2: box "egress port" width .75
B3: box "datapath" width 1.25
"9" at B0.n above
"0" at B0.s below
"15" at B1.n above
"" at B1.s below
"16" at B2.n above
"" at B2.s below
"24" at B3.n above
"" at B3.s below
line <-> invis "" above from B0.nw + (0,textht) to B3.ne + (0,textht)
]
.PE
\}
.\" check if in nroff mode:
.if n \{
.nf
\fL    9          15          16         24
.br
\fL+--------+------------+-----------+--------+
.br
\fL|reserved|ingress port|egress port|datapath|
.br
\fL+--------+------------+-----------+--------+
.br
\fL    0
.fi
\}
.PP
.PP
For connecting to gateways, in addition to Geneve and STT, OVN supports VXLAN, because only VXLAN support is common on top-of-rack (ToR) switches\[char46] Currently, gateways have a feature set that matches the capabilities as defined by the VTEP schema, so fewer bits of metadata are necessary\[char46] In the future, gateways that do not support encapsulations with large amounts of metadata may continue to have a reduced feature set\[char46]