The Ugly Truth About Cisco ACI Bridge Domains

Here are my study notes on the topic ACI Bridge Domains

Bridge Domain: Concepts

A Bridge Domain defines a MAC address space and a L2 flooding domain (layer 2 flooding encompasses layer 2 broadcast traffic and layer 2 multicast traffic), if flooding is enabled on the bridge domain configuration page.

The bridge domain in ACI replaces the function of a VLAN in the traditional network world. Remember we use VLANs to segment broadcast domains and “group” end hosts together according to some business needs? A bridge domain fulfills this function, and more, but without the limitation of a VLAN where we attach one subnet per VLAN. In fact, a bridge domain is a container of 0, 1 or more subnets. In other words, we define IP subnets within bridge domains.

A Bridge Domain that has at least one subnet selects one gateway as the primary IP address of the bridge domain.

We can group subnets altogether in a same bridge domain, or separate them in different bridge domains. The second approach is necessary if you need to place firewall policies between subnets (so you need to have Subnet_A on one bridge domain, and Subnet_B on the other bridge domain, and between both we place the firewall).

ACI bridge domain relationship to other ACI managed objects
ACI bridge domain relationship to other ACI managed objects – copyright

Unlike the traditional network world where we enter complete subnet numbers (like with ACL, route maps and NAT), an IP subnet in ACI is defined by configuring its gateway IP address. This is what we call the Anycast Gateway, aka the Pervasive Gateway. This gateway value will be present on all leafs where an end point attached to an EPG in this Bridge Domain is detected. In this fashion, the gateway of the subnet is no more confined to a single leaf (just like traditional networking, where a gateway IP address is configured on only one router), but spans many leafs.

So which MAC address is associated with the Anycast gateway? The MAC address of the Anycast Gateway is the MAC address we see under the bridge domain configuration page. And we can anytime modify its value. And the same MAC address stays associated to the Anycast gateway, no matter which leaf the gateway IP address is see on. WOW!

We can configure the Anycast gateway in a bridge domain either graphically, or with the NX-OS style CLI on the APIC:

apic# interface bridge-domain {...}

Each Bridge Domain is associated to a unique VNI value.

A Bridge domain attaches to one and only one VRF. In other words, in the configuration menu of the Bridge Domain you are allowed to choose only one VRF to associate to. We can omit to set a VRF initially (APIC GUI allows it), however the system will assign to it the default VRF (named default) which belongs to the tenant common:

default VRF in ACI bridge domains

A Bridge Domain retains a maximum of 1024 IP addresses for each MAC address.

  • A Bridge Domain can be created through context menus (on the left) or graphically with drag and drop. If there is more than one VRF already created, then you must pay attention while dragging the bridge domain symbol: you must release the bridge domain symbol over the desired VRF.
  • The Bridge Domain learns by default all endpoint IP addresses and endpoint MAC addresses in the local (EPG to EPG within the Bridge Domain), inbound and outbound traffic that crosses it. It is however recommended in some cases to restrict the learning process to only subnets local to the Bridge domain. This is performed simply by activating a checkbox.
  • A Bridge Domain is associated with a Multicast group called GIPO (Global IP Outside). Each Bridge Domain has its unique associated GIPO.
  • We say a Bridge Domain is “activated” if an endpoint connects to one of its EPGs.
  • BUM traffic:
    • = Broadcast, Unknown unicast and Multicast traffic. All these types of traffic are considered multi destination traffic: one sends, many receive. Each Bridge Domain is a BUM domain
    • When an endpoint in an EPG in the Bridge Domain has BUM traffic to send, it sets the GIPO associated with the Bridge Domain as the destination address in the frame. The leaf forwards the BUM traffic to the spine that is part of a multicast tree associated with the Bridge Domain. The Spine forwards the BUM traffic to all leaf nodes. Only the EPGs that are part of the Bridge Domain associated with the GIPO will receive the BUM traffic.
    • is transported in VXLAN multicast frames

ARP Flooding

ARP traffic in the ACI fabric is handled in two ways:

  • flood: MAC addresses are learned from the L2 traffic
  • Unicast routing: MAC addresses are learned from L2 traffic and IP addresses are learned from L3 traffic

A Bridge Domain presents the possibility to enable or disable ARP flooding within it:

  • ARP Flooding enabled: the ACI leaf performs traditional ARP protocol operations
  • ARP Flooding disabled + Unicast Routing enabled: the ACI leaf forwards ARP traffic as unicast packets.

Unicast Routing allows the spines to learn endpoint IP-to-VTEP information and insert it in the Mapping database.

If both ARP flooding and Unicast Routing are disabled, ACI forwards traffic as flooding anyway.

Bridge Domain Types

  • A Bridge Domain can be configured as:
    • Legacy Bridge Domain:
    • L2-only Bridge Domain: simply a Bridge Domain with no subnets.
    • L3 Bridge Domain
    • normal Bridge Domain
    • External Bridge Domain:
      • aka External L2 Domain
      • connects an external device to the fabric.
      • together with external EPGs and Node Profiles, the external bridge domain helps define subnets that will be exchanged with the external routers, but does not define whether there is communication in either way between ACI and the external network device. For this matter we need contracts.

Handling L2 Unknown Unicast Traffic

  • proposes two configurable methods to handle L2 Unknown Unicast traffic: Hardware Proxy and L2 Unknown Unicast flooding.
list of configured ACI bridge domains
  • Hardware Proxy: the ACI leaf forwards the unknown unicast frame to a spine, which leverages the MAC-to-VTEP information residing on the Mapping database to determine to which leaf the destination endpoint is attached and sends the frame to it. If the spine finds no information, it discards the frame.
  • Flood: this function uses multicast technology to flood the L2 unknown unicast traffic within the bridge domain.The multicast tree has its base on one of the spines.

L2 Unknown Unicast traffic is by default not flooded, but “hardware proxied”. This is seen on the Bridge Domain configuration menu, where forwarding is by default set to optimize; when we change it to custom, we see that it converts to hardware proxy

first menu in creating an ACI bridge domain
selecting the forwarding method in an ACI bridge domain
selecting how to handle the L2 Unknown Unicast traffic in an ACI bridge domain

If you invoke the help page at the bridge domain menu of the APIC GUI, Cisco official help page displays a checkbox named Clear Remote MAC Entries. Mind you, I searched this field in the Bridge Domain page, on both APIC release 3 and release 4. No success! But I suspect it was substituted a while ago with Limit IP Learning to Subnet. And despite of this fact, Cisco engineers did not take the time to update the Help pages. WTF!

Configuring Bridge Domains

On the APIC, a Bridge Domain is configured under Tenant -> Networking -> Bridge Domains. Rightclick then click on Create Bridge Domain:

Creating ACI bridge domains under Tenant menu

I point my new Bridge Domain to the VRF I desire. I leave the rest of the parameters to their defaults and click Next:

filling in the VRF value in ACI bridge domain

By default the following settings are enabled:

  • Unicast Routing
  • Limit IP Learning to Subnet
L3 configuration in an ACI bridge domain

We can define an Alias to the bridge domain name. In this fashion, the Alias will displayed first, and the name between parenthesis:

properties of an ACI bridge domain

Configuring Bridge Domains with Drag-n-Drop

A Bridge Domain can be configured with drag and drop. Simply drag the Bridge Domain icon onto the desired VRF (im my example Pommy_VRF) and the configuration menu will appear:

configuring ACI bridge domains with drag and drop
configuration menu appears after dragging an ACI bridge domain
result of graphically configuring ACI bridge domain

Configuring Subnets in the Bridge Domain

If you need to define subnets, you can do it within the Bridge Domain configuration menu (clicking on the + sign), or later afterwards. Simply rightclick on the Bridge Domain to add subnets:

create a subnet in an ACI bridge domain

Then add the Gateway IP address of the desired subnet:

setting up the subnet under the ACI bridge domain

You can review the subnets configured under each Bridge Domain:

list of subnets under the ACI bridge domain
  • Subnets can be configured as:
    • private to VRF
    • shared between VRFs
    • advertised externally.

Enabling Legacy Mode

The Legacy Mode option is not visible during the initial configuration menu of the Bridge Domain. But enabling it is a couple of clicks to do:

enabling legacy mode in ACI bridge domains

Then you must define the VLAN encapsulation:

configuring VLAN ID in legacy mode in ACI bridge domain

Displaying configured Bridge Domains in CLI

APIC CommandExplanation
show bridge-domain [BD_NAME] Without the BD_NAME argument, the command displays all configured bridge domains sorted by tenant. With the BD_NAME argument, we can read details about this particular bridge domain.
displaying configured ACI bridge domain in CLI
displaying a particular ACI bridge domain in CLI

Click here to read the rest of my Cisco ACI study notes.

References and further reading


Leave a Comment