Cisco ACI Domains

In this blog post I am laying my study notes on the topic Cisco ACI Domains. We learn the concept of workloads, bare metal servers and virtualisation servers first. Second, we discover the types of networking domains Cisco ACI offers. Third, we read about how to configure an external L2 connection. And we finish with an example of configuring an L3 external connection.

Definitions

A Network Engineer can use the following terms interchangeably:

  • ACI domain
  • ACI networking domain
  • ACI domain profile

For the rest of the article I’ll refer to it simply by ACI domain.

An ACI domain defines where and how to use a VLAN Pool. The answer to the “where” question could be a physical server, a virtualized server, an external switch or an external firewall. The answer to the “how” question is understood through the use of AEP, because the AEP brings the encapsulations (the VLANs defined in the VLAN Pool associated to the Networking Domain) to some or all interfaces of a particular Interface Policy Group.

For each VLAN Pool you associate one domain. It is a 1:1 mapping.

There are 4 types of ACI domains or domain profiles:

  • physical domain profiles
  • VMM (virtual) domain profiles
  • External Bridged domain profiles, aka Bridge outside domain profiles
  • External Routed domain profiles, aka Routed outside domain profiles.

An EPG must be attached to only one ACI domain. However, a Domain can be invoked by one or more EPGs.

To be able to logically connect a physical server to ACI fabric you need to configure a Physical Domain on APIC.

To be able to logically connect a virtualized server to ACI fabric you need to configure a VMM Domain on APIC.

Bare Metal Host Concept

A Bare metal host (BMH) is a physical server that does not have any virtualisation capabilities. It is a regular server, the kind of which that you may see sitting shy in a modern datacenter in a corner, near big chassis and blade systems. Th

The bare metal server provides a physical workload, which can be managed by Cisco ACI when we configure the integration between and the bare metal host. Part of this integration is the configuration of a physical domain.

In lab environments, we can emulate a bare metal host by connecting a Nexus switch to the fabric and:

  • configuring the following objects on the Nexus switch: a context, a VLAN, an SVI, an ip route to the bridge domain gateway, a L2 trunk interface,
  • configuring an EPG with static path binding on the ACI fabric,
  • associating a physical domain to the AAEP (not an external routed domain) on the ACI fabric.

Physical Domain

  • A physical domain in ACI connects a physical end point, such as a bare metal host, to the ACI fabric.
  • the default physical domain that comes with ACI has the name of phys.

To configure a physical ACI domain, go to Fabric -> Access Policies -> Physical and External Domains

aci-physical-and-external-domains-menu
Physical and External domains menu in APIC

Right-click -> create Physical Domain

aci-physical-domain-menu
Creating a physical domain

Virtual Domain, aka VMM Domain

  • A Virtual ACI domain connects a virtualized server (aka virtualization host) to the fabric. The interaction occurs between a Virtual Machine Manager such as Vmware vCenter (the management console in a vSphere Cluster) or Microsoft SCVMM and the APIC.
  • when you create a VMM Domain on APIC – as part of integrating a VMM to ACI-, a virtual switch will be created on the virtualized server, and the virtual environment parameters (in the example of VMware: vCenter name, vNICs, list of VMs) becomes visible on the ACI. For this, you have to choose at the configuration menu between VMware DVS or Cisco AVS.
  • We must assign a VLAN/VXLAN pool to the VMM Domain . The VLAN pool will provide the VLAN IDs that will be assigned dynamically to Port Groups on the vSphere Virtual Switch for example in case of Vmware integration with ACI

By the way, both Physical and Virtual Domains require VLAN pool(s) each.

Here is one quick difference between virtual and physical ACI Domains: a physical domain does not use a virtual machine manager.

When we don’t fully integrate a virtual workload to ACI using VMM Integration, then ACI will treat the virtual workload similarly to a physical workload.

When defining a new VMM Domain, you have the choice between VMware DVS or Cisco AVS. If you opt for AVS, then you must ensure that a couple of requirements are there beforehand: ACI must be installed and all fabric switches registered.

The minimum MTU value is 1600, and we should configure it on all devices in the path between ACI fabric and AVS.

External Routed Domains and External Bridged Domains

External Bridged Domains

We use an External Bridged Domain (aka External L2 Domain) when we want to connect an external L2 network, such as a switch, physically to the ACI fabric.

External Bridged Domains are configured under Fabric –> Access Policies –> Physical and External Domains –> External Bridged Domains.

aci-l2-domain-menu

An external bridged domain needs an AAEP and a VLAN Pool.

External Routed Domains

Comparably to external bridged domains, we use an External Routed Domain (aka External L3 Domain) when we connect an external (i.e. non-fabric) L3 network – such as a router or a firewall- physically to the fabric.

aci-domains-external-routed-domain-main-menu

External Routed Domains are configured under Fabric –> Access Policies –> Physical and External Domains –> External Routed Domains.

The following terms can be used interchangeably:

  • external routed domain
  • Layer 3 domain
  • L3 external domain
  • L3 Out domain

But do not confuse it with the External Routed Network construct.

aci-domains-l3-domain-aaep-vlan-pool

The external routed domain needs the already configured AAEP and VLAN Pool:

The external routed domain requires a VLAN Pool, which must contain the VLAN ID that we will configure on the port connected to the external L3 device. The VLAN Pool can be a one-VLAN range.

We can use the same VLAN Pool for more than one External Routed Network (e.g. one L3 Out with OSPF and one L3 Out with EIGRP; both using the same VLAN Pool).

The external routed domain can leverage the same AAEP from a previous integration (e.g. with a bare-metal host), but for administrative purposes we tend to create an AAEP dedicated to the external bridged domain.

Connecting an ACI Fabric to an External L2 Bridged Network

We have two logical possibilities:

Extending an EPG with Static Binding

The purpose of extending an EPG with Static Path Binding is:

  • to support legacy layer 2 networks,
  • to support a Data Center Interconnect,
  • to support non-ACI-integrated hypervisors.

We extend an EPG by statically assigning the port connected to the L2 external network to an internal EPG, and on the opposite device we simply configure 802.1q trunk links.

The VLAN Pool used must contain the VLAN ID that will be defined during the Static Path Binding configuration.

Extending an EPG does not require contracts to communicate with the external switched network. And it is easier to configure than extending a bridge domain.

L2 Out or Extending the Bridge Domain

Extending the bridge domain occurs by configuring a L2 Out. The configuration of a L2 Out involves the configuration of an ACI construct named External Bridged Network or External L2 Network or Bridged Outside, and associating it with an internal Bridge Domain which we want to “extend” to the outside world. That is why we name this method as Bridge Domain extension.

Please note that the concepts External Bridged Network and External Bridge Domain are not the same.

We configure Bridge domain extension per tenant.

Also with bridge domain extension, the VLAN Pool used must contain the VLAN ID that will be defined during VLAN association in an external EPG, and the external VLAN is mapped in ACI to an internal VLAN.

This method of integration with external L2 networks is more secure than extending EPGs, because we need here a contract between internal and external EPGs.

Configuring L2 Out

First, the L2 Out requires the configuration of access policies:

  • Interface Policies
  • Interface Policy Group
  • Interface Profile, and Interface selector(s)
  • Switch Profile, and associate the interface profile to it
  • VLAN Pool
  • AAEP
  • External Bridged Domain.

Once we create the External Bridged Domain, we can not return back to the menu and associate an AAEP for it:

aci-l2-domain-profile
Associating an AAEP to an external bridged domain

However we still are able to associate an AAEP to the External Bridged Domain on the AAEP configuration page:

aci-aaep-external-bridged-domain
Associating an external bridged domain to an AAEP
aci-external-bridged-domain-aaep
aci-domain-aaep-external-bridged-domain
The external bridged domain is associated to the AAEP, as seen on the AAEP page

We go back to the External Bridged Domain configuration page and I see the associated AAEP:

aci-domain-aaep-external-bridged-domain-2
The external bridged domain is associated to the AAEP, as seen on the L2 domain profile page

Associate the VLAN Pool to the External Bridged Domain:

aci-external-bridged-domain-vlan-pool
Associate VLAN Pool to the external bridged domain
  • Configure the External Bridged Network:

After giving it a Name we associate it to the External Bridged Domain we’ve configured:

We select the internal Bridge Domain that we want to extend to the outside switched network:

We configure the VLAN encapsulation that is transiting between the Leaf port and the external switching device:

We must choose this VLAN ID in a way that it falls within the range of the VLAN Pool associated to the External Bridged Domain we’ve configured earlier. So in our case, VLAN ID 24 falls within the range [23-55]:

In Nodes and Interfaces Protocol Profiles we configure the physical topology that the ACI leaf will support toward the external switched network. And here we can choose between:

  • a normal port,
  • a normal port channel,
  • a virtual port channel vPC.

For example I choose vPC:

Path allows us to select the Interface Policy Group, thus indirectly binding a Switch Leaf Profile + AAEP + Interface Profile:

click on add:

Click Next to go to the second step of the Bridged Outside configuration, which is the configuration of external EPG Networks, or sometimes roughly said external networks. We need to configure one only one external EPG per L2 Out connection. And the external EPG is defined by the external VLAN ID. Note that the concept of external networks with L2 Out is not the same as with L3 Out.

Click on the “+” sign to add an external EPG network:

Click Next then Finish. The External Bridged Network object is there:

After we configure the external L2 network object, we are able to see, on the border leaf, the non-ACI attached endpoints with the following command:

Leaf# show endpoints

Cisco ACI and STP

When a layer 2 network is plugged to the ACI fabric, the ACI fabric participation in the forwarding of the STP BPDUs is very limited. In fact, as soon as a Spanning Tree Protocol BPDU enters the fabric, it will be forwarded within the ACI fabric only on the VLAN associated to the EPG (whether it is a VLAN bound statically to the EPG as part of Static Path Binding configuration, or a VLAN associated to the external EPG).

When connecting two or more external switches to an ACI fabric, we have the following interesting facts:

  • ACI relies on the external switches to have STP activated, in order to avoid bridging loops
  • it is recommended to configure a back-to-back vPC.

ACI and MSTP

We configure a Spanning Tree Policy under Switch Policies and we configure a STP domain. Under the STP domain we configure the list of VLANs (VLAN range) associated to each MSTP instance.

Connecting an ACI Fabric to an External L3 Routed Network

The External Routed Network – aka L3 Out- is a construct that connects the ACI fabric to one or more external router(s) for the purpose of exchanging routes. We can do this by setting up what is called the L3 Out connection in ACI.

aci-domains-external-routed-networks-menu

Note that there is no such object in the ACI GUI called L3 Out. No button, no menu. The term refers sometimes to the set of steps that allow to connect an ACI fabric to an external L3 network. The same is valid for L2 Out.

The following terms are interchangeable:

  • L3 Out.
  • External Routed Network.
  • External L3 Routed Network.
  • External routed connection.
  • Routed Outside.

The external routed network assumes that we already prepared the leaf interface that is connecting to the external router. That’s why a requirement of an external routed network is the configuration of the fabric access policies to establish physical connectivity between the fabric and the external router:

Once we prepared all the underlying building blocks, we move on to the configuration of the External Routed Network.

In an ACI multi-tenant design we can:

  • either attach one L3 Out connection to each tenant, or
  • we centralize one L3 Out connection on a shared tenant (either the tenant common or any tenant that provides shared services to other tenants). In this case the L3 Out object is shared.
aci-domains-external-routed-networks-menu

Independently from the chosen design, to create an External Routed Network object, go to Tenants –> {your-tenant} –> Networking –> Eternal Routed Networks –> right-click: Create Routed Outside.

Defining an external routed network in ACI defines with it an external bridge domain.

We have the main configuration menu of the External Routed Network.

The VRF field points to the VRF containing the bridge domain, containing the subnets we want to make reachable from the outside.

We will see later that a lot of magic takes place under Nodes and Interfaces Protocol Profiles. It is where we configure:

  • the Node Profile: which is the ID of the border leaf doing the peering,
  • The Interface Profile; which is a group of policy settings for the border leaf interface that will run OSPF/EIGRP/BGP,
  • the optional loopback interface IP address.

Just like any router, we configure “visibility” of non-connected subnets between the ACI border leaf and the external router. This is possible by configuring a routing protocol such as static routing, OSPF, EIGRP or BGP between the border leaf and the external router. This configuration takes place:

  • on the external router, on one hand,
  • within the External Routed Network object in ACI,

So this means that we configure OSPF in ACI for example similarly as what we are used to do in conventional networking, but this time in the GUI (which I found weird at first).

To choose the peering protocol (with the exception of static routing), we simply need to choose one of the checkboxes under the Routed Outside menu:

Configuring either one of the available protocols is not a lot different when changing from one protocol to another.

If for whatever reason you changed your mind and want to change the peering protocol, then it is “cleaner” to delete the Routed Outside object completely and recreate it.

If you choose EIGRP as the peering protocol, then you need to configure the autonomous system.

If the peering protocol (the protocol between the border leaf and the external routing device) is BGP, then:

  • the border leaf sets its Router_ID to its loopback interface, and
  • the external device must have reachability to that loopback interface through an static route or an IGP route, and
  • we can configure BGP authentication between ACI border leaf and the external device.
  • if we want to have iBGP instead of eBGP, then the BGP AS number must match on both ends of the link.
  • ACI does not give the possibility to choose the peering interface (i.e. the BGP update source interface) of the leaf until you complete the whole BGP config. Then delete the BGP Peer Connectivity Profile manually and recreate it.
  • the menu Create BGP Route Summarization Policy is equivalent to creating a summary route in BGP.

If we choose to configure static routing instead, we do this under the Node Profile, which is part of Nodes and Interfaces Protocol Profiles menu:

After you click on the “+” sign near Nodes and Interfaces Protocol Profiles, the Node Profile menu pops up. the Node Profile is where you answer questions related to the border leaf settings like: on which node the L3 Out is going to be installed? which static routes do we need? which L3 ports are we going to use? which router ID?

Click on the “+” sign near Nodes:

The menu Select Nodes pops up. To add one or more static routes, click on the “+” sign near Static Routes:

In the Create Static Route menu you define all the parameters of your static route that you want to create on ACI border leaf:

We can have many default routes configured as external subnets in the same L3 External Network, but they must be in different external EPGs.

Since we are configuring an External Routed Network, we need to set a layer 3 interface on the border leaf. This is configured as either:

Under the Nodes and Interfaces Protocol Profiles menu add a new Node Profile:

Under Node Profile menu, add an Interface Profile:

Give a name to the Interface Profile and click Next.

Click Next once more:

A L3 external device (router, switch, server…) is attached to the ACI fabric through one of the following physical topologies:

  • an individual port,
  • a vPC,
  • a Port Channel.

If we use vPC as a connection technology with the external routing device:

  • the external routing device peers with both vPC leafs.
  • the vPC peers “peer” with themselves.

Of course, we install the physical cabling on one or more border leafs, depending on the desired physical topology between ACI fabric and the external routing device.

Do not confuse the upper physical topologies with the types of L3 interfaces a border leaf offers when connecting to an external router.

In the next menu we see three possible types of a L3 interface on the ACI leaf to choose from:

  • a regular routed interface (aka routed port or L3 port), or
  • an SVI, or
  • a sub-interface.

In either of the three cases, we can configure the IP address of the interface.

Although in a L3 Out configuration with routed interface we have a layer 3 connection between the border leaf port and the external routing device, we still need a VLAN Pool.

We can configure a loopback address on the border leaf. It occurs also under Nodes and Interfaces Protocol Profiles –> Nodes:

We can control (allow or filter out) route prefixes into and out of the ACI fabric.

An External Routed Network requires the definition of an external network, aka External EPG, which itself will contain the list of non-ACI network prefixes that will be visible from within the ACI fabric. So we must put all external routers that we want to communicate with under an external EPG. We say that through assignment to external EPGs, we are actually classifying external subnet IDs.

Many companies define only one external EPG and within it they define the route 0.0.0.0/0.

We define the external EPG networks always under the Routed Outside object, in the second step:

aci-domains-routed-outside-external-epg-networks
Getting to the configuration menu of the external EPG, within the External Routed Network menu

If the external routed network is shared among tenants, then we need to configure later contracts between the external routed network and the EPGs, and we must set the scope of the contracts to VRF or global, depending on whether the EPGs are assigned the same VRF or are in different tenants.

Making External Subnets Communicate With Internal Subnets

In order to make BD_1_Subnets “seeable” from the external network, we must:

  • set the internal subnets as “advertised externally”
aci-domains-advertised-externally
  • associate BD_1 with the external routed network from BD_1 L3 configurations menu:
  • Export the internal subnets out of APIC

However, the external router won’t be able to ping the subnets. Therefore we need to define a contract between BD_1_EPG and the external EPG.

Once everything is properly configured, we can see the learned external routes in ACI, by going to Fabric –> Inventory –> Protocols.

Transit Routing in ACI

If

  • ACI peers with Protocol_1 on one border leaf, and
  • ACI peers with Protocol_2 on the same (or another) border leaf

then

  • the ACI fabric learns routes form Protocol_1, injects them into MP-BGP and exports them into Protocol_2 and vice-versa.

But in order to establish communication between subnets from Protocol_1 and Protocol_2, we need contracts between the respective external EPGs.

Conclusion

Cisco designed ACI not to be a silo technology, but rather a building block that connects and integrates to physical and virtual workload platforms. ACI integrates with existing L2 and L3 network blocks too.

How is ACI in your organization connected to the rest of the network?

References and further reading

  • https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/2-x/L2_config/b_Cisco_APIC_Layer_2_Configuration_Guide/b_Cisco_APIC_Layer_2_Configuration_Guide_chapter_01.pdf

Leave a Comment