Cumulus NetQ setup on an existing DC fabric lab (real or virtual)
Note: the NetQ elements below are not possible anymore due to changes to the way the data is stored. Check out Cumulus in the Cloud on the Cumulus Networks website to try the new version
The above lab created in EVE-NG is based on the Quickstart: Dual-Attach (MLAG) Demo from this Cumulus github page. All the devices in the lab above are all the same Cumulus VX images.
Also some instructions here on creating Cumulus Spine - Leaf topologies and the use of unnumbered IP addressing on the interfaces.
Additional elements are the devices outside the DC fabric - the ZTP_DHCP device is acting as the REDIS server in the NetQ setup below. The ztp-net and cumulus_ztp_test devices are not used in this post.
NetQ is a telemetry-based fabric validation system. You can try this out for free using Cumulus VX with the important caveat that you can’t try out the NETQ telemetry server unless you have a Cumulus licence to download the VM. But you can setup all the other components and run NetQ validation commands that query the REDIS server to see the fabric state. Good technical NetQ overview video from Cumulus here.
Another great overview of Cumulus and NetQ @Network Field day 17 in this YouTube video - 38 mins
Changes to all switches in the fabric to enable NetQ
Make sure you have inter-device reachability on your management network (Cumulus uses eth0 as a mgmt. port) - NetQ uses this network to send data to the REDIS server.
Edit /etc/apt/sources.list and add the following line:
sudo nano /etc/apt/sources.list
deb http://apps3.cumulusnetworks.com/repos/deb CumulusLinux-3 netq-latest
Run the following commands in a terminal to install the cumulus-netq package:
sudo apt update
sudo apt install cumulus-netq -y
REDIS server
Use apt to install the packages:
sudo apt update
sudo apt install redis-server redis-tools
Open the redis.conf file and if the bind line links to localhost (127.0.0.1) then change it to the IP address of one or more external ports, such as eth0:
sudo nano /etc/redis/redis.conf
Edit the bind line in the file:
bind 192.168.0.250
Restart the REDIS server service:
sudo systemctl restart redis-server.service
Set the REDIS service to start on boot:
sudo systemctl enable redis-server.service
Check status of REDIS service:
systemctl status redis-server.service
Log location for troubleshooting:
logfile /var/log/redis/redis-server.log
Client switches
Specify the IP address of the REDIS server and start the agent. For example:
sudo netq add server 192.168.0.250
sudo netq agent start
Verification commands
Full fabric checks from any of the devices in the fabric:
netq show agents
netq show bgp
netq show interfaces
netq show macs
netq show services
netq show lldp
netq show clag
To show full list of commands:
netq show
You can query any other device from any other device:
netq spine01 show bgp
netq leaf03 show interfaces
Examples of various queries on the lab:
cumulus@spine01:~$ netq show inventory brief
Node Switch OS CPU ASIC Ports
------- -------- ------------- ------ ------ -------
leaf01 VX Cumulus Linux x86_64 N/A N/A
leaf02 VX Cumulus Linux x86_64 N/A N/A
leaf03 VX Cumulus Linux x86_64 N/A N/A
leaf04 VX Cumulus Linux x86_64 N/A N/A
spine01 VX Cumulus Linux x86_64 N/A N/A
spine02 VX Cumulus Linux x86_64 N/A N/A
cumulus@spine01:~$ netq show inventory os
Node OS Name OS Version
------- ------------- ------------
leaf01 Cumulus Linux 3.3.2
leaf02 Cumulus Linux 3.3.2
leaf03 Cumulus Linux 3.3.2
leaf04 Cumulus Linux 3.3.2
spine01 Cumulus Linux 3.4.0
spine02 Cumulus Linux 3.4.0
cumulus@spine01:~$ netq show agents
Node Status Sys Uptime Agent Uptime
------- -------- ------------ --------------
leaf01 Fresh 13m ago 12m ago
leaf02 Fresh 13m ago 12m ago
leaf03 Fresh 13m ago 12m ago
leaf04 Fresh 13m ago 12m ago
spine01 Fresh 13m ago 12m ago
spine02 Fresh 13m ago 12m ago
cumulus@spine01:~$ netq check mtu
Checked Nodes: 6, Checked Links: 68, Failed Nodes: 0, Failed Links: 0
No MTU Mismatch found
cumulus@spine01:~$ netq show ip neighbors
Matching IP Neighbor records are:
IP Node Interface MAC VRF Last Changed
---------------- ---------------- -------------------- ------------------------ ---------------- ----------------
192.168.0.32 spine01 eth0 ac:fd:ce:fc:16:f6 default 2M ago
192.168.0.250 leaf03 eth0 50:00:00:1b:00:00 default 2M ago
192.168.0.250 leaf02 eth0 50:00:00:1b:00:00 default 2M ago
192.168.0.250 leaf01 eth0 50:00:00:1b:00:00 default 23h ago
192.168.0.250 leaf04 eth0 50:00:00:1b:00:00 default 2M ago
192.168.0.250 spine01 eth0 50:00:00:1b:00:00 default 2M ago
192.168.0.250 spine02 eth0 50:00:00:1b:00:00 default 2M ago
192.168.0.1 spine01 eth0 70:50:af:3c:5c:c1 default 2M ago
192.168.0.1 spine02 eth0 70:50:af:3c:5c:c1 default 2M ago
192.168.0.1 leaf03 eth0 70:50:af:3c:5c:c1 default 2M ago
192.168.0.1 leaf01 eth0 70:50:af:3c:5c:c1 default 2M ago
192.168.0.1 leaf04 eth0 70:50:af:3c:5c:c1 default 23h ago
192.168.0.1 leaf02 eth0 70:50:af:3c:5c:c1 default 2M ago
169.254.1.2 leaf01 peerlink.4094 50:00:00:15:00:03 default 2M ago
169.254.1.2 leaf03 peerlink.4094 50:00:00:17:00:03 default 2M ago
169.254.1.1 leaf04 peerlink.4094 50:00:00:16:00:03 default 2M ago
169.254.1.1 leaf02 peerlink.4094 50:00:00:14:00:03 default 2M ago
169.254.0.1 spine02 swp4 50:00:00:17:00:02 default 1W ago
169.254.0.1 spine01 swp4 50:00:00:17:00:01 default 1W ago
169.254.0.1 spine02 swp3 50:00:00:16:00:02 default 1W ago
169.254.0.1 spine01 swp3 50:00:00:16:00:01 default 1W ago
169.254.0.1 leaf03 swp2 50:00:00:13:00:03 default 2M ago
169.254.0.1 leaf04 swp2 50:00:00:13:00:04 default 2M ago
169.254.0.1 leaf02 swp2 50:00:00:13:00:02 default 2M ago
169.254.0.1 spine02 swp2 50:00:00:15:00:02 default 1W ago
169.254.0.1 leaf01 swp2 50:00:00:13:00:01 default 2M ago
169.254.0.1 spine01 swp2 50:00:00:15:00:01 default 1W ago
169.254.0.1 spine02 swp1 50:00:00:14:00:02 default 1W ago
169.254.0.1 spine01 swp1 50:00:00:14:00:01 default 1W ago
169.254.0.1 leaf02 swp1 50:00:00:10:00:02 default 2M ago
169.254.0.1 leaf04 swp1 50:00:00:10:00:04 default 2M ago
169.254.0.1 leaf01 swp1 50:00:00:10:00:01 default 2M ago
169.254.0.1 leaf03 swp1 50:00:00:10:00:03 default 2M ago
cumulus@spine01:~$ netq show bgp
Matching BGP Session records are:
Node Neighbor VRF ASN Peer ASN State PfxRx Last Changed
---------------- -------------------------------- ---------------- ---------- ---------- ------ ------------ --------------
leaf01 swp2(spine02) default 65011 65020 Estd 6 11m ago
leaf01 swp1(spine01) default 65011 65020 Estd 5 12m ago
leaf02 swp2(spine02) default 65012 65020 Estd 5 11m ago
leaf02 swp1(spine01) default 65012 65020 Estd 6 12m ago
leaf03 swp2(spine02) default 65013 65020 Estd 6 Yesterday
leaf03 swp1(spine01) default 65013 65020 Estd 6 Yesterday
leaf04 swp2(spine02) default 65014 65020 Estd 5 Yesterday
leaf04 swp1(spine01) default 65014 65020 Estd 5 Yesterday
spine01 swp2(leaf02) default 65020 65012 Estd 2 1W ago
spine01 swp3(leaf03) default 65020 65013 Estd 2 Yesterday
spine01 swp1(leaf01) default 65020 65011 Estd 2 3M ago
spine01 swp4(leaf04) default 65020 65014 Estd 2 Yesterday
spine02 swp2(leaf02) default 65020 65012 Estd 2 3M ago
spine02 swp3(leaf03) default 65020 65013 Estd 2 Yesterday
spine02 swp1(leaf01) default 65020 65011 Estd 2 1M ago
spine02 swp4(leaf04) default 65020 65014 Estd 2 Yesterday
cumulus@spine01:~$ netq leaf02 show bgp
Matching BGP Session records are:
Node Neighbor VRF ASN Peer ASN State PfxRx Last Changed
---------------- -------------------------------- ---------------- ---------- ---------- ------ ------------ --------------
leaf02 swp2(spine02) default 65012 65020 Estd 5 14m ago
leaf02 swp1(spine01) default 65012 65020 Estd 6 14m ago
cumulus@spine01:~$ netq leaf02 show interfaces
Matching interface records are:
Node Interface Type State Details Last Changed
---------------- ---------------- -------- ----- --------------------------- --------------
leaf02 bridge bridge up Members: server01,peerlink, 4m ago
vxlan10001,vxlan10200,vxlan
10100,
Root Bridge: leaf01,
Root port , MTU: 1500
leaf02 eth0 eth up LLDP: leaf03:eth0, 2M ago
MTU: 1500
leaf02 lo loopback up MTU:65536 2M ago
leaf02 peerlink bond up Slave: swp3(leaf01:swp3), 4m ago
Slave: swp4(leaf01:swp4),
VLANs: 100 200, PVID: 1,
Master: bridge, MTU: 1500
leaf02 peerlink.4094 vlan up MTU:1500 2M ago
leaf02 server01 bond up Slave: swp5(server01:swp2), 4m ago
VLANs: 100 200, PVID: 1,
Master: bridge, MTU: 1500
leaf02 swp1 swp up LLDP: spine01:swp2, 2M ago
MTU: 1500
leaf02 swp2 swp up LLDP: spine02:swp2, 2M ago
MTU: 1500
leaf02 swp3 swp up LLDP: leaf01:swp3, 2M ago
MTU: 1500
leaf02 swp4 swp up LLDP: leaf01:swp4, 2M ago
MTU: 1500
leaf02 swp5 swp up LLDP: server01:swp2, 2M ago
MTU: 1500
leaf02 vxlan10001 vxlan up VNI: 10001, PVID: 1, 4m ago
Master: bridge,
VTEP: 10.10.10.20,
MTU: 1500
leaf02 vxlan10100 vxlan up VNI: 10100, PVID: 100, 4m ago
Master: bridge,
VTEP: 10.10.10.20,
MTU: 1500
leaf02 vxlan10200 vxlan up VNI: 10200, PVID: 200, 4m ago
Master: bridge,
VTEP: 10.10.10.20,
MTU: 1500
cumulus@spine01:~$ netq leaf02 show interfaces type vxlan
Matching interface records are:
Node Interface Type State Details Last Changed
---------------- ---------------- -------- ----- --------------------------- --------------
leaf02 vxlan10001 vxlan up VNI: 10001, PVID: 1, 1m ago
Master: bridge,
VTEP: 10.10.10.20,
MTU: 1500
leaf02 vxlan10100 vxlan up VNI: 10100, PVID: 100, 1m ago
Master: bridge,
VTEP: 10.10.10.20,
MTU: 1500
leaf02 vxlan10200 vxlan up VNI: 10200, PVID: 200, 1m ago
Master: bridge,
VTEP: 10.10.10.20,
MTU: 1500
cumulus@spine01:~$ netq show macs
Matching MAC table records are:
Origin MAC VLAN Node Name Egress Port Last Changed
------ -------------------- -------- ---------------- ---------------- ----------------
1 52:00:00:19:00:02 1 leaf03 server03 2M ago
1 52:00:00:19:00:02 1 leaf04 server03 2M ago
0 52:00:00:19:00:02 1 leaf01 vxlan10001:leaf0 1W ago
3
0 52:00:00:19:00:02 1 leaf02 vxlan10001:leaf0 1W ago
3
1 52:00:00:19:00:01 1 leaf03 server03 2M ago
1 52:00:00:19:00:01 1 leaf04 server03 2M ago
0 52:00:00:19:00:01 1 leaf01 vxlan10001:leaf0 1W ago
3
0 52:00:00:19:00:01 1 leaf02 vxlan10001:leaf0 1W ago
3
1 52:00:00:18:00:02 1 leaf01 server01 2M ago
1 52:00:00:18:00:02 1 leaf02 server01 2M ago
0 52:00:00:18:00:02 1 leaf03 vxlan10001:leaf0 1W ago
1
0 52:00:00:18:00:02 1 leaf04 vxlan10001:leaf0 1W ago
1
1 52:00:00:18:00:01 1 leaf01 server01 2M ago
1 52:00:00:18:00:01 1 leaf02 server01 2M ago
0 52:00:00:18:00:01 1 leaf03 vxlan10001:leaf0 1W ago
1
0 52:00:00:18:00:01 1 leaf04 vxlan10001:leaf0 1W ago
1
cumulus@spine01:~$ netq show changes between 10m and 1s
No changes to specified interfaces found
No changes to interface addresses found
Matching MAC table records are:
Origin MAC VLAN Node Name Egress Port DbState Last Changed
------ -------------------- -------- ---------------- ---------------- ------- --------------
1 2a:e6:b8:c8:51:4b 1 leaf03 vxlan10001 Del 6m ago
1 2a:e6:b8:c8:51:4b 1 leaf04 vxlan10001 Del 6m ago
1 2a:e6:b8:c8:51:4b 1 leaf02 server01 Del 6m ago
1 2a:e6:b8:c8:51:4b 1 leaf01 server01 Del 6m ago
1 02:1f:79:32:4a:22 1 leaf02 vxlan10001 Del 6m ago
1 02:1f:79:32:4a:22 1 leaf01 vxlan10001 Del 6m ago
1 02:1f:79:32:4a:22 1 leaf03 server03 Del 6m ago
1 02:1f:79:32:4a:22 1 leaf04 server03 Del 6m ago
1 2e:fc:96:f9:3d:cc 1 leaf01 vxlan10001 Del 6m ago
1 2e:fc:96:f9:3d:cc 1 leaf02 vxlan10001 Del 6m ago
1 2e:fc:96:f9:3d:cc 1 leaf03 server03 Del 6m ago
1 2e:fc:96:f9:3d:cc 1 leaf04 server03 Del 6m ago
1 8a:ae:c9:50:84:5e 1 leaf03 vxlan10001 Del 8m ago
1 8a:ae:c9:50:84:5e 1 leaf04 vxlan10001 Del 8m ago
1 8a:ae:c9:50:84:5e 1 leaf01 server01 Del 8m ago
1 8a:ae:c9:50:84:5e 1 leaf02 server01 Del 8m ago
1 fa:82:f6:90:ad:cf 1 leaf02 vxlan10001 Del 8m ago
1 fa:82:f6:90:ad:cf 1 leaf01 vxlan10001 Del 8m ago
1 52:cb:e0:4e:1d:d8 1 leaf02 vxlan10001 Del 8m ago
1 52:cb:e0:4e:1d:d8 1 leaf01 vxlan10001 Del 8m ago
1 fa:82:f6:90:ad:cf 1 leaf04 server03 Del 8m ago
1 52:cb:e0:4e:1d:d8 1 leaf04 server03 Del 8m ago
1 fa:82:f6:90:ad:cf 1 leaf03 server03 Del 8m ago
1 52:cb:e0:4e:1d:d8 1 leaf03 server03 Del 8m ago
Matching IP route records are:
Origin Table IP Node Nexthops DbState Last Changed
------ ---------------- -------------------------------- ---------------- ------------------------ -------------- ------------
0 default ff02::1:ff14:2/128 leaf01 swp2 Del 29s ago
0 default ff02::1:ff17:2/128 leaf04 swp2 Del 59s ago
0 default ff02::1:ff14:1/128 leaf01 swp1 Del 59s ago
0 default ff02::1:ff16:2/128 leaf03 swp2 Del 1m ago
0 default ff02::1:ff15:1/128 leaf02 swp1 Del 2m ago
0 default ff02::2/128 spine02 swp3 Del 2m ago
0 default ff02::1:ff17:1/128 leaf04 swp1 Del 2m ago
0 default ff02::1:ff16:1/128 leaf03 swp1 Del 2m ago
0 default ff02::1:ff15:2/128 leaf02 swp2 Del 4m ago
0 default ff02::1:ff14:2/128 leaf01 swp2 Del 4m ago
0 default ff02::1:ff17:2/128 leaf04 swp2 Del 5m ago
0 default ff02::1:ff14:1/128 leaf01 swp1 Del 5m ago
0 default ff02::1:ff16:2/128 leaf03 swp2 Del 5m ago
0 default ff02::1:ff15:1/128 leaf02 swp1 Del 6m ago
0 default ff02::1:ff17:1/128 leaf04 swp1 Del 6m ago
0 default ff02::1:ff16:1/128 leaf03 swp1 Del 7m ago
0 default ff02::2/128 leaf03 eth0 Del 8m ago
0 default ff02::2/128 leaf04 eth0 Del 8m ago
0 default ff02::2/128 leaf01 eth0 Del 8m ago
0 default ff02::1:ff15:2/128 leaf02 swp2 Del 8m ago
0 default ff02::2/128 leaf02 eth0 Del 8m ago
0 default ff02::2/128 leaf04 swp1 Del 8m ago
0 default ff02::1:ff14:2/128 leaf01 swp2 Del 9m ago
0 default ff02::1:ff17:2/128 leaf04 swp2 Del 9m ago
0 default ff02::1:ff14:1/128 leaf01 swp1 Del 9m ago
No changes to IP neighbor table found
No changes to BGP sessions found
No changes to CLAG session found
No changes to LNV session found