Hey Networking fans, I’m back again this time with a really cool little trick to capture RoCE v2 traffic and view it using WireShark. Now I know what all of you are thinking… How can you capture traffic that is offloaded from one Nic to another NIc directly to the OS. Heck if you watch Task Manager you don’t even see any traffic.
Well there is a trick and here is what is required:
- You need to get yourself a custom build of WireShark from Mellanox. https://community.mellanox.com/docs/DOC-2362
- You will also need to use IBDump.exe which is located at C:\Program Files\Mellanox\MLNX_VPI\IB\Tools\IBDump.exe
Then you just need to make some simple configuration changes to your newly installed version of WireShark.
These next few steps below are courtesy Mellanox.
Attached is a preliminary version of Wireshark with InfiniBand support for RoCEv2. In time, it will be added to the Wireshark upstream.
After the installation, click Edit -> Preferences -> Protocols -> InfiniBand, and perform the following adjustments:
1. Type the UDP port as 4791 (RoCEv2 default).
2. Make sure the two check boxes on top are cleared.
Following the steps above, you can open any *.pcapng file to see the InfiniBand parsing of the RoCE traffic.
Here is a parsing example of the BTH header of a CNP packet, OpCode 0x81 (RoCEv2 Congestion Management Ack):
Here is a look at my system….
Then just go onto your Host System with your Mellanox Adapter and run the following:
Initiating resources …
searching for IB devices in host
MR was registered with addr=000002685AF13E30, lkey=0x40024e16, rkey=0x40024e16, flags=0x8
Device : “ibv_device0”
Physical port : 1
Link layer : Ethernet
Dump file : sniffer.pcap
Sniffer WQEs (max burst size) : 4096
Input: guid 7033ed0003078a24, port_num 1, direction 1, qpn 0xaa
Output: reg_id 0
Input: guid 7033ed0003078a24, port_num 1, direction 2, qpn 0xaa
Output: reg_id 0x1
Ready to capture (Press ^c to stop):
Captured: 9483 packets, 7825031 bytes
Interrupted (signal 2) – exiting …
Captured: 9486 packets, 7826645 bytes
-D- Input: guid 0, reg_id 0x1
-D- Input: guid 0, reg_id 0
Last just open it up in our Mellanox Version of WireShark. Let’s see what this looks like shall we:
And there you have it we are able to sniff RoCE / RDMA Traffic we just need the right tools.
Hope you enjoy and Happy learning,