As promised this is a bonus post that will take us into a bit deeper dive on the core Networking stack of Windows Server 2016. Specifically, the new SET Team and RDMA Configurations required for Storage Spaces Direct.

Configuring Datacenter Bridging

 

The following Windows PowerShell script provides an example of how to enable and configure Data Center Bridging (DCB) for SMB Direct on S2DNode1:


#  
# Turn on DCB  
Install-WindowsFeature Data-Center-Bridging  
#  
# Set a policy for SMB-Direct   
New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3  
#  
# Turn on Flow Control for SMB  
Enable-NetQosFlowControl  -Priority 3  
#  
# Make sure flow control is off for other traffic  
Disable-NetQosFlowControl  -Priority 0,1,2,4,5,6,7  
#  
# Apply policy to the target adapters  
Enable-NetAdapterQos  -Name "Ethernet 3", “Ethernet 4”  
#  
# Give SMB Direct 30% of the bandwidth minimum  
New-NetQosTrafficClass "SMB"  -Priority 3  -BandwidthPercentage 30  -Algorithm ETS 

 

 

You can complete the installation of the DataCenter Bridging on the remaining nodes by running the following PowerShell command.


Invoke-Command -ComputerName S2DNode2 -ScriptBlock {
#  
# Turn on DCB  
Install-WindowsFeature Data-Center-Bridging  
#  
# Set a policy for SMB-Direct   
New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3  
#  
# Turn on Flow Control for SMB  
Enable-NetQosFlowControl  -Priority 3  
#  
# Make sure flow control is off for other traffic  
Disable-NetQosFlowControl  -Priority 0,1,2,4,5,6,7  
#  
# Apply policy to the target adapters  
Enable-NetAdapterQos  -Name "Ethernet 3","Ethernet 4"  
#  
# Give SMB Direct 30% of the bandwidth minimum  
New-NetQosTrafficClass "SMB"  -Priority 3  -BandwidthPercentage 30  -Algorithm ETS 

} 

 

Configuring the SET Team

 

A switch-embedded team must be created at the time the Hyper-V switch is created.

When creating the Hyper-V switch using the New-VMSwitch PowerShell cmdlet, the “EnableEmbeddedTeaming” option must be selected.

 

Below is a screen shot of the network adapter configuration prior to building out the networking stack.

Base Networking Configuration

 

Note: You must ensure that you purchase a supported network adapter for your configuration. RDMA over Converged Ethernet is what you want to look for and we personally recommend the: ConnectX®-3 Pro EN Single/Dual-Port Adapters 10/40/56GbE Adapters w/ PCI Express 3.0. More information can be found at Mellanox’s website: http://www.mellanox.com/page/products_dyn?product_family=162&mtag=connectx_3_pro_en_card

 

To build the SET Team you can run the following PowerShell command

New-VMSwitch -Name TeamedvSwitch02 -NetAdapterName "Ethernet 3","Ethernet 4" -EnableEmbeddedTeaming $true

 

You can complete the installation of the SET Team on the remaining nodes by running the following PowerShell command.

Invoke-Command -ComputerName S2DNode2 -ScriptBlock {


New-VMSwitch -Name TeamedvSwitch02 -NetAdapterName "Ethernet 3","Ethernet 4" -EnableEmbeddedTeaming $true 

} 


The result of the creation of the new SET Team looks like this in the Networking Control Panel.

SET Team created

 

Note: Remember that all configurations of the SET Teams are done through SCVMM or PowerShell. When you look at Server Manager you will see that NIC Teaming is disabled because this is only a view for LBFO Teaming.

 

 

Creating RDMA Enabled Virtual Adapters

 

Now that a SET Team has been created we need to create the RDMA enabled Virtual Adapters. This is what S2D will use its core network communications.

 

Below is a screen shot of RDMA Enabled Virtual Adapters post-configuration. I thought I would show you the finished product before showing the script to create them.

That is right under the screen shot.

 

RDMA Virtual Adapters Created

Add-VMNetworkAdapter -SwitchName TeamedvSwitch02 -Name SMB_4 -managementOS  
Add-VMNetworkAdapter -SwitchName TeamedvSwitch02 -Name SMB_3 -managementOS  
Enable-NetAdapterRDMA "vEthernet (SMB_3)","vEthernet (SMB_4)"   


 

You can complete the installation of the RDMA enabled Virtual Adapters on the remaining nodes by running the following PowerShell command.

Invoke-Command -ComputerName S2DNode2 -ScriptBlock {

Add-VMNetworkAdapter -SwitchName TeamedvSwitch02 -Name SMB_3 -managementOS  
Add-VMNetworkAdapter -SwitchName TeamedvSwitch02 -Name SMB_4 -managementOS  
Enable-NetAdapterRDMA "vEthernet (SMB_3)","vEthernet (SMB_4)"  

} 
 


 

Verifying RDMA Enabled Virtual Adapters

 

To verify that the RDMA Enabled Virtual Adapters have been created properly you can run the following PowerShell command. The below PowerShell commands will allow us to see if the configurations have been enabled or not on our Virtual Network Adapters.

Get-NetOffloadGlobalSetting | select networkdirect
Get-NetAdapterRdma 
Get-NetAdapterHardwareInfo 

 

Verifying the RDMA Configurations

 

Viewing RDMA Performance Counters

 

There are specific performance counters that can be viewed for RDMA. Let’s have a look at those.

RDMA Performance Counters

 

 

Lastly, we need to ensure that the nodes are listening on the RDMA Interfaces by using Netstat.   

netstat.exe -xan | ? {$_ -match "445"} 

 

Viewing SMB Direct Connections using Netstat

 

Hopefully this has helped fill in some of the blanks for you on the Networking Configuration with RDMA and SET Teams used for Storage Spaces Direct.

In Part 6 of this Blog Series we will actually build our Failover CLuster and enable Storage Spaces Direct.

Thanks,

 

Cristal

 

Cristal Kawula

Cristal Kawula is the co-founder of MVPDays Community Roadshow and #MVPHour live Twitter Chat. She was also a member of the S2D Technical Advisory board and is the President of TriCon Elite Consulting. Cristal is also only the 2nd woman in the world to receive the prestigious Veeam Vanguard award.

 

BLOG: http://www.checkyourlogs.net

Twitter: @supercristal1