Today, let’s talking about how to step by step to update Mellanox NIC firmware and driver for non-RDMA Storage Space Direct Cluster Servers. You can easy to do via follow steps.

Download the correct firmware and driver for NIC

  1. Access follow link to download Mellanox Firmware Tools (MFT)

    http://www.mellanox.com/page/management_tools

  2. Click 4.6.0, select Windows, select All, select X64, click WinMFT_x64_4_6_0_48.exe and then Save.
  3. Access follow link to download firmware.

http://www.mellanox.com/supportdownloader/

4. Click Adapter Cards, select ConnectX-3 Pro EN, select MCX312B-XCCT, select MT_120001111023, and then click Check for older versions, because fw-ConnectX3Pro-rel-2_40_7000-MCX312B-XCC_Ax-FlexBoot-3.4.746 has performance issue.

5. Select fw-ConnectX3Pro-rel-2_40_5030-MCX312B-XCC_Ax-FlexBoot-3.4.746.bin.zip and then Save.

6. Unzip fw-ConnectX3Pro-rel-2_40_5030-MCX312B-XCC_Ax-FlexBoot-3.4.746.bin.zip file to local drive.

7. Run WinMFT_x64_4_6_0_48.exe file to install Mellanox Firmware Tools.

8. On the Welcome page click Next.

9. On the License Agreement page, select I Accept the terms in the license agreement, click Next.

10. On the Destination Folder page, use the default path and then click Next.

11. Keep the default settings on the Custom Setup page, click Next.

12. Click Install on the Ready to Install the Program page.

13. Click Yes for the User Access Control.

14. On the InstallShield Wizard Completed page, click Finish.

15. Open Command Prompt as Administrator and type “C:\Program Files\Mellanox\WinMFT”.

16. Type mst status and check the device name.

17. Update NIC firmware, type flint -d mt4130_pciconf0 -I c:\post-install\Drivers\fw-Connectx3Pro-rel-2_40_5030-MCX312B-XCC_Ax-FilexBoot-3.4.746.bin\ fw-Connectx3Pro-rel-2_40_5030-MCX312B-XCC_Ax-FilexBoot-3.4.746.bin burn.

18. Run MLNX_VPI_WinOF-5_35_All_win2016_x64 .exe to install new driver for NIC.

19. Click Yes for User Account Control.

20. On the Welcome page, click Next.

21. On the License Agreement page, select I accept the term in the license agreement and then click Next.

22. Keep the default settings on the Custom Setup page, click Next.

23. On the Firmware Upgrade page, unselect Upgrade the HCA’s firmware version, click Next.

24. On the Maximum Performance page, select Check this box to configure your system for maximum performance (Recommended), click Next.

25. On the Setup type page, select Complete and then click Next.

26. Click Install on the Ready Install the Program page.

27. On the InstallShield Wizard Completed page, click Finish.

28. On the restart your system page, click No., we need to do some procedures with cluster services before reboot server.

29. Repeat steps 7-28 for other cluster node servers.

Rebuild Network Stack after NIC Driver Update

If you can arrange an outage maintenance or your failover cluster is still a non-production, you can follow steps and easy to make change without cluster services issues.

  1. Logon to the cluster Node 3 server.
  2. Open Failover Cluster Manager, right-click the cluster name.
  3. Select More Actions, click Shut Down Cluster.


4. Logon to cluster Node 1, Node 2 server and reboot servers.

5. Logon to the cluster Note 3 server and reboot server, after node 3 server reboot, the cluster service will be back

6. Logon Cluster Node 1 server.

7. Run follow PowerShell cmdlet to remove and rebuild Network Stack.

 

#Rebuild Network Stack post-Nic Driver Update


Get-VMNetworkAdapter -ManagementOS |where-object {$_.name -ne 'internal'} | Remove-VMNetworkAdapter

Get-VMSwitch | Where-Object {$_.name -ne 'internal'} | Remove-VMSwitch

Get-NetLbfoTeam | Remove-NetLbfoTeam



#Build NIC Team


New-NetlbfoTeam PRODVLAN "Ethernet", "Ethernet 3" -TeamingMode LACP –verbose


#Get the Status of the Network Adapters

Get-NetAdapter | Sort Name

#Create the new Hyper-V Vswitch VSW01

new-vmswitch "VSW01" -MinimumBandwidthMode Weight -NetAdapterName "PRODVLAN" -verbose

#Check the Bindings

Get-NetadapterBinding | where {$_.DisplayName –like "Hyper-V*"}

#Check the Adapter Settings

Get-NetAdapter | sort name

#Now Create the Converged Adapters


Add-VMNetworkAdapter –ManagementOS –Name "LM" –SwitchName "VSW01" –verbose

Add-VMNetworkAdapter –ManagementOS –Name "HB" –SwitchName "VSW01" –verbose

Add-VMNetworkAdapter –ManagementOS –Name "CLUSTERCSV" –SwitchName "VSW01" –verbose

#Review the NIC Configuration Again

Get-NetAdapter | Sort name

#Rename the HOST NIC

Rename-NetAdapter –Name "vEthernet (VSW01)" –NewName "vEthernet (Host)" –verbose

#Review the NIC Configuration Again

Get-NetAdapter | Sort name

#Set the weighting on the NIC's

Set-VMNetworkAdapter –ManagementOS –Name "CLUSTERCSV" –MinimumBandwidthWeight 40

Set-VMNetworkAdapter –ManagementOS –Name "LM" –MinimumBandwidthWeight 30


Set-VMNetworkAdapter –ManagementOS –Name "HB" –MinimumBandwidthWeight 20

Set-VMNetworkAdapter –ManagementOS –Name "VSW01" –MinimumBandwidthWeight 10

#Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "LM" -Access -VLanID 2

Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "CLUSTERCSV" -Access -VLanID 259

#Configure IPs for Converged Network

New-NetIPAddress -IPAddress 10.5.1.120 -PrefixLength 18 -InterfaceAlias "vEthernet (Host)" -DefaultGateway 10.5.0.1

Set-DnsClientServerAddress -InterfaceAlias "vEthernet (Host)" -ServerAddresses 10.5.1.2,10.5.1.1

New-NetIPAddress -IPAddress 10.5.1.130 -PrefixLength 18 -InterfaceAlias "vEthernet (HB)"

New-NetIPAddress -IPAddress 10.5.1.140 -PrefixLength 18 -InterfaceAlias "vEthernet (LM)"


New-NetIPAddress -IPAddress 172.16.21.10 -PrefixLength 24 -InterfaceAlias "vEthernet (CLUSTERCSV)" 

 

8. Repeat step 6-7 for cluster Node 2 and Node 3.

Note

If your cluster servers are on production and cannot arrange the outage time, you need to follow steps to do Cluster-Aware update for Storage Space Direct Clusters.

  1. Puts each node of the cluster into node maintenance mode
  2. Moves the clustered roles off the node
  3. Update firmware and Installs the drivers
  4. Performs a restart
  5. Run PowerShell cmdlet
  6. Restores the clustered roles on the node
  7. Moves to update the next node

Hope you enjoy this post.

Cary Sun @SifuSun