Nexus 7000 Series ISSU Upgrade (Part 2)

After Introduction in Part 1 about the ISSU now it is the time to get the hardware and ISSU Upgrade process.This upgrade was completed from 6.2 (8) to 6.2 (10)

Hardware Details

cisco Nexus7000 C7009 (9 Slot) with 2 Supervisor Modules

Modules

N7K-M148GS-11L
N7K-F248XP-25

Fex Models attached to Nexus Switches

N2K-B22HP-P
N2K-C2232TM-E-10GE
N2K-C2248TP-E-1GE

With the above hardware i have 4 VDC's configured on each site so all the servers and other connections are dual home.

Now you need to download the required version of nexus kickstart and system images.Once you have them you can check the ISSU compatibility checks

If you have LACP configured the you need to run the following command

sh lacp issu-impact

If you have some lacp connections are not compatibility with ISSU as follows

For ISSU to Proceed, Check the following:

1. All port-channel member port should be in a steady state.
2. LACP rate fast should not be enabled on member ports.

The following ports are not ISSU ready

Eth4/44,Eth 3/23

So you need to check what devices are connected to this port in my case we have storage device,Unix server connected to these ports

# sh lacp interface ethernet 4/44

Interface Ethernet4/44 is up
Channel group is 2217 port channel is Po2217
PDUs sent: 2914395
PDUs rcvd: 97904
Markers sent: 0
Markers rcvd: 0
Marker response sent: 0
Marker response rcvd: 0
Unknown packets rcvd: 0
Illegal packets rcvd: 0
Lag Id: [ [(1000, 0-21-28-a5-b4-92, 3e9, 1000, 4), (7f9b, 0-23-4-ee-be-2, 88a9, 8000, 3711)] ]
Operational as aggregated link since Fri Nov 7 11:51:01 2014

Local Port: Eth4/44 MAC Address= e4-c7-22-16-45-42
System Identifier=0x8000,e4-c7-22-16-45-42
Port Identifier=0x8000,0x3711
Operational key=34985
LACP_Activity=active
LACP_Timeout=Long Timeout (30s)
Synchronization=IN_SYNC
Collecting=true
Distributing=true
Actor Admin State=(Ac-1:To-1:Ag-1:Sy-0:Co-0:Di-0:De-0:Ex-0)
Actor Oper State=(Ac-1:To-0:Ag-1:Sy-1:Co-1:Di-1:De-0:Ex-0)
Neighbor: 0/4
MAC Address= 0-21-28-a5-b4-92
System Identifier=0x1000,0-21-28-a5-b4-92
Port Identifier=0x1000,0x4
Operational key=1001
LACP_Activity=active
LACP_Timeout=short Timeout (1s)
Synchronization=IN_SYNC
Collecting=true
Distributing=true
Partner Admin State=(Ac-0:To-1:Ag-0:Sy-0:Co-0:Di-0:De-0:Ex-0)
Partner Oper State=(Ac-1:To-1:Ag-1:Sy-1:Co-1:Di-1:De-0:Ex-0)

Solution

The solution for this is LACP Timeout value need to change Long Timeout on storage Devices Ports,Unix server

Once you fixed the LACP times you can run the commands to check ISSU compatibility.

Now you have to copy the new kickstart and system image to active supervisor and The install routine automatically copies the files to the redundant supervisor for you.

Before starting upgrade process make sure you have backup of all your VDC's or you can create checkpoints for your VDC's

Prior to the upgrade of the code, I highly recommend checking the impact of the upgrade. You can use the following command

#show install all impact kickstart bootflash:n7000-s2-kickstart.6.2.10.bin system bootflash:n7000-s2-dk9.6.2.10.bin

In my case i have received the following error

Installer will perform compatibility check first. Please wait.

Verifying image bootflash:/n7000-s2-kickstart.6.2.10.bin for boot variable "kickstart".
[# ] 0%[####################] 100% --- SUCCESS

Verifying image bootflash:/n7000-s2-dk9.6.2.10.bin for boot variable "system".
[# ] 0%[####################] 100% --- SUCCESS

Verifying image type.
[# ] 0%[####################] 100% --- SUCCESS

Extracting "system" version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% --- SUCCESS

Extracting "kickstart" version from image bootflash:/n7000-s2-kickstart.6.2.10.bin.
[# ] 0%[####################] 100% --- SUCCESS

Extracting "bios" version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% --- SUCCESS

Extracting "lc1n7k" version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% --- SUCCESS

Extracting "fexth" version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% --- SUCCESS

Performing module support checks.
[####################] 100% --- SUCCESS

Notifying services about system upgrade.
[# ] 0% --- FAIL.
Return code 0x41180057 (
The running configuration has inactive policies, which need to be removed.you can view them using ‘show running-config aclmgr inactive-if-config'
you can remove them using ‘clear inactive-config acl/qos`).

Service "aclmgr" in vdc 3:

The running configuration has inactive policies, which need to be removed you can view them using ‘show running-config aclmgr inactive-if-config'
you can remove them using ‘clear inactive-config acl/qos`

Service "aclmgr" returned error:
The running configuration has inactive policies, which need to be removed you can view them using ‘show running-config aclmgr inactive-if-config'
you can remove them using ‘clear inactive-config acl/qos`
(0x41180057)(vdc: 3)
Service "aclmgr" has sent the following message:
The running configuration has inactive policies, which need to be removed you can view them using ‘show running-config aclmgr inactive-if-config'
you can remove them using ‘clear inactive-config acl/qos`
(vdc: 3)

Pre-upgrade check failed. Return code 0x41180057 (The running configuration has inactive policies, which need to be removed you can view them using ‘show running-config aclmgr inactive-if-config'you can remove them using ‘clear inactive-config acl/qos`).

I have to run the following commands to fix

# clear inactive-config acl

# clear inactive-config qos

After that i run the command again and i got the following output

Installer will perform compatibility check first. Please wait.

Verifying image bootflash:/n7000-s2-kickstart.6.2.10.bin for boot variable "kickstart".
[# ] 0%[####################] 100% --- SUCCESS

Verifying image bootflash:/n7000-s2-dk9.6.2.10.bin for boot variable "system".
[# ] 0%[####################] 100% --- SUCCESS

Verifying image type.
[# ] 0%[####################] 100% --- SUCCESS

Extracting "system" version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% --- SUCCESS

Extracting "kickstart" version from image bootflash:/n7000-s2-kickstart.6.2.10.bin.
[# ] 0%[####################] 100% --- SUCCESS

Extracting "bios" version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% --- SUCCESS

Extracting "lc1n7k" version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% --- SUCCESS

Extracting "fexth" version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% --- SUCCESS

Performing module support checks.
[####################] 100% --- SUCCESS

Notifying services about system upgrade.
[####################] 100% --- SUCCESS

Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive reset
2 yes non-disruptive reset
3 yes non-disruptive rolling
4 yes non-disruptive rolling
5 yes non-disruptive rolling
101 yes non-disruptive rolling
102 yes non-disruptive rolling
103 yes non-disruptive rolling
104 yes non-disruptive rolling
105 yes non-disruptive rolling
106 yes non-disruptive rolling
111 yes non-disruptive rolling
121 yes non-disruptive rolling
122 yes non-disruptive rolling
123 yes non-disruptive rolling
124 yes non-disruptive rolling
125 yes non-disruptive rolling

Images will be upgraded according to following table:
Module Image Running-Version(pri:alt) New-Version Upg-Required
------ ---------- ---------------------------------------- -------------------- ------------
1 system 6.2(8) 6.2(10) yes
1 kickstart 6.2(8) 6.2(10) yes
1 bios v2.12.0(05/29/2013):v2.12.0(05/29/2013) v2.12.0(05/29/2013) no
2 system 6.2(8) 6.2(10) yes
2 kickstart 6.2(8) 6.2(10) yes
2 bios v2.12.0(05/29/2013):v2.12.0(05/29/2013) v2.12.0(05/29/2013) no
3 lc1n7k 6.2(8) 6.2(10) yes
3 bios v1.10.21(11/26/12):v1.10.21(11/26/12) v1.10.21(11/26/12) no
4 lc1n7k 6.2(8) 6.2(10) yes
4 bios v2.0.32(12/16/13):v2.0.32(12/16/13) v2.0.32(12/16/13) no
5 lc1n7k 6.2(8) 6.2(10) yes
5 bios v2.0.32(12/16/13):v2.0.32(12/16/13) v2.0.32(12/16/13) no
101 fexth 6.2(8) 6.2(10) yes
102 fexth 6.2(8) 6.2(10) yes
103 fexth 6.2(8) 6.2(10) yes
104 fexth 6.2(8) 6.2(10) yes
105 fexth 6.2(8) 6.2(10) yes
106 fexth 6.2(8) 6.2(10) yes
111 fexth 6.2(8) 6.2(10) yes
121 fexth 6.2(8) 6.2(10) yes
122 fexth 6.2(8) 6.2(10) yes
123 fexth 6.2(8) 6.2(10) yes
124 fexth 6.2(8) 6.2(10) yes
125 fexth 6.2(8) 6.2(10) yes

Note: The key piece of information here is the Impact column. As you can see here,this upgrade is non-disruptive (call it minimal) to traffic. The only two modules that need to be reset are the Supervisors (Mod 1 and 2)

Once you have validated the images, you can now proceed with the upgrade. To upgrade the images, use the install command.

#install all kickstart bootflash:n7000-s2-kickstart.6.2.10.bin system bootflash:n7000-s2-dk9.6.2.10.bin

Installer will perform compatibility check first. Please wait.

Verifying image bootflash:/n7000-s2-kickstart.6.2.10.bin for boot variable "kickstart".
[# ] 0%[####################] 100% --- SUCCESS

Verifying image bootflash:/n7000-s2-dk9.6.2.10.bin for boot variable "system".
[# ] 0%[####################] 100% --- SUCCESS

Verifying image type.
[# ] 0%[####################] 100% --- SUCCESS

Extracting "system" version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% --- SUCCESS

Extracting "kickstart" version from image bootflash:/n7000-s2-kickstart.6.2.10.bin.
[# ] 0%[####################] 100% --- SUCCESS

Extracting "bios" version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% --- SUCCESS

Extracting "lc1n7k" version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% --- SUCCESS

Extracting "fexth" version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% --- SUCCESS

Performing module support checks.
[####################] 100% --- SUCCESS

Notifying services about system upgrade.
[####################] 100% --- SUCCESS

Compatibility check is done:

Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive reset
2 yes non-disruptive reset
3 yes non-disruptive rolling
4 yes non-disruptive rolling
5 yes non-disruptive rolling
101 yes non-disruptive rolling
102 yes non-disruptive rolling
103 yes non-disruptive rolling
104 yes non-disruptive rolling
105 yes non-disruptive rolling
106 yes non-disruptive rolling
111 yes non-disruptive rolling
121 yes non-disruptive rolling
122 yes non-disruptive rolling
123 yes non-disruptive rolling
124 yes non-disruptive rolling
125 yes non-disruptive rolling
126 yes non-disruptive rolling
127 yes non-disruptive rolling
128 yes non-disruptive rolling

Images will be upgraded according to following table:
Module Image Running-Version(pri:alt) New-Version Upg-Required
------ ---------- ---------------------------------------- -------------------- ------------
1 system 6.2(8) 6.2(10) yes
1 kickstart 6.2(8) 6.2(10) yes
1 bios v2.12.0(05/29/2013):v2.12.0(05/29/2013) v2.12.0(05/29/2013) no
2 system 6.2(8) 6.2(10) yes
2 kickstart 6.2(8) 6.2(10) yes
2 bios v2.12.0(05/29/2013):v2.12.0(05/29/2013) v2.12.0(05/29/2013) no
3 lc1n7k 6.2(8) 6.2(10) yes
3 bios v1.10.21(11/26/12):v1.10.21(11/26/12) v1.10.21(11/26/12) no
4 lc1n7k 6.2(8) 6.2(10) yes
4 bios v2.0.32(12/16/13):v2.0.32(12/16/13) v2.0.32(12/16/13) no
5 lc1n7k 6.2(8) 6.2(10) yes
5 bios v2.0.32(12/16/13):v2.0.32(12/16/13) v2.0.32(12/16/13) no
101 fexth 6.2(8) 6.2(10) yes
102 fexth 6.2(8) 6.2(10) yes
103 fexth 6.2(8) 6.2(10) yes
104 fexth 6.2(8) 6.2(10) yes
105 fexth 6.2(8) 6.2(10) yes
106 fexth 6.2(8) 6.2(10) yes
111 fexth 6.2(8) 6.2(10) yes
121 fexth 6.2(8) 6.2(10) yes
122 fexth 6.2(8) 6.2(10) yes
123 fexth 6.2(8) 6.2(10) yes
124 fexth 6.2(8) 6.2(10) yes
125 fexth 6.2(8) 6.2(10) yes
126 fexth 6.2(8) 6.2(10) yes
127 fexth 6.2(8) 6.2(10) yes
128 fexth 6.2(8) 6.2(10) yes

Do you want to continue with the installation (y/n)? [n] y

Note: As you can see, it does the impact analysis for you anyways when you do the install.
You will now need to confirm if you do want to do the upgrade by entering Y

Install is in progress, please wait.

Performing runtime checks.
[####################] 100% --- SUCCESS

Syncing image bootflash:/n7000-s2-kickstart.6.2.10.bin to standby.
[# ] 0%[####################] 100% --- SUCCESS

Syncing image bootflash:/n7000-s2-dk9.6.2.10.bin to standby.
[# ] 0%[####################] 100% --- SUCCESS

Setting boot variables.
[# ] 0%[####################] 100% --- SUCCESS

Performing configuration copy.
[# ] 0%[# ] 0%[# ] 0%[# ] 0%[# ] 0%[###### ] 25%[###### ] 25%[###### ] 25%[###### ] 25%[########### ] 50%[########### ] 50%[########### ] 50%[########### ] 50%[################ ] 75%[################ ] 75%[################ ] 75%[################ ] 75%[####################] 100%[####################] 100% --- SUCCESS

Module 1: Upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[# ] 0%[####################] 100% --- SUCCESS

Module 2: Upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[# ] 0%[####################] 100% --- SUCCESS

Module 3: Upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[# ] 0%[####################] 100% --- SUCCESS

Module 4: Upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[# ] 0%[####################] 100% --- SUCCESS

Module 5: Upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[# ] 0%[####################] 100% --- SUCCESS

Module 1: Waiting for module online.
--- SUCCESS

Notifying services about the switchover.
[####################] 100% --- SUCCESS

"Switching over onto standby".

At this point, the supervisor that was the secondary (module 2 in my example) has reload and come up with the new code. This triggers the primary to initiate a Stateful Switch Over (SSO) to the new code running in the control plane. Meanwhile, data is still traversing the switch with no impact.

Since our SSH session was disconnected during the SSO (telnet isn't SSO aware), we need to re-establish the session and issue a command to continue monitoring the upgrade.

You start monitor the installation process using the following command

# show install all status

Module 1:
Waiting for module online.-- SUCCESS

Module 3:
Non-disruptive upgrading.-- SUCCESS

Module 4:
Non-disruptive upgrading.-- SUCCESS

Module 5:
Non-disruptive upgrading.-- SUCCESS

FEX image downloading in progress.-- SUCCESS

Module 101:
Non-disruptive upgrading.-- SUCCESS

Module 102:
Non-disruptive upgrading.-- SUCCESS

Module 103:
Non-disruptive upgrading.-- SUCCESS

Module 104:
Non-disruptive upgrading.-- SUCCESS

Module 105:
Non-disruptive upgrading.-- SUCCESS

Module 106:
Non-disruptive upgrading.-- SUCCESS

Module 111:
Non-disruptive upgrading.-- SUCCESS

Module 121:
Non-disruptive upgrading.-- SUCCESS

Install has been successful.

Now you can check the installed version using the following command

N7K_1# sh ver

Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_series_home.html
Copyright (c) 2002-2014, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

Software
BIOS: version 2.12.0
kickstart: version 6.2(10)
system: version 6.2(10)

BIOS compile time: 05/29/2013
kickstart image file is: bootflash:///n7000-s2-kickstart.6.2.10.bin
kickstart compile time: 10/9/2014 12:00:00 [10/17/2014 04:45:38]
system image file is: bootflash:///n7000-s2-dk9.6.2.10.bin
system compile time: 10/9/2014 12:00:00 [10/17/2014 06:39:33]

Hardware
cisco Nexus7000 C7009 (9 Slot) Chassis ("Supervisor Module-2")
Intel(R) Xeon(R) CPU with 12224948 kB of memory.
Processor Board ID xxxxxxx

Device name: N7K_1
bootflash: 2007040 kB
slot0: 0 kB (expansion flash)

Kernel uptime is 0 day(s), 0 hour(s), 47 minute(s), 45 second(s)

Last reset
Reason: Unknown
System version: 6.2(8)
Service:

plugin
Core Plugin, Ethernet Plugin

Sponsored Link

2 thoughts on “Nexus 7000 Series ISSU Upgrade (Part 2)

  1. How long would you say it took for the 7K to perform the ISSU upgrade once you had your issues resolved?

Leave a comment

Your email address will not be published. Required fields are marked *