After Introduction in Part 1 about the ISSU now it is the time to get the hardware and ISSU Upgrade process.This upgrade was completed from 6.2 (8) to 6.2 (10)
Hardware Details
cisco Nexus7000 C7009 (9 Slot) with 2 Supervisor Modules
Modules
N7K-M148GS-11L
N7K-F248XP-25
Fex Models attached to Nexus Switches
N2K-B22HP-P
N2K-C2232TM-E-10GE
N2K-C2248TP-E-1GE
With the above hardware i have 4 VDC’s configured on each site so all the servers and other connections are dual home.
Now you need to download the required version of nexus kickstart and system images.Once you have them you can check the ISSU compatibility checks
If you have LACP configured the you need to run the following command
sh lacp issu-impact
If you have some lacp connections are not compatibility with ISSU as follows
For ISSU to Proceed, Check the following:
1. All port-channel member port should be in a steady state.
2. LACP rate fast should not be enabled on member ports.
The following ports are not ISSU ready
Eth4/44,Eth 3/23
So you need to check what devices are connected to this port in my case we have storage device,Unix server connected to these ports
# sh lacp interface ethernet 4/44
Interface Ethernet4/44 is up
Channel group is 2217 port channel is Po2217
PDUs sent: 2914395
PDUs rcvd: 97904
Markers sent: 0
Markers rcvd: 0
Marker response sent: 0
Marker response rcvd: 0
Unknown packets rcvd: 0
Illegal packets rcvd: 0
Lag Id: [ [(1000, 0-21-28-a5-b4-92, 3e9, 1000, 4), (7f9b, 0-23-4-ee-be-2, 88a9, 8000, 3711)] ]
Operational as aggregated link since Fri Nov 7 11:51:01 2014
Local Port: Eth4/44 MAC Address= e4-c7-22-16-45-42
System Identifier=0x8000,e4-c7-22-16-45-42
Port Identifier=0x8000,0x3711
Operational key=34985
LACP_Activity=active
LACP_Timeout=Long Timeout (30s)
Synchronization=IN_SYNC
Collecting=true
Distributing=true
Actor Admin State=(Ac-1:To-1:Ag-1:Sy-0:Co-0:Di-0:De-0:Ex-0)
Actor Oper State=(Ac-1:To-0:Ag-1:Sy-1:Co-1:Di-1:De-0:Ex-0)
Neighbor: 0/4
MAC Address= 0-21-28-a5-b4-92
System Identifier=0x1000,0-21-28-a5-b4-92
Port Identifier=0x1000,0x4
Operational key=1001
LACP_Activity=active
LACP_Timeout=short Timeout (1s)
Synchronization=IN_SYNC
Collecting=true
Distributing=true
Partner Admin State=(Ac-0:To-1:Ag-0:Sy-0:Co-0:Di-0:De-0:Ex-0)
Partner Oper State=(Ac-1:To-1:Ag-1:Sy-1:Co-1:Di-1:De-0:Ex-0)
Solution
The solution for this is LACP Timeout value need to change Long Timeout on storage Devices Ports,Unix server
Once you fixed the LACP times you can run the commands to check ISSU compatibility.
Now you have to copy the new kickstart and system image to active supervisor and The install routine automatically copies the files to the redundant supervisor for you.
Before starting upgrade process make sure you have backup of all your VDC’s or you can create checkpoints for your VDC’s
Prior to the upgrade of the code, I highly recommend checking the impact of the upgrade. You can use the following command
#show install all impact kickstart bootflash:n7000-s2-kickstart.6.2.10.bin system bootflash:n7000-s2-dk9.6.2.10.bin
In my case i have received the following error
Installer will perform compatibility check first. Please wait.
Verifying image bootflash:/n7000-s2-kickstart.6.2.10.bin for boot variable “kickstart”.
[# ] 0%[####################] 100% — SUCCESS
Verifying image bootflash:/n7000-s2-dk9.6.2.10.bin for boot variable “system”.
[# ] 0%[####################] 100% — SUCCESS
Verifying image type.
[# ] 0%[####################] 100% — SUCCESS
Extracting “system” version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% — SUCCESS
Extracting “kickstart” version from image bootflash:/n7000-s2-kickstart.6.2.10.bin.
[# ] 0%[####################] 100% — SUCCESS
Extracting “bios” version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% — SUCCESS
Extracting “lc1n7k” version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% — SUCCESS
Extracting “fexth” version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% — SUCCESS
Performing module support checks.
[####################] 100% — SUCCESS
Notifying services about system upgrade.
[# ] 0% — FAIL.
Return code 0x41180057 (
The running configuration has inactive policies, which need to be removed.you can view them using ‘show running-config aclmgr inactive-if-config’
you can remove them using ‘clear inactive-config acl/qos`).Service “aclmgr” in vdc 3:
The running configuration has inactive policies, which need to be removed you can view them using ‘show running-config aclmgr inactive-if-config’
you can remove them using ‘clear inactive-config acl/qos`Service “aclmgr” returned error:
The running configuration has inactive policies, which need to be removed you can view them using ‘show running-config aclmgr inactive-if-config’
you can remove them using ‘clear inactive-config acl/qos`
(0x41180057)(vdc: 3)
Service “aclmgr” has sent the following message:
The running configuration has inactive policies, which need to be removed you can view them using ‘show running-config aclmgr inactive-if-config’
you can remove them using ‘clear inactive-config acl/qos`
(vdc: 3)Pre-upgrade check failed. Return code 0x41180057 (The running configuration has inactive policies, which need to be removed you can view them using ‘show running-config aclmgr inactive-if-config’you can remove them using ‘clear inactive-config acl/qos`).
I have to run the following commands to fix
# clear inactive-config acl
# clear inactive-config qos
After that i run the command again and i got the following output
Installer will perform compatibility check first. Please wait.
Verifying image bootflash:/n7000-s2-kickstart.6.2.10.bin for boot variable “kickstart”.
[# ] 0%[####################] 100% — SUCCESSVerifying image bootflash:/n7000-s2-dk9.6.2.10.bin for boot variable “system”.
[# ] 0%[####################] 100% — SUCCESSVerifying image type.
[# ] 0%[####################] 100% — SUCCESSExtracting “system” version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% — SUCCESSExtracting “kickstart” version from image bootflash:/n7000-s2-kickstart.6.2.10.bin.
[# ] 0%[####################] 100% — SUCCESSExtracting “bios” version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% — SUCCESSExtracting “lc1n7k” version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% — SUCCESSExtracting “fexth” version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% — SUCCESSPerforming module support checks.
[####################] 100% — SUCCESSNotifying services about system upgrade.
[####################] 100% — SUCCESSCompatibility check is done:
Module bootable Impact Install-type Reason
—— ——– ————– ———— ——
1 yes non-disruptive reset
2 yes non-disruptive reset
3 yes non-disruptive rolling
4 yes non-disruptive rolling
5 yes non-disruptive rolling
101 yes non-disruptive rolling
102 yes non-disruptive rolling
103 yes non-disruptive rolling
104 yes non-disruptive rolling
105 yes non-disruptive rolling
106 yes non-disruptive rolling
111 yes non-disruptive rolling
121 yes non-disruptive rolling
122 yes non-disruptive rolling
123 yes non-disruptive rolling
124 yes non-disruptive rolling
125 yes non-disruptive rollingImages will be upgraded according to following table:
Module Image Running-Version(pri:alt) New-Version Upg-Required
—— ———- —————————————- ——————– ————
1 system 6.2(8) 6.2(10) yes
1 kickstart 6.2(8) 6.2(10) yes
1 bios v2.12.0(05/29/2013):v2.12.0(05/29/2013) v2.12.0(05/29/2013) no
2 system 6.2(8) 6.2(10) yes
2 kickstart 6.2(8) 6.2(10) yes
2 bios v2.12.0(05/29/2013):v2.12.0(05/29/2013) v2.12.0(05/29/2013) no
3 lc1n7k 6.2(8) 6.2(10) yes
3 bios v1.10.21(11/26/12):v1.10.21(11/26/12) v1.10.21(11/26/12) no
4 lc1n7k 6.2(8) 6.2(10) yes
4 bios v2.0.32(12/16/13):v2.0.32(12/16/13) v2.0.32(12/16/13) no
5 lc1n7k 6.2(8) 6.2(10) yes
5 bios v2.0.32(12/16/13):v2.0.32(12/16/13) v2.0.32(12/16/13) no
101 fexth 6.2(8) 6.2(10) yes
102 fexth 6.2(8) 6.2(10) yes
103 fexth 6.2(8) 6.2(10) yes
104 fexth 6.2(8) 6.2(10) yes
105 fexth 6.2(8) 6.2(10) yes
106 fexth 6.2(8) 6.2(10) yes
111 fexth 6.2(8) 6.2(10) yes
121 fexth 6.2(8) 6.2(10) yes
122 fexth 6.2(8) 6.2(10) yes
123 fexth 6.2(8) 6.2(10) yes
124 fexth 6.2(8) 6.2(10) yes
125 fexth 6.2(8) 6.2(10) yes
Note: The key piece of information here is the Impact column. As you can see here,this upgrade is non-disruptive (call it minimal) to traffic. The only two modules that need to be reset are the Supervisors (Mod 1 and 2)
Once you have validated the images, you can now proceed with the upgrade. To upgrade the images, use the install command.
#install all kickstart bootflash:n7000-s2-kickstart.6.2.10.bin system bootflash:n7000-s2-dk9.6.2.10.bin
Installer will perform compatibility check first. Please wait.
Verifying image bootflash:/n7000-s2-kickstart.6.2.10.bin for boot variable “kickstart”.
[# ] 0%[####################] 100% — SUCCESS
Verifying image bootflash:/n7000-s2-dk9.6.2.10.bin for boot variable “system”.
[# ] 0%[####################] 100% — SUCCESS
Verifying image type.
[# ] 0%[####################] 100% — SUCCESS
Extracting “system” version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% — SUCCESS
Extracting “kickstart” version from image bootflash:/n7000-s2-kickstart.6.2.10.bin.
[# ] 0%[####################] 100% — SUCCESS
Extracting “bios” version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% — SUCCESS
Extracting “lc1n7k” version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% — SUCCESS
Extracting “fexth” version from image bootflash:/n7000-s2-dk9.6.2.10.bin.
[# ] 0%[####################] 100% — SUCCESS
Performing module support checks.
[####################] 100% — SUCCESS
Notifying services about system upgrade.
[####################] 100% — SUCCESS
Compatibility check is done:
Module bootable Impact Install-type Reason
—— ——– ————– ———— ——
1 yes non-disruptive reset
2 yes non-disruptive reset
3 yes non-disruptive rolling
4 yes non-disruptive rolling
5 yes non-disruptive rolling
101 yes non-disruptive rolling
102 yes non-disruptive rolling
103 yes non-disruptive rolling
104 yes non-disruptive rolling
105 yes non-disruptive rolling
106 yes non-disruptive rolling
111 yes non-disruptive rolling
121 yes non-disruptive rolling
122 yes non-disruptive rolling
123 yes non-disruptive rolling
124 yes non-disruptive rolling
125 yes non-disruptive rolling
126 yes non-disruptive rolling
127 yes non-disruptive rolling
128 yes non-disruptive rolling
Images will be upgraded according to following table:
Module Image Running-Version(pri:alt) New-Version Upg-Required
—— ———- —————————————- ——————– ————
1 system 6.2(8) 6.2(10) yes
1 kickstart 6.2(8) 6.2(10) yes
1 bios v2.12.0(05/29/2013):v2.12.0(05/29/2013) v2.12.0(05/29/2013) no
2 system 6.2(8) 6.2(10) yes
2 kickstart 6.2(8) 6.2(10) yes
2 bios v2.12.0(05/29/2013):v2.12.0(05/29/2013) v2.12.0(05/29/2013) no
3 lc1n7k 6.2(8) 6.2(10) yes
3 bios v1.10.21(11/26/12):v1.10.21(11/26/12) v1.10.21(11/26/12) no
4 lc1n7k 6.2(8) 6.2(10) yes
4 bios v2.0.32(12/16/13):v2.0.32(12/16/13) v2.0.32(12/16/13) no
5 lc1n7k 6.2(8) 6.2(10) yes
5 bios v2.0.32(12/16/13):v2.0.32(12/16/13) v2.0.32(12/16/13) no
101 fexth 6.2(8) 6.2(10) yes
102 fexth 6.2(8) 6.2(10) yes
103 fexth 6.2(8) 6.2(10) yes
104 fexth 6.2(8) 6.2(10) yes
105 fexth 6.2(8) 6.2(10) yes
106 fexth 6.2(8) 6.2(10) yes
111 fexth 6.2(8) 6.2(10) yes
121 fexth 6.2(8) 6.2(10) yes
122 fexth 6.2(8) 6.2(10) yes
123 fexth 6.2(8) 6.2(10) yes
124 fexth 6.2(8) 6.2(10) yes
125 fexth 6.2(8) 6.2(10) yes
126 fexth 6.2(8) 6.2(10) yes
127 fexth 6.2(8) 6.2(10) yes
128 fexth 6.2(8) 6.2(10) yes
Do you want to continue with the installation (y/n)? [n] y
Note: As you can see, it does the impact analysis for you anyways when you do the install.
You will now need to confirm if you do want to do the upgrade by entering Y
Install is in progress, please wait.
Performing runtime checks.
[####################] 100% — SUCCESS
Syncing image bootflash:/n7000-s2-kickstart.6.2.10.bin to standby.
[# ] 0%[####################] 100% — SUCCESS
Syncing image bootflash:/n7000-s2-dk9.6.2.10.bin to standby.
[# ] 0%[####################] 100% — SUCCESS
Setting boot variables.
[# ] 0%[####################] 100% — SUCCESS
Performing configuration copy.
[# ] 0%[# ] 0%[# ] 0%[# ] 0%[# ] 0%[###### ] 25%[###### ] 25%[###### ] 25%[###### ] 25%[########### ] 50%[########### ] 50%[########### ] 50%[########### ] 50%[################ ] 75%[################ ] 75%[################ ] 75%[################ ] 75%[####################] 100%[####################] 100% — SUCCESS
Module 1: Upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[# ] 0%[####################] 100% — SUCCESS
Module 2: Upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[# ] 0%[####################] 100% — SUCCESS
Module 3: Upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[# ] 0%[####################] 100% — SUCCESS
Module 4: Upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[# ] 0%[####################] 100% — SUCCESS
Module 5: Upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[# ] 0%[####################] 100% — SUCCESS
Module 1: Waiting for module online.
— SUCCESS
Notifying services about the switchover.
[####################] 100% — SUCCESS
“Switching over onto standby”.
At this point, the supervisor that was the secondary (module 2 in my example) has reload and come up with the new code. This triggers the primary to initiate a Stateful Switch Over (SSO) to the new code running in the control plane. Meanwhile, data is still traversing the switch with no impact.
Since our SSH session was disconnected during the SSO (telnet isn’t SSO aware), we need to re-establish the session and issue a command to continue monitoring the upgrade.
You start monitor the installation process using the following command
# show install all status
Module 1:
Waiting for module online.– SUCCESS
Module 3:
Non-disruptive upgrading.– SUCCESS
Module 4:
Non-disruptive upgrading.– SUCCESS
Module 5:
Non-disruptive upgrading.– SUCCESS
FEX image downloading in progress.– SUCCESS
Module 101:
Non-disruptive upgrading.– SUCCESS
Module 102:
Non-disruptive upgrading.– SUCCESS
Module 103:
Non-disruptive upgrading.– SUCCESS
Module 104:
Non-disruptive upgrading.– SUCCESS
Module 105:
Non-disruptive upgrading.– SUCCESS
Module 106:
Non-disruptive upgrading.– SUCCESS
Module 111:
Non-disruptive upgrading.– SUCCESS
Module 121:
Non-disruptive upgrading.– SUCCESS
Install has been successful.
Now you can check the installed version using the following command
N7K_1# sh ver
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_series_home.html
Copyright (c) 2002-2014, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
Software
BIOS: version 2.12.0
kickstart: version 6.2(10)
system: version 6.2(10)
BIOS compile time: 05/29/2013
kickstart image file is: bootflash:///n7000-s2-kickstart.6.2.10.bin
kickstart compile time: 10/9/2014 12:00:00 [10/17/2014 04:45:38]
system image file is: bootflash:///n7000-s2-dk9.6.2.10.bin
system compile time: 10/9/2014 12:00:00 [10/17/2014 06:39:33]
Hardware
cisco Nexus7000 C7009 (9 Slot) Chassis (“Supervisor Module-2”)
Intel(R) Xeon(R) CPU with 12224948 kB of memory.
Processor Board ID xxxxxxx
Device name: N7K_1
bootflash: 2007040 kB
slot0: 0 kB (expansion flash)
Kernel uptime is 0 day(s), 0 hour(s), 47 minute(s), 45 second(s)
Last reset
Reason: Unknown
System version: 6.2(8)
Service:
plugin
Core Plugin, Ethernet Plugin
How long would you say it took for the 7K to perform the ISSU upgrade once you had your issues resolved?
It took 30 min or so