📖 Navigation: User Guide | Release Notes | https://pantheon.tech/
OS upgrade
Initiation
The OS Upgrade section allows users to upgrade the device's operating system.
This is where users can access OS upgrade page through the UI:
This is how users can initiate device OS upgrade:
Prerequisites
Operation system
The only supported operation system to upgrade devices is a SONiC.
The newer version of the OS should be supported by the current SandWork version.
Addition of any other OS versions to a running instance of the SandWork is not supported.
Addition of OS versions to the SandWork should be done in the service-controller configuration file.
Device
The device should not change its profile after the OS upgrade.
Devices requested to OS upgrade may be either in MANAGED or STAGING room.
Devices requested to OS upgrade exceptionally may be in OS_INSTALL_FAILURE stage: to re-try the OS upgrade on the previously failed device.
Note: the upgrade of device's OS outside the process described in this document is not supported and may lead to potential errors.
OS upgrade input parameters
OS upgrade UI wizard consist of the following views:
View #1. Select Devices
This view consist of two tables: Available Devices and Selected Devices.
To Select which devices should be upgraded, user must choose them from Available Devices and press “>>“ (Select) button.
In case of any mistake happened during selection, user may Unselect chosen Selected devices and press “<<“ (Unselect) button.
To move further click Continue.
View #2. Configure Upgrade
On this view user have to choose only the OS version to which the devices will be upgraded, others configurations are optional.
Optionally user may define advanced configurations which will be used during the OS upgrade process:
Configurations based on labels management
Perform Warmboot on Devices with Labels. If the selected device has at least one of the selected labels, after installing of the OS image, instead of just reboot the device, it will be warm-rebooted;
Perform Traffic Drain on Devices with Labels. If the selected device has at least one of the selected labels, the traffic through it will be drained before the OS image installation;
Perform Interface Shutdown on Devices with Labels. If the selected device has at least one of the selected labels, its' port interfaces, which has at least one of the selected in Perform Interface Shutdown on Interfaces with Labels labels, will be shut-downed before the OS image installation;
Skip Healthcheck Validation. If this config is enabled, during the OS upgrade process, the prelimitary validation and post validation checks of device’s cables, BGP neighbors, and MCLAG status won’t be executed;
Use optional traffic drain values. If this config is enabled, user could define how traffic drain will be executed on the devices customizing:
Traffic drain max-med value;
Traffic drain timeout value (ms);
Use optional progress log values. If this config is enabled, user could define the interval between audit logging of the OS upgrade process customizing:
Progress log Interval (s);
Use optional OS Image Installation values. If this config is enabled, user could define the OS image install and reboot timings:
OS image installation timings:
Image installation time (s);
Poll interval (s);
Poll attempts;
Reboot timings:
Device reboot time (s);
Poll interval (s);
Poll attempts;
Warmboot timings
Device warmboot time (s);
Poll interval (s);
Poll attempts.
Once all required inputs in the UI wizard are defined, the user can start the device OS upgrade by clicking the "Run Upgrade" button.
Validation of the devices before OS upgrade
To avoid most of the OS upgrade failures it is strongly recommended to run the validation of the devices before the actual upgrade. To do it, the user should select the devices expected to be upgraded, and click on the “Validate Devices” button.
In the opened dialog user may check “Skip Optional Validation Steps” (see which validation checks are optional) and start the validation by clicking “Yes”.
OS upgrade process
OS upgrade is a complicated workflow, so to simplify the tracking of the upgrade status, it is divided to the separate sequential steps described below.
Overall the OS upgrade process consists of following steps:
Preliminary validation (VALIDATION)
Preliminary validation consists of few validation checks :
Mandatory gNMI connection check with a device;
Mandatory SSH connection check with a device;
Optional cable check of a device;
Optional BGP neighbors check (comparison of the intended number of BGP peers with the number of active BGP peers);
Optional MCLAG status check (only for devices in MCLAG pair).
After successful pre-validation, the device will be moved to OS_INSTALL room, which make it behave similarly as in RMA stage - all service intents for this device can be stored to inventory, but no configuration would be pushed to the device outside the OS upgrade process.
Note: even with the disabled Skip Healthcheck Validation checkbox, the optional checks failures will produce only warnings instead of preventing of the execution of the OS upgrade. So the running of the validation and fix all of the founded issues before the actual OS upgrade execution is strongly recommended.
Copy running config to startup config (COPY_CONFIG_TO_STARTUP)
This gNOI operation copies current running configuration of the device to its' startup configuration to ensure its synchronization. It is needed to make sure the current configuration remain consistent after restart.
Configuration backup (DEVICE_BACKUP)
Device backup is created in the form of a device configuration snapshot.
Note: if the OS upgrade is rerun for previously failed devices (which are in the OS_INSTALL staging room), the snapshot is not taken. It was taken before the original OS upgrade was run.
Device traffic drain (TRAFFIC_DRAIN)
If the user has selected some device labels in the "Perform Traffic Drain on Devices with Labels" field in the OS Upgrade Wizard, and the device to be upgraded is labeled with any of them, the device traffic will be drained during the OS upgrade. This operation will be accomplished by configuring the BGP max-med value for each VRF configured on the device.
This step could be customized using “Perform Traffic Drain on Devices with Labels” and “Use optional traffic drain values”, so the user may specify the values of max-med and traffic drain timeout in the OS Upgrade Wizard.
Ports shutdown (PORT_SHUTDOWN)
If the user has selected certain device labels in the "Perform Interface Shutdown on Devices with Labels" field in the OS Upgrade Wizard, and the device to be upgraded has any of those labels, its interfaces (ports) may be shut down based on the following condition:
If the user has also selected interface labels in the "Perform Interface Shutdown on Interfaces with Labels" field, then the interfaces on those labeled devices that have matching interface labels will be shut down during the OS upgrade.
Note: the management interfaces won’t be shut downed regardless of whether they are labeled or not.
OS image installation (IMAGE_INSTALL)
This operation consists of two sequential procedures:
Image installation. During this process, the device will download the OS image by the link provided by the SandWork and trigger its' installation afterwards. The longest time SandWork will wait for gNOI install response could be customized using OS upgrade wizard by checking “Use optional OS image Installation values” and specifying “Image Installation Time (s)” value.
Check the installation status. Periodically, the SandWork will check the status of the installation. OS image installation counts as completed only after a success image installation status read from the device. To customize the status checking process, in the OS upgrade wizard user could provide specific “Poll interval (s)” - the time between poll trying to verify the install of the OS is finished. “Poll attempts” - the amount of polls to verify the install of the OS is finished. To use the new image, the device have to be rebooted (next step).
Note: if optional parameters won’t be provided, the values configured in SandWork’s YAML configs will be used as defaults. Current default values:
gNOI waiting time: 120 sec;
poll attempts: 20;
poll interval: 30 sec.
Rebooting of the device (REBOOT)
In general, the reboot process consists of few sequential procedures:
Reboot triggering. SandWork triggers SSH command on the device and wait some time. The waiting time could be customized by checking “Use optional OS image Installation values” and specifying “Device Reboot Time (s)” and/or “Device Warmboot Time (s)” values.
Verification. SandWork periodically pings the device to verify it is ready to work. To customize the periodic checks, in the OS upgrade wizard user could provide specific “Poll interval (s)” - the time between periodic polls, and “Poll attempts” - the amount of polls itself. The success of the reboot operation checked by:
gNMI connectinity check: verification if device already reachable by gNMI protocol (skipped in case of warmboot).
System status: verification if device’s system is ready (all service containers are up and ready).
System uptime: verification if device system’s up time was restarted.
Note: if any optional parameters won’t be provided, the values configured in SandWork’s YAML configs will be used as defaults. Current default values:
reboot time: 300 sec;
poll attempts: 20;
poll interval: 30 sec.
Warmboot
If the user has selected certain device labels in the "Perform Warmboot on Devices with Labels" field in the OS Upgrade Wizard, and the device to be upgraded has any of those labels, the device will be warmbooted instead of just reboot: all of its' services will be reloaded using the versions of the installed OS image.
Note: although SandWork support OS upgrade warmboot option on all SONiC OS versions, it is not guaranteed to work correctly on all devices. The functionality depends on the OS version to install, currently installed OS version, the device vendor, the device model, etc. Before enabling the warmboot option, please, verify that your current device setup supports it.
Checking status of installation (STATUS_CHECK)
On this step, the SandWork verifies if the running version of the OS image is the requested one.
Enabling shutdown ports (PORT_ENABLE)
SandWork pushes the intended ports admin status (UP/DOWN) stored in the Inventory Service on the device.
Post-validation (POST_VALIDATION)
Validation and health checks are executed to verify whether the OS upgrade was successful on a device or not. It is consistent with the preliminary validation (VALIDATION).
Device traffic enabling (TRAFFIC_ENABLE)
Removes BGP max-med configuration from each VRF on a device configured during traffic drain step.
Clean-up (CLEANUP)
On this step, the OS upgrade of the device counts as successfully completed. The clean-up consists of the sequential steps:
Move the device to the stage before the OS upgrade. This means the device won’t be moved to OS_INSTALL_FAILURE stage anymore, even in case of any failure on this step;
Clear device OS upgrade status tracking. This means that if the user runs the OS upgrade once again, the SandWork will execute as it did the first time, without skipping any operations already performed;
Run reconciliation and move the device from the OS_INSTALL stage based on following rules:
If the reconciliation succeeds, the OS upgrade workflow will be reported as successful and the device will be moved to the original stage where it was before the OS upgrade;
If the reconciliation fails, the OS upgrade workflow will be reported as failed and the device will be moved to a stage based on its original stage before the OS upgrade:
Device’s original stage | Device’s stage in case of failed reconciliation |
|---|---|
CONFIG_PENDING | CONFIG_PENDING |
CONFIG_READY | CONFIG_PENDING |
MANAGED | MANAGED_CONFLICT |
MANAGED_CONFLICT | MANAGED_CONFLICT |
Troubleshooting
In case the process of upgrade has failed, the failed devices will be moved to OS_INSTALL_FAILURE stage. To resolve it, the user can do the following:
try to rerun the OS upgrade on the specific device;
if re-run didn’t helped - view which steps of the device OS upgrade process were executed successfully and which failed;
download the device configuration snapshot to see backed-up configuration from before the upgrade;
manually troubleshoot the device through means outside the SandWork system and possibly fix the upgrade;
within the SandWork system - move the device to the certain device stages:
CONFIG_PENDING stage - traffic is disabled during transition;
RMA stage - if the device is faulty, it can be marked for a replacement; transition will attempt to disable traffic on this device;
MANAGED stage - this transition is possible only if warmboot option was used and the device was previously in MANAGED stage; traffic needs to be enabled on such device and reconciliation run - if reconciliation fails, the device is moved to the MANAGED_CONFLICT device stage instead.
When moving a device from the OS_INSTALL_FAILURE device stage to any other stage, the current OS version is read from the device and is set for this device (in order to use the correct device driver when communicating with this device).
If this version is not supported, failure is thrown and the device is not moved from OS install failure device stage.
To select which remedy option to apply, the user needs to click on the wrench icon next to the device.
When the upgrade process encounters network issues, users can click the Rerun button to resume the upgrade from the point where it failed. If a device begins to malfunction, the user can initiate the RMA process to replace the device.
OS upgrade logging
In order to check the detailed OS upgrade workflow execution live or of the past runs, the user could see the logs.
Filtering logs by workflow id
Logs can be filtered using workflow id to show only logs related to certain workflow. There are several ways to filter logs using workflow id:
select workflow id from provided list
copy workflow id from workflow details and pass it into filter
mouse-over workflow id property and press “Filter for value”
Types of logs based on log level
In order to track progress of OS upgrade, it is recommended to enable INFO and ERROR log level.
Log level INFO
By enabling log level INFO, progress of OS upgrade workflow can be filtered. It includes brief summary after each OS upgrade step was finished and periodic progress log with number of devices currently in progress on each step.
Customize logging interval
By default the SandWork logs OS upgrade progress every minute, but user could also customize the logging interval in the OS upgrade wizard providing optional “Progress log interval (s)” value.
Log level ERROR
By enabling log level ERROR, OS upgrade failures will be logged for each failed device. It consists of device name and error message describing the cause of failure.