Release notes - SandWork - 2.0.4

Bug fixes

• Some switchport mappings not configured on device

• Improve performance of DC plan import (100+ pods)

• [SD] - [Sandwork2.0.3 patch1] Delete of Network segments is allowed when there are dependent switchport mappings

• Missing error notification after trying to delete NS when there are dependent switchport mappings

• [UI] Reconciliation Swithcport mappings, list not comprared correctly

Release notes - SandWork - 2.0.3-patch1

Bug fixes

• [SD] Error- Sync with device failed with status code 422

  • Configured session affinity between Service Controller Worker and Device Controller to ensure requests in single transactions are always sent to the same Pod

Release notes - SandWork - 2.0.3

Bug fixes

• [SD] Sync with device fails

• [SD] Sandwork2.0.2: Reconciliation is wrongly showing a SwitchPort Mapping for a VLAN

• [SD] Sandwork2.0.2: "Data loading failed" error when pushing configuration from the reconciliation page

• SwitchPort Mappings View - Loading mappings takes too long and fails

• [UI] Reconciliation conflicts are sometimes duplicated

• Reconciliation - Autostate conflict not showing

• Customer feedback - Every check fails for a periodic connection check

• Brownfield import - PortChannels are not registered for other intents in the same file

• Workflow queue - "My workflows" are not shown

• [UC] Workflow timed out waiting for a device controller to respond

• [UI] Finish notification of a workflow does not appear

• [UI] Update UI after BE change in Switchport Mappings View

• API api/inventory/network-segments takes too long to finish

• DC Plan Import for a large DC (5k devices) takes too long and times out

• DC Plan Import takes 3 hours in Ashburn

• Missing PortChannels in SwitchPort Mappings

Improvements

• Reconciliation outputs "Node has already been installed"

• Cablecheck - Get LLDP data in parallel

• Inventory File Import - Improve validation performance

• Brownfield Import - Add device identifiers to error messages

• Profile and speedup the Inventory File Import

• Implement redirection logic for an old version of portal

Release notes - SandWork - 2.0.2-hotfix1

Bug fixes

• Sync with device operation failed with node

  • Modified affected SONiC YANG models to allow vni-id set to 0 (original range was 1..16777215)

Release notes - SandWork - 2.0.2

Bug fixes

• [SD] - sandwork2.0: Cannot delete switchport mapping when the mapping for both mclag devices is selected for deletion

• [SD] - sandwork2.0: Reconcilation is showing wrong configuration mismatch

• [SD] - sandwork2.0: Internal exception while importing brownfield file

• [UI] Reconciliation - Switchport mappings - cannot properly show mismatches between intent and actual

Improvements

• [SD] - Sandwork2.0: "no ipv6 nd suppress-ra" is mandatory. Otherwise, RA packets will not be sent out/in.

Known limitations

• [SD] - Device RMA Process

• [SD] - Device snapshot push

Workaround:
For Ethernet port admin status reconciliation, recommended workaround for version 2.0.0:
If ports on device have admin status set to UP, force transition to ONBOARDED stage and repeat transition to CONFIG_READY stage. This will trigger interface admin status change and should allow further device promotion.

Release notes - SandWork - 2.0.1

Bug fixes

Bug

• [SD] - Sandwork2.0: when 184 devices are imported 108 are in disconnected state

• [SD] - Sandwork2.0: After uploading 346 devices using dcplan file and inventory, cannot import new dcplan file

• [SD] - Sandwork2.0: The Network * segment is not deleted on the Switch, but UI shows success

• Traffic reconciliation on non-leaf devices Both mlag devices are not locked during portchannel creation

Story

• Analyze and fix vulnerabilities found in maven projects

• Make gRPC maxMetadataSize configurable

• Wait for device installation to finish during inventory file import worflow

• [DevOps] Template new configuration parameters of Device Controller

• Remove sequential connection check when uninstalling devices

Release notes - SandWork - 2.0.0

New use cases included in this release

Compatibility with SONiC 4.1.1a

This release of Sandwork is fully compatible with SONiC 4.1.1a (tested with latest RC4)

Port breakout configuration on device

It is now possible to configure port breakouts through the Sandwork controller.

Bug fixes

• [UI] Warning for Network Segment creation / removal - use host names of the devices

• [UI] Device details - Configuration snapshots - Description column name is broken

• [UI] push-device-configuration-snapshot - user could be informed about details of the snapshot

• [UI] Fix Date filters, make them human friendly

• [UI] Loopback Reconciliation Visualization Table

• [UI] Add scrolling to Reconciliation results dialog

• [UI] Redundant exclamation mark present in port settings tab

• [UI] Reconciliation takes long time to render when lot of mismatches are reported

• [UI] Canvas sizes change randomly

• [UI] Dashboard | Doughnut gNMI Status of Devices

• [UI] Dashboard API response error

• [UI] No error messages are displayed | UI Dashboard

• [UI] Portal is sending workflow queue requests on the DC selection page

• [UI] UI stops polling for the workflow list API

• [UI] Portchannels: Missing Validation: port is already a member or a portchannel

• [UI] Portchannels: Missing validation: VLAN attached to an interface

• [UI] gNMI Check is run during NS creation when push to bLeaf is unchecked

• [UI] Fix filters on master

• [UI] The device is not in MANAGED stage - align a message to be a standard one

• [UI] Reconciliation doesn't specify a conflict for M&O flags

• [UI] Staging page - Throwing back from new screen of stage transition workflow details (Config_pending to Config_ready) to datacenters screen

• [UI] Workflow timestamp - cannot differentiate AM/PM

• [UI] Default M&O flags are not set correctly in the UI

• [UI] Topology view - Cannot interact with devices in the top part of the view

• [UI] Topology view - device details rendered outside of the screen for device at the bottom

• [UI] Add verification to portchannel members

• [UI] PortGroup validation for Portchannels in the selection

• [UI] Portchannel Creation Portchannel Identifier input validation

• Improvement - Errors marked as Info

• [DevOps] Kibana index pattern not initialized after deployment

• Device controller fails to operate with devices after system upgrade

• Service controller is unable to update supported SONiC version

• [SONiC-78731] SONiC - portchannel uses min-links = 0, when min-links is not explicity set

• [SONiC-78532] Management interface name has changed in SONiC 4.1.1

• [SONiC] DEVICE_CONFIGURATION - Unimplemented CopyConfig for service gnoi.SONiC.SONiCService

• [SONiC-70784] Unable to delete vtep from one vlan and add it to another one in single tx

• Cablecheck shows missing neighbor that is in BLUEPRINT stage

• [API] Push-device-configuration-snapshot - NullPointerException - io.grpc.Context

• [API] PortSettings - Validation for device stage

• [API] MCLAG pair - add-mapping-to-switchport - leaf2 - NullPointerException - updateDevicesStageAsResult

• [API] Filter out devices for graphs

• [DevOps] Job creating kibana index pattern doesn't fail on script failure

• Null pointer when changing speed on port in a PortChannel

• [API] Force device change - Name does not resolve

• [API] Switchport mapping - mclag devices - configuration errors - uniconfig-error - ipv6 on interface and vxlan tunnel modification

• Relation not created between port group and port after DC plan import

• Remove license headers from liquibase migration scripts

• Reconciliation of clean device fails due to errors with underlying VLANs configuration

• [UC] MCLAG pair - RPC issues - add-mapping-to-switchport

• NPE During reconciliation

• Add /api prefix to REST API health endpoint

• Removing multiple switchport mappings removes only 1st item on device

• Device controller produces an error when creating portchannel "Offset: '44': Reason: Path must start with '/'"

• It is not possible to delete portchannel from the device

• Invalid error when attempting to import dc-plan-file with

• [API] device-management-connectivity-check - improve message in case connectivity check faces device installation

• [API] Push-device-configuration-snapshot - DEADLINE_EXCEEDED while pushing config snapshot to running/startup configuration for device

Release notes - SandWork - 2.0.0 - dev5

New use cases included in this release

Periodic device configuration snapshot/backup

Goal of this use-case is to periodically store a configuration snapshot of all devices which are in MANAGED stage. These snapshots will be marked as system snapshots to be differentiated from manually created snapshots. Rotation mechanism for the system snapshots will be implemented. It means that only configured number of snapshots will be stored in Inventory Service and the oldest snapshot(s) will be deleted when a new snapshot is created.

IPv6 RA M&O flags

Add support for IPv6 router-advertisement managedConfig and otherConfig flags for IPv6 Network Segments. The flags will be defined when a new Network Segment is created and will be configured on Leaf devices for each VLAN mapped to the Network Segment.

Audit log improvements

Goal of this use case is to improve the level of detail in the audit log records and increase granularity, in which they may be searched.

LLDP - show cablecheck result in topology view

Goal of this use-case is to provide graphical representation of cablecheck results using the topology view component.

UI Dashboard

Goal of this use-case is to provide user with the UI Dashboard that will contain a high level overview of the selected data center with the ability to drill down into details of each POD in the selected data center.

Improvements

• [UI] ReinstallDevices workflow submit button

• [UI] Add routing parameter when opening Workflows Table and Workflow Details page

• [UI] Introduce actions menu to the fabric topology for selection and triggering Cablecheck workflow

• [UI] Update Stage Transition to new reconciliation

• [UI] Add option to filter out non-conflicting port mappings from output visualization

• [UI] Include autostate value on UI

• Read device connection state before DHCP relay update

• Read device connection state before submitting Network Segment workflows

• Read device connection state before submitting Switch Port mapping workflow

• Add more information to error messages from workflows

• Update data models of device ports adding hwPortName

• Validate hwPortNames and portNumbers

• Parallelize device connection check

• Set connection state of devices created by DC plan file import as UNDEFINED

• Add Port breakout configuration to brownfield file

• Update port settings workflow to be able to push settings to multiple devices

• Add coordinates of the DC to the DC Plan file

• Add auditLogs in device-configuration-snapshot-system-backup workflow

• Add auditLogs in device-management-connectivity-check workflow

Bug fixes

• [UI] Stage transition results showing cablecheck and reconciliation show entries for each device mutliple times

• [UI] Portal is sending workflow queue requests on the DC selection page

• [UI] After importing DC plan, the link in workflow queue does not refresh the staging view

• [UI] Delete Selected button doesn't allow to delete a mapped PortChannel

• [UI] Opening workflow details when workflow is in progress shows no inputs and throws errors in console

• [UI] Empty request Device Details

• [UI] Portchannels tab shows members as mismatch for missing device configuration item

• [UI] Monitor device connection - device-management-connectivity-state call accumulates and duplicates devices

• [UI] Suggestions - Device Not Connected warnings

• [UI] To perform device-management-connectivity-check for every device configuration transaction

• [UI] Start-rma dialog - inputs improvements

• [UI] Switchport Mapping - custom mapping invalid request

• [UI] DHCP Relay update dialog submits the changes and closes after opening

• [UI] push-device-configuration-snapshot - user should be informed about details of the snapshot

• [UI] Warning about disconnected devices shows incorrect information

• [UI] UI sends a request with incorrect port speed for port settings

• [UI] DC Map Grid Rendered only after window resize

• [UI] Show POD details - stage instead of Staging Room is shown.

• [UI] Device profile page requests list of profiles by DC name not namespace

• [UI] Reconciliation details view fails to render

• [API] Password Encoding - Can’t upload an inventory file when the device is in Managed stage

• Device SUSPENDED stage not updated during controller upgrade

• reinstall-devices workflow not available in conductor

• Device connectivity check not executed when device is not installed

• DC Plan import fails on when overriding device profile port speed

• Cablecheck results message should show expected neighbor and interface

• Device staging needs to run new reconciliation during stage transitions

• Reconciliation displays all mappings for VLAN if it has mismatches

• Include autostate value in reconciliation

• Switchport Mapping - NPE during mapping validation when using custom mapping

• Unable to change admin state for interface that is part of portchannel

• [API] Store-device-configuration-snapshot sets device_stage to BLUEPRINT

• [API] Exception thrown during connection check - incorrect workflow status and device connectionState

• [API] device-management-connectivity-check - improve message in case connectivity check faces a timeout

• [DevOps] maxNumberOfSnapshots - set default value to 5

• [API] No errors logged for non existing devices - device list provided

• NPE when importing a brownfield file with port breakouts

• Only compare autostate value for vlans not coming from dc import

• Topology view should contain the nearest neighbors as well as requested devices

• [DevOps] Add latitude and longitude properties to initial dc file in inventory service

• Cablecheck reports failure for interface not marked for cablecheck

• [API] device-configuration-snapshot/device - Returns 500 if device has system snapshots

• Add switchport mappings workflow with custom vlan identifier throws NPE

• Duplicate key Exception in portal backend

• WorkflowId search propertiy is missing

• NPE During reconciliation

Release notes - SandWork - 2.0.0 - dev4

New use cases included in this release

RMA

Add functionality to support an exchange of a physical device where the new device has the same profile. After the physical replacement it is needed to change a management interface MAC address of the device entity in Inventory Service and copy configuration snapshot onto the new device amending only the MAC address (editing the snapshot).

Reconciliation - push missing configuration to a device

Adds capability to push missing intents from the controller to the device. Does not include pulling configuration from device to the controller.

Passwords encoding

Purpose of this use-case is to secure device passwords. The passwords will be encrypted either as part of inventory file or as standalone device passwords before stored in database. Inventory service will always transfer passwords (or inventory files) in encrypted form. Services using the device passwords decrypted will need access to encryption keys.

Monitor device connection

Monitor state of a gNMI connection between Device Controller and all devices in DC. Periodic checks for particular device are started after an inventory file is imported (and the device is transitioned from BLUEPRINT to CONNECTED stage).
Device connection state is visible in UI where device information is displayed especially in a topology view.

Port Breakout UI

Implement UI for Port breakout configuration (currently only service intents)

Device details show intents

Add a view that presents intents related to a particular device

Improvements

• Uninstall all nodes existing in inventory before new DC file import

• Add filtering by deviceId for all-configured-vlans EP

• Add filtering by deviceId for network-segments EP

Bug fixes

• [UI] newPortSpeed property not sent in request body when performing port settings

• [UI] Reconciliation results show device ID instead of device name

• [UI] Cablecheck results show device IDs instead of device names

• [UI] Change icon to open Workflow Details in Workflow Notification

• [UI] Workflow details for promotion to CONFIG_READY stage don't show reconciliation results

• [UI] Workflow details for promotion to ONBOARDED stage don't show cablecheck results

• [UI] Toast notification regarding WORKFLOW COMPLETNESS does not appear sometimes (not always).

• [UI] Force staging is enabled when there are no devices selected

• [UI] Title discrepancy for response toasts

• [UI] Port Breakout workflow type is not displayed correctly

• [UI] User is not warned, when the device is not in managed stage

• [UI] User is not warned, when the device doesn't contain any breakout capable port

• [ZTP service API] - dhcp-config - 'hardware ethernet' parameter returns baseMac address instead of mac address

• [API] Add deviceName field, change the name of the device.

• Mapping of same VNI to different VLAN for mlag pair device fails at wrong stage

• Exception thrown in logs on API calls when namespace is not set

• Reconciliation returns a conflict for the bleaf devices

• No error returned when importing a brownfield file with an IPv4 NS with flags

• Assigned ports show total amount instead of amount of specific device

• Output is missing device name when a device failure occurs

• Add relayName to dhcp relay output

• NPE during reconciliation bleaves with NS imported via brownfield

• NPE when pushing switchport mapping configuration

• Changed DHCP relay doesn't affect the reconciliation

Release notes - SandWork - 2.0.0 - dev3

New use cases included in this release

Port group settings

New UI is introduced to allow configuration of speeds for the entire port group to simplify device configuration, especially in cases, when moving between speeds where all ports in a port group must be changed (25 Gbps → 10 Gbps).

Parallel workflow execution (Device Locking)

This release introduces an internal device locking mechanism in the controller, that allows parallel execution of workflows. The locking mechanism will ensure, that only one workflow configuring a particular device will run at single point of time and other configuration workflows for this device will wait until the lock is released.

Controller upgrade - persistent sessions

Starting with this release, the portal-backend instances share a persistent storage (PostgreSQL) to store active user sessions. This way the deployment no longer requires configuration of sticky sessions and allows actual load balancing of requests across all portal backend instances. This is also a precursor to deploying two versions of portal backend to support multiple releases of Sandwork controller deployed in different DCs.

Staging view - UI rework

The UI for the staging view was reworked based on internal and customer feedback.

Each tab lists devices (from selected Pod) in the respective stage and provides actions related to the stage.

It’s also possible to enforce transition to selected stage.

Improvements

• Speedup brownfield file import

• [UI] Add full screen button for Workflow details

• [UI] Improve usability of date range filter

• [UI] Show loading spinner for data fetching operations in Workflows table, Sidebar and Details

• [UI] Minimum info mode for Workflows sidebar

• [UI] Show notification that Workflow has been finished

• [UI] Add columns control for Workflows table

• [UI] Add placeholder for Inputs tab for Import Inventory file workflow

• [UI] Workflow Queue notification styles

Bug fixes

• Topology view overlaps

• Network segment creation - disable autostate on border leaves

• [UI] Topology view with Watford topology shows overlapping racks

• [UI] Convert datepicker time to UTC

• [UI] Workflow details dialog can't be closed on small screens due to title overlapping

• [UI] Fix outer scroll in Workflow Queue table

• [UI] Workflow queue dialog window - refresh button

• [UI] Typo on UI

• [UI] Hide management address if it is null

• [UI] Switchports mapping Configure Vni-Vlan mappings switch filter not working

• [UI] Switchport mapping | In case of untagged mapping conflict it is possible to choose merge configuration

• Filter uninstantiated ports in response from api/inventory/interfaces

• Hide Workflow inputs and Raw outputs contents for Workflows with file operations

• Switchport mapping | Check ports step doesn't shows configuration conflict if port already has existing mapping

• Add validation to switchport mapping dialog when there is a conflict in VLANs

• Zoom-in/out issues on for CableCheck and Reconciliation Workflow results tables

Deployment requirements

This release introduces new deployment dependencies - PostgreSQL databases for Portal Backend (to provide session persistence) and Service Controller Worker (to facilitate device locking).

It is recommended to create the Portal database inside the DB cluster utilized by RBAC (as it is a global deployment) and the service controller DB in the controller DB cluster (deployed per DC and already utilized for Inventory service, Device controller and Conductor).

The credentials and connection information are provided through respective value files. The configuration follows the same pattern as PostgreSQL configuration for other components.

Optionally the deployment contains a Kubernetes job that can create the DBs for you, but it needs to be enabled and configured with the admin credentials to the DB clusters.

Known issues

• During stage promotion the Cablecheck and Reconciliation results only show pass/fail without the actual details

• Reconciliation and Cablecheck results show Device IDs instead of Device Names

• Workflow output results are not ordered

Release notes - SandWork - 2.0.0 - dev2

This version of SandWork controller was developed for and tested with the 4.0.3 GA version of SONiC.

New use cases included in this release

Workflow queue

Goal of this use case is to provide information to the user about started, scheduled and completed workflows based on user's permissions.

Device staging

Goal of this use-case is to define device stages and transitions between them.

Stage transitions are always done in scope of workflow execution. Either initiated by user directly (i.e. "promote" or "demote" actions) or indirectly (i.e. failed configuration of device or reconciliation).

Bug fixes

• [UI] SwitchportMapping creation, Step 2 Select Ports broken

• [UI] Create Portchannel dialog not displaying port speeds correctly

• [UI] In dc-designer ports column is not filled

• [UI] Decommission Devices button is missing id and aria-label attribute

• [UI] Show spinner on Decomission Devices button when sending Intents check request

• [UI] Elements "flashes" when scrolling in list with paging

• Loopback Number validation of decimals