Skip to main content

Migrating to 2.2.0

Migration Guide​

Here you can find the needed steps to upgrade your current Witboost installation to v2.2.0. Please refer to the release notes for a detailed list of changes and improvements. This guide is intended for users who are currently running Witboost version 2.1.0 or earlier and wish to upgrade to version 2.2.0.

Follow these steps to migrate from the previous version to Witboost version 2.2.0:

Backup Your Data​

Before starting the migration, ensure you have a complete backup of your database and configuration files.

Update Dependencies​

Ensure all the dependencies in the Helm files are updated to their compatible versions for 2.2.0. Refer to the published Helm chart for details.

Upgrade Steps​

Core​

Changes

  • Environment priorities are not stored anymore only in the marketplace database, but in the global configuration. Also,there is no more the need to configure the environments manually in multiple places.

  • The new data contracts page is active by default in app-config.yaml

  • Introduced the startup actions, which will perform a series of actions once when Witboost is first installed or first upgraded to this version. This includes the following actions that will now be executed unless overridden:

    • Default Practice Shaper registration: Starting from this version by default, newly installations of Witboost and existing installations with no taxonomies already registered, will come pre-packaged with the Data Mesh taxonomy automatically registered using the data-mesh-taxonomy.yaml catalog info. If this taxonomy is not desired to be registered, check the Migration notes below to disable this behaviour.
    • Default Access Control Request template registration: Starting from this version by default, newly installations of Witboost and existing installations with no Access Control Request templates already registered, will come pre-packaged with the Witboost Starter Kit Access Control Request templates automatically registered. If these Access Control Request templates are not desired to be registered, check the Migration notes below to disable this behaviour.
    • Default location registration: Witboost is now able to register a set of locations to the catalog at startup time. By default, no locations are registered. Check the configuration documentation on how to add your default locations.

  • Removed support for gridInfo and sideInfo in dataProductPage and outputPortPage. If you have had previously configured gridInfo or sideInfo for dataProductPage or outputPortPage in your app-config.yaml and wish to retain that information, you can move those fields directly to the corresponding CustomView and place them wherever you like them to be. You can find the documentation link of the custom views here.

    This is the list of fields whose support has been removed:

    • mesh.marketplace.ui.outputPortPage.gridInfo
    • mesh.marketplace.ui.outputPortPage.sideInfo
    • mesh.marketplace.ui.dataProductPage.gridInfo
    • mesh.marketplace.ui.dataProductPage.sideInfo

Breaking Changes

  • HELM License configurations have been renamed. As such, they must be renamed in the secret store used.

    • licenseFile becomes CORE_LICENSE_FILE
    • signature becomes CORE_SIGNATURE

  • ACLs are now strictly associated with consumable components or subcomponents. This change affects you only if you directly query the acls or acl_requests tables in the marketplace database.

    • Previous Behavior: Rows in these tables could reference a non-consumable component if it was parent of some consumable subcomponents.

    • New Behavior: Rows now always reference a consumable component or subcomponent.

    • Migration Process: Automatically applied when upgrading to this Witboost release. Permissions (rows in the acls table) previously tied to a parent component are now explicitly assigned to its consumable subcomponents.

      Example:

      Component A (consumable: false)

      • subcomponent A.1 (consumable: true)
      • subcomponent A.2 (consumable: true)
      • subcomponent A.3 (consumable: false)

      ACLs before the migration :

      • Component A → [ user1, user2, group1, group2]
      • Subcomponent A.1 → [ ]
      • Subcomponent A.2 → [ ]
      • Subcomponent A.3 → [ ]

      ACLs after the migration:

      • Component A → [ ]
      • Subcomponent A.1 → [ user1, user2, group1, group2]
      • Subcomponent A.2 → [ user1, user2, group1, group2]
      • Subcomponent A.3 → [ ]

  • Changed the default authorization environment value auth.environment from development to the more simple default value. Instances where authorization is defined, development is used as a key under the provider configuration (that is, the object auth.providers.<provider_name>.development exists) and auth.environment is not overridden, authorization will break; check Migration notes below on how to fix it.

Migration

  • If you want to keep the environment priorities, set them under mesh.builder.environments like this:

    environments:
      - name: production
        priority: 0
      - name: development
        priority: 1

    otherwise leave it as it was before, but then priorities will be inferred by the order in which the environments are written (the first one in the list will be taken as the one with highest priority)

    environments:
      - production
      - development

  • To override the startup actions:

    • Default Practice Shaper registering: To block the Witboost installation/upgrade to automatically register the Data Mesh taxonomy on instances where no taxonomies are already registered, the configuration startupActions.catalog.practiceShaperLocations should be set to []. This change should be done before upgrading, as this action is performed only once and doesn't automatically remove locations afterward. In case the taxonomy is registered, to manually remove it go to the Software Catalog tab and filtering by "Location" find the datamesh-taxonomy-locations entity, then unregister it.
    • Default Access Control Request templates registering: To block the Witboost installation/upgrade to automatically register the default Access Control Request templates on instances where no Access Control Request templates are already registered, the configuration startupActions.catalog.accessRequestLocations should be set to []. This change should be done before upgrading, as this action is performed only once and doesn't automatically remove locations afterward. In case the Access Control Request templates are registered, to manually remove them go to the Software Catalog tab and filtering by "Access Control Request Templates" find the Access Request Template and Revoke Request Template entities, then unregister them.

  • On instances where authorization is defined, development is used as a key under the provider configuration (that is, the object auth.providers.<provider_name>.development exists) and auth.environment is not overridden, either:

    • Override auth.environment setting it to development
    • Set the environment name key <environment> on auth.providers.<provider_name>.<environment> to default. Example:

    Old:

    auth:
      providers:
        microsoft:
          development:
            clientId: #CLIENT_ID
            clientSecret: #CLIENT_SECRET
            tenantId: #TENANT_ID

    New:

    auth:
      providers:
        microsoft:
          default:
            clientId: #CLIENT_ID
            clientSecret: #CLIENT_SECRET
            tenantId: #TENANT_ID

Mock Specific Provisioner​

Breaking changes

HELM Helm chart now references the environment variable CORE_BACKEND_PLUGIN_AUTH_SECRET instead of BACKEND_PLUGIN_AUTH_SECRET.

Coordinator​

Breaking Changes

  • HELM In this release the .Values.extraEnv variable of the helm chart becomes an array. The configuration changes from this

    extraEnv: |-
       - name: SQL_DBNAME
         value: "provisioning_coordinator"

    To this

    extraEnv:
    - name: SQL_DBNAME
      value: "provisioning_coordinator"

  • The secrets used by the provisioning coordinator to access the databases have now different names

    old namenew namefallback
    SQL_DBNAMECOORDINATOR_DB_DBNAME
    SQL_SCHEMACOORDINATOR_DB_SCHEMA
    SQL_PASSWORDCOORDINATOR_DB_PASSWORDWITBOOST_DB_PASSWORD
    SQL_USERNAMECOORDINATOR_DB_USERNAMEWITBOOST_DB_USERNAME
    SQL_HOSTCOORDINATOR_DB_HOSTWITBOOST_DB_HOST
    SQL_PORTCOORDINATOR_DB_PORTWITBOOST_DB_PORT

    Important COORDINATOR_DB_DBNAME and COORDINATOR_DB_SCHEMA take the default values of "provisioning_coordinator" and "cgp". You can avoid specifying them if you already use those values.

    Suggestion If you don't plan to split coordinator and cgp databases, using the fallback env variables is a better choice to reduce the amount of secrets in the secret store.

  • Marketplace Proxy configuration has been officially deprecated by Marketplace-Compliant Services configurations.

  • Several mapping of properties and environment variables have been removed from the default configuration. Now they must be explicitly specified in the values.yaml. You can refer to the umbrella chart values.yaml. If you are setting those variables in your secret store, make sure they are also mapped to the corresponding property in your values.yaml.

    Env var
    EVENTS_BACKEND_BASE_URL
    NOTIFICATION_URL
    NOTIFICATION_SECRET
    CLIENT_CERTIFICATE_*
    SERVER_CERTIFICATE_*
    PLATFORM_SERVICES

    For example, if you are setting the NOTIFICATION_SECRET in your secret store, make sure your values.yaml maps that variable its corresponding property:

    notification = {
     ...
     secret = ${NOTIFICATION_SECRET}
    }

Migration

  • With the deprecation of the Marketplace Proxy configuration, the properties defined under provisioning-coordinator.depends-on.marketplace-proxy are no longer used. Those properties must now be defined under the new Marketplace-Compliant Services configurations. Example. 
    marketplace-compliant-services = {
       ...
     services = [
       {
         id = "witboost-marketplace"
         name = "witboost Marketplace"
         base-url = "http://<ui-service>/api/marketplace"
         is-enabled = true
         # whether a provisioning plan should fail on a failed http call to the witboost marketplace
         is-blocking = true
       },
       ...
    ]

CGP​

Breaking Changes

  • CGP endpoint GET /v1/computational-governance/evaluation-reports has been dismissed

  • Policy violations and metric results endpoints have been dismissed

    • /v1/computational-governance/resources/{resource}/policy-violations
    • /v1/computational-governance/resources/policy-violations/count
    • /v1/computational-governance/resources/{resource}/policy-violations
    • /v1/computational-governance/resources/{resource}/metric-results

  • The secrets used by the provisioning coordinator to access the databases have now different names

    old namenew namefallback
    SQL_DBNAMECGP_DB_DBNAME
    SQL_SCHEMACGP_DB_SCHEMA
    SQL_PASSWORDCGP_DB_PASSWORDWITBOOST_DB_PASSWORD
    SQL_USERNAMECGP_DB_USERNAMEWITBOOST_DB_USERNAME
    CGP_SQL_HOSTCGP_DB_HOSTWITBOOST_DB_HOST
    SQL_PORTCGP_DB_PORTWITBOOST_DB_PORT

    Important SQL_DBNAME and SQL_SCHEMA take the default values of "provisioning_coordinator" and "coordinator". You can avoid specifying them if you already use those values.

    Suggestion If you don't plan to split coordinator and cgp databases, using the fallback env variables is a better choice to reduce the amount of secrets in the secret store.

Migration

Important! This version includes a critical database migration that may take a few minutes to complete. Don't worry if the Coordinator/CGP instance takes some time to start on the first launch after the upgrade, and adjust any related timeouts accordingly.

Make sure to backup the CGP database before the migration.

If you prefer to opt for a manual database migration please reach out the Witboost development team for assistance.

Hasura​

Changes

  • Hasura Helm chart does not require you to set HASURA_GRAPHQL_DATABASE_URL and HASURA_GRAPHQL_METADATA_DATABASE_URL anymore. They are automatically computed by combining the new WITBOOST_DB_* variables and the .Values.dbs values, which default to the marketplace database.

  • The Hasura Helm values now allows you to define on .Values.dbs.sources: a list of sources that the service can track. For each source an environment variable HASURA_GRAPHQL_<SOURCE_NAME>_DATABASE_URL is created storing the JDBC connection, to allow defining source connections using only environment variables. We provide on the Umbrella chart a set of default configurations that define the sources for tracking Provisioning Coordinator, Computational Governance Platform (CGP), and Marketplace schemas. The source configuration values will be stored as part of the witboost-hasura-outputs ConfigMap, rendering them available to external services to automatize the tracking process (read further below).

    The new configuration has the following shape:

    dbs:
      ...
      sources:
        - name: provisioning_coordinator
          db: provisioning_coordinator
        ...

  • HASURA_GRAPHQL_DATABASE_URL environment variable is no longer needed or used, and it should not be configured pointing to a source present on the sources list.

Migration

If you want to get rid of HASURA_GRAPHQL_DATABASE_URL and HASURA_GRAPHQL_METADATA_DATABASE_URL, make sure WITBOOST_DB_* variables are defined in your secret store, and that .Values.dbs match your actual configuration.

It is possible to override the definition of the HASURA_GRAPHQL_<SOURCE_NAME>_DATABASE_URL by having them defined on the secret store, as this takes precedence over the .Values.dbs configurations. These environment variables are mandatory if the Hasura Auto Tracking tool is in use (read further below).

Breaking Changes

Hasura ingress does not default anymore to nginx.

Migration

You must either specify ingress.className or annotate your ingress with kubernetes.io/ingress.class.

Hasura Auto Tracking Tool​

Changes

Introduces the new Hasura Auto Tracking tool, an internal tool to automatically align a Hasura instance with database migrations, removing the need to manually track metadata after each version upgrade. This tool uses a set of configuration values to configure which sources to track and which schemas to track/exclude inside said sources. The tool uses the outputs of our Hasura helm chart to register and track the configured sources.

Breaking Changes

  • By including the Hasura Auto Tracking tool on the platform lifecycle, Hasura metadata shouldn't be manually modified on any situation.

Migration

If you want to start allowing the Hasura Auto Tracking tool to completely handle Hasura's metadata, follow these steps:

  1. Before version upgrade, clear Hasura metadata. If possible, clear all Hasura metadata by going on the UI Settings > Reset metadata. After updating, the configured sources and schemas will be re-registered and re-tracked. Between the manual clear of the metadata and the installation of the new Witboost version including this tool's version, there will be a Witboost degradation interval, where the Marketplace and the Computational Governance Platform won't be available.

    If clearing the whole metadata is not possible or desired, it is still necessary to remove the sources that the tool is configured to track in order to avoid source duplication and conflicts, while the tool is performing the tracking of the instructed tables. This sources will be recreated when the tool is launched.

  2. We provide on the Umbrella chart a set of default configurations that track the Provisioning Coordinator, CGP, and Marketplace schemas. On existing installations, if you require the tracking of additional sources, or to define a custom set of allow/exclude schemas, it is possible to use the .Values.configOverride values on the auto tracking tool Helm values to redefine the application configuration file. The configuration has the following shape:

    configOverride:
     sources:
       - name: sourceName
         allowSchemas: [ "schemaName" ]
         excludeSchemas: [ "schemaName" ]
       - name: marketplace
       - name: provisioning_coordinator
         allowSchemas: [ "cgp" ]
    • .Values.configOverride overrides the whole configuration file, so be sure to add as well the Provisioning Coordinator, CGP, and Marketplace sources. After overriding the values, since this tool uses a generated output by our Hasura service, be sure to also modify the hasura .Values.dbs.sources field to include all new sources, pointing to the correct db and with matching source names to the ones defined on the tool configuration. When all the configuration is correctly set on both sides, proceed with the version upgrade.

For more information on the tool, check the internal repository.

Note: After migrating to Witboost version v2.2.0, Auto Tracking may not be enabled by default. If you intend to enable this feature in v2.2.0 or any version release after it then please ensure the following before proceeding:

  • Ensure that all steps outlined in the v2.2.0 migration notes are properly followed during the setup of Hasura Auto Tracking.
  • Verify that the Hasura ConfigMap witboost-hasura-outputs is correctly updated. Failure to do so may prevent Hasura Auto Tracking from locating the required HASURA_GRAPHQL_<SOURCE_DB_NAME>_DATABASE_URL environment variable, potentially resulting in tracking failures.