Skip to main content

Version 1.0.0

These are the release notes for the v1.0.0 release of witboost.

Please refer to the official documentation for a more in-depth overview of the released features.

Features​

Blueprints​

To simplify the creation of complex use cases where multiple components are needed for a Data Product to fulfill the complete scenario, we introduced the concept of Blueprints. Blueprints are a set of templates that are presented together to users, so they can create complex data products in a guided way.

Blueprints are collections of templates: they contain a main template ID that should refer to a template for a Data Product and a list of other template IDs. Like templates, they can be stored inside the Catalog, which is why they must be compliant with a well-defined structure.

As for templates, the blueprint definition describes the parameters that are rendered in the frontend part of the scaffolding wizard, and the templates bound to the entity.

Computational Governance Platform​

The Computational Governance Platform is a new tool available for platform team members to enforce rules and requirements that must be respected by the whole company.

With the introduction of the Computational Governance Platform, users can track the validation of any kind of resource inside WitBoost. The platform allows defining computational policies on various resources (Data Product, Dashboard, ML Projects, Tables), integrating external and third-party systems for a comprehensive view of governance processes and data.

This platform, fully integrated with WitBoost, allows you to:

  • Create, test, and evolve Computational Policies
  • Backtest a policy before its release to assess impacts and conformity
  • Disable policies or put them in different statuses to better handle their lifecycle
  • Have a grace period where a policy is only generating warnings to educate and inform users
  • Enforce resource compliance at deployment and runtime
  • Visualize statistics on policy usage and results
  • Create policies to check that no breaking changes are introduced in a new deployment
  • Extend the standard CUE policies with microservices developed by the platform team
  • Implement local policies and receive their results at the central level
  • Integrate policy results into the marketplace to increase transparency and trust among data users

Breaking Changes​

With the introduction of the Computational Governance Platform, data platform team members can now define rules checking that no contract is violated between a new version of a resource and an already deployed one.

A breaking change policy has the ability to compare the descriptor of the latest deployed version of the resource against the descriptor of the currently validating version, whether it is a test or deployment operation, to detect breaking changes and deviations between versions.

Breaking Change policies can be used to enforce contract violations among users: a failed breaking change policy will alert that even if the resource is correct, when released, it would not be usable anymore by consumers.

Change Management Helper​

The Change Management Helper is a new capability for notifying users when certain events occur in the platform. Here is a list of new notifications that will be generated automatically by WitBoost:

  • "New Policy": When the platform team publishes a new policy, all builder users are notified
  • "New Template": When the platform team releases a new template or a new version of an existing one, all builder users are notified
  • "New Domain": When a new domain is introduced in the community, all platform users are notified
  • "New DP version": When a team releases a new version of a Data Product, all platform consumers are notified
  • "New DP": When a brand new Data Product is released, all platform users are notified

Registering more than one component from one template​

It is now possible to publish to more than one repository starting from one single template.

This is achieved by creating a template with duplicated phases for extraction, publish and registration.

We introduced a new attribute input.sourcePath in the publish phase which is used to read from the folder specified in the input.targetPath fetch phase attribute.

To make this functionality work, those two variables must be an exact match, and the resulting template could be something like:

  steps:
    - id: templateOne
      name: Fetch Skeleton + Template
      action: fetch:template
      input:
        url: ./skeleton
        targetPath: '${{ parameters.rootDirectory }}'
        values:
          ...
          identifier: '${{ parameters.identifier }}One'
          destination: '${{ parameters.repoUrl | parseRepoUrl }}One'
          ...
    - id: publishOne
      name: Publish
      action: witboostMeshComponent:publish:gitlab
      input:
        allowedHosts: ['gitlab.com']
        description: 'This is ${{ parameters.name }}'
        repoUrl: '${{ parameters.repoUrl }}'
        rootDirectory: '${{ parameters.rootDirectory }}'
        dataproduct: '${{ parameters.dataproduct }}'

    - id: registerOne
      name: Register
      action: catalog:register
      input:
        repoContentsUrl: '${{ steps.publishOne.output.repoContentsUrl }}'
        catalogInfoPath: '/${{ parameters.rootDirectory }}/catalog-info.yaml'

    - id: templateTwo
      name: Fetch Skeleton + Template
      action: fetch:template
      input:
        url: ./skeleton
        targetPath: '${{ parameters.rootDirectory }}/two'
        values:
          ...
          identifier: '${{ parameters.identifier }}Two'
          destination: '${{ parameters.repoUrl | parseRepoUrl }}Two'
          ...
    - id: publishTwo
      name: Publish
      action: witboostMeshComponent:publish:gitlab
      input:
        allowedHosts: ['gitlab.com']
        description: 'This is ${{ parameters.name }}'
        repoUrl: '${{ parameters.repoUrl }}Two'
        rootDirectory: '${{ parameters.rootDirectory }}'
        dataproduct: '${{ parameters.dataproduct }}'
        sourcePath: 'two'
    - id: registerTwo
      name: Register
      action: catalog:register
      input:
        repoContentsUrl: '${{ steps.publishTwo.output.repoContentsUrl }}'
        catalogInfoPath: '/${{ parameters.rootDirectory }}/catalog-info.yaml'

  output:
    links:
      - title: Repository
        url: '${{ steps.publish.output.remoteUrl }}'
      - title: Open in catalog
        icon: catalog
        entityRef: '${{ steps.register.output.entityRef }}'
      - title: Repository
        url: '${{ steps.publishTwo.output.remoteUrl }}'
      - title: Open in catalog
        icon: catalog
        entityRef: '${{ steps.registerTwo.output.entityRef }}'

Be extremely careful to not use special characters(for example -) in the id of the publish phase.

Since we introduced the new input.sourcePath field in the publish phase (which is a new reserved keyword), check if your existing templates are already using this field for other reasons.

Dependencies' vulnerabilities in witboost​

We took some actions to reduce the vulnerabilities of our codebase:

  • Upgraded all the versions of libraries with known vulnerabilities for which a solving version was available.
  • Upgraded the Scala version and some dependencies versions to resolve some of the major known vulnerabilities of the code.
  • Changed the base image to use an alpine-based one instead of the ubuntu-based one.
  • removed some deprecated and problematic libraries, like "moment".