Skip to main content

Version 1.2.0+

These are the release notes for the v1.2.0 release of witboost.

Please refer to the official documentation for a more in-depth overview of the released features.

Features​

Reverse provisioning​

The Reverse Provisioning feature relieves users of the burden of manually filling all the fields in the catalog-info.yaml file of a witboost component.

By initiating a reverse provisioning operation through a user-friendly step-by-step wizard, users delegate the task of retrieving metadata from various infrastructure services (such as AWS, Azure, GCP, on-premises systems, etc.) to specific provisioners. Depending on their implementation, these provisioners may be capable of importing schema definitions, billing details, sample data, tags, descriptions, data sharing information, access permissions, and more.

The provided documentation comprises:

  • A user manual that explains what reverse provisioning is, how it functions, and how to execute it.
  • A platform team manual that provides guidance on configuring and enabling reverse provisioning for a set of components.

Specific Provisioner Logs​

A new log structure is used by the Specific Provisioner and received by the coordinator. The phase field contains the step of the provisioning from where the log is being produced (i.e. validation, provision, etc.) and could also mention specific moments of the process, so it's free for the developer to include (i.e. "Validation - pushing file"). The message field is user-friendly and contains the information to show. The list of logs contains the whole set of logs since the beginning of the provision task.

Furthermore, an optional error object can be included according to the new error structure to give the Specific Provisioner developer the ability to send structured error messages in cases something fails also with possible solutions.

The Provisioning coordinator is taking care of persisting the logs received from the Specific Provisioners..

The Witboost UI is now able to display the Specific Provisioner Logs: a new dedicated button Log file opens a drawer containing the aggregated logs generated by all the steps involved in the Data Product deployment. In addition, by opening the details drawer of each step, a new panel containing the logs related to the specific step is available.

Error handling improvement​

On the backend side, the Provisioning Coordinator has been reworked to introduce a new error structure and new error classes according to the cases. The new error structure includes more fields that allow to specify additional information about the error occurred, to help the user troubleshooting the problem. In particular, the moreInfo field contains a detailed description of the problems found and of possible solutions to overcome them. All the changes to the Provisioning Coordinator are retrocompatible.

On the frontend side, a custom alert system has been developed to replace the default one, to be compliant to the new error structure coming from the backend. It is now used at platform level by the entire application. The new alert system is showing additional details regarding the errors occurred and clearer error message, fixing wrongly parsed error messages and raw technical error messages that were not user-friendly in some cases. In addition, the GraphQL errors have been reworked to fix their handling and the resulting visualization frontend side.

Scaffolding errors and repositories errors improvement​

Scaffolding errors and repositories errors related to all the supported providers have been reworked in order to be more user-friendly. Some edge cases not correctly handled before are now fixed, with more detailed and understandable error messages. Some of the improvements regard errors when specifying an invalid target repository or an invalid path, or when trying to create nested Data Product repositories.

Entity ingestion errors improvements​

Ingestion errors related to all Witboost entities are now more visible on the UI. An error badge in the Data Product table has been introduced to highlight Data Products that contain errors. The badge includes a tooltip that shows the error message, the same message displayed in a dedicated banner inside the Data Product page. This change includes also errors coming from custom processors.

In addition, in case of entity errors affecting a template (e.g. a malformed catalog-info.yaml file) we highlight these errors in the template card page and disable them, to not allow the users to use it until it will be fixed. The error is displayed in the same way inside the UI and the change affects also the Blueprint template page.

Reserved field processor​

A new feature has been introduced to allow to specify reserved fields for each entity kind, whose values must not change. In the local section of the configuration, you can define specific reserved fields for different entity kinds. This allows you to customize reserved fields based on the type of entities you are working with. You can specify the entity kinds and the related paths in the following way:

mesh:
  reservedFields:
    local:
      myKind:
        - 'path.one'
        - 'path.two'
      anotherKind:
        - 'path.one'

A special behavior has been implemented for the fields metadata.name and kind, that unlike other local fields, can only be set at global level.

By adding this configuration, the Reserved Fields Consistency Validator will ensure that the specified field values remain unchanged for the entities of the specified kinds. If any reserved field is modified, the system will detect the change during the next entity refresh, and the entity will display an error in the UI, suggesting to restore the original fields' names inside the entity yaml descriptor. Once it is fixed, the error will disappear in the next entity refresh.

RBAC permission page​

In the General section of the user Settings page, some tables have been added to show the user's Roles and Permissions. They include both personal Roles and Permissions and the ones granted by belonging to Groups. By clicking on the table rows a drawer opens containing additional information and details regarding the association between Roles and Permissions.

Entity picker permission check​

The entity picker has been improved to show a more helpful message when no data is returned. The improvement includes the suggestion to check whether the missing data is due to the lack of required permissions, to make the user aware of it.

Upgraded data products graph​

The data products graph in the marketplace went under a big restyling, that includes the fix for a lot of minor problems, but also new features:

  • empty domains are now displayed (by empty domains we mean domains with no data products deployed)
  • the search functionality now also filters data products on the graph. When searching, the active selected data product is cleared
  • names are displayed with an ellipsis if too long, and they now adapt to the size of the domain

Better support for microfrontends and customizations​

Since the microfrontends were already supported, but it was not easy to integrate them inside witboost, we decided to improve the overall experience with integrating external custom microfrontends. As of now, we added a new configuration that lets users load microfrontends in the marketplace output port page, in the marketplace data product page, or as a brand-new marketplace page.

In the future, we are thinking about adding support also to other points in the application where microfrontends can be loaded.

You can find all the details regarding how you can change the configurations to start including your microfrontends in the official documentation. The documentation has also been updated accordingly to better explain to developers how to start developing a new microfrontend application from scratch.

Revamp of the UI style​

We started a revamp of the different witboost pages: we introduced a design system, and we are working on adapting all the existing pages to it.

With this release, we are releasing a first revamp of all the pages of the Marketplace and Computational Governance modules. Some components could still be updated in the future, but this is a first step towards aligning all the pages to the design system.

We introduced also some new components, like the resizable cards, that we started adopting throughout all the witboost's pages.

Introduced tags in the marketplace​

The witboost search is now able to search for all DPs that have a certain tag, both in the marketplace and in the builder. To search in the tags you need to put the exact value you would like to search for.

Block non empty domain unregistering​

We added a feature to block the unregistration of a domain when it still has data products registered. The feature is disabled by default, but you can enable it by setting this configuration in the UI:

catalog:
    disableNonEmptyDomainUnregister: true

Entity picker for groups and users filtered based on current user​

Entity pickers are now able to filter their options based on the current user and the entities it has access to. To add this capability to a picker a schema template, you must set showOnlyUserOwnGroups: true under ui:options. You can also do this on the access request dialog, by using this configuration:

mesh:
    marketplace:
        ui:
        accessControl:
            showOnlyUserOwnGroups: true

Issues Fixed

  • Fixed an error that displayed [object object] instead of "search"
  • Fixed a marketplace bug that showed in some environments input port relations taken from other environments, resulting in broken links on the graph and dependency pages.
  • Introduced a new configuration variable useHttpsSearchUnauthorized that is used to query through https ignoring the certificate along the way (specific behaviour only introduced because of environments where a certificate is not available)
  • In the marketplace tables we refactored the expandable items to improve responsiveness
  • Renamed the menu items "Create" to "Templates" and "Blueprint" to "Blueprints"
  • Added the version to the entity picker when selecting systems, to make it more clear to users
  • Added the version to the data products displayed in the DP Graph
  • Removed the field infrastructureTemplateId as a required field for catalog-infos of systems
  • Fixed a wrong link that appeared in some of the marketplace question notifications
  • Updated the logic to accept the following formats as values of the subDomainOf field: "domain_name", "domain:domain_name", and "domain:default/domain_name". Example: subDomainOf: domain:default/myDomain
  • Added a new optional string configuration deployStep.resultField field under mesh.provisioner to specify a field name that will be the field extracted from the deploy step result, displayed in the deploy step detail drawer (instead of the complete JSON result)
  • Created a new Blueprint import page dedicated to the import of blueprint entities
  • Transitioned from vendor specific APIs to a native Git library that enables operations to be batched together, improving the performances of all the git operations
  • The data platform team can now restrict templates and blueprints to be visible just to a specific domain. by adding the spec.domain property inside the template or blueprint descriptor. All templates and blueprints without a domain will be visible by all the users
  • Changed the OK icons inside parent test suites that are now filled
  • The number of failed metrics is now correctly displayed while displaying test results
  • Changed the condition for polling on the Control Panel page in the test section. The global status of the test is running as long as at least one task is running
  • HTTP errors are now caught and processed by the coordinator returning a more meaningful message
  • Errors coming from external witboost services are now marked appropriately, and more understandable
  • Handled the cases of where a data product had a null last deployment date
  • Now the /v1/resources/dataproducts API is returning the last deploy time parameter too
  • Fixed a bug where the UI would show OK on a failed validation when the status is null
  • Fixed a bug where console logging wouldn't show any information about the error that was logged
  • The resources to be retrieved by the perimeter resolver are now all the successfully deployed data products
  • Scheduler's failed tasks don't display anymore the wrong date 1970-01-01
  • Fixed on the marketplace the connections for the same descriptor on multiple environments. Now, the input ports are independent among environments
  • Added a method that replaces all dashes separator with _ for all the JSON fields in the marketplace
  • Fixed a bug on the marketplace for the descriptor that overwrites public fields with null
  • Fixed an OOM error in the coordinator
  • Fixed task status after execution plan timeout
  • Fixed a bug in the UI that made backend crash with badly formatted JWT token
  • Reverse provisioning templates now does not require domain
  • Fixed display of values in rbac settings table
  • Added fixed order for independent deploy tasks
  • Prevent task descriptors retrieval when not necessary when fetching existing provisioning plans
  • Fixed random UI crashes when a processor tries to delete an entity
  • Fixed sorting of environments in marketplace and governance selectors
  • The authentication token is now passed to the micro-frontend
  • Fixed various errors with RBAC parsing entity refs
  • Fixed an error that caused errors when deploying even if validation itself succeded
  • Coordinator has now the ability to terminate dangling execution plans at startup
  • Removed an error log that was unnecessarily printed
  • Correctly deactivates ReservedFields when not in config
  • Fixed a bug that caused a JSON parsing error to show when the request was unauthorized
  • Fixed a bug that caused remote engine policies to be unable to be tested
  • Added a new RBAC rule to show resources when no domain is associated with them to all users
  • Fixed the hint message under cron fields and the cron expression validator to reflect that the Day of Week value has a range of 0-6 where 0 is Monday
  • Fixed vulnerability affecting org.apache-commons.text by upgrading it to 0.10.0
  • Fixed vulnerability affecting cue by upgrading it to 0.6.0
  • API errors on external policy evaluations are now handled instead of throwing an exception
  • Version and descriptor of the data product are now available in the marketplace for output port microfrontends.
  • Added the snapshot version on deployments
  • Fixed a bug where when a CUE validation fails, it highlights the wrong line.
  • Fixed a bug where a data catalog is required for deployments on environments where it is not setup.