Skip to main content

Changelog 1.0.0

UI​

Changelog v1.0.0​

[#1021] Resolve an error with component policies on validation​

Bug fixes

Fixed the CUE string sent to the backend which had some syntax errors.

Related issue

Closes #1021

[#1018] Update query to retrieve display_name_field​

New features and improvements

Replace urn with new resource_display_name field inside policy tests.

Related issue

Closes #1018 #972

[#1020] Insert displayName in the control panel​

New features and improvements

Add displayName in the result of validation.

Related issue

Closes #1020

[#1019] Out of policy flag to registry navigation is broken​

Bug fixes

Fixed registry navigation in the breadcrumbs.

Related issue

Closes #1019

[#994] policies in "grace" state are not editable​

New features and improvements

  • Add disabled grace status to policies/metrics
  • The status filter will filter the policies/metric for bot disabled and disabled_grace status when the disabled option is selected

Related issue

Closes #994

[#1001] Version in YAML is not consistent with release version​

New features and improvements

The Preview descriptor window will refresh automatically after each of the operations: commit, release, or new snapshot.

Related issue

Closes #1001

[#1015] Enable the functionality to change the name of a policy​

Bug fixes

Enable changing the name of the policy during the edit.

Related issue

Closes #1015

[#1008] Bad cron expression does not block going to the next step​

Bug fixes

Fix bug on bad cron expression validation. It is related to two problems:

  • Mismatch configuration on the libraries we used for cron expression (one to validate, one to parse)
  • Form validation was applied only on the first library but not on the second one

Related issue

Closes #1008

[#1007] Fix execution strategy not editable​

Bug fixes

Removed disabled from execution strategy field

Related issue

Closes #1007

[#938] Uppercase column names in sample visualizations (output ports) in the marketplace​

Bug fixes

Remove the "capitalize" style from the VAR TYPE column

Related issue

Closes #938

[#985] DP Dependency graph: cropped External Resource name​

Bug fixes

Added a tooltip to dependency graph elements

Related issue

Closes #985

[#995] Resolve confusing policy results page​

Bug fixes

  • Fix Test row style
  • Add version to policy
  • Updated icon's color for non-blocking results

Related issue

Closes #995 #992

[#972] "Policy test panel improvements"​

New features and improvements

  • Hide compliant/not compliant test results if they are empty.
  • Remove ID from result

Bug fixes

  • Hide "no resource found" during test results polling.

Related issue

Closes #972

[#1003] Fix mkdocs dependency​

New features and improvements

  • mkdocs dependent packages were moved to stage 2 of dockerfile

Bug fixes

Accessing observability info should correctly redirect to mkdocs

Related issue

Closes #1003

[#969] Refactor ControlPanel response provisioninplan API​

Bug fixes

Align Test section with the Get provisioningplan api format

Related issue

Closes #969 #966

[#878] Autotest improvement (CGP suite)​

New features and improvements

  • Improvements to the auto testing system
  • Getting rid of the "builder_governance_policies" suite. (Moved it to the deprecated stance ./wdio/test/pageobjects/helper/deprecated tests).
  • Added a new test suit "builder_policies" with one test script for the CGP module instead of the outdated suit "builder_governance_policies".
  • Added some drafts for new tests

Related issue

Closes #878

[#983] Refactor regular expression picker​

New features and improvements

Refactoring RegexPicker to be more generic and allow future extensions.

Related issue

Closes #983

[#975] Fix policy result view on ko results​

Bug fixes

Fix bug on test cue script editor.

Related issue

Closes #975

[#948] CGP review the policies results page​

New features and improvements

Added tags column on top 10 failed policies table in the results page

Bug fixes

Fixed env query on evaluations report

Related issue

Closes #948

[#856] Fix notifications errors​

Bug fixes

  • Updated the witboost logo for informative notifications.
  • Fixed informative notifications with an empty body, now they are not collapsed and expandable. Fixed Accordion component style.
  • Fixed and moved the Mark as read button for each individual notification.
  • Removed unread-to-read notification status update on click (that was making unread notifications disappear without the possibility to open and interact with them).
  • Fixed NotificationProvider and NotificationsIcon in order to have a unique global NotificationProvider to be used by both NotificationsIcon and NotificationPage. In this way the shown notification data will be aligned among all the components that use them.

New features and improvements

  • Implemented a new Mark all as read feature to improve the user experience and allow them to mark all the notifications as read at once.

Related issue

Closes #856

[#826] Data product lifecycle image​

New features and improvements

Added a new image to display the data product lifecycle

Related issue

Closes #826

[#968] Disable name change when editing a policy​

New features and improvements

Disable policy name field on edit.

Related issue

Closes #968

[#970] Marketplace API - remove publish endpoint from documentation​

New features and improvements

  • removed the /publish endpoint description from the Marketplace API summary

Related issue

Closes #970

[#964] Send notifications after question answered in Marketplace​

Bug fixes

  • Notify the user when a question he asked has been answered.

Related issue

Closes #964

[#883] Create new pickers for paths and URLs​

New features and improvements

  • Added a custom validation error for URI format to return a custom error message
  • Added a RegexPicker and introduced a new picker for folder path that validate that using a regex for Linux and Windows Systems

Related issue

Closes #883

[#949] Make graphs in the Mesh Supervision page configurable​

New features and improvements

Add configuration to hide graphs inside the mesh supervision page

Related issue

Closes #949

[#962] CGP small UI Fixes on the Control panel​

Bug fixes

  • Remove the round border on test rows
  • Fix selection highlight
  • Fix test with warnings

Related issue

Closes #962

[#951] Add a new metrics page​

New features and improvements

  • Add a page to create new metrics
  • Add picker to add thresholds (waiting for @Emilio.Cassanese feedback)
  • Handle routing
  • Add types to handle creating metric request
  • Handle validation on threshold ranges (min cannot exceed max).

Related issue

Closes #951

#### [#947] Upgrade the multi stage build

New features and improvements

This change attempts to improve/decrease the size of docker image for UI microservice

Breaking changes

Docker host build was converted to multistage build

Related issue

Closes #947

[#961] Fix query GET_INSPECTOR_RESULTS​

Bug fixes

Fix environment query on evaluation results

Related issue

Closes #961

[#889] Wrong text color on control Panel selected test in Poste theme​

Bug fixes

Fix the selected row hover style in the control panel

Related issue

Closes #889

[#960] Default ordering of policies should be newest to oldest​

Bug fixes

Sort the policies by creation date desc

Related issue

Closes #960

[#959] Cue script not shown on Policy Test page​

Bug fixes

Fix breaking changes policies Cue script

Related issue

Closes #959

[#921] Restyle the control panel validation page​

New features and improvements

Update the status visualization of the test validation results

Related issue

Closes #921

[#954] CUE script editor pane displayed on policies​

Bug fixes

  1. Show Cue Script for Breaking Changes policies
  2. Set to be open by default for non-CUE policies

Related issue

Closes #953 #954

[#956] fix Version generator script​

New features and improvements

Added Checks for symbols at the end of the string and removel.

Bug fixes

Crash of deployment on dev

Related issue

Closes #956

[#945] CGP tags review​

New features and improvements

Added tags chips and modal on policies registry

Related issue

Closes #945

[#952] Align UI a new EvaluationReport​

New features and improvements

  • Update label 'Completed' in test.
  • Update enum 'TestStatus'

Related issue

Closes #952

[#950] CGP display error messages in a clearer way​

New features and improvements

Refactor Test Detail component

Bug fixes

  • Fix preprocessing empty value
  • Show Cue script editor only to cue script policy

Related issue

Closes #950

[#946] Marketplace raises an error when clicking on column flag​

Bug fixes

The marketplace catalog table raises an error trying to sort the flag column. To fix the bug, we disabled sorting on this column.

Related issue

Closes #946

[#936] Better handling for dependsOn and identifiers​

New features and improvements

This feature is meant to allow backward compatibility for the use cases where URNs that we are generating for dependsOnField are always lower_case but we had some old components that still have the camelCaseUrn

Related issue

Closes #936

[#788] RBAC error not so meaningful while registering a domain​

Bug fixes

When you do not have permission to view an entity, display a Warning: Not found error instead of a not user-friendly error message.

Related issue

Closes #788

[#924] [CGP] Resource type values of the form are static​

Bug fixes

Fixed static resource type on policy wizard form

Related issue

Closes #924

[#927] "Policy wizard improvements"​

New features and improvements

  • Add more constraints to policy wizard.
  • Add spaces among wizard inputs.
  • Change cron validator configuration to match with BE

Bug fixes

Enable cron expression field during policy editing if it was custom.

Related issue

Closes #927

[#940] [CGP] Confusion with updating after using the "save" button​

Bug fixes

Fixed policy editor not showing last updated policy content

Related issue

Closes #940

[#944] [CGP] Ability to interact with the text of the descriptor in the test menu​

Bug fixes

Make descriptor not editable

Related issue

Closes #944

[#916] Trigger Downstream pipeline​

New features and improvements

  • added new job to trigger a Downstream pipeline

Related issue

Closes #916

[846] New Metric list page​

New features and improvements

  • Add new types to handle metrics
  • Add metrics list
  • Add a new flag to enable/disable the metric section

Related issue

Closes #846

[#912] Error message is not showing well-formed​

Bug fixes

  • Fixed error message parsing from response.

Related issue

Closes #912

[#928] Cron selector improvements in the wizard​

New features and improvements

Add a list of presets to fill the cron expression field. The user still has the possibility to add a custom one. Presets are:

  • Every 12 hours 0 0 */12 ? * *
  • Every day at noon 0 0 12 * * ?
  • Every 2 days at noon 0 0 12 */2 * ?
  • Every week at noon 0 0 12 */7 * ?

Related issue

Closes #928

[#922] Add support for component CUE policies​

New features and improvements

Add preprocessing field to the policy form that enables the definition of CUE component policies.

Related issue

Closes #922

[#923] Fix the image name​

Bug fixes

Fixed the image name that made it unavailable for some versions.

Related issue

Closes #923

[#842] Fix the components fields overlapping​

New features and improvements

The owner field in the data product/components page is now of ellipsis type so that it can be hidden but we had to wrap it inside a paragraph to do so

Related issue

Closes #842

[#932] fix marketplace query to retrieve evaluation result​

Bug fixes

This MR fix a graphQL query to retrieve evaluation result of policies. the dataProductUrn indicated in the query must be retrieved from the field DataProduct.external_id

Related issue

Closes #932

[#925] Add Data Catalog documentation​

New features and improvements

Docs for data catalog plugin

Related issue

Closes #925

[#932] Fix marketplace query to retrieve evaluation result​

Bug fixes

This MR fix a graphQL query to retrieve evaluation result of policies. the dataProductUrn indicated in the query must be retrieved from the field DataProduct.external_id Related issue

Closes #932

[#926] Fix query to visualize tha lat flag in marketplace details​

Bug fixes

This PR fix the graphQL query in a marketplace dataproduct details section to retrieve the last execution result of policies.

Related issue

Closes #926

[#893] CGP progress bar is stuck to zero on noncompliant policies​

Bug fixes

  • Fixed bug that prevented the progress bar from completing when resources were not compliant

Related issue

Closes #893

[#919] Flag and Score info is not displayed correctly"​

Bug fixes

Fix resource_type in graphql query

Related issue

Closes #919

[#855] CI Package stage too slow​

New features and improvements

Improved the overall build time of the pipeline.

Related issue

Closes #855

[#913] Refactoring models for metrics and policies​

New features and improvements

Since we have to implement the metrics page, it's necessary to create new types, to be ready to accept new future changes. In addition, currently, all types were stored inside one long single file. I added the following folders, trying to have a good division:

  • resource type
  • policy
  • metric
  • environment
  • evaluation report
  • evaluation result
  • common
  • insight

All enums are now listed inside one shared file.

Related issue

Closes #913

[#904] Fix some UX problems​

Bug fixes

Corrected some UI bugs:

  • improved the details shown for each policy
  • better displayed the summary of the create/edit wizard
  • added ellipses and a tooltip in the results
  • better display of the interaction type

Related issue

Closes #904

[#910] Fix React error with environments​

Bug fixes

Remove warning on pagination when policies count is zero

Related issue

Closes #910

[#918] Add confirmation dialog for Draft to Grace policy status update​

New features and improvements

  • Added a confirmation dialog when updating a policy from Draft to Grace state to highlight that after that update, it won't be possible anymore to edit the policy, but only to disable it.

Related issue

Closes #918

[#908] Going forward and backward in the policy form is bugged​

Bug fixes

  • Fixed cronExpression reset when going forward and backward
  • Added a check for passive executionStrategy and runtime timing

Related issue

Closes #908

[#917] Fix not shown descriptor when validation has some errors​

New features and improvements

  • Removed NOT_STARTED filter
  • Added warning icon for NOT_STARTED tasks
  • Fixed returned type in useTestCard method

Related issue

Closes #917

[#914] Policy save button must not always be enabled​

Bug fixes

Fixed the save button behavior for the policy editor.

Related issue

Closes #914

[#915] Modify the test visualization​

Bug fixes

This MR allows you to view all the results of the policies applied to the descriptor. Previously, only the result of the first policy was displayed

Related issue

Closes #915

[#911] Remove old policy section​

New features and improvements

Remove the old policy section

Related issue

Closes #911

[#906] Buttons disappear when selecting 'Breaking change' during policy creation​

Bug fixes

It is possible create a policy of type "Breaking Change"

Related issue

Closes #906

[#909] Problem with cron expression field​

Bug fixes

Fix cron validation

Related issue

Closes #909

[#881] Add "Save" button on test policy detail​

New features and improvements

Add a save button on the policy cue script editor in registry testing

Related issue

Closes #881 #891

[#892] Fix policy deletion fo CGP policies​

Bug fixes

  • Added Delete menu entry for policies in every state.
  • Added confirmation dialog for policy deletion.

Related issue

Closes #892

[#907] Invalid interaction type for passive policies​

Bug fixes

  • Added cron expression in the coordinator request
  • Added error message when creating a a policy with execution strategy Passive and interaction type user2user

Related issue

Closes #907, #902

[#897] review enum in severity fields​

Bug fixes

Remove value "OK" from End2End test

Related issue

Closes #897

[#887] Environments not shown in review and create page​

Bug fixes

Add missing environments to review section

Related issue

Closes #887

[#900] Creating policy leads to a blank page​

Bug fixes

Fix routing to avoid blank page after:

  • create new policy
  • edit new policy
  • click on breadcrumbs

Related issue

Closes #900

[#895] Fix requests to CGP error handling​

Bug fixes

  • Removed the dependency on the builder-backend plugin.
  • Added a new sendRequestToCgp function that is working as expected for the interaction with the CGP endpoints. This change will fix the mismatching HTTP response status and the error messages displayed on the FE side.
  • Updated the getErrorMessage function on the FE side to handle all the possible error cases coming from the CGP endpoints.

Related issue

Closes #895

[#818] "Update Control Panel interaction to handle CGP policies in results"​

New features and improvements

Update control panel test setion with CGP result

Related issue

Closes #818

[#864] RBAC group name is case sensitive​

Bug fixes

  • Add some explanation to the documentation about the RBAC group name

Related issue

Closes #864

[#896] Change add/edit/list for boolean policy​

New features and improvements

  • Rewrite queries to adapt to new schema
  • Add mappers to add/edit actions to easily convert data to new schema
  • Add query fragment to have only one data source
  • Change types
  • Remove "OK" status from governance policy

Breaking changes

  • Policy table severity has been moved inside "content" field as key/value
  • Add and edit actions now requires new fields ("booleanSpecific", "metricSpecific") to correctly handle metrics

Related issue

Closes #896

[#899] Remove resource_type from policy top failed query​

Bug fixes

Fixed useless query parameter

Related issue

Closes #899

[#898] Fix that the three dots menu has only one option​

Bug fixes

Allow edit on the governance policy row menu

Related issue

Closes #898

[#894] CGP name and description input field accepts too many characters​

New features and improvements

  • Added some detailed descriptions of all the fields in the policy editor wizard.
  • Set a limit to name and description lengths.

Bug fixes

  • Changed the cron field evaluator to be six-values-based instead of seven.

Related issue

Closes #894 #890 #882

[#824] New policies summary page​

New features and improvements

  • Add policy results page
  • Add filters component
  • Add insights table and relative components
  • Add chart bar component
  • Add the Top failed policy table and relative components
  • Add utils for CGP policy results components
  • Add CGP results page types and interfaces
  • Add results page to Governance header menu

Related issue

Closes #824

[#880] Resolved a problem with pagination in the policies list​

New features and improvements

  • Add order by inside policies list query to improve readability
  • Reset pagination when the policy is deleted

Bug fixes

Fix bug on pagination when policy status change

Related issue

Closes #880

[#874] Improvement graphics in 'Test policy against DP' section​

New features and improvements

  • Improve graphics of test policy when there are not resources.
  • Change environments handling
  • Move test panel label inside header
  • Change launch tests query to adapt to new schema

Related issue

Closes #874

[#879] Solve dependency graph minor issues​

New features and improvements

Use Roboto font and hide placeholders

Bug fixes

  • Remove the wrong font weight on domain labels
  • Remove placeholders from the graph sidebar

Breaking changes

Bump dependency graph version to 2.0.2

Related issue

Closes #879

[#816] Align string preprocessing between MarketplaceDomainPreprocessor and the method used to build the final descriptor​

Bug fixes

  • Updated the domain id generation when adding a new domain in order to match the value coming from the Data Product descriptor when deploying a Data Product.

Migrations

Note: in order to make this fix work you need to refresh all the affected domain entities using the Witboost UI. This action will trigger the domain insert with the updated value.

Related issue

Closes #816

[#875] CGP APIs wiring - policy status change & new version​

New features and improvements

  • Linked the FE to the new BE API to update the policy status.
  • Linked the FE to the new BE API to create a new policy version.

Bug fixes

  • Removed Edit option from policies in Grace state.
  • Fixed the response parsing in case of errors to display them in a human-readable format inside the notification alert.

Related issue

Closes #875

[#852] Remove the "v" present everywhere as a prefix of many labels​

New features and improvements

Letter "v" or "V" that stands for version everywhere on the control panel is removed (before any version value e.g. v0.1-SNAPSHOT becomes 0.1-SNAPSHOT)

Related issue

Closes #852

[#877] Removed wrong prefix for error messages in the dialogs​

New features and improvements

Patched the error message to "Error" instead of reading the class name (this.constructor.name) as they do in the source library.

Bug fixes

Strange letters will not appear before any error on the frontend.

Related issue

Closes #877

[#858] Autotest for CGP module​

New features and improvements

Add new suit 10 for CGP module, contains test scenario that checks:

  • clickable and existing "Add new policy" button;
  • search feature;
  • filtering feature;
  • existence and the ability to move through the tabs of Policies, Scores and Metrics;
  • workable menu for switching the environment (dropdown menu)

Also, it contains the chrome version update to ^111

Related issue

Closes #858

[#872] APIs wiring - policy new version​

New features and improvements

  • Added the policy new version API.

Related issue

Closes #872

[#804] Integrate the new dependency graph version​

New features and improvements

  • Integrate version 2.0.1 of the dependency graph.
  • Properties and configurations are still the same since inherit from the previous version.

Related issue

Closes #804

[#847] Fixes for Dynamic Analysis Result​

New features and improvements

Removed the moment library and replaced it with luxon

Related issue

Closes #847

[#860] Creates API backend side to change policy status​

New features and improvements

  • Added the update policy status API.

Bug fixes

  • Updated the evaluate policy API to be compliant to the defined specification.
  • Fixed the policy update API to return correctly 400 in case of request validation error.

Related issue

Closes #860

[#873] Adapt queries to CGP updated schema​

New features and improvements

Adapt queries to CGP updated schema

Related issue

Closes #873

[#870] Changes to some fields are not received on create/modify of a policy​

New features and improvements

  • Handle policy selector
  • Handle policy tags

Related issue

Closes #870

[#861] Marketplace tab needs catalog.entity.read permission​

Bug fixes

  • Fix authorization request denied before checking if the entity requested needs no auth

Related issue

Closes #861

[#857] Commits and Snapshots are not created correctly​

New features and improvements

The commit and the new snapshot operations were based on Backstage data, unlike descriptor generation which pulls the data from GitLab. With this refactor you can be sure that what you see on Preview Descriptor and on GitLab will also be committed/created with a new snapshot operation.

Bug fixes

The descriptor is now updated correctly when creating new snapshots or commits. This behavior is fixed for both commits and also with new snapshots.

Related issue

Closes #857

[#868] 'Add new policy' button does not work from time to time​

Bug fixes

Fix bug on Add new policy button

Related issue

Closes #868

[#866] Manage no_resource outcome as a result of a policy test​

New features and improvements

  • Handle new no_resource outcome
  • Show OK and KO icons on evaluation results
  • Reset test panel when launch a new test

Related issue

Closes #866

[#859] CGP APIs wiring - edit policy​

New features and improvements

  • Create proxy BE to BE to call coordinator update policy API
  • Add wiring

Related issue

Closes #831 #859

[#867] Fix test-delpoy tests​

New features and improvements

  • 3-3 test has steps with deployment, and in fact, it should fail, but it works because every DP is passing the check for now. The biggest part of the test is working well except few moves.
  • 3-5 tests. Negative tests are expected to fail in the deploy step (just to check up on the error), but while all tests are passed we can't get an error, so this step is skipped.
  • 4-3 Test has some not working steps with policy validation check

Related issue

Closes #867

[#862] Fix create policy validation error​

New features and improvements

  • Created a new platform-backend plugin to contain all the services and utilities shared among all the backend plugins.
  • Moved the ZodErrorHandler express middleware to the platform-backend plugin to be used by multiple backend plugins and updated all the related references.

Bug fixes

  • Fixed the policy create API that was returning 500 instead of 400 in case of request validation errors.

Related issue

Closes #862

[#805] Add marketplace policies and metrics result panel​

New features and improvements

Insert the flags in the marketplace to give information on the result of the last run of each policy for each instance of the dataproduct present in the marketplace

Warning: Still missing version filter, can be fixed in another issue

Related issue

Closes #805 #808

[#863] Helm chart push to registry​

New features and improvements

Updated ci pipeline to push Helm chart to Gitlab registry

Related issue

Closes #863

[#819] Handle policies scheduling in the create/edit panel​

New features and improvements

  • Add missing field to policy add form
  • Wiring Add with api
  • Fixing types

Related issue

Closes #819

[#790] Alphabetical order not always respected​

Bug fixes

Order by on Domain name was ordering on the wrong column

Related issue

Closes #790

[#843] readsFrom and dependsOn urn are not updated when a new version is created​

Bug fixes

This MR fixes a bug in a new version process. The MR allows you to evolve the versions of the components also contained in the readsFrom and dependsOn fields

Related issue

Closes #843

[#741] Change management email sending​

New features and improvements

  • Used Nodemailer library to deliver notifications by email
  • Implemented new EmailService to handle email notification sending
  • Added the config part related to the email configuration and authentication
  • Added a fixed email template customizable using the related configuration section
  • Supported both login and oauth2 (2LO, 3-legged) authentication methods

Related issue

Closes #741

[#832] CGP APIs wiring - policy test​

New features and improvements

  • Add cgp.dump.sql to seed database
  • Add environment field inside test page (related to #836)
  • Add queries to fetch policy to test and related environments
  • Add url parameter to get current policy environment
  • Handle navigation error if user try to reach policies test page passing wrong parameters

Breaking changes

  • Change types, to include new fields as you can see inside cgp tables schemas
  • Add new interaction type enumerative (user2user, user2platform)

TBD

  • Add API call to provisioning coordinator/evaluate to get evaluation report id. (Waiting for Marco MR)

Related issue

Closes #832 #836

[#850] Review Helm Chart for Best Practices​

New features and improvements

refactored Helm chart:

  • removed postgresql dependency
  • added securityContext spec (optional)
  • added resources spec (optional)
  • added readinessProbe spec (optional)
  • added livenessProbe spec (optional)
  • added recommended labels
  • removed service account token auto-mount

Related issue

Closes #850

[#848] Remove dependencies vulnerabilities in Witboost​

Bug fixes

Removed all the HIGH and CRITICAL vulnerabilities found by Trivy scan of the application.

Related issue

Closes #848

[#845] Creates API backend side to test a policy​

New features and improvements

  • Added new API to contact CGP, launch policy test and retrieve the report id.
  • Added new CGP endpoint parser.

Related issue

Closes #845

[#803] Catalog-info updates are not mapped in DP release​

Bug fixes

When the input YAML contains multiple fields with the same names (differing only for casing) the catalog backend plugin rejects the registration by raising an error. Sadly, this affects also arrays where all the elements are strings, so an array like ["Test", "test"] would make the entity ingestion process fail.

Created a patch for @backstage/plugin-catalog-backend to avoid casing issues on the array.

Related issue

Closes #803

[#851] Autotest improvements (suites 3-5)​

New features and improvements

  • improvements for the autotests suites 3-5

Related issue

Closes #851

[#844] Creates API backend side to creates new policy​

New features and improvements

  • Connected policy creation to the corresponding CGP API

Related issue

Closes #844

[#853] Typos in docs and UI​

Bug fixes

  • UI: "For Request access" instead of "To request access"
  • Docs: Markdown not correctly formatted.

Related issue

Closes #853

[#849] New helm chart​

New features and improvements

Adds a new version of the helm chart without deleting the previous one.

Related issue

Closes #849

[#829] CGP APIs wiring - policies' list​

New features and improvements

Added GRAPHQL queries for:

  • get policies list
  • get resource types
  • get environments

Handle search by text with "name" OR "description" with debounce

Added governance-registry.dump.sql to seed:

  • policy table
  • policy_environment table

Delete previous implementation with REST api

Related issue

Closes #829

[#840] UI does not display data catalog and data product validation errors​

Bug fixes

  • Added the possibility to display the data product and data catalog validation errors.

Related issue

Closes #840

[#839] Fix control page async test id​

Bug fixes

Fix control page async test id

Related issue

Closes #839

[#835] Data Sharing Agreement not displayed metadata​

Bug fixes

  • Fixed DataSharingAgreement field: changed all the occurrences of DataSharingAgreements with the version without the final s

Related issue

Closes #835

[#825] Improvements in part of the control panel test​

New features and improvements

  • Added fix when runner launch test in the control panel
  • Improve existing tests

Related issue

Closes #825

[#833] Fix incorrect the name of the process​

Bug fixes

  • Fixed deploy/undeploy label now displayed consistently
  • Fixed steps order that was changing at each deploy status update (now fixed sorted order)
  • Fixed 4 warnings related to rendering issues

Related issue

Closes #833

[#821] Improve Policy test page​

New features and improvements

  • Refactor results/filters
  • Add mock policy detail

Related issue

Closes #821

[#823] Make all the CGP UI pages read-only in case there is no permission​

New features and improvements

Added check on governance permissions in the frontend

Related issue

Closes #823

[#820] Picker with except condition​

New features and improvements

Entity Picker can now filter out unnecessary data/entries using code as shown:

domain:
  title: Domain
  type: string
  description: the Domain of the Data Product
  ui:field: EntityPicker
  ui:options:
    allowArbitraryValues: false
    allowedKinds:
      - Domain
    excludeValues:
      - domain:DPU
      - domain:it

Related issue

Closes #820

[#768] New policy result page​

New features and improvements

  • Added error info box component
  • Added utils for extracting information from compliant errors
  • Added custom hook to handle scroll on uncontrolled editor component
  • Updated policy detail component to handle compliant errors
  • Added new interfaces for compliant errors
  • Added class name extension on codemirror editor (to make highlighting possible)
  • Added highlight functionalities to the text area editor

Related issue

Closes #768

[#822] Fix async validation FAILED status​

Bug fixes

Fix polling trigger

Related issue

Closes #822

[#817] Autotests improvement​

New features and improvements

Fixed problems:

  • access control (question and notification) in the suite 06
  • the stage when fixing descriptor before deploying in suite 03
  • policies validating different suits

Closes #817

[#800] Create new autotests for the blueprint​

New features and improvements

Adding a new test suit to the autotests blueprint feature

Related issue

Closes #800

[#814] Fix error when launch test in control panel​

Bug fixes

Fix broken keyframe style

Related issue

Closes #814

[#770] Integration with async validation​

New features and improvements

Add Async validation support to the control panel test card. To enable/disable asynchronous validation simply change the relative configuration

catalog:
  enableAsyncValidation: true

Breaking changes

Updated provisioning coordinator endpoints in configuration with the respective version.

Migration

Update the configurations with the new coordinator endpoints. In particular, an existing configuration like

mesh:
  provisioner:
    baseUrl: http://localhost:8088/datamesh.provisioningcoordinator/v1

must be updated to something like

mesh:
  provisioner:
    baseUrl: http://localhost:8088/datamesh.provisioningcoordinator

Related issue

Closes #770

[#767] New policy testing page​

New features and improvements

  • Add icons for each action of the governance registry policies list
  • Add new governance policy test page
  • Mock API to retrieve tests for one policy
  • Add polling to get tests already processed

Related issue

Closes #767

[#813] Template info doc​

New features and improvements

Added a tutorial for adding documentation to entities

Related issue

Closes #813

[#715] Make change management customizable for customers​

New features and improvements

  • Implemented a notification event system: plugins can now trigger informative events by specifying the event name and the relevant entity reference.
  • Customization options available for each event: can now personalize the text message, recipients, email, and notification sending settings for each event.
  • It is now simpler to add functions to retrieve recipients and create new events.
  • Known issues and future improvements*
  • During the registration of a new entity it is not available in the catalog yet, so the display name cannot be retrieved.
  • When creating the new version of a data product, both the new version event and the new register event are called, resulting in a double notification.
  • We can explore further customization of the notification text.

Related issue

Closes #715

[#798] Wrong rendering of subtitle in docs​

Bug fixes

Remove non-breaking space characters to correctly render documentation and readme.

Related issue

Closes #798

[#812] Cannot deploy a new version on Bitbucket​

Bug fixes

Fix control panel actions for bitbucket.

Creating the url to fetch files from the repository no longer happens using backstage.io/source-location annotations. The repoContentsUrl method of the RepositoryManager is used for which there is an implementation for each integration.

Related issue

Closes #812

[#809] Policies list page improvements​

New features and improvements

  • add icons to state and severity
  • change labels of the column names
  • remove "deleted" policies
  • change the filters to add a clearable select
  • add an ellipsis to the description (with a tooltip)
  • add the date as described in the UI sketches
  • add the icon to the menu entry

Related issue

Closes #809

[#731] Fix global policy that succeeds instead of failing​

Bug fixes

  • Fixed the Data Sharing Agreement naming convention to match the global policy

Related issue

Closes #731

[#811] Update chromedriver in wdio​

Breaking changes

Updated crhomedriver version

Closes #811

[#799] Wrong json parsing during audit​

Bug fixes

Catch the error when the parser fails to decode the json to collect the information for the audit.

Related issue

Closes #799

[#787] Policies insert/edit component​

New features and improvements

Add Edit/Create policy page in the Registry section

Related issue

Closes #787

[#810] Fix Blueprint selected data product value​

Bug fixes

Fix Blueprint selected data product value

Related issue

Closes #810

[#801] Enable new policies list page from app config​

New features and improvements

Show/hide new policies list page from app-config.yaml

Related issue

Closes #801

[#802] Fix missing token in template blueprint processor​

Bug fixes

Add missing token in template blueprint processor step

Related issue

Closes #802

[#769] New policy details component​

New features and improvements

  • Added new policy detail component
  • Added new policy overview component (Accordion)
  • Added new policy editor component

Related issue

Closes #769

[#795] Autotest improvements​

New features and improvements

  • Added severity to "it" blocks
  • Added features description to "it" blocks
  • Existing tests improved

Related issue

Closes #795

[#793] Resolve "Blueprint using error"​

Bug fixes

  • Added token on catalog API get entity call
  • Updated relations on blueprint entity dependency graph Added new logo

Related issue

Closes #793

[#766] New policies list page​

New features and improvements

Add the following plugins:

  • wb-governance, which includes a new registry page
  • wb-governance-backend, with proxy to send a request to the coordinator (data is mocked since the real one is still in progress)
  • wb-governance-common, to share types among other plugins

The data model has been created following the interface specification

Related issue

Closes #766

[#790] Add custom sorting for the description column to ignore case​

Bug fixes

Added a custom sorting on the description column on the data product catalog to sort ignoring case

Related issue

Closes #790

[#780] Authenticate catalog-extension calls only with server-to-server token​

New features and improvements

  • Authentication for catalog-extension-backend routes only for calls coming from the backend.
  • Getting groups ancestors in getDataProductAccess is now done by a route in catalog-extension-backend

Related issue

Closes #780

[#774] Add a new test to check accesses and requests​

New features and improvements

  • Adding a new test which makes requests and asks a question
  • Fixing existing tests

Related issue

Closes #774

[#792] Make notifications and marketplace routers able to handle backstage entity refs​

New features and improvements

  • Notifications and marketplace routers now take as input backstage entity refs (user:default/name.surname...) and transform them into compact entity refs (user:name.surname...)
  • Improve error messages in the marketplace

Related issue

Closes #792

[#789] Error when trying to ask a question​

Bug fixes

Fix user entity format in the Data Products page

Related issue

Closes #789

[#783] New version pop-up error​

Bug fixes

  • Fix a bug where the new version fails if there are no consumers
    • Add a check for an empty array in the Knex query
  • Remove unused code which retrieved data product access requests, now handled by ACL router

Related issue

Closes #783

[#764] Use Zod as the validator in notification-backend​

New features and improvements

Use zod as the validator for objects in the notification backend plugin

Related issue

Closes #764

[#772] Schema in OP details is not scrollable​

Bug fixes

Fix OP schema scroll

Related issue

Closes #772

[#778] Fix "could not load user information" error flashing​

Bug fixes

  • Fix "could not load user information" error flashing
  • Fix "Could not load Data Product Instance" display error

Related issue

Closes #778 #735

[#777] Update favicon​

Bug fixes

Updated favicon

Related issue

Closes #777

[#744] Create the new Blueprint pages​

New features and improvements

Implement the new Blueprint page

Related issue

Closes #744

[#775] Fix external-secrets typo​

Bug fixes

Fix secret name typo

Related issue

Closes #775

[#755] Migrate from vault to external secrets​

New features and improvements

This MR drops hashicorp vault as secret backend and uses external secrets

Related issue

Closes #755

[#773] Add trim in IdentifierPicker​

New features and improvements

The trailing and leading dashes are removed from the identifier when there is a leading/trailing space in the name

Related issue

Closes #773

[#763] Newsletter of pipeline results​

New features and improvements

  • Fix 05_04 test
  • Small errors correction in the README file
  • Prepare page object files for the new test 06

Closes #763

[#694] Notify DP owners when a new domain, template, or data product is registered​

New features and improvements

  • Send an informative notification when a new location is registered and the entity type is domain, template, or system (data product).
  • Remove unused code in the notification router, add some minor improvements
  • Move NotificationApi to wb-notification-common
  • Create wb-catalog-extension-backend, a new plugin that exposes GET routes and is able to directly access the relations DB.

Related issue

Closes #694

[#752] Registering more than one component from one template​

New features and improvements

Publishing in multiple repositories

It is possible to publish to more than one repository starting from one template. In that case, you would probably want to create two separate fetch, publish and register phases. Here we are introducing the input.sourcePath value in the publish phase which is used to read from the folder specified in the input.targetPath of the fetch phase so these two variables must be an exact match. An example of this is:

  steps:
    - id: templateOne
      name: Fetch Skeleton + Template
      action: fetch:template
      input:
        url: ./skeleton
        targetPath: '${{ parameters.rootDirectory }}'
        values:
          ...
          identifier: '${{ parameters.identifier }}One'
          destination: '${{ parameters.repoUrl | parseRepoUrl }}One'
          ...
    - id: publishOne
      name: Publish
      action: witboostMeshComponent:publish:gitlab
      input:
        allowedHosts: ['gitlab.com']
        description: 'This is ${{ parameters.name }}'
        repoUrl: '${{ parameters.repoUrl }}'
        rootDirectory: '${{ parameters.rootDirectory }}'
        dataproduct: '${{ parameters.dataproduct }}'

    - id: registerOne
      name: Register
      action: catalog:register
      input:
        repoContentsUrl: '${{ steps.publishOne.output.repoContentsUrl }}'
        catalogInfoPath: '/${{ parameters.rootDirectory }}/catalog-info.yaml'

    - id: templateTwo
      name: Fetch Skeleton + Template
      action: fetch:template
      input:
        url: ./skeleton
        targetPath: '${{ parameters.rootDirectory }}/two'
        values:
          ...
          identifier: '${{ parameters.identifier }}Two'
          destination: '${{ parameters.repoUrl | parseRepoUrl }}Two'
          ...
    - id: publishTwo
      name: Publish
      action: witboostMeshComponent:publish:gitlab
      input:
        allowedHosts: ['gitlab.com']
        description: 'This is ${{ parameters.name }}'
        repoUrl: '${{ parameters.repoUrl }}Two'
        rootDirectory: '${{ parameters.rootDirectory }}'
        dataproduct: '${{ parameters.dataproduct }}'
        sourcePath: 'two'
    - id: registerTwo
      name: Register
      action: catalog:register
      input:
        repoContentsUrl: '${{ steps.publishTwo.output.repoContentsUrl }}'
        catalogInfoPath: '/${{ parameters.rootDirectory }}/catalog-info.yaml'

  output:
    links:
      - title: Repository
        url: '${{ steps.publish.output.remoteUrl }}'
      - title: Open in catalog
        icon: catalog
        entityRef: '${{ steps.register.output.entityRef }}'
      - title: Repository
        url: '${{ steps.publishTwo.output.remoteUrl }}'
      - title: Open in catalog
        icon: catalog
        entityRef: '${{ steps.registerTwo.output.entityRef }}'

Be really careful NOT to use special characters(for example -) in the id of the publish phase!

Breaking changes

Introduced input.sourcePath value in the publish phase which is a new reserved keyword. For old templates, check that they are not using it already in order not to break them.

Related issue

Closes #752

[#743] Define the new Blueprint entity​

New features and improvements

  • Created a new blueprint entity
  • Added processor
  • Updated the builder
  • Refactoring catalog relation mapping

Related issue

Closes #743

[#757] Add a new section in README for autotest​

New features and improvements

  • A new section in the README document that describes and defines the impact of autotests
  • Fix 03_03 test

Related issue

Closes #757

[#762] Add authorization in GET dataProductAccess​

Bug fixes

Add authorization when calling GET dataProductAccess

Related issue

Closes #762

[#758] Error message is not displayed when accessing to Governance Policy section​

Bug fixes

Show an error message if the governance policy section doesn't work properly.

Related issue

Closes #758

[#760] Super domain temp in Marketplace's DP Graph​

Bug fixes

  • In the Dp Graph handle the case when a domain contains no data product but only other domains.

Related issue

Closes #760

[#761] Rectify provisioning coordinator url in config files​

Bug fixes

Update provisioning coordinator URL in local config files.

Related issue

Closes #761

[#759] Technical Information card displayed on values not handled​

Bug fixes

  • Fixed a bug where inconsistent data is rendered when we have invalid data inside the descriptor
  • Updated documentation to better explain how the technical information part works

Related issue

Closes #759

[#529] DependsOn and identifiers mismatch​

Bug fixes

Fix bug on wrong identifiers generated for capital case letters found in domain entities

Related issue

Closes #529

[#756] Update the build yarn script command in the GETTINGSTARTED​

New features and improvements

Updated details of the GETTINGSTARTED document.

Related issue

Closes #756

[#749] Autotest to check removing release after deletion​

New features and improvements

  • Added a new test to check that release will be deleted after deleting the relevant DP
  • Fix test 01_02 (update selectors of extensions)

Closes #749

[#662] Use the same token guard in all the APIs​

New features and improvements

Delete duplicate bearer token getter method. We discarded other proposed solutions since they don't fit our needs, and all the catalog APIs already check if the current user is authenticated.

Related issue

Closes #662

Provisioning Coordinator​

Changelog v1.0.0​

[#387] Add validation task display name​

New features and improvements

Add displayName in table task

Related issue

Closes #387

[#389] CGP fix migration script​

Bug fixes

  • Added UUID extension in validation policies migration script
  • Fixed cgp. missing in some tables references

Related issue

Closes #389

[#388] introduce a resourceDisplayName in evaluation result​

New features and improvements

Add configuration to identify which fields make up the resource-display-name fields. The new configuration is

resource-types = [
        {
            name = "data_product"
            configuration = {
                resource-name = "id"
                resource-display-name = ["domain", "name", "version"]
                resource-filter = "version"
            }
            resolver-configuration = {
                url = "http://localhost:8088/datamesh.provisioningcoordinator"
                path = "v1/resolve"
            }
        }
    ]

Add new column in evaluation_result table to insert this information (resource_display_name)

Related issue

Closes #388

[#382] Additional transitions for policies in grace status​

New features and improvements

  • Added disabled-grace status, new version release of policy is possible from that status.

Breaking changes

  • Breaking change to the UI, it needs to add disabled-grace status for displaying and as a transition to and from grace status

Migration

  • Removed unique (name, version) constraint from the DB

Bug fixes

  • Fixed releasing a new version of the policy when that same version number was already deleted.

Related issue

Closes #382

[#381] Breaking change policy returns error when executing on a not yet deployed data product​

Bug fixes

Breaking change policy does not fail for non deployed data products

Related issue

Closes #381

[#383] Change the coordinator error​

Bug fixes

  • Fixed the error message in case of exception

Related issue

Closes #383

[#310] Persist policy history​

New features and improvements

Policy updates history is persisted

Related issue

Closes #310

[#332] CGP documentation​

New features and improvements

Introduced the documentation for the CGP module.

Related issue

Closes #332

[#380] Remove FailedWithError and insert in result the error message​

Bug fixes

  • Removed FailedWithError status

Related issue

Closes #380

[#379] Refactor ExecutionEvent​

New features and improvements

This PR tries to simplify the code.

  • use ExecutorResult instead ExecutionEvent
  • change level at NotExecuted outcome. NotExecuted causes the report to be CompletedWithError
  • empty policy return a outcome ok in evaluation_result

Related issue

Closes #379

[#378] /v1/provisioningplan return a wrong count in pagination​

Bug fixes

  • Fixed count for provisioning plans
  • Refactored code
  • Fixed limit not displayed in the PaginationModel

Related issue

Closes #378

[#343] CGP - Processing values added as a selector​

Bug fixes

  • Fixed resource selectors

Related issue

Closes #343

[#373] Fix the NotExecuted and Pending result encoder​

Bug fixes

  • Moved the outcome field from the body of NotExecutedEvaluationResult and PendingEvaluationResult to the attribute list.

Related issue

Closes #373

[#369] The name of the policy is not unique​

New features and improvements

  • Added check for unique policy name

Related issue

Closes #369

[#351] Move the model conversion outside the scheduler​

Breaking changes

  • The UI must be aligned with the returning object of /v1/provisioningplan API

Bug fixes

  • Moved the EvaluationReport conversion outside the CueValidationTask
  • Now the value that is saved in the database is a complete report

Related issue

Closes #351

[#333] Get total count of Provisioning Plans in the get response​

New features and improvements

Modifies the returned object of GET v1/provisioningplan to include the pagination results

Breaking changes

The schema of the returned object of the mentioned endpoint will change

Migration

Modify the logic of the caller to accept the correct object

Related issue

Closes #333

[#337] Evaluation report: multiple evaluation results with same policy id and resource name​

New features and improvements

  • Added validation to the push of evaluation results so a single resource cannot have multiple results from the same policy
  • Added functionalities to Outcome enum

Breaking changes

  • The couple (policyId, resource.name) must be unique among the list of sent evaluation results

Bug fixes

  • Fixes an error where Outcome enum parsing in the CGP could throw an error rather than being handled accordingly

Related issue

Closes #337

[#371] [CGP] Environment and result_filter fields not filled in the database​

Bug fixes

  • Now the CGP module saves the result_filter in the evaluation_result table
  • We can avoid to save also the environment since it is a already saved in the evaluation_report table

Related issue

Closes #371

[#370] fix Version generator script​

New features and improvements

Added Checks for symbols at the end of the string and removal.

Bug fixes

Crash of deployment on dev

Related issue

Closes #370

[#368] Pushing evaluation results with errors to boolean policy results in error​

New features and improvements

  • The satifiesPolicy attribute of an EvaluationResultDto is now set depending on the existence of errors on a EvaluationResult

Bug fixes

  • Fixes a bug where the CGP would throw an error when pushing evaluation results with non-empty errors to a boolean policy

Related issue

Closes #368

[#350] Fix 'validate' feature on a specific resource​

New features and improvements

This PR contains the following changes:

  1. EvaluationResult has a new column policy_status
  2. EvaluationResult has a new outcome value ERROR_NOT_BLOCKING
  3. Removes outcome 'no_resource'. Each Report contains a list of EvaluationResult, one for each Resource. If there are no resources, the list is empty.

Related issue

Closes #350

[#360] feature: Generalize marketplace-compliant services​

New features and improvements

  • Implementation: is now possible to define more than one marketplace service in the coordinator's configuration
  • Implementation: isBlocking (default: true) flag on task commands. If false and the corresponding task returns an error, it will not cause the provisioning plan to fail
  • Modification: MarketplaceTaskFunction and MarketplaceAclTaskFunction have been replaced by MarketplaceTasksBuilder

Related issue

Closes #360

[#358] hotfix: Missing checks for passive policies​

New features and improvements

  • Added checks for the creation of passive policies
  • Added checks for the presence of metric/boolean specific values according the resultType field value

Bug fixes

  • A Passive Policy can now be either a metric or boolean policy
  • A Passive Policy will now be created only if it has timing runtime without a cron expression
  • Fixes a bug where the CGP accepted as valid a metric policy with boolean specific values and viceversa

Related issue

Closes #358

[#361] Breaking change - resource composition is not configurable​

New features and improvements

Now how the breaking change executor compose the two resources is configurable

Related issue

Closes #361

[#366] [CGP] Cron expression: fix DayOfWeek range​

New features and improvements

  • Modification: new cron expression converter's DayOfWeek range: 0-6

Related issue

Closes #366

[#338] Persist data catalog info​

New features and improvements

  • Data catalog now takes enriched component descriptors (with deploy info from specific provisioners)
  • Data catalog can further enrich deploy info with its data

Related issue

Closes #338

[#349] Cannot create policy new version​

New features and improvements

  • Moved logic for new version to domain layer

Migration

  • Removed the unique constraint on (groupId, version) for policies

Bug fixes

  • Can now release a new version correctly, from the latest version
  • Deleting a policy doesn't throw an error

Related Issues

Closes #349

[#347] Trigger Downstream pipeline​

New features and improvements

  • added new job to trigger a Downstream pipeline

Related issue

Closes #347

[#359] "Clarify error messages for CGP policy executions"​

New features and improvements

PolicyExecutionResult has been split into ExecutorResult and ExecutionEvent. When an evaluation ends without any internal error or anomaly, i.e. outcome is ok, error, info, warning, an EvaluationResult will be created and an ExecutorResult will describe the details in case there is a non compliant descriptor.

When an unexpected error occurs or no resources are found, a NotExecutedEvaluationResult will be created and an ExecutionEvent will explain what happened (platform failures included).

Breaking changes

  • Introduced outcome not_executed. On UI side, this is not handled but it is not causing any issue

Related issue

Closes #359

[#365] Cron expression: always convert star-slash notation​

New features and improvements

  • Modification: the cron star-slash notation is now always converted to its equivalent extended form

Related issue

Closes #365

[#356] Thresholds of external metric policy are not being persisted​

Bug fixes

Metric specific values (i.e. thresholds) are now correctly persisted

Related issue

Closes #356

[#345] Ability to roll-back policy status​

New features and improvements

  • Added the possibility to filter in and out deleted policies from the query by policyId
  • Update of a policy can now be made only when in Draft status and doesn't modify the status of it

Bug fixes

  • Fixes a bug where it allowed the user to update the policy in any stage
  • Fixes a bug where it allowed the user to "revive" a deleted policy by editing it
  • Fixes a bug where querying for a deleted policy would actually retrieve it

Related issue

Closes #345

[#354] Set UTC as cron scheduler fixed time zone​

New features and improvements

Modification: the cron scheduler now uses the UTC time zone instead of the system default one

Related issue

Closes #354

[#353] add severity to passive policy​

New features and improvements

Add severity to passive policy

Related issue

Closes #353

[#352] Resolve "Preprocessing should generate a list of resources"​

New features and improvements

  • Added unpacking feature for CuePreprocessStep. If the cue script returns a list of elements, the pre-process step will create one resource per item

Related issue

Closes #352

[#344] preprocessor script boundaries​

Bug fixes

  • Added brackets around user code for cue pre-process
  • Fixed evaluation report not to end up as completed when all results are ended with status error
  • Fixed invalid temp file name for cue export executor

Related issue

Closes #344 UI#901

[#348] Modify the ValidationResult object to save in the database​

Bug fixes

CueValidationTask return a list of result, one for each applied policy

Related issue

Closes #348

[#334] Valid cron expression is invalidated​

Bug fixes

Adds a custom converter to adjust cron expressions to the format cron4s uses

Related issue

Closes #334

[#340] CGP endless test launch​

Bug fixes

  • Fixed report being pending when an error occurs in an evaluation task
  • Fixed TransformStep error at start up time

Related issue

Closes #340

[#346] "fix creation breaking change policy"​

Bug fixes

Fix decode for breaking_change value.

Related issue

Closes #346

[#341] External policy unable to find resource-name config when testing​

New features and improvements

  • Removed table from the supported resource_type values, since it not supported yet

Breaking changes

All policies that have resource_type=table will be invalidated

Migration

Edit all policies with resource_type=table to resource_type=data_product or delete them

Related issue

Closes #341

[#318] Coordinator should send specific provisioner results to CGP​

New features and improvements

  • Implementation: scheduler task to send validation results to the cgp (it can be integrated in the validation workflow when EvaluatorReportPort.save() is ready and fully functional)
  • Modification: validation task results decoding methods are now grouped into a decoder object

Related issue

Closes #318

[#335] Fix policy deletion (update status API)​

Bug fixes

A policy can now be deleted regardless of the status it's in

Related issue

Closes #335

[#339] Remove "tag" doesn't work​

Bug fixes

Remove tag from policies

Related issue

Closes #339

[#308] CGP is allowing to evaluate a non-evaluable policy​

New features and improvements

  • Policy is now creating an EvaluationResult whenever a precondition fails (e.g. policy status does not allow for evaluation/testing)

Related issue

Closes #308

[#330] CGP - Refactor metrics management​

New features and improvements

refactoring metrics management

Related issue

Closes #330

[#336] CGP - execute selectors before preprocessing​

New features and improvements

  • Switched operands order of selectors and preprocessing in Evaluator

Related issue

Closes #336

[#323] Coordinator have to throw validation error if cue is not present​

New features and improvements

  • Refactored the method for decoding the CGP validation result
  • Aligned current validation results with the one returned by /v1/provisioningplan

Related issue

Closes #323

[#326] Fix policy new version API error​

New features and improvements

  • Release version API now returns 400 on validation error

Related issue

Closes #326

[#329] Mockito fails randomly during ci​

Bug fixes

fix mockito random fail

Related issue

Closes #329

[#327] Exclude non-evaluable policies from scheduled execution​

New features and improvements

The PolicySchedulingService doesn't schedule anymore (or unschedules, in case of policy updates) policies in a non-evaluable status.

Related issue

Closes #327

[#325] Refactor multi-jvm-tests​

New features and improvements

multi-jvm to single-jvm integration tests

Related issue

Closes #325

[#312] A grace or deprecated policy should only throw warnings​

New features and improvements

  • Added info, warning and error to the outcome of a EvaluationResult.
  • Outcome is chosen based on severity and the status of the Policy

Closes #312

[#316] The V4 migration script does not work​

New features and improvements

  • added relation to table evaluation_result to policy using policy_id
  • added relation to table evaluation_result to resource_type using resource_type field

Related issue

Closes #316

[#322] Evaluation result should have a generic free json field​

New features and improvements

added details json field to evaluation result

Related issue

Closes #322

[#315] Improve error message in create/edit policy​

New features and improvements

  • Modified policy lifecycle error handling, to give more detailed errors
  • Modified evaluation report lifecycle, to give more detailed and accurate errors

Related issue

Closes #315

[#305] Return an error in case of disabled data-deletion configuration​

New features and improvements

Return a validation error instead of defaulting the removeData flag to false, when the data-deletion configuration is not enabled

Related issue

Closes #305

[#299] CGP Cron Scheduler - Persist scheduled jobs in the database​

New features and improvements

  • Modification: scheduled jobs are now persisted in a database table
  • Implementation: cron jobs table and repository
  • Implementation: cron job specification domain concept
  • Implementation: policy execution job domain concept
  • Implementation: local cron job factory
  • Implementation: reload-on-start cron policies configuration
  • Implementation: reload cron policies endpoint

Related issue

Closes #299

[#309] Return whole policy object on creation​

New features and improvements

  • Modified the return of the POST /policies and PUT /policies/{policyId} endpoints to return the whole newly created policy

Migration

  • Modify the parsing of the response of the endpoints to adjust to the new returned schema

Related issue

Closes #309

[#321] the configuration name should be 'reference.conf'​

Closes #321

[#307] Push Evaluation Report is not persisting in the db​

Bug fixes

  • Called upsert method at the end of the save method
  • Refactored save method

Related issue

Closes #307

[#302] CGP add preprocess into policy​

New features and improvements

Preprocessing configuration for policies

Related issue

Closes #302

[#311] Fixed state transition​

Bug fixes

  • Fixed state transition from disabled to enabled.

Closes #311

[#262] Initialize a Database seeding script for local testing​

New features and improvements

  • Created a database seeding SQL script for local testing, according to migration V15
  • Created a Postman collection for CGP endpoints

Bug fixes

  • Fixed a bug where filtering by environments would always return an empty list of policies
  • Fixed a bug where the outcome enumeration was all caps in some parts and in lowercase in others, causing conflicts.

Related issue

Closes #262

[#314] resource_type doesn't have a foreign key relationship in db​

New features and improvements

Policy table has a reference to resource_type table

Related issue

Closes #314

[#303] Missing cron expression field in PolicyCreationModel​

Bug fixes

It is now possible to pass a cron expression as an optional filed in the PolicyCreationModel

Related issue

Closes #303

[#313] A grace policy shouldn't be editable​

Bug fixes

update forbidden for policies in status "grace"

Related issue

Closes #313

[#306] Missing Validation task action mapping​

Bug fixes

Added missing mapping for VALIDATION plans

Related issue

Closes #306

[#297] Insert the version field in the evaluation_report table​

New features and improvements

  • Added result_filter migration
  • Changed "enrichWithName" method in order to include the version
  • Fetched resource-filter and resource-name from configuration

Related issue

Closes #297

[#288] Passive policies​

New features and improvements

  • Added passive policies in the domain layer.
  • Added a check for the validity of the passive policy(interactionType and timing fields).
  • Evaluation a passive policy returns an error.

Related issue

Closes #288

[289] Fix CGP selectors run by evaluator​

Bug fixes

Policy selectors are now always being run by the Evaluator

Related issue

Closes #289

[#304] Helm chart push to registry​

New features and improvements

Updated ci pipeline to push Helm chart to Gitlab registry

Related issue

Closes #304

[#301] CGP - create interface specification for external services​

New features and improvements

Interface specification for perimeter resolver services and external policies providers

Related issue

Closes #301

[#222] Switch coordinator validation to new CGP​

New features and improvements

  • Introduced cgp.evaluate for the "Launch test"
  • Introduced migration script for migrating old validations_policies to cgp
  • Added waitForCompletion method while executing CueValidationTask

Related issue

Closes #222, #273

[#281] Application and domain model names are conflicting​

New features and improvements

  • Created a new ADR to address the naming convention of the CGP application and domain layer models
  • Modified the application layer models to follow the ADR decision
  • Separated API Service implementation in two classes, one for coordinator and one for governance platform

Breaking changes

  • All of the models will now follow the naming convention, so most of the class names will change accordingly (specially the ones defined in the interface specification)
  • New API for CGP will have the new "GovernancePlatform" tag and the implementation will be done in the GovernancePlatformApiServiceImpl rather than the provisioning coordinator

Migration

When developing, have in mind the name change and work with the new names. When developing new classes in the future, follow the guidelines given in the ADR

Related issue

Closes #281

[#290] Evaluate error trying to decode value on failed cursor​

New features and improvements

  • Evaluation result can now contain optionally have a resource
  • Evaluation result can assume status NO_RESOURCE

Bug fixes

  • Fixed a bug causing evaluation report to be on status Complete, while still having pending results

Related issue

Closes #290

[#263] CGP scheduler should accept environment field​

New features and improvements

The policy scheduler now schedules a cron policy in all the policy's environments

Related issue

Closes #263

[#284] Dependencies' vulnerabilities in witboost​

New features and improvements

Changed the Scala version and some dependencies versions to resolve some of the major known vulnerabilities of the code. The code was updated accordingly to remove deprecations.

A known vulnerability linked to the snakeyaml library was not solved since circe and jackson did not release a fixed version yet.

Related issue

Closes #284

[#293] CGP resource types upsert is not working​

Bug fixes

upsert resource type now works

Related issue

Closes #293

[#294] Metric policies are saved with wrong executionStrategy​

Bug fixes

Metric policy has now the correct execution strategy

Related issue

Closes #294

[#267] Modify the update API request​

New features and improvements

  • Updated the interface-specification and consequently all the affected files to include all the expected policy fields for the update policy API.

Related issue

Closes #267

[#286] Review Helm Chart for Best Practices​

New features and improvements

refactored Helm chart:

  • added securityContext spec (optional)
  • added resources spec (optional)
  • added readinessProbe spec (optional)
  • added livenessProbe spec (optional)
  • added recommended labels
  • removed service account token auto-mount

Related issue

Closes #286

[#292] fix pipeline​

Bug fixes

  • Added coalesce

Related issue

Closes #292

[#256] send to specific provisioner the previous descriptor (with additional logic)​

New features and improvements

  • Modified: Scheduler soft dependencies to make them tolerant to NotStarted parents
  • Modified: ProvisioningOperationStatePersistenceTask in order to merge deploy info in case of not started provisioning tasks
  • Implemented: partial undeploy
  • Implemented: deploy cleanup tasks
  • Implemented: 'removeData' field and 'data-deletion' configuration

Related issue

Closes #256

[#287] Add the version field in the execution_plan table​

New features and improvements

  • Added version field in execution_plan table
  • Modified CRUD operations for version field
  • Modified /v1/provisioningplan API in order to include version filter

Related issue

Closes #287

[#258] Check field missing and add implementation logic​

New features and improvements

  • Persisted context, timing and interaction type attributes for the policy, integrating them in all the lifecycle layers
  • When retrieving the policies to schedule, only the policies with Timing runtime will be retrieved
  • When retrieving the policies to evaluate a resource, only the policies with the Timing deployment will be retrieved

Breaking changes

  • The Policy classes and companion objects now accept the new attribute values
  • Policies with runtime timing now must contain a cron expression to be valid, and deployment policies must not have one

Related issue

Closes #258

[#279] CGP async evaluator​

New features and improvements

  • Evaluations are now performed asynchronously

Related issue

Closes #279

[#285] CGP - add selectors field​

New features and improvements

Persisted selectors

Related issue

Closes #285

[#283] Paginate GET v1/provisioningplan​

New features and improvements

Added optional offset, limit and ordering parameters to v1/provisioningplan get method. Default ordering is descending, if either offset or limit (or both) is left out, pagination is not applied.

Related issue

Closes #283

[#261] Initialize a Postman collection​

New features and improvements

Initialized coordinator's postman collection

Related issue

Closes #261

[#282] CGP - add tags filters​

New features and improvements

Tag support for policies

Related issue

Closes #282

[#280] CGP - add status column on evaluation result​

New features and improvements

Changed the implementation of EvaluationResult outcome. Values are ok, error and pending, before they were ok, info, warn, error. Added an outcome field to the evaluation_result table.

Closes #280

[#278] Bug on Policy evaluate post API​

Bug fixes

  • remove hard coded environment in ´getResources´ method
  • used the yaml parser to parse the content of the perimeterResolver response

Related issue

Closes #278

[#270] parametric async validate​

New features and improvements

Configurable async validate during deployment

Related issue

Closes #270

[#254] Evaluator results and reports should have an environment field​

New features and improvements

Added the environment field to the evaluation report, to all layers(infra, domain, api)

Related issue

Closes #254

[#275] improve tests​

New features and improvements

Improved overall testing

Related issue

Closes #275

[#276] Improvement Test​

New features and improvements

This PR simplifies test maintainability.

A builder was created for the instantiated CuePolicy class. In this way you will have a centralized place where the CuePolicy is created and in case of modification of the field you have only one point to act

To create a instance of CuePolicy:

    val cuePolicy = CuePolicyFaker().build()

To create a instance of CuePolicy with specific value for some fields:

    CuePolicyFaker()
      .withPolicyId("policyId-5")
      .withStatus(Status.draft)
      .withEnvironments(List(devEnv, prodEnv))
      .build()

Related issue

Closes #276

[#274] Create a mock of a Data Catalog​

New features and improvements

  • Data Catalog plugin now has a launchable mock.

Related issue

Closes #274

[#264] CGP selectors should be run by evaluator​

New features and improvements

  • Added a filtering layer on the resource perimeter resolver

Related issue

Closes #264

[#249] CGP - Retrieve evaluation report and results​

New features and improvements

  • Save evaluation results now accepts a list of results
  • Retrieve evaluation report and its results by id
  • Retrieve all evaluation reports
  • Retrieve all evaluation reports related to a single policy given the policy id

Related issue

Closes #249

[#266] Modify the creation API request​

New features and improvements

  • Modified create API

Related issue

Closes #266

[#272] New helm chart​

New features and improvements

This MR adds a new helm chart, which will be dependency of the umbrella chart

Related issue

Closes #272

[#271] Resolve "Test Policy"​

New features and improvements

This MR Contains a series of improvements and fixes to successfully terminate an endToEnd test to evaluate a policy on deployed resources present in the system.

Breaking changes

  • The API /v1/resolve require a new query params environment

Migration

  • The key resource-types.[].resolver-configuration.query-format is no longer required in the coordinator configuration

Related issue

Closes #271

[#257] Breaking change policy uses environment​

Related issue

Closes #257

[#268] cgp - refactoring mapping application layer -> domain layer​

New features and improvements

Better mapping between layers

Related issue

Closes #268

[#221] CGP - metrics management​

New features and improvements

  • metrics management
  • external executor

Related issue

Closes #221

[#265] Resource resolver should send a proper json on not found​

Bug fixes

  • Now the resolver receives a proper json when fetching resources from external endpoint

Related issue

Closes #265

[#245] CGP - create policy test endpoint​

New features and improvements

  • Added endpoint for testing policies /v1/governance-platform/policies/{id}/evaluate. will return a report id

Bug fixes

  • Fixed bug in create policy and select environment by policy id. join key was wrong.

Related issue

Closes #245

[#217] CGP - scheduled policies​

New features and improvements

  • Local cron job scheduler based on fs2-cron library
  • Policy scheduling service

Related issue

Closes #217

[#252] remove policyinfrastructureport​

Related issue

Closes #252

[#228] CGP - Define filter​

New features and improvements

  • Added filters to policy query

Related issue

Closes #228

[#219] CGP - breaking change discovery​

New features and improvements

Breaking change policies

Related issue

Closes #219

[#250] Add policyId relation between evaluation_result and policy table​

Bug fixes

  • Added policy_id foreign key relation

Related issue

Closes #250

[#235] CGP Policies can be filtered by environment​

New features and improvements

  • Added environment's tables in CGP module
  • Added repositories relative to the above tables
  • Modified query for creation and update in order to include the environments field
  • Update the interface-specification in order to include the environments in the DTOs
  • Added seeding method for environment table

Related issue

Closes #235

[#239] Deployment unit​

New features and improvements

The coordinator now can deploy single components

Related issue

Closes #239

[#248] Fix environment override in configuration​

Related issue

Closes #248

[#248] CGP transactor not started in coordinator main​

Bug fixes

The cgp related tables are now created

Related issue

Closes #248

[#246] Async validation must return a result also with FAILED status​

Bug fixes

  • Fixed decodeAsyncValidation method

Related issue

Closes #246

[#218] CGP - preprocessing​

New features and improvements

  • Added TransformStep trait, which allows to supply a specific resourceType a list of transformation steps
  • Added CueTransformStep implementation to apply cue export functionality to descriptors
  • Added CueExecutor as a common ground executor for Cue scripts, both for eval and export commands

Related issue

Closes #218

[#236] Enable the functionality to release a new policy version​

Related issue

Closes #236

[#241] Unit Test Bug on Windows - SchedulerV2Spec​

Bug fixes

Fixed non-deterministically failing SchedulerV2Spec tests

Related issue

Closes #241

[#207] Send to specific provisioner it's own previous completed descriptor​

New features and improvements

When calling the provision/unprovison endpoint of a specific provisioner, the coordinator will now send the latest enriched (with provisioning info) data product descriptor, resulting from the last (un)provisioning operation (if any)

Related issue

Closes #207

New features and improvements

  • Added check for existence of resource type when creating or updating a Policy or an EvaluationReport
  • Improved error handling in policy lifecycle

Related issue

Closes #243

[#244] Modify the specific provisioners endpoints for async validation​

Bug fixes

  • Added missing v2 prefix to getStatus endpoint of async validation.

Related issue

Closes #244

[#229] CGP - push validation result use case​

New features and improvements

  • New endpoint to push result report

Related issue

Closes #229

[#242] Remove old scheduler implementation​

Bug fixes

Removed old scheduler

Related issue

Closes #242

[#231] CGP - listing resource types​

New features and improvements

  • Introduced new API GET /resource-types to query the existing resource types
  • Modified the structure of the API endpoints to be more RESTful friendly
  • Fixed some errors

Breaking changes

  • New API endpoint names

Migration

  • Check the new interface specification and modify the API URLs of the calls accordingly

Related issue

Closes #231

[#227] CGP - Modify policy model - add id​

Related issue

Closes #227

[#213] Include the status field for SpecificProvisionerHttpTask​

New features and improvements

  • Added status field for async validations
  • Added v1 and v2 to the requestPaths

Related issue

Closes #213

[#240] hotfix: Bug - UpdateACL workflow​

Bug fixes

Fixed some minor misalignments between v1 tasks and v2 ones.

Related issue

Closes #240

[#226] CGP - Logic of policy fetch is incorrect​

Bug fixes

  • Lifecycle port now correctly handles offset and limit.
  • Lifecycle port now correctly retrieves policies (i.e. it filters out policies which are deleted from the result set)

Related issue

Closes #226

[#205] Store the complete provisioning result​

New features and improvements

  • Implementations for storing a complete aggregate of the provisioning results returned by each specific provisioner
  • Migration from v1 tasks to v2 tasks

Related issue

Closes #205, #233, #220

[#238] configure resource type definition​

New features and improvements

  • Refactored ResolverConfiguration to integrate it with ResourceType

Related issue

Closes #238

[#232] CGP - Cue fields validation not done​

New features and improvements

  • Added BetterEnum trait to handle case insensitive serialization and deserialization of enums to/from strings

Bug fixes

  • Cue policies now require that at least an empty cue script is passed.

Related issue

Closes #232

[#237] Master branch build docker push fails​

Bug fixes

Disable the push image sbt task for the cgp module

Related issue

Closes #237

[#164] Add Coordinator flow documentation​

Related issue

Closes #164

New features and improvements

  • General restyling of the README
  • Removal of useless sections

Bug fixes

  • Fixed broken swagger link

Related issue

Closes #195

[#224] CGP domain and infra layer​

Features

  • Added domain layer including: evaluator, resolver, lifecycle manager
  • Added infrastructure layer: evaluator adapters, resolver adapters, lifecycle manager adapters

Closes #193 #194 #191 #189 #224

[#204] 1 to N relationship between template and specific provisioner​

New features and improvements

  • Handled sub components relation in deploy, undeploy, validate and update ACL

Related issue

Closes #204

[#216] Refactor config external files management​

Breaking changes

This MR changes the way configs and cert are managed

Related issue

Closes #216

[#214] Bump helm chart version​

New features and improvements

This MR bumps the helm chart version to include external secrets

Related issue

Closes #214

[#212] Fix async validation getStatus bug​

Bug fixes

  • Added check for data product and data catalog level provisioning
  • Added a test for the getExecutionPlanById

Related issue

Closes #212

[#208] Cue latest version vet command does not support useful functionalities​

Bug fixes

  • set the cue version to 0.4.3 in every docker image used
  • changed cue command, from "vet" to "eval"

Related issue

Closes #208

[#187] Validate as an async endpoint​

New features and improvements- Introducing the new Scheduler

  • Creating the async validation plan with the new scheduler
  • Exposed two APIs in the coordinator's interface-specification: POST /v2/validate and GET /v2/validate/{token}/status
  • Exposed two APIs in the specific provisioner's interface-specification: POST /v2/validate and GET /v2/validate/{token}/status
  • Renamed from DeployInfo to Info in the interface-specification of the coordinator and specific provisioner

Related issue

Closes #187 #151

[#201] Migrate from vault to external secrets​

New features and improvements

This MR switches the secret backend from vault to external secrets.

Related issue

Closes #201

[#202] Refactor task creation to make explicit when a task will be executed​

New features and improvements

  • Refactored - for better clarity - the task creation logic in the provisioning and validation workflows
  • Added tests to cover the behaviors of the following application configuration flags:
    • data-product-level-provisioning
    • data-catalog-provisioning

Related issue

Closes #202

[#203] Add Windows + WSL + VSCode instructions in the README​

New features and improvements

  • README updated with instructions for the setup of the local dev environment on Windows with WSL and VSCode

Related issue

Closes #203

[#196] Specific Provisioner Documentation​

New features and improvements

Improved descriptions in the OpenAPI interface specification for the specific provisioner

Related issue

Closes #196

Marketplace Plugin​

Changelog v1.0.0​

[#62] Remove the unused publish endpoint​

New features and improvements

Removed the /v1/publish endpoint and its references

Related issue

Closes #62

[#61] fix Version generator script​

New features and improvements

Added Checks for symbols at the end of the string and removed them.

Bug fixes

Cash of deployment on dev

Related issue

Closes #61

[#58] Fix missing bash command for deployment​

Bug fixes

Added a RUN step in the dockerfile to install bash.

Related issue

Closes #58

[#56] Dependencies' vulnerabilities in witboost​

New features and improvements

Changed the Scala version and some dependencies versions to resolve some of the major known vulnerabilities of the code.

Changed also the base image to use an alpine-based one instead of the ubuntu-based one.

A known vulnerability linked to the snakeyaml library was not solved since circe and jackson did not release a fixed version yet.

Related issue

Closes #56

[#54] Review Helm Chart for Best Practices​

New features and improvements

refactored Helm chart:

  • added securityContext spec (optional)
  • added resources spec (optional)
  • added readinessProbe spec (optional)
  • added livenessProbe spec (optional)
  • added recommended labels
  • removed service account token auto-mount

Related issue

Closes #54

[#53] New helm chart​

New features and improvements

This MR adds a new helm chart that will be used by the umbrella chart

Related issue

Closes #53

[#52] Refactor external files management​

Breaking changes

This MR changes how configs are injected. All external configs are placed in the same configmap.

Related issue

Closes #52

[#38] Error when Observability is defined for a Data Product​

New features and improvements

Add 'specific' field in the descriptor on Hasura

Related issue

Closes #38

[#49] Migrate from vault to external secrets​

New features and improvements

This MR introduces external secrets backed by Gitlab secret manager

Related issue

Closes #49