Data Product information

Data Product visualizations

By accessing the Builder module a developer can see all the Data Products that are currently managed by the domain team in a single place; by opening their details we can easily access the source code directly (to open the repository just click the <> button) and even see the documentation (in the docs tab).

As a Data Product consumer, I want to see all the existing Data Products and ask access for to one of their output ports.

First of all, I can start my research by accessing two different pages: the Catalog and the Data Product Graph pages. The two pages contain pretty much the same information overall: a complete list of all the Data Products currently deployed through all the environments. They differ anyway in how they are meant to be used.

The data product catalog is a table that lists all the Data Products at the same level, independently of their relations and how they are connected. From the upper right corner, you can choose which environments you want to see. Please note that since a data product with a different major version is by all means a brand new data product, you can see different versions of the same data product for each environment simultaneously. When you are seeing all the environments at once in the catalog, you can also see in which environments a specific data product version is already deployed and ready to be used.

Here you can also filter and sort Data Products to look for the ones you are interested in. This is the page that should be used when a consumer is looking for a specific Data Product.

The Data Product Graph page contains all the deployed data products for each environment, but differs from the data product catalog since here a user has a broader view of the whole data mesh, and how the various data products are related to each other.

First of all this page organizes the data products into the hierarchy of domains that contains them (the big circles represent domains, while smaller circles represent the data products contained in them). This way a user can see which data products belong to the same domain, and how the domains are related to each other. The other dimension that this graph shows is the connection among all the various data products: which data products read and which are read, actually highlighting at first sight the ones that are extremely connected to the data mesh, and which are isolated and bring less value.

By selecting one deployed Data Product, you can see what output ports it is exposing, and the general details of the Data Product (just select a row of the Catalog or click on a circle representing a Data Product in the Dependencies page). By clicking the view icon on an output port, you can see all of its details, that are extracted from the metadata defined in the building phase.

In the Data Product page, you can also:

• ask questions regarding the Data Product and see the previously asked questions, and the associated answers (Refer to DP rating and info section)
• express a vote to rate the data product from 1 to 5. (Refer to DP rating and info section)
• ask to access one or more of the data product's output ports. (Refer to DP Access section)

Software Catalog

The two main Builder pages, My Data Products and Software Catalog are two views of the same underlying page.

The only difference among the two pages is that in the My Data Products page you do not have a filter that lets you select the type of entities to display (you can only see data products). From the Software Catalog page, you can see all the entities that you are entitled to see inside witboost, and you can change which entities you want to see by just selecting them in the Kind filter.

If you have access to all the entities, you can check all the entity types managed by Witboost, in particular:

• Domains: these are the data domains that compose the structure of the data mesh. All the data products are associated with one Domain that contains them.
• Data Products and Components: these are the building block of the data mesh. By selecting "Data Products" you visualize the same exact page as the "My Data Products" page. By selecting "Components" you see all the components, independently of their Data Product and Domain. If there is any error during the ingestion phase of an entity this will be shown as an error badge next to the data product. Users need to do the necessary updates in the repository and refresh the entity to fix the error.
• Templates: templates are treated as entities as well. All the templates that are shown on the Templates page are collected here.
• Users and Groups: all users and groups that are handled inside witboost, as they are extracted from the external identity provider (e.g. Active Directory or LDAP)
• Resources: these are external resources that are not handled by witboost (or the data mesh), but it can be useful to reference them in the metadata. You could for example register a Database resource and set in the metadata of a workload component that it reads from it.
• Releases: every time a new data product release is created, the corresponding entity is stored inside witboost. Here you can see a list of all the releases among all data products.
• Locations: every entity inside witboost is registered as a location. A location represents a remote repository that contains a catalog-info.yaml file. Every couple of minutes all the locations are fetched again to detect changes in the source files to update the related entities. Almost all of the entities described above have a location associated since they are defined externally in a remote repository.

From the Software Catalog page, there is also a button in the upper-right corner to create new entities from templates: it is just a shortcut for the Builder > Templates page.

Data Product details

When clicking on a Data Product from the My Data Products page, the data product details page is displayed.

From this page you have full control over the data product; using the previous image as a reference let's see what this page is composed of:

1. At the top of the page, you can see the Data Product name, and just below it the breadcrumbs that let you navigate more easily through the various pages.
2. The main tab is the Overview, where you can see the overall details of the data product. The other tabs will show other different aspects of the data product: documentation, edit and test, and deployment.
3. The About card contains general metadata regarding the data product, including the URN that uniquely identifies it. This is the URN that should be used in all the needed references inside the Data Product Descriptor.
4. From this card, there are also some utility buttons:

• the <> button opens the repository containing the source code of the data product
• the refresh button triggers a refresh of the entity inside Witboost by reading its location again

5. The Relations card shows how the data product entity is related to other entities. As introduced, everything is an entity, so in this card, you will see different kinds of relations, depending on the entities connected to it (owned by, part of, etc).

6. In the bottom part of the page, you can see all the components that are part of the data product divided into different tables based on their type. By clicking on their name you can open their details page as well. From this card, you also can quickly add new components.

7. The menu on the three dots on the top of the page will let you:

   • create a new version of the Data Product. This will create a new branch in all the repositories.
   • delete the entity by unregistering it.
   • open the Entity Inspector, where you can see other detailed information regarding the entity and the catalog-info.yaml (like a JSON and YAML summary)

8. By selecting the docs tab, the techdocs are loaded from the repository (in the "docs" folder) and rendered as a documentation page

9. The next tab is the Edit and Test, where you can perform operations on the data product like previewing its descriptor, testing it, and creating/updating its draft release.

10. The last tab is the Deployment tab, where you can check all the releases associated with the data product, and check the deployment status for all the environments.