Property graph
Overview
The Practice Shaper property graph is composed of:
- type nodes - vertices representing abstract concepts (classes)
- instance nodes - concrete entities each instantiating one of the available type nodes
- relations - they are used to define relationships between nodes. For example, an instance node will always express an
instanceOfrelation towards a type node
Type nodes and the relations among them are defined by the Platform Team when configuring a Data Landscape.
Instance nodes and their relationships with type nodes (and other instance nodes) are generated by end users as they interact with the platform, within the context of a data landscape.
The Platform Team can develop and publish a set of creation templates: visual wizards that guide users through the process of creating system or component instances. They ensure that relationships and constraints defined in the property graph are automatically enforced, thereby simplifying the process and hiding complexity.
If you are familiar with object-oriented programming, you can think of type nodes as classes that define relationships with other classes through their attributes. Instance nodes are the objects that instantiate these classes, adhering to the behaviors and attributes of their parent. Creation templates are akin to class constructors, guiding the creation process while ensuring compliance with predefined relationships and constraints.
Witboost catalog
Both type and instance nodes are registered as Witboost entities: their definition is provided by a catalog info YAML file (hosted in a Git repository), and they are part of the Witboost catalog.
The catalog serves as a centralized repository that manages ownership and metadata for all entities within Witboost. It is designed around the use of metadata YAML files, which are parsed, validated, aggregated and displayed in Witboost.
A sample graph
Before digging into the attributes of the different node types, let's look at a sample property graph, and try to understand it at high-level.
This is the same kind of graph the Platform Team can see, edit and customize from the Practice Shaper settings page.
Ideally, this graph should include system, component, and domain instance nodes as well, but for practical reasons, it only focuses on types and data landscapes.
From the graph above, we can infer that:
-
this Witboost instance supports 3 data landscapes: Data Lake, Data Mesh, Machine Learning
-
2 domain types are available: Business Domain, Department (more about domains in the following sections). A domain, instance of Business Domain type, can only be part of (subdomain of) another instance of Business Domain. The same is true for Department.
-
within the Data Lake data landscape:
- users can work on instances of the BI Project system type (i.e., each system in the Data Lake landscape is a BI Project)
- a BI Project can be part of an instance of a Business Domain
- a BI Project can include components of type Consumer View and components of type Dashboard
- a Dashboard component instance can only read from a Consumer View component instance (even if they are not in the same BI Project instance)
- a Consumer View component instance can only read from another Consumer View component instance (even if in different BI projects)
-
within the Data Mesh data landscape:
- users can work on instances of the Data Product system type (i.e., as by definition of Data Mesh, the unit of work is the data product)
- data products can be part of domains of type Business Domain
- data products can contain output ports, storages, schema components, control ports, and workloads
At this point, by looking at the graph, it should also be clear how the Machine Learning data landscape is structured. Notice how a ML Sandbox cannot belong to a Business Domain; instead, it must be defined within a domain of type Department. For instance, if your organization has Finance and Sales business domains, these domains can own and manage data products and BI projects, but not Machine Learning sandboxes. The responsibility for maintaining Machine Learning sandboxes probably lies with your Data Science department. In this example, Finance, Sales, and Data Science are domain instances.
As you can notice in the graph, some nodes may be marked by icons indicating specific attributes, such as consumable and shoppable, which we will explore further in the next sections.
Type nodes
Domain type
A domain is a logical boundary within an organization that groups related data, processes, and responsibilities. Domains are typically aligned with business units or areas of expertise and are intended to decentralize data ownership and management. Each domain is responsible for its own data projects, that are made available for consumption by other domains within the organization.
A domain type represents a class of domains.
Attributes
| Catalog Info property | Type | Required | Description | Example |
|---|---|---|---|---|
apiVersion | string | Yes | Fixed to witboost.com/v2 | |
kind | string | Yes | Fixed to DomainType | |
metadata.name | string | Yes | Unique entity name | business-domain |
metadata.displayName | string | No | Display name to show in the UI. When not provided,metadata.name will be displayed | Business Domain |
Relations
| Display name | Target kind | Catalog Info property | Type | Required | Description | Example |
|---|---|---|---|---|---|---|
| partOf | DomainType | spec.partOfDomain | string | No | Reference to a domain type (it can be reflexive). When provided, an instance of this domain type can optionally be marked as a subdomain of an instance belonging to the referenced domain type. If not defined, instances of this domain type can be subdomains of any registered domain instance, regardless of its domain type. | domaintype:default/business-domain |
System type
Attributes
| Catalog Info property | Type | Required | Description | Example |
|---|---|---|---|---|
apiVersion | string | Yes | Fixed to witboost.com/v2 | |
kind | string | Yes | Fixed to SystemType | |
metadata.name | string | Yes | Unique entity name | dataproduct |
metadata.displayName | string | No | Display name to show in the UI. When not provided,metadata.name will be displayed | Data Product |
metadata.pluralizedDisplayName | string | No | Pluralized display name to show in the UI. If not provided, the display name will be automatically pluralized when necessary | Data Products |
spec.resourceTypeId | string | Yes | Identifier of this class. It must be unique accross all the registered system and component types and must never be changed. This is a critical field as it will appear as the value of the kind field in the descriptor of instances of this type. Thus, it can be used by external systems (provisioners, computational governance platform, marketplace, data catalogs) to identify the specific resource type of the instance they are processing | dataproduct |
spec.resourceTypeConfig | object | No | Resource type configuration for the integration with the Computational Governance Platform. More on this in the dedicated section in the documentation for Platform Teams | |
spec.shoppable | boolean | No (default:false) | Instances of a shoppable type will be marked as shoppable by default, though they can override their parent's attribute. End users can request access to a shoppable system in the Witboost marketplace to get access to all its consumable interfaces (i.e., all the system's components and their subcomponents with the consumable flag enabled). More on shoppable and consumable attributes in the dedicated section of the documentation | |
spec.partiallyDeployableIn | list[string] | No | List of environments in which instances of this system type can be deployed partially (i.e., users can select which system's components to deploy). In environments that are not listed here, the system instance can only be deployed as a whole (i.e., as an atomic deployable unit) | ['development', 'qa'] |
Relations
| Display name | Target kind | Catalog Info property | Type | Required | Description | Example |
|---|---|---|---|---|---|---|
| belongsTo | Taxonomy | spec.belongsTo | string | Yes | Reference to a registered data landscape (aka taxonomy). Assigns this system type and all its related component types and their instances to one of the available data landscapes | taxonomy:default/data-mesh |
| partOf | DomainType | spec.partOfDomain | string | Yes | Reference to a registered domain type. Instances of this system type can only be part of a domain instance of the referenced domain type | domaintype:default/business-domain |
Component type
Attributes
| Catalog Info property | Type | Required | Description | Example |
|---|---|---|---|---|
apiVersion | string | Yes | Fixed to witboost.com/v2 | |
kind | string | Yes | Fixed to ComponentType | |
metadata.name | string | Yes | Unique entity name | outputport |
metadata.displayName | string | No | Display name to show in the UI. When not provided,metadata.name will be displayed | Output Port |
metadata.pluralizedDisplayName | string | No | Pluralized display name to show in the UI. If not provided, the display name will be automatically pluralized when necessary | Output Ports |
spec.resourceTypeId | string | Yes | Identifier of this class. It must be unique accross all the registered system and component types and must never be changed. This is a critical field as it will appear as the value of the kind field in the descriptor of instances of this type. Thus, it can be used by external systems (provisioners, computational governance platform, marketplace, data catalogs) to identify the specific resource type of the instance they are processing | outputport |
spec.consumable | boolean | No (default:false) | Instances of a consumable component type will be marked as consumable by default, though they can override their parent's attribute. A consumable component is ready to be transmitted, used, or processed, indicating that the output is in a format or state that can be readily consumed. The tech adapters of consumable components will be involved in ACL update operations | |
spec.shoppable | boolean | No (default:false) | Instances of a shoppable type will be marked as shoppable by default, though they can override their parent's attribute. End users can request access to a shoppable component in the Witboost marketplace to get access to all its consumable descendants (i.e., all its subcomponents or the component itself having the consumable flag enabled). More on shoppable and consumable attributes in the dedicated section of the documentation |
Relations
| Display name | Target kind | Catalog Info property | Type | Required | Description | Example |
|---|---|---|---|---|---|---|
| partOf | SystemType | spec.partOfSystem | string | Yes | Reference to a registered system type. Instances of this component type can only be part of a system instance of the referenced system type | systemtype:default/dataproduct |
| readsFrom | ComponentType | spec.readsFrom | list[string] | No | List of references to registered component types. They can be in a different data landscape. When provided, instances of this component type can read from consumable component instances of the referenced component types. If not provided, or empty, instances of this component type cannot read from any component instance. In both cases, component instances can always read from resource instances. | ["componenttype:default/outputport", "componenttype:default/consumerview"] |
Instance nodes
As explained above, nodes in the Practice Shaper are implemented as Witboost entities. The following sections describe the main attributes and relations of instance entities, which are relevant to the Practice Shaper.
Data Landscape (aka Taxonomy)
A Data Landscape can also be referred to as a Taxonomy because it involves the systematic classification and organization of various data elements and their relationships within an organization. Just as a biological taxonomy categorizes living organisms into hierarchical groups based on shared characteristics, a data taxonomy organizes data assets, their attributes, and their interconnections into a structured framework.
Attributes
| Catalog Info property | Type | Required | Description |
|---|---|---|---|
apiVersion | string | Yes | Fixed to witboost.com/v2 |
kind | string | Yes | Fixed to Taxonomy |
metadata.name | string | Yes | Unique entity name |
metadata.displayName | string | No | Display name to show in the UI. When not provided,metadata.name will be displayed |
spec.enabled | boolean | Yes | Whether this data landscape and all its underlying types and instances are enabled and accessible |
Domain
Attributes
| Catalog Info property | Type | Required | Description | Example |
|---|---|---|---|---|
apiVersion | string | Yes | Fixed to backstage.io/v1alpha1 | |
kind | string | Yes | Fixed to Domain | |
metadata.name | string | Yes | Unique entity name | |
spec.mesh.name | string | Yes | Display name to show in the UI |
Relations
| Display name | Target kind | Catalog Info property | Type | Required | Description | Example |
|---|---|---|---|---|---|---|
| instanceOf | DomainType | spec.instanceOf | string | No | Reference to the domain type this domain is instance of. If not provided, it will default to the domain type set under configuration practiceShaper.migration.defaultDomainTypeRef (default value: domaintype:default/business-domain) | domaintype:default/business-domain |
| partOf | Domain | spec.subDomainOf | string | Yes | metadata.name of another domain instance, parent of this domain.This relation could be constrained by the domain type of this instance (refer to documentation of the subDomainOf relation in the Domain type section) | parent-domain-name |
| ownedBy | User | Group | spec.owner | string | Yes | Reference to a user or group owner of this domain | group:my-group-name |
System
Attributes
| Catalog Info property | Type | Required | Description | Example |
|---|---|---|---|---|
apiVersion | string | Yes | Fixed to witboost.com/v1 | |
kind | string | Yes | Fixed to System | |
metadata.name | string | Yes | Unique entity name | |
spec.type | string | Yes | Resource type identifier. When spec.instanceOfis provided, this field is automatically filled with the value of the spec.resourceTypeId attribute of the system type this system is instance of | dataproduct |
spec.mesh.name | string | Yes | Display name to show in the UI | |
spec.mesh.version | string | Yes | A regular three elements version with major, minor and patch components | 1.0.0 |
spec.mesh.useCaseTemplateId | string | No | A Witboost URN of the template used to scaffold the system | urn:dmb:itm:aws-workload-dbt-transformation-template:0 |
spec.mesh.infrastructureTemplateId | string | No | A Witboost URN of the technology adapter that will handle the system's deployment | urn:dmb:itm:cdp-aws-dataproduct-provisioner:1 |
spec.mesh.shoppable | boolean | No | End users can request access to a shoppable system in the Witboost marketplace to get access to all its consumable interfaces (i.e., all the system's components and their subcomponents with the consumable flag enabled). More on shoppable and consumable attributes in the dedicated section of the documentation.If not provided, the value will be inherited from the spec.shoppable attribute of the system type this system is instance of | |
spec.mesh.developmentGroup | string | No | Deprecated - refer to spec.ownerReference to the group of the development team of the system | group:my-group-name |
spec.mesh.kind | string | No | Deprecated - refer to spec.typeResource type identifier. |
Relations
| Display name | Target kind | Catalog Info property | Type | Required | Description | Example |
|---|---|---|---|---|---|---|
| instanceOf | SystemType | spec.instanceOf | string | No | Reference to the system type this system is instance of. If not provided, it will be automatically filled with the reference to a system type having spec.resourceTypeId equals to the value of the spec.mesh.kind property of this system, or spec.type (if spec.mesh.kind is undefined) | systemtype:default/dataproduct |
| partOf | Domain | spec.domain | string | Yes | Reference to a domain instance this system belongs to. This relation is constrained by the system type of this instance (refer to the documentation of the partOfDomain relation in the System type section) | domain:domain-name |
| systemOwnedBy | User | Group | spec.mesh.projectOwner or spec.mesh.dataProductOwner (deprecated) | string | No | Reference to a user or group owner of this system | user:user-name |
| developedBy | User | Group | spec.owner | string | No | Reference to a user or group main developer of this system | user:user-name |
Component
Attributes
| Catalog Info property | Type | Required | Description | Example |
|---|---|---|---|---|
apiVersion | string | Yes | Fixed to witboost.com/v1 | |
kind | string | Yes | Fixed to Component | |
metadata.name | string | Yes | Unique entity name | |
spec.type | string | Yes | Resource type identifier. When spec.instanceOfis provided, this field is automatically filled with the value of the spec.resourceTypeId attribute of the component type this component is instance of | outputport |
spec.lifecycle | string | Yes | The lifecycle state of the component | experimental, production, deprecated |
spec.mesh.name | string | Yes | Display name to show in the UI | |
spec.mesh.version | string | Yes | A regular three elements version with major, minor and patch components | 1.0.0 |
spec.mesh.useCaseTemplateId | string | No | A Witboost URN of the template used to scaffold the component | urn:dmb:itm:aws-workload-dbt-transformation-template:0 |
spec.mesh.infrastructureTemplateId | string | No | A Witboost URN of the technology adapter that will handle the system's deployment | urn:dmb:itm:cdp-aws-op-provisioner:1 |
spec.mesh.dependsOn | list[string] | No | List of URNs of other components part of the same system, this component depends on | ["urn:dmb:cmp:marketing:system-a:0:storage-a", "urn:dmb:cmp:marketing:system-a:0:storage-b"] |
spec.mesh.shoppable | boolean | No | End users can request access to a shoppable component in the Witboost marketplace to get access to all its consumable descendants (i.e., all its subcomponents or the component itself having the consumable flag enabled). More on shoppable and consumable attributes in the dedicated section of the documentation.If not provided, the value will be inherited from the spec.shoppable attribute of the component type this component is instance of | |
spec.mesh.consumable | boolean | No | A consumable component is ready to be transmitted, used, or processed, indicating that the output is in a format or state that can be readily consumed. The tech adapters of consumable components will be involved in ACL update operations. If not provided, the value will be inherited from the spec.consumable attribute of the component type this component is instance of | |
spec.mesh.kind | string | No | Deprecated - refer to spec.typeResource type identifier. |
Relations
| Display name | Target kind | Catalog Info property | Type | Required | Description | Example |
|---|---|---|---|---|---|---|
| instanceOf | ComponentType | spec.instanceOf | string | No | Reference to the component type this component is instance of. If not provided, it will be automatically filled with the reference to a component type having spec.resourceTypeId equals to the value of the spec.mesh.kind property of this component, or spec.type (if spec.mesh.kind is undefined) | componenttype:default/outputport |
| partOf | System | spec.system | string | Yes | Reference to a system instance this component belongs to. This relation is constrained by the component type of this instance (refer to the documentation of the partOfSystem relation in the Component type section) | system:system-name |
| ownedBy | User | Group | spec.owner | string | Yes | Reference to a user or group owner of this component | user:user-name |
| readsFrom | Component | Resource | spec.mesh.readsFrom | list[string] | No | List of URNs of components or resources in Witboost read by this component. This relation is constrained by the component type of this instance (refer to the documentation of the readsFrom relation in the Component type section) | ["urn:dmb:cmp:marketing:system-b:0:storage-a", "urn:dmb:cmp:marketing:system-c:0:storage-b"] |
Resource
A resource represents an external system (e.g., an API, a database, a file) that is not managed within Witboost but can be accessed by its components. For instance, while a public dataset might not be owned by the organization using Witboost, the organization's components can still read data from it.
Attributes
| Catalog Info property | Type | Required | Description | Example |
|---|---|---|---|---|
apiVersion | string | Yes | Fixed to witboost.com/v1 | |
kind | string | Yes | Fixed to Resource | |
metadata.name | string | Yes | Unique entity name | |
spec.owner | string | Yes | Reference to a user or group owner (maintainer in Witboost) of this resource | user:user-name |
spec.type | string | Yes | Kind of resource | api, file, database |
spec.mesh.name | string | Yes | Display name to show in the UI |
Template
Attributes
| Catalog Info property | Type | Required | Description | Example |
|---|---|---|---|---|
apiVersion | string | Yes | Fixed to witboost.com/v1 | |
kind | string | Yes | Fixed to Template | |
metadata.name | string | Yes | Unique entity name | |
metadata.title | string | No | Display name to show in the UI | |
spec.owner | string | No | Name of a group, owner of this template | group-name |
Relations
| Display name | Target kind | Catalog Info property | Type | Required | Description | Example |
|---|---|---|---|---|---|---|
| generates | SystemType | ComponentType | spec.generates | string | Yes, if the template includes a fetch:template action | Reference to the system type or component type of the system/component instances generated by this template. This will be used to group templates in the UI, based on the generated types, and to automatically populate the spec.instanceOf property of the generated system or component instances | systemtype:default/dataproduct |
Errors
While processing the entities registered in the Witboost catalog, the Practice Shaper enforces compliance with the defined constraints.
As you already know, Witboost entities are backed by a catalog info file hosted on a Git repository. A pool of scheduled processors periodically read these files to update the attributes and relations of the related entities on Witboost.
When a processor detects a Practice Shaper violation, it will emit an error, that will be displayed in the entity page. For example:
By using the templates provided by your Platform Team, you will rarely encounter these types of errors, as they are designed to create entities that are fully compliant with the Practice Shaper.