Skip to main content

Tech Adapters

Tech Adapters (formerly known as Specific Provisioners or Infrastructure Templates), are microservices responsible for the validation and provisioning of components. In Witboost, provisioning refers to the creation of a specific set of resources on a target technology or service which represent the functional representation and physical implementation of a component.

In order to support provisioning of components on a new technology, a new Tech Adapter microservice compliant with the Technology Adapter API must be created and registered. At a high level, a Tech Adapter provides one endpoint for each major action supported by Witboost, like validation, provisioning, resource import (reverse provisioning), etc.

Successful provision of components of a System, like a Data Product, results in a publication of all its metadata (contained in the descriptor) on the Witboost Marketplace, rendering them available to all consumers. In the following section, we specify how additional fields returned by the Tech Adapter results can be shown in the Marketplace UI pages.

Technology Adapter API details​

Validation and Provisioning​

Provision, and its mirror operation unprovision, are the core tasks of a Tech Adapter. Taking the whole descriptor as an input, its purpose is to extract the relevant information written both by the user and the template developer, and create the appropriate resources that define a specific component in a target technology. Some examples, but not restricted to, of this can be:

  • A Databricks pipeline to represent a Workload
  • A Snowflake view to expose data as an Output Port
  • An S3 bucket to store data to represent a Storage Area
  • A DAG orchestration workflow on Airflow representing a Workload
  • A Kafka topic exposing events as an Output Port
  • and much more.

The specifics on the implementation depend on the target technology, the chosen programming language, and the details of the business logic of your organization. As long as the Tech Adapter respects the API specification, you're free to do whatever necessary.

note

Check out our Python Scaffold and our Java Scaffold: two useful starting points to develop your Tech Adapters. They provide out-of-the-box features commont to all Tech Adapters, like the API layer, parsing logic and more to start building your business logic on top of it.

Provision can be either a synchronous or an asynchronous operation. The decision whether you should use one or another is governed based on the estimated elapsed time of the whole provision task: Each provision operation has a timeout that, albeit configurable, shouldn't be too high to avoid network timeout and to maintain reliability of the application. And since many of the channels to interact with target technologies or services are time-consuming (REST APIs, JDBC connections, etc.), asynchronous provision is heavily recommended on cases where these interaction span a long time. For this we recommend a baseline of 30 seconds: If the tech adapter is expected to consistenly take on average 30 seconds or more to finish its work, asynchronous provision should be used. On the other hand, if operations are quick, a basic synchronous provision is enough.

What defines whether a provision operation is synchronous or asynchronous is the response status and body you return on the /v1/provision endpoint:

  • Synchronous: Response status 200 OK, response body contains the Provisioning Status with status COMPLETED or FAILED and the provision information
  • Asynchronous: Response status 202 Accepted, response body contains a plain text including a string token that Witboost will use to poll for completion

When asynchronous provision is used and the Tech Adapter returns a 202 with a token, Witboost will use this token to regurlarly poll the /v1/provision/{token}/status endpoint of your Tech Adapter. The response of this endpoint is the same to the synchronous provision, but could also return a RUNNING status, meaning the operation is still in course.

The way this asynchronous behaviour is implemented depends entirely on the programming language and used framework. You could have an internal processing cycle or thread pool that processes the requests, or if your provisioning logic consists of polling external endpoints, you could think to synchronize this polling to the one performed by Witboost. Regardless of the choice, the Tech Adapter must still be ready to accept other requests while the asynchronous provision task is executing, as well as be able to response to the status poll for asynchronous operations which are not yet completed, and more.

warning

The same behaviour is true for the validation endpoint. However, instead of using the same endpoint with two different response status codes, validation is implemented as two different endpoints /v1/validate (synchronous) and /v2/validate (asynchronous), and as such it is also necessary to configure Witboost to work with asynchronous endpoints.

Witboost doesn't support a mix of synchronous and asynchronous validation operations on the same installation. If you configure validation as asynchronous, all your Tech Adapters must implement asynchronous validation.

For more information regarding the technical details of the endpoint, check out the Technology Adapter API specification. You can also check the Witboost Starter Kit to see real-life examples of developed Tech Adapters.

For the Witboost user, the visual experience on the Deployment panel of synchronous and asynchronous provision is the same, since it's Witboost to internally poll for completion on asynchronous tasks, updating the provisioning status on the Deployment page accordingly and informing the user in the process of updates received from the Tech Adapter.

Provisioning outcome​

The Provisioning Status returned by the provisioning methods of the Specific Provisioners contains an info field, which allows Technology Adapter developers to add additional details (like URLs, descriptions, etc). The structure of the status returned is similar to:

{
    "status": "RUNNING" | "COMPLETED" | "FAILED",
    "info": {
        "publicInfo": {},
        "privateInfo": {}
    }
}

As you can see, info is a field that can contain different public and private details. This data can be used to show additional information to consumers in the Marketplace by storing them in the publicInfo object using a structure explained further below; while all the values stored in the privateInfo will be stored in the deployed descriptor, but will not be shown in the Marketplace UI.

warning

For Data Contract Guardian components, the provisioning outcome must have the "info" field either set to null, or to merge the "info" field present as part of the input descriptor of the provisioned component with other values you'd like to store on this field. This is crucial since the "info" field for Data Contract Guardian components includes information regarding the target guardian policy used by other components in Witboost. For more information check the Guardian provisioning section

Information passed to dependent components​

During the provisioning process, dependencies can also define how provision information is passed among components, where downstreams components receive as input the outcome of dependent components that have finished provisioning. For example, if a component C1 depends on a component C2, component C1 will wait on the completion of provision of component C2 before starting, as explained on the Provisioning architecture. On top of this, component C1 will also receive as part of the input descriptor, the info field of component C2 stored under the component C2 section of the descriptor. This way, information stored by an upstream provisioning task can be used by dependent components that may require it to perform their own provisioning.

As an example, this is what component C1 would receive as its input Component Descriptor after C2 successfully finished provisioning:

componentIdToProvision: C1
dataProduct:
  components:
    - id: C2
      info:
        privateInfo: { ... }
        publicInfo: { ... }
      ...
    - id: C1
      dependsOn:
        - C2
      ...
  ...
Marketplace publication​

As explained above, you can return information on the provisioning outcome to be shown in the Marketplace by storing the inside the publicInfo field. There is no limit to how many values can be set inside the Public Info section of the provision status, but only the ones compliant with the following specification will be rendered in the "Technical Information" card of the Marketplace module. Invalid data is ignored and if there is no valid data to be shown, the "Technical Information" card will not be displayed.

The info must be an object that contains 2 objects: publicInfo and privateInfo. These objects can contain other objects, one for each field you want to represent. As already explained, only the valid ones from the publicInfo object will be displayed. Each object is related to a specific key of the publicInfo object and must contain the following fields: type, label and value. An optional href field is also supported for links. The values currently supported for the type field are string and date.

The following examples show how to write the field objects for each supported type:

String

To show in the UI a string field with the label "MyString" and value "MyValue", just return the following "publicInfo" object:

{
    ...
    "info": {
        "publicInfo": {
            "aString": {
                "type": "string",
                "label": "MyString",
                "value": "MyValue"
            }
        },
        "privateInfo": {}
    }
}

Links

To show in the UI an anchor field with the label "MyLink" and value "MyValue" that once clicked forwards the user to "http://my.remote.url", just return the following "publicInfo" object:

{
    ...
    "info": {
        "publicInfo": {
            "aLink": {
                "type": "string",
                "label": "MyLink",
                "value": "MyValue",
                "href": "http://my.remote.url"
            }
        },
        "privateInfo": {}
    }
}

As you can see, there is no need to specify a different type, since the clickable link rendering depends on the presence of the optional href field.

Date

To show in the UI a date field with the label "MyDate", just return the following "publicInfo" object:

{
    ...
    "info": {
        "publicInfo": {
            "aDate": {
                "type": "date",
                "label": "MyDate",
                "value": "2011-12-03T10:15:30Z" // ISO UTC INSTANT
            }
        },
        "privateInfo": {}
    }
}

The value must be in ISO UTC instant format: yyyy-MM-ddThh:mm:ssZ.

Upsert of the Access Control List​

The tech adapter is also in charge of updating the Access Control List for consumable, shoppable components through the /v1/updateacl endpoint. This action is always synchronous. As part of the request, Witboost supplies the input descriptor under provisionInfo.request, and the list of identities to be granted access. Witboost always supplies the full list of identities even if some of them already have access to the underlying infrastructural resource, so the Tech Adapter must:

  • Grant access to the new identities present on the request but that have not yet access granted on the target technology.
  • Revoke access to the identities who currently have access granted on the target technology but are not present anymore in the input list.

This way, both lists, the one managed by Witboost and the identities with access in the target technology, are aligned.

Import of existing resources (Reverse Provisioning)​

Reverse Provisioning is the action on which resources present on a target technology can be imported into the Witboost entity lifecycle. This way, instead of creating new resources from a template, Witboost stores the information of an existing resource in the component descriptor, allowing the information to be governed by Witboost and aligning it to the provisioning lifecycle. In fact, after a reverse provisioning operation, the next provision operations will attempt to (re)create the imported resource along with all other components by calling the configured Tech Adapter.

Details on how to implement the Reverse Provisioning endpoint can be found here.

Provision task termination​

Witboost provides the possibility for users to early terminate a provision/unprovision operation via a button in the Deployment panel. In order for this button to have an effect on the provisioning in your Tech Adapter, the service must be able to handle the DELETE operation on your provision endpoint.

info

This operation is only available for asynchronous provision and unprovision operations, as it uses the provision status token to identify the task to terminate.

Internally, your Tech Adapter must be able to receives these requests, identify the operation being performed and perform the necessary tasks to terminate the action, like stopping the execution thread associated to said operation. Be mindful that, as with other asynchronous operations, the state of the Tech Adapter must contain the necessary operation information and eventually be available to all potential instances of the service (e.g. if you're handling asynchronicity via service replication). Since this operation is quite complex, it is not a mandatory endpoint to implement and you can set the response status to 501 Not Implemented as defined in the endpoint specification to signal Witboost that the Tech Adapter doesn't support this feature.

Testing a Tech Adapter​

As part of the Tech Adapter development, you might want to test the service endpoints to ensure the business logic is working correctly against a series of test descriptors. To do this, it's not necessary to have the Tech Adapter running as part of the Witboost installation. You can easily test them locally using a tool like cURL or Postman to directly send requests to the Tech Adapter endpoints without the need of deploying them on your remote cluster.

Testing provision/unprovision​

Provision and unprovision can be tested by calling the /v1/provision and /v1/unprovision endpoints. As input you can create an input descriptor and then send it as part of the request body. The shape of the descriptor is explained here, which translates to the following request body:

{
  "descriptorKind": "COMPONENT_DESCRIPTOR",
  "descriptor": "<descriptor>",
  "removeData": false
}

The descriptor is the main field, which, depending on the descriptorKind field, may represent a Data Product Descriptor (DATAPRODUCT_DESCRIPTOR) or a Component Descriptor (COMPONENT_DESCRIPTOR). The first represents a descriptor as shown on the Edit and Test page stored as a JSON String; it is used on data product level provisioning. The second is the most common and represents the provision of a specific component. This is a JSON String containing the Data Product descriptor plus the component ID to be provisioned stored as YAML as follows:

componentIdToProvision: urn:dmb:cmp:domain:dp:0:component
dataProduct:
  id: ...
  components: [ ... ]
  ... <data_product_descriptor>

On both cases, you can generate the JSON String by using A JSON Stringify tools like this one and dumping the descriptor in it.

If your Tech Adapter requires it, you can also include the latest enriched descriptor as part of the request:

{
  "descriptorKind": "COMPONENT_DESCRIPTOR",
  "descriptor": "<descriptor>",
  "removeData": false,
  "latestEnrichedDescriptor": "<latest published descriptor enriched with provision info>"
}

The response of this endpoint depends on whether you have implemented it as a synchronous or asynchronous endpoint. In case of asynchronous endpoints, you can use the returned token to call the /v1/provision/{token}/status endpoint to retrieve the outcome of the operation.

Testing validation​

The validation endpoint, as explained above, can be implemented either as a synchronous or an asynchronous operation. For the first case, you must implement and test the endpoint /v1/validate. On the asynchronous case, you must implement and test the endpoint /v2/validate.

The request body for this operation is the same as the provisioning operations, while the outcome will either store the outcome of the validation for synchronous validation, or the token to retrieve the validation outcome via the /v2/validate/{token}/status.

Testing Access Control List update​

The Update ACL endpoint receives the list of subjects to grant access to the target technology. Depending on the Access Control Mode, this endpoint will be called during the access grant of a user on the Marketplace after the System owner or the delegated access flow have accepted the grant request.

The request body is tightly related to the provision body, where the content of the provision descriptor field will be sent as part of the provisionInfo.request of the update ACL, but including as well a list of users and groups with granted access:

{
  "refs": ["user:john.doe_witboost.com", "group:dev_users"],
  "provisionInfo": {
    "request": "<descriptor>",
    "result": "<message sent as part of provisioning result>"
  }
}

In addition to this, if an Access Control Request Template is defined, an additional field can be present on the request including the values defined under the fields object of the template:

{
    "refs": ["user:john.doe_witboost.com", "group:dev_users"],
    "provisionInfo": {
        "request": "<descriptor>",
        "result": "<message sent as part of provisioning result>"
    },
    "accessControlFields": {
        "identities": ...,
        "motivation": ""
        ...
    }
}