OpenLineage (beta)

Follow these steps to start pushing your OpenLineage events to Select Star.

Before you start

To connect your Data Sources to Select Star, via OpenLineage, you will need to...

• prepare your OpenLineage job events according to the specifications

Select Star does not need any permissions for your underlying data sources or ETL tools, rather it relies on the events prepared and pushed by you.

Complete the following steps to connect OpenLineage to Select Star.

1. Create a new Data Source in Select Star

2. Configure OpenLineage Producer

3. Sync Metadata in Select Star

1. Create a new Data Source in Select Star

Go to the Select Star Settings. Click Data in the sidebar, then + Add to create a new Data Source.

Fill in the form with the required information:

• Source Type - Choose OpenLineage from the dropdown.

• Display Name - This value is OpenLineage by default, but you can override it.

• Base URL - The URL of your ETL instance. For example, http://airflow.example.com.

Click Save to proceed.

On the next screen, you will see the API Token, Events Endpoint, and the Events URL. You will need these in the next steps to configure your OpenLineage producer environment.

• API Token - This is a secret key that Select Star will use to authenticate the traffic coming from your OpenLineage producer instance.

• Events Endpoint - This is the Select Star endpoint where your producer will send OpenLineage events, containing the metadata about your Jobs, Job Runs, and Datasets.

• Events URL - This is the Select Star Base URL where your producer will send OpenLineage events.

2. Configure OpenLineage Producer

You must use the values provided above in your producer to start sending your OpenLineage events to Select Star.

That's it! OpenLineage events will be sent to Select Star when your producer starts creating events.

Select Star's OpenLineage integration can be used to generate lineage from your Spark, Airflow, and Custom OpenLineage events.

For more details on configuring and producing OpenLineage events, please read the official openlineage documentation.

3. Sync Metadata in Select Star

After you have configured your OpenLineage environment, make sure to trigger your health check jobs. This will send OpenLineage events to Select Star, and help you verify that the integration is working correctly.

Afterwards, you can go to the Select Star Settings and click on the Data in the sidebar. Click on the Sync metadata button on your OpenLineage Data Source.

Note that Select Star does not connect to your sources directly. That means the lineage and your job metadata will be available in Select Star only after you run your jobs and OpenLineage events are sent to Select Star.