AWS Glue (beta)
Last updated
Last updated
Ensure AWS Glue database and tables are setup in your AWS Glue instance. For details, see Adding an AWS Glue crawler.
To connect AWS Glue to Select Star, you will need to...
Select AWS Glue from the Add Data Source menu and provide the
Display Name - This value is AWS Glue by default, but you can overrided.
Region - ID of the AWS region where the cluster was created. For example us-east-2,us-west-1, eu-central-1
Select Star recommends use of AWS CloudFormation to setup integration, which allows you to make necessary changes to the AWS Glue environment in a automatic, transparent, safe and auditable manner.
AWS CloudFormation creates an AWS IAM Role to enable access for Select Star and add it to AWS Glue cluster.
The source code of the CloudFormation template along with build scripts and real-time logs of the continuous deployment system is available on public repository on GitHub "selectstar/cloudformation-templates" to be freely audited.
You can pass the link to CloudFormation to the infrastructure team to enable the integration to be created.
1. A simple form will be displayed in Select Star.
2. Select the "Open CloudFormation" button. A new window will open to proceed to the creation of a CloudFormation stack by AWS Management Console. Make sure you are logged into the AWS account in which the AWS Glue cluster exist.
3. The Create Stack form will be displayed. Fill form in the required information:
4. Review the information and under Capabilities choose "I acknowledge that AWS CloudFormation might create IAM resources".
5. Click Create stack.
6. Wait until the stack changes it status to "CREATE_COMPLETE" from "CREATE_IN_PROGRESS" in tab "Stack Info". The operation should take up to 5 minutes. You need to refresh tab to see the progress.
7. After completing stack creation, the Role ARN is available from the "Outputs". Copy and save the RoleArn for later use.
1. Return to Select Star. You should see a form that allows you to provide "Role ARN". Fill form in the required information:
Role ARN: Identifier of AWS IAM Role to use by Select Star. You'll see this after completing step 2.7 of the instructions.
2. Click Connect.
After you fill in the information, you'll be asked to select the catalogs and databases you'd like to load into Select Star.
Select Star will not read queries or metadata or generate lineage for databases, or tables that are not loaded. Please load all data for which you expect to see lineage.
You can change the catalog and databases you have loaded if needed.
Select the Catalog from the list (if more than one).
Select Databases from the list.
Click Next and your metadata should start loading automatically. Please allow 24-48 hours to completely generate lineage.
When the sync is complete, you'll be able to explore AWS Glue in Select Star.
