ProductBlogFeedback
Search
⌃K
Links

PostgreSQL

Follow these steps to connect your AWS RDS for PostgreSQL to Select Star.

Before you start

To connect AWS RDS for PostgreSQL to Select Star, you will need...
  • access to CloudFormation with permissions to modify IAM, AWS Lambda, AWS RDS instance and VPC
  • access to AWS RDS for PostgreSQL admin
Select Star requires only minimal metadata access to AWS RDS for PostgreSQL. The granted permissions are defined in CloudFormation template:
  • IAM permission defined by resource "CrossAccountRolePolicy" in file SelectStarRDS.json​
  • AWS RDS for PostgreSQL user permission defined in file provision.py​

1. Create a new data source

1. Go to the Select Star Settings. Click Data in the sidebar, then + Add to create a new Data Source.
Animations shows where to click to create a new data source.
2. Fill form in the required information:
  • Source Type: Select "AWS RDS - Postgres"
  • Display Name: This value is AWS RDS - Postgres by default, but you can override it if desired.
  • RDS instance: The name of your RDS instance in the AWS management console. Also known as "DB identifier" by AWS.
  • AWS Region: ID of the AWS region where the cluster was created. For example us-east-2,us-west-1, eu-central-1
Screenshot shows a new data source form.
3. Click Connect.

2. Create CloudFormation stack

Select Star recommends use of AWS CloudFormation to setup integration, which allows you to make necessary changes to the RDS instance environment in a automatic, transparent, safe and auditable manner.
​
AWS CloudFormation will create AWS resources, modify and validate the RDS instance configuration to ensure a safe integration with the Select Star services, to do this, the Stack creation will:
  • validate RDS instance compatibility
  • create an AWS IAM Role to enable access for Select Star
  • enable logging to CloudWatch of all RDS instance queries
  • create a custom parameter group or modify an existing one to set a parameters:
    • log_statement to all
    • log_min_duration_statement to 0
  • create PostgreSQL user and grants permissions in RDS instance
  • configure the security group to allow Select Star access to the RDS instance
The source code of the CloudFormation template along with build scripts and real-time logs of the continuous deployment system is available on our public repository selectstar/cloudformation-templates hosted on Github, to be freely audited.
You can pass the link to CloudFormation to the infrastructure team to enable the integration to be created.
1. A simple form will be displayed in Select Star.
Screenshot shows the access authorization form.
2. Click Open CloudFormation. A new window will open to proceed to the creation of a CloudFormation stack by AWS Management Console. Make sure you are logged into the AWS account in which the RDS instance is hosted.
3. The Create Stack form will be displayed. Some of the values will be filled in by default, fill the rest as required in the following steps:
  • Under Parameters, enter:
  • RDS server name: the field should be filled in automatically on the basis of the data from the previous form. Use the Amazon RDS instance name
  • DbUser and DbPassword: Credentials of PostgreSQL user used to connection for provision access for Select Star. This user is used only by CloudFormation for the purposes of creating a user account with minimal rights, including without access to data necessary for integration. Select Star never gets direct access to this user.
  • Configure CloudWatch logging (if disabled): Accept the automatic configuration of the instance to the extent necessary for the correct operation of the integration by selecting "true". If you do not accept - the created Lambda function will fail during the first configuration inconsistency along with tips on necessary changes.
  • Restart Server (if necessary to apply changes): Accept immediate restart of the instance instance to apply changes by selecting "true". If you do not accept - integration may not fully function eg. missing lineage and popularity until the instance is manually restarted.
  • Provided user name for Select Star to access your metadata: Leave "selectstar" in the field if possible.
  • PostgreSQL schema granted: Leave "." in the field if possible to avoid to minimize the risk of incomplete data. In the next steps you will be able to select loaded databases and schemas in Select Star, hence we recommend leaving "." in CloudFormation to avoid duplicate and potentially inconsistent configuration.
Screenshot shows the "Quick create stack" form.
4. Review the information and under Capabilities choose "I acknowledge that AWS CloudFormation might create IAM resources".
Screenshot shows the "Capabilities" section in "Quick create stack" form.
5. Choose Create stack.
6. Wait until the stack changes it status to "CREATE_COMPLETE" from "CREATE_IN_PROGRESS" in tab "Stack info". The operation should take up to 5 minutes. You need to refresh tab to see the progress.
Screenshot shows the "State" value in "Stack info" tab.
7. A completing stack creation, the Role ARN and Secret ARN is available from the "Outputs". Copy and save the RoleArn and SecretArn for later use.
Screenshot shows where to obtain a Role ARN and Secret ARN in the Outputs tab

3. Confirm authorization

1. Return to Select Star. You should see a form that allows you to provide "Role ARN" and "Secret ARN". Fill form in the required information:
  • Role ARN: Identifier of AWS IAM Role to use by Select Star. You'll see this after completing step 2.7 of the instructions.
  • Secret ARN: Identifier of Secret in AWS Secret Manager to use by Select Star. You'll see this after completing step 2.7 of the instructions.
Screenshot shows authroization in Select Star
2. Click Connect.

4. Choose databases and schemas

After you fill in the information, you'll be asked to select the databases you'd like to load into Select Star.
Select Star will not read queries or metadata or generate lineage for databases, schemas, or tables that are not loaded. Please load all data for which you expect to see lineage.
You can change the databases and schemas you have loaded if needed.
Select the database and click Next.
For each database you selected, you'll be able to select the schemas.
Your metadata should start loading automatically. Please allow 24-48 hours to completely generate popularity and lineage.
When the sync is complete, you'll be able to explore Redshift in Select Star.

Optional steps

Thanks to AWS Secret Manager service, you can configure automatic password rotation to fullfill your organization's password policy. Before each connection, we retrieve current password and never save it, so you can change it as often as your organization need. To configure it, refer to relevant section of AWS Secret Manager service documentation and AWS support if required.
The AWS RDS logging setup may in some cases result in sensitive information being recorded and stored in Amazon CloudWatch. To assess impact, risk, and take precaution and remediation steps corresponding to your organization's workload characteristics, refer to relevant section of AWS RDS service documentation and AWS support if required..
​

Additional information

Sync new Data Assets

If you create a new PostgreSQL schema in your database, we need you to GRANT it permissions explicitly as the CloudFormation stack will not grant the user selectstar with permissions on assets that do not exist at the time of execution.
To sync your new assets into select star, follow these steps:
1. GRANT us access to the new Schema:
GRANT USAGE ON SCHEMA <schema_name> TO selectstar;
GRANT SELECT ON ALL TABLES IN SCHEMA <schema_name> TO selectstar ;
2. Go to Select Star > Settings > Data
3. Find your Data Source, and click on Configure
4. Click Refresh
5. Click Next, and make sure your new Schema is selected.
​
​
​
We are always happy to help if you have any other questions! Send us a mail.
​
Previous
dbt Tests (beta)
Next
Tableau
Last modified 9mo ago