Query Logs

Available query logs sources for MySQL data sources:

• None

• CloudWatch Logs

If you are using a different hosting provider and you want to ingest query logs, please contact our sales team.

CloudWatch Logs

For instances hosted on AWS RDS we have the ability to ingest query log which allows us to get detailed lineage and popularity. Here's a step-by-step guide to enable the General Query Log for a MySQL database hosted on AWS RDS and export the logs to AWS CloudWatch.

Step 1: Sign in to the AWS Management Console

1. Go to the AWS Management Console.

2. Sign in using your AWS account credentials.

Step 2: Navigate to the RDS Dashboard

1. In the AWS Management Console, search for "RDS" in the search bar at the top and select RDS from the search results.

2. This will take you to the RDS dashboard.

Step 3: Select Your RDS Instance

1. In the RDS dashboard, click on Databases in the left-hand menu.

2. Find and select the MySQL RDS instance for which you want to enable the General Query Log.

Step 4: Modify the DB Parameter Group

1. On your selected DB instance page, scroll down to find the Configuration section.

2. Under Parameter group, note the name of the currently associated parameter group. If it's a default parameter group, you need to create a new custom parameter group because default ones cannot be modified.

3. In the left-hand menu, click on Parameter groups.

4. Click Create parameter group if you need to create a new one:

   • Choose the Parameter group family corresponding to your MySQL version.

   • Enter a name for the new parameter group.

   • Click Create.

5. Once created, find your new parameter group in the list, and click on it to modify the parameters.

Step 5: Enable General Query Log in the Parameter Group

1. In the parameter group settings, use the search bar to find the general_log parameter.

2. Set general_log to 1 (enabled).

3. Find the log_output parameter and set it to FILE as required for logging to CloudWatch.

4. Save the changes.

Step 6: Apply the Parameter Group to Your RDS Instance

1. Go back to the Databases section.

2. Select your MySQL RDS instance again.

3. Click on the Modify button in the upper right corner.

4. In the Database options section, change the DB Parameter Group to the new custom parameter group you just modified.

5. Scroll down and choose whether to apply the changes immediately or during the next maintenance window.

6. Click Continue and then Modify DB Instance.

Step 7: Enable Logging to CloudWatch

1. With the RDS instance selected, go to the Logs & events tab.

2. Under Manage export logs to CloudWatch, ensure that General Log is checked.

3. Click on Configure.

Step 8: Confirm Logs in CloudWatch

1. Go to the AWS Management Console and search for CloudWatch.

2. In the CloudWatch dashboard, click on Logs in the left-hand menu.

3. You should see a log group corresponding to your RDS instance. The General Query Log entries should now appear here.

Step 9: Test and Verify

1. Run a few queries on your MySQL database.

2. Check CloudWatch Logs to ensure that the queries are being logged and that the General Query Log is working correctly.

Step 10. Create CloudFormation stack

Select Star recommends setting up integration using AWS CloudFormation, which allows you to make necessary changes to the RDS instance environment automatically, transparently, safely, and auditably.

AWS CloudFormation will create AWS resources to enable us to access AWS CloudWatch logs.

The source code of the CloudFormation template, build scripts, and real-time logs of the continuous deployment system are available on our public repository selectstar/cloudformation-templates, hosted on Github, to be freely audited.

You can pass the link to CloudFormation to the infrastructure team to enable the integration to be created.

1. A simple form will be displayed in Select Star. In option Source Type select "CloudWatch Logs".

2. Click Open CloudFormation. A new window will open to proceed to the creation of a CloudFormation stack by AWS Management Console. Make sure you are logged into the AWS account in which the RDS instance is hosted.

3. The Create Stack form will be displayed. Some of the values will be filled in by default, under Parameters, enter:

• Log Group Name: The name of the log group to be consumed by Select Star. Example: /aws/rds/instance/dev-mysql8/general

4. Review the information and under Capabilities choose "I acknowledge that AWS CloudFormation might create IAM resources".

Screenshot shows the "Capabilities" section in "Quick create stack" form.

5. Choose Create stack.

1. Wait until the stack changes its status to "CREATE_COMPLETE" from "CREATE_IN_PROGRESS " in the tab "Stack info." The operation should take up to 5 minutes. You need to refresh the tab to see the progress.

Screenshot shows the "State" value in "Stack info" tab.

7. After completing stack creation, the Role ARN is available from the "Outputs". Copy and save the RoleArn for later use.

Screenshot shows where to obtain a Role ARN in the Outputs tab

Step 11. Confirm authorization

1. Return to Select Star. You should see a form that allows you to provide "Role ARN" . Fill form in the required information:

• AWS region: Identifier of AWS Region where your MySQL instance is hosted eg. us-east-2

• Role ARN: Identifier of AWS IAM Role to use by Select Star.

• Log group name prefix: The prefix of name of the log group to be consumed by Select Star. Example: /aws/rds/instance/dev-mysql8/general

2. Click Connect.