Query Logs
Last updated
Was this helpful?
Last updated
Was this helpful?
Available query logs sources for MySQL data sources:
None
CloudWatch Logs
For instances hosted on AWS RDS we have the ability to ingest query log which allows us to get detailed lineage and popularity. Here's a step-by-step guide to enable the General Query Log for a MySQL database hosted on AWS RDS and export the logs to AWS CloudWatch.
Go to the .
Sign in using your AWS account credentials.
In the AWS Management Console, search for "RDS" in the search bar at the top and select RDS from the search results.
This will take you to the RDS dashboard.
In the RDS dashboard, click on Databases in the left-hand menu.
Find and select the MySQL RDS instance for which you want to enable the General Query Log.
On your selected DB instance page, scroll down to find the Configuration section.
Under Parameter group, note the name of the currently associated parameter group. If it's a default parameter group, you need to create a new custom parameter group because default ones cannot be modified.
In the left-hand menu, click on Parameter groups.
Click Create parameter group if you need to create a new one:
Choose the Parameter group family corresponding to your MySQL version.
Enter a name for the new parameter group.
Click Create.
Once created, find your new parameter group in the list, and click on it to modify the parameters.
In the parameter group settings, use the search bar to find the general_log parameter.
Set general_log to 1 (enabled).
Find the log_output parameter and set it to FILE as required for logging to CloudWatch.
Save the changes.
Go back to the Databases section.
Select your MySQL RDS instance again.
Click on the Modify button in the upper right corner.
In the Database options section, change the DB Parameter Group to the new custom parameter group you just modified.
Scroll down and choose whether to apply the changes immediately or during the next maintenance window.
Click Continue and then Modify DB Instance.
With the RDS instance selected, go to the Logs & events tab.
Under Manage export logs to CloudWatch, ensure that General Log is checked.
Click on Configure.
Go to the AWS Management Console and search for CloudWatch.
In the CloudWatch dashboard, click on Logs in the left-hand menu.
You should see a log group corresponding to your RDS instance. The General Query Log entries should now appear here.
Run a few queries on your MySQL database.
Check CloudWatch Logs to ensure that the queries are being logged and that the General Query Log is working correctly.
Select Star recommends setting up integration using AWS CloudFormation, which allows you to make necessary changes to the RDS instance environment automatically, transparently, safely, and auditably.
AWS CloudFormation will create AWS resources to enable us to access AWS CloudWatch logs.
You can pass the link to CloudFormation to the infrastructure team to enable the integration to be created.
1. A simple form will be displayed in Select Star. In option Source Type select "CloudWatch Logs".
2. Click Open CloudFormation. A new window will open to proceed to the creation of a CloudFormation stack by AWS Management Console. Make sure you are logged into the AWS account in which the RDS instance is hosted.
3. The Create Stack form will be displayed. Some of the values will be filled in by default, under Parameters, enter:
Log Group Name: The name of the log group to be consumed by Select Star. Example: /aws/rds/instance/dev-mysql8/general
4. Review the information and under Capabilities choose "I acknowledge that AWS CloudFormation might create IAM resources".
5. Choose Create stack.
Wait until the stack changes its status to "CREATE_COMPLETE" from "CREATE_IN_PROGRESS " in the tab "Stack info." The operation should take up to 5 minutes. You need to refresh the tab to see the progress.
7. After completing stack creation, the Role ARN is available from the "Outputs". Copy and save the RoleArn for later use.
1. Return to Select Star. You should see a form that allows you to provide "Role ARN" . Fill form in the required information:
AWS region: Identifier of AWS Region where your MySQL instance is hosted eg. us-east-2
Role ARN: Identifier of AWS IAM Role to use by Select Star.
Log group name prefix: The prefix of name of the log group to be consumed by Select Star. Example: /aws/rds/instance/dev-mysql8/general
2. Click Connect.
The source code of the CloudFormation template, build scripts, and real-time logs of the continuous deployment system are available on our public repository , hosted on Github, to be freely audited.
