Accessing and Searching CloudWatch Logs Insights from AWS Lambda using Python

AWS Lambda is a serverless compute service that allows you to run your code without provisioning or managing servers. AWS CloudWatch Logs Insights is a fully managed service that helps you analyze, visualize, and gain insights from your log data. In this blog post, we will discuss how to access and search CloudWatch Logs Insights from an AWS Lambda function using Python and the Boto3 library.

Prerequisites:

• An AWS account
• Basic knowledge of AWS Lambda and CloudWatch Logs Insights
• Python and Boto3 library installed

Step 1: Set up the IAM role for your Lambda function

To access CloudWatch Logs Insights from your Lambda function, you need to create an IAM role with the necessary permissions. Attach the following policy to the role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:StartQuery",
                "logs:GetQueryResults"
            ],
            "Resource": "*"
        }
    ]
}

You can also first create the lambda function with some default IAM roles, and then come back to modify the existing policy attched to the current role in the lambda function, then just insert the json in the “statement” block to the existing policy.
To modify policy with the existing role, click “cofiguration” on the top of the lamda function page, then click “permissions” on the left pannel, and click the role attached to the lambda function.
Further click the role, you will see polices attached to that role, and click the policy to make any updates.

Step 2: Create a Lambda function with the IAM role

Create a new Lambda function using the AWS Management Console, AWS CLI, or any other method you prefer. Make sure to assign the IAM role you created in Step 1 to the Lambda function.

Step 3: Access CloudWatch Logs Insights from your Lambda function

In your Lambda function, import the Boto3 library and create a CloudWatch Logs client:

import boto3
import time

cloudwatch_logs = boto3.client('logs')

Define a function to execute a CloudWatch Logs Insights query and get the results:

def run_insights_query(log_group_name, query, start_time, end_time):
    response = cloudwatch_logs.start_query(
        logGroupName=log_group_name,
        startTime=int((time.time() - start_time) * 1000),
        endTime=int((time.time() - end_time) * 1000),
        queryString=query
    )
    query_id = response['queryId']

    while True:
        response = cloudwatch_logs.get_query_results(
            queryId=query_id
        )
        status = response['status']
        if status == 'Complete':
            return response['results']
        elif status == 'Failed':
            raise Exception('Query failed')
        else:
            time.sleep(1)

Use the run_insights_query() function to execute a CloudWatch Logs Insights query and get the results:

log_group_name = 'your-log-group-name'
query = 'fields @timestamp, @message | sort @timestamp desc | limit 20'
start_time = 60 * 60  # 1 hour ago
end_time = 0  # Now

results = run_insights_query(log_group_name, query, start_time, end_time)
print(results)

Replace 'your-log-group-name' with the appropriate value for your Log Insight group, and adjust the query, start_time, and end_time variables as needed.

Conclusion:

In this blog post, we demonstrated how to access and search CloudWatch Logs Insights from an AWS Lambda function using Python and the Boto3 library. This allows you to analyze and visualize your log data directly from your Lambda function, enabling you to build powerful serverless applications that can react to log data in real-time.