Log routing using AWS FireLens

You attached CloudWatchLogsFullAccess policy to ecsInstanceRole role and enabled FireLens integration when you create catsdef for this chapter.

FireLens works with Fluentd and Fluent Bit. AWS provide the AWS for Fluent Bit image or you can use your own Fluentd or Fluent Bit image. FireLens for Amazon ECS enables you to use task definition parameters to route logs to an AWS service or AWS Partner Network (APN) destination for log storage and analytics. In this lab, you will learn one of the simplest use cases: Firelens Fluent Bit and CloudWatch Logs.

  1. Move to CloudWatch Log groups.
  2. Filter ecs-demogo-log FilterLogGroups
  3. Click ecs-demogo-log and navigate each log stream. LogStream
  4. Expand each log to find more information. DetailLogs You can find information about container_id, ecs_cluster, etcs_task_definition and logs etc.
  1. You can also easily analyze logs using CloudWatch Log Insights. Select the ecs-demogo-log group and click the View in Logs Insight button to be redirected to CloudWatch Logs Insight. Alternatively, you can click Insight in the left navigation bar.

  2. You should see a sample query like this:

fields @timestamp, @message
| sort @timestamp desc
| limit 20
  • fields: Filters the results of a query based on one or more conditions.
  • sort: Sorts the retrieved log events. Both ascending (asc) and descending (desc) order are supported.
  • limit: Specifies the number of log events returned by the query.

CloudWatch Logs Insights supports a query language you can use to perform queries on your log groups. You may refer to sample queries.

  1. Click Run query to see what results the sample query outputs. It shows 20 results in descending order of timestamp and message of log with simple visualization. Key query statements can also be added to your dashboard by clicking the Add to dashboard button on the top right.

In addition to Log Insight, Amazon CloudWatch Logs integrates with other AWS services to provide endless benefits. For example, you can set an alarm by filtering specific phrases in your logs with CloudWatch Alarm, or you can use AWS Kinesis to process log data in real time.

Amazon CloudWatch Logs is not just for long-term log storage. For simple long-term storage, export to Amazon S3, and unless you must use CloudWatch for permanent storage, set an appropriate Expire period such as 1 day, 1 week, etc.