Container technology is now a cornerstone of every cloud provider’s infrastructure offerings, and AWS provides various managed services, such as Amazon ECS and EKS, to orchestrate these containers. In this article, we will be focusing on the Amazon ECS service, which provides two options for the launch type: EC2 and AWS Fargate.

The EC2 launch type provides flexibility, giving operators fine-grain control of the infrastructure (EC2 instances). DevOps teams need to set up the Linux/Windows instances, along with the required networking as well as auto-scaling, VPC, and subnet configuration for those instances.

Fargate, on the other hand, offers a serverless container hosting option, where DevOps teams only have to do the networking configuration and not worry about the provisioning of servers (EC2 instances).

In this blog, we’ll provide a step-by-step guide and learn how to stream your Amazon ECS and AWS Fargate logs to Elasticsearch.

Need more background on ECS and Fargate? Feel free to check out these posts featuring an Amazon ECS Tutorial and a comprehensive overview of AWS Fargate.

Logging into ECS and Fargate

ECS and Fargate are complex services in terms of the number of components (clusters, services, tasks, app containers), other AWS services they integrate with, networking, security, and many additional configurations. So, logging and monitoring of the ECS solution become very important to maintain its reliability, availability, and performance. You need to collect monitoring and metrics data from all parts of the ECS solution so that debugging a multi-point failure becomes easy.

AWS provides several tools, including a few out of the box, that can monitor your Amazon ECS resources and respond to potential issues.

AWS CloudWatch Logs

ECS has a container agent installed for each container instance that stores the ECS instance logs at /var/log/ecs.

AWS also provides an awslogs driver that can be configured for both the EC2 and Fargate launch type. The container agent uses this driver and sends logs to CloudWatch Logs using APIs. In AWS, users can then search, analyze, and visualize the CloudWatch logs via the CloudWatch Logs Insights tool.

Let’s now discuss how the awslogs driver is configured in ECS tasks.

Fargate Launch Type

If the launch type is Fargate for the ECS tasks, all you need to add are the logConfiguration parameters in the ECS task definition. 

Below is an example of such a configuration:

"logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "awslogs-web",
                    "awslogs-region": "ap-east-2",
                    "awslogs-stream-prefix": "awslogs-web-example"
                }
            }

The above configuration uses awslogs as a log driver and sends log streams to an existing log group in CloudWatch Logs or creates a new log group if it doesn’t exist.

EC2 Launch Type

If the launch type is EC2, other than setting up the awslogs driver, you also need to create an IAM policy to give permission to your container instances to use the CloudWatch Logs APIs. You then need to attach this policy to ecsInstanceRole.

Below is an example for the policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogStreams"
            ],
            "Resource": [
                "arn:aws:logs:*:*:*"
            ]
        }
    ]
}

With the EC2 launch type, you also need to ensure ECS container instances have the 1.9.0 + version of the container agent and use an ECS-optimized AMI so that the awslogs driver is already installed and available for use.

The type of information that is logged by these containers depends on the command that gets executed using ENTRYPOINT in a Dockerfile. It generally sends all the STDOUT and STDERR I/O streams to CloudWatch Logs. 

Drawbacks to Consider

The price of CloudWatch Logs is very low. But over a period of time, the number of logs you store will increase, along with the cost to do so. There are use cases where a team needs to keep the last three months of logs—or even more—for an application to debug production issues. You’ll observe that a significant part of your AWS billing goes toward CloudWatch Logs.

Another drawback of this tool is that it does not provide a filter option to send only a limited number or certain types of logs to storage.

One Solution to Reduce Costs

To reduce the cost of long-term storage, you may use the Lambda service to move logs stored in CloudWatch Logs to a cheaper logging solution, such as Elasticsearch, and others. Below is a sample architecture for sending logs to the Elasticsearch service:

ECS and CloudWatch Logs to Elasticsearch

Figure 1: Customized Solution for CloudWatch Logs to Elasticsearch

However, this solution requires effort to build and maintain.

ECS Log Drivers

CloudWatch Logs may not fulfill every requirement of logging and monitoring, so teams may need other services. For that, ECS does support log drivers other than awslogs. A few of the most common are syslog and Fluentd. These log drivers can be used to flush logs directly from containers to corresponding servers. For example, you can either push Fluent Bit as a daemon service or deploy the Fluentd as an aggregator service with a separate ECS task along with a network load balancer.

Following are the steps required to set this up:

To enable the Fluentd log driver, you need to add it to the ECS environment configuration:

 #!/bin/bash
echo "ECS_AVAILABLE_LOGGING_DRIVERS=[\"awslogs\",\"fluentd\"]" >> /etc/ecs/ecs.config

Fluentd can be installed by using an available Docker image and updating the fluent.conf file with the new source and destination. The example below shows the configuration for an Amazon Elasticsearch destination behind Kinesis Firehose:

<system>
workers 4
</system>



<source>
@type  forward
@id    input1
@label @mainstream
port  24224
</source>



<label @mainstream>
<match frontend*>
@type kinesis_firehose
@id   output_kinesis_frontend
region us-west-2
delivery_stream_name elasticsearch-delivery-stream
<buffer>
flush_interval 1
chunk_limit_size 1m
flush_thread_interval 0.1
flush_thread_burst_interval 0.01
flush_thread_count 15
total_limit_size 2GB
</buffer>
</match>
</label>

You also need to update the ECS task definition with logConfiguration having the fluentd logDriver:

"logConfiguration": {
    "logDriver": "fluentd",
    "options": {
        "fluentd-address": "fluentd-aggregator-service-cfe765812373a246.elb.ap-east-2.amazonaws.com:24224",
        "tag": "web-httpd"
    }
}

Here, the configuration points to a Fluent Loggregator service. 

That’s all for the setup, although you can see it requires a lot of setup and configuration. Also, you need to host and manage Fluentd as a separate service.

FireLens Log Router

Looking at the above approaches, it’s pretty clear that ECS needs a specific service to allow users to send container logs to other services with minimum effort. And that’s why AWS launched the FireLensn Service, a log router built specifically for ECS and AWS Fargate.

FireLens was built using a side-car pattern and creates one container per ECS task. It enables integration with many AWS services and partner solutions built for log storage and allows you to stream logs to both AWS services: CloudWatch and Kinesis Data Firehose destinations, such as Elasticsearch and S3.

FireLens supports both Fluentd and Fluent Bit open-source projects. However, you should prefer Fluent Bit over Fluentd due to the former’s better performance and lower resource usage. You can also easily set it up with some simple configuration in the ECS task definition.

Let’s take a look at the detailed steps needed to configure Elasticsearch as a destination when sending ECS container logs.

FireLens internal architecture for Elasticsearch destination

Figure 2: FireLens internal architecture for Elasticsearch destination

IAM Permission

The first thing you need to do is create an IAM role with permissions for the task to access the AWS service to send the logs. Here, it would be Elasticsearch:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "es:*",
            "Resource": "*"
        }
    ]
}

ECS Task Definition with FireLens

AWS provides its own Fluent Bit image that you can define in the task definition. The task definition must have these three components:

  • The Fluent Bit sidecar is a sidecar container for the log router that contains a FireLens configuration. The container should be marked as essential so that if it fails to start, it errors out:
{
"essential": true,
"Image": "802198612424.dkr.ecr.ap-east-2.amazonaws.com/aws-for-fluent-bit:latest",
"name": "log_router",
"firelensConfiguration": {
"type": "fluentbit",
                                              "options":{
               "config-file-type":"s3 | file",
               "config-file-value":"arn:aws:s3:::fluentbucket/fluent.conf | filepath"
            }
},
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "firelens-container",
"awslogs-region": "ap-east-2",
"awslogs-create-group": "true",
"awslogs-stream-prefix": "firelens"
}
},
"memoryReservation": 50
}

Here, we have externalized the fluent.conf file to an S3 bucket so that it can be customized.

  • One or more application containers must have a FireLens configuration. The log configuration should specify the awsfirelens log driver:
{
"essential": true,
"image": "nginx",
"name": "app",
"logConfiguration": {
"logDriver":"awsfirelens",
"options": {
"Name": "es",
"Host": "vpc-firelens-domain-ce8fjjyo08jawrhmz6nd8mta7z.ap-east-2.es.amazonaws.com",
"Port": "443",
"Index": "index1",
"Type": "my_type",
"Aws_Auth": "On",
"Aws_Region": "ap-east-2",
"tls": "On"
}
},
"memoryReservation": 100
}

Here, the Host is the URL of Amazon Elasticsearch, while other attributes in options are metadata fields.

  • Finally, the IAM Role ARN refers to the ARN of the IAM role created in the first step discussed earlier in this piece. It contains the permissions needed for the task to route the logs and executes the task:
"taskRoleArn": "arn:aws:iam::XXXXXXXXXXXX:role/ecs_task_iam_role",
"executionRoleArn": "arn:aws:iam::XXXXXXXXXXXX:role/ecs_task_execution_role",

If you want to use Fluentd instead of Fluent Bit, the configuration is almost the same except for a few differences when it comes to syntax and attribute names. 

Filter the Logs

There are use cases where you don’t want all logs flowing to the destination service—perhaps to reduce the log size—and thus keep only, say, application error-related logs. FireLens provides filtering of logs based on their content. You just need to add a few attributes with regular expressions to either include or exclude certain types of logs: 

"exclude-pattern":"[Dd]bug",
"include-pattern":"[Ee]rror"

Here, the exclude-pattern key causes all logs that match “Debug/debug” to be dropped, while the include-pattern key makes sure that only logs with the word “Error” are sent to Elasticsearch.

Inject a New Field in Logs

Let’s say you need to inject a new field in the log to be able to identify each record’s version. This can be done using the fluent.conf configuration. Fluentd can pick up the environment variable you define in the ECS task definition and refer it in the conf file to pass along with each record of the logs. 

Below is an example of how this would work:

<filter>
Record app-version ${APP_VERSION}

Summary

In this blog, we covered how to stream your Amazon ECS and AWS Fargate logs to Elasticsearch. Amazon ECS and AWS Fargate are the most popular services among developers to build their solutions for containerized applications. However, nobody wants to remain limited with CloudWatch Logs for monitoring their application. They want to use other services for their logging source and build dashboards around them. 

Read More:

Monitoring Amazon ECS Clusters with Epsagon

How to Stream AWS Lambda Logs to Elastic

Deeper Visibility into ECS and Fargate Monitoring

Why You Can’t Ignore Changes to Monitoring and Logging for Serverless