← back

2022-05-03: using awslogs to send EC2 logs to CloudWatch

CloudWatch is a convenient way to store logs in AWS, especially in Cloud-centric application design.

Sometimes this may mean interacting with virtual machine compute instances, such as EC2, but might also include baremetal servers too.

First, ssh to the desired server and do the following and obtain the awslogs agent:

curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O

Next run the agent install setup, and if this is an EC2 instance, select a region as well using the --region flag.

sudo python ./awslogs-agent-setup.py --region us-west-2

At this point the installer might ask you a bunch of questions, in particular the following.

• aws key: key from an AWS role, note that this role is required to have CloudWatch permissions
• aws secret: secret value from the AWS role mentioned above
• hostname: either the full qualified domain name or the name of the server
• datetime: offers multiple formats, including year-month-day or month-day
• logfile: path to the file to monitor and send data to CloudWatch
• position: either send entire log to CloudWatch from beginning or upload contents appended to end of file
• stream: name of the log entry in CloudWatch

After all of the questions have been answer, then the awslogs service will start and the contents of the log in question will be sent.

Check the status of the server using either service or systemctl:

sudo service awslogs status

[or]

sudo systemctl status awslogs

Afterwards edit the awslogs config:

sudo vim /var/awslogs/etc/awslogs.conf

The content of this file should resemble:

[/var/log/nginx/access.log]
datetime_format = %d/%b/%Y:%H:%M:%S
file = /var/log/nginx/access.log
buffer_duration = 5000
log_stream_name = /var/log/nginx/access.log
initial_position = end_of_file
log_group_name = /aws/ec2/hostname.of.ec2

Then restart the awslogs service, using the method relevant to the Linux distribution present on the EC2 instance or server; typically one of these commands:

sudo service awslogs restart

[or]

sudo systemctl restart awslogs

This will result in the logs being sent to AWS CloudWatch, which can be found under the log group name of /aws/ec2/hostname.of.ec2, with a steam name of /var/log/nginx/access.log.