Configuring CloudWatch Agent to Send NGINX Logs from EC2 to CloudWatch

In this blog, we will guide you through the process of configuring the CloudWatch Agent to send NGINX logs from your EC2 instances to Amazon CloudWatch. This setup will enable you to monitor and analyze your NGINX logs in real-time, providing valuable insights into your web server's performance and activity.

We'll cover the steps to install and configure the CloudWatch Agent on your EC2 instances, and show you how to customize the log collection to include your NGINX log files. Whether you're new to AWS or looking to enhance your log monitoring capabilities, this guide will help you set up an effective logging solution with ease.

Let's get started on optimizing your AWS infrastructure with CloudWatch!

1. Launch Ubuntu EC2 Instance:-

   • Log in to AWS Management Console and navigate to the EC2 Dashboard.

   • Launch an Instance:

     • Click "Launch Instance." Select "Ubuntu Server AMI"

     • Choose instance type as t2.micro.

     • Enable "Auto-assign Public IP."

     • Enable port 22 for ssh, port 80 for accessing the web page in security group.

     • In the "User data" section, add the following script to update the package list and install NGINX.

         #!/bin/bash
         sudo apt-get update
         sudo apt-get install nginx -y
         echo "ANY MESSAGE YOU WANT ON BROWSER" >> /var/www/html/index.html
       

     • Create instance.

       

2. Create/Assign IAM Role for EC2 with CloudWatch permissions:-

   • Log in to the AWS Management Console and go to the IAM Dashboard.

   • Click "Roles" > "Create role." > Select "AWS service" > "EC2." > Click "Next: Permissions."

   • Search for CloudWatchAgentServerPolicy and select it.

   • Name the Role, e.g., EC2CloudWatchAgentRole.

   • Click "Create role."

   • Assign the Role to Your EC2 Instance:

     • Go to the EC2 Dashboard.

     • Select your instance, click "Actions" > "Security" > "Modify IAM role."

     • Choose EC2CloudWatchAgentRole and click "Update IAM role."

       

3. Download/Install CloudWatch Agent package:-

   • Use the following steps to download the CloudWatch agent package, SSH to Instance & Download CW Agent package.

   • Download the cloudwatch agent package using:

       sudo wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
     

     

   • Install the downloaded package:

       sudo dpkg -i -E ./amazon-cloudwatch-agent.deb
     

   • Update Packages & Install collectd: (This will take a few minutes if you haven’t updated your available updates prior).

       sudo apt-get update && apt-get install collectd
     

4. Create the CloudWatch Agent Configuration File:-

   • Before running the CloudWatch agent on any servers, you must create a CloudWatch agent configuration file.

   • The agent configuration file is a JSON file that specifies the metrics and logs that the agent is to collect, including custom metrics.

   • The agent configuration file wizard, amazon-cloudwatch-agent-config-wizard, multiple configuration related questions.

   • We can check json configuration file under /opt/aws/amazon-cloudwatch-agent/bin/config.json

   • Creating configuration file:

       /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
     

   • We are using Linux OS and using EC2 so we will select 1 option:

     

   • Next user we will select root, StatsD daemon = no,

     

     

     

   • Here next give the path for your log file to monitor in our case it is /var/log/nginx/access.log

     

     

     

5. Checking status of CloudWatch Agent:

    /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status
   

   It will appear as stopped, as shown below:

   

   We will start the status of the CloudWatch Agent:

    /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
   

   Now, we will again check for the status after running above command.

   

   Boom!! our cloudwatch agent is ready to send our Nginx logs to cloudwatch agent on AWS.

6. Verifying logs on CloudWatch:

   • We will see log group created with access.log name on CloudWatch.

     

   • Inside log group you will get a log stream containing all our logs.

     

   • Here you will get all your logs on the CloudWatch.

Conclusion: In this blog, we have walked through the steps to configure the CloudWatch Agent to send NGINX logs from your EC2 instances to Amazon CloudWatch. By launching an EC2 instance with NGINX installed and setting up an IAM role with the necessary permissions, you've enabled a robust logging solution that enhances your ability to monitor and analyze your server's performance in real-time.

We hope this guide has been helpful in setting up your EC2 and CloudWatch integration. If you have any questions or need further assistance, feel free to reach out me anytime. Happy monitoring!