Wanted to drop a quick how-to on how to report back up to CloudWatch from a CoreOS instance in Amazon. This is a bit of a lab project, something I was thinking about and decided to try. Fair warning that I haven’t used this with any serious workloads or anything.
If you’ve never used it before, CloudWatch is an AWS service that allows you to gather metrics and trigger alarms based on those metrics. This is core to having a robust architecture in Amazon and being able to scale instances in your autoscaling groups. By default, only hypervisor level data is available to act upon: CPU %, Network I/O, or Disk I/O. In order to get some other actionable metrics like disk and RAM usage, we want to make use of the CloudWatch Monitoring Scripts found here.
Containerize Custom Metrics
Let’s bundle all the necessary bits we need to report up to CloudWatch inside of a container image. That is the CoreOS way after all!
- Create two files,
- And now
- In the bash script, let’s note that the disk-path we’re pointing to is /rootfs. I couldn’t find a great consensus on whether the value reported inside the container a
/was accurate since it points to the overlay. We’ll work around this by mounting the
/directory from the host as read only.
We can see if our container works simply by building and running.
On a CoreOS VM in AWS:
docker build -t cloudwatch-mon .(In the same directory as the Dockerfile and entrypoint.sh script)
- Run the container, filling in the AWS credentials as needed.
docker run -ti -e AWS_ACCESS_KEY_ID="abc" \
-e AWS_SECRET_ACCESS_KEY="123" -v /:/rootfs:ro cloudwatch-mon
You should see some output that shows that it’s sending data to CloudWatch:
- At this point we can push it to the Docker hub so that we can pull it down on other hosts as well.
Wrap It Up
Now that we’ve got a working Docker image, we can wrap it in a systemd service definition. Note that the same service setup could be added to cloud-config and automatically configured for new hosts.
- Create a credentials file at
/root/.aws/creds. It should look something like:
- Now create a service file at
/etc/systemd/system/cloudwatch.service. Populate it with:
sudo systemctl daemon-reload
- Start the monitoring service with
sudo systemctl start cloudwatch
- You should see that it’s reporting upstream with
sudo systemctl status cloudwatch
- Finally, you can head to the CloudWatch dashboard. The link should look something like:
- Once there, you should see metrics under “Linux System” that show the disk and RAM usage.
- You can also create some dashboards. Here’s what I was able to quickly mock up: