Creating ssh key pair and passing them as parameters with terraform

Encrypting some random text with AWS KMS

Uploading files to an S3 bucket always with encription

Prometheus and node_exporter install

Installing Prometheus

useradd --no-create-home --shell /bin/false prometheus

useradd --no-create-home --shell /bin/false node_exporter

mkdir /etc/prometheus

mkdir /var/lib/prometheus

chown prometheus:prometheus /etc/prometheus

chown prometheus:prometheus /var/lib/prometheus

Download Prometheus from https://prometheus.io/download/

Extract the tar ball:

tar xf prometheus-xxx.tar.gz

cd prometheus-xxx

cp prometheus /usr/local/bin

cp promtool /usr/local/bin/

chown prometheus:prometheus /usr/local/bin/prometheus

chown prometheus:prometheus /usr/local/bin/promtool

cp -r consoles /etc/prometheus

cp -r console_libraries /etc/prometheus

chown -R prometheus:prometheus /etc/prometheus/consoles

chown -R prometheus:prometheus /etc/prometheus/console_libraries

You can now rm the tarball and folder created when extracting.

Configuring prometheus:

vi /etc/prometheus/prometheus.yml

# ---------- begin /etc/prometheus/prometheus.yml

global:

  scrape_interval: 15s

scrape_configs:

  - job_name: 'prometheus'

    scrape_interval: 5s

    static_configs:

      - targets: ['localhost:9090']

# ---------- end /etc/prometheus/prometheus.yml

chown prometheus:prometheus /etc/prometheus/prometheus.yml

# Try to run prometheus

sudo -u prometheus /usr/local/bin/prometheus \

        --config.file /etc/prometheus/prometheus.yml \

        --storage.tsdb.path /var/lib/prometheus/ \

        --web.console.templates=/etc/prometheus/consoles \

        --web.console.libraries=/etc/prometheus/console_libraries

Everything went well, right?

Let's create a new systemd file for prometheus:

vi /etc/systemd/system/prometheus.service

# ---------- begin /etc/systemd/system/prometheus.service

[Unit]

Description=Prometheus

Wants=network-online.target

After=network-online.target

[Service]

User=prometheus

Group=prometheus

Type=simple

ExecStart=/usr/local/bin/prometheus \

    --config.file /etc/prometheus/prometheus.yml \

    --storage.tsdb.path /var/lib/prometheus/ \

    --web.console.templates=/etc/prometheus/consoles \

    --web.console.libraries=/etc/prometheus/console_libraries

[Install]

WantedBy=multi-user.target

# ---------- end /etc/systemd/system/prometheus.service

systemctl daemon-reload

systemctl start prometheus

systemctl status prometheus

systemctl enable prometheus

Installing node_exporter

Download node_exporter from https://prometheus.io/download/

Extract it and cd into the newly created folder

tar xf node_exporter-xxx.tar.gz

cd node_exporter-xxx

Copy the node_exporter binary

cp node_exporter /usr/local/bin

chown node_exporter:node_exporter /usr/local/bin/node_exporter

You can now delete the remaining node_exporter folder and tarball

Create a new systemd script for node_exporter

vi /etc/systemd/system/node_exporter.service

# ---------- begin /etc/systemd/system/node_exporter.service

Description=Node Exporter

Wants=network-online.target

After=network-online.target

[Service]

User=node_exporter

Group=node_exporter

Type=simple

ExecStart=/usr/local/bin/node_exporter

[Install]

WantedBy=multi-user.target

# ---------- end /etc/systemd/system/node_exporter.service

systemctl daemon-reload

systemctl start node_exporter

systemctl status node_exporter

systemctl enable node_exporter

So far so good!

Now we need to tell prometheus about how to reach node_exporter:

Add node_exporter to prometheus-yaml

vi /etc/prometheus/prometheus.yml

# ---------- begin /etc/systemd/system/node_exporter.service

global:

  scrape_interval: 15s

scrape_configs:

  - job_name: 'prometheus'

    scrape_interval: 5s

    static_configs:

      - targets: ['localhost:9090']

  - job_name: 'node_exporter'

    scrape_interval: 5s

    static_configs:

      - targets: ['localhost:9100']

# ---------- end /etc/systemd/system/node_exporter.service

Prometheus and node_exporter are installed, configured and working together!

Check prometheus console with your web browser

http://localhost:9090

Click Status, then Target. You should find two targets:

prometheus itself, which serves status about the prometheus server, and
node_exporter will serve stats about the host.

Click the endpoint for prometheus. Should be http://localhost:9090/metrics

At the bottom on the page, check the endpoint named

promhttp_metric_handler_requests_total{code="200"}

Refresh the page a few time, you should see the metric value increasing.

Copy the endpoint for this metric (promhttp_metric_handler_requests_total).

Go back to prometheus dashboard http://localhost:9090/graph

Paste the endpoint to the Expression form, then press execute. You will see only information about the endpoint you requested.

That's it for now!

Creating ssh key pair and passing them as parameters with terraform

We've all been there. Creating EC2 instances with Terraform, using self generated ssh key pairs. And it is always a pain in the neck.

Fear no more! Let me show you how to do it in a few steps:

Generate your ssh key pair:

ssh-keygen -t rsa -b 4096 -f mykey.pem

Fix permissions:

chmod 600 mykey.pem*

Create the public key in the format AWS' likes. AWS is requiring the public key to be in RFC4716 format.

ssh-keygen -e -f mykey.pem > mykey

Fix permissions on the newly generated key:

chmod 600 mykey

Then, your vars.tf file will look like:

variable "path_to_private_key" {

      default = "mykey.pem"

variable "path_to_public_key" {

      default = "mykey"

And your instance.tf will look lile:

resource "aws_key_pair" "mykey" {

      key_name = "mykey"

      public_key = "${file("${var.path_to_public_key}")}"

resource "aws_instance" "example" {

      ami = "${lookup(var.ami_ids, var.aws_region)}"

      instance_type = "t2.micro"

      key_name = "${aws_key_pair.mykey.key_name}"

      provisioner "file" {

      source = "script.sh"

      destination = "/tmp/script.sh"

provisioner "remote-exec" {

      inline = [

      "chmod +x /tmp/script.sh",

      "sudo /tmp/script.sh"

connection {

      host = coalesce(self.public_ip, self.private_ip)

      type = "ssh"

      user = "${var.instance_username}"

      private_key = "${file("${var.path_to_private_key}")}"

And I hope that helps!

See you around!

Encrypting some random text with AWS KMS

Want to use AWS KMS to encrypt text? Want to play with AWS KMS just for fun? Here, I gout your back!

1) Create a CMK on KMS. You'll need to give usage permissions for the username you're using on AWS CLI.

2) List the CMK you've just created:

aws kms list-keys

3) Let's encrypt:

aws kms encrypt --key-id arn:aws:kms:region:12301291029>/key/4323442-re244gdf --plaintext "AWS please encrypt this for me" --query CiphertextBlob --output text

The encrypted text will be displayed in base64.

Cleaning up your AWS Athena

You've been playing with AWS Athena a lot, and Athena's console is starting to look like a mess.

Here's how to clean it up:

1) List the databases you have with the command below

show databases

2) Start deleting the databases you don't want, using the command below. Replace (default) with the database name you find out on step 1.

drop database (default) cascade

That's it for today!

AWS S3 Pre-signed URL via CLI

You always heard about pre-signed URLs for AWS S3, and that you can generate a pre-signed URL thru SDKs.

But you also can generate a pre-signed URL thru AWS CLI. Here's how:

aws s3 presign s3://(bucket-name)/(file.ext) --expires-in (seconds-to-expire)

If you ommit the --expires-in parameter, the default duration for the pre-signed URL will be 60 minutes.

Uploading files to an S3 bucket always with encription

You want to enforce that files uploaded to a bucket are always encrypted?

You need to create a bucket policy to DENY uploads without encryption.

Here's how:

1) On your bucket, go to the Permissions tab, then click the button Bucket Policy.

2) Then you type the policy. OR click on policy generator, and change:

Step 1: Select Policy Type: S3 Bucket Policy

Step 2: Add Statement(s)

Effect: Deny

Principal: *

AWS Service: Amazon S3

Actions: PutObject

Amazon Resource Name (ARN): go back to the bucket tab on the browser, and copy the bucket ARN.

It will be something like "arn:aws:s3:::bucket-always-encripted"

Click the link "Add Conditions (Optional)"

Condition: StringNotEquals

Key: s3:x-amz-server-side-encryption

At Value, specify which encription you want, without quotes:

"AES256" for SSE-S3

"ams:kms" for SSE-KMS

Click the Add condition button.

Click the Add Statement button.

Click the Generate Statement button. A new popup windows will open with the Policy. This policy will not be automatically copied to the bucket policy. You need to select all policy text and CTRL+C.

4) Then go back to the Bucket Policy tab and paste the text from the Policy Generator.

Then you click the Save button and you'll get an error. That's because some AWS services will demand which objects the policy will apply to.

In this case, the policy will apply to all objects on the bucket. So, specify all objects on the bucket with:

"arn:aws:s3:::bucket-always-encripted/*"

Hit the Save button, and you're good to go!

AWS CloudWatch Agent

On AWS, you can use CloudWatch to monitor basic aspects of your EC2 instances: CPU utilization, disk I/O, network utilization. But of course you want to monitor way more things from your environment, RAM utilization to say the least.

You can do it by installing and configuring the CloudWatch Agent on your instances.

Here's how to do it on Amazon Linux 2. For other distros and Windows, check here.

1) Create a role:

Select type of trusted entity: AWS Service

Service that will use this role: EC2

Attach the following policy: CloudWatchAgentServerPolicy

And name the role "CloudWatchAgentServerRole"

2) Download the agent installer into your instance.

wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm

3) Install the CloudWatch Agent with the Linux RPM command:

sudo rpm -U ./amazon-cloudwatch-agent.rpm

The installer will install the application, and also create a new user and group named "cwagent".

You will also need to install and enable the EPEL repo, and install the collectd application, that is needed by the CW Agent:

sudo yum install -y epel-release

sudo yum --enablerepo=epel install collectd

4) Now run the CloudWatch Agent Wizard to create the configuration file for the CloudWatch Agent:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

Pay attention to change some of the default settings:

- Which user are you planning to run the agent?

1. root

2. cwagent

3. others

default choice: [1]:

Choose 2. It will run the agent with less privileges.

- What is the collect interval for StatsD daemon?

1. 10s

2. 30s

3. 60s

default choice: [1]:

I choose 3. One minute is fine for me.

- Do you want to monitor cpu metrics per core? Additional CloudWatch charges may apply.

1. yes

2. no

default choice: [1]:

I choose 2.

- Which default metrics config do you want?

1. Basic

2. Standard

3. Advanced

4. None

default choice: [1]:

I choose 3. Check here what are the metrics for each option

- Do you have any existing CloudWatch Log Agent (http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html) configuration file to import for migration?

1. yes

2. no

default choice: [2]:

- Do you want to monitor any log files?

1. yes

2. no

default choice: [1]:

If you don't have, choose 2

- Do you want to store the config in the SSM parameter store?

1. yes

2. no

default choice: [1]:

You will need to insert the SSM policy into the role if you want yes. Otherwise, choose no.

5) Start the agent, pointing the configuration created by the wizard

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s

6) Check if the agent is running

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status

You should see an output like

  "status": "running",

  "starttime": "2019-09-11T23:04:04+0000",

  "version": "1.227496.0"

7) Now go to the AWS Console, and navigate to the CloudWatch main dashboard.

Choose Dashboard on the left panel, and create a new dashboard.

You will see the new Custom Namespace called CWAgent, with the many new monitored itens. Create the dashboard with the metrics you want.

If you want to see some action on the dashboard, you'll need to put your instance to work.

Some examples of load tests:

CPU:

dd if=/dev/urandom | bzip2 -9 >> /dev/null&

Disk IO: write file.txt with 1GB

while true;do dd if=/dev/zero of=file.txt count=1024 bs=1048576;rm file.txt;done

RAM: write 500MB to RAM

sudo swapoff -a

while true;do sudo dd if=/dev/zero of=/dev/shm/fill count=1024 bs=548576;done