Security | AWS | EC2

Using Session Manager to provide secure EC2 access, whilst improving incident response capabilities with activity recording.

Andrew Blooman

·

Follow

12 min read

·

6 days ago

223

3

What are we trying to solve?

How many times have I seen a security group with port 22 open from 10.0.0.0/8, or worse 0.0.0.0/0? Too many times! But why, why are we still using SSH in 2024 when there are better alternatives? As a security professional I'm often tasked with trying to convince people of “a better way of working”. I often fail. People like quick and easy ways of working (and I don't blame them!). But in a time where the number security incidents is only rising, we must find a better way of working, a way which has security built in, by design.

Moreover, if you watch tech videos on YouTube, you’ll often see public facing EC2 instances, with SSH open to “My IP”. I don’t blame content creators for doing this, after all, they just want to show people how perform a particular task. However, a lot of developers assume this is the ONLY way of working, so when a security professional tells them they need to change how they connect to EC2, its often met with resistance.

So how do you provide secure access to an EC2 instance, with no key pairs, no public IP address, no poorly configured security group, whilst benefiting from enhanced logging AND security?!

The answer? Systems manager and Session Manager.

What is Systems Manager (SSM)?

AWS Systems Manager is the operations hub for your AWS applications and resources and a secure end-to-end management solution for hybrid and multi-cloud environments that enables secure operations at scale.

Some of the services Systems Manager offers include:

• Patch management
• Software distribution
• Inventory of EC2s
• Configuration management
• Session Manager

What is Session Manager?

You can connect to AWS EC2 instances by using either an interactive one-click browser-based shell or the AWS Command Line Interface (AWS CLI). Session Manager provides secure and auditable node management without the need to open inbound ports, maintain bastion hosts, or manage SSH keys. Session Manager also allows you to comply with corporate policies that require controlled access to managed nodes, strict security practices, and fully auditable logs with node access details, while providing end users with simple one-click cross-platform access to your managed nodes.

Why use Session Manager instead of SSH?

• Centralized access control to managed nodes using IAM policies
• No open inbound port 22 or 3389 (Windows) required in Security Groups
• No need to manage bastion hosts or SSH keys
• Logging and auditing session activity.

The Logging capability works on two levels, firstly all sessions are recorded to AWS CloudTrail, making auditing of activities consistent with other AWS services and event types. This ensures non-repudiation to activities performed whilst a session is active.

Secondly, the key strokes can be captured to a centralised logging facility such as CloudWatch or Amazon S3. This means that if you have a particular regulatory requirement to log all actions performed on an EC2 instance via the terminal, you can. This is also highly beneficial to Security Operations teams when performing incident investigation as they can build alerts based on known suspicious activities e.g. Privilege escalation.

Solution Architectural Overview

Designing for a multi-account environment is not simple, especially when you involve Encryption Keys, however the complexity can be alleviated by automating the configuration outlined below, during the account provisioning process. Below is a design which should help clarify how we can solve our SSH challenge, without adding barriers for developers or platform engineers.

Logging Account (S3 and CloudWatch)

The Logging Account is in the Security Organization Unit (OU). The S3 bucket and CloudWatch log group are configured in this account, encrypted using a KMS key.

VPC Endpoints (VPCe)

In each AWS account in the Workloads OU, have VPC endpoints configured to keep traffic internal to the AWS cloud. This provides access via VPCe to the S3 bucket and CloudWatch Log group without having to egress to the Internet.

EC2 Instances

The EC2 instances are configured with a custom instance profile with permissions to send logs to CloudWatch and S3, ensuring they have permission to encrypt using the KMS key. Furthermore there is a manged IAM policy from AWS which enables Systems Manager to operate; this is also attached to the instance profile.

Systems Manager & Session Manager

Session Manager preferences are where the configuration of the above is stored, ensuring that the same S3 bucket or CloudWatch group is used.

Ingest Logs to SIEM (Optional)

A Security Information & Events Management system will capture logs from multiple sources for analysis and alert on potentially malicious activity. Rather than configuring alerts in CloudWatch, it may be easier for security analyst to configure rules within the SIEM. This also means you can expire the CloudWatch logs quickly to save costs, as the logs will be ingested to an external system anyway.

Pre-requisites for Session Recordings

• Use an Amazon Machine Image (AMI) with SSM agent installed (Amazon Linux 2023 has it installed by default).
• Follow the Guide from AWS to configure Systems Manager (some settings are in my terraform config).
• Ensure the VPC endpoints for ec2messages, ssm, ssmmessages, kms, logs and s3 and configured. I also found that a Gateway endpoint for S3 was also required.

Setting up and using Session Manager Logs

Terraform Configuration

Below is the Terraform I used to configure the core aspects of this demo. You can find the terraform on my GitHub.

ec2.tf — Configure the EC2 instance and associated settings

This creates the EC2 in your VPC with a security group which allows egress to all IP addresses. Note there is NO inbound port 22. There is also an IAM policy which is used to provide permission to CloudWatch, S3 and KMS; this ensures we can ship logs and use the correct encryption key.

Next, this will create the role, attach our IAM policy as well as the required AmazonSSMManagedInstanceCore policy. (This is an AWS managed policy and is required for the SSM service to talk to the SSM agent on the EC2 Instance).

Finally an instance profile will be created and attached to the EC2 with the permissions taken from the IAM role.

# Creates EC2 instance
resource "aws_instance" "ssm_recording_example" {
  ami                    = data.aws_ami.amazon_linux_2023.id
  instance_type          = "t2.micro"
  iam_instance_profile   = aws_iam_instance_profile.ec2_ssm_instance_profile.name
  subnet_id              = data.aws_subnet.subnet.id
  vpc_security_group_ids = ["${aws_security_group.session_manager_security_group.id}"]
  metadata_options {
    http_endpoint               = "enabled"
    http_tokens                 = "required" # This enforces the use of IMDSv2
    http_put_response_hop_limit = 1
  }
  tags = {
    Name        = "Session Manager Demo"
    Environment = "Development"
    Owner       = "Medium Article"
  }
}
# Creates Security Group
resource "aws_security_group" "session_manager_security_group" {
  name        = "Session Manager Security Group"
  description = "Session Manager Security Group"
  vpc_id      = data.aws_vpc.main.id
  tags = {
    Name : "Session Manager Security Group"
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}
# Creates EC2 IAM Policy
resource "aws_iam_policy" "session_manager_recording_policy" {
  name        = "SSM-SessionManager-S3-KMS-Logging"
  description = "Policy for SSM Session Manager logging to S3 with KMS encryption"
  policy      = <<EOF
  {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PutObjectsBucket",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Effect": "Allow",
      "Resource": "${aws_s3_bucket.log_bucket.arn}/*"
    },
    {
      "Sid": "ListBucketAndEncryptionConfig",
      "Action": [
        "s3:GetEncryptionConfiguration"
      ],
      "Effect": "Allow",
      "Resource": "${aws_s3_bucket.log_bucket.arn}"
    },
    {
      "Sid": "S3KMSSessionManagerKMS",
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt",
        "kms:GenerateDataKey*"
      ],
      "Resource": [
        "arn:aws:kms:eu-west-1:${aws_kms_key.log_key.arn}"
      ]
    },
        {
            "Sid": "cloudwatch",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams"
            ],
            "Resource": "*"
        }
  ]
}
EOF
}
# Creates IAM role
resource "aws_iam_role" "ec2_ssm_role" {
  name = "ec2_ssm_role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
  managed_policy_arns = ["${aws_iam_policy.session_manager_recording_policy.arn}", "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"]
}
# Creates instance profile
resource "aws_iam_instance_profile" "ec2_ssm_instance_profile" {
  name = "ec2_ssm_instance_profile"
  role = aws_iam_role.ec2_ssm_role.name
}

kms.tf — Used to encrypt CloudWatch and S3

This section creates the KMS key, key alias and associated permissions policy required for multi-account usage.

# Creates KMS Key
resource "aws_kms_key" "log_key" {
  description         = "KMS key for S3 log bucket"
  enable_key_rotation = true
  policy              = <<POLICY
{
    "Version": "2012-10-17",
    "Id": "ExamplePolicy01",
    "Statement": [
        {
            "Sid": "AllowUseOfTheKey",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "kms:Decrypt",
                "kms:GenerateDataKey"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:PrincipalOrgID": "${var.organization_id}"
                }
            }
        },
        {
            "Sid": "EnableIAMUserPermissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::${var.account_id}:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        },
        {
            "Sid": "AllowCloudWatchLogsUseOfTheKey",
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.${var.region}.amazonaws.com"
            },
            "Action": [
                "kms:*"
            ],
            "Resource": "*"
        }
    ]
}
POLICY
}
# Creates KMS Key Alias
resource "aws_kms_alias" "log_key" {
  name          = "alias/logging_key"
  target_key_id = aws_kms_key.log_key.id
}

s3.tf - Bucket Configuration

This is where the bucket is created, versioning enabled (recommended), KMS key defined, along with the bucket policy. It might seem complex defining both resource and identity policies for S3, KMS and EC2, but remember, this is a multi-account configuration, so it takes a little more work to get the permissions correct.

resource "aws_s3_bucket" "log_bucket" {
  bucket = var.bucket_name
}

resource "aws_s3_bucket_versioning" "log_bucket_versioning" {
  bucket = aws_s3_bucket.log_bucket.id
  versioning_configuration {
    status = "Enabled"
  }
}
resource "aws_s3_bucket_server_side_encryption_configuration" "log_bucket_encryption" {
  bucket = aws_s3_bucket.log_bucket.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.log_key.arn
    }
    bucket_key_enabled = true
  }
}
resource "aws_s3_bucket_policy" "log_bucket_policy" {
  bucket = aws_s3_bucket.log_bucket.id
  policy = <<POLICY
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetEncryptionConfiguration",
            "Resource": "${aws_s3_bucket.log_bucket.arn}",
            "Condition": {
                "StringEquals": {
                    "aws:PrincipalOrgID": "${var.organization_id}"
                }
            }
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "${aws_s3_bucket.log_bucket.arn}/*",
            "Condition": {
                "StringEquals": {
                    "aws:PrincipalOrgID": "${var.organization_id}"
                }
            }
        }
    ]
}
POLICY
}

cloudwatch.tf — Log Group

This creates a log group and uses the KMS key to encrypt. (In a production environment, you would be best using a dedicated KMS key for CloudWatch and S3.

resource "aws_cloudwatch_log_group" "session_manager_logs_group" {
  name              = "session_manager_logs"
  kms_key_id        = aws_kms_key.log_key.arn
  log_group_class   = "STANDARD"
  retention_in_days = 30
  tags = {
    Environment = "Sandbox"
  }
}

data.tf — Populate inputs for use later

This is where we specify the tag name of the VPC, so we can query the AWS API using terraform and retrieve its attributes, namely the VPC ID. If you know the VPC ID, you don’t need this data source, just reference the VPC ID as a variable or if it exists as a resource in your terraform configuration, reference it from there. (Similar for the subnet ID)

Finally we are querying the latest AMI, useful if you are using the Amazon Linux 2023 image. We output the AMI ID for use later.

data "aws_vpc" "main" {
  filter {
    name   = "tag:Name"
    values = ["INSERT VPC NAME"]
  }
}

data "aws_subnet" "subnet" {
  filter {
    name   = "tag:Name"
    values = ["INSERT SUBNET NAME"]
  }
}

data "aws_ami" "amazon_linux_2023" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["amzn2-ami-kernel-5.10-hvm-*-x86_64-gp2"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  filter {
    name   = "architecture"
    values = ["x86_64"]
  }
}

output "ami_id" {
  value = data.aws_ami.amazon_linux_2023.id
}

Variables

Variables here are the name of the bucket, the organization ID, the account ID and the region. Some or perhaps all of these may not be needed in your terraform, depending on what is already in your terraform state. Similarly, you may choose to use alternative methods to input these values, to make the code more re-usable e.g. CI Pipeline variables.

variable "bucket_name" {
  type    = string
  default = "medium-session-manager-article"
}

variable "organization_id" {
  type    = string
  default = "INSERT ORG ID"
}

variable "account_id" {
  type    = string
  default = "INSERT ACCOUNT ID"
}

variable "region" {
  type    = string
  default = "eu-west-1"
}

Note, there is no native terraform resource (at the time of writing) in the AWS terraform provider for Session Manager Preferences. However, there are custom modules you can use if you want to automate this process, which can be found here. For the next few steps I will show screenshots from the AWS console instead.

1. Access Systems Manager Service in AWS, under Node Management you will find Session Manager. Click Preferences and then Edit.

2. Within the General Preferences you can specify several options including (you can leave these as defaults if you wish):

• Session timeout
• Maximum session duration — you have the option to resume a session within SSM, so this is something to discuss with your team
• Use KMS instead of TLS
• Specify Run As for Linux EC2 instances — if you have a custom AMI with a specific user, this might be useful to use.

3. Logging Options — Important!

This is where we specify out destination for where we want to our logs to be sent. We can specify CloudWatch, S3 or both.

4. We can specify several options here, in this case we are leaving the defaults. Select the CloudWatch log group you wish to ship logs to and then click Save.

5. We can also send logs to S3 if we desire; this might be useful if you have a retention requirement for logs and you want to keep CloudWatch costs to a minimum.

Select the bucket you wish to send the logs to and also specify an option prefix. I would suggest using the prefix of AccountID/session-logs/ so you can readily identify which AWS account the logs came from. Reminder, there isn’t a Terraform resource for this, so you will need to do this part manually or as part of the account vending process.

Example Logs in CloudWatch

Here is a sample of me typing some commands into my EC2 instance.

Here is the corresponding activity in CloudWatch.

It is important to note that sessions are recorded in AWS CloudTrail, providing the auditing capability, for accountability and non-repudiation.

How Session manager bolsters security posture by logging activity to CloudWatch and S3

By capturing key strokes from EC2 instances, it will aid security teams in detection and response capabilities as they can build rules for when specific commands are typed in, such as elevating to root privileges. Furthermore, by sending the logs to a SIEM, threat intelligence including Indicators of Compromise (IOCs) can be used to enrich the logs, which might reveal an attempt to connect to a suspicious domain (command & control traffic), either to exfiltrate data or be used as part of crypto mining fleet.

The secondary benefit of this solution is that it adds a layer of audit & compliance, enabling security teams to keep a record of changes which have taken place. This allows external auditors to ratify that there are sufficient controls in place to detect unauthorized activity. Furthermore, as SSM uses IAM permissions to grant access to Session Manager, it could be used as a preventative control, if console access is restricted via company security policy. It should be noted that there are other solutions which provide more sophisticated privileged access, one of which is Teleport.

It should be noted that Session Manager would not capture logs if the EC2 was compromised in an alternative fashion e.g. RCE attack leading to remotely executed commands, as such it is always essential to have multiple security controls in place to detect and respond to threats. Examples include centralising Linux operating system logs logs from EC2 instances to CloudWatch and deployment endpoint sensors such as eBPF.

Final Thoughts

AWS is always trying to figure out more secure ways to provide access to their services, but getting adoption of these newer ways of working is often a challenge. People are set in their ways and will struggle to change how they work; the convenience of just connecting via a terminal or using a bastion has been an established practice for over 25 years.

Whilst there are some third-party vendors like Teleport doing good work out there, its sometimes easier to use the native AWS tools, the configuration centrally audited using tools like Wiz and logged out to consistent storage solutions like CloudWatch or S3.

One disclaimer with this article I probably should have said at the start; this assumes a high level of consistency and guardrails already existing in your environment; if you allow a developer to spin up an EC2 instance without the correct IAM role, SSM simply won’t work. So you will need to have controls such as permission boundaries or Service Control Policies to enforce compliance of said Guard Rails. You may even enforce Infrastructure as Code scans to prevent configurations from being deployed without the pre-requisites needed for this solution to work.