There is quite a lot of high profile hack due to supply chain attack. Essentially one of populate package got hack, inject with malicious code, then our application naively upgrade to that package and now we're running malicious code in our production env.
Some example such as log4j and gitlab's github import https://hackerone.com/reports/1679624
How can we defend these kind of attack? Once someone is able to run untrusted code somehow, what can we do to reduce the damage. IMHO if we limit their ability to reach out to their mothership to upload information then we can reduce the damage locally, they can delete our file, destroy our database but won't be able send out anything out.
There is also a kind of application that have to run untrusted code, such as a live editor which enable people to compile and run code on a shared environment.
So I want to start a series to share my though about these defense. In first part, we will talk about how we can limit outgoing connection.
What is limit outgoing connection
Whenever a process want to transfer data out, it has to use either UDP or TCP. There is way to use ICMP to transfer data but it's much trickier, and to make thing simple we gonna look at UDP and TCP first.
UDP and TCP generally works by having the network stack open a connection by giving a pair of (destination ip, destination port). Then a connection to this IP address on that port is made.
So limit outgoing connection means we will setup firewall rule so that only a certain destination IP address or destination port are allow. We can set configure this for non root user so our root user can continue to update the node, run apt, docker pull for example.
Why is limit outgoing connection useful
When a piece of code runs inside our infra, it most likely want to ship the information out, such as creating an SSH tunnel out, or uploading your environment variable which usually contains credential out.
If we prevent this then there is no way for data to be transfered out. The server is pretty much lock down. Imagine someone get into a room with full of treasure but they themselves cannot take any of treasure out.
How can I do it
We can implement this in a few way either in DIY style or using cloud tooling
Using iptables
If the node has no need to make any outgoing connection except to your VPC (database, other deployment service) then you can do
First, we set the defaulut output policy to "DROP" so by default it drops everything, and we implement an allow list
iptables -P OUTPUT DROP
Then say we want to allow it access to our VPC subnet:
iptables -A OUTPUT -d 10.0.0.0/24 -j ACCEPT
If our service also using monitoring system like Sentry or any API we need to find their IP and whitelist as well
iptables -A OUTPUT -d ip-address-of-that-service -j ACCEPT
Normally those services will have a list of their public IP so we may need some sort of cron to update this list.
Now, what is interesting is that we cannot run apt update or docker pull anymore? We can allow root to make HTTP request and DNS request
iptables -A OUTPUT -m owner--uid-owner root -j ACCEPT -p tcp --dport 80 -j ACCEPT
iptables -A OUTPUT -m owner--uid-owner root -j ACCEPT -p tcp --dport 443 -j ACCEPT
iptables -A OUTPUT -m owner--uid-owner root -j ACCEPT -p udp --dport 53 -j ACCEPT
This works for most of the node behind a load balancer. Especially server where it just need to talk to our database, or other server in the same subnet. This works great for those kind of server. It's better if we don't run any application code on the node that can be reached publicly. The node in public subnet should only do load balancing and hold no sensitive secret or application code. What happen if we need the load balancer to be able to return the data to the incoming established request? As in the node cannot make new connection out, but if someone established a connection in first, then you node should be able to return data. Example when we run a load balancer itself to respond to traffic to port 80 or 443. We simply add the below rule
iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
conntrack is a part of Linux network stack which can be view as a big hash table to track a tcp connection and see where this package belongs to. It's usually has 3 state:
NEW: mean this package is for create a new connection, or we never send package in either direction
ESTABLISHED: this package belongs to an established connection
RELATED: this package is a new connection, but it's a result of an existing connection
Setting to ESTABLISHED and RELATED allow the responding package flow back.
Using AWS
On AWS We have twoo primitives are Network ACL or Security Group. We can use both or either but they are a bit different.
Using Network ACL
Network ACL is stateless, and operate at subnet level, so the rule apply to anything, even for established connection. So if our node require responding to external traffic, we cannot use this method.
Let's say we run a load balancer. A user visit our website. Then the TCP connection will be
user (local port 36000 for example) -> connect to our website on port 443
Now, our server needs to send back the data to port 36000 but because we block outgoing traffic, the data cannot be retuned. Therefore we cannot using Network ACL.
However, if our node run in a private subnet we can simply edit "Outbound rule" of the network ACL to only allow outbound rule of the subnet cidr.
Using Security Group
Security group operate at instance level and stateful. It means the outgoing packet respond to an incoming request that match a security group rule, then that rule will be used for outgoing request as well.
So we can block outbound traffic on ephemeral ports if the node initiated connection. However, if the node respond to an established connection, the incoming rule is applied and traffic can be flow.
So in the security group, edit "Outbound rule" and we can block all outgoing traffic and just allow the subnet vpc by edit it to ensure only one rule that contains the subnet CIDR is existed. Make sure to delete the default 0.0.0.0/0 that AWS may pre-generated for you.
How do I updated my node
Now that the node cannot make request outside such as APT or github or docker ? how can I update it? Using security group we cannot know which user is which to allow "root" like ip tables.
The short answer is it depends. A few thing we can do such as temporarily attach a security group that allow outbound when we update the node such as doing routine OS patching, or run docker pull then after that, we revoked the security group. This can be a part of any automation that you used for deployment.
Be careful to allow outgoing traffic to public known website
Note that it's wrong to allow outgoing to Github for example is fine because Github not gonna hack us. The attacker can used Github API/Gist API as a way to store data, and now they will be able to transfered data out.
The best is just make the node isolated completely and use other means to update code.
That's it for today, next time we can look into setting up forward proxy to route external outgoing traffic to some central node and analyze it.