In the previous post in this series I covered the fundamentals of networking on AWS. The central resource for networking on AWS is the Virtual Private Cloud (VPC).

You can control the traffic that is allowed to flow between different EC2 instances and other compute services or database services in your VPC through a few different means.

The basic components for controlling traffic in a VPC are security groups and network access control lists (NACLs). Security groups are stateful and NACLs are stateless. These types of resources form the foundation of network security on AWS, and should not be neglected even after the introduction of more advanced components.

A more advanced and feature-rich appliance for protecting resources in your VPCs is the AWS Network Firewall. For simpler network architectures the AWS Network Firewall is often not included, yet it could be worth considering when network security is important – which it almost always is.

In this blog post we will dive deeper into what the AWS Network Firewall is, how you can provision one using Terraform and best practices for managing your AWS Network Firewall in your cloud environment.

What is AWS Network Firewall?

There are a few options on AWS for controlling what traffic is allowed to flow in your virtual networks.

First, there are security groups. A security group is a stateful construct that you can attach to specific endpoints or services and they allow you to specify what traffic is allowed in or out of the security group.

Next, there are Network Access Control Lists (NACLs) that you can attach to specific subnets to allow or deny traffic for the whole subnet. NACLs are stateless, meaning you have to allow both outgoing traffic and the response.

Security groups and NACLs are limited in functionality. You mostly work directly with IP addresses and ports.

AWS Network Firewall is a service that offers much more granular options for controlling traffic to and from your AWS environment. This is a fully managed service offering by AWS, removing the need for you to manage the underlying resources for running a third-party firewall appliance in your AWS environment.

The AWS Network Firewall service is arguably an expensive service, with pricing starting at $0.395 per hour in the us-east-1 region. This pricing does not include traffic processing costs or other advanced features.

Due to the pricing for the AWS Network Firewall a common architecture solution you will see is to use a single firewall instance for each environment (dev, stage, prod). In some situations you might even use a single instance for your whole organization.

The Network Firewall inspects traffic using two different inspection engines: a stateless rules engine and a stateful rules engine. Stateless rules are similar in spirit to NACL rules for subnets, while stateful rules are similar to VPC security group rules. You can configure your firewall to only use one type of inspection or both.

Traffic is processed by first going through the stateless rules engine, followed by the stateful rules engine. If a matching rule is found, the configured action will be taken and processing will stop.

For stateless rules, each packet is inspected independently from other packets. Stateless rules are fast, and this is why they are processed first. For these types of rules you should include basic filtering, for instance to block traffic to or from specific IP addresses.

Stateful rules on the other hand take the context into account. These rules are slower, but allow much stronger inspections. Stateful rules can inspect layer 7 content, for instance looking at the content body of a HTTP request or HTTP headers. This allows you to build very specific rules of what you want to allow or block.

Building firewall rules is a complex topic, and we will not dive deeper into this topic in this blog post. To start designing firewall rules, see the AWS documentation.

Provision and manage an AWS Network Firewall using Terraform

When you provision an AWS Network Firewall (referred to as simply firewall in the following text) through Terraform there are a few resources you need to configure.

First of all, as with most networking on AWS it all starts with an AWS Virtual Private Cloud (VPC). This resource was covered in detail in the previous article in this series.

When working with a firewall there is an important detail to keep in mind. The resource should be placed in dedicated subnets. The reason for this is because you would not be able to inspect the traffic from resources in the same subnet as the firewall resource.

The details of how to configure the full VPC and its related resources can be found in the previous article, in the following walkthrough we assume we have set up the basic VPC configuration shown in the following image.

In essence we have a VPC with three public subnets and three private subnets. In the private subnets we can have our workloads of different types. All outbound and inbound traffic for the VPC will pass through the firewall endpoints that are deployed in dedicated subnets.

The most important resource type when working with firewalls is the network firewall policy. In the Terraform world this is the aws_networkfirewall_firewall_policy resource type. This is the resource where you configure what rules should apply for the inspected traffic.

Before you configure the firewall policy you can provision individual rules in a resource known as a rule group. In the Terraform world this is known as the aws_networkfirewall_rule_group resource type. An example of configuring a rule group with a single rule denying traffic to a specific target looks like this:

resource "aws_networkfirewall_rule_group" "deny_hackers_com" {
  capacity = 100
  name     = "deny-hackers-com"
  type     = "STATEFUL"
  rule_group {
    rules_source {
      rules_source_list {
        generated_rules_type = "DENYLIST"
        target_types         = ["HTTP_HOST"]
        targets              = ["hackers.com"]
      }
    }
  }
}

There are a lot of different types of rules you can create and the full details are beyond the scope of this blog post.

You can use this rule group in a firewall policy. A simple firewall policy that defines a few default actions as well as references the previous rules group looks like this:

resource "aws_networkfirewall_firewall_policy" "anyshift" {
  name = "example"
  firewall_policy {
    stateless_default_actions          = ["aws:pass"]
    stateless_fragment_default_actions = ["aws:drop"]
    stateful_rule_group_reference {
      resource_arn = aws_networkfirewall_rule_group.deny_hackers_com.arn
    }
  }
  tags = {
    Name = "firewall-policy-${var.aws_region}"
  }
}

A firewall policy can be reused in multiple firewalls if desired. However, a firewall can only have a single firewall policy. Speaking of firewalls, with the firewall policy in place you can configure the firewall resource itself:

resource "aws_networkfirewall_firewall" "anyshift" {
  name                = "anyshift"
  firewall_policy_arn = aws_networkfirewall_firewall_policy.anyshift.arn
  vpc_id              = aws_vpc.default.id
  subnet_mapping {
    subnet_id = aws_subnet.firewall["eu-west-1a"].id
  }
  subnet_mapping {
    subnet_id = aws_subnet.firewall["eu-west-1b"].id
  }
  subnet_mapping {
    subnet_id = aws_subnet.firewall["eu-west-1c"].id
  }
  tags = {
    Name = "anyshift-firewall"
  }
  timeouts {
    create = "40m"
    update = "50m"
    delete = "1h"
  }
}

Note how you configure a subnet mapping for each subnet that the firewall should be active in. In this case I have added three subnet mappings, one for each availability zone in the eu-west-1 region (Ireland).

An important detail related to the VPC network is that you need to route traffic through the firewall from other subnets. To achieve this you need to parse attributes of the firewall resource to obtain the VPC endpoints that it has created in each of its subnets:

locals {
  sync_states = tolist(aws_networkfirewall_firewall.anyshift.firewall_status[0].sync_states)
  endpoint01  = local.sync_states[0].attachment[0].endpoint_id
  endpoint02  = local.sync_states[1].attachment[0].endpoint_id
  endpoint03  = local.sync_states[2].attachment[0].endpoint_id
}
Now you can configure routes for route tables attached to other subnets to go through the firewall:
resource "aws_route" "example" {
  route_table_id         = aws_route_table.workload.id
  destination_cidr_block = "0.0.0.0/0"
  vpc_endpoint_id        = local.endpoint01
}

If you require TLS inspection in the firewall you need to configure a few more resources. The full details are out of scope for this blog post, but in general you need to have your TLS certificates in AWS Certificate Manager (ACM). Then you can create a TLS inspection configuration resource that refers to the certificate in ACM:

resource "aws_networkfirewall_tls_inspection_configuration" "example" {
  name        = "example"
  description = "example"
  tls_inspection_configuration {
    server_certificate_configuration {
      server_certificate {
        # reference to your server certificate
        resource_arn = aws_acm_certificate.example.arn
      }
      scope {
        protocols = [6]
        destination_ports {
          from_port = 443
          to_port   = 443
        }
        destination {
          address_definition = "0.0.0.0/0"
        }
        source_ports {
          from_port = 0
          to_port   = 65535
        }
        source {
          address_definition = "0.0.0.0/0"
        }
      }
    }
  }
}

Then you would connect this configuration to your firewall policy resource.

As an SRE or network administrator we are interested in collecting logs from the firewall. This can be set up using the aws_networkfirewall_logging_configuration resource.

You can send the logs from the firewall to three separate destinations:

• An S3 bucket: this is a good option for cheap long-term storage (e.g. for compliance reasons).

• A CloudWatch log group: this is a good option as an aid in troubleshooting in your AWS environment when issues occur.

• A Kinesis Data Firehose stream: this is a good option if you need to process the logs in some way or integrate with third party systems.

For simplicity in our setup, let's send the logs to CloudWatch. 

Set up a CloudWatch log group as the target for the firewall logs:

resource "aws_cloudwatch_log_group" "firewall" {
  name              = "aws-network-firewall"
  retention_in_days = 7
}

Set the retention_in_days argument to 7 to store logs for the past week. If you expect the amount of logs to be huge you could decrease the retention to 5, 3, or even 1 day.

With the CloudWatch log group in place you can set up the firewall logging configuration:

resource "aws_networkfirewall_logging_configuration" "cloudwatch" {
  firewall_arn = aws_networkfirewall_firewall.anyshift.arn
  logging_configuration {
    log_destination_config {
      log_destination = {
        logGroup = aws_cloudwatch_log_group.firewall.name
      }
      log_destination_type = "CloudWatchLogs"
      log_type             = "FLOW"
    }
  }
}

The firewall supports resource policies for its firewall policies and rule groups. This is similar to how you can configure resource policies for S3 buckets or SNS topics.

Resource policies are required if you are using your firewall in a multi AWS account setup and need to delegate permissions to associate firewall resources to firewall policies across accounts. In this case we will not bother configuring resource policies, if you have the need for this look at the aws_networkfirewall_resource_policy resource.

The AWS Network Firewall resource is complex. The complexity lies in configuring the rules that should be processed by the firewall. To get started configuring rules for your environment you should look into the AWS provider documentation and dive into the documentation on the AWS Network Firewall resource itself.

Best Practices for the AWS Network Firewall

Use more than one AWS Network Firewall

It is tempting to save on costs by provisioning a single AWS Network Firewall and sharing it among all of your cloud environments (dev, stage, prod). However, this increases the possibility that a bad firewall rule update will affect multiple applications across multiple environments.

If you feel that updating firewall rules for your development environment is a scary operation because it might affect your production environment – then this is a sign you need more than one AWS Network Firewall.

Manage your AWS Network Firewall using Terraform

This has already been hinted at in all of this blog post. Use Terraform or another infrastructure as code tool to manage your AWS Network Firewall. This allows you to use versioning of the firewall configuration, promoting collaboration and visibility. In case of issues with a firewall update you can roll back the change easily by applying the last known good configuration.

Build reusable rule groups and firewall policies

Set up a plan for reusing rule groups and firewall policies. This allows you to introduce new rules in a development environment and test them out before you promote the same rules into the staging and eventually  the production environment.

Managing firewall rules is complex, this is why you should reuse the same rules and policies if it is viable in your environment.

Avoid single point of failure

Always deploy more than one firewall endpoint across multiple availability zones in an AWS region. You should generally avoid single points of failure in all your important architectural choices.

Use stateless and stateful rules in unison

Stateless rules are fast and allow you to run basic sanity-checking on incoming or outgoing traffic. You could block traffic to known bad IP addresses or block traffic to and from specific ports.

Use stateful rules when traffic content and context is important. Avoid building too large collections of stateful rules since this could slow down traffic and also increase the costs of your AWS Network Firewall.

Use multiple layers of security

Even after introducing an AWS Network Firewall into your environment you still need to use security groups for your endpoints and NACLs for your subnets. A layered approach to security is required. Remember that some of your internal network traffic might not go through the AWS Network Firewall.

Monitor your AWS Network Firewall

SREs and network administrators require insights into what happens in your AWS Network Firewall to be able to troubleshoot. Add monitoring and logging from day 1. Consider how logs will be consumed and add the appropriate logging configuration. If you are primarily using AWS monitoring you should opt for using AWS CloudWatch logs. If you use an external monitoring solution you could either integrate with CloudWatch or use the Kinesis Firehose logging integration for the firewall.

For long term log storage you can use S3. Note that storing logs in S3 is inconvenient from a real time monitoring situation.

Terraform and Anyshift for AWS Network Firewall

An AWS Network Firewall is often placed centrally to allow inspection of all traffic entering and leaving a given network perimeter. This could be in a hub VPC in a hub-and-spoke network architecture.

In this situation it is imperative that you have a workflow for how you introduce changes into your environment. A bad update to your firewall could potentially affect all applications and users that your infrastructure supports.

Anyshift is a tool that creates a digital twin of your AWS environment and together with your Terraform state files and Terraform configurations has a complete AWS knowledge graph to base insights on. This allows Anyshift to have great insight into your current context and can judge changes to your Terraform infrastructure accurately to assess potential impact.

In the context of the AWS Network Firewall this means Anyshift can know about any hidden dependencies that exist between Terraform configurations such as resource traffic that is allowed to pass through the firewall where resources are defined in different Terraform configurations than the network firewall. The biggest danger when working with the AWS Network Firewall is adding, updating, or removing firewall rules from firewall policies.

An SRE AI-copilot like Anyshift that can inform you before-the-fact that a change in your Terraform configuration could lead to potential issues in a different Terraform configuration. This is a powerful feature that could reduce the on-call load on your SREs. The only better option to resolving issues in your cloud infrastructure quickly is to avoid the issue to start with.

Visit the documentation for more on Anyshift.

Conclusions

Given the importance of an AWS Network Firewall resource in your AWS cloud environments, it is critical that you handle changes to this resource carefully. The firewall is often placed in a central location to allow all ingoing and outgoing traffic to be inspected. 

The AWS Network Firewall is a complex resource. It combines the features of both Network Access Control Lists (NACLs) for subnets and security groups. However, it offers much deeper network traffic inspection capabilities than any other managed solution on AWS.

The AWS Network Firewall is a fully managed resource provided by AWS. You can achieve the same level of protection with third-party self-managed firewalls. However, the simplicity of using managed services is hard to beat.