Portfolio · Notes · Dotfiles

Search everything

Search case studies, engineering notes, and Dotfiles documentation.

    all case studies

    Case study 11

    The NAT bill and the fix that went upstream

    NAT instances became the default egress path with managed-gateway fallback. I contributed the deployment changes needed to run the failover function without a container build.

    5 min read

    My role
    Adopted alterNAT, integrated it into the cell platform, and contributed Zip packaging and dependency removal upstream.
    Evidence
    Two merged upstream pull requests; the production configuration uses the Zip deployment path and checks connectivity every minute.
    costnetworking

    Change the cost structure while retaining fallback

    Our outbound traffic passed through managed NAT gateways, whose processing charges grew with traffic volume. I adopted alterNAT to move the normal path onto EC2 NAT instances while retaining managed gateways as fallback.

    This removes the managed gateway’s per-gigabyte processing charge from traffic that stays on the instance path. EC2, standby gateways, and other applicable network charges remain part of the cost. The trade-off is operating NAT instances and their health checks in exchange for a different cost structure.

    Make the failure path part of the design

    alterNAT checks connectivity every minute. Failed checks or an instance termination trigger a route change to the managed gateway. A replacement instance restores the instance route during startup. Upstream also supports health-based restoration of a recovered instance through a separate opt-in setting.

    ALTERNAT · NORMAL ROUTE AND MANAGED FALLBACK
    Alternat Routing Simplified alterNAT routing for one availability zone. A private workload uses a default route through an EC2 NAT instance and its Elastic IP. If a connectivity check fails or the instance is replaced, Lambda changes the route to a standby managed NAT gateway with a different Elastic IP. Both paths reach the internet through the VPC internet gateway. A replacement instance restores the instance route when it boots. ONE AVAILABILITY ZONE · TWO POSSIBLE EGRESS PATHS Private workloads Outbound traffic Private route table 0.0.0.0/0 NORMAL PATH FALLBACK PATH EC2 NAT instance Auto Scaling group Elastic IP A Managed NAT gateway Standby gateway Elastic IP B Internet gateway VPC egress Internet DETECT Checks every minute or termination event SWITCH Lambda updates the default route RESTORE Replacement boots and restores route
    EXHIBIT 01 — One zone shown; each configured zone has its own instance and standby gateway. The routes use separate Elastic IPs. Based on the upstream architecture.

    The check interval is a detection cadence, not a guarantee of recovery within one minute.

    I introduced the system in the main VPCs, then made it an option in the cell platform. The configuration uses ARM instances. It also allocates separate Elastic IPs for the instance and fallback gateway, allowing both egress addresses to be known and allowlisted ahead of a switch.

    Remove an unnecessary deployment dependency

    The version of alterNAT we adopted required a container image for its failover function. That added a build and registry dependency to infrastructure automation running through pull requests.

    I contributed two changes upstream:

    • Zip deployment support, PR #44, allowing the function to use the standard Lambda runtime without an image registry.
    • Dependency removal, PR #52, replacing requests with the standard library and removing the Terraform-time dependency installation step. The function still uses boto3, supplied by the Lambda runtime.

    The cell configuration uses that deployment path:

    cell/nat.tf
    # Selected inputs from the alterNAT module invocation.
    architecture        = "arm64"
    lambda_package_type = "Zip"
    create_nat_gateways  = true

    What changed

    Normal traffic gained an instance-based egress path with automatic managed-gateway fallback. The integration fit the existing Terraform workflow, and the deployment changes shipped upstream, where they could be maintained and reused beyond our own installation.