Tostada

Overview

In the previous PrivateLink post, we went through the separate resources that make up AWS PrivateLink. In this post, we will be provisioning a PrivateLink configuration which will allow resources in one VPC to connect to a web service in another VPC. You can use several different AWS services to accomplish the same goal, but PrivateLink can simplify some setups and meet security expectations with its standard one-way communication.

PrivateLink

Lesson

  • Building Two VPCs
  • Creating A Web Service To Share Via PrivateLink
  • Deploying PrivateLink Resources in Provider VPC
  • Provision PrivateLink Resources in Consumer VPC
  • Validating Connectivity From The Consumer VPC To Provider VPC

Building Two VPCs

A prerequisit for creating a PrivateLink connection is at least two VPCs. Remember that the VPC providing the service can interface with more than one consumer VPCs. These consumer VPCs can live in the same or different accounts, but they must be in the same region. That being said, we’ll start with creating two VPCs. This module will also create a jumpbox to allow you to connect into the VPC and a wildcard certificate. If you’d like to see the code, you can view it here.

module "vpc" {
  source = "../../modules/vpc"

  for_each = {
    provider = {
      cidr_block         = "10.1.0.0/16"
      vpc_name           = "provider-prod"
    },
    consumer = {
      cidr_block         = "10.2.0.0/16"
      vpc_name           = "consumer-prod"

    }
  }

  cidr_block              = each.value.cidr_block
  env                     = "prod"
  enable_jumpbox_instance = true
  vpc_name                = each.value.vpc_name
}

Note: if you plan on deploying this module on your own, you will need to have a public DNS zone in the AWS account use the public_domain_name to specify your domain.

We now have a provider VPC to start building a new service we’ll call “tostada”. For the sake of keeping things simple of this demo, the service is comprised of an EC2 instance with nginx installed, a security group, and Application Load Balancer resources (ALB, listener, listener rule, target group.)

resource "aws_security_group" "tostada" {

  name        = "tostada-${local.vpc_name}"
  description = "${local.vpc_name} tostada security group"
  vpc_id      = data.aws_vpc.selected.id

  ingress {
    description      = "TLS from VPC"
    from_port        = 443
    to_port          = 443
    protocol         = "tcp"
    cidr_blocks      = [data.aws_vpc.selected.cidr_block]
  }

  ingress {
    description      = "HTTP from VPC"
    from_port        = 80
    to_port          = 80
    protocol         = "tcp"
    cidr_blocks      = [data.aws_vpc.selected.cidr_block]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = merge(
    local.common_tags,
    tomap({
      "Name" = "tostada-${local.vpc_name}"
    })
  )
}

resource "aws_instance" "tostada" {

  ami                    = data.aws_ami.amazon_linux_2.id
  instance_type          = "t4g.nano"
  subnet_id              = data.aws_subnets.private.ids[0]
  vpc_security_group_ids = [aws_security_group.tostada.id]
  user_data              = file("${path.module}/files/user-data.sh")

  tags = {
    Name = "tostada-${local.vpc_name}"
  }
}

resource "aws_lb_target_group" "tostada" {
  name     = "tostada"
  port     = 80
  protocol = "HTTP"
  vpc_id   = data.aws_vpc.selected.id
}

resource "aws_lb" "tostada" {
  name               = "tostada"
  internal           = true
  load_balancer_type = "application"
  security_groups    = [aws_security_group.tostada.id]
  subnets            = data.aws_subnets.private.ids

}

resource "aws_lb_listener" "tostada" {
  load_balancer_arn = aws_lb.tostada.arn
  port              = "443"
  protocol          = "HTTPS"
  ssl_policy        = "ELBSecurityPolicy-2016-08"
  certificate_arn   = data.aws_acm_certificate.issued.arn

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.tostada.arn
  }
}

resource "aws_lb_target_group_attachment" "tostada" {
  target_group_arn = aws_lb_target_group.tostada.arn
  target_id        = aws_instance.tostada.id
  port             = 80
}


resource "aws_route53_record" "lb_cname" {
  zone_id = data.aws_route53_zone.private.zone_id
  name    = "tostada.${local.private_domain_name}"
  type    = "CNAME"
  ttl     = 300
  records = [aws_lb.tostada.dns_name]
}

Note: To see the entire module, you can visit the tostada service module

Once this module is done provisioning, you can use a tool like aws-connect to “ssm” into the jumpbox in the same provider VPC as the tostada service and verify that the service is online via curl:

$ aws-connect -n jumpbox0-provider-prod -r us-east-2


Establishing session manager connection to jumpbox0-provider-prod (i-044687cbb5f964344)

Starting session with SessionId: taccoform-0cad163c26c2c8610
sh-4.2$ curl https://tostada.provider-prod.tacoform.com
<html><body><h1>IT'S TOSTADA TIME</h1></body></html>

At the same time of the tostada service provisioning, additional resource will be created in the provider-prod VPC to support the PrivateLink connection. This will include a VPC Endpoint Service, Network Load Balancer, and Target Group attachment to the tostada service’s ALB:

resource "aws_lb" "privatelink" {
  enable_cross_zone_load_balancing = false
  internal                         = true
  load_balancer_type               = "network"
  name                             = "tostada-privatelink-nlb"
  subnets                          = data.aws_subnets.private.ids
}

resource "aws_lb_target_group" "privatelink" {
  name        = "tostada-privatelink-tg"
  port        = "443"
  protocol    = "TCP"
  target_type = "alb"
  vpc_id      = data.aws_vpc.selected.id

  health_check {
    matcher  = "200"
    path     = "/"
    port     = "443"
    protocol = "HTTPS"
  }
}

resource "aws_lb_listener" "privatelink" {
  load_balancer_arn = aws_lb.privatelink.arn
  port              = 443
  protocol          = "TCP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.privatelink.arn
  }
}

resource "aws_lb_target_group_attachment" "privatelink" {
  port             = aws_lb_listener.privatelink.port
  target_group_arn = aws_lb_target_group.privatelink.arn
  target_id        = aws_lb.tostada.arn
}


resource "aws_vpc_endpoint_service" "privatelink" {
  acceptance_required        = false
  network_load_balancer_arns = [aws_lb.privatelink.arn]
  private_dns_name           = "tostada.${local.private_domain_name}"
}

resource "aws_route53_record" "privatelink" {
    name    = "${aws_vpc_endpoint_service.privatelink.private_dns_name_configuration[0]["name"]}."
    records = [aws_vpc_endpoint_service.privatelink.private_dns_name_configuration[0]["value"]]
    type    = "TXT"
    ttl     = "300"
    zone_id = data.aws_route53_zone.public.zone_id
}


resource "aws_vpc_endpoint_service_allowed_principal" "allowed_aws_accounts" {
  for_each = toset(var.allowed_aws_accounts_list)

  vpc_endpoint_service_id = aws_vpc_endpoint_service.privatelink.id
  principal_arn           = "arn:aws:iam::${each.key}:root"
}

Note: To see the entire module, you can visit the tostada service module

Some things to note for the Provider VPC PrivateLink resources:

  • You want to enable auto acceptance on the VPC endpoint Service resource to make things more automated
  • Enabling private DNS on the VPC Endpoint Service resource will allow Consumer VPCs to use your domain and HTTPS connections (eg. tostada.provider-prod.tacoform.com instead of the Amazon provided DNS name)
  • The VPC Endpoint Service Allowed Principal resource is what will allow Consumer VPCs in different AWS accounts to connect to this Provider VPC. You will need to populate this list prior to attempting any connectivity from a Consumer VPC.
  • The Route53 record is required to automatically verify ownership of the domain to allow you to use your own DNS Zone.
  • When AWS says “private DNS”, it just means your own domain (eg. tacoform.com). It doesn’t mean private or public DNS zones.
  • During initial provisioning, you may run into errors related to the private DNS not validating in time, you can re-run the validation step by grabbing the VPC Endpoint Service ID and re-validating via awscli:
$ VPCE_SERVICE_ID=$(terraform output -raw vpce_service_id)
$ aws ec2 start-vpc-endpoint-service-private-dns-verification --service-id $VPCE_SERVICE_ID --region us-east-2

After the “private DNS” has been verified, you are now ready to connect Consumer VPCs to the tostada service in your Provider VPC. To do this, I would start with another terraform workspace/stack/etc. You only need to provision a Security Group and VPC Endpoint resource, but I would still put them into a module:

resource "aws_security_group" "consumer" {
  name        = "tostada-${local.vpc_name}"
  description = "${local.vpc_name} tostada security group"
  vpc_id      = data.aws_vpc.selected.id

  ingress {
    description = "TLS from VPC"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = [data.aws_vpc.selected.cidr_block]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = merge(
    local.common_tags,
    tomap({
      "Name" = "tostada-${local.vpc_name}"
    })
  )
}


resource "aws_vpc_endpoint" "privatelink" {
  private_dns_enabled = true # allows custom domain usage instead of AWS provided DNS name
  service_name        = data.aws_vpc_endpoint_service.privatelink.service_name
  security_group_ids  = [aws_security_group.consumer.id]
  subnet_ids          = data.aws_subnets.private.ids
  vpc_endpoint_type   = "Interface"
  vpc_id              = data.aws_vpc.selected.id
}

Note: To see the entire module, you can visit the VPC Endpoint module

If you put the two resources in a module, you can group together Consumer VPCs in the same AWS account to avoid needing separate workspaces per Consumer VPC:

module "service_consumer" {
    source = "../../modules/vpc-endpoint"

    for_each = {
        consumer-prod = {
            vpc_name = "consumer-prod"
        },
    }

    env                = "prod"
    service            = "tostada"
    vpc_name           = each.value.vpc_name
}

Note: To see the entire module, you can visit the VPC Endpoint module

Validating Connectivity From The Consumer VPC To Provider VPC

Once the module is provisioned in the service-consumer workspace, you can move on to validating the HTTPS connection from a jumpbox in the Consumer VPC to the tostada service in the Provider VPC. You can use the aws-connect tool again to “ssm” into the Consumer VPC’s jumpbox and curl the endpoint to validate connectivity:

$ aws-connect -n jumpbox0-consumer-prod -r us-east-2
Establishing session manager connection to jumpbox0-consumer-prod (i-01f12a90d03e99d60)

Starting session with SessionId: taccoform-05f4d141e79540cf6
sh-4.2$ curl https://tostada.provider-prod.tacoform.com
<html><body><h1>IT'S TOSTADA TIME</h1></body></html>

You can also check DNS with dig to see how the Consumer VPC DNS resolves the request for tostada.provider-prod.tacoform.com and points them to Elastic Network Interfaces (ENIs) assigned to the VPC Endpoint for the tostada PrivateLink config:

$ dig tostada.provider-prod.tacoform.com +noall +answer

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.amzn2.5.2 <<>> tostada.provider-prod.tacoform.com +noall +answer
;; global options: +cmd
tostada.provider-prod.tacoform.com. 60 IN A     10.2.3.124
tostada.provider-prod.tacoform.com. 60 IN A     10.2.4.142

Note: Notice how the IPs live within the CIDR block we’ve created for the Consumer VPC (10.2.0.0/16)

If you deployed the vpc, tostada, and vpc-endpoint modules, be sure to destroy those workspaces to avoid any unnecessary AWS charges

In Review

We’ve set up two VPCs (one Consumer, one Provider) and created a one way trust allowing resources in the Consumer VPC to reach the tostada web service in the Provider VPC. This simple example can be expanded by mapping one Provider VPC service to many Consumer VPCs (as long as they are in the same region which is an AWS limitation.) In the future, AWS can expand the resources to share across PrivateLink which will make it even more useful. That being said, the main use-case of sharing web services is compelling enough to start using PrivateLink.

As always, feel free to reach out on twitter via @taccoform for questions and/or feedback on this post