AWS (EKS)

This page covers the EKS-specific pieces for running canette on AWS: using the AWS Load Balancer Controller with Application Load Balancer (ALB) as the Gateway implementation, and AWS Certificate Manager (ACM) for TLS — replacing the cert-manager + Let's Encrypt setup used on generic clusters.

PrerequisitesCopy Anchor Link

• An EKS cluster (Kubernetes 1.27+)
• AWS Load Balancer Controller installed
• Gateway API CRDs installed
• A Route 53 hosted zone for your domain (or any DNS provider)
• An ACM certificate covering your domain and wildcard (*.apps.example.com + canette.example.com)

Gateway setupCopy Anchor Link

The AWS Load Balancer Controller can act as a Gateway API implementation, provisioning an ALB per Gateway resource. TLS termination happens at the ALB using your ACM certificate — no cert-manager required.

DNSCopy Anchor Link

Create a wildcard record in Route 53 pointing *.apps.example.com at the ALB's DNS name. You can use external-dns to manage this automatically, or add the record manually after the ALB is provisioned.

Node groups and build jobsCopy Anchor Link

AL2023 and AL2Copy Anchor Link

Rootless BuildKit requires user namespace support. AL2023 nodes support this out of the box. For AL2 nodes, configure user.max_user_namespaces via a launch template userdata script:

echo "user.max_user_namespaces=28633" >> /etc/sysctl.d/99-user-ns.conf
sysctl -p /etc/sysctl.d/99-user-ns.conf

BottlerocketCopy Anchor Link

Bottlerocket uses SELinux in enforcing mode. The default SELinux policy blocks the mount namespace operations that rootlesskit requires, even though user namespaces are available at the kernel level. The rootless buildkitd pod will fail to start on Bottlerocket nodes.

Use rooted mode instead by setting buildkit.rootless: false:

buildkit:
  rootless: false

This runs buildkitd as root with privileged: true. Privileged containers receive the spc_t SELinux label on Bottlerocket, which bypasses the SELinux policy. No other node-level configuration is needed.

ECR integrationCopy Anchor Link

canette can push build images to Amazon ECR using IRSA (IAM Roles for Service Accounts) for secure, credential-free authentication. ECR repositories are automatically created on first build.

IAM role and policyCopy Anchor Link

Create an IAM role for the canette builder with the following trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/oidc.eks.REGION.amazonaws.com/id/CLUSTER_ID"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.REGION.amazonaws.com/id/CLUSTER_ID:sub": "system:serviceaccount:canette-system:canette-builder"
        }
      }
    }
  ]
}

Replace ACCOUNT_ID, REGION, and CLUSTER_ID with your values.

Attach this permissions policy to the role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ECRRepositoryManagement",
      "Effect": "Allow",
      "Action": [
        "ecr:DescribeRepositories",
        "ecr:CreateRepository"
      ],
      "Resource": "arn:aws:ecr:REGION:ACCOUNT_ID:repository/canette/*"
    },
    {
      "Sid": "ECRAuth",
      "Effect": "Allow",
      "Action": "ecr:GetAuthorizationToken",
      "Resource": "*"
    },
    {
      "Sid": "ECRPush",
      "Effect": "Allow",
      "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:BatchGetImage",
        "ecr:PutImage",
        "ecr:InitiateLayerUpload",
        "ecr:UploadLayerPart",
        "ecr:CompleteLayerUpload"
      ],
      "Resource": "arn:aws:ecr:REGION:ACCOUNT_ID:repository/canette/*"
    },
    {
      "Sid": "ECRScanning",
      "Effect": "Allow",
      "Action": [
        "ecr:StartImageScan",
        "ecr:DescribeImageScanFindings"
      ],
      "Resource": "arn:aws:ecr:REGION:ACCOUNT_ID:repository/canette/*"
    }
  ]
}

Configure Helm valuesCopy Anchor Link

domain: apps.example.com

# Disable in-cluster registry
registry:
  enabled: false

# Configure builder to use ECR
builder:
  imageRepo: "ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/canette/"
  registryAuthType: "irsa"  # or omit to auto-detect from URL
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/canette-builder

# Configure controller to pull from ECR
controller:
  pullRepo: "ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/canette/"

Replace ACCOUNT_ID and REGION with your AWS account ID and ECR region.

Node IAM permissions for image pullsCopy Anchor Link

EKS nodes need ECR read permissions to pull app images. Add this policy to your node IAM role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken",
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchGetImage"
      ],
      "Resource": "*"
    }
  ]
}

This allows nodes to pull images from ECR without requiring imagePullSecrets.

How it worksCopy Anchor Link

1. The builder auto-detects ECR from the imageRepo URL (contains .ecr. and .amazonaws.com)
2. Before each build, the builder checks if the ECR repository exists (e.g., canette/my-project/my-app)
3. If missing, it creates the repository using the IRSA credentials
4. BuildKit inherits the same IRSA credentials to push the built image
5. EKS nodes use their IAM role to pull images from ECR

ECR repositories follow the pattern: canette/{project-slug}/{app-slug}

ECR image scanningCopy Anchor Link

When canette detects an ECR registry (auto-detected from the imageRepo URL), it uses ECR basic scanning instead of running a Trivy K8s Job. After each image push the builder calls StartImageScan, then polls DescribeImageScanFindings until the scan is complete — typically within a minute for small images.

ECR basic scanning is enabled by default when the ECR provider is active (it is free and adds no build infrastructure). You can still disable it or make it mandatory via Helm values:

security:
  scan:
    provider: "auto"   # "ecr" is auto-selected for ECR imageRepo URLs
    enabled: "true"    # quoted string; omit or set "" to use the provider-aware default (enabled for ECR)
    mandatory: false   # set true to block deployments when findings exceed failSeverity
    failSeverity: "HIGH"

The ECRScanning IAM statement in the policy above is required for this feature. If you omit it, the builder will log an access-denied error and mark the scan as errored — the deployment still proceeds unless mandatory: true is set.

ValuesCopy Anchor Link

domain: apps.example.com
ui:
  hostname: canette.example.com

controller:
  gateway:
    name: canette-gateway     # name of your ALB Gateway resource
    namespace: canette-system

InstallCopy Anchor Link

helm install canette oci://ghcr.io/canette/canette \
  --namespace canette-system \
  --create-namespace \
  -f values.yaml