In-cluster registry

canette ships with an optional in-cluster container registry (registry:2) enabled by default. This page explains how it works, how credentials are managed, and how to set up TLS for local development clusters where Let's Encrypt isn't an option.

How the registry worksCopy Anchor Link

When registry.enabled=true (the default), the Helm chart sets up:

• A registry:2 deployment in the canette-system namespace, listening on port 5000
• An HTTPRoute that exposes it at registry.{domain} through your Gateway
• Auto-generated credentials stored in the canette-registry-credentials Secret, reused on every helm upgrade
• A docker config Secret (canette-registry-auth) mounted into every build job automatically

There are two distinct paths for registry traffic:

BuildKit pushes over plain HTTP to the internal ClusterIP address. Kubelets pull over HTTPS through the Gateway, so TLS termination on the Gateway is required for image pulls to work.

Storage backendCopy Anchor Link

The in-cluster registry supports two storage backends: filesystem (default) and S3-compatible object storage.

Filesystem storage (default)Copy Anchor Link

By default, the registry stores images on a PersistentVolumeClaim backed by your cluster's default StorageClass:

registry:
  storage:
    type: filesystem  # default
  persistence:
    enabled: true
    size: 20Gi
    # storageClass: ""  # uses cluster default

This works well for development, staging, and single-cluster production deployments. The PVC is tied to a single availability zone and requires manual resizing.

S3 storageCopy Anchor Link

For production deployments requiring durability, disaster recovery, or multi-cluster registry sharing, you can use S3-compatible object storage instead:

registry:
  storage:
    type: s3
    s3:
      bucket: my-canette-registry
      region: us-west-2
      accessKey: AKIA...
      secretKey: ...

When storage.type: s3, the PVC is not created and the registry stores all image layers in the S3 bucket.

When to use S3 vs filesystemCopy Anchor Link

Supported S3-compatible providersCopy Anchor Link

The registry's S3 driver works with any S3-compatible API:

• AWS S3 — leave endpoint empty to use AWS defaults
• MinIO (self-hosted) — requires pathStyle: true
• DigitalOcean Spaces
• Backblaze B2
• Wasabi
• Cloudflare R2

AWS S3 exampleCopy Anchor Link

registry:
  storage:
    type: s3
    s3:
      bucket: my-canette-registry
      region: us-west-2
      # Leave endpoint empty for AWS S3
      endpoint: ""
      pathStyle: false
      # Credentials via Secret (recommended)
      secretRef:
        name: canette-s3-credentials
        accessKeyKey: access-key
        secretKeyKey: secret-key
      # Enable server-side encryption
      encrypt: true
      secure: true

Create the credentials Secret before installing:

kubectl create secret generic canette-s3-credentials \
  --namespace canette-system \
  --from-literal=access-key=AKIA... \
  --from-literal=secret-key=...

IAM Policy (attach to the IAM user):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": "arn:aws:s3:::my-canette-registry"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::my-canette-registry/*"
    }
  ]
}

MinIO exampleCopy Anchor Link

registry:
  storage:
    type: s3
    s3:
      bucket: canette-registry
      region: us-east-1  # MinIO requires a region, any value works
      endpoint: https://minio.example.com
      pathStyle: true  # MinIO requires path-style access
      accessKey: minioadmin
      secretKey: minioadmin
      encrypt: false  # MinIO handles encryption separately
      secure: true

DigitalOcean Spaces exampleCopy Anchor Link

registry:
  storage:
    type: s3
    s3:
      bucket: my-canette-registry
      region: nyc3  # Must match Spaces region
      endpoint: https://nyc3.digitaloceanspaces.com
      pathStyle: false
      accessKey: DO00ABC...  # Spaces access key
      secretKey: ...
      secure: true

Create the bucket via the DigitalOcean Control Panel, then generate Spaces access keys under API → Spaces Keys.

Backblaze B2 exampleCopy Anchor Link

registry:
  storage:
    type: s3
    s3:
      bucket: my-canette-registry
      region: us-west-001  # Must match B2 region
      endpoint: https://s3.us-west-001.backblazeb2.com
      pathStyle: false
      accessKey: 00abc...  # Application Key ID
      secretKey: K00...    # Application Key
      secure: true

Create the bucket in the B2 dashboard, then generate an Application Key with read/write access to the bucket.

Wasabi exampleCopy Anchor Link

registry:
  storage:
    type: s3
    s3:
      bucket: my-canette-registry
      region: us-east-1  # Must match Wasabi region
      endpoint: https://s3.us-east-1.wasabisys.com
      pathStyle: false
      accessKey: WASABI-ACCESS-KEY
      secretKey: WASABI-SECRET-KEY
      secure: true

Available Wasabi regions: us-east-1, us-east-2, us-west-1, eu-central-1, ap-northeast-1

Cloudflare R2 exampleCopy Anchor Link

registry:
  storage:
    type: s3
    s3:
      bucket: my-canette-registry
      region: auto  # R2 uses automatic region selection
      endpoint: https://<account-id>.r2.cloudflarestorage.com
      pathStyle: false
      accessKey: <r2-access-key>
      secretKey: <r2-secret-key>
      secure: true  # R2 encrypts at rest by default

Get your account ID and endpoint from Cloudflare dashboard → R2 → Overview. Generate API tokens under R2 → Manage R2 API Tokens.

Migrating from filesystem to S3Copy Anchor Link

To migrate existing images from PVC storage to S3:

1. Export all images from the current registry
2. Switch Helm values to S3 storage and upgrade canette
3. Re-push exported images to the new S3-backed registry
4. Verify all apps can deploy
5. Delete the old PVC

# 1. List all repositories
kubectl port-forward -n canette-system svc/registry 5000:5000 &
curl http://localhost:5000/v2/_catalog

# 2. Pull and save each image locally
docker pull registry.canette.example.com/my-app:abc123
docker save -o my-app.tar registry.canette.example.com/my-app:abc123

# 3. Switch to S3 storage
helm upgrade canette ./charts/canette \
  --reuse-values \
  --set registry.storage.type=s3 \
  --set registry.storage.s3.bucket=my-bucket \
  --set registry.storage.s3.region=us-west-2 \
  --set registry.storage.s3.secretRef.name=canette-s3-credentials

# 4. Wait for registry to restart, then re-push
docker load -i my-app.tar
docker push registry.canette.example.com/my-app:abc123

# 5. Delete old PVC (after verifying all apps work)
kubectl delete pvc -n canette-system registry-data

S3 security best practicesCopy Anchor Link

1. Credential management

   • Never commit access keys to values files
   • Use registry.storage.s3.secretRef to reference a Kubernetes Secret
   • Rotate credentials regularly (every 90 days recommended)
   • Use separate credentials for each environment

2. Bucket security

   • Enable bucket versioning to recover from accidental deletions
   • Configure lifecycle policies to automatically delete old image layers
   • Block public access at the bucket policy level
   • Enable access logging for audit trails

3. IAM/Access policies

   • Follow principle of least privilege — only grant S3 permissions needed (GetObject, PutObject, DeleteObject, ListBucket)
   • For AWS: create a dedicated IAM user per cluster, never use root credentials
   • For other providers: create application-specific access keys, not account-level keys

4. Encryption

   • For AWS S3: set encrypt: true to enable server-side encryption (AES-256)
   • For other providers: check if encryption at rest is enabled by default
   • Always use secure: true (HTTPS) to encrypt data in transit

5. Network security

   • Use private S3 endpoints (VPC endpoints for AWS) to avoid internet egress when possible
   • For MinIO/self-hosted: use TLS certificates from a trusted CA

Trade-offsCopy Anchor Link

Advantages of S3 storage:

• Durability: 99.999999999% (11 9's) for AWS S3
• Cost: ~$0.023/GB/month vs ~$0.10/GB/month for EBS
• Scalability: No resizing required, unlimited capacity
• Multi-cluster: Share registry data across multiple canette installations
• Backup: Use S3 versioning and lifecycle policies

Disadvantages of S3 storage:

• Network latency: S3 API is slower than local disk for small files
• Egress costs: Pulling images from S3 to nodes incurs data transfer fees (can be significant)
• Complexity: Requires S3 bucket setup and credential management
• No atomic operations: S3 is eventually consistent (can cause rare race conditions)

CredentialsCopy Anchor Link

Registry credentials are auto-generated on first install and reused on every upgrade — no configuration needed. Build jobs receive the credentials automatically via the canette-registry-auth docker config Secret.

To rotate credentials or set your own, see the Helm reference.

Image pull credentialsCopy Anchor Link

When controller.imagePullSecrets.enabled=true (the default), the controller creates a canette-registry-creds Secret in each app namespace and references it in the pod spec so the kubelet can pull images.

In-cluster registryCopy Anchor Link

The internal registry is accessed at two different addresses depending on who is talking to it:

The pull credential placed in app namespaces uses the external pull address as its auth key, so containerd can match it to the image reference.

External registry — separate pull credentialsCopy Anchor Link

For external registries it is good practice to configure a read-only registry account for pulls, separate from the read-write account used by the builder. This limits the blast radius if a pull secret is ever read from an app namespace.

externalRegistry:
  username: "builder-rw"        # push — used by build jobs
  password: "..."
  pullUsername: "puller-ro"     # pull — distributed to app namespaces
  pullPassword: "..."

pullUsername/pullPassword fall back to the push credentials when not set. A pullPasswordSecretRef is also supported if you prefer to keep the password out of values.yaml.

Disabling imagePullSecretsCopy Anchor Link

If your nodes already have registry access configured at the containerd level (via /etc/containerd/certs.d/ or a node credential provider), you can disable Secret-based pull credentials entirely:

controller:
  imagePullSecrets:
    enabled: false

No canette-registry-creds Secret is created and no imagePullSecrets field is added to pod specs. This is the recommended approach for production clusters where node-level credentials are managed by the platform team.

TLS for local development (mkcert)Copy Anchor Link

On a local cluster (k3s, kind, etc.) without a public IP, Let's Encrypt won't work. mkcert generates locally-trusted certificates backed by a CA you install yourself.

Step 1 — Generate a certificateCopy Anchor Link

brew install mkcert
mkcert -install   # installs the local CA into the macOS keychain

mkcert "registry.apps.example.com"
# Produces: registry.apps.example.com.pem  and  registry.apps.example.com-key.pem

Step 2 — Create the TLS Secret in KubernetesCopy Anchor Link

kubectl create secret tls registry-tls \
  --namespace canette-system \
  --cert="registry.apps.example.com.pem" \
  --key="registry.apps.example.com-key.pem"

Step 3 — Configure your Gateway to use the certCopy Anchor Link

How you reference this Secret depends on your Gateway implementation. With Traefik, reference it in your Gateway resource or annotate the HTTPRoute. Consult your Gateway implementation's docs for the exact approach.

Step 4 — Install the mkcert CA on each nodeCopy Anchor Link

Every node that runs app pods needs to trust the mkcert CA so that kubelets can pull images from registry.{domain}.

# Find the CA file
mkcert -CAROOT
# e.g. /Users/you/Library/Application Support/mkcert

# Copy to each node (Ubuntu/Debian)
scp "$(mkcert -CAROOT)/rootCA.pem" root@<NODE_IP>:/usr/local/share/ca-certificates/mkcert-rootCA.crt
ssh root@<NODE_IP> update-ca-certificates

On RHEL/CentOS nodes: copy to /etc/pki/ca-trust/source/anchors/ and run update-ca-trust extract.

For k3s, also add the CA to /etc/rancher/k3s/registries.yaml on each node:

configs:
  "registry.apps.example.com":
    tls:
      ca_file: "/usr/local/share/ca-certificates/mkcert-rootCA.crt"

Then restart k3s: systemctl restart k3s.

Using an external registry insteadCopy Anchor Link

To disable the in-cluster registry and push to an external one (GHCR, ECR, Docker Hub, etc.):

registry:
  enabled: false

externalRegistry:
  username: "<username>"
  password: "<password>"

controller:
  pullRepo: "ghcr.io/myorg/"

Amazon ECR with IRSACopy Anchor Link

For ECR on EKS, you can use IRSA (IAM Roles for Service Accounts) instead of static credentials. ECR repositories are automatically created on first build:

registry:
  enabled: false

builder:
  imageRepo: "ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/canette/"
  registryAuthType: "irsa"
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/canette-builder

controller:
  pullRepo: "ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/canette/"

See the AWS (EKS) page for complete IAM policy setup.

See the Helm reference for the full set of external registry values.

TroubleshootingCopy Anchor Link