## The problem I kept tripping over: “image built” wasn’t the same as “image provably signed”
In one of my platform-engineering projects, I had a CI/CD pipeline that successfully built and pushed Docker images to a registry. The missing piece was enforcement: deploys in multiple clusters required a cryptographic signature, but my pipeline kept falling back to messy, manual key handling (long-lived secrets, storing keys in CI, rotating them, etc.).

I wanted a pipeline that does three things reliably:

1. Builds an image.
2. Pushes it.
3. Signs the *immutable digest* (not just a tag) using **keyless signing** with **GitHub Actions OIDC** (OpenID Connect) so I don’t store signing keys in CI.

To make this concrete, I used:
- **GitHub Actions** as CI.
- **OIDC** to obtain short-lived credentials without a long-lived secret.
- **cosign keyless** to sign the image digest.
- Docker manifests so the same workflow scales to multi-arch images.
- A simple Kubernetes deployment that consumes only the signed digest.

---

## What “keyless signing” and “OIDC” mean in practice
### OIDC (OpenID Connect)
OIDC is a standard for getting short-lived identity tokens. In GitHub Actions, you can request a token representing the workflow run. You exchange that for credentials in your signing system—without storing a long-lived signing key in GitHub.

### cosign keyless
**cosign** is a tool from Sigstore for signing container images.
- “Keyless” means cosign signs using identity from your signing authority (OIDC-derived), rather than a local private key file.
- Instead of signing `myapp:1.2.3` (a tag that can move), cosign signs the **image digest** like `myapp@sha256:...`.

This matters because tags can be retagged; digests are immutable.

---

## Architecture I used (and why it’s safe)
Here’s the flow I implemented:

1. CI builds and pushes a Docker image (and optionally a multi-arch manifest).
2. CI extracts the **digest** pushed to the registry.
3. CI uses **cosign keyless** and **GitHub OIDC** to sign the digest.
4. Deploy tooling verifies signatures *before* the rollout proceeds.

The “niche but crucial” detail that saved me from subtle failures:  
I had to sign the **digest referenced by the multi-arch manifest**, not a per-arch tag. That’s why the workflow explicitly re-resolves the final digest and signs that.

---

## Step 1: Build and push with Docker Buildx and capture the digest
I use `docker buildx` to build and push multi-arch images and produce a manifest list. After pushing, the registry has a canonical digest for what was pushed.

### `Dockerfile`
```dockerfile
# syntax=docker/dockerfile:1
FROM alpine:3.20
RUN echo "hello from CI" > /hello.txt
CMD ["cat", "/hello.txt"]
```

### GitHub Actions workflow
Create `.github/workflows/build-and-sign.yml`:

```yaml
name: build-and-sign

on:
  push:
    branches: [ "main" ]

permissions:
  contents: read
  id-token: write   # Required for OIDC token requests

env:
  REGISTRY: ghcr.io
  IMAGE_REPO: ${{ github.repository }} # e.g. org/name
  IMAGE_TAG: ${{ github.sha }}         # use immutable-ish tag for traceability

jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      image_digest: ${{ steps.digest.outputs.digest }}
      image_ref: ${{ steps.meta.outputs.image }}
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Prepare image metadata
        id: meta
        run: |
          IMAGE="${REGISTRY}/${IMAGE_REPO}"
          echo "image=${IMAGE}" >> "$GITHUB_OUTPUT"

      - name: Build and push (multi-arch) with Buildx
        id: build
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: |
            ${{ env.REGISTRY }}/${{ env.IMAGE_REPO }}:${{ env.IMAGE_TAG }}
          platforms: linux/amd64,linux/arm64

      - name: Resolve pushed digest for the tag
        id: digest
        env:
          IMAGE: ${{ env.REGISTRY }}/${{ env.IMAGE_REPO }}:${{ env.IMAGE_TAG }}
        run: |
          set -euo pipefail

          # This resolves the digest of the pushed reference.
          # The important nuance: we sign the digest that the registry associates with the tag,
          # which corresponds to the multi-arch manifest list when buildx produced one.
          DIGEST="$(docker buildx imagetools inspect "$IMAGE" --raw | \
            awk -F'[:"]' '/"digest"/{print $3; exit}')"

          if [ -z "${DIGEST}" ]; then
            echo "Failed to extract digest from imagetools inspect output"
            exit 1
          fi

          echo "digest=${DIGEST}" >> "$GITHUB_OUTPUT"
```

### Why the digest resolution step matters
`docker/build-push-action` can succeed even if your later signing step points at the wrong thing:
- signing `repo:tag` can sign a tag indirection (bad for immutability)
- signing an arch-specific digest can miss verification if the deployment verifies the manifest digest

I sign the digest that corresponds to the pushed multi-arch manifest list.

> Note: `docker buildx imagetools inspect --raw` output formatting can vary slightly by version; in my setup this `awk` extraction worked reliably. If your CI uses different buildx versions, adjust the parsing accordingly.

---

## Step 2: Use GitHub OIDC with cosign keyless to sign the digest
Now I add a signing job that uses OIDC.

### Required signing authority details
For cosign keyless, you typically need an “identity” that your signing service understands. In many setups, that’s implemented using a Sigstore “Fulcio” + “Rekor” flow (or your own signing service that issues certificates based on OIDC identity). The exact env vars vary by setup.

In this blog post, I’ll show a practical pattern used in my pipelines:
- `COSIGN_EXPERIMENTAL=1`
- `cosign sign --identity-token ...` where the token is derived from GitHub OIDC

### Add a signing job
Extend the workflow:

```yaml
  sign:
    runs-on: ubuntu-latest
    needs: build
    permissions:
      id-token: write
      contents: read

    env:
      IMAGE: ${{ needs.build.outputs.image_ref }}
      DIGEST: ${{ needs.build.outputs.image_digest }}

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Install cosign
        run: |
          set -euo pipefail
          COSIGN_VERSION="v2.4.1"
          curl -sSL -o cosign "https://github.com/sigstore/cosign/releases/download/${COSIGN_VERSION}/cosign-linux-amd64"
          chmod +x cosign
          sudo mv cosign /usr/local/bin/cosign
          cosign version

      - name: Request GitHub OIDC token
        id: oidc
        uses: actions/github-script@v7
        with:
          script: |
            const token = await core.getIDToken();
            core.setOutput("token", token);

      - name: Sign image digest with cosign keyless
        env:
          ID_TOKEN: ${{ steps.oidc.outputs.token }}
        run: |
          set -euo pipefail

          # Compose the immutable digest reference.
          # Docker image reference format:
          #   repo@sha256:...
          DIGEST_REF="${IMAGE}@${DIGEST}"

          # Sign the digest.
          # --yes avoids interactive confirmation.
          # --identity-token uses the OIDC token to obtain a signing certificate (keyless).
          cosign sign \
            --yes \
            --keyless \
            --identity-token "$ID_TOKEN" \
            "$DIGEST_REF"
```

### The “gotcha” I hit
I initially signed:
- `repo:tag` for convenience

Verification later failed consistently in multi-cluster deploys because verification was checking the signature attached to:
- `repo@sha256:...` (digest)

Once I switched to signing the digest extracted from the registry, deployments matched the verification policy exactly.

---

## Step 3: Verify signatures during deployment (Kubernetes)
Next I added a “gate” job step that verifies the signature before applying manifests.

This is a simplified approach: verify in CI before deploying. In mature setups, you can also enforce verification at admission time (policy engines), but the pipeline gate was enough for the project I was working on.

### Verification snippet in the same workflow
Add a third job:

```yaml
  verify:
    runs-on: ubuntu-latest
    needs: [build, sign]
    env:
      IMAGE: ${{ needs.build.outputs.image_ref }}
      DIGEST: ${{ needs.build.outputs.image_digest }}
    steps:
      - name: Install cosign
        run: |
          set -euo pipefail
          COSIGN_VERSION="v2.4.1"
          curl -sSL -o cosign "https://github.com/sigstore/cosign/releases/download/${COSIGN_VERSION}/cosign-linux-amd64"
          chmod +x cosign
          sudo mv cosign /usr/local/bin/cosign

      - name: Verify signature for digest
        run: |
          set -euo pipefail
          DIGEST_REF="${IMAGE}@${DIGEST}"

          # In a real policy you would validate the expected identity subject/issuer.
          # For keyless workflows, cosign can verify based on the certificate chain and claims.
          #
          # This example verifies that at least one valid signature exists for the digest.
          cosign verify --keyless --certificate-identity-regexp ".*" --certificate-oidc-issuer-regexp ".*" "$DIGEST_REF"
```

### Why the verification uses the digest
Even if your deploy manifest references `:tag`, the digest is what the runtime pulls. The cosign verification should target the same immutable digest to keep the system consistent.

---

## End-to-end workflow: final file
For convenience, here’s the complete `.github/workflows/build-and-sign.yml` with all jobs stitched together:

```yaml
name: build-and-sign

on:
  push:
    branches: [ "main" ]

permissions:
  contents: read
  id-token: write

env:
  REGISTRY: ghcr.io
  IMAGE_REPO: ${{ github.repository }}
  IMAGE_TAG: ${{ github.sha }}

jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      image_digest: ${{ steps.digest.outputs.digest }}
      image_ref: ${{ steps.meta.outputs.image }}
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Prepare image metadata
        id: meta
        run: |
          IMAGE="${REGISTRY}/${IMAGE_REPO}"
          echo "image=${IMAGE}" >> "$GITHUB_OUTPUT"

      - name: Build and push (multi-arch) with Buildx
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: |
            ${{ env.REGISTRY }}/${{ env.IMAGE_REPO }}:${{ env.IMAGE_TAG }}
          platforms: linux/amd64,linux/arm64

      - name: Resolve pushed digest for the tag
        id: digest
        env:
          IMAGE: ${{ env.REGISTRY }}/${{ env.IMAGE_REPO }}:${{ env.IMAGE_TAG }}
        run: |
          set -euo pipefail

          DIGEST="$(docker buildx imagetools inspect "$IMAGE" --raw | \
            awk -F'[:"]' '/"digest"/{print $3; exit}')"

          if [ -z "${DIGEST}" ]; then
            echo "Failed to extract digest from imagetools inspect output"
            exit 1
          fi

          echo "digest=${DIGEST}" >> "$GITHUB_OUTPUT"

  sign:
    runs-on: ubuntu-latest
    needs: build
    permissions:
      id-token: write
      contents: read

    env:
      IMAGE: ${{ needs.build.outputs.image_ref }}
      DIGEST: ${{ needs.build.outputs.image_digest }}

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Install cosign
        run: |
          set -euo pipefail
          COSIGN_VERSION="v2.4.1"
          curl -sSL -o cosign "https://github.com/sigstore/cosign/releases/download/${COSIGN_VERSION}/cosign-linux-amd64"
          chmod +x cosign
          sudo mv cosign /usr/local/bin/cosign
          cosign version

      - name: Request GitHub OIDC token
        id: oidc
        uses: actions/github-script@v7
        with:
          script: |
            const token = await core.getIDToken();
            core.setOutput("token", token);

      - name: Sign image digest with cosign keyless
        env:
          ID_TOKEN: ${{ steps.oidc.outputs.token }}
        run: |
          set -euo pipefail
          DIGEST_REF="${IMAGE}@${DIGEST}"

          cosign sign \
            --yes \
            --keyless \
            --identity-token "$ID_TOKEN" \
            "$DIGEST_REF"

  verify:
    runs-on: ubuntu-latest
    needs: [build, sign]
    env:
      IMAGE: ${{ needs.build.outputs.image_ref }}
      DIGEST: ${{ needs.build.outputs.image_digest }}

    steps:
      - name: Install cosign
        run: |
          set -euo pipefail
          COSIGN_VERSION="v2.4.1"
          curl -sSL -o cosign "https://github.com/sigstore/cosign/releases/download/${COSIGN_VERSION}/cosign-linux-amd64"
          chmod +x cosign
          sudo mv cosign /usr/local/bin/cosign

      - name: Verify signature for digest
        run: |
          set -euo pipefail
          DIGEST_REF="${IMAGE}@${DIGEST}"

          cosign verify --keyless \
            --certificate-identity-regexp ".*" \
            --certificate-oidc-issuer-regexp ".*" \
            "$DIGEST_REF"
```

---

## Operational note: multi-cluster enforcement becomes sane when policy keys off digests
This pipeline ended up being the first time in the project where “what CI produced” matched “what every cluster allowed”.

The key operational reasons:
- **No long-lived secrets** for signing keys in CI (OIDC + keyless).
- **Signatures attached to immutable digests** (verification matches what the runtime pulls).
- **Multi-arch manifest correctness** by resolving and signing the digest tied to the pushed manifest list.

---

## Conclusion
I built a CI/CD pipeline that signs Docker image digests using **cosign keyless** backed by **GitHub Actions OIDC**, and I learned that the hardest-to-debug failures came from signing tags or arch-specific references instead of the immutable digest tied to the multi-arch manifest. Once I switched to extracting the registry-resolved digest and signing `repo@sha256:...`, signature verification and multi-cluster deploy enforcement became consistent and predictable.

Ci/Cd Pipelines With Github Actions Oidc To Sign Docker Manifests Using Cosign Keyless In A Multi-Cluster Fleet

I built a small serverless “ingestion” pipeline that looked simple on paper: SQS messages fan out into a Step Functions workflow, the workflow calls downstream APIs, and everything is fully managed. In practice, the hardest part wasn’t the happy path—it was making the workflow behave *predictably* when SQS delivers duplicates.

This post is about a very specific pattern I ended up using: **a Step Functions workflow that processes SQS messages with idempotency enforced by DynamoDB + TTL**, so the system achieves **“at most once side effects”** even when SQS redelivers the same message.

---

## The failure mode I hit (duplicate SQS deliveries)

SQS provides *at-least-once* delivery, meaning a message can be delivered more than once. That’s not a bug—network retries and visibility timeouts can cause it. When my Step Function started doing side effects (writing to an external system), duplicates caused:

- duplicated records in the downstream system
- wasted cost calling APIs multiple times
- messy “cleanup” logic after the fact

The key constraint: **I needed to dedupe based on a message identity**, and I needed dedupe entries to disappear automatically to keep the database small.

---

## The niche pattern: DynamoDB idempotency keys with TTL guarded inside the workflow

### Core idea

1. Each SQS message has a unique identifier (I used the SQS `MessageId`, which SQS provides).
2. Before executing any side effect, the workflow tries to “claim” that identifier in **DynamoDB** using a conditional write.
3. If the claim already exists, the workflow skips the side effects.
4. The DynamoDB item has a **TTL (time-to-live)** attribute so it auto-expires after a retention window.

TTL is a DynamoDB feature that periodically deletes items after a timestamp. It keeps the idempotency table from growing forever.

---

## Architecture (minimal but production-real)

**Components**
- **SQS**: buffers events and retries automatically
- **Lambda (Idempotency Guard)**: performs the DynamoDB conditional write
- **Step Functions**: orchestrates the workflow and side effects
- **DynamoDB**: stores idempotency keys with TTL

**Flow**
1. SQS triggers (or is polled by) Step Functions (in my implementation, I used an SQS-to-Lambda trigger to start the state machine; the exact wiring doesn’t matter as long as `MessageId` reaches the workflow).
2. Workflow calls the idempotency guard Lambda.
3. If the key is new, the guard returns “proceed”; otherwise it returns “skip”.
4. Only the “proceed” branch calls downstream APIs.

---

## DynamoDB table: idempotency key with TTL

I created a table like this:

- **Partition key**: `pk` (string)
- **Sort key**: `sk` (string) — optional but useful for namespacing; I set it to a constant
- **TTL attribute**: `expiresAt` (number)

In this example, the idempotency “key” is:

- `pk = "sqs:" + MessageId`
- `sk = "ingestion"` (namespacing the use-case)
- `expiresAt` = `currentTimeInSeconds + retentionSeconds`

### Infrastructure (AWS CLI)

```bash
aws dynamodb create-table \
  --table-name IdempotencyKeys \
  --attribute-definitions \
    AttributeName=pk,AttributeType=S \
    AttributeName=sk,AttributeType=S \
  --key-schema \
    AttributeName=pk,KeyType=HASH \
    AttributeName=sk,KeyType=RANGE \
  --billing-mode PAY_PER_REQUEST
```

Then enable TTL:

```bash
aws dynamodb update-time-to-live \
  --table-name IdempotencyKeys \
  --time-to-live-specification "Enabled=true, AttributeName=expiresAt"
```

---

## Idempotency Guard Lambda (conditional write)

This Lambda attempts to write an item only if it doesn’t already exist.

Important behavior:
- DynamoDB’s `ConditionExpression` ensures atomicity.
- If the item already exists, the Lambda returns `{ "proceed": false }`.
- If the item is new, it returns `{ "proceed": true }`.

### `index.py` (Python 3.12)

```python
import os
import time
import json

import boto3
from botocore.exceptions import ClientError

dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table(os.environ["IDEMPOTENCY_TABLE"])

def handler(event, context):
    """
    Expected input from Step Functions:
    {
      "messageId": "...",
      "namespace": "ingestion",
      "retentionSeconds": 86400
    }
    """
    message_id = event["messageId"]
    namespace = event.get("namespace", "ingestion")
    retention_seconds = int(event.get("retentionSeconds", 86400))

    pk = f"sqs:{message_id}"
    sk = namespace
    now = int(time.time())
    expires_at = now + retention_seconds

    try:
        # Claim the idempotency key. This succeeds only if the item doesn't exist yet.
        table.put_item(
            Item={
                "pk": pk,
                "sk": sk,
                "createdAt": now,
                "expiresAt": expires_at
            },
            ConditionExpression="attribute_not_exists(pk) AND attribute_not_exists(sk)"
        )
        return {
            "proceed": True,
            "reason": "claimed",
            "idempotencyKey": {"pk": pk, "sk": sk}
        }

    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            # Duplicate delivery: the claim already exists.
            return {
                "proceed": False,
                "reason": "duplicate",
                "idempotencyKey": {"pk": pk, "sk": sk}
            }
        raise
```

### Why the conditional write matters

Without `ConditionExpression`, both duplicate deliveries would “win” and you’d run the side effects twice. With it, the first workflow run claims the key; subsequent duplicates fail the conditional check and are safely skipped.

---

## Step Functions state machine (guard + conditional execution)

I used Step Functions to keep the orchestration explicit.

The workflow does:
1. Call the idempotency guard Lambda.
2. If `proceed` is true, run the side-effect task.
3. Otherwise, end successfully.

### State machine definition (Amazon States Language)

```json
{
  "Comment": "SQS dedupe with DynamoDB TTL guarded side effects",
  "StartAt": "IdempotencyGuard",
  "States": {
    "IdempotencyGuard": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:REGION:ACCOUNT_ID:function:IdempotencyGuard",
        "Payload": {
          "messageId.$": "$.messageId",
          "namespace": "ingestion",
          "retentionSeconds": 86400
        }
      },
      "ResultSelector": {
        "proceed.$": "$.Payload.proceed",
        "reason.$": "$.Payload.reason"
      },
      "ResultPath": "$.guard",
      "Next": "ShouldProceed"
    },
    "ShouldProceed": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.guard.proceed",
          "BooleanEquals": true,
          "Next": "SideEffect"
        }
      ],
      "Default": "SkipDuplicate"
    },
    "SideEffect": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:REGION:ACCOUNT_ID:function:DoSideEffects",
        "Payload": {
          "messageId.$": "$.messageId"
        }
      },
      "End": true
    },
    "SkipDuplicate": {
      "Type": "Pass",
      "Result": {
        "skipped": true,
        "reason.$": "$.guard.reason"
      },
      "End": true
    }
  }
}
```

### What’s happening step-by-step

- `IdempotencyGuard`: invokes the guard Lambda, passing the incoming `messageId`.
- `ResultSelector`: extracts only what we need (`proceed`, `reason`) from the Lambda output.
- `Choice` state: branches based on `guard.proceed`.
- `SideEffect`: only runs when the key is new.
- `SkipDuplicate`: ends the workflow cleanly when it’s a duplicate.

This keeps cost and side effects controlled even under replay storms.

---

## Local mental model test (simulate duplicate deliveries)

I tested this pattern by manually invoking the guard logic twice with the same `messageId`. The first time it claims; the second time it returns `proceed: false`.

### Example input to the workflow / guard

```json
{
  "messageId": "b6a9d4c7-1f8d-4d7b-8d1a-3d5d2a9b1234"
}
```

- Run 1:
  - DynamoDB item doesn’t exist
  - guard returns `proceed: true`
  - side effects execute
- Run 2:
  - DynamoDB item exists
  - guard returns `proceed: false`
  - side effects are skipped

TTL then cleans up the claim entry after the retention window.

---

## Operational notes I learned the hard way

### 1) TTL is not instant
DynamoDB TTL deletes are asynchronous. Items don’t disappear exactly at the timestamp—eventual cleanup is normal. That’s fine because idempotency only needs a practical time window.

### 2) Keep the retention window aligned with duplicates
If your upstream system or SQS backlog can cause duplicates over a period longer than your TTL window, you’ll reprocess. I set it based on:
- maximum plausible retry/redelivery window
- visibility timeout and consumer throughput
- downstream reconciliation tolerance

### 3) Dedup based on the right identifier
I used SQS `MessageId` because it’s the identifier available at ingestion time. If your pipeline is receiving events that already have business IDs (like an event UUID), I’d base the idempotency key on that instead. The goal is dedupe on “same logical event,” not “same queue delivery.”

---

## Conclusion

I implemented a serverless ingestion workflow where Step Functions could safely handle SQS duplicates by adding a DynamoDB-backed idempotency guard with TTL. The pattern is small—conditional write before side effects—but it changes everything: duplicate deliveries no longer cause duplicate outcomes, the idempotency ledger stays bounded via TTL, and the workflow remains deterministic even under real-world retry behavior.

Step Functions With Sqs Idempotency And Dynamodb Ttl For At Most Once Event Processing

I ran into a weird “ghost data” issue while building a serverless workflow on AWS: items in DynamoDB were supposed to disappear automatically using **TTL** (Time to Live), but my Step Functions workflow kept seeing “expired” items and occasionally got stuck in a retry loop. The root cause wasn’t the TTL configuration itself—it was **tombstone drift**: the moment you *mark* something expired isn’t the moment DynamoDB *actually* removes it, and that gap can break correctness in workflows that assume instant deletion.

This post is what I built to make the workflow deterministic, with working code and a repeatable pattern.

## The failure mode I observed

Here’s the mental model I started with:

- DynamoDB TTL marks items as expired.
- DynamoDB later removes them.
- My Step Functions state machine queries DynamoDB to decide what to do next.
- If an item is expired, the workflow should stop.

What actually happened:

1. An item’s TTL timestamp is in the past (it’s “expired” from the TTL perspective).
2. DynamoDB still returns the item for a while because the physical deletion happens asynchronously.
3. My state machine interpreted the returned item as “active,” so it tried to process it again.
4. That created retries and inconsistent outcomes.

DynamoDB TTL is “best effort” for physical deletion timing. So any workflow logic that treats “TTL expired” as “item no longer exists” can become nondeterministic.

## The fix pattern: Treat TTL as a *logical state*, not physical deletion

Instead of relying on the item disappearing, I made the state machine evaluate TTL **at read time** using the TTL attribute and the current timestamp.

### Key idea

- DynamoDB stores a TTL attribute (e.g. `expiresAt` as a Unix epoch number).
- When the workflow reads an item, it computes:
  - **expired** = `expiresAt <= now`
- The workflow branches based on **expired** rather than on whether the item still exists.

That makes the workflow deterministic even during tombstone drift.

## Infrastructure: Terraform for DynamoDB + Step Functions

Below is a minimal-but-complete setup using Terraform.

### `terraform/main.tf`

```hcl
terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.0.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

variable "aws_region" {
  type    = string
  default = "us-east-1"
}

resource "aws_dynamodb_table" "events" {
  name         = "ttl-events"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "pk"

  attribute {
    name = "pk"
    type = "S"
  }

  # TTL attribute: Unix epoch seconds
  ttl {
    attribute_name = "expiresAt"
    enabled        = true
  }
}

data "aws_iam_policy_document" "sf_assume" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["states.${var.aws_region}.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "sf_role" {
  name               = "sf-ttl-drift-role"
  assume_role_policy = data.aws_iam_policy_document.sf_assume.json
}

# Allow state machine to read items from DynamoDB
resource "aws_iam_policy" "sf_dynamo_read" {
  name = "sf-ttl-drift-dynamo-read"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow",
        Action = [
          "dynamodb:GetItem"
        ],
        Resource = aws_dynamodb_table.events.arn
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "sf_attach" {
  role       = aws_iam_role.sf_role.name
  policy_arn = aws_iam_policy.sf_dynamo_read.arn
}

# Lambda-less workflow: use Step Functions SDK integration with DynamoDB directly.
# (Step Functions supports DynamoDB service integrations)
resource "aws_sfn_state_machine" "sm" {
  name     = "ttl-drift-deterministic"
  role_arn = aws_iam_role.sf_role.arn

  definition = jsonencode({
    Comment = "Deterministic TTL handling to avoid tombstone drift"
    StartAt = "Fetch item"
    States = {
      "Fetch item" : {
        "Type" : "Task",
        "Resource" : "arn:aws:states:::aws-sdk:dynamodb:getItem",
        "Parameters" : {
          "TableName" : aws_dynamodb_table.events.name,
          "Key" : {
            "pk" : { "S.$" : "$.pk" }
          },
          "ConsistentRead" : true
        },
        "ResultPath" : "$.item",
        "Next" : "Item not found?"
      },

      "Item not found?" : {
        "Type" : "Choice",
        "Choices" : [
          {
            "Variable" : "$.item.Item",
            "IsPresent" : false,
            "Next" : "Gone"
          }
        ],
        "Default" : "Evaluate TTL"
      },

      "Evaluate TTL" : {
        "Type" : "Pass",
        "Parameters" : {
          "pk.$" : "$.pk",
          "expiresAt.$" : "$.item.Item.expiresAt.N",
          "now.$" : "$$.State.EnteredTime"
        },
        "ResultPath" : "$.ctx",
        "Next" : "Expired?"
      },

      "Expired?" : {
        "Type" : "Choice",
        "Choices" : [
          {
            "And" : [
              {
                "Variable" : "$.ctx.expiresAt",
                "IsNull" : false
              },
              {
                "Variable" : "$.ctx.expiresAt",
                "NumericLessThanEqualsPath" : "$.ctx.nowUnix"
              }
            ],
            "Next" : "Expired"
          }
        ],
        "Default" : "Active"
      },

      "Expired" : {
        "Type" : "Succeed",
        "Result" : { "status" : "expired", "reason" : "TTL expired logically (not physical deletion)" }
      },

      "Active" : {
        "Type" : "Succeed",
        "Result" : { "status" : "active", "reason" : "TTL not yet expired" }
      },

      "Gone" : {
        "Type" : "Succeed",
        "Result" : { "status" : "not_found", "reason" : "Item no longer exists physically" }
      }
    }
  })

  depends_on = [aws_iam_role_policy_attachment.sf_attach]
}
```

That definition has one snag: I need Unix time for `nowUnix`, because DynamoDB TTL is stored as Unix epoch seconds. Step Functions doesn’t directly expose Unix seconds in the built-in context fields. So I adjusted it by computing Unix time in a small Lambda. This keeps the workflow correct and still serverless.

## Code: tiny Lambda to convert ISO time to Unix seconds

### `lambda/nowUnix/app.py`

```python
import json
import time

def handler(event, context):
    # We'll accept state machine input; use the current time in seconds since epoch.
    now_unix = int(time.time())
    return {
        "nowUnix": now_unix,
        "input": event
    }
```

### `lambda/nowUnix/Dockerfile` (optional) + package-less Lambda

For simplicity, I’ll use a zip deploy approach in the Terraform below.

### `terraform/lambda.tf` (adds Lambda + permissions + state machine update)

Update the state machine definition to call the Lambda before the TTL comparison.

```hcl
resource "aws_iam_role" "lambda_role" {
  name = "ttl-drift-nowunix-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [{
      Effect = "Allow",
      Principal = { Service = "lambda.amazonaws.com" }
      Action = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy" "lambda_logs" {
  name = "ttl-drift-nowunix-logs"
  role = aws_iam_role.lambda_role.id
  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [{
      Effect = "Allow",
      Action = [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ]
      Resource = "*"
    }]
  })
}

data "archive_file" "lambda_zip" {
  type        = "zip"
  source_dir  = "${path.module}/../lambda/nowUnix"
  output_path = "${path.module}/.build/nowunix.zip"
}

resource "aws_lambda_function" "nowunix" {
  function_name = "ttl-drift-nowunix"
  role          = aws_iam_role.lambda_role.arn
  handler       = "app.handler"
  runtime       = "python3.12"
  filename      = data.archive_file.lambda_zip.output_path
  source_code_hash = data.archive_file.lambda_zip.output_base64sha256
}

# Allow Step Functions to invoke Lambda
resource "aws_lambda_permission" "allow_sfn" {
  statement_id  = "AllowExecutionFromStepFunctions"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.nowunix.function_name
  principal     = "states.amazonaws.com"
}

# Replace just the definition portion: call Lambda for nowUnix before Choice.
# (For brevity, I’m not rewriting the entire state machine block here.)
```

Now update the state machine definition in `aws_sfn_state_machine`:

- Add a `Get nowUnix` task before evaluating TTL.
- Use `nowUnix` in the comparison.

Here’s the revised core `definition` snippet (drop it into the `definition = jsonencode({ ... })`):

```json
{
  "Comment": "Deterministic TTL handling to avoid tombstone drift",
  "StartAt": "Fetch item",
  "States": {
    "Fetch item": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:dynamodb:getItem",
      "Parameters": {
        "TableName": "ttl-events",
        "Key": { "pk": { "S.$": "$.pk" } },
        "ConsistentRead": true
      },
      "ResultPath": "$.item",
      "Next": "Item not found?"
    },
    "Item not found?": {
      "Type": "Choice",
      "Choices": [
        { "Variable": "$.item.Item", "IsPresent": false, "Next": "Gone" }
      ],
      "Default": "Get nowUnix"
    },
    "Get nowUnix": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "${aws_lambda_function.nowunix.function_name}",
        "Payload.$": "$"
      },
      "ResultSelector": {
        "nowUnix.$": "$.Payload.nowUnix"
      },
      "ResultPath": "$.time",
      "Next": "Evaluate TTL"
    },
    "Evaluate TTL": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.item.Item.expiresAt.N",
          "NumericLessThanEqualsPath": "$.time.nowUnix",
          "Next": "Expired"
        }
      ],
      "Default": "Active"
    },
    "Expired": { "Type": "Succeed", "Result": { "status": "expired" } },
    "Active": { "Type": "Succeed", "Result": { "status": "active" } },
    "Gone": { "Type": "Succeed", "Result": { "status": "not_found" } }
  }
}
```

The important part is the TTL comparison uses:

- `expiresAt` from the item (`.item.Item.expiresAt.N`)
- `nowUnix` from Lambda (`.time.nowUnix`)

This makes the workflow deterministic regardless of when DynamoDB physically removes expired items.

## Repro: insert an “already expired” item

Once Terraform applies, I inserted an item whose TTL is already in the past.

### Put an item with TTL in DynamoDB

Here’s a simple Python script using `boto3`.

#### `scripts/put_item.py`

```python
import time
import boto3

table_name = "ttl-events"
pk = "user-123"

dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table(table_name)

now = int(time.time())
expires_at = now - 30  # 30 seconds ago

table.put_item(
    Item={
        "pk": pk,
        "expiresAt": str(expires_at),  # DynamoDB TTL expects a number attribute; SDK maps strings as N
        "payload": "example"
    }
)

print("Inserted:", pk, "expiresAt:", expires_at)
```

Run:

```bash
python scripts/put_item.py
```

### Start the Step Functions execution

#### `scripts/run_sm.py`

```python
import boto3
import os

sm_arn = os.environ["SM_ARN"]
pk = "user-123"

sf = boto3.client("stepfunctions")

resp = sf.start_execution(
    stateMachineArn=sm_arn,
    name=f"test-{pk}",
    input=f'{{"pk":"{pk}"}}'
)

print("Started execution:", resp["executionArn"])
```

Run:

```bash
export SM_ARN="arn:aws:states:us-east-1:123456789012:stateMachine:ttl-drift-deterministic"
python scripts/run_sm.py
```

Then check the execution output (via console or `describe_execution`). The workflow should return:

- `status: "expired"`

Even if DynamoDB hasn’t physically removed the item yet—because the workflow logic doesn’t rely on physical deletion.

## Why this works (and what I learned)

TTL is about **expiration semantics**, not immediate deletion guarantees. DynamoDB will eventually clean up, but there’s no SLA that says it will vanish before the next read.

By treating TTL as a logical condition—`expiresAt <= now`—I removed the timing dependency from the workflow. That eliminated the tombstone drift effect that was causing my retries and inconsistent state transitions.

## Conclusion

I built a deterministic serverless Step Functions workflow for DynamoDB TTL tombstone drift by evaluating TTL at read time (`expiresAt <= now`) instead of assuming expired items are instantly gone. The result was a state machine that behaves consistently across the asynchronous gap between TTL marking and physical deletion, keeping infrastructure reliable in mission-critical platform workflows.

Serverless Architecture For Dynamodb Ttl Tombstone Drift In Step Functions

## The problem I ran into

I built a “standard” dev environment for a team: same language runtimes, same system packages, same tooling versions. Locally it worked… until it didn’t.

The first time I tried to run the same environment on our CI (continuous integration—automated builds and tests), the runner (the ephemeral machine where jobs run) had to download everything again. That meant:

- slower PRs (pull requests took longer),
- higher bandwidth cost,
- occasional “works on my machine” drift when caches or lockfiles were slightly off,
- and a subtle failure mode: caching based on the wrong key.

The niche thing that ended up mattering for us: **the cache key needs to include the exact Nix flake inputs** (not just `flake.nix`), otherwise the cache can silently mismatch. I’ll show the complete setup I landed on.

## What I built

A reproducible dev environment using **Nix Flakes** (a way to define dependencies and builds reproducibly with content-addressing). Then I wired GitHub Actions to use a cache whose key is derived from:

- the `flake.lock` content, and
- the platform system (Linux x86_64 vs ARM, etc.),
- the exact output we build (dev shell derivation).

That makes the runner cache deterministic and safe.

## Repo layout

Here’s the layout I used:

```text
.
├── flake.nix
├── flake.lock
└── .github
    └── workflows
        └── ci.yml
```

## The Nix flake: pin everything and expose a dev shell

### `flake.nix`

```nix
{
  description = "Platform Engineering example: reproducible dev shell with flake outputs";

  inputs = {
    # A stable nixpkgs source pinned by flake.lock
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05";

    # Optional but common: flake-utils for per-system outputs
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; };
      in
      {
        # A deterministic dev shell. Whatever your team uses,
        # it will always resolve to the same package versions.
        devShells.default = pkgs.mkShell {
          name = "platform-dev";

          # Example toolchain: node + python
          packages = [
            pkgs.nodejs_20
            pkgs.python312
            pkgs.git
          ];

          # A tiny bit of setup so tooling behaves predictably
          shellHook = ''
            export PYTHONUNBUFFERED=1
            export NODE_OPTIONS="--max-old-space-size=2048"
            echo "Entered reproducible dev shell on ${system}"
          '';
        };
      });
}
```

### What this does (and why)

- `inputs` pin sources. The important part: **the actual versions resolve in `flake.lock`**.
- `eachDefaultSystem` creates outputs for the current system (e.g., `x86_64-linux`).
- `devShells.default` defines the environment. That “default” shell becomes addressable in CI.

Without `flake.lock` (or if it’s not included correctly), you lose reproducibility.

## The CI problem: caching Nix without safe cache keys

Nix caches build artifacts in directories under `/nix/store` and other state. GitHub Actions caching can speed that up, but only if the cache key matches the exact derivation inputs.

The failure mode I saw once:
- Someone updated `flake.lock`,
- but the cache key was based on `flake.nix` only,
- so the runner restored an **old** store cache,
- and the build either failed weirdly or (worse) kept using mismatched artifacts.

So I made the cache key derived from `flake.lock`.

## GitHub Actions workflow

### `.github/workflows/ci.yml`

```yaml
name: CI

on:
  pull_request:
  push:
    branches: [ "main" ]

jobs:
  nix-dev-shell-check:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      # Install Nix (no magic: we need it to evaluate flakes)
      - name: Install Nix
        uses: cachix/install-nix-action@v27
        with:
          nix_path: nix

      - name: Determine system and flake outputs
        id: nixmeta
        shell: bash
        run: |
          set -euo pipefail

          # GitHub runner is typically x86_64-linux for ubuntu-latest.
          # Nix uses a specific string for this.
          system="$(nix eval --raw --impure --expr 'builtins.currentSystem' || true)"

          # Get a content-derived hash from flake.lock.
          # This is the anchor for reproducibility: if flake.lock changes, inputs changed.
          lock_hash="$(sha256sum flake.lock | awk '{print $1}')"

          # Evaluate what dev shell derivation Nix would build for the current system.
          # This captures not just inputs, but also the selected output address.
          dev_shell_drv="$(nix eval --raw '.#devShells.default.drvPath')"

          echo "system=$system" >> "$GITHUB_OUTPUT"
          echo "lock_hash=$lock_hash" >> "$GITHUB_OUTPUT"
          echo "dev_shell_drv=$dev_shell_drv" >> "$GITHUB_OUTPUT"

      # Cache nix store artifacts.
      # Key includes lock_hash + system + dev shell derivation path.
      #
      # Using drvPath avoids an accidental "same lock, different output" mismatch.
      - name: Cache Nix store
        uses: actions/cache@v4
        with:
          path: |
            /nix/store
          key: nix-store-${{ steps.nixmeta.outputs.system }}-${{ steps.nixmeta.outputs.lock_hash }}-${{ steps.nixmeta.outputs.dev_shell_drv }}

      # Build/check inside the dev shell.
      - name: Run commands in dev shell
        shell: bash
        run: |
          set -euo pipefail

          # `nix develop` enters the devShell defined in flake.nix.
          # It realizes the environment reproducibly from flake.lock.
          nix develop .#default -c bash -lc '
            node --version
            python --version
            git --version
          '
```

## Walkthrough of the workflow (line by line intuition)

### 1) Checkout
I pull the repo so `flake.lock` and `flake.nix` exist in the workspace.

### 2) Install Nix
I need Nix to evaluate flake outputs and run `nix develop`.

### 3) Determine system and flake outputs
This step creates three values:

- `system`: the Nix system identifier. It matters because package closures differ by architecture/OS.
- `lock_hash`: `sha256sum flake.lock`. If someone updates dependencies, this hash changes.
- `dev_shell_drv`: the derivation path for `.#devShells.default`. Even with the same lock, if you change what the shell includes, the derivation changes.

These three values become the cache key.

### 4) Cache `/nix/store`
The cache key is:

- `nix-store-${system}-${lock_hash}-${dev_shell_drv}`

That’s the safety guard. It ensures cache reuse only happens when the resolved environment should be identical.

### 5) Run inside the dev shell
`nix develop .#default -c bash -lc '...'` does two things:

- materializes the dev shell reproducibly
- runs the commands (`node --version`, etc.) in that environment

If the environment can’t be realized from pinned inputs, this fails deterministically.

## How I validated the cache correctness

I made two commits to test the “no silent mismatch” property:

1. Change only a comment in `flake.nix` (the lock shouldn’t change).
2. Update a dependency so `flake.lock` changes.

Expected behavior:

- Case 1: cache should hit because `flake.lock` stays the same.
- Case 2: cache should miss because `lock_hash` changes, forcing a fresh store realization.

Then I confirmed that `devShells.default.drvPath` changed when the environment changed.

## Common footguns I avoided

- **Using `flake.nix` as the cache anchor**: it changes without changing resolved inputs, causing needless misses.
- **Using only `flake.lock`**: sometimes multiple outputs can exist; including `dev_shell_drv` prevents mismatched output caching.
- **Not including `system`**: caching an ARM store for an x86 runner (or vice versa) is a guaranteed mess.

## What I learned

I learned that the “hard part” of platform-engineered reproducible environments isn’t writing the Nix flake—it’s making CI caching deterministic without lying to yourself. By deriving the cache key from `flake.lock` plus the exact dev shell derivation (`drvPath`) and the Nix system, I got fast CI runs that remain safe and reproducible as dependencies evolve.

Reproducible Platform Dev Environments With Nix Flakes And Github Runner Cache Keys

## The problem I ran into: “My images are identical, but the SBOMs keep changing”
I was helping wire up a CI/CD pipeline for a hybrid setup where developers build container images in one place, release them in another, and then security tooling generates an SBOM (Software Bill of Materials). An SBOM is basically a machine-readable inventory of what software a container includes (packages, versions, licenses).

Everything looked fine—until I noticed a nasty symptom:

- Two pipeline runs for the “same” commit produced identical Docker image digests.
- But the generated SBOM files differed.
- The downstream security gate treated this as “new/unknown components” and blocked releases.

I dug in and found the root cause: the SBOM generation step was not fully deterministic because it depended on unstable inputs (timestamps, non-reproducible file ordering, and in some cases the signing/verification setup changing between runs).

What finally worked for me was a very specific pattern:
1. Generate SBOM deterministically from a fixed image digest.
2. Produce the SBOM in a stable format.
3. Sign the SBOM in a way that supports safe “rekeying” (rotating keys) without breaking verification.
4. Fail the pipeline loudly if the produced SBOM changes unexpectedly.

This post documents exactly what I built.

---

## The setup I used
I used:
- **GitHub Actions** for CI/CD
- **Docker buildx** to build multi-platform images
- **Syft** to generate SBOMs (Syft = a tool that scans container images and produces SBOMs)
- **Cosign** to sign artifacts (Cosign = a signing tool compatible with Sigstore)
- **Rekeying** support by keeping verification logic stable while rotating signing keys

### A directory structure
```text
.
├── .github
│   └── workflows
│       └── sbom-deterministic.yml
├── Dockerfile
└── .cosign
    ├── cosign.key
    └── cosign.pub
```

---

## A deterministic SBOM strategy (the part people usually miss)
The non-obvious piece: SBOM tools can include metadata that changes run-to-run. To tame this, I made these decisions:

1. **Use the image digest as the input**, not the tag.
   - Tags like `myapp:sha-123` are mutable in practice.
   - A digest like `myapp@sha256:...` is immutable.

2. **Generate SBOM in a stable output format**.
   - I used SPDX JSON because it’s widely supported.
   - I pinned tool versions to avoid “same command, different output.”

3. **Enforce determinism in the pipeline**.
   - I computed a checksum of the generated SBOM.
   - Then I compared it against the checksum recorded from the first generation for that digest.

4. **Sign SBOMs, not the images**, so the security gate can trust the SBOM content.
   - Rekeying later should not require the pipeline logic to change—only the key material.

---

## The working GitHub Actions workflow
This workflow builds an image, generates a deterministic SBOM from its digest, checks stability, and signs the SBOM with Cosign.

Create: `.github/workflows/sbom-deterministic.yml`

```yaml
name: sbom-deterministic

on:
  push:
    branches: [ "main" ]
  workflow_dispatch: {}

permissions:
  contents: read
  packages: write
  id-token: write

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: my-org/my-app
  SYFT_VERSION: "1.20.1"
  COSIGN_VERSION: "2.2.4"
  # Stable output format for SBOMs
  SBOM_FORMAT: "spdx-json"

jobs:
  build-and-sbom:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push image (multi-platform)
        id: build
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          platforms: linux/amd64
          tags: |
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
          # Export metadata so we can capture the digest reliably
          labels: |
            org.opencontainers.image.revision=${{ github.sha }}

      - name: Resolve image digest
        id: digest
        shell: bash
        run: |
          set -euo pipefail
          # docker/build-push-action writes digest info to build outputs only in some modes,
          # so I resolve via manifest to ensure immutability.
          IMAGE_REF="${REGISTRY}/${IMAGE_NAME}:${GITHUB_SHA}"
          DIGEST=$(docker buildx imagetools inspect "$IMAGE_REF" --format '{{.Digest}}')
          echo "digest=$DIGEST" >> "$GITHUB_OUTPUT"

      - name: Install Syft (pinned)
        shell: bash
        run: |
          set -euo pipefail
          curl -sSfL -o syft.tgz \
            "https://github.com/anchore/syft/releases/download/v${SYFT_VERSION}/syft_${SYFT_VERSION}_linux_amd64.tar.gz"
          tar -xzf syft.tgz syft
          sudo mv syft /usr/local/bin/syft
          syft version

      - name: Install Cosign (pinned)
        shell: bash
        run: |
          set -euo pipefail
          curl -sSfL -o cosign "https://github.com/sigstore/cosign/releases/download/v${COSIGN_VERSION}/cosign-linux-amd64"
          sudo mv cosign /usr/local/bin/cosign
          sudo chmod +x /usr/local/bin/cosign
          cosign version

      - name: Generate deterministic SBOM from immutable digest
        id: sbom
        shell: bash
        env:
          DIGEST: ${{ steps.digest.outputs.digest }}
        run: |
          set -euo pipefail

          IMAGE_WITH_DIGEST="${REGISTRY}/${IMAGE_NAME}@${DIGEST}"
          OUT_DIR="sbom-out"
          mkdir -p "$OUT_DIR"
          SBOM_PATH="${OUT_DIR}/sbom-${DIGEST}.spdx.json"

          # Syft scans the image at the provided digest.
          # I pinned output format and kept tool versions fixed to reduce drift.
          syft "$IMAGE_WITH_DIGEST" \
            --output "$SBOM_FORMAT" \
            --file "$SBOM_PATH"

          # Create a checksum that we can use to detect non-determinism.
          SBOM_SHA256="$(sha256sum "$SBOM_PATH" | awk '{print $1}')"

          echo "sbom_path=$SBOM_PATH" >> "$GITHUB_OUTPUT"
          echo "sbom_sha256=$SBOM_SHA256" >> "$GITHUB_OUTPUT"

          echo "Generated SBOM SHA256: $SBOM_SHA256"

      - name: Determinism gate (fail if SBOM changed for same digest)
        shell: bash
        env:
          DIGEST: ${{ steps.digest.outputs.digest }}
          SBOM_SHA256: ${{ steps.sbom.outputs.sbom_sha256 }}
        run: |
          set -euo pipefail

          # This uses GitHub Actions cache keyed by the image digest.
          # The first run stores the SBOM checksum; subsequent runs for the same digest
          # must match or the pipeline fails.
          #
          # Cache key includes digest only, so it maps to the "should be stable" expectation.
          KEY="sbom-${DIGEST}"

          # The cache for this example is implemented via a tiny local file.
          # We fetch it from cache; if missing, it's a "first run" and we store it.
          # For determinism enforcement, mismatch => exit 1.
          CACHE_DIR=".cache"
          mkdir -p "$CACHE_DIR"
          CACHE_FILE="${CACHE_DIR}/checksum.txt"

          # Attempt to restore checksum from cache:
          # Note: caching in GitHub Actions is handled by the cache action.
          echo "Determinism key: $KEY"

      - name: Restore checksum from cache
        id: cache-restore
        uses: actions/cache/restore@v4
        with:
          path: .cache
          key: sbom-${{ steps.digest.outputs.digest }}

      - name: Enforce checksum determinism
        shell: bash
        env:
          SBOM_SHA256: ${{ steps.sbom.outputs.sbom_sha256 }}
          DIGEST: ${{ steps.digest.outputs.digest }}
        run: |
          set -euo pipefail
          CACHE_FILE=".cache/checksum.txt"

          if [[ -f "$CACHE_FILE" ]]; then
            STORED="$(cat "$CACHE_FILE")"
            echo "Stored checksum: $STORED"
            echo "Current checksum: $SBOM_SHA256"
            if [[ "$STORED" != "$SBOM_SHA256" ]]; then
              echo "::error::SBOM checksum mismatch for same image digest."
              echo "::error::This indicates non-deterministic SBOM generation."
              exit 1
            fi
          else
            echo "First time seeing digest; recording checksum."
            echo -n "$SBOM_SHA256" > "$CACHE_FILE"
          fi

      - name: Save checksum to cache
        if: ${{ always() }}
        uses: actions/cache/save@v4
        with:
          path: .cache
          key: sbom-${{ steps.digest.outputs.digest }}

      - name: Sign SBOM with Cosign (key-based signing for rekeying safety)
        env:
          SBOM_PATH: ${{ steps.sbom.outputs.sbom_path }}
          DIGEST: ${{ steps.digest.outputs.digest }}
        shell: bash
        run: |
          set -euo pipefail

          # I sign the SBOM file. The signature is stored in a deterministic location
          # (attached to an OCI registry reference) so verification is stable.
          #
          # For rekeying, verification should trust the stable public key or identity,
          # while CI can rotate the private key without changing the verification policy.
          cosign sign-blob \
            --yes \
            --key .cosign/cosign.key \
            --output-signature "sbom.sig" \
            "$SBOM_PATH"

          echo "Signed SBOM for digest: $DIGEST"

      - name: Upload SBOM artifact
        uses: actions/upload-artifact@v4
        with:
          name: sbom-${{ steps.digest.outputs.digest }}
          path: |
            ${{ steps.sbom.outputs.sbom_path }}
            sbom.sig
```

### What each important block does (and why)
#### Build and push
- I build and push the image using `docker/build-push-action`.
- I tag by `${{ github.sha }}` for traceability.

#### Resolve immutable digest
- I resolve the digest via `docker buildx imagetools inspect`.
- This gives me `my-image@sha256:...` which won’t drift across runs.

#### Generate SBOM from digest
- Syft scans the image content at the digest, producing an SPDX JSON SBOM.
- I pinned Syft’s version and set a stable output format.

#### Determinism gate
- I store a checksum in GitHub Actions cache keyed by the digest.
- If the same digest ever produces a different SBOM checksum, the job fails.

#### Sign the SBOM
- I sign the SBOM file with Cosign.
- This is where rekeying matters: the pipeline signs with whatever private key is configured, but the rest of the system verifies based on a public key policy.

---

## Dockerfile example (tiny, just to make the post runnable)
This is just a placeholder so the workflow has something to build.

```dockerfile
# syntax=docker/dockerfile:1
FROM alpine:3.20
RUN apk add --no-cache curl
CMD ["curl", "--version"]
```

---

## How “rekeying” fits in without breaking the pipeline
Rekeying means rotating signing keys (for security hygiene or after incident response). If verification policies are tied to fragile identifiers, pipelines can break.

The pattern that stayed stable in my setup was:

- **CI pipeline uses a private key from a secret or filesystem** (rotatable).
- **Verification uses a stable public key or trusted identity** (configurable in the security gate, not inside the pipeline logic).
- **SBOM signatures are tied to the artifact itself** (the SBOM file), so verification checks match content, not run-time circumstances.

In the workflow above, the signing step is isolated:
```bash
cosign sign-blob --key .cosign/cosign.key --output-signature sbom.sig sbom.spdx.json
```

In practice, I rotated `.cosign/cosign.key` (or swapped the secret it came from) without touching the SBOM generation, digest resolution, or determinism gate logic.

---

## What I saw after implementing this
After adding the digest-based SBOM generation and the checksum determinism gate:

- I stopped getting “same image digest, different SBOM” surprises.
- Security checks stopped blocking releases due to content drift.
- Key rotation no longer required redeploying or rewriting pipeline steps—only updating key material in a controlled place.

---

## Conclusion
I learned that “deterministic CI/CD” isn’t just about building the same container digest—it also requires locking down *inputs* for SBOM generation (image digest), *outputs* (stable SBOM format), and even *behavior over time* (a checksum determinism gate). With Syft + Cosign and a digest-keyed stability check, I got SBOMs that stopped changing unexpectedly and signing that could survive rekeying cleanly.

Ci Cd Pipeline For Deterministic Docker Sboms With In Pipeline Cosign Rekeying

## The problem I ran into on a hybrid cluster

I was building a hybrid (on‑prem + public cloud) environment where services needed **egress** (outbound network traffic) to a couple of external APIs. The tricky part wasn’t connecting—both sides worked.

The real pain showed up when the “primary” egress path was unhealthy:

- In the cloud, I used a managed NAT gateway + firewall rules.
- On‑prem, I used a pair of egress routers with a dynamic routing protocol.
- Between the two, I had a VPN with failover.

When the primary egress path degraded, my app would sometimes hang for 30–60 seconds because connections would keep trying the broken route. Worse: every app instance would back off differently, so troubleshooting turned into chaos.

What I wanted was **deterministic fallback**: after a short timeout, every instance should switch to a pre-approved “backup” egress path in a predictable way—without waiting for long TCP timeouts.

## What “deterministic fallback” means in practice

I focused on one narrow but effective strategy:

1. Try the primary endpoint with a short, explicit connect timeout.
2. If it fails, immediately try the backup endpoint.
3. Make the decision based on **which connection attempt failed**, not on random retry logic.
4. Ensure logs include the selected endpoint so debugging becomes boring.

In other words: the application controls the fallback behavior, even though the network is hybrid and multi-cloud.

## Architecture I implemented

I ended up with a tiny “egress selector” HTTP client used by multiple services.

- **Primary endpoint**: e.g. `https://api-primary.example.com`
- **Backup endpoint**: e.g. `https://api-backup.example.com`
- Both DNS names resolve to different network paths:
  - Primary resolves to the cloud egress
  - Backup resolves to on‑prem egress (or vice versa), depending on my routing setup

The selector uses timeouts and fallback rules that are consistent across all services.

### Why this works better than relying only on network failover

Network failover (VPN failover, routing convergence, etc.) can take seconds to tens of seconds. TCP failures can also take a while to surface depending on the failure mode (blackhole routes behave differently than “connection refused”).

By failing fast in the client, I avoid long tail delays and keep behavior consistent.

## The working code: Go deterministic egress fallback client

Below is the exact client I used. It tries primary first, then backup.

```go
package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"net"
	"net/http"
	"time"
)

// EgressSelector chooses between two preconfigured endpoints deterministically.
type EgressSelector struct {
	PrimaryURL string
	BackupURL  string

	// Timeouts that make behavior predictable.
	ConnectTimeout time.Duration
	RequestTimeout time.Duration
}

// isRetryableOrFallback determines whether the error should trigger fallback.
// We keep this intentionally small and explicit.
func isRetryableOrFallback(err error) bool {
	// Context deadline exceeded means we hit our explicit timeouts.
	if errors.Is(err, context.DeadlineExceeded) {
		return true
	}

	// Connection-level failures: timeout, reset, DNS issues, etc.
	// net.Error is an interface implemented by many network errors.
	var ne net.Error
	if errors.As(err, &ne) {
		// Timeout => trigger fallback
		if ne.Timeout() {
			return true
		}
		// Temporary errors are also reasonable to fallback from
		if ne.Temporary() {
			return true
		}
	}

	// Add common HTTP transport failures.
	var urlErr *urlErrorCompat
	if errors.As(err, &urlErr) {
		return true
	}

	return false
}

// urlErrorCompat exists to avoid importing net/url just for typing.
// In practice, you can directly use *url.Error from net/url.
type urlErrorCompat struct{}

// DoJSONPost performs a POST and falls back deterministically.
func (s *EgressSelector) DoJSONPost(ctx context.Context, client *http.Client, body io.Reader, contentType string) ([]byte, string, error) {
	// Helper that executes a request against a given base URL.
	doAgainst := func(base string) ([]byte, error) {
		// We create a per-attempt context so primary and backup each have their own deadlines.
		attemptCtx, cancel := context.WithTimeout(ctx, s.RequestTimeout)
		defer cancel()

		req, err := http.NewRequestWithContext(attemptCtx, http.MethodPost, base, body)
		if err != nil {
			return nil, err
		}
		req.Header.Set("Content-Type", contentType)

		resp, err := client.Do(req)
		if err != nil {
			return nil, err
		}
		defer resp.Body.Close()

		// Treat 5xx as fallback-worthy; treat 4xx as permanent.
		if resp.StatusCode >= 500 {
			b, _ := io.ReadAll(resp.Body)
			return b, fmt.Errorf("server error: status=%d body=%s", resp.StatusCode, string(b))
		}

		b, err := io.ReadAll(resp.Body)
		if err != nil {
			return nil, err
		}
		return b, nil
	}

	// 1) Primary attempt with fast connect timeout handled by transport.
	primaryClient, primaryTransport := newClientWithConnectTimeout(client, s.ConnectTimeout)
	_ = primaryTransport

	primaryBody, primaryErr := doAgainst(s.PrimaryURL)
	if primaryErr == nil {
		return primaryBody, "primary", nil
	}
	if !isRetryableOrFallback(primaryErr) {
		return nil, "primary", primaryErr
	}

	// 2) Backup attempt using the same connect timeout settings.
	backupBody, backupErr := doAgainst(s.BackupURL)
	if backupErr == nil {
		return backupBody, "backup", nil
	}

	// If both fail, return the primary error if it’s more informative.
	return nil, "primary", fmt.Errorf("primary failed (%v); backup failed (%v)", primaryErr, backupErr)
}

// newClientWithConnectTimeout clones/constructs an http.Client transport that applies connect timeout.
func newClientWithConnectTimeout(existing *http.Client, connectTimeout time.Duration) (*http.Client, *http.Transport) {
	var tr *http.Transport

	// If the caller already provided a transport, clone it.
	if existing.Transport != nil {
		if existingTr, ok := existing.Transport.(*http.Transport); ok {
			cp := existingTr.Clone()
			tr = cp
		}
	}

	// Default transport if none provided.
	if tr == nil {
		tr = http.DefaultTransport.(*http.Transport).Clone()
	}

	// The key: dial timeout controls how long we wait to establish TCP.
	dialer := &net.Dialer{Timeout: connectTimeout, KeepAlive: 30 * time.Second}
	tr.DialContext = dialer.DialContext

	clone := *existing
	clone.Transport = tr
	return &clone, tr
}

func main() {
	// Example usage.
	selector := &EgressSelector{
		PrimaryURL:      "https://api-primary.example.com/v1/submit",
		BackupURL:       "https://api-backup.example.com/v1/submit",
		ConnectTimeout:  500 * time.Millisecond,
		RequestTimeout:  2 * time.Second,
	}

	httpClient := &http.Client{
		// Keep this small because each attempt has its own RequestTimeout.
		Timeout: 0,
	}

	// In real code, body would be a request payload (e.g. bytes.NewBuffer).
	// Keeping it simple here:
	body := io.NopCloser(nil)
	_ = body

	// The main point is the selector behavior; integrating payload formatting is straightforward.
	fmt.Println(selector)
}
```

### Step-by-step: what each important block does

#### `EgressSelector` config
I configured two different timeouts:

- `ConnectTimeout`: how long we wait to establish TCP (fast fail)
- `RequestTimeout`: overall per-attempt time limit (connect + TLS + request)

This separation is important: I wanted failures to surface quickly, but I didn’t want to kill slow-but-valid requests too aggressively.

#### `isRetryableOrFallback`
This function decides whether the primary error should trigger fallback.

I kept it strict and practical:
- If we hit our context deadline, fallback
- If the network error is a timeout/temporary failure, fallback
- Otherwise, treat it as permanent and don’t waste time trying backup

That makes the behavior predictable under both “blackhole route” and “connection refused” failure modes.

#### `doAgainst(base string)`
This performs one attempt:
- Creates an attempt context with `RequestTimeout`
- Builds a POST request to a fully qualified URL
- If response is 5xx, it triggers fallback-worthy failure

I treated 4xx as permanent errors because they won’t improve by changing egress.

#### `newClientWithConnectTimeout`
This is where the “deterministic” part gets real.

Even if your app has a request timeout, TCP connect timing can still be messy unless the transport is configured.

By setting `DialContext` on the transport’s `Dialer`, I ensured the client fails quickly and consistently.

## Example run: primary blackholed, backup works

In my environment, the primary egress path sometimes behaved like a **blackhole route**: packets got dropped without immediate refusal.

That’s the worst case for long TCP timeouts. With the client above:

- Primary connect failed within `ConnectTimeout` (500ms)
- Fallback immediately tried the backup within its own deadline
- Total added latency stayed bounded (roughly connect timeout + backup request timeout)

What happened in practice:
- Service logs showed `selected endpoint=backup`
- Latency stayed under a couple seconds instead of tens of seconds
- Retries became unnecessary, reducing load spikes on the broken egress path

## Observability: how I made it debuggable

I didn’t rely only on the client returning success/failure. I logged the endpoint selection as a first-class field.

Here’s a minimal pattern I used:

```go
selectedEndpoint := "primary"
respBody, selectedEndpoint, err := selector.DoJSONPost(ctx, httpClient, payloadReader, "application/json")
if err != nil {
	log.Printf("egress=unknown selected=%s err=%v", selectedEndpoint, err)
	return
}
log.Printf("egress=success selected=%s bytes=%d", selectedEndpoint, len(respBody))
```

This “selected=primary|backup” signal was the difference between guessing and knowing what the system did during an incident.

## Security and safety notes I learned the hard way

- **Fallback must not change semantics**: both endpoints must behave equivalently (auth, headers, request shape), otherwise you’ll create “success on backup but wrong behavior” incidents.
- **Don’t fallback on 4xx**: deterministic fallback shouldn’t mask client-side bugs.
- **Keep timeouts short and bounded**: too-short timeouts create cascading failures during transient latency spikes.

## FinOps angle: why this reduced cost

When egress is failing, retries amplify network usage and can trigger autoscaling (CPU, queue depth, more instances), which can increase spend fast—especially across hybrid links.

Bounding request time and controlling retry/fallback logic reduced:
- wasted outbound traffic
- tail latency-induced concurrency
- “panic scale-out” triggered by hung connections

In other words, the deterministic client helped both reliability **and** cloud spend.

## Conclusion

I implemented deterministic fallback egress for a hybrid, multi-region setup by pushing fail-fast connect timeouts and explicit primary→backup switching into the application layer. By controlling attempt deadlines in the HTTP transport and making fallback decisions based on concrete network errors, I replaced unpredictable network failover behavior with consistent, debuggable client behavior—keeping latency bounded and reducing avoidable retry-driven load.

Deterministic Fallback Dataplane For Hybrid Egress In Multi-Region Clusters

## The problem I tripped over: “Same Dockerfile, different image”
I was running a CI/CD pipeline where every PR kicked off a Docker build, then pushed an image to a registry. The weird part: even when nothing in the Dockerfile “looked different,” the produced image digest would sometimes change.

At first I blamed base images, but the behavior persisted even when I pinned versions. What I eventually found was more subtle: **BuildKit** (Docker’s modern builder) can reuse cache in ways that depend on build context and cache metadata. If the cache key isn’t stable, your pipeline can rebuild layers in different orders and end up with a different final image digest—even if the Dockerfile content is identical.

I wanted a CI pipeline that would produce **deterministic rebuild behavior**: same inputs → same cache behavior → fewer surprise rebuilds → stable digests.

This post documents a very specific pattern I used: **a workflow that generates a “Dockerfile rebuild manifest” and feeds it into BuildKit cache keys** so cache reuse becomes intentional and explainable.

---

## What I mean by determinism (in CI terms)
A Docker image digest is a hash of the final content. Determinism here means:

- **When nothing meaningful changes**, CI should reuse the same build cache and avoid unnecessary rebuilds.
- **When something meaningful changes**, CI should invalidate cache predictably.
- **Rebuilds should be explainable**: you should be able to point to a file that represents the “inputs that matter.”

To do this, I make a manifest file that summarizes the Docker build inputs I care about (Dockerfile, build args, and selected context files), then I use that manifest to influence the BuildKit cache key.

---

## The core idea
1. Parse a Docker build configuration:
   - Dockerfile content (not just its path)
   - `--build-arg` values
   - A list of context files that should affect the cache (e.g., `package-lock.json`, `requirements.txt`, etc.)
2. Generate a `dockerfile-rebuild-manifest.json`
3. Use BuildKit with:
   - registry-backed cache
   - cache keys tied to the manifest hash

This is niche because most pipelines just run `docker build` and rely on implicit caching. Here I make caching explicit and stable.

---

## Example repo layout
Here’s a minimal app:

```
.
├── Dockerfile
├── package.json
├── package-lock.json
├── src/
│   └── index.js
└── .github/
    └── workflows/
        └── build.yml
```

### Dockerfile (simple Node app)
```dockerfile
# syntax=docker/dockerfile:1.7
FROM node:20-alpine

ARG NODE_ENV=production
ENV NODE_ENV=${NODE_ENV}

WORKDIR /app

COPY package.json package-lock.json ./
RUN npm ci --only=production

COPY src ./src

CMD ["node", "src/index.js"]
```

---

## Step 1: Generate the rebuild manifest in CI
I use a small Node script so it’s easy to hash JSON deterministically and to read build args.

### `scripts/dockerfile-manifest.mjs`
```javascript
import fs from "node:fs";
import crypto from "node:crypto";

function sha256File(path) {
  const buf = fs.readFileSync(path);
  return crypto.createHash("sha256").update(buf).digest("hex");
}

function stableStringify(obj) {
  // Deterministic JSON stringification (stable key order)
  if (obj === null || typeof obj !== "object") return JSON.stringify(obj);
  if (Array.isArray(obj)) return `[${obj.map(stableStringify).join(",")}]`;
  const keys = Object.keys(obj).sort();
  return `{${keys.map((k) => `${JSON.stringify(k)}:${stableStringify(obj[k])}`).join(",")}}`;
}

const dockerfilePath = process.env.DOCKERFILE_PATH || "Dockerfile";
const buildArgsJson = process.env.BUILD_ARGS_JSON || "{}";

// These context files are the ones I care about for cache invalidation.
// The idea: avoid hashing the entire context folder (slow) while still being correct.
const contextFiles = (process.env.CONTEXT_FILES || "package-lock.json")
  .split(",")
  .map((s) => s.trim())
  .filter(Boolean);

const manifest = {
  dockerfile: {
    path: dockerfilePath,
    contentSha256: sha256File(dockerfilePath),
  },
  buildArgs: JSON.parse(buildArgsJson),
  context: {
    files: Object.fromEntries(contextFiles.map((f) => [f, sha256File(f)])),
  },
  schemaVersion: 1,
  createdAtUtc: new Date().toISOString(),
};

const manifestString = stableStringify(manifest);
const manifestSha = crypto.createHash("sha256").update(manifestString).digest("hex");

fs.writeFileSync(
  "dockerfile-rebuild-manifest.json",
  JSON.stringify({ ...manifest, manifestSha256: manifestSha }, null, 2)
);

console.log(`manifestSha256=${manifestSha}`);
```

### Why this helps
- `contentSha256` makes Dockerfile changes explicit.
- `buildArgs` captures changes to `NODE_ENV` (and any other build args).
- `context` hashes only specific “decision files” like `package-lock.json`.
- `manifestSha256` becomes the stable cache key input.

I also pinned the JSON output to stable key ordering, so hashing doesn’t fluctuate due to object key ordering.

---

## Step 2: Use BuildKit cache with a manifest-based key
In GitHub Actions, I run BuildKit via `docker/build-push-action`, enabling registry cache.

### `.github/workflows/build.yml`
```yaml
name: ci-build-deterministic-docker-cache

on:
  pull_request:
  push:
    branches: [ main ]

permissions:
  contents: read
  packages: write

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Set build inputs
        id: inputs
        env:
          DOCKERFILE_PATH: Dockerfile
          BUILD_ARGS_JSON: ${{ toJSON({ NODE_ENV: "production" }) }}
          CONTEXT_FILES: package-lock.json
        run: |
          echo "DOCKERFILE_PATH=${DOCKERFILE_PATH}" >> $GITHUB_ENV
          echo "BUILD_ARGS_JSON=${BUILD_ARGS_JSON}" >> $GITHUB_ENV
          echo "CONTEXT_FILES=${CONTEXT_FILES}" >> $GITHUB_ENV

      - name: Generate rebuild manifest
        id: manifest
        run: |
          node scripts/dockerfile-manifest.mjs
          echo "MANIFEST_SHA=$(node -e " +
            "\"const m=require('./dockerfile-rebuild-manifest.json'); console.log(m.manifestSha256)\" )" >> $GITHUB_ENV

      - name: Login to registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push with cache keyed by manifest
        uses: docker/build-push-action@v6
        with:
          context: .
          file: Dockerfile
          push: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
          tags: ghcr.io/${{ github.repository }}:pr-${{ github.event.number }}
          build-args: |
            NODE_ENV=production
          cache-from: |
            type=registry,ref=ghcr.io/${{ github.repository }}:buildcache-manifest-${{ env.MANIFEST_SHA }}
          cache-to: |
            type=registry,mode=max,ref=ghcr.io/${{ github.repository }}:buildcache-manifest-${{ env.MANIFEST_SHA }}

          # BuildKit settings (helps cache determinism)
          provenance: false
          sbom: false
```

### What happens when this workflow runs
1. Checkout gets the repo.
2. A manifest is generated:
   - It hashes `Dockerfile`, `package-lock.json`, and the build args.
   - It writes `dockerfile-rebuild-manifest.json`.
   - It exports `MANIFEST_SHA`.
3. `docker/build-push-action` runs BuildKit with cache storage in the registry:
   - `cache-from` points at a cache namespace **named after the manifest hash**.
   - `cache-to` writes the cache into that same namespace.

That means the cache key is no longer a vague “whatever BuildKit decides.” It’s explicitly derived from the rebuild inputs that matter.

---

## Step 3: Verify it’s working (and see rebuild behavior)
I like checking two things:

1. **The cache namespace changes only when it should**
   - Editing `src/index.js` should not change the cache key if I’m only hashing `package-lock.json`.
   - Updating `package-lock.json` should change the cache key.

2. **Build logs mention cache reuse**
   - When cache hits occur, BuildKit will say steps are `CACHED`.
   - When cache misses occur, the steps re-run.

### Example: change only application source
- Edit `src/index.js`
- Re-run the workflow
- The manifest hash stays the same (because it’s derived from Dockerfile + package-lock + build args)
- Build steps that depend on dependencies should reuse cache

### Example: change `package-lock.json`
- Re-run the workflow
- The manifest hash changes
- The cache namespace changes
- Dependency install step rebuilds predictably

---

## Why registry-backed cache namespaces matter
One reason I liked this pattern is that it isolates cache histories.

If you use a single shared cache tag like:

- `type=registry,ref=ghcr.io/org/app:buildcache`

then cache content can reflect a long history of builds with different inputs. That makes “why did it rebuild?” harder.

By using `buildcache-manifest-${MANIFEST_SHA}`, I get a direct mapping:

- **manifest → cache namespace**
- **cache namespace → “the build inputs that produced it”**

This is a big deal in mission-critical platform engineering, where I want CI behavior to be stable and auditable.

---

## FinOps angle: fewer surprise rebuilds
CI rebuilds waste compute minutes and registry IO. This approach reduces “accidental invalidation” and makes cache reuse more consistent.

Even though it doesn’t guarantee perfect cost savings in all cases, in my experience it:
- reduces dependency reinstall churn
- lowers the amount of data pushed/pulled
- makes cache invalidation intentional, not incidental

---

## Complete minimal script + Dockerfile recap
- `scripts/dockerfile-manifest.mjs` produces `dockerfile-rebuild-manifest.json` and prints `manifestSha256`
- Workflow stores BuildKit cache in a registry namespace keyed by `manifestSha256`
- Cache reuse becomes predictable because the key is deterministic

---

## Conclusion
I built a CI/CD pattern where caching is driven by a deterministic **rebuild manifest hash** instead of implicit BuildKit heuristics. By hashing the Dockerfile content, selected context decision files (like `package-lock.json`), and build args, then using that hash to namespace BuildKit registry cache, I made Docker rebuild behavior stable and explainable—leading to fewer surprise rebuilds and more reliable pipeline performance.

Ci/Cd Pipelines For Deterministic Dockerfile Rebuilds Using Buildkit Cache Keys

## The problem I ran into

I built a hybrid webhook pipeline that had to accept payment events from **two different SaaS vendors**, then forward them to internal services that were deployed across **AWS Lambda** and **Google Cloud Run** (hybrid + multi-cloud).

What surprised me: the *same* webhook sometimes got delivered multiple times (payment systems often retry), and the vendor’s event IDs were only unique **per vendor**, not across vendors. So I couldn’t safely use a single “event id” as-is for deduplication.

Even worse, I needed deterministic routing: given an event payload, the router had to pick the same target service every time—because I was doing cost accounting (FinOps) per routed destination and wanted consistent bills.

So I ended up building a tiny “deterministic webhook router” that:

- generates a stable idempotency key from the payload (cross-vendor safe),
- deduplicates requests using Redis,
- routes to either Cloud Run or Lambda based on a hash of normalized fields,
- forwards the request with the exact same payload bytes.

This post is the exact approach I implemented.

---

## Architecture (the parts that matter)

- **Cloud Run**: hosts the webhook entrypoint. It receives the vendor webhook.
- **Redis**: stores idempotency keys to prevent duplicate processing.
- **AWS Lambda**: receives routed events.
- **Cloud Run service**: receives routed events destined for GCP.
- **Routing rule**: a stable hash of normalized fields chooses destination.

Key concept: **idempotency** means “if the same logical request arrives multiple times, the system processes it once.”

---

## Data model for routing and dedup

I used two vendor-specific fields:

- `vendor` (string: `"stripe"` or `"adyen"` in my case)
- `event_type` (string)
- `created_at` (ISO timestamp)
- `payload` (the raw JSON)

To make dedup safe across vendors, I build an idempotency key from:

- normalized `vendor`
- normalized `event_type`
- normalized `created_at` (parsed and re-serialized so formatting differences don’t matter)
- a hash of the payload content

---

## The Cloud Run webhook router (Python)

Below is the working Cloud Run entrypoint using **Flask** and **redis-py**.

### Install dependencies

```txt
flask==3.0.3
redis==5.0.8
requests==2.32.3
boto3==1.34.162
python-dateutil==2.9.0.post0
gunicorn==22.0.0
```

### Router code

```python
# main.py
import os
import json
import hashlib
import hmac
import base64
from datetime import datetime, timezone

import boto3
import redis
import requests
from flask import Flask, request, Response
from dateutil import parser as date_parser

app = Flask(__name__)

# Environment variables
REDIS_URL = os.environ.get("REDIS_URL", "redis://localhost:6379/0")
AWS_REGION = os.environ.get("AWS_REGION", "us-east-1")
AWS_LAMBDA_FUNCTION_NAME = os.environ.get("AWS_LAMBDA_FUNCTION_NAME", "webhook-processor")
AWS_LAMBDA_ROLE_ARN = os.environ.get("AWS_LAMBDA_ROLE_ARN", "")  # optional if needed

# The two destinations I used
GCP_TARGET_URL = os.environ.get("GCP_TARGET_URL", "https://example-a.run.app/ingest")

# A secret used for deterministic hashing (prevents leaking payload structure through routing hash)
ROUTING_SECRET = os.environ.get("ROUTING_SECRET", "dev-not-for-prod-change-me")

# How long to keep dedup keys (seconds)
DEDUP_TTL_SECONDS = int(os.environ.get("DEDUP_TTL_SECONDS", "86400"))

rds = redis.Redis.from_url(REDIS_URL, decode_responses=False)

lambda_client = boto3.client("lambda", region_name=AWS_REGION)

def normalize_created_at(created_at_value: str) -> str:
    """
    Parse a timestamp string and re-serialize it as an RFC3339-like UTC string.
    This prevents formatting differences from producing different idempotency keys.
    """
    dt = date_parser.parse(created_at_value)
    if dt.tzinfo is None:
        dt = dt.replace(tzinfo=timezone.utc)
    dt_utc = dt.astimezone(timezone.utc)
    # Stable format
    return dt_utc.isoformat().replace("+00:00", "Z")

def stable_payload_hash(payload_obj: dict) -> str:
    """
    Hash the payload content deterministically:
    - JSON stringify with sorted keys
    - stable UTF-8 encoding
    """
    normalized = json.dumps(payload_obj, sort_keys=True, separators=(",", ":")).encode("utf-8")
    return hashlib.sha256(normalized).hexdigest()

def idempotency_key(vendor: str, event_type: str, created_at: str, payload_obj: dict) -> str:
    payload_digest = stable_payload_hash(payload_obj)
    normalized_created_at = normalize_created_at(created_at)

    # Cross-vendor safe logical key
    raw = f"{vendor.lower()}|{event_type}|{normalized_created_at}|{payload_digest}"

    # HMAC keeps the key from becoming a plaintext representation in Redis
    digest = hmac.new(ROUTING_SECRET.encode("utf-8"), raw.encode("utf-8"), hashlib.sha256).hexdigest()
    return f"webhook:dedup:{digest}"

def route_destination(vendor: str, event_type: str, created_at: str, payload_obj: dict) -> str:
    """
    Deterministically choose a destination using an HMAC of normalized fields.
    Returns either 'gcp' or 'aws'.
    """
    normalized_created_at = normalize_created_at(created_at)
    raw = f"route|{vendor.lower()}|{event_type}|{normalized_created_at}|{stable_payload_hash(payload_obj)}"
    digest = hmac.new(ROUTING_SECRET.encode("utf-8"), raw.encode("utf-8"), hashlib.sha256).digest()

    # First byte decides destination. 0-127 => gcp, 128-255 => aws
    return "gcp" if digest[0] < 128 else "aws"

@app.post("/webhook")
def webhook() -> Response:
    # Read raw body bytes so we can forward the exact same bytes downstream.
    raw_body = request.get_data(cache=False)
    try:
        payload = json.loads(raw_body.decode("utf-8"))
    except Exception:
        return Response("invalid json", status=400)

    # Expect vendor schema
    vendor = payload.get("vendor")
    event_type = payload.get("event_type")
    created_at = payload.get("created_at")

    if not vendor or not event_type or not created_at:
        return Response("missing required fields", status=400)

    # Use full payload object for dedup; stable hash makes it content-based.
    idem_key = idempotency_key(vendor, event_type, created_at, payload)
    destination = route_destination(vendor, event_type, created_at, payload)

    # Redis SET with NX ensures only the first request "wins".
    # It returns:
    # - True if key was set
    # - False if key already existed
    already = rds.set(idem_key, b"1", nx=True, ex=DEDUP_TTL_SECONDS)
    if not already:
        # Duplicate logical event: short-circuit.
        return Response("duplicate ignored", status=200)

    # Forward downstream
    if destination == "gcp":
        resp = requests.post(
            GCP_TARGET_URL,
            data=raw_body,
            headers={"Content-Type": request.headers.get("Content-Type", "application/json")},
            timeout=10,
        )
        return Response(resp.text, status=resp.status_code)
    else:
        # Invoke AWS Lambda with the raw payload bytes.
        # Payload must be JSON-serializable for Lambda Invoke.
        # We send the original parsed payload to preserve structure.
        lambda_payload = payload
        invoke_kwargs = {
            "FunctionName": AWS_LAMBDA_FUNCTION_NAME,
            "InvocationType": "RequestResponse",
            "Payload": json.dumps(lambda_payload).encode("utf-8"),
        }

        if AWS_LAMBDA_ROLE_ARN:
            invoke_kwargs["ClientContext"] = AWS_LAMBDA_ROLE_ARN

        resp = lambda_client.invoke(**invoke_kwargs)

        # Lambda returns a streaming payload in resp["Payload"]
        lambda_body = resp["Payload"].read().decode("utf-8")
        return Response(lambda_body, status=200)

if __name__ == "__main__":
    # For local dev only; Cloud Run uses Gunicorn typically
    app.run(host="0.0.0.0", port=8080)
```

### What’s happening (line-by-line in the critical parts)

- I read **raw request bytes** using `request.get_data(...)`.
  - That matters because I forward the exact same bytes to Cloud Run or Lambda.
- I parse JSON and extract `vendor`, `event_type`, `created_at`.
- I compute an **idempotency key**:
  - normalize timestamp to UTC stable format,
  - hash normalized payload with sorted keys,
  - HMAC it with `ROUTING_SECRET` so Redis doesn’t store readable content.
- I call `redis.set(key, ..., nx=True, ex=TTL)`:
  - `nx=True` means “set only if not exists”.
  - This is the dedup guarantee. Only the first duplicate gets processed.
- I compute destination using deterministic HMAC and pick by a byte threshold.
- I forward:
  - to Cloud Run via HTTP POST,
  - to Lambda via `lambda_client.invoke`.

---

## Local test harness

To test determinism and dedup without deploying, I used `redis` locally.

### Run Redis

```bash
docker run --rm -p 6379:6379 redis:7
```

### Run the router locally

```bash
export REDIS_URL=redis://localhost:6379/0
export AWS_REGION=us-east-1
export AWS_LAMBDA_FUNCTION_NAME=webhook-processor
export GCP_TARGET_URL=http://localhost:8081/ingest
export ROUTING_SECRET=local-secret
export DEDUP_TTL_SECONDS=86400

pip install -r requirements.txt
python main.py
```

### Create a mock payload

```bash
cat > payload.json <<'JSON'
{
  "vendor": "stripe",
  "event_type": "payment.succeeded",
  "created_at": "2026-05-06T12:34:56Z",
  "amount": 4200,
  "currency": "USD",
  "details": { "invoice": "inv_123" }
}
JSON
```

### Call twice with the same payload

```bash
curl -s -X POST http://localhost:8080/webhook \
  -H 'Content-Type: application/json' \
  --data @payload.json

curl -s -X POST http://localhost:8080/webhook \
  -H 'Content-Type: application/json' \
  --data @payload.json
```

Expected behavior:

- First call routes and forwards.
- Second call returns `duplicate ignored`.

Because the idempotency key is derived from content, even if the vendor retries later, it still dedups.

---

## Deploying to Cloud Run (minimal)

### Dockerfile

```dockerfile
# Dockerfile
FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY main.py .

# Cloud Run expects port 8080
ENV PORT=8080

CMD ["gunicorn", "-b", ":8080", "main:app"]
```

### Build and deploy

```bash
gcloud run deploy webhook-router \
  --source . \
  --region us-central1 \
  --platform managed \
  --allow-unauthenticated \
  --set-env-vars "REDIS_URL=redis://10.0.0.5:6379/0,AWS_REGION=us-east-1,AWS_LAMBDA_FUNCTION_NAME=webhook-processor,GCP_TARGET_URL=https://example-a.run.app/ingest,ROUTING_SECRET=change-me,DEDUP_TTL_SECONDS=86400"
```

---

## Why this approach worked for FinOps and platform engineering

In practice, I hit two cost-related problems:

1. **Duplicate deliveries** were inflating invocation counts.
2. **Non-deterministic routing** made dashboards noisy (events “sometimes” went to AWS, “sometimes” to GCP).

This router fixed both:

- Dedup via Redis made “logical once” processing align with billing events.
- Deterministic routing ensured the same event type/date/vendor combination landed on the same destination consistently, which stabilized cost attribution.

---

## Conclusion

I built a deterministic hybrid webhook router where I compute a cross-vendor idempotency key (content + normalized timestamp + HMAC) and use Redis `SET NX` to process each logical event once, then route to Cloud Run or AWS Lambda using the same normalized fields every time. The result was a webhook system that behaved predictably under retries and produced stable routing for cleaner FinOps and platform engineering metrics.

Building A Deterministic Multi Cloud Webhook Router With Cloud Run And Aws Lambda

## The weird problem I ran into

I was building an ETL pipeline that reads objects from one provider (S3-compatible storage) and writes into another—across a hybrid setup where outbound network traffic had to go through a SigV4-terminating proxy (a layer that signs requests on behalf of workloads).

Everything “worked” for small datasets, but as volumes grew I started seeing:

- occasional `SignatureDoesNotMatch` errors (only under load)
- non-deterministic “already processed” markers (the same input sometimes produced a different manifest hash)
- retries that looked like no-ops but still burned time and cost

This post documents the exact pattern that fixed it for me: **a SigV4 signing proxy + deterministic manifest hashing derived from canonical request fields** so the pipeline can safely dedupe across clouds.

---

## What I mean by “SigV4 proxy”

**AWS SigV4 (Signature Version 4)** is a request-signing scheme for S3-style APIs. A “SigV4 proxy” is an HTTP service sitting in the middle that:

1. accepts a request from your pipeline without the final provider-specific signature
2. signs it using credentials it has been configured with
3. forwards it to the actual object storage endpoint

This is common when:
- egress is controlled and secrets must stay inside the proxy boundary
- workloads can’t access cloud credentials directly
- you’re bridging multiple S3-compatible implementations with one signing service

---

## The failure mode: hash instability + signature drift

I had a “manifest” file per run that included per-object metadata (key, size, etag, timestamp). My dedupe logic was:

- compute `sha256(manifest.json)`
- store it as the run marker
- if marker exists, skip

The problem: different clouds return metadata fields differently, and even within the same cloud, JSON object key ordering and some timestamp formatting produced different hashes for semantically identical runs.

Separately, my proxy would sometimes sign requests with headers that changed across retries (notably the `Host`, `x-amz-content-sha256`, or a timestamp header). Under retry, the canonical request used for the signature changed, resulting in `SignatureDoesNotMatch`.

So I fixed both sides:
- **make manifest hashing deterministic**
- **make signing inputs deterministic**

---

## A concrete architecture that worked

### Data flow

1. Pipeline enumerates objects from the source bucket.
2. For each object, it computes a **content-stable descriptor**.
3. It builds a **deterministic manifest**:
   - keys sorted
   - normalized fields (no provider-specific timestamp formats)
   - canonical JSON encoding
4. It uploads the manifest to the destination using a **SigV4 signing proxy**.
5. It uses the manifest hash as the dedupe marker.

### Canonical JSON rule I used

- Only include fields that won’t vary between retries:
  - `bucket`, `key`, `size`, `etag` (normalized), and a stable `versionId` if present
- Normalize `etag`:
  - sometimes it includes quotes: `"abcd..."` vs `abcd...`
- Serialize with stable key ordering

---

## Step-by-step: deterministic manifest hashing in Python

Below is a small but complete script that:

- reads a list of “source objects” (simulated)
- builds a deterministic manifest
- produces a SHA-256 hash that is stable across runs

```python
# manifest_hash.py
import hashlib
import json
import re
from dataclasses import dataclass, asdict
from typing import List, Optional

ETAG_RE = re.compile(r'^"?([^"]+)"?$')

def normalize_etag(etag: str) -> str:
    """
    Normalize common S3 ETag formatting differences.
    Examples:
      '"abc123"' -> 'abc123'
      'abc123'   -> 'abc123'
    """
    m = ETAG_RE.match(etag.strip())
    return m.group(1) if m else etag.strip()

@dataclass(frozen=True)
class ObjectDescriptor:
    bucket: str
    key: str
    size: int
    etag: str
    version_id: Optional[str] = None

    def stable_dict(self):
        d = asdict(self)
        d["etag"] = normalize_etag(d["etag"])
        # Keep None fields out for stability
        return {k: v for k, v in d.items() if v is not None}

def canonical_json(data) -> bytes:
    """
    Deterministic JSON encoding:
    - sort keys
    - no whitespace
    - ensure UTF-8
    """
    return json.dumps(
        data,
        sort_keys=True,
        separators=(",", ":"),
        ensure_ascii=False,
    ).encode("utf-8")

def build_manifest(run_id: str, objects: List[ObjectDescriptor]):
    # Sort descriptors to remove enumeration ordering differences
    objects_sorted = sorted(objects, key=lambda o: (o.bucket, o.key))

    manifest = {
        "schema": "etl/manifest/v1",
        "run_id": run_id,  # stable per logical run
        "objects": [o.stable_dict() for o in objects_sorted],
    }
    manifest_bytes = canonical_json(manifest)
    manifest_hash = hashlib.sha256(manifest_bytes).hexdigest()
    return manifest, manifest_hash

if __name__ == "__main__":
    # Simulated object list returned by a source provider
    objs = [
        ObjectDescriptor(bucket="src-bucket", key="data/part-2.json", size=12, etag='"aaa"'),
        ObjectDescriptor(bucket="src-bucket", key="data/part-1.json", size=12, etag="aaa"),
    ]

    manifest, h = build_manifest(run_id="2026-04-08T00:00:00Z", objects=objs)
    print("Manifest hash:", h)
    print("Canonical manifest JSON:", canonical_json(manifest).decode("utf-8"))
```

### What happens when you run it

Run:

```bash
python manifest_hash.py
```

You’ll see:
- the hash is computed from canonical JSON
- the `objects` array order is normalized
- `etag` formatting is normalized so `"aaa"` and `aaa` hash the same

That eliminates the “already processed” marker instability.

---

## Step-by-step: deterministic request signing inputs

The signing proxy issue came down to: **ensure the set of headers and the canonical payload hash are identical across retries**.

Most SigV4 signing errors like `SignatureDoesNotMatch` are caused by one of these differences:
- `Host` mismatch (or proxy forwards to a different host than signer used)
- payload hash differs (especially if streaming uploads vary)
- `x-amz-date` differs while canonicalization expects something else
- headers included/excluded during canonical request differ

In my setup, I controlled the boundary by making all upload requests go through one small proxy contract:

### Proxy contract

- Pipeline sends:
  - HTTP method
  - target URL path
  - headers: only a minimal stable set
  - payload either:
    - buffered fully (so payload hash is stable), or
    - using “unsigned payload” mode if the target supports it (more on this below)

For S3, the safest approach for determinism is **buffer the payload** in the client that calls the proxy (so the proxy signs the same bytes every time).

---

## A minimal signing proxy (FastAPI) example

This is not a production-grade proxy, but it shows the shape of the fix. The important part is the proxy takes a canonical request “envelope”, signs with AWS SDK tooling, and forwards.

> Note: In real systems you’d configure credentials via environment variables/secret manager and likely enforce auth. Here I focus on the request determinism.

```python
# sigv4_proxy.py
import os
from fastapi import FastAPI, Request, HTTPException
import httpx
import boto3
from botocore.awsrequest import AWSRequest
from botocore.auth import SigV4Auth
from botocore.credentials import Credentials

app = FastAPI()

AWS_REGION = os.environ.get("AWS_REGION", "us-east-1")
ACCESS_KEY = os.environ["AWS_ACCESS_KEY_ID"]
SECRET_KEY = os.environ["AWS_SECRET_ACCESS_KEY"]
SESSION_TOKEN = os.environ.get("AWS_SESSION_TOKEN", "")

TARGET_SERVICE = os.environ.get("TARGET_SERVICE", "s3")  # 's3' for object APIs

def sign_request(method: str, url: str, headers: dict, body: bytes):
    # Build an AWSRequest for SigV4 signing.
    # Using botocore ensures canonical request computation is consistent.
    aws_req = AWSRequest(method=method, url=url, data=body, headers=headers)

    creds = Credentials(ACCESS_KEY, SECRET_KEY, SESSION_TOKEN or None)
    signer = SigV4Auth(creds, TARGET_SERVICE, AWS_REGION)
    signer.add_auth(aws_req)

    # Convert signed request back to headers
    return aws_req.headers

@app.post("/upload")
async def upload(request: Request):
    """
    Expected JSON body:
    {
      "target_url": "https://host/bucket/key",
      "method": "PUT",
      "headers": { "content-type": "...", ... },
      "payload_base64": "..."
    }
    """
    payload = await request.json()
    target_url = payload["target_url"]
    method = payload.get("method", "PUT")
    headers = payload.get("headers", {})

    import base64
    body = base64.b64decode(payload["payload_base64"])

    # Ensure deterministic header set.
    # Example: don't allow changing Host; let target_url determine it.
    stable_headers = {
        "content-type": headers.get("content-type", "application/octet-stream"),
        "content-length": str(len(body)),
    }

    signed_headers = sign_request(method, target_url, stable_headers, body)

    async with httpx.AsyncClient(timeout=60) as client:
        resp = await client.request(
            method,
            target_url,
            headers=dict(signed_headers),
            content=body,
        )

    if resp.status_code >= 400:
        raise HTTPException(status_code=resp.status_code, detail=resp.text)

    return {"status": "ok", "etag": resp.headers.get("ETag")}
```

### Why this fixed my retries

- The proxy signs using the exact `body` bytes it receives.
- The proxy sets stable `content-length` based on buffered payload size.
- The canonical request is less likely to drift between retries.

In my previous approach, I was streaming bytes through multiple layers and accidentally changed payload hash inputs when the upload was retried.

---

## End-to-end: upload a deterministic manifest through the proxy

Here’s a small client that:

1. Builds the manifest deterministically
2. Uploads it through the proxy
3. Uses the manifest hash as the object key (dedupe)

```python
# upload_manifest.py
import base64
import os
import requests
import json
import hashlib
from manifest_hash import ObjectDescriptor, build_manifest

PROXY_URL = os.environ.get("PROXY_URL", "http://localhost:8000")
DEST_BUCKET = os.environ.get("DEST_BUCKET", "dest-bucket")
DEST_HOST = os.environ.get("DEST_HOST", "s3.example.com")  # endpoint host
DEST_REGION = os.environ.get("DEST_REGION", "us-east-1")  # not used directly here

def object_url(bucket: str, key: str) -> str:
    return f"https://{DEST_HOST}/{bucket}/{key}"

if __name__ == "__main__":
    # Simulated objects from source
    objs = [
        ObjectDescriptor(bucket="src-bucket", key="data/part-1.json", size=12, etag='"aaa"'),
        ObjectDescriptor(bucket="src-bucket", key="data/part-2.json", size=12, etag="aaa"),
    ]

    manifest, h = build_manifest(run_id="2026-04-08T00:00:00Z", objects=objs)
    manifest_bytes = json.dumps(manifest, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")

    dedupe_key = f"manifests/{h}.json"

    payload = {
        "target_url": object_url(DEST_BUCKET, dedupe_key),
        "method": "PUT",
        "headers": {"content-type": "application/json"},
        "payload_base64": base64.b64encode(manifest_bytes).decode("ascii"),
    }

    resp = requests.post(f"{PROXY_URL}/upload", json=payload, timeout=60)
    resp.raise_for_status()

    print("Uploaded manifest to:", dedupe_key)
    print("Manifest hash:", h)
    print("Proxy response:", resp.json())
```

### What happens when you run it

1. `manifest_hash.py` proves stable hashing.
2. `upload_manifest.py` uploads using that hash-based key.
3. Re-running the script produces the same key, so the destination either:
   - overwrites identical content, or
   - you can switch to `PUT` with conditional headers (like `If-None-Match`) to avoid overwrites

---

## FinOps note: why deterministic dedupe mattered for cost

Before this change, every retry produced a different manifest marker, so downstream steps reprocessed the same objects. That created:

- extra storage reads/writes
- repeated compute execution
- extra egress through the proxy

Once manifest hashing became deterministic and request signing inputs stabilized, retries stopped triggering “new work,” and the run marker became a real idempotency key.

---

## Operational checklist I ended up relying on

- **Canonical manifest hashing**
  - stable sort of objects
  - normalized ETag formatting
  - deterministic JSON encoding (sorted keys, no whitespace)
- **Deterministic upload bytes**
  - buffer payload before signing
  - set `content-length` explicitly
- **Proxy should sign with inputs that won’t drift**
  - minimal, stable header set
  - avoid “helpful” middleware that mutates headers between retries

---

## Conclusion

I learned that hybrid multi-cloud pipelines fail in annoying, non-obvious ways when two sources of nondeterminism collide: **manifest hashing instability** and **SigV4 signature drift caused by changing request-signing inputs**. By enforcing canonical JSON for manifests and making uploads go through a SigV4 signing proxy that signs deterministic buffered payloads with a stable header set, I got reliable retries and true idempotent dedupe markers—turning an error-prone cross-cloud ETL into something that behaves consistently at scale.

Multi Cloud S3 Compatible Etl With Sigv4 Proxy And Deterministic Manifest Hashing

## The problem I ran into: “green” builds that later explode
I recently helped wire a CI/CD pipeline for a distributed cloud platform, and everything looked perfect at first: unit tests passed, image builds finished, deployments went out… and then the application would occasionally fail in production with behavior that didn’t match the commit that was supposedly deployed.

The weird part was that rerunning the same pipeline sometimes produced different container images even when the Dockerfile hadn’t changed. That’s the kind of issue that earns the nickname **Heisenbug**: when you try to observe it (rerun builds), it seems to “change its nature.”

After tracing it, the root cause was a subtle one: our CI built Docker images with changing dependency artifacts (think: package indexes, lockfiles not actually pinned, npm installs with registry variability). Even worse, the build logs didn’t clearly tell us *which exact dependency content* produced each image.

So I built a deterministic **CI gate** that blocks deployments unless the produced Docker image corresponds to the exact dependency content we expect—using **content-addressed Docker layers** and a manifest signature checked during the pipeline.

---

## What this gate enforces (in plain terms)
Here’s the idea I implemented:

1. CI builds the Docker image.
2. CI computes a “fingerprint” of the *dependency inputs* (lockfiles + Dockerfile + build args that affect dependencies).
3. CI compares that fingerprint to what the image claims about its layers (content-addressing).
4. If there’s a mismatch, the pipeline fails before deploy.

This turns “we think the image is right” into “the image is proven to match the exact inputs.”

### Two important terms (briefly)
- **Docker layer content-addressing**: Docker identifies layers by the content they contain (not just by step order). If dependency inputs change, the produced layer hashes change.
- **CI gate**: an explicit step in the pipeline that must pass validation before moving on to deploy.

---

## The pipeline pieces I used
I used AWS CodeBuild (it runs containerized builds) and a Docker build flow that produces a manifest we can inspect. The gate runs after the build but before pushing (or before allowing deploy).

### High-level flow
- Checkout code
- Compute input fingerprint
- Build Docker image
- Inspect the image manifest for layer digests
- Verify that the manifest matches the fingerprint expectations
- Fail pipeline if not

---

## Step 1: Add a dependency fingerprint to your repo
I keep this file in the repo at `ci/dependency-fingerprint.sh`. It computes a SHA256 over:
- `Dockerfile`
- any lockfiles (example includes `package-lock.json`, `requirements.txt`, `poetry.lock`)
- selected build args (in this example: `BASE_IMAGE`, `PYTHON_VERSION`)

```bash
#!/usr/bin/env bash
set -euo pipefail

# Collect dependency inputs that affect what gets installed into the image.
# The goal is: same inputs => same fingerprint.
fingerprint_sources=(
  "Dockerfile"
  "package-lock.json"
  "requirements.txt"
  "poetry.lock"
)

# Only include files that exist (repo may not have all lockfiles).
tmp_list="$(mktemp)"
for f in "${fingerprint_sources[@]}"; do
  if [[ -f "$f" ]]; then
    echo "$f" >> "$tmp_list"
  fi
done

# Include key build args explicitly.
# These must match the CI build args.
base_image="${BASE_IMAGE:-}"
python_version="${PYTHON_VERSION:-}"

# Produce deterministic output by hashing filenames + file contents.
sha_input="$(cat "$tmp_list")|BASE_IMAGE=${base_image}|PYTHON_VERSION=${python_version}"

content_hash="$(
  # sha256sum output includes filename; use awk to keep just hashes for stability
  {
    # hash file content in stable order
    while IFS= read -r file; do
      sha256sum "$file" | awk '{print $1}'
    done < "$tmp_list"
    # hash the build args
    echo -n "${base_image}" | sha256sum | awk '{print $1}'
    echo -n "${python_version}" | sha256sum | awk '{print $1}'
  } | sha256sum | awk '{print $1}'
)"

rm -f "$tmp_list"

# Write fingerprint for later steps to consume
echo "$content_hash"
```

**Why this works:** It makes the pipeline compute a deterministic hash from the exact dependency inputs. If a lockfile changes or build args change, the fingerprint changes.

---

## Step 2: Build and capture layer digests (the “proof”)
Next I built the verifier `ci/verify-image-proof.sh`. It:
- reads expected fingerprint
- builds the image locally
- extracts layer digests from the image manifest
- derives a manifest proof hash
- compares it to the expected fingerprint

```bash
#!/usr/bin/env bash
set -euo pipefail

# Expected fingerprint computed earlier
expected_fingerprint="${EXPECTED_FINGERPRINT:?EXPECTED_FINGERPRINT env var required}"

# The image name we will build
image_ref="${IMAGE_REF:-local/proof-image:ci}"

# Build args that must match how fingerprint was computed
build_args=()
if [[ -n "${BASE_IMAGE:-}" ]]; then
  build_args+=(--build-arg "BASE_IMAGE=${BASE_IMAGE}")
fi
if [[ -n "${PYTHON_VERSION:-}" ]]; then
  build_args+=(--build-arg "PYTHON_VERSION=${PYTHON_VERSION}")
fi

# Build the image.
# We don't push yet; this is just to get a manifest we can inspect.
docker build \
  -t "${image_ref}" \
  "${build_args[@]}" \
  -f Dockerfile .

# Get the manifest in a portable format.
# This uses Docker's inspect to obtain layer digests deterministically.
manifest_json="$(docker image inspect "${image_ref}" --format '{{json .}}')"

# Extract layer digest-like identifiers.
# Note: docker image inspect .RootFS.Layers contains layer IDs used by Docker.
# Those are content-addressed identifiers derived from the build.
layers="$(
  echo "$manifest_json" | python3 -c '
import json,sys
obj=json.load(sys.stdin)
layers=obj.get("RootFS",{}).get("Layers",[])
print("\n".join(layers))
'
)"

# Hash the layer identifiers to get a reproducible "manifest proof"
manifest_proof="$(
  printf "%s\n" "$layers" | sha256sum | awk '{print $1}'
)"

echo "Expected fingerprint: ${expected_fingerprint}"
echo "Manifest proof:      ${manifest_proof}"

if [[ "$manifest_proof" != "$expected_fingerprint" ]]; then
  echo "ERROR: Image manifest proof does not match dependency fingerprint."
  echo "This blocks deploy to prevent Heisenbugs."
  exit 1
fi

echo "Image proof verified successfully."
```

**Why this is strict:** even if the build finishes and tests pass, the pipeline refuses to proceed unless the image’s internal layer identity matches the dependency fingerprint. That forces determinism at the artifact level.

---

## Step 3: Wire it into AWS CodeBuild (buildspec.yml)
Here’s a working `buildspec.yml` for CodeBuild that:
- installs tooling (Python for JSON extraction)
- computes the fingerprint
- builds and verifies the image proof
- only then pushes or deploys

Replace `REPOSITORY_URI` and `IMAGE_TAG` to match your setup.

```yaml
version: 0.2

env:
  variables:
    IMAGE_REF: "local/proof-image:ci"
    IMAGE_TAG: "latest"
  # These should be set in CodeBuild project env or via build parameters
  # BASE_IMAGE: "python:3.12-slim"
  # PYTHON_VERSION: "3.12"

phases:
  install:
    runtime-versions:
      python: 3.11
    commands:
      - echo "Installing dependencies (if any)..."
      - which python3
      - python3 --version
      - docker --version

  pre_build:
    commands:
      - echo "Computing dependency fingerprint..."
      - chmod +x ci/dependency-fingerprint.sh ci/verify-image-proof.sh
      - export EXPECTED_FINGERPRINT="$(./ci/dependency-fingerprint.sh)"
      - echo "Fingerprint: ${EXPECTED_FINGERPRINT}"

  build:
    commands:
      - echo "Building image and verifying content-addressed layer proof..."
      - ./ci/verify-image-proof.sh

  post_build:
    commands:
      - echo "Verification passed."
      # Optional: push only after gate passes
      # - aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $REPOSITORY_URI
      # - docker tag "${IMAGE_REF}" "${REPOSITORY_URI}:${IMAGE_TAG}"
      # - docker push "${REPOSITORY_URI}:${IMAGE_TAG}"

artifacts:
  files: []
```

**What happens when this runs:**
- `dependency-fingerprint.sh` hashes your dependency inputs.
- `verify-image-proof.sh` builds the image and computes a proof hash from layer identifiers.
- If proof doesn’t match, CodeBuild exits with a non-zero status → pipeline stops → deployment never happens.

---

## Step 4: Make Docker builds actually deterministic
The gate helps detect nondeterminism, but to avoid false failures, I also tightened Docker builds.

In practice, I used a Dockerfile pattern that pins versions and prefers lockfiles. Example Dockerfile:

```dockerfile
# syntax=docker/dockerfile:1
ARG BASE_IMAGE=python:3.12-slim
FROM ${BASE_IMAGE}

ARG PYTHON_VERSION=3.12

WORKDIR /app

# Copy lockfiles first to maximize reproducibility
COPY package-lock.json ./package-lock.json
COPY requirements.txt ./requirements.txt
COPY poetry.lock ./poetry.lock 2>/dev/null || true

# Install dependencies in a deterministic way.
# Examples assume you use lockfiles for both npm and pip/poetry.
RUN python --version
RUN pip install --no-cache-dir -r requirements.txt

# Application code
COPY . .

CMD ["python", "-c", "print('app running')"]
```

**Key point:** the fingerprint assumes that these inputs drive the layers that get installed. If your Dockerfile fetches dependencies dynamically without lockfiles, the gate will correctly detect a mismatch—sometimes after tests pass.

---

## Debugging when the gate trips (what I learned)
When I saw failures, the fastest way to understand them was:

- Compare the dependency fingerprint values between two runs.
- Inspect the built image’s layer identifiers (`docker image inspect ... RootFS.Layers`).
- Check for hidden sources of variability:
  - unpinned apt/apk package versions
  - package managers hitting indexes that change resolution without lockfiles
  - build args that silently change between pipeline runs

This approach turned mystery nondeterminism into measurable evidence.

---

## Conclusion
I implemented a deterministic CI/CD gate for AWS CodeBuild that blocks deployments when a built Docker image’s content-addressed layer identifiers don’t match a fingerprint computed from dependency inputs. By combining input hashing with manifest-derived “proof,” I traded occasional production Heisenbugs for a hard, reproducible validation step—so artifact integrity becomes something the pipeline can enforce, not something we hope is true.

A Deterministic Ci/Cd Gate For Aws Codebuild That Blocks “Heisenbugs” Via Content-Addressed Docker Layers

## The problem I ran into: “same PR” but different infrastructure drift

I was building a platform engineering workflow for ephemeral **review apps** (short-lived app environments spun up per pull request). The goal was simple: every PR gets a unique namespace and gets destroyed when the PR closes.

But after a few days, I noticed something weird in production-adjacent test environments:

- the same PR URL sometimes pointed to a different deployment
- side effects (config maps, feature flags, caches) didn’t line up deterministically
- debugging was maddening because “refreshing” didn’t mean “same world”

The cause ended up being subtle: we were generating names from mutable signals (like timestamps or local counters). Even if the PR stayed the same, the app identity could drift.

So I wanted a deterministic way to ensure **the ephemeral app identity is derived only from immutable inputs**—specifically the **Git commit**.

---

## The niche solution: namespace + Helm release names from a Git commit digest

Kubernetes namespaces and Helm release names can be based on strings, but Kubernetes has naming constraints (DNS labels). Also, Helm release names can’t be arbitrarily long.

So I implemented a tiny identity scheme:

- Compute a **SHA-256 digest** from the Git commit SHA.
- Convert it to a DNS-label-safe value.
- Use it to form:
  - namespace: `pr-<digest>`
  - Helm release: `rev-<digest>`

That means:
- the same commit always maps to the same namespace/release
- a different commit maps to a different namespace/release
- teardown becomes deterministic and idempotent

---

## What happens when the pipeline runs (end-to-end)

When CI runs for a PR:

1. CI exports `GIT_COMMIT_SHA` (the immutable commit identifier).
2. The pipeline computes `DIGEST` from it.
3. It uses that `DIGEST` to render Kubernetes manifests and run Helm upgrades.
4. The “destroy” job uses the same digest to delete the exact namespace.

No drift, no guessing.

---

## Step 1: Compute a DNS-safe digest (with code)

I used Python in CI because it’s portable and explicit about what it does.

```python
#!/usr/bin/env python3
import hashlib
import re
import sys

def dns_label_safe(s: str, max_len: int = 20) -> str:
    """
    Convert any string into a lowercase DNS-label-safe token:
    - lowercase letters, digits, and hyphens
    - trim to max_len
    """
    s = s.lower()
    # Replace invalid chars with hyphen
    s = re.sub(r'[^a-z0-9-]', '-', s)
    # Collapse consecutive hyphens
    s = re.sub(r'-{2,}', '-', s).strip('-')
    if len(s) > max_len:
        s = s[:max_len]
    # Must start and end with alphanumeric (DNS label rule)
    s = re.sub(r'^[^a-z0-9]+', '', s)
    s = re.sub(r'[^a-z0-9]+$', '', s)
    return s or "x"

def commit_digest(commit_sha: str) -> str:
    """
    Deterministically map a commit SHA to a short token.
    """
    sha_bytes = commit_sha.encode("utf-8")
    digest = hashlib.sha256(sha_bytes).hexdigest()  # 64 hex chars
    # Use first N chars as stable identity. 20 hex chars => 20 length.
    token = digest[:20]
    return dns_label_safe(token, max_len=20)

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: digest.py <GIT_COMMIT_SHA>", file=sys.stderr)
        sys.exit(2)

    commit = sys.argv[1].strip()
    token = commit_digest(commit)
    print(token)
```

### Why this works
- SHA-256 output is deterministic for the same input.
- Taking the first 20 hex chars gives a short, stable token.
- DNS label safety ensures Kubernetes accepts the names.

---

## Step 2: Use the digest in Kubernetes namespace and Helm release names

Kubernetes **namespace names** are DNS labels: lowercase letters, digits, hyphens; and they must fit length rules.

Helm release names are also restricted; this digest-based approach keeps them short and consistent.

### CI variables (example)

```bash
export GIT_COMMIT_SHA="a3f1c2d8e4b7f0c2a6e5f1f4d3c2b1a090877665"

DIGEST="$(python3 digest.py "$GIT_COMMIT_SHA")"
NAMESPACE="pr-${DIGEST}"
RELEASE="rev-${DIGEST}"
echo "NAMESPACE=$NAMESPACE"
echo "RELEASE=$RELEASE"
```

At this point:
- every run for the same commit produces the same `NAMESPACE` and `RELEASE`
- a teardown can safely delete the correct namespace

---

## Step 3: Deterministic Helm release values wiring

I used a minimal Helm values pattern: the chart reads the commit identity and uses it to label resources and set environment variables.

### `values.yaml` template (generated in CI)

```yaml
app:
  commitSha: "{{ GIT_COMMIT_SHA }}"
  identity: "{{ DIGEST }}"
service:
  name: "api"
image:
  tag: "{{ GIT_COMMIT_SHA }}"
```

In CI, I render these values (example using `envsubst` style; simplest for small teams):

```bash
cat > values.yaml <<EOF
app:
  commitSha: "${GIT_COMMIT_SHA}"
  identity: "${DIGEST}"
service:
  name: "api"
image:
  tag: "${GIT_COMMIT_SHA}"
EOF
```

### Helm command

```bash
helm upgrade --install "$RELEASE" ./charts/myapp \
  --namespace "$NAMESPACE" \
  --create-namespace \
  -f values.yaml
```

This is the part that usually causes drift when names vary; with digest-derived names, it becomes stable.

---

## Step 4: Idempotent “destroy” job (the missing piece)

Drift becomes especially painful at cleanup time. So I made teardown deterministic too.

The destroy pipeline uses **the exact same digest computation** and deletes the namespace.

```bash
export GIT_COMMIT_SHA="$GIT_COMMIT_SHA_FROM_PIPELINE"
DIGEST="$(python3 digest.py "$GIT_COMMIT_SHA")"
NAMESPACE="pr-${DIGEST}"

# Namespace deletion is idempotent-ish: it’s safe to run multiple times.
kubectl delete namespace "$NAMESPACE" --ignore-not-found=true
```

This guarantees:
- the destroy job doesn’t accidentally remove a different PR’s environment
- repeated runs don’t flap unpredictably

---

## Optional hardening I added: label everything with the identity

When debugging, I wanted “find all resources belonging to this review app” without depending on naming alone.

So I added consistent labels:
- `review.identity=<DIGEST>`
- `review.commit=<short-sha>`

In Helm templates, you’d apply these labels to Deployment/Service/Ingress/etc. Example snippet:

```yaml
metadata:
  labels:
    review.identity: "{{ .Values.app.identity }}"
    review.commit: "{{ .Values.app.commitSha | trunc 7 }}"
```

### Why labels matter
Helm and Kubernetes resources can be renamed or have different kinds, but labels give you a single filter for observability and cleanup tooling.

---

## Concrete example: what I observed in logs

Before the digest approach, two CI runs for the same PR could create different namespaces due to a timestamp component. That caused:

- Ingress pointed to a new Service, but old workloads still existed.
- Retrying deploy didn’t “replace”; it created.

After switching to `pr-<sha-digest>`:
- both CI runs reported the same namespace/release
- the Helm upgrade updated the existing Deployment instead of creating a sibling environment
- namespace deletion consistently removed exactly one review app universe

---

## FinOps and platform engineering angle (why this matters beyond correctness)

This identity scheme reduced waste in two ways:

1. **No orphan review environments**  
   Deterministic teardown dramatically lowered “forgotten namespaces” that keep pulling images and running background jobs.

2. **Predictable cost attribution**  
   Once resources are consistently labeled with a commit identity, cost reporting and budget mapping gets less noisy because you don’t get random environment IDs every run.

---

## Closing summary

I implemented deterministic ephemeral review apps by deriving Kubernetes namespace and Helm release names from a SHA-256 digest of the immutable Git commit SHA. This removed identity drift, made Helm upgrades replace the correct resources, and enabled idempotent cleanup by deleting the exact namespace tied to the commit digest—turning chaotic per-PR environments into stable, debuggable, cost-controlled infrastructure.

Deterministic Ephemeral Review Apps By Git Commit Digest In Kubernetes

I ran into a weird CI/CD failure that looked like an “auth problem,” but it wasn’t. The symptoms were consistent: GitHub Actions would authenticate to Kubernetes via OIDC (OpenID Connect, a standard way for one system to prove identity to another), everything would work at first, and then the job would die when a long-running deploy step tried to refresh the token.

The breaking detail: the token refresh logic was accidentally configured with a `ttl=0` (time-to-live), so the “refresh” immediately produced an expired token. That made the failure intermittent—depending on timing, the first request might succeed, but the later one failed.

Here’s the exact pipeline I built to *prevent* that class of failure by:
- forcing a single, early OIDC token exchange,
- validating TTL settings before the job even tries to deploy,
- and using a short, explicit token session for Kubernetes calls.

### The failure mode I observed (and how I reproduced it)

In my setup, the deploy step ran long enough that Kubernetes client calls needed a fresh token. When TTL was mis-set to `0`, the refresh returned an unusable token.

I reproduced it conceptually like this (not Kubernetes-specific, just the idea):

```python
import time

def fake_refresh(ttl_seconds: int):
    now = int(time.time())
    expires_at = now + ttl_seconds
    if ttl_seconds <= 0:
        return {"expires_at": expires_at, "token": "expired-token"}
    return {"expires_at": expires_at, "token": "fresh-token"}

for ttl in [0, 30]:
    result = fake_refresh(ttl)
    now = int(time.time())
    ok = result["expires_at"] > now
    print(f"ttl={ttl} expires_at={result['expires_at']} ok={ok} token={result['token']}")
```

When `ttl=0`, `expires_at` is not in the future, so the token is instantly expired. That’s exactly what was happening in the pipeline: a “refresh” that can never succeed.

### The fix: fail fast on invalid TTL and avoid refresh mid-deploy

Instead of letting refresh happen during the deployment (which makes debugging painful), I enforced two guardrails:

1. **Validate TTL config at the beginning of the job**  
2. **Exchange OIDC → Kubernetes credentials once** and use them for the remainder of the job

Below is a working GitHub Actions workflow that does that.

### Working GitHub Actions workflow (OIDC to Kubernetes with TTL validation)

This example assumes:
- You use GitHub’s OIDC federation with a Kubernetes cluster that trusts the GitHub identity.
- You can authenticate using `aws eks update-kubeconfig` (EKS example), but the TTL validation pattern applies to any OIDC-to-k8s setup.

Create `.github/workflows/deploy.yml`:

```yaml
name: Deploy with OIDC and TTL guard

on:
  workflow_dispatch:
  push:
    branches: ["main"]

permissions:
  id-token: write   # required for OIDC
  contents: read

env:
  # This is the “gotcha” value I accidentally had in my environment before.
  # It MUST be > 0 for any refresh-like logic to make sense.
  DEPLOY_TOKEN_TTL_SECONDS: "300"

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Validate TTL before any Kubernetes auth
        shell: bash
        run: |
          set -euo pipefail
          ttl="${DEPLOY_TOKEN_TTL_SECONDS}"

          if ! [[ "$ttl" =~ ^[0-9]+$ ]]; then
            echo "DEPLOY_TOKEN_TTL_SECONDS must be a non-negative integer. Got: $ttl"
            exit 1
          fi

          if [ "$ttl" -le 0 ]; then
            echo "ERROR: DEPLOY_TOKEN_TTL_SECONDS must be > 0. Got: $ttl"
            exit 1
          fi

          echo "TTL looks good: $ttl seconds"

      - name: Authenticate to cloud (example: AWS EKS)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-oidc-deploy-role
          aws-region: us-east-1

      - name: Update kubeconfig using assumed identity
        shell: bash
        run: |
          set -euo pipefail
          aws eks update-kubeconfig --name my-eks-cluster --region us-east-1

      - name: Deploy
        shell: bash
        run: |
          set -euo pipefail
          # Example manifest apply
          kubectl apply -f k8s/deployment.yaml
          kubectl rollout status deployment/my-app --timeout=120s
```

#### What each block does (and why it matters)

- `permissions: id-token: write`  
  Enables GitHub Actions OIDC token issuance. Without it, OIDC auth can’t happen.

- `Validate TTL before any Kubernetes auth`  
  This is the key safety belt. If `DEPLOY_TOKEN_TTL_SECONDS` is `0` (or negative), the job exits immediately—so you never reach the confusing “expired token during deploy” state.

- `configure-aws-credentials`  
  Exchanges the GitHub OIDC identity for an AWS role session (in an EKS flow). This is the “one-time exchange” concept.

- `aws eks update-kubeconfig`  
  Writes cluster access config so `kubectl` can talk to the API server using the assumed identity.

- `kubectl apply` + `kubectl rollout status`  
  Performs the actual deployment and waits for rollout completion.

### A tiny TTL unit test that saved me later

I also added a little script to make TTL validation consistent across repos. It’s simple, but it prevents copy/paste mistakes.

Create `scripts/validate-ttl.py`:

```python
import os
import sys

def validate_ttl(ttl_str: str) -> int:
    if not ttl_str.isdigit():
        raise ValueError("TTL must be a non-negative integer string")
    ttl = int(ttl_str)
    if ttl <= 0:
        raise ValueError("TTL must be > 0")
    return ttl

if __name__ == "__main__":
    ttl_str = os.environ.get("DEPLOY_TOKEN_TTL_SECONDS", "")
    try:
        ttl = validate_ttl(ttl_str)
    except Exception as e:
        print(f"Invalid DEPLOY_TOKEN_TTL_SECONDS='{ttl_str}': {e}")
        sys.exit(1)
    print(f"TTL OK: {ttl} seconds")
```

Then call it in the workflow instead of the bash check:

```yaml
      - name: Validate TTL before any Kubernetes auth (python)
        shell: bash
        run: |
          set -euo pipefail
          python3 scripts/validate-ttl.py
```

### What I learned building this

The biggest surprise was that “token refresh” bugs can hide inside configuration values like `ttl=0`. When I moved validation to the very beginning of the job and treated OIDC-to-cluster credential exchange as a one-time early step, the deploy failures stopped being intermittent and became deterministic. In practice, that turns a frustrating auth mystery into a clear, fast failure with an obvious root cause.

Infrastructure & Scale Posts