Step Functions With Sqs Idempotency And Dynamodb Ttl For At Most Once Event Processing

I built a small serverless “ingestion” pipeline that looked simple on paper: SQS messages fan out into a Step Functions workflow, the workflow calls downstream APIs, and everything is fully managed. In practice, the hardest part wasn’t the happy path—it was making the workflow behave predictably when SQS delivers duplicates.

This post is about a very specific pattern I ended up using: a Step Functions workflow that processes SQS messages with idempotency enforced by DynamoDB + TTL, so the system achieves “at most once side effects” even when SQS redelivers the same message.

---

The failure mode I hit (duplicate SQS deliveries)

SQS provides at-least-once delivery, meaning a message can be delivered more than once. That’s not a bug—network retries and visibility timeouts can cause it. When my Step Function started doing side effects (writing to an external system), duplicates caused:

• duplicated records in the downstream system
• wasted cost calling APIs multiple times
• messy “cleanup” logic after the fact

The key constraint: I needed to dedupe based on a message identity, and I needed dedupe entries to disappear automatically to keep the database small.

---

The niche pattern: DynamoDB idempotency keys with TTL guarded inside the workflow

Core idea

1. Each SQS message has a unique identifier (I used the SQS MessageId, which SQS provides).
2. Before executing any side effect, the workflow tries to “claim” that identifier in DynamoDB using a conditional write.
3. If the claim already exists, the workflow skips the side effects.
4. The DynamoDB item has a TTL (time-to-live) attribute so it auto-expires after a retention window.

TTL is a DynamoDB feature that periodically deletes items after a timestamp. It keeps the idempotency table from growing forever.

---

Architecture (minimal but production-real)

Components

• SQS: buffers events and retries automatically
• Lambda (Idempotency Guard): performs the DynamoDB conditional write
• Step Functions: orchestrates the workflow and side effects
• DynamoDB: stores idempotency keys with TTL

Flow

1. SQS triggers (or is polled by) Step Functions (in my implementation, I used an SQS-to-Lambda trigger to start the state machine; the exact wiring doesn’t matter as long as MessageId reaches the workflow).
2. Workflow calls the idempotency guard Lambda.
3. If the key is new, the guard returns “proceed”; otherwise it returns “skip”.
4. Only the “proceed” branch calls downstream APIs.

---

DynamoDB table: idempotency key with TTL

I created a table like this:

• Partition key: pk (string)
• Sort key: sk (string) — optional but useful for namespacing; I set it to a constant
• TTL attribute: expiresAt (number)

In this example, the idempotency “key” is:

• pk = "sqs:" + MessageId
• sk = "ingestion" (namespacing the use-case)
• expiresAt = currentTimeInSeconds + retentionSeconds

Infrastructure (AWS CLI)

aws dynamodb create-table \
  --table-name IdempotencyKeys \
  --attribute-definitions \
    AttributeName=pk,AttributeType=S \
    AttributeName=sk,AttributeType=S \
  --key-schema \
    AttributeName=pk,KeyType=HASH \
    AttributeName=sk,KeyType=RANGE \
  --billing-mode PAY_PER_REQUEST

Then enable TTL:

aws dynamodb update-time-to-live \
  --table-name IdempotencyKeys \
  --time-to-live-specification "Enabled=true, AttributeName=expiresAt"

---

Idempotency Guard Lambda (conditional write)

This Lambda attempts to write an item only if it doesn’t already exist.

Important behavior:

• DynamoDB’s ConditionExpression ensures atomicity.
• If the item already exists, the Lambda returns { "proceed": false }.
• If the item is new, it returns { "proceed": true }.

index.py (Python 3.12)

import os
import time
import json

import boto3
from botocore.exceptions import ClientError

dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table(os.environ["IDEMPOTENCY_TABLE"])

def handler(event, context):
    """
    Expected input from Step Functions:
    {
      "messageId": "...",
      "namespace": "ingestion",
      "retentionSeconds": 86400
    }
    """
    message_id = event["messageId"]
    namespace = event.get("namespace", "ingestion")
    retention_seconds = int(event.get("retentionSeconds", 86400))

    pk = f"sqs:{message_id}"
    sk = namespace
    now = int(time.time())
    expires_at = now + retention_seconds

    try:
        # Claim the idempotency key. This succeeds only if the item doesn't exist yet.
        table.put_item(
            Item={
                "pk": pk,
                "sk": sk,
                "createdAt": now,
                "expiresAt": expires_at
            },
            ConditionExpression="attribute_not_exists(pk) AND attribute_not_exists(sk)"
        )
        return {
            "proceed": True,
            "reason": "claimed",
            "idempotencyKey": {"pk": pk, "sk": sk}
        }

    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            # Duplicate delivery: the claim already exists.
            return {
                "proceed": False,
                "reason": "duplicate",
                "idempotencyKey": {"pk": pk, "sk": sk}
            }
        raise

Why the conditional write matters

Without ConditionExpression, both duplicate deliveries would “win” and you’d run the side effects twice. With it, the first workflow run claims the key; subsequent duplicates fail the conditional check and are safely skipped.

---

Step Functions state machine (guard + conditional execution)

I used Step Functions to keep the orchestration explicit.

The workflow does:

1. Call the idempotency guard Lambda.
2. If proceed is true, run the side-effect task.
3. Otherwise, end successfully.

State machine definition (Amazon States Language)

{
  "Comment": "SQS dedupe with DynamoDB TTL guarded side effects",
  "StartAt": "IdempotencyGuard",
  "States": {
    "IdempotencyGuard": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:REGION:ACCOUNT_ID:function:IdempotencyGuard",
        "Payload": {
          "messageId.$": "$.messageId",
          "namespace": "ingestion",
          "retentionSeconds": 86400
        }
      },
      "ResultSelector": {
        "proceed.$": "$.Payload.proceed",
        "reason.$": "$.Payload.reason"
      },
      "ResultPath": "$.guard",
      "Next": "ShouldProceed"
    },
    "ShouldProceed": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.guard.proceed",
          "BooleanEquals": true,
          "Next": "SideEffect"
        }
      ],
      "Default": "SkipDuplicate"
    },
    "SideEffect": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:REGION:ACCOUNT_ID:function:DoSideEffects",
        "Payload": {
          "messageId.$": "$.messageId"
        }
      },
      "End": true
    },
    "SkipDuplicate": {
      "Type": "Pass",
      "Result": {
        "skipped": true,
        "reason.$": "$.guard.reason"
      },
      "End": true
    }
  }
}

What’s happening step-by-step

• IdempotencyGuard: invokes the guard Lambda, passing the incoming messageId.
• ResultSelector: extracts only what we need (proceed, reason) from the Lambda output.
• Choice state: branches based on guard.proceed.
• SideEffect: only runs when the key is new.
• SkipDuplicate: ends the workflow cleanly when it’s a duplicate.

This keeps cost and side effects controlled even under replay storms.

---

Local mental model test (simulate duplicate deliveries)

I tested this pattern by manually invoking the guard logic twice with the same messageId. The first time it claims; the second time it returns proceed: false.

Example input to the workflow / guard

{
  "messageId": "b6a9d4c7-1f8d-4d7b-8d1a-3d5d2a9b1234"
}

• Run 1:
  • DynamoDB item doesn’t exist
  • guard returns proceed: true
  • side effects execute
• Run 2:
  • DynamoDB item exists
  • guard returns proceed: false
  • side effects are skipped

TTL then cleans up the claim entry after the retention window.

---

Operational notes I learned the hard way

1) TTL is not instant

DynamoDB TTL deletes are asynchronous. Items don’t disappear exactly at the timestamp—eventual cleanup is normal. That’s fine because idempotency only needs a practical time window.

2) Keep the retention window aligned with duplicates

If your upstream system or SQS backlog can cause duplicates over a period longer than your TTL window, you’ll reprocess. I set it based on:

• maximum plausible retry/redelivery window
• visibility timeout and consumer throughput
• downstream reconciliation tolerance

3) Dedup based on the right identifier

I used SQS MessageId because it’s the identifier available at ingestion time. If your pipeline is receiving events that already have business IDs (like an event UUID), I’d base the idempotency key on that instead. The goal is dedupe on “same logical event,” not “same queue delivery.”

---

Conclusion

I implemented a serverless ingestion workflow where Step Functions could safely handle SQS duplicates by adding a DynamoDB-backed idempotency guard with TTL. The pattern is small—conditional write before side effects—but it changes everything: duplicate deliveries no longer cause duplicate outcomes, the idempotency ledger stays bounded via TTL, and the workflow remains deterministic even under real-world retry behavior.