Serverless Architecture For Dynamodb Ttl Tombstone Drift In Step Functions

I ran into a weird “ghost data” issue while building a serverless workflow on AWS: items in DynamoDB were supposed to disappear automatically using TTL (Time to Live), but my Step Functions workflow kept seeing “expired” items and occasionally got stuck in a retry loop. The root cause wasn’t the TTL configuration itself—it was tombstone drift: the moment you mark something expired isn’t the moment DynamoDB actually removes it, and that gap can break correctness in workflows that assume instant deletion.

This post is what I built to make the workflow deterministic, with working code and a repeatable pattern.

The failure mode I observed

Here’s the mental model I started with:

• DynamoDB TTL marks items as expired.
• DynamoDB later removes them.
• My Step Functions state machine queries DynamoDB to decide what to do next.
• If an item is expired, the workflow should stop.

What actually happened:

1. An item’s TTL timestamp is in the past (it’s “expired” from the TTL perspective).
2. DynamoDB still returns the item for a while because the physical deletion happens asynchronously.
3. My state machine interpreted the returned item as “active,” so it tried to process it again.
4. That created retries and inconsistent outcomes.

DynamoDB TTL is “best effort” for physical deletion timing. So any workflow logic that treats “TTL expired” as “item no longer exists” can become nondeterministic.

The fix pattern: Treat TTL as a logical state, not physical deletion

Instead of relying on the item disappearing, I made the state machine evaluate TTL at read time using the TTL attribute and the current timestamp.

Key idea

• DynamoDB stores a TTL attribute (e.g. expiresAt as a Unix epoch number).
• When the workflow reads an item, it computes:
  • expired = expiresAt <= now
• The workflow branches based on expired rather than on whether the item still exists.

That makes the workflow deterministic even during tombstone drift.

Infrastructure: Terraform for DynamoDB + Step Functions

Below is a minimal-but-complete setup using Terraform.

terraform/main.tf

terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.0.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

variable "aws_region" {
  type    = string
  default = "us-east-1"
}

resource "aws_dynamodb_table" "events" {
  name         = "ttl-events"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "pk"

  attribute {
    name = "pk"
    type = "S"
  }

  # TTL attribute: Unix epoch seconds
  ttl {
    attribute_name = "expiresAt"
    enabled        = true
  }
}

data "aws_iam_policy_document" "sf_assume" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["states.${var.aws_region}.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "sf_role" {
  name               = "sf-ttl-drift-role"
  assume_role_policy = data.aws_iam_policy_document.sf_assume.json
}

# Allow state machine to read items from DynamoDB
resource "aws_iam_policy" "sf_dynamo_read" {
  name = "sf-ttl-drift-dynamo-read"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow",
        Action = [
          "dynamodb:GetItem"
        ],
        Resource = aws_dynamodb_table.events.arn
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "sf_attach" {
  role       = aws_iam_role.sf_role.name
  policy_arn = aws_iam_policy.sf_dynamo_read.arn
}

# Lambda-less workflow: use Step Functions SDK integration with DynamoDB directly.
# (Step Functions supports DynamoDB service integrations)
resource "aws_sfn_state_machine" "sm" {
  name     = "ttl-drift-deterministic"
  role_arn = aws_iam_role.sf_role.arn

  definition = jsonencode({
    Comment = "Deterministic TTL handling to avoid tombstone drift"
    StartAt = "Fetch item"
    States = {
      "Fetch item" : {
        "Type" : "Task",
        "Resource" : "arn:aws:states:::aws-sdk:dynamodb:getItem",
        "Parameters" : {
          "TableName" : aws_dynamodb_table.events.name,
          "Key" : {
            "pk" : { "S.$" : "$.pk" }
          },
          "ConsistentRead" : true
        },
        "ResultPath" : "$.item",
        "Next" : "Item not found?"
      },

      "Item not found?" : {
        "Type" : "Choice",
        "Choices" : [
          {
            "Variable" : "$.item.Item",
            "IsPresent" : false,
            "Next" : "Gone"
          }
        ],
        "Default" : "Evaluate TTL"
      },

      "Evaluate TTL" : {
        "Type" : "Pass",
        "Parameters" : {
          "pk.$" : "$.pk",
          "expiresAt.$" : "$.item.Item.expiresAt.N",
          "now.$" : "$$.State.EnteredTime"
        },
        "ResultPath" : "$.ctx",
        "Next" : "Expired?"
      },

      "Expired?" : {
        "Type" : "Choice",
        "Choices" : [
          {
            "And" : [
              {
                "Variable" : "$.ctx.expiresAt",
                "IsNull" : false
              },
              {
                "Variable" : "$.ctx.expiresAt",
                "NumericLessThanEqualsPath" : "$.ctx.nowUnix"
              }
            ],
            "Next" : "Expired"
          }
        ],
        "Default" : "Active"
      },

      "Expired" : {
        "Type" : "Succeed",
        "Result" : { "status" : "expired", "reason" : "TTL expired logically (not physical deletion)" }
      },

      "Active" : {
        "Type" : "Succeed",
        "Result" : { "status" : "active", "reason" : "TTL not yet expired" }
      },

      "Gone" : {
        "Type" : "Succeed",
        "Result" : { "status" : "not_found", "reason" : "Item no longer exists physically" }
      }
    }
  })

  depends_on = [aws_iam_role_policy_attachment.sf_attach]
}

That definition has one snag: I need Unix time for nowUnix, because DynamoDB TTL is stored as Unix epoch seconds. Step Functions doesn’t directly expose Unix seconds in the built-in context fields. So I adjusted it by computing Unix time in a small Lambda. This keeps the workflow correct and still serverless.

Code: tiny Lambda to convert ISO time to Unix seconds

lambda/nowUnix/app.py

import json
import time

def handler(event, context):
    # We'll accept state machine input; use the current time in seconds since epoch.
    now_unix = int(time.time())
    return {
        "nowUnix": now_unix,
        "input": event
    }

lambda/nowUnix/Dockerfile (optional) + package-less Lambda

For simplicity, I’ll use a zip deploy approach in the Terraform below.

terraform/lambda.tf (adds Lambda + permissions + state machine update)

Update the state machine definition to call the Lambda before the TTL comparison.

resource "aws_iam_role" "lambda_role" {
  name = "ttl-drift-nowunix-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [{
      Effect = "Allow",
      Principal = { Service = "lambda.amazonaws.com" }
      Action = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy" "lambda_logs" {
  name = "ttl-drift-nowunix-logs"
  role = aws_iam_role.lambda_role.id
  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [{
      Effect = "Allow",
      Action = [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ]
      Resource = "*"
    }]
  })
}

data "archive_file" "lambda_zip" {
  type        = "zip"
  source_dir  = "${path.module}/../lambda/nowUnix"
  output_path = "${path.module}/.build/nowunix.zip"
}

resource "aws_lambda_function" "nowunix" {
  function_name = "ttl-drift-nowunix"
  role          = aws_iam_role.lambda_role.arn
  handler       = "app.handler"
  runtime       = "python3.12"
  filename      = data.archive_file.lambda_zip.output_path
  source_code_hash = data.archive_file.lambda_zip.output_base64sha256
}

# Allow Step Functions to invoke Lambda
resource "aws_lambda_permission" "allow_sfn" {
  statement_id  = "AllowExecutionFromStepFunctions"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.nowunix.function_name
  principal     = "states.amazonaws.com"
}

# Replace just the definition portion: call Lambda for nowUnix before Choice.
# (For brevity, I’m not rewriting the entire state machine block here.)

Now update the state machine definition in aws_sfn_state_machine:

• Add a Get nowUnix task before evaluating TTL.
• Use nowUnix in the comparison.

Here’s the revised core definition snippet (drop it into the definition = jsonencode({ ... })):

{
  "Comment": "Deterministic TTL handling to avoid tombstone drift",
  "StartAt": "Fetch item",
  "States": {
    "Fetch item": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:dynamodb:getItem",
      "Parameters": {
        "TableName": "ttl-events",
        "Key": { "pk": { "S.$": "$.pk" } },
        "ConsistentRead": true
      },
      "ResultPath": "$.item",
      "Next": "Item not found?"
    },
    "Item not found?": {
      "Type": "Choice",
      "Choices": [
        { "Variable": "$.item.Item", "IsPresent": false, "Next": "Gone" }
      ],
      "Default": "Get nowUnix"
    },
    "Get nowUnix": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "${aws_lambda_function.nowunix.function_name}",
        "Payload.$": "$"
      },
      "ResultSelector": {
        "nowUnix.$": "$.Payload.nowUnix"
      },
      "ResultPath": "$.time",
      "Next": "Evaluate TTL"
    },
    "Evaluate TTL": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.item.Item.expiresAt.N",
          "NumericLessThanEqualsPath": "$.time.nowUnix",
          "Next": "Expired"
        }
      ],
      "Default": "Active"
    },
    "Expired": { "Type": "Succeed", "Result": { "status": "expired" } },
    "Active": { "Type": "Succeed", "Result": { "status": "active" } },
    "Gone": { "Type": "Succeed", "Result": { "status": "not_found" } }
  }
}

The important part is the TTL comparison uses:

• expiresAt from the item (.item.Item.expiresAt.N)
• nowUnix from Lambda (.time.nowUnix)

This makes the workflow deterministic regardless of when DynamoDB physically removes expired items.

Repro: insert an “already expired” item

Once Terraform applies, I inserted an item whose TTL is already in the past.

Put an item with TTL in DynamoDB

Here’s a simple Python script using boto3.

scripts/put_item.py

import time
import boto3

table_name = "ttl-events"
pk = "user-123"

dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table(table_name)

now = int(time.time())
expires_at = now - 30  # 30 seconds ago

table.put_item(
    Item={
        "pk": pk,
        "expiresAt": str(expires_at),  # DynamoDB TTL expects a number attribute; SDK maps strings as N
        "payload": "example"
    }
)

print("Inserted:", pk, "expiresAt:", expires_at)

Run:

python scripts/put_item.py

Start the Step Functions execution

scripts/run_sm.py

import boto3
import os

sm_arn = os.environ["SM_ARN"]
pk = "user-123"

sf = boto3.client("stepfunctions")

resp = sf.start_execution(
    stateMachineArn=sm_arn,
    name=f"test-{pk}",
    input=f'{{"pk":"{pk}"}}'
)

print("Started execution:", resp["executionArn"])

Run:

export SM_ARN="arn:aws:states:us-east-1:123456789012:stateMachine:ttl-drift-deterministic"
python scripts/run_sm.py

Then check the execution output (via console or describe_execution). The workflow should return:

• status: "expired"

Even if DynamoDB hasn’t physically removed the item yet—because the workflow logic doesn’t rely on physical deletion.

Why this works (and what I learned)

TTL is about expiration semantics, not immediate deletion guarantees. DynamoDB will eventually clean up, but there’s no SLA that says it will vanish before the next read.

By treating TTL as a logical condition—expiresAt <= now—I removed the timing dependency from the workflow. That eliminated the tombstone drift effect that was causing my retries and inconsistent state transitions.

Conclusion

I built a deterministic serverless Step Functions workflow for DynamoDB TTL tombstone drift by evaluating TTL at read time (expiresAt <= now) instead of assuming expired items are instantly gone. The result was a state machine that behaves consistently across the asynchronous gap between TTL marking and physical deletion, keeping infrastructure reliable in mission-critical platform workflows.