Building a Replayable Workflow Engine.

At first glance, the job seems straightforward: define some tasks, connect them, run them, and mark the workflow done. But that is not enough to make it a real workflow system.

I do not just want something that executes tasks.

I want something that can explain itself, recover from failure, replay history, and stay predictable even when execution becomes asynchronous and distributed.

A Few Definitions First

Asynchronous, at least in the sense I care about here, boils down to processes, events, or communication that occur independently. They do not all happen at the same time, and they do not need to rely on one shared timer.

Distributed systems? That is really just a collection of independent computers or nodes connected through a network, but functioning together as one cohesive system.

I pulled these definitions from a quick search, and that is part of the point. If anything here feels unclear, it is worth chasing the concept properly instead of letting the term stay fuzzy.

That line of thinking pushed me toward a few core ideas early:

• DAG execution
• event-sourced state
• replayability
• async workers

This is not a final spec. It is a design pass on what matters, what I would build first, and which tradeoffs I would accept.

Why I Want DAG Execution

The natural model for workflows is a Directed Acyclic Graph.

Wait, why can I just claim that it is the natural model? Well, a directed graph naturally represents the order in which tasks must execute. It also needs to be acyclic because cycles create infinite or undefined execution paths, which means completion stops making sense.

A linear workflow is really just a special case of a DAG. But a DAG gives me the vocabulary for dependency-aware execution. Some tasks must happen after others. Some can happen in parallel. Some should only run if their parents succeed.

A typical workflow might fetch data, transform it, call an external API, wait for a response, and then fan out into multiple follow-up tasks. If I model everything as a simple list, I either lose parallelism or I start inventing awkward abstractions later. Starting with DAG semantics feels cleaner.

That matters because most real systems are not truly sequential. A simple anecdote would be the media pipeline I built a while back for a client. It involved so many nodes that I honestly could not fit the entire model in my head at once. The immediate question I had was: how is something like n8n able to handle all that cleanly?

The tradeoff, of course, is complexity.

The moment I try to build something even remotely close to n8n, the scheduler becomes harder. State reconstruction becomes harder. Failure handling becomes harder because what should run next is no longer trivial. But I think that complexity is worth it because DAGs are the point where workflow systems stop being glorified job queues and start becoming orchestration engines.

That said, I still do not want to build the full DAG engine on day one. My minimum viable slice should stay linear first, because the hard part is not just graph traversal. The hard part is correctness.

Why Event-Sourced State Feels Right

The biggest design decision here is that I do not want execution state to be the source of truth.

I want events to be the source of truth.

That means instead of storing the workflow as currently in some latest state record, I append events like workflow started, task scheduled, task started, task completed, task failed, retry requested, and workflow completed. From there, the state is derived.

This makes sense to me for a few reasons.

First, it makes the system explainable. If something fails, I do not just see the latest row in a status table. I can inspect the history that led there.

Second, it makes replay possible. If I trust my event log, I can rebuild the workflow state at any point without re-running the tasks themselves. That is powerful. State stops feeling fragile and starts feeling reconstructable.

Third, event sourcing fits the mental model of workflows surprisingly well. A workflow execution is really a timeline. Treating it as an append-only stream feels more honest than pretending it is just a mutable record.

Of course, this comes with tradeoffs too.

Event sourcing increases implementation difficulty. You need a rebuilder. You need to think carefully about event schemas. You need deterministic state transitions. You need to decide what is truly an event and what is just derived metadata.

It also means debugging the system requires more discipline. If I am careless with event design, I can create a log that is technically complete but practically useless.

Still, I think this is the right choice because replayability is not a nice-to-have here. It is one of the reasons to build the system at all.

Replayability Is More Than a Cool Feature

Replay sounds flashy, but I think it is actually a core systems property.

If I can take the same initial input plus the same ordered event log and always reconstruct the same execution state, then I get a much stronger guarantee about the system. I know the workflow engine is behaving predictably.

That determinism matters.

I do not expect task execution itself to be deterministic. External HTTP calls are not deterministic. LLM calls are definitely not deterministic. Any interaction with outside systems should be treated as a black box that emits outcomes.

What I do want to make deterministic is everything around that:

• event ordering
• state transitions
• dependency resolution
• retry decisions
• final workflow status

In other words, I am not trying to make the world deterministic. I am trying to make the orchestration layer deterministic.

That distinction feels important. If I blur it, I risk designing something unrealistic. External effects will always be messy. The engine's job is to record those effects faithfully and react to them predictably.

Why Async Workers Are Necessary

The moment workflows include HTTP calls, delays, external systems, or heavy internal tasks, synchronous execution stops making sense.

I do not want the API process to be responsible for actually doing the work. I want it to accept an execution request, persist the right events, and hand off actual task execution to workers.

That immediately suggests a few components:

• a scheduler that figures out which nodes are runnable
• a queue that distributes work
• workers that execute tasks
• an event stream that records the results
• a state rebuilder that can reconstruct current status from the log

I like this separation because it keeps responsibilities clear.

The scheduler decides what is eligible to run. The executor runs the task. The event log records what happened. The rebuilder tells me what the workflow currently looks like.

This is cleaner than having one giant service try to do everything.

The tradeoff is operational complexity. Once workers exist, I have to think about duplicate execution, crashes, idempotency, retries, and partial progress. But those are not optional concerns anyway. Async workers do not create those problems. They simply force me to face them honestly.

Core Entities I Think Matter

The minimal model in my head has four main entities.

WorkflowDefinition holds the durable design of a workflow:

• id
• version
• DAG
• retry policy
• created_at

This lets the workflow itself evolve over time without breaking historical runs. Versioning matters because once workflows start doing real work, I do not want the current definition to retroactively redefine the past.

WorkflowInstance represents one run of one version of one workflow:

• id
• workflow_definition_id
• version
• status
• start_time
• end_time

This is the high-level execution record.

TaskExecution captures node-level activity:

• node_id
• status
• input
• output
• retry_count

I may not store this as a primary mutable table if event sourcing is strict, but conceptually I still need this shape.

EventLog is the actual append-only source of truth.

That is the most important part of the model. Everything else is either definition or projection.

Node Types: Starting Small on Purpose

I do not think the engine needs a giant plugin ecosystem on day one.

The first three node types are enough:

• Function nodes
• HTTP nodes
• Delay nodes

Function nodes let me run internally registered tasks. HTTP nodes let me interact with external systems. Delay nodes let workflows wait.

That combination is already enough to express a lot.

It is tempting to add conditionals, branching policies, webhooks, human approval steps, and LLM nodes immediately. But I think that is the wrong instinct. If the execution semantics are not solid yet, more node types just multiply failure modes.

I would rather build a small set of nodes with strong guarantees than a wide set of features on a weak core.

Scheduling, Retries, and Failure

Once I accepted that this system would be async and event-sourced, failure handling became one of the most important parts of the design.

Retries need to be policy-based, not improvised. Backoff needs to be explicit. Duplicate task execution cannot break correctness. Worker crashes have to be survivable.

That naturally led me to a few control operations I think are essential:

• cancel workflow
• retry failed workflow or node
• replay workflow

These are not just convenience APIs. They are operational tools. If something goes wrong, the system should let me intervene without corrupting history.

I am still undecided on how far to go with partial failure handling in the first version. There is a spectrum here.

On one end, I can keep the model strict: if a required node fails, the workflow fails unless retry succeeds. On the other end, I can support more nuanced semantics around partial success, compensations, or optional branches.

My current instinct is to stay strict early. Simpler failure semantics are easier to reason about, and reasoning matters more than feature breadth at this stage.

Observability Is Not Optional

A workflow engine without observability becomes miserable very quickly.

If it is going to coordinate multiple tasks across multiple workers, I need at least:

• workflow status
• per-node execution logs
• event inspection
• execution history

This is another reason I like event sourcing. The observability story is not something bolted on later. It is already part of how the system thinks.

I do not just want to know whether a workflow failed. I want to know which node failed, what input it saw, how many retries happened, what events led up to the failure, and whether the scheduler made the right decisions.

That level of visibility is what makes systems maintainable.

API Design: Practical, Not Clever

The API shape I am leaning toward is intentionally boring.

Definitions:

• POST /workflows
• GET /workflows
• GET /workflows/{workflow_id}
• POST /workflows/{workflow_id}/versions
• GET /workflows/{workflow_id}/versions

Executions:

• POST /executions
• GET /executions/{execution_id}
• GET /executions

Task visibility:

• GET /executions/{execution_id}/tasks
• GET /executions/{execution_id}/tasks/{node_id}

Control:

• POST /executions/{execution_id}/cancel
• POST /executions/{execution_id}/retry
• POST /executions/{execution_id}/replay

Scheduling:

• POST /schedules
• GET /schedules
• DELETE /schedules/{schedule_id}

Events:

• GET /executions/{execution_id}/events

There is nothing fancy here, and that is intentional. I do not think infrastructure APIs benefit from being clever. They benefit from being obvious.

What I Would Build First

The minimum viable slice is intentionally smaller than the vision.

I would start with:

• linear workflows
• function and HTTP nodes
• append-only event logging
• basic retry support
• one queue
• one worker model

Then I would extend into:

• DAG branching
• delayed and cron scheduling
• replay controls
• versioning improvements

The reason is simple: the first milestone is not support every workflow shape. The first milestone is prove that execution, logging, and reconstruction are correct.

If those are not correct, adding branching just creates a more sophisticated failure machine.

Final Thought

What keeps pulling me toward this project is that it sits at the intersection of systems design and product clarity.

A workflow engine is not just about running tasks. It is about making execution understandable.

I want a system where history is visible, failures are recoverable, retries are intentional, and replay is possible because the architecture was built for it from the start.

That is the standard I want to hold this project to.

Not just execution.

Execution you can trust.