Workflow orchestration: the part that remembers where you are
Several steps, in an order, with retries, waiting and branches. An orchestrator holds that shape outside your code, so a job that takes an hour survives a function that takes thirty seconds.
What is workflow orchestration?
As soon as a job has more than one step, something has to remember which step it is on, what to do when step three fails, and how long to wait for a human to approve step four. You can write that yourself, and people do, until the first time the process dies halfway and nobody can say what already happened.
An orchestrator takes that state out of your code and puts it somewhere durable. The steps stay small and stateless, the shape of the process lives in one place you can draw, and every run leaves a record of what happened and where it stopped. For an AI pipeline this matters twice over, because the slow expensive steps are the ones you do not want to repeat after a crash.
Four ways to run it
Two shapes exist here: a managed state machine you describe declaratively, and a code-first engine you run yourself. They solve the same problem from opposite ends.
AWS Step Functions[1]
The state machine is a document, not code. It calls Lambda and most other AWS services directly, waits for up to a year, and shows each run as a picture you can point at in a review.
Official documentation →Azure Logic Apps[2]
Built around a large catalogue of ready-made connectors to other systems, which is why it tends to win where the process crosses into Microsoft 365, SAP or a line of business application.
Official documentation →Cloud Workflows[3]
A light YAML-defined orchestrator that chains Google Cloud services and any HTTP endpoint. Less machinery than the other two, which is the point of it.
Official documentation →Temporal or Apache Airflow[4]
Temporal keeps the process in ordinary code and replays it after a failure, so developers never leave their language. Airflow is the older scheduler that data teams already run. Both can be self-hosted or bought as a service.
Project documentation →Where I used it
The video pipeline
Step Functions holds the order of the whole thing: cut the frames, label them, let the agents choose, render, deliver. When one clip fails, that run stops where it failed and the others carry on.
Read the write-up →Managed and self-hosted together: hybrid
The orchestrator sees the shape of your process and the metadata of every run, which is more than people expect. A common split is to keep the orchestration in the cloud and have it call a worker inside your network for the steps that touch regulated data, so the orchestrator knows that a step ran and not what it read.
What would help you next?
Learn to decide
The course “From Managed to Hybrid” teaches where AI should run: what each layer costs, what you operate and where your data has to stay. It is in preparation, with a free live session first.
Linda MohamedAI and cloud consultant, AWS Community Hero, Vienna
Build it with me
Tell me what you want to build. In 30 minutes we check which of these ways fits your data, your rules and your budget, and whether AWS funding for a proof of concept can cover part of it. How AWS funding works →
You already know what you need
Book the workshop directly:
- AI Concept and Prototype workshops →
Architecture first, priced per service and tested on your data, then a prototype with a go or stop decision. - AI Discovery Workshop →
Sort your ideas into a scored shortlist, so you know which use case to build first.
Read more on ai-solutions.wiki
My open knowledge base, with longer comparisons, pricing notes and sources.
Sources
- AWS Step Functions documentation. https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html
- Azure Logic Apps documentation. https://learn.microsoft.com/en-us/azure/logic-apps/logic-apps-overview
- Google Cloud Workflows documentation. https://cloud.google.com/workflows/docs/overview
- Temporal documentation. https://docs.temporal.io/
- AWS Step Functions, ai-solutions.wiki. https://ai-solutions.wiki/tools/aws-step-functions/
- Azure Logic Apps, ai-solutions.wiki. https://ai-solutions.wiki/tools/azure-logic-apps/
- Temporal, ai-solutions.wiki. https://ai-solutions.wiki/tools/temporal/
- Apache Airflow, ai-solutions.wiki. https://ai-solutions.wiki/tools/apache-airflow/
- Airflow vs Step Functions, ai-solutions.wiki. https://ai-solutions.wiki/comparisons/airflow-vs-step-functions/
Links checked October 2026. Product names and features change: check the linked pages before you decide.
