Metaflow

Metaflow is a Python framework for building and managing real-life AI and ML systems, originally developed at Netflix and now supported by Outerbounds. It lets data scientists write workflows as plain Python — @step decorators define a DAG — and run the same code locally on a laptop or scaled out on AWS with S3-backed versioning of code, data, and artifacts.

At Netflix alone Metaflow supports 3,000+ AI/ML projects and executes hundreds of millions of compute jobs, and adopters include Amazon, DoorDash, Dyson, and Goldman Sachs. Its pitch is human-centricity: excellent local development, automatic experiment tracking without instrumentation, and Kubernetes/GPU scaling via decorators rather than YAML.

Limitations: the AWS integration is the most mature — other clouds are second-class; DAG topology is less flexible than Kubeflow or Flyte for exotic workflow shapes; and teams wanting a fully managed platform may prefer SageMaker or Vertex over operating Metaflow themselves.

pip install metaflow; write a FlowSpec with @step methods and run python flow.py run locally, then scale with --with kubernetes or AWS Batch.