Metaflow is unclaimed —
M
Metaflow
A framework for real-life ML, AI, and data science·metaflow.org
Metaflow is an open-source framework that helps build and manage real-life machine learning, AI, and data science projects. It provides tools for modeling, versioning, orchestration, and scaling compute from a laptop to the cloud, and for deploying workflows to production. It was originally developed at Netflix and open-sourced in 2019, and is now used by hundreds of companies.
What it's for
Building and managing ML/AI and data science projectsTracking and versioning experiment variables automaticallyDeploying workflows to production with a single commandScaling compute to the cloud using GPUs, multiple cores, and multiple instancesRunning distributed training over many GPU instancesProcessing large datasets or running hyperparameter searches in parallel via foreachAccessing data from data warehousesReacting to updating data and other events automaticallyCheckpointing long-running model training tasksComposing flows with reusable custom decorators
Features 20
- @resources decorator for on-demand cloud compute
- Automatic variable tracking and versioning for experiments
- Branching for parallel step execution
- Checkpointing progress with @checkpoint decorator
- Configurable flows via Config object
- Custom, reusable decorators for composing flows
- Data access from data warehouses with versioning across steps
- Dependency management with uv
- Distributed computing with ephemeral compute clusters
- Event-based flow triggering
- foreach for data-parallel task execution at scale
- Library dependency management locally and in the cloud
- Programmatic run and deploy APIs
- Real-time, dynamic Cards for observability
- Scalable compute with GPUs, multiple cores, and large memory
- Single-command deployment to production
- spin command for incremental flow development
- Spot instance support with graceful termination handling
- Support for recursive and conditional steps
- Workflow orchestration in plain Python
At a glance
free tieropen sourceself-hostableAPI
TypeLibrary / SDK
DeploymentHybrid
ForML/AI engineers, data scientists
CompanyNetflix, Inc.
Integrations
AWS BatchAWS Step FunctionsAmazon EKSAmazon S3Azure AKSAzure Blob StorageGoogle Kubernetes EngineGoogle Cloud StorageKubernetesPyTorchXGBoostuvKubeflowAWS Trainium
Resources