Kubeflow for MLOps: A Practical Crash Course

In the Feast guide you deployed a feature store for the employee attrition project. In the Airflow + DVC guide you automated dataset versioning with Airflow on Kubernetes. In an enterprise environment, you need an orchestration platform that automates the entire data preparation, model training, tuning, and retraining flow whenever new data becomes available. That's where Kubeflow comes in.

When the project started in 2017, Kubeflow was primarily built as an MLOps platform for Kubernetes to manage traditional ML pipelines. It has since grown into a cloud-native AI platform for GenAI, LLMOps, agentic AI, distributed AI training, and fine-tuning foundation models. This guide focuses on the orchestration layer — Kubeflow Pipelines — and how it compares to Airflow.

The Kubeflow Stack

The Kubeflow ecosystem has nine subprojects, each solving a specific use case in the AI/ML lifecycle. The subprojects can run independently or as part of the full stack — you can adopt only Pipelines, only Trainer, or only Notebooks instead of installing the whole platform.

The Feast feature store from an earlier edition, and KServe covered previously, are both part of the Kubeflow ecosystem — so you have already been using Kubeflow subprojects without realizing it. In a typical MLOps capstone project, Kubeflow Pipelines handles data preparation and model training orchestration, and for the training step, the pipeline delegates to Kubeflow Trainer.

What is Kubeflow Pipelines (KFP)?

Kubeflow Pipelines is a standalone DAG runtime that lets you describe your entire ML workflow in Python. The interesting part: KFP uses Argo Workflows as its execution engine.

Argo Workflows is a Kubernetes-native DAG orchestration engine, but you write the DAG declaratively as a YAML CRD rather than as Python code, and Argo executes each task as a Kubernetes pod — the same model as Airflow's KubernetesPodOperator. Kubeflow uses Argo as the backend and adds an AI/ML layer on top of it, so instead of just orchestrating containers, it understands datasets, model artifacts, experiment tracking, metrics, caching, and reproducible ML pipelines.

When you write a pipeline, you write Python DAGs — the same mental model as Airflow. Kubeflow then compiles your DAG into an Argo Workflow and submits it to Kubernetes for execution.

Kubeflow Pipelines Architecture

Before working with pipelines, it helps to know the components that make up KFP:

Three more pod types get deployed dynamically for every pipeline run:

• system-dag-driver-* — created once per run; records run details in MLMD
• system-container-driver-* — runner pod that gathers input data and hands it to the implementation pod
• system-container-impl-* — runner pod that executes your actual component code

Pipeline Structure

The employee attrition data preparation pipeline has multiple steps — ingestion, validation, feature engineering, preprocessing — mostly sequential, though real-world pipelines often run some tasks in parallel or repeat them.

A KFP pipeline uses the @dsl.pipeline decorator to define the workflow. Inside the pipeline function, you call each component (task) and connect them via inputs and outputs. You never explicitly write a DAG — Kubeflow builds it automatically from task dependencies. Whenever one task consumes another task's output, a dependency edge is created.

How Kubeflow Pipelines Work

A Kubeflow Pipeline executes as a Directed Acyclic Graph. Each task depends on the outputs of one or more previous tasks; if two tasks have no dependency between them, Kubeflow runs them in parallel. Every pipeline task runs in its own Kubernetes pod — instead of running everything in a single process like a plain Python script, Kubeflow schedules each step as an independent containerized workload.

Pipeline Execution Flow

1. The KFP SDK compiles your Python pipeline definition into an Intermediate Representation (IR) YAML describing all tasks, inputs, outputs, and dependencies.
2. The compiled IR YAML is submitted to the Kubeflow API server, which records the run in MySQL and creates an Argo Workflow CRD.
3. The Argo Workflow controller reads the CRD and creates pods to run each task.
4. A system-dag-driver pod initializes the run and creates the execution context in ML Metadata.
5. For each task, a system-container-driver pod checks for a cache hit; if found, the task is skipped entirely.
6. If there's no cache, the container-driver gathers inputs and generates an executor_input JSON for the task.
7. A system-container-impl pod runs your actual component code.
8. When the task finishes, Kubeflow stores parameters, artifacts, metrics, and lineage in MLMD for tracking and future caching.

Task-Level Caching

One of the most useful features of Kubeflow Pipelines is task-level caching. If a pipeline fails at any stage, re-running it reuses the previous results from cache for every step whose code and inputs haven't changed, and skips re-executing them. This saves time and avoids unnecessary re-computation on retries.

Triggering the Pipeline

During development, you typically trigger a pipeline from your local machine using the Kubeflow Pipelines SDK. In production, pipelines are rarely triggered manually — they run in one of three ways:

• On a schedule — for example, every night or every week
• On an event — for example, when new training data lands in an S3 bucket
• From a CI/CD pipeline — the Kubeflow Pipelines API triggers the workflow automatically after code merges to the main branch

Airflow vs Kubeflow Pipelines

If you already have Airflow, why would you need another workflow orchestration tool? Airflow was designed to orchestrate general-purpose workflows — ETL pipelines, data engineering jobs, scheduled tasks — and it works well for those use cases. Kubeflow Pipelines was built specifically for machine learning workflows: it understands ML concepts out of the box and provides capabilities essential for building and operating production ML pipelines, such as artifact tracking, experiment comparison, and task-level caching tied to model lineage.