Kubeflow

Term 3 · Sep-Nov · 3 projects

• Abstracting Pod Lifecycle Diagnostics for Kubeflow Pipelines

  LFX Mentorship · Listed in the official cohort

  @alyssacgoinsSource IssueLFX portal

  Background Kubeflow Pipelines (KFP) is an orchestrator for containerized ML workloads. Featuring a Python SDK as well as a UI that visualizes pipeline run workloads as directed acyclic graphs, KFP is designed to be a Kubernetes abstraction for ML engineers and data scientists scaling their containerized training and experimentation workflows. But this abstraction breaks when a Kubernetes pod hits lifecycle failure. These failures can occur at the provisioning level (e.g., ImagePullBackOff or Unschedulable), runtime level (e.g., CrashLoopBackOff or OOMKilled), or node level (e.g., NodeLost or Preempted). On failure, the KFP UI displays a pipeline frozen at the current pod – not succeeding, progressing, or failing. The KFP console provides visual support for pod failures that result from errors in user-supplied pipeline code. But this support does not extend to the pod lifecycle failures defined above, forcing a user to debug with the Kubernetes CLI. Additionally, even after a user has retrieved pod status, they also require an advanced understanding of Kubernetes pod events and infrastructure. While AI tooling can ease the burden of Kubernetes debugging, it should not be a prerequisite for the project’s target users. This proposal aims to reduce technical friction in KFP by abstracting away low-level Kubernetes details for ML engineers and data scientists. Qualifications A successful applicant for this project is proficient in Go and Typescript and has practical experience with Kubernetes and pod debugging. Experience with Kubeflow is preferred. Project Deliverables This project introduces a new abstraction layer to visualize and manage Kubernetes pod lifecycle failures directly within the KFP UI. Spanning the entire KFP stack, the implementation is divided into three core phases: - UI-level Diagnostic Support: An upgraded visual console featuring color-coded pod lifecycle statuses, inline failure messages, and educational hover tooltips linked to documentation. - API Server-level Pod Failure Management: A more robust API layer capable of handling per-error custom timeouts and a repaired data pipeline that prevents Argo Workflow failure logs from being dropped. - Enablement & Docs: Complete feature documentation (within the KFP repository and the Kubeflow website) alongside an interactive website demo.

  • Kubernetes (including kubectl CLI)
  • Go
  • Typescript

• Evolve SparkClient into Kubeflow's Unified Data Processing Layer

  LFX Mentorship · Listed in the official cohort

  @shekharrajakSource IssueLFX portal

  Kubeflow SparkClient (KEP-107) gives Python users a simple way to run Apache Spark on Kubernetes: interactive Spark Connect sessions via connect(), batch jobs via submit_job(), lifecycle APIs, and a pluggable backend (Kubernetes, future scope - gateway/livy). The foundation lands this term through KEP-107 and GSoC 2026. This program evolves SparkClient into the unified, Pythonic data-processing layer of the Kubeflow SDK, so users go from raw data to trained models through one consistent API, alongside Trainer, Pipelines, Katib, and Model Registry. The mentee advances four connected workstreams: 1. Production readiness & observability — surface metrics, events, and structured logs through the client; add debugging tooling and a richer job/session status model for running Spark at scale. 2. Kubeflow ecosystem integration — Integrate with Kubeflow ecosystem and components like Kubeflow Notebook, Kubeflow pipeline, and other. Explore to make SparkClient the SDK's data layer. Flagship deliverable: hand a dataset written by SparkClient to Kubeflow Trainer via a DataFusion-backed cache, with hooks toward Pipelines, Katib, and Model Registry. 4. Multi-mode execution — extend beyond interactive/batch to scheduled (ScheduledSparkApplication) and streaming jobs behind one consistent Python API. We would like to make it for simple for ML users but making sure advance usecases are also available for data engineers. 5. Spark ML experience — Explore Spark MLlib workflows (preprocessing, training, evaluation) through SparkClient, registering results to Model Registry. 6. Writing blogs, examples, and improve IT, unit tests as well as making user AI assistants are helping well writing Spark Client. Expected Outcome: - Observability APIs on SparkClient: job/session metrics, events, and log streaming; an expanded status model; a debugging guide. - A working Spark → Trainer data handoff through a DataFusion cache, with a runnable end-to-end example. - Scheduled and streaming execution modes added to SparkClient, with tests and docs. - Explore Spark MLlib workflow example wired through SparkClient into Model Registry. - KEP-107 updates, unit and integration tests, and user-facing documentation for each capability and what to add in AI tools/Kubeflow MCP for easy usage for users.

  • Python
  • Java
  • Apache Spark
  • Kubernetes
  • distributed data processing
  • ML workflows
  • +2

• OptimizationJob CRD: HPO Engine for Kubeflow Trainer

  LFX Mentorship · Listed in the official cohort

  @tariq-hasanSource IssueLFX portal

  Hyperparameter optimization is critical for maximizing model performance, and Katib has long provided it through the Experiment CRD. But that CRD was built for broad use cases, including Neural Architecture Search and arbitrary workloads, which makes ordinary tuning verbose, hard to validate, and reliant on a stateful helper pod and database per experiment. The community has agreed to replace it with OptimizationJob, a resource focused solely on tuning TrainJobs. That work is half finished: the design is merged, the API is in review, and the push-based metrics foundation has shipped - but nothing yet watches these resources, launches trials, reads results, or picks a winner. This project builds that missing runtime.

  • Go
  • Python
  • Kubernetes controllers
  • CRDs
  • HPO frameworks