Build a Pipeline
This page describes how to author pipelines and components, and submit them to the Kubeflow Pipeline system to run. We will demonstrate the process using the Kubeflow Pipelines notebooks and samples to create a pipeline for deployment.
Using a notebook
You can choose to build your pipeline in a Jupyter notebook.
We suggest that you use the JupyterHub that is installed in the same cluster as the pipeline system. Follow these steps to start a notebook:
Follow the quickstart guide to deploy a Kubeflow cluster and open the Kubeflow Pipelines UI.
Click Notebooks in the left-hend menu. If this is the first time you’ve visited JupyterHub, you need to sign in with any username. Password can be blank. Then click the Spawn button to create a new instance. After a few minutes, the Jupyter UI opens. You can switch to the JupyterLab UI by changing
/labin the URL.
Download the sample notebooks from https://github.com/kubeflow/pipelines/tree/master/samples/notebooks.
Upload these notebooks from the Jupyter UI. In Jupyter, go to the tree view and find the upload button in the top right-hand area of the screen.
Open the uploaded notebooks and make sure you are on Python 3. The Python version is at the top right-hand corner in the Jupyter notebook view. You can run the notebooks now.
Note: The notebook samples don’t work on Jupyter notebooks outside the same cluster, because the Python library communicates with the Kubeflow Pipelines system through in-cluster service names.
The following notebooks are available:
KubeFlow pipeline using TFX OSS components: This notebook demonstrates building a machine learning pipeline based on TensorFlow Extended (TFX) components. The pipeline includes a TFDV step to infer the schema, a TFT preprocessor, a TensorFlow trainer, a TFMA analyzer, and a model deployer which deploys the trained model to
tf-servingin the same cluster. The notebook also demonstrates how to build a component based on Python 3 inside the notebook including building a docker container.
Lightweight python components: This notebook demonstrates building simple Python components based on Python 3 and using them in a pipeline with fast iterations. Going this route, building a component does not require building a docker container so it is faster, but the container image may not be self contained because the source code is not built into the container.
By following the notebooks you learn how to build pipelines and components with the Kubeflow Pipelines SDK (a Python DSL).
Using the command line
Instead of using a notebook, you can choose to set up things yourself using the command line. But currently you can’t use the Python SDK to submit pipelines to a cluster, and you can’t build container images using the SDK. The DSL compiler works as usual.
Set up Python
Python 3.5 or above is required. If you don’t have Python 3 set up, we suggest the following steps to install Miniconda.
In a Debian/Ubuntu/Cloud shell environment:
apt-get update; apt-get install -y wget bzip2 wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh bash Miniconda3-latest-Linux-x86_64.sh
In a Windows environment, download the installer and make sure you select the “Add Miniconda to my PATH environment variable” option during the installation.
In a Mac environment, download the installer and run the following command:
Create a clean Python 3 environment:
conda create --name mlpipeline python=3.6 source activate mlpipeline
conda command is not found, be sure to add Miniconda to your path:
Install the Kubeflow Pipelines SDK
Run the following to install the Kubeflow Pipelines SDK:
pip3 install https://storage.googleapis.com/ml-pipeline/release/0.1.7/kfp.tar.gz --upgrade
After successful installation the command
dsl-compile should be added to your
Compile the samples
The pipelines are written in Python, but they must be compiled to an intermediate representation before submitting to the Kubeflow Pipelines service:
dsl-compile --py [path/to/py/file] --output [path/to/output/tar.gz]
dsl-compile --py [ML_REPO_DIRECTORY]/samples/basic/sequential.py --output [ML_REPO_DIRECTORY]/samples/basic/sequential.tar.gz
Deploy the samples
Upload the generated
.tar.gz file through the Kubeflow Pipelines UI.
Advanced: Build your own components
See how to build your own pipeline components.