Install the Kubeflow Pipelines SDK

Setting up your Kubeflow Pipelines development environment

This guide tells you how to install the Kubeflow Pipelines SDK which you can use to build machine learning pipelines. You can use the SDK to execute your pipeline, or alternatively you can upload the pipeline to the Kubeflow Pipelines UI for execution.

All of the SDK’s classes and methods are described in the auto-generated SDK reference docs.

Set up Python

You need Python 3.5 or later to use the Kubeflow Pipelines SDK. This guide uses Python 3.7.

If you haven’t yet set up a Python 3 environment, do so now. This guide recommends Miniconda, but you can use a virtual environment manager of your choice, such as virtualenv.

Follow the steps below to set up Python using Miniconda:

  1. Choose one of the following methods to install Miniconda, depending on your environment:

    • Debian/Ubuntu/Cloud Shell:

      apt-get update; apt-get install -y wget bzip2
      wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
      bash Miniconda3-latest-Linux-x86_64.sh
    • Windows: Download the installer and make sure you select the option to Add Miniconda to my PATH environment variable during the installation.

    • MacOS: Download the installer and run the following command:

      bash Miniconda3-latest-MacOSX-x86_64.sh
  2. Check that the conda command is available:

    which conda

    If the conda command is not found, add Miniconda to your path:

    export PATH=<YOUR_MINICONDA_PATH>/bin:$PATH
  3. Create a clean Python 3 environment with a name of your choosing. This example uses Python 3.7 and an environment name of mlpipeline.:

    conda create --name mlpipeline python=3.7
    source activate mlpipeline

Install the Kubeflow Pipelines SDK

Run the following command to install the Kubeflow Pipelines SDK:

latest_version=$(curl --silent https://api.github.com/repos/kubeflow/pipelines/releases/latest | jq -r .tag_name)
pip install https://storage.googleapis.com/ml-pipeline/release/${latest_version}/kfp.tar.gz --upgrade

After successful installation, the command dsl-compile should be available. You can use this command to verify it:

which dsl-compile

The response should be something like this:

/<PATH_TO_YOUR_USER_BIN>/miniconda3/envs/mlpipeline/bin/dsl-compile

Next steps