Upgrade Kubeflow

Upgrading your Kubeflow installation on Google Cloud

Before you start

To better understand upgrade process, you should read the following sections first:

This guide assumes the following settings:

The ${MGMT_DIR} and ${MGMT_NAME} environment variables are the same as in Management cluster setup.
The ${KF_DIR}, ${KF_NAME}, ${CLIENT_ID} and ${CLIENT_SECRET} environment variables are the same as in Deploy using kubectl and kpt.

General upgrade instructions

Both management cluster and Kubeflow cluster follow the same instance and upstream folder convention. To upgrade, you’ll typically need to update packages in upstream to the new version and repeat the make apply-<subcommand> commands in their respective deployment process.

However, specific upgrades might need manual actions below.

Upgrading management cluster

Upgrade management cluster from v1.1 to v1.2

Note, there is no change for management cluster from v1.2.0 to v1.2.1. So there is no need to upgrade if you are already on v1.2.0.

The instructions below assume that your current working directory is
```
cd "${MGMT_DIR}"
```

Use your management cluster’s kubectl context:

# Look at all your contexts
kubectl config get-contexts
# Select your management cluster's context
kubectl config use-context "${MGMT_NAME}"
# Verify the context connects to the cluster properly
kubectl get namespace

If you are using a different enviroment, you can always reconfigure the context by:

make create-context

Check your existing config connector version:

# For Kubeflow v1.1, it should be 1.15.1
$ kubectl get namespace cnrm-system -ojsonpath='{.metadata.annotations.cnrm\.cloud\.google\.com\/version}'
1.15.1

Uninstall the old config connector in the management cluster:

kubectl delete sts,deploy,po,svc,roles,clusterroles,clusterrolebindings --all-namespaces -l cnrm.cloud.google.com/system=true --wait=true
kubectl delete validatingwebhookconfiguration abandon-on-uninstall.cnrm.cloud.google.com --ignore-not-found --wait=true
kubectl delete validatingwebhookconfiguration validating-webhook.cnrm.cloud.google.com --ignore-not-found --wait=true
kubectl delete mutatingwebhookconfiguration mutating-webhook.cnrm.cloud.google.com --ignore-not-found --wait=true

These commands uninstall the config connector without removing your resources.

Replace your ./Makefile with the version in Kubeflow v1.2.0: https://github.com/kubeflow/gcp-blueprints/blob/v1.2.0/management/Makefile.

If you made any customizations in ./Makefile, you should merge your changes with the upstream version. We’ve refactored the Makefile to move substantial commands into the upstream package, so hopefully future upgrades won’t require a manual merge of the Makefile.
Update ./upstream/management package:
```
make update
```

Use kpt to set user values:

kpt cfg set -R . name ${MGMT_NAME}
kpt cfg set -R . gcloud.core.project ${MGMT_PROJECT}
kpt cfg set -R . location ${LOCATION}

Note, you can find out which setters exist in a package and what there current values are by:

kpt cfg list-setters .

Apply upgraded config connector:
```
make apply-kcc
```
Note, you can optionally also run make apply-cluster, but it should be the same as your existing management cluster.

Check that your config connector upgrade is successful:

# For Kubeflow v1.2, it should be 1.29.0
$ kubectl get namespace cnrm-system -ojsonpath='{.metadata.annotations.cnrm\.cloud\.google\.com\/version}'
1.29.0

Upgrading Kubeflow cluster

DISCLAIMERS:

The upgrade process depends on each Kubeflow application to handle the upgrade properly. There’s no guarantee on data completeness unless the application provides such a guarantee.
You are recommended to back up your data before an upgrade.
Upgrading Kubeflow cluster can be a disruptive process, please schedule some downtime and communicate with your users.

To upgrade from specific versions of Kubeflow, you may need to take certain manual actions — refer to specific sections in the guidelines below.

General instructions for upgrading Kubeflow:

The instructions below assume that:

Your current working directory is:
```
cd ${KF_DIR}
```

Your kubectl uses a context that connects to your Kubeflow cluster

# List your existing contexts
kubectl config get-contexts
# Use the context that connects to your Kubeflow cluster
kubectl config use-context ${KF_NAME}

Edit the Makefile at ./Makefile and change MANIFESTS_URL to point at the version of Kubeflow manifests you want to use
- Refer to the kpt docs for more info about supported dependencies
Update the local copies:
```
make update
```
Redeploy:
```
make apply
```
To evaluate the changes before deploying them you can run make hydrate and then compare the contents of .build to what is currently deployed.