kustomize-controller

The kustomize-controller will keep the cluster state in sync with the resources defined in a kustomization.yaml file.

The kustomize-controller is looking at a Kustomization custom resource in the cluster (One entry point Kustomization was created on flux bootstrap). Kustomization custom resource will contain a sourceRef and a path to a folder. That folder can contain a kustomization.yaml file, which will contain a list of resources, and the kustomize-controller will strive to get the cluster state according to those resources. If the kustomization.yaml file is not present, then the kustomize-controller will generate a kustomization.yaml file based on the resources in the folder.

kustomize, kustomize-controller, kustomize.config.k8s.io/v1beta1 - Kustomization, kustomize.toolkit.fluxcd.io/v1 - Kustomization

This is where things get a bit confusing. There are two different Kustomization resources, one from the kustomize.config.k8s.io/v1beta1 API group, which is client only and not created in the cluster, and one from the kustomize.toolkit.fluxcd.io/v1 API group which is a custom resource added to our cluster. In addition there is the kustomize tool, and the kustomize-controller which is part of the flux project. We need to understand each of these components and how they relate to each other.

kustomize

kustomize kind of remind me of a very simplified version of helm. You feed kustomize with a Kustomization resource that looks like so:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- flux-system
- namespace.yaml

Here's an example of a Kustomization resource, kustomize based on this file will create the group of resources which we can feed kubectl and those resources will be created. It's important to note that this Kustomization resource is not a custom resource in the cluster, it's just a file that kustomize will use to generate resources.

kustomize can be externally installed but it's also now baked as part of kubectl.

kubectl kustomize <kustomization_directory>

Will show you the resources that will be created.

kubectl apply -k <kustomization_directory>

Will create the resources. This is not a kustomize lesson and there are plenty good lessons about this tool, and I'm assuming you have a bit of experience with it. We will however do a recap of the important features of this tool which we are going to utilize.

Install kustomize

Although kustomize is now baked into kubectl, you can still install it separately. The easiest way is using homebrew:

brew install kustomize

After installation you will have the kustomize command available. This command will not effect your cluster, and personally I love to use it as a tool to understand what is created, and patched.

kustomization.yaml

There is a special meaning for files named kustomization.yaml in a directory. Kustomize is given a directory, and it will look for a file named kustomization.yaml in that directory. If that file does not exist then kustomize will fail to generate the resources.

kustomize create

kustomize can help you create a kustomization.yaml file based on the resources in a directory.

cd clusters/staging && kustomize create --autodetect --recursive

You can cd into a directory containing yaml resources, and you can run kustomize create --autodetect --recursive and it will generate a kustomization.yaml file based on the resources in that directory. This is a very useful feature, and it's the feature that the kustomize-controller will use if the kustomization.yaml file is not present.

kustomize build

I use this command often, it's a way to see the resources that will be created by kustomize-controller. The command will not actually create the resources, it will just show you the resources that will be created.

kustomize build <kustomization_directory>

patch

It's important to go over this feature in the Kustomization resource since we will use it often, especially when we have similar configurations between clusters.
For example we might have a base infastructure configuration between the clusters, but there are some minor configuration changes in that base infastructure between the clusters, often kustomization patches will allow us that distinction.

There are 2 types of patches, patchesStrategicMerge and patchesJson6902.

patchesStrategicMerge

This is a list of file paths, and kustomize will merge these files with the resources in the kustomization.yaml file.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- flux-system
- namespace.yaml
patches:
- patch.yaml

In this example we will use the patch.yaml to add annotation to the source-controller ServiceAccount

apiVersion: v1
kind: ServiceAccount
metadata:
	name: source-controller
	namespace: flux-system
	annotations:
		iam.gke.io/gcp-service-account: source-controller@prj-academeez.iam.gserviceaccount.com

This is the content of the file patch.yaml, and it will add the annotation to the source-controller ServiceAccount. It will look for the ServiceAccount with the name source-controller in the flux-system namespace and it will add the annotation.

patchesJson6902

In this merge type, we can use a json patch to merge with the resources in the kustomization.yaml file. This can be used for a more surgical approach to patching a field or multiple fields instead of merging bulk of data.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- flux-system
- namespace.yaml
patches:
- target:
	kind: ServiceAccount
	name: source-controller
	namespace: flux-system
	version: v1
	path: patch.yaml

and in the patch.yaml file we can have the following content:

- op: add
  path: /metadata/annotations/iam.gke.io~1gcp-service-account
  value: "source-controller@prj-academeez.iam.gserviceaccount.com"

We will use this patch technique when we talk about workload identity and accessing out private helm repository.

overlays

Another feature we will use often is the overlays feature. This is a way to have a base configuration and then have different configurations for different clusters. Let's examine the following simple example. In the root directory create a folder called apps, this folder will contain our clusters app configurations. Later in the course we will have a slight difference between the staging and production apps, they will be installed by our private helm which we will configure in future lesson, and we will distinguish between chart versions with canary that will be loaded to staging cluster and the stable version that will be loaded to production cluster.

In the apps directory create a base directory, which will contain app configurations that are common to all clusters. Let's create the file release.yaml in the apps/base directory with the following content:

apiVersion: helm.toolkit.fluxcd.io/v2beta2
kind: HelmRelease
metadata:
  name: hello
  namespace: www
spec:
  interval: 10m
  timeout: 5m
  chart:
    spec:
      chart: hello
      repository: https://charts.example.com      
      # version: "..." modify this per cluster
      sourceRef:
        kind: HelmRepository
        name: hello

Notice that we omitted the version field, we will use the overlays feature to add this field per cluster, loading canary versions to the staging cluster. In the apps/base create also a kustomization.yaml file with the following content:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - release.yaml

For now this kustomization.yaml file only package all the resources in the directory, but we will use it later to add common labels, namespace and other configurations that relates to all the resources in this directory.

In the apps directory create a staging directory, which will contain app configurations that are specific to the staging cluster. In the apps/staging directory create a kustomization.yaml file with the following content:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base
patches:
  - path: release.yaml
    target:
      kind: HelmRelease

We added a patches field that targets the HelmRelease kind, and we will use this field to add the version field to the HelmRelease resource.

Create the file release.yaml in the apps/staging directory with the following content:

- op: add
  path: /spec/chart/spec/version
  value: ">=1.0.0-canary"

We will create the same files in the apps/staging also in the apps/production directory, but the version field will be different. Modify the file apps/production/release.yaml to have the following content:

- op: add
  path: /spec/chart/spec/version
  value: ">=1.0.0"

We can test our overlays by running the following command:

kubectl kustomize apps/staging
kubectl kustomize apps/production

You should see how the configurations for the 2 clusters are identical except for the version field. Overlays usually combines with patches to create similar clusters with slight differences.

This is the recap we need about the kustomize tool, Now let's move to the kustomize-controller and the Kustomization custom resource.

kustomize-controller

The kustomize-controller is a kubernetes operator which is using a Kustomization custom resource that points a path in a source (usually a git repository). Based on kustomization.yaml in the path (or if not present it is generated) resources are created in the cluster, and the state of our cluster is maintained.

Kustomization custom resource

After installing flux we now have 2 Kustomization resources, one from the kustomize.config.k8s.io/v1beta1 API group, and one from the kustomize.toolkit.fluxcd.io/v1 API group. It was confusing for me at first, and it's important to understand the difference between the two, and look at the apiVersion field to distinguish. Remember that flux resources have <controller>.toolkit.fluxcd.io in the apiVersion. Here's an example of a flux Kustomization resource and we will go over the important fields:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: www
spec:
  interval: 10m0s
  path: ./clusters/staging
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system

Let's go over the important fields:

metadata.namespace - The Kustomization resource doesn't have to be in the flux-system namespace, Although I usually put it there.
spec.interval - The interval in which the kustomize-controller will examine the resource, check the cluster, and correct any drifting to match the cluster state to the resources in the kustomization.yaml file.
spec.path - The path to the folder in the source (git repository in this case) that contains the kustomization.yaml file (if no kustomization.yaml it will be generated - more on that later).
spec.prune - If set to true removed resources will be garbage collected, I turn this on must of the times.
spec.sourceRef - The sourceRef field is a reference to a source-controller custom resource, in this course we will use GitRepository custom resource - this points to where the path directory is located, and where our yaml files to apply reside.

kustomize-controller entry point

When we ran flux bootstrap a flux Kustomization resource was created in the flux-system namespace:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 10m0s
  path: ./clusters/staging
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

This is our entry point for the kustomize-controller and the path is set to ./clusters/staging according to the --path flag in the flux bootstrap command.

kustomization.yaml generation

Notice that the path ./clusters/staging does not contain a kustomization.yaml yet the kustomize-controller did not complain. If the Kustomization resource is pointing to a path that does not contain a kustomization.yaml file, then the kustomize-controller will generate a kustomization.yaml (virtually not as an actual file). The way it creates the kustomization.yaml is by running kustomize create --autodetect --recursive on the path. Let's try to run it ourselves:

cd clusters/staging && kustomize create --autodetect --recursive

You will see that a kustomization.yaml file was created in the clusters/staging directory, and it contains the resources in the directory and also the subdirectories (like flux-system dir). It also included the namespace.yaml from the first lesson, this is how our namespace was created in the first lesson, by the kustomize-controller automatically generating a Kustomization resource, in the clusters/staging folder.

Arranging the repository

Now that we understand the kustomize-controller and the Kustomization resource, let's arrange our repository, to be more fitting for larger projects.

Move the clusters/staging/namespace.yaml to the folder: apps/base/namespace.yaml this will represent the namespace where we will place our apps (It's important to note that I usually recommend not to throw all the apps in a single namespace if you have a large project, more logical seperation is recommended, but for the sake of simplicity we will use a single namespace for all our apps in this course). We do need to modify the apps/base/kustomization.yaml file to include the namespace.yaml file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml

Delete all the release.yaml resources that we created, those were just for demonstration purposes, we will create them again in future lesson when we learn about the helm-controller. Modify the apps/staging/kustomization.yaml and apps/production/kustomization.yaml file to include the apps/base directory:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base

Also make sure you have a clusters/staging/apps.yaml with the following code:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  path: ./apps/staging
  prune: true
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system

we prepared an apps overlay, and place the namespace creation in that directory. If we push our code we should see a namespace as well as an additional Kustomization resource created:

> kubectl get kustomization -n flux-system

Summary

In this lesson we learned about:

kustomize - a tool that can generate resources based on a kustomization.yaml file, built-in in our kubectl client.
kustomize-controller which responds to Kustomization custom resource and keeps the cluster state in sync with the resources defined in a kustomization.yaml file.

We also learned about the Kustomization resource, and created a basic overlay for our apps as well as learned how to use patches which will be handy to create slight differences in our overlays.

The full source code is available here

Previouslesson: Installing Flux Nextlesson: helm-controller