Skip to content
Yoke

Air Traffic Controller

Overview

The Air Traffic Controller (ATC) is a Kubernetes controller that enables the creation of CustomResourceDefinitions (CRDs) and implements them through Yoke flights.

The Problem with Current Deployment Tools

Client-Side Package Managers:

Yoke introduces an innovative “packages as code” model, shareable as WASM executables. However, client-side package managers are limited in Kubernetes environments. These tools:

  • Operate outside Kubernetes APIs, deploying resources ad hoc into clusters.
  • Fail to track package lifecycles effectively without reliance on resource labels or implementation specific secrets.

Such reliance on tool-specific implementations makes managing packages less transparent and harder to standardize.

Limitations of ArgoCD:

Tools like ArgoCD rely on generic container resources (e.g., Argo Applications), which offer flexibility but introduce challenges:

  • Permission escalation: Users creating Argo applications, either via kubectl or Git commits, can bypass organizational RBAC, effectively assuming Argo’s deployment permissions.
  • Underuse of Kubernetes API validation: Applications are generic and pass unchecked values to their underlying resource renderer. For example, if using helm, values are unchecked by the kubernetes api and we must rely on chart implementations to validate and surface errors later down the pipeline.
  • Lack of specificity: Applications are deployed as generic resources, obscuring their types and requiring labels or naming conventions to organize and filter them.

Goal of the Air Traffic Controller

The ATC makes package management native to Kubernetes by defining packages as specific, well-structured resources. This approach:

  • Utilizes Kubernetes features like OpenAPI validation for robust package definitions.
  • Enforces RBAC policies, limiting resource deployment and enhancing security.

By aligning with Kubernetes’ native capabilities, ATC improves security, governance, and control over deployments.

How It Works

The ATC consists of two components: the controller deployment and the Airway CustomResourceDefinition (CRD). An Airway is an API that enables you to define and connect two core elements: a new CRD specification of your creation and its corresponding flight implementation, which is specified as a URL to a WASM executable flight.

The ATC monitors Airways and creates the specified CRD, and spawns Flight Controllers within its process to manage the corresponding CRs. When a CR is created, updated, or modified, the relevant Flight Controller invokes the corresponding WASM flight to compute the desired state of your resources and applies these changes to the cluster.

This design allows you to define custom packages and enables your users to deploy them as native Kubernetes resources.

Flow Diagram

How to install the ATC

Terminal window
# Substitute the following variables as desired for your deployment.
NAMESPACE=atc
VERSION=latest
RELEASE=atc
URL=https://github.com/yokecd/yoke/releases/download/$VERSION/atc-installer.wasm.gz


# Use the yoke cli to deploy the Air Traffic Controller!
yoke takeoff --create-namespace --namespace $NAMESPACE $RELEASE $URL

Getting started

We will be following along with the example found at https://github.com/yokecd/examples/tree/main/atc

To begin we will need two things:

  • The definition of our custom resource
  • A flight (executable wasm program) to implement our resource as a package

For the sake of example let’s assume that for our enterprise we wish to create a package representing a “Backend” or “API” service.

First let’s define a Go package containing our new Custom Resource type.

source: atc/backend/v1/backend.go

backend.go
package v1


import (
  "encoding/json"
  "fmt"


  metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)


const (
  APIVersion  = "examples.com/v1"
  KindBackend = "Backend"
)


// Backend is the type representing our CustomResource.
// It contains Type and Object meta as found in typical kubernetes objects and a spec.
// Do not provide a Status Object as that is automatically generated by the ATC.
type Backend struct {
  metav1.TypeMeta   `json:",inline"`
  metav1.ObjectMeta `json:"metadata,omitempty"`
  Spec              BackendSpec `json:"spec"`
}


// Our Backend Specification
type BackendSpec struct {
  Image       string            `json:"image"`
  Replicas    int32             `json:"replicas"`
  Labels      map[string]string `json:"labels,omitempty"`
  NodePort    int               `json:"nodePort,omitempty"`
  ServicePort int               `json:"port,omitempty"`
}


// Custom Marshalling Logic so that users do not need to explicity fill out the Kind and ApiVersion.
func (backend Backend) MarshalJSON() ([]byte, error) {
  backend.Kind = KindBackend
  backend.APIVersion = APIVersion


  type BackendAlt Backend
  return json.Marshal(BackendAlt(backend))
}


// Custom Unmarshalling to raise an error if the ApiVersion or Kind does not match.
func (backend *Backend) UnmarshalJSON(data []byte) error {
  type BackendAlt Backend
  if err := json.Unmarshal(data, (*BackendAlt)(backend)); err != nil {
    return err
  }
  if backend.APIVersion != APIVersion {
    return fmt.Errorf("unexpected api version: expected %s but got %s", APIVersion, backend.APIVersion)
  }
  if backend.Kind != KindBackend {
    return fmt.Errorf("unexpected kind: expected %s but got %s", KindBackend, backend.Kind)
  }
  return nil
}

Next we will want to implement a program to transform this resource into an array of resources.

Note that nothing in our example implementation is specific to backends or any type of workload. For this example our package will output the backend as a kubernetes Deployment and Service.

source: atc/backend/v1/flight/main.go

flight/main.go
package main


import (
  "cmp"
  "encoding/json"
  "fmt"
  "io"
  "maps"
  "os"
  "strconv"


  appsv1 "k8s.io/api/apps/v1"
  corev1 "k8s.io/api/core/v1"
  metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
  "k8s.io/apimachinery/pkg/util/intstr"
  "k8s.io/apimachinery/pkg/util/yaml"


  // path to the package where we defined our Backend type.
  v1 "github.com/yokecd/examples/atc/backend/v1"
)


func main() {
  if err := run(); err != nil {
    fmt.Fprintln(os.Stderr, err)
    os.Exit(1)
  }
}


func run() error {
  // When this flight is invoked, the atc will pass the JSON representation of the Backend instance to this program via standard input.
  // We can use the yaml to json decoder so that we can pass yaml definitions manually when testing for convenience.
  var backend v1.Backend
  if err := yaml.NewYAMLToJSONDecoder(os.Stdin).Decode(&backend); err != nil && err != io.EOF {
    return err
  }


  // Configure some sane defaults
  backend.Spec.ServicePort = cmp.Or(backend.Spec.ServicePort, 3000)


  // Make sure that our labels include our custom selector.
  if backend.Spec.Labels == nil {
    backend.Spec.Labels = map[string]string{}
  }
  maps.Copy(backend.Spec.Labels, selector(backend))


  // Create our resources (Deployment and Service) and encode them back out via Stdout.
  return json.NewEncoder(os.Stdout).Encode([]any{
    createDeployment(backend),
    createService(backend),
  })
}


// The following functions create standard kubernetes resources from our backend resource definition.
// It utilizes the base types found in `k8s.io/api` and is essentially the same as writing the types free-hand via yaml
// except that we have strong typing, type-checking, and documentation at our finger tips. All this at the reasonable
// cost of a little more verbosity.


func createDeployment(backend v1.Backend) *appsv1.Deployment {
  return &appsv1.Deployment{
    TypeMeta: metav1.TypeMeta{
      APIVersion: appsv1.SchemeGroupVersion.Identifier(),
      Kind:       "Deployment",
    },
    ObjectMeta: metav1.ObjectMeta{
      Name:      backend.Name,
      Namespace: backend.Namespace,
      Labels:    backend.Spec.Labels,
    },
    Spec: appsv1.DeploymentSpec{
      Replicas: &backend.Spec.Replicas,
      Strategy: appsv1.DeploymentStrategy{
        Type: appsv1.RollingUpdateDeploymentStrategyType,
      },
      Selector: &metav1.LabelSelector{MatchLabels: selector(backend)},
      Template: corev1.PodTemplateSpec{
        ObjectMeta: metav1.ObjectMeta{Labels: backend.Spec.Labels},
        Spec: corev1.PodSpec{
          Containers: []corev1.Container{
            {
              Name:            backend.Name,
              Image:           backend.Spec.Image,
              ImagePullPolicy: corev1.PullIfNotPresent,
              Env: []corev1.EnvVar{
                {
                  Name:  "PORT",
                  Value: strconv.Itoa(backend.Spec.ServicePort),
                },
              },
              Ports: []corev1.ContainerPort{
                {
                  Name:          backend.Name,
                  Protocol:      corev1.ProtocolTCP,
                  ContainerPort: int32(backend.Spec.ServicePort),
                },
              },
            },
          },
        },
      },
    },
  }
}


func createService(backend v1.Backend) *corev1.Service {
  return &corev1.Service{
    TypeMeta: metav1.TypeMeta{
      APIVersion: corev1.SchemeGroupVersion.Identifier(),
      Kind:       "Service",
    },
    ObjectMeta: metav1.ObjectMeta{
      Name:      backend.Name,
      Namespace: backend.Namespace,
    },
    Spec: corev1.ServiceSpec{
      Selector: selector(backend),
      Type: func() corev1.ServiceType {
        if backend.Spec.NodePort > 0 {
          return corev1.ServiceTypeNodePort
        }
        return corev1.ServiceTypeClusterIP
      }(),
      Ports: []corev1.ServicePort{
        {
          Protocol:   corev1.ProtocolTCP,
          NodePort:   int32(backend.Spec.NodePort),
          Port:       80,
          TargetPort: intstr.FromString(backend.Name),
        },
      },
    },
  }
}


// Our selector for our backend application. Independent from the regular labels passed in the backend spec.
func selector(backend v1.Backend) map[string]string {
  return map[string]string{"app": backend.Name}
}

The flight executable can now be built via the Go Toolchain:

Terminal window
GOOS=wasip1 GOARCH=wasm go build ./path/to/main

Where and how you choose to host this binary such that it is fetchable by the ATC controller is up to you. Common choices include:

  • In a public or private github release
  • In an internal service you deploy in your cluster
  • Any service that hosts assets for you and allows you to fetch them over http/https.

Since we are following the example from the yokecd/examples repository, we will be using the latest github release of this flight: https://github.com/yokecd/examples/releases/download/latest/atc_backend_v1_flight.wasm.gz

Now all that’s left is to build an Airway using our v1.Backend type in conjunction with our Flight. Let’s define it in code.

source: atc/backend/airway/main.go

airway/main.go
package main


import (
  "encoding/json"
  "fmt"
  "os"
  "reflect"


  apiextv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
  metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"


  "github.com/yokecd/yoke/pkg/apis/airway/v1alpha1"
  "github.com/yokecd/yoke/pkg/openapi"


  v1 "github.com/yokecd/examples/atc/backend/v1"
)


func main() {
  if err := run(); err != nil {
    fmt.Fprintln(os.Stderr, err)
    os.Exit(1)
  }
}


func run() error {
  return json.NewEncoder(os.Stdout).Encode(v1alpha1.Airway{
    ObjectMeta: metav1.ObjectMeta{
      Name: "backends.examples.com",
    },
    Spec: v1alpha1.AirwaySpec{
      WasmURLs: v1alpha1.WasmURLs{
        Flight: "https://github.com/yokecd/examples/releases/download/latest/atc_backend_v1_flight.wasm.gz",
      },
      Template: apiextv1.CustomResourceDefinitionSpec{
        Group: "examples.com",
        Names: apiextv1.CustomResourceDefinitionNames{
          Plural:     "backends",
          Singular:   "backend",
          ShortNames: []string{"be"},
          Kind:       "Backend",
        },
        Scope: apiextv1.NamespaceScoped,
        Versions: []apiextv1.CustomResourceDefinitionVersion{
          {
            Name:    "v1",
            Served:  true,
            Storage: true,
            Schema: &apiextv1.CustomResourceValidation{
              OpenAPIV3Schema: openapi.SchemaFrom(reflect.TypeFor[v1.Backend]()),
            },
          },
        },
      },
    },
  })
}

With this final piece we have built a flight that installs the Airway that will bind our Backend resource to its implementing flight. Fortunately for the rest of this example, it is also hosted in the github release of our examples repository.

Putting it all together let’s install yoke, setup a local cluster, install the atc, install our airway, and finally install our new backend component.

Terminal window
# install yoke cli
go install github.com/yokecd/yoke/cmd/yoke@latest


# create a local cluster
kind delete cluster && kind create cluster


# install the atc
yoke takeoff -wait 30s --create-namespace --namespace atc atc 'https://github.com/yokecd/yoke/releases/download/latest/atc-installer.wasm.gz'


# install the yokcd/examples Backend-Airway
yoke takeoff -wait 30s backendairway "https://github.com/yokecd/examples/releases/download/latest/atc_backend_airway.wasm.gz"


# You are done! You can now create Backends!
kubectl apply -f - <<EOF
apiVersion: examples.com/v1
kind: Backend
metadata:
  name: nginx
spec:
  image: nginx:latest
  replicas: 2
EOF

Flight Overrides

Generally, the custom resources we create do not expose flight implementation details. Users can create resources without being aware of the underlying implementation details of the CRD.

However, this abstraction can pose challenges during flight development, as Airways only accept a single Flight Module URL. Any changes to the flight implementation will update all custom resources associated with that airway.

During flight development, you may want to test a new implementation in production with a non-critical application as a canary.

To enable resource-specific flight implementation updates, you can override the flight URL using the following annotation: overrides.yoke.cd/flight: <url>.

The ATC will use the module located at the specified URL and apply it to the custom resource. For example, you can add the annotation as shown below:

Terminal window
kubectl apply -f - <<EOF
apiVersion: examples.com/v1
kind: Backend
metadata:
  name: nginx
  annotations:
    overrides.yoke.cd/flight: http://path/to/development/module.wasm
spec: ...
EOF

The development module is not cached, as it is assumed to be neither stable nor versioned. This allows you to iterate on the module by updating its source without changing the URL. However, this also means that overrides are not recommended for production use. They are significantly less performant than standard modules because each update forces the ATC to download and compile the WASM module.

Conversion Webhooks via Converter Wasm Programs

Airways describe the CustomResourceDefinition you wish to create. Therefore, it is best to follow the recommended best practices outlined in the official Kubernetes documentation: Versions in CustomResourceDefinitions.

  • Avoid breaking changes.
  • Use conversion webhooks to translate between different versions of your CustomResource.

Setting up your own Conversion Webhook Server and deploying it to support breaking changes for your APIs can be cumbersome. To simplify the process of evolving your Airways and APIs over time, the ATC supports conversion webhooks for Airways that specify a converter Wasm executable.

apiVersion: yoke.cd/v1alpha1
kind: Airway
metadata:
  name: example
spec:
  wasmUrls:
    flight: http://wasmcache/example.flight.wasm.gz
    # Specifying a converter enables the ATC to handle conversion webhooks for our custom resource.
    converter: http://wasmcache/example.converter.wasm.gz
  template:
    # crd definition

A converter program reads a conversion review object, as defined in the conversion webhook documentation, from standard input and writes the resulting conversion review, with the conversion response populated, back to standard output.

An example conversion program written in Go can be found within the yoke project here.