A follow-up to Cloud Foundry in 10 minutes, from the SAP Team.
A while back we showed how to get Cloud Foundry running on your own machine in about ten minutes, with kind and a single make up. The idea was to give you a real Cloud Foundry to try out, with no BOSH and no full deployment behind it.
The question we kept getting back was whether this only works on a laptop. It isn’t. The laptop was only the easiest place to start. What’s running is ordinary Cloud Foundry, and it runs on any Kubernetes cluster you point it at — kind on your machine, Amazon EKS in the cloud, or anything else. The cluster has to provide a load balancer, persistent volumes on demand, and the node pools Cloud Foundry schedules onto with the labels and taints it expects. This time we run it on EKS, so the commands below are EKS-specific.
We use EKS here, but it’s only one example. The cluster-provisioning steps below are AWS-specific — eksctl, the AWS Load Balancer Controller, EBS-backed volumes — so they won’t port to GKE or AKS unchanged, but Cloud Foundry itself will. It installs onto any cluster that can give it a LoadBalancer, provision persistent volumes on demand, and offer node pools with the labels and taints it expects. On another platform you’d use that platform’s equivalents for the provisioning, and the Cloud Foundry install at the end stays the same.
This takes longer than ten minutes. Budget around half an hour, most of it spent waiting for AWS to build the cluster; the Cloud Foundry part at the end is quick.
Why this works this time
Cloud Foundry on Kubernetes has been tried before, more than once, and the projects that tried it — Eirini, cf-for-k8s, Korifi — are all archived now, so some skepticism is fair.
Each of them swapped out a piece of Cloud Foundry for something that behaved a bit differently. Eirini handed app-instance scheduling to Kubernetes instead of Diego’s BBS and auctioneer. cf-for-k8s staged apps with Cloud Native Buildpacks instead of the classic buildpacks. Korifi went furthest, reimplementing the v3 API on top of Kubernetes CRDs, and even after all that work it never fully covered the Cloud Foundry API. The result was close to Cloud Foundry but not the same, and your existing apps and scripts didn’t always keep working.
We went the other way and left Cloud Foundry itself intact: the Cloud Controller, Diego’s BBS and auctioneer, the classic buildpacks and the staging path, all unchanged. We had to change only one thing at the bottom of the stack: how a container gets created.
Diego’s rep, the agent on each cell that creates containers, talks to its container backend through a Go interface called garden.Client. That interface has been the seam between Diego and Garden for years, so we wrote a second implementation of it, one that creates a Kubernetes Pod against any conformant Kubernetes API. Everything above that seam is the same Cloud Foundry that runs in production, which means an app that runs on a BOSH-deployed Cloud Foundry runs here too, without changes. The exception is per-VM syslog metrics, since the components are no longer deployed on dedicated
VMs.
The Cloud Foundry Acceptance Tests (CATS), the same suite that gates a BOSH-deployed Cloud Foundry, pass here at 100%. They cover what an app and its operator actually do, and all of it works.
What you’ll need
Alongside the tools from the first post, you’ll want:
· An AWS account you can create a cluster in, and the aws CLI configured for it.
· eksctl, the EKS cluster tool.
· kubectl and helm, which you already have from last time.
· A DNS domain you control, for the Cloud Foundry API and app routes. We’ll use cf.example.com as a placeholder — substitute your own.
· A clone of the kind-deployment repository. The same make targets that ran locally do the work here too.
Everything below assumes you’re running the commands from that repository.
Step 1: Stand up the cluster
We start with a bare cluster that has a control plane but no worker nodes yet.
eks-cluster.yaml:
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: byo-cluster
region: eu-central-1
version: ‘1.36’
autoModeConfig:
enabled: false
addonsConfig:
disableDefaultAddons: true
addons:
– name: coredns
– name: aws-ebs-csi-driver
iam:
withOIDC: true
We turn off the default add-ons and pull in only what we need: CoreDNS for in-cluster DNS and the EBS CSI driver for persistent volumes. OIDC is on because the AWS Load Balancer Controller, a couple of steps down, gets its AWS permissions through a Kubernetes service account mapped to an IAM role (IRSA), and that mapping needs OIDC.
eksctl create cluster -f eks-cluster.yaml
This is the slow part, since AWS is spinning up a VPC, subnets, and a control plane for you, which takes roughly fifteen minutes.
Step 2: Networking with Cilium
Before we add any nodes, we install Cilium as the cluster’s CNI. Installing the CNIfirst means each node joins a cluster that already has networking, instead of booting and waiting for it to appear.
helm repo add cilium https://helm.cilium.io/
helm upgrade –install cilium
cilium/cilium –version 1.20.2 -n kube-system \
–set eni.enabled=true –set ipam.mode=eni –set
egressMasqueradeInterfaces=eth+ \
–set envoy.enabled=false
–set routingMode=native –set
kubeProxyReplacement=true \
–set
kubeProxyReplacementHealthzBindAddr=0.0.0.0:10256 –set operator.replicas=1
\
–set k8sServiceHost=$(aws eks
describe-cluster –name byo-cluster –region eu-central-1 –query ‘cluster.endpoint’ –output text | sed ‘s#https://##’) \
–set k8sServicePort=443
Step 3: Add the nodes
Now for the workers, which is also where we create the IAM service account for the load balancer controller, since both live in the same config file.
eks-nodegroup.yaml:
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: byo-cluster
region: eu-central-1
version: ‘1.36’
iam:
withOIDC: true
serviceAccounts:
– metadata:
name: aws-load-balancer-controller
namespace: kube-system
wellKnownPolicies:
awsLoadBalancerController: true
nodeGroups:
– name: controlplane
instanceType: m5.xlarge
desiredCapacity: 1
volumeSize: 80
privateNetworking: true
– name: cells-1a
instanceType: m5.xlarge
privateNetworking: true
desiredCapacity: 1
availabilityZones: [eu-central-1a]
volumeSize: 100
labels: { cloudfoundry.org/cell: “true”, cloudfoundry.org/zone: “a” }
taints:
– { key: cloudfoundry.org/cell, value: “true”, effect: NoSchedule }
– name: cells-1b
instanceType: m5.xlarge
privateNetworking: true
desiredCapacity: 1
availabilityZones: [eu-central-1b]
volumeSize: 100
labels: { cloudfoundry.org/cell: “true”, cloudfoundry.org/zone: “b” }
taints:
– { key: cloudfoundry.org/cell, value: “true”, effect: NoSchedule }
There are two kinds of node here. The controlplane group runs the Cloud Foundry system components, though despite the name it’s not the Kubernetes control plane, which EKS manages for you. The two cells groups, one per availability zone, are where applications run; they’re the Kubernetes equivalent of Diego cells. The labels and taints keep the two apart: the taints keep the system components off the cell nodes, and Cloud Foundry uses the labels to place app Pods onto them. Splitting the cells across cells-1a and cells-1b spreads them over two zones for resilience, the same way you’d spread Diego cells.
eksctl create nodegroup -f eks-nodegroup.yaml
eksctl create
iamserviceaccount -f eks-nodegroup.yaml –approve
Step 4: Ingress and storage
Two AWS-specific pieces are left. The first is the AWS Load Balancer Controller, which turns a Kubernetes Service into an actual network load balancer so outside traffic can reach Cloud Foundry.
helm repo add eks https://aws.github.io/eks-charts
helm install
aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system \
–set
clusterName=byo-cluster –set serviceAccount.create=false \
–set
serviceAccount.name=aws-load-balancer-controller –version 1.14.0
The second is a default storage class, so the components that need persistent volumes — the database, the blobstore — get EBS volumes automatically.
sc.yaml:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ebs-sc
annotations:
storageclass.kubernetes.io/is-default-class: “true”
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
csi.storage.k8s.io/fstype: ext4
type: gp3
kubectl apply -f sc.yaml
Step 5: Install Cloud Foundry
With the cluster ready, installing Cloud Foundry onto it is the same as installing it locally.
The load balancer needs a little AWS-specific config here. We want it to be an internet-facing NLB managed by the controller we just installed, so we pass the gateway a few annotations through an extra values file.
values-eks.yaml:
gateway:
infrastructure:
annotations:
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
And then:
export CLUSTER_DOMAIN=cf.example.com
export ADDITIONAL_VALUES_FILES=values-eks.yaml
make init install
If you did the local walk-through, you’ll notice we run make init install here rather than make
up. On your laptop, make up created a kind cluster for you first. Here there’s nothing to create: you already have a cluster and kubectl is pointing at it, so init install deploys the same components onto whatever your current context targets. Set CLUSTER_DOMAIN to your domain, and on AWS pass the values file for the NLB.
Step 6: Point your domain
at it and log in
Point your domain at the load balancer that just came up. Its address is on the gateway’s LoadBalancer Service; read the hostname off it:
kubectl get svc -n cf-system istio-gateway-istio -o jsonpath=‘{.status.loadBalancer.ingress[0].hostname}’
Create a DNS record for *.cf.example.com pointing at it, then let the setup finish wiring things up and log you in:
make login
bootstrap-complete
The cluster is ready, and it’s the same Cloud Foundry you already know. cf
push works the way it always has: the app stages with the classic buildpacks, Diego schedules it, and rep creates a Pod for it on one of the cell nodes. To the cf CLI this is an ordinary Cloud Foundry; it can’t tell the deployment runs on EKS.
Where this goes
We took a setup that started life as a way to run Cloud Foundry on a laptop and ran it, unchanged, on a managed EKS cluster. The laptop and AWS were only ever examples. What matters is that both run the same real Cloud Foundry, the components a production deployment uses, with no local shortcut underneath.
BOSH is how Cloud Foundry reaches production today, proven over many years, and it stays the reliable choice for a production workload. What we add is another option: if your organization already lives on Kubernetes, here’s a way to run Cloud Foundry on the infrastructure you already have.
If you want to dig into the internals, the Pod backend is in k8s-garden-client, the deployment lives in kind-deployment, and the Helm charts and images it installs come from cf-k8s-releases. Give it a try on your own cluster.
