Cloud Foundry Logo
blog single gear
App Dev

Azure Node Resource Groups and cf-for-k8s

As a developer advocate at the Cloud Foundry Foundation, the most important thing to focus on is to make the Cloud Foundry projects resonate with the developer community at large. This means trying different things each day and testing the Cloud Foundry platform on various environments. It is a relentless pursuit in ensuring that every permutation of infrastructure, languages, and frameworks are covered and edge cases uncovered. In particular, the recent focus for me (and the team here) has been the cf-for-k8s project.

All details abstracted away, the cf-for-k8s project is a load of CRDs running on a Kubernetes cluster, which when installed provides the Cloud Foundry experience over k8s. This means simple deployments, simpler maintenance, and the most simplified day 2 operations!

One of the means to success for this project (like many other open source ones), is to have it work on various managed Kubernetes providers. In the past we have documented the process with Google Kubernetes Engine (GKE) and DigitalOcean Kubernetes (DOKS) clusters. This post is meant to clarify a certain caveat when it comes to installing Cloud Foundry for Kubernetes on Azure Kubernetes Service (EKS).

Installing cf-for-k8s is rather simple. Create a reasonably sized cluster, install kubectl, cf-cli, bosh, ytt, and kapp locally. Generate the required YAML for deployment and fire away with the kapp tool. Complete details available here.

However, what complicates this process slightly is the way different infrastructure providers create Kubernetes clusters. In particular, Azure Kubernetes – well, all Azure infrastructure in general – employ what are known as Resource Groups.

Resource groups hold all related Azure compute resources together. In the case of a Kubernetes cluster, this includes the virtual machines comprising the node pool, networking components (Istio), storage, etc. In the case of cf-for-k8s this also includes the Cloud Controller, Eirini, blobstore, kpack, and all the other blocks that are needed to enable the Cloud Foundry experience.

For architectural reasons, every Azure Kubernetes Service spans two resource groups –

  1. The first is the one the user creates.
  2. The second is a system-generated one.

This Azure FAQ provides some clarification about the two resource groups that are created.

The second resource group is known as the node resource group. It is this second resource group that holds all of the functional components that make up the Cloud Foundry Platform. During installation, the load balancer of cf-for-k8s is configured against an IP or a domain, located in the resource group. This static IP or domain needs to be located within this system generated resource group.

What if the domain or IP is not within this resource group? The cf-for-k8s installation will not complete and the resulting installation will be unable to discover other components. Internally, the Istio ingress controllers will continue to throw errors because of the inability to route requests and not resolve internal components.

Therefore, if you’re using Azure Kubernetes Service, and would like to install cf-for-k8s on your clusters, please remember to create the static IP in the system-generated node resource group. 

Have questions about using cf-for-k8s on your choice of infrastructure? No problem – please reach out to us via Slack! Our community is always available to answer any questions you may have.  

Ram Iyengar Profile Image

Ram Iyengar, AUTHOR