Bringing Cloud Foundry To Proxmox VE

How we built the API client, the BOSH CPI, the operator command line, and the foundation that connected the two platforms.

The Cluster BOSH Could Not Reach

At FiveTwenty we had a cluster ready to go, three Proxmox VE nodes with Ceph underneath them, and we had a Cloud Foundry foundation that needed to run on it. What we did not have was any way to connect the two. BOSH talks to every infrastructure through one small contract called the Cloud Provider Interface, or CPI. The contract has twenty-one methods, and they cover VMs, disks, stemcells, and networks. At the time, nobody had written those twenty-one methods for Proxmox VE. That meant nothing BOSH could deploy would land there, Cloud Foundry included, however capable the cluster underneath happened to be.

The same contract is also the good news. Any platform that implements it inherits the whole BOSH ecosystem at once, and that includes the stemcells, the releases, cf-deployment, and the acceptance suites. Closing the gap took three pieces of software, and we built them in a deliberate order: first a typed Go API client for Proxmox, then a BOSH CPI on top of that client, and finally a proxmox cli pmx which is a command line for the operators who prepare the clusters. After the tools came the foundation itself, and after the foundation came the workloads that moved onto it.

The API Client Came First

The CPI was the missing piece, but we started one layer lower. A CPI and an operator’s tooling both bottom out in the same Proxmox REST API, and if we wrote that layer twice we would discover every Proxmox quirk twice. So the first repository was proxmox-apiclient-go, a typed Go API client generated endpoint by endpoint from Proxmox’s own apidoc.json.

The API client owns typed requests and authentication, and each consumer decides for itself how long to wait and what counts as a failure. When we ran into a Proxmox quirk, such as a disk-resize endpoint that is a PUT rather than a POST, the fix landed once in the API client, and the CPI and pmx both picked it up from there.

Four Proxmox Answers To One CPI Contract

The BOSH Director never links against a CPI. It runs the binary once per request, writes one JSON-RPC request to stdin, and reads one response from stdout. There is no daemon, no session, and no shared state, so the identifiers the CPI returns for stemcells, VMs, and disks are the only memory it will ever get back. Four of the contract’s expectations had no direct Proxmox equivalent, and each of them forced a specific design answer.

Persistent disks have no volume object. AWS has EBS volumes, OpenStack has Cinder, vSphere has VMDK descriptors, and Proxmox has nothing like them. A volume that no VM config references is just bytes on storage, with no record of who created it or why. So we made the disk identifier itself the record. It encodes exactly where the volume lives, and a token written onto the drive’s serial= attribute survives every rename Proxmox performs. When a disk sits between owners, we park it on a protected VM that never starts, so an administrator browsing the cluster does not destroy what looks like an unused guest and take the disk with it.

First boot has no registry. A freshly cloned VM knows nothing about itself. It does not know its name, its networks, or where its Director is. Classic CPIs solved that with a registry service, which meant one more thing to deploy before anything else existed. Our CPI builds a small config-2 ISO for each VM and attaches it as a CD-ROM. The stock OpenStack KVM stemcell from bosh.io reads that ISO at first boot, because both platforms are QEMU and KVM underneath. The whole bootstrap has one dependency, and that dependency is the Proxmox API.

The host expects pets. A Cloud Foundry deploy can put a dozen concurrent create_vm calls on one node within a second, and those calls go through the same API the web UI uses for one click at a time. The API daemon defaults to three workers, and when a worker recycles in the middle of a burst it drops every in-flight connection without sending an HTTP response. So we wrap every call in exponential backoff with jitter, and we gate mutating calls through a per-node semaphore before they ever reach Proxmox. The Director only ever sees the faults that survived all of that.

Two systems want to heal the same VM. Proxmox HA restarts guests that fail, and the BOSH Resurrector rebuilds VMs whose agents stop responding. Point both at the same VM and they race, and the prize for finishing second is a duplicate. So the CPI registers nothing with Proxmox HA by default and writes onboot: 0 on every VM it creates. The Resurrector stays the only healer unless an operator opts a deployment into HA and switches resurrection off.

The release lives at bosh-proxmox-cpi-release, and the design behind each of those four answers is written up at BOSH On Proxmox VE.

Preparing The Cluster With pmx

A CPI translates requests. Somebody still has to build the cluster it talks to, and that preparation has teeth. Proxmox creates API tokens with privilege separation switched on, which gives the token an empty ACL even when it belongs to root, and every call returns 403 until the token is recreated with that flag off. Since Proxmox VE 8, a non-root identity that assigns a bridge needs the SDN.Use privilege, and no built-in role carries it. We transcribed a least-privilege role from the CPI’s own endpoint inventory, and it is written up in A Least-Privilege PVE API Token For BOSH.

Doing that work across several clusters with the stock tools meant living in SSH sessions and keeping node names and VMIDs in our heads, and day-two operations were no better. bosh create-env builds the Director through the same CPI, so it carries the same onboot: 0 and stays down after a node reboot until someone starts it. Nobody wants to run bosh create-env again for that. Starting a VM we knew only by its name took a pvesh query, a jq filter, and a second pvesh call against the right node.

With pmx, which sits on the same Go API client as the CPI, that whole dance becomes pve qemu start cpi-bosh-bosh-0. The guest resolves by name across the cluster, an ambiguous name returns an error rather than a guess, and the command blocks until the Proxmox task has actually completed.

flowchart TB
    accTitle: Three tools, one API client
    accDescr: The BOSH Director drives bosh-proxmox-cpi through the CPI v2 contract, and a platform engineer drives the pmx command line at a terminal. Both consumers sit on the same typed Go API client, proxmox-apiclient-go, which speaks the Proxmox VE, Backup Server, and Datacenter Manager APIs.

    director["BOSH Director<br/>cf-deployment above it"]
    operator["Platform engineer<br/>at a terminal"]
    cpi["bosh-proxmox-cpi<br/>one process per call"]
    pmx["pmx<br/>pve · pbs · pdm"]
    client["proxmox-apiclient-go<br/>typed API client from apidoc.json"]

    subgraph proxmox["Proxmox"]
        direction LR
        pve["Proxmox VE"]
        pbs["Backup Server"]
        pdm["Datacenter Manager"]
    end

    director -->|"CPI v2 over stdin/stdout"| cpi
    operator -->|"commands, contexts, labs"| pmx
    cpi --> client
    pmx --> client
    client -->|"REST API"| proxmox

A Foundation With A Route

Getting from a fresh node to a logged-in Director takes one bosh create-env run with the CPI release tarball and a vars file. On our lab hardware that run finishes in under half an hour, and most of the time goes to compiling packages. Then comes cf-deployment, which has no idea Proxmox exists. It asks the Director for VMs, disks, and networks, and the only object in the whole deploy that names Proxmox is the cloud-config. The three availability-zone aliases in that file bind to one node in a lab, or to separate nodes or clusters in production.

The deploy itself is the standard sequence. We push the runtime-config and the cloud-config, upload a stemcell, and run bosh deploy with the compiled-releases and HAProxy ops files.

Two changes along the way show up directly in those numbers. Linked clones from a per-cluster stemcell cache template took VM creation from roughly four minutes down to seconds, and compiled release tarballs cut the deploy phase from 21m32s to about 11 minutes. The full run is written up in Cloud Foundry On Proxmox: The Full Stack, End To End.

Moving Workloads Onto It

Everything platform-specific in a BOSH deployment lives in the cloud_properties blocks and in the CPI job. Moving a foundation onto Proxmox therefore means swapping the CPI, rewriting the cloud-config’s cloud_properties, and choosing the matching stemcell line, while the deployment manifests above that layer stay mostly unchanged. In practice we build the target foundation alongside the current one and move applications, services, data, and traffic across in waves. Developers keep the same cf push, the same buildpacks, the same routing, and the same service bindings.

The choice that matters most is storage. On a multi-node cluster, Ceph gives persistent disks failure independence, and it allows live migration. ZFS suits smaller clusters where each node keeps its own storage. LVM-thin is fine for ephemeral disks, but persistent data on it dies with its node.

Zones are the next decision. Nodes in one cluster share a control plane. A zone that has to survive a quorum incident or a cluster-wide upgrade therefore gets a cluster of its own, and one Director drives all of them through multiple CPI entries.

The Proof Is In The Repository

We wanted the CPI’s correctness demonstrated rather than asserted, so the proof runs in the open and it runs on a schedule. At the base sits a lifecycle harness that drives the methods in the order a real Director calls them, and it takes none of the CPI’s answers on trust. After each step a separate verifier queries the Proxmox API with the same credentials and confirms that the cluster agrees with what the CPI just claimed. A CPI that returns success while the cluster disagrees fails the run exactly as an error would.

Above the harness sit the two upstream suites, the BOSH Acceptance Tests and the certification suite’s Director Upgrade Test. Both run in GitHub Actions against live Proxmox servers, and both pass. The upgrade test carries a Director and its deployment through a release upgrade on the same state, and every disk and attachment stays intact across the version change.

A scheduled workflow repeats both suites unattended every Saturday on a self-hosted runner fleet. Each report lands on main through an auto-merged pull request, with the environment tuple and the per-example timings committed beside it. That schedule catches the drift nobody thought to look for, such as a dependency that shifted underneath the harness or a change whose consequence only shows up two weeks later. Every result is committed as it came, so the defect a Ceph run surfaced in the delete path for parked disks sits in the repository beside the passes.

The Contract Was Always There

BOSH was built without any knowledge of Proxmox. It only needed someone to teach Proxmox the twenty-one methods it has always spoken. Our team from FiveTwenty will be at Cloud Foundry Summit in Heidelberg on September 21 and 22, 2026, with the design lessons from this work and the run reports in hand. We would welcome a conversation there with anyone running Cloud Foundry on hardware they own.

Further Reading

• Design Decisions Behind A Modern CPI In Go
• Persistent Disks On PVE: Attach, Detach, Survive
• Bootstrapping Agents Without A Registry: ConfigDrive
• Tuning PVE Hosts For BOSH Workloads
• Certifying A BOSH CPI
• Introducing pmx: One Command Line For The Whole Proxmox Fleet
• One Director, Many Clusters: Multi-CPI BOSH On Proxmox
• Day-2 Operations With The Proxmox CPI

 

(Sponsored Post: This post was made by the team at FiveTwenty who are sponsors of Cloud Foundry Summit 2026)