This post has been authored by CAPI maintainers within the Cloud Foundry community. It is a reflection of their work on improving certain aspects of the Cloud Foundry API layer, relieving some of the technical debt that arises with a challenging project like this. This update focuses on Blobstore interaction specifically.
Cloud Foundry’s API layer (CAPI) – known as the Cloud Controller – uses the fog gem family to interact with blobstores on cloud providers like Azure, AWS, GCP, and Alibaba Cloud. The Cloud Controller is responsible for storing critical artifacts occurring during cf push: application packages, buildpacks, and droplets. Over time, several of the fog gems have become unmaintained, rely on archived SDKs, or require custom forks just to keep working. This blog post explains why we decided to replace fog, how we discovered a better approach by looking at how BOSH solves the same problem, and what the migration path looks like for operators.
If you are using cf-deployment, the blobstore ops files have already been updated to use the new storage-cli – no manual changes are needed from v55.2.0 onwards. Fog support is deprecated as of May 2026 and removal will begin shortly after, starting with the fog Azure gems. We encourage all operators to migrate as soon as possible.
The Problem: Technical Debt Blocking Progress
For years, the fog gem family has provided Cloud Controller with a unified Ruby interface for blobstore interactions across multiple cloud providers. While fog served the community well initially, it has become a growing source of technical debt. Several fog provider gems haven’t been updated in years like fog-azure-rm which was last released in January 2022 and thus is effectively abandoned. Microsoft archived the Azure SDK for Ruby in 2021, leaving no official upgrade path, and we maintain two custom git forks just to keep Azure blobstore functionality working. On top of this, fog dependencies prevent a confident upgrade to Ruby 3.4, increasing our CVE exposure window.
On top of the maintenance burden, fog is a large, multi-purpose library. Cloud Controller only needs blobstore operations like put, get, delete, exist. However, fog pulls in far more than that. We were carrying the weight of a full cloud abstraction library when all we needed was a small, focused client.
The Search and Discovery
In 2025 the CAPI maintainers began exploring alternatives. Newer Ruby gems carried the same maintenance risks. Forking the existing fog gems would still leave us with a Ruby-based solution that didn’t address the core problem. Building our own client would have been too much effort. No clear winner emerged.
The breakthrough came at Cloud Foundry Day North America 2025. Sitting together with other CAPI maintainers, the question came up: “Wait… how does BOSH handle blobstores? They’re also Ruby, right?” BOSH had faced identical challenges years ago and solved them with standalone Go-based CLI tools using official provider SDKs – actively maintained, consistent interface across providers (put, get, delete, exists), and proven in production for years across CF foundations worldwide. Instead of reimagining blobstore clients, we could reuse what already works.
The Solution: Storage CLI
A proof of concept with bosh-azure-storage-cli (PR #4397) validated the approach: core Cloud Foundry workflows including cf push, buildpack caching, and app bits handling all worked successfully. The gaps identified like bulk delete and direct blob copy were solvable with targeted additions to the storage CLIs.
This led to RFC-0043, which the community agreed on: a unified storage-cli consolidating BOSH’s provider-specific CLIs with the additional commands Cloud Controller needs. The result supports Azure (azurebs), AWS S3 (s3) including S3 compatible providers, GCP (gcs), and Alibaba Cloud (alioss).
Cloud Controller now accepts blobstore_type: storage-cli in its configuration, with fog remaining supported during the transition.
Migration Guide
Azure, Alibaba Cloud, GCP, AWS and other S3 compatible providers are supported from cf-deployment v55.2.0 onwards. Earlier versions include partial support: Azure from v54.9.0, Alibaba Cloud from v54.10.0, and GCP from v54.14.0.
If you use cf-deployment ops files, no manual changes are needed. Simply redeploy with cf-deployment v55.2.0 or later. The standard ops files (use-s3-blobstore.yml, use-azure-storage-blobstore.yml, use-gcs-blobstore-service-account.yml, use-alicloud-oss-blobstore.yml) already use storage-cli.
If you use custom deployments or ops files, you will need to update your configuration manually. Further information can be found in the Cloud Controller Blobstore Configuration documentation. Typically, the fog_connection block is replaced by connection_config and a new blobstore_provider field:
# Before (fog):
blobstore_type: fog fog_connection: provider: AWS aws_access_key_id: <key> aws_secret_access_key: <secret>
# After (storage-cli):
blobstore_type: storage-cli blobstore_provider: s3 connection_config: aws_access_key_id: <key> aws_secret_access_key: <secret> region: us-east-1
If you use custom fog configuration parameters, please review your fog configuration carefully. Some fog-specific parameters have no direct equivalent in storage-cli. Pay particular attention if you have customized timeout values, proxy settings, or provider-specific options.
Deprecation Timeline
The time to migrate is now. Fog support in Cloud Controller is deprecated as of May 2026 and gem removal will follow shortly after.
What to expect:
- May 2026: Fog deprecated – the fog code path will remain functional but receives no further maintenance
- Late May 2026: Azure gems removed first (fog-azure-rm, azure-storage-ruby) – most critical maintenance issues
- End of Q2 2026 and onwards: Remaining fog gems removed progressively (fog-aliyun, fog-openstack, fog-google, fog-aws, fog-local)
Not all fog providers have a direct replacement. fog-local is replaced by a native local blobstore client for development purposes only which does no longer support NFS. fog-openstack will be removed without a direct replacement. Swift users may be able to use the S3 provider as an alternative via its S3-compatible API. If you have any special blobstore configuration impacted by this, please reach out to us.
Stay informed and plan ahead: the removal order will broadly follow the order in which storage-cli support was introduced. Keep an eye on the cf-deployment and capi-release release notes for exact dates and announcements.
Further Details: S3 and the “S3-Compatible” Challenge
While Azure was our most critical provider, the S3 implementation brought its own set of challenges.
Performance Regressions and Tuning
During testing we discovered performance regressions in certain S3 workloads. The root cause: the initial S3 client implementation did not reuse TCP connections, resulting in higher latency under concurrent upload and download workloads.
We addressed this by switching to a keep-alive HTTP client and introducing configurable parameters to tune S3 transfer behavior:
{ "single_upload_threshold": 5242880, "download_concurrency": 5, "upload_concurrency": 5, "upload_part_size": 5242880, "request_checksum_calculation_enabled": true, "response_checksum_calculation_enabled": true, "uploader_request_checksum_calculation_enabled": true }
These settings allow operators to tune behavior for their specific infrastructure and workloads. See the storage-cli S3 README for the full list of configuration options.
A Note on “S3-Compatible” Storage
“S3-compatible” is a broad term. Many on-premise and alternative cloud storage systems advertise S3 compatibility but differ in subtle ways like checksum handling, multipart upload support, signature versions, or SSL behavior.
If you are running an S3-compatible blobstore (e.g. MinIO, Ceph, or a private object store), we strongly recommend testing Cloud Controller with storage-cli in a non-production environment first. The configurable parameters above exist precisely to handle these edge cases.
Summary
Replacing fog is not just a dependency cleanup – it is a step towards a more maintainable and secure Cloud Foundry platform. By reusing BOSH’s proven approach and collaborating across working groups, the CAPI maintainers have laid out the groundwork for removing a significant source of technical debt. With all major providers supported in cf-deployment v55.2.0 and a clear migration path in place, we encourage operators to make the switch.
Have questions or running into issues? Reach out in Cloud Foundry Slack (#capi channel) or open an issue in the cloud_controller_ng or storage-cli repositories.
