Cloud Foundry Diego Approaching 250,000 Containers in a Single Cluster, And What That Means For the Enterprise
Interest in containers reached a broad audience nearly two years ago. My, how time flies! Since then, adoption has been, well, up and to the left.
Today, Cloud Foundry is announcing it’s retiring its legacy DEA backend in favor of Diego, which is rapidly approaching the scale of 250,000 containers in a single cluster.
Why should you care?
Consider the iPhone. When you first purchased one, it was likely an 8gb phone. Chances are, you thought you’d never use all the space on that phone. Chances are you were dead wrong. Fast forward to last week’s iPhone 7 announcement. Max size for those devices was 256gb. If you’re anything like me, you probably plan to order the largest capacity device available, just to be safe.
EOL Timeline for Legacy Cloud Foundry DEA Backend
Cloud Foundry’s next generation runtime, Diego, is production ready and has been running production workloads since October 2015. Work to increase its scalability substantially past that of the DEAs is nearing completion. After this work completes we will be phasing out support for the DEA, Warden, HM9k, DEA Logging Agent, and Collector components. In this document, “DEAs” will be used to stand for all these components.
Though the specific date is subject to change, we expect Diego to meet its scaling targets of 250,000 containers spanning 1000 Cells during Q3 of 2016. This is being accomplished primarily by transitioning Diego from etcd to a relational database (Cloud Foundry’s HA MySQL database by default, though postgres is also supported).
Meet the Photon Platform BOSH CPI
Today, we’re excited to continue addressing the needs of developers by announcing the release of the Photon Platform BOSH CPI. We’re building upon the recent release of v0.8 of Photon Controller: a distributed, multi-tenant and elastic platform purposefully-built for modern applications.
BOSH was designed to address the challenges in building both small and large distributed system. A BOSH CPI, or Cloud Provider Interface, is an API that is used to interact with an underlying IaaS to create and manage objects on an infrastructure, including images, VMs and disks.
How To Bill on Cloud Foundry
This is a guest blog post by Greg Cobb, Ben Watts, Michael Smykowski and Dieu Cao from Pivotal.
As a Cloud Foundry operator, you may wish to monetize your application runtime and services usage. While Cloud Foundry does not include a built-in billing system, it does provide features to allow operators to construct their own billing tools that meet their individual needs and requirements.
Cloud Foundry exposes usage events for both applications and services. These usage events detail when Cloud Foundry users start, stop, and upgrade apps and services. These events contain additional information about the resource usage of the apps and services. Usage events are guaranteed to only appear for operations that have successfully completed and to be in chronological order.
Garden and runC
Today, Julz Friedman (an engineer from IBM and the Anchor for Cloud Foundry’s Garden project) posted an important update from the Garden project team to the cf-dev mailing list about the future of the Garden project’s approach to the Linux container runtime. TL;DR here is that the Garden project will be transitioning from it’s own backend for Linux containers to using runC from the Open Container Initiative.
As he says in his email, the team did an initial prototype as part of this decision (fair warning, that repo is a spike).
The Forklifted Application
This content was originally posted on the Pivotal blog and is part of the upcoming book – on Cloud Native Java – that I am writing with Kenny Bastani on building cloud native applications with Spring and Cloud Foundry. –Josh
Community and customers alike are moving as much of their workloads to platforms like Cloud Foundry as possible. Cloud Foundry aims to improve velocity by reducing or at least making consistent the operational concerns associated with deploying and managing applications. Cloud Foundry is an ideal place to run online web-based services and applications, service integrations and back-office type processing.
New Fellow Travelers Join the Cloud Foundry Foundation Mission
Pivotal CEO Paul Maritz wrote about Cloud Foundry back in February when we announced our intention to form the non-profit Cloud Foundry Foundation. The Foundation aims to formalize Pivotal’s commitment to the open governance of a new platform that serves businesses in a multi-cloud world, anchored around Cloud Foundry.
I’m delighted with Pivotal’s announcement of the eight new Gold level members who have stated their intention to join the Cloud Foundry Foundation when it launches this Fall. They include Accenture, BNY Mellon, Capgemini, Ericsson, GE, Intel, NTT and Verizon, bringing the total level of Platinum and Gold members to 17.
Software continues to disrupt every aspect of the consumer and enterprise experience.
Packaged and Offline Buildpacks
With the recent addition of the cf create-buildpack and cf update-buildpack commands, we’ve been looking into changes to how the Java Buildpack is deployed on Cloud Foundry. There is a move afoot to remove the buildpacks from the DEAs in favor of using those commands (with an automated initial install to preserve user-experience) and an outcome of this was the need to more easily create ‘packaged’ buildpacks. In the process of delivering these packages we also realized that we could also deliver “offline” buildpacks. To understand why “offline” buildpacks are interesting, it helps to understand why the buildpack is designed to look for dependencies on the Internet in the first place.
Wrapping libyaml in go
We recently released version 6.0.0 of cf, the command line client for Cloud Foundry. cf was previously written in Ruby, and we have rewritten it in Go. This allowed us to package cf as as a single binary and simplified our deployment strategy enormously.
The YAML problem
Two weeks before our release, we realized that the go YAML library we were using, goyaml, is distributed under the LGPL license. This makes it unusable for cf. We had to quickly find another way to parse YAML in cf.
The definitive YAML implementation is a C library called libyaml. We knew that Go had good support for interfacing with C libraries, so we decided to write our own go bindings to libyaml. This felt a bit risky given that we were releasing in two weeks, so we named our creation ‘gamble’.
HM9000: Ready for Launch
Cloud Foundry (CF) is a platform-as-a-service that, once deployed, makes it easy for developers to deploy, run and scale web applications. Powering this elegant PAAS is a complex distributed system comprised of several interoperating components: the Cloud Controller (CC) accepts user input and directs Droplet Execution Agents (DEAs) to stage and run web applications. Meanwhile, the Router maps inbound traffic to web-app instances, while the Loggregator streams log output back to developers. All these components communicate via NATS, a performant message bus.
It’s possible to boil CF down to a relatively simple mental model: users inform CC that they desire applications and Cloud Foundry ensures that those applications, and only those applications, are actually running on DEAs.