How to Push an App to Cloud Foundry with Sidecars

This post was written by the CAPI Product team.

What are sidecars?

Sidecars in Cloud Foundry are additional dependent processes that are run in the same container as the main app process. If you’ve ever found yourself wanting to include an APM agent or proxy alongside your app, read on!

Why use sidecars?

You might be wondering how using sidecar processes differs from just pushing separate apps as microservices. While it’s true that these use cases overlap, the ability of the sidecar process to run in the same container as its main app grants us several affordances. For example, you may want to use sidecar processes if you have two processes that:

• need to communicate over a unix socket or via localhost.
• need to share the same filesystem.
• need to be scaled and placed together.
• need to have fast interprocess communication.

How sidecars work in Cloud Foundry

Sidecars are currently an alpha feature available in Cloud Foundry. All required code and configuration needed to run the sidecar and application are packaged together in the same droplet. This droplet is deployed in a single container on Diego and both processes within the container are health checked independently. To learn more about how sidecars work in Cloud Foundry, check out our sidecar documentation.

How to push an app to Cloud Foundry with a sidecar process

To demonstrate how sidecars work in Cloud Foundry, we will deploy a simple Ruby app that talks to a separate Golang binary called `config-server` that is deployed as a sidecar process.

Required Setup

To follow along with the tutorial, it’s recommended that you have access to the following:

• git
• cf cli
• A Cloud Foundry installation running capi-release `1.79.0` or greater.

First, clone the capi-sidecar-samples git repository:

git clone https://github.com/cloudfoundry-samples/capi-sidecar-samples.git

In this repository you will find two apps: a Ruby app that serves as the “main application” called `sidecar-dependent-app` and a Golang “configuration server” sidecar called `config-server-sidecar` that produces a `config-server` binary.

The `config-server` binary provides applications with their required configuration over its `/config/` endpoint and only accepts connections over localhost on the `CONFIG_SERVER_PORT` port. For example:

vcap@f00949bd-6601-4731-6f7e-e859:~$ curl localhost:$CONFIG_SERVER_PORT/config/
{"Scope":"some-service.admin","Password":"not-a-real-p4$$w0rd"}

Since the `config-server` sidecar process only accepts connections directly over localhost, it needs to share the same network namespace as the main app. Or, in other words, it needs to be co-located in the same container. This makes the `config-server` process a prime candidate for the sidecar pattern. For demonstration purposes, we’ve added a `/config` endpoint to the main app that simply calls out to the `config-server` sidecar and will echo back the configuration in its response:

get '/config' do
puts "Sending a request to the config-server sidecar at localhost:#{ENV['CONFIG_SERVER_PORT']}/config/"
response = Typhoeus.get("localhost:#{ENV['CONFIG_SERVER_PORT']}/config/")
puts "Received #{response.body} from the config-server sidecar"
response.body
end

The diagram below demonstrates this architecture:

In order to push our main application and its sidecar we first have to build the `config-server` binary. If you have Go installed on your machine, simply enter the `config-server-sidecar` and compile it using the following command:

GOOS=linux GOARCH=amd64 go build -o config-server .

If you don’t have Go installed, don’t fret! You can download the binary directly here.

Next, we will copy it alongside our main application’s source code so that it is included with the app when we deploy it.

cp capi-sidecar-samples/config-server-sidecar/config-server capi-sidecar-samples/sidecar-dependent-app/.

Next, let’s create the app on Cloud Foundry.

cf v3-create-app sidecar-dependent-app

We then configure the sidecar via our application manifest so that it is included when we push the app. An example manifest is shown below:

applications:
  - name: sidecar-dependent-app
    env:
      CONFIG_SERVER_PORT: 8082
    sidecars:
      - name: config-server
        process_types:
          - web
        command: './config-server'

Here we are setting the `CONFIG_SERVER_PORT` environment variable so that the sidecar knows which port to listen on and the main app knows which port to connect to. We also name the sidecar, assign it to the app’s main `web` process, and configure the start command for the sidecar.

This manifest can then be applied using the following command:

cf v3-apply-manifest -f <app-manifest-path>

Finally, we push the app using the V3 CF API.

cf v3-push sidecar-dependent-app

You can see both processes running by `cf ssh`ing on to the app container. You can run `ps aux` and see both the `config-server` sidecar process and `rackup` main web process command running.

vcap@f00949bd-6601-4731-6f7e-e859:~$ ps aux
USER         PID %CPU %MEM    VSZ RSS TTY  STAT START TIME COMMAND
root           1 0.0 0.0 1120     0 ? S<s 22:17 0:00 /tmp/garden-init
vcap           7 0.0 0.0 106716  4508 ? S<sl 22:17   0:00 ./config-server
vcap          13 0.0 0.1 519688 35412 ?        S<sl 22:17 0:00 /home/vcap/deps/0/vendor_bundle/ruby/2.4.0/bin/rackup config.ru -p 8080
vcap          24 0.0 0.0 116344 10792 ?        S<sl 22:17 0:00 /tmp/lifecycle/diego-sshd --allowedKeyExchanges= --address=0.0.0.0:2222 --allowUnauthenticatedClients=false --inhe
root          82 0.0 0.0 108012  4548 ? S<sl 22:17   0:00 /etc/cf-assets/healthcheck/healthcheck -port=8080 -timeout=1000ms -liveness-interval=30s
vcap         215 0.3 0.0  70376 3756 pts/0    S<s 23:12 0:00 /bin/bash
vcap         227 0.0 0.0  86268 3116 pts/0    R<+ 23:12 0:00 ps aux

You can also see that it’s listening on the port specified by `CONFIG_SERVER_PORT` and that our main `ruby` process is connected to it:

vcap@f00949bd-6601-4731-6f7e-e859:~$ lsof -i | grep $CONFIG_SERVER_PORT
config-se   7 vcap 3u  IPv4 17265901     0t0 TCP *:8082 (LISTEN)
config-se   7 vcap 5u  IPv4 17265992     0t0 TCP localhost:8082->localhost:42266 (ESTABLISHED)
ruby       13 vcap 11u  IPv4 17274965    0t0 TCP localhost:42266->localhost:8082 (ESTABLISHED)

Now, as a demonstration, we can navigate to our app at its route and view the configuration that it is fetching from the `config-server` sidecar. In your browser, you should see something like this:

Next, if we run `cf logs sidecar-dependent-app` and hit our app’s `/config` endpoint, we will see logs from both the sidecar and the main app process intermingled in the same log stream. For example:

   2019-04-17T16:46:47.41-0700 [APP/PROC/WEB/SIDECAR/CONFIG-SERVER/0] OUT Received a request for config.
   2019-04-17T16:46:47.41-0700 [RTR/0] OUT sidecar-dependent-app.treasure-boot.capi.land - [2019-04-17T23:46:47.398+0000] "GET /config HTTP/1.1" 200 0 64 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" "209.234.137.222:61388" "10.244.0.139:61004" x_forwarded_for:"209.234.137.222" x_forwarded_proto:"http" vcap_request_id:"a2cb121e-f40d-4c22-73fa-867bb5ea3602" response_time:0.013113447 app_id:"21df1eb8-f25d-43b2-990b-c1a417310553" app_index:"0" x_b3_traceid:"f186933c826c162d" x_b3_spanid:"f186933c826c162d" x_b3_parentspanid:"-" b3:"f186933c826c162d-f186933c826c162d"
   2019-04-17T16:46:47.41-0700 [RTR/0] OUT

Finally, let’s kill our sidecar process and see what happens. While running `cf logs` in one terminal, open up a second and start another `cf ssh` session. Now kill the sidecar process using the following command:

vcap@a8db0eed-7371-4805-5ad3-4596:~$ kill -9 $(pgrep config-server)
vcap@a8db0eed-7371-4805-5ad3-4596:~$ FAILED

In your terminal that’s tailing the app logs, you should see logs indicating the sidecar process crashed and that Diego restarted the app container. For example:

2019-04-17T16:48:55.41-0700 [API/0] OUT App instance exited with guid 
21df1eb8-f25d-43b2-990b-c1a417310553 payload: 
{"instance"=>"a8db0eed-7371-4805-5ad3-4596", "index"=>0, 
"cell_id"=>"86808ce7-afc2-47da-9e79-522a62a48cff", "reason"=>"CRASHED", 
"exit_description"=>"APP/PROC/WEB/SIDECAR/CONFIG-SERVER: Exited with status 137",
"crash_count"=>1, "crash_timestamp"=>1555544935367052708, 
"version"=>"50892dcb-274d-4cf6-b944-3eda1e000283"}

Tutorial Resources

Well there you have it, a basic CF app and sidecar running side by side. Again, if you would like to try these steps out yourself, we used the following applications to create this tutorial. They are open source and available at the links below for experimentation:

• Example sidecar-aware Sinatra application
• Example “config-server” Golang sidecar
• A helpful bash script to put it all together

Additional resources

• Cloud Foundry API documentation
• A detailed explanation of the sidecar pattern from Microsoft