Securing Cloud Foundry Volume Services: Part 2

In our last blog post, we covered the principal challenges we face in making volume services secure. This post will outline the available options for adding security back in. Read on!

(More) Secure Options

Option 1: SMB

This is our current recommendation for customers interested in security. SMB is sometimes also referred to as CIFS or Samba. There are subtle differences among CIFS, Samba and SMB, but typically when someone mentions any of these terms, they mean SMB. SMB is most prevalent in Windows environments, and originated as a way to attach to Windows file shares.

Because it was the enabling technology behind the Windows “net use” command, SMB is designed to be a single-user protocol, typically requiring a username and password or other single user authentication in order to perform a mount. This makes it an odd fit for most Unix server applications, but a much better fit for Cloud Foundry applications. The requirement to provide username and password to complete any mount means that SMB Volume Services act much more like any other Cloud Foundry service; the server can be left open to all of Cloud Foundry, and application developers will not be able to gain access to it without a set of credentials.

Connections using SMB Version 3 (the default version for recent releases of Linux) are also encrypted in flight. This means that traffic to and from the server cannot be easily read with a packet sniffer.

Option 1 PROS:

• Password security is built into the protocol.
• No additional configuration is necessary in Cloud Foundry. The service just needs to be enabled.
• Performance is similar to a standard kernel mount.
• Data is encrypted in flight when the server supports SMB version 3.

Option 1 CONS:

• Most existing/legacy shares in Linux environments will be using NFS. Storage teams and application developers may be unfamiliar with SMB.
• The NAS and network may require additional configuration to support SMB connections, and username/password authentication.

You can find more instructions for enabling and using the Cloud Foundry SMB volume service here:

• https://docs.cloudfoundry.org/running/deploy-vol-services.html#smb-example
• https://docs.cloudfoundry.org/devguide/services/using-vol-services.html#smb

Option 2: NFS with LDAP

This option attempts to work around the various security challenges described in Part 1. When properly implemented, it allows customers to achieve heterogeneous access control and to approximate the same level of security they would normally have when using standard NFSv3 IP security. It requires nontrivial configuration steps however, and still does not offer encryption of data in flight.

The Cloud Foundry NFS Volume Service contains an optional mounting feature that operates on top of the standard NFS kernel mounter. This feature uses the FUSE module to substitute alternate UIDs and default GIDs when sending traffic to the NFS server. The result is that application developers can choose a specific UID when binding to an NFS service, and file accesses from their application will appear to come from that user.

In the default configuration, however, there are no restrictions on which UIDs can be specified in a service binding, and as a result, any developer can impersonate any user for purposes of file access.

To lock down that interaction, we have added an LDAP integration into the NFS volume service. When LDAP is enabled, developers can no longer use the UID and GID configurations, and instead must specify a username and password. Immediately prior to performing the mount, the NFS volume driver contacts the LDAP server to validate the password, and to look up the UID corresponding to the given username. That UID is then used in the volume mount.

The final step to secure this configuration is to disallow NFS traffic to the NAS from within application containers. This blocks malicious application code from communicating directly with the NFS server, thus making an end-run around the LDAP security. We recommend one or more of the following steps:

1. Avoid using insecure NFS exports, as that will allow malicious application code running as vcap to connect on port 2049 to your share.
2. Avoid enabling Docker application support as that will allow root users to connect on port 111 even when your share is not insecure.
3. Use CF Security groups to block direct application access to your NFS server IP, especially on ports 111 and 2049.

Option 2 PROS:

• Works with existing NFS shares and servers, and is familiar to storage teams.
• Provides a similar level of security to the IP security used for many NFS servers.

Option 2 CONS:

• Requires significant additional configuration, particularly for customers who do not have an LDAP server. In many enterprises this requires collaboration among Cloud Foundry, Storage, and ADFS/LDAP teams to fully implement.
• Performance is significantly worse than a kernel mount due to the use of FUSE in the mount chain.
• Data is unencrypted in flight.

Instructions for configuring the NFS volume service, LDAP integration, and CF security groups can be found here:

• https://docs.cloudfoundry.org/running/deploy-vol-services.html#example
• https://github.com/cloudfoundry/nfs-volume-release/blob/master/USING_LDAP.md
• https://docs.cloudfoundry.org/concepts/asg.html

Option 3: NFS with Isolation Segments

The final option for customers that cannot use SMB and cannot use option 2 — either for performance or functional reasons — is to set up an isolation segment. Isolation segments allow customers to create a separate pool of VMs and control access to those VMs by org and space through the Cloud Foundry CLI.

Using NFS IP security to only allow mounts from the VMs in an isolation segment, and restricting the spaces that can run workloads on that segment, provides another mechanism to restrict access to a file share.

Option 3 PROS:

• Works with existing NFS shares and servers, and is familiar to storage teams.
• Provides coarse-grained access control without requiring configuration of an LDAP server.
• Provides an access control mechanism that doesn’t incur the performance overhead from FUSE mounts.

Option 3 CONS:

• Only provides space-level access control.
• Requires a separate pool of cell VMs just to have different levels of access to a file share, which may cause the overall deployment to grow.
• May conflict with other use cases for isolation segments, leading to a proliferation of isolation segments.
• Data remains unencrypted in flight.

Conclusion

We hope this post has been helpful, and that it will assist you in creating a secure configuration for your volume services in Cloud Foundry. If you have questions or concerns about the material covered here, or anything else related to volume services on Cloud Foundry, please reach out to us in the #persi channel on Cloud Foundry Slack. You can also post an issue to the nfs-volume-release or smb-volume-release github repositories.