I spent a bit of time visiting KubeCon 2015, the first Kubernetes conference held in San Francisco. Kubernetes is a Docker container orchestration system created by Google. I went there mostly to see what’s going on in the networking side of Kubernetes (written in shorthand as K8S).
The Container Network Interface: Network Plugins for Kubernetes and beyond talk by Eugene Yakubovich of CoreOS on the Container Network Interface was interesting. Its aim is to make network plug-ins work among different container engines. It works with so many different flavors of networking – ranging from MAC VLANS (a reverse VLAN that takes a single network interface and creates many virtual ones with different MAC addresses) to newer SDN systems like Weave and flannel.
I think this reflects the fact that in this varied container world, one cannot force a single way to do networking, so it’s better to plug in to different underlying network abstractions. Indeed, this system is a plug-in that can invoke even more lower level plug-ins to make it work. It started off as a networking layer for CoreOS’ rkt, a container runtime that now supports K8s. The lesson seems to be that in various containers systems, networking has been complex, hard for app developers to deal with, and has often been tacked on after the core runtime was designed. There was a danger it would happen for K8S, but this time, there are some early efforts to make it less painful.
The Dark Art of Container Monitoring talk by Gianluca Borello of SysDig was a demo tour-de-force. He first showed how hard it is to monitor K8S and how hard it was to correlate so much data from disparate sources. Then he showed some simple command line tools that textually displayed status, similar to the classic GNU/Linux ‘top’ command. Then he showed the graphical, cloud-based edition called sysdig cloud that lets you zoom into and out of K8S pods, containers, and showed graphs such as the latency of top transactions sent via HTTP. It’s hard to explain how good this demo was in this blog, but it suffices to say that it brought several rounds of spontaneous applause from the audience. Although sysdig does not go deep into networking other than tracking requests sent over the network, Gianluca (who has experience from Riverbed and CACE Technologies, the firm that did Wireshark) said that collecting and correlating app data with networking data is often requested by users, and it may be possible to collect data from switches via APIs or perhaps agents running on Linux-based switch platforms (Note: these types of data are programmatically available on systems as varied as the Linux-based network OSs: Big Switch’s Switch Light OS, Cumulus Linux, Juniper Junos, or mainstream systems like Cisco's NX-OS via NX-API or Arista EOS).
Kubernetes at eBay
I also saw the Cloud-Scale Kubernetes at eBay presentation on how eBay does cloud scale K8S by Ashwin Raveendran nair. The networking address management is interesting in that they chose many different methods – such as plain DHCP or IPAM, and their implementation is atop OpenStack. The viewpoint here is a highly production oriented one, where they need to be pragmatic to make it run for their heaily loaded environment. It’s always good to have users talk about their experiences, since it’s unvarnished.
Another interesting item shown on the exhibit area was Juniper’s Contrail, which is well-established SDN system (and got listed in a survey at the recent OpenStack summit as the most deployed OpenStack SDN system) that also integrates into K8S. The point about Contrail is that it’s multi-platform – it has a vRouter that runs on many platforms. It runs on OpenStack, but isn’t tied to specific OpenStack abstractions. It tunnels traffic over many methods – MPLS, VXLAN – and can work with plain OpenStack, K8S, and talks to bare metal via a VXLAN tunnel endpoint (VTEP) on the switch and interacts with peer controllers via BGP. The viewpoint from a networking vendor is to respect a wide variety of systems and platforms, since they need to work with a customer who already has an IaaS or PaaS platform. Via this approach, they’ve landed customers as varied as Lithium Technologies (a social platform) and AT&T.
SignalFX was showing their app monitoring system. This was interesting since its origin was in the experiences learned at large operators such as Facebook and now it's being made available for app operators (as a SaaS). The viewpoint taken by SignalFX is totally app driven. Whereas sysdig shows the components of the app that’s very useful for developers, The performance, such as app response is tied to areas of the infrastructure, such as networking, as well as many other elements of the stack. SignalFX shows the top level performance that is highly useful for operator of apps.
Metaswitch was showing Calico, a layer 3 virtual networking system that is highly scalable. This is quite unlike a traditional networking overlay system that creates tunnels – like VMware NSX or Juniper Contrail. Instead all the packets to and from a workload are routed by the Linux routing table and iptables and the traffic to other hosts are managed by BGP. Their viewpoint is highly network-centric, by exploiting the basic elements of BGP.
The takeaway to me is that at least for networking, the eye of the beholder makes a difference in design for the solution.
- A network-centric view such as Metaswitch or Juniper builds on a networking foundation and uses capabilities such as BGP as a scalable foundation.
- An app-centric view of SignalFX looks at high level application stats since it caters to app managers.
- A developer-oriented firm like CoreOS constructs systems in a framework of plug-ins.
- An operator like eBay takes pragmatic approaches by using every function available to them to make a production system work at scale.
The approaches to Kubrnetes network are quite varied, all tailored to who you are. It's a matter of what's comfortable and what you have experience with. This also translates into experiences appropriate for different types of end-users.
From a business perspective of those who may want to run K8S to be a foundation for modern cloud apps, they can see the emergence of an ecosystem of development components to support netwoking and a set of tools to help operate and monitor them. The rapidly growing popularity of K8S means that the pace of innovation is fast, and tools need to quickly evolve to catch up with the users and developers, but this show seems to indicate that many companies, large and small, are doing a decent job.