Red Hat, Google Engineers Work on a Way for Kubernetes to Run Containers Without Docker
In 2015, when the Open Container Initiative (OCI) was launched to create industry standards around containers, it used Docker’s container runtime and image format as the base. But now a number of companies are undertaking a project that would break the OCI stack away from Docker in preference of Kubernetes, Google’s open source container orchestration engine.
This new project is geared for Kubernetes. It will directly interface with Kubernetes pods. It will enable Kubernetes — not Docker — to launch and manage containers at scale.
“What we want is a daemon that can be used by Kubernetes for running container images that are stored on Docker registries,” said Dan Walsh, the long-time SELinux project lead, and consulting engineer with Red Hat, speaking with The New Stack. Red Hat’s and Google’s developers are taking the lead with this project, for now, called simply OCID (OCI daemon). “In order to do that,” Walsh continued, “we wanted to build a series of libraries, to be able to facilitate running these container images.”
The maintainers of the project assert that OCID is not a “Docker fork,” though the project serves as evidence of a split in the container ecosystem, especially as Kubernetes continues to gain its own traction in the container marketplace.
Earlier this year, Docker, Inc. made its own orchestration engine, Docker Swarm, a part of its Docker Engine as per of release 1.12. As a result, Docker Engine now allows users to manage complex containerized applications without additional software. The Docker orchestration capabilities are opt-in; they must be activated by the user, though many worry that not opting in may lead to backward compatibility issues down the road.
Docker’s move to add Swarm to Docker Engine added friction to a community already at odds. In late August, the interest in forking Docker surfaced in discussions with vendors and users. Publicly, developers were vocal about Docker’s aggressive release schedule and how it put third-party system providers at odds with their own customer base.
ocid – A fully featured, standards based, container runtime that implements Kubernetes' Container Runtime Interface. https://t.co/GVDVcqLs7U
— Kelsey Hightower (@kelseyhightower) September 15, 2016
At present, Red Hat’s engineers are taking the lead with the OCID project, which kicked off in June of this year. It’s being developed as part of the Kubernetes Incubator initiative, which supports projects that, in turn, support Kubernetes. Its lead maintainers are Google’s Vishnu Kannan, and Red Hat’s Mrunal Patel.
“We don’t really need much from any container runtime, whether it’s Docker or [CoreOS’] rkt — they need to do very little,” said Kelsey Hightower, Google’s staff developer advocate, “mainly give us an API to the kernel. So this is not just about Linux. We could be on Windows systems… if that’s the direction the community wanted to go in, to support these ideas. Because it’s bigger than Docker Inc., at this point. This is about, how do you run a containerized application?”
Front and Center
If OCID (pronounced O-C-I-D) does become, as some suggest, a reference implementation of a container engine, it will provide a free and open source option for running OCI containers at scale. It will include runc, the container runtime based on libcontainer, which Docker donated to the OCI for use as a free standard. It will also include the code necessary for pushing images to and pulling them from repositories hosted by container registries. It will support the Container Network Interface, built by CoreOS, for modeling plug-ins independently from the engine hosting the containers.
Yet the component that is OCID’s raison d’être is called oci-runtime. As its project page on GitHub describes it, “This is an implementation of the Kubernetes Container Runtime Interface (CRI) that will allow Kubernetes to directly launch and manage Open Container Initiative (OCI) containers.”
OCI not only needs to specify how to run containers, Kelsey Hightower explained, but also how to download and network them, giving people something they can actually use.
In the Kubernetes environment, there is a component named the kubelet whose job is to manage pods — clusters of containers that typically comprise an application. A kubelet is capable of acting as a stand-alone daemon, a kind of local overlord for the containers in its pod. The key project in OCID, oci-runtime, will implement a new class of interface between kubelets and Kubernetes, enabling the orchestrator to effectively manage the entire container lifecycle.
“What we want to do now is refactor the kubelet code base,” remarked Google’s Hightower, “so that the way we integrate with Docker, rkt, and now OCID is much cleaner and consistent. We want to provide a well-defined interface to say, in order for a container runtime engine to be compliant with Kubernetes, we’re going to have a Container Runtime Interface. The CRI will be an API abstraction of what your container runtime needs to support, in order for us to certify it as something we can run underneath Kubernetes.”
Fill in the Blank
If OCID sounds like a wholly new, feature-complete container engine, it is not. Despite how it has been characterized to date, there is one very significant omission.
“Right now, at this point, OCID is not implemented to be able to build an OCI image,” said Red Hat’s Walsh. “You wouldn’t have the ability to create the image in the first place… You still need a tool like Docker for building the OCI image — [which] is basically a Docker image.”
“Our goal — and, I think, the industry’s goal — is to have a standard implementation of the container runtime, and a standard format for how container images are built,” remarked Joe Fernandes, Red Hat’s senior director of product management for OpenShift (Red Hat’s commercial orchestration engine, based on Kubernetes). “So that different vendors can build different solutions on top and the ecosystem can grow.”
The Docker image format has established common ground for the development ecosystem, Fernandes explained. He perceives the publication of that format as a standard to be part of the job of OCI.
“Once you have the format, then it comes down to the runtime,” he said. “Some people thought that, when Docker contributed libcontainer — which became runc — that was the runtime. It was a piece of the runtime — the lowest piece. But these other modules also make up the runtime. So what we’re doing here is creating standard implementations for these modules.”
In December, that actually appeared to be true, after presentation slides made it appear to have been firmly decided that OCI’s focus would be restricted to the container runtime (runc) and the container image format.
But Red Hat’s developers, aided by Google’s developers, are arguably extending the definition of “container runtime.” Fernandes explained it as the foundation tier of a much broader implementation. And Hightower explained it as, by definition, including an interface to an orchestrator component — for now, Kubernetes, but theoretically not restricted to Kubernetes, assuming any other orchestrator wants to pick up the same API and run with it.
“OCID is really the natural progression of OCI,” explained Hightower. “Their goal is to specify some standards around pretty much what Docker started: this idea that we can take our applications, package them into an image format, put them on a repository, and share them with anyone. And once they’re shared, pull them down, extract them, and run them in a very consistent way. ”
In order to do that, OCI not only needs to specify how to run containers, Hightower explained, but also how to download and network them, giving people something they can actually use.
“If they’re going to be a bunch of documents and libraries, let’s give people something that is, I guess in many ways, comparable to Docker itself,” he continued, “that is focused only on those bits that I just mentioned. That way, it’s less of a mystery on what OCI is; now you’ll have a fully specified, integrated thing that you can actually use in some of these projects.”
One of OCID’s principal components is skopeo, a tool for validating and fetching container images from Docker repositories, that today is one of Red Hat’s major contributions to Docker itself.
Walsh explained skopeo as originally having been developed as a tool that examined a container image, and produced a JSON file that could be used as its manifest. That tool was later leveraged for pulling and pushing images from a Docker registry. Red Hat then enrolled it as part of its Project Atomic, and the Go library-based derivative is now available on GitHub as containers/image.
In addition to skopeo, Walsh said, the OCID project includes a copy-on-write file system based on a Docker component called the graph driver.
“We started working several months ago,” he told us, “on splitting out the copy-on-write file system, so the other tools besides Docker could share [it]. We actually found it difficult to continue to work underneath Docker to get this as a separate module, so we decided to pull it out and concentrate on getting the module totally correct. Hopefully, at some point, we want to open up a pull request to get this back into Docker.”
In the meantime, he said, the current form of the library also appears on GitHub as containers/storage. While Docker does have its own copy-on-write file system, Walsh pointed out, it runs entirely in its own exclusive memory.
“We’re using the same shared libraries that OCID is using,” he said. “We would love to see some of these libraries going into future versions of rkt, and we would actually like to see them going back into Docker… What we want to do with storage is move the locking of file system storage to file lock, so that multiple applications could share the storage at the same time. We would love to get that back into the Docker daemon at some point.”
Motivation
As a project in development, OCID may be certain things today that it will cease to be in the future, and vice versa. One of the things OCID is not, at least today — surprisingly — is a mechanism for running OCI containers exclusively. Nor will OCID fall under the scope of the OCI project, said Patel, for certifying containers as OCI compliant. Documentation co-authored by Patel and Kannan explicitly calls for flexibility of container format support, including continuing to support Docker as OCI goes forward, and also supporting everyday TAR files.
As Joe Fernandes reminded us, Docker began its existence as a complete container system, and over the preceding months (it’s still a very young product) became more modularized — runc being one of those modules. Although runc’s predecessor was designed to be pulled into Docker Engine, runc can just as easily be pulled into CoreOS’ rkt today.
“You can view OCID along those lines,” Fernandes explained. “There’s other modules here that we’d like to carve out and make them standards… We want to work with the Docker community, who could pull these back in, now that they’re standard interfaces — versus having a sort of monolith that has it all blended together. It’s the whole UNIX philosophy of, ‘Do One Thing Well.’ We created the OCID project to house that work — to test out these ideas, and come up with standard implementations that then can be pushed back into OCI, and then ultimately pulled into the different implementations, Docker being the obvious one.”
As Fernandes and Walsh put it to us, the omission of an “engine” component as the creator of container images for OCID, is by design. They agreed that it’s the engine where vendors such as Red Hat, Docker, CoreOS, and VMware could attribute their respective “value-adds” and compete with one another on value and service.
Yet at the time of this writing, there remain two very curiously phrased elements of the open source documentation on GitHub (the entirety of which, of course, remains under continual review).
First, there’s this description that appears on the project page of the kubelet interface: “For the first release, oci-runtime will continue to use docker-engine for managing images. The image management functionality will be separated from the runtime functionality so each could have different implementations which could potentially be switched. It will be ideal to support the OCI image specification for images once it reaches v1.0.”
That description may be interpreted as stating that the runtime component will lean on Docker code for support, at least in the initial rounds, but it also sets the stage for OCID for using other engines, decreasing its reliance on the Docker Engine itself.
The second, equally curious, phrase is the hook in the OCID documentation that links OCID to Google: specifically, its explicit self-declaration as a part of a Kubernetes environment. Up front, OCID states its purpose in life is to provide management for the members of Kubernetes’ pods — and that declaration takes precedence over its management of containers.
Google’s Hightower believes that Kubernetes has earned a position in the container community that could perhaps be described as the arbiter of fairness.
“This is more about leveling the playing field,” he said. “Right now, Docker is the most well-supported container runtime, because we at Google and the community have done the majority of the work to make Docker work as a first-class runtime. But now that we see that people want choice, and it has always been our vision to support multiple container runtimes, one step in doing that is creating this Container Runtime Interface. As part of that, we’re going to refactor our current code base to implement the [CRI].”
Singularity
For the CRI to work as Hightower explained it, however, will require a certain kind of component that strikes at the heart of the very first argument that ever erupted over Docker: specifically, a system container, capable of launching itself. Such a component effectively kicks systemd, Linux’s now preferred application initialization mechanism, back out from under the rug.
“We want to ship everything in the form of containers, so we need a container runtime that doesn’t use orchestration,” explained Red Hat’s Walsh, “because the orchestration needs it. So we have a chicken-and-egg situation. By using system containers, we’re able to pull an image using the containers/image library, we’re able to store it on containers/storage, and then we’re going to execute runc, but in this case, we’ll be running it as part of systemd.”
It’s one way, Walsh believes, that a containerized environment can effectively launch itself — and Kubernetes in turn — in production.
Unquestionably, the scope of OCI has expanded in its first year of operation. There had been some talk at the beginning that, once the container format specification was released, OCI’s work would be done. Now, we see two of its most prominent members pushing for an open copy-on-write file system library, an open registry maintenance library, a networking scheme, and a bootstrap mechanism that relies upon an architectural feature Docker would prefer to remain deprecated. Could a container security library be far behind?
The vital component — the container engine itself — will be a matter left to each data center to resolve for itself. There, Docker might have an early advantage in having created the market in the first place. But it could find itself competing one-on-one with OpenShift, on terms that Red Hat helped create.
“I would say that OCID is going to be used by Kubernetes. Our goal, in an OpenShift environment, is to make Kubernetes use OCID as its container runtime environment. Then OpenShift will use Kubernetes, so we’ll get the advantages of OCID.”
So if you were to use Kubernetes in the future, you’d have OCID, and the decision would be made for you.
