Some part of the content from this post was migrated to our Docker Platform tutorial. Content left here will be getting gradually more and more outdated.
Docker Containers provide some level of isolation, but unfortunately, not enough if you want to give full access to someone who you do not trust.
🔐 In this blog post we will show you a much safer alternative: Kata Containers that provide VM-like isolation for your container, while keeping the simplicity of Docker ecosystem.
💡 As an example use-case we will configure a VPN client and SSH server inside Kata Container, that can be securely exposed to untrusted users of your service. Doing the same with pure Docker would be very risky.
Thanks to Husarnet P2P VPN used in this example you can provide a secure and quick access to isolated containers in an easy way.
Kata vs runc
Docker is a very extensible tool. One of it's building blocks is called "runtime". Runtime is responsible for actually running the containers. The default runtime is called runC and it uses Linux namespaces and cgroups in order to isolate the container from the rest of the system. This is a relatively thin layer of separation implemented in the Linux kernel that tries to maximize the performance, minimize the startup times while providing as much security as possible. This is fine for most of the cases but it shouldn't be considered secure when running untrusted apps. An alternative designed for such cases is for example Kata Containers presented in this article. The main difference here is that each of the containers is being run in a separate virtual machine (so called MicroVM). The main benefit of this is a much greater separation between your container and the host - thus increased security. What's more - Kata Containers is designed to be a drop-in replacement for runc so using it is super easy - which we will show in this blog post.
What we're going to do?
In this blog post we're going to:
- install Docker (you can skip this step if you already have Docker installed)
- install Kata runtime, make it able to run VPN inside and let your Docker know about alternative runtime
- clone and configure the example so you actually use it
- start the containwe using two different methods so you can use your favourite for production use
Tested on a clear installation of the following host system:
Operating System: Ubuntu 20.04.1 LTS
Kernel: Linux 5.4.0-62-generic
Architecture: x86-64
Install Docker
If you have Docker already installed - you can skip this step.
The official instruction is the best tutorial but here's a quick rundown for you:
sudo -E apt-get -y install apt-transport-https ca-certificates software-properties-common && \
curl -sL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - && \
arch=$(dpkg --print-architecture) && \
sudo -E add-apt-repository "deb [arch=${arch}] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" && \
sudo -E apt-get update && \
sudo -E apt-get -y install docker-ce docker-compose
sudo systemctl daemon-reload
sudo systemctl restart docker
Get and prepare the Kata containers runtime
Even if you have Kata Containers already installed do NOT skip this step. Kata's kernel needs some extra kernel options in order to be able to run Husarnet VPN Client.
As this step is a little bit cumbersome we've provided a script that'll do all the necessary configuration steps for you.
Begin with cloning the GitHub repository.
git clone https://github.com/husarnet/blog-examples/
cd kata-example
If you do not have git installed, add it with:
sudo apt-get install -y git
Then - run our script. It won't ask you for any additional info (except maybe for sudo password ;-P) and it won't override any of your existing Docker defaults. Kata runtime will be installed and added as a new runtime to Docker. Kernel in Kata Containers will be rebuilt (this step may take a couple of minutes!) and installed as it's default.
sudo ./prepare-host.sh
…and voilà! Your host is now ready to run Husarnet Client in Kata Container.
Install Husarnet Client on the host
It's not necessary for production usage, but in order to test this example, you'll need an access to a host with Husarnet Client installed and running. The easiest way would be to install it on the same host you're running Docker on.
Use our getting started on Linux tutorial in order to do that.
Start the example
Using docker-compose
Create .env file
…and specify Husarnet JoinCode and hostname there. The file should look something like this:
JOINCODE=fc94:b01d:1803:8dd8:3333:2222:1234:1111/xxxxxxxxxxxxxxxxx
HOSTNAME=my-container-1
You will find your JoinCode at https://app.husarnet.com
-> Click on the desired network
-> Add element button
-> Join code tab
Build an image and start a container
sudo docker-compose up
Skip the docker run method and go to the result paragraph.
Using docker run
Build an image
Make sure init-container.sh is executable. If not:
sudo chmod +x init-container.sh
Then build an image:
sudo docker build -t kata-example .
Get your JoinCode
You will find your JoinCode at https://app.husarnet.com
-> Click on the desired network
-> Add element button
-> Join code tab
…and change the JOINCODE variable in the listing below to your JoinCode.
Start a container
Execute in a Linux terminal:
sudo docker run --rm -it \
--env HOSTNAME='my-container-1' \
--env JOINCODE='fc94:b01d:1803:8dd8:3333:2222:1234:1111/xxxxxxxxxxxxxxxxx' \
--volume husarnet-ssh-v:/var/lib/husarnet \
--volume husarnet-ssh-keys:/srv/sshd \
--device /dev/net/tun \
--cap-add NET_ADMIN \
--sysctl net.ipv6.conf.all.disable_ipv6=0 \
--runtime kata-runtime \
kata-example
Note that we provide --runtime kata-runtime while starting a container, not before.
If you remove this line, the container will run based on the default runtime for Docker - runc. So basically you use Kata almost the same way, as standard Docker. You can even run both types of containers on the same host.
That's a great featue: you can use kata-runtime only for containers needing an extra level of isolation, that however means also utilizing a little bit more resources of your host system. A "trusted containers" that don't need deep isolation can run the old way, being a little bit faster than the same container running on Kata runtime.
Result
After running a container you should see a log like this:
⏳ [1/2] Initializing Husarnet Client:
waiting...
waiting...
waiting...
waiting...
success
🔥 [2/2] Connecting to Husarnet network as "my-container-1":
[693849] joining...
[695852] joining...
done
*******************************************
💡 Tip
To SSH your container execute in a new terminal session:
ssh johny@fc94:2106:8423:68d7:695e:f6cf:8130:c059
(default password is "johny" as well)
*******************************************
The container is up and running. This example uses SSH as the "main application" but you can change it to whatever you want.
You can now access it from other machines in the same Husarnet network using standard SSH client:
- Husarnet IPv6 Address
ssh johny@fc94:2106:8423:68d7:695e:f6cf:8130:c059
- Hostname provided as
--envindocker runor in.envfile
ssh johny@my-container-1
Remarks
If you like this example and want to run this in production, here are some tips that you may find helpful.
prepare-host.sh script - performs a pretty standard installation of Kata Containers, the only tricky part is where it adds extra modules to the kernel. Look for enable_option function usages in that file to get the current list of required modules.
init-container.sh script - this file shows:
- which processes have to be running in the background (
husarnet daemon) - how to delay starting your application until the Husarnet Client is ready
- how to get the IPv6/Husarnet address of your container
docker-compose.yml file
The most important part here is runtime: kata-runtime. If you've manually installed kata as the default runtime - it's not necessary, but we'd still advise to be explicit here.
In order to preserve the IPv6/Husarnet address between runs of the container you need to persistently store /var/lib/husarnet hence the volume.
/etc/sshd volume is just a sugar to albo persist SSH keys in case you'd really like to run SSH in the container.
The way you specify environment variables is completely up to you. Just remember, in case you store your confings in some kind of version control system, that JoinCode is a sensitive data and should be treated as a secret - it gives the access to the whole specific network.
sysctls, cap NET_ADMIN and access to /dev/net/tun are all required in most of the cases.
Dockerfile
Pretty much everything here is up to you, as long as you:
- install Husarnet Client
- make
ip6tablesuseip6tables-legacyas it's what we're configuring the kernel for in the script above. If your OS in the container needs a differentip6tablesbackend - please configure the kernel accordingly - start
husarnet daemonin the background as we've mentioned above.
Summary
In this article you've learned:
- how to run untrusted containers in a secure way with Kata Containers
- how to provide a secure, remote SSH access to the Kata Containers even if you are being behind NAT or firewall
We hope you will find this article helpful.
We would be happy to discuss the topic covered in this blog post on Husarnet Community Forum.