Nabla Containers

A new approach to Container Isolation

© 2021. All rights reserved.

Go over Nabla: App Safety meets Host Isolation

Written by Sahil Suneja 29 Oct 2018

This blog presents how an application written in Golang can be built and run as a Nabla container. We take an example of a web server application, part of the nabla-demo-apps repository. To enable/port Go over Nabla, we also briefly present certain key modifications made to the Go runtime.

Application build

The httpd example highlights Nabla’s support for Go applications organized across multiple .go files, multiple packages, as well as third-party package imports from github.

In the sample code, the main() function in httpd.go uses a utility function from the helper package (helper/process_args.go) to parse the commandline arguments. After some trivial commandline parsing, the webserver listens for HTTP GET commands on port 3000 from the outside world.

Building this Go webserver as a Nabla container is pretty straightforward. Dockerfile.Nabla lists the steps required for a multi-stage build of the Go app. A typical image build with docker build -t go-httpd-nabla -f Dockerfile.nabla . performs the following operations:

  1. First, the app’s source directory is added to the nabla-go-base image, which contains the Go runtime ported over Nabla (see Section ‘Under the Covers’).
  2. This is followed by the usual go get of the dependencies.
  3. After this, a generic Makefile.Goapp (part of the base image itself) is triggered, which builds the Go app as a static Nabla binary.
  4. The resulting binary then becomes the entrypoint of the Go app Nabla container, which can be run using the runnc Nabla-container runtime as docker run --rm --runtime=runnc go-httpd-nabla random-arg1 random-arg2. An expected output would look something like:
[/nabla-run --mem=512 --net-mac=62:b0:e3:51:0c:06 --net=/dev/tap8591 --disk=/rootfs.iso /goapp.seccomp {"env":"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","env":"HOSTNAME=78d9833120c3","env":"TERM=xterm","cmdline":"/goapp.seccomp arg1 arg2 arg3","net":{"if":"ukvmif0","cloner":"True","type":"inet","method":"static","addr":"172.17.0.2","mask":"16","gw":"172.17.0.1"},"blk":{"source":"etfs","path":"/dev/ld0a","fstype":"blk","mountpoint":"/"},"cwd":"/"}]

            |      ___|
  __|  _ \  |  _ \ __ \
\__ \ (   | | (   |  ) |
____/\___/ _|\___/____/
Solo5: Memory map: 512 MB addressable:
Solo5:     unused @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0x4f18a3)
Solo5:     rodata @ (0x4f18a4 - 0x6a1ccf)
Solo5:       data @ (0x6a1cd0 - 0x88a4bf)
Solo5:       heap >= 0x88b000 < stack < 0x20000000
rump kernel bare metal bootstrap

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 7.99.34 (RUMP-ROAST)
total memory = 251 MB
timecounter: Timecounters tick every 10.000 msec
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
cpu0 at thinair0: rump virtual cpu
root file system type: rumpfs
kern.module.path=/stand/amd64/7.99.34/modules
mainbus0 (root)
timecounter: Timecounter "bmktc" frequency 1000000000 Hz quality 100
ukvmif0: Ethernet address 62:b0:e3:51:0c:06
rumprun: call to ``_sys___sigprocmask14'' ignored
/dev//dev/ld0a: hostpath XENBLK_/dev/ld0a (25580 KB)
mounted tmpfs on /tmp

=== calling "/goapp.seccomp" main() ===

Hi
Got commandline args:
[random-arg1 random-arg2]
You can now call `wget <ip>:3000' now

(where <ip> is 172.17.0.2 as emitted in the preamble)

Under the covers

This Section describes certain internal details behind Nabla’s Go port, and assumes some familiarity with the Rumprun unikernel (one of the Library OSes Nabla supports). The Nabla Go base is built on top of Gorump, a port of Go for Rumprun. Nabla currently supports Go-1.5.1, one of the two versions supported by Gorump.

Gorump modifies Go to add a new supported ‘GOOS’- rumprun. Although this enables running Go applications as unikernel VMs, they can’t be run as-is as a Nabla container, unlike python or node applications. This happens because Go runtime’s requirement on platform-specific Thread Local Storage (TLS), is at odds with the host system interface Nabla permits to offer increased container isolation. Specifically, the Go runtime uses TLS to store a pointer to a ‘G’ structure containing internal details specific to each individual Go routine (a nice description can be found here). In AMD64, the TLS implementation translates to setting segment registers (FS/GS). The corresponding syscall to access these registers (arch_prctl(ARCH_SET_FS,..) in Linux, _lwp_setprivate() in BSD) is not part of the default Nabla seccomp policy1. While it could be added, the resulting impact on system isolation would need to be explored, which is a non-trivial task.

Another alternative is to handle TLS in software (potentially at a performance cost). Our port of Go on Nabla uses this approach, by using the POSIX pthread_setspecific() interface (managed inside rumprun) to provide TLS to the Go runtime.

The corresponding changes made to Gorump can be found in the solo5 branch of Nabla’s Gorump fork.

1 Other TLS implementations, such as modify_ldt() for i386 architectures, and set_thread_area(), would also require expanding Nabla’s seccomp policy to allow the respective syscalls.

Related Posts