FreeBSD 13.0 Jails with ZFS and VNET

As of late I have had some pain points with iocage, which I used since I started using FreeBSD in 2017. I came from an Ubuntu with LXD + ZFS background and iocage had the command line interface I wanted that felt familiar with LXD at the time.

Well, iocage seems dead now. Its last release was in 2019 and its last commit (at the time of writing this) was September 30, 2021. Of course, that commit isn’t in what’s in FreeBSD ports, unless you use the devel package..and, that package has some issues (for me, iocage list doesn’t work right).

Because of this, I decided to take up the challenge of making my own base jails. To start, I will give credit where credit is due and say I followed these resources to get me to where I am:

Michael W Lucas' FreeBSD Jail Mastery book
https://clinta.github.io/freebsd-jails-the-hard-way/
https://www.skyforge.at/posts/a-note-in-sysvipc-and-jails-on-freebsd/
iocage and their fstab files

Creating the Release

First off, we need to create a release jail. This is a base image that we can use to make cloned jails, thick jails, or our base jails from. I’m going to start by making a new jails dataset and mounting it at /jails

$ zfs create -o mountpoint=/jails zroot/jails

Here is the foundation for everything. Now I’ll create a few other datasets for our releases and templates and running jails, as well as our first release dataset (13.0-RELEASE)

$ zfs create -p zroot/jails/releases/13.0-RELEASE
$ zfs create zroot/jails/templates
$ zfs create zroot/jails/jails

Next, we need to download the base OS as well as lib32 for our jail. The contents should be extracted into /jails/releases/13.0-RELEASE in the end.

$ fetch https://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/13.0-RELEASE/base.txz -o /tmp/base.txz
$ fetch https://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/13.0-RELEASE/lib32.txz -o /tmp/lib32.txz
$ tar -xf /tmp/base.txz -C /jails/releases/13.0-RELEASE
$ tar -xf /tmp/lib32.txz -C /jails/releases/13.0-RELEASE

Now let us update the jail contents

$ env UNAME_r=13.0-RELEASE freebsd-update -b /jails/releases/13.0-RELEASE fetch install
$ env UNAME_r=13.0-RELEASE freebsd-update -b /jails/releases/13.0-RELEASE IDS

Then, we can copy our /etc/localtime and our /etc/resolv.conf files into the jail $ cp /etc/localtime /jails/releases/13.0-RELEASE/etc/localtime $ cp /etc/resolv.conf /jails/releases/13.0-RELEASE/etc/resolv.conf

Nice. Now we have our base. Lets snapshot it so we can clone it. We will clone this to our templates folder after we take the snapshot:

$ zfs snapshot zroot/jails/releases/13.0-RELEASE@p4
$ zfs clone zroot/jails/releases/13.0-RELEASE@p4 zroot/jails/templates/base-13.0-RELEASE

This part is done. The release is made, now we have a base created for our base jails.

Creating Our Skeleton

Since we want to be using nullfs mounts for our base jail, we are going to want to make another clone and wipe out the contents of that new clone. Here I think you can debate whether or not you want to take a clone of the base-13.0-RELEASE clone from earlier, or if you want to clone from the release. I opted to clone from the release. Maybe one is a proper way, but at this time I can’t see a drawback to either way myself.

$ zfs clone zroot/jails/releases/13.0-RELEASE@p4 zroot/jails/templates/skeleton-13.0-RELEASE

Now we want to hollow out the skeleton we just made. In Clinta’s example, this is where I started to encounter issues, namely with /usr/local since it doesn’t seem you can mount /usr/local if /usr is nullfs’d…this could create a bunch of other problems too, namely, too much stuff is set up with nullfs mounts. The way that is shown by Michael W Lucas and the iocage people is more sensible in this case. We need to EMPTY the contents of some directories without deleting the directories. Note, some directories (such as the lib directories) have immutable files, so you’ll have to make the files mutable if you encounter the error:

$ rm -rf /jails/templates/skeleton-13.0-RELEASE/bin/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/boot/*
$ chflags -R noschg /jails/templates/skeleton-13.0-RELEASE/lib/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/lib/*
$ chflags -R noschg /jails/templates/skeleton-13.0-RELEASE/libexec/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/libexec/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/rescue/*
$ chflags -R noschg /jails/templates/skeleton-13.0-RELEASE/sbin/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/sbin/*
$ chflags -R noschg /jails/templates/skeleton-13.0-RELEASE/usr/bin/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/usr/bin/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/usr/include/*
$ chflags -R noschg /jails/templates/skeleton-13.0-RELEASE/usr/lib/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/usr/lib/*
$ chflags -R noschg /jails/templates/skeleton-13.0-RELEASE/usr/libexec/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/usr/libexec/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/usr/sbin/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/usr/share/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/usr/libdata/*
$ chflags -R noschg /jails/templates/skeleton-13.0-RELEASE/usr/lib32/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/usr/lib32/*

And if you run into the unable to delete issue, run: chflags -R noschg on that directory.

Now we are at the stage where we can create our jails. We will be snapshotting the skeleton from here on out when we want to have a new jail. So, lets snapshot the skeleton and call this new jail "testboxen_13-0".

$ zfs snapshot zroot/jails/templates/skeleton-13.0-RELEASE@skeleton
$ zfs clone zroot/jails/templates/skeleton-13.0-RELEASE@skeleton zroot/jails/jails/testboxen_13-0

Neat! It’s created. We can’t use it yet, but it’s there.

Templates

Here is a quick tidbit on templates…kind of a sidebar. Earlier we made that base-13.0-RELEASE clone and it’s not doing much. It’s the same as what’s in the release, as a matter of fact. I never utilized the templates feature in iocage, but reading Michael’s book tipped me off to how to do this for my own self. For my job I have roughly 10 jails doing the same thing. Do I want to go and update the same packages each time for each jail? They’re all running the same software, so why make 10 identical downloads? You can clone the release jail and make a base jail named, say, "nginx-13.0-RELEASE". You could go in that jail and install nginx and anything else each of that jail might need and use that nginx-filled template along with the aforementioned skeleton. If you do this however, note you will additionally have to nuke usr/local and var/db/pkg in the skeleton.

I would like to note it wasn’t until AFTER I set everything up for work that I messed up my implementation. So, for this section I’ll just make this mention here and suggest you check out Michael’s book, ch6, page 123.

FSTABs

Next, we must create our fstab files. This will tell the jail creation tools to use the nullfs mounts inside our skeletoned jail we just created. Here’s what it should look like for our "testboxen_13-0"

/jails/templates/base-13.0-RELEASE/bin                /jails/jails/testboxen_13-0/bin          nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/boot               /jails/jails/testboxen_13-0/boot         nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/lib                /jails/jails/testboxen_13-0/lib          nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/libexec            /jails/jails/testboxen_13-0/libexec      nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/rescue             /jails/jails/testboxen_13-0/rescue       nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/sbin               /jails/jails/testboxen_13-0/sbin         nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/usr/bin            /jails/jails/testboxen_13-0/usr/bin      nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/usr/include        /jails/jails/testboxen_13-0/usr/include  nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/usr/lib            /jails/jails/testboxen_13-0/usr/lib      nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/usr/libexec        /jails/jails/testboxen_13-0/usr/libexec  nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/usr/sbin           /jails/jails/testboxen_13-0/usr/sbin     nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/usr/share          /jails/jails/testboxen_13-0/usr/share    nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/usr/libdata        /jails/jails/testboxen_13-0/usr/libdata  nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/usr/lib32          /jails/jails/testboxen_13-0/usr/lib32    nullfs  ro      0       0

I like to keep these files in /jails/fstabs, so this file would be /jails/fstabs/testboxen_13-0.fstab.

The jail.conf File

Now lets make our jail.conf file. This will tell our system how to put together our jail. Here’s the contents of mine for this case. Note, you can copy/paste the testboxen_13-0 block over and over for each jail you make with its new name.

$jails="/jails/jails";

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

testboxen_13-0 {
        host.hostname = "testboxen_13-0";
        path = "$jails/testboxen_13-0";
        vnet;
        vnet.interface = "e0b_test1";
        exec.prestart += "sh -x /usr/local/bin/jib addm test1 jeth";
        exec.poststop += "/usr/local/bin/jib destroy test1";
        mount.fstab = "/jails/fstabs/testboxen_13-0.fstab";
}

Let’s break this down a little:

path - where the jail is located
vnet - specifying we are running a vnet jail
vnet.interface - the name of the interface inside the jail
exec.prestart/poststop - how we are going to create/destroy our interfaces
mount.fstab - the fstab file for our base jails

Neat. This is all done…but, this Still won’t work. Our host does not yet know how to use this stuff.

Host Setup

For the host, we need to setup a few things:

Enable the service
Get jib in the right place
Configure our interface (optional)

Enable the Service

sysrc jail_enable="YES"

Get jib in the Right Place

Either copy or symlink jib to /usr/local/bin

$ cp /usr/share/examples/jails/jib /usr/local/bin/jib

Configure Our Interface

This part is optional. I have an extra interface and I decided to give it a name. That is the "jeth" in the config above. It could be em0, igb0, re0, em1, whatever interface you want to use, don’t even need to rename it.

$ sysrc ifconfig_igb1_name="jeth"
$ sysrc ifconfig_jeth="up"

Since that will only work on the next boot, you can run ifconfig igb1 name jeth and ifconfig jeth up to get the interface up.

Okay, NOW our jail should start

Jails Needing Shared Memory

I ran into one instance where software needed shared memory (PostgreSQL). Because of that, I initially thought I needed to enable an insecure mode of sharing memory, but it turns out since, I think, FreeBSD 11.1 they added new security controls to create new shared memory namespaces to keep separate shared memory…pools? for each jail you set it up for. iocage did this out of the box for everything, but I find it’s probably only necessary on an as-needed basis. Basically, for each jail (or just your entire jail.conf file) add this line:

sysvshm = "new";

This is pretty well explained in the skyforge URL above. == Wrapping it Up

With everything in place, let’s start this jail up:

$ service jail start testboxen_13-0
$ jls
   JID  IP Address      Hostname                      Path
    1                  testboxen_13-0               /jails/jails/testboxen_13-0

HOORAY! But wait, there’s no networking. That’s because we need to have the jail do it for us. We could either put some of the exec stuff in the jail.conf, or pass the buck to the jail and put it in its rc.conf, which is what I will do. Note, with vnet jails you won’t see the ip address in the jls output. So, console into that jail:

jexec testboxen_13-0

Now lets get some stuff in our jail’s rc.conf. I’ll give it an address of 10.0.0.10/24, and a default router of 10.0.0.1

hostname="testboxen_13-0"
# note the name here matches what we put for vnet.interface
ifconfig_e0b_test1="10.0.0.10/24"
defaultrouter="10.0.0.1"

# Enable IPv6 if you want
#rtsold_enable="YES"
#ifconfig_e0b_test1_ipv6="inet6 accept_rtadv"

Cool. Now get out of that jail with an exit and restart it to ensure networking is good: service jail restart testboxen_13-0. You should be able to console back in and ping. It should be good! Now you have a fully functional jail. Lastly, if you want these to boot up, you can add it to the jail list:

sysrc jail_list+="testboxen_13-0"

Boom. Done. With all this out of the way, you can take all these same principles with the cloning of the skeleton, fstab, and jail.conf entry to create all the base jails you want!

Update 2022-01-20

I have seen others in places suggest the use of BastilleBSD for jail management too. I also think BastilleBSD is a good tool, so here’s a shoutout BastilleBSD and the people behind the project 🙂

https://bastillebsd.org/

Creating the Release#

Creating Our Skeleton#

Templates#

FSTABs#

The jail.conf File#

Host Setup#

Enable the Service#

Get jib in the Right Place#

Configure Our Interface#

Jails Needing Shared Memory#

Update 2022-01-20#