How is SmartOS So Far?

Right after I got setup, it didn’t take long for me to start digging into FMA. FMA was one of the selling points for me, since I thought it would be cool to get some insights into the hardware I was running. This turned out to be to my advantage.

I ran fmadm faulty to look at my system’s faults and found out there was a fault detected not long into running my server with SmartOS. What was cool was it showed me the path of what was faulty. It mostly went like:

chassis → cpu → motherboard → dimm0/dimm1

Something like that. The message showed that a feature in my memory was not working in 2 DIMM slots. I took out the 2 memory sticks in those slots and those messages went away after I cleared them. I still had faults, however. The message indicated that the MMU was failing. So, I found out that modern Intel processors have the MMU on the CPU die, so I replaced the processor with a "new" 12 core Intel Xeon and cleared the FMRI, and the fault message was gone!

I’m not sure if Linux or FreeBSD have anything like this. I have not encountered errors in my hardware like this before, so it was cool to see where the issue was and manage it.

There’s an existing module available in Ansible-galaxy community. I’ve been using it to deploy new zones to my server. I use it for my one-off zones and ones I repeatably deploy for testing. It isn’t as flexible as I’d like though. There’s still some manifest settings I would like, such as owner_uuid and flexible_disk_size for bhyve zones. They don’t exist, however. If I get around to it, I may modify the module to include more options. Or, just switch to doing ansible.builtin.command and run vmadm straight with JSON manifests for each zone.

owner_uuid is more of a variable for Triton Datacenter, but I like to use it because I run my own zones, some zones for friends, and work. I set a uuid so I can nicely sort out these zones when listing them.

I have a JSON file called /usbkey/owner.json.

In my bashrc file I have a function to make it easy to sort by UUID

So I can run vml me or vml friend1 and get that subset of zones.

I’ve been enjoying native SmartOS zones so far. I use them as much as possible. I’m an OpenBSD nut, so I have quite a number of OpenBSD VMs. I also have a few Linux-only applications. However, the rest I have are native zones. So far, my regular setup is like this:

I have a bunch of other small stuff too.

What I like about the native zones are the package availability from pkgsrc and how I can mix and match php versions and postgresql versions for Nextcloud. I also like how updating the global zone updates the SmartOS zones too. That make management much easier. With FreeBSD I’d either have to freebsd-update a bunch of jails or weave my way around thin jails and the management weirdness that comes with that. Using native zones lets me get easy access to ZFS tooling and such too, though, those tools are available and mostly functional in an LX zone too. I also get easy access to running SMB and NFS servers too with delegated datasets. Because of the easy management and performance, I run native zones as much as I can.

I haven’t gotten much into this, but I’ve been messing around a little with fwadm to set firewall rules for zones. It is pretty easy to create new rules and apply tags to zones to apply rules.

I came across the etherstub last week, but I have a concept that I want to do for my zones that require reverse proxies. The etherstub is essentially a virtual switch.

I already run haproxy on my router to do TLS passthrough for my internal services that use ports 80/443. However, I don’t like this setup as much. I woulnd’t mind doing TLS termination on haproxy and let each application know it is being proxied. In some cases I have 2 proxies running because of this setup. Not very efficient. I also don’t like the idea of letting unencrypetd traffic run over my local network. This is where etherstubs come into play.

I can create an etherstub and then create new interfaces with IPs on my zones so they can communicate locally over the etherstub. My goal then is to have haproxy do TLS termination up front and proxy locally over an etherstub. That will be my next upcoming project to take some load off my weak router and hopefully make my setups a bit simpler.

One issue I had with FreeBSD was the lack of built-in tooling to deploy jails and bhyve virtual machines. I really do not like the default jail.conf method of managing jails. I also don’t like relying on software in ports, namely, vm-bhyve (which is great software) to manage jails. SmartOS makes deploying all zone types (native, lx, bhyve) painless. There’s also a web GUI I can setup too to manage my zones. Since zone management has been such an easy job for me, I find myself running more on my server. I only have so much mental bandwidth before I say "okay, no more. I am out of time". However, I have been able to add projects and I have had to touch my server less. YMMV, but that’s been my experience these last 6 months.

Another fun fact: later in 2024 there was a bhyve vuln that allowed a guest to execute code on the host. Because of the security design built into illumos zones, the bhyve vulnerability was fully mitigated. Maybe this could have been mitigated on FreeBSD too if bhyve VMs were always deployed in a jail by default.

There isn’t much of that. I grabbed zrepl from github and set up my own SMF manifest to run backups. I haven’t had to do a whole lot otherwise. I have some monitoring set, but otherwise I flash a new image to a flash drive, poweroff, swap flash drives, start up, and move on. Going from, say, FreeBSD 14.0 → 14.1 would have involved all the host update processes, zpool upgrades and praying I don’t screw up boot code, upgrading all my jails.

I know I am shooting fire at FreeBSD. I don’t hate FreeBSD, having been a FreeBSD user since 2017 and still actively maintain FreeBSD boxes today. However, coming from that background, these are things that I have found that make quality of life as an admin much better, and I think, if any FreeBSD devs ever read this, would be some very great changes to FreeBSD that would make sysadmin life much nicer.

There’s more I could write over time. I will likely add bits here and there and document more of the fun things I do on my virtualization host. These are my thoughts 6 months later running SmartOS. I’m a very happy user, performance is good, management is very nice, and I am happy that I can trust this OS to run my servers. I don’t miss the OpenZFS features that I lost either.