Courtnix Blog

Migrating My Email to Stalwart

Courtney — Tue, 23 Dec 2025 15:00:00 PST

I have been self hosting my own email for quite some time now on OpenBSD. In the beginning, I had a little trouble with getting out of peoples spam boxes, but eventually my mail server became my least troublesome server since there has only been 2 people on the server, not much room to get on a spam list.

My configuration has been to use OpenSMTPd, dovecot, and rspamd to handle everything. My server was never quite complete, admittedly, since I never got spam filtering right. I never got spam though until I put one alias out on a website. I expected spam, and never got much. The amount was so little that when I did get spam to that recipient, I just marked it as spam (which did nothing) and moved on. Maybe I was getting hit with spam a lot, but I had some basic filters already with OpenSMTPd and I imagine that handled a lot of the real crud. It was smooth sailing.

Why The Switch?

My real pain points were with getting my spam filter set, which, I just needed to add spam headers at the right place. I also was seeing that Dovecot has been in the works of removing features from their open source version and putting them behind a "Pro" plan. When I saw that dovecot is in the market of ensh**ification, I knew it was time to move on. I don’t mind the FOSS core, proprietary extensions model. I don’t like when a project starts rug pulling features. Looking for an alternative made me realize some of these alternatives also support CalDAV/CardDAV, which I am interested in, and will touch on that later as a side bar.

Stalwart was an option I stumbled upon some time ago and forgot about. I think because there was no OpenBSD support. It did land in ports earlier this year, then again when I was ready to kick the tires I saw it got updated to the latest (at the time) v0.14.1. What got me interested was the CalDAV/CardDAV support, JMAP, and all the batteries included to do email. Bonus points to OIDC, which I very crappily have set up but it would be really cool to integrate. I also really like the admin UI. I am not much of a point and click guy, but this is nice. I also like that there are all sorts of backends to scale up a Stalwart mail cluster. I don’t have that need right now though, so I am just using RocksDB.

The Switch

I wanted to detach my email domain from my daily sending domains. So, I created a new domain name and spun up my server under that. I really wanted to try out the Enterprise features, mostly because I like pretty graphs and metrics and wanted to look into the tenant feature. So, I created tenants, domains, user accounts, groups, etc. I tried to get close to what I had on my previous server, but with changes to make up for bad decisions.

I also had all the DNS records and such figured out for my sending domain. Stalwart has a very handy page under domains that shows you all the DNS records you will ever want. I let that run for a while but with no real email being sent. None of my daily accounts had been moved, but I was ready to start getting things moved over.

The Rush

So, I only thought I would start with one domain today, and figured I would start with the email accounts for the previous sending domain. That wasn’t all that smart as I had my other domain pointing to that for the mail records. So heck, I was here now and just started the whole migration. Was this the right path? Probably not.

Migration Steps

Here are the steps I took. I have very few accounts, so only me and one other would notice the disruption, and I know they wouldn’t notice.

Note that prior to this, I lowered TTLs to 5 minutes on all necessary DNS records. That way stuff wouldn’t be going to the old server potentially for many days.

Made sure all the DKIM, DMARC, SPF records and all were correct for each domain
Updated MX records with priority to the old server and the new one (with lower priority)
used imapsync to sync mailboxes between servers.
1. This only worked because I had credentials to all the accounts
Removed old MX record priority and kept only the new one
Did a check with imapsync again to look for mail that went to the old server, this is what I wanted:
1. "The sync looks good, all 1410 identified messages in host1 are on host2."
Once all boxes were synced, I started bumping TTLs up higher on my DNS records

Looking Back

This was majorly a thought experiment as I have intentions on doing this on a larger deployment and with users that really, heavily rely on email. I think this is something that will take more user interaction to do neatly for each inbox in the environment I am in. I will also refine my steps and timing. I will cross that bridge when I get there. However, this was a very helpful test for me.

I am very happy with my switch. I like how it is easier to manage accounts and domains. I like the built-in features Stalwart has. I don’t mind touching other servers, but this is just a nice setup and I look forward to the 1.0 realease that is incoming. I think Stalwart has a bright future. Now I just hope the clients will catch up and we get some JMAP support coming along. ;)

Last thing I am working on is Sieve scripts. Dovecot would do INBOX.Example, but apparenty Stalwart does syntax INBOX/Example. Still trying to confirm that, but so far seemingly my inbox isn’t being sorted right..I’ll get there.

Final Thoughts (Sidebar)

Earlier I mentioned my interest in CalDAV/CardDAV. Since people rely on contact syncing a lot for sending email, it makes sense to make this a part of Stalwart. Also, I have entered into an environment where calendar sharing and invites is very heavily used. It is handy to do. So it also makes sense to have Stalwart handle calendar since invites are often sent via email. Nothing against Nextcloud, but I wouldn’t mind moving calendar and contacts off Nextcloud. Some combo of Apple and Nextcloud has issues with syncing contacts, and I would be okay with putting calendar/contacts in a more logical place.

One Problem

I was initially drafting up doing that, but I ran into a problem with calendar sharing. Stalwart DOES support sharing calendars via CalDAV for individual users, but practically no clients do that. Not on iOS (but I hear you can on macOS?), nor Thunderbird, anything really. There are a couple but it is just well out of the reach of the users I am targeting on my own server. I can do a shared calendar with groups and all, but because of lack of client support, my users cannot share without admin intervention. Nextcloud at least makes this easy to share calendars with other Nextcloud users. In my realm of use, this may not matter an inch. But at scale, this won’t work well. Since there’s word about more JMAP clients coming around, maybe we will see that change. JMAP is pretty new and not well adopted yet. It could provide for a better ecosystem, at least I hope.

For now, I will play with what I have, and hope I don’t cause too much trouble.

Disable Fish Shell Greeting

Courtney — Mon, 22 Dec 2025 16:30:00 PST

I have been using the fish shell recently. I have it tweaked some, and one day (probably after an update), I started to get this message:

Welcome to fish, the friendly interactive shell
Type help for instructions on how to use fish

> ~

Not sure why I need a greeting every time, don’t want it. So I found two ways to disable this functionality:

1. Create an Empty Function File

The method I chose was to create an empty file at $HOME/.config/fish/functions/fish_greeting.fish.

2. Update config.fish

Adding this line to your config.fish file will suppress the message:

set fish_greeting

That’s all you should need to do. Enjoy the quieter prompt.

My SmartOS Live Demo Failure, and Success

Courtney — Thu, 21 Aug 2025 18:00:00 PDT

This summer, I was tasked with a workshop: spend roughly 8 hours of the day teaching Ansible. The workshop went great. I got good feedback to improve and good remarks. I have grown to love to teach, and I always appreciate what people think about how I do. As a part of that workshop, I wanted to have VMs available for any of the people there that couldn’t virtualize on their own computers. The group was small, with 10 members in the workshop so I felt I could take my extra Thinkpad T480 and throw SmartOS on there and let that be the virtualization platform. I configured it at home for the trip.

My first step in the workshop was to show the attendees a working playbook that they could copy/paste and then run on my server to spin up virtual machines. I was ready, flew to the conference, and the next day was showtime.

The Troubles Begin

Before the conference, I requested a block of IPs on the same network so that we could all be on the same network as the virtualization host. I also needed an ethernet port. Some of the network issues this day included the port in the wall not being connected to the right port on the patch panel and switch so there was no access, and then come to find out the cable I brought decided to die along the way. Thankfully there were extras, but it was fun to find these things out as the workshop was starting. Additionally, The DHCP addresses were not correct, so static IPs were needed.

Showing My Ignorance

As I was trying to figure these things out, SmartOS was spending an eternity booting. I could not get the system to boot, only single user mode! I thought maybe I had dorked something up, and I didn’t know what the fix was. So I figured the best thing was to reinstall the OS. What I later found out is SmartOS will get stuck booting for an eternity if it is configured to get a DHCP address but there is no DHCP answer. This situation left me with a questionable network, no hypervisor, and the clock was ticking, it was my teaching time.

The Resolution

With my Framework laptop handy, I was able to confirm that the networking was complete and functional. By then, it was too late. I had already chosen the path of destruction. I was already reinstalling SmartOS. The bad news though was because there existed a previous zpool on the disk, I had to dd the beginning/end of the disk to reinstall. I was able to wipe the beginning/end of the disk at this point and was plugging through the installer. As I was doing that, I was teaching the workshop, getting up to the point of running our first playbook. In about 10 minutes or so after starting the SmartOS installer, I had a running system! I knew which Ubuntu 22.04 image I wanted to pull down, so I grabbed it with imgadm and was then able to tell the users to add their ssh pubkeys to the authorized_keys file on the SmartOS server. They ran their first playbook, which contained community.general.vmadm and installed the new Ubuntu 22.04 BHYVE zones. Great success!

The Failure and Success

This could be counted as a failure if this was any other operating system. If I had to nuke Debian or FreeBSD or something else, it would take a lot more for me to get from 0 to deploying VMs. I didn’t have to nuke my install, but in a hurry (and at the time not knowing the best way to fix the dhcp issue), I knew my setup was simple and would not require much effort to put things back together. As for the success, things timed out just right despite reinstalling mid-presentation and everything ran smoothly from there. I told the attendees to update their VM, take a snapshot, and then from there if there was ever a need, it was easy for us to rollback and fix mistakes. It was daring for me to take the OS out on the road and hook it up to an unfamiliar network, but all was well and the workshop was a success.

Core Ultra Framework Laptop and OpenBSD

Courtney — Thu, 30 Jan 2025 16:45:00 PST

Software

I started with a Fedora iso to update firmware stuff easily
tried with the efi updater but it would not work

OpenBSD installer

ure0 ethernet card works iwx0 wireless works

Post-Install

Here is a list of what does/doesn’t work:

audio - yes
battery status - yes
bluetooth - no
keyboard backlight - yes
ethernet - yes
wireless - yes
ssd - yes
suspend/resume - mostly
touchpad - yes
video - yes
webcam - haven’t tried
microphone - also haven’t tried
fingerprint reader - not even sure what I’d use to test this
graphics - 6/10
2.8k display - good

I assume the webcam, mic, and fingerprint reader still work since they have worked on older Framework models according to other OpenBSD blog posts about OpenBSD on the Framework laptop

Overall Experience

I’ve been using the new Intel Core Ultra Framework laptop for a few weeks by now, and so far it has been both better and worse than I thought.

What surprised me was I was able to get the Anker 553 USB-C hub and have everything work off of it. I have 2 HDMI monitors plugged in, power delivery, a USB switcher plugged in, ethernet, and it all works. All off a single type C port on the laptop. The downside is, unplugging this doesn’t result in friendly experiences. I have to hard power off the laptop if I unplug it from the laptop. Not the biggest deal since my setup requires I reload some stuff anyway when I go from many monitors to just the laptop one.

The other downside right now, which may more be the Intel driver’s fault, is that I get very bizarre graphical glitches, mostly when I have multi monitors hooked up. I noticed that graphical issues with this Intel iGPU is nothing new, so I am hopeful that the next driver pull (kernel 6.12?) will have fixes for all these oddities.

Still a little bummed that I have to deal with weird xhci interrupts really messing with my USB input/output. I can lose keypresses sometimes and audio will cut out on occasion. Scrolling can be laggy. All of this is over that single xhci interface. It’s all going over xhci1, so I may see if I can get some of that stuff plugged into xhci0 (assuming that’s the right side of the laptop) and see if spreading the load out will help. Maybe also since video is pretty demanding it is really making the interrupt count very high.

Despite all that though, I am pleased that everything is working, even if some of it isn’t providing me the best experience. I found it is best to keep the apm setting at auto. When driving my displays on performance mode, the Framework gets quite toasty and the fan runs a lot. Auto is actually quite good. And the laptop stays around 65C with fans off most of the time on auto.

Battery life has been good? I don’t expect good battery life if it isn’t macOS or perhaps even Windows. I haven’t done a proper test but I think I could be getting 6 hours of battery off this with regular use.

Sleep works some times, and doesn’t work other times (like me).

I’ve tried Wayland some and was happy that Wayland could handle the hidpi display very well with my 1440p and 1080p display. But Wayland on OpenBSD (maybe Linux too, haven’t tried) just isn’t there and I was ready to get back on Xenocara after a day of using it. I just keep my laptop display off and have the dpi at 96 and use my 2 monitors. That makes the laptop display unusable since the text is SO tiny unless the dpi is set to 192.

Cool Battery Feature

The Framework lets you set the max charge capacity. Since I have this as a desktop replacement, I set the max charge to 80% and it works. What is really cool too is if I had it at 100% before and set it to 80%, it will drain my battery back down to 80%. Very nice. Doing this within OpenBSD doesn’t work. I understand it is very vendor-specific. So it’s something that Framework would have to provide some sort of driver compatibility for? Either way, I am perfectly fine having that feature in the BIOS.

Conclusion

Overall, I am actually very happy with this laptop. I don’t regret my choice. It does sadden me that I am putting up with these audio stutters again. I solved that issue on my desktop by having my USB DAC connected via SPDIF and that has worked incredibly well. Maybe I’ll get tipped over the edge with the stuttering to the point where I say enough is enough since I am listening to something all day, every day. But I really do believe that I have the most easy, comfy, and manageable setup having OpenBSD on my laptop all other things considered. I just have to take some occasional copium with the audio stutters.

How is SmartOS So Far?

Courtney — Thu, 02 Jan 2025 20:30:00 PST

In an earlier post, I wrote on my migration to SmartOS. It has been about 6 months now and I thought I would write up some on my experience so far.

Fault Management Architecture (FMA)

Right after I got setup, it didn’t take long for me to start digging into FMA. FMA was one of the selling points for me, since I thought it would be cool to get some insights into the hardware I was running. This turned out to be to my advantage.

I ran fmadm faulty to look at my system’s faults and found out there was a fault detected not long into running my server with SmartOS. What was cool was it showed me the path of what was faulty. It mostly went like:

chassis → cpu → motherboard → dimm0/dimm1

Something like that. The message showed that a feature in my memory was not working in 2 DIMM slots. I took out the 2 memory sticks in those slots and those messages went away after I cleared them. I still had faults, however. The message indicated that the MMU was failing. So, I found out that modern Intel processors have the MMU on the CPU die, so I replaced the processor with a "new" 12 core Intel Xeon and cleared the FMRI, and the fault message was gone!

I’m not sure if Linux or FreeBSD have anything like this. I have not encountered errors in my hardware like this before, so it was cool to see where the issue was and manage it.

Ansible Usage

There’s an existing module available in Ansible-galaxy community. I’ve been using it to deploy new zones to my server. I use it for my one-off zones and ones I repeatably deploy for testing. It isn’t as flexible as I’d like though. There’s still some manifest settings I would like, such as owner_uuid and flexible_disk_size for bhyve zones. They don’t exist, however. If I get around to it, I may modify the module to include more options. Or, just switch to doing ansible.builtin.command and run vmadm straight with JSON manifests for each zone.

Owner UUID Fun

owner_uuid is more of a variable for Triton Datacenter, but I like to use it because I run my own zones, some zones for friends, and work. I set a uuid so I can nicely sort out these zones when listing them.

I have a JSON file called /usbkey/owner.json.

{
        "me": "00000000-0000-0000-0000-000000000000",
        "work": "632ce8ef-ea4b-4d38-bd94-5ee1da596b28",
        "friend1": "1bb332ea-5d5c-448a-b8ad-c6bf8322d798",
        "friend2": "2704b08b-772e-49b4-82d9-d8bdacee3f30"
}

In my bashrc file I have a function to make it easy to sort by UUID

function vml () {
        if [ -z "$1" ]; then
                vmadm list -s alias
        else
                vmadm list -s alias owner_uuid="$(cat $zone_owners | json $1)"
        fi
}

So I can run vml me or vml friend1 and get that subset of zones.

How Are Native Zones?

I’ve been enjoying native SmartOS zones so far. I use them as much as possible. I’m an OpenBSD nut, so I have quite a number of OpenBSD VMs. I also have a few Linux-only applications. However, the rest I have are native zones. So far, my regular setup is like this:

Ansible - SmartOS zone (WIP actually, migrating from OpenBSD to SmartOS)
SMB/storage - SmartOS zone
Collabora CODE - Debian 12 VM. LX zone had an issue
factorio - Alma Linux 9 LX zone
gitea - SmartOS zone
Vault - OpenBSD
Immich - Debian 12 VM since it’s a docker-composed program
Jellyfin - Alpine Linux LX zone
Synapse matrix - OpenBSD
Nextcloud - SmartOS zone

I have a bunch of other small stuff too.

What I like about the native zones are the package availability from pkgsrc and how I can mix and match php versions and postgresql versions for Nextcloud. I also like how updating the global zone updates the SmartOS zones too. That make management much easier. With FreeBSD I’d either have to freebsd-update a bunch of jails or weave my way around thin jails and the management weirdness that comes with that. Using native zones lets me get easy access to ZFS tooling and such too, though, those tools are available and mostly functional in an LX zone too. I also get easy access to running SMB and NFS servers too with delegated datasets. Because of the easy management and performance, I run native zones as much as I can.

fwadm

I haven’t gotten much into this, but I’ve been messing around a little with fwadm to set firewall rules for zones. It is pretty easy to create new rules and apply tags to zones to apply rules.

etherstub

I came across the etherstub last week, but I have a concept that I want to do for my zones that require reverse proxies. The etherstub is essentially a virtual switch.

I already run haproxy on my router to do TLS passthrough for my internal services that use ports 80/443. However, I don’t like this setup as much. I woulnd’t mind doing TLS termination on haproxy and let each application know it is being proxied. In some cases I have 2 proxies running because of this setup. Not very efficient. I also don’t like the idea of letting unencrypetd traffic run over my local network. This is where etherstubs come into play.

I can create an etherstub and then create new interfaces with IPs on my zones so they can communicate locally over the etherstub. My goal then is to have haproxy do TLS termination up front and proxy locally over an etherstub. That will be my next upcoming project to take some load off my weak router and hopefully make my setups a bit simpler.

Deploying zones/VMs is Easy

One issue I had with FreeBSD was the lack of built-in tooling to deploy jails and bhyve virtual machines. I really do not like the default jail.conf method of managing jails. I also don’t like relying on software in ports, namely, vm-bhyve (which is great software) to manage jails. SmartOS makes deploying all zone types (native, lx, bhyve) painless. There’s also a web GUI I can setup too to manage my zones. Since zone management has been such an easy job for me, I find myself running more on my server. I only have so much mental bandwidth before I say "okay, no more. I am out of time". However, I have been able to add projects and I have had to touch my server less. YMMV, but that’s been my experience these last 6 months.

Another fun fact: later in 2024 there was a bhyve vuln that allowed a guest to execute code on the host. Because of the security design built into illumos zones, the bhyve vulnerability was fully mitigated. Maybe this could have been mitigated on FreeBSD too if bhyve VMs were always deployed in a jail by default.

Host Management

There isn’t much of that. I grabbed zrepl from github and set up my own SMF manifest to run backups. I haven’t had to do a whole lot otherwise. I have some monitoring set, but otherwise I flash a new image to a flash drive, poweroff, swap flash drives, start up, and move on. Going from, say, FreeBSD 14.0 → 14.1 would have involved all the host update processes, zpool upgrades and praying I don’t screw up boot code, upgrading all my jails.

Shots Fired

I know I am shooting fire at FreeBSD. I don’t hate FreeBSD, having been a FreeBSD user since 2017 and still actively maintain FreeBSD boxes today. However, coming from that background, these are things that I have found that make quality of life as an admin much better, and I think, if any FreeBSD devs ever read this, would be some very great changes to FreeBSD that would make sysadmin life much nicer.

bhyve VMs always jailed
easier jail management
better tooling in base for jail/bhyve management

Conclusion

There’s more I could write over time. I will likely add bits here and there and document more of the fun things I do on my virtualization host. These are my thoughts 6 months later running SmartOS. I’m a very happy user, performance is good, management is very nice, and I am happy that I can trust this OS to run my servers. I don’t miss the OpenZFS features that I lost either.

I Fixed My Blog...Again

Courtney — Tue, 31 Dec 2024 06:00:00 PST

I’ve had this blog go through some different softwares.

First time I did it all in raw AsciiDoc and used the asciidoctor tool to export it all and one of their stock css styles. Worked fine, but I wanted more utility. I went onto Hugo, but had some issues there, and now am on 11ty.

Hugo Problems

I like writing in AsciiDoc. I’ve tried Markdown, I just can’t get used to it. Its feature set is too narrow. The issue I had with Hugo was that eventually Hugo made changes to syntax highlighting which broke my theming. Then the theme I was using broke. All my code blocks were messed up and unreadable. I was getting tired of the constant moving target of the theme and Hugo, so I came across 11ty.

What I like About 11ty

I had already made my own static site generator for my church’s website with a basic Python script, Jinja2 templates, and Bootstrap. The theming is not easy for me as someone who doesn’t work with CSS or Javascript at all. However, the site turned out really nice! Well, with my blog being broken and needing to add some stuff to the church site, I needed more flexibility. So, I picked 11ty. What I like is there’s a lot of different ways to build a site, data can be fetched and manipulated with different modules and such, and best of all, there’s a community module for AsciiDoc!

I was able to move all my Hugo blog pages into 11ty without having to modify them any, the rest was just making the site look nice and doing the pagination. I lost my easy copy button, but I’ll add that back eventually.

What I like now is that I have the flexibility to do my own theming on my own time, and upgrade versions of 11ty on my own time. It’s just static pages, I don’t need constant moving targets for themes and site generators. I don’t want to futz with that stuff. 11ty so far seems it will fit the bill. Time will tell. But for now, I like that it is easy to just write again and hit npm run build (which I put in the packages.json file) and get the directory slapped up on my server.

Really, the most I had to do was make 2 templates for my main pages and my blog pages (even the latter wasn’t necessary), then do some Liquid templating for my few page types, and the rest is just writing my content in AsciiDoc.

What’s Next

Now that this is all back, I’m going to get back into writing. I have some drafts I want to finish, some I want to toss, and some things I need to catch up on, like some more of my SmartOS stuff.

Fix MTR Report in Ubuntu 24.04

Courtney — Thu, 29 Aug 2024 17:45:00 PDT

TL;DR

Run this:

#!/usr/bin/env sh

sed -i 's/Types: deb/Types: deb deb-src/g' /etc/apt/sources.list.d/ubuntu.sources
cd /tmp
apt-get update
apt-get install -y dpkg-dev
apt-get source mtr
apt-get build-dep mtr
cd mtr-0.95

# Get patch
curl -o debian/patches/5908af4c19188cb17b62f23368b6ef462831a0cb.patch https://github.com/traviscross/mtr/commit/5908af4c19188cb17b62f23368b6ef462831a0cb.patch

# Apply patch
patch -p1 < debian/patches/5908af4c19188cb17b62f23368b6ef462831a0cb.patch

# Build package
dpkg-buildpackage -rfakeroot -uc -b

# Install package
find /tmp -name "*mtr-tiny*.deb" -exec dpkg -i {} \;

The Fix

I’ve been meaning to upgrade some computers to Ubuntu 24.04 LTS, but have run into a problem with the MTR package. I frequently use the -r flag to generate reports. However, there is a known issue with the report feature in Ubuntu 24.04 due to Ubuntu enabling "string_fortified" by default. The good news is the fix is pretty simple, as seen in the patch below:

From 5908af4c19188cb17b62f23368b6ef462831a0cb Mon Sep 17 00:00:00 2001
From: Marcus Meissner <meissner@suse.de>
Date: Tue, 11 Apr 2023 16:05:36 +0200
Subject: [PATCH] fixed the sizes passed into snprintf

---
 ui/report.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/ui/report.c b/ui/report.c
index 93014808..61f2c74b 100644
--- a/ui/report.c
+++ b/ui/report.c
@@ -140,7 +140,7 @@ void report_close(
             continue;

         snprintf(fmt, sizeof(fmt), "%%%ds", data_fields[j].length);
-        snprintf(buf + len, sizeof(buf), fmt, data_fields[j].title);
+        snprintf(buf + len, sizeof(buf) - len, fmt, data_fields[j].title);
         len += data_fields[j].length;
     }
     printf("%s\n", buf);
@@ -172,10 +172,10 @@ void report_close(

             /* 1000.0 is a temporary hack for stats usec to ms, impacted net_loss. */
             if (strchr(data_fields[j].format, 'f')) {
-                snprintf(buf + len, sizeof(buf), data_fields[j].format,
+                snprintf(buf + len, sizeof(buf) - len, data_fields[j].format,
                          data_fields[j].net_xxx(at) / 1000.0);
             } else {
-                snprintf(buf + len, sizeof(buf), data_fields[j].format,
+                snprintf(buf + len, sizeof(buf) - len, data_fields[j].format,
                          data_fields[j].net_xxx(at));
             }
             len += data_fields[j].length;

And the patch was committed here. Since the patch was back in April 11, 2023, I was hoping that after a bug report being filed on 2024-07-04 on launchpad that it would hopefully make its way to 24.04.1. Sadly, it didn’t. So, for anyone who wants to patch this for themselves, here is how.

Process

First, you want to update /etc/apt/sources.list.d/ubuntu.sources and include deb-src in the "Types" lines to look like so:

Types: deb deb-src

Next, we need to install dpkg-dev

cd /tmp; apt-get update; apt-get install -y dpkg-dev

Next, you can get the source for mtr and get the build dependencies for it

apt-get source mtr; apt-get build-dep mtr

Next, I fetched the patch and put it in the debian/patches dir, you can put it anywhere you like

cd mtr-0.95; curl -o debian/patches/5908af4c19188cb17b62f23368b6ef462831a0cb.patch https://github.com/traviscross/mtr/commit/5908af4c19188cb17b62f23368b6ef462831a0cb.patch

When you get the source, the patches get applied too, so you need to apply the patch yourself

patch -p1 < debian/patches/5908af4c19188cb17b62f23368b6ef462831a0cb.patch

With the patch applied, build the package with this command:

dpkg-buildpackage -rfakeroot -uc -b

Finally, install the package;

dpkg -i ./mtr-tiny_0.95-1.1build2_amd64.deb

I install the mtr-tiny package because I don’t need the x11 version, but it is also there.

My OpenBSD Desktop Revisited

Courtney — Tue, 20 Aug 2024 19:30:00 PDT

In May 2022 I wrote about switching from FreeBSD to OpenBSD. Today I want to revisit that post, as a lot has changed since then in OpenBSD land.

The Hardware

My hardware has changed since:

Ryzen 5800X
Some B450 motherboard
32GB DDR4 at 2666MHz
OS Drive: 500GB WD Black SN750
MSI RX 6700XT
ZSA Ergodox EZ keyboard
Ploopy Adept trackball
AudioEngine D1 DAC/amp

Environment/Software

i3 (i3bar + i3status)
Nextcloud client
keepassxc
neovim (also dabbling with Helix as I write this)
Firefox
Audacious music player - Strawberry no longer compiles, and same with Clementine

First Impressions Revisited

When I was running FreeBSD, my Ethernet port didn’t work out of the box, my graphics card was not supported, and U2F was not supported in base. I can’t confirm if that is still true today, but I do know now that my graphics card is supported with the latest drm-kmod package.

My Elecom HUGE Problem

I stated that not all my buttons worked on the Elecom. I found a solution by editing the /etc/X11/xorg.conf file and adding

Section "InputClass"
        Identifier "mouse driver"
        MatchIsPointer "on"
        Driver "mouse"
EndSection

I still have that with my Ploopy trackball. Would it all work the same without it? Maybe, haven’t bothered to try.

FreeBSD Revisited

My experience with what I stated then is outdated. I hear that Wifi with Intel chips is getting better. Maybe it is catching up to OpenBSD? On OpenBSD with an Intel AC7265(?) I was able to get 400x400Mbps. I don’t know if I was on 2.4GHz or 5GHz, but at any rate, that was totally acceptable to me.

I also hear there’s been great strides made with USB audio and disconnecting devices. If so, that is also fantastic as it was such a pain if I accidentally disconnected my DAC and found it being almost easier to reboot my computer than do the dance to get everything back in order.

OpenBSD Experience Revisited

USB Audio

When I started using OpenBSD, USB audio was probably worse than it was on FreeBSD. The constant stuttering was so bad that I would have to about every 30 minutes kill Firefox to get my audio in order.

Eventually I got an AudioEngine D1 anticipating using the SPDIF port on it. But to my dismay, SPDIF on my system wasn’t supported by OpenBSD! BUT, switching devices did really help my problem. Instead of restarting Firefox every 30 minutes, it was more like twice a day. Still don’t understand that one. Even better, over time I only had to restart Firefox once a day or not at all in a day to solve the stuttering audio issue.

Even better better, some kind soul submitted updates to OpenBSD that supported the sound device on my motherboard! So now I have my DAC doing audio through the SPDIF port on my motherboard and it has been smooth sailing.

Firefox

These days the Download button in the file manager in Firefox actually takes me to the Downloads folder. Previously, I had to train it at first to remember. From time to time, Firefox may crash when trying to upload/download a file. It can be very frustrating. But I assume there’s some sort of pledge or unveil violations going on and assume that these security controls are doing their job.

keepassxc-proxy

All works great. Though I did eventually stop and switched to hosting Vaultwarden and using the browser extension. I still use KeePassXC for a lot of credentials, ones I don’t want in Vaultwarden.

Erlang

Works! The port maintainer has chosen to support multiple versions, so currently there’s erlang25, erlang26, and erlang27 in ports and I’d have to go through the steps of symlinking a bunch of stuff so I didn’t have to type erl27 and stuff like that. So I just download the tarball, set my own feature flags and build and make install the package myself. Overall, I don’t mind this.

hostname.if for Wireguard Clients

I still use wg-quick. I think there’s a way to not autostart the interface, but I prefer my method of having an up/down script to wrap wg-quick.

sndiod…is Great!

My issues have been solved here, I like using sndiod. I’m still not super used to the different tools, but the man pages are helpful, sndio is pretty straightforward. I’ve gotten sndiod to work well with my USB microphones, minus one I mentioned before. Oh well, now I hardly use a microphone.

One issue I have encountered though, not sure if it is sndio or usb related on OpenBSD, but I tried to record some audio with aucat in over USB and there was a constant audible tick in the recording. This problem did not happen on Linux where I also tested for this problem.

Overall Thoughts

OpenBSD is still slow compared to FreeBSD. But I find every release something gets better and better in OpenBSD. New stuff supported, new security features, better hardware support, performance gains here and there, it has been a fun 2 years of using OpenBSD on my computers.

I stated previously that I would likely switch back to FreeBSD on my desktop once my graphics card was supported. At this point, I am very happy with OpenBSD and will not. But, I will revisit FreeBSD on the desktop again some time again. I have accumulated extra SSDs so maybe it’s time for a triple boot and have some fun!

I see both FreeBSD and OpenBSD improving release after release and that is exciting to me. 2 great operating systems that deserve much love. In the end, I don’t regret moving my desktop to OpenBSD. Maybe sometimes I miss a package that might be on Linux or FreeBSD, but I haven’t missed anything so much that I feel like I have to abandon OpenBSD as my daily driver, and it will be here to stay on my systems.

Complete SmartOS Migration

Courtney — Tue, 20 Aug 2024 19:00:00 PDT

Today I continue my SmartOS saga. Actually, it’s done! I have SmartOS on my main server. It was quite a task. In addition to my dad’s server being replaced with SmartOS in April (which has been running very well), I have yet to get SmartOS on my friend’s server. Previously, I said I would probably install SmartOS on a friend’s box before mine, but events lead me to get ahead on my box first.

Recap: Why The Switch?

I’ve originally been using FreeBSD as my main server OS for 7 years. It has been great, reliable, never gave me an issue, and performant. My first motivation is because this is a home lab server and I like to try and learn new things. SmartOS has some overlap in features that I like:

ZFS
bhyve
Same bootloader as FreeBSD
zones (like jails, but not entirely)

The other reason has to do with a couple shortcomings in FreeBSD that SmartOS very uniquely solves:

VM images - native OS zones, Linux (LX) branded zones, a variety of Linux VMs, and I can still make my own from scratch
Unified tooling for the sysadmin (vmadm, imgadm, zoneadm, fwadm, etc.)
JSON manifests to create zones easily from images
Available Ansible module for zone management
It’s easier to wrap my mind around firewalls for each zone with fwadm and tags

Some of these things I could solve in ways with some third party tools and build some of my own stuff, but as I get busier with life, I sometimes need something that I can do fast or with minimal thought. SmartOS makes it a bit easier to automate some of the stuff that I do with my server.

The Long-winded Process

Since I wasn’t familiar with SmartOS and wanted to learn and kick the tires, I pulled out the only extra computer I had, which was an 8GB RAM Intel i5 4570 Thinkcentre. After learning, I decided it was time and I was comfortable enough to get the migration on. It was quite the journey learning and starting to move my virtual machines over to this desktop initially. I only had a single 4TB NAS drive and the amount of data I have on my server is just over 4TB. So I ended up removing some extra data that was unneeded and planned to move another chunk of data somewhere else while I was preparing this migration. So I started with moving some of my simpler VMs, ones that didn’t take up too much space and didn’t require much memory or CPU time.

I proceeded to one of my larger servers, which was Plex. Since I don’t update my Plex library very often and the amount of space it uses up was okay for me to move it over, it wasn’t that big of an issue. A day later I had noticed that ZFS foudn 8 corrupt files in my Plex library. I eventually found out that the issue was a faulty SATA cable. After replacing it, everything worked fine, I did a memtest to ensure my memory was okay and did a scrub on the zpool. Afterward, I replaced the corrupt files. I was concerned then with the reliability of this drive. It had been running for 4 years 24/7, then I let it sit 3 years. That had my concerned about the identical drive in my existing server which has been running 24/7 for 7 years. To be safe, I decided it was time to spend the cash (which I was trying to avoid) and I ordered 2x14TB drives.

New Drive Configuration

In my intial plan, I had intended to install my zones zpool on one of my SSDs and then have the data in its own data zpool. However, I was advised not to do that since I would not see any gains from using these SSDs. Additionally, I’d be having to do workarounds with SmartOS such as LOFS and such to get the data properly in each zone, which would have added complexity to my server. So at the advice of the maintainers, I decided to use only my new hard drives.

The Process Begins

Once I got the new drives, I couldn’t wait. I decided to start the migration over the upcoming weekend. I installed my new drives in my Thinkcentre, with one hanging out the chassis since it only supported a single 3.5" drive. I started copying data from my jails to my new zones to delegated datasets. However, I started encountering issues. One issue was the constant battle of low memory and trying to move everything over to this new "server" without overloading the desktop. I had been using rsync up until this time, but since my destination zones had too little memory allocated to them, rsync would eventually fail. I found sftp to work, and eventually I reached a point where I had to just turn stuff off and allocate more memory to these zones to get data over efficiently.

I did my best to recreate all my jails as zones. Most of them were done beforehand since they had low CPU/memory requirements. I had some others such as Nextcloud, my samba server, Plex, and my friend’s minecraft server that required some messing around to get things right and get it correct in joyent/lx branded zones. Once it was all done, I was ready to power down both machines and swap the guts.

Failure

At this point, I had my 2 disks in my server and all else hooked up, but encountered an error: I was booting into single user mode constantly. After looking through the logs and beating my head against the wall, I had to revert everything and put stuff back in their messed up state with both computers running their duties. I kept Nextcloud, Immich, Matrix, and Minecraft on my old server for the night, and most of the next day. In the meantime, it was the end of the night.

The following day, I scoured the internet for some sort of answer, and came upon this post. In it the writer notes to size up the dump and swap space on the zones zpool. Even mentioning the exact same error I had gotten. So I followed his instructions, but kept swap the same since I was doing this on my live system, and not single user mode. But I increased my dump volsize:

zfs set volsize={Installed RAM ÷ 2}G zones/dump
dumpadm -d /dev/zvol/dsk/zones/dump

I have 64GB RAM, where my old system which I built SmartOS on had 8GB. So the dump size was too small, and I set it to 40GB or something. When I put everything back in my server, it all worked!

Finishing the Move

The remaining bits were to put back together Nextcloud, Immich, Matrix, and Minecraft. I had ensured my data was current on Nextcloud, did a database dump/restore and Nextcloud was running well then. Immich, I had already moved the data and samba (namely, SmartOS' in-kernel SMB server) share. So my steps were to just restore the database stuff I backed up and spin Immich up in a new VM. Matrix and Minecraft restores went well too. At this point, I was fully on SmartOS.

Finishing Touches

Once done, I had some improvements to make. For one, I think going from a raidz1 to a mirrored zpool slowed down my write speeds some, which I was thinking might hav been the case. And it was. However, I also noticed that Nextcloud was performing particularly slow. For one, I noticed I didn’t install the php redis module, opcache php modules were missing too, and it wasn’t configured right either. After making those changes, Nextcloud is running where it’s been at. Some things feel a little slower, and I have high certainty it is disk I/O bound.

Monitoring and Backups

I like to receive alerts for problems, so I put in place SMART data monitoring, as well as monitoring to detect if a zpool is not in an ONLINE state or experiencing problems. Additionally, I have a monitor that will alert me if one of my zones goes offline.

The last bit is backups, which, I am using znapzend to backup my delegated ZFS datasets. I have backups of my bhyve virtual machines going to my storage zone. My last plan of action is to get all my configs backed up to a better spot so I can use Ansible or something to neatly rebuild my servers.

Final Thoughts

This project was a lot of fun to do, and so far I have been enjoying SmartOS. My hope is that it will continue to run well as it is now. I still have some fwadm work to do with this server, get into tagging and improving my zone manifests for future deployments of whatever I want to do. I am also hoping that my vision of this being a time saver will be true. As other obligations keep me busy, and family, I want something that I can let lowkey hum along in the back and check up on it when I feel like it. Since SmartOS seems to be more rolling, this is helpful for me. Likewise too since when I update the global zone, it also updates my native zones too.

Pause on SmartOS

Courtney — Mon, 10 Jun 2024 22:32:58 PDT

I’ve been writing about doing more with SmartOS and switching to it. The more I try it and look into it, the more I am interested in the operating system for my personal use. I have already deployed one lighter use server with it, and it was a pleasure. I will also be deploying one more that will likewise be somewhat lighter use. However, I will have to pause for my own personal server. It will be quite the job.

What is keeping me behind is I have not been satisfied with my backup plan on my current server. While I am migrating to SmartOS, I need to be certain that I have good backups. When that is in a comfortable position, I will move onward when I get some time…the other element is a lack of time. :)

In the meantime as I correct my backups, I am also creating playbooks for everything that I do repeatably on my server. SmartOS was an excuse for me to beef up my Ansible playbooks for personal use. So instead of waiting, I will do some of those in the meantime. Recently, I created a system to sign my SSH certificates and regenerate new hostkeys for servers on creation. I also have about completed automating SSL certificate updates with Lego and playbooks to deploy my certificates. I also have a nearly-complete collection to handle setting up Nextcloud in a SmartOS zone.

Writing has not been a priority, as time as filled to the top with other projects. But hopefully I can return to this. I think I will have lost out on my 100 days to offload so I am throwing in the towel there :(

Flair 58 Review

Courtney — Sat, 18 May 2024 11:32:15 PDT

I’ve been a Flair user since the beginning of my making home espresso. I started with the Classic, got the Pro, and pre-ordered the original 58. I’ve used it near daily for the last almost 3 years now. I did just have to set it aside to put a semi-auto machine on my coffee bar so both my wife and I could make espresso - she didn’t like coffee much up until recently - so I wanted to make a long term reveiw of the 58.

Having the original, I have a few differences in design compared to the ones out today. The ones I am aware of are the puck screen always being mandatory, as the original 58 had its own shower screen in the chamber, the plunger design is different. There’s likely other features that are different.

Workflow

My workflow for my 58 has been to start heating it on setting 2 and then walk away for maybe 15 minutes. I’ll come back and start my kettle and heat it to 210F for medium roasts (I overshoot the water temp because it drops when I pour it into the chamber), and as the water is heating I will start my puck prep.

As the water gets hot I will balance my portafilter on top of my kettle to steam the basket, since it isn’t hot enough after 15 minutes. During that time I will grind my coffee, and at that point the basket is usually warm enough to dry it off and then prepare my basket.

I lock the the portafilter in place, pour in my hot water, then start pulling.

Clean Up

My clean up involves purging the rest of the water from the chamber, dumping out my puck, then taking my purge cup and portafilter to the kitchen sink to clean it. Lastly, I put everything back in place for the next cup.

My Overall Thoughts

I’ve loved making espresso on the Flair 58 all these years. Maintenance has been simple, I will take the plunger out of the chamber and remove the o-rings and clean them all once every 2-3 weeks, and that’s been about all it needs. I have not needed to replace or repair anything, there isn’t much to repair. I know in the beginning there were controller failure issues, but I didn’t have that problem. Users were warned to plug/unplug the machine in the right order, since early designs shorted out from doing this in the wrong order.

Coffee Quality

The quality of the espresso has always been excellent. Every now and again I found something was lacking, but couldn’t figure out what. Maybe it was the beans or the way I made my coffee, but sometimes I would get a magical shot from a cafe using a La Marzocco and I couldn’t match it. However, all my coffee was excellent. The lever made it fun to try different techniques, but it also saved my coffee many times. If pulling too fast or too slow, it was always easy to manipulate the lever to get something good that probably wouldn’t have been as good on a traditional semi-auto machine.

Downsides

For me, the downsides were always the need to heat up the portafilter and basket on top of my water kettle. It was never hot enough unless I let the basket hang out over some steam for a while. Also, the constant use of the purge cup wasn’t my favorite thing either, and needing to fill up and manage a water kettle at the same time was just an added step to my process.

The biggest issue though was that my wife did not want to make espresso on it. She could, but didn’t like doing the job herself. So the biggest con to me is it is definitely a personal pleasure tool, unless others in your house like to do the fun process of making their coffee.

Conclusion

I love my 58. Currently, it got set aside by a Profitec GO so my wife could also make espresso. However, I intend on getting a new bar so I can put the 58 back on it. I love the GO, and a review will come for that.

The Flair 58 makes coffee fun and with great results. The price is good for what you get, and the fact that there is nearly 0 maintenance on this machine and few parts to break is excellent. If something DOES break, it doesn’t require a technician. Flair’s support is top notch in my experience and it makes the art of good quality lever espresso achievable.

3 years later, I would buy this machine again. And I am sad to put this machine on the side for a season, but I can’t wait to have it back for the times I want to experiment with my coffees.

OpenBSD 7.5 on SmartOS

Courtney — Mon, 29 Apr 2024 22:05:03 PDT

I use OpenBSD a lot. I have 14 jails/VMs right now and 5 of them are OpenBSD. As a part of wanting to move to SmartOS, one of my requirements is that I get OpenBSD running on SmartOS. Since there is no image, I made it my own way following the Virtual Machine from Scratch docs on the SmartOS wiki.

It makes it easy with OpenBSD 7.5 supporting EFI boot for the .iso images. I started with using this manifest:

{
    "brand": "bhyve",
    "alias": "OpenBSD75",
    "autoboot": false,
    "hostname": "obsd",
    "resolvers": [
        "9.9.9.9",
        "149.112.112.112"
    ],
    "ram": 1024,
    "vcpus": 2,
    "device": null,
    "nics": [
        {
            "interface": "vio0",
            "model": "virtio",
            "nic_tag": "admin",
            "ip": "dhcp",
            "primary": true

        }
    ],
    "disks": [
        {
            "boot": true,
            "model": "virtio",
            "size": 16384
        },
        {
            "model": "virtio",
            "media": "disk",
            "path": "/install75.iso",
            "size": "654",
            "nocreate": true
        }
    ],
    "bootrom": "uefi",
    "bhyve_extra_opts": "-w"
}

SmartOS Progress

Courtney — Mon, 29 Apr 2024 22:00:07 PDT

A while back I briefly wrote about using SmartOS and wanting to give it a spin. I ran short on time and excuses to try it, so I put it on the back burner, despite wanting to get into it.

Well, great news! My dad’s server died and a friend’s TrueNAS CORE box needs replacing (it is having problems too, the plugins really do suck at times). With my dad having a timeline to meet to get a server up with a couple VMs, it was a great time to bite the bullet on SmartOS for real. I still have plenty to learn, but I believe now I have a good server ready to go with the 2 VMs he needs.

The Server

It’s just an old beater HP i3 desktop with 2 cores, 4 threads and 6GB RAM, and we stuck in an SSD that we will do a mirrored zpool on later.

I installed SmartOS to a flash drive and set it up. This server only has 2 functions:

A UISP server - so I pulled down Debian 12 for that
OpenBSD wireguard VM - so I did the process to install from iso

I’ll write later on what I did to install OpenBSD 7.5.

Ansible

I use Ansible at work and always wanted to get it up for home use. I finally pulled the trigger and have a couple roles and such to manage setting up servers with SSH and adding users. I’ll add more as time goes on, but this is all these two servers need as far as Ansible goes right now.

Everything is running well right now, I am quite pleased with the end result.

Last Steps

I need to get this server out to its destination, finish the UISP setup, connect clients to the VM at its new destination, and then pair up the Ubiquiti devices with the UISP server, and then this should be good to roll for now. There’s lots of changes happening AROUND this server, which I will get to at some point. I will also probably get a DNS server on this too and use a zone-dataset fixed up with that and run unbound on it or something.

This is a good start for me. Next steps will be to move onto the next server and migrate my friend’s Nextcloud to a zone-dataset, and I think he wants an Immich server too, plus some sort of local DNS as well. But yeah, overall quite pleased, more to come as I set these boxes up and go into more detail.

#100DaysToOffload #Post11

From Nextcloud Photos to Immich

Courtney — Mon, 29 Apr 2024 21:30:46 PDT

Lately I’ve been growing a little disappointed with Nextcloud’s offering. I’ve been running it since its first release, and overall it has been a great piece of software. However, lately I’ve encountered issues like sometimes the desktop client won’t use secret service (and there’s been lots of yelling about this on GitHub with 0 developer response), Nextcloud’s pursuit for new shiny stuff like AI without fixing the core software, and lastly, utterly breaking core software in big ways and shipping it anyway. So, I’ve been looking to replace components of Nextcloud since there isn’t exactly a one-to-one replacement for it.

My first shot is at photos. This has actually been a pain point already. I was frustrated when Nextcloud seriously broke Photos 2.0 in Nextcloud 26. It was broken until a later release and I had to get over it. I switched to using Memories, which is actually incredibly awesome and should just replace what Nextcloud has, but the big problem is with albums.

Albums are great. Nextcloud doesn’t make it easy to sync any of that. Also, if my family is mainly phone users, as they are, I can’t tell them to pull out a laptop and go and add pictures to albums. They don’t do that. But, if they could from their phones, they could, and likely would, collaborate on family albums. After playing with Piwigo some, I settled on Immich. Piwigo is also very good, and it settled my strong desire to run my software on FreeBSD and not use Docker, but Immich is just too cool.

What’s Cool About Immich?

Here’s a nice bullet point list of what I like:

Don’t have to do community albums to allow single users to have their own photo syncing
Web UI is nicer
Administration is straightforward
A stupid amount of picture recognition stuff out of the box
Pretty maps for geotagged photos
Transcoding
Easy album sharing
Lots of search options, like searching for a mountain, my wife, a plane
Sync photos from my phone
OIDC support - this will be fun maybe

How I Am Attempting This

At first I started with the idea of cloning my ZFS dataset with my Nextcloud data, adding it to Immich, and deleting the other surrounding documents since a lot of the storage is taken up in photos and videos from our phones. However, I noticed that Immich copies the files into a library. So, that would mean importing and then deleting all the data in the dataset pretty much. Not super effective.

What I Want

I don’t like having big data tied up in a disk image. That’s also not useful for me with the way I do backups. So, how from a FreeBSD box do I take data from dataset A, and import it into dataset B from a Linux (Debian 12) virtual machine?

Create the VM
ZFS clone my Nextcloud dataset and set it up for an NFS mount
Create a new ZFS dataset and NFS mount it: this is will be the end destination
Create a docker user and spin up Immich
Mount both NFS filesystems
Use immich cli to upload my Nextcloud camera rolls
When done, delete the old NFS mount and dataset

This process has the added bonus of letting me upload to both Nextcloud and Immich while I get things in order.

I’m Still in the Middle of This

I am still moving data as I speak. I am kicking the tires on only my user at this moment, but if I get it all working right, I’m sliding the family over.

My biggest concerns at this moment are how well will the iOS app sync? And, will the app pick up where I left off in my photos and not make me have to upload my whole entire library? That would be a huge pain.

First Attempt

My first attempt was messy. I did what I described in "What I Want" but it is a lot of work for many users, and I ran into some roadblocks with the features currently present in Immich.

First, I can’t tell the iOS app to start backing up my photos from NOW. It has to do everything. Secondly, I converted all my files to jpg with Nextcloud, Immich operates on the HEIC files. This means I have to sync everything because the file hashes will not match.

So, I used the immich cli to import my Nextcloud photos, then uploaded all my pictures on my phone with Immich. This left me with duplicates for HEIC and jpg files. To alleviate this, I thought to change the Storage Template to be the same thing, applied that template to my existing files, then uploaded the images on my phone. This meant the file names were identical, minus the HEIC and jpg. I looked for all the HEIC files I had and removed the equivalent files named jpg

for f in $(find /mnt/nfs/library/library -type f -name "*.HEIC"); do
   jpg_file="$(echo $f | sed 's/.HEIC/.jpg/g')"
   rm $jpg_file
done

Then I had to remove these files from the database. This required I exec into the postgresql container and run some commands on the database. I took the orphans.json file that Immich provides in the Repair page, having only the Offline Paths entries. I pulled out the entityIDs and stuck them into a file, which I then ran a for loop on to delete the entries that matched

for entry in $(cat /entityIds.txt); do
	psql -d immich -c "DELETE FROM ASSETS WHERE id='$entry';"
done

After that, I had a bunch of untracked files to clear out, since they’re all thumbnails and what not that were generated for those images. So I downloaded the untracked.txt again, this time putting it in the immich-server container. I didn’t have to do any grepping or awk, it was already in a nice format. SO I ran a somewhat similar command as before

for file in $(cat /untracked.txt); do
	rm $entry
done

I did a refresh on the repair page to make sure it was all cleared out.

Next Steps

I got my family on Immich for now. It will be a job to move their old stuff over, but I think my next attempt will be to match time info in the filenames between Nextcloud and Immich, and if the time info matches, it is probably the same file. If that file does not exist in immich, then upload it.

After that, at a much later date, I’ll clean out all the Nextcloud data.

This will probably be a "do this over months" project, especially since I have family on Nextcloud that isn’t in the area, so I have to migrate them when they get here.

Anyway, this is a big job. It could have been easily solved if I could backup the camera roll from NOW. But, so far I still find Immich worth it. I do have my first upgrade coming up. Lots of irons in the fire right now though…I’m also super duper behind on my 100 days to offload :)

#100DaysToOffload #Post10

Pondering Leaving Nextcloud

Courtney — Wed, 10 Apr 2024 20:26:58 PDT

I’ve been using Nextcloud since its first release in 2016. It has been fun to watch it grow and improve over all these years. I sometimes hear people complain about how difficult it is to maintain, or it breaking and such. However for me, 8 years later, it has been incredible and put up with a lot from me.

I started using Nextcloud on a Raspberry Pi 2 with a 32GB flash drive for storage, a Raspbian 8(?) LAMP stack. Then I got the Nextcloud Box and upgraded it to 1TB storage, then moved to an Ubuntu 16.04 server in an LXD container, then to a FreeBSD 11.1 server in an iocage jail, changed from Apache2 to nginx, changed from MariaDB to PostgreSQL, upgraded hardware again in there, moved it to a hand-spun jail, and now to today. I have run it on all sorts of configurations and over all of it, it has worked. Until recently.

Over time, I have watched the scope of Nextcloud grow broader and broader. I don’t blame them though. It is a great one stop shop and pluggable. However, I have recently seen core parts of Nextcloud languish. With Nextcloud 25, Photos 2.0 was released. And it was a horrible launch. Every feature that was touted in the marketing was entirely broken. It really took until Nextcloud 27 for it to feel adequate, but still not impressed. In 28, Files got a facelift. However, still, core functionality is broken. When creating new files/folders it just sticks with "New Folder" or whatever, requiring extra clicks, even having to go to the "Rename" option to rename it from the default, where before I could just create new and type the name. I can no longer download files in bulk. It pops open a new tab and does nothing. The nextcloud client only seems to behave well for me with kwallet, and does not work with any other secret service for me. I noticed that there were multiple issues open on GitHub, all of which had received 0 developer response.

I have been seeing a trend. A focus on new shiny features with big promises and hugely under delivering. I can never trust E2EE after its debut years ago, Guests was a promised feature at one point that still does not exist, all their efforts to polish existing core software has left them in a condition with all sorts of bugs at release. Really, a release that should not have happened at all. I am experiencing too many bugs today to be comfortable with Nextcloud like I had been for years. So, it is time I consider new pastures.

So, Onto What?

I really only use Nextcloud for a small set of things:

Files
- Sharing with people on the server, and via public links
- Versioning
- Desktop sync client
Photos
- Syncing from my phone
- sharing with friends/family
- Allowing family/friends WITHOUT accounts to upload
- Albums
CalDAV/CardDAV
Kanban (kinda)

So, all my options aren’t 100% solidified, but here is where I plan on going.

Files

If I didn’t do file sharing, I would have gone with syncthing. I used it for a long time before I switched to Nextcloud. Maybe I will look at it again, but I also have to consider that I am wanting strangers to possibly have an upload spot without installing a client, as well as sharing with family on the server. I don’t think Syncthing does any of this, but it also has been many years since I looked at the project.

Really, I see myself possibly using Owncloud OCIS. It does files. Done. Neat. The UI is familiar too. I’ve poked at it with a stick over the years, and I have a feeling it is good enough for me to do this job. Hopefully the desktop client is still good.

Photos

If you’ve read my posts, you’ll probably know that I have basically 0 Linux servers in my house. It’s FreeBSD or OpenBSD (or maybe even SmartOS soon, time will tell). I’m not AGAINST Linux, I just like running those operating systems from a management and design perspective. However. I was recommended Immich. Even worse, it basically requires I use something like Docker which 99% of the time I refuse to even consider using….but Immich struck that 1% chance. It is still in active development and they have warning signs saying thar be dragons. But I have ZFS snapshots to save the day if something goes south.

Immich is just…awesome. It does everything I wished Nextcloud could do, but couldn’t. I don’t like Nextcloud albums. I like Memories, but one fatal flaw has been that album sharing and all that really requires being on a computer. I tried making it a PWA on my phone, it didn’t bode very well. Immich still has some work to be done for it to be that golden program for me, but it is so good that I am pressing on. I’ll be writing soon about migrating from Nextcloud to Immich. It isn’t easy. But I will make it work!

CalDAV/CardDAV

This was THE feature that got me onto Nextcloud in 2016. I noticed a project called Radicale, and I will be using that. No one in my family uses the Nextcloud Calendars app (which, also has had all sorts of problems, forgot to mention that). We use our phones or Thunderbird to manage calendars. So this will be easy to replace.

Kanban

I use Deck. Deck is awesome. But I don’t need it. In fact, where I do use it is on the work’s Nextcloud server, and I am all good with keeping my server at work on Nextcloud. It has served its purpose for us very well there.

Moving Onward

Nextcloud has served me very well over the years. I still don’t know if I will be dropping Files and Cal/CardDAV for other software. I also have to consider my family who has also been using this. I also have to consider my own time managing all this. But I think I am ready to move on.

The last bit I am interested in possibly is setting all this up with a Keycloak server. Only Radicale doesn’t seem to support some sort of SSO, but they suggest I can do this at the web server level. This could be that nice "bow on top" piece here, but I may not care either, since I have all my family using password managers.

So yeah, let’s see where this goes. I have another post lined up on my migration from Nextcloud photos → Immich experience. I actually started with that before this post, but I have more to add to that.

#100DaysToOffload #Post9

Pork Top Loin and Mushrooms

Courtney — Wed, 03 Apr 2024 23:03:17 PDT

I’ve only written tech stuff here, but I wanted to share a good recipe that I threw together for dinner tonight. It’s a pork top loin chop from Costco with buttery garlic mushrooms, and a white wine pan sauce. I made 2 loins for my wife and myself.

Ingredients

This is only an approximation, I throw stuff together on the fly.

The meat:

2 pork loins
Salt and pepper
Avocado or olive oil

The mushrooms:

Button mushrooms - about 4-6oz
4 cloves of garlic
2Tbsp unsalted butter
Salt and pepper

The sauce:

White wine - I had a chardonnay, maybe 3 fl oz? A good splash and glug.
2Tbsp unsalted butter
1-2 cloves of garlic
Heavy cream - just under 1/4 cup
Thyme (I had dried)
Salt and pepper

Cooking the Meat

Take the pork out, pat it dry and apply a generous amount of salt to all sides
Let sit on the counter for about 30 minutes
While resting the meat, get a cast iron skillet and put it in a 250F oven for that 30 minutes
Cook the meat about 25 minutes, or until it hits 110-115F
Just before the meat is ready to come out, get a stainless steel skillet and put it on medium heat
Put 2Tbsp of oil in the skillet and bring it up to temp, and put the meat in the new skillet
Cook on all sides to get it golden brown and delicious
Cook until it reaches 145F internally
Set the meat aside on a wire rack

The Mushrooms

With the skillet fresh out of the oven, put it on the burner at about medium heat

Add in some butter and let it melt and warm up
Add the sliced mushrooms
Add some salt and a lot of black pepper
When 3/4 of the way done, add more butter, then add minced garlic
Cook on medium-low to let the garlic slowly cook and not burn
When the mushrooms are done, I took my pork and put it back in the skillet to keep it warm. Set the burner to the lowest setting

The Pan Sauce

With the meat out, set the heat to medium-low
Add 1Tbsp of butter, let it melt, and add the minced garlic
Once lightly brown and fragrant, dump in the white wine to deglaze the pan
Scrape the meat off the bottom, turn up the heat a little to maintain a simmer
Add a bunch of black pepper and some thyme, a few good shakes of it
I also dumped in the drippings from the pork as it sat on the wire rack
When the wine is reduced some, about half the amount of liquid gone, add the heavy cream
Lower the heat a little and bring the cream up to temp, just a couple minutes of stirring
Turn the heat down and add the butter and melt it in
Check the seasoning for anymore salt or pepper

Serving

Put the meat on a plate, load some of the mushroom on top, and drown it all in the pan sauce.

Enjoy.

#100DaysToOffload #Post8

Trying SmartOS

Courtney — Mon, 25 Mar 2024 22:32:40 PDT

Today, I am going to try something very different. I am going to give TRITON SmartOS a shot.

A number of years back, I tried OmniOS and loved it. But I wasn’t ready to move to it or anything. After trying out some of the Solaris zone and SMF stuff, I was hooked. It was really cool tooling to use and felt thought out.

Now again, I am going to try again, but with SmartOS. I found the idea of running the OS off a flash drive interesting. Rotating flash drives for my updates and clearing up my SSDs for JUST zones is a cool idea. Especially since I don’t have any more cables in my server to add anymore drives :)

If I like it, I may even try moving my 7 year FreeBSD installation to SmartOS. I really like the idea of the zone manifests and all the other cool management stuff I can do, and possibly even integrating it all with Ansible, which I use a lot at work. Now I have a nice place to integrate it all in with, perhaps.

I have an installation already with data on an external USB drive. I think I should just be able to plug that into the computer, plug in my flash drive, and pick up where I left off before. If I like it enough, this is how I imagine moving to SmartOS would look like.

Right now, I have FreeBSD 14.0 installed on a mirrored zpool that has my jails and VMs. I have a RaidZ1 pool with a few VMs and all my data. My imagined steps would go like:

Try out SmartOS and get acquainted with it on my flash drive/extra USB external drive
If I like it, detach an SSD from my mirrored zpool
Start from the ground up, making the new SSD my zones drive
Build new SmartOS zones on top to replace what my jails do, and probably a Linux one to get Plex on there

Note
I prefer FreeBSD for Plex, but I like the idea of mounting ZFS datasets into a jail/zone like I am now, and LX zones will let me keep that.
Since FreeBSD and SmartOS use bhyve, I would test moving some VMs over one at a time
If all looks good:
1. zpool export my data zpool
2. move my flash drive and SSD to my server
3. Import the data zpool
4. Map my zpool datasets to my new zones and get it all running
Clean up anything that needs cleaning
Clean my extra SSD and then attach it to the new zones zpool

By then, FreeBSD should replaced with SmartOS.

What is my motivation? Just to do something fun and different. After running FreeBSD for 7 years, I’ve grown a bit bored with it just working all the time :) I like to try new things still and build new things, and SmartOS checks all the boxes of different, interesting, and has a lot of the things I like about FreeBSD like ZFS and bhyve, and then some. It’s time to mix it up some on my own servers. Some of my motivations include:

Freeing up my internal drives since SmartOS is designed to run from a flash drive
SMF and Crossbow are really cool. These management tools are great
To my understanding, Type 1 hypervisors are a little more performant
zone manifests are neat. And with existing Ansible collections for this stuff, I can more easily automate some of my server deployments
- Also, being able to pass in some init commands with the customer_metadata stuff to install SSH keys or run cloud-init or what have you is neat
pkgsrc has also been a cool feature to me, and I like seeing it used here

I think there’s a lot to like. And, this may be an odd view from the perspective of an outsider looking in, but when I see some of what Illumos has, it looks like an OS from the future, but also in other ways, like hardware support, an OS of the past. So, it’s all just interesting to me and I can’t wait to play with it more, jot down my thoughts, and see if I really do decide to commit to it. If I don’t replace FreeBSD on my server, I at least know I can say I tried out a very interesting operating system.

#100DaysToOffload #Post7

Hebrew and Greek With asciidoctor-pdf

Courtney — Wed, 13 Mar 2024 17:29:34 PDT

I was tasked with trying to get both Hebrew and Greek to be readable with Asciidoc, primarily for epub and PDF, but I also wanted to try with HTML too. Both HTML and epub worked great, but I had troubles with getting it to work with PDFs.

Installation

I’m on OpenBSD, but there are similar steps. In fact, I also did this on Linux and the steps were pretty similar. First I had asciidoctor installed on my computer:

pkg_add asciidoc asciidoctor

This also installed ruby, but if not for you, install that so you can get the gem package manager. Here is how I installed asciidoctor-pdf and asciidoctor-epub

gem32 install --user asciidoctor-pdf

On OpenBSD I have to specify gem32.

NOKOGIRI_USE_SYSTEM_LIBRARIES=1 gem32 install --user asciidoctor-epub3

That’s all is needed to make HTML, PDF, and epub.

HTML Output

HTML output didn’t need any work. I just had my source content which contained English, Hebrew, and Greek.

asciidoctor myfile.adoc

Done. I now had a single HTML page.

EPUB Output

This one contains some extra metadata in the document to make it work

:doctype: book
:author: Author Name Here
:copyright: CC0
:epub-chapter-level: 2
:toc:
:media: print

Then I ran the command to get an ebook

asciidoctor-epub332 myfile.adoc

That generated the epub file, looks great, renders all languages great.

PDF Output

This one was tricky. I guess the HTML uses FontAwesome, maybe the epub tool does too, but the PDF tool handles it differently. I was suggested to get Society of Bible Literature’s TTF font file. I had to modify the theme to make it use this font. In the working directory where my adoc file is, I created this yaml file:

extends: default

font:
  catalog:
    merge: false
    SBL BibLit:
      normal: /path/to/Documents/mypdf/fonts/SBL_BLit.ttf
      bold: /path/to/Documents/mypdf/fonts/SBL_BLit.ttf

base:
  font-family: SBL BibLit

I called the file mytheme-theme.yml.

I also created a fonts directory and placed the downloaded ttf file there.

Note	These fonts do not support italics, so it is vital to avoid styling that will put Greek/Hebrew texts in italics, such as quote blocks with `----`. Use ____ (4 underscores) instead.

The header metadata is also very similar to the epub

:doctype: book
:author: Author Name Here
:copyright: CC0
:epub-chapter-level: 2
:toc:
:media: print
:pdf-page-margin: [0.5in, 0.25in]

You should be able to run this command now to generate the PDF:

asciidoctor-pdf32 --theme mytheme-theme.yml -a pdf-fontsdir=/path/to/Documents/mypdf/fonts test.adoc

#100DaysToOffload #Post6

i3 Fix Firefox Alerts

Courtney — Mon, 26 Feb 2024 16:30:36 PST

Somewhere around Firefox 120/121 I noticed my Firefox notifications were getting messed up. At first I didn’t see them, then I realized they were transparent and would freeze the part of the screen where the notification would appear until it eventually faded away. But It was hard to know when that was, so I would have to reload i3 constantly when I needed to access that part of the screen since I didn’t have any sort of x button to close out.

The Fix

Enter Firefox 123. Not sure what they changed, but in version 123 notifications started to pop up again…and tile. So I would have half my screen tile out of nowhere because i3 treated it as a regular window. I was able to quickly run xprop and get the window’s information. I decided the best method in my case was to modify how i3 (I should note: version 4.23) treated "alert" windows as this is what the window role popped up as. So I added these two lines to my i3 config file:

for_window [window_role="alert"] floating enable
no_focus [window_role="alert"]

This made the alert windows float, and not focus since these windows were also drawing focus.

After that, things seem to be back to normal.

#100DaysToOffload #Post5

Ploopy Adept Initial Review

Courtney — Mon, 19 Feb 2024 18:16:13 PST

The other day, I received my Ploopy Adept in the mail, after having waited since mid-December when I placed my preorder. I have been a trackball user for a few years now, mainly having used the Elecom HUGE and the Kensington Orbit Fusion. The Kensington was awesome, but the angle was a bit too steep for my wrists. The Elecom has been my favorite by far, but the stock bearings caused me a lot of grief with the insane amount of stiction. I eventually switched the bearings to chrome plated steel ball bearings and those worked wonderfully. However, with how much I use the trackball, about every 6-8 months I had to swap the bearings. No fun.

Enter: the Ploopy devices.

I’ve eyeballed these trackballs for a couple years now, but was never ready to settle on one. Plus, I wanted something with a couple more buttons. The ploopy had all I wanted:

A good price
some extra buttons (6)
I wanted to try out a blade-style trackball

Now below are some of my initial thoughts on this trackball, after having used it only a couple days so far. I will have something more in-depth later

What’s Nice

I ordered mine in black and with a red ball. Looks great. I like bright colors, but I am also easily distracted and liked the more utilitarian look of just black and red. However, I think it is really cool that they have so many different color options.

Stiction?

None to speak of. The ball glides so well. I’ve never had a trackball that felt so good in terms of ball movement. It is taking some getting used to honestly. The roller bearings feel great too. They are here to stay.

Button Quality

The 6 buttons on this trackball feel great. Nice and clicky, not mushy, not hard to press down either. They feel durable, and like they will last.

Ergonomics

This is subjective and I think depends on who you are. I like this shape and design. It does feel nice. The buttons above the ball are a little hard to reach, but no surprise there with the ball pretty well in the way.

The Chassis

I was expecting the chassis to feel like it was 3D printed, but it is actually nice and smooth. And it has some of the lines from the printing so it has some character to it, so I can have my cake and eat it too.

What’s Not Nice

So far, can’t say much negative about it. They suggest that you use Via to customize the buttons and such for this trackball. I was shocked that making changes were instant. The downside though is I don’t see any option to adjust my DPI, and I would have preferred the drag scroll (nice feature by the way) to be an on button press, not holding down the button.

The default layout doesn’t feel all that good either, I arranged the default keys around and that was good to my taste. Again, subjective.

Modifying This on Linux

This is where it was a bit odd. I couldn’t find any information on how to modify this mouse’s layout except on Reddit I think. To use Via, I needed to use a chromium browser, but by default on Linux you have to find the device from the command line and chmod a+rw that device. I had to find which device it was using in chromium’s log. In my case, it was /dev/hidraw6. After adjusting permissions, I could use Via. Then afterwards I reset it back to 0600. I think that some newer Linux users who are using this mouse won’t like having to enter a console and chmod the device after finding it to modify the layout.

What I Want

I’m not very creative with fancy custom layouts, but what I would like to be able to do easily is

Adjust the DPI - it’s too fast for my taste out of the box
have drag scroll be toggle feature, not hold
allow different functionality if I am holding or pressing a button

For the last point, I could see myself saying "on click for the bottom right button, enable drag scroll. On hold, enter layer 2". Or even vise-versa. I am pretty sure that is something that can be done in QMK…my Ergodox EZ does that sort of stuff anyway, which is also a QMK board. So perhaps those features are there. However, I recall seeing I will have to disassemble my Adept if I want to make these changes.

Conclusion

Overall, I give this trackball a 9/10 on initial use and a couple days in. I really love how this mouse feels. The more I get used to it, the better it is. And lastly, maybe some of my nitpicks are easily addressable, or maybe not. I’m really not sure. But I am hoping that my couple requests are possible. If not, I think I will just get used to these things and still be a happy user. Thank you for the great device!

#100DaysToOffload #Post4

My Dumb Roasting Script

Courtney — Mon, 05 Feb 2024 19:32:14 PST

I roast coffee at home regularly. I haven’t updated my roasting log, I need a more lazy man approach to converting it all nicely. But I still do roast, and I use the neatest, dumb script ever..which I know works on OpenBSD and Linux. And it’s public domain

#!/usr/bin/env sh

COFFEEDIR="$HOME/Documents/coffeeroasts"

echo "Coffee country of origin?"; read origin
mkdir -p "$COFFEEDIR/$origin"

filecount=$(ls "$COFFEEDIR/$origin" | wc -l | xargs)

start_timer()
{
	terms="xterm konsole gnome-terminal xfce4-terminal mate-terminal rxvt urxvt xterm-256color"
	if [ $(command -v xterm) ]; then
		xterm -e sh ./timer.sh &
	elif [ $(command -v gnome-terminal) ]; then
		gnome-terminal -- sh ./timer.sh &
	fi
}

seconds_to_ts()
{
	TIME=$1
	if [ $(uname | grep -c "BSD") -eq 1 ]; then
		date -u -r $TIME +%T
	elif [ $(uname) = "Linux" ]; then
		date -d @${TIME} +%T
	fi
}

configure_roast()
{
	echo "What is the coffee's name?"; read coffeename
	dirname=$(echo $coffeename | sed 's/ /_/g')
	roastdir="$COFFEEDIR/$origin/$dirname"
	mkdir -p "$roastdir"
}

write_details()
{
	START="$1"
	DETAILS="$2"
	FILE="$3"

	#timefromstart=$(($(date +%s) - $START))
	timefromstart=$(expr $(date +%s) - $START)
	#printtime=$(date -u -r $timefromstart +%T)
	printtime=$(seconds_to_ts $timefromstart)
	echo "$printtime $DETAILS"
	echo "$printtime $DETAILS" >> "$FILE"
}


calculate_phases()
{
	START="$1"
	ROASTFILE="$2"

	drop=$(grep "TAG:DROP" "$ROASTFILE" | awk '{print $3}')
	#roasttotal=$((($drop - $START)))
	roasttotal=$(expr $drop - $START)

	drystart=$(grep "TAG:DRY" "$ROASTFILE" | awk '{print $3}')
	#drytime=$((($drystart - $START)))
	drytime=$(expr $drystart - $START)
	drypercent=$(echo $(echo "scale=2; $drytime/$roasttotal" | bc -l)*100 | bc)

	onecrackstart=$(grep "TAG:1CSTART" "$ROASTFILE" | awk '{print $3}')
	#browningtotal=$((($onecrackstart - $drystart)))
	browningtotal=$(expr $onecrackstart - $drystart)
	browningpercent=$(echo $(echo "scale=2; $browningtotal/$roasttotal" | bc -l)*100 | bc)

	#developtotal=$((($drop - $onecrackstart)))
	developtotal=$(expr $drop - $onecrackstart)
	developpercent=$(echo $(echo "scale=2; $developtotal/$roasttotal" | bc -l)*100 | bc)

	echo "Roast complete at $(date +%T) in $(seconds_to_ts $roasttotal) minutes"
	echo "Roast complete at $(date +%T) in $(seconds_to_ts $roasttotal) minutes" >> "$ROASTFILE"
	echo "Dry time: $(seconds_to_ts $drytime)\t\t\tDry time percent: $drypercent"
	echo "Dry time: $(seconds_to_ts $drytime)\t\t\tDry time percent: $drypercent" >> "$ROASTFILE"
	echo "Caramelization time: $(seconds_to_ts $browningtotal)\t\tCaramelization percent: $browningpercent"
	echo "Caramelization time: $(seconds_to_ts $browningtotal)\t\tCaramelization percent: $browningpercent" >> "$ROASTFILE"
	echo "Develop time: $(seconds_to_ts $developtotal)\t\t\tDevelop percent: $developpercent"
	echo "Develop time: $(seconds_to_ts $developtotal)\t\t\tDevelop percent: $developpercent" >> "$ROASTFILE"

}

if [ $filecount -eq 0 ]; then
	configure_roast
else
	ls "$COFFEEDIR/$origin" | nl
	echo "Are you roasting any of these? Pick the number or any other key for no"; read pickroast
	dirname=$(ls "$COFFEEDIR/$origin" | nl | grep " $pickroast" | awk '{print $2}')
	if [ -z "$dirname" ]; then
		configure_roast
	else
		roastdir="$COFFEEDIR/$origin/$dirname"
	fi
fi



start=$(date +%F)
roastfile="$roastdir/$start.txt"
touch "$roastfile"

echo "Batch size?" ; read batchsize
echo "Batch weight $batchsize" >> "$roastfile"

echo "Press any key to start the roast timer"; read x
startepoch=$(date +%s)
#xtimer &
start_timer
timer_pid=$!

echo "Enter 1 to note end of drying phase"
echo "Enter 2 to note start of 1 crack"
echo "Enter 3 to note end of roast"
echo "Enter q to exit the program entirely"

while read detail; do
	case "$detail" in
		q)
			break
			exit
			;;
		1)
			write_details $startepoch "TAG:DRY $(date +%s)" "$roastfile"
			continue
			;;
		2)
			write_details $startepoch "TAG:1CSTART $(date +%s)" "$roastfile"
			continue
			;;
		3)
			write_details $startepoch "TAG:DROP $(date +%s)" "$roastfile"
			break
			;;
		*)
			write_details $startepoch "$detail" "$roastfile"
			continue
			;;
	esac
done

calculate_phases $startepoch "$roastfile"
#pkill xtimer
if [ -z $timer_pid ]; then
	pkill timer.sh
else
	kill $timer_pid
fi

echo "Final roast size?" ; read finalbatchsize

echo "Drop weight: $finalbatchsize" >> "$roastfile"

startbatchnum=$(echo $batchsize | sed 's/g//g')
endbatchnum=$(echo $finalbatchsize | sed 's/g//g')
percentloss=$(echo "100*(1-($endbatchnum/$startbatchnum))" | bc -l)
printf "Weight percent loss: %.2f\n" $percentloss >> "$roastfile"

grep "TEMP" "$roastfile" | awk '{print $1" " $3}' | sed 's/C//g' > temp.dat
cat graphit.gp | gnuplot
graphfile="$roastfile-graph.png"
rm temp.dat
mv temps.png "$graphfile"

It’s pretty dumb, but works great. I roast with a BocaBoca250 so I have no way to input data in except to watch the thermometer and do my best. This script is nice because you can select the coffee origin, the name of the coffee you are roasting, batch size, then input information you want such as roaster settings, smells, etc. The part that matters is temperature entry. For example, TEMP 100, TEMP 180, etc. Then numbers 1,2,3 for different phases

Dry phase finish
First crack start
Roast done

Looking at the contents here, you can see how I entered the values. I didn’t enter the times though. That is handle automatically as you enter temperatures and such.

When done, it calculates the time and percentages in dry phase, caramelization phase, and development phase, and lets you enter the final weight which will get you the % of moisture loss while roasting. Lastly, with gnuplot installed you will get a nice graph too like in the above post.

The last 2 scripts needed are timer.sh

#!/usr/bin/env sh

seconds_to_ts()
{
        TIME=$1
        if [ $(uname | grep -c "BSD") -eq 1 ]; then
                date -u -r $TIME +%H:%M:%S
        elif [ $(uname) = "Linux" ]; then
                date -d @${TIME} +%H:%M:%S
        fi
}

start=$(date +%s)
while true; do
	time="$(($(date +%s) - $start))"
	printf '%s\r' "$(seconds_to_ts $time)"
done

and graphit.gp

#!/usr/bin/env gnuplot

set terminal png size 800,600
set output "temps.png"

set xdata time
set timefmt "%H:%M:%S"

set title "Roasting Temperature Profile"
set xlabel "Minutes/seconds"
set ylabel "Temperature (C)"

set key below
set grid
plot "temp.dat" using 1:2 title "Time" with lines

Very basic and dumb but I’ve had fun using it the last year, and it has helped me make consistent roasts.

#100DaysToOffload #Post3

Intro to SSH Certificates

Courtney — Mon, 05 Feb 2024 19:25:00 PST

Like many of us, I have been using SSH since the beginning of learning how to learn Linux, or some other UNIX-like OS. My usage has moved from password logins, to SSH keys, and now I have picked up on SSH certificates. This is an introduction to using SSH certificates with OpenSSH.

Before I get to my pros/cons, what the heck are SSH certificates? Think of getting an SSL/TLS certificate. First, I would generate my private/public RSA keypair, then a trusted certificate authority confirms my identity, being the owner of the website I want to enable HTTPS on, then they give me a signed public certificate to use alongside my private key, the new public key, and the intermediate certificate. That is similar to how SSH certificates work, except I am the certificate authority (CA).

The way it is done, is I generate two unique SSH keypairs:

To sign my servers' host keys
To sign my users' personal keys

I then tell the servers to trust the user keys I sign and vice versa. So, why should I do this instead of using only a password, or dropping my public key in the authorized_keys file on my server? Well here’s the reasons I found that make this worthwhile:

When I create new keys, I don’t have to password login to then add my new SSH key

With the trust relationship in place, I don’t have to accept host keys on a new connection

Note	This basically remedies the TOFU situation (Trust on First Use). But not 100% of the time if you don’t have direct console access to the server. But it will solve the problem for users logging in after the host key signing has been finished.

If a host dies, instead of installing all users' public keys, I just sign the host keys and the clients won’t complain. It is already trusted.
It makes key expiration policies much more feasible and enforcable.
This can be automated pretty easily

And the cons:

It’s not as well known as passwords or plain old keys, so there is a learning curve for policy/use changes
You probably want 2 dedicated, and preferably offline hosts for key signing
Digging into the weeds with principals can be hard to track to get right
If you plan on expiring keys, you will want processes in place to manage and sign new keys for other users

Does it sound more complicated than just dumping a key on a server or using a password? Yeah probably. But the scope of certificates go beyond just signing in. It is a system for good security practices. And I argue, does add some convenience in some occasions.

Getting Started

Now that that is out of the way and some of the what’s and why’s are in the open, lets drill down into the application.

First, consider having 2 separate hosts for this job. But if you are just doing this for home use, a single host that isn’t internet accessible should do. But I find it nicer to have 2 virtual machines running that I can turn on/off as I need. From here, I will act on the assumption you have 2 servers.

Host Keys

On host A, create the keypair. If you want to get extra fancy, password protect these keys. That way, if someone somehow got one of the keypairs, they would still need the password. So lets generate the keys

# ssh-keygen -t ed25519 -a100 -f /root/.ssh/hostkey
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase): SuperSpecialAwesomeOptionalPassword
Enter same passphrase again: SuperSpecialAwesomeOptionalPassword
Your identification has been saved in hostkey
Your public key has been saved in hostkey.pub
The key fingerprint is:
SHA256:WaOws18CKpgntRnFSgmJeKWD9utZqjqN8cmaLNtyw10 root@hostkeys
The key's randomart image is:
+--[ED25519 256]--+
|+. ..            |
|+o.+             |
|.o= o .   o      |
|...+   o + .     |
|  +.  + S        |
|.+ +..E+         |
|+B=+.o. . .      |
|*=X.=  . o       |
|BOo=    .        |
+----[SHA256]-----+

Neat. That’s one down. A quick rundown of the flags I specified

-t ed25519 sets the key type to ed25519. The default before OpenSSH 9.5 was RSA. So if you are running 9.5 or newer, you don’t need this
-a 100 - rounds of KDF used. Makes the private key more brute-force resistant. Useful only if you are password protecting your key
-f - tells ssh-keygen where to create the key

User Keys

Now, same as before but on the other host, lets generate the user keys

ssh-keygen -t ed25519 -a100 -f /root/.ssh/userkey
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase): SuperSpecialOtherOptionalPassword
Enter same passphrase again: SuperSpecialOtherOptionalPassword
Your identification has been saved in userkey
Your public key has been saved in userkey.pub
The key fingerprint is:
SHA256:NP4fKqdvbILpwfOhKx9u3voefyKR4+/ugTTjnsj5NY4 root@userkeys
The key's randomart image is:
+--[ED25519 256]--+
|                 |
|                 |
|        o        |
|       o .       |
|        S.       |
|     . o+=       |
|      =+=+= .    |
|    .o+@=O*=..   |
|     *%*E#Xo.    |
+----[SHA256]-----+

Signing Host Keys

Onward to signing our first keys. Let’s start with the host keys.

On each host when you first enable and run the sshd server, it will create some keys for itself. Usually RSA, ED25519, and ECDSA. I only use ED25519, so I will sign only those here in this example. In some cases, I still sign the RSA keys if some clients are doing RSAkeys. These are the default public keys that are to be found on a host:

/etc/ssh/ssh_host_ecdsa_key.pub
/etc/ssh/ssh_host_ed25519_key.pub
/etc/ssh/ssh_host_rsa_key.pub

Copy these to the host that is doing the key signing. Now, this is perhaps where the TOFU I talked about earlier gets defeated. You need to sign into that computer once to get the public host keys. This is a predicament with cloud servers that you do not have console access to. This is only a problem once. After this, all subsequent logins to that server can be trusted. Here’s how the key signing will go:

ssh-keygen -s /root/.ssh/hostkey -I example.com -h -n example.com ssh_host_ed25519_key.pub

Breakdown:

-s - the signing key
-I - the key identity. Can be used for key revocation
-h - specifies that this is a host certificate, not a user certificate
ssh_host_ed25519_key.pub - the public key being signed

This generates a ssh_host_ed25519_key-cert.pub file. This needs to be copied back to /etc/ssh on the host we got it from.

Cool. Now what’s next is we need to tell sshd to accept and use this certificate. Let’s add these lines to the /etc/ssh/sshd_config file

HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub
TrustedUserCAKeys /etc/ssh/userkey.pub
RevokedKeys /etc/ssh/revoked_keys

HostCertificate - the certificate that we just got back from signing the public key
TrustedUserCAKeys - this is the /root/.ssh/userkey.pub file created on the User Keys server
RevokedKeys - currently, just an empty file. But if any user keys ever need to be revoked for some reason, you’d add the public key here

There’s more that can be done, but I’ll reference it later as bonus material.

Next, run sshd -t which will let you know if a configuration is wrong or not. If all is good, run rcctl reload sshd or service sshd reload or systemctl ssh reload or whatever init system command you need to do.

Signing User Keys

The server is done, so now what? We just told the server that we want to trust keys signed by the Host Certificate Authority. Now our user keys need to be signed, and tell our computer to trust the host certificate.

First generate ssh keys

ssh-keygen -t ed25519 -a100 -f ~/.ssh/mycoolkey

Same as before, optional password, the -t and -a are optional to what you want. I want to get the ~/.ssh/mycoolkey.pub file to the USER key signing server this time. Once there, this is the command I want to run

ssh-keygen -s /root/.ssh/userkey -I me@example.com -n bob -V +180d mycoolkey.pub

The flags:

-s - the key used for signing
-I - the key ID. This can be useful for tracking logins. You could make this mylaptop or myphone or mycomputer and correlate each key to each sign in
-n - this is the permitted username, or principal (more on this later). If my user is bob, I should make this be bob for now
-V +180d - this sets an expiration of 180 days after creation. If you do not specify this, it will be valid forever

Now I have a shiny mycoolkey-cert.pub file. This will go in my ~/.ssh directory alongside my other keys.

With the file in place…I still cannot sign in to the server. Why? I will get a prompt asking me whether or not I trust the destination server. Kind of defeats the purpose. What to do now? This is one of my favorite parts. Instead of having a billion entries in my ~/.ssh/known_hosts file, I can instead litter these with @cert-authority lines! This actually gets much cleaner when you have many servers using the same host key. For example, I have about 70 servers that I signed with the same host key and can use a SINGLE line in my known_hosts file, instead of multiple (one per host key, which will probably be 3, then if I signed in with the IP and then later with the hostname, that can be 4 lines per host!). Ashamedly, I have gotten my known_hosts file over 700 lines from all my sign ins and servers that I nixed and never cleaned out my known_hosts file. So what do I add to my known_hosts:

@cert-authority *.example ssh-ed25519 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILw3E3Pa1CvvdCDQ8EkMpjl9kD72Sxbms9R1QCU8/ni6 root@example.com

To break down this line:

@cert-authority - Indicates the public key of the accepted certificate authority
*.example.com - any example.com host that is signed with the aforementioned host key
everything else - this is the contents of hostkey.pub from the Host Keys section

Signing In

So far, we

Created a user key signing keypair
Created a host key signing keypair
Signed a server’s host keys and told it to use that certificate, and trust the user keys we sign
Signed the user keys, and told our ssh client to trust the host keys that we signed

If all went right, you should be able to sign into your server and NOT get the yes/no prompt to trust that host’s keys. If I signed into that server before, I remove the previous trust entries from my known_hosts file. I do this by making a backup copy, in case I totally dorked stuff. Don’t want to get locked out! If you are getting a yes/now prompt, something went wrong and steps taken should be revisited. From here, you can just go with this. But I want to explore key revocation as well as principals a little.

Key Revocation

There are 2 kinds of key revocation to consider:

Host key revocation
User key revocation

Host Key Revocation

This is useful for SSH clients, if you want to enforce not trusting host keys. Say, if a host’s keys were compromised and you want to ensure your users will not get tricked into logging into a server with the compromised keys. You can add this info to /etc/ssh/ssh_config, or even your own ~/.config file. You would update your Host * section to have this:

Host *
	RevokedHostKeys		/etc/ssh/revoked_hostkeys

Host *
	RevokedHostKeys		~/.ssh/revoked_hostkeys

In this file, add the public keys of the host keys you want to revoke. Your ssh client will look to see if the pubkey is in your revoked hostkeys file and cancel the login if the keys are revoked.

User Key Revocation

This is very similar to host key revocation, just the other way around. Earlier, I created a config like in the sshd_config file

RevokedKeys /etc/ssh/revoked_keys

Like before, if a user’s keypair instead gets compromised, you can add the public key to this RevokedKeys file and any login attempts with this keypair will get denied.

Principals

I don’t want to get too deep into principals, but what makes it handy is you can do stuff like say user1 can sign into host1.example.com but not host2.example.com. By default, user1 can sign into any host that has their key signed by the user CA. We can modify user1’s principal to be something like "host1only"

ssh-keygen -s /root/.ssh/userkey -I user1@example.com -n host1only -V +180d mycoolkey.pub

To make this work, a couple tweaks need to be made to our sshd_config file. Namely:

AuthorizedPrincipalsFile /etc/ssh/principals/%u

Next, make the file /etc/ssh/principals/user1 on host1.example.com and have it only contain host1only. Although user1 may exist on host2.example.com, this setup will mandate that user1 has their keypair principal be host1only. This can be useful, for example, if you have a backups user on all your hosts but only want the backups user to be allowed to SSH into only the backup server.

Conclusion

This sounds like a lot, but definitely give it a try if you have a bunch of servers you log into! This is definitely more useful on larger scale deployments, but even in my home of a handful of servers I find it nice to be able to just create a key, sign it, and I can sign into all my hosts with ease, no needing to copy around keys or copying public keys to all my authorized_keys files.

#100DaysToOffload #Post2

Device Busy

Courtney — Mon, 05 Feb 2024 15:08:57 PST

Once a week I pop out the microSD card on my church’s recorder and get the sermon uploaded. I do this on OpenBSD, and also mount/umount the microSD card via the command line. On another note, I use the SSH ControlMaster feature when logging into my servers. Why is this related? You’ll see. The other day I mounted the microSD card

doas mount /dev/sd3i /mnt/usb

Then cd’d to that directory to copy out the music file. While I was there I SSH’d into a server for some reason. I finished my copying of the audio, exited the terminal, edited and uploaded the Sunday sermon. Then went to umount the microSD card

$ doas umount /mnt/usb
umount: /mnt/usb: Device busy

Huh. I wasn’t cd’d into the directory, no open files, I checked fstat

$ fstat -f /mnt/usb
USER     CMD          PID   FD MOUNT        INUM  MODE R/W    SZ|DV
courtney ssh        40229   wd /mnt/usb        2  drwxr-xr-x r     4096
$ ps -ux | grep 40229
courtney 40229  0.0  0.0  2312  2664 ??  Ip     12:14PM    0:00.00 ssh: /home/courtney/.ssh/controlmasters/courtney@example.com22 [mux] (ssh)

Well okay then. Why on earth is my control master file there? I found that strange. Turns out, this isn’t some SSH thing. For some reason in the UNIX world if you create an open file from your shell, it creates a file descriptor from that directory, something I wasn’t aware of. I thought since I wasn’t cd’d into the directory and that the file was on my /home partition, this wouldn’t be a thing. Sure enough, closing the SSH connection solved the issue

$ ssh -O exit example.com
Exit request sent.
$ doas umount -v /mnt/usb
/dev/sd3i: unmount from /mnt/usb

All good. I tried this with httpd out of curiosity. I started httpd from /mnt/usb

# httpd -f /etc/httpd.conf

This had the same effect. I had to stop httpd so I could then umount /dev/sd3i. So many small things that you may never encounter unless you really know operating systems, or have a random encounter like I did here, I guess.

#100DaysToOffload #Post1

OpenBSD Is My Desktop of Choice

Courtney — Tue, 31 Oct 2023 17:28:44 PDT

It has been almost a year and a half now of using OpenBSD on my desktop. From previous blog posts I have noted some of my frustrations with the desktop experience in OpenBSD, and in some ways, these have been more or less addressed. So, why do I really love using my computer with OpenBSD?

sndiod

This may sound funny, since in my previous blog posts and in other public forums, I have frequently complained about OpenBSD audio. I still do on occasion get that stutter with USB audio. My solution was to buy a new DAC (the Audioengine D1, great device), which sounds better anyway than my previous FiiO E10k, but also means I only have to restart Firefox every maybe 1-2 days instead of every half hour. There was no solution for that. I also have a problem with a USB audio input device that has a ticking audio sound. Couldn’t be resolved. It probably requires an OpenBSD developer to make some changes in the kernel.

All that said, these problems are offset with audio issues in other operating systems. Like with Linux, when an issue arises you have to have familiarity potentially with alsa, pulseaudio, and pipewire now. I actually like pipewire, but really, I find the documentation hard to navigate. I really love sndiod because it really has been a set and forget sort of thing for me. It reliably works all the time. The man pages are simple, the FAQ for audio is simple, I don’t have to go very far if I have a problem.

Release or Current?

Since the beginning of using OpenBSD on my desktop in May 2022, I was using -current. I really had no issues to speak of. It’s honestly very rare that I would encounter an issue. But, it is a development branch. I did have an issue on occasion. For all of my uses with OpenBSD -current, I just do my usual sysupgrade -s and update packages and carry on. However, I did just reinstall my desktop with 7.4. I’ve hit a bad sysupgrade a couple times and it did cause me to lose time at work trying to fix the issues. Those minutes aren’t mine, and the problems were likely entirely avoidable if I had sat on a release. So, I moved my desktop to the release branch, but I do have my laptop (and a few servers) on -current still. It is great flexibility. I like that on one hand, I can camp on a release and have things hum along without any care in the world to speak of. -current requires a little more attention, but I am happy that I can keep up with the latest updates and ports tree for the applications where I need it.

I did like that every 1-2 weeks doing my sysupgrade and updating packages. However, there is also a good feeling knowing that the OS is gonna pretty well stay put for the next 6 months. So far, after 3 years of release upgrades, I have had 0 issues. I trust (and hope, we are talking computers) in April 2024 I will have a clean upgrade to OpenBSD 7.5 and have another 6 months of not having to bother with my computer.

How I Backup My Openbsd Desktop

Courtney — Wed, 25 Oct 2023 23:40:00 PDT

Table of Contents

It’s not often I backup and restore my desktop, but I do like having good backups for my desktop computer, especially since I rely on it for work. I use OpenBSD for my desktop operating system. I’ve been running -current since May 2022. 99.9% of the time it has been very good to me with no headache. Every now and then though I run into an issue, and it is something I can work around and maybe even just wait a day and sysupgrade. I’ve come to find right now though, I need something more predictable for my daily workstation since it is the tool I use to put dinner on the table. So, I decided today to downgrade to OpenBSD 7.4 from -current. I still run -current on a handful more servers and my laptop, but this machine I need to take a step back on. So, I got to verify my backup and restore plan actually works!

My Tools

I use Restic to do all my backing up, and I piece it together with some scripts. To get some of my config files, my setup is a little inconvenient, so I am pondering if I should do backups as the root user or something to get some of the files I don’t have permissions to read unless I escalate with doas. So, those files aside, let’s just pretend I back those up and they make it into my restic backups safely.

I backup over SSH to my FreeBSD server as the destination.

Scripts

Backing Up

Here is my backup script

#!/usr/bin/env sh

RESTIC_PASSWORD_FILE="$HOME/.local/bin/.restic_pass"
export RESTIC_PASSWORD_FILE
RESTIC_REPOSITORY="sftp:homeBackup:/usr/home/username/rstc-desktop"
export RESTIC_REPOSITORY

excludes="/home/username/.local/bin/EXCLUDEFILE"

cp /etc/X11/xorg.conf /etc/rc.conf.local /etc/doas.conf /etc/httpd.conf /etc/man.conf /etc/sysctl.conf /etc/fstab /etc/login.conf $HOME/Documents/desktop_configs/OpenBSD
pkg_info -m -z -- > $HOME/Documents/desktop_configs/OpenBSD/packages.txt

restic --compression=auto --exclude-file $excludes backup $HOME /var/www/htdocs/username
complete=$(echo $?)

rm $HOME/Documents/desktop_configs/OpenBSD/*


if [ $complete -eq 1 ]; then
        dunstify "Restic backup failed"
fi

Some files are missing here, notably my pf.conf file. I should make a better way of backing those up as said earlier.

My favorite line is:

pkg_info -m -z -- > $HOME/Documents/desktop_configs/OpenBSD/packages.txt

This does a fuzzy export of packages installed. Very handy.

Pruning Backups

Backups accumulate over time, and I don’t need the old ones. So I have a similar script that prunes backups:

#!/usr/bin/env sh

RESTIC_PASSWORD_FILE="$HOME/.local/bin/.restic_pass"
export RESTIC_PASSWORD_FILE
RESTIC_REPOSITORY="sftp:homeBackup:/usr/home/username/rstc-desktop"
export RESTIC_REPOSITORY

restic forget --keep-daily 10 --keep-weekly 5 --keep-monthly 3 --prune
complete=$(echo $?)


if [ $complete -ne 0 ]; then
        dunstify "Restic pruning failed"
fi

I have dunst installed so I can get desktop notifications of backups or prunes failing. So far, hasn’t happened yet and listing backups shows that to be true.

The restic_pass file is permissions 400 so only my user can read the file.

Cron

This is my crontab that runs these commands. I backup hourly, and at the end of the workday I clear out old backups

@hourly -ns /bin/sh /home/courtney/.local/bin/rstbkp
30 17 * * * /bin/sh /home/courtney/.local/bin/rstprune

Nuke

I prepared my flash drive following the faq and did a backup of my computer. I did a test restore of a few files to ensure my backups actually worked like I expected

$ RESTIC_PASSWORD_FILE="$HOME/.local/bin/.restic_pass"
$ export RESTIC_PASSWORD_FILE
$ RESTIC_REPOSITORY="sftp:homeBackup:/usr/home/username/rstc-desktop"
$ export RESTIC_REPOSITORY
$ restic restore --target restore/ -i /home/username/.Xresources latest

This restored the .Xresources file from the latest snapshot. It worked, I’ve done similar file/subdirectory restores in the past, I trust the process. Before nuke and pave time, I wanted to grab just a handful of directories to have separate from the restic repo so I could conveniently grab them. Just my ssh files in this case, so I could do my key sign into the server easily to restore my backups. I encrypted those and stored them elsewhere safely. Nuke and pave time.

I installed OpenBSD the way I wanted, then it was onward to restoration.

Restoration

I signed into root and first things first, I installed restic so I could restore my backups. Then I copied my ssh files back to my user. I then started restoring my backups:

$ RESTIC_REPOSITORY="sftp:homeBackup:/usr/home/username/rstc-desktop"
$ export RESTIC_REPOSITORY
$ restic restore --target restore/ latest

I was prompted to enter my restic password, then I waited for the restoration. Once done, I put all my files where they needed to go.

Restoring Packages

This is my favorite part about OpenBSD. The fuzzy export of package names is super handy. FreeBSD needs this (wink wink, nudge nudge). My backup contained a packages.txt file which I can install from doing this:

pkg_add `cat packages.txt`

Done. Easy.

Bow on Top

My data is restored, files are where they were before, packages installed, last bit was to syspatch and then reboot. I was greeted with a working system as it was before. I wasn’t missing much, since my snapshot was only a few days newer than the latest release, so now I don’t have to feel like I am lacking something I did not have.

I’m very grateful to the OpenBSD devs and their awesome work. OpenBSD -current is the most stable development branch I have used on any OS. It is expected to run into bugs. I knew that, and was still surprised with how well things ran all this time. But, I’m at a point in my work life where I cannot risk stability. Even a day or a little time of minor inconveniences is just a bit much for me at this moment in life. I’m sure it won’t be long before I find myself back on current though on my workstation. It really makes computing fun.

So yes..I am typing this now from my 7.4 installation and am content with where I am right now. Great operating system, and backup/restore with OpenBSD and restic is quick and convenient.

Creating FreeBSD Thick Jails

Courtney — Fri, 01 Sep 2023 21:06:11 PDT

In an older blog post I focused on creating FreeBSD base jails with VNET on ZFS. Today, I want to do the same thing with Thick jails, since I have some use cases where base jails don’t quite do what I want to do in terms of management.

Thick Jail?

A thick jail in creation is basically half the steps of a base jail. It is a jail with its own complete userspace, no nullfs mounts and the sort.

My reasons for wanting a thick jail is because I used base jails to have an easy to manage jail. freebsd-update the base they were built on, restart the jails, then I’m done. But to keep my jail versions pure and be able to delete older FreeBSD versions, I’ve migrated my jails from base to base. For most, this is not a problem. However, I have grown very tired of doing this with my Nextcloud jail. Now yeah yeah I could use Ansible to do it all for me but for this single jail, that’s a bit much.

The thick jail is has its own copy of the FreeBSD userland, which then means it uses more space the more thick jails you add. It also means I have to independently manage each jail. That makes it easier for me with Nextcloud, and some other single use jails where it’s probably easier to just treat each jail as a virtual machine, so to speak.

Creating the Jail

First off, we need to create the jail. I’m going to start by making a new jails dataset and mounting it at /jails

zfs create -o mountpoint=/jails zroot/jails

Here is the foundation for everything. Now I’ll create a few other datasets for our releases and templates and running jails, as well as our first release dataset (14.0-RELEASE)

zfs create -p zroot/jails/jails/myjailname

Next, we need to download the base OS as well as lib32 for our jail. The contents should be extracted into /jails/jails/myjailname in the end.

fetch https://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/14.0-RELEASE/base.txz -o /tmp/base.txz

fetch https://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/14.0-RELEASE/lib32.txz -o /tmp/lib32.txz

tar -xf /tmp/base.txz -C /jails/jails/myjailname

tar -xf /tmp/lib32.txz -C /jails/jails/myjailname

Now let us update the jail contents

env UNAME_r=14.0-RELEASE freebsd-update -b /jails/jails/myjailname fetch install

env UNAME_r=14.0-RELEASE freebsd-update -b /jails/jails/myjailname IDS

Then, we can copy our /etc/localtime and our /etc/resolv.conf files into the jail

cp /etc/localtime /jails/jails/myjailname/etc/localtime

cp /etc/resolv.conf /jails/jails/myjailname/etc/resolv.conf

Jail Configuration

Now that the jail is done, we need to configure it to start it. This is done by editing the /etc/jail.conf file. For this single jail, it should look something like this

$jails="/jails/jails";

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
exec.system_user = "root";
exec.jail_user = "root";
mount.devfs;
vnet;
vnet="new";
allow.raw_sockets = 1;
devfs_ruleset="5";

myjailname {
        host.hostname = "myjailname";
        path = "$jails/myjailname";
        #securelevel="2";
        vnet.interface = "e0b_jeth";
        exec.prestart += "sh /usr/local/bin/jib addm jeth em0";
        exec.poststop += "/usr/local/bin/jib destroy jeth";
}

Note	em0 is my hosts interface and jeth is the name of the interface that `jib` expects

We will need to drop some basic configs in the jail soon such as an rc.conf and pf.conf file since this configuration is set up to run pf in each jail on its own.

Inside the Jail

Inside the jail, we need a working rc.conf and pf.conf. To start, my rc.conf looks like this

cron_flags="$cron_flags -J 15"

# Disable Sendmail by default
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

# Run secure syslog
syslogd_flags="-c -ss"

hostname="myjailname"
# NOTE the usage of jeth here with /etc/jail.conf
ifconfig_e0b_jeth="192.168.1.100/24"
defaultrouter="192.168.10.1"

pf_enable="YES"

And the pf.conf file (definitely be sure to configure this properly, this is only an example)

block all
pass out

Host Configuration

For the host, we need to setup a few things:

Enable the service
Get jib in the right place
Configure our interface (optional)

Enable the Service

sysrc jail_enable="YES"
sysrc jail_list="myjailname"

The above enables the jail server and jail_list will autostart the jail when the service gets started and stopped when the service stops

Get jib in the Right Place

Either copy or symlink jib to /usr/local/bin

cp /usr/share/examples/jails/jib /usr/local/bin/jib

Wrapping It Up

The jail should be able to start now. Here is one way the jail can be started

service jail start myjailname

And now it is possible to jexec into it

jexec myjailname

Now you can do whatever you wish with this jail like any other jail.

UPDATES

2024-01-15 - Updated release numbers to 14.0

My Matrix Server Migration

Courtney — Tue, 15 Aug 2023 00:20:00 PDT

Table of Contents

I have been paying to host my Matrix synapse server for 2.5 years now. It has been one of my favorite projects to host because I have found a great communities this way and have been able to learn a lot being on Matrix. The problem for me though is, the synapse server is a bit of a resource hog, and has grown to be expensive.

Synapse

Synapse is the current Matrix server, in addition to dendrite which has been a gradual work in progress. I think dendrite is ready to go as a replacement, apparently…anecdotally. However, there is no way to migrate from synapse to dendrite yet.

The biggest issue I have run into with Synapse is it is pretty resource intensive. I started out using a VPS with 2 cores and 1GB RAM, but with more activity, it couldn’t keep up and my server would basically DoS itself from the extra load and become unavailable. So, I upgraded to a 2 core and 2GB RAM server. That bumped my prices from $5 to $15 per month. A price I didn’t want to pay. But, I did it anyway.

My Setup

I have been running synapse on OpenBSD with PostgreSQL, in addition to hosting element-web on my server. With that added performance, the server is basically sitting idle most of the time, all the while I do need that extra RAM because of what synapse gobbles up.

My New Plan

I already paid for my server at home, as well as the electric bill and everything else. Why am I letting this perfectly good hardware go to waste? After hearing a good idea on the OpenBSD Matrix group, I decided it is a good idea to follow a similar idea. My home internet is already pretty good, so why not move synapse and PostgreSQL to my local server? This isn’t a comprehensive migration solution, but a loose documenting of some of the stuff I did to accomplish my task.

I don’t want to move EVERYTHING back my house, because I don’t want my home IP exposed. So my new gameplan is:

Set up an OpenBSD 7.3 virtual machine on my local server
Set up a site to site VPN for migration
Restore my data to this server
Spin up a new OpenBSD 7.3 VPS (my current one is 7.2 so might as well go afresh)
Set up all the relayd, httpd, etc. etc. on it
Set up a site to site VPN between the 2
Shut down the old
Dump and restore the database to my local server
Restore the last bit of data with an rsync
Turn on all the lights
Make DNS changes

With this done, let’s move forward.

Set up a Virtual Machine

My local server is running FreeBSD 13.2, and I am managing my VMs with vm-bhyve. Thankfully, I already created an OpenBSD 7.3 image, so I provisioned it.

sudo vm provision <long-id-here> matrix

I then used restic to restore backup configurations locally. Mainly user stuff, some config files and my element-web stuff. More will come later as I move along.

Set up a VPN

For this I chose WireGuard. It handles domain names, which works for my ever changing IP address here at home. I take advantage of Namecheap’s A+ record to handle my ISP’s dynamically given IP with ddclient. The WireGuard setup was easier than I thought.

This is only a temporary VPN, since it will be to conveniently get my data off the old server and onto the new. I will repeat this setup later, but with the new server. Same thing, different keys. And I will delete this temporary tunnel.

On my local server, I created /etc/hostname.wg0

wgkey mykeyLocal wgport 51820
inet 10.25.50.2 255.255.255.0
wgpeer pubkeyRemote wgendpoint remote.example.com 51820 wgaip 10.25.50.1

And on my remote server

wgkey mykeyRemote wgport 51820
inet 10.25.50.2 255.255.255.0
wgpeer pubkeyLocal wgendpoint local.example.com 51820 wgaip 10.25.50.2

On each, I also have firewall rules. I will tighten this down later, this is just temporary:

pass in on egress proto udp from any to any port 51820
pass on wg0
pass out on egress inet from (wg0:network) nat-to (egress:0)

Load the firewall rules pfctl -f /etc/pf.conf. Later I want to ratchet this down more so the VPN only works and is exposed between my home and the server. Another downside which I will have to figure out later, is to monitor if my IP changes and do something about adding it to PF. This is outside the scope of this, but…I’ll put my script in later in this post.

Restore my data

I have my new server set up with its own disk at /var/synapse to use. With my servers communicating, let’s sync up /var/synapse.

rsync -az /var/synapse/ root@10.25.50.2:/var/synapse/

Let it run, eventually it will be synced up. Later when I turn the old server off, I have to sync these again. Now that I think about it, Python versions are probably incorrect too so I will likely have to handle that since I build synapse instead of relying on ports. I just found this easier for my own taste since I don’t want to ride on OpenBSD -current (actually, that’s what I did in the end, more on that later).

Restoring Packages

When I backup my matrix server, I also run pkg_info -m -z — > $HOME/packages.txt as a part of my script. This dumps my installed packages in a fuzzy format so I can quickly restore them on the new server like this

pkg_add `cat packages.txt`

The New VPS

Now to set up the new server. Previously, I was partitioning this server by hand because I wanted to maximize space for /var/synapse. However, since I have plenty of free space on my local server to store my Matrix data, I went with the auto partitioning in the VPS since it won’t be storing anything.

This server will only be handling:

httpd
relayd
pf
sshd

I copied all these configs, as well as crontabs, my personal scripts, and user data to the VPS and set up my VPN as I did earlier. I did make adjustments to my firewall. I have a script running that will do a DNS check every minute to the authoritative DNS server for my domain name to see if the A record changed. And if it did, add the new IP address to PF by using a table, then removing the old IP address.

Here’s my configs for each file:

/etc/httpd.conf

types {
        include "/usr/share/misc/mime.types"
}

server "web.example.com" {
        listen on * port 8080
        location "/*" {
                root "/htdocs/web.example.com"
                #request strip 1
        }
}

server "example.com" {
        listen on * port 80
        alias autoconfig.example.com
        location "/.well-known/acme-challenge/*" {
                root "/acme"
                request strip 2
        }
        location * {
                block return 302 "https://$HTTP_HOST$REQUEST_URI"
        }
}

server "example.com" {
        listen on * port 443
        alias autoconfig.example.com
        hsts { subdomains, preload }
        location "/.well-known/acme-challenge/*" {
                root "/acme"
                request strip 2
        }
        location "/.well-known/matrix/*" {
                root "/matrix"
                request strip 2
        }
        location "/*" {
                directory auto index
                root "/matrix"
        }
}

/etc/relayd.conf

ip4="$publicIP4"
ip6="$publicIP6"

table <element> { 127.0.0.1 }
table <matrixserver>  { 10.25.50.2 } # the IP of my synapse server in the VPN's network

log connection

http protocol "https" {
        tls { tlsv1.2, tlsv1.3, ciphers secure }
        tls keypair "example.com"
        tls {session tickets }
        match header set "X-Forwarded-For"   value "$REMOTE_ADDR"
        match header set "X-Forwarded-Proto" value "https"
        #match header set "X-Forwarded-Proto" value "SERVER_ADDR:$SERVER_PORT"

        match response header set "Server" value "flippinburgers"

        # set CORS header for .well-known/matrix/server, .well-known/matrix/client
        # httpd does not support setting headers, so do it here
        match request path "/.well-known/matrix/*" tag "matrix-cors"
        match response tagged "matrix-cors" header set "Access-Control-Allow-Origin" value "*"

        pass request quick header "Host" value "web.example.com" forward to <element>

        # pass on non-matrix traffic to webserver
        #pass                                 forward to <webserver>
        #pass                                 forward to <element>
}

relay "https_traffic" {
        listen on $ip4 port 443 tls
        protocol "https"
        forward to <matrixserver> port 8008 check tcp
        forward to <element> port 8080 check tcp
}

relay "https_traffic6" {
        listen on $ip6 port 443 tls
        protocol "https"
        forward to <matrixserver> port 8008 check tcp
        forward to <element> port 8080 check tcp
}

http protocol "matrix" {
        tls { tlsv1.2, tlsv1.3, ciphers secure }
        tls keypair "example.com"
        block
        pass quick path "/_matrix/*"         forward to <matrixserver>
        pass quick path "/_synapse/client/*" forward to <matrixserver>
}

relay "matrix_federation" {
        listen on $ip4 port 8448 tls
        protocol "matrix"
        forward to <matrixserver> port 8008 check tcp
}

relay "matrix_federation6" {
        listen on $ip6 port 8448 tls
        protocol "matrix"
        forward to <matrixserver> port 8008 check tcp
}

/etc/pf.conf

table <homeip> persist file "/etc/homeip"
table <bruteforce> persist
table <rfc1918> const { 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8 }

set skip on lo

set loginterface egress
match in all scrub (no-df random-id max-mss 1440)
antispoof quick for egress
block in on egress from { <bruteforce> <rfc1918> } to any
block in log all
pass from (self)
pass out on egress

pass in on egress inet proto tcp from any to (egress) port { http, https, 8008, 8448 } keep state
pass in on egress inet6 proto tcp from any to (egress) port { http, https, 8008, 8448 } keep state

pass proto tcp from any to (egress) port ssh \
        keep state (max-src-conn 15, max-src-conn-rate 5/3, \
                overload <bruteforce> flush global)

pass in on egress proto udp from <homeip> to any port 51821
pass on wg1
pass out on egress inet from (wg1:network) nat-to (egress:0)

pass in on egress inet6 proto icmp6 all icmp6-type { echoreq routeradv neighbrsol neighbradv }
pass in on egress inet proto icmp icmp-type { echoreq, unreach }

# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010

# Port build user does not need network
block return out log proto {tcp udp} user _pbuild

And here is my script for checking the IP and updating my firewall rules. This runs every minute in cron

/usr/local/bin/changeip

#!/usr/bin/env sh

currentip=$(dig +short @dns1.registrar-servers.com myhomeip.example.com)
fileip=$(cat /etc/homeip)

if [ "$currentip" != "$fileip" ]; then
        echo "$currentip" > /etc/homeip
        pfctl -t homeip -T add "$currentip"
        pfctl -t homeip -T delete "$fileip"
fi

Note	This script isn’t tested, my IP hasn’t changed yet :)

With all these configuration on hand, I should be able to accept connections and pass them along back to my home server’s virtual machine, once that part is ready. Last bit is the WireGuard tunnel part. It’s basically what I did earlier between my old VPS and my VM at home, but change the IP address to something different for the new VPS.

Moving to the New Server

For moving to the new server, I turned off synapse on the old server so I wasn’t getting new data. I then did a dump of the database and a restore to the database on the new one. I basically followed the Option 1 method in /usr/local/share/doc/pkg-readmes/postgresql-server to get this.

Admittedly, I didn’t really change my pf rules a whole lot on this server since the only publicly listening service is synapse. And I set my bind address in the homeserver.yaml file to be the VPN address. And only servers in this VPN tunnel can talk with each other. And, it’s a public endpoint anyway and with my router and all in the way, it’s not a threat for me.

Now that PostgreSQL is restored to my local database I did one last rsync of my data and made a couple tweaks to the homeserver.yaml file, like the bind address.

The Problem

Earlier I said I did not want to play the OpenBSD -current game with this server. Well, meh. I was having troubles getting the server to work right with building synapse for OpenBSD 7.3. I could have made it work, but I didn’t want to deal with it. So I upgraded to -current and am now using the synapse package that is in ports. I feel safer with this because I can easily and quickly power off my VM and do a ZFS snapshot of it before doing my sysupgrade.

Wrapping Up

The last part then was to start the synapse server, ensure relayd and httpd were playing nice on my VPS. I made changes to DNS to ensure my domain names were pointing to the VPS, and then I got new SSL certs. I then made sure that everything was talking with DNS propagated, and it was! I was able to chat over Matrix again.

I got backups working of my synapse server too. I don’t care to backup my web server. That’s all documented here so I can copy/paste ;) I’ll probably back it up anyway but it isn’t a priority at the moment of writing.

So yeah. That was my migration process. I’m saving a few bucks a month now. I think performance is a bit better. I get a 404 here and there though that I need to diagnose. Maybe I’m losing connection between my home server and this VPS but it is only ever for a minute right now at any given time. Not the biggest issue, and it is a bit expected.

Setting Up An OpenBSD Mail Server

Courtney — Tue, 07 Mar 2023 18:45:30 PST

Table of Contents

I’ve mulled over setting up a mail server for years. I tried once 5 years ago, and failed. I don’t know why I wanted to do this, it’s just something I wanted to do. So I did.

I have been on a kick to move a lot of servers to OpenBSD. I love OpenBSD. It is simple (as in simplicity) to use, and I find it easier for me to maintain more than other operating systems. I also liked having OpenSMTPD built in. I read a lot of articles, and found none of them really worked. I guess email isn’t a one size fits all thing. I pulled together everything I found and built it my own way. I’m sure the way I did it will be yet another one of those blog posts. Works for me, doesn’t work for you.

Here, I have this documented for myself, and hopefully it can help you. I didn’t understand a LOT of what I was setting up. I really learned a lot as I did it. I would encourage reading the man pages as you go and documentation. There’s a lot of moving parts. I’ll do my best to explain some of this stuff. So here we go!

OpenBSD

First, you need a stable provider. If you want to send mail, you want a VPS provider with good uptime, will give you a clean IP, and of course you want one that will let you install OpenBSD.

One thing that sucks is if you want to send emails to the bigger email companies like Google, Microsoft, Apple, you may find out you are SOL only when you have everything set up. You can plug your IP into a site like MXToolbox and see for yourself if you have a clean IP, but these other companies have their own blackbox lists and you probably won’t find out until you send your first email to them and get the sad bounce. Vultr could be a good option, but you have to ask for permission to open the right ports. This usually means you won’t get blocked by the big email companies. Currently, I am blocked on RamNode’s network. Hoping they can resolve the issue, otherwise I have to move everything and try again myself.

Initial Setup

Once your system is up and running, you’ll want to do your syspatches and what not. You can do any other tweaking you might like here. Assign your IP addresses, go set your A and AAAA records as well as your PTR (or rDNS) records. I set these records:

A - mail.example.com CNAME - smtp.example.com CNAME - imap.example.com

Faster SSH Connections with Control Master

Courtney — Sat, 07 Jan 2023 18:45:30 PST

Table of Contents

Many years ago I was told about a way to speed up my SSH connections to servers I have recently made a connection to. The idea was that a session would be kept open for a certain amount of time, even when you had closed out of your sessions. I couldn’t remember the name, so I forgot about it until today when I was trying to find out ways to speed up my Ansible connections.

If you’re like me, you may have many SSH connections opened up to the same server. This trick will make it so your first connection to that server is slow, but any following connections to that server will be sped up because you will have an already opened multiplexed connection.

How to Setup

Setting it up is simple. You can either set this in your system-wide ssh_config file at /etc/ssh/ssh_config, or set this on in your own user’s ssh_config file at ~/.ssh/config.

These are the lines I set:

Host *
	IdentitiesOnly	yes
	ControlPath   	~/.ssh/controlmasters/%C
	ControlMaster 	auto
	ControlPersist	60m

ControlPath is where the control socked will be stored. %C will be a hash and not human readable. If you want it human readable, you could do something like %r@%h:%p.

ControlMaster is what lets you create the sessions and handle the control sockets. auto will do it automatically for all hosts, and you can specify autoask if you want to be asked to confirm it.

ControlPersist says how long that session will stay active in the background when you close your session. So if you disconnect, and remember you wanted to do something on that server, you won’t have to wait to establish an entire new connection, you will connect of the session that is sitting in the background.

Resources

My OpenBSD Audio Woes and My (Hopeful) Resolution

Courtney — Mon, 28 Nov 2022 18:45:30 PST

Table of Contents

I have been an OpenBSD desktop user since May. In my blog post, I mentioned I was having some audio woes with OpenBSD. Namely, when running Firefox on OpenBSD it wouldn’t take long for the audio to sound like someone was taking a hammer to my ears. Now that may be an exaggeration, but someone might as well have because the stuttering frustrated me to no end. I tried a handful of solutions:

Reducing core usage in Firefox
Switching Firefox cache to use an mfs filesystem
Handful of knobs to reduce as much I/O to disk as possible

No luck.

The Trouble

After (likely) annoying people too much on the OpenBSD Matrix channel (sorry!), I have concluded a couple things:

USB on OpenBSD sucks
Audio on OpenBSD sucks

Now in many ways I most certainly feel like an outsider to OpenBSD. The community is great, but because of the way the OpenBSD dev community operates, it is perhaps unlikely to get anywhere with any sort of audio bugs. After perusing the bugs and misc mailing lists, there seems to be a lot of unanswered and unresolved threads relating to audio issues.

I get it. OpenBSD is a small group, basically all donation based, and there are so many hands. So on one hand I feel my complaint is valid, but my expectation to have something fixed simply should not be an expectation. I have pretty well learned that a lot of OpenBSD is by and for the developers. If you aren’t one, then, well, you probably aren’t going to get what you want. I don’t fault the project with this. In fact, even if there are some things that frustrate me, such as this audio issue, I really appreciate what the project has become because of their approach to who contributes what.

USB on OpenBSD Sucks

At first, I pointed my fingers at Firefox. After Firefox 91 ESR these problems ensued. But this seems to only be a problem because (as the devs tell me) USB is far from perfect on OpenBSD, and yes, apparently the stack is flimsy enough that a (heavy) browser can thrash the crap out of it.

New Hardware Time

Since I pretty well decided I won’t be using chromium browsers anytime soon, and there’s probably no way I could expect USB to get to a place I want (maybe if I had donation money and someone who would take up the work), I decided to try out S/PDIF audio.

I bought an AudioEngine D1 DAC deciding that if USB audio sucks hardcore, then maybe using the S/PDIF out on my motherboard could bypass that and I’ll get good sound.

Boy, I didn’t think it could possibly get worse. It seems this issue has been mentioned in the past at least twice

No resolution. Basically, not too long after use, whatever is playing constantly loops the last second of audio until finally sndiod just says "NOPE" and now nothing else will play. Restarting sndiod doesn’t work. In fact, the daemon successfully runs, and running in debug mode shows nothing once this happens. You have to restart your computer.

I’ve had a 1 hour session of no issues, and a session last half a song. It is unpredictable.

Audio on OpenBSD Sucks

So, USB audio stutters, the azalia drivers seems to just blow up over S/PDIF, and standard headphone jack may be usable, but I want good sounding audio darn it!

So…What Now?

Well, my previous DAC was a FiiO E10k. Stuttered horribly on OpenBSD. SO FAR this AudioEngine D1 does not seem to exhibit this behavior over USB. I am REALLY hoping this continues to function okay.

This seems to be a very hardware-specific issue over USB then for audio. So in conclusion, I will +1 the AudioEngine D1 for OpenBSD. It sounds good, is fairly compact, and does not stutter when running Firefox on OpenBSD.

My Home Server

Courtney — Sat, 29 Oct 2022 19:45:30 PDT

Table of Contents

I have been self hosting since about 2015. It is probably something I could say is a decision that I regret the least and have enjoyed the most. When I say regret the least I mean it has been a rewarding endeavor for me and has given me much value since I started using it. I want to share my history of self hosting, what my setup is now, and my thoughts on why you should do so yourself. If you want to skip the personal stories, you can go here

Where I began

To quickly recap my history prior to self hosting, I started using Ubuntu 13.04 when I started learning more about computers. I started using Linux to prep for my college education, since I found one of my upcoming courses in my first year of my cyber security program I would learn Linux, and I saw it was something I could get a head start on.

What started my desire to self host was a need I found I had with my school work. I would work on my laptops and desktop and wanted easy syncing access between my devices for my school work. I conveniently had a Raspberry Pi 1B at the time and thought to set up Syncthing on it so I could sync my school work. So, I set up Raspbian 7 on it, installed Syncthing and let it do its magic.

That did well for about a year. At that time I began to think "gee, I sure would love to synchronize calendars and contacts between my phone and computers". This was in 2016. I was about to hop onto ownCloud, but around that exact same time that I was playing around with it, the split to Nextcloud happened. I already knew how to do a LAMP stack at the time. I had done it dozens of times to learn how to build simple servers, normally setting up Wordpress and not doing much with it. I had also purchased a Pi 2 around that time, then set up Nextcloud 9 with the Raspberry Pi and had a 32GB flash drive plugged in. I stopped using Syncthing at the time and used Nextcloud’s syncing instead and utilized its WebDAV support to handle syncing calendars and contacts with my iPhone and Thunderbird client.

I had a great time running Nextcloud. In fact, I’m still running the same database that I started out with with Nextcloud 9 in 2016. Eventually, I introduced the idea to my family, and they liked it. So in the nature of me using more files and letting my family use Nextcloud with me, I got the Nextcloud Box that came with a 1TB hard drive. I got an upgrade from my 32GB flash drive then, woohoo!

From Pis to Towers

I ran my Pi server for quite some time. However, I wanted to do more. Just having a Nextcloud box in my house wasn’t enough. I wanted to try out different things. Also the Pi wasn’t keeping up with my use. I actually didn’t stick with the Nextcloud Box too long. After 4 months I got an HP Proliant ML110 G6 off eBay for $110. I upgraded it to 16GB and switched to Ubuntu Server 16.04. I wanted to also increase storage, so I got some refurbished (mistake #1) enterprise HDDs so I could do LXD containers and ZFS to mirror the disks and "jail" off my services.

I set up a bunch of random stuff. I made a website for one of my classes with just HTML, CSS, and a little JS, moved Nextcloud over, got into running Plex, samba server, and some other stuff I forget.

Changing Hardware to Changing Operating Systems

I was really digging my setup. Ubuntu 16.04 was great, I really loved the simple command line interface for LXD, ZFS was running solid and I loved the data protection and easy disk mirroring it offers. However, by July of 2017 I was wanting to switch operating systems. At that time there were murmurs of Canonical starting an IPO. I didn’t like the idea of my OS being dictated by the mandates of meeting quarterly earnings and becoming less and less community focused (granted, it probably wasn’t all that much to begin with). I really liked my LXD/ZFS combo. Nobody in the Linux distro community was quite as bold as Canonical for offering an easy way to run ZFS, and nobody was offering LXD.

At that point, the thought struck me. I had the opportunity to talk with Allan Jude and Rodney Grimes at a Linuxfest a few times. I remembered each time I’d talk to these two gentlemen they’d blow my mind with how much they knew. Clearly they were leagues ahead of me. Being one of the uppermost Linux gurus in my classes (and really, the rest of the Linuxfest folks advertising their stuff on more of a social merit than a technological one), I felt like I had never come by people so smart when it came to operating systems. Sure there were other booths of geniuses, but no one talked tech like these two. At that time, I investigated FreeBSD. I found out pretty quickly that FreeBSD offered better ZFS integration, and iocage was an excellent replacement for LXD. In fact, a lot of the commands were very similar.

Note	Don’t use iocage. It’s dead. You can read my older article on how I do FreeBSD jails now

I put FreeBSD on an extra computer, set up identical jails and tried it out. After a month of playing around, I didn’t feel fully comfortable with FreeBSD yet, but it felt like the replacement for me. It fit my needs exactly. So, I went through the process of reinstalling the OS on my main server and rebuilding everything on the new installation, and restored data from backups to get Nextcloud back and running among my other content.

More New Hardware!

Around 2019 my demands grew larger with my home server. I wanted to start virtualzing operating systems, and for some reason my processor would not virtualize. It was time for an upgrade. I found myself an HP Z440 with a 6 core Xeon and 64GB DDR4 ECC for $300! Wow! I moved my drives over aaaaand failed to boot. My old system was doing a traditional BIOS boot and the new server was an EFI system. So I had to reinstall. I think that time I had used the chance to upgrade to 12.0 or whatever the latest FreeBSD was.

Finally, last year I wanted to upgrade from my 120GB drives to 500GB so I could do more and utilize the speed. I bought new drives and made some big changes.

Rearranging My Jails and Data

In my initial setup, I made the mistake of putting my jails on my hard drives and lumped it in with my data. I found out pretty quick it is not fun to couple jails and data. Sure, you can. But I was starting to like the idea of thin and thick jails and utilizing ZFS mounts better. At the same time I also got 2 more 4TB drives. I was going to take my old 2 and existing 2 and do a striped mirror of sorts, but I don’t have 6 SATA connectors and I wanted my system on the new stuff. So I detached a drive, turned my pool into a raidz1, and built my jails on my SSDs, then copied my data to new datasets on my raidz1 zpool.

When I was done I had my jails on my SSDs, started making VMs to put on the SSDs, and had my data separated on my drives. This made managing much more flexible.

Where Am I At Now?

Now, in the current time I have the same existing setup, however, I moved away from iocage. Like I said earlier in a note, I changed how I did jails and switched to FreeBSD’s built-in management. This works out just fine for my own server use. There are great tools out there such as Bastille BSD, but I wanted basic tooling. Bastille is very advanced for the simple stuff I do.

I also have my network set up with some fun gear. It is still a work in progress though. In January of last year I install OpenBSD 6.8 on my router and made my first OpenBSD router, having switched from pfSense. Today, I have upgraded that router to 7.2 and it has been great. It is running on a PC Engines APUc4. The 3rd port isn’t used. I intend to learn vlans on OpenBSD and FreeBSD and break out my jails and VMs more, since it has too much access to my home network as it is.

Why You Should Self Host

Simulate Packet Loss with iptables

Courtney — Wed, 28 Sep 2022 19:45:30 PDT

I work primarily with Linux servers during $dayjob and sometimes come across needs where I need to have packet loss. Of course, I can’t be sitting around waiting for it to happen. Turns out, there’s a pretty simple way to do this.

iptables -A INPUT -s <SOURCE_IP_ADDRESS> -m statistic --mode random --probability 0.1 -j DROP

You can find more info in the iptables-extensions(8) man page. There are some other settings beyond random and probability to make packets drop. So, if you need a more precise kind of packet loss, you could do something like drop every nth packet.

Fixing OpenBSD Bootcode

Courtney — Wed, 28 Sep 2022 19:45:30 PDT

Table of Contents

Back in April(?) of this year when I got my AMD RX 6700XT graphics card I came across FreeBSD still not supporting the card. So, I switched my desktop to OpenBSD. In that same step, I had to fix Windows because it put some of its bootcode on my FreeBSD drive. Long story short, I switched to Linux to do my gaming for a few months because Windows wouldn’t install. My last desktop died, and over time found Linux didn’t fit my needs entirely and switched back to Windows.

Windows Install Woes

Windows is dumb. In the installer, I went to wipe my Linux drive (good luck figuring out which drive is which for starters…disk 1,2,3 is SO descriptive). When I selected the option to configure my now empty drive to be my install target, Windows complained that my System partition was too small. What System partition? This is an empty drive!

…ahhh… said I, when I realized Windows wanted to be the only OS on my machine. In other words, it said "that OTHER System partition must go", being my OpenBSD EFI partition. I wanted Windows, so I had no choice. Bye bye OpenBSD EFI partition. I had Windows up and running, now to fix OpenBSD since it was no longer bootable with the EFI stuff gone.

Fixing OpenBSD Bootcode

This part was a little tricky. I think this could have been easily achieved by booting into the OpenBSD live installer, but after talking with some people on the OpenBSD Matrix channel, I learned you can point to the existing drive at the boot> prompt from a live USB. Allegedly, at the boot> prompt you coan type diskinfo and get your disks listed? Didn’t work for me anyway. But I found out that typing boot hd1a:/bsd booted straight to my OpenBSD installation! It was just like I had never screwed up its boot code. For you it could be hd2a or hd3a or whatever so if you don’t get the expected outcome, try incrementing the drive number.

Now I was in my system…now what? Allegedly, installboot(8) is supposed to be a solution, as it says in the man page

installboot installs bootstrap on the specified disk

That didn’t work for me. With uncertainty, I went to the Using fdisk section of the OpenBSD FAQ where it says:

A totally blank disk will need to have the master boot record's boot code written to the disk before it can boot. Normally, all you need to do is:

Followed by the command:

# fdisk -iy sd0

In the past, I did this following the GPT section shown below in the Full Disk Encryption section. There, it specifies the command for GPT for UEFI booting is different:

# fdisk -gy -b 960 sd0

So, I did that since my drive is in fact sd0. Did a reboot and bing bang boom, my computer booted OpenBSD again!

If You Read This In The Future

I wanted to make a note here that since this is a "how to fix X" article to always reference the latest documentation WITHIN OpenBSD and their FAQ pages in case anything changes. But, if you landed in the same predicament as me and don’t want to nuke your install, this should be your fix.

FreeBSD Backups with Zap

Courtney — Tue, 30 Aug 2022 19:45:30 PDT

Table of Contents

A while back I tried to develop a script to handle ZFS backups from my server to my external hard drive. Well, in the end I really didn’t like what I came up with and thought it best to look at sysutils/zap to do my backups. The tool has worked great for me and reliably does snapshots and sends them to my destination. I have ran into some issues though. Maybe this will come by the developer or maybe I can learn to make the improvements I would like to see…or maybe there’s something better! But, I’d rather improve this tool than redo backups if it came down to it.

My Backup Process

I have a 3-2-1 process of doing backups

3 copies of data 2 on different media 1 offsite

I won’t go through any of the setting up because the GitHub repository and a blog linked in the developer’s website is plenty sufficient to get started.

With it set up, I rotate my drive out every month. Since this is just me and my family, I don’t have a more rapid rotation schedule. I have a cron job that will daily take a snapshot that has a TTL of 6 weeks, and once a month takes a snapshot with a 3 month TTL.

#!/usr/bin/env sh

zpool import backups
/usr/local/bin/zap rep -Fv
/usr/local/bin/zap destroy
zpool export backups
camcontrol standby da0

This imports the zpool, does the replication, deletes old snapshots, exports, and lastly does a standby on the drive so it powers down until next time.

A Problem with Rotations

Not too long ago, I found there was an issue with drive rotation. Namely, there is an order to how to get the drive rotated in up to date. If I just plugged in my drive and did a zap rep, I would not get everything to sync or get errors. That is because there is a mismatch in snapshots between the source and destination. So, I found the best method is to

import the zpool
run zap destroy to delete old snapshots on the backup drive
run zap rep -Fv to get everything up to date

The other way around does not work as you would expect.

Lazy Roation?

Just last night I came across an issue where I had everything automated and "set and forget" my backups for a couple months.

Important

Bad idea by the way

I found out 2 datasets were not syncing. Somehow I missed them somewhere and I had all my ZAP snapshots get destroyed, likely due to no new ones getting sent over somehow and then getting destroyed because they expired.

With that incident, I was not getting any new ZAP snapshots syncing to my external hard drive. I wish I had the exact message, but the issue was that there was an older snapshot on my external drive that wasn’t taken by ZAP, and thus it would not send a snapshot.

My solution then was to delete all snapshots on the destination for that ZFS dataset, then zfs send data/my/snapshot@ZAP_oldest_one | zfs receive backups and follow that with a zap rep -Fv which would sync from my oldest ZAP snapshot to my newest one. I did that for the 2 datasets I had with this issue and backups were properly underway.

OpenBSD Cron Jobs and Borg: A Lesson in Learning to Read Slower

Courtney — Thu, 25 Aug 2022 19:45:30 PDT

For the longest time I was running borg on my server to do backups. Then one day I changed my destination’s name to something different (different domain name). I know with borg, if the backup destination changes, borg will prompt you on the next run that the destination is different than before, and to confirm the change. I ran my borg backup script, confirmed the destination and all was good… until next run. Suddenly, I found my backup script failing. Wondering why, I appended > /root/backup.log 2>&1 to the end of the cron job so I could see why.

Warning: The repository at location ssh://backup@mybackups.example.com/usr/home/backup/borg was previously located at ssh://backup@oldbackups.example.com/usr/home/backup/borg
Do you want to continue? [yN] Aborting.
Repository access aborted

Obviously since this was a cron job, I wasn’t there and the backup aborted. So, I went in and removed the borg data in $HOME/.cache and $HOME/.config. No dice.

Turns out, I should be a better reader, because as I typed this I thought I came across a hidden "WAT" decision. But if you look at the top of the root user’s crontab (which is the user that ran my backup script) you will see HOME=/var/log. Sure enough, my borg data was showing up in /var/log. So in my backup script I set HOME=/root so I didn’t mess with any defaults.

Now, I’m not sure why the default is /var/log, but if there is one thing I have learned from using OpenBSD is to read carefully and you will get the answer. These days it becomes so easy for me to skim something and look for the quick answer, and then miss out on the important and obvious details. I constantly fall into this trap, but I think over the last year of me using OpenBSD full time in some respect, I have learned about some personal setbacks in how I approach technical problems.

If you are like me and are trying to take a shortcut, maybe invest some time too in learning to take it slower and really think critically about what your problem is, and not be so quick to go to a forum or something. Critical thinking and patience are good traits to have.

0 Dependency Websites with OpenBSD & AsciiDoc

Courtney — Mon, 27 Jun 2022 19:45:30 PDT

Table of Contents

Note	I want to clearly state this is 0 dependencies on the server. The server does not need AsciiDoc installed to operate. AsciiDoc is for the client to use. The server operates with 0 `pkg_add`s

I have fallen in love with a recent combination of software to make good looking websites, and having an easy to manage web server. I’m a minimalist in many ways. Really, I find that it makes my life easier. I like to keep my blog up here and don’t want to deal with database updates, language exploits, weird migrations, and so on. I also manage my church’s website. I can’t be having complex solutions that require large amounts of maintenance at multiple times a month because of a dozen pieces of software requiring version upgrades or what have you. Sure, some people have needs for elaborate content management systems, but sometimes you only need a simple site too, and why overcomplicate it?

This is why I like my current combo:

OpenBSD
AsciiDoc

OpenBSD is a simple operating system with simple management needs. I also like AsciiDoc because it is very easy to write content with (like right now for this blog), the output looks nice and clean, and it’s basic script away from being simple HTML/CSS.

OpenBSD

Why do I like OpenBSD for this job? I like OpenBSD because it comes with batteries included

webserver
firewall
acme client for SSL certs
openrsync for backups

I also like its strong focus on security, as well as how easy it is to maintain the server with the way I am demonstrating here. Really, the only commands someone has to know to do upkeep is:

syspatch
reboot
sysupgrade

Next, I will talk about how I configured my server to run a simple, static site.

Firewall Configuration

Firewall configuration only requires you edit the /etc/pf.conf file. This is what I needed to make my webserver work.

table <bruteforce> persist

set skip on lo0

block in log all
pass out on egress

pass in on egress inet proto tcp from any to (egress) port { http, https } keep state
pass in on egress inet6 proto tcp from any to (egress) port { http, https } keep state

pass proto tcp from any to (egress) port ssh \
     keep state (max-src-conn 15, max-src-conn-rate 5/3, \
     overload <bruteforce> flush global)

pass in on egress inet6 proto icmp6 all icmp6-type { echoreq routeradv neighborsol neighboradv }
pass in on egress inet proto icmp icmp-type { echoreq unreach }

block return in on ! lo0 proto tcp to port 6000:6010
block return out log proto {tcp udp} user _pbuild

Enable these rules with pfctl -f /etc/pf.conf

acme-client configuration

OpenBSD comes with a built-in acme client that you can use to generate new SSL certificates. I get mine from Let’s Encrypt. I then create a cron job under the root user to run every so often to see if there is a renewal that can happen, and then reload.

Here is my configuration for the /etc/acme-client.conf file

domain example.com {
       domain key "etc/ssl/private/example.com.key"
       domain certificate "/etc/ssl/example.com.pem"
       domain full chian certificate "/etc/ssl/example.com.fullchain.pem"
       sign with letsencrypt
}

Then I add the cron job as the root user with crontab -e

30 0 * * 1 acme-client example.com && rcctl reload httpd

Webserver Configuration

I use OpenBSD’s built in HTTP server, which does its job well. They have a file at /etc/examples/httpd.conf that you can use and mock up. Here is mine

server "example.com" {
       listen on * tls port 443
       hsts preload
       tls {
           certificate "/etc/ssl/example.com.fullchain.pem"
           key "/etc/ssl/private/example.com.key"
           ciphers "HIGH:!aNULL"
       }

       location "/*" {
           root "/htdocs/example.com"
       }
}

With the SSL cert and the web server configuration set, you can start and enable the server:

rcctl enable httpd; rcctl start httpd

doas.conf

You can use doas to make superuser configurations without always being root. Edit the doas.conf file and add

permit persist <username> as root

AsciiDoc

So, you might be asking why I chose AsciiDoc. I like AsciiDoc because it is a simple markup language that lets me focus on my writing and layout. Markdown can do this too, and is more common than AsciiDoc. However, the fact that there isn’t a standard Markdown and I don’t like any of the rendered output. With AsciiDoc, they give a simple and clean default result in HTML with CSS. I also like that there are other themes I can use readily, or I can make my own. But I end up using the default theme and the dark theme that is available.

With this in mind, it makes it easy for anyone to see what is written and figure out what does what since the syntax is simple. And, if you use editors like Visual Studio Code or AsciiDocFX, you get a nice preview. For me, I use vim. Here is an example of what the Webserver Configuration section looks like:

[#webserver-config]
=== Webserver Configuration

I use OpenBSD's built in HTTP server, which does its job well. They have a file at `/etc/examples/httpd.conf`
that you can use and mock up. Here is mine

[source]
\----
server "example.com" {
       listen on * tls port 443
       hsts preload
       tls {
           certificate "/etc/ssl/example.com.fullchain.pem"
           key "/etc/ssl/private/example.com.key"
           ciphers "HIGH:!aNULL"
       }

       location "/*" {
           root "/htdocs/example.com"
       }
}
\----

With the SSL cert and the web server configuration set, you can start and enable the server:

`rcctl enable httpd; rcctl start httpd`

Note	The source section starts with a `\` so as to not interpret it literally here :)

CSS Styles

To get the best out of CSS styles if I am not using the builtin, I like to do what I am doing here for my site:

:stylesheet: https://blog.passwordclass.xyz/css/dark.css
:linkcss:

How I Generate My AsciiDoc

I use a simple script to pull together my content, as well as generate my AsciiDoc into HTML format. I also want to note that I did create some extra custom HTML for my page to be inserted so I could offer a navbar, since in some cases I have come by the Table of Contents feature of AsciiDoc is good, but other times people wanted a standard navbar.

#!/usr/bin/env sh

CWD=$(pwd)
DEST="$CWD/example.com"

rm -rf "$DEST"
mkdir "$DEST"

# For any images you may have
cp -r "$CWD/images" "$DEST/"
# For the CSS to use with adoc
cp -r "$CWD/css" "$DEST/"
# In case you add any extra JS to run
cp -r "$CWD/js" "$DEST/"
# Any custom HTML you may insert into your adoc files
cp -r "$CWD/asciidoc/html" "$DEST/"
# I had a zipfile of favicons, this could just be changed do copying a favicon.ico
unzip "$CWD/favicon.zip" -d "$DEST/"

# Generate index file
asciidoctor -D "$DEST" "$CWD/asciidoc/index.adoc"

adocfiles=$(find asciidoc -type f -name "*.adoc" | xargs)

for f in $adocfiles; do
    relative_path=$(echo "${f%/*}/")
    mkdir -p "$CWD/$relative_path"
    asciidoctor -D "$DEST/$relative_path" $f
done

cd "$DEST"
mv asciidoc/* ./
rmdir asciidoc
cd -

So, I will be in my terminal at this point and be in the root of my working path, where I have my asciidoc folder and my compile.sh script. Then, when I want to, I run sh ./compile.sh to generate the site. At that point, I have a newly made example.com dir.

Uploading Site Content

With a created site, it is time to upload your work. I tend to use sftp or scp to do this. You could even do rsync if you wanted.

Swapping in Your New Site

Now with your content uploaded, presumably to your home directory, you can either run a few commands, or use a script like this to

Change ownership of the new website content
Change permissions
Move the old site
Move the new one

Something like this, assuming you have your web content at $HOME

#!/usr/bin/env sh

chown -R root:wheel $HOME/example.com
find $HOME/example.com -type f -exec chmod 644 {} \;
find $HOME/example.com -type d -exec chmod 755 {} \;
mv /var/www/htdocs/example.com /var/www/htdocs/example.com-$(date "+%Y-%m-%d-%H%M%S")
mv $HOME/example.com /var/www/htdocs

This will move the old site away as a backup, then put the new one in place.

Wrap Up

With all of this, if you can install OpenBSD, you can make a simple solution for basic static websites that are also easy to create since you don’t have to learn any (or very little) HTML, no CSS, no heavy site creator to get in the way, and little command line knowledge to configure and maintain.

Bonus

I had previously mentioned backups, so I won’t forget this one. You may find backups "unnecessary" since you should theoretically always have your site content on a local computer. However, if you want to backup your configurations and such, you can accomplish this with scp or openrsync.

Even though I don’t have any secret content, I do like to encrypt my backups. So this is something you can do when creating yours.

This is more or less what my backup script looks like:

#!/usr/bin/env sh

keyfile="/root/.enckey"
key="$(cat $keyfile)"

backupdir="/root/backups"
name="$backupdir/$(date +'%F_%T')_<optional backup name here>"
mkdir -p $name
mkdir -p $name/etc

# copy files to backup directory
etc_files="hostname.if,httpd.conf,pf.conf,rc.conf.local,sysctl.conf,acme-client.conf,doas.conf"

cp /etc/{$etc_files} $name/etc/

other_files="/etc/ssh/sshd_config"

for file in $other_files; do
	dir=$(echo "${file%/*}")
	bkp_dir="$name/${dir#/}"
	mkdir -p $bkp_dir
	cp $file $bkp_dir/
done

cp -r /var/www/htdocs $name/

# Compress and encrypt
tar -czf $name.tar.gz $name
openssl enc -aes-256-cbc -md sha512 -pbkdf2 -iter 100000 -salt -in $name.tar.gz -out $name.tar.gz.enc -pass file:$keyfile

# Clean up
rm $name.tar.gz
rm -rf $name

result=$(openrsync -a --delete $backupdir remoteuser@example.com:/path/to/backup/dest 2>&1 > /dev/null)

# Delete files older than 30 days
find $backupdir -mtime +30 -delete

Bonus Bonus

I like to know when my backups are working. So, I want to get notified when a backup is failing. In the previous backup script, you can insert a conditional to say "if the backup happened, submit a pass. If it didn’t for some reason, send a fail". I do this to NodePing with their PUSH checks. Which, if you can make a POST request and format some JSON, you can submit any data.

My method has been to just submit an empty JSON object {"data":{}} to accomplish my needs. One issue though, doing a POST request with OpenBSD from just base isn’t quite as cut and dry. I reached out, and the 2 most common answers were to write some Perl, which, that’s too much for me, or use nc(1). I opted to use nc.

I created a PUSH check on NodePing and got the checktoken and checkid for the POST request. The nc command looks like:

head='POST /v1?id=<checkid>&checktoken=<checktoken> HTTP/1.1\r\nHost: push.nodeping.com\r\nContent-Type: application/json\r\nContent-Length: 11\r\n\r\n{"data":{}}'
echo -ne $head | nc -c push.nodeping.com 443

Bonus X3

I appreciate knowing when a syspatch is available. OpenBSD provides a handy syspatch -c flag that will output any patches available. I submit this info to my Matrix server, but of course you could create another NodePing check or anywhere else you want to do a POST request with any sort of data to a service that will notify you.

My script basically looks like

#!/bin/sh

if [ ! -z "$(syspatch -c)" ]; then
	body="{\"body\":\"$(hostname) syspatch available\"}"
	contentlength=$(expr $(echo "$body" | wc -m) - 1)

	head="POST /web/path/post/somewhere HTTP/1.1\r\nHost: example.com\r\nContent-Type: application/json\r\nContent-Length: $contentlength\r\n\r\n$body"

	echo -ne "$head" | nc -c example.com 443
fi

Next create a daily cron job. We can call this /root/checksyspatch.sh

0 2 * * * /bin/sh /root/checksyspatch.sh

Now you should be notified if there are any updates available for your OpenBSD release.

Conclusion

Now, you have:

A running OS with high security
Easy OS management, really only needing to run syspatch, reboot, and sysupgrade
A nice, clean static website that really only requires simple syntax to know to write
Backups with openrsync
Backup notifications with NodePing and nc (definitely optional)
Update notices

Since it’s not often that a website update happens on some of my sites, I really only need to keep track of running syspatch and upgrading if a new release comes out. Then, any updates that DO need to happen are a basic edit, compile, upload, and put the new site content in place.

EDIT 2022-06-29

I didn’t state clearly enough that my 0-dependency claim is you don’t need to install anything on the server. The server does not need AsciiDoc. The client will, but my point is that the server doesn’t need any pkg_add and thus, only requires syspatch, reboot, and sysupgrade to maintain the host, and no needing to worry about maintaining other software on top of the OpenBSD base install. :)

Upgrade FreeBSD Base Jails to 13.1-RELEASE

Courtney — Sun, 29 May 2022 19:45:30 PDT

From 13.0-RELEASE to 13.1-RELEASE

$ zfs create -p zroot/jails/releases/13.1-RELEASE
$ fetch https://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/13.1-RELEASE/base.txz -o /tmp/base.txz
$ fetch https://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/13.1-RELEASE/lib32.txz -o /tmp/lib32.txz
$ tar -xf /tmp/base.txz -C /jails/releases/13.1-RELEASE
$ tar -xf /tmp/lib32.txz -C /jails/releases/13.1-RELEASE
$ env UNAME_r=13.1-RELEASE freebsd-update -b /jails/releases/13.1-RELEASE fetch install
$ env UNAME_r=13.1-RELEASE freebsd-update -b /jails/releases/13.1-RELEASE IDS
$ zfs snapshot zroot/jails/releases/13.1-RELEASE@p0
$ zfs clone zroot/jails/releases/13.1-RELEASE@p0 zroot/jails/templates/base-13.1-RELEASE
$ zfs clone zroot/jails/releases/13.1-RELEASE@p0 zroot/jails/templates/skeleton-13.1-RELEASE

rm -rf /jails/templates/skeleton-13.1-RELEASE/bin/*;
rm -rf /jails/templates/skeleton-13.1-RELEASE/boot/*;
chflags -R noschg /jails/templates/skeleton-13.1-RELEASE/lib/*;
rm -rf /jails/templates/skeleton-13.1-RELEASE/lib/*;
chflags -R noschg /jails/templates/skeleton-13.1-RELEASE/libexec/*;
rm -rf /jails/templates/skeleton-13.1-RELEASE/libexec/*;
rm -rf /jails/templates/skeleton-13.1-RELEASE/rescue/*;
chflags -R noschg /jails/templates/skeleton-13.1-RELEASE/sbin/*;
rm -rf /jails/templates/skeleton-13.1-RELEASE/sbin/*;
chflags -R noschg /jails/templates/skeleton-13.1-RELEASE/usr/bin/*;
rm -rf /jails/templates/skeleton-13.1-RELEASE/usr/bin/*;
rm -rf /jails/templates/skeleton-13.1-RELEASE/usr/include/*;
chflags -R noschg /jails/templates/skeleton-13.1-RELEASE/usr/lib/*;
rm -rf /jails/templates/skeleton-13.1-RELEASE/usr/lib/*;
chflags -R noschg /jails/templates/skeleton-13.1-RELEASE/usr/libexec/*;
rm -rf /jails/templates/skeleton-13.1-RELEASE/usr/libexec/*;
rm -rf /jails/templates/skeleton-13.1-RELEASE/usr/sbin/*;
rm -rf /jails/templates/skeleton-13.1-RELEASE/usr/share/*;
rm -rf /jails/templates/skeleton-13.1-RELEASE/usr/libdata/*;
chflags -R noschg /jails/templates/skeleton-13.1-RELEASE/usr/lib32/*;
rm -rf /jails/templates/skeleton-13.1-RELEASE/usr/lib32/*;

$ zfs snapshot zroot/jails/templates/skeleton-13.1-RELEASE@skeleton
$ zfs clone zroot/jails/templates/skeleton-13.1-RELEASE@skeleton zroot/jails/jails/testboxen_13.1

MY OpenBSD Desktop: Move From FreeBSD

Courtney — Fri, 06 May 2022 19:45:30 PDT

Just last week (as of the date on this post) I ordered an AMD RX 6700XT graphics card. I saw it at a great price, probably $20 above what it COULD have been if prices were normal for graphics cards. I couldn’t pass it up, I had been wanting a new card anyways, and I was wanting OpenCL on FreeBSD. Well bad news for me, FreeBSD only uses AMD drivers from Linux kernel 5.7, and it turns out the RX 6000 series landed in the kernel at kernel 5.10. I was very sad finding this out because I had just set up poudriere to do weekly builds, I had a custom kernel, and FreeBSD was ripping fast. I wasn’t ready to table my new shiny card. I have been wanting to try an OpenBSD desktop for years anyways but I always had roadblocks that would prevent me from using it (in addition to my last GPU being an Nvidia), and I rely on my desktop for work since I work from home.

Well, here I am today, a few days into using OpenBSD 7.1-current and I wanted to compare both my FreeBSD and OpenBSD desktop, what I love, what I miss, what I don’t know, and yeah.

The Hardware

Here are my hardware specs right now:

Intel i7 10700K
MSI Z490 chipset motherboard
32GB DDR4 ad 2666MHz…ECC but can’t use it cuz Intel
OS drive: 500GB WD Black SN750 m.2 drive
Extras: Different OS on a 480GB SATA SSD and 2TB WD Blue
MSI RX 6700XT
ZSA Ergodox EZ keyboard
Elecom HUGE trackball
FiiO E10K USB DAC/amp

Environment/Software

Currently I primarily use:

i3 (w/ i3bar and i3status)
Nextcloud client
keepassxc
Emacs editor
Firefox
Strawberry (Clementine on OpenBSD since Strawberry isn’t in OpenBSD ports)

First Impressions

In here, I’m going to compare/contrast setting up Free/OpenBSD on my desktop since the setup experience stood out to me.

FreeBSD

What didn’t work (or work right) after install:

Ethernet: I have a 2.5Gbps Realtek RTL8125 which I had to get the pkg on a different computer, copy it over, then pkg install to get ethernet
No xorg, but that is no issue
No graphics driver (had to install)
No U2F support

What worked after install:

keyboard/mouse worked as expected (though not all Elecom buttons work, I blame Elecom)

OpenBSD

So, with all the other times I’ve poked around with OpenBSD, everything Just Worked(tm). Same case here:

Ethernet worked
Got the AMD drivers and my screen defaulted to 1440p/144Hz right away
Xorg
U2F

Now, this is after install impressions. As I take you down my journey I’ll explain everything I’ve had to change or fix.

FreeBSD Experience

I’ve been a FreeBSD desktop user for 3 years. I love it. For the most part, there is little I would want changed. There’s a lot of good software, performance, good security controls, virtualization and jails are the absolute bomb, the FreeBSD Handbook is great. As far as general use goes, the only real papercuts I have are around enabling Nvidia drivers, but after that in these last 2 years Nvidia drivers have been totally painless for me. I loved how easy it was to compile the OS. Poudriere is mildly frustrating and could be documented a bit better I think, but eventually it was up and running. ZFS snapshots were pretty big for me, as I would snapshot my home dir and borg backup the snapshots.

What I Have Absolutely Hated

FreeBSD has had 2 total pain points for me.

WiFi
USB Audio

WiFi

Now, the FreeBSD Foundation has sponsored WiFi work for FreeBSD, and that is exciting. It will be awesome when FreeBSD sees something better than G wireless speeds. Really, what has required any medium amount of downloading has been painful.

USB Audio

OSS is amazing. I loved all the knobs I could touch with sysctl and how obvious different settings were. I didn’t have one single weird sound issue with FreeBSD….except on my Ryzen desktop when I had the Ryzen 3600. Not sure why, but sometimes audio would cut out. Not sure if that’s fixed today or not, I’m on Intel again.

But…the BIGGEST pain of them all: turn on some music and play a song, then unplug your DAC,I dare ya. Oh oops, the programs didn’t properly release the audio device, go and kill every process that used your DAC, otherwise you aren’t getting your DAC back. This wasn’t a HUGE problem for me with i3, I usually had just Firefox and Strawberry to kill then life was back to normal. But with a desktop like KDE, you’re better off restarting your computer than killing the processes and finding out how to get KDE Plasma to a clean state again.

All in all, FreeBSD was a dream on my desktop, just don’t unplug the DAC. It was also very smooth on my Thinkpad T480 just get over the wireless speeds.

OpenBSD Experience

This is my first time ever getting OpenBSD on my desktop. Aside from the Nvidia card, there was always a couple things that didn’t quite sit right with me to trust it for my day do day work OS. I think now a lot of that has changed.

Earlier I said everything Just Worked(tm), and that is mostly true. However, here’s some of the things I have had to change:

modify /etc/login.conf` and add myself to staff group to get enough max files/mem for stuff like Firefox
Audio stuttered like a dying car
Some sysctl flags

Really, this wasn’t much. I regularly pull some bits and pieces from here to set up OpenBSD. The audio though…yikes. Sounds great, but that stutter was (is) awful! I think it’s 95% fixed now. I eventually discovered that the way OpenBSD handles CPU scheduling (I guess) would cause interrupts if anything is too taxing at any give time. Now, most software I used hasn’t been taxing to the point where it cuts out my audio. But Firefox is the one.

Fun fact: out of the box, Firefox wants 8 processes. That’s all 8 of my CPU cores, and probably all 8 of your 4 CPU cores. My solution:

Go to about:config → change dom.ipc.processCount to something lower than your core count. I set mine to 5. Audio problems gone.

This still isn’t fully resolved for me either. I’ve been trying to tweak everything I can think of. I’m about to the point where I’m near going against my conscience and using chromium because chromium does not induce the stuttering audio issues the way running Firefox does. There’s one issue with chromium though, which I’ll get to.

Firefox

Couple notes I have here on Firefox are namely that it can be finicky getting it to point to your Downloads directory. At least make sure it is present so it can download since unveil actually does what it says it does. This goes for using the keepassxc-proxy too. For a while, unveiling it didn’t work for me. Now it appears to work. Yay!

Now for the worst of it. I still cannot figure this one out. Seems no one on Reddit, Matrix, or the mailing list has an answer to this. But for whatever reason, Firefox constantly interrupts audio on OpenBSD. Audio, and even other USB input gets totally murdered by Firefox. Tried a TON of stuff. Nothing fixes the issue. At this point, I have to abandon Firefox until this is resolved, which sucks a lot because, I think Firefox is technically and feature-wise a better browser than chromium.

Chromium

Chromium is chromium. Works just fine out of the box, no weird issues in terms of regular use. The one thing that does suck is I can’t get keepassxc-proxy working with it. It seems even with some unveil tweaking I can’t get it to properly communicate with NativeMessagingHost, and, I think there’s some other unveil issues that are keeping it from working right. Don’t know what it is yet.

Where’s Muh Erlang?

What has been AWESOME about FreeBSD is how it has Erlang 21-24. OpenBSD is on Erlang 21, and I think the last time it got any love was in 2020. Well, I am grateful to the Erlang people, it seems they have some love for OpenBSD, despite it not being up to date in the ports tree. But downloading the source code, installing gmake, then ./configure, gmake, gmake install and there. You have current Erlang. Similar steps for rebar3. I’ve never maintained a port before, but I use Erlang regularly, and Elixir. So if I can, I may see about maintaining the Erlang/Elixir port in OpenBSD some day.

Not a Fan of vmm

Between bhyve and vmm, bhyve wins every time in my opinion. To be fair, I haven’t tried vmm but when I look at what it supports, it always looks to be very lacking. That’s okay though, my server will be running FreeBSD indefinitely so I can utilize ZFS, jails, and bhyve.

Me No Like OpenBSD’s Wireguard Client

A few months ago, I tried to get OpenBSD to be a Wireguard client…didn’t go well. I have 2 Wireguard OpenBSD SERVERS and they have been champs. For some reason though, can’t get the client to pass through traffic. Tried a lot of stuff and for some reason the answer I always get is "use OpenIKED".

On the bright side: I did get everything working great now with wg-quick. I could probably shim that into using hostname.if, but the problem with hostname.if is I don’t think I can selectively enable interfaces. So, the wg-quick package does what I need best.

sndiod is great…and not

OpenBSD’s sndiod has solved my USB audio issue I have complained about for years. In FreeBSD, even inadvertently unplugging the USB DAC will result in a blow up. It used to completely hang every single USB device which required a force restart. Now as I said, you have to restart or kill every single process that held onto that device. In OpenBSD I can plug/unplug my DAC as many times as it wants. No explosions, and on plug in my audio works again no problem.

The not-so-great is it takes more to understand sndio than OSS. OSS was very straightforward to me. But sndio…it’s a must that you read the mixerctl(8), sndioctl(1), sndiod(8), and multimedia FAQ to get anything working or understand what a lot of the settings are in mixerctl and sndioctl output, configuring sndiod, understanding what rsnd/X is vs snd/X and so on.

Also, maybe my USB microphone is unsupported, but it SHOULD be rsnd/2 (or audio2, or uaudio1 because I have 2 USB audio devices but 3 audio devices), but whatever I try doesn’t seem to produce any working input and I just get errors even though the device shows up in dmesg.

Mouse is Missing Button Functions

I have an Elecom HUGE. I haven’t had all the buttons work right on Linux or FreeBSD either because I guess some of the buttons only work Right with their desktop Windows software. However, the forward/back buttons worked right on every OS except OpenBSD. They’re scroll up/down. And I can’t go in and make X11 edits because then my scroll wheel will misbehave, since according to X11 (I’m no display server expert), they’re the same button. So that sucks, I have to actually click the back/forward button in the GUI now.

Overall Thoughts

I have to say, I love both operating systems for both their places. On one hand, I want to shake a stick at FreeBSD for being behind on graphics and wireless, but they’ve done such a great job at shipping a performant operating system with some awesome features. Likewise too, OpenBSD is SLOW often times and has some IO quirks sometimes it seems, it performs worse on weaker hardware by a lot (at least with browsers, an old dual core refused to run OpenBSD well), but without browsers it runs fine.

Currently, I am super happy with OpenBSD on my laptop. In fact, I think it will stay like that for the foreseeable future. I am not happy with it on my desktop though. Really, it all circles around to Firefox throat punching OpenBSD it seems when I do something as simple as open a new tab.

My conclusions for now:

OpenBSD, great on laptop. Better than FreeBSD for me. Maybe it also has similar audio issues to my desktop, but I don’t use my laptop for audio of any sorts really. Maybe I will play something from the speakers on occasion.

I’ll probably switch back to FreeBSD on my desktop though once the RX 6700XT is properly supported. Really, I see no future for my audio problem being solved. Maybe some day, but I have my doubts it will happen in 2022. I work from home, so I listen to music for a majority of the day. If I’m constantly being battered by poor audio, I can’t work. I also far prefer Firefox…one big reason is because Firefox has better features and screw Google.

I think though, if this issue were solved today I would stay here on OpenBSD. Not out of any disdain or dislike for FreeBSD…I do think it has a leg up right now. But I really appreciate OpenBSD’s simplicity and security controls.

Addition

2022-05-07

I forgot to add what WiFi has been like on OpenBSD. Wireless worked out of the box, got 180Mbps down, which is excellent compared to the 25Mbps I get on FreeBSD.

FreeBSD 13.1 Release

Courtney — Wed, 13 Apr 2022 19:45:30 PDT

Table of Contents

According to the FreeBSD 13.1 release schedule, the team is close to putting out the 13.1-RELEASE. I found the page and have been looking through the notes that are there so far to see what is of interest to me.

On my desktop and laptop I have been running 13 STABLE, albeit blindly to what has changed, so it is nice to see the team put together a document with what is upcoming for the release itself.

I already have plans to move my personal server over within the first week or two of release, in which I hope to create another post documenting my upgrade process from 13.0 → 13.1 for my jails which I wrote about before. Additionally, I’ll be doing some VPS upgrades and retiring some hardware, so migrating to 13.1 will be a perfect opportunity to move away and also fix my old design mistakes.

So now, with my rambling done, let’s move on to what interests me.

Mmmmm, PIE

The page states that they are enabling PIE, or Position Independent Executable.

For 64-bit architectures, the base system is now built with Position Independent Executable (PIE) support enabled by default. It may be disabled using the WITHOUT_PIE knob. A clean build is required. 396e9f259d96

What is this? This basically means that instructions can be positioned anywhere in virtual memory and be executed properly. And, software in memory will not be in the same memory address each time it is loaded. This means it is harder to execute certain exploits, like Return Orieinted Programming attacks since the attacker no longer knows the memory addresses of functions, and those locations will be different each time the process is restarted, so if they find what they want once, they’ll have to find it again if the process is restarted.

zfskeys

here is a new zfskeys rc(8) service script, which allows for automatic decryption of ZFS datasets encrypted with ZFS native encryption during boot

I haven’t used ZFS’s encryption yet, but I think it’s neat that it’s possible now to automatically decrypt ZFS datasets. This was actually one of the reasons why I hadn’t yet used this feature of ZFS.

SSH

A few things that interest me with the SSH update is the bump to v8.8p1. 2 Features I’m interested in:

scp gaining support for replacing rcp protocol with sftp. It says this is experimental, which is understandable since OpenSSH 9.0 is said to be the release that fully transitions to sftp by default. In the meantime with 8.8, if you use scp -s it will enable sftp mode

Note
Seems you can’t do any ~ for path expansion
u2f keys. Don’t use one for my SSH stuff, but it’s cool FreeBSD FINALLY has a release that supports this in base.

iwm(4) updates

I’m happy to see more work is being put into better WiFI support in FreeBSD. This has probably been the worst part about FreeBSD for me. I haven’t done a speed test since running STABLE, but soon I hope to, and see if maybe they’ll be backporting more of what’s being dumped into 14-CURRENT into the 13 branch…if not, 14 is I’m guessing about a year off. Really, I just want something better than the 25Mbps I cap at.

Boot Time

In the notes, they write:

Performance improvements have been made to loader(8), nvme(4), random(4), rtsold(8), and x86 clock calibration, which collectively yield a significant speedup in system boot time.

I have to say, this is one thing I have noticed since running the stable branch. I run rtsold and boot off NVMe and it certainly has been faster than when I booted FreeBSD in the past.

Wrapping Up

This seems like it’s going to be another cool release. I like that FreeBSD steadily and reliably improves the operating system and I am excited to see what more they do in the future too!

FreeBSD 13.0 Jails with ZFS and VNET - Part 2: Updating

Courtney — Fri, 25 Feb 2022 18:45:30 PST

Table of Contents

In part 1 of my FreeBSD jail adventure, I switched from iocage to my own homespun jails. This will be a short piece on updating those base jails.

Since they are all nullfs mounting the same directories, you only have to update things once.

Note	Allan Jude mentioned doing symlinks instead of nullfs, but I haven’t tried that. You can hear his comments on BSD Now.

Update the Host

First, you will want to update the host. However you want to go about this, go for it. For me, I use freebsd-update. I do a trick I learned from vermaden, though, it was something I found in a tweet that was posted, maybe there’s an article…

bectl create 2022-02-25_13.0

bectl mount 2022-02-25_13.0

chroot /tmp/be_bootenv_id

mount -t devfs devfs /dev

freebsd-update fetch install

pkg update

pkg upgrade

exit

bectl activate 2022-02-25_13.0

shutdown -r now

The system will be rebooted and then you will boot into the new boot environment with the updates applied.

Jail Updates

Now let us update the jails. From the previous article, note I made the zfs dataset that is nullfs mounted from at zroot/jails/templates/base-13.0-RELEASE. This is what we will be updating. First I take a snapshot of it (in case things go sideways)

zfs snapshot zroot/jails/templates/base-13.0-RELEASE@2022-02-25_update

Then I update it with the freebsd-update utility:

env UNAME_r=13.0-RELEASE freebsd-update -b /jails/templates/base-13.0-RELEASE fetch install

Now with the base image updated, I don’t know if it’s necessary (didn’t check, will update this in the future to add clarity), but I restarted all my jails too while I was at it with service jail restart. Of course, if not all your jails are using the same base, you can restart them individually. Or, you can verify before/after to see if the changes were applied without restarting the jails by taking a uname -a before and after and confirming that the change did happen. I inadvertantly updated the jails before the host because I rebooted without applying the boot environment from before :)

And that’s it! All the base jails you have should be updated now, then you can do your pkg update/upgrades in your jails as necessary.

Once I’m done I also like to remove the old snapshot. I know the system is working right so I don’t need it anymore.

Why I Stopped Going Out For Coffee

Courtney — Sat, 19 Feb 2022 18:45:30 PST

Table of Contents

I am a big lover of coffee. If you know me, you are probably aware that I am rather knowledgeable and enthusiastic about my favorite drink. For many years (probably 10 now) I would go out for an espresso 1-2x each day. Now, I may go out once a week for a social occasion and that’s about it. But why?

Coffee is Expensive

Back when I started drinking espressos, they started around $1.40 each espresso. Not bad! It was rare that I had a bad one, and I figured for $2.80 each day, it was worthwhile since it gave me a break in the day, a moment to chat with my barista friends, and have a place to chill away from home with a good (albeit fast) drink. Over time it has crept up in cost. When I graduated from the university in 2018 it was just over $2/shot, while I was going to the university it was $1.81. Still not too bad when you consider the return on investment that has to be made on espresso machines. But now, those espressos are $2.80, or over $5 each day for coffee.

Fun fact: you can get 1lb of unroasted coffee for about $6-$9, which will net you about 12oz of roasted coffee in the end which could get you 17-22 espressos, depending on how much coffee you use per espresso.

Now, I took a break from roasting, which, I should pick back up, but you can see that with that, or even buying a 12oz bag for $15 will still net you an espresso for $0.68 to $0.77.

I Make Better Coffee at Home

This isn’t an insult to my baristas. You all do an AWESOME job. And if you’re reading this knowing the coffee shop you work(ed) at and who I am, know I am beyond grateful for the kindness and attention you put into making my espresso. However, most coffee shops will have a blend that is good, it isn’t always what I’m looking for. And, since there’s only so much you can do to make a "perfect" espresso when there are other customers, it becomes clear that the quality will go down.

At home, I am in charge of how often I clean my machine, the exact amount of beans that go in, how it’s prepped, and how it is extracted. With what I know of how to make coffee, and what I like, I know how to tune my machine to make what I want consistently.

What Do I Use?

I started making espresso with the Flair Classic which is an awesome espresso maker. However, in 2021 they released the Flair 58, which is a bigger, and electric version of the Classic. Coming in at $530 when I got it, to get features akin to this machine, I would be paying probably $3k minimum for a standard machine you’re more familiar with. Which features? I guess just one…which is the pressure profiling that you can do with the lever. Most machines will usually try to pull shots at a straight 9 bars. However, I get control of the pressure the entire extraction. So, full coffee cart lineup:

Espresso machine: Flair 58
Grinder: Eureka Mignon Notte (with single dose hopper & silicon bellow)
10/15/20/22g Decent Espresso baskets (and the 18g high flow that came withe the Flair)
Decent Tamper v3

This is my favorite setup, costing just over $1k, which sounds like a lot (maybe it is because most people will go out and get a $30 drip maker) but Real espresso isn’t a cheap ordeal.

Can I Make Espresso at Home?

YES! Anyone can make espresso at home. But please don’t cheapen your experience with cheap espresso machines. James Hoffman made a great video on espresso machines under £100, and you can see for yourself what the outcome is. If you want a coffee machine the cheapest I can recommend is the Gaggia Classic Pro. It’s an actual machine, and not an appliance like the cheaper Breville espresso machines. The difference being cup quality and appliances being harder or impossible to service when parts fail. From there my next recommendation is the Rancilio Silva. There’s other good ones too, but most good machines will be $800 and up.

Next, you need a good grinder. If your grinder sucks, your coffee sucks. Doesn’t matter how good your machine is. I like my Eureka, which cost $300. Anything in their Mignon lineup will be good. There’s many others too, but usually good grinders will be around $400+

But That’s Way Too Much Money!

Yes, yes I agree. That’s actually what drove me to start with the Flair Classic, which I got my hands on for about $150. You still need a good grinder, but you can get the Flair Neo for $125 and any cheap (please get a burr grinder!) grinder will do. You could probably be making espresso at around $180.

But, if you could, I would HIGHLY suggest a whole kit under $500, which will make you GOOD espresso once you learn. What do you need?

Flair Pro 2
1Zpresso JX Pro hand grinder

These 2 tools will make you some of the best espresso ever, and for under $500. The best part? There’s no maintenance! Just wash it when you are done, takes maybe 2 minutes.

Wrapping Up

Thanks for reading my random opinion piece :) this was just a bit of filler writing for the sake of having something on my blog that isn’t computer related, and to kick the tires on my writing process. Hope this at least inspired someone to consider brewing at home. With my Flair machines, I have a pretty lengthy process. When I used the Pro model I would spend maybe 15 minutes from taking everything out to drinking my espresso and cleaning everything up. With my 58, I can do all that in maybe 5 or so minutes. It’s really a great way to take a pause in your day and make a good coffee.

Oh, and didn’t mention milk drinks in here, but I make my milk drinks with the Nanofoamer.

Also, I have no affiliation with Flair, I’m just a fanboy who has really loves their product and have been so happy to have a low cost machine that can compete with the "big boys".

How I Made My Old Blog

Courtney — Sat, 19 Feb 2022 18:45:30 PST

Table of Contents

For quite some time now, I’ve always wanted a basic static blog, but never really knew what I wanted. I settled on AsciiDoc because it has a very nice template for basic websites, and is pretty versatile. When trying to build a site in the past, I thought I had to tie together some language and build routes and bla bla bla. Turns out I way overcomplicated things.

Templating

I templated my pages to meet a scheme. For all my pages, I start with this header:

= Site Title
:icons:
:toc: left
:encoding: utf-8
:stylesheet!:

++++
<button id="theme-switcher">Switch Themes</button>
++++

https://blog.passwordclass.xyz[HOME]

This lets me get the theming I want and get a home button. My index page does not have the home button for obvious reasons. From there on, I write what I want.

Site Layout

Next I have an expected site layout. At the webroot, I have a css dir with "light.css" and "dark.css" for my light and dark themes. Next, I have a js directory with my "switcher.js" which lets me handle light and dark themes. For any pictures I’ll probably have a "media/images" directory to pull from, but currently none of my pages have images. Unfortunately, I don’t have any relative path work I can do in my blog pages, so I have to point to absolute paths on my website for anything. No problem for me though.

Building

Next is the building of AsciiDoc to HTML files. I have a working directory, and from there I have a directory called "asciidoc" which I have layed out exactly the way I want it to be on my webroot. I run this script here:

#!/usr/bin/env sh

CWD=$(pwd)
DEST="$CWD/blog.passwordclass.xyz"
NEWLINE='<script src="https://blog.passwordclass.xyz/js/switcher.js"></script>'

rm -rf "$DEST"
mkdir "$DEST"

# Copy JS files
cp -r "$CWD/js" "$DEST/"
# Copy CSS files
cp -r "$CWD/css" "$DEST/"

# Generate index file
asciidoctor -D "$DEST" "$CWD/asciidoc/index.adoc"

# Find adoc files
adocfiles=$(find asciidoc -type f -name "*.adoc" | xargs)

# Generate all the adoc files and dump them in the webroot dest
for f in $adocfiles; do
    relative_path=$(echo "${f%/*}/")
    mkdir -p "$CWD/$relative_path"
    asciidoctor -D "$DEST/$relative_path" $f
done

# Some cleanup
cd "$DEST"
mv asciidoc/* ./
rmdir asciidoc
cd -

# Find the HTML files
htmlfiles=$(find "$DEST" -type f -name "*.html" | xargs)

# Find a line in the HTML header and insert my switcher.js
for f in $htmlfiles; do
    sed -E -i "" "/\<meta charset\=/a\\
$NEWLINE
    " "$f"

done

Finishing Touches

With a new article done, I update my index.adoc to contain my new article listing, then I run the script above to generate the updates. Lastly, all I have to do is copy my newly generated local webroot to the new one.

Then it’s done!

Freebsd Connect To Public WiFi

Courtney — Fri, 04 Feb 2022 18:45:30 PST

Table of Contents

I’ve been running FreeBSD on my laptop for under a year now. One of my biggest pain points so far has been completely around WiFi. It’s slow, janky to connect sometimes, and very manual (sometimes not a big deal on that last part). However, I still prefer to use it. Lately, I’ve left my house more to do some stuff on coffee shop WiFi. I used to come out more before I got married and got my Flair 58 espresso maker, however, sometimes it’s nice to get out or, in my more recent cases, my wife has been needing rides to meet people so I’m "stuck in town".

This is a bit of a reminder to self, and maybe this will help you out when you’re banging your head against the wall trying to figure out why you can’t connect to a captive portal.

If you have a better way to do what I’m doing here, hit me up on Twitter!

TL;DR

Add the wifi network to wpa_supplicant
Find the resolver IP in /var/unbound/forward.conf and set it on top in /etc/resolv.conf
Remove "WPA" from your wlan interface in /etc/rc.conf
service netif restart
Connect to captive portal
Test connection
Connect to VPN
Re-enable your own resolver in /etc/resolv.conf

Things to Note

I’ve noticed a few things when it comes to public WiFi:

A lot of the time, it isn’t encrypted
Captive portal won’t let you use a local resolver like unbound
When switching your VPN on, you have to make even More changes

Now, I believe the last 2 parts can be completely left in the back of your mind if you are doing all your DNS with resolvconf editing your resolv.conf file. I however, run local unbound (for better or worse).

wpa_supplicant

You will need to add an entry to your /etc/wpa_supplicant.conf file. Should just be like this:

network={
	ssid="public wifi"
}

rc.conf

In the rc.conf file, you will have to make one edit. You’ll have to knock out the "WPA" part for your wireless interface (assuming the network you are connecting to does not do any sort of encryption). I like to have 2 lines in my rc.conf file: one with WPA and one without, otherwise identical, and I’ll comment out each line according to where I’m at.

resolv.conf

If you have local_unbound running, and you connect to the wireless network, you can find the network’s DNS resolver in the /var/unbound/forward.conf file. I put that at the top of my resolv.conf file.

Connecting

With these edits in place, you can try a service netif restart and check with ifconfig to see if you are connected to the right SSID. If so, open a browser and a search should bring up a captive portal.

From here, you may be able to uncomment their resolver from the resolv.conf file and use your own, but in my experience, you might be stuck using their resolver. This is where a VPN becomes handy.

VPN

On public WiFi, you should ALWAYS use a VPN. As I pointed out, all the traffic is unencrypted. Anyone could sit and sniff your internet traffic with ther right hardware. Another handy thing to a VPN is you can use any resolver you please.

Enabling the VPN

I found you have to use the local network’s resolver to connect to your VPN first, then you can switch to your own. So ensure you are still using their VPN first, then connect. I use my own WireGuard server so I just run service wireguard onestart to temporarily enable my VPN. From there, you can uncomment out the network resolver from your resolv.conf file. At that point for me, I use 127.0.0.1 to use unbound.

Conclusion

Connecting to public wifi on FreeBSD is still pretty janky. You definitely get a smoother experience with Windows/Mac/Linux. Maybe it’s better on OpenBSD even? Who knows. But this is my process, and I hope this helps somebody out. If not, I at least know where to find my own instructions :)

FreeBSD 13.0 Jails with ZFS and VNET

Courtney — Fri, 14 Jan 2022 18:45:30 PST

Table of Contents

As of late I have had some pain points with iocage, which I used since I started using FreeBSD in 2017. I came from an Ubuntu with LXD + ZFS background and iocage had the command line interface I wanted that felt familiar with LXD at the time.

Well, iocage seems dead now. Its last release was in 2019 and its last commit (at the time of writing this) was September 30, 2021. Of course, that commit isn’t in what’s in FreeBSD ports, unless you use the devel package..and, that package has some issues (for me, iocage list doesn’t work right).

Because of this, I decided to take up the challenge of making my own base jails. To start, I will give credit where credit is due and say I followed these resources to get me to where I am:

Michael W Lucas' FreeBSD Jail Mastery book
https://clinta.github.io/freebsd-jails-the-hard-way/
https://www.skyforge.at/posts/a-note-in-sysvipc-and-jails-on-freebsd/
iocage and their fstab files

Creating the Release

First off, we need to create a release jail. This is a base image that we can use to make cloned jails, thick jails, or our base jails from. I’m going to start by making a new jails dataset and mounting it at /jails

$ zfs create -o mountpoint=/jails zroot/jails

Here is the foundation for everything. Now I’ll create a few other datasets for our releases and templates and running jails, as well as our first release dataset (13.0-RELEASE)

$ zfs create -p zroot/jails/releases/13.0-RELEASE
$ zfs create zroot/jails/templates
$ zfs create zroot/jails/jails

Next, we need to download the base OS as well as lib32 for our jail. The contents should be extracted into /jails/releases/13.0-RELEASE in the end.

$ fetch https://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/13.0-RELEASE/base.txz -o /tmp/base.txz
$ fetch https://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/13.0-RELEASE/lib32.txz -o /tmp/lib32.txz
$ tar -xf /tmp/base.txz -C /jails/releases/13.0-RELEASE
$ tar -xf /tmp/lib32.txz -C /jails/releases/13.0-RELEASE

Now let us update the jail contents

$ env UNAME_r=13.0-RELEASE freebsd-update -b /jails/releases/13.0-RELEASE fetch install
$ env UNAME_r=13.0-RELEASE freebsd-update -b /jails/releases/13.0-RELEASE IDS

Then, we can copy our /etc/localtime and our /etc/resolv.conf files into the jail $ cp /etc/localtime /jails/releases/13.0-RELEASE/etc/localtime $ cp /etc/resolv.conf /jails/releases/13.0-RELEASE/etc/resolv.conf

Nice. Now we have our base. Lets snapshot it so we can clone it. We will clone this to our templates folder after we take the snapshot:

$ zfs snapshot zroot/jails/releases/13.0-RELEASE@p4
$ zfs clone zroot/jails/releases/13.0-RELEASE@p4 zroot/jails/templates/base-13.0-RELEASE

This part is done. The release is made, now we have a base created for our base jails.

Creating Our Skeleton

Since we want to be using nullfs mounts for our base jail, we are going to want to make another clone and wipe out the contents of that new clone. Here I think you can debate whether or not you want to take a clone of the base-13.0-RELEASE clone from earlier, or if you want to clone from the release. I opted to clone from the release. Maybe one is a proper way, but at this time I can’t see a drawback to either way myself.

$ zfs clone zroot/jails/releases/13.0-RELEASE@p4 zroot/jails/templates/skeleton-13.0-RELEASE

Now we want to hollow out the skeleton we just made. In Clinta’s example, this is where I started to encounter issues, namely with /usr/local since it doesn’t seem you can mount /usr/local if /usr is nullfs’d…this could create a bunch of other problems too, namely, too much stuff is set up with nullfs mounts. The way that is shown by Michael W Lucas and the iocage people is more sensible in this case. We need to EMPTY the contents of some directories without deleting the directories. Note, some directories (such as the lib directories) have immutable files, so you’ll have to make the files mutable if you encounter the error:

$ rm -rf /jails/templates/skeleton-13.0-RELEASE/bin/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/boot/*
$ chflags -R noschg /jails/templates/skeleton-13.0-RELEASE/lib/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/lib/*
$ chflags -R noschg /jails/templates/skeleton-13.0-RELEASE/libexec/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/libexec/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/rescue/*
$ chflags -R noschg /jails/templates/skeleton-13.0-RELEASE/sbin/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/sbin/*
$ chflags -R noschg /jails/templates/skeleton-13.0-RELEASE/usr/bin/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/usr/bin/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/usr/include/*
$ chflags -R noschg /jails/templates/skeleton-13.0-RELEASE/usr/lib/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/usr/lib/*
$ chflags -R noschg /jails/templates/skeleton-13.0-RELEASE/usr/libexec/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/usr/libexec/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/usr/sbin/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/usr/share/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/usr/libdata/*
$ chflags -R noschg /jails/templates/skeleton-13.0-RELEASE/usr/lib32/*
$ rm -rf /jails/templates/skeleton-13.0-RELEASE/usr/lib32/*

And if you run into the unable to delete issue, run: chflags -R noschg on that directory.

Now we are at the stage where we can create our jails. We will be snapshotting the skeleton from here on out when we want to have a new jail. So, lets snapshot the skeleton and call this new jail "testboxen_13-0".

$ zfs snapshot zroot/jails/templates/skeleton-13.0-RELEASE@skeleton
$ zfs clone zroot/jails/templates/skeleton-13.0-RELEASE@skeleton zroot/jails/jails/testboxen_13-0

Neat! It’s created. We can’t use it yet, but it’s there.

Templates

Here is a quick tidbit on templates…kind of a sidebar. Earlier we made that base-13.0-RELEASE clone and it’s not doing much. It’s the same as what’s in the release, as a matter of fact. I never utilized the templates feature in iocage, but reading Michael’s book tipped me off to how to do this for my own self. For my job I have roughly 10 jails doing the same thing. Do I want to go and update the same packages each time for each jail? They’re all running the same software, so why make 10 identical downloads? You can clone the release jail and make a base jail named, say, "nginx-13.0-RELEASE". You could go in that jail and install nginx and anything else each of that jail might need and use that nginx-filled template along with the aforementioned skeleton. If you do this however, note you will additionally have to nuke usr/local and var/db/pkg in the skeleton.

I would like to note it wasn’t until AFTER I set everything up for work that I messed up my implementation. So, for this section I’ll just make this mention here and suggest you check out Michael’s book, ch6, page 123.

FSTABs

Next, we must create our fstab files. This will tell the jail creation tools to use the nullfs mounts inside our skeletoned jail we just created. Here’s what it should look like for our "testboxen_13-0"

/jails/templates/base-13.0-RELEASE/bin                /jails/jails/testboxen_13-0/bin          nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/boot               /jails/jails/testboxen_13-0/boot         nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/lib                /jails/jails/testboxen_13-0/lib          nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/libexec            /jails/jails/testboxen_13-0/libexec      nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/rescue             /jails/jails/testboxen_13-0/rescue       nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/sbin               /jails/jails/testboxen_13-0/sbin         nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/usr/bin            /jails/jails/testboxen_13-0/usr/bin      nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/usr/include        /jails/jails/testboxen_13-0/usr/include  nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/usr/lib            /jails/jails/testboxen_13-0/usr/lib      nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/usr/libexec        /jails/jails/testboxen_13-0/usr/libexec  nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/usr/sbin           /jails/jails/testboxen_13-0/usr/sbin     nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/usr/share          /jails/jails/testboxen_13-0/usr/share    nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/usr/libdata        /jails/jails/testboxen_13-0/usr/libdata  nullfs  ro      0       0
/jails/templates/base-13.0-RELEASE/usr/lib32          /jails/jails/testboxen_13-0/usr/lib32    nullfs  ro      0       0

I like to keep these files in /jails/fstabs, so this file would be /jails/fstabs/testboxen_13-0.fstab.

The jail.conf File

Now lets make our jail.conf file. This will tell our system how to put together our jail. Here’s the contents of mine for this case. Note, you can copy/paste the testboxen_13-0 block over and over for each jail you make with its new name.

$jails="/jails/jails";

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

testboxen_13-0 {
        host.hostname = "testboxen_13-0";
        path = "$jails/testboxen_13-0";
        vnet;
        vnet.interface = "e0b_test1";
        exec.prestart += "sh -x /usr/local/bin/jib addm test1 jeth";
        exec.poststop += "/usr/local/bin/jib destroy test1";
        mount.fstab = "/jails/fstabs/testboxen_13-0.fstab";
}

Let’s break this down a little:

path - where the jail is located
vnet - specifying we are running a vnet jail
vnet.interface - the name of the interface inside the jail
exec.prestart/poststop - how we are going to create/destroy our interfaces
mount.fstab - the fstab file for our base jails

Neat. This is all done…but, this Still won’t work. Our host does not yet know how to use this stuff.

Host Setup

For the host, we need to setup a few things:

Enable the service
Get jib in the right place
Configure our interface (optional)

Enable the Service

sysrc jail_enable="YES"

Get jib in the Right Place

Either copy or symlink jib to /usr/local/bin

$ cp /usr/share/examples/jails/jib /usr/local/bin/jib

Configure Our Interface

This part is optional. I have an extra interface and I decided to give it a name. That is the "jeth" in the config above. It could be em0, igb0, re0, em1, whatever interface you want to use, don’t even need to rename it.

$ sysrc ifconfig_igb1_name="jeth"
$ sysrc ifconfig_jeth="up"

Since that will only work on the next boot, you can run ifconfig igb1 name jeth and ifconfig jeth up to get the interface up.

Okay, NOW our jail should start

Jails Needing Shared Memory

I ran into one instance where software needed shared memory (PostgreSQL). Because of that, I initially thought I needed to enable an insecure mode of sharing memory, but it turns out since, I think, FreeBSD 11.1 they added new security controls to create new shared memory namespaces to keep separate shared memory…pools? for each jail you set it up for. iocage did this out of the box for everything, but I find it’s probably only necessary on an as-needed basis. Basically, for each jail (or just your entire jail.conf file) add this line:

sysvshm = "new";

This is pretty well explained in the skyforge URL above. == Wrapping it Up

With everything in place, let’s start this jail up:

$ service jail start testboxen_13-0
$ jls
   JID  IP Address      Hostname                      Path
    1                  testboxen_13-0               /jails/jails/testboxen_13-0

HOORAY! But wait, there’s no networking. That’s because we need to have the jail do it for us. We could either put some of the exec stuff in the jail.conf, or pass the buck to the jail and put it in its rc.conf, which is what I will do. Note, with vnet jails you won’t see the ip address in the jls output. So, console into that jail:

jexec testboxen_13-0

Now lets get some stuff in our jail’s rc.conf. I’ll give it an address of 10.0.0.10/24, and a default router of 10.0.0.1

hostname="testboxen_13-0"
# note the name here matches what we put for vnet.interface
ifconfig_e0b_test1="10.0.0.10/24"
defaultrouter="10.0.0.1"

# Enable IPv6 if you want
#rtsold_enable="YES"
#ifconfig_e0b_test1_ipv6="inet6 accept_rtadv"

Cool. Now get out of that jail with an exit and restart it to ensure networking is good: service jail restart testboxen_13-0. You should be able to console back in and ping. It should be good! Now you have a fully functional jail. Lastly, if you want these to boot up, you can add it to the jail list:

sysrc jail_list+="testboxen_13-0"

Boom. Done. With all this out of the way, you can take all these same principles with the cloning of the skeleton, fstab, and jail.conf entry to create all the base jails you want!

Update 2022-01-20

I have seen others in places suggest the use of BastilleBSD for jail management too. I also think BastilleBSD is a good tool, so here’s a shoutout BastilleBSD and the people behind the project 🙂

https://bastillebsd.org/

Courtnix Blog

Migrating My Email to Stalwart

Why The Switch?

The Switch

The Rush

Migration Steps

Looking Back

Final Thoughts (Sidebar)

One Problem

Disable Fish Shell Greeting

1. Create an Empty Function File

2. Update config.fish

My SmartOS Live Demo Failure, and Success

The Troubles Begin

Showing My Ignorance

The Resolution

The Failure and Success

Core Ultra Framework Laptop and OpenBSD

Software

OpenBSD installer

Post-Install

Overall Experience

Cool Battery Feature

Conclusion

How is SmartOS So Far?

Fault Management Architecture (FMA)

Ansible Usage

Owner UUID Fun

How Are Native Zones?

fwadm

etherstub

Deploying zones/VMs is Easy

Host Management

Shots Fired

Conclusion

I Fixed My Blog...Again

Hugo Problems

What I like About 11ty

What’s Next

Fix MTR Report in Ubuntu 24.04

TL;DR

The Fix

Process

My OpenBSD Desktop Revisited

The Hardware

Environment/Software

First Impressions Revisited

My Elecom HUGE Problem

FreeBSD Revisited

OpenBSD Experience Revisited

USB Audio

Firefox

keepassxc-proxy

Erlang

hostname.if for Wireguard Clients

sndiod…​is Great!

Overall Thoughts

Complete SmartOS Migration

Recap: Why The Switch?

The Long-winded Process

New Drive Configuration

The Process Begins

Failure

Finishing the Move

Finishing Touches

Monitoring and Backups

Final Thoughts

Pause on SmartOS

Flair 58 Review

Workflow

Clean Up

My Overall Thoughts

Coffee Quality

Downsides

Conclusion

OpenBSD 7.5 on SmartOS

SmartOS Progress

The Server

Ansible

Last Steps

sndiod…is Great!