Complete SmartOS Migration

In my intial plan, I had intended to install my zones zpool on one of my SSDs and then have the data in its own data zpool. However, I was advised not to do that since I would not see any gains from using these SSDs. Additionally, I’d be having to do workarounds with SmartOS such as LOFS and such to get the data properly in each zone, which would have added complexity to my server. So at the advice of the maintainers, I decided to use only my new hard drives.

Once I got the new drives, I couldn’t wait. I decided to start the migration over the upcoming weekend. I installed my new drives in my Thinkcentre, with one hanging out the chassis since it only supported a single 3.5" drive. I started copying data from my jails to my new zones to delegated datasets. However, I started encountering issues. One issue was the constant battle of low memory and trying to move everything over to this new "server" without overloading the desktop. I had been using rsync up until this time, but since my destination zones had too little memory allocated to them, rsync would eventually fail. I found sftp to work, and eventually I reached a point where I had to just turn stuff off and allocate more memory to these zones to get data over efficiently.

I did my best to recreate all my jails as zones. Most of them were done beforehand since they had low CPU/memory requirements. I had some others such as Nextcloud, my samba server, Plex, and my friend’s minecraft server that required some messing around to get things right and get it correct in joyent/lx branded zones. Once it was all done, I was ready to power down both machines and swap the guts.

At this point, I had my 2 disks in my server and all else hooked up, but encountered an error: I was booting into single user mode constantly. After looking through the logs and beating my head against the wall, I had to revert everything and put stuff back in their messed up state with both computers running their duties. I kept Nextcloud, Immich, Matrix, and Minecraft on my old server for the night, and most of the next day. In the meantime, it was the end of the night.

The following day, I scoured the internet for some sort of answer, and came upon this post. In it the writer notes to size up the dump and swap space on the zones zpool. Even mentioning the exact same error I had gotten. So I followed his instructions, but kept swap the same since I was doing this on my live system, and not single user mode. But I increased my dump volsize:

I have 64GB RAM, where my old system which I built SmartOS on had 8GB. So the dump size was too small, and I set it to 40GB or something. When I put everything back in my server, it all worked!

The remaining bits were to put back together Nextcloud, Immich, Matrix, and Minecraft. I had ensured my data was current on Nextcloud, did a database dump/restore and Nextcloud was running well then. Immich, I had already moved the data and samba (namely, SmartOS' in-kernel SMB server) share. So my steps were to just restore the database stuff I backed up and spin Immich up in a new VM. Matrix and Minecraft restores went well too. At this point, I was fully on SmartOS.

I like to receive alerts for problems, so I put in place SMART data monitoring, as well as monitoring to detect if a zpool is not in an ONLINE state or experiencing problems. Additionally, I have a monitor that will alert me if one of my zones goes offline.

The last bit is backups, which, I am using znapzend to backup my delegated ZFS datasets. I have backups of my bhyve virtual machines going to my storage zone. My last plan of action is to get all my configs backed up to a better spot so I can use Ansible or something to neatly rebuild my servers.