I have been paying to host my Matrix synapse server for 2.5 years now. It has been one of my favorite projects to host because I have found a great communities this way and have been able to learn a lot being on Matrix. The problem for me though is, the synapse server is a bit of a resource hog, and has grown to be expensive.

Synapse

Synapse is the current Matrix server, in addition to dendrite which has been a gradual work in progress. I think dendrite is ready to go as a replacement, apparently…​anecdotally. However, there is no way to migrate from synapse to dendrite yet.

The biggest issue I have run into with Synapse is it is pretty resource intensive. I started out using a VPS with 2 cores and 1GB RAM, but with more activity, it couldn’t keep up and my server would basically DoS itself from the extra load and become unavailable. So, I upgraded to a 2 core and 2GB RAM server. That bumped my prices from $5 to $15 per month. A price I didn’t want to pay. But, I did it anyway.

My Setup

I have been running synapse on OpenBSD with PostgreSQL, in addition to hosting element-web on my server. With that added performance, the server is basically sitting idle most of the time, all the while I do need that extra RAM because of what synapse gobbles up.

My New Plan

I already paid for my server at home, as well as the electric bill and everything else. Why am I letting this perfectly good hardware go to waste? After hearing a good idea on the OpenBSD Matrix group, I decided it is a good idea to follow a similar idea. My home internet is already pretty good, so why not move synapse and PostgreSQL to my local server? This isn’t a comprehensive migration solution, but a loose documenting of some of the stuff I did to accomplish my task.

I don’t want to move EVERYTHING back my house, because I don’t want my home IP exposed. So my new gameplan is:

  1. Set up an OpenBSD 7.3 virtual machine on my local server

  2. Set up a site to site VPN for migration

  3. Restore my data to this server

  4. Spin up a new OpenBSD 7.3 VPS (my current one is 7.2 so might as well go afresh)

  5. Set up all the relayd, httpd, etc. etc. on it

  6. Set up a site to site VPN between the 2

  7. Shut down the old

  8. Dump and restore the database to my local server

  9. Restore the last bit of data with an rsync

  10. Turn on all the lights

  11. Make DNS changes

With this done, let’s move forward.

Set up a Virtual Machine

My local server is running FreeBSD 13.2, and I am managing my VMs with vm-bhyve. Thankfully, I already created an OpenBSD 7.3 image, so I provisioned it.

sudo vm provision <long-id-here> matrix

I then used restic to restore backup configurations locally. Mainly user stuff, some config files and my element-web stuff. More will come later as I move along.

Set up a VPN

For this I chose WireGuard. It handles domain names, which works for my ever changing IP address here at home. I take advantage of Namecheap’s A+ record to handle my ISP’s dynamically given IP with ddclient. The WireGuard setup was easier than I thought.

This is only a temporary VPN, since it will be to conveniently get my data off the old server and onto the new. I will repeat this setup later, but with the new server. Same thing, different keys. And I will delete this temporary tunnel.

On my local server, I created /etc/hostname.wg0

wgkey mykeyLocal wgport 51820
inet 10.25.50.2 255.255.255.0
wgpeer pubkeyRemote wgendpoint remote.example.com 51820 wgaip 10.25.50.1

And on my remote server

wgkey mykeyRemote wgport 51820
inet 10.25.50.2 255.255.255.0
wgpeer pubkeyLocal wgendpoint local.example.com 51820 wgaip 10.25.50.2

On each, I also have firewall rules. I will tighten this down later, this is just temporary:

pass in on egress proto udp from any to any port 51820
pass on wg0
pass out on egress inet from (wg0:network) nat-to (egress:0)

Load the firewall rules pfctl -f /etc/pf.conf. Later I want to ratchet this down more so the VPN only works and is exposed between my home and the server. Another downside which I will have to figure out later, is to monitor if my IP changes and do something about adding it to PF. This is outside the scope of this, but…​I’ll put my script in later in this post.

Restore my data

I have my new server set up with its own disk at /var/synapse to use. With my servers communicating, let’s sync up /var/synapse.

rsync -az /var/synapse/ root@10.25.50.2:/var/synapse/

Let it run, eventually it will be synced up. Later when I turn the old server off, I have to sync these again. Now that I think about it, Python versions are probably incorrect too so I will likely have to handle that since I build synapse instead of relying on ports. I just found this easier for my own taste since I don’t want to ride on OpenBSD -current (actually, that’s what I did in the end, more on that later).

Restoring Packages

When I backup my matrix server, I also run pkg_info -m -z — > $HOME/packages.txt as a part of my script. This dumps my installed packages in a fuzzy format so I can quickly restore them on the new server like this

pkg_add `cat packages.txt`

The New VPS

Now to set up the new server. Previously, I was partitioning this server by hand because I wanted to maximize space for /var/synapse. However, since I have plenty of free space on my local server to store my Matrix data, I went with the auto partitioning in the VPS since it won’t be storing anything.

This server will only be handling:

  • httpd

  • relayd

  • pf

  • sshd

I copied all these configs, as well as crontabs, my personal scripts, and user data to the VPS and set up my VPN as I did earlier. I did make adjustments to my firewall. I have a script running that will do a DNS check every minute to the authoritative DNS server for my domain name to see if the A record changed. And if it did, add the new IP address to PF by using a table, then removing the old IP address.

Here’s my configs for each file:

/etc/httpd.conf

types {
        include "/usr/share/misc/mime.types"
}

server "web.example.com" {
        listen on * port 8080
        location "/*" {
                root "/htdocs/web.example.com"
                #request strip 1
        }
}

server "example.com" {
        listen on * port 80
        alias autoconfig.example.com
        location "/.well-known/acme-challenge/*" {
                root "/acme"
                request strip 2
        }
        location * {
                block return 302 "https://$HTTP_HOST$REQUEST_URI"
        }
}

server "example.com" {
        listen on * port 443
        alias autoconfig.example.com
        hsts { subdomains, preload }
        location "/.well-known/acme-challenge/*" {
                root "/acme"
                request strip 2
        }
        location "/.well-known/matrix/*" {
                root "/matrix"
                request strip 2
        }
        location "/*" {
                directory auto index
                root "/matrix"
        }
}

/etc/relayd.conf

ip4="$publicIP4"
ip6="$publicIP6"

table <element> { 127.0.0.1 }
table <matrixserver>  { 10.25.50.2 } # the IP of my synapse server in the VPN's network

log connection

http protocol "https" {
        tls { tlsv1.2, tlsv1.3, ciphers secure }
        tls keypair "example.com"
        tls {session tickets }
        match header set "X-Forwarded-For"   value "$REMOTE_ADDR"
        match header set "X-Forwarded-Proto" value "https"
        #match header set "X-Forwarded-Proto" value "SERVER_ADDR:$SERVER_PORT"

        match response header set "Server" value "flippinburgers"

        # set CORS header for .well-known/matrix/server, .well-known/matrix/client
        # httpd does not support setting headers, so do it here
        match request path "/.well-known/matrix/*" tag "matrix-cors"
        match response tagged "matrix-cors" header set "Access-Control-Allow-Origin" value "*"

        pass request quick header "Host" value "web.example.com" forward to <element>

        # pass on non-matrix traffic to webserver
        #pass                                 forward to <webserver>
        #pass                                 forward to <element>
}

relay "https_traffic" {
        listen on $ip4 port 443 tls
        protocol "https"
        forward to <matrixserver> port 8008 check tcp
        forward to <element> port 8080 check tcp
}

relay "https_traffic6" {
        listen on $ip6 port 443 tls
        protocol "https"
        forward to <matrixserver> port 8008 check tcp
        forward to <element> port 8080 check tcp
}

http protocol "matrix" {
        tls { tlsv1.2, tlsv1.3, ciphers secure }
        tls keypair "example.com"
        block
        pass quick path "/_matrix/*"         forward to <matrixserver>
        pass quick path "/_synapse/client/*" forward to <matrixserver>
}

relay "matrix_federation" {
        listen on $ip4 port 8448 tls
        protocol "matrix"
        forward to <matrixserver> port 8008 check tcp
}

relay "matrix_federation6" {
        listen on $ip6 port 8448 tls
        protocol "matrix"
        forward to <matrixserver> port 8008 check tcp
}

/etc/pf.conf

table <homeip> persist file "/etc/homeip"
table <bruteforce> persist
table <rfc1918> const { 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8 }

set skip on lo

set loginterface egress
match in all scrub (no-df random-id max-mss 1440)
antispoof quick for egress
block in on egress from { <bruteforce> <rfc1918> } to any
block in log all
pass from (self)
pass out on egress

pass in on egress inet proto tcp from any to (egress) port { http, https, 8008, 8448 } keep state
pass in on egress inet6 proto tcp from any to (egress) port { http, https, 8008, 8448 } keep state

pass proto tcp from any to (egress) port ssh \
        keep state (max-src-conn 15, max-src-conn-rate 5/3, \
                overload <bruteforce> flush global)

pass in on egress proto udp from <homeip> to any port 51821
pass on wg1
pass out on egress inet from (wg1:network) nat-to (egress:0)

pass in on egress inet6 proto icmp6 all icmp6-type { echoreq routeradv neighbrsol neighbradv }
pass in on egress inet proto icmp icmp-type { echoreq, unreach }

# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010

# Port build user does not need network
block return out log proto {tcp udp} user _pbuild

And here is my script for checking the IP and updating my firewall rules. This runs every minute in cron

/usr/local/bin/changeip

#!/usr/bin/env sh

currentip=$(dig +short @dns1.registrar-servers.com myhomeip.example.com)
fileip=$(cat /etc/homeip)

if [ "$currentip" != "$fileip" ]; then
        echo "$currentip" > /etc/homeip
        pfctl -t homeip -T add "$currentip"
        pfctl -t homeip -T delete "$fileip"
fi
Note
 This script isn’t tested, my IP hasn’t changed yet :)

With all these configuration on hand, I should be able to accept connections and pass them along back to my home server’s virtual machine, once that part is ready. Last bit is the WireGuard tunnel part. It’s basically what I did earlier between my old VPS and my VM at home, but change the IP address to something different for the new VPS.

Moving to the New Server

For moving to the new server, I turned off synapse on the old server so I wasn’t getting new data. I then did a dump of the database and a restore to the database on the new one. I basically followed the Option 1 method in /usr/local/share/doc/pkg-readmes/postgresql-server to get this.

Admittedly, I didn’t really change my pf rules a whole lot on this server since the only publicly listening service is synapse. And I set my bind address in the homeserver.yaml file to be the VPN address. And only servers in this VPN tunnel can talk with each other. And, it’s a public endpoint anyway and with my router and all in the way, it’s not a threat for me.

Now that PostgreSQL is restored to my local database I did one last rsync of my data and made a couple tweaks to the homeserver.yaml file, like the bind address.

The Problem

Earlier I said I did not want to play the OpenBSD -current game with this server. Well, meh. I was having troubles getting the server to work right with building synapse for OpenBSD 7.3. I could have made it work, but I didn’t want to deal with it. So I upgraded to -current and am now using the synapse package that is in ports. I feel safer with this because I can easily and quickly power off my VM and do a ZFS snapshot of it before doing my sysupgrade.

Wrapping Up

The last part then was to start the synapse server, ensure relayd and httpd were playing nice on my VPS. I made changes to DNS to ensure my domain names were pointing to the VPS, and then I got new SSL certs. I then made sure that everything was talking with DNS propagated, and it was! I was able to chat over Matrix again.

I got backups working of my synapse server too. I don’t care to backup my web server. That’s all documented here so I can copy/paste ;) I’ll probably back it up anyway but it isn’t a priority at the moment of writing.

So yeah. That was my migration process. I’m saving a few bucks a month now. I think performance is a bit better. I get a 404 here and there though that I need to diagnose. Maybe I’m losing connection between my home server and this VPS but it is only ever for a minute right now at any given time. Not the biggest issue, and it is a bit expected.