It has finally come time to “upgrade” the home server to hardware more suited for the scale, and free up the old gaming hardware for other projects.

I spent a few hours researching hardware options and settled on an N150 Mini PC, a USB Disk Enclosure, and 2 more 12TB drives.

Build

Proxmox wasn’t too difficult to get going, installed with default settings on the entire SDD drive. Being new to the software I poked around first to familiarize myself - found and set up an ACME certificate for the webui.

For storage, I removed one of the existing 12TB drives from the RAID and added that along with the two new drives into the drive bay. Proxmox found the drives with little issue, so I set up a ZFS pool of type raidz1.

I’m trying to use the least amount of drives to retain the 6TB of media we have, expand available space, and maintain single-drive redundancy. This plan should enable me to keep media apps live, build the new storage and move, then put the final 12TB drive in as a hot spare.

File Sharing

The next step of my plan was to get the storage accessible via Samba and NFS. I determined the best wey to do that would be from a container running on Proxmox, so to youtube I went. I found a very useful youtube video and followed along.

That got Samba going, and by poking around in the file sharing app I found the NFS settings and got that going as well. I mounted the new drive to my PC and to the old home server in preparation for migration.

Ubuntu & Microk8s

After that I moved on to the Ubuntu VM and getting microk8s going. I checked the box for microk8s during install and it was ready a few minutes after first boot. I validated that base things were running and checked which addons were enabled. Canonical has done a great job with creating something that works well out of the box, but uses older versions of addons/plugins and I like to keep things updated.

NGINX Ingress

Rather than enable the microk8s ingress addon, I followed the helm install guide. Once the CRDs and app itself were installed, I refreshed myself on how to make it accessible without a load balancer and manually edited the service to be type NodePort with a set ExternalIP.

ArgoCD

Another guide, a little tricker to get going. After install I was able to port-forward and access ArgoCD, but it was having SSL errors when connecting through the ingress. This took a few different tries of mixing the annotations on ArgoCD’s ingress and restarting until it finally worked.

The First apps I wanted to deploy in ArgoCD were cert-manager and external dns. Because these apps contain all configuration inside the values files, and they don’t have external dns names that could overlap - I was able to use the same Github Repo as my other apps to get them deployed initially.

Next app to move was homepage. This app also contains the entire config inside the values file, but does have an ingress that could overlap. I got around this for now by creating another ingress resource in my Github folder for the app and using exclusions in the manifests to filter the wrong ones out. This created my second accessible app on the new server.

I then added NGINX Ingress to ArgoCD using the same helm URL that I had installed with earlier, and may have been a little hasty in my review of the diff before syncing…

Roadblock

The first issue I noticed was that none of the external API based elements of homepage were loading. Digging deeper it seemed that all of the pods were having this issue, and it didn’t immediately seem like DNS (an issue i’ve created and fixed for myself before). I also noticed that the ingress resources no longer had an IP address, and external dns was no longer updating DNS. I was in the mood for movement so I switched gears.

Media Apps

My crazy idea is to add the apps to ArgoCD, create the pvc’s, stop both old and new apps, and then copy the app files on the backend. Barring any permission issues, this should enable me to move the apps from one server to other without having to reconfigure them.

For Prowlarr, Radarr, Sonarr, and qBittorrent the pvc isn’t under the deployment for the pod, so I was easily able to stop the apps via the ArgoCD UI. Plex would have been tricker but I overlooked the cpu and mem allocation and it was still looking for far more of both than the new cluster has and was stuck waiting for it.

I went through both ArgoCDs and gathered the pvc names for both the old and new and typed up the bash cp code to copy all of them over at once. I did this from the old media server where I mounted the new drive to earlier to get whatever hardware advantage I could, pasted in the code and hit enter. As expected it threw some permission errors, but not as many as I expected and more on the destination than the source.

I used the Proxmox console to access the file sharing container and did a chmod and chown there, then ran the copy script again on the old home server - no errors!

Roadblock Pt.2

Rather than messing with the old server’s URLs, I decided it was time to clone the project repo and start working in a new space. After setting that up on the new server in VS Code and Github, I added the new repo to ArgoCD and migrated the existing manifests to the new source URL. This enabled me to set new URLs for the new apps without affecting the old ones.

Now I’m back to my original issue, and when I ask gemini if it could be the ingress, coredns, or external dns, it tells me it could totally be one of those things. After a good hour of messing around with settings, I was tired and wanted to see something work so I deleted the ingress from argocd and microk8s helm and reinstalled only with microk8s helm as was working before.

That fixed my ingress issues with the apps, with ArgoCD, and with external dns. It still leaves the issue of app external access, I validated again that the pods can resolve addresses and decided it was a problem for another day. Good Night!

Project is continued here

---

← Back to blog