Server-ShenanigansI somehow got convinced to try running Arch for my Pi-Server, when my only prior experience with Linux is Mint. I will make extremely stupid mistakes and probably do absolute newbie errors.
This thread will record my suffering for your enjoyment.
re: Server-ShenanigansSo, First Mistake. Its not even software related, but hardware. Planning on using a store bought (and already used) MicroSD Card for my Operating system.
Now for a very low intensity Usage that might work, but most common (and cheap) MicroSD Cards are not built for running longterm and with frequent writing cycles and apparently might suddenly fail without warning.
Now I either gotta buy a better MicroSD Card or accept that I wont be physically seperating Operating System from Data Storage (which I would highly prefer).
re: Server-Shenanigans, Booting ArchSo, today I actually started and I actually managed to boot arch.
So first mistake, I tried to use regular arch, but well, its a Pi, so it has to be ARM-Arch.
Then silly me thought I could just flash it onto my MicroSD (now the proper one), which of course did not work.
Because this isnt Mint, this is arch, so I had to set up manually the partitions, Filesystems for the bootloader and root, then copy the files into it. Then I had to tell Arch, that it needs to actually mount its bootloader and where to find that.
Funnily enough, the only guide I found actually contained an error (luckily something that was pretty self explanatory and easy to do manually).
re: Server-Shenanigans, MountingToday I despaired, after managing to change the keyboard layout I realised that none of the ctl commands were working and the system needed ages to boot. Lots of essential services, like systemd, dbus werent working.
After way to long I figured out that my boot partition wasnt mounting, despite me putting it into fstab with the name it had while connected to my laptop.
Well, turns out that a devices name (unsurprisingly) isnt static, but can change depending on the connected system.
So it could try mounting that /dev/sdb1 all it wanted, if it was now called /dev/mmcblk1p1.
re: Server-Shenanigans, PacmanAfter some smaller stuff, I finally wanted to to update the basic software I had already installed.
Minor issue, the package authentification didnt work, so I couldnt update or download anything.
Now I had the hen and egg problem. I need to verify packages to install them, but I couldnt verify, as I thought I had deleted something needed for that.
Luckily you can turn verification off (dont do this unless absolutely necessary).
Turns out later the actual issue was that I was trying to populate my authentification (keys) from the wrong source (cause archlinux != archlinuxarm)
Now I can install silly things like the man pages...
re: Server-Shenanigans, First ReinstallAfter trying to relocate Home onto a different Device for later easier Reinstalls (which seemingly worked), now my Arch No longer recognizes that said drive is even physically connected.
According to various sources the issue might be connected to the Bootloader. Funny thing, I cannot access any Bootloader. Like the Device Boots, but the Pi-Bootloader never shows up.
As I cannot find any solution online, I guess we will try to do a clean reinstall?
re: Server-Shenanigans, WiFi and SSHA somewhat clean reinstall later I now managed to set up WiFi and SSH.
(Please dont tell anyone that so far SSH is completely unsecured and only works locally)
WiFi did give me some trouble and there is still an error message I couldnt resolve yet (or find online) and my router constantly tries to connect to my Server for some so far unknown reason, but thats tomorrow Nikas Problem.
re: Server-Shenanigans, Basic SSH-HardeningToday I didnt get much done. I only fixed some errors and made my SSH slightly safer and more convenient. - Static Local IP Address - removing Password Login in exchange for Key Based Login - change the port SSH runs on - Login Shortcut, so I dont have to remember the IP Address
Next Up is the First Major Boss (cause I still need to figure out how to do it): Making SSH work from Outside my Home Network...
re: Server-Shenanigans, Remote-Access/WireguardIt only took me a workweek, a lot of trial and error and some help (thanks again @fugi) to make my Device Remote Accessible.
The issue was less the process, but more my own self imposed restriction of using as few external services as possible and the fact that my server is behind CGNAT (So no IPv4 Adress from the Outside).
My solution was to create a Tunnel over Wireguard, directly connected via IPv6 to my server. I also set up DDNS with a Domain, but technically that would not be necessary, but might be useful later.
re: Server-Shenanigans, The Part were using Arch Bites meUsing Arch as a Server might not have been the best idea. Arch rolling release model kind of expects you to manually administer updates and doesnt really seem to differentiate between Security and Other Updates.
So I either make sure I dont forget to regularly update my server (Ahahahahaha) or I built a job, that will probably at some point brick my system or (which is the most complicated, but probably most sustainable solution), build myself a reminder system.
re: Server-Shenanigans, FirewallsSo. I could not access my Server remotely, despite it temporarily working.
I was really confused, cause I didnt change anything settings wise. Technically true, but I did enable something again. UFW.
I installed it as my Firewall, but the Forwarding of the VPN Interface has been done via IP Tables.
I did not realise that both are firewalls and UFW overwrites IP Tables (which I would have known if I had checked Iptables setting). So my VPN Interface isnt actually getting forwarded anywhere, so can never be found or seen for connection.
re: Server-Shenanigans, Finally Remote AccessAfter trying it over IPv6, which often doesnt work, when you remote device doesnt get an Address, I finally found a working solution.
I got myself a VPS and use it as a Headscale (Selfhoste Tailscale) Server Node.
With that I get through CGNAT (as the Homelab is in Tailscale the Device establishing the connection), its still mostly under my control, as the Headscale Server actually only initialises the Connection, the Rest is a Wireguard Tunnel between the Individual Nodes as far as I know...)
Apart from Docker it wasnt actually that difficult to set up, and now I have a reasonably modular, vaguely safe way to Remote Access my Homelab (still need to do a lot of Server Hardening tho and see if it works in the locations I had in the past issues).
re: Server-Shenanigans, First ServiceI have my first proper service up (A Seafile) and running and despite having to fight for several days, I managed to actually fix all the initial issues.
It works on my own Caddy Reverse Proxy instead of the one they use, which initially seemed impossible and it also doesnt expose ports anymore.
re: Server-Shenanigans, PrideSetting this Homelab up was a mess, but I am genuinly quite proud of the result.
My only not self controlled Tool I use for my homelab is the VPS on which I run headscale (needed a Public IP) for my own Tailnet. Everything else is Open Source and Self Hosted.
All my Services (so far only 1 admittetly), are dockerized and are only accessible via Reverse Proxy to Minimise Port Exposure.
Was this way to much effort and I could have used Cloudflare, yes, but I dont care, this is my security, I want to screw it up myself.
re: Server-Shenanigans, ExpandingSo today I went from 1 Service to 4, with two more already in work.
So far I have set up Vikunja (a Task Manager), Mealie (Recepie Management), Seafile (Data Storage) and Hedgedoc (Notetaking).
I am still planning for Radicale (a Calendar Server) and Syncthing (mainly to Synchronise Obsidian)
Next step after that will probably to add TLS back in, simply so my browser stops annoying me (its already encrypted, due to being only accessible via Tailnet)
re: Server-Shenanigans, TLSSo, all my for now desired Services are Up and Running, now I need to make this construction actually safe (ish)
I do have a baseline of Safety due to it only being accessible via Tailnet and the Server only Opening Ports towards the Tailnet, but I feel I should do a bit more.
Next up is TLS. That one mainly cause the constant reminders of of "This is just HTTP, so very unsafe" are annoying.
Just need to figure out if I want to keep it fully internal, with a self Signed Certificate or try to essentially misuse a Public Certificate for solely internal use.
re: Server-Shenanigans, TLS@Quz I have done both, and I find getting a (wildcard) cert from letsencrypt via DNS challenge (works without being reachable from outside) to be nicer. It's not even misusing it, imo
