Homelab

What’s the shape? #

The homelab is one of my passion projects. It’s not even birthed yet and there’s so much to do and experiment with. If things go alright, I’ll probably have my dream homelab in 1-2 years. I want it well documented , well maintained , reproducible, high quality and of-course do all the things I want it to do. Why? Vendor independence, privacy, costs, peace of mind, fun. pro: you can fix the problem, con: you have to fix the problem.

Properties #

Should have solid observability. This is also my playground for things.
Upto-date and mostly automated
Good and live documentation and should be easily debug-able
It’s not like I want to selfhost everything, if there are external hosted services which satisfy my needs I would skip selfhosting. I want this to be seamless, controlled and do not want to spend hours on debugging any issue. Same goes for hardware.
I am not building enterprise network here, I can have fun so stupid stuff is sort of allowed

Components components #

NOTE:

I have not built the homelab, this is first draft plan.

These are not actual boundaries, I am just laying out an abstract layout here for me to build on.

Things might be not be technically correct(I might be placing things in the wrong places of the stack as-well)

I have not thought all of this properly but merely dumping ideas here. this is NOT the topology

Name	Remark
Goku	Bastion server, External VPS
Dobbies	Any service, many service, whatever random thing. These will run it.
SRK	Anything media management goes here, runs locallty
Warehouse	multi purpose storage server(s)
Cloud ZEPEEYOU	AI experiments helper
Rasta	Throwaway servers
Piccolo	Trusted, Persistent Good ol webserver
daCNC	My phone

Goku #

Sort of a bastion host . Idea is to have access to all my services from one place. Eg. I should be able to ssh from my phone to this machine and manage things even if I am away from my laptop. It should have the tools installed I need in a dev/sysadmin machine.

Location: VPS
Threat Model: Assume that it can be compromised and reduce attack surface accordingly.
Possible stuff here: centralized logging , centralized observability center, orchestrator center, Teleport w 2FA
Concerns: The usecase and motive of this component sort of contradicts. I am expecting this to be target but at the same time making this the most powerful and yet SOP in a way? Need to think.

Dobbies #

Local RPi(s)/Small computers/NUCs, can name them dobby-1, dobby-2 etc. Host small tools or whatever that I want to use locally or maybe expose some to the public internet as-well.

Location: Local
Possible stuff here: These will basically do anything. ArchiveWarrior stuff, bespoke scripts, see my secondary toolchest for complete list of tools that are already selfhostable/can be made selfhostable to fit my needs.

SRK #

The media server , connected to a NAS most likely. . I wanted to be local first, requiring internet to reach my media does not make sense but I probably would want to have public access to this in-case.

Location: Local
Useful stuff
- Perfect Media Server
- Self-hosted media center, based on open source software | Lobsters
- *arr services like Prowlarr, Lidarr, Sonarr Radarr, Tdarr etc
- gerbera/gerbera

Warehouse #

Some kind of storage server/multiple servers. I have to explore this, zfs, btrfs etc. This will store archives, media files etc. This is not the backup, it’ll be done separately.

Cloud ZEPEEYOU #

To carry out AI experiments. Not worrying about this much rn as this will be specific to usecase but definitely want this to be billed on usage lol.

Rasta #

Location: VPS, needs to be ephemeral

A test server / dummy that i can trash and recreate anytime, installs my necessary tools automatically on creation etc.

Piccolo #

Trusted, Persistent Good ol webserver. This will have a solid reverse proxy in place so that I spin up random APIs/Websites for public quickly.

daCNC #

This is my phone. This is more like a remote control for things and I’ve set some phone specific tasker profiles which are super useful. Eg. Taking picture and Uploading it to my Google Drive via SMS trigger etc.

Orchestration #

We have few options.

Name	Remark
Nomad	Based on what I read it seems like this would be suitable for my homelab
Kubernetes	There are things like k3s
Promox	Runs LXC and VMs, Min 3 nodes needed

What keeps everything together? I have not decided yet but guess it’ll be a mix of terraform and ansible .

I think I’ll go with Nomad like I mentioned.

Networking #

Goals

I should be able to access(ssh) certain private devices/services securely from the public internet.
I should be able to access certain public services securely from the public internet. (eg. fileserver, mediaserver etc)
Local devices should be able to talk to each other, preferably put local devices into a different VLAN and internet facing stuff into a DMZ.
Should have a proper way to access geoblocked content
Security, have not thought about my threat model properly.

Anti-Goals

Not trying to be anonymous here
Even though I want to build a mesh network, in this iteration it’s not the goal. At most I might be use tailscale or something similar.

VPN #

Mesh VPN setup
- Goal: Allow my devices to talk to each other
- Something like Tailscale is looking juicy here
Road warrior setup (VPN VPS)
- Goal: Something that allows me to access my devices at home when I am out.
Encrypted Traffic + Hide source IP(geo) setup
- Goal: Not anonymity but more of privacy and bypassing censorship. Eg. When using insecure public wifi or anything else that fits.
- I can selfhost this but with that I cannot keep switching countries etc. So might be good idea to go with something like Mullvad VPN
Tunnels
- Goal: Expose public only services quickly, give temporary access to something that I am running locally etc.

Proxy #

Forward Proxy #

I do not really feel the need of a forward proxy as such at the moment. But I can see one usecase: Censorship bypass. Setting up shadowsocks, vray and cloak along w tor proxy(whatever combination makes sense for the usecase) might be a good idea. Because you don’t need them until you need them :)

Reverse Proxy #

I can use these things to do load-balancing/ssl termination/reverse proxy/protocol demultiplexing/HA /failover/caching/rate-limiting etc. Here’s a more complete list . After some comparison, I think i’ll be going with either Traefik or Caddy.

Router #

We have the options of OpenWRT and OPNSense here. We can mix and match, will think of exact topology later.
Point web services logs to fail2ban and let it handle rate-limiting etc.
For extra points can check Crowdsec

DNS #

This is one bad boi. I probably just want to run local resolver. Maybe an authoritative server replicated to secondaries later. But for now, I plan PiHole/Blocky+Unbound.

Once we have a reverse-proxy setup, you can have your local DNS server point to your reverse proxy for whatever domain. eg. *.home. Also see what domain name to use for your home network? home.arpa
Some people recommend doing split-horizon DNS along with reverse-proxy if running multiple services, I don’t see a point rn but maybe I’ll later.

Local Network #

VLANs and Subnets #

Reason: It’s nice to separate things with vlans and firewall rules + IoT devices are known to be insecure . (Sort of an overkill but who cares)
Subnets
- VLAN 1 for home devices LAN
- VLAN 2 for trusted IoT which cannot run VPN client, access to the Internet allowed
- VLAN 3 for isolated (untrusted) IoT devices
- VLAN 4 for DMZ for publicly hosted services etc
VPN runs on VLAN1
What comes and goes out of these VLANS to be configured via firewalls
Check if we’ll need a managed switch or OpenWRT will cut it

DMZ #

Reason: Because I plan to host public facing services it makes sense to have a DMZ.
Objective is to provide firewall capabilities between hosts in the DMZ and hosts on the internal network.

Monitoring the network #

I haven’t explored this properly, so just link dumping.

Backup Plan #

NOTE ⚠

I have not started backing up anything at the moment, there are just scattered copies etc.

This will be an incremental process, but will start soon. (18th Feb'23)

In some cases I need to do some prior work, eg. my video files are scrattered all over the internet and different drives. I have to put them together into one place before I even think of backing them up.

After some reading and going through various backup solutions, I decided that the primary tool to make my backups will be restic . I initially considered borg with rysnc.net , but using restic lets me use cheaper storage alternatives and at the time of this writing I am trying to cut costs. I haven’t really looked into tarsnap but I wanted to.

The main strategy I am going to follow is the 3-2-1 strategy . (3 copies, 2 different media, 1 offsite) + restore tested.
I am not backing up emails, DMs etc as I consider them ephemeral and I try to set disappear timer in most of them.

Data inventory #

Name	What about it?	Priority	Backed Up?
Passwords & 2FA passphrases	Strengthen master pass. Create regular encrypted export from bitwarden. Backup local `pass` store.	5/5	👎
2FA	Google Authenticator, no backups nothing, do something.	5/5	👎
PC	Nothing worth backing up here	0/5	👎
Laptop	Installed package list and configurations(dot files). Browser profile+ext. configurations	5/5	👎
Phone	Tasker configuration. App list + configuration	2/5	👎
Access & Encryption Keys	Put SSH and Age private keys somewhere safe, make way for automatic backup of rotated keys	5/5	👎
Homelab configuration	I don’t have the homelab ready now so would not know	0/5	👎
Public and Private repositories	Github+Bitbucket mirrors. Offsite(forked+own+custom repo) backup.	1/5	👎
eBooks	I have a book collection on google drive. Setup automated organization. Then backup.	4/5	👎
Internet Documents	Research papers and other random PDFs. Put them in appropriate place first. Backup.	1/5	👎
Internet memes&photos&videos	Make a media browser/search engine first for this. Backup everything as application backup afterwards.	0.2/5	👎
Personal Photos	Photos from Google drive/photos	2/5	👎
Personal Screenshots	Screenshots from Google drive	1/5	👎
Personal Documents	Google drive, Physical copies. Put them in appropriate place first. Backup.	3/5	👎
Personal Social Media Dumps	First organize. Then backup.	1/5	👎

Backup details #

This section will be incrementally populated with details about how I am doing the backups etc. I’ll probably do it in literate programming fashion.

Threat model of data loss and disaster recovery is no longer hardware failure: it’s account lock out. So make sure to use replicate stuff to different media/providers.

Resources & Links #

Compute providers #

Name	Remark
Vultr	Heard good things
Exoscale	One person said good thing about this
Hetzner	Good value for VPS, support, transparent, peering issues (Now as ARM64! cheap :))
Time4VPS	Idk, probably good and cheap
Uberspace	Unique “shared server” concept. In theory you can use as much ressources as you want but in that case other customers are impacted.
Scaleway	Complaints about support
Oracle	It’s a free tire but lot of complaints about dark patterns. Use it w caution.
Tornado VPS	Idk, probably good and cheap
Linode	Little pricy but trusy
DigitalOcean	Little pricy but trusy(2)
RackNerd	Black friday yearly deal is juicy
netcup	Old fellow, probably good
SSD Nodes	Cheap stuff but good
OVH	French company, once data center caught fire but otherwise reviews are mixed. Interesting bare metal offerings

Other server resources #

Storage providers #

Name	Remark
Hetzner storage boxes	have not checked but good things heard
Blackblaze B2	moi wants 2 use this for offsite backup

Storage resources #

Best practices #

Hardening system #

Reverse proxy only accepting domain-name queries instead of the IP.

Environment #

Observability #

samber/awesome-prometheus-alerts : Collection of Prometheus alerting rules
monitoringsucks/metrics-catalog : Catalog of valuable metrics you might want to collect
Enapiuz/awesome-monitoring : List of tools for monitoring and analyze everything.
AnalogJ/scrutiny