Homelab

NOTE

  • Everything about this page and my homelab and selfhosted tools are very much WIP ✨ 🚧
  • geekodour/workshopexternal link : Keeping track of homelab and other workshop things.

TIP:

  • I did some internet comparison/study when preparing this selfhosting setup. I always take notes when studying anything, all my notes related to selfhosting can be found here in my wikiexternal link .
  • Check “tags” and “Links to this note” in the wiki page.

What’s the shape? #

The homelab is one of my passion projects. It’s not even birthed yet and there’s so much to do and experiment with. If things go alright, I’ll probably have my dream homelab in 1-2 years. I want itexternal link wellexternal link documentedexternal link , well maintainedexternal link , reproducible, high quality and of-course do all the things I want it to do. Why? Vendor independence, privacy, costs, peace of mind, fun. pro: you can fix the problem, con: you have to fix the problem.

Properties #

  • Should have solid observability. This is also my playground for things.
  • Upto-date and mostly automated
  • Good and live documentation and should be easily debug-able
  • It’s not like I want to selfhost everything, if there are external hosted services which satisfy my needs I would skip selfhosting. I want this to be seamless, controlled and do not want to spend hours on debugging anyexternal link issue. Same goes for hardware.
  • I am not building enterprise network here, I can have fun so stupid stuff is sort of allowed

Components components #

NOTE:

  • I have not built the homelab, this is first draft plan.
  • These are not actual boundaries, I am just laying out an abstract layout here for me to build on.
  • Things might be not be technically correct(I might be placing things in the wrong places of the stack as-well)
  • I have not thought all of this properly but merely dumping ideas here. this is NOT the topology
Name Remark
Goku Bastion server, External VPS
Dobbies Any service, many service, whatever random thing. These will run it.
SRK Anything media management goes here, runs locallty
Warehouse multi purpose storage server(s)
Cloud ZEPEEYOU AI experiments helper
Rasta Throwaway servers
Piccolo Trusted, Persistent Good ol webserver
daCNC My phone

Goku #

Sort of a bastion hostexternal link . Idea is to have access to all my services from one place. Eg. I should be able to ssh from my phone to this machine and manage things even if I am away from my laptop. It should have the tools installed I need in a dev/sysadmin machine.

  • Location: VPS
  • Threat Model: Assume that it can be compromised and reduce attack surface accordingly.
  • Possible stuff here: centralized loggingexternal link , centralized observability center, orchestrator center, Teleportexternal link w 2FA
  • Concerns: The usecase and motive of this component sort of contradicts. I am expecting this to be target but at the same time making this the most powerful and yet SOP in a way? Need to think.

Dobbies #

Local RPi(s)/Small computers/NUCs, can name them dobby-1, dobby-2 etc. Host small tools or whatever that I want to use locally or maybe expose some to the public internet as-well.

  • Location: Local
  • Possible stuff here: These will basically do anything. ArchiveWarrior stuff, bespoke scripts, see my secondary toolchest for complete list of tools that are already selfhostable/can be made selfhostable to fit my needs.

SRK #

The media serverexternal link , connectedexternal link to a NAS most likely. . I wanted to be local first, requiring internet to reach my mediaexternal link does not makeexternal link sense but I probably would want to have public access to this in-case.

Warehouse #

Some kind of storage server/multiple servers. I have to explore this, zfs, btrfs etc. This will store archives, media files etc. This is not the backup, it’ll be done separately.

Cloud ZEPEEYOU #

To carry out AI experiments. Not worrying about this much rn as this will be specific to usecase but definitely want this to be billed on usage lol.

Rasta #

  • Location: VPS, needs to be ephemeral

A test server / dummy that i can trash and recreate anytime, installs my necessary tools automatically on creation etc.

Piccolo #

Trusted, Persistent Good ol webserver. This will have a solid reverse proxy in place so that I spin up random APIs/Websites for public quickly.

daCNC #

This is my phone. This is more like a remote control for things and I’ve set some phone specific tasker profiles which are super useful. Eg. Taking picture and Uploading it to my Google Drive via SMS trigger etc.

Orchestration #

We have few options.

Name Remark
Nomad Based onexternal link whatexternal link I readexternal link it seemsexternal link like thisexternal link would be suitable for my homelab
Kubernetes There are things like k3sexternal link
Promox Runs LXC and VMs, Min 3 nodes needed

What keeps everything together? I have not decided yet but guess it’ll be a mix of terraform and ansibleexternal link .

I think I’ll go with Nomad like I mentioned.

Networking #

Goals

  • I should be able to access(ssh) certain private devices/services securely from the public internet.
  • I should be able to access certain public services securely from the public internet. (eg. fileserver, mediaserver etc)
  • Local devices should be able to talk to each other, preferably put local devices into a different VLAN and internet facing stuff into a DMZ.
  • Should have a proper way to access geoblocked content
  • Security, have not thought about my threat model properly.

Anti-Goals

  • Not trying to be anonymous here
  • Even though I want to build a mesh network, in this iteration it’s not the goal. At most I might be use tailscale or something similar.

VPN #

  • Mesh VPN setup
    • Goal: Allow my devices to talk to each other
    • Something like Tailscale is looking juicy here
  • Road warrior setup (VPN VPS)
    • Goal: Something that allows me to access my devices at home when I am out.
  • Encrypted Traffic + Hide source IP(geo) setup
    • Goal: Not anonymity but more of privacy and bypassing censorship. Eg. When using insecure public wifi or anything else that fits.
    • I can selfhost this but with that I cannot keep switching countries etc. So might be good idea to go with something like Mullvad VPNexternal link
  • Tunnels
    • Goal: Expose public only services quickly, give temporary access to something that I am running locally etc.

Proxy #

Forward Proxy #

I do not really feel the need of a forward proxy as such at the moment. But I can see one usecase: Censorship bypass. Setting up shadowsocks, vray and cloak along w tor proxy(whatever combination makes sense for the usecase) might be a good idea. Because you don’t need them until you need them :)

Reverse Proxy #

I can use these things to do load-balancing/ssl termination/reverse proxy/protocol demultiplexing/HAexternal link /failover/caching/rate-limiting etc. Here’s a moreexternal link complete listexternal link . After some comparison, I think i’ll be going with either Traefik or Caddy.

Router #

  • We have the options of OpenWRT and OPNSense here. We can mix and match, will think of exact topology later.
  • Point web services logs to fail2ban and let it handle rate-limiting etc.
  • For extra points can check Crowdsec

DNS #

This is one bad boi. I probably just want to run local resolver. Maybe an authoritative server replicated to secondaries later. But for now, I plan PiHole/Blocky+Unbound.

  • Once we have a reverse-proxy setup, you can have your local DNS server point to your reverse proxy for whatever domain. eg. *.home. Also see what domain name to use for your home network? home.arpaexternal link
  • Some people recommend doing split-horizon DNS along with reverse-proxy if running multiple services, I don’t see a point rn but maybe I’ll later.

Local Network #

VLANs and Subnets #

  • Reason: It’s nice to separate things with vlans and firewall rules + IoT devices are known to be insecureexternal link . (Sort of an overkill but who cares)
  • Subnets
    • VLAN 1 for home devices LAN
    • VLAN 2 for trusted IoT which cannot run VPN client, access to the Internet allowed
    • VLAN 3 for isolated (untrusted) IoT devices
    • VLAN 4 for DMZ for publicly hosted services etc
  • VPN runs on VLAN1
  • What comes and goes out of these VLANS to be configured via firewalls
  • Check if we’ll need a managed switch or OpenWRT will cut itexternal link

DMZ #

  • Reason: Because I plan to host public facing services it makes sense to have a DMZ.
  • Objective is to provide firewall capabilities between hosts in the DMZ and hosts on the internal network.

Backup Plan #

NOTE ⚠

  • I have not started backing up anything at the moment, there are just scattered copies etc.
  • This will be an incremental process, but will start soon. (18th Feb'23)
  • In some cases I need to do some prior work, eg. my video files are scrattered all over the internet and different drives. I have to put them together into one place before I even think of backing them up.

After some reading and going through various backupexternal link solutions, I decided that the primary tool to make my backups will be resticexternal link . I initially consideredexternal link borg with rysnc.netexternal link , but using restic lets me use cheaper storageexternal link alternatives and at the time of this writing I am trying to cut costs. I haven’t really looked into tarsnapexternal link but I wanted to.

  • The main strategy I am going to follow is the 3-2-1 strategyexternal link . (3 copies, 2 different media, 1 offsite) + restore tested.
  • I am not backing up emails, DMs etc as I consider them ephemeral and I try to set disappear timer in most of them.

Data inventory #

Name What about it? Priority Backed Up?
Passwords & 2FA passphrases Strengthen master pass. Create regular encrypted export from bitwarden. Backup local pass store. 5/5 πŸ‘Ž
2FA Google Authenticator, no backups nothing, do something. 5/5 πŸ‘Ž
PC Nothing worth backing up here 0/5 πŸ‘Ž
Laptop Installed package list and configurations(dot files). Browser profile+ext. configurations 5/5 πŸ‘Ž
Phone Tasker configuration. App list + configuration 2/5 πŸ‘Ž
Access & Encryption Keys Put SSH and Age private keys somewhere safe, make way for automatic backup of rotated keys 5/5 πŸ‘Ž
Homelab configuration I don’t have the homelab ready now so would not know 0/5 πŸ‘Ž
Public and Private repositories Github+Bitbucket mirrors. Offsite(forked+own+custom repo) backup. 1/5 πŸ‘Ž
eBooks I have a book collection on google drive. Setup automated organization. Then backup. 4/5 πŸ‘Ž
Internet Documents Research papers and other random PDFs. Put them in appropriate place first. Backup. 1/5 πŸ‘Ž
Internet memes&photos&videos Make a media browser/search engineexternal link first for this. Backup everything as application backup afterwards. 0.2/5 πŸ‘Ž
Personal Photos Photos from Google drive/photos 2/5 πŸ‘Ž
Personal Screenshots Screenshots from Google drive 1/5 πŸ‘Ž
Personal Documents Google drive, Physical copies. Put them in appropriate place first. Backup. 3/5 πŸ‘Ž
Personal Social Media Dumps First organize. Then backup. 1/5 πŸ‘Ž

Backup details #

This section will be incrementally populated with details about how I am doing the backups etc. I’ll probably do it in literate programming fashion.

  • Threat model of data loss and disaster recovery is no longer hardware failure: it’s account lock out. So make sure to use replicate stuff to different media/providers.

Compute providers #

Name Remark
Vultrexternal link Heard good things
Exoscaleexternal link One person said good thing about this
Hetznerexternal link Good value for VPS, support, transparent, peering issues (Now as ARM64! cheap :))
Time4VPSexternal link Idk, probably good and cheap
Uberspaceexternal link Unique “shared server” concept. In theory you can use as much ressources as you want but in that case other customers are impacted.
Scalewayexternal link Complaints about support
Oracleexternal link It’s a free tire but lot of complaints about dark patterns. Use it w caution.
Tornado VPSexternal link Idk, probably good and cheap
Linodeexternal link Little pricy but trusy
DigitalOceanexternal link Little pricy but trusy(2)
RackNerdexternal link Black friday yearly deal is juicy
netcupexternal link Old fellow, probably good
SSD Nodesexternal link Cheap stuff but good
OVHexternal link French company, once data center caught fire but otherwise reviews are mixed. Interesting bare metal offerings

Storage providers #

Name Remark
Hetzner storage boxes have not checked but good things heard
Blackblaze B2 moi wants 2 use this for offsite backup

Best practices #

Hardening system #

  • Reverse proxy only accepting domain-name queries instead of the IP.

Observability #

Other Homelabs #

Hardware #

Guides #

Issues #

USB ova IP #

Wayland x Windows KVM #