Homelab

NOTE

  • Everything about this page and my homelab and selfhosted tools are very much WIP ✨ 🚧
  • geekodour/workshop external link : Keeping track of homelab and other workshop things.

TIP:

  • I did some internet comparison/study when preparing this selfhosting setup. I always take notes when studying anything, all my notes related to selfhosting can be found here in my wiki external link .
  • Check “tags” and “Links to this note” in the wiki page.

What’s the shape? #

The homelab is one of my passion projects. It’s not even birthed yet and there’s so much to do and experiment with. If things go alright, I’ll probably have my dream homelab in 1-2 years. I want it external link well external link documented external link , well maintained external link , reproducible, high quality and of-course do all the things I want it to do. Why? Vendor independence, privacy, costs, peace of mind, fun. pro: you can fix the problem, con: you have to fix the problem.

Properties #

  • Should have solid observability. This is also my playground for things.
  • Upto-date and mostly automated
  • Good and live documentation and should be easily debug-able
  • It’s not like I want to selfhost everything, if there are external hosted services which satisfy my needs I would skip selfhosting. I want this to be seamless, controlled and do not want to spend hours on debugging any external link issue. Same goes for hardware.
  • I am not building enterprise network here, I can have fun so stupid stuff is sort of allowed

Components components #

NOTE:

  • I have not built the homelab, this is first draft plan.
  • These are not actual boundaries, I am just laying out an abstract layout here for me to build on.
  • Things might be not be technically correct(I might be placing things in the wrong places of the stack as-well)
  • I have not thought all of this properly but merely dumping ideas here. this is NOT the topology
Name Remark
Goku Bastion server, External VPS
Dobbies Any service, many service, whatever random thing. These will run it.
SRK Anything media management goes here, runs locallty
Warehouse multi purpose storage server(s)
Cloud ZEPEEYOU AI experiments helper
Rasta Throwaway servers
Piccolo Trusted, Persistent Good ol webserver
daCNC My phone

Goku #

Sort of a bastion host external link . Idea is to have access to all my services from one place. Eg. I should be able to ssh from my phone to this machine and manage things even if I am away from my laptop. It should have the tools installed I need in a dev/sysadmin machine.

  • Location: VPS
  • Threat Model: Assume that it can be compromised and reduce attack surface accordingly.
  • Possible stuff here: centralized logging external link , centralized observability center, orchestrator center, Teleport external link w 2FA
  • Concerns: The usecase and motive of this component sort of contradicts. I am expecting this to be target but at the same time making this the most powerful and yet SOP in a way? Need to think.

Dobbies #

Local RPi(s)/Small computers/NUCs, can name them dobby-1, dobby-2 etc. Host small tools or whatever that I want to use locally or maybe expose some to the public internet as-well.

  • Location: Local
  • Possible stuff here: These will basically do anything. ArchiveWarrior stuff, bespoke scripts, see my secondary toolchest for complete list of tools that are already selfhostable/can be made selfhostable to fit my needs.

SRK #

The media server external link , connected external link to a NAS most likely. . I wanted to be local first, requiring internet to reach my media external link does not make external link sense but I probably would want to have public access to this in-case.

Warehouse #

Some kind of storage server/multiple servers. I have to explore this, zfs, btrfs etc. This will store archives, media files etc. This is not the backup, it’ll be done separately.

Cloud ZEPEEYOU #

To carry out AI experiments. Not worrying about this much rn as this will be specific to usecase but definitely want this to be billed on usage lol.

Rasta #

  • Location: VPS, needs to be ephemeral

A test server / dummy that i can trash and recreate anytime, installs my necessary tools automatically on creation etc.

Piccolo #

Trusted, Persistent Good ol webserver. This will have a solid reverse proxy in place so that I spin up random APIs/Websites for public quickly.

daCNC #

This is my phone. This is more like a remote control for things and I’ve set some phone specific tasker profiles which are super useful. Eg. Taking picture and Uploading it to my Google Drive via SMS trigger etc.

Orchestration #

We have few options.

Name Remark
Nomad Based on external link what external link I read external link it seems external link like this external link would be suitable for my homelab
Kubernetes There are things like k3s external link
Promox Runs LXC and VMs, Min 3 nodes needed

What keeps everything together? I have not decided yet but guess it’ll be a mix of terraform and ansible external link .

I think I’ll go with Nomad like I mentioned.

Networking #

Goals

  • I should be able to access(ssh) certain private devices/services securely from the public internet.
  • I should be able to access certain public services securely from the public internet. (eg. fileserver, mediaserver etc)
  • Local devices should be able to talk to each other, preferably put local devices into a different VLAN and internet facing stuff into a DMZ.
  • Should have a proper way to access geoblocked content
  • Security, have not thought about my threat model properly.

Anti-Goals

  • Not trying to be anonymous here
  • Even though I want to build a mesh network, in this iteration it’s not the goal. At most I might be use tailscale or something similar.

VPN #

  • Mesh VPN setup
    • Goal: Allow my devices to talk to each external link other
    • Something like Tailscale is looking juicy here
  • Road warrior setup (VPN VPS)
    • Goal: Something that allows me to access my devices at home when I am out.
  • Encrypted Traffic + Hide source IP(geo) setup
    • Goal: Not anonymity but more of privacy and bypassing censorship. Eg. When using insecure public wifi or anything else that fits.
    • I can selfhost this but with that I cannot keep switching countries etc. So might be good idea to go with something like Mullvad VPN external link
  • Tunnels
    • Goal: Expose public only services quickly, give temporary access to something that I am running locally etc.
  • TODO: check dns leaks external link

Proxy #

Forward Proxy #

I do not really feel the need of a forward proxy as such at the moment. But I can see one usecase: Censorship bypass. Setting up shadowsocks, vray and cloak along w tor proxy(whatever combination makes sense for the usecase) might be a good idea. Because you don’t need them until you need them :)

Reverse Proxy #

I can use these things to do load-balancing/ssl termination/reverse proxy/protocol demultiplexing/HA external link /failover/caching/rate-limiting etc. Here’s a more external link complete list external link . After some comparison, I think i’ll be going with either Traefik or Caddy.

Router #

  • We have the options of OpenWRT and OPNSense here. We can mix and match, will think of exact topology later.
  • Point web services logs to fail2ban and let it handle rate-limiting etc.
  • For extra points can check Crowdsec

DNS #

This is one bad boi. I probably just want to run local resolver. Maybe an authoritative server replicated to secondaries later. But for now, I plan PiHole/Blocky+Unbound.

  • Once we have a reverse-proxy setup, you can have your local DNS server point to your reverse proxy for whatever domain. eg. *.home. Also see what domain name to use for your home network? home.arpa external link
  • Some people recommend doing split-horizon DNS along with reverse-proxy if running multiple services, I don’t see a point rn but maybe I’ll later.

Local Network #

VLANs and Subnets #

  • Reason: It’s nice to separate things with vlans and firewall rules + IoT devices are known to be insecure external link . (Sort of an overkill but who cares)
  • Subnets
    • VLAN 1 for home devices LAN
    • VLAN 2 for trusted IoT which cannot run VPN client, access to the Internet allowed
    • VLAN 3 for isolated (untrusted) IoT devices
    • VLAN 4 for DMZ for publicly hosted services etc
  • VPN runs on VLAN1
  • What comes and goes out of these VLANS to be configured via firewalls
  • Check if we’ll need a managed switch or OpenWRT will cut it external link

DMZ #

  • Reason: Because I plan to host public facing services it makes sense to have a DMZ.
  • Objective is to provide firewall capabilities between hosts in the DMZ and hosts on the internal network.

Monitoring the network #

I haven’t explored this properly, so just link dumping.

Backup Plan #

NOTE ⚠

  • I have not started backing up anything at the moment, there are just scattered copies etc.
  • This will be an incremental process, but will start soon. (18th Feb'23)
  • In some cases I need to do some prior work, eg. my video files are scrattered all over the internet and different drives. I have to put them together into one place before I even think of backing them up.

After some reading and going through various backup external link solutions, I decided that the primary tool to make my backups will be restic external link . I initially considered external link borg with rysnc.net external link , but using restic lets me use cheaper storage external link alternatives and at the time of this writing I am trying to cut costs. I haven’t really looked into tarsnap external link but I wanted to.

  • The main strategy I am going to follow is the 3-2-1 strategy external link . (3 copies, 2 different media, 1 offsite) + restore tested.
  • I am not backing up emails, DMs etc as I consider them ephemeral and I try to set disappear timer in most of them.

Data inventory #

Name What about it? Priority Backed Up?
Passwords & 2FA passphrases Strengthen master pass. Create regular encrypted export from bitwarden. Backup local pass store. 5/5 👎
2FA Google Authenticator, no backups nothing, do something. 5/5 👎
PC Nothing worth backing up here 0/5 👎
Laptop Installed package list and configurations(dot files). Browser profile+ext. configurations 5/5 👎
Phone Tasker configuration. App list + configuration 2/5 👎
Access & Encryption Keys Put SSH and Age private keys somewhere safe, make way for automatic backup of rotated keys 5/5 👎
Homelab configuration I don’t have the homelab ready now so would not know 0/5 👎
Public and Private repositories Github+Bitbucket mirrors. Offsite(forked+own+custom repo) backup. 1/5 👎
eBooks I have a book collection on google drive. Setup automated organization. Then backup. 4/5 👎
Internet Documents Research papers and other random PDFs. Put them in appropriate place first. Backup. 1/5 👎
Internet memes&photos&videos Make a media browser/search engine external link first for this. Backup everything as application backup afterwards. 0.2/5 👎
Personal Photos Photos from Google drive/photos 2/5 👎
Personal Screenshots Screenshots from Google drive 1/5 👎
Personal Documents Google drive, Physical copies. Put them in appropriate place first. Backup. 3/5 👎
Personal Social Media Dumps First organize. Then backup. 1/5 👎

Backup details #

This section will be incrementally populated with details about how I am doing the backups etc. I’ll probably do it in literate programming fashion.

  • Threat model of data loss and disaster recovery is no longer hardware failure: it’s account lock out. So make sure to use replicate stuff to different media/providers.

Compute providers #

Name Remark
Vultr external link Heard good things
Exoscale external link One person said good thing about this
Hetzner external link Good value for VPS, support, transparent, peering issues (Now as ARM64! cheap :))
Time4VPS external link Idk, probably good and cheap
Uberspace external link Unique “shared server” concept. In theory you can use as much ressources as you want but in that case other customers are impacted.
Scaleway external link Complaints about support
Oracle external link It’s a free tire but lot of complaints about dark patterns. Use it w caution.
Tornado VPS external link Idk, probably good and cheap
Linode external link Little pricy but trusy
DigitalOcean external link Little pricy but trusy(2)
RackNerd external link Black friday yearly deal is juicy
netcup external link Old fellow, probably good
SSD Nodes external link Cheap stuff but good
OVH external link French company, once data center caught fire but otherwise reviews are mixed. Interesting bare metal offerings

Other server resources #

Storage providers #

Name Remark
Hetzner storage boxes have not checked but good things heard
Blackblaze B2 moi wants 2 use this for offsite backup

Storage resources #

Best practices #

Hardening system #

  • Reverse proxy only accepting domain-name queries instead of the IP.

Environment #

Observability #

Security #

Other Homelabs #

Homelab as a service kind of projects #

Tips from others #

Aesthetics #

Network #

Hardware #

Products #

Guides #

Issues #

USB ova IP #

Wayland x Windows KVM (Input switch / desktop switch) #

Sleep #