Steps to Build a Personal Homelab

These are my notes on building a homelab Kubernetes cluster using three Raspberry Pi 4B boards running k3s. The goal was to have a fully GitOps-managed environment where ArgoCD handles all deployments, MetalLB provides load balancer IPs on my local network, and NGINX Ingress routes traffic to services. I also added monitoring with kube-prometheus-stack and TLS certificates via cert-manager.

I'm writing this so I can reproduce the setup from scratch if needed. If you're planning something similar, these notes should save you some debugging time.

Prerequisites

• Basic familiarity with Kubernetes concepts (pods, services, ingress, namespaces)
• Experience with kubectl and Helm charts
• A GitHub account (for the GitOps repo and ArgoCD integration)
• Comfort with SSH and basic Linux administration

Goals

• Flash and configure 3 Raspberry Pi nodes with Ubuntu Server
• Install k3s with 1 master and 2 workers
• Bootstrap ArgoCD and set up the app-of-apps pattern for full GitOps
• Deploy MetalLB for LoadBalancer IPs, NGINX Ingress for routing, and cert-manager for TLS
• Add Grafana and Prometheus monitoring tuned for low-resource nodes
• Verify the full pipeline with a test NGINX app

Hardware

• 3x Raspberry Pi 4B (2GB RAM), 1 master + 2 workers
• 3x 64GB SD cards
• 5-port Gigabit PoE Switch
• Cluster case with cooling fans
• Ethernet cables

Network / IP Plan

Before starting, I planned out the IP allocation for services that need fixed addresses. MetalLB will manage a pool of IPs and assign them to Kubernetes services of type LoadBalancer.

MetalLB pool: 192.168.1.210 - 192.168.1.220

To access these services by name from my local machine, I added entries to /etc/hosts:

192.168.1.210   argocd.local
192.168.1.211   grafana.local
192.168.1.212   nginx.local

Node Config

Each Raspberry Pi gets a static IP and a hostname that makes it easy to identify in kubectl get nodes output.

k3s-master      192.168.1.29
k3s-worker-01   192.168.1.31
k3s-worker-02   192.168.1.32

SSH user: diogo

OS Setup (all nodes)

Flash Ubuntu Server 22.04 LTS 64-bit using Raspberry Pi Imager.

Then on each node, install the basic packages and enable SSH:

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl wget git vim jq openssh-server
sudo systemctl enable ssh && sudo systemctl start ssh
hostname -I  # note the IP
sudo reboot

Enable cgroups

Raspberry Pi nodes need cgroups enabled for Kubernetes to manage container memory limits. Edit /boot/firmware/cmdline.txt and append the following to the end of the existing line (don't add a new line):

cgroup_memory=1 cgroup_enable=memory

Disable swap

Kubernetes requires swap to be disabled. This applies to all nodes:

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
sudo reboot

k3s Install

k3s is a lightweight Kubernetes distribution that works well on ARM devices like the Raspberry Pi. I disabled Traefik and the built-in service load balancer because I'm using NGINX Ingress and MetalLB instead.

Master node

curl -sfL https://get.k3s.io | K3S_TOKEN=<token> sh -s - \
  --write-kubeconfig-mode 644 \
  --disable traefik \
  --disable servicelb

After installation, verify the master is running:

sudo systemctl status k3s
sudo k3s kubectl get nodes

Save the kubeconfig for remote kubectl access from your workstation:

sudo cat /etc/rancher/k3s/k3s.yaml

Note: Copy this file to your local machine's ~/.kube/config and replace the server address 127.0.0.1 with the master node's IP (192.168.1.29).

Worker nodes

On each worker, join the cluster by pointing to the master's API server:

curl -sfL https://get.k3s.io | \
  K3S_URL=https://192.168.1.29:6443 \
  K3S_TOKEN=AS2DG46s1e sh -

Then check from the master that all nodes joined:

sudo k3s kubectl get nodes
# Should show all 3 nodes as Ready

ArgoCD Bootstrap

ArgoCD is the GitOps controller that will manage every deployment in the cluster. The idea is to install it manually once, then let it manage itself (and everything else) from a Git repo.

# Install ArgoCD using the official manifest
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Temporary access via port-forward (before ingress is up)
sudo k3s kubectl port-forward svc/argocd-server -n argocd 8080:443

# Get initial admin password
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d; echo

# Login via CLI
argocd login localhost:8080 --username admin --password <password> --insecure

Once MetalLB and NGINX Ingress are live, you can create an Ingress resource for ArgoCD and stop using port-forward.

Repo Structure

The GitOps repo is organized so that ArgoCD Application manifests are separate from the actual Kubernetes configs they deploy:

homelab/
├── apps/
│   ├── argocd/
│   ├── metallb/
│   ├── monitoring/
│   ├── cert-manager/
│   └── nginx/
├── argocd-apps/
├── LICENSE
└── README.md

argocd-apps/ contains ArgoCD Application manifests that tell ArgoCD what to deploy and where to find it. apps/ contains the actual Kubernetes configs (Helm values, raw manifests, Kustomize files) that those Applications point to. This separation keeps things clean: you change a config in apps/, ArgoCD detects the diff and syncs it.

App of Apps: root-app

The app-of-apps pattern is an ArgoCD approach where a single "root" Application watches a directory of other Application manifests. When you add a new Application YAML to that directory, ArgoCD picks it up automatically. This means you only need to kubectl apply one resource manually, ever.

# argocd-apps/root-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: 'https://github.com/diogofrmota/homelab.git'
    path: argocd-apps
    targetRevision: main
    directory:
      recurse: true
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      selfHeal: true
      prune: true

Apply this once manually, and everything else is managed by ArgoCD from this point:

kubectl apply -f argocd-apps/root-app.yaml

Sync Wave Order

ArgoCD sync waves control the order in which Applications are deployed. This matters because some components depend on others. For example, NGINX Ingress needs MetalLB to be running so it can get a LoadBalancer IP, and ClusterIssuers need cert-manager's CRDs to exist first.

Set the wave via an annotation on each Application:

annotations:
  argocd.argoproj.io/sync-wave: "-3"

MetalLB

MetalLB gives you LoadBalancer-type services in a bare-metal cluster. Without it, services of type LoadBalancer would stay in Pending state forever because there's no cloud provider to provision an IP.

Note: I initially tried using helm.cattle.io/v1 HelmChart (the k3s-native CRD) inside ArgoCD. This doesn't work because ArgoCD can't reconcile those resources. The fix is to point ArgoCD directly at the MetalLB Helm repo.

ArgoCD Application

# argocd-apps/metallb-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: metallb
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-3"
spec:
  project: default
  source:
    repoURL: https://metallb.github.io/metallb
    chart: metallb
    targetRevision: 0.13.12
  destination:
    server: https://kubernetes.default.svc
    namespace: metallb-system
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
      - CreateNamespace=true

IP Pool and L2 Advertisement

The IPAddressPool and L2Advertisement go in a separate ArgoCD app that points to apps/metallb/, deployed in a later sync wave so MetalLB's CRDs are ready:

# apps/metallb/metallb-config.yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  namespace: metallb-system
  name: home-pool
spec:
  addresses:
  - 192.168.1.210-192.168.1.220
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  namespace: metallb-system
  name: home-adv

NGINX Ingress

The NGINX Ingress Controller acts as the single entry point for HTTP/HTTPS traffic into the cluster. It gets a LoadBalancer IP from MetalLB and routes requests to backend services based on Ingress rules.

Setting externalTrafficPolicy: Local preserves the client's real IP address in the X-Forwarded-For header. Without it, you'd only see the node's internal IP.

# argocd-apps/ingress-nginx-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ingress-nginx
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  project: default
  source:
    repoURL: https://kubernetes.github.io/ingress-nginx
    chart: ingress-nginx
    targetRevision: 4.13.2
    helm:
      values: |
        controller:
          service:
            type: LoadBalancer
            externalTrafficPolicy: Local
          publishService:
            enabled: true
  destination:
    server: https://kubernetes.default.svc
    namespace: ingress-nginx
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

cert-manager

cert-manager automates TLS certificate provisioning. I'm using it with Let's Encrypt so that Ingress resources can request certificates automatically via annotations.

This is a two-step setup: first deploy cert-manager itself (wave -2), then apply ClusterIssuers (wave 0) after the CRDs are ready. If you try to apply them at the same time, the ClusterIssuer will fail because the cert-manager.io/v1 API doesn't exist yet.

cert-manager Application (wave -2)

# argocd-apps/cert-manager.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cert-manager
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-2"
spec:
  project: default
  source:
    repoURL: https://charts.jetstack.io
    chart: cert-manager
    targetRevision: v1.18.2
    helm:
      values: |
        installCRDs: true
  destination:
    server: https://kubernetes.default.svc
    namespace: cert-manager
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
      - CreateNamespace=true

ClusterIssuer (wave 0)

# apps/cert-manager/cluster-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    email: istarkillerpt@gmail.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - http01:
        ingress:
          class: nginx

Tip: Also create a letsencrypt-staging issuer pointed at https://acme-staging-v02.api.letsencrypt.org/directory for testing without hitting Let's Encrypt rate limits.

ArgoCD Service (MetalLB fixed IP)

Once MetalLB is running, give ArgoCD a fixed LoadBalancer IP so you can access it reliably at argocd.local instead of relying on port-forward:

# apps/argocd/argocd-server-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: argocd-server
  namespace: argocd
spec:
  selector:
    app.kubernetes.io/name: argocd-server
  ports:
  - name: http
    port: 80
    targetPort: 8080
  - name: https
    port: 443
    targetPort: 8080
  type: LoadBalancer
  loadBalancerIP: 192.168.1.210

Monitoring (kube-prometheus-stack)

The kube-prometheus-stack bundles Prometheus, Grafana, and Alertmanager. Grafana gets a LoadBalancer IP so I can access dashboards at grafana.local. Prometheus and Alertmanager stay as ClusterIP since I only access them through Grafana or kubectl port-forward.

The Raspberry Pi 4B only has 2GB RAM, so resource tuning is important. I also disabled the etcd, kubeScheduler, and kubeControllerManager alert rules because k3s bundles those components into a single process and doesn't expose their metrics separately. Without disabling them, you'd get constant firing alerts for targets that will never be reachable.

Key settings in the Helm values:

grafana:
  adminPassword: "admin"
  service:
    type: LoadBalancer
    port: 80
prometheus:
  service:
    type: ClusterIP
  prometheusSpec:
    retention: 5d
    retentionSize: "2GB"
defaultRules:
  rules:
    etcd: false
    kubeScheduler: false
    kubeControllerManager: false

Note: Change the default Grafana admin password after first login.

Test Nginx App

A simple NGINX deployment to verify the full pipeline works end to end: ArgoCD syncs the manifests, MetalLB assigns an IP to the ingress controller, NGINX Ingress routes traffic based on the host header, and cert-manager provisions a TLS certificate.

The NGINX service itself is ClusterIP (not LoadBalancer) because it sits behind the NGINX Ingress Controller. The ingress controller is the one with the LoadBalancer IP.

# apps/nginx/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
  namespace: nginx
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
  - hosts:
    - nginx.local
    secretName: nginx-tls-secret
  rules:
  - host: nginx.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-service
            port:
              number: 80

If everything is working, visiting https://nginx.local should show the default NGINX welcome page with a valid TLS certificate.

Issues I Hit

Here are the problems I ran into and how I fixed them. Documenting these so I don't debug the same things twice.

MetalLB failing via ArgoCD. I was using helm.cattle.io/v1 HelmChart, which is a k3s-internal resource. ArgoCD can't reconcile those properly because it doesn't understand the k3s Helm controller CRD. The fix was to point ArgoCD directly at the MetalLB Helm repo instead of wrapping it in a HelmChart resource.

Nginx app namespace mismatch. The deployment was going to the default namespace but the service and ingress were in nginx. Everything in apps/nginx/ needs to use namespace: nginx consistently. The Kustomization should also include the namespace manifest so it gets created first.

cert-manager CRDs not ready. ClusterIssuers were being applied before cert-manager finished installing its CRDs. Fixed with sync waves: cert-manager at wave -2, ClusterIssuers at wave 0.

Grafana serve_from_sub_path breaking access. I had serve_from_sub_path set to true initially, which broke direct IP access. Set it to false since Grafana is served at the root path, not behind a sub-path like /grafana.

Useful Commands

# Check all nodes
sudo k3s kubectl get nodes

# Watch pods in a namespace
kubectl get pods -n metallb-system -w

# ArgoCD port-forward (before ingress is up)
kubectl port-forward svc/argocd-server -n argocd 8080:443

# Get ArgoCD password
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d

# Check MetalLB got an IP assigned
kubectl get svc -A | grep LoadBalancer

# Check ingress
kubectl get ingress -A

# Check certificate status
kubectl get certificates -A

# ArgoCD app sync status
argocd app list

Conclusion

Here's a summary of what was set up and the role of each component:

The whole cluster is now managed via Git. To add a new service, I create a manifest in apps/, add an ArgoCD Application in argocd-apps/, push to main, and ArgoCD handles the rest. No more kubectl apply after the initial bootstrap.