Development Journey Update

Development Journey Update - End of 2025

As we approach the end of 2025, it’s a good time to reflect on the achievements of the year.

I’ve not been blogging much recently; this was a combination of a change of job and focusing on some personal projects. In this post, I’ll summarise some key homelab projects I’ve undertaken this year.

Move to Ubiquiti

In the physical changes to my home lab, one of the biggest changes was the transition out of consumer-grade networking equipment (Asus Gaming Router) to sub-enterprise-grade Ubiquiti Unifi equipment.

Why

So like most home users, I pretty much had a flat network design. All my devices existed in the same /24 subnet, although I utilised the guest network feature to emulate some segmentation, my previous router only supported 4 guest networks so I was limited in what I could do, that and you didn’t have much control over the routing between the main and guest networks.

Now I could have easily upgraded to a higher-end consumer router that allowed network segmentation and VLANs. However, I also was finding that the amount of traffic the k3s cluster was generating was beginning to become too much for the small ARM processor in the router. I was constantly finding that the router’s CPU and memory were being maxed out.

What

So what is Ubiquiti Unifi? Ubiquiti is a networking hardware manufacturer that produces a range of networking equipment from access points, switches, routers and security gateways. Their Unifi product line is targeted at small to medium businesses and prosumers, offering a range of features and capabilities that are typically found in enterprise-grade networking equipment.

For my setup I started with a Dream Machine Pro (UDM-Pro) which is an all-in-one router, switch and security gateway, a U7 Pro access point for WiFi coverage and a 24-port USW-24 switch for wired connectivity.

The aim in having an L3 switch with the USW-24 was to offload some of the routing and switching tasks from the UDM-Pro. With the heavier networking load generators like my TrueNAS server and media server connected directly to the switch, this allowed for better performance and reduced latency. This was short-sighted though, as in hindsight I should have gone for the PoE version of the switch to power my Raspberry Pi K3s nodes as by this point each of them was running with a PoE hat to avoid having multiple power adapters cluttering up my setup.

How

As mentioned before, I was previously using a consumer grade Asus router. One of my key goals here was to segment the network putting different types of devices in their own VLANs. I settled on the following structure:

Name	Description
default	Ubiquiti devices / management
home lab	All homelab servers and devices
IOT	Smart Home devices and untrusted devices
guest	Guest devices and visitors
work from home	Work devices
personal	phones, tablets and personal laptops

Once the physical hardware was setup and connected, I then proceeded to configure the network. Now, Ubiquiti does not segment cross VLANs by default, so I had to ensure that trusted subnets could not be accessed from untrusted devices. It’s remarkable when you study the network traffic of some IoT devices how nosey they can be. I regularly found my Amazon Echo devices snooping for other devices on the network. To mitigate this, I created firewall rules to block traffic between the IOT VLAN and all other VLANs except for the default management VLAN.

I made an exception for my Home Assistant so that IOT devices can still be seen and controlled, but all other traffic is blocked.

Finally, I also wanted to implement regional blocking. Many IoT devices seem to communicate to dubious locations around the world. I’d commonly see the Philips Hue hub trying to communicate with servers in China, and Dreame robotic vacuum devices trying to reach servers in Russia. To mitigate this, I created firewall rules to block traffic to and from certain regions. I’d rather have to debug and create exceptions for devices that need to communicate internationally only to find my data being part of a data breach or sold.

graph TD
    subgraph "Physical Infrastructure"
        Internet["Internet"]
        UDMPro["UDM-Pro<br/>Router & Security Gateway"]
        USW24["USW-24<br/>L3 Switch"]
        U7Pro["U7 Pro<br/>WiFi Access Point"]

        Internet --> UDMPro
        UDMPro --> USW24
        UDMPro --> U7Pro
    end

    subgraph "Network Segmentation - VLANs"
        DefaultVLAN["Default VLAN<br/>Ubiquiti Management<br/>192.168.1.x/24"]
        HomeLabVLAN["Home Lab VLAN<br/>Servers & Infrastructure<br/>192.168.10.x/24"]
        IOTVLAN["IoT VLAN<br/>Smart Home Devices<br/>192.168.20.x/24"]
        GuestVLAN["Guest VLAN<br/>Visitor Access<br/>192.168.30.x/24"]
        WFHVLAN["Work From Home VLAN<br/>Work Devices<br/>192.168.40.x/24"]
        PersonalVLAN["Personal VLAN<br/>Phones & Laptops<br/>192.168.110.x/24"]
    end

    subgraph "Connected Devices"
        TrueNAS["TrueNAS Server"]
        MediaServer["Media Server"]
        K3sNodes["K3s Raspberry Pi Cluster"]
        IoTDevices["Smart Home Devices"]
        WorkDevices["Work Laptops"]
        PersonalDevices["Phones & Tablets"]
        GuestDevices["Guest Devices"]
    end

    %% VLAN routing through UDM-Pro
    UDMPro -.-> DefaultVLAN
    UDMPro -.-> HomeLabVLAN
    UDMPro -.-> IOTVLAN
    UDMPro -.-> GuestVLAN
    UDMPro -.-> WFHVLAN
    UDMPro -.-> PersonalVLAN

    %% Device connections
    USW24 --> TrueNAS
    USW24 --> MediaServer
    USW24 --> K3sNodes
    U7Pro -.-> IoTDevices
    U7Pro -.-> WorkDevices
    U7Pro -.-> PersonalDevices
    U7Pro -.-> GuestDevices

    %% VLAN assignments
    TrueNAS -.-> HomeLabVLAN
    MediaServer -.-> HomeLabVLAN
    K3sNodes -.-> HomeLabVLAN
    IoTDevices -.-> IOTVLAN
    WorkDevices -.-> WFHVLAN
    PersonalDevices -.-> PersonalVLAN
    GuestDevices -.-> GuestVLAN

    %% Styling with Ubiquiti colors
    style Internet fill:#2a2a2a,stroke:#61dafb,stroke-width:2px,color:#e0e0e0
    style UDMPro fill:#2a2a2a,stroke:#00ff88,stroke-width:3px,color:#e0e0e0
    style USW24 fill:#2a2a2a,stroke:#00ff88,stroke-width:2px,color:#e0e0e0
    style U7Pro fill:#2a2a2a,stroke:#00ff88,stroke-width:2px,color:#e0e0e0

    %% VLAN styling
    style DefaultVLAN fill:#1a1a1a,stroke:#ffcc00,stroke-width:1px,color:#e0e0e0
    style HomeLabVLAN fill:#1a1a1a,stroke:#61dafb,stroke-width:1px,color:#e0e0e0
    style IOTVLAN fill:#1a1a1a,stroke:#ff6b9d,stroke-width:1px,color:#e0e0e0
    style GuestVLAN fill:#1a1a1a,stroke:#ff6b6b,stroke-width:1px,color:#e0e0e0
    style WFHVLAN fill:#1a1a1a,stroke:#a855f7,stroke-width:1px,color:#e0e0e0
    style PersonalVLAN fill:#1a1a1a,stroke:#00ff88,stroke-width:1px,color:#e0e0e0

    %% Device styling
    style TrueNAS fill:#1a1a1a,stroke:#444444,stroke-width:1px,color:#a0a0a0
    style MediaServer fill:#1a1a1a,stroke:#444444,stroke-width:1px,color:#a0a0a0
    style K3sNodes fill:#1a1a1a,stroke:#444444,stroke-width:1px,color:#a0a0a0
    style IoTDevices fill:#1a1a1a,stroke:#444444,stroke-width:1px,color:#a0a0a0
    style WorkDevices fill:#1a1a1a,stroke:#444444,stroke-width:1px,color:#a0a0a0
    style PersonalDevices fill:#1a1a1a,stroke:#444444,stroke-width:1px,color:#a0a0a0
    style GuestDevices fill:#1a1a1a,stroke:#444444,stroke-width:1px,color:#a0a0a0

Network Segmentation

Building a 48TB ZFS Network Attached Storage

What

For the last couple of years I used a 4TB HDD inside of my old gaming PC (now a Proxmox server) and a TrueNAS VM to provide my network storage needs. However, as my media collection grew and I started to worry about the reliability due to the age of the drive, which was 5 years old at this point, I decided to build a dedicated NAS server.

Why

The main requirements for the new NAS server were: - Power Efficiency: Living in the UK we’re charged quite high rates for electricity, so having a power-hungry server running 24/7 was not ideal. - Storage Capacity: I want breadth of storage to grow my media collection and to be able to run backups of my homelab servers and host MinIO object storage. - Reliability: I wanted a solution that would provide data redundancy and protection against drive failures. - Performance: While not needing enterprise level performance, I wanted to be able to maximise my gigabit network speeds for file transfers and media streaming.

I planned to do more with my monitoring stack this year so I wanted to ensure that there was enough storage to capture long-term logs and metrics. I had been working a lot this year professionally on improving the efficiency of log retention and metric cardinality, so I wanted to ensure my home lab practiced what I preached. There’s also performance considerations with tin that you worry more about then in the cloud where you can just scale up resources.

How

I settled on the following:

AMD Ryzen 3 4300G CPU
64GB DDR4 RAM (initially 16GB but upgraded later in the year after realising my mistake)
4 x 12TB Seagate IronWolf Pro HDDs (RAIDZ2 for 48TB usable storage)
10Gbe Network Card (Asus XG-C100C) for future proofing
TrueNAS Scale as the operating system

I settled on Ryzen 3 4300G as it offered a good balance of performance and power efficiency as it only drew around 65W TDP. The integrated graphics also meant I didn’t need a dedicated GPU, further reducing power consumption and cost. The 64GB of RAM was chosen to ensure that ZFS had enough memory for caching and to support future expansion. ZFS is known to benefit from ample RAM, especially when dealing with large datasets and multiple concurrent accesses. The 4 x 12TB Seagate IronWolf Pro HDDs were selected for their reliability and performance. Using RAIDZ2 provided a good balance between storage efficiency and data protection, allowing for two drive failures without data loss.

Major Change! - Homelab APM Setup

What

Like you’d expect from a workplace environment, monitoring is key for getting insight on how systems are performing, their availability, and how they’re being used. However, I don’t have tens of thousands to spare on commercial APM solutions like Datadog, New Relic or Dynatrace. Now many out there probably already run Prometheus and Grafana on their homelabs and for a few hundred metrics and a handful of hosts a simple Prometheus on bare metal, VM or Kubernetes setup is perfectly fine. But, as my home lab is much more then a few development projects now, it actually captures measurements as small as biometrics from my smart watch to the power consumption of my entire house.

So I needed something more scalable, reliable and flexible. After researching a few options I settled on the following stack:

Grafana (Hosted/K3S) This is the same as last year, no change here.
Prometheus (Hosted/K3S) Again, same as last year, slightly more storage and Im now using Raspberry Pi 5s with a NVMe SSD for storage.
Thanos (Hosted/K3S) This is the big change, Thanos sits on top of Prometheus and provides long term storage, downsampling and global querying across multiple Prometheus instances. This is then stored in an S3 compatible object store (MinIO in my case).
Loki (Hosted/K3S) Update on the previous year, I now capture logs with a combination of Loki and Alloy. Alloy ships system logs and metrics to prometheus and Loki. Loki is then configured to store the logs in MinIO.

This setup was more focused on covering the big three pillars: metrics, logs, and traces.

Why

Mainly “because I can”, as I wanted to do more deep diving into observability and monitoring. I’ve always engaged with this area of work and even my initial blog platform I use to use a self hosted analytics platform (Matomo) to track usage. I wanted a setup that went beyond just simple metrics of user visits. I wanted a system that could give me insight into my own life.

As we wander about with our smartphones and watches, living in our smart homes with IoT devices and driving our Google-powered cars, we generate a lot of data. All of which we’re handing off to big tech companies to monetise. I wanted to take back control of my own data and use it give me insight into my own life. For example, I now track:

Power consumption of my entire house (via a smart meter and Home Assistant)
Temperature, Humidity and Air Quality of each room in my house (via multiple sensors and Home Assistant)
Biometrics from my smart watch (Heart Rate, Sleep, Activity etc)
Exercise data from my smart watch and Peleton bike.
Car metrics from my electric car, on milage, location, system state, battery power, range. I even have automations to check if the door or windows have been left unlocked. (via Home Assistant and Volvo API)
Network usage of all devices in my home (via Ubiquiti Unifi)
CPU, Memory, Disk and Network usage of all my homelab servers and VMs/Containers

How

In the years gone by I’ve had the habit of wandering between different technologies and platforms. One year I’ll do everything in VMs in a hypervisor, I will then change the hypervisor then decided to containerise everything then move to Kubernetes. I would then flip-flop between which technology Im using: ansible, terraform, helm thats even before deciding on which application stack to use.

This year I decided to all-in on Kubernetes (K3S) as my main platform for everything. Making only minor changes to the language and framework based on which is best suited for the task at hand.

Once I’d locked in on the platform technologies, the rest wrote itself in a few months. I started with the creation of a new Helm chart called monitoring that would deploy all the components needed for the monitoring stack. This included:

Prometheus
Thanos
Loki
Grafana
Alertmanager

I expanded the existing prometheus-operator helm chart (for note, it does have thanos support however it missed some key features I needed) to include thanos sidecars, store gateway and querier components.

The following diagram shows the architecture of my monitoring stack:

graph TD
    subgraph "TrueNAS (External)"
        Minio[MinIO S3]
        Minio -- "Bucket: prometheus-data" --> ThanosData[(Thanos Data)]
        Minio -- "Bucket: loki-chunks" --> LokiData[(Loki Chunks)]
        Minio -- "Bucket: loki-ruler" --> LokiRulerData[(Loki Rules)]
    end

    subgraph K3S Cluster
        Endpoint[K3s External Endpoint] <--> Minio

        subgraph "Namespace: monitoring"
            Grafana(Grafana UI)
            Alertmanager(Alertmanager)

            subgraph "Metrics Stack (Thanos/Prometheus)"
                PromOp(Prometheus Operator) -.-> Prom
                Prom(Prometheus + Thanos Sidecar)
                TQuery(Thanos Query)
                TStore(Thanos Store Gateway)
                TCompact(Thanos Compactor)
                TRuler(Thanos Ruler)

                Prom -- "Uploads Blocks" --> Endpoint
                TStore -- "Reads History" --> Endpoint
                TCompact -- "Compacts & Downsamples" --> Endpoint
                TRuler -- "Stores Rule Results" --> Endpoint

                TQuery -- "Queries Live" --> Prom
                TQuery -- "Queries History" --> TStore
                TQuery -- "Queries Rules" --> TRuler
            end

            subgraph "Logs Stack (Loki)"
                LokiLB[MetalLB Gateway\n192.168.55.10]
                LDistributor(Loki Distributor)
                LIngester(Loki Ingester)
                LQuery(Loki Querier)
                LQueryFE(Loki Query Frontend)
                LIndexGW(Loki Index Gateway)
                LCompactor(Loki Compactor)
                LRuler(Loki Ruler)
                LChunkCache[Chunks Cache]

                LokiLB --> LDistributor
                LDistributor --> LIngester
                LIngester -- "Flushes Chunks" --> Endpoint

                LQuery --> LQueryFE
                LQueryFE --> LIndexGW
                LQueryFE --> LChunkCache
                LIndexGW -- "Reads Indices" --> Endpoint
                LChunkCache -- "Caches Chunks" --> Endpoint

                LCompactor -- "Retention & Compaction" --> Endpoint
                LRuler -- "Reads Logs" --> LQuery
                LRuler -- "Remote Write Metrics" --> Prom
            end

            Alloy(Alloy Collector)

            Prom -- "Scrapes Metrics" --> Alloy
            Alloy -- "Pushes Logs" --> LokiLB

            Grafana -- "Data Source: Metrics" --> TQuery
            Grafana -- "Data Source: Logs" --> LQuery

            LRuler -- "Sends Alerts" --> Alertmanager
            Prom -- "Sends Alerts" --> Alertmanager
        end

        K3sNodes(K3s Nodes & Apps)
        ExternalSystems(External Systems\nHome Assistant VM)
        Alloy -- "Collects Telemetry" --> K3sNodes
        ExternalSystems -- "Pushes Logs" --> LokiLB
    end

    %% Retro Terminal Color Scheme
    %% Storage - Magenta/Pink accent
    style Minio fill:#2a2a2a,stroke:#ff6b9d,stroke-width:2px,color:#e0e0e0
    style ThanosData fill:#1a1a1a,stroke:#ff6b9d,stroke-width:1px,color:#e0e0e0
    style LokiData fill:#1a1a1a,stroke:#ff6b9d,stroke-width:1px,color:#e0e0e0
    style LokiRulerData fill:#1a1a1a,stroke:#ff6b9d,stroke-width:1px,color:#e0e0e0

    %% Grafana - Amber/Orange (brand color)
    style Grafana fill:#2a2a2a,stroke:#ffcc00,stroke-width:3px,color:#e0e0e0

    %% Alertmanager - Red accent
    style Alertmanager fill:#2a2a2a,stroke:#ff6b6b,stroke-width:2px,color:#e0e0e0

    %% Prometheus/Thanos Stack - Cyan
    style PromOp fill:#1a1a1a,stroke:#61dafb,stroke-width:1px,color:#e0e0e0
    style Prom fill:#2a2a2a,stroke:#61dafb,stroke-width:2px,color:#e0e0e0
    style TQuery fill:#2a2a2a,stroke:#61dafb,stroke-width:2px,color:#e0e0e0
    style TStore fill:#1a1a1a,stroke:#61dafb,stroke-width:1px,color:#e0e0e0
    style TCompact fill:#1a1a1a,stroke:#61dafb,stroke-width:1px,color:#e0e0e0
    style TRuler fill:#1a1a1a,stroke:#61dafb,stroke-width:1px,color:#e0e0e0

    %% Loki Stack - Green
    style LokiLB fill:#2a2a2a,stroke:#00ff88,stroke-width:2px,color:#e0e0e0
    style LDistributor fill:#1a1a1a,stroke:#00ff88,stroke-width:1px,color:#e0e0e0
    style LIngester fill:#1a1a1a,stroke:#00ff88,stroke-width:1px,color:#e0e0e0
    style LQuery fill:#2a2a2a,stroke:#00ff88,stroke-width:2px,color:#e0e0e0
    style LQueryFE fill:#1a1a1a,stroke:#00ff88,stroke-width:1px,color:#e0e0e0
    style LIndexGW fill:#1a1a1a,stroke:#00ff88,stroke-width:1px,color:#e0e0e0
    style LCompactor fill:#1a1a1a,stroke:#00ff88,stroke-width:1px,color:#e0e0e0
    style LRuler fill:#1a1a1a,stroke:#00ff88,stroke-width:1px,color:#e0e0e0
    style LChunkCache fill:#1a1a1a,stroke:#00ff88,stroke-width:1px,color:#e0e0e0

    %% Alloy - Magenta (collector/agent)
    style Alloy fill:#2a2a2a,stroke:#ff6b9d,stroke-width:2px,color:#e0e0e0

    %% Infrastructure
    style Endpoint fill:#1a1a1a,stroke:#444444,stroke-width:1px,color:#a0a0a0
    style K3sNodes fill:#1a1a1a,stroke:#444444,stroke-width:1px,color:#a0a0a0
    style ExternalSystems fill:#1a1a1a,stroke:#ffcc00,stroke-width:1px,color:#e0e0e0

Move to Solar & Battery Power

What

In mid-2025 I finally took the plunge and installed a solar panel system with battery storage at my home. After months contemplating the decision and researching the different options and the costs, I settled on a 4kW solar panel system with a 10kWh battery storage solution.

Why

The main reasons for the move to solar and battery power were:

Reduce Electricity Bills: With the rising cost of electricity in the UK, generating my own power would help reduce my 200 GBP monthly electricity bills.
Environmental Impact: I wanted to reduce my carbon footprint and contribute to a more sustainable future by using renewable energy.
Energy Independence: With rising energy prices and the fluctuation in energy markets since the war in Ukraine, I wanted to be less reliant on the grid and have a more stable and predictable energy source.
Time obfuscation: With battery storage, I could store excess solar energy or cheap off-peak grid energy to use during peak times, further reducing my electricity costs.

The solution needed to be capable of the following:

Have an offline API to monitor power generation, consumption and battery status.
Ideally already have an integration with Home Assistant.
Warranty of at least 15 years.
Allow isolation from the EV charger to prevent house battery wear and tear.

Implementation

After researching various solar panel and battery storage options, I settled with Octopus Energy for the setup.

Device	Model/Specification	Number of Units	Warranty (Years)
Solar Panels	455W JA Solar 430-455	15	30
Inverter	Enphase IQ8AC-72-M-INT	15	25
Battery Storage	Enphase IQBATTERY-5P-1P-INT	2	15

It’s only been a few months since the installation; however, I can report that even in the autumn and winter months, the system has been performing well and has reduced my electricity bills by 37%, mostly driven from the time obfuscation.

Conclusion

2025 has been a year of significant changes and improvements to my home lab and personal projects. From upgrading my networking equipment to Ubiquiti Unifi, building a robust ZFS NAS, implementing a comprehensive monitoring stack with Thanos and Loki, to transitioning to solar and battery power, each project has brought new challenges and learning opportunities. As for 2026, I plan to do more with AI and machine learning, as I write this article with the assistance of an AI language model. I’m excited to see what the new year brings and look forward to sharing more updates on my development journey.