The Cost of Homelab

You there. The one in the back. Yes, you, the weird-looking one with the unicorn costume. Come closer and hear my tale of woe.

Did you know that some people have computers at home? Maybe even… more than one? Some would say… too many?

Nerds, amirite? Why have a bunch of computers at home when you can open a smoothie shop in the airport instead? That is what I always say. Can’t beat a good airport smoothie.

Or at least, that is what I used to say, until I took an arrow in the knee became a nerd with a bunch of computers at home.

I have a homelab. My homelab contains several computers, various networking equipment, 0 dogs or mythical creatures, and lots of cables. And, more recently, a 25U server rack, because why not. What is a homelab, though?

I think traditionally a homelab was a place where you can test out business-grade software and hardware that isn’t a production environment. Production as in the fancy word that means real people rely on it and get mildly to substantially unhappy when it isn’t working. So the idea is that when you are learning how to set up something more complicated, test it out in the lab first. That way, nothing bad can happen because it is separate from the real, production environment. Because we definitely test things before putting them in production. Wink. Sorry, there was something in my eye.

That’s the lab part. The home part is… basically what it sounds like. You set up the lab in your home. This is a lab with computers and servers, mind you. If you want a different lab at home, you will need to bark up a different windmill because I can’t help you there.

So testing things before putting them in production is one use case of a lab. The other large, related use case is learning. And in particular, in the case of a homelab, learning as a personal development exercise. You can learn quite a bit about the fundamentals of technology when you set it up yourself.

Maybe somebody buys a bunch of Cisco switches secondhand and learns how to setup a Cisco network. Or, maybe somebody sets up a windows domain controller and a few client machines and decides to learn how group policies work. There are lots of things you can learn in a homelab. But the general theme is that these things, both hardware and software, wouldn’t traditionally ever be found at home; they are usually more complex things that are intended or generally used in a business environment.

I have an analogy I use to explain a homelab to those more sensible people that don’t have a home full of computers. Having a homelab is kind of like somebody who is really into cars and has a “project car” in a constant state of repair, reassembly or upgrades in their garage. It is a combination of hobby and learning, and maybe some actual functional benefit sprinkled in here and there.

Though I will admit, every time I have used this analogy I get met with blank stares, and eventually something like “well that is cool, I am sure it is useful for something”, as their eyes glaze over and they regret their life choices that led to them asking me what my hobbies are.

I don’t actually have a homelab, though.

Off in the distance, we hear the sound of a record scratch. Or something is very wrong inside my head, hard to say.

I have something I call a homelab, but it isn’t really a lab. It would be more accurately called home infrastructure.

I think there are several very closely related communities that work on similar types of projects or have similar hobbies. A few of these communities go by terms like “homelab”, “home server”, “home networking”, and “self-hosted.” These all blend into each other. I feel that homelab is the most popular term, or at least the most fun to say, so that is what I call mine, even though it isn’t a lab.

When I say my homelab isn’t a lab, what I mean is that I don’t do test things for fun. I have the resources and equipment to do it, certainly, but everything in my homelab is in production. That doesn’t necessarily mean it wasn’t fun to do or that I didn’t derive enjoyment from it, it just means that I depend on everything in there for my daily activities. There isn’t, for example, some virtual machine running that I could just shut off and not notice. Any absence or downtime would be noticeable.

I guess, in a way, the venn diagram for my homelab would have one circle for homelab, one circle for home infrastructure, and they are the same circle. What I mean by that is that I got the homelab type benefits out of it (some enjoyment, lots of learning), but it is also home infrastructure. I didn’t spin things up just to learn for a certification, in other words, but I did still learn many things as a result of solving my real-world wants and needs. Though I will admit some of the upgrades that felt dire and necessary at the time were, perhaps, more wants than needs.

So where does that leave us? I am a nerd with a bunch of computers at home that I call a homelab. A thrilling life story, I know. But there is a problem.

The Problem

DUN DUN DUN

That is the sound problems make in my head, in case you were wondering.

What is the problem, then? The problem is that running a homelab, or home infrastructure, or in general self-hosting anything, costs you something.

Money, for one thing, but that part is pretty obvious and not my greatest concern. You can go surprisingly far on used hardware on the cheap. Most of my homelab came from ebay.

The greater cost, I have found, is time and stress.

Why does having a homelab use up time and cause stress? Well, because computers are hard, and something something entropy of the universe just makes things tend to break eventually even if you don’t change anything. And, naturally, if you do change something, that will also cause something else to break, as is tradition. Here are a few examples of the stresses I have inflicted upon myself:

  • I was building a new computer with a mini-ITX form factor for use as a NAS. I spent 3 hours trying to figure out why I couldn’t “overclock” my DDR4 RAM to its rated speed (3200 MHz). Every time I tried it failed to post and I had to reset the bios again. Eventually I just left it at the stock 2133Mhz speed because I was very stressed and stayed up too late troubleshooting it and it didn’t really matter for what I was using it for anyway.
  • I currently run a web-based note-taking application called Trilium in a docker container. One time I updated the container, and then the web UI failed to come back up. I eventually learned, via a github issue, that the docker file had been changed to use a different user, for permission reasons, and you had to change file permissions or it failed to read its database and couldn’t start.
  • I tried to set up Zabbix once, for monitoring uptime status of various home services. In hindsight I didn’t really need this one, this was more of a fun/hobby/lab thing than home infrastructure. I got it installed but couldn’t figure out how to use it to save my life. I like to think I am smart with computers, but trying to use Zabbix made me feel incompetent. The problem could be me, but I feel the documentation was poor – a common occurrence with a lot of software, I have found. It was particularly frustrating because it was easy to find online mentions like “I use Zabbix and it saved my marriage” or “I use Zabbix and setting it up was like a walk on the beach.” Then I tried to use it and struggle for hours, and I wonder: what am I missing that other people aren’t? WHAT DID YOU SEE, NINJAMELON420? Anyway, I gave up on getting this working.
  • I wanted to get some smart plugs to do smart plug stuff. I am, at times, a bit of an idealist, and try to do things in an open-source, self-hosted way. So I got a smart plug preflashed with tasmota, and I set up home assistant in a virtual machine and tried to get them connected over MQTT or some other such fancy thing. This was another case where I had heard so many good things about home assistant, about how easy it was to use and how it had an integration for everything. But to me it felt like a project that just had a big UI overhaul or new revision and all the documentation online was out of date. Eventually I gave up and just got some proprietary TP Link smart plugs, and I have been much happier with them so far. In this particular case, the stress wasn’t just the troubleshooting in trying to get it setup. It was also the stress of all the research into how to set it up and what smart plugs to buy and what integrations supported what devices.
  • Every single time I have set up VLANs on my network – or, for that matter, any other network – I run into some new problem. Sometimes it is that different vendors use different terms or UIs to configure VLANs, and figuring out how to set the option in the UI that actually does what you want on the wire is surprisingly hard. There is also fun stuff like equipment that can’t properly handle a tagged vlan 1 (cough Ubiquiti cough). Bonus points when you run a router on a stick, where all your VLANs, including your WAN connection, all go into one port. Hope you remembered to trunk ports on all the switches in between your devices. Of all the homelab stuff I have done I actually enjoy networking the most, but it still can take a surprisingly long time to get working.
  • NAS/backup infrastructure has been my biggest source of stress by a country mile. Or a nautical mile. Whichever is longer. The problem isn’t so much getting it working originally. It is the question “what if something goes wrong?” I am, unfortunately, painfully aware of all the ways a backup can fail. Ransomware, lightning strike, disk failure, accidental deletion, fire, filesystem driver bugs, sata cables corrupting data, power outage, rogue unicorn attacks from the skylands… it is such a long list. Maybe your backups ran, but how do you know if the backup created can actually be restored? What if you changed something and a backup that used to work stopped working? Would you get an alert or notification if a routine scheduled backup stopped running? As far as I can tell this is a genuine, certified, organic, non-GMO, Hard Problem™. It is also one of the worst risk profiles. If you have a problem somewhere else in your homelab – maybe the network is down, or your favorite self-hosted app isn’t working – that can all be fixed. But if you lose your data, it is just gone. You can’t recreate it.

These examples I listed here are only a tiny fraction of what has gone on in my homelab. There have been many, many things that were stressful and often take hours to complete.

Somedays, I wonder why I have a homelab. I say it is my hobby. But I have also accumulated a ton of stress from it. There have been upsides – I have learned a lot, I value the functions of my home infrastructure, and I do get some enjoyment out of it. But I also get a lot of pain.

Is it worth it?

Everybody has some life juice. That is a silly term I made up, just now, and it makes me happy, so I will continue to use it. I got life juice. You got life juice.

Each and every day, you get to choose where to spend that life juice. Life juice is primarily your time and energy. You could spend your life juice making the fanciest, bedazzle-est homelab that has ever existed. Or you could spend your life juice with your family. Or you could go out in your boat and eat a taco, if it so pleased you to do so. It is up to you.

But the question you have to ask yourself is this: is the juice worth the squeeze? Are you happy with where your life juice is going? Or do you have some regrets?

It is also important to consider the opportunity cost of your time and energy. It isn’t simply a question of “do I enjoy this activity enough to continue doing it?” In many ways the more important question is “if I allocated less time and energy to this activity, what else could I do instead?”

A feeling I have had a lot in the last few years is “this homelab causes me so much stress; why do I keep doing this? Is it worth it?”

My gut feeling has generally been that… I don’t think it is worth it.

Though it doesn’t have to be all-or-nothing. The choice isn’t limited to having a large, complicated, time-consuming, stressful homelab, or no homelab at all. There are plenty of options in the middle. It can also vary by season of life. At some points in life there will be more time for these sorts of projects and more room for additional stress load, and at other times there will be less time and less room for stress.

My more specific feeling is “what I have been doing is unhealthy, and I need to alter it or reduce it to a point where it becomes healthy again.” I don’t necessarily know exactly what that means in practice, but this feeling serves as a guide for which direction I need to be moving. For me, this feeling that it is unhealthy consists of both a push and a pull. The push – away from this activity – is the stress. The pull – towards other activities – is being aware of the other things I want to do with my time and energy that I can’t because they have been allocated to my homelab.

Why I still have a homelab, and why it is still unhealthy

I still have a homelab.

Why? Didn’t you just say it wasn’t worth it?

Yes. Yes, I did. And my belief in that hasn’t changed. Over a long period of time, I have often felt “this isn’t worth it” or “this is too much stress”. But I have rarely, if ever, felt “I am glad I worked on this.” And if I did feel that way, it was a brief feeling that didn’t have lasting impact, which is not true for the recurring stress.

So why do I keep doing this thing that causes me so much pain?

I have asked myself that many times. And the answer so far has been “I want to stop but I don’t feel like I can.”

Theoretically, my homelab should serve me. It exists to benefit me. But in practice, for me specifically, it often feels inverted. It feels like I serve the homelab. It feels like the technology controls me more than I control the technology. I am stuck in a trap of my own making, in a way.

My homelab has been an oddly masochistic hobby. Though I am not sure if masochistic is even the right word for it. It suggests that it is enjoyable because it is painful, but sometimes it just feels painful with no enjoyment.

I think there are two primary forces within me that have maintained this cycle:

  • Anxiety
  • Drive/obsession/fixation

These recurring forces are why I keep going back and working on my homelab.

First, the anxiety. I got some of that, as it turns out. More than I would like, in fact, and it affects more than just the homelab. Here are some examples of worries/questions that cause me anxiety:

  • What if I lose my files/data?
  • What if there is a hardware failure?
  • What if there is a security vulnerability?

And these are just general top-level worries. Each of these can have many more specific sub-worries, such as:

  • What if I lose files relating to taxes?
  • What will I do if my router fails? Will I go several days without internet?
  • What if there is a security vulnerability in OpenVPN? Because it is exposes directly to the internet, it could be exploited at any time and I wouldn’t even know.

The part of my brain that generates anxiety – let’s call him Frank – is quite the pessimist. Frank often assumes that the worst possible outcome is not only possible, but very likely, maybe even certain – unless I take action to remove the danger. Frank sees a lit candle and expects the house to burn down. Is it likely that the house will burn down? Not very. Is there a chance that the house could burn down because the candle is lit? Yes. So Frank would insist that the candle must be put out. It is the only safe thing to do.

So there I am, sitting there, minding my own business, and here comes Frank. Frank whispers in my ear “Remember that open port on the firewall? What if attackers are probing it RIGHT NOW? You better rush and do something about it.” And so, because I feel anxious, I take some action – incur stress, spend time on the homelab, and so on – to resolve the anxiety.

This isn’t healthy, as I already identified. I think there are two key takeaways to dealing with anxiety that would help here:

  • While the anxiety feels intense, how likely is it that the feared outcome will actually happen? What is the probability?
  • There are very few things in life where certainty can be achieved, and it makes more sense to aim for a reasonably likely successful outcome rather than trying to eliminate every risk in life.

The problem is I don’t remember either of these things in the moment when anxiety strikes. The intensity of the anxiety makes me assume, without question, that the feared outcome is very likely to happen if I don’t do something. And, furthermore, it feels like certainty is achievable, or at least it is possible to make changes that make something feel more certain or make the feared outcome less likely.

So that is the anxiety force. What about the other force? I called it “Drive/obsession/fixation”, which is a bit of a mouthful. Because I am lazy and typing is too hard, I am just going to call it drive, but I think all of these words are applicable.

I would describe myself as a driven person. I like to always be working towards a goal. I don’t like to be idle. That doesn’t mean I can’t enjoy the present; it is just that my goal in that case would be to maximize that present experience. For example, if I am having a conversation with somebody, I tend to prefer a deeper, more meaningful conversation, whereas small talk feels like idleness. I just bring a default intensity to everything implicitly. I tend to either be all or nothing in many areas of life. Being halfway anything is hard for me.

Then, in addition to that baseline drive, I also tend to get deeply focused or fixated on certain interests. For that reason, my hobbies have historically been few but intense. Computer related things in general is one of those interests.

I have this insatiable curiosity. It isn’t simply that I want to learn about how computers work; It is that I feel like I must. But, unlike the anxiety, this isn’t a negative force. This drive to learn more feels like a positive force. I have, on the spot, come up with an absurd hypothetical to illustrate how it feels. Imagine that a local car dealership offered you a deal: any car on the lot for $10. And, for the sake of this example, I am not talking about a car junkyard. These are brand-new cars. If you saw a brand-new car on sale for $10, would you be able to pass on it? Could you continue on with your day, just knowing that deal was sitting there waiting for you? Or would you be, inevitably, drawn to it?

That is how the intense interest feels to me. I feel very drawn to it in a positive way.

In the context of the homelab, this drive can take two basic forms:

  • “I need to learn how this layer/technology works.”
  • “I know this can be built or setup better, and I can’t relax until I do that.”

Here are some more specific examples:

  • If I setup IPv6 on my network, how can I have both globally routable IPs but also static IPs for local servers?
  • If I trigger a database backup on a live database, at the moment the backup completes, is it a snapshot of what the database looks like at the moment the backup started, or the moment the backup finished? How is the backup mechanism implemented?
  • I need to have backups that follow best practices, like the 3-2-1 rule.
  • I need my router to have two NICs, one for WAN and one for LAN. I could run it as a router on a stick all through one port, but having a separate port dedicated for WAN feels more proper.
  • I need to have a 10 gigabit network on my LAN because the 1 gigabit connection bottlenecks file transfer to the NAS.

I would say that my anxiety is almost exclusively a net negative force. I don’t feel I am better off for having it, except in the way that difficult things in life build character and empathy. By contrast, this drive and obsession force often does help me. I consider it a net positive. However, it is positive to a fault.

What’s the problem? The problem is that I can’t turn it off. Let’s think back to that car sale example. Suppose that you could get the car of your choice for $10, but you learned about this right before an important academic exam. You have two basic paths you can follow:

  • Skip the exam and buy the car. You satisfied the drive, but probably made unbalanced life decisions that will cost you later.
  • Take the exam but be distracted because you are still thinking about the car.

Granted, this example is a bit silly, because the benefit of a $10 car probably vastly outweighs the cost of skipping the exam. My point is just that, while the drive and intense interest is extremely useful when it points me towards goals I want to move towards or tasks I want to complete, it also costs me when the thing I am interested in isn’t the most important thing, or when moving towards it makes me unbalanced in other areas of life. It would be like skipping an important exam to save $200 on a car purchase instead.

If you were to look at how I spend my discretionary/free time, you might conclude that my homelab is one of the most important things in my life. But if you were to ask me what the most important things in my life were, I would not say my homelab is one of those things. It is a nice thing to have, certainly. I wouldn’t want to lose it. But it isn’t important to me to the degree that my time allocated to it would suggest it is. Which means I am unbalanced and making unhealthy decisions that don’t align with my values.

So, as a summary, I still have a homelab. And I still have a homelab, despite believing it is largely unhealthy for me, due to the combined forces of anxiety and drive.

There is another unhealthy thing I haven’t mentioned yet: burnout. It is a result of the anxiety and drive forces, and the resulting time I spend on the homelab. I push myself too hard for too long, all the while carrying the anxiety and stress with me. For example, I have “broken” sleep many times by staying up too late solving a technical problem. The technical problem could have waited until the next day, but I felt like I had to keep working on it until it was fixed. But staying up late like that, for me, is effectively like taking out a very high-interest, short-term loan. Poor sleep results in the entire next day being worse off, often in exchange for spending only an extra hour working on a technical problem at night. Not worth it. Yet I keep making that mistake.

And, of course, just because the stress and burnout are caused by the homelab, doesn’t mean they stay there. They follow me wherever I go.

The thing I realize about myself is that I have a tendency to burnout, because I don’t quit or take breaks when I should. My body signals to me that I should, but my drive pushes me past that signal, usually to my own detriment.

These moments of homelab burnout are probably what made me realize it was unhealthy. I kept going through the same cycle, and it was painful. I asked myself, why do I keep doing this? It is completely self-inflicted pain. For me, the burnout makes me more apathetic, maybe because I just don’t have the energy to care anymore. But I find that the apathy often gives me insights about what corrections I need to make in my life.

You would think I would just burnout completely and give up on the homelab. But that hasn’t happened. Usually I feel burned out for a few days, but then once I recover I end up back in the same cycle again.

What am I going to do about that?

Nothing quickly, apparently. Because I had this key insight many months ago, and still find myself trapped in the cycle. But I am working on it.

In an ideal world, I could just scrap the entire thing and go on with my life. That way it would free up my time and I would incur less stress. I would be healthier.

The problem is, I feel like I can’t get rid of my homelab because it actually does provide some functional benefits and it would be difficult to not have those things. For example, if I got rid of my NAS and backup solution, I would worry about losing files. And in the likely event I responded to that worry by using a cloud backup solution, then I would be worried about my privacy.

It is kind of like how there is a recurring maintenance cost to home ownership. For example, if you have a home that is surrounded by that grass stuff, you probably need to mow it every so often. Maybe you don’t like to mow your lawn. But you likely believe the benefits of owning the home outweigh the cost of mowing the lawn, so you put in the effort to upkeep the home.

But there are varying levels of upkeep. There is a difference between just trimming the grass every once and a while and having an elaborate garden with topiaries (aka fancy shrubs), birdhouses, water fountains, and other such elaborate decorations. There is a baseline level of effort to maintain certain functions, but it is also possible to go far beyond the basics. What I have been doing, in a way, is complaining about how much effort I spend on my lawn, meanwhile I am trimming a shrub to look like Mickey Mouse.

So the goal I have been working towards is just enough. I want the homelab to be setup just enough to satisfy the functional needs, but at the same time be as simple as possible and require very little maintenance and time.

My hope is that, with some work, I can get the homelab to this “finished state”, and then ride off into the sunset on a unicorn with all my free time, leaving the stress behind.

Unfortunately, that goalpost always seems to be just out of reach. Close enough that it feels like just a few more days of work will get me there, but then months later it is still just as far away. That sneaky goalpost is moving, as they tend to do. I am not as spry as I used to be, and I can’t seem to catch it.

The problem here is that, while I am aware of the cycle I am stuck in and the underlying forces that sustain it, I don’t change my actions or behavior consistently, so I get the same results. For example, I am aware that many of my homelab decisions are motivated by anxiety, but I rarely remember to question my anxiety in the moment. I am not consistently changing the inputs in the ways I need to, so I am getting the same undesirable outputs (results).

In my mind, here are the steps that I need to take, and need to take consistently, in order to get the results I want:

  • Learn to identify anxious thinking and handle it in a rational way by asking about real-world probabilities and outcomes, rather than making an emotional decision.
  • Learn to not give in to the drive. I recognize what it feels like while it is happening. I just need to develop habits to pause, slow down, and think about what the consequences of following the drive would be, and sometimes decide to not follow what it wants me to do.

There are also some more practical or immediately actionable things I have tried that I believe are effective:

  • Any time I want to work on a new homelab project, I should first write up a project proposal, which needs to identify the expected costs and benefits of the project. The intent is that it makes me more likely to slow down and make a rational decision rather than a gut emotional decision.
  • Timeboxing. Say things like “I can work on this project, but only for an hour today, and then I must be done for the day.” And actually back it up by using timers, alarms, or similar.
  • On-and-off days. Something like “I cannot work on the homelab on consecutive days; there must always be a break in between project days”.

I think all of these are good ideas that have helped me in the past. The only reason they haven’t worked so far is consistency. I don’t consistently apply these ideas. I do well for a few days or weeks, but then I tend to forget and revert back to whatever I did before.

The actual underlying task here is behavior change. Behavior change is hard. Knowing what behaviors you want to change is the easy part. But making changes that stick in the longer term is surprisingly difficult.

I could write about behavior change techniques for a long time. But this post is already long enough, so I will not do that today.

My personal conclusion for myself is:

  • My current actions related to my homelab are unhealthy.
  • I understand that I take these actions as a result of two forces: anxiety and drive.
  • My current goal for my homelab is to get it to a just enough state.
  • I understand what techniques I can use to avoid the negative cycle, and I need to work on how to consistently remember to use them.

Should you have a homelab?

You might read this and think that having a homelab is a bad idea. After all, I droned on and on about all the problems it has caused me. But I don’t think “homelabs are always bad” is the correct takeaway here.

The things I described in this article are about me. Most people with homelabs probably don’t have the problems I have had, because they aren’t me. I am sure there are people out there who take great joy in their homelab as a healthy hobby, and don’t go through negative cycles and burnout as I have.

Ultimately, whether or not a homelab is right for you is a question only you can answer. This article is just food for thought, basically. You can think about what your own core drives and tendencies are, and what impact having a homelab may have on you. You can also think about what your core values are, and to what extent having a homelab aligns with those values.

I just want you to be aware that there is a cost of homelab.

 

Jacob Clarity

 

Leave a Reply