“Upgrade your server!” they said. “It’ll be EASY!!!” they said…

Posted on by Scott / 0 Comment

It’s been a while since I’ve done much tinkering with my home servers – frankly because I’ve been pretty satisfied how they’ve been running as-is!

I think it was back in 2018 that I migrated the first half of my setup to a rackmount (old enterprise) hardware, with Plex and everything else running under virtualization for the first time; and then in 2019 I added 4U of NAS boxes to move all of my media and put running on desktop hardware to an end.

For the most part, it’s been great – I’ve got a couple of beefy batteries to help endure blips with the power, and Unraid is so much better at managing disks than me shuffling stuff from one external hard drive to another … particularly when you’re looking at upwards of a dozen disks…

I actually believe I’m running 15 now and I’ll probably add more if I can find a good sale in a few months for Black Friday, but regardless…

The limitation of using old enterprise hardware is that there’s only so much you can do to upgrade them. Dell considers all of the machines that I’m running to be 11th generation, whereas currently they’re on 15th generation, so when it comes to things like processor upgrades and even drive support, there’s not a ton of wiggle room.

Not a big deal, considering that as I said they’re running pretty smoothly, but still … I guess you could say I got bored and felt like seeing what I could tweak anyways? 😛

About a month ago, I settled on three relatively simple upgrades that I could do to bulk my main server that runs my VMs up a bit to at least help prolong until I’m able to replace it with a new build altogether…

  • New CPUs – This sounds like a big endeavor, but I literally found a pair of matched CPUs on eBay for like $40.
  • Add SSDs – This is the one I just finished, and it wasn’t as easy as I had assumed!
  • 10 gigabit ethernet – I’ll do this one later on this fall … I don’t have a 10-gig switch yet, so I figured in the short term I could run 10-gig just between the two servers and at least see some speed boosts there.
  • Graphics Card for Plex Transcoding – Probably my last update because A) the card alone runs about $500, and B) I have to do some other mods like replacing a riser in the case just to get the thing to fit. Still, this will be the biggest impact because it should render all Plex transcoding a walk in the park…

Swapping out the CPUs probably made me the most nervous – I mean, you’re basically doing brain surgery on your computer – but aside from a brief scare where I couldn’t find my thermal paste, but the replacement went super smoothly and it booted up no problem with the new chips a few minutes later!

I went from 4 cores each running 2.5 Ghz (16 cores total w/hyperthreading) to 6 cores each running at 3 Ghz (24 cores total), which will basically give me a little more overhead for Plex transcoding while I wait to add a real graphics card.

And now on to today’s adventure, or should we say last week’s adventure???

My Plex libraries have gotten pretty massive – think thousands of movies and TV shows across over 100 TB of media – and one recommendation that I’ve heard to improve performance as you grow bigger is to move your metadata and database over to an SSD. I guess it’s not so much the interface speed itself, but the IOPS because with the metadata you’re dealing with tons of tiny files with all of the images.

Anyways, I originally wanted to go nVME because I understand they’re by far the fastest drives available right now. Unfortunately, there was no good way to get them into my system. Sure, I could buy a card to add them, but RAID wouldn’t be supported and I’ve come to appreciate the redundancy because I know how easy it is for drives to fail…

So instead I went with just regular, old SSDs because I could fit them right into my normal drive bays and then my existing RAID controller could just manage them like normal. Easy, right?!

Well, apparently the PERC6i RAID controller that I had in my Dell R610 doesn’t like SSDs, or at least doesn’t like Western Digital Red SSDs, because it would detect the drives at startup but refuse to do anything with them, reporting failures.

I tried swapping drive bays, thinking it was a bad cable.

I tried upgrading the firmware for the card, which OMG took forever because the server is so old that the online firmware update process no longer works.

Finally I decided to try replacing the card altogether because I knew that the PERC6i couldn’t use larger drives, either … not a problem in this case, but still. I ordered an H700 off of eBay instead and waited a week, then waited another week while I tried to figure out how to backup all of my VMs just in case!

As of now, everything is up and running as expected, so luckily I didn’t end up needing those backups, but it sure was close for a minute this afternoon…

Installing the card itself was pretty easy – thanks to this video which walks through the whole process…

That said, the cables that shipped with the card that I bought weren’t long enough to route through my case correctly, so now it won’t close! I have replacements coming in a couple of days that cost me another $20 on top of the $40 I spent for the card, cables, and new battery.

Once I booted up the new card, the new SSDs showed up immediately as Online!

Not so much for the existing drives, but that was an error on my part because I missed a prompt to “Import the Foreign Array”. A second reboot to correct that showed everything and a few minutes later I was initializing the new array and doing a background initialization on the old array which finished by the time I was done with lunch.

But we weren’t out of the woods yet!

Upon booting back into ESXi, it too saw the new SSDs and I was able to create my new datastore where I wanted to move my Plex VM to … but the old array was nowhere in sight and as a result, it thought that all of my existing VMs were invalid.

That sort of made sense because I figured that the identifiers for the drives probably changed with the introduction of the new card. After a healthy amount of digging, I found that the trick was to SSH into ESXi itself and then do this…

esxcli storage vmfs snapshot list
esxcli storage vmfs snapshot mount -n -l "SNAPSHOT_NAME"

The first command confirmed that my datastore still existed and the second one mounted it so that ESXi would recognize it as usual. (this link explains it all)

From here it was simply a matter of unregistering each VM, moving them one by one to the new SSD datastore, and then re-registering them as existing VMs. Apparently the one downside of doing it this way is that the disk provisioning type for each VM changes from thin to thick, which means they take up their full disk allocation instead of only what they’re using at the time … but I’m really only using the new SSD datastore for two VMs and they’ll both probably get rebuilt eventually for OS upgrades anyways, so it wasn’t worth the hassle with vCenter/vSphere/whatever that I honestly don’t understand yet anyways.

They’re moved, they’re running again, and we’re done!

Now can I tell much of a difference???

I was hoping to see faster download speeds on the one VM, but not so much there.

For Plex itself, the interface both on my laptop, phone, and a TV locally as well as my phone over cellular data does seem a lot snappier. Artwork seems to pop onto the screen in maybe a second per page on the TV when it would take a couple to browse through pages of large libraries, so that’s a win in my book.

And again, I wasn’t exactly expecting huge gains with any of these, but aside from pounding my head against the wall for the last week about the stupid RAID card, both were fun, little upgrades that added incremental gains to the performance, so that’s cool in my book.

Next up will likely be the 10-gig upgrade because I know what parts I need and just need to get them ordered. The goal is to string the two servers together for starters, and then maybe add a 10-gig switch next year along the same time I upgrade to a new Macbook that can also support 10-gig, too!

My longer term goal, I’ve decided, is going to be to eventually build new servers myself to replace all of these ones. Still rackmount, but I want to do them myself because new enterprise gear is ridiculously expensive and I think I’m comfortable enough with it now to go at it alone. I’ve gotten a lot of inspiration from watching Linus Tech Tips and the evolution of their own server room at their office, so ideally it would wait until we build a new house in a couple of years where I can:

  • Install a full rack in a dedicated room (that isn’t our bedroom closet!)
  • Build two identical servers so that I can upgrade to a cluster to make it easier to make changes without taking Plex down
  • Build a replacement NAS, probably in a single 4U case
  • Along with a matching backup NAS

As you can see, it’ll be a big endeavor that will take some time to put together, but I’m in no rush and I think my current rig will keep us moving forward just fine until then.

We’ll see if the other two upgrades are crazy enough to warrant a blog post of their own!

Warm Water Weekend

Posted on by Scott / 0 Comment

This week, amid Tropical Storm/Hurricane Elsa, we did something that we honestly should’ve done years ago – added a solar heater to our pool.

Once the bad weather passed and the sun had a chance to come out for more than an hour, the effect was seriously like swimming in bath water!

Yesterday I set the pool at 92 degrees and it was almost too warm, compared to the 82-85 degrees we’ve been seeing the last couple of weeks in June since I started measuring.

Today I backed it off to 90 and at 11:30am when we first went, it was just about perfect.

I’ve found that 80+ is enough to enjoy during the summer and feels refreshing when the sun is out, but you still get that shock when you get in for the first time – particularly when the boys in your swim trunks first hit the water – and after dark once the sun is down and the kids are asleep, it’s a bit chilly to relax and unwind in.

90, on the other hand, is like stepping into a giant bathtub and brings back memories of swimming down in the Florida Keys in the middle of July … without the salt water, thankfully. I was even able to use the heater to get the hot tub up to 102 degrees during the day, which fell to around 97-98 by the time I got the kids to bed and could actually enjoy it for myself.

Considering that our spa heater has literally been broken for years, this was a nice little bonus!

The only downside that’s going to take some getting used to is that Matthew is a lot more comfortable in the warmer water, which has kept me on my toes the last couple of days. Lately I’ve been taking all three kids swimming by myself because Sara has been having problems with her ear, and I can mostly handle it because Christopher and David can usually handle themselves and Matthew would spend a lot of time playing on the pool deck, so I could relax and just catch him when he wants to jump in…

…now, however, he’s gotten a lot braver between jumping in without waiting for me and swimming over his head where he can’t make it across without assistance. He’s also been doing this thing that Christopher used to do where he can’t judge when he’s tired and needs a break, so he just keeps going and going and struggling more along the way. ☹️

Hopefully we can push him to build up his skills, just like his brothers did, to where these won’t be as big of a deal, and once Sara’s feeling better that will give me an extra set of hands, too.

Anywho, I’m really curious to see how these solar panels manage once we get out of the hottest months of the year because the selling point that we heard wasn’t so much warmer temps but a longer swimming season – as in, this should in theory add another month or two to each end of when we normally use our pool, which is typically around May – October for the kids and maybe June – September for us thin-skinned adults!

If it could keep the temps into the high 80s or even 90 for March/April and October/November, that would be amazing, but there are a lot of other factors like the position of the sun in the sky that I’m just not familiar enough to say, so we’ll see what happens.

In the meantime, though, the consensus is that solar heat is a win! 😉

Writing Prompt Sundays – Fix a Movie

Posted on by Scott / 0 Comment

Writing Prompt – What’s the last movie you saw and how would you make it better?

This is an easy one – Fatherhood, with Kevin Hart, and I’d improve it by not killing off the wife in the first act.

Aside from the fact that it’s been done a thousand times, I kind of shook my head when it played out on the screen because I was expecting a solid comedy with Kevin Hart and this plot device just made it way too sad too quick. Plus, as a father myself, there are plenty of wacky fatherhood scenarios that the movie could’ve played out without him needing to be a single Dad. Or if you want him to explore that helplessness anyways, make the wife a career woman and put him in the stay at home Dad role.

It just really set the movie off on the wrong foot for me, which admittedly was already going kind of slow, to have that traumatic scene, and the funeral, and everyone gathered at the house, and then all of the pressure from his mother-in-law about them wanting to take the baby home with them?!

Maybe it wasn’t meant to be nothing but laughs like his other movies, but I was definitely looking for something lighthearted and funny that night so to be hit with more of a drama was a little disappointing.

a good Florida day…

Posted on by Scott / 0 Comment

By my best estimation, I think we spent something like 3-4 hours in the pool today. It was basically all we did today, and it was fantastic.

The kids admittedly didn’t get as much pool time as we would’ve liked last week due to scheduling and rain conflicts. I think the last two nights we had to cancel on account of thunderstorms, which of course is very common this time of year in Florida, so out of fear that the same could happen again today for a third day in a row, I ended up taking them out this morning to get in an hour or two before lunch…

…which also has the added bonus of quieting down the house so that Sara could sleep because she both worked overnight last night and is working again tonight…

Anyways, arguably the pool is one of my easiest ways of entertaining the kids because they’re allowed to get a little wild (within reason) on account of being outside, it wears them out like crazy which makes naptime/bedtime so much easier, and it’s a good workout for me, too.

And it’s fun, of course!

Usually I’ll spend some time diving with David & Christopher while Matthew plays around on the pool deck, and then once his brothers start jumping that’s what he wants to do which turns me into the aquatic version of an air traffic controller to both keep his head above water and at least try to prevent everyone from bumping into each other … the latter of which is still very much a work in progress!

After our pool time, Christopher asked if we could eat lunch outside on the patio – another thing that we’ve also been enjoying lately, so we ate and watched iPads for a while before one by one I started peeling them off for naptimes. Christopher and I then sat around by ourselves a while longer, with me doing some work and him playing with his Mario Kart cars, until we eventually made a quick trip to the gas station for ice and snacks which resulted in us getting a bit drenched by the afternoon downpour.

Thankfully, however, it only lasted maybe twenty minutes and the heat outside still made the pool a welcoming place, so after naps and saying goodbye to Mom, we headed back out for round #2 and swam for another hour while waiting for pizza to get delivered for dinner!

In reflection, it just reminded me of one of those classic, lazy summer days where all of your time is either spent in the water or eating nearby it. It would’ve been nice to have a grill or something for maybe some shrimp or fish, but then again it would’ve been nice for Sara to be able to join us, too, because I certainly can’t man a grill and supervise three kids in the pool at the same time anyways!

We need more days like this over the summer. I’ll circle back to the grill point at a later date… 😉

Coronavirus, Day 474 – What Now???

Posted on by Scott / 0 Comment

I honestly just don’t know what to think about this pandemic anymore.

It frustrates me that vaccinations have slowed down so significantly – a month ago on May 22nd, 2.3 million doses were given out, compared to just 647k doses yesterday on June 22nd.

It also confuses me that the case and death numbers have fallen as much as they have, which don’t get me wrong is obviously a good thing. It just leaves me wondering if we’re missing something … are states getting lax about their reporting (Florida switched from daily to weekly recently) or are more cases going unreported???

I wish I knew what it’s supposed to look like for me to believe that it’s over because although the graphs above look promising even despite vaccinations running at only 45% of people vaccinated instead of the 70% that I thought we were aiming for a few months ago.

I guess I just worry that after we’ve all been through and sacrificed so much, what if there’s another surge that we could’ve prevented by taking precautions just a little bit longer or by more people being eager to get vaccinated themselves. My wife and I are both vaccinated and we still wear our masks everywhere because we’re worried about bringing it home to our kids, and yet it’s hard to see mask rules pretty much on a whim at this point where you know that those most adamantly against them just aren’t wearing them and still refuse to get vaccinated.

It’s weird for me to see Disney not only removing the last of its precautions around the parks, but also absolutely packing them in with people who are ready to move on with their lives … despite today seeing 365k cases around the world and 8,600 deaths.

And it’s damn scary to see Republican politicians going out of their way to pass legislature against things like schools requiring vaccinations or even masks, or businesses being allowed to require proof of vaccination for going on cruises in the name of “personal liberty.”

I’ve been taking my shoes off at the airport now for twenty years because one time a guy tried to blow up a plane with bombs in his shoes – what about that personal liberty?!

I think a lot of it makes me uncomfortable because it forces me to be more judgmental of the people around me for the sake of protecting my family. If everyone still wore masks or even if the vaccination rates were still up to show that the public trust was high around everyone just being careful, it would make me feel better about trying to ease back into some of the things that we did before COVID-19 changed all of our lives.

I saw one person joking about becoming agoraphobic over the last year and I don’t know if I’d go that far, but my anxiety is definitely a bit more on edge when I have to be around people who I don’t know if I can entirely trust because I don’t know if they’ve shared the same attitude towards the virus and staying safe as I have.

It leaves me wondering what back to normal will look like for me and just how long it’s going to take regardless of whatever the world chooses to do around me… 🙁

It’s Finally GONE!!!

Posted on by Scott / 0 Comment

I can’t tell you how long I’ve been looking forward to this day.

Actually, I can!

I’ve despised this 30-some-odd foot monstrosity pretty much since we bought this house back in 2012. It drops leaves and acorns like it’s slowly dying on my front lawn, even though it’s definitely not, it’s impossible for me to prune because of how huge it’s gotten, and it’s even started to tear up the sidewalk and our driveway.

If left to its own devices, I have little doubt that before too long this innocent-looking oak tree would’ve displaced the entire house, leaving just one gargantuan behemoth of a tree in the center of our tiny, suburban lot and the family that once lived there forced out onto the area formerly known as our sidewalk before it ripped that whole thing to absolute shreds, too!

Admittedly it actually was pretty impressive to see the crew work. It wasn’t cheap, but these guys rolled up with three trucks and two massive cranes along with an industrial-sized wood chipper that buzzed through 8.5 years of my oak dread like a hot knife through a tree.

I griped a lot about our HOA over the years, but the process was surprisingly easy. I probably spent a week or two getting quotes from various tree removal companies – the one we picked wasn’t the cheapest option, mind you, but I was more confident in these guys than the next quote that was about $300 cheaper and when I watched them work, I knew we’d made the right choice. We didn’t have to get a local permit because the tree was considered a nuisance (tell me about it!) due to the damage its roots were doing, and I think even the HOA had their approval back to me in less than a week!

From the final approval, it took about a month for scheduling to get these guys out, and once they arrived the whole thing was GONE in maybe an hour?!

They still need to come back in a few days to grind out the stump so that we can try to grow grass over the oak tree’s unmarked grave, but all in all I was pretty impressed.

And relieved. We literally just had the AC worked on in our van because acorns were clogging up the pipes and causing water to back up into the system. Just before that, I raked 4-5 bags of leaves myself and then paid somebody who raked another 10 bags out of my tiny, little front yard!

Good riddance – don’t let the wood chipper hit ya where the lumberjack split ya… 😛

Writing Prompt Sundays – Medieval Travel Plans

Posted on by Scott / 0 Comment

Writing Prompt – Where would you fly right now if you could hop in a plane?

Let’s assume that this implies we’re not in the middle of a global pandemic because my travel anxiety is high enough without adding in the threat of communicable disease!

I’ve always wanted to travel to Europe to check out all of the castles of the Middle Ages. I’m not sure the extent to which they let you just randomly wander around – if it’s simply a look at the castle walls or you can actually walk the dungeons and towers and explore all sorts of secret passages that for all I know are based more in movies than reality.

When I was a teenager, we had a camping trip at Northern Michigan’s closest equivalent to a castle and I got to ditch my lowly tent on the grounds for an opportunity to actually sleep inside! I don’t remember a whole lot about it, except that the room we stayed in was made of stone and it definitely felt the part to a 14 year-old who’d always looked up to the likes of Robin Hood and the Three Musketeers as veritable role models…

It does make me wonder with so many castles over there, does looking at them ever get “old” as in, “If you’ve seen one castle, you’ve seen ’em all…” or are there enough varieties and details to explore that it’s like people who tour classic churches or museums where there’s always something new to discover?

I don’t know if it will happen before we retire, but it’s definitely a trip that I’d like to make eventually … and if there’s an opportunity to spend the night inside, well, then that’s just a bonus! 😉

Mini-Games

Posted on by Scott / 0 Comment

Very much on a whim, I found myself ordering one of the Super Mario Bros Game & Watch handhelds.

I honestly wasn’t really swayed last fall when this was released because I didn’t have any of these things as a kid and admittedly most handhelds are just too small for my big hands and old eyes to play for any length of time these days. But then Nintendo just announced that this year they’re releasing a Zelda version – which is easily in my Top 5 favorite games of all-time – and I also happened to find the Mario one on sale, so here we are…

Even though by now I’ve played the original Super Mario Bros on probably half a dozen different platforms (NES, SNES, Wii, WiiU, Switch, Arcade Cab, Computer Emulator), for some reason it still amused me to see that iconic screen in full color on a tiny, 2.5-inch portable screen. And it’s not like I don’t have scads more graphically impressive games on my phone, or that I could carry around literally thousands of games on a 1 TB micro-SD card the size of my thumbnail.

Fun Fact – Super Mario Bros on the NES was only 40 KB in size!

I think what really makes me smile is that Christopher has gotten a kick out of playing it, even though it’s not nearly as shiny and feature-rich as Super Mario Odyssey and Super Mario 3D World that he’s playing right now. Here’s a pic from 2018 of him trying to play, but more or less just running into pits and Goombas…

And yet three years later he can play along with the game that not only introduced his Dad to video games, but also was the predecessor for the Switch games that he loves exploring today!

I’m curious to see if the next one will spark a little interest in the Zelda franchise for him because although he hasn’t had much interest to date, you never know what will catch their eye next! 😉

…either way, I’m sure I’ll enjoy playing through the first dungeon randomly before my eyes start seeing double or my hands cramp up on these tiny, tiny buttons…

Dealing with Death

Posted on by Scott / 0 Comment

This week started as Monday morning I learned that one of my coworkers passed away unexpectedly.

We worked together for something like eight years – nearly half of my career and a quarter of hers – and although she was going to be out on leave for most of the summer for medical reasons, I never could’ve imagined it turning to this.

I guess I’ve been kind of lucky recently because I think the last funeral I went to was my Grandpa’s in 2014 and then my wife’s Grandma a couple of years before that. Even with the global pandemic, I don’t think that I knew of anyone personally who has died from it, and so other than Cleo passing last fall, I’ve been mostly spared from the sorrow that comes from that kind of loss…

Monday was mostly a day of shock for me and I spent the rest of the day working in my office alone with the lights off after sharing what limited news I had with my co-workers.

Tuesday was when I started to feel the sadness seeping in. I’d learned more specifics about what had happened and more people were reaching out to me, which has felt very stressful in a guilty sort of way.

By today I just didn’t really want to interact with anybody, which has been tough because Sara isn’t feeling well and the kids have been a lot to handle today on top of trying to squeeze in a little work here and there because the job doesn’t stop. In fact, it only grows because not only do I have her extra work, but also the work involved with her passing … which I also feel guilty even complaining about because it is my job, but here we are regardless.

Tonight I went swimming after the kids went to bed for all of about ten minutes until a thunderstorm rolled in and called me out of the water. Even though it was a little cold because it rained some today, and it was dark so the only lights were the pool lights, it was nice just to embrace the quiet for a few minutes and try to take a breath and get my bearings around all of it.

I feel like I really haven’t had a chance to grieve yet, per se, because I need to work on an announcement to go out to the whole company about her career, and there are a lot of other unknowns that I still need to talk to HR about because they don’t exactly have a documented process for this sort of thing. Everyone around me has offered to help in any way that they can, but when you’re trying to figure out what it is you’re supposed to do yourself, there’s just not much that others can offer at this point.

Tomorrow I need to talk to her family again and see if they’ve made funeral arrangements, and I need to get a handle on what my own responsibilities in all of this are. Apparently I also have to get access to her email to see if there’s anything we need to keep before IT removes her account, which makes me a little uncomfortable.

And yet through all of the sadness and discomfort, I keep reminding myself that my feelings are nothing like what her husband and daughters are going through right now. I worked with her for 40 hours a week – a good portion of the time remotely where we didn’t even see each other face to face – whereas she was the rock of their family 24/7/365.

I will say that I’ve tried to hug my family a little tighter and appreciate everything around me, despite the chaos, ever since hearing the news. It’s scary to realize that things can change on you just that quickly, and that nothing is guaranteed in life except right here and now.

It reminds me of this eulogy that Richard Hunt gave at Jim Henson’s funeral back in 1990 that I turn to in times when I need a little inspiration…

Writing Prompt Sundays – My Next Five Years…

Posted on by Scott / 0 Comment

Writing Prompt – Write about the goals you’d like to work toward in the next five years.

In five years’ time, I’ll be 46 years old.

The kids will be 12 (6th grade) and 9 (3rd grade).

If I continue at the same job that I have today, I’ll have been with them for 22 years!

Over the last few months, I’ve spent a lot of time thinking about goals near and far, so I’m honestly not sure if most of them are much of a surprise…

  • I’d like to continue working towards taking better care of my health. This is probably the most important one – for obvious reasons.
  • I’d like to have built and moved into our dream house, so here’s hoping that real estate and construction costs have dropped considerably before then!
  • I’d like to go on some fun adventures with our family once COVID-19 is finally behind us, including travels out of state and possibly even under the sea… 😉
  • I’d like to explore some new avenues with my writing, and also continue adding more books to my bookshelf – both digitally and in person.
  • In general, I’d like my life to be less stressful and more within my control so that I can better focus on the things that are really important to me.

I actually think that every one of these are pretty reasonable goals for a five-year period if I’m able to practice what I’ve learned lately about making small, impactful changes over time. None of these are things that I’d expect to solve over a single year, but five years gives a person a lot of time to work with as long as you’re actually willing to put in the work.

And I’m not saying that by 46 I want to own a theme park or write The Great American Novel, though the money from the latter would be nice for funding that dream house of mine!

My goals are really to be more or less of healthy, reasonably wealthy, and just wise enough to fill a few tombs with my thoughts for my own amusement. With enough time and inspiration, how hard could that be?!