To Migrate 70 Terabytes…

Posted on by Scott / 0 Comment

It’s true.

The project that I started almost two months ago to migrate all of my home server data from rapidly aging desktop hardware to a rackmount NAS is finally completed.

What was previously around 60 TB spread across 9 hard drives of varying shapes, sizes, and ages has now been moved to a dozen 10-12 TB drives all born within the last year, including the addition of two parity drives for redundancy in a new-to-me server that will be dedicated to nothing but storing files, thus finally separating out Plex and the various apps that I use to download media to their own hardware where disk conflicts should officially be a thing of the past!

Of course, it didn’t take that full two months solely to move the data from one set of drives to the other … even though at times it certainly felt like it…

A good chunk of time was spent waiting for Unraid to clear and format new drives – a little over a day for 12 TB drives. 😯

I also had to limit when I could migrate data so as to not impact Plex, considering both that copying at full tilt ate up a lot of CPU on my old server AND I found that copying at full tilt into the new server would make it difficult to stream media from the affected drive at the same time.

I ended up counteracting the latter by adding a 1 TB SSD cache drive to Unraid, which unfortunately limited me to moving about 1 TB at a time because the mover process that moves data from the cache to the array (normally at night) is equally intensive.

That said, most of the speeds I got from the old server weren’t enough to matter anyways. For drives attached directly to the motherboard, I could average speeds of 60 MBps, however a good chunk of my media was living on external USB drives which meant that it was more likely for my transfers to crawl along at 30 MBps instead…

Comical when the SSD can do upwards of 90-100 MBps and even higher read speeds, but hey – I knew that speed wasn’t one of the selling points of going with Unraid, anyways.

Those two months also included a disk recovery … truth be told, I actually lost two disks that prompted me finally putting all of this into motion! One was a lost cause and I just made a list and re-downloaded everything over time, but the second I left alone until everything else was done and then was able to recover using this great free app that I found called testdisk. It turns out that basically the partition table had gotten corrupted somehow – a problem that actually already affected me once before that I previously had been able to repair, but this time once I realized that testdisk would allow me to copy the contents over to another disk that I now had to spare, I opted to just do that instead and about 12 hours later roughly 490 movies were sitting on a fresh disk and ready to migrate over to the new array!

So anyways, as of yesterday now everything is living on the new server and I’m basically ready to power down old faithful and prepare it for its afterlife. I think I decided that once I get the cables I’m waiting on to move the new NAS into my rack in thcloset, I’m going to bring that old server down to my office and disassemble it, give it a good cleaning and actually remove the dead drives that are still installed, and then I’m going to wipe the thing and turn it into a sandbox of sorts for a few random things that don’t really have a place on my other servers…

  • Plex media local backup – Until I can build out a proper backup NAS, I’m going to take a couple of leftover 8 TB drives and backup the most essential 16 TB of media in my collection for an additional backup on top of the 1 TB that I’m now backing up to the cloud.
  • Torrent seeding – I found this great docker for Transmission that incorporates OpenVPN for a seamless experience, which makes me feel a little more comfortable having it running full time to help seed some of the more difficult to find files that I had to hunt for after not being able to get them from usenet.
  • DVD ripping station – Right now the only computers left in this house that still have CD drives are one rackmount server and a very old laptop that I first trialed Plex on before moving it to my desktop hardware. I actually bought an internal bluray drive shortly after I started getting into Plex because I thought that I’d end up ripping all of my media instead of just downloading it, so it’s been buried underneath my desk for about four years now. Nonetheless, I want to install it in this version of the computer to have something a little more accessible for the random DVDs that I have to rip myself.

Not sure how much else will end up there simply because the CPU inside is pretty weak at this point, but I’m really trying to keep my main environments a little cleaner and not just install any old random thing that I come across, so this will be a good place for that because it won’t really matter if there are disruptions.

Looking forward to writing up a separate post outlining all of the reasons why I love Unraid now that I’ve been using it for a couple of months, and of course, I’m already working on expansion plans to move beyond the 106 TB limit that I currently have installed in my already very full, new NAS today! 😉

Things I Want for Christmas… – 2019 Edition

Posted on by Scott / 0 Comment
  1. I need to figure out why the ice from the ice maker in my freezer all melts together into an unusable blob in the bucket so quickly.
  2. I want someone to fix the sprinkler pipe in my garden that I broke with a shovel while grumbling about our HOA … not because I’m not capable of doing it myself, but because every time I think about it I just get angry all over again!
  3. I’d like to go an entire week without stepping on a single toy in my house.
  4. I’d love to not have to drive 60 miles round trip to get rid of my styrofoam recycling.
  5. I wish the lights in my garage that went out a month and a half ago would magically start working again.
  6. It would be a blessing to have someone properly trim the hideous oak tree in my front yard.
  7. I really want to have all of our Christmas lights up by December 1st this year.
  8. I’d like to solve the problem of us constantly losing our remote controls once and for all.
  9. Both of our cars could use a thorough cleaning.
  10. I’m really curious to know how much electricity my latest server upgrades are going to cost me.

Things That I’m Excited About Right Now! – November 2019 Edition

Posted on by Scott / 0 Comment

Thanksgiving Dinner
We’ve never actually done a traditional Thanksgiving dinner all by ourselves. Typically we’ll get a pre-cooked turkey breast from HoneyBaked Ham, and if we’re feeling really lazy, sometimes we’ve even gotten the sides and dessert pre-made, too, so “dinner prep” is more of a reheating exercise than anything else.

So this year we’re doing things a little different.

We actually don’t have any family coming over, so it’ll just be the five of us, but we’re going to try our hands at actually cooking a turkey, making the sides in something other than a microwave, and hopefully getting a bit of family time out of the whole ordeal, too. We still need to plan a menu and go grocery shopping, but I’m surprisingly looking forward to the challenge!

Christmas Shopping
The last year or so, including Christmas 2018, was kind of tight, so we’ve had to cut back in areas a lot more than we would like, however at least for the time being things are turning around … mostly because my wife has been picking up loads of extra shifts … and so after getting a bunch of bills finally caught up, I’m looking forward to splurging a bit and have already caught myself digital window shopping for the various folks on our list.

Spoiler Alert – It’s all about the clam candy canes!

Disk Space!!!
Even though it’s also been stressful and there’s been a lot to learn, I’ve really been enjoying building out my new NAS over the past couple of months – particularly this past weekend when I dropped in five brand new 12 TB drives that I picked up on sale at Best Buy to max the beast out at a whopping 106 TB of usable storage!

Granted, it’s a little concerning that this thing is already almost 70% full, and unfortunately every drive bay is filled in the current enclosure, but it should at least tide me over for a few months until I figure out where to go from here…

So what did you do this weekend?

Posted on by Scott / 0 Comment

I myself was surprisingly productive!

  • Brought in server rack from the garage and started reorganizing servers in closet.
  • Spun up a new VM and installed Home Assistant to start tinkering with the next level of home automation.
  • Started working on a new script to keep up with downloading YouTube videos into Plex.
  • Cleared off two shelves of expired foods in the pantry.
  • Put away Halloween.
  • Made somewhat of a dent in cleaning the living room and playroom.
  • Added new hard drives to NAS and expanded parity volumes from 10 TB to 12 TB. (I love Unraid!)
  • Gave the dog a bath … after the kids covered her in applesauce.
  • Cleaned out bedroom closet to make way for said ginormous server rack.
  • Reorganized recycling in garage … because it’s a giant pet peeve of mine that they can’t just all go to the same place.

Pay the Writer

Posted on by Scott / 0 Comment

I’ve really been enjoying this thread by Heidi Moore over on Twitter about writers getting exploited through low rates…

This morning it got me thinking about the days long ago when I ran Just Laugh because some smaller publications are starting to give her pushback for being called out because “they don’t have the budget to afford to pay writers any more.”

And I sort of get that to a point, however that point lies in a time almost 20 years ago when the publishing landscape was much different … and it’s for this same reason why when I tried to relaunch the site a few years ago, I just didn’t have the guts to ask people to write for free anymore…

When I first created Just Laugh back in 1999, I had just graduated from high school and was still living in my Mom’s basement. I worked the same job I had in school that paid me about $9/hour, and I built Just Laugh in my spare time to help distract myself from not having gone off to college right away like the rest of my friends had.

The Internet was still pretty new then, so Adsense and Amazon affiliates and all of these revenue options for small publishers just didn’t exist yet. I don’t know if we all just happened to be in the right mindset at the right time, but when I emailed writers asking to syndicate their work or publish something new for them, it worked for everyone at the time.

In fact, the site always had a slot for a 468×60 banner ad in the header, but it was perpetually filled by either house ads for our various projects or ads for random sites that I liked.

I never made a dime from it, and by the time advertising had become a thing, the site was basically defunct and the only place I ran anything was in our Joke Database which still never made enough to cover the $14.95 a month I spent on hosting.

I actually remember being surprised when I looked at the account years later and found that it had accumulated a couple hundred bucks in it, quite literally by making $1-2 a month for years and years on end! 😯

Anyways, I wanted to share this because I feel like today online publishing is very different.

Today it’s so much easier to build a website, and a following of your own through social media, and even earn financial support whether it’s through ads or merch or directly from fans via sites like Patreon.

Most creators don’t need a publisher anymore, and I can’t tell you the number of times during that attempted relaunch when I would come across writers and artists who I would’ve loved to work with, only to think two things…

  1. But I can’t afford to pay them.
  2. And their following is bigger than mine to the point that they don’t need me anyways!

It’s been a really sore spot as I’ve learned more about publishing in general over these years how lopsided the equation actually is for most creators, and how messed up it is for someone to take the lion’s share of your revenue and also not do as much for your own work as you could just do yourself.

You hear about it all of the time from people on the NYT bestsellers list, and yet people still scramble to make it there despite knowing that getting actual support from a big publisher like JK Rowling or John Green might is the equivalent of winning the lottery.

So it was interesting to read about several smaller publications fighting in support of their lower wages because they were doing the best that they could, and writers were grateful for their voice. 

One actually noted in its “expense report” that their expenses had gone up “because they had to pay for health insurance for their staff” … which is kind of the point that the whole thread is trying to make because a writer shouldn’t have to work for pennies so that their editor can have health insurance!!!

The thing is, I get passion projects. I really do.

Just Laugh was always very much a passion project for me, and I sincerely hope that the other people who helped contribute to what we did looked at it in the same way.

In hindsight, I actually had one squabble with a writer who thought that I was just using his work to sell merch, and despite having only sold I think one single mousepad the entire time, now as an almost-40 writer I can see a little better where he was coming from, and I would’ve been pissed thinking some cocky, 20-something was profiting off of my work and not paying me for it, too!

It’s for that reason why I think that it would be really hard to run a publication like that today because just like any other field, if you can’t afford to pay your writers a decent wage, you don’t deserve to take one yourself, either. Maybe if everyone is onboard with your mission, or it’s a close-knit group chasing a dream, but if you’ve got a budget of any sorts, there’s really no excuse to not be fairly compensating the folks who are literally putting the words on the page for your cause.

To put it another way…

Rediscovering Music, Digitally

Posted on by Scott / 0 Comment

Recently I decided to revisit the seemingly gargantuan task of reorganizing my digital music collection.

It’s something that I’ve been putting off even longer than updating my backup plan because I honestly don’t listen to music much except for maybe when I’m in the car, even though it seems silly to only have access to about 20 albums on my phone when I’ve got upwards of 100 GB of music sitting on the server at home.

But really, therein lies the problem – I’ve found that while Plex has been my loving savior for roughly 99.5% of my digital media woes, the one area where it seems to fall short is in organizing my music because of how it identifies songs … or at least tries to, anyways.

It turns out that despite going through the steps many, many moons ago to convert all of my mountains of CDs that I acquired through the likes of BMG and Columbia House to MP3s, apparently the tags that got embedded in the files are inconsistent as all crap. It never really bothered me because I had the files themselves organized by genre, artist, and album, and I’d play everything with Windows Media Player (or WinAmp if we really want to whip the llama’s ass…).

#geeknostalgia

Anyways, it turns out that when you tell Plex to use a file’s tags, it takes that directive very seriously, even when to my regular, human eyes some of them are absolute garbage! Completely ignoring directory structure, it would mix tracks among different albums and sometimes even classify music under several different artists if their names were spelled incorrectly across the various tracks!

It sucks, which is why I’ve put the project off for so long, however lately I’ve been thinking a lot about the data I hoard and how it makes all of the sense in the world to store it in a format that’s actually useful for its consumption, so it was time to finally start addressing the problem…

…which in my case means importing one or two artists at a time and refreshing my Plex library, then reviewing the results and making any manual changes to group songs together correctly, list multi-disk sets correctly, and so forth.

After several hours of work, Plex tells me that so far I’ve added a whopping 19 artists to my library, so I’ve clearly got a long ways to go, but the plus side of all of this is that I’ve been stumbling back across all of these great songs that I used to love at various times over the years.

So I thought it might be fun to share a few here – most are from my college days, though Led Zeppelin I listened to pretty much religiously back in high school! It’s amazing how beautiful some of those guitar tracks are that I’ve completely forgotten about…

Maybe as I come across others, I’ll write a little something about select favorites and what they mean to me … seeing as that was actually the original intention of this blog post when I first started writing. 😉

A Change of Backup…

Posted on by Scott / 0 Comment

This past weekend I finally made the decision to switch backup providers from CrashPlan to Backblaze.

I think.

It seems like I’ve been using CrashPlan forever at this point – at least 5 years now – and it’s honestly something that I just setup a long time ago and left alone … like you’re supposed to with any good backup! 😉

The problem is, and it’s one that I’ve admittedly been ignoring for a while now, is that over the last couple of years CrashPlan’s price has crept up while at the same time its feature set crept down, so I honestly haven’t been getting the value out of it that I once was oh so long ago…

I believe the cost was $5/month/computer when I first started using CrashPlan, and I used it for both my laptop as well as critical files on my home server (which was cool because they had a Linux client that was really easy to use!). Then a few years later, they unexpectedly dropped home support, which was going to double the price in the long term … though in their defense, they offered a 50% discount off home pricing for one year to ease in the transition.

So basically my pricing went from $10 -> $5 -> $20 per month over a few years time!

The bigger hit was that this summer they added a special exclusion for Plex files, which was a big part of what I backed up off my server. I didn’t try to send them my entire library of dozens of TB, mind you, but it seemed reasonable to send them 20 GB of config and metadata so that I could restore Plex easily if the server bit the big one.

In total, I had something like 400 GB backed up with CrashPlan – roughly 200 GB of personal photos and writing and everything else from my laptop, and another 200 GB of Plex config data and some music and other hard to replace archived stuff from my server.

So anyways…

It’s been eating at me for a while that I needed to make a change.

I’ve actually followed Backblaze for a long time because I love how open they are with how they store massive amounts of data. I guess I always just thought that their usage-based plan was too expensive for my needs because I didn’t want to go with another $5/month plan and their unlimited plan doesn’t support Linux anyways.

The funny thing is, apparently when you’re already spending $20/month on backups, that’s enough to store about 4 TB of data using Backblaze’s B2 system!

I think part of the problem has been that whenever I looked at their pricing in the past, I always equated it to backing up my entire data collection – including what’s now 60+ TB of TV shows and movies for Plex – which in turn ends up being something like $300/month and is completely unreasonable for a simple backup strategy!

Yet after now having endured a couple of hard drive failures across my collection, I’m starting to realize that there are certainly subsets of my data that are easier to replace than others. And so instead of B2 being this out of reach backup strategy for all of my data, it suddenly became a new opportunity to go from 400 GB backed up with CrashPlan to nearly 4 TB backed up with Backblaze for about the same monthly cost.

😯

Maybe I’ll do a separate post that’s a little more technical when I finally pull the plug … CrashPlan renews again on 10/10, so I’ve got a couple of weeks to test the waters to make sure I’m truly happy with Backblaze before I cancel one account and fully commit to the other. But so far, I’m pretty satisfied.

I found this free, open source software called Duplicati to manage the backups themselves, and it was super easy to install on both MacOS and CentOS. Within about 36 hours time this past weekend, I had 220 GB from three separate machines backed up to B2, which according to their calculator will run me about $1.10/month, so that’s cool! 🙂

I still need to do some testing on restores to see how that works, but it seems fairly straightforward via Duplicati.

I think in all of my years of using CrashPlan, I had to do one restore and it was 100 GB of music when a drive failed in my server. Their client made it just about seamless, so here’s to hoping for a similar experience with the new guard as well…

Mini-Me Mario Kart

Posted on by Scott / 0 Comment

Today ended up turning into kind of a disappointing day, so instead of writing about that I thought I’d share something a little more fun…

The last couple of nights I’ve been playing Mario Kart 8 on the Wii U with David & Christopher to entertain them while Matthew has his therapy, and they’re not quite there yet, but it’s still surprisingly fun!

I’ve found that I have to take all of the controllers while we’re getting the game setup because otherwise they’ll push random buttons and back us out of whatever I just finished doing, and occasionally both of them will still get caught up with pressing the Home button that interrupts the game when they start getting bored.

But they really enjoy choosing characters because they get to pick whoever they want – even if they both end up wanting the same one, which is super helpful! Christopher has been playing Mario a lot, though he switched to Princess Peach (and knew her by name?!), while David primarily enjoys being “the monster” (Bowser) and thinks it’s hilarious to see such a huge character riding a motorcycle with a parachute that looks like himself!

I think one place where this game has really shined – on both the Wii and the Wii U – is that it looks beautiful with tons of detail in the background, so when I noticed on multiple occasions as we raced one of the beach tracks that they weren’t actually going anywhere and asked if everything was ok, I got the response, “We’re watching the fish swim by!” which was pretty cool to hear how amused they were by something as simple as the background art!

Truth be told, neither of them ends up getting very far around the track, and when Christopher finally complete a few laps the other night it was because he kept getting “rockets” (Bullet Bill items) to move him ahead, but for 15-20 minutes at a time they seem to really enjoy it, and I can get them to get along by choosing which tracks we’re going to do together, and overall they’re relatively quiet which is about the best that I’ve come to expect for a house with three boys and an obnoxious dog. 😉

I need to go back through my collection to see what others we could branch out into next.

What would you tell your past self???

Posted on by Scott / 0 Comment

I had kind of an interesting dream last night…

I had met up with my scout troop in the middle of an outing, and randomly I asked them how they fared enduring the hurricane! This was somewhat shocking to them because as we talked, we quickly came to realize that the terrible storm that I was asking about … hadn’t actually happened yet.

I woke up shortly after that, so I didn’t really get to see how this new knowledge of the future would actually play out, but it got me thinking while I was laying there trying to fall back asleep – if I was able to tell myself something at a specific point in my past, what would I say?

“I know that you feel broken and alone and hopeless right now, and I’d be lying if I said that you weren’t going to feel this way for a while, but eventually it will pass and you’ll meet someone who completes you in ways that you can’t even fathom right now, and everything is going to be ok.”

“I know that having kids is the last thing on your mind right now, but someday you’re going to be a Dad. It’s going to be a very hard road to get there and you’re going to spend a lot of money and relationships with the people around you will change, but everything is going to be ok.”

“I know that it doesn’t seem like a big deal to postpone moving across the country for just one more year, but there’s so much more out there for you than this little town has to offer and you really need to spread your wings and explore and be around a diverse populace. Take the leap – everything is going to be ok.”

I found it intriguing that regardless of the scenario that I considered, I didn’t really have much in terms of new advice to tell myself for the past. The best that I could do was offer that hint of reassurance that despite how terrible struggles like heartbreak and infertility and moving away from home seemed at the time, somehow in hindsight everything had its way of working out ok.

One of my favorite quotes along these lines right now goes something like this…

“Never forget that so far you’ve survived 100% of the worst days of your life that you’ve ever had.”

It’s still hard to think about in the moment, though, so maybe we need to get to work on figuring out time travel just the same. 😉

How to Grow a Virtual Disk… (the long way)

Posted on by Scott / 0 Comment

So to any sysadmins who do this kind of stuff on a daily basis, this is going to seem way obvious, but for somebody who doesn’t and has been struggling with this literally for months … let’s just say I’m pretty happy to finally have figured this out!

Also, this post is mostly for documentation’s sake so that I have a place to look back to when I need to do it again sometime many moons into the future…

It’s hard to believe that it’s been over a year already since I migrated my Plex server off of my old desktop hardware over to a proper rackmount server. Or at least Plex itself migrated, while the bevy of hard drives that 50+ TB of media lives on still resides in that aged and ever-waning PC.

Anyways, last June when I made the big leap to server-grade hardware, I only had a single hard drive to run VMs from for the new machine. For simplicity’s sake, I set it up as a RAID 0, single disk array, with the understanding that I could “easily” add more disks a few months later and re-configure that array into a more resilient RAID 5.

In fact, according to Amazon I did buy two more drives to use for said purpose in September 2018.

And just yesterday I finally got them working!

You see, it was probably too easy for me to setup that initial RAID 0 array via the new server’s BIOS. At the time, it seemed simple enough to add more drives to the pool and then reconfigure the array itself.

But one thing I’ve learned somewhat painfully since I first set this server up is that everything is more picky than that. Versions have to line up with the hardware, and older versions lack features supported by newer versions, even while they’re all being supported by the companies in parallel. This isn’t really news to me, but it’s certainly something that I never had to scrutinize to this extent.

With my old desktop server…

  1. Connect new hard drive.
  2. Find it in the CentOS Disks GUI, quick format it, and mount it.

With my new server…

  1. Connect new hard drive.
  2. Try to add it to my RAID pool via the RAID controller, but you can’t.
  3. Try to add it via ESXi, but you can’t.
  4. Try to connect via Dell OpenManage, but I didn’t install the server-side software in ESXi right because Dell’s support page for this server only goes up to ESXi 6.0 even though I’m running 6.5 and then I finally find the right software on a support doc found via Google.
  5. Try to connect via Dell OpenManage, but they only make a Windows client so I have to find a laptop to do that.
  6. Try to connect via Dell OpenManage, but the server doesn’t have a certificate and the login failure doesn’t mention that this is a big deal, so you just guess until you see a checkbox mentioning ignoring that and finally it works!
  7. Add new disks to RAID pool and reconfigure from RAID 0 to RAID 5 … and wait a very long time.
  8. Worry that I didn’t make backups of my VMs because I couldn’t figure out how to do it precisely the entire time.
  9. Try to expand virtual disk via ESXi now that the extra space is available, but it still doesn’t see it.
  10. Confirm via Dell OpenManage that the reconfigure is definitely done now and showing the extra space as available.
  11. Wait until 1:30am when nobody is using Plex and just reboot the whole thing, just in case.
  12. Try to expand virtual disk via ESXi, and now it sees it!
  13. Allocate additional space to new VM and reboot that VM, but it does nothing.
  14. Spend an hour Googling for instructions about how to allocate the new space inside of the guest OS until I finally found this random support post that ends up working not unlike magic!
  15. Verify that the new disk space is finally ready to use in the VM, and then debate whether it’s going to be enough or if I should’ve bought yet another disk just in case…

I mean, looking back logically it does make sense – first add the physical drives, then add them to the RAID pool, then rebuild the RAID array, then add new space to the Virtual Disk, then allocate the new space to a specific VM, then update the VM to recognize its new resources … maybe I was just hoping it would be slightly more seamless, even if only in parts! 😛

If anything, I guess it should be a tad easier the next time around, and now that I’ve gotten the bugs worked out of OpenManage, that alone is one less headache to worry about.

That said, I don’t want to rely on my work laptop for managing this server (and others down the road) indefinitely, so it also means I need to put together some sort of Windows box to sit in the corner and collect dust until it’s needed once in a blue moon…

Still, my Plex environment … minus the media itself … now lives on a cushy, new RAID 5 array that could sustain a single disk failure without missing a beat, plus I’ve got some extra cushion for downloading new stuff to boot.

Not too shabby for only ten months worth of work!