Granted, I was probably long overdue for one because I can’t even remember the last time I had to toil away in my server closet / bedroom closet for more than a minute or two, but it sure would’ve been nice to do before or after I was done recovering from my kidney transplant so that I could twist and move to mess with cables and heavy servers in such a manner!
But then again, power outages come at a time when we least expect it, which is basically what happened about a week ago in the early hours of my birthday when we lost electricity for about two hours and when it came back up, my NAS decided that it didn’t want to…
For reference, the server in reference is a Dell PowerEdge C2100 – yes, it’s pretty old, but it gets the job done for now.
My first suspects were all power-related, figuring that maybe a power supply went bad when the electricity came back on or something. I did notice a battery error on one of the UPSes, which was odd because one ran out of juice and the other didn’t, but in hindsight I think that was an unrelated way of reminding me that it’s time to replace the batteries in them – makes sense, considering I bought them back in 2019!
For the longest time, I was at a loss because essentially the lights on the power supplies told me that it was getting power, but a single, non-flashing orange light on the front indicated a hardware failure that prevented booting. I started stumbling through Dell’s troubleshooting guide of unplugging components until I could get it to boot and was about ready to cry when I found a random forum post hinting to try resetting the CMOS via a really tiny jumper that was a pain in the ass to find, and suddenly it booted!
Once!
It spit out all sorts of random errors and sometimes even random characters while it was trying to boot, eventually telling me that it couldn’t find the boot device (which is a USB drive for Unraid) and then went silent again when I killed the power to try a reboot.
After a couple of rounds of resetting the CMOS to work through the various settings changes and watching them disappear shortly thereafter, it clicked that maybe it was the actual CMOS battery that had died, which was correct because once I replaced that … and also plugged back in one of the SATA cables for the backplane for the disks, it started booting fine!
But it still wouldn’t boot into Unraid.
Well, chalk one up to Oops because it turns out that *I* had unplugged the dongle for the USB boot drive when I was trying to locate that CMOS jumper, and although I had replaced it, I guess I must’ve put it in backwards or something because after reorienting it, I was able to go into BIOS and it actually saw the USB as a boot option finally!
So thankfully after a full week of dabbling with this thing a few times a day for as long as I could crouch down in the closet, it’s finally back online and now I just have to endure a 2.5 day parity check because Unraid failed to shutdown nicely when my UPS ran out of battery.
A couple of follow-ups:
- I need to look into that and make sure all of my servers are set to gracefully shutdown if they switch to UPS power for more than, say, a minute. We get a lot more flickers than actual outages where we are, so that should be the best scenario to shoot for.
- I also need to replace the batteries in those UPSes … which wouldn’t be so bad at about $25 a piece except that each one contains three batteries!
- And lastly, but most urgently, I need to verify that the right BIOS is now running because one goofy bug that I encountered when I first got this machine was that the newer BIOSes from Dell didn’t control the fans correctly, thus resulting in it absolutely screaming all of the time, which my wife does not approve of! Backing down to a specific, lower version fixes it, and I even know exactly which one because apparently I made my own forum post when I did it back in 2019, so I need to see whether the resets lost that as well or what.
That said, right now I’m very tired because it’s very hot inside of that closet, especially with one server running at full tilt, so we’ll see what all that involves and how loud it is with the closet door closed to determine whether I get to it tonight or not!
2 Comments