So what happened? Where is everything? Why is everything 404’ing without those nice WordPress powered pages? I fucked up, other things fucked up, and it combined for the mother of all fuck-ups. (Which could also be assumption…but I’m also not hijacking a satellite.)
Just keep readin…..
So as far as failure of the old server…look, the node was kind of old. I’d been on it since December of 2017. This was also a pretty cut-rate provider that had…for the most part…been working. But the last few weeks it’d gotten a little more unreliable network wise; it also hadn’t been pushing full speed in over a year.
So I’ve gotten two stories from the provider; the first was the RAID card failed and they were unable to rebuild the raid; the second was that two disks in the 10 array failed simultaneously. Did they have backups? Someone asked me what kind of place doesn’t back this stuff up…and I have to be honest in that I’ve never expected a place to. They always charged for it.
I had my own backup solution in place…which is where my fuck-ups come in to play. Once a month the server would tar up the www directories; make a backup copy of all SQL databases, and then upload them here at home. It was literally just a cronjob set to a script to push it all over SSH. This failed when my SSH machine here at the house blew up from the lightning strike last September. So the backups weren’t getting done monthly…more like every other month…sometimes 3.
When I got my homeserver back and started getting it setup….I was in the process of reconfiguring and setting everything back up when I was struck with a major issue; hard drive failure. This time, it was the NAS drive. I found this out after hacking Linux on to the NAS appliance. AT first I thought this was just a mild annoyance; afterall….putting Linux on the thing required a complete total backup of all the data. I’d spent the previous two weeks getting my media off of there in a massive reorganization; and I’d copied all of the data on to another hard drive.
Guess what? The backup drive….also failed! So I was able to do a partial recovery of the data…but there’s still about 30% of it that’s missing. I had all this straightned out by mid-June and was ready to modify my scripts to start backing up again…but I didn’t. Why?
Because I was planning on switching servers anyway. I wanted to move away from the OpenVZ garbage and back to KVM. So I basically stopped worrying about that and focused on finding a new host and then figuring out what I’d done on my server for 3 years that I needed to restore. I planned on doing gradual migrations.
Well….we are on the new server. I went ahead and decided on a host and purchased a VPS. We moved away from the cut-rate providers; I’m spending serious money on something now; but I actually know this guy. Like I physically know the guy. Those are the services I always had the best luck with.
Of course a new server also brings in some changes on the back-end. I’ve migrated from Apache to nginx…which has caused some of it’s own problems. The new server also has an IPv6 address; so I added AAAA records to the domains and set all that up.