Yesterday was an event filled day, but I'm finally here...
We started off our morning at 7AM, heading out to a plant and tree sale in Qualicum. Twice per year, a (semi) local nursery has a huge sale, with plants and trees being sold for about 50% of their regular retail prices, direct from the grower. As we'd cut down the tree in front of our house, we wanted to replace it, and this seemed like as good a time as any. We managed to pick up a 12' Sunset Maple tree for all of $35, which is a great deal. (The 15' ones were $189 regular, so that gives you some idea of the prices.) After packing our car full of rhodo's and other varieties of plants, we began to head towards Fairwinds.
The reason we were heading in to work is that we were having a grand opening of our newly renovated cafe/pub that afternoon. I was going to miss it, and it had been a long time since Dee had been to the cafe, so I was taking her to show her around. We also wanted to stop by the pro shop to pick up some logo wear for the summit, so it all seemed to tie together. But as we were heading there, things started to turn... well... you be the judge...
There was an accident on the highway. So we all came to a screeching halt, turned around through the median, and went the long scenic route through the backroads to get there. I said to Dee at the time... "There's always accidents on that part of the highway. And every time there is, it knocks out power to Fairwinds." The location is MILES away from us, but for some reason, it seems to be tied in to the grid.
As we're clipping down the backroads, my cell phone gets a text message... "UPS on battery power." Damn... Then another "UPS battery is low and will soon be exhausted." Nice timing is all I can say. We were going to stop in briefly, then head to Seattle via the Victoria ferry. At this point it is 10AM, and we have to be on the road out of Fairwinds no later than noon.
Another text message comes in... "Cold Start sequence initiated." This is good. The UPS is back online and should be rebooting.
Now, we've got our stuff set up so that the MSA (Storage Area Network Drive Array) boots back up when the power comes on. After 6 minutes, my two VMWare ESX servers boot up. After they've booted up, the system slowly starts to bring up, in order, the following virtual servers: Our two Active Directory servers, the File & Print server, the SQL server, the Exchange server, our two Citrix servers, and finally the Citrix secure gateway (remote access) server. The whole process is staged in increments, and should have the system back up and running from a cold start in about 25 minutes tops. (Yes, it seems like a long time, but we put delays in so that everything can settle down nicely, and that nothing runs into issues by being too quick.)
So with the restart sequence initiated, we arrive at the cafe. I talk to the staff, let them know that things are in process, and even stopped by a terminal to make sure I could ping my two ESX servers. All looks good and I give my wife the tour of the place. I'll admit that I was a little surprised that things were taking so long to come back, but had every confidence that it was going to work. After a while, we moved on to the pro shop.
At the pro shop, much of the same. I talked to the crew there, assured them things were under control, and picked up some shirts. I then decided to stop by my office on the way out, just to check. And this is where things went to hell in a hurry...
The ESX servers had booted, but they could not see the SAN. This is not good, as all of my virtual servers reside on the SAN. Without it, I'm kind of dead in the water. So I placed a call to my VMWare tech guy in Vancouver, and we rebooted the ESX servers. After waiting 10 minutes for a "can't sync time" error on the ESX hosts (VMWare, fix this issue!) we were finally back up to where I could log in to the VMWare Infrastructure client on the host. Although it could see the servers, it still couldn't start them.
Down go the ESX hosts again... down goes the SAN. We reboot the SAN, and it takes 10 minutes (give or take) to settle down. I reboot the ESX hosts again...
By this time, it is 11:50AM, and I have to leave... like NOW. The ESX server is about 5 minutes into the time error message, we still don't know if that's all that needs to be done to fix things, and I have to go... I called up Kelly, (her husband is a tech, but not her,) gave her a 3 minutes lesson on how to switch consoles, what server was what, and where the power was controlled from, and sprinted out the door. I jumped into the drivers seat of the car while dialing my local support firm, and got instructed by Deanna to pull over about 50 meters down the road. She drove, and I made calls.
After 45 minutes, 2 calls to the emergency support line requesting support for a critical "business down" issue and a phone call direct to the support company's president, I had a tech dispatched to Fairwinds. (I won't go into the frustration around that, but after 40 minutes, I escalated the issue to the very top of the support company, as I wasn't getting anywhere. Funny how quickly things get in action when you do that!) So out went a tech to relieve Kelly from the console. His instructions were basically to work with the lead from Vancouver, and do as instructed. The thing here is that the local support guys don't have a good knowledge of VMWare or our setup, and the Vancouver guys do. So what I needed at the time was someone on the ground with tech savvy to be able to relay messages back to the Vancouver lead, and recognize if something might be problematic.
They got the system back up an running within another half hour, and the "all clear" light was issued as I was about 1/2 hour into my drive down to Victoria. I fielded a few more calls yesterday, but nothing urgent, and now I'm just praying that things go smoothly.
We caught the Victoria Clipper down to Seattle, which was an interesting experience. We cleared US immigration in Victoria, and the guard there was, by far, the friendliest border guard I have ever met. She smiled, laughed, and was genuinely happy! A fantastic "border crossing" experience, I have to say.
When then boarded the ferry, and sat next to a collection of frat boys on their way to a drinking event, I think. Honestly, I have no idea what the heck they were going down for, but I hope that the local hospitals are equipped to deal with four serious cases of alcohol poisoning, as I see that in there immediate future. They were hilarious... the instant the boat cut loose from shore, they were at the bar. "Give us the darkest beer you've got." They returned to the table with four Heineken. LOL! I'm thinking that the Kokanee beer they had later is darker than Heineken, but whatever. They seemed to have a good time playing cards, even though one guy kept dealing one guy six cards instead of five.
They pretty much drank their way across the 2.75 hour trip, even pulling beer out of their bags after last call on the boat. And then they each bought a litre of spirits from the duty free for their party. All in all, though, funny guys. They weren't obnoxious at all, and were having a good time.
The weather for the trip was incredible. I've never seen the ocean so calm, it was beautifully sunny, and the sunset was gorgeous. We got in to Seattle at around 8:45 and cleared customs with ease. Again the customs guy was super nice and friendly. We had a short conversation about the MVP summit, (he asked where I was going,) and let us go on our way. A quick cab ride, and we're at the Westin.
From there, we headed out to dinner at the Palace Kitchen. The food was great, and then it was time to turn in, as both Dee and I were pretty exhausted from being up early, and dealing with the stress of the Fairwinds server issues.
Today's plans include a lazy morning, (already partly accomplished,) a little bit of shopping, and meeting up with Robert before heading up to the summit registration. After that we've got plans for dinner, rounding up some Excel MVP's and the Rock Bottom pilgrimage, although not necessarily in that order. 😉
