|
|
Post by MartinT on Sept 17, 2014 7:19:12 GMT
Tomorrow we will have no power to the entire site. I have to switch off everything: 18 servers, about 100 workstations, all network cabinets and switches, the lot. When we had to do this earlier this year, things broke on the power-up surge (two core network switches), so I'm not taking the chance with leaving anything on this time. I can't rely on the UPS batteries as they only give us 30 mins or so.
Tomorrow's weather forecast is cloudy with rain. It's going to be a dark, dingy day. We have our torches at the ready and will be tidying up! |
|
|
|
Post by ChrisB on Sept 17, 2014 7:23:21 GMT
Just to clarify, Martin is talking about the site where he works and not The Audio Standard website!
|
|
Deleted
Deleted Member
Posts: 0
|
Post by Deleted on Sept 17, 2014 7:32:42 GMT
I do sympathise Martin. I have to switch off my modem and Voip box oh....almost every week as the satellite feed gets clobbered by any storm within 50km of us. Its 9.30 in the morn and its like night out there now. Oh the pressure....will it switch back on? Ah...it has.  I used to get involved in huge networks in a previous existence. My nephew was digital manager for one of the big publishing houses and I used to spend happy weekends with him in his wiring hell on earth. |
|
|
|
Post by MartinT on Sept 17, 2014 7:44:05 GMT
Sorry, yes of course, my place of work and not TAS!!
|
|
|
|
Post by Chris on Sept 17, 2014 14:00:17 GMT
If it makes you feel any better I head back to the rig tomorrow so it'll be a shit day for me as well.
Oh well,always got TAS to keep me amused....
|
|
|
|
Post by MartinT on Sept 17, 2014 15:28:34 GMT
I'll be accessing TAS on the phone!
|
|
|
|
Post by zippy on Sept 17, 2014 15:54:47 GMT
Look on the bright side - at least you have the luck to know about the power outage in advance.
A while ago we had an outage of several hours - the UPS couldn't keep going that long and although most of the servers shut down gracefully, we had problems starting up again from the backups.
The most annoying thing was that the outage was scheduled, but nobody thought to tell us !
|
|
|
|
Post by MartinT on Sept 17, 2014 17:19:03 GMT
That's nasty, Chris. I would be immensely displeased with whoever had been informed. My UPS batteries only give about 30 mins per server so are not a solution for long term outages either. I have belt-and-braces backups (including a caddy in my car and cloud backup) so hopefully we'll be ok. The old adage "you can never have too many backups" comes to mind!
|
|
|
|
Post by MartinT on Sept 18, 2014 5:36:44 GMT
Today's the day! Shutdown at 8am  |
|
|
|
Post by MartinT on Sept 18, 2014 14:44:14 GMT
We're back up and survived without too high a body count: a total of three network devices and two workstations damaged. Unfortunately, we have to do it once more tomorrow for the weekend  |
|
|
|
Post by MikeMusic on Sept 18, 2014 17:21:19 GMT
Pain
Damaged how ?
|
|
|
|
Post by MartinT on Sept 18, 2014 20:08:06 GMT
Blown from the start-up surge. It's impossible to go around the site switching every single device off. When the electricity company kick the power back on, the surge is immense.
|
|
|
|
Post by MikeMusic on Sept 19, 2014 7:20:49 GMT
Doesn't anyone make a soft switch on type device ?
|
|
|
|
Post by MartinT on Sept 19, 2014 20:40:08 GMT
Yes, they're called anti-surge devices and we have them protecting almost everything. However, the switch-on surge involved when the power company kicks on the power to a site which is full of inductive and capacitive reactance cannot be prevented with such simple devices and they themselves tend to blow when hit by such a surge.
|
|
|
|
Post by MikeMusic on Sept 21, 2014 15:10:25 GMT
This is not a good thing.
What's next up, a big generator you would hardly ever use ?
|
|
|
|
Post by MartinT on Sept 21, 2014 21:04:48 GMT
That's the problem - I'm faced with batteries the size of a room or an external generator that would hardly ever be used. We almost always end up with the solution that we have: UPS batteries capable of sustaining the servers for 30 mins, sufficient to cope with most brown-outs, and anti-surge strips everywhere to save equipment from the worst of surge damage.
Anyway, I will be in extra-early tomorrow (about 05:30) to hopefully power it all up in the right sequence and have all services up by around 08:00.
|
|
|
|
Post by MikeMusic on Sept 22, 2014 8:20:19 GMT
Have you manged to get a 'sensible' UPS ?
IE. one that sounds an alarm but carries on providing power when it fails itself
I went right off UPSs when they themselves failed and shut us down
|
|
|
|
Post by MartinT on Sept 22, 2014 13:07:29 GMT
The APC range are good and will usually fail in straight-through mode if the battery fails. We have nine of them and the battery replacement costs can be rather alarming. However, they do the job and scream / send e-mail when power goes down.
All done now, network back up and this time no damage, only a couple of frozen/glitched switches.
|
|
|
|
Post by MikeMusic on Sept 22, 2014 14:18:00 GMT
Good, sensible APC UPS then
Have the leccy boys finished ?
|
|
|
|
Post by MartinT on Sept 22, 2014 16:37:03 GMT
Done! For now, but since it's a new building going up, who knows whether there might be other disruptions ahead.
|
|
