In a classic example of the saying "never put off until tomorrow what you can do today", the fact I had not yet finished migration of all services from the already crappy datacentre meant that there was little method of recovery from them destroying their own network.
Technical details available here, but not really a worthwhile read. The migration to the new datacentre should be (mostly) complete by my unchanged planned date of the 16th Sept, at which point we shouldn't be affected again.
p.s. bancho could have been available the whole time, was it not for the people who abused it when I had it running in offline mode. But I have learnt this is near impossible to achieve without stupidity so will not be attempting the same fallback again.
For anyone who was left in the dark, please follow @osustatus (or bookmark it at very least) for updates in case of future issues.
Total downtime was 19h 13m.
Technical details available here, but not really a worthwhile read. The migration to the new datacentre should be (mostly) complete by my unchanged planned date of the 16th Sept, at which point we shouldn't be affected again.
p.s. bancho could have been available the whole time, was it not for the people who abused it when I had it running in offline mode. But I have learnt this is near impossible to achieve without stupidity so will not be attempting the same fallback again.
For anyone who was left in the dark, please follow @osustatus (or bookmark it at very least) for updates in case of future issues.
Total downtime was 19h 13m.
. Oh well, I'm just glad that catastrophe is over.
