It is impossible to cover every possible contingency for a lot of reasons. Like programming software you don’t always know what you do not know. It is possible to mitigate some of these situations. If you expect that your data center could be flooded or destroyed by a hurricane you may have a contract with a disaster recovery provider or if you are worried about a system reliability you may have redundant features such as hard disks, network interfaces or controller cards. There is no firm in the world that can afford to have complete redundancy.
During one of my projects we had very ambitious goals for the Treasury, so ambitious that we could not implement it all at one time. Support of the productive system was handed over to a small group of people in headquarters while the project team worked on new functionality. Of course the project team was considered the second level support in case serious problems occurred in production.
Well one of the contingency plans was that if for any reason there was problems in the project building we should all reconvene at headquarters. This would ensure that second level support would always be available. It turned out that nearby the project building was a construction site for a new high-rise.
Germany, like many other countries that have experienced war, sometimes finds unexploded ammunition during construction. During the excavation for the foundation of a new building a WWII bomb was found that needed to be disposed of.
The standard procedure is to bring in a bomb disposal unit to explode the ordinance but first all buildings within a kilometer are evacuated. We were just sitting at our desks when those nice policeman came in and told us that we had to gather our possessions and leave. Just to make sure we followed instructions they stood around to make sure we packed up and left.
Our project management may have known what the contingency plan was, but it was never disseminated to most of the “second level” staff. Even if this had been done most of us did not possess the required pass so we would be allowed onto the corporate campus. It was all for the best that we didn’t go to headquarters as was originally planned because they forgot one more thing. We couldn’t have done any support due to logistics. There were no rooms for us to work in nor computers for us to work on. You cannot plan for every contingency.
I was blissfully unaware and so that afternoon I went to the pool.