The upgrade goals when seen from the management level must have sounded fairly modest. Change the existing process flow for all transactions so they are only entered once and processed end to end without any “manual processing”.
This may not sound like a revolutionary development to bring in straight through processing starting with data entry up until final settlement. That would be really nice but that wasn’t the goal at all. Due to limitations of the existing financial system had some pretty terrible inefficiencies. One example was to enter the data into the backend system, print it out and have someone re-enter it into the front end system so it could be seen on the company reports.
In order to placate the different departments there were a lot of promises of new and better functionality that would be delivered as part of the new upgrade. After all, the vendor had made big changes since the last version of this system.
One of the first problems encountered was the upgrade of our STADS system. During the initial installation of this system the project team built an application of their own using Microsoft Access and Visual basic. This system, STADS, was intended to hold various static data but more importantly to provide the ability to populate an empty database with all of the configuration.
This system was a convenient way to upload the data during the initial project phase. It was so convenient that our client decided to use this as the official storehouse of static data for this system. The problem is that that this homebrew tool when combined with the complexity of the vendors software created a situation where it simply was no longer possible to manually maintain this setup.
A big part of the problem was the IT decision to virtually clone the table structures from the vendor system. Over the years the vendor’s product added new functionality and thus the structure of some the tables changed a little while other tables and relations were totally redesigned. The vendor didn’t have a data model for the original product, nor did they have a data model for the upgrade, and to make matters worse no good product release notes were available.
Nobody from the IT group knew up front how long it was going to take to internally upgrade the STADS solution just to support the basic functions from the previous version. Not only that but IT also had no idea how long it would take to modify STADS to support the new functionality. Each time the vendor delivered a new patch the database structure seemed to have massive changes which again caused the STADS group to re-write large portions of their system.
The goals of the upgrade ended up being like a stick of salami at a party – over time more and more of planned new functionality was being sliced off. The first casualty was the target release of 7.4. The project leaders came up with a lot of official sounding reasons why that was not feasible but the real reason was that the permission structure had changed so much and the code was so poorly documented in STADS they did not feel confident to go beyond the 7.3 patch.
Then straight through processing was abandoned mainly due to integration. The part of the vendors back end system had been replaced with something purchased from another vendor. The vendor’s integration of the two products really was to maintain separate tables with a lot of duplication that needed to be kept in sync.
Even worse, the old system had a lot of data that was no longer available in the new system. The new front end did allow data entry but it was nowhere as rich as the data the existing backend system needed. Because of this multiple payment and extract interfaces would continue to exist in both the new system including the printout and manual entry of data as well as the old system.
Another major feature to be cut from the feature list was the full reporting database. The number reporting tables were limited and the hierarchy was relatively flat but due to other technical issues all of the developers were too busy with other problems to even begin to implement this new reporting system1.
Not all of the problems were related to software. A big resource blackhole was the desire to have a lot of different testing environments – six to be exact. Three environments existed for the existing production (prod, pre-prod and test), but somehow double that number were being maintained for accounting, risk, trading, development not to mention the new pre-prod and related databases.
The tech guys were busier than a one armed paper hanger trying to keep all of the necessarily changes and fixes in sync on all the environment. When they weren’t keeping environments running they were fighting with 20+ hour upgrade scripts that should have worked but did not most of the time.
- Ineffeciencies
- Conflicting requirements
- Political infighting
- Multiple unnecessary environments
- None of the promised enhancements
The upgrade took two years, cost a lot of money, and delivered none of the new functionality.
1Actually the vendor had just added this feature. It wasn’t in use by any other client.