Why don’t I get my confirmations – revisited

I have never worked for the government but sometimes I feel like I do. In January the bank’s email account expired and was deleted. It was not possible to get the exact same id so a new one was requested and tested. Between the ordering, configuration, testing, and just getting it to work took some weeks. Yet to get it installed on production took a downtime and downtime’s occur infrequently. Finally the software was ready to be installed in march but the production downtime was commandeered thus forcing the bank to wait another two months for the next downtime.

Finally the software was installed in may, and so I forgot about this issue. It was a month later when I noticed on the production server that these files were still not being sent out. I tried to follow up, all according to proper channels but never did hear anything back. So one evening I decided to pick up the phone and informally call someone in that department to ask what was going on.

It seems that the person who was in charge of IT in that group had moved to another group internally and as he was the “official” owner of the database it was basically moved to his new group when he got a transfer. So the reason the new email address doesn’t work is now a permission issue because their database was migrated to another group within the company and for security reasons the different groups cannot access each others resources.

It is unclear if the database can be ever be transferred back to the bank despite the fact it is not needed or even wanted by its new group. It also may not be possible to get a new database as the company is trying to get rid of Lotus Notes yet the new email solution has been put on hold due to other technical difficulties and no budget to continue with the replacement.

Thus it is now about eight months after the initial problems and the situation has not only gotten worse but there is no real end in sight.

Why don’t I get my confirmations

(excerpt from a trouble reporting system)

The bank has let their email account expire and thus it was automatically deleted after 60 days due to the company’s standard policy. They will need to provide a new one, the configuration files will need to be amended and transported to production. Once this is done then fax confirmations will then resume.

The upgrade

The upgrade goals when seen from the management level must have sounded fairly modest. Change the existing process flow for all transactions so they are only entered once and processed end to end without any “manual processing”.

This may not sound like a revolutionary development to bring in straight through processing starting with data entry up until final settlement. That would be really nice but that wasn’t the goal at all. Due to limitations of the existing financial system had some pretty terrible inefficiencies. One example was to enter the data into the backend system, print it out and have someone re-enter it into the front end system so it could be seen on the company reports.

In order to placate the different departments there were a lot of promises of new and better functionality that would be delivered as part of the new upgrade. After all, the vendor had made big changes since the last version of this system.

One of the first problems encountered was the upgrade of our STADS system. During the initial installation of this system the project team built an application of their own using Microsoft Access and Visual basic. This system, STADS, was intended to hold various static data but more importantly to provide the ability to populate an empty database with all of the configuration.

This system was a convenient way to upload the data during the initial project phase. It was so convenient that our client decided to use this as the official storehouse of static data for this system. The problem is that that this homebrew tool when combined with the complexity of the vendors software created a situation where it simply was no longer possible to manually maintain this setup.

A big part of the problem was the IT decision to virtually clone the table structures from the vendor system. Over the years the vendor’s product added new functionality and thus the structure of some the tables changed a little while other tables and relations were totally redesigned. The vendor didn’t have a data model for the original product, nor did they have a data model for the upgrade, and to make matters worse no good product release notes were available.

Nobody from the IT group knew up front how long it was going to take to internally upgrade the STADS solution just to support the basic functions from the previous version. Not only that but IT also had no idea how long it would take to modify STADS to support the new functionality. Each time the vendor delivered a new patch the database structure seemed to have massive changes which again caused the STADS group to re-write large portions of their system.

The goals of the upgrade ended up being like a stick of salami at a party – over time more and more of planned new functionality was being sliced off. The first casualty was the target release of 7.4. The project leaders came up with a lot of official sounding reasons why that was not feasible but the real reason was that the permission structure had changed so much and the code was so poorly documented in STADS they did not feel confident to go beyond the 7.3 patch.

Then straight through processing was abandoned mainly due to integration. The part of the vendors back end system had been replaced with something purchased from another vendor. The vendor’s integration of the two products really was to maintain separate tables with a lot of duplication that needed to be kept in sync.

Even worse, the old system had a lot of data that was no longer available in the new system. The new front end did allow data entry but it was nowhere as rich as the data the existing backend system needed. Because of this multiple payment and extract interfaces would continue to exist in both the new system including the printout and manual entry of data as well as the old system.

Another major feature to be cut from the feature list was the full reporting database. The number reporting tables were limited and the hierarchy was relatively flat but due to other technical issues all of the developers were too busy with other problems to even begin to implement this new reporting system1.

Not all of the problems were related to software. A big resource blackhole was the desire to have a lot of different testing environments – six to be exact. Three environments existed for the existing production (prod, pre-prod and test), but somehow double that number were being maintained for accounting, risk, trading, development not to mention the new pre-prod and related databases.

The tech guys were busier than a one armed paper hanger trying to keep all of the necessarily changes and fixes in sync on all the environment. When they weren’t keeping environments running they were fighting with 20+ hour upgrade scripts that should have worked but did not most of the time.

  • Ineffeciencies
  • Conflicting requirements
  • Political infighting
  • Multiple unnecessary environments
  • None of the promised enhancements

The upgrade took two years, cost a lot of money, and delivered none of the new functionality.

1Actually the vendor had just added this feature. It wasn’t in use by any other client.

Green Milk

The refrigerator is small but there is more than enough space to put your salad until lunch time. I always thought that I had good taste in food, but perhaps not. On no occasion over my career as a consultant has my food looked good enough that someone took it from the fridge. Nope, it was always there when I went for it.

But milk is apparently different kettle of fish. The kitchen was small but large enough to have a place for a refrigerator, a Nespresso coffee machine and a water boiler thus making it the unofficial nerve center of the project.

I like to have a small bit of milk in my tea and milk likes to be kept at a constant cool temperature so the most reasonable thing was to store it in the refrigerator. I cannot say how long it had been going on but it seemed like my milk was evaporating at an ever increasing rate. I don’t have that many cups of tea each day and I am generally too lazy to try and track the quantity used. I may have never really caught on, except one bright egg decided to take the last of the milk and put the empty carton back in the refrigerator.

I had a chat with Marcus and Martin who also had been suspecting increased evaporation but they could not prove it either. Drastic times call for drastic measures but I was a bit hesitant to put laxative into the milk but there were a few options still open. I tried to put goats milk into a carton and that must have had some effect as they never finished off that particular carton. I tried watering down the milk, which didn’t slow them down – that carton was finished very quickly.

My favorite milk experiment was to add green food coloring to the milk. At first it was only a little and perhaps not enough that you could readily distinguish it, so the next day I really added a lot. It would have fit into a St. Patrick’s day parade it was so green. Perhaps that was a bit much as the next time I needed milk someone had thrown the entire nearly full carton away.

Penny wise and pound foolish

During the initial implementation of the companies risk system we were told to write programs and just make things work. The computers were from Sun Microsystems who also happen to be the creator of the programming language Java. We had the Java language and we also had a Unix server so we should have been ready to get to work. Yet software development usually requires a bit more than a simple editor (vi) and a Java compiler. This method of software development is hardly any different than using a simple editor to write a book.

Sure we could actually get things programmed but it wasn’t the most productive way to create programs. The good thing, or bad thing depending on your point of view, is that software developers are usually hired for their problem solving abilities. It didn’t take long for our group to quietly download some of the typical programmer tools from the Internet ( Eclipse, ant, vi, notepad++, winmerge, putty, cvs and winscp). This gave us the proper tools for developing and refactoring code in addition to managing the changes so nothing got lost. The official rules are don’t download anything from the Internet but our boss knew that we needed tools in order to produce.

I was speaking with someone from another group who was telling me about the group that develops on the internal TAP project which is an internal bug request tracking system. It seems that in addition to that group not having an enlightened manager they were also saddled with a rather odd technology. They were programming in language Delphi, there is nothing inherently wrong about using object oriented pascal to solve your problems but it turns out that this one didn’t appear on the list of company supported programming languages.

The set of tools that our group officially uses is not much better than a using flint and steel to start a fire but they were basically waiting for a lighting strike to start there fire. Because Delphi is not on the list of official technologies means that the client won’t officially purchase this Delphi development environment. Those developers simply spent their time taking notes and then trying to compile and test the changes on their laptop. This isn’t a very good solution as the database with the real world tests and bugs existed in the client’s network but it is forbidden to connect unapproved equipment to the network. I have no idea but I suspect that copying over the final executable is also breaking about 20 different company policies.

To this day, which is about four years later the client still refuses to purchase even a starter edition of the Delphi development environment. The cost of the starter edition is probably the cost of half of a single consultant day for these two developers – oh did I fail to mention that they were external consultants?

A clean cut hurts the least and heals the quickest

Back in the day my company was transferring their development staff to Chicago due to entering a new market and the staff was terribly excited. So excited in fact that productivity suffered. Yet not every one of the developers were going to be transferred. Not everyone wanted to leave London nor was everyone invited. For those that were not invited, they could see the career opportunities would be rather limited in the future and so they all started to find new jobs.

The bad news was this was around Christmas time and some of the departments were having small Christmas parties at the “pub” during lunch. Coming back to work after being in a bar was not the most productive, but the real problem was the different developers were having leaving parties for all their coworkers once they found a job – also celebrated at the pub. The party would start about lunch time in a nearby pub. The culture in London is a bit more laid back about have a pint or two over the lunch break thus having a leaving party at lunchtime is also accepted. It only took a few weeks of the entire development department going to lunch and either not coming back or they coming back after having “a few too many”.

Thus the company’s official line from that point on was that leaving parties must be scheduled after 4pm1 It probably would have been more productive to transfer everyone and fire the rest over a month period than the fun yet not so productive six month period that it took to transfer all of us.

1 Darn those American kill-joys in management

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close