Our Enterprise's Finest Software Solutions


There is a big idea out there in the startup world that you should build a Minimum Viable Product, e.g. The Lean Startup tells you to do that. The general principle is to put together an unpolished product that is the core of your idea, presumably get someone to use it, and iterate from there. This is sometimes confused with a prototype or a demo that you put together and show to investors, or only manifests itself with the founders' idea that they need to build something.

Surprisingly, I have encountered very similar strategies during my short but thrilling early career in an investment bank. As teams tried to navigate the complex landscape of random requirements, unclear priorities - my boss's boss's boss actually ordained during a yearly town hall that we should be prioritising stability while delivering on new features - and random interruptions of "we can't deploy software because there's gonna be an announcement by a central bank and lots of trading" or "we can't hire anyone this quarter, because we didn't make enough money in Japan" - anyway, I'm rambling. People still got stuff done, thanks to the MVP principle: do most of what you have to do with the limited means available to you, i.e. the minimum viable effort that still gets you a promotion. I will share a couple of examples: may you find them educative and inspiring.

Fast real-time system: cache a slow system

I worked on a system that showed anyone who cares numbers about risk and P&L for portfolios in interest rate products. It had a feature that was introduced to me along with the phrase "X is the poor man's Y" by James, my manager there. The real-time numbers were sourced from upstream systems, and they were very slow. The Japanese desk wanted to see their real-time numbers faster. They requested an extension that queries a particular upstream system every five minutes and stores the result, so they got it. Unfortunately the feature was not implemented to a very high quality so there were lots of support issues about it.

What would Akka do?

We had a nightly feed into reporting and accounting systems - massively complicated operation with lots of mutually dependent tasks. After adding another domino piece to it, it started performing dramatically badly and the bosses came to know that the feed suffers from a design mistake. The task of fixing it went to Rob the consultant, the best programmer I worked with there and maybe ever, and they flew him to Budapest and hosted him in nice hotels. He did come up with a better architecture - essentially a reactive system - and successfully adapted our sprawling Java webapp. "What would Akka do? " comes from one of the progress updates he gave us and I don't remember what it would do but it was a solution to what the Java app should be made to do. Rob did not add Akka to the feeds, but implemented what he needed with thread pools, synchronized blocks, and other usual hairy concurrency primitives.

CSV Importer

There was a big new program calculating finance numbers in construction, set off to replace lots of other systems, and there was a big complicated issue about how our program would interface with it. It wouldn't be complicated anywhere else, but there were lots of extra constraints, like, nobody was allowed to call the new program or use it as a library. As a proof of concept, our program opened a port where you could tell it there's a CSV file to be read in, and this was the interface for when I was there. It spent lots of time and memory in java.lang.Double.parseDouble, and I remember Rob messing with his own Double parser, but otherwise it worked really well.

It made management quite uncomfortable that this is the best that can happen in the face of all the ridiculous constraints in place - for example, the new program stored the data in a fancy way, and you needed to interface it with a library in the codebase of the new program, but you weren't allowed to have the new program as a library. Eventually - this was unfolding in my final days there - they've agreed on some other file format to read and write to. Later Rob became the team lead and in due course achieved the next milestones towards whatever the thing was that needed to happen on this issue.

The minimally viable support strategy

y team constituted level three support for the program we wrote. Level one was a guy on the trading floor that could install it, and level two were guys in India who fielded issues and complaints about stuff. One of us rotated as level three, and we were meant to have an advisory role. It would be a mixture of randomly incoherent issues that I had to punt over after finding somewhere to blame - they wouldn't really be our problems, they were issues in the systems giving us data, confusing but correct behaviour- that could be understood and explained away, and real massive actual problems.

The actual problems were almost always in the old program that gave us numbers, which we shared the rota with, and they were really big, say, a batch job was scheduled overnight to transfer the currencies between various legal entities of the bank, shifting hundreds of millions, and would choke on bad input. I'd tell the support person to "rerun the job, with the output sent to my display" and soon the job would log to me computer through an X11 window. I could stop it and examine the variables, hack something up, and push it through. Well, the issues only got so big rarely, and I'd try to give the big ones to Krisz the old system's manager or one of the developers.

I didn't quite have what it took to be a hero, and I barely knew the proprietary dialect of APL the system was written in (later open sourced), although I had fun learning and later got more or less comfortable with it. Whatever happened, the goal was to do anything to make the issue go away. Otherwise it got escalated, and that meant being put on an increasingly larger conference call which would take real balls to hang up on. Every random person on a related support rota had to dial in, regardless of whether they knew what was going on or wanted to fix anything, so the calls took long time to establish any basic facts. Eventually a hero showed up - maybe from the next shift, the New Yorkers were waiting for the Japanese to come into the office - and did something, and then it stopped being a problem for everybody on call, and maybe some of them went home to a cold dinner and children already in beds, or an empty apartment with a single blinking microwave light.