How projects fail

Remember the ill fated NHS IT project? Launched by the UK government in 2002 with a budget of £6bn and scrapped in 2013 after £12bn was spent, this easily wins our vote for the worse IT fiasco ever. There have been other projects that one way or another have wasted as much if not more money, but the NHS project was by far the most galling because what they were trying to achieve was not particularly complicated. In terms of data volume it was a big project, but in terms of what it actually needed to do with the data, it really wasn't that big. It could have (just about), fallen within the scope of what HadronZoo could pull off with just a dozen developers over three or four years. Much of what was needed could have been achieved with very little software indeed. Putting 60 million or so patient records is a lot of data entry work but it was hardly a challenge in design terms as the data fell naturally into partitions suitable for deployment across multiple servers.

So how did it happen? The following four tips should shed some light on the matter.

1) Know what 'big' means and what it doesn't

One thing that is clear from the NHS debacle was that nobody working on the procurement side, knew anything about computing fundamentals. Or if they did, they kept quiet about it! As a result they fell into a simple trap. The word 'big' came into their minds and they were fixated by it. And when they walked into the offices of the various suppliers the word 'big' popped into the minds of the salesmen. Gormless ministers can be spotted a mile off and when they walk into your office looking to buy something you just know you are in for a big fat bonus. All you have to do is play to their ego and keep reinforcing the notion of 'big'.

But 'big' does not necessarily mean anything. As any child knows, you can make a number 10 times bigger by putting a zero on the end. But does adding a zero make the number harder to read or understand? Well data is a bit like that. Big data means lots of data, it does not mean complex data. The data might be complex but if it is a small volume of it will be just as complex as a large volume.

Managing the data can be complicated by large volumes because of the need for data distribution. If you cannot store all the data on one server, it has to be stored across multiple servers. Likewise if the processing of that data cannot all be done on one server then this also has to be spread across multiple servers. And multiple server scenarios are a whole different ball game. None of it is rocket science however. Data distribution can be tricky at times but in the case of the NHS system this was not really an issue or at least, it need not have been because of the obvious natural partitions. Where it is an issue there is only one answer - think through carefully how to partition things to get as close as you can to meeting the objectives. And to do this you have to know the computing fundamentals. Know how the CPU works. Know how the disks work. Know what the networks can handle. Know how the data is actually manipulated inside the machine.

2) Beware of managers returning from courses bright eyed and bushy tailed

Without mentioning anybody by name we know of one case where a manager who thought he was a system architect (he actually had that title in his employment contract and a salary to match), attended a check-box management course where he was indoctrinated with the idea that developers could not be trusted to write 'big programs'. Upon his return he was full of enthusiasm and immediately issued an edict to all the developers, that from now on all system functionality must be implemented with very small programs that only did one thing and then exited. Two weeks later he was sat in the director's office being asked to explain why a particular screen was taking a whole minute to populate when previously it had taken less than two seconds. Instead of one program that ran all the time, populated many different screens and logged into the Oracle once upon start-up, they now had more than 60 very small programs that ran every time the screen was invoked, with each one logging onto the Oracle and populating one field before exiting. What a donkey! Not a man who knew too much about the computing fundamentals, it would seem.

3) Never believe that a project is too complex for any given individual to navigate

Lets pick on something we are all familiar with, the human body. Now if you want complexity, you have it right there! The megabytes of data in our DNA. The many trillions of cells. The intricate chemistry. The auto-immune system. The eyes. The brain. Scientists are not just a long way from understanding how it all fits together. They are a very very long way from this state of affairs. In fact they have only really gotten started. The body is way beyond what an individual can fathom but is it beyond what an individual can navigate? For things that are not known by anyone, yes. But it is relatively easy to navigate the vast volume of knowledge science has uncovered about our bodies. If anything goes wrong, most of the time it is pretty easy to get an accurate diagnosis.

But the body is not a project so lets pick something that is. The large Hadron Collider (LHC). Probably our most complex project to date but much less complex than the body! And it is much easier to navigate. All the teams that plan the experiments, do so in the full knowledge of how the collider was put together. They confer on the detail of course but that is to save time. If they need to they can check anything they want in the design documentation. They can do this because they all understand the science behind all the components. They understand the fundamentals.

And big data projects do not compare to the LHC on complexity - either in terms of the number of components, the number of types of components and certainly not on the fundamental science behind it. Boolean logic and data object models are much simpler than quantum mechanics and general relativity! So when you hear phrases like "need to know basis" you need to know what they are hiding and why!

4) Remember that software is not stone and cannot be cast as such

A project becomes a 'mission impossible' when either the development regime or the object model (the collective sum of data classes and processes), or both are not fit for purpose, and where this state of affairs coexists with a perception that some or all aspects of the regime or model are 'cast in stone'.

The clue is in the name. It is called 'software' for a reason and absolutely nothing whatsoever about software, can in any way be said to be cast in stone! Providing the objectives are not mathematically impossible, the project cannot be a mission impossible. It can only be crippled by a poor development regime imposing onerous methods upon developers or an ill fitting object model. And there is only one answer. Fix it! Worse case scenario? You go back to square one. You probably won't have to go back that far but even if you do have to start again, it will still be worth it. At least you will know how not to do the project! Better late than never and you might not even be late. Once you get the right methods and the right object model, things can slot into place quite quickly.

So how can you know what the right methods are and the right object model is? Real simple. KNOW THE FUNDAMENTALS!