TL;DR

Welcome to a new blog series where we return to the main raison d’etre of Zombie Code Kill: Software Craftmanship.

I would like to begin by looking at the possible social harm caused by poor quality software development. As this is allegedly the case, is still going through legal processes with issues remaining in dispute (in the United Kingdom), I do my best to present you just the facts so that you can decide for yourself whether:

• The stellar work of Post Office Ltd has brought a number of evil criminals to justice, or
• There are some signs of a possible cover up

I’ve chosen to highlight this case because it is a big current affairs issue. I could have chosen a bug such as heartbleed which had a much bigger financial impact, but that was last year’s news. The following story is currently on-going.

Horizon

I would like to thank Dave Morgan, who, at the inaugural Software Technology Meetup raised this case study to me.

This is the story of the Post Office Ltd Horizon system accounting problems where many staff, and former staff, went bankrupt or sent to jail for false accounting while there is strong reason to suspect that the Horizon software, and problems in other Post Office computer systems, have been to blame all along.

More than 150 sub-postmasters (approximately 225 according to one source) say they were wrongly prosecuted or made to repay money because of a faulty system. The first complaints were originally made almost 15 years ago and it recently went under an independent review by Second Sight until the Post Office decided to also terminate their appointed reviewers’ employment.

“They are a faceless organisation who couldn’t give a damn about your personal situation.” – Case 21

The Post Office used the tactic of offering to drop the theft charge if the defendant’s pleaded guilty to false accounting, and the defendants were almost obliged to agree due to their employment contract, which runs to an almighty 114 pages and many say they never even received.

“Although Post Office has likened the Standard Contract to a franchise agreement, we note that the British Franchise Association recommends that a franchisee should always seek independent legal advice before entering into a franchise agreement and we are surprised that Post Office does not make a similar recommendation.” – Second Sight

The Horizon system originated in 1996 in a joint Department of Social Security-Post Office PFI deal for an automated benefits payment system with Pathway, part of ICL (now Fujitsu) and was first rolled out in 2000. The Post Office’s position has always been:

“there is absolutely no evidence of any systemic issues”

There also appears to be a very strikingly similar response from the National Federation of Sub Postmasters:

“We continue to have complete confidence in the Horizon system, which carries out hundreds of millions of transactions every week at 11,500 Post Office outlets across the country.” – George Thomson, NFSP

Perhaps it is hard to produce conclusive evidence of software errors? If it happened that the defects were since quietly rectified, then it would be much harder to show conclusive evidence of the claims of so many staff. The investigators found many of their questions on how the software used to work were answered in terms of how the software currently works.

“I can’t explain what they do to you — the bullying, the harassment,” – Sarah Burgess-Boyde

In 2009, Mr Alan Bates, a former subpostmaster in Wales, set up the Justice for Subpostmasters Alliance. Alan has been described as a hero for the tireless work he has done to help distressed ex-colleagues up and down the country.

“Why should we remain loyal to a company that has no regard for our income or well being? ” – David Willis, Ipswich

Following campaigning by this alliance, the Post Office agreed with Rt Hon James Arbuthnot MP to undertake an independent review into the Horizon system. Angela Van Den Bogerd, Post Office’s Head of Partnerships, signed the following:

Post Office is determined to ensure that Horizon and any associated processes are fair, effective and reliable, and that Subpostmasters can have confidence in the system.

Parliament has since been as surprised by as anyone else by the way events have unfolded:

“The way in which the Post Office has treated sub-postmasters and Members of Parliament who have expressed concern about the matter is so worrying, and to my mind shocking” – Rt Hon James Arbuthnot MP

The forensic accountants never had the opportunity to finish the work they started in full, but delivered their final report on 9th April this year. As well as software issue, they found problems with the interface between Horizon and other systems, lack of effective investigative support, and incomplete co-operation from Post Office.

“We have experienced significant difficulty in obtaining access to a number of documents we believe are necessary for the purposes of our investigation, notwithstanding Post Office’s commitment to make requested documents available to us” – Second Sight

“Throughout the process we all lost faith and trust in the Post Office’s willingness to investigate the issue properly and thoroughly” – Andrew Bridgen MP

As I read through the independent report, I was left wondering if I had ever seen anything like it before?

“Post Office has found no reason to conclude that any original prosecution was unsafe” – Post Office

“Is there not something disgraceful in the fact that criminal charges were pressed against these sub-postmasters and sub-postmistresses when the fault lay with the Post Office, yet nearly two years after the investigations those charges have not been lifted?” – Jim Shannon MP

…after some consideration, the only conclusion that I could come to was no I had not.

In the light of this apparent conflict of views between the Post Office and the independent body set up to administer the Scheme (‘the Working Group’) chaired by Sir Anthony Hooper, a retired Court of Appeal Judge, we would normally have asked the Working Group to provide guidance on this matter. Unfortunately, it has not been possible to do this, as on 10 March 2015 Post Office announced that the Working Group had been wound up with immediate effect. This was the day before we were due to circulate a draft of this Report to all members of the Working Group. It was also the day that Post Office notified us that our contract to conduct an independent investigation into the matters raised by Applicants was being terminated. Consequently, Post Office instructed us to issue a final version of this Report no later than 10 April, irrespective of whether or not our independent investigation was complete.

I have never worked for either the Post Office or Fujitsu, the makers of the software, and there appears to be much much more to this story than just alleged software construction errors here. The majority of the criticism has been aimed at the Post Office rather than Fujitsu.

“the P/O walked away with £75,000 of my business” – Case 15

The Post Office have had almost 15 years to address a variety of problems that their own subpostmasters approached them with hoping for a little help.

“At my interview, I was told that I would receive full training and back up from Post Office Ltd. Sadly the Post Office broke their promises and a few months later when I needed help, it was not there. Instead, all I received were demands and threatening letters.” – Case 14

“…some of the goalposts have been moved by the Post Office” – Kay Linnell, Chartered Accountant

Has the Post Office shown a much greater interested in apportioning personal blame than in due diligence? In one case it demanded over £645,000 back.

“That a national body like the Post Office can behave this way is absolutely unbelievable” – Jo Hamilton, former subpostmaster

“The Post Office mediation scheme has proven to be a sham, Second Sight has proven to be far too
independent for the Post Office to stand, and the disdain that has been shown to Members of this
House and to sub-postmasters is a disgrace.” – Andrew Bridgen MP

“Our concern overall is the hundreds of postmasters who seemingly have not had justice.” – Andy Furey, Communication Workers Union

Security concerns

When asked about the risk of malware attacks stealing large amounts of cash to be dispensed by an ATM without leaving any record, the Post Office replied that it was

“not aware of any form of fraud (including retract fraud) that creates a loss to Subpostmasters provided they follow the correct accounting procedure”

And has given many other similar statements regarding how very unaware it is. Banking Industry experts have stated that external ATM cash thefts have been relatively commonly encountered over the last 7 years. For example, last year £1.3 million was stolen from UK banks over the course of just one week. Worldwide, the estimated cost of just one criminal gang is $1 Billion.

“The ATM was producing really odd figures. I ended up with a £40,000 anomaly. I have lost my income, my reputation, my confidence. My life will never be the same again.” – Sarah Burgess-Boyde

I encourage you to read the findings of Second Sight in full, so that you can draw your own conclusions. However be aware that it is 50 pages long, and requires a good hour to absorb in full.

“We went bankrupt in April 2008, we have lost our house our business all our savings and live in rented accommodation.” – Case 12

“It has not only not been transparent; it has gone out of its way to delay cases and hide evidence.” – Mr Kevan Jones MP

However, as a software developer, I am also interested to know what has happened with those who built the system.

“Post Office is confident that there are no systemic problems with branch accounting on Horizon and all existing evidence overwhelmingly supports this position”.

“Dealing with the Post Office is like being in a nightmare, they are right and they can never, ever be wrong.” – Kevin Carter

“there has been failing after failing on the part of the Post Office” – Andrew Bridgen MP

£1 billion system

The system in question is not a cheap system. It was a multinational project involving US federal funds. At the cost approximately £1 billion, you would expect a pretty high level of sophistication and Fujitsu, who also work on the Universal Credit Project, (formerly using a doomed Agile approach) appear to agree with this and be taking all the credit.

“Throughout the process, Fujitsu has provided comprehensive multivendor management and services to ensure everything runs smoothly—from initial planning to deployment”

The independent forensic accountants found:

• Hardware and technology that is old and suffers from avoidable rates of failure
• Telecommunication equipment that is prone to failure or to poor signal reception in
  some rural locations
• Limited usability testing prior to deployment of new facilities on Horizon
• An icon based touch screen that does not auto-calibrate
• Software that does not detect and prevent password sharing or multiple logons by the
  same user at a different branch
• The lack of secure, token-based, user identification that would uniquely identify the
  actual user
• Software that does not prevent or detect suspicious out of hours transactions
• Software that does not require additional process steps such as two person approval
  or an additional approval measure for high value or high risk transactions.
• Possibility of losses, not recoverable by the subpostmaster, to occur as a result of power and telecommunication interrupts
• A failure to record transactions accurately

In short, the forensic accountants have questioned whether it is fit for purpose and concluded that in some cicumstances “Horizon could not be described as ‘fit for purpose'”

These findings agree with the statements made by many of the defendants, such as

Just a few weeks after the system had been installed an overnight software upgrade resulted in over £5000 of system generated duplications having to be reversed out of just one weeks’ balance (and those were ones I found).” – Defending Subpostmaster Case 9

This is a very complex system, and involves the sale of over 170 products. It is beyond the realm of sanity to think that the alleged defects could be entirely due to a sole rogue programmer or manager.

“…my area manager came down and spent 4 hours trying to find out what had happened. Even she could not understand the complexity of the horizon system” – Case 15

Although complex, it was be very difficult to describe all parts of it as elegant with a straight face. Horizon is just a single-currency system and only able to account for transactions in pounds sterling. I find it utterly incredible that in the year 2015, staff are still entering in foreign currency holdings into Horizon as ‘bulk totals’ in pounds sterling.

“We remain concerned that in some circumstances Horizon can be systemically flawed from a user’s perspective and Post Office has not necessarily provided an appropriate level of support.” – Second Sight

Integrity of Second Sight

“CWU is not convinced by the claims made by the Post Office in their own investigations which effectively call into question the professional integrity and competence of Second Sight. We find it unfair that the Post Office challenges this report whilst having gagged Second Sight so that they cannot defend themselves.” – Andrew Furey, Communication Workers Union

“Second Sight has been extremely fair, professional and accurate in its analysis of both systemic and thematic issues within Post Office Ltd. However, the same cannot be said of the Post Office itself.” – Andrew Brigden MP

The Big Rewrite

There is now a suggestion of the system being replaced with a new one to be created by IBM.

“there’s never enough time to do something right, but there’s always enough time to do it over.” – Melvin Conway 1968

Although this could potentially be good news, care must be taken to avoid second system syndrome. Is another £1 billion or more going to be spent?

“You’re just going to make most of the old mistakes again, and introduce some new problems that weren’t in the original version.” – Joel Spolsky, 2000

I hope that whoever is in charge with making the big decision will have a chat with or read of Fred Brooks so that the project has a better chance of getting off on the right start.

Beyond reasonable doubt

I would like to take a step back from this one case to think a little more broadly. If these or other defendants had been charged with theft and pleaded not guilty to the court, given the computer records indicating their guilt, could they possibly be considered guilty beyond reasonable doubt?

If like me, you are a software developer, you are likely to be thinking “of course not!”. Of course there is a reasonable risk that the software contains one or more defects. Software is complex, so much so that error free software is very rare. If you are ever called up for jury service on a case like this I urge you let the defendant go free.

Which raises the question, what would it take in order for it to be beyond reasonable doubt that a piece of software had serious errors?

This is of course far from an easy question. Even if every possible path through the system had been thoroughly tested prior to release (a highly rare situation for complex software) there are still risks that there might have been an error in the testing process somewhere.

Also, even if it were proven that the software adhered entirely to the specification, how can we prove that there aren’t any errors in the specification? Once again, in the case of the Post Office, the report found fundamentally flawed procedures to be part of the issue. 100% certainty of perfection for any complex software is unfortunately not realistic.

Since the early days of computing, software developers have gone from finding it very hard to produce even the most simple software, to finding it relatively easy to produce highly complex software. This is certainly not due to today’s developers being much smarter than those of 50 years ago.

It is due to newer the frameworks, languages, platforms and tooling, that frequently allows us to think at a much more abstract level. This means that if bugs exist at a lower level of abstraction, they are harder to spot.

The law must recognise the risk of errors in any digital evidence.

Lawyers and judges now deal with digital evidence regularly, even if it mainly in the form of email correspondence, the authenticity of which may not be in question… ” – Stephen Mason, Barrister

However I believe that there are a number of best practices and processes that can help reduce this risk and should be employed more often. In Part 2 we will begin to look at what Software Craftmanship is and some of the arguments both against and in support for it.

Thank you very much for reading, and please have your say.

Have you, or anyone you know, been affected by any of these events?

Do you believe that the Post Office and the NFSP are correct in saying there are no systemic issues and that the subpostmasters stole money?

Or do you believe the sub postmasters and the Members of Parliament’s side of the story?

“…there needs to be an independent judicial inquiry into this —as he described it — national scandal” – Mr Kevan Jones MP

Update 17th August: This has now been covered on BBC Panorama and is available on iPlayer:Trouble at the Post Office

Further Reading on Software Development
Melvin E. Conway’s thesis from 1968: How Do Commitees Invent
Joel on Software: Things you should never do

Further Reading on Post Office Scandal
BBC News Post Office ‘failings’
Post Office Computer Bug
Second Sight Briefing Report Part 2
Justice For Subpostmasters Alliance
UK Campaign 4 Change: Jailed and Bankrupt Because of Unfit Post Office
What Do They Know: Internal Review Response To The Refusal To Answer Yes/No To Problems With Horizon
Post Office is a disgrace
The effects of imprisonment
Stress and Suicide
Scores of Sub postmasters driven to ruin or suicide

Further reading on Fujitsu Services
Working at Fujitsu Services