5.1 This chapter considers the common law presumption in the law of England and Wales that ‘In the absence of evidence to the contrary, the courts will presume that mechanical instruments were in order at the material time’. The Law Commission formulated this presumption in 1997.¹ The concept of ‘judicial notice’² is also considered in this chapter.

5.2 The reasons given by the Law Commission for the introduction of this presumption make it clear that the words ‘mechanical instruments’ include computers and computer-like devices – even though computers and computer-like devices are not mechanical instruments. Judges have, although not exclusively, used the term ‘reliable’ in relation to computers, and lawyers have also bypassed the use of the word ‘reliable’ by using the word ‘robust’. The purpose of this chapter is to consider the introduction of a presumption of ‘in order’ or ‘reliability’ or ‘working properly’ in relation to mechanical instruments generally, and to explain why the term ‘reliable’ in relation to computers and computer-like devices is not accurate, although we now have definitive evidence from computer scientists that the words ‘reliable’ and ‘robust’ as used by lawyers and judges have been exposed as not having the meaning attributed to them by the legal profession.¹ It must be emphasized that the examples of the failure of computers and similar devices discussed in this chapter are provided to demonstrate the problems that occur, and do not represent the totality of illustrations that could be used, nor the volume of errors that have occurred or will occur in the future. It is suggested that judicial notice be taken of these examples, particularly because they contradict the presumption that computers are ‘reliable’.²

The purpose of a presumption

5.3 The aim of a presumption, which allocates the burden of proof, is to alleviate the need to prove every item of evidence adduced in legal proceedings, to reduce the need for evidence in relation to some issues and to save ‘the time and expense of proving the obvious’.¹ In an appeal before the Supreme Court of South Australia, Travers J explained the rationale in the case of Barker v Fauser² regarding the accuracy³ of the readings of a weighbridge:

5.4 This explanation justifies the rationale for the presumption that mechanical instruments were in order at the material time. However, it appears that this presumption exists on the basis of expediency. In admitting evidence from a mechanical instrument or similar device, judges have not justified the presumption on the basis of relevant scientific evidence, but have substituted for it concepts such as ‘common use’, ‘ordinary experience’ or ‘substantial correctness’.

5.5 Consider the accuracy of a watch. Just because a watch has passed tests of accuracy at one moment in time does not preclude its mechanical parts from failing subsequently. In Cheatle v Considine, Travers J put the discussion of the accuracy of mechanical instruments into its overall context:

Presumptions and mechanical instruments

5.6 The presumption that scientific instruments work properly has a long history.¹ For instance, scales benefit from the presumption.² Timing devices also take advantage of the presumption. In Plancq v Marks,³ in an appeal against conviction for driving a motor car in excess of the speed limit of 20 mph, the evidence of the police officer was challenged. The stopwatch used by the police officer was produced in court. The appeal focused on the ground that the police officer gave opinion evidence as to the speed of the vehicle. This appeal was dismissed on the basis that the police officer was merely giving oral evidence of the actions of the stopwatch, which did not constitute the giving of opinion evidence. The real issue was whether the police officer was telling the truth.

5.7 Arguments that a watch used to prove that the defendant was speeding ought to be tested have been ignored,¹ as in the case of Gorham v Brice.² The Lord Chief Justice dismissed the appeal against conviction for driving a motor car in excess of the speed limit of 12 mph without considering the point. In comparison, the members of the Divisional Court in Melhuish v Morris³ allowed an appeal against speeding because the speedometer of the police vehicle had not been tested for accuracy.⁴ The court in Nicholas v Penny⁵ subsequently overturned this decision. Lord Goddard CJ commented:

5.8 The judge went on to suggest that because the defendant was accused of exceeding the speed limit by 10 mph, it ‘would be a considerable error in the speedometer if it were as much out as that’.¹ Such a comment was not intended, it is suggested, to create a presumption that such devices are reliable, especially as Lord Goddard CJ commented that ‘the justices need never accept any evidence if they do not believe it, or feel that for some reason they cannot accept it’.² A similar issue arose in the case of H. Gould and Company Limited v Cameron,³ where the pressure in the tyres of a heavy motor vehicle was tested in July and found to be over the legal limit. The instrument used to test the tyre pressure had itself been tested in March of the previous year, and in August in the year following the reading. The defence argued that the instrument might have developed an error after being tested in March. It was known and accepted that, at certain pressures, the device would be in error of 1 lb over a range of tests between 70 lb and 100 lb. This error had been taken into account in this case. Northcroft J said:

5.9 The observations by Shadbolt DCJ in the New South Wales case of Re Appeal of White¹ put the matter into perspective when hearing an appeal for exceeding the speed limit, where he noted, at 430:

5.10 It does not follow, however, that every measuring device is accurate.

Judicial formulations of the presumption that mechanical instruments are in order when used

Judicial notice

5.11 There are a number of reasons for the doctrine of judicial notice:¹ to expedite the hearing of a case where obvious facts do not need proving; to promote uniformity in judicial decision making and to prevent the possibility of a decision which is demonstrably erroneous or false.² Brett JA summed up the concept in R v Aspinall: ‘Judges are entitled and bound to take judicial notice of that which is the common knowledge of the great majority of mankind and of the greater majority of men of business.’³ In the High Court of Australia,⁴ Isaacs J emphasized the guiding principle of the doctrine:

5.12 The practical approach was considered in Commonwealth Shipping Representative v Peninsular and Oriental Branch Service¹ by Lord Summer:

5.13 The doctrine of judicial notice is restricted to very clear knowledge,¹ and it can be more severe in its effect than a presumption, as noted by Susan G. Drummond:

5.14 Given that it appears as if this doctrine has been extended to electronic evidence in Canada, this observation by Drummond illustrates the importance of ensuring judges more fully understand the nature of the world in which they now live. Thorson JA discussed judicial notice in R. v Potts before the Ontario Supreme Court, Court of Appeal:

5.15 In R. v Find,¹ before the Supreme Court of Canada, McLachlin CJC, at [48], held that the threshold for judicial notice is strict:

5.16 The concept of ‘notorious’ is considered in Phipson:

5.17 The judge can conduct her own research, and the United States Court of Appeals, Ninth Circuit reached conclusions regarding automatic programs in this way, as in U.S. v Lizarraga-Tirado, where Kozinski CJ said:

5.18 In justifying judicial notice, David M. Paciocco comments: ‘If a court could not rely on a notorious and incontrovertible material fact because it had not been proved, verdicts would not conform to reality. The repute of the administration of justice would be harmed.’¹ Paciocco went on to illustrate his argument with the following example of how a brake on a motor vehicle operates:

5.19 There is a distinction between the purpose of a brake on a motor vehicle (which is the fact in issue in the above illustration) and how the braking system operates (if the fact in issue is whether the brakes actually worked). In the example above, Paciocco made assumptions about how braking systems work and failed to understand the nature of the technology. Most braking systems in motor vehicles are controlled by a mix of electronic systems and software code (a fact so notorious that no citation ought to be required¹). It is more accurate, using a high-level functional description of the brake system, to explain the braking technology in vehicles as involving the use of brakes primarily under the control of electronics or software code. The failsafe fallback strategy for most modern brake systems is that if the electronics or software code fails, the system reverts to a standard hydraulic brake system. It does not necessarily follow that the function is always performed correctly or as normally expected in the situation where the action is mediated by electronic systems. For instance, anti-lock braking systems (ABS), electronic stability control (ESC) and traction control are predicated on interactions between the engine torque output and brake control on individual wheels and so on (such as using data from accelerometers). This means that there is a possible difference between the fact that a braking event took place and whether or not a braking event was requested, and vice versa.² This example is far from the strict application of the doctrine as noted in the Supreme Court of Canada by McLachlin CJC. If judicial notice is extended to such an extent, then the question of whether justice is served by this doctrine must be carefully scrutinized.

A ‘notorious’ class

5.20 In the Victoria case of Crawley v Laidlaw,¹ Lowe J considered, at 374, the basis upon which a presumption might apply – in this case regarding a scientific instrument:

5.21 The prosecution sought to adduce evidence from two weighing machines called ‘loadometers’ to prove a motor truck was carrying a greater weight than that allowed by the regulations. The Police Magistrate who heard the case had dismissed it on the basis that there was no evidence to demonstrate the correctness of the instruments. On appeal, Lowe J concurred, holding that there was no evidence that the devices were scientific instruments, and there was no foundation for a presumption that the instruments worked properly. Emphasizing the need to establish a foundation for the presumption, Lowe J observed:

5.22 Herring CJ made comments similar to Lowe J’s in the Victoria case of Porter v Koladzeij.¹ This case involved the review of the refusal of a Stipendiary Magistrate to admit evidence of an analogue device to measure the amount of alcohol in a sample of breath. The judge observed that certain instruments of a scientific or technical nature fell into a ‘notorious’ class that by general experience are known to be trustworthy.² He placed a speedometer into this class. However, the evidence from the device to measure breath alcohol was rejected because it was not a standard device, and because the evidence given by the witness regarding the device was not adequate. The judge said that once breath analysis devices were used more often, they would become standard, and then judicial notice would be taken of their existence as scientific or technical instruments,³ although it was necessary to present relevant evidence to the court:

5.23 The failure to obtain such evidence can lead to scenarios such as that described by Thomas E. Workman below:

5.24 In this context,¹ it is relevant to consider the decision of the Supreme Court in New Jersey in the United States, which ordered the software of a breath-testing device to be reviewed in detail in the case of State of New Jersey v Chun.² In his judgment, Hoens J began by stating that: ‘For decades, this Court has recognized that certain breath testing devices, commonly known as breathalyzers, are scientifically reliable and accurate instruments for determining blood alcohol concentration.’ This comment was based on the old technology. With the introduction of a new device, the Alcotest 7100 MK111-C, which was selected by the department of the Attorney General, the court agreed to test the scientific validity of the machine. After extensive testing, the court concluded that the Alcotest, utilizing New Jersey Firmware version 3.11, ‘is generally scientifically reliable’, but modifications were required to enable its results to be admitted into legal proceedings.³ The testing of the software revealed the following issues, among others:⁴

5.25 The analysis of the source code indicated that there was a fault when a third breath sample was taken that could cause the reading to be incorrect, and the court saw fit to order a change in one of the formulae used in the software. Save that the extensive analysis of the device and the source code took some time and some expense, little of substance was found to be wrong with the machine. However, there are two significant points that arise as a result of this case: the first is that the software that controlled the device, written by a human, was defective, which in turn meant that the data relied upon for the truth of the statement was defective and therefore affected the accuracy and truthfulness of the evidence; and the decision by the court to intervene by ordering certain changes and modifications to be carried out, one of which was a change in a formula, meant that part of the evidence used against drivers in the future would be a set of instructions provided by the Supreme Court of New Jersey.¹

5.26 However, it is not necessary to rely on a presumption that an instrument is accurate or reliable in lieu of other evidence that the data produced by the instrument is accurate.¹ For instance, a satellite navigation system was the subject of discussion in Chiou Yaou Fa v Thomas Morris² before the Supreme Court of the Northern Territory of Australia. In this case, the commander of the vessel established his position by using the satellite navigation system, radar and sextant. The court accepted the evidence that a variety of methods were used to establish the position at sea, including the expertise of qualified navigators. Even though the court heard their testimony as to the accuracy of the satellite navigation system, it concluded that it was not necessary to determine, and therefore rely upon, the satellite navigation system as being in the ‘notorious’ class, and accepted the radar and sextant evidence in its place.³

Common knowledge

5.27 Another justification for accepting that a mechanical instrument is in order when it is used is the assertion that it is a type of instrument that is commonly held to be – more often than not – in ‘working order’. In discussing mechanical instruments, it does not appear that lawyers or judges have ever concerned themselves with how the instrument has been maintained, or have considered the maintenance history of the instrument. In a case before the full court of the Supreme Court of Western Australia, Zappia v Webb,¹ the question was whether an amphometer, used to determine the speed of a vehicle, could be considered an accepted scientific instrument. Jackson CJ discussed this as follows:

5.28 The Chief Justice referred to the ‘common knowledge’ of the use of amphometers without referring to any evidence to demonstrate that they were reliable. He also asserted that somehow it was generally accepted that the device would give a reliable reading of speed (without discussing whether the amphometer was calibrated, and if so, to what standard) and concluded that because he was not aware of any complaints about the devices, they were therefore to be considered an accepted speed-checking device.

5.29 In Castle v Cross,¹ the prosecution relied on the presumption that mechanical instruments were in order when they were used. In the judgment, Stephen Brown LJ cited a passage from Cross on Evidence (1979)² regarding this presumption:

5.30 The Latin tag omnia praesumuntur rite esse acta means ‘all acts are presumed to have been done rightly and regularly’ or ‘all things are presumed to have been done regularly and with due formality until the contrary is proved’. Such a presumption cannot operate in a vacuum, as indicated by Stephen Brown LJ’s preference for the above formulation in Cross on Evidence, which requires the basic fact – proof that the instrument be one of a kind which is common knowledge that they are more often than not in working order – to be established before the presumption could operate, as opposed to the same formulation of the presumption in Phipson on Evidence, which did not adopt the basic fact.¹

5.31 In this case, counsel for the Crown put forward the case that the device in question, a Lion Intoximeter 3000, was a sophisticated machine that depended in part on software code, but this did not set it in a different class from other sophisticated mechanical devices and instruments. The presumption stood unchallenged because the defence ‘argued forcefully that the potential for computer error renders the consideration of evidence stemming from a computer particularly sensitive and places it into a separate class in relation to its admissibility’.¹ It is unclear from the judgment of Stephen Brown LJ whether His Lordship relied on the presumption in admitting the printout from the Lion Intoximeter 3000, because the central issue in this case appears to be the admissibility of the printout as real evidence.

5.32 The case of Anderton v Waring¹ also concerned the reading from a Lion Intoximeter 3000. In giving the judgment of the court, May LJ stated that the ‘Intoximeter ought to have been assumed by the justices to have been in good working order unless the contrary was proved’.² Counsel for the prosecution cited from the fourth edition of Cross on Evidence:³ ‘In the absence of evidence to the contrary, the courts will presume that [mechanical instruments] were in order at the material time’.⁴ However, the barrister omitted to continue, and cite the basic fact that ‘the instrument must be one of a kind as to which it is common knowledge that they are more often than not in working order’.⁵ This has to be a misapplication of the presumption, because a presumption cannot operate in a vacuum without the basic fact or facts. In addition, the manufacturers of intoximeters (and almost all forms of software) refuse to share their code, so there is no way to establish any such basic fact or facts – in addition to which, the US cases (discussed above) illustrate that such devices are not reliable.

5.33 A more recent reformulation of the presumption has been articulated by Kerr LCJ, as he then was, when he rejected the suggestion that the machine in question ought to be commonly known to be – more often than not – in working order. In Public Prosecution Service v McGowan,¹ Kerr LCJ said:

5.34 Kerr LCJ’s deviation from the formulation of the presumption, which requires proof of the basic fact, is unwarranted. Furthermore, Kerr LCJ’s formulation of the presumption without the basic fact leads to the extraordinarily broad assumption that all devices and machines are operating properly and in working order, an assumption for which His Lordship did not cite any relevant evidence in support. In particular, there was nothing in the judgment to indicate what he understood by ‘equipment’, or how the equipment was ‘properly constructed’, nor did he provide any evidence as to what he meant by ‘operating correctly’ or ‘proper setting’.¹

Evidential foundations of the presumption

5.35 It is suggested that the correct articulation of the presumption for mechanical instruments is as follows:¹

5.36 This formulation is consistent with Crawley v Laidlaw¹ and Porter v Koladzeij² in that if the presumption is to be recognized, it is necessary for the proponent to provide sufficient evidence – the basic fact – to merit the introduction of such a presumption. It this respect, it is pertinent to note the observation by Lord Griffiths in Cracknell v Willis³ that ‘“trial by machine” is an entirely novel concept and should be introduced with a degree of caution’.⁴ He went on to indicate that it would be unthinkable that somebody should be convicted by a machine that is not ‘reliable’, although he did not make it clear what he meant by ‘reliable’.

5.37 Conversely, in DPP v McKeown (Sharon), DPP v Jones (Christopher)¹ Lord Hoffmann voiced the opinion in 1997 that ‘It is notorious that one needs no expertise in electronics to be able to know whether a computer is working properly’.² This comment, akin to the ‘aura of infallibility’,³ is an extreme view that is contradicted by the evidence, and did not bear a great deal of scrutiny at the time the comment was made. The observation by Lloyd LJ in R v Governor Ex p Osman (No 1), sub nom Osman (No 1), Re⁴ is of a similar nature:

5.38 The judge did not indicate what evidence was before him to demonstrate that there was no ‘internal evidence of malfunction’. Just because a bank or a stockbroker will rely on computer data as part of its records, it does not follow that a judge should accept that such records are what a party asserts they are. Indeed, Professor Seng observed that such comments made by judges are ‘extravagant judicial statements … [that] are incomplete and are actually misleading because accurate computer output depends not just on the proper operation of computers, but also proper human use (or abuse) of computers’.¹ There is a significant difference between functioning ‘correctly’ – meaning, as intended – and being correct, namely that the intentions of the programmers were correct and free of any errors.

5.39 The ‘instrument in working order’ relies on the presumption that transitions between ‘being in working order’ and ‘not being in working order’ are reasonably rare.¹ In other words, the instrument cannot capriciously alternate between giving correct readings and incorrect readings, with arbitrary lengths of the sequences of correct and of incorrect readings. These arbitrary sequences happen rapidly and often with software. Although there is generally a reason for these sequences – something in the exact values and timings of the sequences of inputs determines which outputs will be correct and which ones will be wrong, given the defects in the software – identifying the law that governs them and the software defects causing a problem may be impossibly time-confusing, even for well-equipped experts.

How judges assess the evidence of devices controlled by software

5.40 When discussing the admission of evidence from devices controlled by software code, judges do not distinguish between a single, highly specialist device that is self-contained and a linked network containing any number of devices each independently operating on its own set of software code. As noted above, when considering cases dealing with specialized devices such as breath-testing machines and blood-testing machines, judges have used nebulous terms in the absence of scientific analysis, such as ‘notoriety’, ‘common knowledge’ and ‘properly constructed’. There is little evidence to demonstrate that proper evidential foundations have been adduced to permit such presumptions to be admitted. In this regard, it is useful to consider, although not exclusively, the case law in Australia, where these devices have been subjected to stricter judicial analysis.

5.41 The Southern Australian case of Mehesz v Redman¹ concerned the method of analysing a blood sample. At trial, the Special Magistrate categorized the blood sample-testing device as a scientific instrument with the presumption that it was in the category of a ‘notorious’ instrument whose accuracy is presumed. On appeal, Zelling J rejected this on the basis that the device was not a mere calculator, although the interpretation of the data was a result of its software program. There was no evidence to demonstrate that the machine was accurate or reliable. The appellant was tried a second time, convicted again and appealed to the Supreme Court once more. This appeal was referred to the full court.² The main argument of counsel for the appellant related to the evidence tendered by the prosecution regarding the analysis of a blood sample, in that the evidence relied on the use of two instruments (a gas chromatograph and the ‘Auto-lab system 4B’ data analyser) whose accuracy had not been established. King CJ rejected the submission that the Auto-lab was an instrument that could not be relied upon because there was no evidence as to the ‘correctness’ of its software program. He said:

5.42 Proof of the accuracy of a particular instrument will ‘ordinarily be proved by those who use and test it’, and the results obtained are acceptable in evidence ‘provided that the expert witness has himself formed an opinion that the methods used are apt to produce the correct result’.¹ Notwithstanding the inability of the operator of a machine controlled by software code to demonstrate the accuracy or otherwise of the code that he does not control and has no ability to alter, this proviso is important. (White J also made a similar point.²) This means that the operator of such a machine ought to be able to assess when the machine produces results that are not expected, even if the operator is not able to establish why those results are wrong. If a machine produces results that are not anticipated, the operator is put on notice that the machine (and the software code) might not be reliable. In such circumstances, it will be necessary to have the machine tested before it is relied upon for future analysis.

5.43 Dealing with the submission that the prosecution failed to provide proper foundations for the Auto-lab analyser, White J set out the conditions that must be fulfilled before evidence will be admitted regarding the measurements of scientific instruments:

5.44 At the second trial, the prosecution called evidence from Professor Northcote, Chairman of the School of Mathematics and Computers at the Institute of Technology in South Australia, and an expert in mathematics, physics and computers. He gave evidence about the workings of the Auto-lab from his reading of the manufacturer’s manual and his understanding of the content of the manual. He was not able to read the software code, because the manufacturer had sealed the program against inspection, tampering and modification. Although Professor Northcote was not an expert in relation to the Auto-lab, the members of the Court of Appeal in the Supreme Court were of the opinion that both Professor Northcote and Mr Vozzo, who gave evidence at both trials, were sufficiently qualified to give evidence, even though neither witness had access to, nor any knowledge of, the software code. The Chief Justice also stated: ‘It is sufficient that the expert who uses it is able to say that it is an instrument which is accepted and used by competent persons as a reliable aid to the carrying out of the scientific procedures in question and that he so regards it.’¹ He also prayed in aid the observations of Wigmore on Evidence² to support this comment:

5.45 In this case, the court emphasized that there was evidence other than the trustworthiness of the software code that enabled the evidence from the machine to be admitted as being accurate. White J set out the following analysis of the problem:

5.46 By implication, the court concluded that it would be extreme to establish the reliability of a software controlled device in a court of law by analysing the software code – the very software code that controlled the device and provided the evidence. The court considered that evidence from the operator of the device was sufficient for the trial court to assess the accuracy of the evidence. The appeal was dismissed.

5.47 Given these comments, it is understandable that the court reached the conclusions it did in Mehesz v Redman (no 2). At issue was a self-contained device that was used by trained operators with suitable qualifications. On the basis that the readings from such devices were, at any time, not within the expected range, the suitably trained and qualified operators were expected to use their professional judgement to verify the reliability of the device before submitting the evidence for legal proceedings. In such a case, the court would not require the software code to be challenged.

5.48 The case of Bevan v The State of Western Australia¹ illustrates the approach taken when considering the admission of evidence from computers and computer-like devices. One of the grounds of appeal in this case was the admissibility of mobile telephone data in the form of text messages downloaded by a computer software program. An investigating police officer carried out two separate downloading operations using two separate tools, Cellebrite and XRY. At the beginning of the trial, counsel for the accused objected to the text messages being received into evidence. The trial judge held that the text messages were admissible. Questions were raised as to the reliability of the software and of the officer’s correct use of it. The Court of Appeal concluded that the trial judge erred in law in admitting the text messages into evidence. This was because the officer did not explain the process of how he downloaded it in any detail at trial: it was the first time he had used the relevant software, and he did not have any formal training in its use. When considering the rebuttable presumption at common law as to the accuracy of ‘notorious’ scientific or technical instruments, Blaxell J said that ‘when evidence from a new type of scientific instrument or process is adduced for the first time, there must be proof of its reliability and accuracy’.² He went on to say that:

5.49 Blaxell J approved of the observations by White J¹ in Mehesz v Redman (no 2) as noted above. He continued:

5.50 In essence, Blaxell J is saying that if the user of a smartphone can give evidence to demonstrate that he can use the smartphone, it follows that he is sufficiently knowledgeable to give evidence indirectly that the software code that controls the device is ‘working properly’, ‘reliable’ or ‘accurate’. It is as if the software programs that form the device are irrelevant. Additionally, no attempt was made to define how software code can be determined to be ‘working properly’, ‘reliable’ or ‘accurate’.

5.51 In Bevan v The State of Western Australia, the Court of Appeal heard a second appeal in the same case after a re-trial.¹ The same argument arose regarding the method of downloading the data from the mobile telephone. There was a trial within a trial concerning the evidence of Detective Tomlinson. (Buss J referred to him as a First Class Constable, and set out his qualifications.²) Counsel for the appellant conceded that the witness was qualified to operate the equipment used to perform the download, but argued that he was not qualified to give evidence about the accuracy of the download material and the reliability of the material itself. In cross-examination, Detective Tomlinson explained he did not hold a certificate in relation to the Cellebrite and XRY software packages, but that he had been shown how to use them on about ten occasions. The following exchange took place regarding how the software worked:

5.52 In deciding to allow the evidence before the members of the jury, the trial judge said:

5.53 Pullin and Mazza JJA agreed the trial judge did not err in overruling the objection to the tendering of the text messages. In essence, because Detective Tomlinson was qualified as an expert, he could testify about the performance of the machines and the software. It was inferred that as an expert (in the opinion of the court), he considered the process to be accurate, and that because he had performed such actions previously, the actions undertaken on this particular occasion were properly performed – even though the user of the program will not know that it was giving inaccurate results.¹ There was no requirement for the Detective to understand how the software worked, or whether there were any problems with the software he used.² Pullin JA said: ‘His evidence provided sufficient assurance that the results produced by the machines were reliable and accurate, because he (a trained operator of the machines) observed them to be so.’³ But it does not follow that any operator of an electronic device will be able to detect if the device was malfunctioning in any way. As noted by Eric Van Buskirk and Vincent T. Liu:

5.54 They suggest there are two approaches to resolve the problem in the abstract of the paper:

5.55 The important question is:

5.56 More significantly, the question should be:

5.57 In addition to which, it is necessary to allow for human factors: such as whether an operator focusing on getting a job done has the cognitive capacity to notice errors. It is well known that errors cause ‘interference’, which makes them very hard to recall even if they were noticed – that is, noticing and interpreting the error requires a different sort of thinking than doing the main task, so it interferes and makes it harder to do either properly.

5.58 In the minority, Buss J considered that none of the relevant basic facts and circumstances were proven. The judge considered the applicable legal principles in detail. He cited the relevant case law, and also extracts from The Science of Judicial Proof (3rd edn, 1937, para 111) by Professor Wigmore:

5.59 The judge continued:

5.60 And logically, as Professor Thimbleby has indicated,¹ (3) the appropriateness and correctness of the use of the instrument as used in the particular case.

5.61 Buss J rejected the evidence of the constable, partly because he was not qualified to comment on the software and because the ‘machines/software’ were not so well known that their accuracy may be assumed as a matter of common experience.¹ Evidence was required to demonstrate their accuracy. It followed that the State had to produce evidence from a suitably qualified expert of the trustworthiness of the machines and software in general, and of the correctness of the particular instruments for the purposes of downloading data from mobile telephones.² Arguably, had the State produced sufficient evidence to convince a judge of the accuracy of the machines and software, it would not have been necessary to reply on the presumption. Notwithstanding this observation, the approach by Buss J is to be preferred. His brother judges appear to accept the astonishing conclusion that not having any knowledge of how a device works is irrelevant to the results of the analysis. In their approach, the work of software programmers is immaterial. Software code is not germane when determining causation. If this approach were accepted, no longer would decisions in legal proceedings be based on relevant evidence.

5.62 Contrast this decision to a similar set of facts discussed by the United States Court of Appeals, First Circuit in the case of U.S. v Chiaradio.¹ The Federal Bureau of Investigation (FBI) used a software tool called LimeWire, a commercially available peer-to-peer file sharing program that enables users to transmit files to and from other members of the LimeWire network. The FBI adapted this software for the purposes of investigations into abusive images of children. It was called ‘enhanced peer-to-peer software’ or EP2P. The software adapted by the FBI differed from LimeWire in three principle respects: (1) the software permitted downloading from only one source at a time, thus ensuring that the entire file was available on the computer of the accused; (2) in the commercially available version, LimeWire responds to a search term by displaying the names of the available files, file types, and the file sharers’ IP addresses, whereas EP2P displays the same data and the identity of the Internet Service Provider (ISP), together with the city and state associated with the IP address sharing a particular file, and (3) EP2P was modified so that an agent could easily compare the hash value of an available file with the hash values of confirmed videos and abusive images of children.

5.63 The defence requested discovery of the source code at an evidentiary hearing before the District Court. The application was refused. The purpose of the request was to determine whether the reliability of the technology could be credibly challenged; the defence argued that the inability to examine the source code prevented the accused from mounting such a challenge. The District Court denied the motion to compel discovery of the source code and the Appeal Court agreed with the District Court. Agent P. Michael Gordon testified that the software had no error rate; he demonstrated how the results of an investigation could be independently verified, and that the software had never yielded a false positive. The court considered that this alone provided sufficient evidence of the reliability of the tool. The defence also cited the lack of a peer review, but the Appeal Court indicated that the Daubert¹ factors were not a definitive checklist, and there was a sound explanation for the absence of peer review:

5.64 The evidence in this example enabled the court to resist the discovery of the source code on the basis that the software was proven to be ‘reliable’ in respect of the specific purposes for which it had been developed, although it is not clear what evidence of its correctness, if any, was offered.¹ That no errors (for example there were no false positives) are found does not mean that there are none.

Mechanical instruments and computer-like devices

5.65 The discussion in this chapter focuses on software code that provides instructions. In the case of firmware, which is software that is incorporated into hardware, the absence of visible programs does not mean that software is absent: the commentary in this chapter applies equally to this form of implementation of software.

The nature of software errors

5.66 It can be said that a computer can be both ‘reliable’ (but not infallible) and yet perform functions without the authority or knowledge of the owner or software writer. This may be when the code executes in a way, because of a strange or unforeseen conjunction of inputs, which neither the owner nor the writer had imagined. For instance, one Jonathan Moore designed and produced forged railway tickets that were accepted by ticket machines controlled by computers. It took a ticket inspector to notice subtle differences in the colour and material of the ticket, which led to his arrest and prosecution for forgery.¹

5.67 It is important to understand that programmers are aware of the limitations of their software, as famously articulated by Ken Thompson:

5.68 These comments are decidedly relevant, given that Thompson demonstrated how to create a C program fragment that would introduce Trojan horse code into another compiled C program by compromising the C compiler. Thomas Wadlow explained this process as follows:

5.69 The description could have continued. The Trojan, as described above, remains easily detected: there is one in the source code of the compiler and one in the object code of the compiler. In Thompson’s scheme, he went one step further: modify the compiler to insert the compiler’s Trojan. Now the source code Trojan in the compiler (which inserts the Trojan into the login) can be removed. Furthermore, as Thompson points out, you can now remove all trace of the Trojan in all source code. There is now no readable evidence of any Trojan attack.¹

5.70 Just because a person is in physical control of a computer or shop cash till,¹ it does not follow that she will be aware whether it is working ‘reliably’, ‘properly’, ‘consistently’, ‘correctly’ or ‘dependably’.² As indicated above, even the writer of the software will not be in such a luxurious position. It therefore follows that the following comment by Kerr LCJ was not correct:

5.71 That software code is imperfect and remains so may be illustrated by the comments of an early pioneer in computing, the late Professor Sir Maurice V. Wilkes FRS FREng:

5.72 This observation has been repeated many times since.¹ Programmer errors are caused by a mix of novelty (applying software to previously unsolved problems) and the difficulty of the tasks software is required to perform, including their magnitude and complexity.² And the errors reach back. Programmers use programming languages, and the languages are themselves subject to errors; there are errors even if the programmers are somehow perfect and ensure there are no errors, because other programmers further back will have left errors. This is why software has always been released in new versions: primarily to correct previously unknown (or ignored) errors.

5.73 To address this problem, the approach of many of the existing software safety standards is to define requirements for and put constraints on the software development and assurance processes.¹ Using the taxonomy of the provision of services, Algirdas Avižienis and colleagues have defined a ‘correct service’ as one where the service implements the system function. Its failure is an event that occurs when the service does not do what the function provides. This deviation is described as an ‘error’. For instance, if the function when using an ATM is to dispense the correct quantity of cash, and the ATM dispenses the correct amounts of cash, then there is a correct service, and the service is carried out in accordance with the function. If the amount of cash withdrawn from an ATM is greater or less than the amount keyed in, or no cash is provided, this is a service failure that can be an error or fault. The authors go on to say:

5.74 For instance, an ATM might provide a receipt that £100 has been withdrawn, but does not dispense the money. Given this set of facts, clearly a fault has occurred. One reason might be that the sensors or the software code (or both) in the machine failed to detect the lack of movement of cash. The bank might provide a printout of the machine’s internal functioning that shows the purported balance of cash held in the machine before the transaction, and again after it. This proves very little. In the New York case of Porter v Citibank, N.A.,¹ a similar set of facts occurred. The customer used his card, but no money was dispensed. Employees of the bank testified that on average machines were out of balance once or twice a week. From the point of view of evidence, the information on the printout is restricted to a single transaction. For the bank to prove that the machine actually dispensed £100 (and therefore the customer is lying), it is necessary for the bank to balance the ATM and report the results for the material time. The overall balance might indicate that it had gone down by £100. But the report might be inaccurate. This is because of a number of associated variables, such as (this is not an exhaustive list): there are multiple layers of outsourcing, the fact that people cover up mistakes and the fact that people rely on other people to be diligent in dual-control tasks (whatever they are). Equally, if the machine happens to overpay someone else by £100, the error will cancel out the previous error and the end result might not have been detected by human intervention either. Human cross-checks may suggest that everything appears correct, but the system is failing repeatedly. A further reason for the machine to be in error is that a third party may have successfully inserted code to bypass the software in the machine, leaving the thief to recover the cash after the customer left the scene.²

5.75 For all these reasons and more, it is difficult to show that a computer is working ‘properly’, even for highly skilled professionals.¹ Part of the problem is that computers fail in discontinuous ways (they cannot fail slightly), which is a characteristic of discrete complexity, unlike most mechanical devices.

Why software appears to fail

5.76 People across the world increasingly depend on computers and computer-like devices for mundane uses such as recording (cameras and recorders on mobile telephones), for critical uses such as lifesaving devices that control delicate medical equipment in hospitals and for important infrastructural uses such as systems for the supply of gas, electricity and fuel, underground trains,¹ buses² and financial software that assesses risk in financial products.

5.77 In the light of the ubiquitous nature of software, it is important to be aware that software code can function as intended by the programmer, but it can also be the cause of failure. Alternatively, software code may fail to function in the way the programmer intended, or it might continue to function but undertake actions that the programmer did not originally intend or instruct the device to undertake. Problems can occur for a number of reasons, such as where software code has a mistake, or because of improper installation, or because the people hired to undertake the work were not sufficiently qualified.¹ A range of consequences might follow, such as the failure of air traffic control systems² and lost baggage from baggage handling systems in airports,³ preventing couples from obtaining mortgages because of incorrect records,⁴ dispensing more cash than is recorded via faulty software in ATMs,⁵ miscalculating assets in family cases,⁶ and causing injuries and deaths.⁷ The increasing complexity of software and interconnections act to exacerbate the problems that occur.⁸

Classification of software errors

5.78 The word ‘bug’ is a term commonly used in the information technology industry to describe a variety of issues.¹ When a technician uses this term, it can have a number of meanings.² Professor Thomas offered his view at a lecture he gave in 2015:

5.79 Lay people, not without some justification, consider the term ‘bug’ to be a cloak that hides the correct meaning, namely that what is being described is an error, flaw, mistake, failure or fault in a software program or system.¹ Drawing from the work of Professor Ladkin, it is possible to classify most software errors into the following non-exhaustive categories:² human errors in coding and software development; software design or specification errors; unintended or unanticipated software interactions, input data flaws and deliberate errors caused by operators or hackers remotely.

Human errors and biases in the software code

5.80 Notwithstanding the best software development tools that catch and identify coding errors, human errors in writing software code account for a large number of software errors. This problem is going to be exacerbated, given the increasing size of written codes. An example of human error in software code is that of Mariner I, the spacecraft that was sent to Venus and launched on 22 July 1962. The software code indicated that the booster had failed, and the rocket was destroyed on command from the control centre. In fact, the rocket was behaving correctly and it was the computer system on the ground that was at fault, partly because of a defect in the software and partly because of a hardware failure. The error in the software arose because the person who wrote the software failed to include an overbar in the guidance equations.¹

5.81 Two further examples are the Clementine mission and the Ariane 5 failure. The Clementine mission was a joint project between the Strategic Defense Initiative Organization and NASA. After the spacecraft left lunar orbit, a malfunction in one of the onboard computers on 7 May 1994 caused a thruster to fire until it had used up all of its fuel, leaving the spacecraft spinning at about 80 rpm with no spin control. The spacecraft remained in geocentric orbit and continued testing the spacecraft components until the end of mission.¹ In the case of the Ariane 5 rocket failure in 1996, the disintegration of the rocket 40 seconds after launch was due to a software failure – because, in the words of Professor Les Hatton, ‘the programmers had arranged the code such that a 64-bit floating point number was shoe-horned into a 16-bit integer’.² As pointed out by Professor Ladkin, ‘Code was reused from the Ariane 4 guidance system. The Ariane 4 has different flight characteristics in the first 30 seconds of flight and exception conditions were generated on both inertial guidance system (IGS) channels of the Ariane 5.’³

5.82 Human bias has also begun to be more fully understood, especially when analysing systems marketed as artificial intelligence, usually developed with a form of machine learning.¹ Hidden biases and flawed datasets are, in all probability, normal.²

Failure of specification

5.83 The problem might not be in the software code, but with the specification,¹ such as with the loss of the Mars Climate Orbiter spacecraft in 1999. On this occasion, the failure resulted from not using metric units in the coding of a ground software file. The thruster performance data used in the software application code entitled SM_FORCES (small forces) was in Imperial units instead of metric units.² Roy Longbottom, Head of the Large Scientific Systems Branch of the Central Computer Agency, observed that:

5.84 It is a pervasive characteristic of software code that design will not be quite correct or coding errors will be present, and there will be occasions when a fault cannot be replicated.¹ At the beginning, the attitude taken by NASA towards software code was to consider it of secondary importance. Although this view had changed over time, and a rigorous methodology has since been implemented to provide for the better control and development of software code, NASA has never produced error-free software code.²

Unintended software interactions

5.85 Software code might function correctly, as intended by the programmer, but the interactions between individual components of the software code can be the cause of failure, because the designers of the system fail to account for all potential interactions. This is because the possible number of defects in software relates not only to the components (lines of code), but also to the number of ways in which they interact – the number of interactions increases faster than the number of components, thus making large systems with many components proportionally harder to get right. As the work of Bianca Schroeder and Garth A. Gibson demonstrates, the more complex the system becomes, the more likely it is that different types of failure will occur,¹ and the number of reasons that complexity causes failure also increases.² To put the problem into perspective, it is necessary to understand not the number of defects per device but the proportion of design decisions that contain defects.³ A typical design decision in software looks like this:

5.86 Illustrating the point with this simple example means that each design decision creates at least two choices for the software to handle, and further design choices will have to be made within the ‘do something’ bits, as well as in the ‘do something else’ bit as needed. One decision will have 2 choices, then as it is developed, 4, then 8, 16, 32, 64 and so on, increasing exponentially in complexity. Very quickly the choices go beyond human comprehension. This demonstrates that in software, a very few decisions rapidly create a far more complex thing than humans can reliably analyse and about which they can be confident they have made the right decisions, in even a modest fraction of the possible cases.¹ Since there are typically thousands of design decisions in the software for even relatively small products, there will be millions and millions of design choices, and hence it is easy to overlook hundreds of defects in the final products.² An average defect level of one to five defects per thousand lines of code could translate into hundreds if not thousands of defects for devices that have several hundred thousand to a million or more lines of code.³ This is the typical size of most software that controls aircraft,⁴ motor vehicles and many other common systems. The user is affected by how often the software fails. This is because one defect may cause failures frequently, and another defect may very seldom cause failures.⁵

5.87 Consider, by way of example, the 2003 power outage that affected large portions of the Midwest and Northeast United States and Ontario, Canada. The outage affected an area with an estimated 50 million people and 61,800 megawatts of electric load, and power was not restored for four days in some parts of the United States. Parts of Ontario suffered blackouts for more than a week before full power was restored. The subsequent investigation indicated a number of failures, a significant one being the failure of a computerized energy management system, XA/21 EMS. This system failed to detect the tripping of electrical facilities. After weeks of testing and analysis, a software coding error was discovered. It was a subtle incarnation of a common programming error called a race condition,¹ brought to light by a series of events and alarm conditions in the equipment being monitored. The race condition involved times measured in milliseconds. Mike Unum, manager of commercial solutions at GE Energy, explained the problem: ‘There was a couple of processes that were in contention for a common data structure, and through a software coding error in one of the application processes, they were both able to get write access to a data structure at the same time … And that corruption led to the alarm event application getting into an infinite loop and spinning.’²

5.88 This issue is further magnified by what are called ‘legacy’ systems. For instance, the computer systems used by airlines are very complex. There are a number of reasons for this: airlines introduced computer systems in the 1950s; as airlines merge, or take over other airlines, they might combine or adopt the computer systems they have inherited; over time, as new functions are added, this process has created systems of great complexity. The banking sector has the same problem. Replacing such systems is not an easy decision, because it would take a considerable amount of money and time, and it is doubtful whether any IT firm has sufficient skills and knowledge to provide all the software needed for a complete replacement.¹

5.89 Consider a practical example. The display on a screen has a meaning, and if that meaning is not veridical, then an accident may result. Where the moon rising over the horizon causes a system to interpret it as a massive ICBM launch, semantic safety is violated: that is, the display (it might be a warning signal or something else) was not veridical. This problem has been linked to the possibility that a nuclear war has been averted by human intervention despite computer warnings of imminent attacks at least twice.¹

5.90 It should be observed that the increasing use of machine-learning systems complicates this issue, because the software code is instructed to make further decisions when running, which increases the complexity. In addition, the veridicality of machine-learning systems, like neural nets, cannot be easily understood or verified.¹ Machine learning (ML) systems can learn (correctly or incorrectly) after they have been programmed: the errors they can make will typically not have been subject to the sort of scrutiny we expect of standard non-ML software. In particular, ML systems are easy to fool. There is a whole field of ‘adversarial ML’ which seeks training data to teach ML perverse things. One commonly quoted example is to spray STOP signs with innocuous-looking graffiti, and sign recognition software used in cars will read the STOP sign as 40 mph;² a more recent example is to mislead the software in autopilots by inserting split-second images into roadside billboards.³

Input data flaws

5.91 There are also what are known as ‘input-data flaws’, meaning that the data entered into the machine was not correct, thus ensuring the information coming out is also incorrect – colloquially known as ‘garbage-in-garbage-out’. In a well-designed system, the software should check, insofar as that is possible, that the input data is not wrong, corrupted or unexpected, and subject the output to a warning, perhaps via the user interface. This is a common problem in fairly simple systems such as databases, even in critical uses as in the medical field.

Operational errors

5.92 Another manifestation of human error would be operational error. Professor Leveson observed that it is ‘often very difficult to separate system design error from operator error: In highly automated systems, the operator is often at the mercy of the system design and operational procedures’.¹ The accuracy of this comment applies to virtually every automated system that includes computers and software code, and has, indirectly, caused significant loss of life. For instance, ‘user interface errors’ have been blamed for several aviation accidents, where the pilot as the user did not do anything wrong, but did not know the correct way to do what she wanted to do. Even in situations where people are part of a controlled and trained user community, such as ambulance controllers or air traffic controllers, human error rates in many tasks are high enough to stress systems in ways that are unpredictable. Examples of such situations in high stress industries are further explored in the rest of this chapter.

The development, maintenance and operation of software

5.93 As general-purpose computing systems have become more powerful and flexible, users have devised new uses for them in ways that the systems developers never envisaged. This, coupled with the increase in complexity and the speed at which computers work, especially in modern automated systems, means that developers can never completely anticipate how users will use their software, or how their software will interact with other systems and software. Even where the developers have tested their systems in the ways that most users use them (and possibly fail to test them against less conventional methods of use), they may subsequently need to issue upgrades that provide more functions, or updates to remedy any defects that have been found. In doing so, the developers will have modified the software and its operating conditions. Such changes will result in new modes of operation that have not been previously tested, causing the users to encounter defects they have not previously experienced. This problem is compounded when complex operations such as banking systems are connected to networks and can be attacked by hackers – internal or external to the organization – or just simply affected by unintentional actions of third parties, such as making errors during recovery from backups.

5.94 While it might appear that exhaustive testing could be the answer to these problems, it is impractical and does not necessarily work, and there is no workable theory that would constitute an adequate test. Professor Thomas notes that ‘The main way that software developers assure the quality of their work is by running tests, even though computer scientists have been saying for the past forty years that testing can never show that software is secure or correct’.¹ For even relatively small systems, the number of possible test cases required for comprehensive testing is enormous. It is also not always certain whether or not the software has passed or failed the test, and it is necessary to repeat the tests after all software changes. Furthermore, a single test case can only expose a system to a very specific set of conditions and data values. The number of variations is, in practical terms, unbounded because a robust test must consider, among other things, different data values, the number of simultaneous jobs running, the system memory configuration, the hardware configuration, all of the connected devices or systems, the operators’ actions, user errors, data errors, device malfunctions, and so forth. However, just because testing is a complex affair does not mean that testing should not be carried out.² This is so especially when people can be killed or injured,³ as in the case of the sudden unintended acceleration problems experienced by owners of some modern motor vehicles which operate with electronic control systems.⁴ Michael Barr, in giving evidence as the expert witness for the plaintiffs in the trial of Bookout v Toyota Motor Corporation Case, gave the following in oral testimony:

Developmental issues and software errors

5.95 In examining the nature of a software fault, even at a time when software was less complex than now, Professor Randell and his colleagues made the following astute observation:

5.96 Professor Randell also commented that ‘What is significant about software faults is, of course, that they must be algorithmic faults stemming from unmastered complexity in the system design’.¹ This is a telling observation, in that the primary source of software errors lies in its development process. There are numerous issues in the development of software that will generate errors, including but not limited to the speed at which a developer is required to work to write proprietary software within the contractual time frame, the consistent failure within the industry to provide for suitable quality control procedures, the creation of a climate of fear to suppress concerns relating to errors and safety,² and the lack of knowledge that programmers may have of the domain in which the software is to work (for instance, the programmer might be knowledgeable about mathematics, but have no knowledge of how acceleration systems work in motor vehicles³). In addition, it is extremely difficult to develop good software without well-designed and mature engineering processes, and impossible to do so consistently. Such processes involve the production of essential documents that enable effective communication between members of the development team and those who will accept, install, use and modify the software. The existence of such documents does not guarantee that the software is of high (or adequate) quality, but the absence or lack of rigorous quality control is a strong indication of poor-quality software.⁴

5.97 In addition, unrealistic estimates of how long it will take to write and test software also undermine accuracy,¹ which means that those responsible for writing software code will not have the time or resources to be comprehensive in developing the software.² It is also necessary to have a comprehensive design that has been subjected to peer review that should precede any coding. Often, the writing of lines of code remains the ready and easily quantifiable measure of progress, which means that writing code starts much too soon, and too little emphasis is placed on good design.

5.98 This is not to say that all software programmers are incompetent or that they do not wish to undertake work of a high quality. In their writings about software errors, Algirdas Avižienis and colleagues define ‘human-made faults’¹ as including faults of omission, and wrong actions that lead to faults of commission. ‘Human-made faults’ are, in turn, divided into malicious faults and ‘non-malicious’ or guileless faults. These faults can be introduced during the development of the system by a developer or during use by an external third party. Guileless faults can be classified as faults resulting from mistakes, and deliberate faults that are brought about because of poor decisions – usually caused when choices are made to accept having less of one thing in order to get more of something else, for instance to preserve acceptable performance, or because of economic considerations. Developers who commit such faults may unintentionally or deliberately violate an operating procedure with or without understanding the consequences of their action.²

5.99 The main part of the problem is that writing software is an exceedingly difficult and challenging field, and the methods used by management to control quality are not necessarily the most competent that can be used. However, writing software is now getting easier. Advanced development environments generate some code automatically, although writing software to perform complex functions that works well in all circumstances continues to be demanding. Many amateurs have had the experience of being able to build software that achieves impressive effects with very little effort. This may well lead them to believe that because they find it easy to program a simple video game or puzzle-solver (whose failures do not matter and will probably go unnoticed), or some simple program that seems reliable enough for their personal, everyday use, then completely finishing or building other complex software systems that are correct must be just as easy.

5.100 A further barrier arises when an organization is collectively incompetent.¹ This in turn means that inherent problems in software used in large organizations may not be identified for a long time. For instance, in 2003 Oates Healthcare began to use a new software product that was written for the company. At that time it was not known that the code written by the programmer was defective, in that it failed to calculate overtime for employees correctly. The problem was identified when a previous employee took legal action against the company five years after the software was implemented. As a result of discovering this problem, the company had to undertake two exercises. First, the simple solution was to write new software code to permit the program to begin calculating overtime correctly from the point in time that the software was amended. Second, because the changes to the software were not capable of affecting the previous calculations, the previous records had to be recalculated manually, which is an admission of poor programming: it would not be necessary do it manually if a computer could do it. Apparently there were over 10 million records that needed to be recalculated.²

5.101 A software project can fail partly because of a combination of the failure of management, an unrealistic time frame to develop the software, and a failure to develop and test software properly. There are many examples of such failure, and more importantly, some failures do not come to light until after the project is complete.¹

Increasing the risk of errors through modification of software

5.102 Software typically goes through modification cycles, called updates or upgrades, to fix existing errors in code or enhance or improve software functionality. One of the major causes of software failure is that, as software code is modified, each modification is capable of increasing the risk of failure. Some of the changes that are meant to fix errors may create another one, resulting in a greater or smaller probability of failure. Where a vendor releases a significant number of new features or a major redesign, there is typically a sudden increase in the probability of failure, after which the risk is reduced once further error updates begin to resolve the errors discovered, thus reducing the risk again over time.

5.103 It is useful to observe that when safety-related software code is modified, there is usually documentation to explain how this risk has been reduced, although this is routine only in the case of dangerous failures, and not necessarily all failures. By way of example, consider the case of Saphena Computing Limited v Allied Collection Agencies Limited¹ in which Mr Recorder Havery QC commented:

5.104 The problem with this remark is that proprietary software code can be (and indeed often is) affected by updates, which means it does not necessarily stay ‘fit for purpose’. It can also be affected by updates in other code, for instance – and quite commonly – in updates to the operating system on which it runs. Flaws can become manifest at any time, and some flaws can remain for years, which means if they are detected by a malicious person or state agency, they can be manipulated for purposes other than that which the users intend. There is a more fundamental flaw in the statement that ‘it stays fit for its purpose’. If the software is used unchanged for a different purpose, which may be no more than the original purpose but applied to different data, it may still fail.

5.105 This is illustrated in the Heartbleed exposé.¹ Cryptographic protocols are used to provide for the security and privacy of communications over the Internet, such as the World Wide Web, email, instant messaging and some virtual private networks. A current protocol is called the Transport Layer Security (TLS). To implement this protocol, a developer will use a cryptographic library. One such library, which is open source, is OpenSSL. In 2011, a doctoral student wrote the Heartbeat Extension for OpenSSL, and requested that his implementation be included in the protocol. One of the developers (there were four) reviewed the proposal, but failed to notice that the code was flawed. The code was included in the repository on 31 December 2011 under OpenSSL version 1.0.1. The defect allowed anyone on the Internet to read the memory of any system that used the flawed versions of the OpenSSL software. It was possible for a hacker using this flaw to steal user names and passwords, instant messages, emails and business documents. No trace would be left of the attack. The attack did not rely on access to privileged information or credentials such as username and passwords. Taking into account the length of exposure, the ease by which it could be exploited, the fact that an attack would not leave a trace and that it is estimated to have affected up to two-thirds of the Internet’s web servers, this weakness was taken very seriously. On 7 April 2014, the same day the Heartbleed vulnerability was publicly disclosed, a new version that applied a fix to the flaw was released.

5.106 Software can also be affected by changes in the environment, such as the operating system or other components, rather than any specific application, although it is necessary to distinguish between modification of software in situ and the reuse of software in an environment that is presumed to be similar. An example is the Ariane 5 incident, where a malfunction arose from a changed environment and assumptions that were poorly understood, rather than a defect in the original development. Where the software is modified in situ, the environment does not change; where software is reused in an environment that is presumed to be similar, the software has not changed, but the environment has. The results in either case are that there may be a mismatch where there was none before.

5.107 Generally speaking, programmers who modify someone else’s code often do not fully understand the software, and may also be less well trained than the people who originally wrote it. Software can (if appropriately designed) be relied upon to produce verifiably correct results, but to have such a degree of certainty, it is necessary to be assured that the operating conditions remain identical and that nothing else malfunctions. Peter G. Neumann has indicated that even though the utmost care and attention might be devoted to the design of a system, it may still have significant flaws.¹ This was illustrated in a 1970 report edited by Willis H. Ware. The authors noted, under ‘Failure Prediction’ within section V System Characteristics, that:

5.108 The authors of the report went on to observe the following in Part C, Technical Recommendations:

Security vulnerabilities

5.109 Software vulnerabilities are software errors generally hidden from view. While they generally cause users no harm, they may be exploited by state security services, malicious hackers and professional thieves for various advantages, including theft of personal data (to sell on), control of vulnerable systems, drug smuggling,¹ blackmail and other forms of financial gain. The market in selling packets of software code known as ‘exploits’ has become significant. Legitimate businesses may sell a vulnerability in a software code to business and government agencies, and hackers may sell a vulnerability to anyone who will buy them. These vulnerabilities, particularly those against which there are no pre-existing defences, known as ‘zero day exploits’,² may be exploited, whether legally or illegally, for criminal investigation as well as for the purposes of cyber espionage, including the violation of confidentiality (stealing information), availability (denial of service for political intimidation or blackmail) and integrity (corrupting information to steal from banks or to cause an embedded computer system to cause accidents).

5.110 To address these vulnerabilities, software vendors often, but not always, issue ‘security patches’ regularly each month (sometimes referred to as ‘software updates’ to conceal the nature of the update) in recognition of the failure of their software. Yet these may give rise to more problems. For instance, an important security weakness was discovered in relation to the distribution of software patches (which, ironically, was put in place to address security weaknesses). This meant that attackers who receive the patch first might compromise vulnerable hosts who have yet to receive the patch.¹

5.111 Software security vulnerabilities are particularly pertinent to businesses and industries that operate or rely on digital security infrastructures. For these industries, there are other issues to consider. The first is whether the design of the security protocol is robust. An example of a failure in this category is with banking systems,¹ and although a design can be modified, at best it is only possible to take a provisional view in respect of this point, because designs constantly change and are therefore liable to failure. The second is whether the security protocol is implemented properly. For instance, a number of ATMs were tested around Cambridge in the UK, and it was found that nonce generation was predictable. A nonce is supposed to be a unique object in a protocol, a one-time ‘security code’, but it was found that some ATMs were using a small supply of tokens as nonces and reusing them in a predictable order, thereby compromising their security.²

5.112 Furthermore, security may be associated with safety. If there is a safety-related system with security vulnerabilities, it is possible for the safety functions in the system to be deliberately subverted and give rise to a safety issue. For instance, the nuclear industry has developed a draft international standard for safety and security.¹ The vital problem in this area, which nobody has solved, is that while updates of safety functions in code that control nuclear reactors are slow, deliberate and highly analytical, updates for security purposes have to be rapid, to forestall anticipated attempts via zero-day exploits. These two modi are obviously incompatible.

5.113 It follows that software security vulnerabilities expose the software to manipulations without the authority or knowledge of the software vendor.¹ Many of the vulnerabilities arise specifically from the errors in the original implementation. For instance, it might be possible for a person to control another owner’s computer as part of a botnet² or enter the control system of an aircraft in flight via the in-flight entertainment system.³

5.114 At this point, the reader might consider that such problems can be solved fairly easily – by the introduction of anti-virus software (this is not to imply that all attacks occur through the use of malicious software). But it must be understood that the fundamental nature of most anti-virus software limits its effectiveness – and the anti-virus software itself might not be error-free. A sophisticated attacker will have access to all types of anti-virus software, and he will program round the detection mechanisms and test his code against the anti-virus systems to ensure it is not detected.¹ Most anti-virus software is reactive, in that it searches for known threats. As such, anti-virus software is far from perfect. It can fail to stop some malicious software² and should not be relied upon as the sole method of securing a computer.

5.115 It is a truth universally acknowledged that the majority of hackers concentrate on the most widely used software and on vulnerable applications that can be found by using Internet search engines. The development of the Stuxnet virus illustrates that governments are now probably responsible for some of the most effective viruses that are written, although organized criminals can be equally effective.¹ Software need only include a low number of defects to create enough vulnerabilities for hackers to manipulate them to their advantage. Jim Nindel-Edwards and Gerhard Steinke usefully sum up the position:

Software testing

5.116 Most software organizations test their products extensively, including in the ways that they anticipate that their customers will use them. Indeed, most software has become so complex that in a process called beta testing, software has been provided to volunteers to test before it is sold as a product. It has also been suggested that the problems of the composition of components in large systems can be mitigated by programmers reusing components in ways that they know from experience tend to work,¹ although this view is not generally accepted.² However, there will continue to be malfunctions, because many problems in hardware, software and configuration are only exposed when the system runs under real workloads.³ A number of issues arise in this respect, including the use of tools to test software fault tolerance or robustness,⁴ the degree to which the testing accurately reflects the way users will actually use the software, the unconventional ways in which people may attempt to use the program and testing how the software works when connecting and communicating with different software and hardware. It is well known that testing software is inadequate for uncovering errors, because there is never enough time to cover all the cases, as the illustrations mentioned in this chapter vividly show. Professor Thimbleby has indicated that the only solutions are:

5.117 The problem with a presumption that a computer is deemed to be ‘reliable’ is that, as systems become more complex, it has become progressively more challenging to test software to reflect the way the users will actually use the product. This is because of the large number of functions that software is required to perform and the unpredictability of its users.¹ Professor Partridge reiterates the point that ‘no significant computer program is completely understood’,² and goes further by indicating that systems are now so complex that humans are no longer able to deal with the problems:

5.118 This weakness is now recognized by some of the organizations that produce devices and software. Microsoft and Apple are among a number of companies that have adopted a ‘bug’ bounty programme to reward professionals who test and find errors in the software.¹ The US Department of Defense has also taken this approach, as has Google in respect of cryptographic software libraries.² Yet claims that software code and hardware products have been independently tested does not necessarily lead to the conclusion that they can be relied upon. In his ACM Turing Lecture of 1972, Professor Dijkstra said this of testing:

5.119 In other words, good-quality testing might discover the failings of the developer, but is less capable of resolving the issues in the overall design of software: there are significant limits to testing.

Writing software that is free of faults

5.120 As Professor Thomas indicates, it is possible to design and develop software so that it is almost completely free of faults.¹ Many applications are now built without the developer writing any code at all. The coding is done in building the tools that generate the code when given parameters by the developer – and this is premised on the fact that the software tools that generate the code are themselves error free.

Software standards

5.121 Where an organization produces safety critical software for aeroplanes, motor vehicles, air traffic control or power stations, it will be necessary to conform to the requirements of an international standard on the functional safety of programmable electronic systems.¹ For instance, security in the banking sector relies on certification standards such as FIPS-140 Information Technology Security Evaluation Criteria (ITSEC) and the Common Criteria for Information Technology Security Evaluation. It should be noted that these schemes only focus on aspects of security, and not on overall functionality. It is possible to have an accredited product that implements the security functions well, but its business functions badly.

5.122 The ITSEC scheme, which is no longer as active as it once was, assesses a product based on a document prepared by the organization that wants that product to be evaluated. In general terms, a document that is submitted to ITSEC describes what the product is designed to do, the situation in which it is intended to operate, the risks the product is likely to encounter and the mechanism by which the product acts to protect against the risks. It is for ITSEC to determine whether the claims are substantiated. Only the risks identified by the applicant are tested. A product is given one of seven levels from E0 (no formal assurance) to E6 (the highest level of confidence), with each level representing increasing levels of confidence. The assessment and granting of a position on the E scale is a judgement that a certain level of confidence has been met; it is not a measure of the strength of the security in place. It is important to realize that the organization submitting the product for evaluation sets out the criteria by which it will be evaluated, and it may be that this organization will not have included the risks associated with the use of the product by the end user. The evaluation includes an assessment of the confidence to be placed in whether the security features are the correct ones and how effectively they work. This means that a security mechanism might be applied correctly, but it will not be effective unless it is appropriate for the purpose for which it has been designed. In this respect, it is necessary to know why a particular security function is necessary, what security is actually in place and how the security is provided. It does not follow that if a product has a high E level that it will provide a high level of security.

5.123 The ‘Common Criteria for Information Technology Security Evaluation’ and ‘Common Methodology for Information Security Evaluation’ comprise the technical basis for an international agreement called the Common Criteria Recognition Agreement. The manufacturer submits its product to an independent licensed laboratory for an assessment. The way a product is evaluated is similar to the way ITSEC undertakes such assessments. There are problems with this, because it creates a conflict of interest: there are no known examples of the revocation of licences of laboratories that conduct evaluations, both parties are able to subvert the process, and determining the name of the organization that conducted the evaluation might be impossible without an order for disclosure. In addition, claims will sometimes be made that a device has been certified when, in fact, it might only have been evaluated. Often, a bank will ask a judge to rely on the certification process without disclosing the relevant report. In the Norwegian case of Bernt Petter Jørgensen v DnB NOR Bank ASA, Journal number 04-016794TVI-TRON, Trondheim District Court, 24 September 2004,¹ Assistant Judge Leif O. Østerbø, who tried the case, said at 121–122 (emphasis added):

5.124 If the purpose of a trial is to test the evidence, should a judge assume that the standard security systems used by the bank were effective in the absence of any evidence? Should a judge accept untested assurances that audits actually take place, not knowing whether such audits are conducted internally or by the Banks Control Center, whether the audits revealed problems that might affect the systems for ATMs and PINs, or whether the audits were conducted by people with appropriate qualifications? Given the lack of such evidence, can a judge conclude that there was no reason to doubt the bank’s security systems could be at fault?¹ For instance, it has been demonstrated that independent external examination continues to validate and approve devices and cryptographic software code that are open to failure and subversion.²

5.125 Two observations are worthy of note: that standards¹ regarding aviation, space and medical devices are usually much more prescriptive that those used in other domains, and even within the aviation, space and medical industries a great deal of commercial software is developed against no formal process model at all. The relevant standard for medical devices is ‘ISO 13485:2003 Medical devices – Quality management systems – Requirements for regulatory purposes’ (now revised by ‘ISO 13485:2016 Medical devices – Quality management systems – Requirements for regulatory purposes’). This standard has historically placed much less focus on tracing the details of internal product structure than, for instance, DO-178B, Software Considerations in Airborne Systems and Equipment Certification, which is a guideline dealing with the safety of critical software to be used in certain airborne systems. Yet, although having software evaluated against standards is a laudable goal, it does not follow that, by conforming, errors are eliminated.²

Summary

5.126 In summary, faults in software and errors relating to the design of software systems are exceedingly common.¹ And while defects in hardware have been relatively rare,² they are not unknown.³ Hardware is increasingly developed using high-level languages similar to those used for software. Furthermore, hardware is being released with firmware which may be reconfigured for other purposes. In addition, hardware faults can also be introduced by the improper use or configuration of software tools designed for developing hardware, which may themselves be error-prone. Like software, hardware errors, too, can be exploited to cause security failures.⁴

5.127 Every part of a program is different, and must be independently correct. In the case of machines, there are two important differences: things are almost always continuous, and after a time the system is back where it started. When they are not continuous, problems always occur. For example, a wheel turns, and once it has turned, it is (not withstanding wear and tear) likely to be able to turn again. Each time it turns, it gets back to an indistinguishable state. This is called a symmetry. Symmetries are very general ideas. For example, if one moves a cup of coffee a foot to the left, it stays the same and works exactly as before. This is because the world we live in has translational symmetry – everything is the same if it is moved. Wheels have rotational symmetry, and so on. This means that almost all of the design decisions in mechanical devices ‘collapse’ because of symmetries, and there is not the exponential growth of cases that happens in software. On the other hand, no part of a software program is the same as any other part. Indeed, if it was, one would ask why it was so inefficiently designed. Thus there are no symmetries in software that amplify the ‘how it works’ thinking that so readily simplifies physical design. The other advantage of physical systems is that where a response is expected to be continuous, it can be verified. Where continuity holds, interpolation tells what the behaviour for any input will be. Digital systems do not have this helpful property.

5.128 In particular, it might be obvious that the behaviour of a stopwatch used by a policeman is the ‘same’ as the behaviour of the ‘same’ stopwatch presented in court as evidence, or in the laboratory where it was tested. Thanks to symmetries, moving a watch from the roadside to the laboratory does not change it. There is no symmetry to justify software adduced in court behaving as it did anywhere else. Software is not constrained, as any physical device is, to work in the universe with all its symmetries. Software does not obey any of them, and thanks to human error (known and unknown) in its design, its behaviour cannot be taken for granted.¹

5.129 Software will continue to be unreliable. By providing a general presumption of reliability to software, the law acts to reinforce the attitude of the software industry that the effects of poor-quality work remain the problem of the end user. In many circumstances, because the user can himself cause errors, the industry may seek to pin the blame on the user, further obfuscating the true origin and source of the errors.¹ For these reasons, it is rare for a customer to take legal action against the software supplier, let alone attempt such an action and be successful.²

5.130 This discussion apart, the central issue for lawyers is dealing with the presumption that a computer is working properly. The following summary of the problems of software by Professor Partridge help to remind us of the landscape:

5.131 This poses a question for lawyers, experts and the courts: how the reliability of software should be reviewed in a court of law.

Challenging ‘reliability’

5.132 When seeking to challenge the underlying software of a computer or computer-like device,¹ lawyers frequently have great difficulty in overcoming the presumption that a machine is working properly, although general assertions about the failure of software code are often made without providing any foundation for the allegations. This problem is compounded when a party refuses to deliver up relevant evidence, usually citing confidentiality as the reason for the refusal, and relying, directly or indirectly,² on the presumption that a computer is ‘reliable’. In such circumstances, it is difficult to convince a judge to order the disclosure of relevant data.

5.133 Yet, paradoxically, it is a well-known fact in the industry that software could hardly be said to be ‘reliable’, as noted by Steyn J in Eurodynamic Systems Plc v General Automation Ltd:

5.134 Professor Matt Blaze reinforces this view:

5.135 The late Professor Lawrence Bernstein and C. M. Yuhas also acknowledged this observation:

5.136 Finally, companies that write software code include a contract term in the software licence that makes it clear that writers of software code are not perfect. Here is an example:

5.137 This section aims to provide a broad outline of the problems relating to computers and computer-like devices experienced by different industries, and to illustrate the importance of software and how there may be times when the output of a computer may not necessarily be ‘reliable’ and is therefore not to be trusted. Software code should be open to scrutiny, and should not necessarily share the benefit of a presumption of ‘reliability’ that is incapable of being effectively challenged.

5.138 One of the problems with understanding the role of the presumption is that people fail to distinguish software from computer systems. Computers are merely devices that are remarkable in that they can be turned to do many tasks rather than being limited to a single purpose. In order to perform a useful purpose, they must be instructed by software. A computer and its software together can be taken to form a system. No machine is ‘reliable’ or ‘unreliable’ in an absolute sense. Machines may be more or less reliable. The term ‘reliable’ in everyday use is an abbreviation of what in technical terms is ‘reliable enough for the intended purpose’. All machines have some probability of failing, so none is ‘reliable’ in the sense that one can rely on it without any doubt, while many are reliable enough (their probability of failing to perform correctly at any one use is small enough) to be worth using. The problem with using the word ‘reliable’, as though reliability were a binary quality, is that we risk taking it to mean ‘reliable enough’ without allowing for the fact that what is ‘enough’ depends on the use to which we put the machine, or rather its outputs. For instance, a machine may be reliable enough to be worthwhile in everyday use, and yet not reliable enough to use as evidence in a specific case. The speedometer in a motor car is reliable enough to use as an aid for driving at reasonable speed, because this level of reliability is sufficient for the purpose. In such circumstances, precision is not necessary. Compare this to instruments in an aircraft: the same level of reliability could be catastrophic. It is not a matter of whether or not the instrument is ‘reliable’, but of ‘how reliable’ it is.

5.139 It follows that lay people are not aware of the inherent design faults, and trust their personal experience to reassure themselves that computers are ‘reliable’ machines. Yet lay users regularly experience problems with devices, which illustrates their failure to grasp that ‘reliability’ and software code are impossible to guarantee.¹

5.140 Lay people are not the only people to make this mistake. This is illustrated by the judicial claim that computers are ‘reliable’ because in current times their use is widespread. Villanueva JAD made just such an assertion without providing any evidence to sustain his claim that computers are ‘presumed reliable’ in the case of Hahnemann University Hospital v Dudnick:

5.141 This observation by Villanueva JAD was made in the same year as the failure of the software that caused the Ariane 5 rocket to be destroyed shortly after take-off.

5.142 That computers are deemed to be ‘reliable’ because they are used more frequently now than when they were first developed is a poor substitute for a rigorous understanding of the nature of computers and their software.¹ However, it is accepted that long-term use can be an important element of justified trust in a software system. This comes about because there might be a long history of valuable and seemingly error-free use, but also because the long-term user typically gets to know the idiosyncrasies of the system.

Aviation

5.143 Errors in aviation software can have disastrous, or near disastrous, consequences. They can be caused by something as simple as bad coding. By way of example, consider the F-22A Raptor advanced tactical fighter, which entered service with the US Air Force in 2005. In February 2007, 12 of these aircraft were flying from Hickham AFB in Hawaii to Kadena AB on Okinawa. All of the aircraft experienced simultaneous and total software failure in their navigational console when their longitude shifted from 180 degrees West to 180 degrees East. The jets were accompanied by tanker planes, which meant the pilots in the tankers were able to guide the jets back to Hawaii. Major General Don Sheppard spoke about the problem on CNN on 24 February 2007. The relevant part of the transcript is set out below:

5.144 In practice, this means that most commercially produced software will have thousands of undetected defects.¹

5.145 In conventional flight control, the flight control commands from the cockpit are conveyed mechanically through steel cables or pushrods, often servo-assisted, to hydraulic actuators which then physically move the aerodynamic control surfaces on the wings and tailplane. In ‘fly-by-wire’, the flight control commands are converted to electrical signals transmitted by wires to the control surface actuators (in some cases in modern fly-by-wire aircraft the actuators may also be electric). Flight control is completely intermediated by software code, so a more accurate description would now be ‘fly-by-software-code’. Besides fly-by-wire, the autopilot and flight management systems of even conventionally controlled aircraft are software-based. The more reliable and functional the autopilot and flight management systems software have become, the more pilots have relied on them, even to the detriment of their piloting skills, as demonstrated by a number of accidents and ensuing loss of life. Accidents involving aircraft can exhibit a series of anomalous pilot–system interactions, and aviation regulations and investigators, with few exceptions, tend to assign the responsibility for the results of those interactions ultimately to the pilots.¹ This is so even in circumstances where it is clear that the software code and the system design are so faulty that a human being is not able to respond correctly – or with sufficient speed. In the case of American Airlines Flight 965 near Cali, Colombia, on 20 December 1995,² 151 passengers and all of the cabin crew members died in the crash. In this case, a significant error occurred, as explained by Highsmith DJ:

5.146 The critical importance of verifying the design of aviation software based on industry standards was noted in the Aviation Occurrence Investigation Final Report: In-flight upset 154 km west of Learmonth.¹ In this case, a problem with the software controlling the aeroplane was the cause of the accident. In this investigation report, the authors cited text relating to software requirements from Software Considerations in Airborne Systems and Equipment Certification,² produced by the Radio Technical Commission for Aeronautics:

5.147 Perhaps it is not necessary to indicate that the Boeing 737 Max crashes that killed 346 people (189 on Lion Air Flight 610 and 157 on Ethiopian Airlines Flight 302), were, it appears, caused by a hardware–software interaction.¹ What is pertinent is that the problems originated in design changes that were apparently small and presumed to be unlikely to make any significant difference to the system’s behaviour, and were intended to make the new system appear to the users like the old system.²

Financial products

5.148 In August 2006, the rating agency Moody’s gave constant proportion debt obligations (CPDOs) an AAA rating, which was close to making an investment in a CPDO free of risk.¹ In comparison, a competing rating agency, Fitch, could not understand why such a high rating was given to such ‘investments’, because its own models put CPDOs at almost the grade of ‘junk’.² It transpired that the software used by Moody’s for the purpose of rating CPDOs had a number of faults. A fault was found in early 2008 that, when corrected, failed to give the AAA rating, increasing the likelihood of defaults. The rating committee failed to disclose the error to investors or clients, and although the error was eventually corrected, other changes were made to the code to ensure the AAA rating continued to be assigned.³ A subsequent external investigation by the law firm Sullivan & Cromwell established that members of staff had engaged in conduct contrary to Moody’s Code of Professional Conduct.⁴ Moody’s subsequently received a ‘Wells Notice’⁵ from the Securities and Exchange Commission (SEC) on 18 March 2011.⁶ The Division of Enforcement of the SEC later issued a Report of Investigation into the matter.⁷ In a section of the Report, there was an examination of the attitude of the people responsible for dealing with the software error. It is revealing, and it merits setting out in full:

5.149 Because the rating committee met in France and the UK and not in the US, the SEC declined to take any further action, ‘[b]‌ecause of uncertainty regarding a jurisdictional nexus to the United States in this matter’.

5.150 Although the SEC declined to take action in this case, it did take action against AXA Rosenberg Group LLC, AXA Rosenberg Investment Management LLC and Barr Rosenberg Research Center LLC. In this instance, an employee discovered an error in the computer code of a quantitative investment model used to manage client portfolios. The employee brought the matter to the attention of senior management, but was told to keep quiet about the error and not to inform others about it. The error adversely affected 608 of 1,421 client portfolios managed by AXA Rosenberg Investment Management and caused US$216,806,864 in losses. Cease-and-desist proceedings were instituted and the respondents were jointly and severally ordered to pay a civil money penalty in the amount of US$25 million to the US Treasury.¹

5.151 Another example that might be considered to be mundane is that of software systems for the use of stockbrokers. Stockbrokers used to be regulated by the Financial Services Authority (FSA) (now by the Financial Conduct Authority), and were required to conduct their business in accordance with relevant legislation and the rules laid out by the FSA. Failure to follow the rules may have caused the FSA to take disciplinary action against the firm. In the case of SAM Business Systems Limited v Hedley and Company (sued as a firm),¹ the partners of Hedley used to handle their stockbroking business with a system known as ANTAR, but late in 1999 they decided it might not work after the century date change, so they decided to buy a new product from SAM, a small software company whose only product was an item of software known as InterSet. SAM claimed this product was a ready-made package of software modules made by SAM for stockbrokers and others (such as banks) dealing in stocks and shares in administering their systems. Hedley agreed to buy the new system, but immediately after it went live, serious problems were apparent, many of which were fixed, some speedily. (The word ‘fix’ is the telling word here: a local fix within a large and complex piece of software often generates problems elsewhere.) Hedley continued to use InterSet, but problems persisted. Eventually, they decided to find another product for their purposes. In his judgment, Judge Bowsher QC discussed the issue of defaults in software:

5.152 The full nature of the problems encountered with this software that purported to be written for the specific purpose for which it was supplied merits setting out in full:

5.153 Two experts were appointed to give evidence in this case, and they signed an agreement which was, in fact, a schedule of defects alleged by Hedley with comments on each defect from SAM. This schedule of faults ran to 34 pages. Judge Bowsher QC offered some pertinent comments in relation to the attitude of the software supplier in this case:

Motor vehicles

5.154 Software can be manipulated to give whatever reading the writer wishes. Because software is presumed to be ‘reliable’, software that gives deliberate false data is also presumed to be ‘reliable’. It is well known that traffic lights are now generally controlled by software code across a network, and the code can be written in such a way as to break the law. Stefano Arrighetti, an engineering student from Genoa, is reported to have developed the T-Redspeed traffic light system in Italy. The traffic lights were apparently programmed to remain on amber before turning to red for less than the time set out in regulations.¹

5.155 The ‘sudden unintended acceleration’ incidents involving the unintended, unexpected and uncontrolled acceleration of modern vehicles with electronic controls raises the issue of the reliability of complex electronic vehicle systems.¹ Consider the prosecution of Ann Diggles, aged 82, who was found not guilty at Preston Crown Court (R v Ann Diggles T20157203 before Mr Justice Fraser) for causing death by dangerous driving and death by careless driving when her Nissan Qashqai hit and killed Julie Dean, aged 53, while she was attempting to park.² The prosecution’s case was that the driving of Mrs Diggles caused the accident. The prosecution relied on the evidence from the motor car manufacturer, as reported by the BBC:³

5.156 The expert witness for the defence was Dr Antony F. Anderson CEng FIEE. Dr Anderson pointed out the following:

5.157 In addition to the evidence from Dr Anderson, two other women came forward at a late stage in the trial to give evidence that they had also had identical experiences. The evidence was that Mrs Diggles and the other two witnesses had their vehicles fully serviced in line with the manufacturer’s recommendations.¹ The evidence put before the members of the jury is not readily available, which means it can only be observed that deaths and injuries appear to occur as a result of software failure. It would be of interest to know how the police and prosecution assessed the evidence, including the complexities between the software code and the mechanical and electronic systems.

5.158 Another prominent example involves Toyota and Lexus motor vehicles, some of which have involved deaths of drivers and their passengers.¹ Michael Barr, in giving expert evidence for the plaintiffs in the case of Bookout v Toyota Motor Corporation,² stated that:

5.159 Software in vehicles can also be manipulated to give the false assurance of regulatory compliance. In September 2015, the United States Environmental Protection Agency issued a notice of violation of the Clean Air Act to Volkswagen AG, Audi AG and Volkswagen Group of America, Inc.¹ The notice alleged that four-cylinder Volkswagen and Audi diesel cars manufactured in the years 2009–2015 included software that circumvented the emissions standards for some air pollutants. The State of California Air Resources Board had issued a separate In-Use Compliance letter to Volkswagen,² and the two agencies initiated investigations based on the allegations. A software algorithm on certain Volkswagen vehicles switched the full emissions controls on only when the car detected it was undergoing official emissions testing.³ Thus the effectiveness of the emission control devices was greatly reduced during normal driving. This meant that motor vehicles met the emissions standards in the laboratory or testing station, but during normal operation the vehicles emitted nitrogen oxides, or NOx, at up to 40 times the standard. Over a one-year period of operation, the emission of this extra pollutant by Volkswagen was estimated to have resulted in 5 to 50 premature deaths.⁴ The Department of Justice subsequently filed a complaint for alleged violations of the Clean Air Act.⁵

5.160 Manufacturers of motor vehicles are rapidly increasing the amount of software in vehicles, partly with the aim of manufacturing autonomous vehicles.¹ Semi-autonomous or fully autonomous vehicles will not provide the panacea that the industry constantly asserts. Vehicles controlled wholly or partially by software code will continue to cause accidents and kill and injure people.² Also, because the software in vehicles is open to being attacked, it is far from safe.³

Emergency services

5.161 In 1992, the London Ambulance computer-aided dispatch system failed. A complex set of circumstances resulted in an effective failure of the dispatching system, which are set out in paragraph 1996 of the Report.¹ Apparently ‘the computer system itself did not fail in a technical sense … However, much of the design had fatal flaws that would, and did, cumulatively lead to all of the symptoms of systems failure’.² Among the contributing factors were ‘exception messages’ and ‘requests for attention’ which scrolled off the screen because of the large number of messages generated.³ There is also a suggestion that one member of staff was not using the system as expected,⁴ and the problems were compounded by ‘a genuine failure of crews to press the correct status button owing to the nature and pressure of certain incidents’.⁵ This was so even though the individuals who used the new system were from a skilled and trained pool of staff, namely ambulance crews and controllers. Other problems have occurred since.⁶

5.162 In 2014, an outage for 911 calls in the United States of America occurred because of a preventable software coding error in a 911 Emergency Call Management Center automated system in Englewood, Colorado, operated by Intrado, a subsidiary of West Corporation. This prevented non-PI-enabled long-distance assignments, which meant calls could not be routed to the appropriate destination.¹

Medical

5.163 The widespread use of computer devices in the medical industry has also given rise to incidents where the reliability of devices and software has been called into question. The rules for approving medical devices leave a lot of scope for software failure. The device does not need new approval if it is ‘substantially similar’ to an existing approved device. This allows for errors to go unconsidered or for incremental changes to take the latest device far from the original design.¹ Most of the ‘apps’ promoted on smartphones are not licensed or inherited from an ‘equivalent’ device. Consider the ‘Babylon health app’ – a triage chatbot that is notoriously poor and not approved, but would pass the examination taken by final year doctors, as noted by Dr Margaret McCartney:

5.164 For instance, patients have been affected by an error in clinical IT software¹ and by the failure to timely correct an error,² and one study of a hospital computerized physician order entry system in the USA illustrated a number of errors that the system was supposed to resolve, such as an increased probability of prescribing errors. There were 12 flaws in the interface used by humans that reflected machine rules that in turn did not correspond to how work was organized or the usual behaviour of those using the system.³ There is an increasing volume of articles on this topic,⁴ and it would appear that some, and not all, of the problems were due to software defects,⁵ but it is now very clear that software helps to kill people in hospitals.⁶

The Post Office Horizon scandal

5.165 Between 2000 and 2019, the Post Office¹ operated a computerized accounting and electronic point of sale IT system called Horizon. This system was installed in its branch Post Offices around the country. It was not long before sub-postmasters and sub-postmistresses (SPMs) began experiencing balancing errors that they could not explain. Post Office employees did not attempt to find out why balance errors were occurring; they merely required the SPMs to make-up any shortfall from their own funds. The balancing errors ranged from small amounts to tens of thousands of pounds. Some SPMs would make up the shortfall, and some would not. The Post Office initiated a substantial number of prosecutions for theft and fraud (the Post Office itself is a prosecuting authority), relying on the presumption that computers are reliable.²

5.166 In response to the failure of the Post Office to consider that SPMs were not defrauding or stealing from the Post Office, a group of ex-sub-postmasters and sub-postmistresses formed the Justice For Subpostmasters Alliance (JFSA)¹ in 2009 as the result of experiencing significant problems with how Post Office Limited dealt with apparent shortfalls in their accounts after the introduction of the Horizon IT system in 2000.² Following years of campaigning with the support of many MPs, in 2012 the Post Office appointed Second Sight Support Services Limited, a firm of independent forensic accountants, to investigate the claims being made about the Horizon system and the associated issues. On 8 July 2013, Second Sight published an Interim Report on its findings up until that date, which led to MPs raising questions with the Minister for Postal Affairs in the House of Commons on 9 July 2013.³ The Interim Report demonstrated that there were issues that required further investigation, and in August 2013 an Initial Complaint Review and Mediation Scheme was established to investigate individual cases. The Scheme was open to both serving and ex-sub-postmasters and sub-postmistresses who had concerns relating to Horizon, and offered them an opportunity to have their cases independently reviewed and raised directly with the Post Office. A Working Group, comprising representatives from Second Sight, the Post Office and the JFSA, was established with an independent chair. The Scheme closed to applicants on 18 November 2013. During the 12 weeks it was open 150 applications were received. On 9 April 2015, the Post Office terminated the Scheme Working Group, and also terminated the contracts with Second Sight and the independent chairman. The draft of the Second Sight Report Part Two was due to be released to the Working Group on 10 April 2015, but the action of the Post Office prevented this from taking place. The second part of the Second Sight Report (version 2) eventually appeared on a journalists’ website.

5.167 In 2015, the law firm Freeths LLP agreed to represent those ex-sub-postmasters and sub-postmistresses who wanted to take part in any future legal action. Therium Group Holdings Limited funded the litigation.¹ A Group Litigation Order was subsequently made on 22 March 2017 by Senior Master Fontaine, and approved by the President of the Queen’s Bench Division. Procedural issues before the first trial were dealt with in the first judgment, Bates v Post Office Ltd,² and a second judgment dealt with a further application by the Post Office to strike out part of the claim in Bates v Post Office Ltd (No 2).³ It was anticipated that there would be four trials. In the event, only two trials have taken place.

5.168 The first trial concerned, in the main, the contractual position between the Post Office and the sub-postmasters and sub-postmistresses. The judgment is in Bates v Post Office Ltd (No. 3: Common Issues).¹ In this judgment, the judge included a comprehensive introduction to the issues generally between the parties at [2]‌–[43]. Orders in respect of costs of the Common Issues trial were determined in Bates v Post Office Ltd (No. 5: Common Issues Costs).² The second trial, dealing with the Horizon software, took place between 11 March 2019 and 22 July 2019. During the course of this trial, the Post Office issued an application that the judge recuse himself as Managing Judge in this group litigation, and stop the Horizon Issues trial so that it could be recommenced at some later date before a replacement Managing Judge. That application was refused, for which see Bates v Post Office Ltd (No. 4: Recusal Application).³ Permission to appeal was refused by the single Lord Justice on 9 May 2019.⁴ Between the end of the second trial and the judgment, the parties sought mediation. An agreement was reached on 11 December 2019.⁵ The judge handed down his judgment in the second trial on 16 December 2019 – a comprehensive judgment that clearly indicated that the Horizon system had a significant number of software errors, including the ability of employees of Fujitsu to enter the computers of SPMs remotely and change data without the SPM being aware of what was happening.⁶ When handing down his judgment, the judge indicated that he:

5.169 In 2015, the Criminal Case Review Commission (CCRC) began reviewing claims of wrongful prosecution for offences such as theft and false accounting, caused, the complaints allege, as a result of problems with the Post Office’s Horizon IT system. In 2020, the CCRC referred 47 Post Office cases on the abuse of process to the Court of Appeal,¹ and in October 2020 the government initiated a non-statutory inquiry into the Post Office’s Horizon IT dispute led by Sir Wyn Williams.² The Court of Appeal Criminal Division heard the appeal of 42 appellants on 22, 23 and 24 March 2021 and handed down judgment on 23 April 2021 in which the appeals of 39 appellants were quashed.³ The court also reached a rare determination: that the prosecutions were an affront to the conscience of the court.⁴ In delivering the judgment of the court, Holroyde LJ noted that the Post Office constantly asserted that the Horizon system was ‘reliable’ [20] and [125], ‘accurate and reliable’ [68] or ‘robust and reliable’ [121]. He went on to say, at [137]:

5.170 Not only was it factually incorrect that the Horizon system was reliable, but the failure to disclose relevant information meant:

Banking

5.171 The presumption that computers are reliable is particularly relevant with regard to banking. Banks across the world have introduced very complex systems and networks to control the flow of transactions, many of which are no longer under the sole control of the banks themselves. That a bank benefits from the presumption that its computers and networks, including the computers and networks it relies upon over which it has no direct control, were in order at the material time, puts an impossible burden on the customer. If a customer in dispute with his bank wants to challenge this presumption, he will require significant knowledge of the computers, systems and networks operated by the bank, how they work and where the vulnerabilities might lie, including the results of relevant audits, both internal and external – a task well beyond the majority of customers, including most lawyers without the benefit of expert advice, which in itself is difficult to obtain.

5.172 Issues regarding the reliability of banking systems manifested themselves in the problems in the UK in June and July 2012 with RBS, NatWest and Ulster banks.¹ On 19 June 2012, an important item of software known as CA-7 was updated. This software controls the batch processing systems that deal with retail banking transactions. It is used to automate large sequences of batch mainframe work, usually referred to as ‘jobs’. The jobs take transactions from various places, such as ATM withdrawals, automatic salary payments and such like, so that accounts are credited and debited with the correct amounts by the next morning. The software initiates jobs, and when one job is finished, a new job will be initiated. Accounts are processed overnight when the mainframes are less busy, and finish by updating the master copy of the account in a system known as Caustic. It appears that the update made to CA-7 caused the files to run incorrectly or not to run at all for three nights. David Silverstone, delivery and solutions manager for NMQA, which provides automated testing software to a number of banks, is quoted to the effect that such problems can always be avoided if there is sufficient testing of the update before it is put into operational use.² Michael Allen, director of IT service management at Compuware, is reported to have said:

5.173 The complexity of the problem is highlighted in an article written by Hilary Osborne in The Guardian in 2014, in which the issues were explained:

5.174 The effects of the CA-7 imbroglio were considerable. In some cases people were left homeless after the computer problems meant house purchases fell through; others were stranded abroad, unable to obtain access to funds which should have been in their account; wages and direct debits were not paid; and it is reported that one person spent the weekend in prison because the computer failure meant his bail money was not processed.¹ The problems continued into 2014.² In December 2014, the Royal Bank of Scotland Plc, National Westminster Bank Plc and Ulster Bank Ltd faced a combined financial penalty of £42 million by the Financial Conduct Authority for breaches of Principle 3 of the ‘principles for businesses’, forming part of ‘the principles of good regulation’, which requires a firm to take reasonable care to organize and control its affairs responsibly and effectively with adequate risk management systems,³ and the Prudential Regulation Authority imposed a financial penalty of £14 million on the same banks for their failure to meet their obligations to have adequate systems and controls to identify and manage their exposure to IT risks.⁴

5.175 The problem of complexity and the difficulties in understanding and maintaining another banking system were emphasized in the report by Deloitte of the failure of the Real-Time Gross Settlement (RTGS) system operated by the Bank of England in 2014.¹ The report stated:

5.176 In this case, there was a design defect. The defect was mentioned at paragraph 151 of the report, but it had been redacted to such an extent that there was no meaningful text. The only information available is that a process known as ‘Process A functionality’ was changed in April 2014 and tested in May 2014 in preparation for the anticipated transfer of CHAPS members, and a design defect was introduced at this stage. This was the cause of the failure.¹

5.177 Other examples include Deutsche Bank AG, where a coding error caused Deutsche to reverse the buy/sell indicator for its CFD Equity Swaps in 2013. This meant it reported them inaccurately to the Financial Conduct Authority (FCA). The FCA imposed a financial penalty of £4,7818,800 on Deutsche for failing to provide accurate reports in accordance with the provisions of the Markets in Financial Instruments Directive.¹ In 2014, the Co-operative Bank identified that statements on a number of loans had been issued three days late because of a software error. Under the provisions of s 6 of the Consumer Credit Act 2006, which inserted s 77A into the Consumer Credit Act 1974, it is necessary to provide an annual statement to each borrower for a fixed-sum credit agreement, which should set out the amount borrowed, the money paid, the interest and the outstanding amount. If the creditor fails to provide the debtor with an annual statement, the creditor is not entitled to enforce the agreement during the period of the failure to comply, and the debtor is not liable to pay any interest during the period. The bank set aside £109.5 million to refund interest payments for this breach of the Act.²

Interception of communications

5.178 In the half-yearly report in July 2015, the Report of the Interception of Communications Commissioner illustrated the effect that errors in software code had had on the interception of communications.¹ Although the number of technical errors were low in comparison to the overall number of requests made, nevertheless the effect such errors had on innocent parties was significant. In paragraph 5.28, it was indicated that eight out of ten errors made in relation to resolving IP addresses to individuals related to investigations into the sexual exploitation of children or cases where serious concerns were raised in relation to the welfare of a child.² The Commissioner commented, at paragraphs 5.29 and 5.37:

5.179 In his Report, the Commissioner said that the Crown Prosecution Service used funds provided by the government to work with vendors and the Home Office to develop secure disclosure systems – and although money has been spent on this issue, technical issues nevertheless continue to arise.¹ As a result of the disclosure of the technical errors, the Commissioner made a number of recommendations regarding technical system errors:

5.180 Technical errors continue to be reported by successive Commissioners.¹

Most computer errors are either immediately detectable or result from input errors

5.181 Let us consider the proposition that most computer errors are either immediately detectable or result from errors in the data entered into the machines. The evidence is to the contrary: Mr Adams demonstrated that a third of software faults in a large IBM study took at least 5,000 execution years to appear for the first time (this was one of the largest studies of all time);¹ Professor Les Hatton and Andy Roberts conducted a study that demonstrated that seismic programs developed by oil companies were shown to have been used for many years even though they were defective;² and Nancy G. Leveson and Clark S. Turner demonstrated that between June 1985 and January 1987 the Therac-25 medical linear accelerator was involved in massive radiation overdoses, causing the deaths of six people, while others were seriously injured. The detailed investigations eventually indicated that the main cause of the deaths was software errors. Some of the lessons gleaned from the work by Nancy Leveson included the following: too much confidence was placed in the software, an assumption by lay people that software will not or cannot fail, and engineers ignoring software when analysing faults, because it was assumed the hardware was at fault, not the software.³ In this respect, opinions have not changed since 1987.⁴ When investigating sudden unintended acceleration in some of its motor cars in the US, Toyota did not include software engineers in its investigations, and incorrectly ruled out software as the cause of the resulting deaths and injuries.⁵

5.182 Uncovering the faults in devices controlled by software used in medicine is now considered to be an important research area,¹ and in November 2000, 28 patients at the National Cancer Institute in Panama were given massive overdoses of gamma rays partly due to limitations of the computer program that guided use of a radiation therapy machine. A number of patients died.²

5.183 The observations by Professor Leveson will invariably remain relevant: the Toyota recall exercise in late 2009 and early 2010 serves to illustrate this point.¹ The US Congressional Committee on Energy and Commerce heard evidence on this matter, and a report by The National Highway Traffic Safety Administration and the National Aeronautics and Space Administration (NHTSA–NASA), which conducted a study into the problem entitled ‘Study of unintended acceleration in Toyota vehicles’, a revised version of which was published on 15 April 2011,² concluded that it was not proven that faulty software caused the problems, although it was accepted that just because no software faults could be found did not mean that software faults did not occur. The methods used to investigate this matter were challenged.³

5.184 Civil proceedings were subsequently initiated by a number of people across the US. In Bookout v Toyota Motor Corporation,¹ Michael Barr, an expert in embedded software, gave evidence for the plaintiff regarding the software code in the relevant motor vehicles. He was also cross-examined about aspects of the NHTSA Report, among other issues. His evidence demonstrated that there were a significant number of errors in the software (referred to as ‘bugs’ in the transcript):

5.185 He also demonstrated that motor cars are now largely run by software. In fact, motor cars have more software code than aircraft, and are prone to software recalls.¹ Drivers no longer have total control over their vehicles.² For instance, it was explained how the driver is no longer in direct control of the throttle:

5.186 Mr Barr established that the motor vehicle had errors in the throttle system:

5.187 Mr Barr stated his overall opinion in the following terms: ‘ultimately my conclusion is that this Toyota electronic throttle control system is a cause of [unintended acceleration] software malfunction in this electronic throttle module, can cause unintended acceleration.’¹ The members of the jury found in favour of the plaintiffs, and awarded damages of US$1.5 million to each of the plaintiffs. The US Department of Justice subsequently concluded a criminal investigation into the Toyota Motor Company regarding the widespread incidents of unintended vehicle acceleration that caused panic for Toyota owners between 2009 and 2010. It was established with certainty that Toyota intentionally concealed information and misled the public about the safety issues behind these recalls.² It was alleged that Toyota made misleading public statements to consumers, gave inaccurate facts to Members of Congress and concealed the extent of problems that some consumers encountered from federal regulators. For instance, Betsy Benjaminson, a translator for Toyota, realized what she was translating was highly significant:

5.188 In its settlement with the Department of Justice, Toyota admitted its wrongdoing in making such misleading statements in the Statement of Facts filed with the criminal information, and also admitted that it undertook these actions as an act of concealment as part of efforts to defend its brand. In consequence, Toyota paid a financial penalty of US$1.2 billion under the settlement.¹

Challenging the authenticity of digital data – trial within a trial

5.189 Laying the evidentiary foundations for the authenticity of electronic evidence is discussed elsewhere in this text, but if the authenticity of evidence is raised by one of the parties, it is appropriate to deal with it in a trial within a trial.¹ This will be a rare occurrence, as noted in R. v Wayte (William Guy)² by Bedlan J:

5.190 In R. v Stevenson (Ronald), R. v Hulse (Barry), R. v Whitney (Raymond),¹ Kilner Brown J was required to establish whether audio tapes were originals. After a lengthy and careful examination of the evidence held in a trial within a trial, it became clear that there was an opportunity for someone to have interfered with the original tape, and there was evidence that some interference might have taken place. Given the nature of the evidence before him, he said:

5.191 In the case of R v Robson (Bernard Jack), R v Harris (Gordon Federick),¹ the defence raised the issue of the admissibility of the evidence of 13 tape recordings. The judge had to consider whether, on the face of it, the tapes were authentic in the absence of the members of the jury. Shaw J heard evidence in a trial within a trial from a number of witnesses who gave evidence of the history of the tapes, from the actual process of recording to the time they were produced in court. He also listened to four experts called on behalf of the defence, whose examination of the tapes led them to question their originality and authenticity. The prosecution called a separate witness in rebuttal. After hearing the evidence, Shaw J decided that the tape recordings were originals and authentic, commenting that:

5.192 Professor Tapper expressed the view that this exercise should be conducted first by the judge, and if, on the balance of probabilities, the judge determines the evidence could go before the jury, it would then be necessary to cover the same ground again in the same way as any other question of fact that must be decided at trial.¹ On the standard of proof to be used by the judge, O’Connor LJ indicated the criminal standard of proof is to be used in the context of handwriting,² and in the case of R v Minors (Craig), R v Harper (Giselle Gaile),³ Steyn J, as he then was, set out the opinion of the Court of Appeal on this matter in relation to a computer printout:

5.193 He went on to indicate that the judge should apply the ordinary standard of criminal proof in reaching a decision, and in the case of R. v Neville,¹ the members of the Court of Appeal also noted that trial judges ‘should examine critically any suggestion that a prior computer malfunction has any relevance to the particular computer record tendered in evidence’.² The decision of the Court of Appeal in R v Minors (Craig, R v Harper (Giselle Gaile) to require a judge to apply the ordinary standard of criminal proof in reaching a decision when hearing evidence in a trial within a trial overrules the decision of Shaw J in R v Robson (Bernard Jack), R v Harris (Gordon Federick) (in which he reached an opinion that the standard was on a balance of probabilities³), although there is much to commend the view of Shaw J when he suggested that the prosecution need do no more than set up a prima facie case in favour of the authenticity of the evidence:

5.194 The standard that a judge must apply in determining the admissibility of a videotape was considered by Cameron JA in the Canadian case of R v Penney¹ before the Newfoundland and Labrador Court of Appeal in 2002. In this instance, the prosecution sought to adduce evidence of the killing of marine animals. The evidence comprised a video recording of the killing of a seal. The recording had been frequently switched on and off as the operator of the camera selected scenes to record. The recording was filmed in mini-digital format, transferred to Beta format and then to VHS format. Before the Crown took possession of the tape, it had been in the possession of a professional editing studio for several months. There was no attempt to provide for the security of or to restrict access to the tape. The Crown called the camera operator and the owner of the company for whom the camera operator worked to give evidence during the trial within a trial. The trial judge concluded that the witnesses were not credible and failed to tell the truth. He therefore refused to admit the video recording in any format. The Crown appealed to the summary appeal conviction court, which allowed the appeal. A subsequent appeal to the Newfoundland and Labrador Court of Appeal reversed the summary appeal conviction court decision and the decision of the trial judge was restored. Cameron JA addressed the issue of the standard that a trial judge should apply in determining the admissibility of videotape evidence, indicating that:

5.195 He went on:

5.196 Reference was made to R v Nikolovski,¹ which established that where a videotape has not been altered or changed, and where it depicts the scene of a crime, then it becomes admissible and relevant evidence.² In R v Bulldog,³ the members of the Court of Appeal of Alberta considered this issue, and emphasized that ‘What matters with a recording, then, is not whether it was altered, but rather the degree of accuracy of its representation’.⁴ In R v Penney, the judge addressed the problem of the falsification of evidence by pointing out that the members of a jury ‘can be expected to have, if not experience with, knowledge of the possibilities for manipulating the content of photographs and videotapes’, and concluded that the ‘standard by which the trial judge is to determine the question is on the balance of probabilities’.⁵

5.197 If the standard of proof of a trial within a trial is the criminal standard, it can be argued that the prosecution is required to prove its case twice: once to the trial judge and a second time before the members of the jury. Arguably, the duty of the trial judge is to sift the evidence sufficiently to establish whether it is to go before the members of the jury in cases where the authenticity of the evidence is questioned by the defence.

A protocol for challenging software in devices and systems

5.198 Should it become the norm for the defence to challenge the authenticity of evidence in digital form, it is suggested that consideration might be given to the development of a protocol to deal with such challenges:

5.199 Such an approach would be entirely consistent with the trial management procedures set out in Part 3, rule 3.3(2)(c)(ii) of the Criminal Procedure Rules 2015 (as amended). If this first hurdle is overcome, then it will be for the trial judge to decide whether a trial within a trial is necessary, and if so, to set out the parameters, including the standard of proof, for which a ruling is required.

5.200 There is something missing in the suggestion noted above regarding criminal proceedings: there is no discussion regarding the sufficiency of the evidence the defence must adduce to persuade a judge to order appropriate disclosure.¹ Professor Imwinkelried has also considered this problem,² and has proposed a two-step process, the first part of which is:

5.201 Professor Imwinkelried then indicates, at 128, that ‘The judge should certainly not accept the ipse dixit assertion of the defense counsel that the omitted condition is material in the sense that its presence could affect the outcome of the test’. Providing the defence has met the burden of part one, the second part of the test provides as follows:

5.202 Professor Imwinkelried points out, as 129, that:

5.203 There are criticisms of this proposal. Professor Martyn Thomas has pointed out that an argument or proposal that depends on the outcome of testing to provide evidence of the correctness or of the specific, required reliability of some software is almost always based on erroneous or unverified assumptions.¹ At the very least, it should always be challenged by the following questions:

5.204 The answers to these questions will either reveal a fundamental flaw or provide the basis for challenging the assumptions. In addition, it is not clear how ‘reliable’ a court requires a forensic test to be. If a forensic system was known to be right more than half the time and randomly wrong otherwise, the question is whether a single positive result will pass the on-the-balance-of-probabilities requirement for a civil case.

5.205 As all judges are only too well aware, there is a danger that the trial judge may be seen to usurp the functions of the members of the jury in reaching preliminary decisions on authenticity when conducting a trial within a trial. Marshall J, in delivering the judgment of the Court of Appeal in the case of R. v Ali (Maqsud), R. v Hussain (Ashiq),¹ indicated that conducting a trial within a trial should be a rare occurrence:

5.206 This restricted view was reinforced by the comments in R. v Stevenson (Ronald), R. v Hulse (Barry), R. v Whitney (Raymond)¹ of Kilner Brown J:

5.207 Where the matter of authentication is raised, the trial judge is required to decide whether to conduct a trial within a trial. Where the decision is made to hold a trial within a trial, it will be useful for the judge to set out the scope of the hearing. In R v Robson (Bernard Jack), R v Harris (Gordon Federick), Shaw J said that where such a hearing takes place, it should be defined narrowly.¹ This must be right.

5.208 In respect of the costs of such an exercise, in R. v Saward (Steven Kevin), R. v Bower (Steven Kevin), R. v Harrison (Keith),¹ the prosecution sought the admission of recordings of telephone conversations that were intercepted by the Dutch police and stored on a CD. The judge was invited to conduct a trial within a trial to determine whether or not the data recorded on the CD, transferred from a mainframe computer located in the Netherlands, was admissible in evidence as authentic, accurate and a reliable copy. The trial within a trial lasted for four days, and a number of witnesses, including British officers and a Dutch police officer, were called to give evidence. Lady Justice Hallett commented, at [44], on the costs of such an exercise:

5.209 The defence drew a number of errors in the CD recording to the attention of the trial judge, and it was only right that this issue should be considered.

5.210 When collecting electronic evidence, the investigator needs to pay careful attention to the process by which the evidence was obtained, and to demonstrate the provenance of the evidence. In R. v Skinner (Philip),¹ the defence called into question evidence of screen images obtained by a police constable when conducting an investigation into indecent photographs of children. During the trial within a trial, the police officer gave evidence that he had a ‘source’ for the screen images. He admitted entering a website that he was not prepared to identify, and could only provide limited information about the provenance of the material he produced for the purposes of the investigation: namely, images that appeared on screen that were produced in the form of a printout. He refused to name or identify the website he had entered. It was held by the members of the Court of Appeal that the trial judge wrongly admitted the evidence. First, the members of the Court accepted that it was probable that the screen images were real evidence, because their content did not require any computer input, and likened the image to somebody switching on a television set. However, the printouts were not authenticated properly under the provisions of s 27 of the Criminal Justice Act 1988, and for that reason, the trial judge should not have admitted them. Second, there was no public interest immunity hearing to enable the judge to decide whether the prosecution need not disclose or need not give evidence as to the process by which the screen image reached the police officer, or in the absence of a proper explanation, how the screen image came to be on the police officer’s computer. It was conceded that a public interest immunity hearing should have been requested, and in such circumstances the trial judge was wrong to admit the evidence.

Reintroduction of the common law presumption

5.211 The Law Commission proposed the repeal of s 69 of the Police and Criminal Evidence Act 1984 and a return to the common law presumption:

5.212 The grounds for justification were set out in paragraphs 13.7 – 13.11, and are reproduced below with the references omitted: