The role of good recordkeeping in providing evidence for measuring the Sustainable Development Goals (SDGs) has been discussed in depth in Chapter 1 of this volume, ‘Records as evidence for measuring sustainable development in Africa’, as has the fact that data, documents and other forms of recorded information are increasingly digital in form. The need for digital preservation will only continue to grow, an issue that cannot be avoided if SDGs are to be met. Furthermore, the SDG agenda requires that these records remain accessible to 2030 and beyond, which, in the digital world, can only be achieved through continuous active management.

Digital preservation is not just a technical issue. It also requires an ecosystem of organisations, policies and standards, resources and people. Demonstrating whether or not SDGs have been achieved requires the management of the evidence base, which in turn is dependent on the application of digital preservation methodologies. The significance of developing digital preservation capacity for maintaining reliable SDG measurements over time, as a fundamental component to the SDG agenda, cannot be overstated.

The main challenges of collecting and preserving information specified by the SDGs in ways that will allow it to be combined and compared will be discussed further in David Giaretta’s chapter on ‘Preserving and using digitally encoded information as a foundation for sustainable development’ (Chapter 10 in this volume).

This chapter considers the practical implications for developing digital preservation capabilities. It begins by considering the component elements of such a capability, and it examines how the concept of maturity models can be used to help organisations define models for digital preservation that are appropriate to their needs, as the chapters by Shepherd and McLeod and McDonald also emphasise. It then looks at the variety of models available for delivering a digital preservation service and concludes with a summary of the operational implications.

Elements of a digital preservation capability

There are many definitions of digital preservation, but that provided by the Digital Preservation Coalition is a good starting point:

In an archival context, we can refine this definition to encompass the concept of authenticity which, as set out in ISO 15489,² depends on three essential characteristics:

Fundamentally, therefore, digital preservation involves acquiring authentic digital records, storing them and making them accessible to users for as long as they are required. Given this definition, we can begin to identify the requirements through which this goal can be achieved. At its most fundamental level, all digital information is encoded as a sequence of 1s and 0s – a bitstream – which may be written to a storage medium or transmitted across a network. These bitstreams have no intrinsic meaning but require layers of technology to transform them into meaningful information. Preserving meaningful digital information requires us to maintain both the underlying bitstream and the means to correctly interpret it. Digital preservation is, therefore, often considered to require two fundamental activities:

However, this is not a purely technical exercise. For any organisation to undertake digital preservation, in practice, it also needs broader capabilities, encompassing:

While any organisation must have these capabilities in order to preserve digital content over the long term, digital preservation is not a one-size-fits-all enterprise – every organisation, large or small, whether well-funded or underfunded, is unique and will have its own requirements, opportunities and constraints. Any organisation wishing to develop the capability to preserve digital records through time must therefore develop an understanding of its current capability, design a blueprint for the future that meets its specific requirements, identify the gap between the latter and the former, and introduce a strategy for building the necessary capacity to bridge that gap.

Maturity models are well-established tools that can help an organisation assess its capabilities in a particular area against a benchmark standard as a means of articulating its current and desired future operating models. They typically describe organisational capacity in two dimensions: first, they define the component parts that together constitute the specific service or function to be addressed, and second, they define a scale for measuring capability against each of these requirements.

The key elements of digital preservation capability, including organisational factors such as governance, resources and policy, and technical or functional aspects, have been discussed above. In the language of maturity models, these are often referred to as ‘process perspectives’. A typical scale for measuring capability will comprise five or six steps, spanning the stages from developing awareness to building increasing capability, such as set out in Table 9.1.

These steps can be mapped to the records management capacity levels in Julie McLeod and Elizabeth Shepherd’s chapter, which also identifies the role and responsibilities of the major players involved. These also correspond with the maturity levels defined in John McDonald’s chapter.³ McDonald sets out capacity levels as:

Achieving Level 4 will require a basic level of digital preservation capability, while Level 5 corresponds to the more advanced, managed and optimised capabilities. Taken together, we can begin to develop detailed definitions of each maturity level against every process perspective. To illustrate the principle, the ‘basic’ level of maturity (that is, Step 3) might be defined as shown in Table 9.2.

This should be considered the minimum standard for any organisation to measure SDGs and provide a genuine digital preservation service. For many, this may be an achievable target, sufficient to meet their objectives. It should not be assumed that every organisation should strive to reach Step 5 in every process, which could be excessive and infeasible, particularly in lower resourced countries. The power of maturity models is that they make it possible to define nuanced and proportionate levels of service capability. In practice, most organisations will wish to define different target levels for different process perspectives.

In many cases, a relatively modest level may be entirely appropriate. The value of maturity models lies primarily in providing a means of thinking about digital preservation as a broad spectrum of capabilities, rather than a single, and almost-certainly unobtainable, ideal. This should help organisations to think about what ‘good enough’ preservation looks like in their own particular circumstances, and therefore to ensure a proportionate and appropriate level of investment.

The approach summarised here is intended to provide a flexible and realistic methodology.⁴ However, other maturity models for digital preservation exist. For example, the National Digital Stewardship Alliance has developed its own framework of levels of digital preservation,⁵ while the EU-funded E-ARK project has created a model which has been tested across a range of institutions.⁶ Most recently, the Digital Preservation Coalition has developed a Rapid Assessment Model based on that described in this essay.⁷ Whichever model is used, it is essential for any organisation that is planning to develop a digital preservation capability to have a clear and pragmatic definition of the level of capability it wishes to build. Having done so, it will then need to consider how to establish that capability in practice.

Implementation options

Many different models are possible for implementing a practical digital preservation capability.

This means that options are available to meet the needs of a wide variety of sizes and types of organisation in many different contexts with very different levels of resources at their disposal. The market for digital preservation solutions is still comparatively young, but its vast potential size (one study estimated its potential value in 2011 as in excess of $1 billion,⁸ and this figure has surely only increased in the intervening period) has encouraged a growing number of increasingly mature products to emerge to meet a wide range of requirements.

It is very important to recognise that digital preservation is achievable not only by large institutions with substantial budgets – practical solutions are possible with much more modest means. The main options are summarised below, with a brief discussion of their advantages and disadvantages. Mention of specific products or tools does not constitute an endorsement or recommendation.

Doing nothing

Any analysis of options should always include the status quo, as this allows a true comparison to be made with other, more positive, options, and it highlights the implications of not taking action. Doing nothing is not a cost-free option; actually, through time it is likely to be very expensive because of the continuing burden of maintaining archival data on inappropriate storage infrastructure and the actual or lost opportunity costs of having to recreate digital records that will inevitably be lost through inaction.

Without active management, data loss is inevitable over the longer term, and the associated costs are likely to be much higher than the costs of establishing appropriate control systems. The reactive, ad hoc rescue of digital information will almost inevitably be much more expensive than proactive management. Doing nothing will also, of course, incur significant risks and will not meet the requirement of maintaining access to vital records. It will have major consequences for the delivery of government programmes and for the ability to measure the SDGs reliably.

Using open source software

It is possible to develop a digital preservation solution entirely from open source tools. A number of complete open source digital repository management systems are available, although they vary in the level of preservation functionality that they offer directly. The most widely used systems include Archivematica,⁹ DSpace,¹⁰ EPrints,¹¹ Fedora,¹² LOCKSS¹³ and RODA.¹⁴ In addition to these complete systems, a number of toolkits and individual utilities have been developed that can be used to add preservation functionality to existing repository systems. These include characterisation tools,¹⁵ such as DROID¹⁶ and JHOVE,¹⁷ forensic tools, including BitCurator,¹⁸ and web archiving tools, such as Heritrix.¹⁹

Open source solutions offer a number of attractions. They are often free to use, may have thriving user and developer communities, and can offer very flexible solutions. The fact that users have complete access to, and control over, the source code, can also be very attractive from a sustainability perspective. However, open source solutions are not cost-free: resources are likely to be required to adapt and configure the software, either from in-house staff or procured from a third party. Also, the organisation will have to bear all the risks, rather than sharing them with suppliers.

It is possible to develop a functioning digital archive capacity very affordably, by using simple, cheap and readily available open source tools and existing infrastructure. Compromises may be needed to keep costs down – limited integration will involve a lot of manual or semi-manual processes, and this approach probably won’t be appropriate for managing larger volumes of records or high numbers of users. It may also be difficult to support or sustain as a service over time. Nonetheless, for many smaller organisations or those with limited resources, this may be a good starting point. Investment in limited configuration of these tools to meet local needs, and with good support arrangements, will be very worthwhile, and may compare favourably with the costs of commercial products.

Developing a bespoke solution

Developing a bespoke, or tailor-made, solution is an option, especially for organisations with significant in-house software development capacity, or with the resources to commission external developers. A bespoke solution can be developed to meet the organisation’s specific requirements, but software development is an expensive, complex, uncertain and time-consuming option. Given the availability of a number of mature third-party solutions, this is unlikely to be an economical approach except for organisations with very unique requirements and substantial resources.

Procuring a commercial solution

Commercial off-the-shelf (COTS) digital preservation solutions are a common approach, thanks to a growing and comparatively mature market for such products. They typically command a relatively high price, including one-off licence fees, annual support costs and potentially expensive customisation and configuration. They can also create a degree of dependency on an external supplier, and on proprietary software. At the same time, they can offer a high level of flexibility and support, usually have well-established user communities, and can therefore provide a comparatively low-risk approach. Building a strong relationship with the supplier is often vital.

There is a small, but now well-established, community of commercial digital preservation products that caters not only to the library, archive, museum and gallery sector but increasingly to customers in the private sector with long-term data retention requirements. It includes industries such as banking, pharmaceuticals, aerospace, gas and oil exploration, government and the nuclear energy sector. Examples of commercial products include Preservica²⁰ and Rosetta.²¹

Outsourcing the service

One of the most recent developments in the digital preservation market has been the growth of preservation-as-a-service, whereby third-party suppliers provide an end-to-end solution hosted on their own infrastructure. This option minimises the direct impact on the organisation, including the need to host and support significant infrastructure, and can be easily scaled up or down to meet demand. It also has a very low barrier to entry, making it particularly attractive to smaller organisations or those with limited resources. These services are typically priced on the basis of actual usage, removing the need for significant up-front capital investment, which can make them financially attractive. However, it is always important to consider the long-term costs when comparing the economics of different models.

A growing number of suppliers offer a full range of digital repository services. In some cases, these have emerged to meet the needs of specific communities. For example, the UK Data Archive (UKDA)²² and the Inter-University Consortium for Political and Social Research (ICPSR)²³ both provide data archive services to the international social sciences community. The international library community is served by services such as Portico, which preserves e-journals, e-books and digitised historical collections on behalf of publishers and libraries,²⁴ and by the OCLC CONTENTdm digital collections management service.²⁵ A number of cloud-based commercial services have also emerged, such as ArchivesDirect,²⁶ DuraCloud,²⁷ Preservica Cloud Edition,²⁸ while services are also available for specific functions such as web archiving, such as Archive-It.²⁹

Partnership approaches

A very collaborative community has developed around digital preservation. This can include the actual provision of services, with a group of organisations that share a common set of requirements establishing a partnership to develop and share services. Given the long timeframes involved and high level of confidence required, such arrangements are generally formalised in some way, whether through a contract, consortium agreement, or less legally binding instruments such as memoranda of understanding or letters of agreement. A partnership may even be set up as a separate legal entity representing the shared interests of the partners. This might be a non-profit entity, such as a charity, trust, foundation or private company limited by guarantee.

The precise forms of non-profit organisation allowed vary from country to country, but most enjoy tax-exempt status. For instance, an existing legal entity might host a partnership entity that exists independently from its members but is not itself legally constituted. Partnerships can establish their own shared infrastructure, which may use a distributed model such as LOCKSS (see above) to jointly procure a service from a third party, or they can designate one partner to operate the service on behalf of the others.

A partnership approach can offer significant economies of scale and allow better deals to be negotiated with suppliers than individual partners could achieve on their own. However, they depend on a strong and ongoing alignment of objectives between partners that can be complex to establish and require compromise. There are some excellent examples of the partnership model in practice, many based on LOCKSS technology. These include CLOCKSS (Controlled LOCKSS), an international, not-for-profit community partnership between libraries and publishers using a private network based on LOCKSS technology to provide a distributed archive for electronic scholarly content;³⁰ the Alabama Digital Preservation Network (ADPNet), which provides a low-cost distributed LOCKSS-based digital preservation service for cultural heritage organisations in Alabama;³¹ and the MetaArchive Cooperative, a community-based digital repository solution, serving over 20 libraries, archives and other cultural memory institutions in three countries.³²

In the context of SDGs, there are obvious opportunities for partnerships, both between national archives and national statistical agencies and between countries. Such partnerships could facilitate the establishment of shared digital preservation capabilities.

Hybrid approaches

It is always possible to adopt more than one of these approaches for different elements of the solution. Such a hybrid approach can be very flexible and cost-effective, but it may also increase complexity.

Using consultancy services

Consultancy can provide important support for organisations in understanding their preservation needs. Typically, this may involve specific projects, for example auditing existing holdings, defining requirements, developing policies and procedures, or advising on standards. To benefit from a consultancy – which can be costly – it is essential to have a clear, focused brief and to choose both the project and the consultant with great care. However, at its best such consultancy can bring an impartial and expert perspective to issues that must ultimately be implemented internally.

Implementation and operational implications

Implementing a digital preservation service

The task of establishing facilities for managing and preserving digital records and data can vary enormously, depending on the complexity of the requirements and the approach taken. In some cases, this will involve a major technical and organisational change programme; for others it will be much more straightforward. However, in all cases, success will depend on careful planning and organisation. The steps involved, from galvanising organisational support to delivering an operational service, have been described in detail elsewhere,³³ and are summarised briefly here.

No initiative of such importance can succeed without first securing the organisation’s support, not only in terms of allocating the necessary resources but also of achieving the cultural change that is essential to achieving the benefits of digital preservation and ensuring its long-term viability. In turn, such support can only be achieved by first understanding the specific concerns that will trigger action for a particular organisation or in a specific context. This must also be rooted from the start in an understanding of the needs of all relevant stakeholders, from data creators to those who will use the information now and in the future. An early objective should be to develop and agree a digital preservation policy and strategy, including a commitment to regularly review and update them to address evolving issues. These policies will help to build an institutional commitment, laying the foundations for developing a detailed business case to secure the time and resources needed to make that commitment a reality.

There must be a convincing argument for the long-term value of digital records and data, based on evidence such as case studies and data about, for example, the economic or societal value of specific information or data, and the costs that would be incurred if it were to be lost. In the case of SDGs there is ample evidence that without reliable information, the goals cannot be measured reliably. It also will be important to consider the various available options, such as those discussed above, to identify the most appropriate and economically achievable approach, taking account of the benefits and risks, and any dependencies on other projects or activities. It will take time to create and gain approval for such a business case, but the reward will be a practical and achievable path to developing a digital preservation capability that is appropriate and proportionate to the need. The Digital Preservation Coalition has developed a toolkit that is thoroughly recommended for anyone creating a business case.³⁴

Once the case has been approved, the next step will be to define detailed requirements, which again must be firmly rooted in the needs of stakeholders, including content creators, information managers and those with curatorial responsibilities, end users, decision-makers and funders, and technology suppliers. All key business processes will need to be modelled, showing, for instance, how information is used in the organisation. Having a comprehensive requirements catalogue is one of the most important building blocks for developing a successful digital preservation capability. A wide range of organisations have completed similar exercises. The digital preservation community is very open and collaboration-minded, and many excellent examples are available to draw upon.

The requirements catalogue should inform the design and implementation of the solution, whether this be through a commercial procurement exercise, building a solution in-house, or some form of partnership. The solution should cover all the key digital preservation functions, as outlined above, from acquisition and ingest, metadata management, bitstream and logical preservation, and dissemination. Implementation will always involve designing, building and testing both the technologies and the procedures for using them, often through a number of iterations. In parallel, the other aspects of the service will need to be put in place, as illustrated below.

Governance

Organisational and governance structures needed to deliver the service must be established, including staffing, which is discussed separately below. This must include legacy planning in the event of organisational change.

Roles and responsibilities

While the precise types and numbers of staff required to operate a digital repository will vary considerably, a number of generic roles are likely to be needed, including:

In all cases, some redundancy is highly desirable, to avoid any single-points-of-failure. If a failure does occur in a digital preservation process, other processes (duplicate or otherwise) can negate the impact of the failed process.

Training

Although digital preservation draws on a wide range of skills, including many that will already be part of the core expertise of librarians, archivists and other information management professionals, it is a specialism in its own right and therefore needs to be supported by high-quality training in digital preservation theory and practice. Digital preservation awareness and expertise is increasingly recognised as being a core part of information management professionals’ core skill set, and it is now explicitly addressed within relevant graduate and postgraduate training courses, particularly in well-resourced countries, where most library and archives courses now cover digital preservation in some depth.

Training is, of course, also essential for existing staff who need to develop new expertise. A number of established face-to-face and online training courses are available and are highly recommended for anyone seeking to develop practical skills and knowledge. Notable examples of online training and related resources that are suitable for and accessible to an international audience (albeit primarily in English only), include:

Policies and procedures

All the policies and procedures necessary to underpin the service need to be developed and then maintained. These should cover issues such as collections development, collections management, documentation, preservation, business continuity and access.

Conclusion

Preserving digital information is essential both for programme management and for measuring the SDGs. It requires sustained active management, which in turn necessitates specialised skills and technologies, as well as institutional will to establish and maintain new capabilities. The operational demands of digital preservation, for instance setting up governance structures and determining training needs, are just as important to consider as the technical ones. The long-term success of any service will depend on both being met appropriately.

Digital preservation is, and will continue to be, a growing challenge for organisations across the world, but it is a challenge that can and must be addressed. Practical solutions are available to suit a wide range of needs and are achievable by organisations of all sizes and with widely varying resources at their disposal. Many different approaches to implementation are possible, and it is essential to carefully choose the right option to suit the needs of the organisation. For organisations in lower resource environments, the open source, preservation-as-a-service or partnership models may offer the most practical way forward. The need for digital preservation must be addressed if SDGs are to be met-careful planning for this fundamental aspect of global development from the outset will have wide benefits for transparency and accountability.