Safeguarding Audiovisual Information for Future Generations
Dietrich Schüller / Consultant - Phonogrammarchiv, Austrian Academy of Sciences - Austria, dietrich.schueller@oeaw.ac.at
Keynote: INFORUM 2016: 22nd Annual Conference on Professional Information Resources, Prague, May 24-25, 2016
With their development in the course of the 19th century, audiovisual technologies opened an entirely new perspective of information gathering, storage, and retrieval. While traditional text documents are representations of human thoughts captured by script, audiovisual documents are equivalents to physical – acoustical and/or optical – phenomena. All are machine made, and, except photographs, also machine readable documents.
While today products of the entertainment industry are a dominating factor in this spectrum of documents, it is noteworthy that it was scientific interest that triggered the development of audiovisual technology, not the vision of an audiovisual industry which today has a global dimension of more than 100 bn dollars a year. A strong incentive for the development of phonography was research into the physics of human language, and the interest in folk and exotic musics and languages. The exploration of details of movement stood at the cradle of film. Consequently, as Edison cylinder recorders became available around 1890, scholars systematically employed specifically sound recording for their research, which created many collections of early sound recordings held in research institutions and museums. This led also to the establishment of sound archives: the first was the Vienna Phonogrammarchiv, established by the Academy of Sciences in 1899, soon followed by similar archives in Paris, Berlin (1900), and St Petersburg (1908). The quantitative dimension of scholarly collections was rather modest, as early sound recording equipment was bulky, film recording was rare, because of the considerable costs.
Immediately after the science pioneers, right at the end of the 19th century, followed the conquest of sound recording and film technology for entertainment purposes, which developed, as explained, fairly quickly to a global industry of considerable dimensions. The systematic collection of commercial sound records and films, however, only started in the 1920s, when national phonotheques, record collections at libraries, radio and film archives were established. A great step forward was made after World War II, after magnetic tape as a recording and storage medium became internationally available, for audio first, but from 1956 also for video, which quickly made broadcasters next to the record and film industry a second creative sector. And with the availability of portable magnetic tape audio recorders in the mid1950s, and portable video recorders around 1980, the scientific community had affordable and viable tools to catch up in their endeavours to systematically record languages, musics, dances and rituals from cultures all over the world, which today constitute the sources proper of the linguistic and cultural diversity of human kind.
To date, these three creative sectors – the record and film industry, the radio and television broadcasters, and the academic and cultural bodies – have produced a considerable legacy of audiovisual materials: film, television and music productions, which are artistic creations in their own right, political and cultural documentations, which are meanwhile the key sources to understand contemporary history, and the academic corpora that provide the modern basis of many scholarly disciplines.
Instability, vulnerability, and traditional concepts of preservation
The safeguarding of audiovisual documents is organised along a variety of structures: Libraries often maintain audiovisual departments, some of them concentrating on published materials, while several countries have established audiovisual national archives holding published and unpublished documents. Normally, radio and television broadcasters run their own archives. Also film preservation, because of its very specific needs, is mostly in the hands of (national) film archives. As audiovisual preservation is demanding in terms of equipment and expertise, concentration on few institutions is a natural trend.
Originally, all audiovisual archives followed the traditional preservation paradigm that also governs museums: to preserve the object, the original document, placed in their care. With the traditional materials associated with text documents this is, with several limitations such as degradation of wood pulp paper, a viable strategy. Audiovisual carriers, however, are more vulnerable, and more susceptible to chemical and physical deterioration than materials of traditional text documents. The most widely known preservation problem is associated with cellulose nitrate film in use until the mid-1950s. Chemically unstable, it has a tendency to self ignition, but also safety film is not resistant to deterioration. This calls for specific climatic storage parameters. Amongst audio carriers, instantaneous lacquer discs, often originals of outstanding historical value used before the availability of magnetic tape, are disintegrating, and in the 1980s, severe stability problems of magnetic audio and video tape became apparent. Apart from chemical stability problems, film and mechanical sound carriers are prone to mechanical damage by normal use.
Audiovisual archives managed these problems by separating “untouchable” archival masters, which were held under specific storage conditions, from working (access) copies. Machine readability, the other inherent structural problem of audiovisual documents (except photographs), was not so much a problem in analogue times as formats have been fairly stable, in audio even over decades.
Change of preservation paradigm around 1990
With the advent of digital recording technology in the early 1980s, great hopes have been placed in the development of stable archival formats. However, technology developed fast, and professional applications were increasingly following trends developed for the consumer market. Audio and more typically video formats of ever shorter life cycles emerged and even vanished from the market shortly thereafter.
Around 1990 it became clear that sooner or later all audiovisual carriers will deteriorate beyond retrievability, and, furthermore, the considerable variety of dedicated formats would constitute an unmanageable problem to keep machines and spare parts ready for accessing the documents. This led to a change of the preservation paradigm: Preservation should concentrate on the content of audiovisual documents, by extracting the signals from their original carriers, converting analogue information into digital files, and by copying these losslessly from one preservation platform to the next. Because of the ever growing amounts of holdings, and the relatively frequent migration cycles, automation was one of the key issues of this concept: archives’ holdings would become easily accessible, automatically controlled for their data integrity, refreshed if needed, and with a minimum of manual intervention migrated to new preservation platforms before the old ones become obsolete.
This concept was not accepted without controversy. Traditionally minded archivists saw it as a betrayal of archival principles, and several open minded voices were sceptical about its viability. However, its practical implementation started already in 1992/93 at the initiative of German Broadcasters. Apart from technical considerations, the driving force was the intention of public broadcasters to activate their archival treasures as weapon against the upcoming private broadcasters. The pilot project was started by the Südwestfunk Baden-Baden in cooperation with IBM. At the time, the European Broadcasting Union extended the Wave file to BWF, the Broadcast Wave Format, which became a de-facto standard. Work stations with computer aided quality control of the analogue-to-digital transfer were developed that permitted the simultaneous conversion of several analogue originals by one operator.
Digital Mass Storage Systems started out as near line tape robots, but with increasing affordability, RAID systems took over ever greater amounts of holdings. Today, files are generally held online, normally, while tapes are used as back-ups. Typical storage capacities of radio and national archives are in the order of hundreds of terabytes, while television archives often reach the order of petabytes.
By the mid-1990s, this concept of content migration was generally accepted. As costs for IT equipment and storage media came down considerably, national and research sound archives followed, and video archives also joined the principle of digital file archiving. Meanwhile, the change of film production and film projection to digital, and the associated retreat from raw film production and laboratory services forces film archives, the last stronghold of analogue archiving, to change over to digital content migration.
Principles of content migration
The separation of the content from the carrier is a critical issue. The principles of this concept can only be accepted if the signal can be completely extracted from the original carrier. The pragmatic answer to this question is “yes”, but with some limitations which concern more secondary technical information than the signal. The separation from the original, however, must be compensated to the best possible extent by rigorous metadata assessment.
For audio, the Technical Committee of IASA, the International Association of Sound and Audiovisual Archives, has defined the principles and practices in its standards TC 03 and TC 04; those for video digitisation are forthcoming. The most important parameters can be summarised as follows: the use of modern equipment in full compliance with the historical format parameters of the original carriers, the unmodified transfer to openly defined high definition digital target formats, the interdiction to use data reduction (“lossy compression”) for digital preservation masters as well as attempts to “improve” the signal through signal restoration. The rules for the preservation of digital objects are in line with the general standards set by computer experts: sufficient copies in separate technologies and locations, adequate data integrity checking, and timely migration.
The aim of IASA recommendations is to extract the information from the carriers, not to reconstruct the signal as it may have been perceived by a listener at the time. Modern reproduction normally retrieves more and less distorted information than original equipment. While in principle trusting in the viability of the concept, IASA recommends to keep transferred originals “as long it is possible”. This is a safety measure for possible improvement in transfer technology and the extraction of technically encoded metadata, which are presently beyond transfer routines. Broadcast archives often question this policy by establishing a hierarchy of importance to judge between keeping and dumping originals.
Looking at the original carriers as an object of art, or part of an artistic performance, however, adds another dimension to the scene. Video art will need different extraction criteria and historical equipment, mainly CRTs, for faithful reproduction. And as film archiving comes into the scene, and with it the aspect of historic film screening, reluctance to admit separation of the signal from the carrier, at least for parts of the legacy, is getting some ground.
Present situation and global perspective
From the 1970s to the 1990s, the instability of carriers has been the major concern of audiovisual archivists. However, some of the painted scenarios, specifically those for magnetic tape and cellulose nitrate film, did not happen (to date) to the extent feared. This may, however, also be the result of the improvement of storage conditions.
But the aspect of obsolescence of analogue and digital carrier based formats, and the speed of withdrawal of manufacturers from spare part production and professional service support around and after 2000 has been underestimated. As audio and later also video moved to computer based production and storage, magnetic tape based formats gradually lost their importance, until they totally vanished from the scene, after flash cards of sufficient storage capacity had become affordable and had replaced magnetic cassettes in professional and consumer recorders. Spare part supply and professional service support is fading rapidly, and missing equipment can only be obtained from second hand sources. While the construction of new replay equipment for historical audio formats is possible, offering considerably higher replay fidelity than original equipment, it is an illusion to think that e.g. digital video cassette players can be “handmade”, after serial production has ceased. Present estimates warn that for holdings still sitting on original magnetic tape transfers can only be kept up until 2025.
Apart from lack of replay equipment, another endangering factor is shrinking expertise. Substantial knowledge of an ever increasing sector of audiovisual preservation, specifically signal extraction from analogue and digital single carriers, is only found in a relative small number of audiovisual archives and specialized service providers. In the absence of other appropriate professional institutions, they have to maintain and further develop the expertise and the training of young experts to take over the care for the audiovisual legacy.
Under the prevailing circumstances, wealthy broadcasting and national audiovisual archives have already – at least selectively – brought their holdings into professional digital repositories, or have at least a realistic option to ingest their holdings in time. Many archives in developing countries, but also underfunded collections worldwide may not be able to finance their transfer in time. And this will lead to the unprecedented situation that documents in good physical state will end up as useless orphans, unplayable because sophisticated equipment for their retrieval will be missing.
This most significant threat will hit the many small and scattered collections made over the past 60 years by linguists, folklorists, social anthropologists, oral historians and other scholars, who have used magnetic audio recorders, later video recorders to produce a considerable stock of audiovisual recordings, which form the documents proper of the cultural and linguistic diversity of human kind. These documents are the primary sources of our present knowledge of many sectors of the humanities. Typically, and most unfortunately, only a small part is adequately preserved, while the greater portion – estimates are around 80% – are without archival custody in the narrower sense. Many of these collections are part of research institutes without any preservation budgets, often even without awareness of the immediate threat of loss. Many even still sit on the shelves of the scholars that have recorded them. This unprofessional practice has worked somehow in analogue times, but digitisation and further management of keeping audiovisual preservation files alive for the generations to come needs awareness of the threat, financial resources, and readiness for cooperative models. A noteworthy example of such cooperation is the Media Digitization and Preservation Initiative of the University of Indiana, Bloomington, which has recently started to transfer the more than 500,000 single carriers, held by the various institutions of the university, in a concerted action to a central digital repository. Similar initiatives would be needed at university and also at national levels to safeguard the many scattered collections, too small for autonomous preservation, which otherwise would inevitably be lost. It would be an unprecedented scenario that libraries of secondary sources would survive, while the primary sources would be lost, because of unawareness and neglect of organisation.
Acknowledgement and support of linguistic and cultural diversity is one of the basic aims of UNESCO. Consequently, as audiovisual documents are the most appropriate sources to document this diversity beyond written descriptions, UNESCO is presently preparing a survey which – under the name “Magnetic Tape Alert” – aims to warn governments, decision makers, and stake holders of the imminent threat and to get a realistic view of the quantitative dimensions of endangered collections as a basis for further planning.
In summarising, long-term audiovisual preservation by subsequent content migration is not only viable, but without realistic alternative. Because of the precarious situation of vanishing replay equipment, magnetic tape formats are in need of urgent action. Success will not only depend on appropriate funding, but substantially on strategic and cooperative planning, preferably at national levels.
References:
IASA Technical Committee: The Safeguarding of the Audio Heritage: Ethics, Principles and Preservation Strategy, edited by Dietrich Schüller. (= IASA Technical Committee - Standards, Recommended Practices and Strategies, IASA TC-03), Version 3, 2005
web version: http://www.iasa-web.org/downloads/publications/TC03_English.pdf
Also available in French, German, Italian, Russian, Spanish, Swedish, and Chinese
IASA Technical Committee: Guidelines on the Production and Preservation of Digital Audio Objects, edited by Kevin Bradley. (= IASA Technical Committee - Standards, Recommended Practices and Strategies, IASA TC-04).Second edition 2009
web version: http://iasa-web.org/audio-preservation-tc04
Indiana University Bloomington: Media Digitization and Preservation Initiative. https://mdpi.iu.edu/
Schüller, Dietrich: Towards the Automated "Eternal" Sound Archive. In: Boston, G. (Ed.), Archiving the Audiovisual Heritage. Proceedings of the Third Joint Technical Symposium, Ottawa 1990, 1992, 106-110
