A Brief History of Data Storage

Eli Holderness likens the brain to a Sherlock Holmes attic, suggesting that individuals must prune outdated facts to integrate new knowledge, as he defends his own gaps in basic universe-related information.

He highlighted that keeping useless information necessitates discarding other potentially useful data, demonstrating the brain's finite capacity.

Technologists, in particular, require access to more data than human biological memory can reliably store, underscoring the necessity of external data storage.

This concept underpins the fundamental need for external data storage solutions that can reliably retain vast amounts of information beyond individual recall.

While nonvolatile storage must retain its state without power, Eli Holderness noted that physical durability remains secondary to the ability to actually interpret encoded data.

Holderness explained that encodings demand specialized technology or knowledge for interpretation, noting that even if an object is indestructible, its data is useless if it cannot be read.

As an example, he mentioned a QR code, which despite its physical resilience, becomes meaningless if the decoding technology disappears.

The core challenge lies in the dual requirement of a durable physical medium and a persistent, understandable encoding scheme, lest the information be lost to time despite its physical presence.

The Beowulf manuscript, the sole surviving copy of the Old English poem that inspired Tolkien, underscores the fragility of cultural heritage when dependent on a single physical artifact.

Part of the Cotton Library, this manuscript sustained significant damage during the Ashburnham House fire, demonstrating parchment's vulnerability to environmental factors like fire and aging.

Unlike the enduring nature of clay tablets, parchment is highly susceptible to such damage, making its survival a testament to luck and careful preservation efforts.

This historical example illustrates how human cultural heritage frequently relies on a single, fragile point of failure, risking immense loss from unforeseen disasters.

Early photographic processes, such as the 1826 Niépce image and the 1839 Daguerreotype, recorded data directly onto light-sensitive surfaces where light intensity created physical texture.

Unlike modern JPEGs, which embed human design choices into their encoding, chemical photography inherently records information through a direct physical reaction to light.

The Daguerreotype's long exposure time incidentally captured what may be the first candid photo of a person getting their shoes shined in 1838, illustrating this direct, unmediated data capture.

This method avoids the human bias inherent in digital encoding, where design decisions influence how data is stored and interpreted.

Vinyl records pioneered a persistent paradigm in data storage by recording information as variations in physical texture along a single spiral track on a spinning disc.

This design, read by a movable needle, served as a foundational precursor to later disk-based storage technologies.

The physical fragility of vinyl, however, makes it highly susceptible to dust, grease, and scratches, which can cause the needle to skip or repeat sections of the recorded data.

Despite their vulnerabilities, vinyl records established a critical design principle that influenced subsequent generations of data storage devices.

The 80-column standard common in early computer terminals originates from Herman Hollerith's punch cards, used to tabulate the 1890 US census.

These cards were processed by machines that completed electrical circuits through mercury pools, efficiently categorizing demographic data.

This 80-column width was subsequently adopted by early computer terminals, directly influencing contemporary coding line limits, such as the widely recognized 79-character rule.

The enduring legacy of this 19th-century census technology demonstrates its profound impact on subsequent computing standards.

Core memory utilized small rings of magnetic material and wires to store binary states, representing ones and zeros based on their magnetic polarity.

A peculiar characteristic of this technology was that reading from core memory was a destructive action, automatically resetting the value to zero after retrieval.

The need to immediately restore the state after each read added complexity to its operation, yet the non-volatile nature of these magnetic cores led to the enduring computing term 'core dump'.

Maintaining consistent temperatures through heating improved core memory performance by ensuring stable magnetic states.

Because bits are spaced too far apart on outer tracks, spinning disks read data at varying physical speeds, a limitation Apple once sought to address with costly, incompatible Variable Angular Velocity hardware.

This difference means bits on the outer sections of the disk are spaced farther apart, leading to underutilization of the storage material.

Apple once experimented with Variable Angular Velocity (VAV) to achieve denser data packing across the entire disk surface, making better use of available space.

However, VAV was eventually abandoned due to its requirement for expensive, specialized, and incompatible hardware readers, limiting its practical application.

While Wikipedia labels them as 'obsolescent,' 5.25-inch floppy disks persist within the US nuclear weapons program and other specialized fields.

The speaker pointed out that even outdated hardware maintains a market, which in turn ensures long-term employment for individuals possessing niche expertise.

A more reliable indicator of a technology's true decline, according to the speaker, is the absence of modern hardware compatibility, such as the lack of USB-C external floppy drives, rather than the perceived obsolescence of the media itself.

This persistence highlights how critical infrastructure may prioritize proven, albeit older, technologies for stability and security.

Eli Holderness emphasized that possessing the physical storage medium and the drive to read it is meaningless if the computer cannot decipher the binary data it contains.

This critical issue means that even if data is physically accessible, it becomes useless garbage binary without the corresponding knowledge of its format.

The speaker linked this problem to the concept of a 'digital dark age,' where future generations might retrieve data but find it functionally undecipherable.

Ensuring the long-term interpretability of stored data is a central challenge in data preservation, transcending mere physical accessibility.

During the 10-year period after the 1976 NASA Viking lander mission, the inability to access original tapes highlighted the risks of failing to preserve protocols alongside physical storage, especially as the lead engineers retired or died.

After the mission, the data tapes were unread for a decade, during which many engineers who understood the specific data format either left NASA or passed away.

This loss of human knowledge meant that it took months for NASA to reconstruct the original data format, as the institutional memory had vanished.

The case highlights the critical importance of documenting and preserving not just the data, but also the knowledge required to interpret it, to prevent a 'digital dark age.'

Domains smaller than 50 nanometers risk polarity flips due to heat, a physical constraint that modern drives counter by using ruthenium-separated dual-layer designs.

This effect represents a hard physical limit on how densely data can be packed onto magnetic storage media while maintaining stability.

Modern hard drives overcome this by using two oppositely magnetized layers separated by a three-atom ruthenium boundary, preventing unwanted magnetic interactions.

The engineering challenge of superparamagnetism drives innovation in material science and storage technology to push the boundaries of data density.

While SMR is efficient for packing data, the complexity of writing to these drives makes them more appropriate for 'cold data'—infrequently accessed archives—rather than random access applications.

The design's properties, particularly its density and sequential writing preference, make it excellent for long-term data preservation where write operations are infrequent but storage capacity is paramount.

Lasers create physical points in a 3D matrix within non-photosensitive fused quartz, offering a resilient archival format that survives electromagnetic pulses and magnetism.

This technique creates physical points within a three-dimensional matrix, allowing for highly dense and multi-layered data storage.

The resulting medium is exceptionally resistant to magnetism, electromagnetic pulses, and visible light, providing extreme durability for long-term preservation.

Microsoft and Hitachi have explored this concept, and current research is actively being led by Southampton University, pushing the boundaries of archival technology.

Eli Holderness emphasized that while the physical lifespan of storage mediums has largely been solved, the critical problem for long-term data preservation lies in ensuring future legibility and interpretation.

Encoding methods must be designed to guarantee that data remains understandable and usable for future generations, even millennia from now.

Translating information across vast stretches of time involves profound philosophical challenges regarding reliable human-to-human communication.

The difficulty is compounded by the fact that even contemporary fluent speakers often struggle to convey meaning accurately, magnifying the problem for future receivers who lack context or shared understanding.

Computational abstractions aim to influence the physical world, which according to software engineering principles, serves as the primary medium and ultimate target for code.

The physical world serves as both the medium for software's operation and a constant threat to its longevity and reliability.

The durability of data encoding is not merely a technical specification but a crucial aspect of human-to-human communication across time.

Ultimately, data is useless unless another human, including one's future self, can access and make effective use of it, highlighting the communicative purpose of all data storage.