AN INDUSTRY PERSPECTIVE - Malcolm F. Davidson
The Hidden Costs and Complexities of Storage, or "Pay As You Play"

As the basic cost of storage continues to decline, many entertainment companies and individuals are beginning to consider archival projects for digitizing their valuable physical assets. With many possible solutions available, picking the right technology can clearly be a challenge.

In previous decades, conventional wisdom dictated that a project and system had to be clearly defined and designed prior to development and subsequent deployment. This often meant a fully centralized design was implemented in technology. The problem was that it quickly became outdated. However, such are the rapid changes in technology today that it is now feasible and preferable to design distributed systems that employ a mixture of technologies that can be easily replaced with new sub systems as necessary.

When looking at hard disk solutions one should consider the costs of hardware and software required to support the spinning media.

When looking at hard disk solutions one should consider the costs of hardware and software required to support the spinning media. With system speeds increasing, power consumption is also increasing and this means increased electricity costs. In addition to the folks actually doing the transfers, staff must be employed to support all the backend equipment.

Server farms with RAID arrays can be an option, however it is often a wise move to rent space from a company providing such a service. Their operations are optimized for storage and systems, whereas yours may not be. Hidden costs exist to cover such things as air conditioning, disaster recovery and security. Also, 24 X 7 support can drive up the costs and service contracts are often obligatory with all of the equipment one purchases.

Alternatively, tape backups are a solution with both AIT (Advanced Intelligent Tape) and LTO (Linear Tape Open) being the most common. Some companies have chosen robotic tape systems. If tape is used as a long-term archive it may be acceptable to keep the tapes on shelves. Access requirements to files will dictate whether the tape cartridges have to be near line or offline.

File integrity is an important consideration and some systems have employed checksum techniques such as MD5 hashing for file verification. In the mid 1990's a technique called IFL (Infinite File Life) was proposed to ensure tapes were cross-copied prior to the occurrence of any uncorrectable errors.

Data communication speeds can be a limiting factor for any distributed systems given the large file sizes of high-resolution audio (and any video) and removable hard disks have been utilized to transfer many gigabytes of data between data centers within reasonable distances. Also, Internet2 (an industry, university and government collaboration) and an all-fiber network, promises high transfer rates.

Having a clear understanding of system needs, with well defined lines of demarcation for online, near line, and offline requirements, coupled with an accurate sense of present and future technology options can ensure success as one embarks upon the path towards digitization of valuable physical assets. So indeed, in regards to conceiving and laying out preservation projects, what years ago might have seemed poor planning, indecision and lack of foresight now could mean real wisdom and the resulting cost savings. Pay as you play seems the best motto for today.

Malcolm Davidson is the owner of Yestertainment.com, a Web site devoted to the distribution and consumption of music, books and movies from yesterday's entertainment world. Mr. Davidson also consults on Systems and Process analysis for a number of clients.

Malcolm previously served as Vice President of Technology at Sony Music, where he headed up the effort to design and build a Global Archiving System for the US as well as other affiliates worldwide. He can be reached at: malcolm_davidson@yestertainment.com.