USC Prevents Media from Rotting with Large Private Cloud

University of Southern California is working toward a goal of building one of the largest scholarly data collections in the world.

In November, USC announced that it is moving 8 Petabytes of data to a private Nirvanix Enterprise Cloud. To give you some perspective, that's similar to moving 8 billion 500-page books. According to Nirvanix and USC, this is the biggest private cloud implementation in the industry to date. 

With this cloud, the university can control its resources and make them available to other universities. It also has data available in multiple geographic locations. That way, if one data center goes down, USC can access information from another data center.

From a preservation perspective, everything rots, said Sam Gustman, executive director of USC Digital Repository, associate dean of USC Libraries and chief technology officer of the USC Shoah Foundation Institute. Conservatively, films last 50 years. Video tapes last 20 years. Hardrives last five years. And DVDs last as little as two years.

"The newer the technology, the faster it's rotting," Gustman said. "And so what we have are systems in place that prevent files and media from rotting."

Through the USC Digital Repository, USC will provide cloud services both in-house and to customers that include digitization, cataloging, preservation, Web access and file-server access. The repository includes three partners: The Shoah Foundation Institute, USC Libraries and USC Information Technology Services.

The Shoah Foundation Institute has 107,000 video testimonies from Holocaust survivors that the repository has digitized, made searchable, placed in a preservation system and put online in its cloud. Currently, the repository has four petabytes of data in its archives, and its IT infrastructure is ready for up to 40 petabytes.

Instead of using multiple cloud vendors to do everything, the university went with one cloud service that has all the pieces it needs in a repository for archives and content.

"You can find companies that will do this maybe let's say for documents or you can find some companies that will do some of these things for video, but not all of them," Gustman said. "We're the first to be able to do it for all kinds of media."

One of the major challenges USC has faced during this transition is making sure the cloud service and system meets both university and customer security policies. Over six months, Gustman's team talked with different groups at all levels of the university to make sure the infrastructure and proper policies were in place.

To safeguard the data in the cloud, the Digital Repository has multiple back-ups and takes a SHA-1 fingerprint of all files. This cryptographic hash function encodes the data and is also built into the system.

If the data on a film or other type of media is damaged, they instantly retire that piece of media and move it to another piece of media. They don't trust any piece of media to last more than three years. 

"It's big to be a part of an 8 petabyte cloud at this point," Gustman said. "It's a lot of data. It's a lot of responsibility. And you want to make sure that nothing goes wrong with any piece of it."