Dealing with the massive wave of data flowing into businesses will be one of the defining management issues of the next decade. One company that is already dealing with this is New Zealand’s Weta Digital.
Wellington based Weta that’s best known for its work on Lord of the Rings and is part owned by director Peter Jackson employs 1400 staff for its movie special effects work and has won five visual effects Academy Awards over its 23 years of operations.
Kathy Gruzas, WETA Digital’s CIO, spoke to Decoding the new Economy at the Oracle OpenWorld forum in San Francisco this week about some of the challenges in dealing with the massive amount of data generated by the movie effects industry.
“We have some very heavy loads.” Kathy states. “We push our systems to the limit.”
Applying powerful systems
One challenge is the sheer computing power required, ‘the render frame processes one frame per server until you have four seconds of footage. Sometimes that takes over night or even longer and for that we use a lot of storage,” Kathy says. “The render farm being six thousand servers will write 60 to 100 terabytes of data a day and read a quarter to half a petabyte each day.”
“We need systems that will be very large to handle the volume of data we generate but also be very quick to handle those read and writes.”
“One render could use a thousand computers, sometimes more, and all of those will be reading and writing against the same block of storage so we have our own software layer that directs those loads but we try to minimise the load on our storage but we have the worst work load you can imagine with lots of servers, lots of small reads and writes and many of them random and concurrent with pockets of hot files.”
Despite the automation, the business is still extremely capital intensive. “In visual effects you probably need at least three hundred artists to work on one film, it’s a very labour intensive process to do the artistry and much like a production line.”
Going mobile
The nature of modern movie production means the effects teams are now part of the shoot which adds another level of complexity for Weta. “Although we are visual effects which is largely post-production we do go out with crews when they’re shooting the movie so we can do reference photography,” says Kathy.
“We do 3D scans so if we need to do something digitally and we do motion and facial capture as well,” she says. “There are 240 muscles that we tweak individually to get the expression. That’s a huge amount of data to capture.”
To do this, Weta created their own ‘road case’ that contains everything they need to grab the shots and store the data they need, “you can’t ask the director retake the shot because we missed something.”
Into the forest
“We have to take the case into the forest and into the rain and everywhere. It’s good having that roadcase that has storage, networking and servers in it.” The case, which was self assembled by Weta’s team is “probably the most travelled Oracle system on the planet,” laughs Kathy with “lots of data capture and sub-rendering.”
Weta’s story illustrates just how managing data is becoming a critical issue for companies. While movie special effects is very much a specialised field that’s far ahead of the curve in its technology use than most businesses, they do show the importance of managing and securing their data.
For other businesses, lessons from Weta is understanding your company’s – including staff and customers’ – needs then investing in the right tools to deliver is essential.
One important difference between technology intensive businesses like Weta and most other organisations is the New Zealand company is doing most of its processing and storage in house. Those without the same needs will almost certainly be shifting these tasks onto the cloud.