Jan 172017
 
Is Yahoo! recovering under new CEO Marissa Mayer

Slowly it’s dawning on government agencies how serious online data breaches can be. That can only be a good thing.

With a billion account details exposed the Yahoo! data breach announced last year was the greatest internet security failure to date.

Now Australian government agencies are worried about the scope of the breach and the number of politicians and officeholders whose credentials may have been affected.

Other government officials compromised include those carrying out sensitive roles such as high-ranking AFP officers, AusTrac money laundering analysts, judges and magistrates, political advisors, and even an employee of the Australian Privacy Commissioner.

The ramifications of this breach are far broader than just a few malcontents grabbing the contents of disused Yahoo! mail accounts or being able to hack Flickr profiles, many of the passwords will have been used on other services, compromised profiles linked to other platforms and the possible for identity fraud is immense.

With social media and cloud computing services coupled to these accounts, it’s quite possible for someone’s entire life to be hijacked thanks to one insecure service as Wired’s Matt Horan discovered a few years ago.

Just like individuals and businesses, the ramifications of careless organisations allowing private information to be stolen can be severe for governments. It’s right that Australian agencies are concerned about where this data has gone.

The official response to continued data breaches has been weak at best so it is good that suddenly agencies are having to face the consequences of the biggest one.

A widespread scare about insecure data may be what’s required to see governments start taking data security and citizen privacy seriously. That may be the positive side of the Yahoo! breach.

Jan 112017
 

Last year the Australian Federal government had a smart idea. To fix its chronic budget deficit, it would use data matching to claw back an estimated three billion dollars in social security overspending.

Unfortunately for tens of thousands of Australians the reality has turned out to very different with the system mistakenly flagging thousands of former claimants as being debtors.

How the Australian government messed up its welfare debt recovery is a cautionary tale of misusing data.

Data mis-match

At its core, the problem is due to the bureaucrats mismatching information.

Australia’s social security system requires unemployment or sickness benefit claimants file a fortnightly income statement with Centrelink, the agency that administers the system, and their payments are adjusted accordingly.

Most of those on benefits only spend a short time on them. According to the Department of Social Services, two thirds of recipients are off welfare within twelve months of starting.

Flawed numbers

Despite knowing this, the bureaucrats decided to take annual tax returns, average the individual’s income across the year and match the result against the fortnightly payment.

That obviously flawed and dishonest method has meant hundreds of former welfare recipients have been falsely accused of receiving overpayments.

Compounding the problem, the system frequently mis-identifies income because it fails to recognise employers may use different legal names, leading to people having their wages double counted and being accused of not reporting work.

Shock and awe

Under pressure from their political masters, the aggressive tactics of Centrelink and its debt collectors have left many of those accused shocked and distressed.

I can barely breathe when I think about this. My time period to pay is up tomorrow. I asked them for proof before I pay and I have heard horror stories of debt collection agencies, people being asked to pay so much, people being told there will be a black mark on their credit. I am so terrified. It’s so stupid for me to be terrified but I can’t help it. I am a student, I can’t afford anything!

Reading the minister’s response to criticisms, it’s hard not to come to the conclusion that intimidation was a key objective.

The numbers of people involved are staggering. The department of Social Services reported 732,100 Australians received the Newstart unemployment allowance in 2015-16. Should 66% of those have moved off the benefit during the tax year then up to 488,000 people will receive ‘please explain’ notices.

Nearly half a million people being falsely accused of welfare fraud is bad enough, but that is only last year’s figures – due to a  law change by the previous Labor government, there is no limit to how far back Centrelink can go to recover alleged debts.

The System is working

Claiming the Centrelink debacle is a failure of Big Data and IT systems is wrong – the system is working as designed. The false positives are the result of a deliberate decision by agency bosses and their ministers to feed flawed data into the system.

How this will work out for the Australian government as tens of thousands more people receive unreasonable demands remains to be seen. Recent comments from the minister indicate they are hoping their ‘tough on welfare cheats’ line will resonate with the electorate.

Regardless of how well  it turns out for the Australian government, the misuse of data by its agencies is a worrying example of how governments can use the information they collect to harass citizens for short term political advantage.

Beyond welfare

While many Australians can dismiss the travails of Centrelink ‘clients’ as not concerning them, the same data matching techniques have long been used by other agencies – not least the Australian Taxation Office.

With the Federal Treasurer threatening a campaign against corporate tax dodging and the failure of the welfare crackdown to deliver the promised funds, it’s not hard to see small and medium businesses being caught in a similar campaign using inappropriate data.

More importantly, the Australian Public Service’s senior management’s incompetence, lack of ethics and proven inability to manage data systems is something that should deeply concern the nation’s taxpayers.

In a connected age, where masses of information is being collected on all of us, this is something every citizen should be objecting to.

Jan 092017
 

For most of its existence, Uber hasn’t been shy about claiming to be at the forefront of the future of transport which fits into yesterday’s announcement of Uber Movement which promises to provide aggregated and anonymised trip data to give communities and businesses an overview of road usage in their districts.

Jordan Gilbertson,  one of the company’s Product Managers, and Andrew Salzberg, Head of Transportation Policy, described how Uber intends to make transit time data available.

Uber trips occur all over cities, so by analyzing a lot of trips over time, we can reliably estimate how long it takes to get from one area to another. Since Uber is available 24/7, we can compare travel conditions across different times of day, days of the week, or months of the year—and how travel times are impacted by big events, road closures or other things happening in a city.

As the Washington Post reports, transport agencies do already have a lot of data on some aspects of commuter behaviour – particularly public transport usage – and the Uber information fills as ‘missing part of the puzzle’.

Taxis and buses are also increasing equipped with real time tracking equipment that also gives this data while traffic services like Wayze have been collecting this information for a decade.

So agencies aren’t short of this data and the concentration of Uber’s customer base in more affluent areas means their information may be skewed away from poorer areas. Recently a Sydney taxi driver mentioned to me how he’d stopped driving for Uber because most of the city’s sprawling Western Suburbs where he tended to drive didn’t use the service.

Uber’s offer is another piece in their data strategy that sees the company being a data hub for the logistics industry. It also helps if you’ve co-opted governments into your scheme.

Dec 012016
 

As with every vendor conference, this year’s AWS Re:Invent convention in Las Vegas bombarded the audience with new product announcements and releases.

One of the interesting aspects for the Internet of Things was the announcement of Amazon Greengrass, a service that stores machine data on remote equipment which combines the company’s Lambda serverless computing and IoT services.

Further pushing Amazon’s move into the IoT space was CEO Andy Jassy’s announcement that chip makers such as Qualcomm and Intel will be building Lambda functions into their chipsets, further embedding AWS into the ecosystem.

Jassy also touted the company’s new Snowball Edge, a slimmed down version of their Snowball data transfer unit that also include some processing features, that is aimed at storing machine data at remote or moving locations such as ships, aircraft, farms or oil rigs.

That latter function ties into one of the key aspects about the Internet of Things – that most data doesn’t have to, or can’t, be transmitted over the internet. This is something companies like Cisco have focused on in their edge computing strategies.

With AWS dominating the cloud computing industry – Gartner estimates the company is ten times bigger than the next 14 companies combined – the worry for customers and regulators will be how much control the organisation has of the world’s data.

It’s hard though not to be impressed at the range of products the company has, and the speed they get them to market, the onus is on companies like Microsoft, Google and Facebook to allocate the resources and talent to match AWS in the marketplace.

Sep 272016
 
Kennedy Nixon Presidential Debate 1960

As the 2016 US Presidential race enters its final stages, it’s interesting to see how data is being used by American political candidates and what this means for business.

During last week’s Oracle Open World in San Francisco a panel hosted by the company’s Political Action Committee featured Stephanie Cutter, who worked on Obama’s 2008 and 2012 campaigns, and Mike Murphy, a Republican operative whose most recently worked on Jeb Bush’s primary effort against Donald Trump.

While the discussion mainly focused on the politics – “Crazy times seem to require crazy candidates” says Murphy – it was the technology aspect of modern elections that was notable.

Setting the data standard

The Obama campaign of 2008 set the standard for how modern political campaigns used social media and information, “we revolutionized how data analytics helps predict how people will vote and how they will persuade voters to turn out.” Cutter said.

“We put a big investment into it and Republicans have caught up,” she continued. “The key though was we relied on our own data and nothing that was out in the public domain. We didn’t rely on one piece of data, we had multiple sources. We had an analytics program where we were making 9,000 calls a night where we were predicting the votes.”

Murphy agreed with the political campaigns using data, “the kind of polling you see in the media has kind of vanished in campaigns where they have money to spend on research.” He said, “we don’t do telephone polling any more because we have so much data we can collect.”

Capturing everything

“We capture everything. We have about four hundred data points on the American voter and we’ll have five hundred in the next two years. We’ll be able to build massive data models without phone polling,” Murphy pointed out. “We’re waiting for the tech folk to get ahead on AI so we can predict what voters are going to do in two weeks.”

Despite the amount data collected by US political parties, the real key to success is the candidate’s organisation and management. Cutter made a strong point about the strength of Obama’s campaign team in both the 2008 and 2012 campaigns.

How the US political parties use data points to how businesses will be managing data in the future. Increasingly using information well is going to be the measure of successful organisations in both politics and industry.

Sep 232016
 

Dealing with the massive wave of data flowing into businesses will be one of the defining management issues of the next decade. One company that is already dealing with this is New Zealand’s Weta Digital.

Wellington based Weta that’s best known for its work on Lord of the Rings and is part owned by director Peter Jackson employs 1400 staff for its movie special effects work and has won five visual effects Academy Awards over its 23 years of operations.

Kathy Gruzas, WETA Digital’s CIO, spoke to Decoding the new Economy at the Oracle OpenWorld forum in San Francisco this week about some of the challenges in dealing with the massive amount of data generated by the movie effects industry.

“We have some very heavy loads.” Kathy states. “We push our systems to the limit.”

Applying powerful systems

One challenge is the sheer computing power required, ‘the render frame processes one frame per server until you have four seconds of footage. Sometimes that takes over night or even longer and for that we use a lot of storage,” Kathy says. “The render farm being six thousand servers will write 60 to 100 terabytes of data a day and read a quarter to half a petabyte each day.”

“We need systems that will be very large to handle the volume of data we generate but also be very quick to handle those read and writes.”

“One render could use a thousand computers, sometimes more, and all of those will be reading and writing against the same block of storage so we have our own software layer that directs those loads but we try to minimise the load on our storage but we have the worst work load you can imagine with lots of servers, lots of small reads and writes and many of them random and concurrent with pockets of hot files.”

Despite the automation, the business is still extremely capital intensive. “In visual effects you probably need at least three hundred artists to work on one film, it’s a very labour intensive process to do the artistry and much like a production line.”

Going mobile

The nature of modern movie production means the effects teams are now part of the shoot which adds another level of complexity for Weta. “Although we are visual effects which is largely post-production we do go out with crews when they’re shooting the movie so we can do reference photography,” says Kathy.

“We do 3D scans so if we need to do something digitally and we do motion and facial capture as well,” she says. “There are 240 muscles that we tweak individually to get the expression. That’s a huge amount of data to capture.”

To do this, Weta created their own ‘road case’ that contains everything they need to grab the shots and store the data they need, “you can’t ask the director retake the shot because we missed something.”

Into the forest

“We have to take the case into the forest and into the rain and everywhere. It’s good having that roadcase that has storage, networking and servers in it.” The case, which was self assembled by Weta’s team is “probably the most travelled Oracle system on the planet,” laughs Kathy with “lots of data capture and sub-rendering.”

Weta’s story illustrates just how managing data is becoming a critical issue for companies. While movie special effects is very much a specialised field that’s far ahead of the curve in its technology use than most businesses, they do show the importance of managing and securing their data.

For other businesses, lessons from Weta is understanding your company’s – including staff and customers’ – needs then investing in the right tools to deliver is essential.

One important difference between technology intensive businesses like Weta and most other organisations is the New Zealand company is doing most of its processing and storage in house. Those without the same needs will almost certainly be shifting these tasks onto the cloud.

Jul 122016
 
Big data takes our online, shopping and social media use it is the business challenge for our time

“We want to be the Wayze of enterprise software” is the line being repeated by executives at the Inforum2016 conference in New York today.

This is an interesting strategy for Infor, who provides a range of enterprise software tools to help companies track what is going on in their business, as Wayze is built upon aggregating user data to identify traffic problems to improve commuting times. It’s no surprise that Google bought the company a few years ago.

Infor position though is slightly different as it’s aggregating individual clients’ data for them. In a world where organisations are struggling not to be overwhelmed by information, Informa are in a good position, even if their executives do overdo it on the buzzwords.

Which leads us to another buzzphrase – design thinking – which has been drifting in and out of fashion over recent years. During the opening keynotes one of the comments was about the rise of  “network thinking.”

“Eighty percent of what most companies do deals with data from outside of their organisation,” says Kurt Cavano, Infor’s General Manager of their commerce cloud division. “We’ve seen in the power of networks with sites like Facebook, LinkedIn and Wayze.”

“Nobody wants to be on a network but everyone’s on a network. It takes a long time to build but once you have one it’s magical. That’s what we’re thinking for business, they need to evolve.”

In one respect this is another take on the ecosystem idea, that one vital corporate asset in the connected world is an ecosystem of partners, suppliers and users, however the Infor view articulated by Cavano is much more about the flow of data rather than the goodwill of a community.

So we may well be entering a world of ‘networked thinking’ where thinking about the effects of data flows and being able to understand them – if not manage them – becomes a key executive skill.

Paul travelled to New York as a guest of Infor