Category: Big Data

Hacking the connected vehicle

American farmers hacking their tractors with Ukrainian software are a taste of what’s to come in the connected economy.

What happens when a vehicle manufacturer locks down their products’ software? John Deere’s customers are finding out as American farmers turn to Ukrainian software vendors for software to maintain their tractors.

John Deere’s behaviour is extreme as almost every component of a modern tractor has a software component which leaves farmers at the mercy of the company’s dealers and authorised mechanics.

So understandably the farmers are finding ways to hack their equipment to reduce downtime and costs, something permitted in the US after an exemption to the Digital Millennium Copyright Act (DCMA) was granted to vehicle software.

Vendor control over connected vehicles is a bigger problem for consumers than just maintaining the software, as the information collected from these devices becomes more valuable who controls that data becomes more important.

With global supply chains, increased regulatory requirements and demanding markets, the agricultural industries are probably leading the world in applying the Internet of Things and Big Data, so the challenges faced by farmers are things which will affect us all.

As everything from toasters to motor cars become connected and dependent upon code, the conflict between proprietary software, open markets and user rights is going to grow.

Consumers and the free market can only do so much to control the flows of data and who owns them. It’s hard to see how governments can’t become involved in how information is owned, traded and stored.

Deeper in data and debt

Data tools are getting more powerful as the information collected about us grows. It presents us with some important choices

Data collection agency Experian’s deal with Finicity to collect and process borrower information is an example of the how Big Data is being used by the financial services sector.

Recently I wrote a piece for Fairfax Media on the Science of Money which included some quotes from Experian’s Australian managers. They were quite explicit about their use of data.

That a company like Experian is adopting more advanced analytics isn’t surprising given the power of the tools available. What’s also driving the adoption is the proliferation of devices available to track people.

Notable among those devices are personal assistants, as David Pogue writes in Scientific American, household technologies like Amazon Alexa, Google Home and Apple Siri are vacuuming up huge amounts of data on our behaviour, likes and dislikes.

Increasingly all of this is being fed into machines that determine our suitability for marketing campaigns, credit and financial services.

For companies like Experian this is a massive opportunity although the focus on credit suitability betrays a mindset more suited to the 1980s finance boom than the more complex times of the early 21st century.

It’s hard though not to think that given a choice the finance sector will happily use these tools to take us into another subprime lending crisis which would be a shame as these technologies’ potential for allowing us to make better decisions is immense.

How we use these tools will define our businesses, economies and communities over the next thirty years. We need to be careful about some of the choices we make.

Governments count the cost of massive data breaches

The massive Yahoo! data breach is forcing governments to come to terms with the costs of data breaches.

Slowly it’s dawning on government agencies how serious online data breaches can be. That can only be a good thing.

With a billion account details exposed the Yahoo! data breach announced last year was the greatest internet security failure to date.

Now Australian government agencies are worried about the scope of the breach and the number of politicians and officeholders whose credentials may have been affected.

Other government officials compromised include those carrying out sensitive roles such as high-ranking AFP officers, AusTrac money laundering analysts, judges and magistrates, political advisors, and even an employee of the Australian Privacy Commissioner.

The ramifications of this breach are far broader than just a few malcontents grabbing the contents of disused Yahoo! mail accounts or being able to hack Flickr profiles, many of the passwords will have been used on other services, compromised profiles linked to other platforms and the possible for identity fraud is immense.

With social media and cloud computing services coupled to these accounts, it’s quite possible for someone’s entire life to be hijacked thanks to one insecure service as Wired’s Matt Horan discovered a few years ago.

Just like individuals and businesses, the ramifications of careless organisations allowing private information to be stolen can be severe for governments. It’s right that Australian agencies are concerned about where this data has gone.

The official response to continued data breaches has been weak at best so it is good that suddenly agencies are having to face the consequences of the biggest one.

A widespread scare about insecure data may be what’s required to see governments start taking data security and citizen privacy seriously. That may be the positive side of the Yahoo! breach.

When governments misuse data

The Australian government’s misuse of data in harassing welfare recipients is something that should worry all citizens

Last year the Australian Federal government had a smart idea. To fix its chronic budget deficit, it would use data matching to claw back an estimated three billion dollars in social security overspending.

Unfortunately for tens of thousands of Australians the reality has turned out to very different with the system mistakenly flagging thousands of former claimants as being debtors.

How the Australian government messed up its welfare debt recovery is a cautionary tale of misusing data.

Data mis-match

At its core, the problem is due to the bureaucrats mismatching information.

Australia’s social security system requires unemployment or sickness benefit claimants file a fortnightly income statement with Centrelink, the agency that administers the system, and their payments are adjusted accordingly.

Most of those on benefits only spend a short time on them. According to the Department of Social Services, two thirds of recipients are off welfare within twelve months of starting.

Flawed numbers

Despite knowing this, the bureaucrats decided to take annual tax returns, average the individual’s income across the year and match the result against the fortnightly payment.

That obviously flawed and dishonest method has meant hundreds of former welfare recipients have been falsely accused of receiving overpayments.

Compounding the problem, the system frequently mis-identifies income because it fails to recognise employers may use different legal names, leading to people having their wages double counted and being accused of not reporting work.

Shock and awe

Under pressure from their political masters, the aggressive tactics of Centrelink and its debt collectors have left many of those accused shocked and distressed.

I can barely breathe when I think about this. My time period to pay is up tomorrow. I asked them for proof before I pay and I have heard horror stories of debt collection agencies, people being asked to pay so much, people being told there will be a black mark on their credit. I am so terrified. It’s so stupid for me to be terrified but I can’t help it. I am a student, I can’t afford anything!

Reading the minister’s response to criticisms, it’s hard not to come to the conclusion that intimidation was a key objective.

The numbers of people involved are staggering. The department of Social Services reported 732,100 Australians received the Newstart unemployment allowance in 2015-16. Should 66% of those have moved off the benefit during the tax year then up to 488,000 people will receive ‘please explain’ notices.

Nearly half a million people being falsely accused of welfare fraud is bad enough, but that is only last year’s figures – due to a  law change by the previous Labor government, there is no limit to how far back Centrelink can go to recover alleged debts.

The System is working

Claiming the Centrelink debacle is a failure of Big Data and IT systems is wrong – the system is working as designed. The false positives are the result of a deliberate decision by agency bosses and their ministers to feed flawed data into the system.

How this will work out for the Australian government as tens of thousands more people receive unreasonable demands remains to be seen. Recent comments from the minister indicate they are hoping their ‘tough on welfare cheats’ line will resonate with the electorate.

Regardless of how well  it turns out for the Australian government, the misuse of data by its agencies is a worrying example of how governments can use the information they collect to harass citizens for short term political advantage.

Beyond welfare

While many Australians can dismiss the travails of Centrelink ‘clients’ as not concerning them, the same data matching techniques have long been used by other agencies – not least the Australian Taxation Office.

With the Federal Treasurer threatening a campaign against corporate tax dodging and the failure of the welfare crackdown to deliver the promised funds, it’s not hard to see small and medium businesses being caught in a similar campaign using inappropriate data.

More importantly, the Australian Public Service’s senior management’s incompetence, lack of ethics and proven inability to manage data systems is something that should deeply concern the nation’s taxpayers.

In a connected age, where masses of information is being collected on all of us, this is something every citizen should be objecting to.

Uber’s sharing strategy

Ubers offer to share information with city governments and agencies is part of their bigger data strategy

For most of its existence, Uber hasn’t been shy about claiming to be at the forefront of the future of transport which fits into yesterday’s announcement of Uber Movement which promises to provide aggregated and anonymised trip data to give communities and businesses an overview of road usage in their districts.

Jordan Gilbertson,  one of the company’s Product Managers, and Andrew Salzberg, Head of Transportation Policy, described how Uber intends to make transit time data available.

Uber trips occur all over cities, so by analyzing a lot of trips over time, we can reliably estimate how long it takes to get from one area to another. Since Uber is available 24/7, we can compare travel conditions across different times of day, days of the week, or months of the year—and how travel times are impacted by big events, road closures or other things happening in a city.

As the Washington Post reports, transport agencies do already have a lot of data on some aspects of commuter behaviour – particularly public transport usage – and the Uber information fills as ‘missing part of the puzzle’.

Taxis and buses are also increasing equipped with real time tracking equipment that also gives this data while traffic services like Wayze have been collecting this information for a decade.

So agencies aren’t short of this data and the concentration of Uber’s customer base in more affluent areas means their information may be skewed away from poorer areas. Recently a Sydney taxi driver mentioned to me how he’d stopped driving for Uber because most of the city’s sprawling Western Suburbs where he tended to drive didn’t use the service.

Uber’s offer is another piece in their data strategy that sees the company being a data hub for the logistics industry. It also helps if you’ve co-opted governments into your scheme.

Computing on the edge

Amazon announced another range of services and products at their annual Las Vegas conference, but are they becoming too powerful?

As with every vendor conference, this year’s AWS Re:Invent convention in Las Vegas bombarded the audience with new product announcements and releases.

One of the interesting aspects for the Internet of Things was the announcement of Amazon Greengrass, a service that stores machine data on remote equipment which combines the company’s Lambda serverless computing and IoT services.

Further pushing Amazon’s move into the IoT space was CEO Andy Jassy’s announcement that chip makers such as Qualcomm and Intel will be building Lambda functions into their chipsets, further embedding AWS into the ecosystem.

Jassy also touted the company’s new Snowball Edge, a slimmed down version of their Snowball data transfer unit that also include some processing features, that is aimed at storing machine data at remote or moving locations such as ships, aircraft, farms or oil rigs.

That latter function ties into one of the key aspects about the Internet of Things – that most data doesn’t have to, or can’t, be transmitted over the internet. This is something companies like Cisco have focused on in their edge computing strategies.

With AWS dominating the cloud computing industry – Gartner estimates the company is ten times bigger than the next 14 companies combined – the worry for customers and regulators will be how much control the organisation has of the world’s data.

It’s hard though not to be impressed at the range of products the company has, and the speed they get them to market, the onus is on companies like Microsoft, Google and Facebook to allocate the resources and talent to match AWS in the marketplace.

Connecting 400 points of voter data

US political parties are showing how organisations can use data in a targeted, sophisticated way

As the 2016 US Presidential race enters its final stages, it’s interesting to see how data is being used by American political candidates and what this means for business.

During last week’s Oracle Open World in San Francisco a panel hosted by the company’s Political Action Committee featured Stephanie Cutter, who worked on Obama’s 2008 and 2012 campaigns, and Mike Murphy, a Republican operative whose most recently worked on Jeb Bush’s primary effort against Donald Trump.

While the discussion mainly focused on the politics – “Crazy times seem to require crazy candidates” says Murphy – it was the technology aspect of modern elections that was notable.

Setting the data standard

The Obama campaign of 2008 set the standard for how modern political campaigns used social media and information, “we revolutionized how data analytics helps predict how people will vote and how they will persuade voters to turn out.” Cutter said.

“We put a big investment into it and Republicans have caught up,” she continued. “The key though was we relied on our own data and nothing that was out in the public domain. We didn’t rely on one piece of data, we had multiple sources. We had an analytics program where we were making 9,000 calls a night where we were predicting the votes.”

Murphy agreed with the political campaigns using data, “the kind of polling you see in the media has kind of vanished in campaigns where they have money to spend on research.” He said, “we don’t do telephone polling any more because we have so much data we can collect.”

Capturing everything

“We capture everything. We have about four hundred data points on the American voter and we’ll have five hundred in the next two years. We’ll be able to build massive data models without phone polling,” Murphy pointed out. “We’re waiting for the tech folk to get ahead on AI so we can predict what voters are going to do in two weeks.”

Despite the amount data collected by US political parties, the real key to success is the candidate’s organisation and management. Cutter made a strong point about the strength of Obama’s campaign team in both the 2008 and 2012 campaigns.

How the US political parties use data points to how businesses will be managing data in the future. Increasingly using information well is going to be the measure of successful organisations in both politics and industry.

Data and the modern movie producer

WETA Digital shows the way on how other businesses will have to manage data in coming years.

Dealing with the massive wave of data flowing into businesses will be one of the defining management issues of the next decade. One company that is already dealing with this is New Zealand’s Weta Digital.

Wellington based Weta that’s best known for its work on Lord of the Rings and is part owned by director Peter Jackson employs 1400 staff for its movie special effects work and has won five visual effects Academy Awards over its 23 years of operations.

Kathy Gruzas, WETA Digital’s CIO, spoke to Decoding the new Economy at the Oracle OpenWorld forum in San Francisco this week about some of the challenges in dealing with the massive amount of data generated by the movie effects industry.

“We have some very heavy loads.” Kathy states. “We push our systems to the limit.”

Applying powerful systems

One challenge is the sheer computing power required, ‘the render frame processes one frame per server until you have four seconds of footage. Sometimes that takes over night or even longer and for that we use a lot of storage,” Kathy says. “The render farm being six thousand servers will write 60 to 100 terabytes of data a day and read a quarter to half a petabyte each day.”

“We need systems that will be very large to handle the volume of data we generate but also be very quick to handle those read and writes.”

“One render could use a thousand computers, sometimes more, and all of those will be reading and writing against the same block of storage so we have our own software layer that directs those loads but we try to minimise the load on our storage but we have the worst work load you can imagine with lots of servers, lots of small reads and writes and many of them random and concurrent with pockets of hot files.”

Despite the automation, the business is still extremely capital intensive. “In visual effects you probably need at least three hundred artists to work on one film, it’s a very labour intensive process to do the artistry and much like a production line.”

Going mobile

The nature of modern movie production means the effects teams are now part of the shoot which adds another level of complexity for Weta. “Although we are visual effects which is largely post-production we do go out with crews when they’re shooting the movie so we can do reference photography,” says Kathy.

“We do 3D scans so if we need to do something digitally and we do motion and facial capture as well,” she says. “There are 240 muscles that we tweak individually to get the expression. That’s a huge amount of data to capture.”

To do this, Weta created their own ‘road case’ that contains everything they need to grab the shots and store the data they need, “you can’t ask the director retake the shot because we missed something.”

Into the forest

“We have to take the case into the forest and into the rain and everywhere. It’s good having that roadcase that has storage, networking and servers in it.” The case, which was self assembled by Weta’s team is “probably the most travelled Oracle system on the planet,” laughs Kathy with “lots of data capture and sub-rendering.”

Weta’s story illustrates just how managing data is becoming a critical issue for companies. While movie special effects is very much a specialised field that’s far ahead of the curve in its technology use than most businesses, they do show the importance of managing and securing their data.

For other businesses, lessons from Weta is understanding your company’s – including staff and customers’ – needs then investing in the right tools to deliver is essential.

One important difference between technology intensive businesses like Weta and most other organisations is the New Zealand company is doing most of its processing and storage in house. Those without the same needs will almost certainly be shifting these tasks onto the cloud.

Thinking about networked thinking

In a world awash with data managers may have to start thinking about networked thinking

“We want to be the Wayze of enterprise software” is the line being repeated by executives at the Inforum2016 conference in New York today.

This is an interesting strategy for Infor, who provides a range of enterprise software tools to help companies track what is going on in their business, as Wayze is built upon aggregating user data to identify traffic problems to improve commuting times. It’s no surprise that Google bought the company a few years ago.

Infor position though is slightly different as it’s aggregating individual clients’ data for them. In a world where organisations are struggling not to be overwhelmed by information, Informa are in a good position, even if their executives do overdo it on the buzzwords.

Which leads us to another buzzphrase – design thinking – which has been drifting in and out of fashion over recent years. During the opening keynotes one of the comments was about the rise of  “network thinking.”

“Eighty percent of what most companies do deals with data from outside of their organisation,” says Kurt Cavano, Infor’s General Manager of their commerce cloud division. “We’ve seen in the power of networks with sites like Facebook, LinkedIn and Wayze.”

“Nobody wants to be on a network but everyone’s on a network. It takes a long time to build but once you have one it’s magical. That’s what we’re thinking for business, they need to evolve.”

In one respect this is another take on the ecosystem idea, that one vital corporate asset in the connected world is an ecosystem of partners, suppliers and users, however the Infor view articulated by Cavano is much more about the flow of data rather than the goodwill of a community.

So we may well be entering a world of ‘networked thinking’ where thinking about the effects of data flows and being able to understand them – if not manage them – becomes a key executive skill.

Paul travelled to New York as a guest of Infor

Evolving into a data centric company

The newly demerged HP Enterprise is dealing with a shifting market and a change in product focus.

I’m currently at the HP Enterprise Seize the Data roadshow in Singapore where the recently split company is showing off its range of data analytics tools.

Like companies such as IBM and Google, HPE are looking to make money out of data feeds and analytics with a key part being a platform for developers to create applications.

In launching their Haven OnDemand service, HPE are entering a crowded field with IBM, Salesforce, AWS and Splunk – among others – offering similar products. What compelling difference HPE will add to the field will be something I’ll be asking the company’s executive later.

One of the other services, HP Vertica, looks running data analytics against structured and ‘semi-structured’ sources. Again this is a field where other companies are well established and have an advantage in being able to examine unstructured data.

The overwhelming question though is how big, and lucrative, the market is for these data products. It’s not clear exactly how all of these companies are going to monetize these services and, should they be able to, their profitability.

As a company finding its feet less than a year after being split in two with the added problem of seeing its core server hardware business being eroded, HP Enterprise is realigning its business around data analytics and cloud services.

The challenge for the company is differentiating itself and providing competitive products in these markets, this will be a tough challenge.

Guessing ethnic affinity

Big data can create big risks, particularly when a service like Facebook starts racially profiling

What’s your ethnic affinity? Apparently Facebook thinks its algorithm can guess your race based upon the nature of your posts.

This application is an interesting, and dangerous, development although it shouldn’t be expected that it’s any more accurate than the plethora of ‘guess your age/nationality/star sign’ sites that trawl through Facebook pages.

Guessing your race is something clumsy and obvious but its clear that services like Google, LinkedIn and Facebook have a mass of data on each of their millions of users that enables them to crunch some big numbers and come up with all manner of conclusions.

Some of these will be useful to governments, marketers and businesses and in some cases it may lead to unforeseen consequences.

The truth may lie in the data but if we don’t understand the questions we’re asking, we risk creating a whole new range of problems.

Actuaries and the future of Public Relations

Will actuaries become the most valued profession in the PR industry? Some think so.

One of the truisms of modern industry is we’re going to need more workers with data skills. Could it be actuaries will be the profession of the information age.

Much of the focus around how companies will deal with an information rich age come down to the need for ‘data scientists’, those with a combination of statistical, analytical and coding skills will be required to coax insights out of complex and rapidly changing data sets.

At a Future of PR meetup in Sydney earlier this week, one of the panellists raised the possibility that tomorrow’s most valued agency employees will be actuaries as data analytics comes to dominate the industry.

That boring old actuaries – one particularly cruel joke is atuaries are accountants who failed the personality test – could be the hottest profession in the sexy PR industry is quite a delicious scenario.

Should that turn out to be the case though, it won’t just be the PR industry chasing actuaries, almost every industry is going to demanding the same set of skills.

In a strange way it could be the staid professions of today that are the exciting jobs of tomorrow, we’ll reserve judgement on the actuaries though.