big data | Paul Wallbank - Communications Professional

Smart cities and the sensors in your pocket

Community wide sensors promise to change government

National Public Radio’s Parallels program has story on how the Spanish city of Santander is wiring itself as a ‘smart city’ with a network of sensors wiring everything from garbage bins to parking spots.

The hope with the sensors is they’ll will improve local government’s services, allowing things like more efficient garbage collection and better pricing of parking meters.

What’s notable about the story is that smartphones are included as ‘sensors’ with Santander residents being able to submit data from their handsets.

The idea of smartphones as sensors isn’t new — pothole reporting apps were early to the iPhone — the increased sophistication of handsets and improved tracking technology is making them more powerful.

So we have another Big Data problem with local councils being flooded with information.

Processing all this information is going to require the community pitching in so the data is going to have to open.

Once governments make the data open it also creates opportunities for smart entrepreneurs to create new services and technologies.

Creating new opportunities is a hope of government sensor programs around the world, including Tasmania’s Sense-T project .

With factors like water quality and weather being monitored, existing sectors become more efficient and new industries are being created.

Hopefully the urge to hoard this rich, community data will be resisted by governments.

Big data’s big truths

There’s a lot of hype around Big Data but it doesn’t mean we should ignore the risks or opportunities.

One thing former Obama 2012 campaign CTO Harper Reed cannot be accused of is subtlety so his statement at the Sydney CeBIT conference last week that Big Data is Bullshit wasn’t wholly surprising.

Reed has a good point – like all IT industry buzzwords there is a fair degree of hype and BS around Big Data although his referring to it as a storage problem misses the point.

Data storage is a problem largely solved; when we’re talking about Big Data today, we’re talking more about analysing the information and managing the life cycle of an organisation’s data.

Not that these issues are new, the tech industry has been dealing with the challenges of storing, managing and analysing data since computers first appeared. In fact, that’s the reason computers were invented.

An excellent NY Times Bits blog post expands on Harper’s views and rebuts many of the myths and hype around big data.

Most important is the point that big data is not the truth, we can torture those bits and bytes to tell us anything we like.

Claims that Big Data can tell us everything or that it will conquer discrimination and make cities smarter are fanciful. It all depends on how we choose to use the data.

There are downsides with Big Data too — we live in an age where it’s easier to let the algorithm do the work and if the computer says ‘no’, then we can shrug and say “sorry it’s beyond our control.”

Letting the algorithms run our lives is one of many risks, but it doesn’t change the opportunities for businesses, governments and communities Big Data presents. If we can understand our world better, we can do smarter things.

That’s the real opportunity with Big Data and we don’t need the hype to tell us that.

ABC 702 mornings – Storage and your computer

How we deal with the information explosion in the age of Big Data is the topic of today’s 702 Sydney segment with Linda Mottram

This morning on 702 Sydney I’m talking to Linda Mottram on the decidedly unsexy topic of storage – hard drives, cloud computing and the struggle to keep up with ever expanding file sizes of documents, photos and downloads.

It’s an opportunity to revisit the How Much Data Does The Internet Need topic which I covered for Radio National last year, although almost certainly that needs updating.

Earlier this year networking vendor Cisco released its 2013 Virtual Networking Index which forecast global data traffic growing fourteen fold over the next five years.

Those bytes slopping around the internet have to come to rest on someone’s hard drive and this is what’s driving the storage crisis.

Yesterday US business site Venture Beat had an op-ed by an executive from Seagate, the world’s biggest hard drive manufacturer where he discussed the storage challenges with a claim from industry consultants IDC that worldwide computer storage is 2.7 zettabytes.

A zettabyte is a trillion gigabytes, or ten followed by twenty zeros – it’s the equivalent of a billion one terabyte hard drives that are standard on most cheap desktop computers.

Where those hard drives are located is the big challenge, is it on your laptop, smartphone or on a somewhere on a cloud service?

The other big challenge is what do you do with all this information – which is where the Big Data discussion comes in.

While data storage is a mundane topic, it’s a big one that matters. I hope you can tune in.

We’d love to hear your views so join the conversation with your on-air questions, ideas or comments; phone in on 1300 222 702 or post a question on ABC702 Sydney’s Facebook page.

If you’re a social media users, you can also follow the show through twitter to @paulwallbank and @702Sydney.

Skills, data scientists and the decade’s big IT trends

As the amount data flooding into our lives explodes, we’ll all need to think about how we can get the skills to manage and understand data.

As we all get buried under a tsunami of data, the challenge is managing it. The MIT Technology Review this week looks at the rise of the data scientist, a job title unknown a few years ago.

The problem for industry is the skill sets required to become a data scientist are fairly esoteric.

Data scientist has become a popular job title partly because it has helped pull together a growing number of haphazardly defined and overlapping job roles, says Jake Klamka, who runs a six-week fellowship to place PhDs from fields like math, astrophysics, and even neuroscience in such jobs. “We have anyone who works with a lot of data in their research,” Klamka says. “They need to know how to program, but they also have to have strong communications skills and curiosity.”

Over the last twenty years we’ve done a pretty poor job teaching maths and statistics which is going to create a skills shortage as industry struggles to find people qualified to figure out what all of this data means.

While Big Data might be to this decade what plastics were to the 1960s, it’s not the only technology change that’s affecting business as the McKinsey Quarterly describes the ten IT trends for the decade ahead.

The thing that really stands out with McKinsey’s predictions is the degree of reskilling the workforce is going to need, today’s workers are going to need an understanding of programming, logic and statistics as much the kids currently at school.

If you’re planning on being in the workforce at the end of this decade right now may be the time to consider getting some of these skills.

Just as businesses will be separated by how they use Big Data, workers may too find those skills divide the winners from the losers.

As the amount of data flooding into our lives explodes, we’ll all need to think about how we can get the skills to manage and understand data.

Can maps change the way we work?

Big data and mobile computing are changing the way business operates as maps become an important part of our normal work and leisure time.

“Work the Way You Live” is Google’s motto for their enterprise maps service which the search engine giant hopes to make as ubiquitous in business as it is in the home.

At Google Atmosphere the company showed off their mapping technology and how it can be used by large organisation. It’s a compelling story.

The technology behind Google Maps is impressive – twenty petabytes of images, one billion active monthly users, 1.6 million map tiles served every second and a target of getting those tiles onto the users screen within ten milliseconds.

Maps are one of the Big Data applications that cheap computing makes possible, until a few years ago even desktop computers would have struggled with the sort of mapping technology that we take for granted on our smartphones today.

Rethinking products

Adding mapping technologies to products allows businesses to rethink their products. A good example of this is the internet connected treadmill.

Using the treadmill a jogger, or a walker, can map out a route anywhere in the world and the screen will show them the Google Street View as they travel along the route. The treadmill even adjusts to the changing gradients.

The Google Maps driven treadmill is a trivial example of the internet of machines, but it gives a hint of what’s possible.

The search for truth

The success of a map depends on whether it can be trusted – this is what caught Apple out with their mapping application which was released before it was ready for prime time. Google, and most cartographers, take seriously errors and changes.

In the early days of Google Maps, the company would pass errors and changes onto the private and government mapping providers they licensed the data from. It could take months to fix a problem.

“It was really hard, you have to get maps from all over the world to create the product,” says Louis Perrochon, the Engineering director of google maps for business.

“That’s a limitation if you work with third party data so we started a project called Ground Truth where we build our own maps.”

Google pulls together its Street View data, satellite images and information sent in from the public through their Map Maker site and the Maps Engine Lite to build an accurate map of an area.

Changing consumer behaviour

Having accurate and accessible maps has changed the way consumers have behaved; “this revolution hasn’t happened slowly,” says Google Enterprise Directore Richard Suhr, “it’s happened really quickly.”

“Customers have become savvy about spatial. What this means is that businesses are starting to rethink the problem.”

“What are the exciting things I can do with maps, what else can I do with my data.”

That’s a big question of all businesses – how they use the massive amount of information in their organisation will mark the winners from also runs over the next decade. Maps are one way to visualise their data.

While Google Atmosphere was a marketing event for the companies mapping technologies the message is clear – mapping is changing the way we work and play and it’s affecting business.

How is mapping changing the way your business works?

Fifty trillion shades of grey

Something that’s missed when we talk about Big Data is the risk of false positives – if you dip into the stream, you can prove anything against person.

If you give me six lines written by the hand of the most honest of men, I will find something in them which will hang him said the 17th Century French politician Cardinal Richelieu.

Today those six lines could be written on a social media site or be six disparate points drawn from a database. Without context those six lines could condemn us.

Something that’s missed when we talk about Big Data is the risk of false positives – if you dip into the stream, you can prove anything against person.

The world isn’t black or white, there are fifty trillion shades of gray and that’s why it’s important to think before posting an image on the web, firing someone or calling the cops.

In an era where we’re quick to judge and condemn people, the stakes are very high.

Recruiting big data

Software company Evolv is an example of how businesses can use big data

One of the predictions for 2020 is that decade’s business successes will be those who use big data well.

A good example of a big data tool is recruitment software Evolv that helps businesses predict not only the best person to hire but also who is likely to leave the organisation.

For employee retention, Evolv looks at a range of variables which can include anything from gas prices and social media usage to local unemployment rates then pulls these together to predict which staff are most likely to leave.

“It’s hard to understand why it’s radically predictive, but it’s radically predictive,” Venture Beat quotes Jim Meyerle, Evolv’s cofounder.

There are some downsides in such software though – as some of the comments to the VentureBeat story point out – a blind faith in an alogrithm can destroy company morale and much more.

Recruiters as an industry haven’t a good track record in using data well, while they’ve had candidate databases for two decades and stories abound of poor use of keyword searches carried out by lazy or incompetent headhunters. The same is now happening with agencies trawling LinkedIn for candidates.

Using these tools and data correctly going to separate successful recruitment agencies and HR departments from the also-rans.

It’s the same in most businesses – the tools are available and knowing them how to use them properly will be a key skill for this decade.

Job classifieds image courtesy of Markinpool through SXC.HU

Leadership in a connected world

How do managers and executives lead in a connected world?

Managing a business or government agency as information pours into organisations is one of the great challenges for modern executives.

As part of the Australian Cisco Live event, a panel looked at Public Sector Leadership in a Connected World, many of the issues discussed apply to private sector executives as they do to public sector managers.

Cisco’s Director of Global Public Sector Practice, Martin Stewart-Weeks, kicked off the panel with the observation that “we now live in a world where information has become completely unmanageable.”

Martin quoted from David Weinberg’s book Too Big To Know, Rethinking Knowledge Now That the Facts Aren’t the Facts, Experts Are Everywhere, and the Smartest Person in the Room Is the Room. The author has a good explanation of his book in this YouTube clip.

Trusting the community seems to be the biggest problem facing politicians and the public service, policy consultant Rod Glover puts the general distrust towards governments on the failure of leaders to consult over changes and decision.

Economist Nick Gruen and Australian Industry Group adviser Kate Pound echoed this problem in that a change of culture is needed among leaders towards the way information is controlled and managed.

Nick sees that culture changing while Rod thinks there will need to be demonstrated successes before risk adverse public service leaders will be prepared to adopt new ways of managing.

Kate’s view is that culture change will require a realignment of incentives which will make managers accountable for the delivery of services. She cites a situation where businesses are obligated to register online but the agency’s website doesn’t work.

So the problem is as much gathering the right data along with processing the information inside an agency. Both are challenges for organisations with rigid hierarchies and information flows.

Information is no longer power — it’s how you use it. But the structures are still based around access and control of knowledge.

The big culture shift for politicians, public servants and corporate executives is we can no longer hoard information.

For managers in both the public and private sectors, the task is now to share information and trust the right people will use it well.

Paul travelled to Cisco Live courtesy of Cisco Systems

Dealing with the data explosion

Supply the mobile base stations for data hungry customers is one of the great challenges for telcos. How they resolve this will create some unusual alliances.

“Last year’s mobile data traffic was nearly twelve times the size of the entire global Internet in 2000.”

That little factoid from Cisco’s 2013 Virtual Networking Index illustrates how the business world is evolving as various wireless, fibre and satellite communications technologies are delivering faster access to businesses and households.

Mobile data growth isn’t slowing; Cisco estimate global mobile data traffic was estimated at 885 petabytes a month and Cisco estimate it will grow fourteen fold over the next five years.

Speaking at the Australian Cisco Live Conference, Dr. Robert Pepper, Cisco Vice President of Global Technology Policy and Kevin Bloch, Chief Techincal Officer of Cisco Australia and New Zealand, walked the local media through some of the Asia-Pacific results of Virtual Networking Index.

Dealing with these sort of data loads is going to challenge Telcos who were hit badly by the introduction of the smartphone and the demands it put on their cellphone networks.

A way to deal with the data load are heterogeneous networks, or HetNets, where phones automatically switch from the telcos’ cellphone systems to local wireless networks without the caller noticing.

The challenge with that is what’s in it for the private property owners whose networks the telcos will need to access for the HetNets to work.

One of the solutions in Dr Pepper’s opinion is to give business owners access to the rich data the telcos will be gathering on the customers using the HetNets.

This Big Data idea ties into PayPal’s view of future commerce and shows just how powerful pulling together disparate strands of information is going to be for businesses in the near future.

But many landlords and wireless network owners are going to want more than just access to the some of the telco data — we can also be sure that the phone companies are going to be careful about what customer data they share with their partners.

It may well be that we’ll see telcos providing free high capacity fibre connections and wireless networks into shopping malls, football stadiums, hotels and other high traffic locations so they can capture high value smartphone users.

One thing is for sure and that’s fibre connections are necessary to carry the data load.

Anyone who thinks the future of broadband lies in wireless networks has to understand that the connections to the base stations doesn’t magically happen — high speed fibre is essential to carry the signals.

Getting both the fibre and the wireless base stations is going to be one of the challenges for telcos and their data hungry customers over the next decade.

Paul travelled to the Cisco Live event in Melbourne courtesy of Cisco Systems.

Using big data to find the cupboard is bare

Yahoo! Chief Executive Marissa Mayer is an example of how modern managers are diving into big data to figure out what is going on in their company

Last week this blog discussed whether telecommuting was dead in light of Marissa Mayer’s banning of the practice at Yahoo.

While I don’t think telecommuting is dead, Marissa Mayer has a big problem figuring out exactly who is doing what at the company and abolishing remote working is one short term way of addressing the issue.

If Business Insider is to be believed, Yahoo!’s absent staff problem is bad.

After spending months frustrated at how empty Yahoo parking lots were, Mayer consulted Yahoo’s VPN logs to see if remote employees were checking in enough.

Mayer discovered they were not — and her decision was made.

Business Insider’s contention is that Mayer makes her decisions based on data analysis. At Google she drove designers mad by insisting on reviewing user reactions to different layouts and deciding based on the most popular results.

If this is true, then Marissa Mayer is the prototype of tomorrow’s top executives – the leaders in business by the end of this decade will be the ones who manage data well and can sift what matters out of the information deluge.

For all of us this is going to be a challenge with the probably the biggest task of all being able to identify which signals are worth paying attention to and which should be ignored.

Of course, all this assumes the data is good quality in the first place.

An assumption we’ve all made when talking about Big Data is that it’s about marketing – we made the same assumption about social media.

While Big Data is a good marketing tool, it’s just as useful in areas like manufacturing, logistics, credit evaluations and human resources. The latter is what Yahoo!’s staff are finding out.

In age of Big Data it may not pay to a slacker, but it’s going to be handy if you want to know what’s going on your business.

Smelling digital garbage

Excel spreadsheets lie at the core of business computing, but what happens when they go wrong?

James Kwak writing in the Baseline Scenario blog describes how Excel spreadsheets have an important role in the banking industry and their key role in one of the industry’s most embarrassing recent scandals.

In the early days of the personal computer spreadsheets; it was company accountants and bookkeeping clerks who bought the early PCs into offices to help them do their jobs in the late 1980s .

From the accounts department, desktop computers spread through the businesses world and the PC industry took off.

Over time, Microsoft Excel displaced competitors like Excel 1-2-3 and the earliest spreadsheet of all, VisiCalc, and became the industry standard.

With the widespread adoption of Excel and millions of people creating spreadsheets to help do their jobs came a new set of unique business risks.

The weakness with Excel isn’t with the program itself, it’s that the formulas in many spreadsheets aren’t properly tested and often incorrect data is put into the wrong fields.

In his story Kwak cites the JP Morgan spreadsheets that miscalculated the firms Value-At-Risk (VAR) calculations for synthetic derivatives. The result was the London Whale debacle where traders were allowed to take positions – some would call them bets – exposing the bank to huge potential losses.

It turns out that faulty spreadsheets had a key role as traders cut and paste data between various spreadsheets and the formulas that made the calculations had basic errors.

That a bank would have such slapdash procedures is surprising but not shocking, almost every organisation has a similar setup and it gets worse as a project becomes more complex and bigger numbers become involved. The construction industry is particularly bad for this.

Often, a spreadsheet will show out a bunch of numbers which simply aren’t correct. Someone made a mistake entering some data or one of the formulas has an error.

The business risk lies in not picking up those errors, JP Morgan fell for this and probably every business has, thankfully to less disastrous results.

My own personal experience was with a major construction project in Thailand. One sheet of calculations had been missed and the entire budget for lights – not a trivial amount in a 35 storey five star hotel – hadn’t been included in the contractor’s price.

This confirmed in my mind that most competitive construction tenders are won by the contractor who made the most costly errors in calculating their price. Little has convinced me otherwise since.

In the computer industry there’s a saying that “garbage in equals garbage out” which is true. However if the computer program itself is flawed, then good data becomes garbage.

Excel’s real flaw is that it can make impressive looking garbage that appears credible if it isn’t checked and treated with suspicion. The responsibility lies with us to notice the smell when the computer spits out bad figures.

Spreadsheet image courtesy of mmagallan through sxc.hu

Democratising Big Data

Why a not for profit disrupting Google and the Big Data industry is important for business and society

Common Crawl is a not-for-profit web crawler service that makes the data collected open for all to use. A post on the MIT Technology Review blog speculates how the initiative might spawn the next Google.

One of the problems with Big Data is that it’s held mainly by large corporations and government agencies, both of which have the tendency to keep their data private on that basis that information is power and power means money.

We see this in the business models of Facebook, Google and many of Silicon Valley’s startups; the information garnered about users is as, if not more so, valuable as an utility from the product.

Initiatives like Common Crawl tilt the balance somewhat back towards consumers, citizens, and smaller businesses.

How well Common Crawl and other similar initiatives fare remains to be seen – Wikileaks was a good example of how such projects can flare out, collapse under the weight of egos or be harrassed by corporatist interests.

In search, Google are open to disruption as they tweak their results to suit initiatives like Google Plus. During the company’s earnings call earlier this week Larry Page spoke of the challenges of staying focused on the opportunities that matter, it may well be the company is more distracted from its core business than it should be.

Whether Common Crawl disrupts Google is up to history, it could just as well be a couple of kids called Sergei and Larry with a smart idea.

The imperative now though is to try and keep as much public data available for everyone to use and not lock it away for the privileged few. That will let the future Googles develop while making our societies more fairer and open.