Big Data | Paul Wallbank - Communications Professional

Video and the internet of things

High resolution video coupled with the IoT are part of the Big Data explosion

A few days ago we discussed how 4k video cameras are going to change the sports broadcasting industry.

Yesterday executives from modular data center supplier VCE held a media lunch where they discussed some of their industrial applications. One of the areas they discussed was the monitoring of power stations with large resolution cameras.

The 4k cameras are trained on machine rooms with software watching for irregular conditions such as excessive vibrations, leaks or smoke. Should something out of the ordinary be detected, warnings can be triggered and potentially affected equipment spun down.

With the 4k resolution the cameras are able to watch large areas and like the sports coverage can zoom in for a detailed view of an affected area.

The use of 4k video cameras shows how the internet of things won’t just be about the data gathered from smart devices but also matching the information coming from IoT equipment with that of other environmental factors.

For companies like VCE these sort of applications are an opportunity as they need large amounts of data storage and processing power in local centres.

In many respects these small scale data centers are a large scale example of the fog computing being touted by companies like Cisco where most of the operational tasks are carried out by local equipment with only reports and exceptions being transmitted to the cloud.

This sort of application also shows the demands different industries are going to have for local data processing and storage with the VCE executives suggesting hospitals, mines and sports stadiums are also going to need these facilities.

For VCE – a troubled joint venture between Cisco, storage company EMC and computer virtualisation firm VM Ware – these are the sort of clients they are hoping to find to keep their business running.

Regardless of VCE’s prospects, the need for equipment to manage the data being collected by devices on the Internet of Things and 4k video is going to grow. That could give us one of the clues of where the jobs of the future are going to come from.

Data driven lending

Square enters the small business lending space, will be they successful in a very competitive field?

Banking has always been a data driven business, understanding borrowers and the risks they present is one of the essential skills in making money from lending.

The new wave of payment startups present a new way for lenders to analyse risks; with real time data aggregated across businesses and regions, lenders can quickly decide wether a borrower is likely to able to pay the money back with the conditions asked for.

Payments company Square in its latest pivot has partnered with Victory Park Capital and claims to have extended more than $100 million in capital to more than 20,000 merchants writes the New York Times.

Like other payment companies that have entered this market, Square uses their own deep understanding of their customers’ incomes to be able to make a data based decision on the creditworthiness of applicants.

Square also offers ancillary data-driven products created for small businesses. The new instant deposit product, which is still in testing and will be fully available in the spring, will give businesses faster access to money they put into a debit account. And the company’s new charge-back protection service will cover some disputes between consumers and merchants.

Those products also rely on data that Square has collected. They will be available only to small businesses that have a solid financial track record, based on a history of accepting payments with Square.

Square is by no means the first business to do this, last year we wrote of PayPal’s move into small business lending and Point of Sale hardware manufacturer Verifone retreated from the market two years ago calling it ‘fundamentally unprofitable.’

The competition in the space and the fact assessing financial risks isn’t exactly a core competence of Silicon Valley start ups indicate Square’s and other companies may find small business lending a tough business as well.

Despite that, small business lending is a field that is overdue for disruption. With companies like Apple, Google and Amazon all offering payment services, the logical expansion is into evaluating risk and profit.

It may not be Square, Verifone or PayPal who ultimately redefines the sector, but it will be one of today’s tech businesses that does.

Clawing back our data – Telstra makes metadata available to customers

Australia’s Telstra responds to government data legislation by opening metadata to users

Today Australian incumbent telco announced a scheme to give customers access to their personal metadata being stored by the company.

In a post on the company’s Telstra Exchange blog the company’s Chief Risk Officer, Kate Hughes described how the service will work with a standard enquiry being free through the web portal with more complex queries attracting of fee of $25 or more.

The program is a response to the Australian Parliament’s controversial intention to introduce a mandatory data retention regime which will force telcos and ISPs to retain a record of customer’s connection information.

We believe that if the police can ask for information relating to you, you should be able to as well.

At present the scheme is quite labor intensive, a request for information involves a great deal of manual processing under the company’s current systems however Hughes is optimistic they will be able to deal with the workload.

“We haven’t yet built the system that will enable us to quickly get that data,” Hughes told this website in an interview after the announcement. “If you came to us today and asked for that dataset it wouldn’t be a simple request.”

The metadata opportunity

In some respects the metadata proposal is an opportunity for the company to comply with the requirement of the Australian Privacy Principles that were introduced last year where companies are obliged to disclose to their customers any personally identifiable information they hold.

For large organisations like Telstra this presents a problem as it’s difficult to know exactly what information every arm of the business has been collecting. Putting the data into a centralised web portal makes it easier to manage the requirements of various acts.

That Telstra is struggling with this task illustrates the problems the data retention proposals present to smaller companies with far fewer resources to gather, store and manage the information.

Unclear requirements

Another problem facing Hughes, Telstra and the entire Australian communications industry is no-one is quite clear exactly what data will be required under the act, the legislation proposed the minister can declare what information should be retained while the industry believes this should be hard coded into the act which will make it harder for governments to expand their powers.

What is clear is that regardless of what’s passed into law, technology is going to stay ahead of the legislators, “I do think though this will be very much a ‘point in time’ debate,” Hughes said. “Metadata will evolve more quickly than this legislation can probably keep pace with so I think we will find ourselves back here in two years.”

In many ways Australia’s metadata proposals illustrates the problems facing governments and businesses in managing data during an era where its growing exponentially, it may well turn out for telcos, consumers and government agencies that ultimately less is more.

Reducing big data risks by collecting less

Just because you can collect data doesn’t mean you should

“To my knowledge we have had no data breaches,” stated Tim Morris at the Tech Leaders conference in the Blue Mountains west of Sydney on Sunday.

Morris, the Australian Federal Police force’s Assistant Commissioner for High Tech Crime Operations, was explaining the controversial data retention bill currently before the nation’s Parliament which will require telecommunications companies to keep customers’ connection details – considered to be ‘metadata’ – for two years.

The bill is fiercely opposed by Australia’s tech community, including this writer, as it’s an expensive and unnecessary invasion of privacy that will do little to protect the community but expose ordinary citizens to a wide range of risks.

One of those risks is that of the data stores being hacked, a threat that Morris downplayed with some qualifications.

As we’re seeing in the Snowden revelations, there are few organisations that are secure against determined criminals and the Australian Federal Police are no exception.

For all organisations, not just government agencies, the question about data should be ‘do we need this?’

In a time of ‘Big Data’ where it’s possible to collect and store massive amounts of information, it’s tempting to become a data hoarder which exposes managers to various risks, not the least that of it being stolen my hackers. It may well be that reducing those risks simply means collecting less data.

Certainly in Australia, the data retention act will only create more headaches and risks while doing little to help public safety agencies to do their job. Just because you can collect data doesn’t mean you should.

Reading the golden records – can we avoid a digital dark age?

Changing computer formats mean we risk a ‘digital dark ages’ industry experts warn.

In 1977 NASA’s Voyager mission launched from Cape Canaveral to explore the outer solar system, included on the vessel in case it encountered other civilisations were a plaque and a golden record describing life on Earth.

The record was, is, “a 12-inch gold-plated copper disk containing sounds and images selected to portray the diversity of life and culture on Earth.” It containing images, a variety of natural sounds, musical selections from different cultures and spoken greetings in fifty-five languages.

Most American households in 1977 could have listened to the sounds on Voyager’s golden disk but were the spaceship to return today it would be difficult to find the technology to read the record.

This is the concern of Google Fellow and internet pioneer Vint Cerf who told the American Association for the Advancement of Science’s annual meeting in San Jose this week we are “facing a forgotten century” as today’s technologies are superseded rendering documents unreadable.

A good example of ‘bit rot’ is the floppy disk – the icon used by most programs to illustrate saving files is long redundant and few organisations, let alone households, have the ability to read a floppy disk.

For corporations the problem of dealing with data stored on tape is an even greater problem as proprietary hardware and software from long vanished corporations becomes harder to find or engineer.

As the Internet of Things rolls out and data becomes more critical to business operations, the need for compatible and readable formats will become even more important for companies and historical information may well become a valuable asset.

With libraries, museums and government archives having digitised historic information, this issue of accessing data in superseded formats becomes even more pressing.

It may be that important documents need to be kept on paper – although there’s still the problem of paper deteriorating – to make sure the 21st Century doesn’t become the digital dark ages and our golden records remain unread.

At the mercy of machines

Automation and algorithms are changing business but they are not without risks

Automation is the greatest change we’re going to see in business over the next decade as companies increasingly rely upon computers to make day to day decisions.

Giving control to algorithms however comes with a set of risks which managers and business owners have to prepare for.

Earlier this week the risks in relying on algorithms were shown when car service Uber’s management was slow to react to a situation where its formulas risked a PR disaster.

Uber’s misstep in Sydney shows the weaknesses in the automated business model as its algorithm detected people clamouring for rides out of the city and applied ‘surge pricing’.

Surge pricing is applied when Uber’s system sees high demand – typically around events like New Year’s Eve – although the company has previously been criticised for alleged profiteering during emergencies like Hurricane Sandy in New York.

In the light of previous criticism, it’s surprising that Uber stumbled in Sydney during the hostage crisis. Shortly after criticism of the surge pricing arose on the internet, the company’s Sydney social media manager sent out a standard defence of surge pricing.

We are all concerned with events in CBD. Fares have increased to encourage more drivers to come online & pick up passengers in the area.

— Uber Sydney (@Uber_Sydney) December 15, 2014

That message was consistent with both Uber’s business model and how the algorithm that determines the company’s fares works; however it was a potential disaster for the business’ already battered reputation.

An hour later the company’s management had realised their mistake and announced that rides out of Sydney’s Central Business District would be free.

Uber rides out of the CBD today are free for all riders to help Sydneysiders get home safely. See http://t.co/UIwoom25Bm for more info. — Uber Sydney (@Uber_Sydney) December 15, 2014

User’s mistake is a classic example of the dangers of relying solely on an algorithm to determine business decisions; while things will work fine during the normal course of business, there will always be edge cases that create perverse results.

While machines are efficient; they lack context, judgement and compassion which exposes those who rely solely upon them to unforeseen risks.

As the Internet of Things rolls out, systems will be deployed where responses will be based upon the rules of predetermined formulas.

Businesses with overly strict rules and no provision for management intervention in extreme circumstances will find themselves, like Uber, at the mercy of their machines. Staking everything on those machines could turn out to be the riskiest strategy of all.

Rigging the Internet of Things

The Internet of Things offers many new opportunities for hackers

Hackers are infiltrating public companies to gain an edge on Wall Street warns a story on financial website Finextra.

This is not news, companies’ networks have been the target of insider traders since the early days of corporate computing. What is different today though are the nature of the risks as Chinese and even North Korean hackers are probing networks containing vast amounts of information to find weaknesses and confidential information.

For insider traders, it may be the internet of things turns out to be a boon. By hijacking delivery or supply data, traders may have an advantage over the market.

Things could get very nasty if those hackers subtly alter the data, say over reporting production yields, so a company gives the wrong income guidance based on faulty information.

Security is one of the big issues facing the internet of things sector and the consequences of poorly protected sensors or systems could be immense when governments, businesses and communities come to rely on a stream of data they can trust.

The bad guys are only just starting to explore the possibilities of the connected world.

A question of ethics

Uber’s missteps remind us that ethics matter in business

At this week’s Australian Gartner Symposium ethics was one of the key issues flagged for CIOs and IT workers; as technology becomes more pervasive and instrusive, managers are going to have to deal with a myriad of questions about what is the moral course of action.

So far the news isn’t good for the tech industry with many businesses failing to deal with the masses of data they are accumulating on users, suppliers and competitors.

A failure of transparency

One case in point is that of online ride service, Uber. One of Uber’s supposed strengths is its accountability and transparancy; the service can track passengers and drivers through their journey which should, in theory, make the trip safer for everybody.

In reality the tracking doesn’t do a great job of protecting riders and drivers, mainly because Uber has Silicon Valley’s Soviet attitude to customer service. That tracking also creates an ethical issue for the company’s management and one that isn’t being dealt with well.

Compounding Uber’s ethical problem is the attitude of its managers, when a Senior Vice President suggests smearing a journalist who writes critical stories then its clear the company has a problem and the question for users has to be ‘can we trust these people with our personal data?’

With Uber we may be seeing the first company where data management and misuse results in senior management, and possibly the founder, falling on their sword.

Journalists’ ethics

Another aspect of the latest Uber story is the question of journalistic ethics; indeed the apologists for Uber counter that because some journalists are corrupt that justifies underhand tactics from companies subject to critical articles.

That argument is deeply flawed with little merit and tells us more about the people making it than any journalist’s ethical compass, however there is a discussion to be had about the behaviour of many reporters.

As someone who regularly receives corporate largess — I attended the Gartner Symposium as a guest of BlackBerry and will be going to an Acer event tomorrow night — this is something I regularly grapple with; my answer (or rationalisation) is that I disclose that largess and let the reader make up their own mind.

However one thing is clear at these events; everything is on the record unless explicitly stated by the other party. This makes Michael Wolff’s criticism of Ben Smith’s original Uber story in Buzz Feed pretty hollow and gives us many pointers on Wolff’s own moral compass as he invites other writers to ‘privileged’ dinners where the default attitude is that everything is off the record.

Playing an insider game

Ultimately we’re seeing an insider game being played, where journalists like Wolff put their own egos above their job of telling their audience what is happening; Jay Rosen highlighted this problem with political coverage but in many respects it’s worse in tech, business and startup journalism.

It’s not surprising when a game is being played by insiders that they take offense at outsiders criticizing them.

Once the customers become outsiders though, the game is drawing to an end. That’s the fate Uber, and much of the tech industry, desperately want to avoid.

Uber in particular has many powerful enemies around the world and clumsy management mis-steps only play into the hands of those who see the company as a threat to their cosy cartels. It would be a shame if Uber’s disruption of the many dysfunctional taxi markets was derailed due to the company’s paranoia and arrogance.

Eventually ethics matter. It’s something that both the insular tech industry and those who write on it should remind themselves.

Democratising the internet of things

A primary school science project shows how communities can start using open data to monitor their neighbourhood’s environment

Last year Alicia Asin of Spanish sensor vendor Libelium spoke to this site about her vision of the internet of things improving transparency in society and government.

A good example of this democratisation of data was at the New South Wales Pearcey Awards last week where the state’s winners of the Young ICT Explorers competition were profiled.

Coming in equal first were a group of students from Neutral Bay’s state primary school with their Bin I.T project that monitors garbage levels in rubbish bins.

The kids built their project on an Arduino microcontroller that connects to a Google spreadsheet which displays the status of the bin in the school’s classrooms. For $80 they’ve created a small version of what the City of Barcelona is spending millions of Euro on.

With the accessibility of cheap sensors and cloud computing its possible for students, community groups and activists to take the monitoring of their environment into their own hands; no longer do people have to rely on government agencies or private companies to release information when they can collect it themselves.

Probably the best example of activists taking action themselves is the Safecast project which was born out of community suspicion of official radiation data following the Fukushima.

We can expect to see more communities following the Safecast model as concerns about the effects of mining, industrial and fracking operations on neighbourhoods grow.

The Bin I.T project and the kids of Neutral Bay Public School could be showing us where communities will be taking data into their own hands in the near future.

Connecting people to spaces

Smartphones, beacons and smart software are the key to future retail success believes Proximity Insight’s Steve Orell

Beacon technologies are one of the hottest items in the Internet of Things with retailers, sports stadiums and hotels looking at how they can use these devices to improve their operations and customer experiences.

At Dreamforce 2014 Proximity Insight’s Steve Orell spoke on the event’s wearable panel about how their service plugs into beacon technology and customer service.

Proximity Insight was born out of the 2013 Dreamforce Hackathon where Orell and his team were finalists. From that, the company set up operations in New York with a focus on customer relationship management in the retail industry.

Retail isn’t the only the field that Orell sees for Proximity Insight with the hotel and casino industries as being other targets.

“With the hotel, why check-in? Why not walk in and let your smartphone do it for you?” Orell asks.

“It’s all about making live so much more seamless and slick,” Orell adds. “There’s opportunities in every sector.”

For businesses looking at rolling out beacon technologies the key is to be adding value to enhance the customer experience, Orell believes.

“You have to be delivering something to the customer beyond tracking them, it’s about making the whole retail or hospitality experience better. It has to benefit the customer.”

With beacon technologies now becoming common and the supporting hardware being built into all smartphones, we can expect to see more applications coming onto the market. It’s worth considering how your business can use them to enhance the customer experience.

Paul travelled to Dreamforce 2014 as a guest of Salesforce

Salesforce faces the end of the database era

Cloud CRM giant Salesforce faces a challenge as searching unstructured data and analytics companies like Splunk change the business model.

Last week we looked at the way we organise information is changing in the face of exploding data volumes.

One of the consequences of the data explosion is that structured databases are beginning to struggle as information sources and business needs are becoming more diverse.

Yesterday, cloud Customer Relationship Management company Salesforce announced their Wave analytics product which the company says “with its schema-free architecture, data no longer has to be pre-sorted or organized in some narrowly defined manner before it can be analyzed.”

The end of the database era

Salesforce’s move is interesting for a company whose success has been based upon structured databases to run its CRM and other services.

What the company’s move could be interpreted that the age of the database is over; that organising data is a fool’s errand as it becomes harder to sort and categorise the information pouring into businesses.

This was the theme at the previous week’s Splunk conference in Las Vegas where the company’s CTO, Todd Papaioannou, told Decoding The New Economy how the world is moving away from structured databases.

“We’re going through a sea change in the analytics space,” Papaioannou said. “What characterised the last thirty years was what I call the ‘schema write’ era; big databases that have a schema where you have to load the data into that schema then transform before you can ask questions of it.”

Breaking the structure

The key with programs like Salesforce and other database driven products like SAP and Oracle is that both the data structures — the schema — and the questions are largely pre-configured. With the unstructured model it’s Google-like queries on the stored data that matters.

For companies like Salesforce this means a fundamental change to their underlying product and possibly their business models as well.

It may well be that Salesforce, a company that defined itself by the ‘No Software’ slogan is now being challenged by the No Database era.

Paul travelled to San Francisco and Las Vegas as a guest of Salesforce and Splunk respectively