Paul Wallbank – Brilliant Communications

Tag: big data

  • The messy business of metadata

    The messy business of metadata

    “We kill people based on metadata,” former US National Security Agency director Michael Hayden told a panel at John Hopkins University last year.

    That statement brings sharply into focus the current Australian debate about Telcos being mandated to retain metadata. This information is powerful and can be used in ways not expected by the community.

    How metadata is used depends upon how it is defined; one of the worrying parts of the Australian debate is just how nebulous the definition of ‘metadata’ is.

    Defining metadata

    The basic definition of metadata is that it is ‘data about data’, every packet of data that’s transmitted across the internet is wrapped in information to help the network deliver it to its intended location in a useful form.

    Strictly speaking this is where the glib ‘address on the envelope’ line so beloved by clueless politicians promoting data retention comes in; this information is useful but not private as anyone can see it.

    That view underlies the defining metadata case, the 1979 US Supreme Court’s Smith v Maryland case, that held telephone numbers are not private information as they’ve been disclosed to the phone company.

    Metadata everywhere

    With the rise of digital technologies, metadata became much more than just phone numbers and addresses; digital cameras encode photographs with EXIF information that includes details of the date, time camera, shutter speed, focal length and increasingly the location where the photo was taken.

    The smartphone you might take that photo with is logging into base stations which telcos are tracking, this is one of the applications law enforcement agencies regularly use to catch criminals, and increasingly near field communications like BlueTooth are being logged by everything from toll gates to bus shelters to track phone usage.

    Every email sent has dozens of lines of metadata describing the servers, routing and software used in sending it. Only a tiny fraction of a Twitter message is the 140 character content, the rest is metadata.

    This site has hundreds of lines of metadata to tell your browser how to display it and your browser itself is storing dozens of cookies that tell other sites where you’ve been – all of this is metadata.

    When we add the Internet of Things to the metadata debate things get even more complex. As home devices, smartcars and wearable technology generating packets of data, it becomes far easier for governments and private enterprise to stitch together a complete picture of an individual even without looking at the actual content of the packets.

    The risks for service providers

    As Australian ISP iiNet raised in an excellent submission to the Senate committee on data retention, collecting all of this data creates costs and risks for service providers as they have to store, manage and protect massive amounts of data.

    Most of that data will never be looked at, but agencies want the information stored because they can. The bizarre thing is that under Australian law, specifically section 313 of the Telecommunications Act, is that a public servant – not just a a law enforcement officer – only has to ask for the information.

    For most people the dangers of government surveillance are remote however all of this information is also available to private organisations ranging from health insurers to credit checking bureaus; the real debate is just how much data do we want others to have on our private lives.

    At present we’ve defined metadata too broadly and private information too narrowly.

    What’s notable in every discussion about increasing government or police powers is the mantra of getting the laws to catch up with technology.

    In the case of metadata it may well be the definition of individual rights has to catch up with modern technology, not the powers of government.

    Similar posts:

  • A big reset button on business

    A big reset button on business

    “Every large company is just another color of a spore in a petrie dish.”

    For the latest Decoding the New Economy video Internet Pioneer Doc Searls discusses The Respect Network, online privacy and the future of business on the web.

    Doc Searls is one of the internet’s pioneers who helped write The Cluetrain Manifesto that laid out many of the ideas that underpinned the philosophies driving the early days of the internet.

    Searls’ visit to Sydney was part of the rolling worldwide launch of the Respect Network, a system designed to improve internet users’ privacy through ‘personal clouds’ of information where people can choose to share data with companies and others.

    A big reset button on business

    In many ways The Respect Network shows how the internet has evolved since the days of the Cluetrain Manifesto, something that Searls puts in context.

    “We wrote the Cluetrain Manifesto in 1999,” says Searls. “At that time Microsoft ruled the world, Apple was considered a failure – Steve Jobs had come along and they had the iMac but it was all yet to be proven – Google barely existed and Facebook didn’t exist at all.”

    “On the one hand we saw the internet, we being the four authors of the Cluetrain Manifesto, and this whole new thing in the world that basically hit a big reset button on ‘business as usual’”

    “It did that. I think we’re vindicated on that.”

    Resetting business

    “What we have now are new industrial giants; Apple became an industrial giant, Microsoft are fading away, Nokia was the number one smartphone company and they’re all but gone.”

    One of the key things with today’s markets in Searls’ view is the amount of information that businesses can collect on their customers; something that ties into the original Cluetrain idea of all markets being conversations.

    With the evolution of Big Data and the internet of things, Searls sees challenges for companies using old marketing methods which rely upon online tracking. Something that’s a challenge for social media services and many of the existing internet giants.

    “The interesting thing is there’s a lot more intelligence that a company can get directly from their customers from things they already own than following us around on the internet.”

    Breaking the silos

    Searls also sees the current trend towards the internet being divided into little empires as a passing phase, “every company wants a unique offering but we need standards.”

    For Searls the key thing about the current era internet is we’re only at the beginning of a time that empowers the individual,  “the older I get, the earlier it seems.”

    “Anyone of us can do anything,” Searls says. “That’s the power – I’m optimistic about everything.”

    Similar posts:

    • No Related Posts
  • The limits of big data

    The limits of big data

    A story in the Atlantic – Why Poor Schools Can’t Win At Standardized testing – illustrates the limits of Big Data.

    When Meredith Broussard tried to computerise the text book inventory of her son’s school district she found the project limited by poor systems, fragmented record keeping and siloed management.

    Broussard found the records were manually collated, collected on Microsoft Word documents and emailed to an under resourced office that entered details into an Excel Spreadsheet.

    The Philadelphia schools don’t just have a textbook problem. They have a data problem—which is actually a people problem. We tend to think of data as immutable truth. But we forget that data and data-collection systems are created by people.

    The human factor is a key limitation with any technology; if people aren’t collecting or using data properly than the best computer system in the world is useless. Garbage In, Garbage Out is a long standing IT industry saying.

    Management systems are more than computer networks, they go to the very core of an organisation’s culture which in itself is probably a better indicator of how well a company or institution will survive the current period of change.

    Were the Philadelphia public school system a business it would be a very good example of a company on its way to being digital roadkill, that it’s an educational network should worry anybody concerned about the economy’s future. That’s a bigger issue than Big Data.

     

    Similar posts:

    • No Related Posts
  • Staying healthy with Big Data

    Staying healthy with Big Data

    US medical centre chain Carolinas HealthCare has started mining patients’ credit card data to predict health outcomes reports Bloomberg Businessweek.

    The idea is that by looking at credit information and purchasing records, doctors can anticipate what ailments their patients will present with.

    Carolinas Healthcare’s matching of spending patterns to healthy is an obvious application of Big Data which illustrates some of the benefits that mining information can deliver for individuals and the community.

    Should the project overcome patients’ valid privacy concerns, this is the sort of application that is going to be increasingly common as organisations figure out how to apply software to their mountains of information.

    Similar posts:

    • No Related Posts
  • A bot named Willy and the risk of trusting data

    A bot named Willy and the risk of trusting data

    For two years we were captivated by spectacular rise of the Bitcoin virtual currency. Allegations those gains were a result of market fixing raise important questions about the integrity of our data networks.

    The Coin Desk website discusses how the Mt Gox Bitcoin exchange was being ramped by computer bot network nicknamed Willy.

    Rampant market ramping – where stock prices are pushed up to attract suckers before those in know sell at a profit – has a proud financial market history; during the 1920s US stock boom, fortunes were made by inside players before the crash and its subsequent banning in 1934.

    So it wouldn’t be a surprise that some smart players would try to ramp the Bitcoin market to make a buck and using a botnet – a network of infected computers – to run the trades is a good technological twist.

    Blindly trusting data

    The Willy botnet though is a worry for those of us watching the connected economy as it shows a number of weaknesses in a world where data is blindly trusted.

    As Quinn Norton writes on Medium, everything in the software industry is broken and blindly trusting the data pouring into servers could be a risky move.

    The internet of things is based upon the idea of sensors gathering data for smart services to make decisions – one of those decisions is buying and selling securities.

    Feeding false information

    It’s not too hard to see a scenario where a compromised service feeds false data such as steel shipments, pork belly consumption or energy usage to manipulate market prices or to damage a competitor’s business.

    Real world ramifications of bad data could see not only honest investors out of pocket but also steel workers out work, abattoirs sitting on onsold stocks of pig carcasses or blackouts as energy companies miscalculate demand.

    The latter has happened before, with Enron manipulating the Californian electricity market in the late 1990s.

    When your supply chain depends upon connected devices reporting accurate information then the integrity of data becomes critical.

    Like much in the computer world, the world of big data and the internet of things is based up trust, the Mt Gox Bitcoin manipulation reminds us that we can’t always trust the data we receive.

    Similar posts:

    • No Related Posts