Monday, August 12, 2013

Big Data Meets the Cloud (Post 2 of 5)

Big Data Meets the Cloud: Takeaways Pt 2 
(see the first article in the series, Big Data Explosion in the Cloud)

Imagine you are a marketing executive at a large retailer. How do you know who to target, when to target them, and what products they want to buy? Now imagine you know everything each customer has purchased, when they purchased it and even have a solid understanding of why. You have millions and millions of data points profiling your customers, giving you the opportunity to individually target each one – you could send the recent graduate a coupon for hampers, the new mom an e-mail about baby food.

Sounds good, and it’s possible with today’s technology, data and analytic innovation, but getting to this enviable potential requires navigation of numerous options (to use cloud or not? What platform? Data? Methods?). With so much information available, what do you do first? How do you turn this data into an actionable solution, and what are the tools you need to do it?

By Carmen Augustine -- August 12, 2013

Big Data itself is a relatively new phenomenon – the term has become synonymous with the recent explosion in the size, diversity and spread of both creation and processing of available data – which often requires new approaches to data storage and processing. Now that technology has been developed to take advantage of big data in cloud architecture platforms, the question is: how do we make use of what we’ve built? Who can benefit, and to what degree does the cloud democratize big data?

In last week’s articleThe Big Data Explosion Meets the Cloud, I introduced the growing role of Big Data in the business world and how cloud computing platforms have changed the shape of the industry. This week, I’ll take a deeper dive into trends in the industry, innovations on the horizon and opportunities for improvement. Read on to discover the insights I absorbed from the front row of the New York Technology Council’s June 26th live panel event “The Big Data Explosion: How to Process, Analyze and Visualize in the Cloud”!
  • Scott Rose of Think Big was an advocate of using big data to cultivate more innovation and increase the focus of teams on innovative work, as opposed to day-to-day work keeping the lights on:
    • “[Benefit] is only limited by the imagination of the organization…I’d be hard pressed to look at an organization in any industry that can’t benefit.”
    • Mr. Rose added that in practice about 80-90% of information management budgets go towards ETL while only 10% go toward innovation – a ratio that should be flipped.
  • Stuart Sim of PlaceIQ agreed that big data infrastructure has opened the door to creative business model restructuring, but cautioned that with big data comes big data processing:
    • Imagine an advertising campaign that can take real time consumer data and spit out a custom-tailored advertisement.
    • Though he has experienced the incredible power of real time data collection, he was quick to note that it isn’t always the most effective solution, echoing Mr. Rose’s woes about implementation of big data systems.
    • “The biggest challenge around big data is assembling meaningful data you can do the science on. I spend an awfully large chunk of my time making sure data is not mangled.” 
    • It’s not simply a matter of collecting petabytes of consumer data – you have to know what to look for to capitalize on this data.
  • Edouard Servan-Schreiber of 10gen, who had extensive experience with predictive modeling before becoming a MongoDB specialist, agreed that data cleansing is an unfortunate fact of life but agreed that despite this necessity big data is a platform for innovation:        
    • “To me it's just, ‘we won't resolve this, we'll just manage how much we want to clean [the data] before we use it’.”
    • “Everything is so heavy to move, when these big organizations are trying to do anything, there is an enormous amount of data processing to do what seems like a trivial thing.”
    • Cleaning and sorting through data will always be a part of the equation – in some ways there’s no way around that reality
    • Despite the drawbacks, it may be worth the rigorous processing – Mr. Servan-Schreiber asserted that he could use data alone to predict if someone was falling in love
When refining the incredible potential stored in big data, it’s important to think about what you need, where it comes from and what it will take to turn that data into an asset – by keeping the end goal in mind, it’s easier to go from data to dollars focusing on innovating solutions rather than wading through an ocean of data.