Originally posted on SAS Voices
It’s time. The first of the big data V’s, volume, seems to be coming under control technology wise, even if that technology has not been implemented everywhere. Prices of disks and memory are tumbling and the introduction of technologies, such as Hadoop, are making vast amounts of data cheap to store.
On top of that, numerous technologies have made it easier to process and manage that massive volume of data. In short, the technologies and tools organizations need to deal with the problem of volume are now readily available.
Just when it seemed we could all take a small break from the big data assault on our organizations, since now we can store and process massive amounts of data, the second V, velocity, is gaining in importance – and most organizations are not ready.
Increasingly, the most valuable data that organizations need to handle will no longer be predominantly at rest, waiting for questions to be asked of it. Instead, it will be fluid and somewhat dynamic when it is at its most valuable. This streaming data is the key to the next generation of real-time services.
Streaming data is not a new concept. Some industries have been taking advantage of streaming data for sometime. If you think about it, the trading floor digitized itself from a paper based system into a digital system that needed to handle huge flows of data. If you like, it was a front runner in this digitization process.
We are starting to see more and more flowing data everywhere. As the digitization of many aspects of the world gathers pace, flows of data will become ever more ubiquitous, diverse and rich.
This leads to the question of whether you really need to store all the data and analyse it. Perhaps more importantly, what services can you deliver using it in REAL-TIME, as it flows into your organization or even as it is created by the sensor and devices themselves? And which technologies do you need to enable those services?
Effectively, this new flow of data requires you to handle streaming data using technologies such as event stream processing and to have access to analytics to make intelligent informed decisions automatically as the data is flowing.
A great TED Talk by Kenneth Cukier, “Big data is better data,” mirrors a lot of my recent thinking. In that talk Cukier speaks to how the digitization of the world is going to be vital to solve the world challenges we have ahead of us, such as feeding everyone, providing energy, better healthcare and more. For example, he explains that if you can analyze human waking postures and build patterns based on size, shape, etc, you could potentially monitor driver’s postures in cars (in real-time) and determine if someone is starting to slump and potentially fall asleep at the wheel. This data could be used to alert drivers to avoid accidents or worse.
In essence, he is saying that once you digitize something, you can start to innovate and build out services that are good for everyone. As the world digitizes, there will be more and more data we have never had available to us before.
Cukier’s talk ties in to one of my favorite topics, the Innovation Lab, where you can try to find ways to use this new stream of digital data. More importantly, Cukier encourages listeners to overcome the fear of big data, which is prevalent in the world today, by showing big data can be used to enhance and extend our existence not to monitor it.
Digitization of most things is inevitable. When you add hyper-connectivity to the equation, along with automation, it is clear to see that streaming data is the next frontier which could dramatically alter our lives and the services organizations can offer. Are you already looking at the velocity angle of Big Data? If not, maybe the time is now to consider it!