Originally posted on SAS Voices.

Is the future of analytics within your enterprise architecture a rapidly changing agile innovation platform (Lab), separated from operations and broad enterprise audience usage, alongside a consistently slowly changing enterprise analytics platform (factory) that supports operations and a broad enterprise audience? That is the question this blog digs into?

This is an important IT topic at the moment as working out how innovate and modernize at the same time is one of the biggest challenges organizations are facing right now. In the past this has been like changing to new wings while flying the plane which has always been slow and risky.

Introducing Lambda architecture

Many people will be familiar with the Lambda architecture. If not then this Wikipedia page is perhaps a good place to start. In short it is a data processing framework which Wikipedia states:

“… attempts to balance latency, throughput, and fault-tolerance by using batch processing to provide comprehensive and accurate views of batch data, while simultaneously using real-time stream processing to provide views of online data”

It has been clear for some time that Lambda as an architecture for data processing is on the rise although not everyone is a fan. The rise in interest is caused by the clear need to handle not only regular batch updates of data but to also the move into the realm of handling rapidly arriving streaming data. I do not want to go into the weeds of the Lambda architecture here but if you are pursuing deploying such an architecture then you will need a broad range of technologies to support your including streaming analytics technologies and data management technologies amongst many others to consume, manage and govern your data.

The reason for starting with a discussion about Lambda in this blog is that it got me thinking about the broader challenges in Enterprise Architecture around Analytics in particular. I started to wonder if those same principles of one thing moving fast and another moving slowly could offer a solution to the challenge of bringing in the new analytics technologies and approaches of the day alongside established well trodden paths. As I have explored this further I think there are real merits in looking at such an approach.

The modern organization

If we look at the modern organization today CIOs/CTOs face significant challenges. They have all, unless they are a start-up, generally got a legacy mix of IT systems supporting their critical business processes to which they have added analytical technologies over time.

This overall mix of analytical technologies and systems has got so complex, and so engrained into the organizational operational fabric, that the inability to change it is holding back organizations as they strive to become more digital and move to exploit the latest analytical capabilities. Modernizing the “factory”, as I like to call the analytic platform supporting operations, is not as simple as just removing technology and replacing it with the latest and greatest. To some extent this is akin to the batch data processing side of a Lambda architecture. Here we need time to make change and change needs to be carefully assessed to determine its impact. Change impacts processes, people, culture and much more. It is not just about the technology, the integration points, the downstream systems and all the normal things we consider often when we think about IT technology projects.

This is why change is hard on the factory side of things and why many companies are unable to move quickly to modernize and transform into the digital organization they will need to be in the future. For these organizations the question is what can they do to overcome this inertia and reduce the risk when they do move?

Supporting digital transformation

There are really two approaches organizations can take to move forwards in my opinion.

Approach One

On one hand you can work exclusively on the modernization of the your enterprise architecture to update all your analytics to drive a better overall factory filled with the latest technologies and ways of working. Many organizations out there have this focus right now based on my experience. They are trying to understand the capabilities of the new analytic technologies on the market, react to the need to do something with “big data”, look for cost savings and more efficiency and they are doing all of this while the same exercise is going off with core operational systems which are also under scrutiny. All of this is being done to remain relevant in the market, cut IT costs to free up budget for other things, support regulatory requirements and meet new business requirements. There is nothing wrong with this approach. It is something that needs to be happening. My view is that focusing just on this means you cannot expect to stay on the cutting edge as movement forwards will generally be slow and deliberate as your business is in flight. You might make a big jump once but will you really do this at every generation of major capability evolution?

Approach Two

The second approach marries the ongoing aim to modernize the enterprise architecture when it comes to analytics (the factory) with the addition of a second platform to support analytics driven innovation. This second platform, often provisioned as something like an innovation lab, provides an agile environment to deveop a dynamic frequently changing platform free of the constraints of the more slowly moving platform, where ideas can be tried and incubated, along with a clearly defined set of services and processes to use it on an ongoing basis thorughout the organization. This platform can provide the directional focus on where to make changes in your analytics in your main enterprise architecture first to get the best returns and give guidance on your target architecture as you make that change.

To some extent the second platform can be seen as being akin to the streaming or real-time side of the Lambda architecture.

Thinking forward – acting quickly

This dual approach will let IT meet the requirements of the organization to be agile and innovative. It will also ensure that any changes we will make to our enterprise platform around analytics are well defined and scoped with a clear benefit lined up once that work is complete. This is also just like Lambda which clearly states that the two world have to come back together at some stage.

Your feedback wanted

I am very interested in your thoughts around this. Any other thoughts on how to tackle this issue? Feel free to follow me on Twitter @mark_torr to see what else I’m passionate about.