DataOps: What It Is, Why It’s Important Now, and How to Achieve It

It conquered the challenge of managing the exponential growth in data scale, variety, and speed. The second was DataOps — the alignment of people, automated technologies, and business agility to enable an order of magnitude improvement in the caliber and reduced cycle time of data analytics. DataOps expedites the flow of data for effective operations on both traditional and big data, by leveraging self-service capabilities to bypass traditional methods of engineering customized programs.

Here, we take a deeper dive into DataOps to discover why it’s important now.

DataOps: A Paradigm Shift

DataOps is not named randomly. It builds in the use of DevOps, which is a widely adopted and well-understood practice that accelerates software development through monitoring and automation to facilitate collaboration across application designers, operations staff and business users. While DataOps does have some capabilities like DevOps, it is a more sophisticated capability, and the comparison downplays its importance and distinction. DataOps is a paradigm shift in the fundamental way data is delivered and managed and completely challenges the typical way to integrate data.

• DataOps enables existing and new data services and products to be delivered quickly, despite changing infrastructures, environments, semantics and requirements, while also mitigating data threats.
• DataOps helps applications to interact more easily despite the data drift that emerges unexpectedly from dynamic technologies.
• DataOps transforms stodgy, centralized Business Intelligence “dashboards and reports” into real-time and democratized analytics capability that unlocks the huge potential.
• DataOps automatically handles data drift (the unpredictable, unannounced and unending mutation of data characteristics caused by the changing operation, maintenance and modernization of systems that produce the data) to reliably deliver data.
• DataOps changes the typical approach of designing and building custom data movement software into self-service capabilities that people simply operate.

Why Now?

Three forces have united to create a “perfect storm” that today require DataOps for data delivery and management — a combination of events that are not individually dangerous but occurring together produce a disastrous outcome.

The first force is the four V’s of big data (volume, variety, velocity, and veracity), which is like a hurricane making it hard to find, deliver and access data. Once you feel you have data under control, it changes. The quantity, speed and endless variety of data (unstructured, structured, batch, real-time, streaming, cloud, IoT, etc.) feels like chaos of a hurricane. It all must be rationally defined to be trusted, make sense, be truthful and be protected from people who may damage it or steal it. This is a scale of complexity that didn’t exist in earlier years.

The second force is an unceasing wave of technology change. Data management technology is endlessly being improved to find data in new devices and structures. It needs to be transformed, delivered to where is needed, and cataloged, analyzed, monitored, secured, compressed, archived, and the list goes on and on. In its totality, it feels like a tidal wave — a tsunami of technology.

The third force is that data is more valuable than ever. Data is now independent from applications and must be managed explicitly in all states and forms in order for the enterprise to operate its business-critical requirements. Data is highly valuable since it is no longer just facts about your business or operations, analytics are predictive and prescriptive and, in many respects, data is your business. The third force is that data is an asset that needs to be governed and secured and at the same time needs to democratized and used widely.

The DataOps Center of Excellence

Bottom-up evolution or top-down transformation are the two main strategies for implementing DataOps. Bottom-up evolution is the fastest way to start. Data professionals can do so by incrementally applying new methods. The quickest way to finish a mature practice that is embedded company-wide is top-down by following a transformation roadmap with senior management support. The transformation strategy takes longer to get started than the bottom-up evolution, but it will establish a foundation to scale DataOps and support its company-wide adoption.

Whether bottom-up evolution or top-down transformation, both strategies need change agents, which could be new hires or long-term employees that have the in-house connections to get things done in the company. Once change agents are identified, work with them and begin to apply simple Agile or Lean methods such as flow of value, waste elimination and fail fast. With each success, evolve the capability.

The top-down transformation of DataOps leverages some of the same methods as bottom-up, but with more structure and formality. First, you need support from an executive sponsor since you will run into resistance from team processes, policy changes, funding needs and other roadblocks. Second, formally define and document your vision and charter. The third step is to launch your DataOps blueprint, which consists of three elements:

• Strategic roadmap — A “checklist” of milestones or outcomes arranged in multiple tracks and phases.
• Program roadmap — List and obtain approval for items like timelines, business justification change drivers, costs, current state models (as well as future), constraints and risks. The program roadmap adds concrete steps to the strategic roadmap and parses them out in phases.
• Project plan — Details efforts for a program initiate showing a detailed breakdown of activities, resources, dependencies, costs, deliverables and other elements.

The next steps for transformation are ongoing activities to grow and sustain your DataOps capability:

• Execute and advertise/market: The program owners develop detailed project plans, secure the resources, and then make it happen. Keep stakeholders and the data community up to date with progress. Make specific efforts to highlight successes and measurable outcomes.
• Periodically assess and renew your plan: Do a periodic review with the executive sponsor and leadership team; depending on the pace and speed of your transformation, you should do a review every month, quarter or year. At least once a year, you should review and possibly reshape the plan, especially if company strategies have evolved or significant technologies or other best practices are now possible.
• Reinforce the DataOps culture: This is an ongoing process to ensure it remains part of the fabric of your company.
• In terms of the DataOps name or brand, you may be able to build on a competency center or center of excellence that already exists in the enterprise; such as a Security Competency Center, Integration COE, BI COE, Network Operation Centers, and so on. One option is to structure it as the Chief Data Office or Data Operations Center that serves as a center of excellence across the enterprise as it is influencing and directing the practice being developed across the enterprise.

Data is no longer an intangible asset; rather, it is a vital pillar of corporate strategy. As such — and because of data’s dynamic nature — businesses must adopt DataOps strategies, tools and practices to ensure their data’s integrity is maintained and its value is realized.