What is DataOps? (What is data operations?)
DataOps, standing for data operations, is an approach used by data teams to improve the speed of delivering results while maintaining data quality.
It is a scalable approach to managing data that includes:
- Preparing and cleaning data 🧹
- Setting up data pipelines and workflows 🛠
- Creating metadata ⚙
- Cataloguing data 🗂
- Managing data infrastructure 🛢
What is DevOps? Is it connected to DataOps?
DataOps applies the agile methodology, DevOps and manufacturing quality principles (SPC or Statistical Process Control) to data management.
We get this a lot—DevOps and DataOps are the same. Well, they're not. Let's understand why.
DevOps combines software engineering, quality assurance and IT operations with agile practices to build, test and release software. It is based on building better collaboration between the previously siloed software development and operations teams.
Belgian project manager and agile practitioner Patrick Debois coined the term "DevOps" in 2009 at Devopsdays in Ghent, Belgium.
DevOps has helped several organizations reduce their software release cycle time, thereby reducing the time to deploy, time to market and helping them grow quickly.
Similar to DevOps, DataOps is an agile methodology. The goal of DataOps is to reduce the overall time taken by the data lifecycle—from acquisition and analysis to visualization and reporting.
DataOps fosters collaboration between the members of a data team (aka the humans of data) such as data scientists, data engineers, analysts and business users. DataOps removes bottlenecks and helps the humans of data perform data analysis, build models and automate processes.
Another core element of DataOps is building, managing and maintaining data pipelines.
Where did the term DataOps come from?
The term DataOps was first coined by Lenny Liebmann in a blog post for IBM Data & Analytics Hub. It was further popularized by Andy Palmer at Tamr.
Several professionals and organizations that support DataOps have put together a DataOps Manifesto, which has 18 DataOps principles that cover the mission, values, goals and best practices for DataOps practitioners worldwide.
Think we're missing something? 🧐 Help us update this article by sending us your suggestions here. 🙏