What is big data?

Big data refers to massive volumes of structured, semi-structured or unstructured data that is too complex to be processed by traditional data systems (RDBMS and data warehouses).

Examples of big data include social media data, transactional data (stock prices, purchase histories), sensor data (location data, weather data) and satellite data.

Why traditional data systems don't work for big data

Traditional data systems work best with structured data, but aren't fully equipped to process large amounts of unstructured data.

Gartner estimates that more than 80% of enterprise data is unstructured.

Big data is massive and complicated because it grows exponentially with time.

According to Domo, we generate over 2.5 quintillion bytes (2.5 exabytes) of data every day. They estimate that by 2020, 1.7 MB of data will be created every second for every person on Earth!

Holy cow, that's a lot of data 😱!

The volume, velocity, variety and veracity of big data

Big data is commonly defined by the 3Vs—Volume, Variety and Velocity of data. Some organizations also use additional Vs such as the Veracity, Value and Variability of data.

The 4 Vs of big data by IBM. Image courtesy: IBM Big Data & Analytics Hub

How can you analyze big data?

Usually, these datasets can’t be fit on a single machine and require distributed computing.

A distributed storage system uses cloud technology to store data on several different computers. You can store data on multiple systems, which divide the computing load and increase data processing speed.

Analyzing big data requires complex big data technologies such as Hadoop technologies, Spark, Kafka, NoSQL databases and cloud-based data warehouses.

To analyze big data, organizations collect raw data using data acquisition methods. The data collected is cleaned, processed and stored in large data warehouses or data lakes for further analysis.

Who uses big data?

Big data is revolutionizing almost every industry, business and organization worldwide as it helps you make intelligent decisions and be more efficient.

Banks, healthcare providers, manufacturers, retailers, government agencies and educators use big data.

Think we're missing something? 🧐 Help us update this article by sending us your suggestions here. 🙏

See also

Articles you might be interested in

  1. The complete beginner’s guide to data everyone can understand
  2. How big data can influence decisions that actually matter?
  3. How big data can drive big decisions?