Why Big Data Resists Analysis

Overview

Recall questions on why a lack of structure beats size as an obstacle, why relational databases do not fit, and what machine learning is asked to do instead of a query.

What you need to know

Answer each one before you reveal it.

Questions from this subtopic

Each one with the model answer underneath.

1. 1

   Why is a lack of structure a greater obstacle than sheer size?

   Model answer

   Size has a known remedy: divide the data across more machines and divide the work with it. A lack of structure has no such answer. Without a consistent layout there is nothing for a program to depend on, so the data cannot be loaded into a conventional table, and a query written against particular field names will simply miss anything that does not carry them. The analysis itself becomes the hard part, not the storage.

2. 2

   Why is a relational database not a suitable home for most Big Data?

   Model answer

   For two reasons. The first is shape: a relational database expects every record to fit a row-and-column format decided in advance, and a great deal of Big Data has no such regular form. The second is scale: relational systems were designed to run on a single machine, and once the data exceeds what one server can hold it must be spread across a cluster — which is precisely the thing relational databases handle worst.

3. 3

   What does it mean to say relational databases do not scale well across multiple machines?

   Model answer

   It means that spreading one relational database over several servers costs far more than it gains. Its usefulness rests on joining tables and on guaranteeing that a transaction either completes fully or not at all, and both become expensive once the rows involved sit on different machines and every operation needs coordination across the network. The system can be made to work, but it slows down as machines are added rather than speeding up.

4. 4

   What is a schema, and why does fixing one in advance cause difficulty with Big Data?

   Model answer

   A schema is the declared structure of a dataset — which fields exist, what type each holds and how the tables relate. Fixing it in advance requires knowing the shape of the data before any of it is stored. Big Data rarely allows that: it arrives from many sources in forms nobody agreed on beforehand, and it changes as new sources appear, so any schema decided up front either rejects valid data or has to be rewritten constantly.

5. 5

   What are machine learning techniques used for with Big Data?

   Model answer

   They are used to find the regularities hidden in the data and pull useful information out of it. Instead of being told in advance what to look for, the technique works the other way round and infers the regularities from the data itself. That reversal is what makes it valuable here, because it removes the requirement to know the answer's shape before starting — the very requirement that a conventional query cannot do without.

6. 6

   What kinds of pattern is machine learning typically asked to find?

   Model answer

   Grouping items that resemble one another, such as sorting customers into segments that behave alike; picking out the case that resembles nothing else, which is how fraudulent transactions and faulty equipment are flagged; and predicting the next value in a sequence, such as forecasting demand or the next reading from a sensor. In each instance the technique works out the rule from the examples rather than being handed the rule to apply.

Other ways to work this

The same content, four other ways.

What is inside Levo

Six ways to work the same material.