Introduction to Data Mining & Big Data
Why we need data mining, what it is, the 5 V's of big data, and where data mining is used — banking, health and the public sector.
▶Lecture videos
Click any timestamp — here or inside the notes — to jump the video to that moment. Your position is remembered.
Introduction to Big Data
What big data is and the V's that describe it.
00:01Why data science and big data matter02:47Examples of the data explosion03:32Volume: the amount of data today04:22Variety (complexity) of data05:11Extracting knowledge: integration and cleaning05:56Applications: marketing, e-promotions, industry1What data mining is about
Data mining is the process of discovering interesting patterns or knowledge from massive amounts of data. "Interesting" means non-trivial, implicit, previously unknown and potentially useful.
The lesson introduction describes data mining as:
- An iterative process where progress means discovery, made automatically or manually.
- Most useful in exploratory analysis, where you have no fixed idea of what an "interesting" result will be.
- The search for new, valuable and non-trivial information in large volumes of data.
- A cooperative effort of humans and computers. The best results come from balancing human experts' knowledge (describing problems and goals) with computers' search power.
2Why data mining is required
- Huge volumes of data fill our computers, networks and lives. Governments, scientific institutions and businesses spend enormous resources collecting and storing it.
- Only a small amount of it is ever used. The volumes are too large to manage, or the structures are too complicated to analyse.
The original effort to create a dataset usually focuses on storage efficiency. It doesn't include a plan for how the data will later be used and analysed.
- Understanding large, complex, information-rich datasets matters in virtually all fields of business, science and engineering.
- In business, corporate and customer data are a strategic asset. Being able to extract hidden knowledge and act on it matters in a competitive world.
- The whole process of applying computer-based methods to discover knowledge from data is called data mining.
3Big Data and the 5 V's
- 2005: Roger Mougalas of O'Reilly Media first coined the term "Big Data", a year after O'Reilly created the term Web 2.0.
- Big data means data sets so large they are almost impossible to manage and process with traditional business intelligence tools.
- Social networks and Web 2.0 drove the growth. Start-ups and then governments began big-data projects.
Big Data is the large volume of data, both structured and unstructured, that floods a business every day. It's not the amount of data that matters but what organisations do with it.
The original 3 V's
| V | Meaning | Examples |
|---|---|---|
| Volume | Data collected from many sources in huge amounts | Business transactions, social media, sensor / machine-to-machine data |
| Velocity | Data streams in at unprecedented speed and must be handled promptly | RFID tags, sensors, smart metering in near-real time |
| Variety | Data comes in all types of formats | Structured numeric DB data → unstructured text, email, video, audio, stock ticker, financial transactions |
Two more V's added later
| V | Meaning |
|---|---|
| Veracity | Trustworthiness of the data. More data doesn't mean clean or accurate data. Data must stay consolidated, cleansed, consistent and current to support the right decisions. |
| Value | Worth of the data extracted. Endless data isn't always useful. You must understand the costs and benefits of collecting and analysing it, so the mined data can ultimately be monetised. |
Volume = how much. Velocity = how fast. Variety = how many kinds. Veracity = how true. Value = how useful.
"Explain big data using the V's" is a typical question. Give all five, a one-line meaning and an example each. Mention that the first three are the original definition and Veracity and Value were added later.
4Defining data mining properly
The customer loyalty example
The problem is fickle customer loyalty in a competitive market. A database of customer choices and profiles holds the answer:
- Analyse patterns of former customers to find the characteristics of those who left and those who stayed loyal.
- Use those characteristics to identify current customers likely to "jump ship".
- Target that group with special treatment. This would be too costly to give the whole customer base.
- On the positive side, the same techniques find customers likely to want another service the company offers, so they can be targeted with special offers.
In today's competitive, customer-centred economy, data is the raw material that fuels business growth, if only it can be mined.
The formal definition
The four requirements:
- The process must be automatic or (more usually) semi-automatic.
- The patterns must be meaningful: they lead to some advantage, usually economic.
- The data is always present in substantial quantities.
- Useful patterns let us make non-trivial predictions on new data.
How are patterns expressed?
Black box
The inside is effectively incomprehensible. It makes predictions, but you can't see why.
e.g. a complex neural network
Transparent box
Its construction reveals the structure of the pattern. You can read and understand it.
e.g. a decision tree or rule "IF age < 25 AND student THEN buys laptop"
Both make useful predictions. The difference is whether a human can understand the pattern.
5Applications of data mining
Data mining is used in banking, stock exchanges, marketing, retail, signal processing, insurance, telecoms, e-commerce, health, medicine, biology, genetics, industry, construction, education, intelligence, science and engineering. The notes focus on three sectors:
1. Banking and finance
Used to find what, when and why customers prefer.
| Application | Idea |
|---|---|
| Market basket analysis | If a customer buys a certain group of items, they're likely to buy another group. This helps retailers understand buying patterns and change the store layout (e.g. bread near butter). |
| CRM (Customer Relationship Management) | Strategies to acquire and retain customers. Mining customer data shows where to focus on retention. |
| Fraud detection | Billions are lost to fraud each year. Use historical data to model fraudulent behaviour and flag similar new cases. |
2. Health sector
Reduces the complexity of studying healthcare transactions.
| Application | Idea |
|---|---|
| Measuring treatment effectiveness | Compare symptoms, causes and treatments to find the most effective course of action. Compare patient groups on different drug regimens to find what works best and saves most money, which helps standardise treatment. |
| Bioinformatics | Biology is data-rich. Uses: gene finding, protein function inference, disease diagnosis/prognosis, treatment optimisation, protein/gene interaction networks, data cleansing, protein subcellular location prediction. |
| Fraud detection | Establish normal patterns, then find unusual medical claims by clinics, physicians or labs: inappropriate referrals or prescriptions, insurance fraud. |
3. Public sector
Predicting public safety and security problems and optimising decisions.
| Application | Idea |
|---|---|
| Weather forecasting | Build prediction models from historical data. Needed for climate monitoring, agriculture, energy planning, aviation and communication. |
| Criminal investigation | Criminology identifies crime characteristics. Crime data is high-volume with complex relationships, which suits mining. Text crime reports can be converted and used for crime matching. |
Name the sector → application → what is mined → benefit. For example: Banking → fraud detection → historical transaction data modelled for fraud patterns → flags new suspicious transactions early.
6Quick revision sheet
| Concept | Remember it as |
|---|---|
| Data mining | Discovering interesting (non-trivial, implicit, unknown, useful) patterns from massive data |
| Nature | Iterative, exploratory, human + computer cooperation |
| Why needed | Huge data, mostly unused; created for storage efficiency, not analysis |
| Big Data term | Roger Mougalas, O'Reilly Media, 2005 |
| 3 V's | Volume, Velocity, Variety |
| +2 V's | Veracity (trust), Value (worth) |
| Process | Automatic or semi-automatic; patterns give economic advantage |
| Pattern forms | Black box vs transparent box |
| Banking | Market basket, CRM, fraud detection |
| Health | Treatment effectiveness, bioinformatics, fraud detection |
| Public | Weather forecasting, criminal investigation |
?Self-check quiz
10 questions on this week. Answer them all, then press Check my answers. Your best and latest scores are saved and shown on the subject page.