Data Mining Foundations and Core Concepts

Core Disciplines of Data Mining:

• Database Systems: Provides efficient disk storage, indexing schemes (B-trees, hash indices), transactional integrity, and out-of-core query execution necessary to manage massive, persistent data repositories.
• Machine Learning and Artificial Intelligence: Supplies inductive learning models, non-linear function approximation, heuristics, search strategies, and pattern recognition algorithms.
• Statistics: Provides mathematical rigor, probabilistic distributions, hypothesis testing, sampling theory, confidence intervals, and uncertainty estimation.
• High-Performance and Distributed Computing: Offers parallel processing frameworks and distributed storage architectures required when datasets exceed single-machine memory or storage limits.

Formal Definition: Data mining is the non-trivial extraction of implicit, previously unknown, and potentially useful information or patterns from large quantities of data.

Deconstruction of Key Terms:

• Non-trivial: The pattern cannot be uncovered by simple eyeballing or elementary SQL aggregations (e.g., finding MAX(Salary)). It requires sophisticated algorithmic modeling.
• Implicit: Knowledge is hidden within complex data relationships rather than directly recorded.
• Previously Unknown: The extracted patterns reveal novel structural insights rather than confirming obvious domain knowledge.
• Potentially Useful: The discovered patterns provide actionable intelligence that can drive automated systems or business decisions.

The 6-Layer BI Pyramid (Bottom to Top — Increasing potential to support business decisions):

1. Layer 1: Data Sources: Operational databases, paper files, web documents, scientific experiments, and transaction logs.
2. Layer 2: Data Preprocessing & Warehousing: Data cleaning, schema integration, and loading into Enterprise Data Warehouses (EDW).
3. Layer 3: Data Exploration: Statistical summaries, OLAP slicing/dicing, querying, and reporting.
4. Layer 4: Data Mining: Pattern discovery, information extraction, and model training.
5. Layer 5: Data Presentation: Interactive dashboards, visualization tools, and analytical reports.
6. Layer 6: Decision Making: Executive strategy formulation, campaign execution, and automated business decisions.

The KDD Pipeline: Formal mapping of KDD process pipeline from Input Data through Pre-processing, Mining, and Post-processing to Knowledge:

$$\text{Input Data}⟶\text{Data Pre-processing}⟶\text{Data Mining}⟶\text{Post-Processing}⟶\text{Knowledge}$$

Detailed Phase Breakdown:

1. Input Data: Raw heterogeneous data repositories (relational tables, text files, streams).
2. Data Pre-processing:

• Data Integration: Merging disparate data sources into a unified structure.
• Data Normalization: Scaling feature ranges (e.g., $[0,1]$ min-max scaling or $Z$-score standardization).
• Feature Selection: Filtering out irrelevant or redundant variables.
• Dimension Reduction: Compress high-dimensional feature spaces (e.g., PCA).

1. Data Mining Core: Applying algorithmic techniques (Classification, Clustering, Association Mining) to extract patterns.
2. Post-Processing:

• Pattern Evaluation: Validating patterns using objective statistical metrics (support, confidence, lift, $R^2$).
• Pattern Selection: Pruning trivial or duplicate rules.
• Pattern Visualization: Presenting results visually to human domain experts.

Fayyad et al. (1996) Advances in Knowledge Discovery and Data Mining Taxonomy:

1. Predictive Tasks: Target mapping function for predictive methods fitting dependent variables from independent feature vectors. Use known explanatory variables ($X$) to fit a target model $f:X\to Y$ that predicts an unknown, future, or target variable ($Y$).
2. Descriptive Tasks: Uncover human-interpretable patterns, groupings, and underlying structural relationships that describe the data without targeting a specific predefined label.