Data mining

Data mining can be defined as "the nontrivial extraction of implicit, previously unknown, and potentially useful information from data". Data mining may also be defined as "the science of extracting useful information from large data sets or databases".

Data mining is the principle of sorting through large amounts of data and picking out relevant information. It is usually used by business intelligence organizations, and financial analysts, but it is increasingly used in the sciences to extract information from the enormous data sets generated by modern experimental and observational methods.

Discussion

Metadata, or data about a given data set, are often expressed in a condensed data mine-able format, or one that facilitates the practice of data mining. Common examples include executive summaries and scientific abstracts.

Although data mining is a relatively new term, the technology is not. Companies for a long time have used powerful computers to sift through volumes of data such as supermarket scanner data to produce market research reports. Continuous innovations in computer processing power, disk storage, and statistical software are dramatically increasing the accuracy and usefulness of analysis.

Data mining identifies trends within data that go beyond simple analysis. Through the use of sophisticated algorithms, users have the ability to identify key attributes of business processes and target opportunities.

The term data mining is often used to apply to the two separate processes of knowledge discovery and prediction. Knowledge discovery provides explicit information that has a readable form and can be understood by a user. Forecasting, or predictive modeling provides predictions of future events and may be transparent and readable in some approaches (e.g. rule based systems) and opaque in others such as neural networks. Moreover, some data mining systems such as neural networks are inherently geared towards prediction and pattern recognition, rather than knowledge discovery.

Misuse of the term

The term "data mining" is often used incorrectly to apply to a variety of unrelated processes. In many cases, applications may claim to perform "data mining" by automating the creation of charts or graphs with historic trends and analysis. Although this information may be useful and timesaving, it does not fit the traditional definition of data mining, as the application performs no analysis itself and has no understanding of the underlying data. Instead, it relies on templates or pre-defined macros (created either by programmers or users) to identify trends, patterns and differences.

A key defining factor for true data mining is that the application itself is performing some real analysis. In almost all cases, this analysis is guided by some degree of user interaction, but it must provide the user some insights that are not readily apparent through simple slicing and dicing. Applications that are not to some degree self-guiding are performing data analysis not data mining.

Related terms

Although the term "data mining" is usually used in relation to analysis of data, like artificial intelligence, it is an umbrella term with varied meanings in a wide range of contexts. Unlike data analysis, data mining is not based or focused on an existing model which is to be tested or whose parameters are to be optimized.

A promising application of Knowledge discovery is in the area of software modernization which involves understanding existing software artifacts. Usually the knowledge obtained from existing software is presented in the form of models to which specific queries can be made when necessary. An entity-relationship model is a common way to represent knowledge obtained from existing software. The Object Management Group (OMG) developed the Knowledge Discovery Metamodel (KDM), which defines an ontology for software assets and their relationships, for the purpose of performing knowledge discovery of existing code. Knowledge discovery from existing software systems, also known as software mining is closely related to data mining, since existing software artifacts contain enormous business value, key for the evolution of software systems. Knowledge Discovery from software systems addresses structure and behavior as well as the data processed by the software system. Instead of mining individual data sets, software mining focuses on metadata, such as database schemas. The OMG Knowledge Discovery Metamodel provides an integrated representation for capturing application metadata as part of a holistic existing system metamodel. Another OMG specification, the Common Warehouse Metamodel focuses entirely on mining enterprise metadata.

In statistical analysis where there is no underlying theoretical model, data mining is often approximated via stepwise regression methods wherein the space of 2^k possible relationships between a single outcome variable and k potential explanatory variables is smartly searched. With the advent of parallel computing, it became possible (when k is less than approximately 40) to examine all 2^k models. This procedure is called all subsets or exhaustive regression. Some of the first applications of exhaustive regression involved the study of plant data.

Data dredging

Data dredging or data fishing are terms one may use to criticize someone's data mining efforts when it is felt the patterns or causal relationships discovered are unfounded. In this case the pattern suffers of overfitting on the training data.

Data dredging is the scanning of the data for any relationships, and then when one is found coming up with an interesting explanation. The conclusions may be suspect because data sets with large numbers of variables have by chance some "interesting" relationships. Fred Schwed said:

"There have always been a considerable number of people who busy themselves examining the last thousand numbers which have appeared on a roulette wheel, in search of some repeating pattern. Sadly enough, they have usually found it."

Nevertheless, determining correlations in investment analysis has proven to be very profitable for statistical arbitrage operations (such as pairs trading strategies), and correlation analysis has shown to be very useful in risk management. Indeed, finding correlations in the financial markets, when done properly, is not the same as finding false patterns in roulette wheels.

Some exploratory data work is always required in any applied statistical analysis to get a feel for the data, so sometimes the line between good statistical practice and data dredging is less than clear.

Most data mining efforts are focused on developing highly detailed models of some large data set. Other researchers have described an alternate method that involves finding the minimal differences between elements in a data set, with the goal of developing simpler models that represent relevant data.

When data sets contain a big set of variables, the level of statistical significance should be proportional to the patterns that were tested. For example, if we test 100 random patterns, it is expected that one of them will be "interesting" with a statistical significance at the 0.01 level.

Cross validation is a common approach to evaluating the fitness of a model generated via data mining, where the data are divided into a training subset and a test subset to respectively build and then test the model. Common cross validation techniques include the holdout method, k-fold cross validation, and the leave-one-out method.

Privacy concerns

There are also privacy concerns associated with data mining - specifically regarding the source of the data analyzed.

Data mining government or commercial data sets for national security or law enforcement purposes has also raised privacy concerns.

There are many legitimate uses of data mining. For example, a database of prescription drugs taken by a group of people could be used to find combinations of drugs exhibiting harmful interactions. Since any particular combination may occur in only 1 out of 1000 people, a great deal of data would need to be examined to discover such an interaction. A project involving pharmacies could reduce the number of drug reactions and potentially save lives. Unfortunately, there is also a huge potential for abuse of such a database.

Essentially, data mining gives information that would not be available otherwise. It must be properly interpreted to be useful. When the data collected involves individual people, there are many questions concerning privacy, legality, and ethics.

Combinatorial game data mining

• Data mining from combinatorial game oracles:

Since the early 1960s, with the availability of oracles for certain combinatorial games, also called tablebases (e.g. for 3x3-chess) with any beginning configuration, small-board dots-and-boxes, small-board-hex, and certain endgames in chess, dots-and-boxes, and hex; a new area for data mining has been opened up. This is the extraction of human-usable strategies from these oracles. Current pattern recognition approaches do not seem to fully have the required high level of abstraction in order to be applied successfully. Instead, extensive experimentation with the tablebases combined with an intensive study of tablebase-answers to well designed problems and with knowledge of prior art i.e. pre-tablebase knowledge is used to yield insightful patterns. Berlekamp in dots-and-boxes etc. and John Nunn in chess endgames are notable examples of researchers doing this work, though they were not and are not involved in tablebase generation.

Notable uses of data mining

• Data mining has been cited as the method by which the U.S. Army unit Able Danger supposedly had identified the September 11, 2001 attacks leader, Mohamed Atta, and three other 9/11 hijackers as possible members of an al Qaeda cell operating in the U.S. more than a year before the attack.
  • See also: Able Danger, wikinews:U.S. Army intelligence had detected 9/11 terrorists year before, says officer.
  • It has been suggested that both the Central Intelligence Agency and their Canadian counterparts, Canadian Security Intelligence Service, have put this method of interpreting data to work for them as well, although they have not said how.

Example

An example of data mining, often called the Market Basket Analysis, relates to its use in retail sales. If a clothing store records the purchases of customers, a data mining system could identify those customers who favour silk shirts over cotton ones. Although some explanations of relationships may be difficult, taking advantage of it is easier. The example deals with association rules within transaction-based data. Not all data are transaction based and logical or inexact rules may also be present within a database. In a manufacturing application, an inexact rule may state that 73% of products which have a specific defect or problem, will develop a secondary problem within the next 6 months.

See also

• Artificial intelligence
• Bayesian network
• CRISP-DM
• Data analysis
• Data farming
• Data fusion
• Descriptive statistics
• Fuzzy logic
• Hypothesis testing
• k-nearest neighbor algorithm
• Machine learning
• Pattern recognition
• Predictive analytics
• Preprocessing
• Statistics

Structured Data Mining

• Concept mining
• Database mining
  • Relational data mining
  • Database
  • Document warehouse
  • Data warehouse
• Graph mining
  • Molecule mining
• Sequence mining
  • Data stream mining
  • Learning from time-varying data streams under concept drift
• Tree mining
  • Decision tree learning
• Web mining
• Software mining
• Text mining

Induction algorithms

Supervised learning

• Artificial neural network
• Boosting
• Decision tree learning
• Linear discriminant analysis
• Logit (in reference to logistic regression)
• Naive Bayes
• Nearest neighbor (pattern recognition)
• Neural network
• Quadratic classifier
• Random forest
• Support Vector Machine

Unsupervised learning

• Apriori algorithm
• Data clustering
• Self-organizing map (SOM)

Dimensionality reduction

• Feature selection
  • Information gain
• Feature extraction
  • Principal components analysis (PCA)

Application areas

• Business intelligence
• Business performance management
• Discovery science
• Loyalty card
• Cheminformatics
  • Quantitative structure-activity relationship
• Bioinformatics
• Intelligence services

Software

• CART
• DataTime Universal
• Funnelback
• GeneXproTools
• InforSense
• Java Data Mining (JSR-73, JSR-247)]
• MATLAB
• Microsoft Analysis Services
• MicroStrategy
• Neural network software
• Oracle Data Mining
• Orange Micro
• Prospero
• R
• RapidMiner (formerly YALE)
• ROOT
• SAS
• Splunk
• SPSS
• STATA
• STATISTICA
• TANAGRA (software)
• Teradata
• Weka
• YALE

Sources

• Kurt Thearling, An Introduction to Data Mining (also available is a corresponding online tutorial)
• Dean Abbott, I. Philip Matkovsky, and John Elder IV, Ph.D. An Evaluation of High-end Data Mining Tools for Fraud Detection published a comparative analysis of major high-end data mining software tools that was presented at the 1998 IEEE International Conference on Systems, Man, and Cybernetics, San Diego, CA, October 12-14, 1998.
• Mierswa, Ingo and Wurst, Michael and Klinkenberg, Ralf and Scholz, Martin and Euler, Timm: YALE: Rapid Prototyping for Complex Data Mining Tasks, in Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-06), 2006.
• Peng, Y., Kou, G., Shi, Y. and Chen, Z. "A Systemic Framework for the Field of Data Mining and Knowledge Discovery", in Proceeding of workshops on The Sixth IEEE International Conference on Data Mining (ICDM), 2006
• Hari Mailvaganam and Daniel Chen, Articles on Data Mining

Books

• Peter Cabena, Pablo Hadjnian, Rolf Stadler, Jaap Verhees, Allesandro Zanasi, Discovering Data Mining: From Concept to Implementation (1997), Prentice Hall, ISBN 0137439806
• Ronen Feldman and James Sanger, The Text Mining Handbook, Cambridge University Press, ISBN 9780521836579
• Pang-Ning Tan, Michael Steinbach and Vipin Kumar, Introduction to Data Mining (2005), ISBN 0-321-32136-7 (companion book site)
• Galit Shmueli, Nitin R. Patel and Peter C. Bruce , Data Mining for Business Intelligence (2006), ISBN 0-470-08485-5 (companion book site)
• Richard O. Duda, Peter E. Hart, David G. Stork, Pattern Classification, Wiley Interscience, ISBN 0-471-05669-3, (see also Powerpoint slides)
• Phiroz Bhagat, Pattern Recognition in Industry, Elsevier, ISBN 0-08-044538-1
• Ian Witten and Eibe Frank, Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations (2000), ISBN 1-55860-552-5, (see also Free Weka software)
• Mark F. Hornick, Erik Marcade, Sunil Venkayala: "Java Data Mining: Strategy, Standard, And Practice: A Practical Guide for Architecture, Design, And Implementation" (Broché)
• Weiss and Indurkhya, Predictive Data Mining, Morgan Kaufman
• Yike Guo and Robert Grossman, editors: High Performance Data Mining: Scaling Algorithms, Applications and Systems, Kluwer Academic Publishers, 1999
• Trevor Hastie, Robert Tibshirani and Jerome Friedman (2001). The Elements of Statistical Learning, Springer. ISBN 0387952845 (companion book site)

External links

• Template:Dmoz
• SIGKDD (KDD.org), ACM Special Interest Group on Knowledge Discovery and Data Mining, USA
• KDnuggets, Data Mining and Knowledge Discovery site
• Data Mining Meta Directory
• Data Mining, University of Houston - Clear Lake, USA
• Data Mining Program, University of Central Florida, USA
• Data Mining, MIT OCW, USA
• Data Mining Group, Artificial Intelligence Unit, University of Dortmund, Germany
• The Institute of Analytics Professionals of Australia Limited
• Data Mining and Knowledge Discovery Search Engine, a google co-op search engine for Data Mining
• An overview of commercial, free and open source data mining tools