‘Wirtschaftsinformatik’ did a little digging through its archives and reprinted our 1993 article on data mining. We feel honored and sum up what we have learned over the past 16 years.

Wirtschaftsinformatik magazine is celebrating its 50th anniversary with a commemorative issue containing a selection of its best articles. The ‘Data Mining’ article written by Dr. Jürgen Hagedorn and me truly unexpectedly made it into the final cut. The essay was written in the days when we both worked as assistants at the chair of Prof. Peter Mertens. Today, Jürgen works for SAP.

The hype around data mining is long gone. The discipline, however, continues to mature, free from all false expectations. The headlines focus on other subjects. So what’s the story on data mining? What has happened since then? The experience that we have gathered across all types of commercial sectors and industries over the past 16 years can be summarized as follows:

1. Easter egg paradox The possibilities and limitations of data mining endure. Yet, the nightmarish degree of automation that people once romanticized is still a long way off. Back then, we called it an ‘Easter egg paradox’ – people have to hide the information that they hope to find through clever data modeling and method selections. In other words, it still makes sense that companies are managed by people.
2. Machine-based process control After a certain point in the data mining process, human intervention is not desirable: Only very few users have the discipline to continuously adhere to the advice of a data mining system. If you let an algorithm such as DeltaMaster’s Selector choose the addresses for a direct marketing campaign, you should use the suggested data if you later want to measure how effective it actually was. Succeeding here leads to verifiable results.
3. Automation of routines You can always achieve fast results, if you consequently delegate repetitive analysis tasks to a computer and define calculation rules that reflect human decision-making and problem solving practices. In that case, a computer can do the same job as a human being would – assuming, of course, that that person had enough time and that his or her time wasn’t expensive. Some samples include searching for the underlying causes of variances in operating results or segmenting markets based on social demographics.
4. Structural poverty This assumption has proven itself over the years. What is even worse than poor data quality, however, is the general lack of structure in data. Nevertheless, most companies generally focus on data cleansing. That alone, however, won’t help them reach their goals. Data mining can cope when a few customers are listed under the wrong industry. But you can’t learn anything about industry patterns when you don’t categorize your data by industry in the first place.
5. Data shortage Most companies complain that they are flooded with data. When it comes to data analysis, however, they are actually experiencing a major drought. Huge investments in operational IT systems are no guarantee for suitable analytical data. If the employees processing orders always type 9999 instead of properly worming the secrets out of their customers, you will soon have large volumes of worthless data. This fact disheartens many before they even get started. Too bad, because the solution is obvious and inexpensive as well. You simply start with an adequate sample and collect the necessary data.
6. Management suitability We have discovered that managers achieve the highest levels of progress and satisfaction when the analysis methods are designed for their needs. Today, there are many different types of software with association features for shopping cart analysis. As far as I know, however, the only one that reduces the typically large number of rules to a manageable, understandable number is ours.

Although it’s not always possible to discover a gold mine, it is easy to avoid a mine disaster. In addition, you will always uncover a few valuable nuggets along the way.