Data Integration Tools
Tags: Extract, transform and load (ETL), Enterprise data integration (EDI), Service-oriented architecture (SOA) data services, Data integration tools, VIEW ALL TAGS
These Apps Can Help You Know Your Customers
CRM (customer relationship management) is tricky business. First there are the CRM tools themselves: big, bulky systems from mega-vendors such as Siebel, Oracle, and SAP, or lesser players such as Vantive and Clarify. Installing them can be a major project; teaching users to use them can be even bigger.
As if that weren't enough, most firms store their data in different silos across the enterprise, each with its own format and subset of data. The result? Customer data—your lifeblood—is often fractured, flawed, and messy.
What's more, companies can't build unified views of data across their many systems, meaning they can't get a grasp on who their customers are, what they buy, when they buy it, how they buy it, and most crucially, why they buy it.
A Triad Of Tools
Of course, there are tools to help you cure your data disease. In fact, there are many, but the top three are ETL, EII, and DQ.
ETL. Short for extract, transform, and load, ETL tools are used to grab data from one database and merge it into another. The first step, extraction, reads data from the first database and stores it in memory. Next comes the transformation phase: The ETL tools convert the data into a form that matches the new database. Last is loading, in which the tool writes the data to the target database.
Of course, a tool that can do this has a broader range than simple data migration. ETL can be used to form data marts (a database for a single department or division) and even the odd data warehouse (a larger system that's used across the enterprise and often pools the data from several database systems at once).
EII. Next on the list of tools is EII, short for enterprise information integration. As the name implies, these tools are used to merge data from different systems across the enterprise and show the data in a single, unified (also called "federated") view, as though it came from a single source. Unlike its cousin ETL, EII does not write to a new database or change any data source. Instead, it holds the data in a cache that lets you read—but not write—different records.
DQ. Last is DQ, or data quality tools. These are the simplest of the three. They're used to "scrub" or "cleanse" dirty data (data that's missing fields or contains other errors, such as spelling mistakes).
DQ is utterly needed to keep a clean database that you can rely on (after all, it's awkward to send a form letter to a customer and spell his name wrong). But it's merely a first, and partial, step in cleaning your data. For quality business use, cleansed data needs to be carefully vetted by human eyes, a costly step that not every enterprise can afford.
The Goal
Here's yet another acronym for you to master: CDI. Short for Customer Data Integration, it refers to the process of building a customer hub: a single, central place to store and view data from multiple (and often disparate) systems across the enterprise. A customer hub is one of the seminal concepts of data integration, as it can be used as an end product in itself (a tool for users such as managers and business strategists to view and vet data) or a source for downstream applications (a "feeder" that sends data to other tools, such as analytics and finance software).
You can buy a soup-to-nuts CDI solution from several vendors, the most famous of which is Siperian (www.siperian.com), whose Master Reference Manager and Activity Manager let you troll internal and external silos for customer data and integrate that data into a single, sensible hub.
Or you can build one of your own using the tools mentioned above. After all, ETL, EII, and DQ are meant to gather and clean data. But do they work for such a daunting task as CDI? Remember, the average enterprise can have upward of 20 data silos in just as many departments.
The answer is yes: ETL, EII, and DQ tools can help you with CDI, but they're largely inefficient and ill-suited to the task. First, these tools support only one data modality (or method of moving data from place to place): batch or real-time. Yet most CDI apps need both because there are times when you'll need to update the hub with a high-volume batch process that loads data into it from a new silo, as well as times when you'll need to retrieve single records in real-time. And second, none of the ETL, EII, and DQ tools uses a metadata framework. Metadata, or data about the data in your database, lets data stewards and others quickly configure and extend your data systems without having to write reams of code.
Anurag Wadehra, vice president of marketing for Siperian, notes that using ETL, EII, and DQ for customer data integration is like "having a plumber build your house. Like plumbing in a house, the tools that push data through the pipes don't represent the overarching blueprint needed for CDI architecture."
It's a point well taken, and one you'll need to ponder as you begin to integrate your customer data. Proceed with care: It's one of the hardest tasks that IT can face.
by David Garrett
Data Lifecycle Management
Data, like everything else in this world, has a lifecycle, from the time it's first written to a database to the time it grows obsolete and gets deleted from the system.
Short for data lifecycle management, DLM tools let you manage data from youth to old age, storing newer, more vital data on faster servers or systems, and moving old, rarely used data to slower, cheaper storage, such as tape.
CDI (customer data integration) and DLM systems need to be linked in the enterprise, since the former unifies data from different silos into a customer hub, and the latter decides how and where to store the hub's information. When planning your CDI architecture, be sure to use experts who can take DLM into account.
CDI Vendors
CDI (customer data integration) is the process of building a single, central place to store and view data from many systems across the enterprise. Its major vendors include:
Acxiom | www.acxiom.com ; (501) 342-1000
Ascential | www.ascential.com ; (508) 366-3888
Journee | www.journee.com ; (512) 634-5111
Siebel | www.siebel.com ; (650) 295-5000
Siperian | www.siperian.com ; (650) 350-2200
