Data Virtualization Insights
Welcome to the World of Data Virtualization Insights
I'd like to begin with some thoughts on the universe of data virtualization. We need to understand the road the pioneers assembled with basic data virtualization from the early framework of enterprise information integration (EII).
We need to understand the incremental and compelling advantages of advanced data virtualization and the new application areas for which this technology can provide compelling advantage and return on investment.
Lets review what we know, identify the issues, and highlight useful information that can help all of us be more successful.
Michael Zuckerman, Chief Marketing Officer
Legacy Vendor "Triple Play"
Written by Michael Zuckerman Friday, October 07, 2011 08:25 AM
Instant Video - Excess Expense in Data Integration Projects
I recently listened to a customer story as they shared their view of the benefits of our Advanced Data Virtualization for data integration, business intelligence and data warehouse deployment.
During this conversation I gained an even better sense of how much they are forced to spend with large legacy enterprise vendors. One of the fundamental root causes of the excess expense is clearly caused by the lack of a Unified Platform - every component they have is sold to them separately and by a different division in their vendor's company. The second root cause of excess expense is the perpetual professional services harvest ("every day's a holiday" for these vendors) wrapped around an installed base using a product based upon legacy extract, transform and load (ETL) technology. And the third root cause, if they have the features to play the game, is the scope creep, managed with style and time tested legacy vendor process, around every data integration engagement. The customer called this the "Triple Play" and related the story to us.
Lets look at a typical scenario, as I learned:
Your legacy vendor finds another need for data integration and because of the legacy spiderweb of ETL in your enterprise, is likely the first vendor to the show. You are already spending many thousands of dollars with this vendor annually on both maintenance and professional services. They sit in your meetings, have an office in your facility and you know them by first name. The CIO thinks they are part of your institution. But they don't have one integration platform. Not even two. They have, at my last count, perhaps seven. And all incompatible and different.
The CIO is often misdirected by the relationship and installed base. It is the same way old IBM-MVS(r) systems used to run your business. It is what we know. Data integration and ongoing operations usually fall into the baseline expense portion of your CIO's budget. This part of your CIO's budget, the baseline operating expense, should be constantly pruned and trimmed. The portion of the budget that really counts is how much your CIO has for basic business transformation. This is where potentially great rewards lie - but instead of focusing on ways to achieve this these items get missed again.
So the legacy vendor delivers a professional services proposal, albeit a bit pricey, to get the integration done. Looks like the way to go, doesn't it? We have them in 27 other integrations ... will one more hurt? Meetings (and hourly billing) begin in earnest. The first business flow question of substance arises. How will we relate these business objects? What is the key (or candidate keys) that we will use to take data from several sources and match it together in our new project? It is the same issue in data integration, business intelligence, data warehousing and (virtual) MDM. The same. You need to align these four sources but you need one good key to do it.
Now we need to refer you to our other group and they will give you a 2nd proposal for Data Profiling. They will ascertain the state of your data, your data quality, and determine potential key candidate, and deliver this in a beautiful report, all for yet another $45,000.
Well, now, of course, we have started our Data Integration engagement. We are slowing down, due to no fault of our own, with this particular business unit, because they don't know enough about their data, but even worse, there may be problems with their data. But wait, the results of the Data Profiling engagement are in! Good news! We understand the problems with your data - look at the various data quality issues.
Some can be fixed with tools and will map nicely to a 3rd proposal for a Data Quality engagement. We need a bunch of meetings with various data stewards (again...). This will be another $65,000 we think and this will get the basic Data Integration engagement on the road.
This is a well orchestrated "Triple Play" for the legacy vendor. What could be better - 3 engagements for more than three times than the price of one! The 1st for data integration, the 2nd for data profiling and the 3rd for data quality. They do this, even in knowledgeable accounts, over and over, because this the "accepted approach" for this entrenched legacy vendor. And so ends this typical scenario. The business user pays the bill and the information technology team pays the opportunity cost.
With Advanced Data Virtualization, we have all of the tools in one system, accessible without programming or SQL, to the business user to do everything. So if you want to align and integrate perhaps four sources, which is tremendously hard, time-consuming and difficult using legacy ETL, we provide the tools to immediately ascertain the state of data quality, profile your data, build out candidate keys, implement business rules - and more.
Advanced Data Virtualization may cut your professional services expense by 75% or more. We'll reduce risk, speed deployment and provide an open, auditable platform for your organization. We'll scale better to meet the needs of the largest organization in the world and save you money while you do it. Such is the benefit with new technology - break the bonds of legacy data integration and find out how much benefit you can derive from Advanced Data Virtualization.
Customer Polarization
Written by Michael Zuckerman Monday, September 19, 2011 12:00 AM
I do believe that the road for ETL is coming to an end. Perhaps a very graceful one. Perhaps not.
Today I heard about a project at a $1 billion+ company called "Kill ETL." I kid you not. I could not make this up if I tried. Formal project name.
Evidently the customer is sufficiently motivated over the excess professional services fees and cost upgrades being charged by their dominant, legacy ETL vendor that they have an internal project to find a way to displace and replace their existing nest of ETL. ETL teams are generally our friends and sponsors in large enterprise. They bring us into accounts to support data warehousing, business intelligence and a variety of integration challenges across the board. These same people are leading the charge at this $1 billion+ company to "Kill ETL."
When the wave turns, and the tide moves out, things can happen quite fast. Two years ago the blackberry was king and now RIM is on the verge of being irrelevant.
Small and medium business has no problem with ETL - they are using new world integration technologies most of the time. But a $1 billion+ company with a project like this is the mainstream market - albeit a bit close to the edge.
I'll watch this project and keep you updated. Perhaps our 15 minutes of fame have begun in earnest.
Plate-Forme Unifiée
Written by Michael Zuckerman Wednesday, September 07, 2011 03:42 AM
A Unified Platform (or a Plate-Forme Unifiée as our potential new VAR in France has reminded me) is a tremendously important concept for many of our customers, prospects and channel partners. It is part of the new world that defines the sea change moving across the data integration landscape being driven faster (and faster) by the new cloud applications. It is building, quietly but surely, and will reach broadscale importance over the next year or two.
The cognoscenti know already that the back of data integration has been broken for many years. Moderately innovative product technology has been shuffled aside in favor of a professional services model which empowers large companies to consume (and subsume) innovative software companies that can enable them to drive the hourly rate deliverables they enjoy. The two largest vendors in the global enterprise space are in the business of selling professional services. Data integration software is an afterthought. The need to deliver greater return on investment to their customers is obviated by the P&L in charge through the available and perpetual harvest of professional services. Even laptop companies can see the benefit to their services business. ;-)
These large companies are not building the landscape in the SMB (small-medium-business) companies. There is a large variety of new software cloud applications pouring in from the cloud-scape world and it continues to grow. This is the same small wave that Salesforce(r) rode into a Tsunami in the customer relationship management space. It manifests a new wave the Netsuite will surely ride which will ultimately subsume some very large ERP vendors. You can see it in every space and the legacy vendors are fighting it hard.
Data integration, data quality and master data management are all experiencing the same challenge. Not only is the industry framework of extract, transform and load (ETL) poorly designed to support the new cloud world, but the growing number of connections desired by customers, especially in the SMB space makes it almost impossible to use these legacy technologies. You face both cost barriers and architectural barriers. It is a lose-lose situation for most customers. The early vendors to this ultimately massive market have done no more than placed 20 year old ETL technology in a 1 year old cloud. When you get to 3 or 4 applications the story wears thin. You cannot lay pipe from "A to B", "B to C" and back from "C to A." The machine won't work and the model breaks down.
On top of this the best cloud architectures, if they exist, are 100% isolated from the legacy, on-premise architectures. They don't work together. This is most obvious to the largest global enterprises. They fought cloud early on for security and compliance reasons but now see it coming in through their partner, B2B, sales automation (that division in India that went with Salesforce ... remember them? now the one in Des Moines has done the same ...) and more. Now that they must integrate cloud and add order to the chaos they find that the lack of a Unified Architecture or Single Platform makes it difficult to impossible. How will our MDM or VMDM applications work with these new cloud applications? How will we load the data warehouse from these new cloud applications? The list of poorly answered questions is large and growing.
The legacy vendors are no help. They offer words and distant roadmaps. CTO's with nice presentations about what they will do "someday." Their cloud architectures are almost totally different from the legacy, on-premise architectures. "Here is our cloud architecture - but pay no attention to our on-premise products. Here is our on-premise product line - you use it - forget about our cloud architecture." Let the facts speak for themselves - these architectures are totally incompatible and they know it. "In 2 (or 3 or 4) years we'll have all of this integrated and unified." Yes, someday.
Today, right now, at this moment we offer our customers a Unified Platform. 100% integrity between what you can do in our QueCloud and what you can do On-Premise with our Virtual Data Manager. Every interface, every user interface, every facet of deployment is 100% identical and compatible. And you can mix them together. One architecture that a business user can implement economically, quickly and with low risk in a small business that can scale to an SMB through the largest global enterprise in the world. One architecture that will outperform, outscale and outmanage all of your data integration, data management, data quality and master data management for any mix of cloud and on-premise deployment. One architecture empowered by advanced data virtualization which allows us to do things that legacy architectures assembled on ETL frameworks will never do.
Five Best Practices in Data Integration
Written by Michael Zuckerman Thursday, August 04, 2011 09:07 AM
SEE THE VIDEO: The Five Best Practices in Data Integration
The explosion in information technology tools and capability has accelerated. Our information technology environment is moving at absolute light-speed. Yet the back of data integration is broken. 90% of the customers are using 20 year old ETL technology designed to move batch files into the data warehouse at 1 am. It is a manual process. Worst of all, is that some vendors have even put this 20 year old technology in their newly minted 1 year old cloud. Who wants 20 year old technology in a 1 year old cloud?
To add insult to injury, data quality is … “another product.” I humbly submit that without resolving data quality issues and providing for their ongoing maintenance you cannot integrate even two applications, load your data warehouse, consider a future Virtual MDM deployment.
The cause of this is simple. Look at the revenue mix for the two major vendors that control the market for large enterprise. Their business is about professional services – not software tools. They are in business to sell you, their customer, as much professional services at the highest rates you can imagine. Legacy ETL technology is a bonanza to a company that sells your company services by the hour.
Even worse, they don’t offer you the right mix of services and technology. The integration was done to specification but it doesn’t seem to work. I guess we’ll just have to sell you additional consulting to do the data quality clean-up.
The five best practices in data integration break this model. We turn the model upside down. By using advanced data virtualization, and pushing back at the legacy monopoly attached to your wallet, you can reduce your costs and speed your time to implementation.
BEST PRACTICE #1: Use an Automated Metadata Discovery Tool and Data Dictionary. IEEE did a seminal study recently in the area of data integration to understand how the time was allocated. Over 40%+ of the time in any data integration project is spent on discovery. What is the structure of the data? Which field corresponds to others so that we can identify the source record of truth and harmonize it over to the target? Advanced data virtualization can completely automate the discovery of data structures, provides insight into how things relate (semantics) and then automates the integration between the systems. All of this is automated and easy to deploy. You will be able to reduce your time, in this project phase alone, by 75% or more because you use a metadata discovery tool.
BEST PRACTICE #2: Data Quality Must Be Integrated At Every Step in Your Project. Data quality is an essential project component for a small, medium or large company. Not a separate project. Advanced data virtualization technology provides full data quality tool sets, fully automated, with the product. It is the starting point to any data integration, the core to best practices implementation and the ongoing steady hand that keep your data harmonization working into the future. Without good data quality, each data harmonization will reject hundreds to thousands of records that “don’t synchronize.” And this creates tons of additional work.
BEST PRACTICE #3: Understand Your Needs Versus Vendor Architecture. “Tomorrow never comes” is not the right way to deploy critical applications that serve your business. Advanced data virtualization brings a hub and spoke architecture that easily extends and adds the next application to your integration architecture. This is in sharp contrast to the pipe (or pipe and hub for mdm) architectures you must assemble with ETL.
BEST PRACTICE #4: Bring in Tools Suitable For Business Users – No Programming Required. Advanced data virtualization enables your business user to point, click and select to build out integrations. Data transformations between systems come together using a formula builder very similar to Microsoft® Excel® – highly intuitive and obvious. This works the same with relational data, objects, complex API’s with products like SAP®, XML and even NoSQL data sources. One simple object framework designed to work with everything – fast, easy and automated.
BEST PRACTICE #5: Data Enrichment is Easy – Automate It For Business Benefit. Today products like D&B360® and others offer clean data sources for many elements of your important customer data. Yet most of our customers are still manually correcting data at the time of data entry in the ERP system, and then hoping that the support and SFA systems catch up. There is a better way – you can harmonize data directly with a source of data enrichment using business rules. This improves business process and provides strong ROI.
These are some excellent white papers on our technology - check them out - instant download - no registration required:
6.23.2011 - IdealNet - Analysis of Data Integration Technologies
12.20.1010 - White Paper Thinking of Master Data Management? Don't Do It! Find Out Why?
11.22.2010 - White Paper Advanced Data Virtualization - Function, Capability, Benefits
10.2010 - White Paper 2010 Report - Application and Data Integration for Cloud and On-Premise Applications - Tools and Technologies
10.20.2010 - White Paper Overview of Advanced Data Virtualization - Differentiation Verses Basic Data Virtualization
Analysis of Data Integration Technologies Report
Written by Michael Zuckerman Wednesday, June 29, 2011 09:29 AM
You can get immediate access to a new report on data integration solutions and technology. The report, An Analysis of Data Integration Technologies, summarizes the findings of an evaluation comparing ETL and open source solutions, and offers recommendations as to which technologies would be most appropriate based on an organization's data integration requirements. This report shows Informatica® as the lead vendor in the ETL category and reviews many other options for technology and vendor data integration solutions, both on-premise and in the cloud. ETL is legacy technology so we're happy to concede that small victory to Informatica®.
Queplix, however, is firmly positioned as the lead innovative solution and frames a strong position against Informatica® as you go beyond the batch file oriented transfer so typical of legacy ETL. We're pleased and proud with the results.
Areas the report covers include:
• Business Applications
• Platform Deployment (on-premise versus cloud)
• Connectivity to Data Sources
• Synchronization (Data Harmonization)
• Transformation
• Data Movement
• Test, Development and Operations Environments
• Data Modeling
• Data Quality and Data Governance
• Architecture and Standards
• Conclusions: Leading ETL Vendor, Open Source ETL Vendor, Alternate Innovative Solutions
Here is the link to the report: http://www.idealnetinc.com/IdealNet_Analysis_of_Data_Integration_Technologies.pdf
More Articles...
Page 1 of 2
