We recently exhibited Ratchet-X at Microsoft’s TechEd 2009 show in Los Angeles California. Thank you to all who visited our booth and made the show such a big success for us. While there was no shortage of ideas as to how Ratchet-X could be used within the organizations of those who took the time to receive a demo, there was one repeated use case that stood out. Let’s just say that despite all efforts the industry has made to create better integrated systems, ETL (Extract, Transform and Load), is alive and well. We heard a number of stories about how organizations still continue to struggle with creating and maintaining reliable ETL processes.
No doubt, modern application development and deployment technologies coupled with “smart” infrastructure are enabling organizations to integrate more-and-more systems in real time. However, despite the advances, ETL is still a reality of life in organizations of all sizes. So the question asked often asked of us at the show was; “Where does Ratchet-X fit within the ETL process?” Glad you asked.
Let me first say that while Ratchet-X does extract, transform and load data between systems, it is not a traditional ETL tool. When I think about ETL, I think of scheduled batch jobs that process and move data between queues and systems. Generally, ETL processes are executed during non-prime time usage or maintenance hours for the purpose synching data across systems. While the term ETL can be used to describe other related processes, I think this is what most folks mean when they refer to ETL.
So, if Ratchet-X extracts, transforms and loads data, why isn’t it an ETL tool? It’s mostly a matter of perspective. Whereas most ETL processes are scheduled to run in batch, work with large data sets and have little end-user interaction, Ratchet-X is an on demand data processing platform that processes a single record set at a time and is often guided by the end-user.
Most of our customers use Ratchet-X to add new features to existing applications without changing those applications in any way. These new features usually involve:
- Acquiring data from an external source (e.g. web services, web site, electronic form, database, API, etc).
- Sending application screen data to an external source.
Processing application screen data for the purposes of either transforming it in some way or kicking off a down stream process.
In most cases, Ratchet-X is processing one or more application screen’s worth of data that collectively constitutes a single record set (i.e. a composite customer record, an insurance policy, a shipping manifest, etc.). Ratchet-X is not usually used to process thousands of these record sets at a time. Further, Ratchet-X is almost always used under the watchful eye of the end-user. In reality, Ratchet-X is an “assistive” technology that allows an end-user to acquire, push or process data to and from the systems they use without having to rekey data and reducing errors. This doesn’t sound like ETL to me.
So, uh, where is it that Ratchet-X fits into the ETL process? Oh yeah. No ETL process is perfect. In fact most ETL processes have an error handling stage where exceptions and records that fail validation are kicked out to an exceptions queue. Once in the queue, the user then must manually review and rekey the corrected information into the target system or error handling interface. This is precisely where Ratchet-X can be very useful. For example, we have customers that set up Ratchet-X’s task list as the exception handling queue so users can easily review the data, make whatever corrections are necessary and immediately paste the corrected record into the target application.
So while Ratchet-X is not an ETL tool per se, it can serve as an important part of the error handling stage of most ETL implementations. If you have any questions or comments regarding how Ratchet-X can be used to supplement your ETL process, please drop us a line. We’d love to hear from you. If you’d like to see a video regarding how Ratchet-X can be used as part of your ETL efforts, click here.