Scalability

A single process can handle every LifeMasters' customer, no matter how many new customer formats are added as their business grows.

Extensibility

LifeMasters appreciates the many interfaces and APIs Pervasive® Data Integrator™ offers, allowing LifeMasters to easily implement a new function or a specific path through either APIs or coding, and to leverage existing solutions that work well.

Highly accessible support and services

LifeMasters enjoys a great relationship with the Pervasive support and services teams. Steve Hromyko points out that "The staff is knowledgeable and proactive when working with us to resolve issues, and they listen to us as a company. They are genuinely interested in how we use their software for our business."

Printable Format

Manual load procedures can't scale for growing data volumes

LifeMasters Supported SelfCare, Inc. is a leading provider of health promotion and disease management programs and services that create health partnerships among individuals, their physicians and payors. By helping people with long-term illnesses like diabetes or asthma maintain a healthier lifestyle, LifeMasters saves insurance companies money while helping people live more fulfilling lives. Business growth is rapidly increasing, and with it comes lots of data–so much, in fact, that data manageability was growing challenging. LifeMasters wanted to automate ETL data loading processes, and turned to Pervasive® Data Integrator™ to do the job.

Automating a manual multi-step data loading process

LifeMasters receives and processes large amounts of data from a broad spectrum of sources. With growth in business came the realization that data volumes would ultimately outpace their manageability. Assessing and loading incoming customer data required a significant amount of scarce and expensive time from data analysts, so the only way to accommodate a growing number of customers and grow the business was to hire more people.

LifeMasters wanted to address the challenge by replacing the manual, multi-step process with a smooth, automated procedure with no error-prone human intervention. LifeMasters' ETL Manager Steve Hromyko said, "We needed a set of tools that could not only digest and transform a variety of file types (ASCII, COBOL, EDI), but that also provided added functionality to enable full lights-out automation of our loading process."

The existing manual data loading process began with incoming data files from insurance companies in various X12 healthcare formats like HIPAA, HL7 and NCPDP as well as non-standard flat and delimited ASCII and COBOL formats. The files were dropped into an FTP server, which data analysts would manually query periodically to check for files. The analysts would download, decompress, decrypt and manually process the file, and then run it through 10 to 20 different stored procedures to load data through LifeMasters' system into a number of tables. Once loaded, the tables were accessed by the company's home-grown data-driven Web-based application.

Data load times reduced from 5 - 10 days down to 2 hours

With Pervasive Data Integrator, LifeMasters now has the ability to automatically transform ASCII, COBOL and EDI file types and proceed with automated data loading. The old process typically took about five days for larger existing customers and even longer to tackle new customers’ formats. With Pervasive integration, data loads can now be accomplished in a couple of hours. According to Hromyko, "The process of loading the data is down to two hours. We now pull files from a secure FTP server, decrypt, stage, transform, and load them to our SQL databases without the need for human manipulation. This significantly reduces the time required to process data, and the data itself is inherently more secure. All this compared to a process that was taking 7 to 10 days to load custom data files."

Hromyko adds, "As we continue to enhance automation, the analyst’s time will approach zero. All the analysts’ time spent loading data is going to go away, and they can consequently use their time to do what they were hired to do–analyze data and turn it into useful information."

One master process + one map per file format = fast, scalable data load 

Two of Hromyko’s favorite Pervasive Data Integrator tools are Process Designer and Map Designer. Process Designer provides components to access existing technologies (SQL procedures, DTS routines, and custom applications) so LifeMasters can leverage the skill set of employees and reuse what is already built and works well. Map Designer provides considerable built-in functionality to implement transformations quickly, without the need for development resources. LifeMasters handles more than 120 different file formats, many of which require unique transformation logic. Using Process Designer, LifeMasters built a single master process into which any file from any source can be fed. The process queries the FTP server to see if a file has arrived, downloads the file, and then parses, decrypts and unzips it, writing associated information such as where the file is located and what type of file it is to tables.

The approach is straightforward, yet very flexible–it is designed to process files, not customers. As a file comes in, it’s married programmatically to its associated transformation and loaded. This relationship, along with additional information needed to manipulate the file, is stored in meta tables and gathered during runtime using RIFL.

LifeMasters metadata driven dynamic master process design

Map Designer power saves time and money

Pervasive Data Integrator allows LifeMasters to have effective, economical automated ETL. As ETL Manager Steve Hromyko confides, "Honestly, even if we took all the stored procedures and just hooked them together with statements and put them in a chain of processes…that alone would justify the cost of Pervasive’s software. Keeping the analysts from sitting there and monitoring these stored procedures is worth a lot of money. I really value the automation and the full cycle of support that Pervasive provides."

Steve Hromyko also emphasizes, "To keep the process simple, when processing a file, no matter how complex the layout, we use a single map step to transform and load the data into a standard set of tables. Even with the challenges of looping file structures, the powerful Map Designer makes this possible. We haven’t found any data file that can’t be transformed with a single map."

(This process design is often used in clearinghouses and other centers of data processing. See Integration Hub for a solution specifically designed to address the challenge of multiple data feeds.)

Pervasive Data Profiler saves LifeMasters from "Code, Load, and Explode"

LifeMasters uses Pervasive Data Profiler™ to assess the quality of clients' data prior to loading. Pervasive Data Profiler lets LifeMasters generate metrics quickly and easily so that files can be analyzed and, if necessary, recycled back to the customer before the loading process begins. This allows the customer to address the problem up front, saving tons of time.

In Steve Hromyko's words, "Prior, we were unable to assess the quality of data coming into our systems. We had no way to generate a system of metrics to really understand, is this good data, is this bad data. It was a very manual process and eventually, the system would break. That’s how we would find out about bad data. So, we may go through five, six, seven days worth of data loading just to get to the point where, oh, geez, we need more information from this customer. So, we just wasted all that time. We have to go back to the customer and have them resubmit it. Now that we’re using the Data Profiler, we’re able to do that automatically, and then identify which records are bad, pull those records out, so that the good data can continue on and load into our system, so we can get patients enrolled much quicker. And then off line, be able to address the issues with the bad data. It doesn’t stop our process."

When the problems with the data don't require more information from the customer, those can be automatically remediated using Pervasive Data Integrator. As Hromyko says, this process allows LifeMasters "to not only get the data loaded quicker, but to get the data loaded cleaner and quicker."

For more information on the products used, see Data Integrator and Data Profiler

For more information on integrating industry data, see Healthcare integration