OpenMRS: Record Linkage Project

Performance Test: Analyzing token frequencies

sarp — Sun, 22 Jul 2007 22:16:32 +0000

For analyzing token frequencies, I have implemented two different designs.Design A: For different data sources, create specific data readers that convert your data into Records. For frequency analysis, operate on the abstraction of RecordAdvantage:

Easy to maintain, implement once and for all
Reduces complexity of the project

Disadvantage:

Poor performance
High memory requirement, since you are doing analysis record by record, instead of column by column, you need to keep frequency data for ALL columns at the same time

Design B: For each type of data source (database, character delimited file etc.) we implement a specific analyzer in addition to a data readerAdvantage:

Better performance in some data sources since you are operating on a lower level of abstraction
Possibility of doing analysis column by column, the way it is supposed to be

Disadvantage:

Each data source needs a different analyzer, increases complexity
Could be tiresome to implement

Performance difference becomes apparent in analyzing databases. Instead of loading our data into memory, processing it and writing it back, we can just ask MySQL to tell us token frequencies with a query, and store this information for later use. Below is the result for a data source of 5000 records (fileA_5000) consisting of 19 columns. Token frequency analysis of one column takes:Design B: 1843 millisecondsDesign A: 4035 millisecondsDesign A does not improve much even if we reduce column count to one. This could mean that execution time consists of mostly overhead from iterating over all records and numerous function calls it brings.Last word:Even though there is significant performance difference for database data, I wouldn’t argue passionately for design B. This decision depends on how many different data sources we are planning to support, their nature (possibility of performance gain or not) and whether execution time stays within reasonable limits in design A or not.

Midterm update

sarp — Thu, 19 Jul 2007 20:24:29 +0000

As many of you know, we are working on a patient matching module for OpenMRS that will allow users to identify records that belong to the same patient among different data sources.In the first part of SoC, I’ve completed adding weight scaling functionality to the existing record linkage framework.Matching records are determined by assigning a score to each possible record pair. Weight scaling improves the accuracy of patient matching because fields that match on a common value, for instance James for first name, will be scaled down, and they will contribute less to the overall score for the given pair.In order to introduce weight scaling, we first analyze data sources (could be database or character delimited file) that will be used in linkage for token frequencies. We store this data in a relational database and use it later during calculating scores for possible pairs. We have the ability to use different lookup tables for token frequencies (top N most/least frequent tokens, top N% most/least frequent tokens and frequencies above/below N).There are other possible improvements for scoring, therefore we’re currently working on refactoring the framework to make it easier to adjust matching scores.

Code Spotlight: Analyzing token frequencies

sarp — Tue, 10 Jul 2007 19:48:52 +0000

I’d like to read a text file and store the frequency of each word in it into a database. Memory is fast, database is slow. At the two extremes, we have:1) I don’t use any memory, for each word I read, I query the database to learn it’s frequency, and update it in the database.Problem: Very Inefficient2) Everything is stored in the memory, I keep a hashtable in which I store frequencies of words. After reading the whole file, I write all entries to the database.Problem: Very fast, but I may have so many words that they won’t fit into memory.Solution:I keep part of the words in a hashtable, for those entries that are not in the hashtable, I query the database.SubProblem: How do I decide if I’ll put a word in the hashtable or not? I’d like to store the most frequent words in the hashtable so that I will minimize the number of queries to the database. But I don’t know frequencies of words in advance, they change as I read the file. My hashtable has to organize itself as I read the text file.Solution: If the next word I read is not in the hashtable, and is more frequent than the *least* frequent word in the hashtable, I put the new word into the table and remove the least frequent one. The reasoning is that if a word has high frequency, it is likely that I will see more of the same word.SubProblem: I need a way to efficiently determine the least frequent word in the hash table. Further more, after I replace the least frequent one, which word will be the least frequent one now? What about next round?Solution: Along with the hashtable, I also keep a data structure called PriorityQueue. If implemented using a heap, it provides insert, remove and findMin operations in O(logN) time.Here are some experimental results for analyzing one column of 10,000 records:

Without a lookup table, only using database: 16 seconds
With a lookup table of 250 records (covers 60% of data): 8 seconds
With a lookup table of 500 records (covers 80% of data): 5 seconds

Any ideas or comments on this approach will be appreciated

Phase3 Complete

sarp — Tue, 10 Jul 2007 19:48:04 +0000

Now that Phase3 is ready for review, we have added weight scaling functionality to patient matching module. In fact, I have both versions of weight scaling implemented (Framework A and B). This week, I will be performing performance tests this week to determine if there is noteworthy performance difference between these two approaches.

3. Runtime Component, operational: Modify the ScorePair method to incorporate frequency scaling. This process should be performed incrementally, in two phases.The first phase will hard code the frequency scaling equation, into the existing ScorePair method. Once the entire linkage process (from analytics to operational phase) has been tested and successfully implements frequency scaling as a prototype, we will proceed to phase 2.In the second phase ScorePair will be re-factored to accommodate a framework that accepts future modifications to linkage scores established by the Felligi-Sunter model. These modifications include the frequency scaling, and will also include modifying the agreement weight based on the degree of string similarity as established by various string comparators.

Phase2 Complete

sarp — Mon, 02 Jul 2007 13:40:10 +0000

I refactored the code for Phase1 according to the feedback I received, and completed Phase 2 (Here is the changeset). I will make a more detailed post tomorrow (before the phone call). The next step (changing the formula for weight scaling) should be fairly easy, after I figure out where scores are calculated in the existing code. Testing and commenting the code should not take too long either, since I know that the code is functional, just have to check for boundary cases etc.

Runtime Component, start-up: Implement functionality instantiating a data structure that provides fast lookup of individual token frequencies. This data structure will likely be a hash table, where the key is the token value (eg, last name of “SMITH”) and the value is the token frequency (eg, 2,102). This data will be loaded from the persistent data structure created in task 1(e).Because the primary performance constraint for weight-based frequency scaling will be the lookup, we will need to be able to configure the number of elements loaded into the hash table. For example, it is likely that some fields will have hundreds of thousands of unique tokens (eg, name fields), while others will have on the order of 10 or 20 (middle initial, month of birth).Also, weight scaling can be used to either increase or decrease individual field weights. If an individual token frequency is less than the average frequency it will be increased, if it is above the average frequency it will be decreased. Consequently, there needs to be some ability to configure the total number of tokens loaded into the lookup structure for each field.a. Implement functionality to load top ‘N’ most/least frequent tokens from the persistent data structure, where top, bottom, and ‘N’ are specified in the configuration file. Ifb. Other (future) options may include top or bottom N%, frequencies above or below N.He

Phase1 Complete

sarp — Mon, 25 Jun 2007 15:56:35 +0000

I completed Phase 1, the Analytic Component. It includes:

Implement and analytic object that performs the following tasks:a. Read linkage configuration file and determine which fields/columns need to be scaled.b. Connect to data source containing individual tokens. Assume that the data source is either a flat file such as a CSV file or a relational database.c. Add new information to configuration file that indicates location of token frequenciesd. Count frequencies of individual tokens.e. Store token frequency results in persistent structure (eg, a relational database table). In order to access the token frequency data at runtime, the frequency tables need to be identified in the configuration file. Thus, will need to develop a programmatic scheme to identify each token frequency table associated with a given data source, eg:

What has changed in the source code?Testing code has been moved into a new package called org.openmrs.testingorg.regenstrief.linkage.analysis,A new abstract class for analyzers called DataSourceAnalyzerTwo classes that extend it are CharDelimFileAnalyzer and DataBaseAnalyzerThese classes contain a CharDelimFileReader/DataBaseReader to go over/query recordsIn this schema, existing classes such as DataSourceAnalysis, Analyzer and ScaleWeightAnalyzer are not used. The schema provided by existing classes was more general, and if we can find a way to fit my classes into it, I’m willing to refactor the code. Otherwise, I’ll delete them.org.regenstrief.linkage.dbAnalyzers contain a LinkDBManager to insert token frequencies into the database. I’m not sure if this was the most suitable class for adding this code.org.regenstrief.linkage.ioDataSourceReader has a new parameter to determine if it will be used for analysis or reading. This change was necessary because in reading, some columns are excluded and blocking is done to make the source ready for linkage. However, we don’ want these in analyzing the data.org.regenstrief.linkage.utilLinkDataSource: Added a variable to store a unique identifier for each linkdatasource (is used in storing token frequencies)RecMatchConfig: Added a LinkDBManager to create a connectionXMLTranslator: Modified to include id of the linkdatasource and a new parameter for storing token frequencies

Database schema for analysis results

sarp — Sat, 16 Jun 2007 21:36:27 +0000

Here is a draft on how to store analytical phase results in a relational database table. I created this diagram using DBDesigner 4, the same tool used to create OpenMRS Data Model. Right click here and choose ”Save target as” if you’d like to load this schema in DBDesigner and modify it.Datasource_analysis table mimics LinkDataSource class in org.regenstrief.linkage.io package. I am imagining a GUI in OpenMRS where the user manually chooses among existing data sources, or adds a new data source in which the datasource_id is automatically assigned by the database.Field table contains changed and data_changed attributes to determine how fresh the statistics are.

Explaining grandma record linkage

sarp — Sat, 16 Jun 2007 20:45:19 +0000

I was so eager to start working on this project that I forgot to introduce the problem that I will be working on during this summer.Record linkage, roughly, is the task of identifying pieces of scattered information that refer to the same thing. Patient matching is a specific application, in which we try to identify records that belong to the same patient among different data sources. These sources can range from patient data collected at different hospitals to external information from governmental institutions, such as death master file etc.One of the interesting and challenging aspects of this project is to deal with erroneous data, for instance when your name is misspelled or your birth date is entered incorrectly. These kinds of things often happen in reality, and we can account for them by using flexible distance metrics and statistical models. Why is then record linkage important and what are the benefits?Well, we are living in an exciting period of globalization, where computers and internet make world-wide collaboration easy and necessary. Patient linkage and data aggregation techniques will allow medical institutions to store their own data, yet at the same time work together with others to offer better treatment to patients. For instance, patients often forget their test results at home, or old tests get lost eventually. Imagine that all your medical records are stored in digital format, and when you go to Hospital A, a doctor there can examine your tomogram taken 4 years ago at Hospital B where your name was misspelled by the clerk I hope that record linkage functionality will be a step forward to increase collaboration between OpenMRS implementers.

Code reviews

sarp — Thu, 07 Jun 2007 21:35:38 +0000

Google was cool enough to send out Karl Fogel’s Producing Open Source Software book as a suprise gift to all Summer of Code ‘07 participants. I have already read that book at our local library after getting accepted, therefore I was hoping to receive another book, Open Sources 2.0: The Continuing Evolution Nevertheless, the book we received is more suitable in terms of the goals of this program, so there is nothing to complain :)Karl Fogel was one of the developers of Subversion, and a past Google employee. In his book, he shares his experience about different aspects of open-source software development, from setting up the technical infrastucture to licensing and copyright issues. This book is worth reading because it contains specific examples and good advice.Fogel mentions a developer named Greg Stein, who decided to set an example by reviewing every line of every single commit that went into the code repository and eventually started a tradition of code reviews among developers of Subversion project.As a young developer, code reviews would be a great way to get useful feedback and improve my coding abilities. Fogel argues that one could contribute as much to the project by reviewing others’ changes as by writing new code because bugs and non-optimal coding practices that would go unnoticed would be caught on the fly.I agree with him, and would love to receive some reviews after my first few commits

Agenda for the week

sarp — Tue, 29 May 2007 20:30:19 +0000

After receiving a great guideline on how to implement weight scaling from James, I examined part of the existing code today. In the analytic phase, for each field, we need to calculate:(1) number of unique values(2) frequency of each unique value(3) total recordsThese calculated values should be stored in the database for future analysis.My idea for the data structure to hold these values, ScaleWeightData, is:- integer array indexed by column id for each field regarding (1) and (3)- hash table indexed by token that contains most frequent K records for (2), with database lookups for tokens that are not found in the hash table (for large data sets)A few questions as usual:1) Scalability: What magnitude of scalability are we trying to achieve? Will scalability be a major design concern right from the beginning or are we trying to make it work first, and later worry about scalability? Should I worry about a real system with millions of patient records that may not fit into memory?2) Null values: How do they affect our statistical analysis? Do we ignore them when calculating total number of records?