OpenMRS: Record Linkage Project

I refactored the code for Phase1 according to the feedback I received, and completed Phase 2 (Here is the changeset). I will make a more detailed post tomorrow (before the phone call). The next step (changing the formula for weight scaling) should be fairly easy, after I figure out where scores are calculated in the existing code. Testing and commenting the code should not take too long either, since I know that the code is functional, just have to check for boundary cases etc.

Runtime Component, start-up:  Implement functionality instantiating a data structure that provides fast lookup of individual token frequencies.  This data structure will likely be a hash table, where the key is the token value (eg, last name of “SMITH”) and the value is the token frequency (eg, 2,102). This data will be loaded from the persistent data structure created in task 1(e).Because the primary performance constraint for weight-based frequency scaling will be the lookup, we will need to be able to configure the number of elements loaded into the hash table.  For example, it is likely that some fields will have hundreds of thousands of unique tokens (eg, name fields), while others will have on the order of 10 or 20 (middle initial, month of birth).Also, weight scaling can be used to either increase or decrease individual field weights.  If an individual token frequency is less than the average frequency it will be increased, if it is above the average frequency it will be decreased.  Consequently, there needs to be some ability to configure the total number of tokens loaded into the lookup structure for each field.a. Implement functionality to load top ‘N’ most/least frequent tokens from the persistent data structure, where top, bottom, and ‘N’ are specified in the configuration file.  Ifb. Other (future) options may include top or bottom N%, frequencies above or below N.He