Comments for OpenMRS: Record Linkage Project http://soc.sarpcentel.com Google Summer of Code™ 2007 Fri, 29 Aug 2008 05:46:55 +0000 http://wordpress.org/?v=2.6 Comment on Performance Test: Analyzing token frequencies by sarp http://soc.sarpcentel.com/2007/07/23/performance-test-analyzing-token-frequencies/#comment-440 sarp Wed, 10 Oct 2007 23:41:28 +0000 http://soc.sarpcentel.com/2007/07/23/performance-test-analyzing-token-frequencies/#comment-440 It would be interesting to see what possible collaborations are possible. Please contact Shaun Grannis who is leading this project. You can find his contact details here: http://www.regenstrief.org/bio/full?member=sgrannis It would be interesting to see what possible collaborations are possible. Please contact Shaun Grannis who is leading this project. You can find his contact details here:
http://www.regenstrief.org/bio/full?member=sgrannis

]]> Comment on Performance Test: Analyzing token frequencies by Andre http://soc.sarpcentel.com/2007/07/23/performance-test-analyzing-token-frequencies/#comment-437 Andre Wed, 10 Oct 2007 18:53:26 +0000 http://soc.sarpcentel.com/2007/07/23/performance-test-analyzing-token-frequencies/#comment-437 Hello. I've been working with Record Linkage for some time and i'm building a FrameWork (which i've already been using). Maybe we could exchange ideas, and maybe merging the project could be a good idea too. Hope you answer this soon André Hello. I’ve been working with Record Linkage for some time and i’m building a FrameWork (which i’ve already been using).
Maybe we could exchange ideas, and maybe merging the project could be a good idea too.

Hope you answer this soon

André

]]> Comment on Code Spotlight: Analyzing token frequencies by OpenMRS: Record Linkage Project » Blog Archive » Midterm update http://soc.sarpcentel.com/2007/07/11/code-spotlight-analyzing-token-frequencies/#comment-12 OpenMRS: Record Linkage Project » Blog Archive » Midterm update Thu, 19 Jul 2007 20:24:34 +0000 http://soc.sarpcentel.com/2007/07/11/code-spotlight-analyzing-token-frequencies/#comment-12 [...] order to introduce weight scaling, we first analyze data sources (could be database or character delimited file) that will be used in linkage for token [...] [...] order to introduce weight scaling, we first analyze data sources (could be database or character delimited file) that will be used in linkage for token [...]

]]> Comment on Code Spotlight: Analyzing token frequencies by Darius Jazayeri http://soc.sarpcentel.com/2007/07/11/code-spotlight-analyzing-token-frequencies/#comment-9 Darius Jazayeri Thu, 12 Jul 2007 20:24:23 +0000 http://soc.sarpcentel.com/2007/07/11/code-spotlight-analyzing-token-frequencies/#comment-9 Hi Sarp, That's probably a conversation worth having more broadly. For example when I was working on cohort builder I did some back-of-the-envelope math and decided that I'd be taking up 5MB of RAM, and that's okay because it's important enough. Yesterday Christian and Justin and I we were discussing the Find a Patient search widget, and whether we might want to build a list of tokens for that. We decided that was also important enough. :-) We probably need a less ad-hoc process for deciding "this feature is worth X megs of RAM". -Darius Hi Sarp,

That’s probably a conversation worth having more broadly.

For example when I was working on cohort builder I did some back-of-the-envelope math and decided that I’d be taking up 5MB of RAM, and that’s okay because it’s important enough.

Yesterday Christian and Justin and I we were discussing the Find a Patient search widget, and whether we might want to build a list of tokens for that. We decided that was also important enough. :-)

We probably need a less ad-hoc process for deciding “this feature is worth X megs of RAM”.

-Darius

]]> Comment on Database schema for analysis results by sarp http://soc.sarpcentel.com/2007/06/17/database-schema-for-analysis-results/#comment-8 sarp Wed, 20 Jun 2007 08:05:55 +0000 http://soc.sarpcentel.com/2007/06/17/database-schema-for-analysis-results/#comment-8 <p>1. access refers to the delimiter character in flat files, for databases it could be referring to username-password information etc. James would know better because it is a variable he used while designing LinkDataSource class.</p> <p>2. The motivation for these fields was that since analytic phase may take a long time for large data sources, we may want to use old statistics in the linkage process to skip the analytic phase. </p> <p>changed field will be either 0 or 1, depending on whether any update/insert/delete operations were done to the database after our statistical analysis. This field would need to be maintained by functions outside of patient linkage module that make changes to the database.</p> <p>date_changed is to store the last date in which changes to the database were made. It would be good to know how fresh the calculated statistics are, to decide whether to run the analytic phase again or not.</p> <p>3. Thanks, I'll add that</p> 1. access refers to the delimiter character in flat files, for databases it could be referring to username-password information etc. James would know better because it is a variable he used while designing LinkDataSource class.

2. The motivation for these fields was that since analytic phase may take a long time for large data sources, we may want to use old statistics in the linkage process to skip the analytic phase.

changed field will be either 0 or 1, depending on whether any update/insert/delete operations were done to the database after our statistical analysis. This field would need to be maintained by functions outside of patient linkage module that make changes to the database.

date_changed is to store the last date in which changes to the database were made. It would be good to know how fresh the calculated statistics are, to decide whether to run the analytic phase again or not.

3. Thanks, I’ll add that

]]> Comment on Database schema for analysis results by Shaun Grannis http://soc.sarpcentel.com/2007/06/17/database-schema-for-analysis-results/#comment-7 Shaun Grannis Tue, 19 Jun 2007 15:21:44 +0000 http://soc.sarpcentel.com/2007/06/17/database-schema-for-analysis-results/#comment-7 Sarp, a great start to the the data model! A couple of questions: 1. In the patient_matching_datasource_analysis table, there is a field labeled "access", can you describe its use? 2. In the patient_matching_field table, can you describe how the "date_changed" and "changed" fields will be used? I'm not certain how the "changed" field will be used 3. A given field (such as last name) will have an overall entropy. Additionally, each individual token possesses an individual entropy. I would suggest adding an "entropy" field to the patientmatching_token table. Calculating the entropy of individual tokens is not top priority, but I anticipate potentially using those values in the future. Sarp, a great start to the the data model! A couple of questions:

1. In the patient_matching_datasource_analysis table, there is a field labeled “access”, can you describe its use?

2. In the patient_matching_field table, can you describe how the “date_changed” and “changed” fields will be used? I’m not certain how the “changed” field will be used

3. A given field (such as last name) will have an overall entropy. Additionally, each individual token possesses an individual entropy. I would suggest adding an “entropy” field to the patientmatching_token table. Calculating the entropy of individual tokens is not top priority, but I anticipate potentially using those values in the future.

]]> Comment on Agenda for the week by Shaun Grannis http://soc.sarpcentel.com/2007/05/30/agenda-for-the-week/#comment-6 Shaun Grannis Wed, 30 May 2007 02:14:58 +0000 http://soc.sarpcentel.com/2007/05/30/agenda-for-the-week/#comment-6 Regarding NULLs: For frequency scaling, I anticipate that we will ignore nulls in most cases. However, because there is sure to be variation in how other users want to treat nulls, we want to capture sufficient information on null frequencies for those who may want to use them. That information includes: a) total number of non-null tokens, b) total number of null tokens, c) total number of unique non-null tokens. From the OpenMRS matching design pages, there should be the following options pertaining to nulls contained in the config file (talk to James to see "if" and "how" these are implemented): - A flag indicating whether to use null tokens when scaling agreement weight based on term frequency (default-no) - A flag indicating how to establish agreement among fields when one or both fields are null (eg, apply disagreement weight, apply agreement weight, or apply zero weight) (default-apply zero weight) Regarding NULLs:
For frequency scaling, I anticipate that we will ignore nulls in most cases. However, because there is sure to be variation in how other users want to treat nulls, we want to capture sufficient information on null frequencies for those who may want to use them. That information includes: a) total number of non-null tokens, b) total number of null tokens, c) total number of unique non-null tokens.

From the OpenMRS matching design pages, there should be the following options pertaining to nulls contained in the config file (talk to James to see “if” and “how” these are implemented):

- A flag indicating whether to use null tokens when scaling agreement weight based on term frequency (default-no)
- A flag indicating how to establish agreement among fields when one or both fields are null (eg, apply disagreement weight, apply agreement weight, or apply zero weight) (default-apply zero weight)

]]> Comment on Agenda for the week by Shaun Grannis http://soc.sarpcentel.com/2007/05/30/agenda-for-the-week/#comment-5 Shaun Grannis Tue, 29 May 2007 21:32:29 +0000 http://soc.sarpcentel.com/2007/05/30/agenda-for-the-week/#comment-5 Response to question 1) Scalability: Given that it is still uncertain how end-users will specifically implement scaling, we'll need flexibility to limit the number of entries in the data structure. Options that immediately come to mind include: a) Store all token frequencies b) Limit by percent of total unique tokens, eg, "store the 10% most frequent tokens", or "store the 10% least frequent tokens" c) Limit by absolute number of tokens, eg, "store 1,000 most frequent tokens", or "store 3,000 least frequent tokens" d) conditional on token frequency: store all tokens with frequency above 'n', eg 'store all tokens with frequency above 100', or 'store all tokens with frequency below 100' e) may need to be able to combine the above criteria, eg 'store top 1,000 tokens with frequency greater than 2,000' Will get to 2) later... Thanks! Response to question 1) Scalability: Given that it is still uncertain how end-users will specifically implement scaling, we’ll need flexibility to limit the number of entries in the data structure. Options that immediately come to mind include:
a) Store all token frequencies
b) Limit by percent of total unique tokens, eg, “store the 10% most frequent tokens”, or “store the 10% least frequent tokens”
c) Limit by absolute number of tokens, eg, “store 1,000 most frequent tokens”, or “store 3,000 least frequent tokens”
d) conditional on token frequency: store all tokens with frequency above ‘n’, eg ’store all tokens with frequency above 100′, or ’store all tokens with frequency below 100′
e) may need to be able to combine the above criteria, eg ’store top 1,000 tokens with frequency greater than 2,000′

Will get to 2) later… Thanks!

]]> Comment on Connecting the dots by Shaun Grannis http://soc.sarpcentel.com/2007/05/21/connecting-the-dots/#comment-4 Shaun Grannis Sat, 26 May 2007 04:22:38 +0000 http://soc.sarpcentel.com/2007/05/21/connecting-the-dots/#comment-4 In response to comment C) above: string comparators in fact can produce a value between 0 and 1. In practice, if the comparator value exceeds a pre-specified threshold, then the fields are considered to agree, and $latex \gamma_{k}$ for that field is set to 1. Conversely, if the comparator value is less than a pre-specified threshold, then the fields are considered to disagree, and $latex \gamma_{k}$ for that field is set to 0. Another interesting (and unproven) approach is to make the $latex m_{k}$ and $latex u_{k}$ rates conditional on the string comparator score. For instance, if we have an arbitrary comparator that produced values (x) between 0 and 1, we can divide these values into 2 ranges: range a: $latex 0.50\leq{ x}\leq0.75$ range b: $latex 0.75\textless{ x}\leq1.00$ Then for each field there can be 2 likelihood scores depending on the value of the string comparator. For example, if the string comparator returned a score of 0.78, then the agreement weight would be $latex \frac{m_{b}}{u_{b}}$, the disagreement weight would be $latex \frac{1-m_{a}-m_{b}}{1-u_{a}-u_{b}}$ In response to comment C) above: string comparators in fact can produce a value between 0 and 1. In practice, if the comparator value exceeds a pre-specified threshold, then the fields are considered to agree, and \gamma_{k} for that field is set to 1. Conversely, if the comparator value is less than a pre-specified threshold, then the fields are considered to disagree, and \gamma_{k} for that field is set to 0.

Another interesting (and unproven) approach is to make the m_{k} and u_{k} rates conditional on the string comparator score. For instance, if we have an arbitrary comparator that produced values (x) between 0 and 1, we can divide these values into 2 ranges:
range a: 0.50\leq{ x}\leq0.75
range b: 0.75\textless{ x}\leq1.00

Then for each field there can be 2 likelihood scores depending on the value of the string comparator. For example, if the string comparator returned a score of 0.78, then the agreement weight would be \frac{m_{b}}{u_{b}}, the disagreement weight would be \frac{1-m_{a}-m_{b}}{1-u_{a}-u_{b}}

]]> Comment on Connecting the dots by Shaun Grannis http://soc.sarpcentel.com/2007/05/21/connecting-the-dots/#comment-3 Shaun Grannis Fri, 25 May 2007 19:29:21 +0000 http://soc.sarpcentel.com/2007/05/21/connecting-the-dots/#comment-3 Sarp, Great work above! I'll try to cover several issues in this response. In this reply, the term "token" refers to a specific instance in a given field. For example, tokens for the Last_Name field might be 'SMITH', 'GRANNIS', 'CENTEL', etc. Tokens for Zip_code may be '46224', '02139', '44235', etc. Not sure I follow the equation you list above, which I repeat below as equation (1). Specifically, which are the token frequencies, and which are the weights? $latex (1) \hspace{20pt}\sum_{k=1}^n w_{k} \cdot \log(\frac{m_{k}}{u_{k}}) ^ {w^{j}_{k}\gamma^{j}_{k}} \cdot \log(\frac{1-m_{k}}{1-u_{k}}) ^ {({1-w^{j}_{k}\cdot\gamma^{j}_{k}})}$ The Felligi-Sunter (F-S) "weight" for fields that *agree* is: $latex (2) \hspace{20pt}agree\hspace{5pt}weight = \log(\frac{m_{k}}{u_{k}})$ And the Felligi-Sunter "weight" for fields that *disagree* is: $latex (3) \hspace{20pt}disagree\hspace{5pt}weight = \log(\frac{1-m_{k}}{1-u_{k}})$ Where: $latex m_{k}$ is the agreement rate (frequency) among *true* matches $latex u_{k}$ is the agreement rate (frequency) among *false* matches ... So it seems that the F-S field weights are included twice in equation (1)? We propose the following general scaling factor (s) for each field, which decreases wieghts for frequently occuring tokens, while: $latex (4) \hspace{20pt}s_{k} = \sqrt{\frac{T_{k}}{(Q_{k} \cdot I_{k})}}$ Where: $latex T_{k}$ is the total number of tokens for the field (constant for the field) $latex Q_{k}$ is the total number of *unique* tokens for the field (constant for the field) $latex I_{k}$ is the specific frequency of the current Individual token (varies for each token) Since $latex T_{k}$ and $latex Q_{k}$ are constant for the field, equation (4) can be re-written as: $latex (5) \hspace{20pt}s_{k} = \sqrt{\frac{A_{k}}{I_{k}}}$ Where: $latex A_{k} = \frac{T_{k}}{Q_{k}}$, and is constant for the field. It represents the average token frequency for the field. The F-S log-likelihood equation is: $latex (6) \hspace{20pt}likelihood=\sum_{k=1}^n\log(\frac{m_{k}}{u_{k}}) ^{\gamma_{k}} \cdot \log(\frac{1-m_{k}}{1-u_{k}}) ^{(1-\gamma_{k})}$ Where: $latex m_{k}$ is the agreement rate among *true* matches $latex u_{k}$ is the agreement rate among *false* matches $latex \gamma_{k}$ is the binary agreement status for the current field (eg, 1=agree, 0=disagree) Incorporating the scaling factor from equation (5) into equation (6), we get the following modified F-S equation: $latex (7) \hspace{20pt}likelihood=\sum_{k=1}^n\log(s_{k}\cdot\frac{m_{k}}{u_{k}}) ^{\gamma_{k}} \cdot \log(\frac{1-m_{k}}{1-u_{k}}) ^{(1-\gamma_{k})}$ So equation (7) is largely the goal. Please let me know your thoughts. Thanks! Sarp,

Great work above! I’ll try to cover several issues in this response. In this reply, the term “token” refers to a specific instance in a given field. For example, tokens for the Last_Name field might be ‘SMITH’, ‘GRANNIS’, ‘CENTEL’, etc. Tokens for Zip_code may be ‘46224′, ‘02139′, ‘44235′, etc.

Not sure I follow the equation you list above, which I repeat below as equation (1). Specifically, which are the token frequencies, and which are the weights?

(1) \hspace{20pt}\sum_{k=1}^n w_{k} \cdot \log(\frac{m_{k}}{u_{k}}) ^ {w^{j}_{k}\gamma^{j}_{k}} \cdot \log(\frac{1-m_{k}}{1-u_{k}}) ^ {({1-w^{j}_{k}\cdot\gamma^{j}_{k}})}

The Felligi-Sunter (F-S) “weight” for fields that *agree* is:
(2) \hspace{20pt}agree\hspace{5pt}weight = \log(\frac{m_{k}}{u_{k}})
And the Felligi-Sunter “weight” for fields that *disagree* is:
(3) \hspace{20pt}disagree\hspace{5pt}weight = \log(\frac{1-m_{k}}{1-u_{k}})
Where:
m_{k} is the agreement rate (frequency) among *true* matches
u_{k} is the agreement rate (frequency) among *false* matches

… So it seems that the F-S field weights are included twice in equation (1)?

We propose the following general scaling factor (s) for each field, which decreases wieghts for frequently occuring tokens, while:

(4) \hspace{20pt}s_{k} = \sqrt{\frac{T_{k}}{(Q_{k} \cdot I_{k})}}
Where:
T_{k} is the total number of tokens for the field (constant for the field)
Q_{k} is the total number of *unique* tokens for the field (constant for the field)
I_{k} is the specific frequency of the current Individual token (varies for each token)

Since T_{k} and Q_{k} are constant for the field, equation (4) can be re-written as:
(5) \hspace{20pt}s_{k} = \sqrt{\frac{A_{k}}{I_{k}}}
Where:
A_{k} = \frac{T_{k}}{Q_{k}}, and is constant for the field. It represents the average token frequency for the field.

The F-S log-likelihood equation is:
(6) \hspace{20pt}likelihood=\sum_{k=1}^n\log(\frac{m_{k}}{u_{k}}) ^{\gamma_{k}} \cdot \log(\frac{1-m_{k}}{1-u_{k}}) ^{(1-\gamma_{k})}

Where:
m_{k} is the agreement rate among *true* matches
u_{k} is the agreement rate among *false* matches
\gamma_{k} is the binary agreement status for the current field (eg, 1=agree, 0=disagree)

Incorporating the scaling factor from equation (5) into equation (6), we get the following modified F-S equation:
(7) \hspace{20pt}likelihood=\sum_{k=1}^n\log(s_{k}\cdot\frac{m_{k}}{u_{k}}) ^{\gamma_{k}} \cdot \log(\frac{1-m_{k}}{1-u_{k}}) ^{(1-\gamma_{k})}

So equation (7) is largely the goal. Please let me know your thoughts.

Thanks!

]]>