Notes from 10/21/2008 Patient Matching Call

(A) We're experiencing out of memory errors in Tomcat with 100,000 patients in OpenMRS and 512 MB allocated to Tomcat. The first blocking run (blocking on postal code) appears to be completing; the out of memory error appears to occur during the analysis phase (either during random sampling or during Expectation Maximization) of the second blocking run (blocking on SSN).

Memory snapshots from the profiler reveal that a large amount of memory is allocated for the MySQL database connection. Large result sets are not being released and closed. Even though we are attempting to close the result set, it remains open, we believe because unidentified resources are still accessing the result set.

James has updated the code to address releasing the result set, and Win is testing the revision on his computer
If this code update doesn't resolve the issue, we'll re-examine the profiler output for other places where memory use could be further optimized.

Update 10/22/2008: The new code successfully ran with no out-of-memory errors observed with the new code. Therefore


(B) In some cases, the duplicate report listing may be large. So rather than displaying the report in a web browser (which is not designed to display large amounts of line-list data), it may be more practical to output the duplicate report directly to a file. Consequently, we're in the process of modifying the module to accommodate this. Win is implementing a more robust reporting work flow using AJAX. The user will be able to initiate a report, navigate away from the administrator interface, return to check status, and when the report is complete, a link to that report is displayed.

We’ve implemented a lock-out feature so that once a report is started, no reports can be initiated until the current report is complete.


(C) Currently we have been unable to run the “Yourkit” java profiler on Linux. We are using Windows to profile memory usage. If this issue further hinders progress, we will need to address the barrier we face to run Yourkit on Linux.