Monday, April 6, 2009
The President Speaks on International Cooperation to Fight Pandemics
Click here to see and listen to short video
De-duplication
I've been thinking about the "De-duplication" grid service a bit lately and want to get my thoughts out here in case someone else has been mulling it over as well.
De-duplication is related to the De-identification/Re-identification problem of Anonymized data across multiple data sources (interesting MIT paper here). But in a grid environment the problem becomes tougher because the traditional, really-hard-to-do-solution of checking the various MPIs is even tougher because data sources 1) don't frequently have MPIs and 2) don't want to share them if they do.
Here's the scenario I use to talk about the de-duplication problem set:
The scenario is this:
3 hospitals in Atlanta with overlapping catchment areas. A federated query is sent out for SyndromeX for April 1, 2009. Hospital A returns a count of 10, Hospital B returns a count of 15 and Hospital C returns a count of 20. The combined result from the federated query shows a total of 45 counts of SyndromeX for April 1, 2009.
There exists a question of data quality as it's possible that the same patient visited multiple hospitals and had his SyndromeX recorded at multiple locations. In a federated/grid configuration this is even more probable as Hospitals may use data from multiple feeder sources (labs, small clinics, etc) that may report the same patient encounter multiple times.
Dr. Lenert had a good idea where each Hospital could send over a list of PII (name, address, dob, social) without the medical information for purposes of de-duping. This requires a trusted intermediary to receive these matches and send them on to each Hospital to respond back with number of matches. A hospital cannot send directly to each other hospital because it would reveal its patients to potential competitors.
So the de-duping scenario may change to something like this:
1) User submits query for SyndromeX on April 1, 2009 and selects de-duplication.
2) Trusted intermediary receives the query and passes it on to Hospitals A,B,C.
3) Hospital A returns count of 10, Hospital B returns count of 15, Hospital C returns count of 20. Interim total count 55 is not returned to user.
4) Trusted intermediary then requests patient id info from each hospital in a separate request.
5) Trusted intermediary compares the PII from each hospital (perhaps using a tool like febrl) and determines that there is overlap of patients and actual count. Hospital A shares 1 patient with Hospital B. Hospital A shares 2 patients with Hospital C. Hospital B shares 3 patients with Hospital C. Therefore there are 6 unique duplicates.
6) Total count of 39 is returned to the user (45-1-2-3).
Of course, this is going to require some pretty serious data use agreements with the hospitals and the trusted intermediary to control the storage of PII (i.e. don't persist any PII)
The alternative without the trusted intermediary:
1) User submits query for SyndromeX on April 1, 2009 and selects de-duplication.
2) Each hospital receives the query and list of nodes to which is was submitted.
3) Hospital A identifies a preliminary count of 10. To check for duplicates, it submits only the PII without any condition data and a query ID to Hospital B and C for comparison.
4) Hospital B compares the PII to its own results for query ID and returns the number of matches to Hospital A (duplicate count=1).
5) Hospital C compares the PII to its own results for query ID and returns the number of matches to Hospital A (duplicate count=2).
6) At the same time, Hospitals B&C are performing similar communications with the other nodes participating in the query.
7) Hospital B identifies a preliminary count of 15. To check for duplicates, it submits only the PII without any condition data and a query ID to Hospital A and C for comparison.
8) Hospital A compares the PII to its own results for query ID and returns the number of matches to Hospital B (duplicate count=1).
9) Hospital C compares the PII to its own results for query ID and returns the number of matches to Hospital B (duplicate count=3).
10) Hospital C identifies a preliminary count of 20. To check for duplicates, it submits only the PII without any condition data and a query ID to Hospital A and B for comparison.
11) Hospital A compares the PII to its own results for query ID and returns the number of matches to Hospital C (duplicate count=1).
12) Hospital B compares the PII to its own results for query ID and returns the number of matches to Hospital C (duplicate count=3).
13) Hospital A returns a count of 10 and notes that duplicates with B=1 and duplicates with C=2.
14) Hospital B returns a count of 15 and notes that duplicates with A=1 and duplicates with C=3.
15) Hospital C returns a count of 20 and notes that duplicates with A=2 and duplicates with B=3.
16) The user then feeds these results into another service that processes the results and determines that the actual count is 39 based on the following calculated relationships: (A|B)-1; (A|C)-2; (B|C)-3. These reductions are subtracted from the preliminary total of 45 to yield 39.
Approach #2, while not requiring a trusted intermediary requires a lot of coordination with the queried nodes and also involves the sharing of patient info with other partners that may not be desired. Also, due to set math will grow exponentially with the number of nodes involved in the query (for 3 nodes, the number of service calls is 9 [3 from user to nodes, then 6 among the nodes-using n!/{n-r}!]; but for 4 nodes, the number of service calls is 16; and for 5 nodes the number of service calls is 25). This is also suspect due to my deficient math skills and just some back of the napkin calculations.
Anyway, I wanted to get these two approaches to de-duping out there and will update with more as I keep thinking.
De-duplication is related to the De-identification/Re-identification problem of Anonymized data across multiple data sources (interesting MIT paper here). But in a grid environment the problem becomes tougher because the traditional, really-hard-to-do-solution of checking the various MPIs is even tougher because data sources 1) don't frequently have MPIs and 2) don't want to share them if they do.
Here's the scenario I use to talk about the de-duplication problem set:
The scenario is this:
3 hospitals in Atlanta with overlapping catchment areas. A federated query is sent out for SyndromeX for April 1, 2009. Hospital A returns a count of 10, Hospital B returns a count of 15 and Hospital C returns a count of 20. The combined result from the federated query shows a total of 45 counts of SyndromeX for April 1, 2009.
There exists a question of data quality as it's possible that the same patient visited multiple hospitals and had his SyndromeX recorded at multiple locations. In a federated/grid configuration this is even more probable as Hospitals may use data from multiple feeder sources (labs, small clinics, etc) that may report the same patient encounter multiple times.
Dr. Lenert had a good idea where each Hospital could send over a list of PII (name, address, dob, social) without the medical information for purposes of de-duping. This requires a trusted intermediary to receive these matches and send them on to each Hospital to respond back with number of matches. A hospital cannot send directly to each other hospital because it would reveal its patients to potential competitors.
So the de-duping scenario may change to something like this:
1) User submits query for SyndromeX on April 1, 2009 and selects de-duplication.
2) Trusted intermediary receives the query and passes it on to Hospitals A,B,C.
3) Hospital A returns count of 10, Hospital B returns count of 15, Hospital C returns count of 20. Interim total count 55 is not returned to user.
4) Trusted intermediary then requests patient id info from each hospital in a separate request.
5) Trusted intermediary compares the PII from each hospital (perhaps using a tool like febrl) and determines that there is overlap of patients and actual count. Hospital A shares 1 patient with Hospital B. Hospital A shares 2 patients with Hospital C. Hospital B shares 3 patients with Hospital C. Therefore there are 6 unique duplicates.
6) Total count of 39 is returned to the user (45-1-2-3).
Of course, this is going to require some pretty serious data use agreements with the hospitals and the trusted intermediary to control the storage of PII (i.e. don't persist any PII)
The alternative without the trusted intermediary:
1) User submits query for SyndromeX on April 1, 2009 and selects de-duplication.
2) Each hospital receives the query and list of nodes to which is was submitted.
3) Hospital A identifies a preliminary count of 10. To check for duplicates, it submits only the PII without any condition data and a query ID to Hospital B and C for comparison.
4) Hospital B compares the PII to its own results for query ID and returns the number of matches to Hospital A (duplicate count=1).
5) Hospital C compares the PII to its own results for query ID and returns the number of matches to Hospital A (duplicate count=2).
6) At the same time, Hospitals B&C are performing similar communications with the other nodes participating in the query.
7) Hospital B identifies a preliminary count of 15. To check for duplicates, it submits only the PII without any condition data and a query ID to Hospital A and C for comparison.
8) Hospital A compares the PII to its own results for query ID and returns the number of matches to Hospital B (duplicate count=1).
9) Hospital C compares the PII to its own results for query ID and returns the number of matches to Hospital B (duplicate count=3).
10) Hospital C identifies a preliminary count of 20. To check for duplicates, it submits only the PII without any condition data and a query ID to Hospital A and B for comparison.
11) Hospital A compares the PII to its own results for query ID and returns the number of matches to Hospital C (duplicate count=1).
12) Hospital B compares the PII to its own results for query ID and returns the number of matches to Hospital C (duplicate count=3).
13) Hospital A returns a count of 10 and notes that duplicates with B=1 and duplicates with C=2.
14) Hospital B returns a count of 15 and notes that duplicates with A=1 and duplicates with C=3.
15) Hospital C returns a count of 20 and notes that duplicates with A=2 and duplicates with B=3.
16) The user then feeds these results into another service that processes the results and determines that the actual count is 39 based on the following calculated relationships: (A|B)-1; (A|C)-2; (B|C)-3. These reductions are subtracted from the preliminary total of 45 to yield 39.
Approach #2, while not requiring a trusted intermediary requires a lot of coordination with the queried nodes and also involves the sharing of patient info with other partners that may not be desired. Also, due to set math will grow exponentially with the number of nodes involved in the query (for 3 nodes, the number of service calls is 9 [3 from user to nodes, then 6 among the nodes-using n!/{n-r}!]; but for 4 nodes, the number of service calls is 16; and for 5 nodes the number of service calls is 25). This is also suspect due to my deficient math skills and just some back of the napkin calculations.
Anyway, I wanted to get these two approaches to de-duping out there and will update with more as I keep thinking.
Saturday, April 4, 2009
Friday, April 3, 2009
Updated programming playground rules
I updated the initial blog posting from June 5, 2008 and updated based on additional developments and moved it over to the wiki for easier updating. This will be the place where the NCPHI Lab's and eventually the PHGrid's programming conventions and policies are captured:
- Use Sourceforge Subversion site to store changes - All changes are checked into the Subversion site on the sourceforge project. Only security related changes (passwords, etc) are not checked in, and these are factored out into a property or configuration file.
- Everything builds - Maven is used to build (compile and package) and deploy changes. This will allow for changes to be made on different desktops without spending time trying to manually configure a new environment. This also means that if the build breaks then the developer who broke it needs to fix it as soon as possible.
- Follow the Sun Code Conventions for Java. These are old, but still applicable for what we're trying to do. We picked this sort of as a default and this can change based on feedback and practice, but we want some uniform style to the project.
- Document code- document code to improve clarity. Use javadocs where necessary. This is not documentation for documentation's sake, but enough so that someone coming across the code will be able to follow. This doesn't absolve a programmer from clean, concise coding but should improve on the clarity of a source file.
- Write a Use Case first- the first step to development should be to make a post on this blog describing the use case. Nothing too formal, just a description of the steps, who will perform them, alternate flows and error handing. This will provide a way to capture the requirements before any code is written.
- Next write a Test Case before coding- after the use case is posted and there's some sort of agreement, then a test case is written while the coding is performed. This speeds up development by institutionalizing testing in a standard manner.
- Each project will have a separate folder in the sourceforge root with a Maven pom.xml file for the build (including dependencies).
- All properties are stored in a configuration file to allow for projects to be easily built for various target environments. Actual values are not checked into sourceforge as these property values are likely sensitive (server names, passwords, etc.). For example a project named foo would build based on the properties stored in foo.properties. The file checked into SVN would be named foo.properties.template and contain descriptions for each of the properties, but not values. When a developer is building, they will take the proeprties file as configured for an environment, rename it to foo.properties and run their build. It is likely that you will have foo-dev.properties, foo-training.properties, foo-staging.properties, foo-production.properties (none checked into SVN) and choose a properties file based on where your deployment is targeted.
- All changes are tracked using the SourceForge tracker for Bugs and Feature Requests. This is done to project project transparency to all our changes and prioritization of changes.
- Every Friday, the current project state is built to the training node (ncphi.phgrid.net) so that the community can see progress and comment. Eventually, our CM processes will be run nightly and deployed more frequently than once a week.
- Each evening, before leaving each developer ensures that their changes adhere to these rules and checks them into SVN.
learning grid services.
So right now I am essentially reading the grid book, learning how to make grid services, and learning how to secure grid services. Then, I will be playing with Introduce to see how it makes all this stuff much easier and much more automatic with the help of Felicia.
I also hope to pick Felicia's brain about implementing security using the client.
Either way, Globus services are a lot more than I thought they were, they have a lot more involved for maintaining state and essentially having all of their interactions marshaled and grouped, and it is just crying out for a shiny (if not leaky) abstraction to make it all pretty and less scary to people who aren't programmers used to fighting with Webservices and some of the gooey "I was generated by a machine" code that can come out of them. I look forward to those abstractions, but until then, I am going to be learning a lot of transport layer fun to get secure/amds-based quicksilver running.
Cheers
I also hope to pick Felicia's brain about implementing security using the client.
Either way, Globus services are a lot more than I thought they were, they have a lot more involved for maintaining state and essentially having all of their interactions marshaled and grouped, and it is just crying out for a shiny (if not leaky) abstraction to make it all pretty and less scary to people who aren't programmers used to fighting with Webservices and some of the gooey "I was generated by a machine" code that can come out of them. I look forward to those abstractions, but until then, I am going to be learning a lot of transport layer fun to get secure/amds-based quicksilver running.
Cheers
Thursday, April 2, 2009
Minor deploy, and Introduce
So, the next step for Quicksilver is to make an AMDS-intermediary layer that is a secure Globus service. Thus, tonight I am going to be going over some of the GRID/Globus books again and tomorrow I will probably install introduce and run the tutorial.
The reason I want to read the grid books is so I can see what goes into creating a grid service so that I can see what the introduce client will do for me. Then, I can start better understanding the heuristics of making an intermediary layer for poison data showing up in AMDS web.
Oh yeah, I have to go over the AMDS schema as well.
The end result is that the viewer part of Quicksilver, will end up becoming more of a generic AMDS viewer. Thus, one picks (a) secure service(s) that return(s) AMDS data, fills out the form that it/they require(s) (based on retrieved metadata for selected services), compiles the results into some sort of mappable data, and displays it in the map... and one of those secure AMDS services will be poison data.
One of the concerns I had was that making the NPDS->AMDS converter secure would be completely heuristic... then I remembered that people have to be given logins to be able to access poison data, and it would be much easier if they just were able to access it if they had a node on the grid.
So yeah, big context switch over the next couple of days, expect flagellating expectations as I realize the can, can't, and how.
Also, I ported a bunch of the improvements I made to npdsgmaps-web (the viewer part of quicksilver) back to gmap-poly-web (a demonstration library made on top of Gmap-Polygon). That should be deployed to training tomorrow so the two apps act the same (and because the version currently on training doesn't use encoded polygons, and is therefore much slower and error prone).
Cheers!
The reason I want to read the grid books is so I can see what goes into creating a grid service so that I can see what the introduce client will do for me. Then, I can start better understanding the heuristics of making an intermediary layer for poison data showing up in AMDS web.
Oh yeah, I have to go over the AMDS schema as well.
The end result is that the viewer part of Quicksilver, will end up becoming more of a generic AMDS viewer. Thus, one picks (a) secure service(s) that return(s) AMDS data, fills out the form that it/they require(s) (based on retrieved metadata for selected services), compiles the results into some sort of mappable data, and displays it in the map... and one of those secure AMDS services will be poison data.
One of the concerns I had was that making the NPDS->AMDS converter secure would be completely heuristic... then I remembered that people have to be given logins to be able to access poison data, and it would be much easier if they just were able to access it if they had a node on the grid.
So yeah, big context switch over the next couple of days, expect flagellating expectations as I realize the can, can't, and how.
Also, I ported a bunch of the improvements I made to npdsgmaps-web (the viewer part of quicksilver) back to gmap-poly-web (a demonstration library made on top of Gmap-Polygon). That should be deployed to training tomorrow so the two apps act the same (and because the version currently on training doesn't use encoded polygons, and is therefore much slower and error prone).
Cheers!
Potential PHGrid-related projects for PHI/CS Fellows
Hi all,
Please take a look…and if you think of other ideas….reply back with comments or email the team.
http://sites.google.com/site/phgrid/Home/resource-documents/PHGridtasksforfellows.pdf?attredirects=0
Wednesday, April 1, 2009
Quicksilver Code Complete
The deploy of Quicksilver to training has been complete. I have verified that the other applications are still working, and tested quicksilver on IE, Firefox, and Safari.
It can be reached here
The passwords have been changed to be more complex. Please contact Brian Lee at fya1@cdc.gov for your new password or login if you haven't gotten it.
Quicksilver now uses encoded gmap polygons. This reduces the size of the page sent back to the browser and greatly improves the performance. Internet Explorer no longer throws the "this script is taking a very long time" errors, and the polygons get more complex as you zoom in and less complex as you zoom out. They were encoded using the Douglas-Peucker algorithm described on this site.
The other improvements are tinkerings with the submit button (thus, it becomes disabled and says "loading..." when clicked) positioning improvements and a link to the Wiki help page.
This should be the Code Complete version of Quicksilver that will be deployed to the CDC SDN, and it has been labeled as version 1.0 and branched. New Features and behavior changes will be done to the 1.x branch and deployed to training, while bug fixes will be done to both the 1.x branch and the 1.0 branch and deployed to training and (if allowed) CDC SDN.
Thanks for your patience, I am very excited about Quicksilver and I think it has grown by leaps and bounds in functionality and niftiness from it's poicondai roots.
Otherwise, it looks like the next steps for me are to turn the service end of Quicksilver into an AMDS secure service, and to modify the visual end of Quicksilver (Gmap-polygon) to be flexible enough to combine and display data from multiple AMDS sources.
More immediately, I am going to merge some of the improvements I made for Quicksilver into the gmap-poly-web example project, and work with Felicia to learn the rudiments of a secure AMDS service and to work on making secure service clients coexist happily on the same server.
It can be reached here
The passwords have been changed to be more complex. Please contact Brian Lee at fya1@cdc.gov for your new password or login if you haven't gotten it.
Quicksilver now uses encoded gmap polygons. This reduces the size of the page sent back to the browser and greatly improves the performance. Internet Explorer no longer throws the "this script is taking a very long time" errors, and the polygons get more complex as you zoom in and less complex as you zoom out. They were encoded using the Douglas-Peucker algorithm described on this site.
The other improvements are tinkerings with the submit button (thus, it becomes disabled and says "loading..." when clicked) positioning improvements and a link to the Wiki help page.
This should be the Code Complete version of Quicksilver that will be deployed to the CDC SDN, and it has been labeled as version 1.0 and branched. New Features and behavior changes will be done to the 1.x branch and deployed to training, while bug fixes will be done to both the 1.x branch and the 1.0 branch and deployed to training and (if allowed) CDC SDN.
Thanks for your patience, I am very excited about Quicksilver and I think it has grown by leaps and bounds in functionality and niftiness from it's poicondai roots.
Otherwise, it looks like the next steps for me are to turn the service end of Quicksilver into an AMDS secure service, and to modify the visual end of Quicksilver (Gmap-polygon) to be flexible enough to combine and display data from multiple AMDS sources.
More immediately, I am going to merge some of the improvements I made for Quicksilver into the gmap-poly-web example project, and work with Felicia to learn the rudiments of a secure AMDS service and to work on making secure service clients coexist happily on the same server.
Once-secret 'cloud manifesto' sees light of day
http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9130706&source=NLT_PM
KMWorld Article: Tackling extreme data volume
Tackling extreme data volume
By Hugh McKellar - Posted Apr 1, 2009
We all know of the imperative of managing the extreme—and sometimes debilitating—volume of data in today’s organizations. In early March, Digital Reef emerged from two years of stealth mode to tackle that problem by introducing what it claims is a new approach to discovering and managing unstructured and semi-structured data.
The company says its massively scalable platform is designed to address critical business issues—such as e-discovery, data risk mitigation, knowledge reuse and strategic storage initiatives—that aren’t currently properly addressed using traditional solutions. Digital Reef says that as the volume of data expands unchecked, enterprises take on more expense and more risk.
"In the sea of solutions focusing on e-discovery and enterprise search, Digital Reef uniquely understands the requirements of data center infrastructure and contextual content management," says Tony Asaro, founder and senior consultant, The INI Group (http://contemplatingit.com). "This combination sets Digital Reef apart from other players. From a data center perspective, Digital Reef’s solution uses grid architecture to meet the performance requirements for indexing massive amounts of data.
"[Digital Reef] has a truly federated view of all of the data regardless of physical location, and the solution transparently works with all file systems from NTFS to WAFL to ZFS. From a content management perspective, Digital Reef combines intelligent keyword search with its unique similarity engine to enable users to efficiently access the appropriate and relevant information they seek."
Digital Reef says its namesake platform allows customers to: locate specific kinds of data, including sensitive data like Social Security and credit card numbers; identify regulated data for compliance; pinpoint relevant documents for pending legal action; and find intellectual property that can be reused for competitive advantage.
By Hugh McKellar - Posted Apr 1, 2009
We all know of the imperative of managing the extreme—and sometimes debilitating—volume of data in today’s organizations. In early March, Digital Reef emerged from two years of stealth mode to tackle that problem by introducing what it claims is a new approach to discovering and managing unstructured and semi-structured data.
The company says its massively scalable platform is designed to address critical business issues—such as e-discovery, data risk mitigation, knowledge reuse and strategic storage initiatives—that aren’t currently properly addressed using traditional solutions. Digital Reef says that as the volume of data expands unchecked, enterprises take on more expense and more risk.
"In the sea of solutions focusing on e-discovery and enterprise search, Digital Reef uniquely understands the requirements of data center infrastructure and contextual content management," says Tony Asaro, founder and senior consultant, The INI Group (http://contemplatingit.com). "This combination sets Digital Reef apart from other players. From a data center perspective, Digital Reef’s solution uses grid architecture to meet the performance requirements for indexing massive amounts of data.
"[Digital Reef] has a truly federated view of all of the data regardless of physical location, and the solution transparently works with all file systems from NTFS to WAFL to ZFS. From a content management perspective, Digital Reef combines intelligent keyword search with its unique similarity engine to enable users to efficiently access the appropriate and relevant information they seek."
Digital Reef says its namesake platform allows customers to: locate specific kinds of data, including sensitive data like Social Security and credit card numbers; identify regulated data for compliance; pinpoint relevant documents for pending legal action; and find intellectual property that can be reused for competitive advantage.
Subscribe to:
Posts (Atom)
Posts
