After Tom received some feedback from Farzad Mostashari, we changed the AMDS schema a bit to make it more usable.
Now Level 1 has:
Syndrome
Syndrome classifier
Age Group (0<2;2<5;5<18;18<54;54<65; 65+)
Patient State
Patient Zip (either zip3 or zip5)
Date Start
Date End (in at least 1 day increments)
Count
We removed denominator as that's not really necessary (since you can query for all syndromes).
Level 2 (not developed yet):
Gender
Race
Ethnicity
Detailed Geography
Also, Dr. Lenert created different stages of access support for AMDS:
Stage 0: Scheduled transfers of AMDS documents to a supernode/central database
Stage 1: AMDS data available using secure grid services
Stage 2: AMDS data de-duped via secure grid services
Stage 3: AMDS data de-duped and combined together from multiple nodes (although after Stage 1, results can be combined together)
Wednesday, January 7, 2009
Update on Collaboration with SatScan on Cloud
Ron Price and I chatted yesterday about some progress that he has made with the SatScan on the cloud service;
I have build a simulation dataset with 53,018 random points within NC, SC, and GA that is based upon a RODS dataset that was provided to me.
I am working with a statistician in NCZVED to write scripts to automatically generate SatScan parameter files (.prm) as well as case files (.cas), population files (.pop) and coordinate files (.geo) that would all be used as inputs
to the SatScan batch mode (for the grid service).
Also, I have emailed the CCID TB group several singular genotype spatial cluster maps as examples as well as built one web mapping service to visualize spatial clusters. It is my hope that Ron and I can collaborate on a demo of the grid service and simulated data to reignite the interest of the CCID TB epidemiologists.
Jim Tobias
I have build a simulation dataset with 53,018 random points within NC, SC, and GA that is based upon a RODS dataset that was provided to me.
I am working with a statistician in NCZVED to write scripts to automatically generate SatScan parameter files (.prm) as well as case files (.cas), population files (.pop) and coordinate files (.geo) that would all be used as inputs
to the SatScan batch mode (for the grid service).
Also, I have emailed the CCID TB group several singular genotype spatial cluster maps as examples as well as built one web mapping service to visualize spatial clusters. It is my hope that Ron and I can collaborate on a demo of the grid service and simulated data to reignite the interest of the CCID TB epidemiologists.
Jim Tobias
R and Rweb
R is an open source statistical software package: http://www.r-project.org/
R is used extensively by epidemiologists and geographers.
One can download R (free): http://cran.cnr.berkeley.edu/
The base install of R can be extended with R packages.
A fairly comprehensive list of R packages is here: http://cran.r-project.org/web/packages/
The R software is level III approved for use at the CDC.
The R software runs on your desktop but the R code can be run through a web service.
An example of an R web service is RWeb: http://www.math.montana.edu/Rweb/
Specific packages such as EpiTools have been developed by UC Berkeley and they support full epidemiology coursework with R.
If you have questions or wish to see this in action...just stop by;
Jim Tobias
R is used extensively by epidemiologists and geographers.
One can download R (free): http://cran.cnr.berkeley.edu/
The base install of R can be extended with R packages.
A fairly comprehensive list of R packages is here: http://cran.r-project.org/web/packages/
The R software is level III approved for use at the CDC.
The R software runs on your desktop but the R code can be run through a web service.
An example of an R web service is RWeb: http://www.math.montana.edu/Rweb/
Specific packages such as EpiTools have been developed by UC Berkeley and they support full epidemiology coursework with R.
If you have questions or wish to see this in action...just stop by;
Jim Tobias
Tuesday, January 6, 2009
package changes and SVN
So, most of this morning was spent doing some data file massaging into CSVs, which were then loaded into the postgresql database.
It was sort of humbling, because my prior approaches were to build a big parser library and then spend another day or two debugging it. In this case, I just took the files, got a macro on them, and them told the database to import them. The result is I spent just 2/3rd of a day getting all the data I wanted in the right spot. The thing that will be bad is I'll have to do it all again (well, probably spend 1/3 of a day) when database updates need to be made, then again, I probably would have to tweak things in my loader scripts if they needed to be made, and it's still the same cost to the end user... and in fact probably less. Before it was download files, download code, adjust properties, package, build, run. Now it's download CSV, create table in database, import CSV into table in database. Yeah. It's easier now.
Otherwise, there are lists of relational data (all the zip3s in Nevada, all the zip5s in a given zip3, etc). Now I get to start building the code libraries that will serve it up as needed. Before I did that I had to do a package update... which involves changing one line in a lot of places.
Luckily eclipse handled the refactoring for me beautifully, and it terribly broke the subversion management of it all. Thus, I spent the rest of the day deleting out the appropriate files and committing the changes by hand.
Tomorrow is the implementation of the lists, and updating documentation for the loader and the list-libraries and, well, everything else.
It was sort of humbling, because my prior approaches were to build a big parser library and then spend another day or two debugging it. In this case, I just took the files, got a macro on them, and them told the database to import them. The result is I spent just 2/3rd of a day getting all the data I wanted in the right spot. The thing that will be bad is I'll have to do it all again (well, probably spend 1/3 of a day) when database updates need to be made, then again, I probably would have to tweak things in my loader scripts if they needed to be made, and it's still the same cost to the end user... and in fact probably less. Before it was download files, download code, adjust properties, package, build, run. Now it's download CSV, create table in database, import CSV into table in database. Yeah. It's easier now.
Otherwise, there are lists of relational data (all the zip3s in Nevada, all the zip5s in a given zip3, etc). Now I get to start building the code libraries that will serve it up as needed. Before I did that I had to do a package update... which involves changing one line in a lot of places.
Luckily eclipse handled the refactoring for me beautifully, and it terribly broke the subversion management of it all. Thus, I spent the rest of the day deleting out the appropriate files and committing the changes by hand.
Tomorrow is the implementation of the lists, and updating documentation for the loader and the list-libraries and, well, everything else.
Dallas Network Issues
- The Tarrant node is back online. The root cause of the problem was the VM failed to restart after a host server reboot. The VM was manually restarted successfully.
- The Dallas node is experiencing slow response times due to network issues that are currently being addressed by the Dallas network team. They are in the process of investigating the root cause of the issue. This issue currently affects the RPDSAdai demo and any data transfers to and from the Dallas node. I was informed that this issue would be corrected by close of business on 01-06-2009.
Link Plus: From CDC's National Program of Cancer Registries (NPCR)
Link Plus is a probabilistic record linkage program developed at CDC's Division of Cancer Prevention and Control in support of CDC's National Program of Cancer Registries (NPCR). Link Plus is a record linkage tool for cancer registries. It is an easy-to-use, standalone application for Microsoft® Windows® that can run in two modes: to detect duplicates in a cancer registry database, or to link a cancer registry file with external files. Although originally designed to be used by cancer registries, the program can be used with any type of data in fixed width or delimited format. Used extensively across a diversity of research disciplines, Link Plus is rapidly becoming an essential linkage tool for researchers and organizations that maintain public health data. http://www.cdc.gov/cancer/NPCR/tools/registryplus/lp.htm
Download Application Here:
ftp://ftp.cdc.gov/pub/Software/RegistryPlus/Link_Plus/RPLinkPLus-2.0.exe
Features
http://www.cdc.gov/cancer/NPCR/tools/registryplus/lp_features.htm
Download Application Here:
ftp://ftp.cdc.gov/pub/Software/RegistryPlus/Link_Plus/RPLinkPLus-2.0.exe
Features
http://www.cdc.gov/cancer/NPCR/tools/registryplus/lp_features.htm
Febrl - An Open Source Data Cleaning, Deduplication and Record Linkage System with a Graphical User Interface
One of the key services that will eventually have to be provided on the grid, given the vast amounts of data to be integrated, is one that focuses on Data Cleaning, Deduplication and Record Linkage. This domain has been a significant challenge for years- even off the grid. Clearly, this is a very important and exciting area of research.
I found out about Febrl yesterday. It has been developed since 2002 as part of a collaborative research project conducted between the Australian National University in Canberra and the New South Wales Department of Health in Sydney, Australia.
The group is working on Parallel Large Scale Techniques for High-Performance Record Linkage
Link to their efforts: http://datamining.anu.edu.au/linkage.html
The NCPHI Grid team is going to investigate this - and find out what, if any, aspects of Febrl might be leveraged to help provide this service on the PHGrid.
Note: It requires Python to be installed.
Here are two papers on Febrl.
http://cs.anu.edu.au/~Peter.Christen/publications/kdd2008christen-febrl-demo.pdf
http://crpit.com/confpapers/CRPITV80Christen.pdf
Here is the sourceforge link to Febrl
http://sourceforge.net/projects/febrl
Features include:
* Probabilistic and rules-based cleaning and standardisation routines for names, addresses, dates and telephone numbers.
* A geocoding matching system based on the Australian G-NAF (Geocoded National Address File) database.
* A variety of supplied look-up and frequency tables for names and addresses.
* Various comparison functions for names, addresses, dates and localities, including approximate string comparisons, phonetic encodings, geographical distance comparisons, and time and age comparisons. Two new approximate string comparison methods (bag distance and compression based) have been added in this release.
* Several blocking (indexing) methods, including the traditional compound key blocking used in many record linkage programs.
* Probabilistic record linkage routines based on the classical Fellegi and Sunter approach, as well as a 'flexible classifier' that allows a flexible definition of the weight calculation.
* Process indicators that give estimations of remaining processing times.
* Access methods for fixed format and comma-separated value (CSV) text files, as well as SQL databases (MySQL and new PostgreSQL).
* Efficient temporary direct random access data set based on the Berkeley database library.
* Possibility to save linkage and deduplication results into a comma-separated value (CSV) text file (new).
* One-to-one assignment procedure for linked record pairs based on the 'Auction' algorithm.
* Supports parallelism for higher performance on parallel plat- forms, based on MPI (Message Passing Interface), a standard for parallel programming, and Pypar, an efficient and easy-to-use module that allows Python programs to run in parallel on multiple processors and communicate using MPI.
* A data set generator which allows the creation of data sets of randomly generated records (containing names, addresses, dates, and phone and identifier numbers), with the possibility to include duplicate records with randomly introduced modifications. This allows for easy testing and evaluation of linkage (deduplication) processes.
* Example project modules and example data sets allowing simple running of Febrl projects without any modifications needed.
-----
An Evaluation of Febrl
http://www.qsinc.com/dedup/results_FEBRL.html
Conclusion of Evaluation:
Febrl is an excellent research tool to study the deduplication process. It provides refinements in record standardization, geocoding and field matching. It provides flexibility in how data is selected for scoring, how record fields are compared, and how the untimate score is calculated. Febrl is also useful as an outboard record standardizer, of particular interest where names, addresses and phone numbers are not already separated into component parts.
It became clear, as we understood the inherent flexibility of Febrl that with sufficient effort, we could have used our long experience in deduplication to improve our results. It's all in the configuration.
However, we couldn't think of any way to deal effectively with twins. Probabilistic record linking schemes that rely solely on aggregating scores of individual field comparisons do not perform well on data with twins. Febrl couldn't distinguish between a true match with data errors and a twin. It didn't "understand" data in context.
Febrl places considerable resources at the researcher's command. It provides a deduplication laboratory allowing the experimenter lots of knobs to turn in order to arrive at a deduplication solution. But a laboratory is not a factory. We found ourselves "trying just one more thing" over and over again in the interests of fairness for this comparison with no way to decide when enough is enough. With endless opportunities for tinkering, adjustment and trying "just one more thing", it's an hourly consultant's paradise. For this reason, look very carefully at any consultant's recommendation to use Febrl.
While Febrl is free open source software, we believe that the actual cost of moving it from the laboratory into production could be significant. Its inability to deal with twins is a fatal flaw.
I found out about Febrl yesterday. It has been developed since 2002 as part of a collaborative research project conducted between the Australian National University in Canberra and the New South Wales Department of Health in Sydney, Australia.
The group is working on Parallel Large Scale Techniques for High-Performance Record Linkage
Link to their efforts: http://datamining.anu.edu.au/linkage.html
The NCPHI Grid team is going to investigate this - and find out what, if any, aspects of Febrl might be leveraged to help provide this service on the PHGrid.
Note: It requires Python to be installed.
Here are two papers on Febrl.
http://cs.anu.edu.au/~Peter.Christen/publications/kdd2008christen-febrl-demo.pdf
http://crpit.com/confpapers/CRPITV80Christen.pdf
Here is the sourceforge link to Febrl
http://sourceforge.net/projects/febrl
Features include:
* Probabilistic and rules-based cleaning and standardisation routines for names, addresses, dates and telephone numbers.
* A geocoding matching system based on the Australian G-NAF (Geocoded National Address File) database.
* A variety of supplied look-up and frequency tables for names and addresses.
* Various comparison functions for names, addresses, dates and localities, including approximate string comparisons, phonetic encodings, geographical distance comparisons, and time and age comparisons. Two new approximate string comparison methods (bag distance and compression based) have been added in this release.
* Several blocking (indexing) methods, including the traditional compound key blocking used in many record linkage programs.
* Probabilistic record linkage routines based on the classical Fellegi and Sunter approach, as well as a 'flexible classifier' that allows a flexible definition of the weight calculation.
* Process indicators that give estimations of remaining processing times.
* Access methods for fixed format and comma-separated value (CSV) text files, as well as SQL databases (MySQL and new PostgreSQL).
* Efficient temporary direct random access data set based on the Berkeley database library.
* Possibility to save linkage and deduplication results into a comma-separated value (CSV) text file (new).
* One-to-one assignment procedure for linked record pairs based on the 'Auction' algorithm.
* Supports parallelism for higher performance on parallel plat- forms, based on MPI (Message Passing Interface), a standard for parallel programming, and Pypar, an efficient and easy-to-use module that allows Python programs to run in parallel on multiple processors and communicate using MPI.
* A data set generator which allows the creation of data sets of randomly generated records (containing names, addresses, dates, and phone and identifier numbers), with the possibility to include duplicate records with randomly introduced modifications. This allows for easy testing and evaluation of linkage (deduplication) processes.
* Example project modules and example data sets allowing simple running of Febrl projects without any modifications needed.
-----
An Evaluation of Febrl
http://www.qsinc.com/dedup/results_FEBRL.html
Conclusion of Evaluation:
Febrl is an excellent research tool to study the deduplication process. It provides refinements in record standardization, geocoding and field matching. It provides flexibility in how data is selected for scoring, how record fields are compared, and how the untimate score is calculated. Febrl is also useful as an outboard record standardizer, of particular interest where names, addresses and phone numbers are not already separated into component parts.
It became clear, as we understood the inherent flexibility of Febrl that with sufficient effort, we could have used our long experience in deduplication to improve our results. It's all in the configuration.
However, we couldn't think of any way to deal effectively with twins. Probabilistic record linking schemes that rely solely on aggregating scores of individual field comparisons do not perform well on data with twins. Febrl couldn't distinguish between a true match with data errors and a twin. It didn't "understand" data in context.
Febrl places considerable resources at the researcher's command. It provides a deduplication laboratory allowing the experimenter lots of knobs to turn in order to arrive at a deduplication solution. But a laboratory is not a factory. We found ourselves "trying just one more thing" over and over again in the interests of fairness for this comparison with no way to decide when enough is enough. With endless opportunities for tinkering, adjustment and trying "just one more thing", it's an hourly consultant's paradise. For this reason, look very carefully at any consultant's recommendation to use Febrl.
While Febrl is free open source software, we believe that the actual cost of moving it from the laboratory into production could be significant. Its inability to deal with twins is a fatal flaw.
Monday, January 5, 2009
Setting up the locational library.
Well, right before I went off for the holidays, I completed the part of the gmap polygon libraries that would easily provide javascript for an array of polygons.
Now, I am in the process of building lists so that any application that wants to provide locations will be able to get lists of states, zip3's in said states, and zip5s in said zip3s. That, and improving the graphic awareness to be able to center on given areas after they are selected.
This of course means massaging large KML lists, and discussions on what would be most useful to someone trying to build their own databases on their own systems. Right now it seems the best answer would be "Comma delimited files", considering every database has a way to very quickly ingest such files.
I also am happy to have gotten some examples of text massaging using macros, and am once again making sure to earmark some time in the new year for basic scripting niftiness, namely because there are just some times when creative uses of grep and regular expressions will do so much more with a file than trying to create a SAX parser.
Now, I am in the process of building lists so that any application that wants to provide locations will be able to get lists of states, zip3's in said states, and zip5s in said zip3s. That, and improving the graphic awareness to be able to center on given areas after they are selected.
This of course means massaging large KML lists, and discussions on what would be most useful to someone trying to build their own databases on their own systems. Right now it seems the best answer would be "Comma delimited files", considering every database has a way to very quickly ingest such files.
I also am happy to have gotten some examples of text massaging using macros, and am once again making sure to earmark some time in the new year for basic scripting niftiness, namely because there are just some times when creative uses of grep and regular expressions will do so much more with a file than trying to create a SAX parser.
Friday, January 2, 2009
NIH Webcast - Transforming Science: Cancer Control on caBIG
Videocast of the November 10th 2008 Forum:
http://videocast.nih.gov/Summary.asp?File=14755.
Special thanks to Abdul R Shaikh, PhD, MHSc, Behavioral Scientist Program Director, Health Communication and Informatics Research Branch, Behavioral Research Program, Division of Cancer Control and Population Sciences, National Cancer Institute.
FIY - the NCI is moving forward to launch a division web site on grid-enabled research in population health.
http://videocast.nih.gov/Summary.asp?File=14755.
Special thanks to Abdul R Shaikh, PhD, MHSc, Behavioral Scientist Program Director, Health Communication and Informatics Research Branch, Behavioral Research Program, Division of Cancer Control and Population Sciences, National Cancer Institute.
FIY - the NCI is moving forward to launch a division web site on grid-enabled research in population health.
Thursday, January 1, 2009
HealthGrid 2009 conference Berlin - Call for Papers
KEY DATES
Call for papers opens: 18 December 2008
Call for papers closes: 1 February 2009
Registration opens: 1 February 2009
More information about submission:
http://berlin2009.healthgrid.org/fileadmin/templates/download/HealthGRID2009_Call-for-Papers.pdf
Subscribe to:
Posts (Atom)
Posts
