Conflicting PostgreSQL drivers

While deploying the AMDSService (soon to be renamed to GIPSEService) to the training node, we noticed this error:
"SET AUTOCOMMIT TO OFF is no longer supported" being thrown by the AMDSService operations.

This was caused because the training node uses PostgreSQL 8.1 yet the globus_database_common package deploys the JDBC driver for PostgreSQL 7.3 (pg73jdbc2.jar). This causes a problem with the way that ibatis handles postgres connections (specifically ibatis turns autocommit off to maximize performance).

The AMDSService includes the postgreSQL 8 driver (postgresql-8.3-604.jdbc3.jar). To get around this error, we removed the 7.3 driver from the training node's Globus and the training node's Tomcat. Globus is happy using the updated driver, AMDSService is happy using the updated driver and RODSA-DAI is still happy using the updated driver.

This is not a problem for WS-Core Globus installs (like our Windows nodes) since WS-Core doesn't include the 7.3 driver.

Something to keep in mind for any Linux PHGrid nodes that use AMDSService is that they will need to remove the 7.3 postgres driver after installing Globus and the AMDSService. (This is accomplished by renaming $GLOBUS_LOCATION\lib\pg73jdbc2.jar to pg73jdbc2.jar.old; then redeploying Globus to tomcat).

GIPSE Loader

I added the top 30 BioSense Sub-Syndromes (as determined by Roseanne and Jerry) to the amds-db project and tested a load using the past 40 days of sub-syndrome RT data. This is increasing the size of the test data set that we use for testing the GIPSE services.

PS- In case you haven't noticed from Tom's massive renaming of all the AMDS-related wiki pages, AMDS has been renamed to Geocoded Interoperable Population Summary Exchange (GIPSE) by the NCPHI Director. So whenever you see GIPSE think AMDS.

AMDS Service Beta updates

I worked with the CSC/NEDSS programmer to test out their usage of AMDSService. I left out some of the boilerplate jars, but after adding those they were able to configure, build and successfully test out the AMDSService running on their local node. So this is progress.

New box!

So, a large portion of today was spent moving from the Linux development station I am used-to to a new Windows development station I am not-as-used-to.

Some of the things were easier to set up. The biggest help was being able to share other windows development boxes and being able to nab their already-downloaded copies of Java, eclipse, and Globus-ws core files. The setup for globus also seems to be a lot snappier in Windows (but at the same time, I am doing a much less involved install, and this isn't the first time I've done it).

Otherwise, I was able to build and run the client from the AMDS Service that was pulled over from another computer. Now I need to build and run the client from a fresh client downloaded from SVN and configured myself... so I anticipate a few "I am not sure what this property is" issues, but then it will be configuring the client, building the jars that grid viewer needs on my box (gmap-polygon) and figuring out what is needed between those steps.

BioSense AMDS Service - Beta Release

The AMDS Service for BioSense is now in beta release. Please view details of how to download,configure, build, deploy and use on the service registry page.

You can also download the raw gar from sourceforge. But I recommend getting the source and building with your own configuration.

This service uses the updated 5/31 AMDS draft.

This service is specifically developed to share BioSense aggregate data over PHGrid, but can actually be used for any JDBC data source that wants to be shared using the AMDS spec.

This release is significantly different than the 4/30 alpha release. Specifically we're using Introduce 1.3 (big improvement over 1.2) for service development and configuration management and iBATIS for easy db access / ORM. This release is smaller than the alpha release in size and lines of code so theoretically it will be easier to use. Please let me know any comments. We'll be following the weekly build schedule with a target of July 8 for code freeze.

Update: Tom asked me to explain that we're using iBATIS rather than Hibernate. Both are decent JDBC/ORM frameworks. I chose iBATIS because it has a lighter footprint and I got it working in about 15 minutes. This doesn't mean we won't use Hibernate in the future, but just that we're using iBATIS for now.

AMDS Draft Schema Work

After some productive conversations with Bill and Ian from the University of Washington CoE, I've made some changes to the forthcoming May 31 draft of the AMDS schema. The changes are checked into SVN and can be viewed here (schema), here (metadata response example), here (request example), here (response example).

Please take a look and let me know your comments. This does a few things:
1) It flattens the xml structure as much as possible. Simplicity is good.
2) It allows for more fine-grained geographic regions to be selected. We now have wildcard (*) and parent qualifiers so you can stratify your result however is appropriate.
3) It adds additional, optional stratifiers of Age, Facility and Service Area (called "bucket" by BioSense). These optional stratifiers won't be supported by all AMDS services/providers/publishers but they are now in the schema to allow for it. DiSTRIBuTE supports these stratifiers.
4) Renames "condition" to "indicator" since AMDS can really be used to query and report any population indicator as definied by the AMDS provider.

I plan on using this schema for the service operations on the AMDS-BioSense beta service (due May 31).

One thing that is becoming apparent is that we will need a good registry / registry services to capture all the metadata for the indicator classifiers, age classifiers, service area classifiers, facility classifiers, etc. that are used by all of the AMDS services. I'm noting this now, but it won't be built for a while (for the typical reasons).

Finally, I'll add that this is a very DRAFT schema and will likely be updated again between now and PHIN (especially based on your comments). We will also change a lot once the science side of the house determines what standards we need to meet (Ken is working on this with the epidemiologists).

AMDS Technical updates

I added a project called db-importer that can be used to import CSV (and eventually xml) AMDS extracts into an AMDS Store database based on the schemas we're using at CDC. This isn't the fastest or best importer, but it works on MS SQLServer and PostgreSQL and will be used to bring in daily report extracts from systems like BioSense.

I also talked a bit with Vaughn to transition over his projects. I'll be fixing up / re-writing AMDSCore and AMDSPoison over the next few weeks in time for the PHIN Conference until we find another Java developer.

Finally, we had a productive meeting with the PHIN Messaging steward and team about ways to directly collaborate in the solution development of PHGrid, PHINMS and PHIN-SRM. Nothing major was covered, but lots of good ideas that we'll follow up on in the near future.

AMDS Data Store

I checked in the first batch of database definitions for the AMDS Data Store to SVN. This is basically the DDL for all the tables, views, triggers, etc. that we're using for the AMDS Store that will be populated with, initially, synthetic BioSense data. This will help out anyone else who is looking to build their own aggregate store.

I'm also starting a list of conditions and classifiers that we'll use internall in the AMDS Store but will be useful to any others developing AMDS services.

Eventually this will have DDL for postgres, any other dbs we can create scripts for by request.

AMDS Architecture clarification

After talking with Tom, Barry, John and Ken I thought it would be useful to clarify some of the ideas we have around the AMDS architecture. You can view the full page on the wiki, but I'll describe the concepts a little here.

Option #1 for AMDS is what we've typically be dealing with. A grid node is installed as a publisher and the publisher runs an AMDS service to query the biosurveillance database and return the counts based on the query. This requires hosting a service and getting your IT organization to open up port 443 for internal connections.


Option #2 for AMDS is what we're calling the producer-collector scenario. Some partners cannot/will not host a service. To accomodate this, they generate AMDS reports containing only aggregate data and transfer it to a collector node. CDC doesn't want to be the collector node, but they may need to be to demonstrate the capability. These periodic reports are received and loaded into an AMDS store.


And if we put these both together, we see that producers send data to collectors, collectors then store the data in a database where a publisher can access it. Of course, there will be publishers that connect directly to biosurveillance data sources.

Gar issues...

Today Vaughn and I tried to get the client for the amds service on my machine so we could develop against it. It didn't work.

First we tried to build his code on my machine, which failed because my machine is not his machine and therefore had a slightly different version of globus... or ant... or user setup... or some other variable that is yet to be discovered that usually pops up during the first couple of attempts like this.

Then we tried to bring over the gar he had built on his machine... and deploy it... only to have the client blow up whenever I tried to run it citing missing class exceptions... despite the fact that all the classes that should have been in the gar.

So, I am still wanting for an AMDS client, and the resulting checklist of code and build and configurion points is long and mildly upsetting. I'm wondering if making a quick little AXIS or CXF service would be worth it just so I could develop against the resulting classes and client examples while the Globus connections were ironed out.

I'm sure it will get sorted out eventually, but when I was working with Quicksilver, I had the benefit of having the service from day one. This time, I have build out a lot of the grid viewer on assumptions of how the client and service will behave, and I worry about increasing the chance of needing to refactor things once it gets booting and connected because my assumptions were off.

Oh well, growing pains. We'll get through it... and I'm sure what we make will be rather cool. It's just "that phase" in the project where things are going wacky all over the place when one was hoping they would just fall into place and it's extra difficult to take the calming breaths you need to collectedly analyze the situation.

Updated AMDS Schema + web app

I updated the AMDS schema with a new version in preparation for the release of AMDS-BioSense's Alpha services tomorrow.

I'm sure Vaughn will tell you about the details. Basically, the service tool we're using didn't like referenced types, so I remade the schema to use inline defined elements. I know this is bad form, but it's what we had to do to get the tool to work. If anyone can generate a WSDL file with Introduce using the March 30 schema we'll be very grateful. Otherwise, the example XML didn't change at all so documents valid under the March 30 schema are still valid under the April 30 schema.

Other changes are we changed all our xs:date and xs:dateTime elements to be xs:string, again to make it easier for the AMDS-BioSense Alpha services. Axis1.4 didn't really like date objects very much.

Finally, I added an optional "AllCount" sibling element to "Count" so that services that want to explicitly return the denominator can do so. This was added at the suggestion of the Harvard ESP:SS folks as their data provider doesn't actually return the counts for every single condition type so a query for Classifier=ALL would not actually return the denominator.

I also created a sourceforge project for the schemas so 1) we can do some more detailed version control than the wiki provides and 2) others can host the schemas if they so desire.

gridviewer is up

Greetings all,

So, this morning I posted gridviewer to the training node. Right now it is an empty shell being prepared for data and metadata, so it behaves a lot like the gmap-poly-web application, in that there is no data so the polygons will always show zero counts.

you can see it at http://ncphi.phgrid.net:8080/gridviewer/gmap-pane.jsp

The next steps are to set up a framework for handling multiple servers and metadata, reflect that capability in the framework by showing a changing server-list depending on which regions/time-periods were selected, and prepare for integrating Vaughn's AMDS client so that data can be fetched.

I am hoping to get to a "rodsadai" like state (click server A, get some data, click server B, get other data, click both, get combined data) for next Friday's release.

Cheers, and have a good weekend!

AMDS-UI

So, AMDS-UI is going to be broken into three parts, and the structure will be very similar to Quicksilver but will be replicated because the two apps will be doing two very different things.

Amdsmulticlient will be the main model generator. It will contact AMDS clients with queries(securely or non-securely, depending on the URL), and deal with marshalling data into objects which can be understood by...

Amdsgmap which is the main controller. It will take the data objects and convert them into data better understood by gmap-polygons (and will be using the gmap-polygon jar). It will pack up the data into gmap-polygons through a suite of JSP backing code for...

Amdsgmap-UI, which is the main view. This will take data from amdsgmap and display it and handle the requests and responses to the actual viewer. It will also hopefully have some admin pages for adding and removing AMDS sites and configuring some of the properties.

The look should be a lot like quicksilver, but the options will have shifted, and some of the features wanted for quicksilver will be provided by for amds-web.

I have created all these projects, and tomorrow I will extend the code and set mockups for handling data. Soon after that will be integrating Vaughns client, and then many cycles of testing and coding.

My hope is by the end of the week to have Quicksilver-esque functionality coming from AMDS test data.

Service Philosphy

Last week I worked on merging the service and the core functionality of AMDS. The bulk of that effort was spent trying to get the GDTE tool to call the specific JaxBElements as specified by the AMDS XSD. At one point I thought I had this licked but after some initial testing I realized that there was some additional issues that would take even longer to resolve. So for now, I shelved trying to get the service to work in that fashion and just created manual marshal and unmarshal routines that would get the job done. I guess for the programmer that likes to just look at an interface and begin extending through raw guts, intuition and instinct, it forces them to read the README file before moving forward. I can talk about those type of programmers because I am guilty of the same. :-)

The entire code check in process took place over the weekend. I wanted to wrap my head around a directory structure for services going forward so that it can be easy for someone who wants to download the code and build. So through my thinking I came up with the idea that a service should have the following directory structure:

+--ServiceProjectName:

• +--Service
• +--Common
• +--Client
• +--lib
• --build.xml
• --README.txt
• --POM.xml

This structure follows my philosophy for grid service development which follows:

A grid service should have 3 components:

1. The core or common components - these components do all of the work for the service from making database calls, structuring inputs and outputs, etc.
2. Service Implementation -Since there are a couple of techniques for creating Globus services its interface should be implemented from the core components making the service implementation very thin. I have based my service implementation on GDTE which is a more generic service implementation than Introduce. However, if someone wanted to implement a service using Introduce then the common components are preserved in such a way that this is very easily done. Additionally, configuration file location and environment variables are accessible in locations as prescribed by the GTK4 development manuals.
3. Client Implementation - the client implementation is based on the service implementation. In my TODO list I am going to try and make this more generic and base the client off of the WSDL GDTE tool generates a client stub, I will use it as a base. I will transform it into something generic later.

Included in my service is a GUI for the service configuration. A snap shot is below:


The AMDS configuration is fairly complex and since this service will be moving to production, I wanted to make it as easy as possible to administer from a UI as oppose to just having a configuration file. From the UI it is easily seen that its design is intended to manage a suite of AMDS services at one node. This taps into my ideas about the service ecosystem: The service should be easy to extend/develop, to deploy/undeploy, management/configure, and include a default usable client. The intended audience for this targeted functionality is not just for developers but also for the non-technical person when using the service. These specific ideas follow the IPhone App-Store Model of application development, a topic to go into detail later.

Lastly, here are my TODOs for this week:

• Complete the generic configuration of the AMDS service.
• Build the default client to be used with the AMDS Service
• Begin to think about multi-node access for the client (The client should be able to aggregate multiple node endpoints running the AMDS service)
• Upload the updated service and remaining artifacts to source control, with a base level instruction set.
  

More to come...

De-duplication Insight from the Field

After Monday's post Dave Heinbaugh, the tech manager for the Tarrant County APC, was kind enough to spend some time with me on the phone walking through some of the scenarios for de-duplication.

He basically broke it down that based on the statistical analysis, it's really not worth trying to account for a single patient who gets served by multiple hospitals or clinics. Based on Dave's experience near the data, this does not happen.

So, de-duplication is an issue for NHIN where they are trying to figure out if 5 different EMR/EHRs are the same person, but for AMDS it may not be as an immediate, data quality issue as feared.

This doesn't get rid of the question of de-duplication, but shifts it a bit from trying to reduce specific instances of patients recorded multiple times to trying to make sure that a data source is not reported multiple times (for example Tarrant County is a BioSense data source so if you queried BioSense-AMDS and Tarrant-AMDS services for Respiratory conditions and combined them together you would get double counts).

AMDS Store for BioSense data

So part of the BioSense program strategic plan is to share national data sources in a federated manner. In order to support this using grid and AMDS, we've started working with the BioSense Data Quality team (they run the analysis on the Data Mart) to generate some sample files that we can use to populate an AMDS data store with all the aggregate counts of BioSense syndromes by zip code and by day.

So far, we've got a sample file for all the zip codes covered by VA, DoD and Real-Time data. Of course all the counts are redacted as we don't deal with real data in the lab. But this will let me start working on an ETL process to get this into the AMDS extract data base that Vaughn is developing against.

So far it's only about 5MB/day of data and that's before I normalize out a lot of the content (we won't actually store syndrome names in the tables, etc. etc.) so this is still a good sign.

Also, it looks like Tom has started calling AMDS the Population Data Object (PDO) some times so we may be relabling soon. This actually makes more sense as we're not talking about a data set so much as we are a specification for sharing population data (as opposed to patient specific data that is NHIN's currency).

De-duplication

I've been thinking about the "De-duplication" grid service a bit lately and want to get my thoughts out here in case someone else has been mulling it over as well.

De-duplication is related to the De-identification/Re-identification problem of Anonymized data across multiple data sources (interesting MIT paper here). But in a grid environment the problem becomes tougher because the traditional, really-hard-to-do-solution of checking the various MPIs is even tougher because data sources 1) don't frequently have MPIs and 2) don't want to share them if they do.

Here's the scenario I use to talk about the de-duplication problem set:
The scenario is this:
3 hospitals in Atlanta with overlapping catchment areas. A federated query is sent out for SyndromeX for April 1, 2009. Hospital A returns a count of 10, Hospital B returns a count of 15 and Hospital C returns a count of 20. The combined result from the federated query shows a total of 45 counts of SyndromeX for April 1, 2009.

There exists a question of data quality as it's possible that the same patient visited multiple hospitals and had his SyndromeX recorded at multiple locations. In a federated/grid configuration this is even more probable as Hospitals may use data from multiple feeder sources (labs, small clinics, etc) that may report the same patient encounter multiple times.

Dr. Lenert had a good idea where each Hospital could send over a list of PII (name, address, dob, social) without the medical information for purposes of de-duping. This requires a trusted intermediary to receive these matches and send them on to each Hospital to respond back with number of matches. A hospital cannot send directly to each other hospital because it would reveal its patients to potential competitors.

So the de-duping scenario may change to something like this:
1) User submits query for SyndromeX on April 1, 2009 and selects de-duplication.
2) Trusted intermediary receives the query and passes it on to Hospitals A,B,C.
3) Hospital A returns count of 10, Hospital B returns count of 15, Hospital C returns count of 20. Interim total count 55 is not returned to user.
4) Trusted intermediary then requests patient id info from each hospital in a separate request.
5) Trusted intermediary compares the PII from each hospital (perhaps using a tool like febrl) and determines that there is overlap of patients and actual count. Hospital A shares 1 patient with Hospital B. Hospital A shares 2 patients with Hospital C. Hospital B shares 3 patients with Hospital C. Therefore there are 6 unique duplicates.
6) Total count of 39 is returned to the user (45-1-2-3).
Of course, this is going to require some pretty serious data use agreements with the hospitals and the trusted intermediary to control the storage of PII (i.e. don't persist any PII)

The alternative without the trusted intermediary:
1) User submits query for SyndromeX on April 1, 2009 and selects de-duplication.
2) Each hospital receives the query and list of nodes to which is was submitted.
3) Hospital A identifies a preliminary count of 10. To check for duplicates, it submits only the PII without any condition data and a query ID to Hospital B and C for comparison.
4) Hospital B compares the PII to its own results for query ID and returns the number of matches to Hospital A (duplicate count=1).
5) Hospital C compares the PII to its own results for query ID and returns the number of matches to Hospital A (duplicate count=2).
6) At the same time, Hospitals B&C are performing similar communications with the other nodes participating in the query.
7) Hospital B identifies a preliminary count of 15. To check for duplicates, it submits only the PII without any condition data and a query ID to Hospital A and C for comparison.
8) Hospital A compares the PII to its own results for query ID and returns the number of matches to Hospital B (duplicate count=1).
9) Hospital C compares the PII to its own results for query ID and returns the number of matches to Hospital B (duplicate count=3).
10) Hospital C identifies a preliminary count of 20. To check for duplicates, it submits only the PII without any condition data and a query ID to Hospital A and B for comparison.
11) Hospital A compares the PII to its own results for query ID and returns the number of matches to Hospital C (duplicate count=1).
12) Hospital B compares the PII to its own results for query ID and returns the number of matches to Hospital C (duplicate count=3).
13) Hospital A returns a count of 10 and notes that duplicates with B=1 and duplicates with C=2.
14) Hospital B returns a count of 15 and notes that duplicates with A=1 and duplicates with C=3.
15) Hospital C returns a count of 20 and notes that duplicates with A=2 and duplicates with B=3.
16) The user then feeds these results into another service that processes the results and determines that the actual count is 39 based on the following calculated relationships: (A|B)-1; (A|C)-2; (B|C)-3. These reductions are subtracted from the preliminary total of 45 to yield 39.

Approach #2, while not requiring a trusted intermediary requires a lot of coordination with the queried nodes and also involves the sharing of patient info with other partners that may not be desired. Also, due to set math will grow exponentially with the number of nodes involved in the query (for 3 nodes, the number of service calls is 9 [3 from user to nodes, then 6 among the nodes-using n!/{n-r}!]; but for 4 nodes, the number of service calls is 16; and for 5 nodes the number of service calls is 25). This is also suspect due to my deficient math skills and just some back of the napkin calculations.

Anyway, I wanted to get these two approaches to de-duping out there and will update with more as I keep thinking.

learning grid services.

So right now I am essentially reading the grid book, learning how to make grid services, and learning how to secure grid services. Then, I will be playing with Introduce to see how it makes all this stuff much easier and much more automatic with the help of Felicia.

I also hope to pick Felicia's brain about implementing security using the client.

Either way, Globus services are a lot more than I thought they were, they have a lot more involved for maintaining state and essentially having all of their interactions marshaled and grouped, and it is just crying out for a shiny (if not leaky) abstraction to make it all pretty and less scary to people who aren't programmers used to fighting with Webservices and some of the gooey "I was generated by a machine" code that can come out of them. I look forward to those abstractions, but until then, I am going to be learning a lot of transport layer fun to get secure/amds-based quicksilver running.

Cheers

Minor deploy, and Introduce

So, the next step for Quicksilver is to make an AMDS-intermediary layer that is a secure Globus service. Thus, tonight I am going to be going over some of the GRID/Globus books again and tomorrow I will probably install introduce and run the tutorial.

The reason I want to read the grid books is so I can see what goes into creating a grid service so that I can see what the introduce client will do for me. Then, I can start better understanding the heuristics of making an intermediary layer for poison data showing up in AMDS web.

Oh yeah, I have to go over the AMDS schema as well.

The end result is that the viewer part of Quicksilver, will end up becoming more of a generic AMDS viewer. Thus, one picks (a) secure service(s) that return(s) AMDS data, fills out the form that it/they require(s) (based on retrieved metadata for selected services), compiles the results into some sort of mappable data, and displays it in the map... and one of those secure AMDS services will be poison data.

One of the concerns I had was that making the NPDS->AMDS converter secure would be completely heuristic... then I remembered that people have to be given logins to be able to access poison data, and it would be much easier if they just were able to access it if they had a node on the grid.

So yeah, big context switch over the next couple of days, expect flagellating expectations as I realize the can, can't, and how.

Also, I ported a bunch of the improvements I made to npdsgmaps-web (the viewer part of quicksilver) back to gmap-poly-web (a demonstration library made on top of Gmap-Polygon). That should be deployed to training tomorrow so the two apps act the same (and because the version currently on training doesn't use encoded polygons, and is therefore much slower and error prone).

Cheers!

Updated AMDS Draft Schema

I went ahead and formalized a lot of the talks around the AMDS schema we've been having this winter and spring and updated the schemas on the wiki. Although this namespace is 20090330, it really has been around since January and isn't a major change from the v1 that was posted back on December 15.

The amds schema boils down to just two messages for the service:

• MetadataQuery/Response - Returns the appropriate metadata for each service. This exists so that clients and registries can determine metadata through a runtime API rather than rely on an administrator to manually enter.


• AMDSQueryRequest/Response- This takes in a query of conditions, regions and a date range and returns an array of counts by condition, region and day.

This is much smaller than the earlier version since we may as well start with a single operation.

The MetadataQuery is built into the service spec so that a user or registry can programatically check for what capabilities a service provides. Thanks Jeremy for this idea as it's a lot easier than trying to keep the service registry up to date manually.