Updated AMDS Schema + web app

I updated the AMDS schema with a new version in preparation for the release of AMDS-BioSense's Alpha services tomorrow.

I'm sure Vaughn will tell you about the details. Basically, the service tool we're using didn't like referenced types, so I remade the schema to use inline defined elements. I know this is bad form, but it's what we had to do to get the tool to work. If anyone can generate a WSDL file with Introduce using the March 30 schema we'll be very grateful. Otherwise, the example XML didn't change at all so documents valid under the March 30 schema are still valid under the April 30 schema.

Other changes are we changed all our xs:date and xs:dateTime elements to be xs:string, again to make it easier for the AMDS-BioSense Alpha services. Axis1.4 didn't really like date objects very much.

Finally, I added an optional "AllCount" sibling element to "Count" so that services that want to explicitly return the denominator can do so. This was added at the suggestion of the Harvard ESP:SS folks as their data provider doesn't actually return the counts for every single condition type so a query for Classifier=ALL would not actually return the denominator.

I also created a sourceforge project for the schemas so 1) we can do some more detailed version control than the wiki provides and 2) others can host the schemas if they so desire.

Updated DRN Design drafts

I updated the DRN Design Drafts wiki page with a data flow diagram to show the flow of detailed data --> aggregate data --> combined aggregate data based on some feedback from Roy Pardee, Ross Lazarus and Jeff Brown.

DRN Design drafts

I've put together some design sketches to show the planned deployment and sequence of events for the DRN SAS automation scripts that Dan has begun writing.

http://sites.google.com/site/phgrid/Distributed-Research-Network/drn-sas-grid-design

Basically we'll have three components:

1. ClientScript.pl - transfers files to Globus nodes and aggregates the output documents into a single TSV report (to be developed)


2. Secure Simple Transfer Service - runs on the Globus nodes and allows for listing, getting and putting of sas programs and output files (already completed)


3. SASProcessorScript.pl - configures and runs sas programs against databases (to be developed)

Please let me know any comments you may have as Dan is beginning to develop the scripts. The source code will be stored in http://phgrid.svn.sourceforge.net/viewvc/phgrid/drn/

(here's the visio file should anyone want to make direct comments and changes.)

VMWare Appliance Now available online

Dan made a VMWare Appliance of the PHGrid Globus node and it's available at http://phgrid.net/documents/node/v1/.

If you download both of the files here, you can run a SUSE installation of the Globus Toolkit. This will make configuration easier as all you have to do is make a few configuration changes(set the IP, hostname, user accounts, request a user and host cert, etc).

This will reduce the install time down even lower (5 minutes) and allow PHGrid nodes to run on Windows (through the free VMWare player).

This first release is still pretty large (4.5GB download, 8GB necessary to run) so it may take you a while to download (took me over 30 minutes to upload).

Aggregate MDS Service planning

Tom and John presented the FY09 plan to the NCPHI governance council last week. A rather decent size block of work that the NCPHI R&D lab is planning is the development of a set of summary data services and associated coordination services.

These services will be an extension of the RODSA-DAI work and will provide access to various biosurveillance systems hosted by partners out in the states. Each service will return a set of aggregate syndrome counts that will map to a new common data structure that for now we are calling the "Aggregate Minimum Data Set". The AMDS will use the AHIC/HITSP MDS and select the minimum useful fields for aggregate reporting. The work on developing a scientifically vetted AMDS is in progress and so far is involving the Centers of Excellence and CDC BioSense personnel.

So the idea is that there will be multiple implementations of the AMDS services for each participating biosurveillance system and then a set of coordination services that know how to run a federated query across the participating nodes and combine together the results from the query. This is basically what the PHDGInet and RODSA-DAIWeb demos shows as a proof of principle, but 2009 will bring this to a proper pilot by developing services for actual installed biosurveillance systems running at partner sites.

Here's a list of the initial set of services that will support the pilot:

• AMDSX-DAI (Where X is one service for each participating system, in 2009 probably 5 different implementations planned for RODSA, ESSENCE, BioSense and specific state systems)


• AMDSCoordinator.RunFederatedQuery


• AMDSCoordinator.QueryAvailableCoverageArea

DiSTRIBuTE's Aggregate Data Model

John Stinn reminded me of the DiSTRIBuTE's Aggregate Data Model. This summer, the PHGrid team met with Ross Lazarus and the BioSense epis about the potential structure of an Aggregate Minimum Data Set (written about earlier on this blog).

The International Society for Disease Surveillance has a project called DiSTRIBuTE (aside to future project namers- name your project something that is easily googlable) that seeks to collect aggregate data on ILI. DiSTRIBuTE uses a minimal aggregate data structure of:
| date | zip3 | age group | fever count | denominator |

This is similar to that proposed by Dr. Lazarus for use at Harvard ESP.

Integrating caGridTransfer and RFT

I participated in a conference call with Justin Permar and Shannon Hastings to discuss the possibility of integrating caGridTransfer Service with RFT. Some interesting points came up during the meeting:

• To date no one has suggested the integration of caGridTransfer Service with RFT but Justin will bring the topic up on some upcoming calls that he currently participates in to determine if interest exists in other domains

• Justin will have further discussions with Shannon, Ravi (Madduri), and Rachana Ananthakrishnan to discuss feasibility of using caGridTransfer Service with RFT
  

• Shannon pointed out that caGridTransfer Service persists transfer information and adding in automated restart capability for transfer would not be difficult to add

• Justin would like an informal explanation of the drivers behind the desire to use caGridTransfer Service with RFT. I will provide him with this information by Oct. 1 at the latest

• Justin would also like to know if there is a definitive timeline for the integration of caGridTransfer Service with RFT. I will speak with stakeholders and provide this information back to Justin.

All in all it was a good first call.

caGridTransfer Service

The basic caGridTransfer Service was successfully deployed to the staging and dallas nodes.

Found a interesting twist with the caGridTransfer Service, when trying to undeploy the service, it breaks the Globus container. Currently investigating what is causing this problem.

caGridTransfer Service

Made progress I was able to build and successfully deploy the default caGridTransfer Service to my local environment today.


Next Step: Add functionality to satisfy use cases.

More successful testing of secure grid services!

Ron Price, with the University of Utah CoE, sent over a new demo project this morning to test out the secure invocation of his grid services.

Utah is using cagrid's gridgrouper to manage security configuration (phirg uses manual maintenance of certs at each node), so I created a userid with cagrid (https://cagrid-portal.nci.nih.gov/web/guest/register) and Ron added my cagrid user the PH Grid group.

Ron made it pretty easy to test by providing an Eclipse project export and the required certificates to authenticate with cagrid's Dorian services.

I copied the certificates to the ~.globus folder on my workstation, specified my cagrid provided userid and password, and ran the service from inside Eclipse. Pretty straightforward

When I tried with an invalid password I get an exception thrown back from Dorian (gov.nih.nci.cagrid.authentication.stubs.types.InvalidCredentialFault).

So we tested securely invoking a Utah phGrid service over SSL using the cagrid authentication management from the NCPHI Research Lab grid.

Next steps will be to test out invoking the service using phirg specific credentials.

New person / NLP Service Wrapper

Hello, I'm Brian Alexander Lee and I'm starting to work with the public health research grid team to help out on some of the proof of concept projects. I'm an enterprise architect and my background is in web services and software development, so I hope that I'll be of use to the projects.

To work with the Centers of Excellence, we had a call with Albert Lai of Columbia University on how to provide an interface to the Columbia Natural Language Processor (I think is called Medley).

We talked about some of the best ways to provide a wrapper to the existing NLP components and right now the thinking is to provide a web service wrapper through Globus that accepts medical free text (in either body text or files) that can invoke the NLP and respond with structured data in xml format. We'll post more as we start testing and designing more.

Also, Albert shared the URLs for two potential web service frameworks that we might be able to use in the test:

• NBCR Opal Toolkit
  
• caGrid Introduce Grid Authoring Toolkit