More updates for H1N1

I'm just putting out a general updates post since Peter and Chris haven't said anything yet.

Marcelo updated the GIPSEService google code project with some performance tuning and configuration enhancement changes. It won't affect the input/output but makes the code easier to maintain.

Chris and Peter put out an update to the gridViewer that looks pretty cool, but more importantly, allows you to overlay different indicators, data sources and services on top of or next to each other.

We also now have an EDVisits indicator so now calculations can be performed that require a denominator.

Correcting Globus Handshake Errors

The Problem:
When Globus-WS is deployed to Tomcat 5.5.x within Windows, a Handshake error is thrown when secure Globus commands are issued. The secure commands run fine in a standalone container but fail when Globus is deployed to Tomcat.

Example:
C:\>counter-client -m conv -z none -s https://192.168.20.120:8443/wsrf/services/Se
cureCounterService

Error: ; nested exception is:
javax.xml.rpc.soap.SOAPFaultException: ; nested exception is:
org.globus.common.ChainedIOException: Authentication failed [Caused by:
Failure unspecified at GSS-API level [Caused by: Handshake failure]]

The Solution:
(Commands are based on JDK 1.5.x)

This is caused when the SSL client does not trust the CA that signed the certificate. The solution is to add the CA certificate as a trustedCA.


1. Create a Java Key Store:

keytool -genkey -alias servercert -keyalg RSA -dname "CN=Your_host_name, OU=yoursite.net, O=your_organization, L=city, ST=state C=country" -keypass changeit -keystore server.jks -storepass changeit

2. Create a PKCS12 Keystore:

keytool -genkey -alias globus -keystore globus.p12 -storetype pkcs12 -keyalg RSA -dname "CN=Your_host_name, OU=yoursite.net, O=your_organization, L=city, ST=state C=country" -keypass changeit -storepass changeit

3. Export your PKCS12 Keystore:
keytool -export -alias globus -file globus.cer -keystore globus.p12 -storetype pkcs12 -storepass changeit

4. Import your PKCS12 Ketstore file into you Java Keystore:

keytool -import -keystore server.jks -alias globus -file globus.cer -v -trustcacerts -noprompt -storepass changeit

5. Import the 3rd Party CA into your Java Keystore as a Trusted CA

keytool -import -keystore server.jks -alias globusCA -file c:\etc\grid-security\certificates\31f15ec4.0 -v -trustcacerts -noprompt -storepass changeit

6. Import the host certificate issued by the 3rd Party CA into your Java Keystore.

keytool -import -keystore server.jks -alias containercert -file c:\etc\grid-security\importcontainercert.pem -v -trustcacerts -noprompt -storepass changeit

Based on the proceedure above, your server.xml file should look like this:

className="org.globus.tomcat.coyote.net.HTTPSConnector"
port="8443" maxThreads="150"
minSpareThreads="25" maxSpareThreads="75"
autoFlush="true" disableUploadTimeout="true"
scheme="https" enableLookups="true"
acceptCount="10" debug="0"
protocolHandlerClassName="org.apache.coyote.http11.Http11Protocol"
socketFactory="org.globus.tomcat.catalina.net.BaseHTTPSServerSocketFactory"
keystoreFile="C:\apache-tomcat-5.5.27\conf\server.jks"
keystorePass="changeit"
cacertdir="c:\etc\grid-security\certificates"
encryption="true"/>

---------------------------------------------------------
The commands change slightly when using JDK 1.6.x.

1. Create a Java Key Store:

keytool -genkeypair -alias servercert -keyalg RSA -dname "CN=Your_host_name, OU=yoursite.net, O=your_organization, L=city, ST=state C=country" -keypass changeit -keystore server.jks -storepass changeit

2. Create a PKCS12 Keystore:

keytool -genkeypair -alias globus -keystore globus.p12 -storetype pkcs12 -keyalg RSA -dname "CN=Your_host_name, OU=yoursite.net, O=your_organization, L=city, ST=state C=country" -keypass changeit -storepass changeit

3. Export your PKCS12 Keystore.

keytool -exportcert -alias globus -file globus.cer -keystore globus.p12 -storetype pkcs12 -storepass changeit

4. Import your PKCS12 Ketstore file into you Java Keystore.

keytool -importcert -keystore server.jks -alias globus -file globus.cer -v -trustcacerts -noprompt -storepass changeit

5. Import the 3rd Party CA into your Java Keystore as a Trusted CA.

keytool -importcert -keystore server.jks -alias globusCA -file c:\etc\grid-security\certificates\31f15ec4.0 -v -trustcacerts -noprompt -storepass changeit

6. Import the host certificate issued by the 3rd Party CA into your Java Keystore.

keytool -importcert -keystore server.jks -alias containercert -file c:\etc\grid-security\importcontainercert.pem -v -trustcacerts -noprompt -storepass changeit

H1N1 Response

I just realized that I haven't made any posts about our H1N1 response. The grid team has been working with the rest of NCPHI preparing for the flu season and the H1N1 surveillance that will be required. H1N1 response has been consuming a lot of all of our time.

I've been working on a data format that is modeled after the DiSTRIBuTE data use agreement's Appendix A that is a form of "GIPSE Lite" that defines a CSV format that can be used for immediate H1N1 surveillance. The data format is available on the wiki.

This format can be used in the "Producer-Collector" style of GIPSE architecture where data sources wish to provide daily data feeds that produce a report for transfer to a collector node (such as DiSTRIBuTE or CDC's GIPSE store repository). This is showing to be a viable option for sites that do not wish to host a query service, which requires a dedicated web service host with inbound access on port 443.

This is also helpful as this is the first DUA from CDC I'm aware of that specifically addresses summary and aggregate data collection and transfer.

Prepping for Monday.

Greetings!

Over the past couple of weeks I have been dealing with two fronts: On one front, GridViewer is being modified heavily with lots of cool new javascript and AJAX type functionality, so I have been making simple data-fetching interfaces as needed to support said functionality. On the other front, I have found out that GIPSEPoison works everywhere but the training node and have been investigating why.

There are now two new methods in the new gridviewer class PHMapper and a bunch of new classes, code, and even some new bits in gmap-polygon which will help data be pulled relatively easily. I'm basically mapping out better ways to associate and pull data (things like keeping the regional data separate from the time-series data so that one can be adjusted without messing with the other) but maintaining all the old structures to keep the old grid-viewer working as an example. Thus there has been some function splitting, but luckily I have managed to avoid duplicating lots of code.

Meanwhile, one of the foibles of GIPSEPoison is that it relied on the maven-built NPDS-Client-jar-with-dependencies... The "Jar with dependencies" is something maven does in case you want to run a client from the command line so that you don't have to mess with classpaths or worry about conflicts (shove the jar with dependencies in a directory and hit go). The dangerous temptation is to include it in something that is not maven-related that still needs the code (like an introduce project, more specifically, GIPSEPoison) because it's one jar rather than 20 (NPDS-client uses a lot of CXF and xml/jaxb jars).

Thus, one of the things I hope to complete early next week is to wrangle together all the jars needed separately and include them in the lib of the GIPSEPoison project. Right now GIPSEPoison does not work on the training node... and while it seems like a library conflict it could be something else. If it's a library conflict, having separate jars will be necessary to fix the issue. Otherwise, having separate jars is good form (it allows other potential conflicts to be discovered easier) and we might just run GIPSEPoison off a different globus container.

So, next week should be some adventures in jar hunting and library tweaking. If GIPSEPoison gets sorted out relatively early and the various installs I need to support go well, I hope to clean up some of the packaging of gridviewer (pull out the java code into a 'ServiceUtils class' leaving grid viewer a pure web project) and gmap-polygon (try and see if more of the data can be cleaned up and made simpler to download/sync) and brush up the documentation.

Cheers,
Peter

Updated sample data on NCPHI node

One of our partner asked for better distribution of Denominator and flu data on our sample node's GIPSEService (https://ncphi.phgrid.net:8443/wsrf/services/gipse/GIPSEService).

So I generated some synthetic data for the 12 BioSense Denominator and flu indicators (EDVisits,EDVisits-LowTemp,EDVisits-HighTemp,EDVisits-UnknownTemp,ILI-b,ILI-b-LowTemp,ILI-b-HighTemp,ILI-b-UnknownTemp,ILI-n,ILI-n-Lowtemp,ILI-n-HighTemp,ILI-n-UnknownTemp) for Jan,Feb,March,April and June.

The values are all random but at least you can generate some ratios and percentages and such (previously we only had about 35 days spread out over the year).

So please check out gridviewer and the service API and let me know if it is helpful at all.

GIPSEService and GIPSEPoison.

GIPSEService was updated, so I updated the client and other libraries in the PHGrid repository (the old client/common/stubs was version 0.1, the current version is (like the schema) 20090831) and tested the integration. If you are interested in updating to the new versions of GIPSEService and GIPSEPoison, just update your gridviewer and it should be able to connect.

Otherwise GIPSEPoison is now returning state data. I had to adjust how the CXF service was being called, but thanks to Jeremy and a lot of legwork by Chris, we now have service re-sending enabled within a globus service... which is neat!

Tomorrow, we are going to try and deploy a bunch of this new stuff, so hopefully you will be able to pull from GIPSEPoison and the new GIPSEService and see some of the cool changes Chris has been making too.

Otherwise, this marks the first step in The Big Shift of a large amount of the UI code to Chris. He is (at least from my vantage point) excruciatingly good with several web technologies like JSON, REST, AJAX, JQuery, (a lot of which fit under an umbrella of JavaScript) and frameworks. Thus, a lot of my job over the next couple of weeks will be itemizing processes and adding to the infrastructures of geolocation code and the service-wrangling code. The end goal is for me to make a bunch of simple and obvious functons to be called by the architecture Chris is setting up. Thus, many of the persistence and state needs will move into a fancy web controller which will allow for much niftier User Interface stuff (like sliding date ranges and AJAX-populated drop down lists).

I'm looking forward to a lot of the new stuff!

Cheers,
Peter

Easy upgrading GIPSEService to 20090831 gipse.xsd schema

In case you have customized the GIPSEService, you can easily update it to use the http://ncphi.phgrid.net:8080/schemas/gipse/20090831 namespace gipse.xsd by following the few steps below: (thanks to Peter for collecting the details)

1. Update the gipse.xsd within the ./schema directory with the latest version (available in the sourceforge project)


2. Replace all the occurrances of "20090630" with "20090831" within the following files:
   ./schema/GIPSEPoison/gipse.xsd (replaced entirely)
   ./schema/GIPSEPoison/GIPSEPoison.wsdl
   ./GIPSEQueryRequest-ForClient.xml
   ./introduce.xml
   ./introduce.xml.prev
   ./namespace2package.mappings
   


3. Run "ant clean"


4. Run "ant all"


5. Run "ant createDeploymentGar"


6. Undeploy your old service


7. Deploy your updated service (created in step 5)

That's it, you shoudln't need to modify any Java code.

GIPSEService update

I just checked in updates to the GIPSEService project to include support for the Age and Service Area (Disposition) stratifiers as well as the Data Source filter.

This has pretty major ramifications as the version of gipse.xsd changed from 6/30 to 8/31. This means that anyone using the source code for GIPSEService should redownload and rebuild their local gar so that the latest version of gipse.xsd is supported.

You can see the latest copy of the gipse.xsd in its sourceforge project.

Taha wanted us to try out google code instead of the usual sourceforge project, so you can grab the latest version of the GIPSEService project through google code's svn. (btw, so far google code is a lot cleaner, but I miss the ability to download a tarball through the browser)

I'll update the NCPHI lab node during Friday's usual deployment, so you have until then to update your clients to use the latest updates.

If we were actually in production (planned for after Sep 1), we would actually version the service and provide a support window where both versions were supported. We're only breaking interfaces because there are no production users yet.

If you don't update, you will see an exception like this:

[java] org.xml.sax.SAXException: Invalid element in gov.cdc.ncphi.phgrid.services.gipse.stubs.QueryMetadataRequestQuery - MetadataQuery
[java] at org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:221)
[java] at org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:128)
...

This is caused because the client expects a specific xml namespace to be used by gipse. Older clients will expect http://ncphi.phgrid.net:8080/schemas/gipse/20090630, while this latest version uses http://ncphi.phgrid.net:8080/schemas/gipse/20090831.

This should be a rather painless upgrade for any existing users. If you customized the service code then you'll struggle (but what did you expect, next time contribute your changes back so we can make sure your changes make it into future versions).

For anyone still using the amds* database structure, you will need to update your db. Use the gipse-store project to generate ddl for SQLServer or PostgreSQL. Then you can copy over all your data from amds_extract using a query like:

INSERT INTO gipse_store (date, indicator_id, value, data_source_id, zip5, state)
SELECT date, condition, count, source_oid, zip5, state FROM amds_extract

PHMap and GIPSEPoison

So, I went away for vacation last week and when I got back, a lot of stuff had changed, but I feel it will make for better progress.

First up, I am supporting Chris in making PHMap... which may combine some of my service wrangling code and geo-fetching code with Chris' web code. This is fortuitous because Chris has a much better hang of actual web technologies like CSS, Spring, Javascript, and AJAX... and the concepts that make those all work smoothly together like REST and JSON. Thus, the code should be relatively clean and followable and adjustable, which is much harder when you have things like Java Objects which spitting out Javascript/HTML strings (like in my code).

This also means some of my service-wrangling and geo-fetching logic will probably be broken out or otherwise added-to to better enable the paradigm Chris wants to support . So essentially, refactors are upon us, and it will pretty much be the good old process of splitting function A into functions B and C since Chris will only need functon B.... and updating libraries and setting up property-less jar files for easy downloads instead of complicated builds.

Part of me feels a bit sheepish about how all the code sort of ended up mashed together and a lot of MVC lines were blurred... but then I remember that grid-viewer is essentially an evolution of RODSA-dai, which was a teeny little mashup that would have been overkill to load up with separate controllers and lots of different views. Then, suddenly, we needed polygons, and we needed to interact with the NPDS service, then we needed lots more UI tweaks and the ability to hit different services with different metadata.... and I must have been doing something right because I was able to make those modifications without too many large refactors, it's just now I get to deal with someone else actually needing to use it and not just install it, and their always runs the chance that the two paradigms for how something should work will be a bit obtuse, and larger refactors will be needed to accomodate both of them.

The other thing I will be doing mainly is getting GIPSEPoison working. I implemented a client and several tests and found that it is indeed getting data, but the aggregation needs to be tweaked, and some of the zip3 translations need to be taken into account. I think for that I will just split out some of the code I made for npdsgmaps (so it doesn't have any axis components) and make a new library that can be used by both npdsgmaps and GIPSEPoison.

Otherwise, lots of changes, lots of work, lots of progress, one hopes.

Cheers,
Peter

Vector-based maps

Ran into this interesting project out of MIT. Instead of taking image tiles like Google maps, it renders real-time maps in HTML5 and a custom styling language (like CSS and JSON combined) called GSS (Geographic Style Sheets). You will have to use Firefox 3.5 or Safari 4+.

Check out the project page:
http://wiki.cartagen.org/wiki/show/HomePage

Sample:
http://cartagen.org/

Refreshing to see a new approach for browser-based maps. We should definitely look to Cloudmade API as well.