AMDS Technical updates

I added a project called db-importer that can be used to import CSV (and eventually xml) AMDS extracts into an AMDS Store database based on the schemas we're using at CDC. This isn't the fastest or best importer, but it works on MS SQLServer and PostgreSQL and will be used to bring in daily report extracts from systems like BioSense.

I also talked a bit with Vaughn to transition over his projects. I'll be fixing up / re-writing AMDSCore and AMDSPoison over the next few weeks in time for the PHIN Conference until we find another Java developer.

Finally, we had a productive meeting with the PHIN Messaging steward and team about ways to directly collaborate in the solution development of PHGrid, PHINMS and PHIN-SRM. Nothing major was covered, but lots of good ideas that we'll follow up on in the near future.

A few things...

I have been busy with a few things over the past couple of days:

I upgraded GridViewer and Quicksilver so that they used timers (so that one can see which proportion of waiting lies in service calls or data rendering) and had an option for plotting pinpoints instead of polygons (pinpoints being that much faster and clear-cut, polygons looking neater and showing better spatial orientation). Quicksilver also has a link to metadata for servers so you can see what data is avialable from a particular server.

Right now I am working on making grid viewer behave based on what the servers can provide, this means tailoring the lists (states, zip3s, zip5s, conditions) so that one can only select regions that a server can provide data-for, and not everything in the region-relations table or conditions properties. I am hoping to have all the regions and the conditions selecting appropriately tomorrow, that way it will become much less confusing which regions have been asked-for.

After that, more User Interface niftiness is planned. Including hover-over for metadata stats, and a server list that changes based on the selected regions and conditions, but first I want to get the service pulling back data more reliably.

Cheers!

PAOH-BIG Notifiable Disease Sharing Grid Application

We created a video of the early stages of our notifiable disease sharing application.


The application uses Globus for service deployment, Introduce for service stub generation, Dorian for authentication, Grid Grouper for service and method level authorization and the Credential Delegation Service for authorizing a proxy to access data on the client's behalf. We also implemented an additional layer of authorization in the application so that users can only see cases that are relevant to their jurisdiction



http://www.youtube.com/watch?v=GjoU82i5Vac

Thanks everyone!!!

I may not say it enough, but I am extremely proud and privileged to be a part of this effort.  

I would just like to thank everyone (the NCPHI grid team and all of our national and international colleagues) for all your support and efforts.    Thank you for your posts on the blog, your submissions to the wiki, your code development (apps, viewers, services, etc), your entries in sourceforge, your artistic diagram creations, your use case documents, your abstract and conference submissions, your presentations, and your white papers. (..and the list goes on).

We've made truly incredible progress since our first post on Wednesday, November 14th, 2007 at 10:37 AM.  

What a ride.  Thanks again everyone!

Google Maps and privacy

I get asked a lot about how Google Maps works with our data (like the grid viewer, quicksilver, rodsa-dai, etc). Specifically, concerns about privacy and security of sending your data to Google.

There is a popular misconception about Google Maps and how it handles data. Way back in the olden days (~2006) the easiest way to use google maps was to hand google a KML file and google would draw it on their maps site. This was convenient but involved passing your data points to google within a KML file.

PHGrid does NOT do this. We use the Google Maps API which sends no data on to google maps other than requests for map images. No counts or conditions or calls or any data is passed to the google servers. All that stays on PHGrid web servers and users' web browsers. All of the polygons and pins are drawn using Javascript without passing data points back to maps.google.com.

So from a privacy concern, Google can tell that a random user is requesting a map of the state of georgia (and scrolling and zooming) but will not know the polygons drawn, the pins placed, the counts that affect the colors or any of the actual AMDS data.

NHIN and PHGrid Interoperability Testing

The NHIN 2.0 'connect' and 'adapter' packages have been obtained. The lab now has two machines ready for usage for testing (thanks TS!) so now we get to digest the NHIN source code and install it to prepare for testing PHGriD <--> NHIN interoperability.

Pinpoints and Polygons

It has been requested to have the ability to display both pinpoints (located centrally) and polygons... namely because some zipcodes are a bit confusing and have multiple polygons.

I have spent a good part of yesterday and most of today enabling that functionality. It involved modifying several handlers and creating new classes to execute interfaces based on what was selected... Luckily, I did not have too many incidences of "oops, not OO enough", so it went smoother than I thought it would.

Otherwise, this is part of a host of speed-increasing tweaks. One of the other adjustments I am planning is to place timers throughout the application structure that both Quicksilver and Grid-Viewer use so that execution times can be displayed and give everyone a better idea of where time is spent between data loads.

After that, it's enabling the metadata in grid-viewer, and beginning the transition to an "available versus selected" framework which will better show that only certain regions will be selectable.

Deploying Grid Trust Service (GTS)

The CaGrid Grid Trust Service has been deployed on the 1001 internal grid node. At this point I still need to deploy the GAARDS UI in order to test the installation. The install was done using the cagrid-installer jar, which takes care of downloading dependencies and deploying Globus to Tomcat.

For future reference, GTS can be installed from source much faster than going through the entire CaGrid install process.

KNIME

KNIME is the Konstanz Information Miner: http://www.knime.org/

It looks like a workflow tool such as TAVERNA and seems to integrate R, Python, in an Eclipse-like environment....

KNIME FEATURES: http://www.knime.org/introduction/features

KNIME SCREENSHOTS: http://www.knime.org/introduction/screenshots

KNIME SCREENCASTS: http://www.knime.org/introduction/screencasts

Getting Started: http://www.knime.org/documentation/getting_started

Planned updates.

Getting data into gridviewer has been a bit cantankerous... but now that it's there I know what I want to happen over next week.

- Update the structure of the grid-viewer to deal with items on an available versus selected. Having all the states show up when the service only allows for the Southeastern ones is going to be confusing... Thus, limit the possible selections to what is available. The same goes for services (if metadata could not be fetched for a service, don't let it show up as an option). The other added bonus is allowing for multi selection of regions... thus, you can pull data back for zip3s in two different states.

- Update the graph popups to populate zero values in a time series, as opposed to having them populated upon a service return. It's a bit complicated and behind the scenes, but it essentially moves some workload around and should speed up the client by having it do less work upon return.

- Start working with diversified clients. Things on different computers. Things behind different servers. Things in different states.

- Add bullets, and server return versus view render times.

- Have a histogram view, and update the time series to allow for differentiation of data source.

Either way, I am excited.