Updated sample data on NCPHI node

One of our partner asked for better distribution of Denominator and flu data on our sample node's GIPSEService (https://ncphi.phgrid.net:8443/wsrf/services/gipse/GIPSEService).

So I generated some synthetic data for the 12 BioSense Denominator and flu indicators (EDVisits,EDVisits-LowTemp,EDVisits-HighTemp,EDVisits-UnknownTemp,ILI-b,ILI-b-LowTemp,ILI-b-HighTemp,ILI-b-UnknownTemp,ILI-n,ILI-n-Lowtemp,ILI-n-HighTemp,ILI-n-UnknownTemp) for Jan,Feb,March,April and June.

The values are all random but at least you can generate some ratios and percentages and such (previously we only had about 35 days spread out over the year).

So please check out gridviewer and the service API and let me know if it is helpful at all.

GIPSEService and GIPSEPoison.

GIPSEService was updated, so I updated the client and other libraries in the PHGrid repository (the old client/common/stubs was version 0.1, the current version is (like the schema) 20090831) and tested the integration. If you are interested in updating to the new versions of GIPSEService and GIPSEPoison, just update your gridviewer and it should be able to connect.

Otherwise GIPSEPoison is now returning state data. I had to adjust how the CXF service was being called, but thanks to Jeremy and a lot of legwork by Chris, we now have service re-sending enabled within a globus service... which is neat!

Tomorrow, we are going to try and deploy a bunch of this new stuff, so hopefully you will be able to pull from GIPSEPoison and the new GIPSEService and see some of the cool changes Chris has been making too.

Otherwise, this marks the first step in The Big Shift of a large amount of the UI code to Chris. He is (at least from my vantage point) excruciatingly good with several web technologies like JSON, REST, AJAX, JQuery, (a lot of which fit under an umbrella of JavaScript) and frameworks. Thus, a lot of my job over the next couple of weeks will be itemizing processes and adding to the infrastructures of geolocation code and the service-wrangling code. The end goal is for me to make a bunch of simple and obvious functons to be called by the architecture Chris is setting up. Thus, many of the persistence and state needs will move into a fancy web controller which will allow for much niftier User Interface stuff (like sliding date ranges and AJAX-populated drop down lists).

I'm looking forward to a lot of the new stuff!

Cheers,
Peter

Easy upgrading GIPSEService to 20090831 gipse.xsd schema

In case you have customized the GIPSEService, you can easily update it to use the http://ncphi.phgrid.net:8080/schemas/gipse/20090831 namespace gipse.xsd by following the few steps below: (thanks to Peter for collecting the details)

1. Update the gipse.xsd within the ./schema directory with the latest version (available in the sourceforge project)


2. Replace all the occurrances of "20090630" with "20090831" within the following files:
   ./schema/GIPSEPoison/gipse.xsd (replaced entirely)
   ./schema/GIPSEPoison/GIPSEPoison.wsdl
   ./GIPSEQueryRequest-ForClient.xml
   ./introduce.xml
   ./introduce.xml.prev
   ./namespace2package.mappings
   


3. Run "ant clean"


4. Run "ant all"


5. Run "ant createDeploymentGar"


6. Undeploy your old service


7. Deploy your updated service (created in step 5)

That's it, you shoudln't need to modify any Java code.

GIPSEService update

I just checked in updates to the GIPSEService project to include support for the Age and Service Area (Disposition) stratifiers as well as the Data Source filter.

This has pretty major ramifications as the version of gipse.xsd changed from 6/30 to 8/31. This means that anyone using the source code for GIPSEService should redownload and rebuild their local gar so that the latest version of gipse.xsd is supported.

You can see the latest copy of the gipse.xsd in its sourceforge project.

Taha wanted us to try out google code instead of the usual sourceforge project, so you can grab the latest version of the GIPSEService project through google code's svn. (btw, so far google code is a lot cleaner, but I miss the ability to download a tarball through the browser)

I'll update the NCPHI lab node during Friday's usual deployment, so you have until then to update your clients to use the latest updates.

If we were actually in production (planned for after Sep 1), we would actually version the service and provide a support window where both versions were supported. We're only breaking interfaces because there are no production users yet.

If you don't update, you will see an exception like this:

[java] org.xml.sax.SAXException: Invalid element in gov.cdc.ncphi.phgrid.services.gipse.stubs.QueryMetadataRequestQuery - MetadataQuery
[java] at org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:221)
[java] at org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:128)
...

This is caused because the client expects a specific xml namespace to be used by gipse. Older clients will expect http://ncphi.phgrid.net:8080/schemas/gipse/20090630, while this latest version uses http://ncphi.phgrid.net:8080/schemas/gipse/20090831.

This should be a rather painless upgrade for any existing users. If you customized the service code then you'll struggle (but what did you expect, next time contribute your changes back so we can make sure your changes make it into future versions).

For anyone still using the amds* database structure, you will need to update your db. Use the gipse-store project to generate ddl for SQLServer or PostgreSQL. Then you can copy over all your data from amds_extract using a query like:

INSERT INTO gipse_store (date, indicator_id, value, data_source_id, zip5, state)
SELECT date, condition, count, source_oid, zip5, state FROM amds_extract

PHMap and GIPSEPoison

So, I went away for vacation last week and when I got back, a lot of stuff had changed, but I feel it will make for better progress.

First up, I am supporting Chris in making PHMap... which may combine some of my service wrangling code and geo-fetching code with Chris' web code. This is fortuitous because Chris has a much better hang of actual web technologies like CSS, Spring, Javascript, and AJAX... and the concepts that make those all work smoothly together like REST and JSON. Thus, the code should be relatively clean and followable and adjustable, which is much harder when you have things like Java Objects which spitting out Javascript/HTML strings (like in my code).

This also means some of my service-wrangling and geo-fetching logic will probably be broken out or otherwise added-to to better enable the paradigm Chris wants to support . So essentially, refactors are upon us, and it will pretty much be the good old process of splitting function A into functions B and C since Chris will only need functon B.... and updating libraries and setting up property-less jar files for easy downloads instead of complicated builds.

Part of me feels a bit sheepish about how all the code sort of ended up mashed together and a lot of MVC lines were blurred... but then I remember that grid-viewer is essentially an evolution of RODSA-dai, which was a teeny little mashup that would have been overkill to load up with separate controllers and lots of different views. Then, suddenly, we needed polygons, and we needed to interact with the NPDS service, then we needed lots more UI tweaks and the ability to hit different services with different metadata.... and I must have been doing something right because I was able to make those modifications without too many large refactors, it's just now I get to deal with someone else actually needing to use it and not just install it, and their always runs the chance that the two paradigms for how something should work will be a bit obtuse, and larger refactors will be needed to accomodate both of them.

The other thing I will be doing mainly is getting GIPSEPoison working. I implemented a client and several tests and found that it is indeed getting data, but the aggregation needs to be tweaked, and some of the zip3 translations need to be taken into account. I think for that I will just split out some of the code I made for npdsgmaps (so it doesn't have any axis components) and make a new library that can be used by both npdsgmaps and GIPSEPoison.

Otherwise, lots of changes, lots of work, lots of progress, one hopes.

Cheers,
Peter

Vector-based maps

Ran into this interesting project out of MIT. Instead of taking image tiles like Google maps, it renders real-time maps in HTML5 and a custom styling language (like CSS and JSON combined) called GSS (Geographic Style Sheets). You will have to use Firefox 3.5 or Safari 4+.

Check out the project page:
http://wiki.cartagen.org/wiki/show/HomePage

Sample:
http://cartagen.org/

Refreshing to see a new approach for browser-based maps. We should definitely look to Cloudmade API as well.

White House Mulls NASA / Center for Cloud Computing

White House Mulls NASA / Center for Cloud Computing:

http://www.nextgov.com/nextgov/ng_20090724_6498.php?oref=topstory

8th Australian Symposium on Parallel and Distributed Computing

-------------------------------------------------------------------------
8th Australasian Symposium on Parallel and Distributed Computing
(AusPDC 2010)
Brisbane, Australia, January 18-22, 2010
http://www.cse.unsw.edu.au/~rajivr/auspdc2010/
in conjunction with
Australasian Computer Science Week, 18-22 January 2010
-------------------------------------------------------------------------
Overview/Scope:
The 8th Australasian Symposium on Parallel and Distributed (AusPDC 2010) will be held in January, in Brisbane, Australia in conjunction with Australasian Computer Science Week, 18-22 January 2010.

Scope of the Symposium
AusGrid event has been broadened to include all aspects of parallel and distributed computing and hence it will be called as Australasian Symposium on Parallel and Distributed Computing (AusPDC) from 2010. In both New Zealand and Australia parallel and distributed computing has been recognised as strategic technologies for driving their moves towards knowledge economies. A number of projects and initiatives are underway in both countries in these areas. There is a natural interest in tools which support collaboration and access to remote resources given the challenges of the countries location and sparse populations.

Topics of interest for the symposium include but not limited to:
* Multicore
* GPUs and other forms of special purpose processors
* Cluster computing
* Grid computing
* Green computing

* Cloud computing
* Peer-to-peer computing
* Service computing and workflow management
* Managing large distributed data sets
* Middleware and tools
* Performance evaluation and modeling
* Problem-solving environments
* Parallel programming models, languages and compilers
* Runtime systems
* Operating systems
* Resource scheduling and load balancing
* Data mining
* Reliability, security, privacy and dependability
* Applications and e-Science

The symposium is primarily targeted at researchers from Australia and New Zealand, however in the spirit of parallel and distributed computing, which aims to enable collaboration of distributed virtual organizations, we encourage papers and participation from international researchers.

Best Paper Award:
A best paper award sponsored by Manjrasoft Pty. Ltd, Australia will be presented to a paper receiving the highest quality rating. In addition, a special issue in a high quality international journal will be organized for selected best papers.

Program Committee Chairs:
- Jinjun Chen, Swinburne University of Technology
- Rajiv Ranjan, University of Melbourne

Program Committee:
David Abramson, Monash University
Mark Baker, University of Reading, UK
David Bannon, Victoria Partnership for Advanced Computing
Rajkumar Buyya, University of Melbourne
Paul Coddington, University of Adelaide
Neil Gemmell, University of Otago, NZ
Andrzej Goscinski, Deakin University
Kenneth Hawick, Massey University, NZ
John Hine, Victoria University of Wellington, NZ
Jane Hunter, University of Queensland
Martin Johnson, Massey University, NZ
Nick Jones, University of Auckland, NZ
Laurent Lefevre, University of Lyon, France
Andrew Lewis, Griffith University
Piyush Maheshwari, Perot Systems
Teo Yong Meng, National University of Singapore
Manish Parashar, Rutgers University, USA
Srikumar Venugopal, University of New South Wales
Yun Yang, Swinburne University of Technology
-------------------------------------------------------------------
--
Dr. Jinjun Chen
Lecturer in Information Technology
CS3 - Centre for Complex Software Systems and Services
Faculty of Information and Communication Technologies
Swinburne University of Technology,
1, Alfred Street, Hawthorn,
Melbourne, Victoria 3122, Australia.
Tel: +61 3 9214 8739
Fax: +61 3 9819 0823
Office: EN508a, Engineering Building, Hawthorn Campus
Email: jinjun.chen@gmail.com
URL: http://www.swinflow.org/~jchen/

GIPSEPoison getting closer

Today I spent some time playing with GIPSEPoison trying to see if I could invoke the service... it seems I could but I kept getting caught up in classpath issues... first needing to import Dr. Jeremy Espino's CXF client for accessing the NPDS service... and then talking a bit with Brian about how he set up the build scripts to include data (properties files) from a "resources" directory in the classpath. So tomorrow I have a good list of things to tinker-with that will hopefully successfully punt data from npds through GIPSE and to the grid viewer.

Then I need to clean up a bit of the fun javascript logic around the magical autopopulating drop-downs (there are some strange inital load issues).

Otherwise, I have gone ahead and made a service registry entry for gridviewer to help aid people who want to install and set up their own gridviewers. I have also added some database definitions to the database loader project to help aid in geodata creation and described their use in that entry. Right now one needs to build and install the gmap-polygon project before they are able to build and use gridviewer, and it has pointed out to me that it's an annoying step that could probably be replaced with a no-props version of the gridviewer jar... so I am going to look into doing that soon as well.

Cheers,
Peter

Introduce Metadata working.

So, with some help from Dan and Brian, I got GIPSEPoison deployed to my local dev box... and then I pointed grid-viewer to it... and got metadata back in the standard metadata format (which means that I properly imported some of the metadata code Chris already wrote).

Thus, tomorrow will be sorting out the client side of the code and then, hopefully, GIPSEPoison.

Cheers,
Peter