Gridviewer is close

So, I managed to get gridviewer working in a sense last night.

It was pulling data and displaying it Quicksilver style when installed in tomcat and connecting to the GIPSE Globus Service running on a non-secure globus container. In Windows.

Today, I tried to build it for a training deploy when connecting to secure GIPSE Globus service running in a secure tomcat container. I found some more dependencies I needed to add. And then I found out that the GIPSE globus service seemed to be having configuration issues.

Thus, I figure the next best step is to get that GIPSE Globus Service running on a secure tomcat container in Windows... or move development over to my old Ubuntu development box (where I would still have to get the GIPSE service running in Linux). Both would be beneficial and work towards the goal of getting an environment more like the training node.

Otherwise, I hope to get that done relatively soon, and then start focusing on the refactors I have been planning for Grid Viewer but haven't been able to really work on in the attempts to just get something Grid-Viewer-ey completed and working (hopefully correctly).

The other thing that needs to happen is more services. One that is starting development will provide NPDS data in GIPSE form, and we should probably also deploy a smaller GIPSE service some place like Dallas so that we can pretend to have a smaller service that only has data for a few states and/or a few conditions. Multiple services with varying metadata is where Grid Viewer should get most of it's niftiness.

Cheers!

June 2009 GeoInformatics - Cloud computing and GIS

The June 2009 issue of GeoInformatics is here:

http://fluidbook.microdesign.nl/geoinformatics/04-2009/

If you go to page 36....there is an article on Neogeography and Google;

They interview Jack Dangermond (ESRI) as well as Intergraph and other GIS vendors;

Jack Dangermond makes an interesting comment about the future of Desktop GIS and the increase in demand for cloud computing....

DHS project aims to bring open-source software to state and local agencies

The Homeland Security Department is funding a program that will help federal, state and local agencies better understand their options for using open-source software. More information can be found here.

Tomcat SSL

Testing the latest Windows installation document for the grid node. The goal is to create a clear and simple guide for users to get a grid node up and running quickly. The primary issues I am running into are all in the certificate/SSL area.

It seems to me we could link to the installation documents already written by Apache on configuring SSL with Tomcat in a pre-requisite section. There are so many variables like the user account running tomcat, where to store the keystore file, whether they can use a 3rd party certificate or self-signed, is there load-balancing, etc. Either way, once a user has Tomcat running with a certificate then they could begin stepping through our Globus installation.

Basically I think we should not duplicate existing documentation, and it will allow us to focus on Globus.

Cloud vs Grid Comment

Good post. Thanks for the post.

To further articulate the differences and similarities between the EGEE Grid, Amazon Cloud and the Public Health Grid...

The EGEE Grid service focuses on "short-lived batch-style processing (job execution)". The Public Health Grid has these services available and plan to research and deploy in the future but our current focus is on public health and health specific services.

The Amazon Cloud (and other clouds) service is "long-lived services based on hardware virtualization". The Public Health Grid does have a virtualization appliance, which we ship on DVD, and we are currently researching Grid in the cloud methodologies (University of Utah and Argonne specifically are working in this area).

The Public Health Grid services are long-lived services built using service-oriented architecture (SOA) methodologies and technologies. The team has build the first services: GIPSE, Grid Publisher and Grid Viewer. EAch of these services may be accessed through this web site and downloaded through our open source repository.

Grid vs Cloud

I was asked yesterday what the primary differences were between grid and cloud computing. Struggling for a good answer, I did some quick google searches and I came across this interesting paper called EGEE Comparative Study: Grids and Clouds Evolution or Revolution? done in Nov 2008. I imagine most people have read it, but thought I would post just in case.

building! testing! Deploying?

So, I got a list of the jars (which can be found in the $GLOBUS_LOCATION$\lib directory after an introduce install) which grid-viewer now needs to run the GIPSE client:

addressing-1.0.jar
caGrid-metadata-security-1.3.jar
caGrid-ServiceSecurityProvider-client-1.3.jar
caGrid-ServiceSecurityProvider-stubs-1.3.jar
caGrid-ServiceSecurityProvider-common-1.3.jar
axis.jar (this has to be the version that is shipped with globus)
jaxrpc.jar
saaj.jar
cog-axis.jar
cog-jglobus.jar
commons-logging.jar
commons-discovery.jar
wss4j.jar
wsdl4j.jar
wsrf_core.jar
wsrf_core_stubs.jar
jce-jdk13.jar (and this one seems to have those pesky bouncycastle.org security libs)
puretls (*for secure access*)
cryptix32 (*for secure access*)
cryptix-asn1 (*for secure access*)

These are stored in my private repository and downloaded by maven because they are classified as "provided" in the POM (which means they are needed for compilation and testing, but will be provided in the classpath when the war is installed). Thus, I copied them all over to the tomcat/commons/lib directory.

Initial attempts seem to be resulting in NoClassDefFound errors. So it seems there may be a few jars needed still in tomcat alone... or that tomcat is not properly loading the jars in the common space (and that might be the case, I remember RODSA-DAI having issues running in tomcat becuse of library/classpath issues).

Either way, I am hoping to get it sorted out early tomorrow, have it returning data to the map. Then I want to get it tested on a secure-globus environment (in case some more security libraries are needed and to make sure that it plays nice with other secure clients like RODSA-DAI), and then I'll be ready to continue with the refactors.

Right now, the main refactorings have to do with shifting from a "state, zip3, zip5" paradigm to a "region" paradigm. This should allow for showing zip3s and zip5s and states all on the same map. A complimentary paradigm shift will be allowing multiple loads on one map (load one query with one set of pushpins, load another query with another set of pushpins... so you can click both sets of pushpins and compare data from two queries on the same map). All with more services and more realtime options.

Examining the relationship betweeen GIPSE and SDMX-HD

I was recently pointed to a very interesting entity known as SDMX-HD.

Google Groups for SDMX-HD

SDMX-HD is a Statistical Data and Metadata Exchange (SDMX)-based data exchange format intended to serve the needs of the Monitoring and Evaluation community. It has been developed by WHO and partners to facilitate exchange of indicator definitions and data in aggregate data systems.

Experience with the UNAIDS IXF version 2.0 provided a basis for developing the SDMX-HD. New features in SDMX-HD include better support for domain-specific content for various stakeholders, cross-domain 'Metadata Common Vocabulary', hierarchical codelists, and the ability to generate generic or compact XML from a common data model. It will be based on the ISO SDMX standard and be a 'Content-Oriented Guideline' for the exchange of public health indicator data typically for Monitoring and Evaluation (M&E) activities and international reporting, e.g. PEPFAR.

Jar hunt complete

So, I managed to find all the jars that are needed to test (and eventually run) the use of the GIPSE client (at least the metadata portion of it)... and it is a long list pulling from both the CAGrid and globus libs (which thankfully end up in the globus libs). The very long list is in the pom file of the grid viewer project and I plan to post it in more plain text on the wiki tomorrow.

The thing is, I had to hand-install these items into the maven repository... which means that anyone wanting to test or build the gridviewer project at this point would have to do the same... and after setting up a grid-node and the GIPSE service, it is arduous and annoying.

Thus, I am hoping to get most of the jars needed into sourceforge. It was done once before for RODSA-dai already, and it should allow for people to get all the jars needed for testing after a simple property file change (which should be easily scriptable)

Furthermore, I am going to test running grid-viewer from the tomcat-enabled globus container... which should have all the libraries co-located in the lib directory, which will hopefully make it a much easier to install grid-viewer after setting up a tomcat instance.

I'm sure there will need to be some finessing either way.... and will still need to check all this in a secure globus environment and get the data returning in the grid view.

But, it's exciting. Nothing feels more gratifying than seeing your test code come back with something other than "NoClassDefFoundError: "

Compiling

So, I managed to take out all the conflicts and update all the code in grid viewer, so now everything is successfully compiling.

Tomorrow will be re-engineering the tests to match the new test cases, and then, hopefully, a war that can be installed and run (even if the initial phase is just having the service loader pull back the metadata options).

Then, it's implementing some new features in line with the current refactor, which will ultimately make things simpler.

I'm excited, I think grid viewer is turning into something that will work rather well, and still be flexible enough to work in ways not yet anticipated.