More updates for H1N1

I'm just putting out a general updates post since Peter and Chris haven't said anything yet.

Marcelo updated the GIPSEService google code project with some performance tuning and configuration enhancement changes. It won't affect the input/output but makes the code easier to maintain.

Chris and Peter put out an update to the gridViewer that looks pretty cool, but more importantly, allows you to overlay different indicators, data sources and services on top of or next to each other.

We also now have an EDVisits indicator so now calculations can be performed that require a denominator.

Re-introduce introduce

I managed to fix a few bugs and do a few deployments with grid viewer over the past few days, and it looks like the next thing I get to play with is the GIPSEPoison service... but GIPSEPoison has been built in introduce, and I need to play more with introduce to figure out how those services work and are built and can be modified.

But it will be about the third time I have started to learn introduce. The other two times priorities shifted and I started working on other things... So, perhaps third time is the charm.

I am also going to play a bit with graphics and see if I can plan out concurrent load items.

Cheers!

Client flexibility.

So, one of the little issues that cropped up over the last couple of days is that if you do not really watch how you set up your service in Introduce... there is a very good chance that it will use new packages and namespaces and use an entirely different service in the generated service code.

Right now I am using the client half of the client/server code that was generated by introduce for GIPSE-Biosense. Meanwhile, the client half of the client/server code that was generated by introduce for GIPSE-Poison was different.

Really different. Different packages different.

This means I would have to import both the clients of GIPSE-Poison and GIPSE-Biosense and have logic in my code to differentiate between the servers being called to get data from both... and the idea of having to do that for every service someone wants to generate makes the solution not scalable.

Thus, a big focus over this upcoming week is going to be making a very flexible client. I was chatting with Jeremy about things like Yahoo Pipes and he also just pointed out that the data is XML and I could probably just run things through an XSLT to get data from one point to another... and it was a really good idea. Hopefully, it means that all one would have to do is create an XSLT in case their schema didn't match ours exactly and drop somewhere in the classpath of gridviewer and set a few properties to say "this might need a bit of extra processing" and then gridviewer will be able to display the data.

Also, I am going to be thinking of pinwheels and set-the-URL pushpin styles... but I am not sure if I am going to be able to get to them this week.

Cheers,
- Peter

Got multiple region searches.

Part of the improvements that will hopefully deploy successfully tomorrow is the ability to have multi-region searches.

So, there will be a new button that says "add to current" which, when pressed, will add data without clearing off the old data. This means you can have all the zip3s in Florida post next to all the zip3s in Georgia in case you want to compare between the two.

Currently, there is no way to differentiate between two searches on the same region (so if you select for all the zip3s in Georgia for one condition, and then add all the zip3s in Georgia for another condition... it will just overwrite the first set)... this is going to be part of a suite of improvements that allow for different sorts of pinpoints (pinwheel) and probably for selecting regions and legends.

But, I think it's neat and I hope you will too (and I actually got it working yesterday but spent a lot of yesterday and today debugging it, as I essentially had to rip apart the gmap-polygon project to get it working (and I branched before I did, thank goodness))

Cheers!

multisets

Gmap Polygon and Grid Viewer are now essentially doing cumulative searches... it's just not displaying them in any particular order... Thus tomorrow will be spent creating some updated pinpoint libraries (pinwheel), modifying the flot-plot screens, adding a "clear map" function... and filtering to make sure that empty data sets aren't added.

Also, there is a distinction for "empty set". If servers are queried and no data comes back, I'll show the pinpoints/polygons as empty. If, however, a search is run without any servers selected.... it will be considered an empty set and not added to the search tree.

Thus, when you first load the page, you'll get a map of the US, but there will be no pinpoints until you actually search for data. This behavior can change, of course, but for now it will help debug the behavior of how data is added.

Then, it's testing to see how it all behaves, and seeing whether a collection of collections will suffice or whether I will have to upgrade it to a map of some sort (a bit more complex to navigate, but probably much better "make sure these are in order" behavior)

Enterprise...

So, I was chatting with a friend yesterday and indicated that I was a bit sad because the next set of changes (modifying GmapPolygon to hold multiple sets of polygons/indicators and writing a new polygon color handler that allows for URL's to be set) will sort of make things ugly and a bit more confusing and threatens to mottle the design patterns even more...

The friend spake (and I paraphrase): "Don't worry about it, it's enterprise code. You go in expecting A... and you end up making A, which eventually needs B after the user sees it and decides it wants B-esque features... but you don't have the time to completely remake it into B, so you make it into a mashup of A and C which works just as good but is not as pretty. So just be happy that A-C is doing good enough and make it into B if you have time before the next deadline." We then went on to discuss how the whole idea that code as a start-do-end thing is more myth than reality... as specifications change and user wishes change and sometimes you change from linux to windows or the tool you need only works with a certain version &c &c.

So yes, The next step is going to be changing the GmapPolygon and the Grid Viewer to work with indexed collections of polygons instead of just one collection of polygons. This will allow for sequential searches (search for all the fever in the US... now search for all the nausea in the US... now search for all the poison-related cellulitis in the US... now do those again for all of Texas...) to be maintained and shown simultaneously on the map until they are explicitly cleared. This means the pinpoint visualizations will need to be changed so that later searches don't show up on top of the older searches (right now I am thinking of using rotating pins... so sequential searches start filling in a pin-wheel). The flot-plotting mechanics are also going to need to change a bit to search on region type, name, and index (thus, it knows to look for the 3rd search or the 2nd search). Eventually the selection mechanisms might also be tweaked to become more intuitive or allow for more specific searches... but right now it's just about getting multiple searches on a map.

I'm sure it will actually not be as ugly or unintuitive as I fear when I'm done, but it is a lot of changes in the name of niftiness,

Cheers!

grid-viewer now pulls it's library from the phgrid repository

I uploaded the jars that grid-viewer uses to the phgrid repository, and updated the pom file to pull from the repositories.

So now, instead of having to track down the various files from globus, amds service, and introduce... and then install them into the local repository... one can just type "maven package" and maven will download the files.

I think this will help me a lot, and now that several people are using my code I am hoping it will help them a lot.

The one thing that will still need to be written up is that some of the files (all the ones specified by "provided" in the POM file) will need to be copied into the web containers shared lib directory. I am debating whether it will be more useful to change them from provided to default (meaning they will be included in the war) so long as whoever is trying to set-up grid-viewer isn't trying to use multiple grid-viewers (or other things that will be using the secure globus libraries), it should work.

Otherwise, I have made some minor changes to grid viewer, but I'm planning to make some progress to gridviewer behavior and and performance tomorrow and next week.

Gridviewer is close

So, I managed to get gridviewer working in a sense last night.

It was pulling data and displaying it Quicksilver style when installed in tomcat and connecting to the GIPSE Globus Service running on a non-secure globus container. In Windows.

Today, I tried to build it for a training deploy when connecting to secure GIPSE Globus service running in a secure tomcat container. I found some more dependencies I needed to add. And then I found out that the GIPSE globus service seemed to be having configuration issues.

Thus, I figure the next best step is to get that GIPSE Globus Service running on a secure tomcat container in Windows... or move development over to my old Ubuntu development box (where I would still have to get the GIPSE service running in Linux). Both would be beneficial and work towards the goal of getting an environment more like the training node.

Otherwise, I hope to get that done relatively soon, and then start focusing on the refactors I have been planning for Grid Viewer but haven't been able to really work on in the attempts to just get something Grid-Viewer-ey completed and working (hopefully correctly).

The other thing that needs to happen is more services. One that is starting development will provide NPDS data in GIPSE form, and we should probably also deploy a smaller GIPSE service some place like Dallas so that we can pretend to have a smaller service that only has data for a few states and/or a few conditions. Multiple services with varying metadata is where Grid Viewer should get most of it's niftiness.

Cheers!

building! testing! Deploying?

So, I got a list of the jars (which can be found in the $GLOBUS_LOCATION$\lib directory after an introduce install) which grid-viewer now needs to run the GIPSE client:

addressing-1.0.jar
caGrid-metadata-security-1.3.jar
caGrid-ServiceSecurityProvider-client-1.3.jar
caGrid-ServiceSecurityProvider-stubs-1.3.jar
caGrid-ServiceSecurityProvider-common-1.3.jar
axis.jar (this has to be the version that is shipped with globus)
jaxrpc.jar
saaj.jar
cog-axis.jar
cog-jglobus.jar
commons-logging.jar
commons-discovery.jar
wss4j.jar
wsdl4j.jar
wsrf_core.jar
wsrf_core_stubs.jar
jce-jdk13.jar (and this one seems to have those pesky bouncycastle.org security libs)
puretls (*for secure access*)
cryptix32 (*for secure access*)
cryptix-asn1 (*for secure access*)

These are stored in my private repository and downloaded by maven because they are classified as "provided" in the POM (which means they are needed for compilation and testing, but will be provided in the classpath when the war is installed). Thus, I copied them all over to the tomcat/commons/lib directory.

Initial attempts seem to be resulting in NoClassDefFound errors. So it seems there may be a few jars needed still in tomcat alone... or that tomcat is not properly loading the jars in the common space (and that might be the case, I remember RODSA-DAI having issues running in tomcat becuse of library/classpath issues).

Either way, I am hoping to get it sorted out early tomorrow, have it returning data to the map. Then I want to get it tested on a secure-globus environment (in case some more security libraries are needed and to make sure that it plays nice with other secure clients like RODSA-DAI), and then I'll be ready to continue with the refactors.

Right now, the main refactorings have to do with shifting from a "state, zip3, zip5" paradigm to a "region" paradigm. This should allow for showing zip3s and zip5s and states all on the same map. A complimentary paradigm shift will be allowing multiple loads on one map (load one query with one set of pushpins, load another query with another set of pushpins... so you can click both sets of pushpins and compare data from two queries on the same map). All with more services and more realtime options.

Jar hunt complete

So, I managed to find all the jars that are needed to test (and eventually run) the use of the GIPSE client (at least the metadata portion of it)... and it is a long list pulling from both the CAGrid and globus libs (which thankfully end up in the globus libs). The very long list is in the pom file of the grid viewer project and I plan to post it in more plain text on the wiki tomorrow.

The thing is, I had to hand-install these items into the maven repository... which means that anyone wanting to test or build the gridviewer project at this point would have to do the same... and after setting up a grid-node and the GIPSE service, it is arduous and annoying.

Thus, I am hoping to get most of the jars needed into sourceforge. It was done once before for RODSA-dai already, and it should allow for people to get all the jars needed for testing after a simple property file change (which should be easily scriptable)

Furthermore, I am going to test running grid-viewer from the tomcat-enabled globus container... which should have all the libraries co-located in the lib directory, which will hopefully make it a much easier to install grid-viewer after setting up a tomcat instance.

I'm sure there will need to be some finessing either way.... and will still need to check all this in a secure globus environment and get the data returning in the grid view.

But, it's exciting. Nothing feels more gratifying than seeing your test code come back with something other than "NoClassDefFoundError: "

Compiling

So, I managed to take out all the conflicts and update all the code in grid viewer, so now everything is successfully compiling.

Tomorrow will be re-engineering the tests to match the new test cases, and then, hopefully, a war that can be installed and run (even if the initial phase is just having the service loader pull back the metadata options).

Then, it's implementing some new features in line with the current refactor, which will ultimately make things simpler.

I'm excited, I think grid viewer is turning into something that will work rather well, and still be flexible enough to work in ways not yet anticipated.

Service Changes

The spec for GIPSE has changed, and so has the client, since the last time I got grid viewer working. Thus, I have been doing a lot of refactoring, and planning more refactoring for some of the places we want to take grid viewer, and it's taking some time.

On the service side, all the names of the objects being returned have changed, and that means all the classes have to be changed and the ways they are loaded too. That also means the metadata has changed and needs to be set up.

Also, the grid service will now allow for cumulative loads... in that one can load more than one set of results onto the map.. which is a completely new paradigm that will have to be handled, in addition to the already shaky paradigms of multiple, variable regions. And the new idea of having services come from a central repository instead of a database (like a Wiki page or UDDI).

But, I think I have finally gotten it all mapped out in my mind, and have built the task lists, and have started the massive refactoring that will be needed.

The thing is, I know this will happen again (new service bits), so I need to keep in mind where all these changes occur and try to make them obvious and as isolated as possible. Then it is more likely that a change in the GIPSE structure can be propogated without a change in the grid view dynamics, and generally less changes the better.

Finally, I think I am going to introduce a simple "this is the data we got back from the service" page for the sake of debugging and sanity. It will help immensely to see what is coming back to figure out how the grid viewer is interpreting it and illuminating other options or assumptions.

New box!

So, a large portion of today was spent moving from the Linux development station I am used-to to a new Windows development station I am not-as-used-to.

Some of the things were easier to set up. The biggest help was being able to share other windows development boxes and being able to nab their already-downloaded copies of Java, eclipse, and Globus-ws core files. The setup for globus also seems to be a lot snappier in Windows (but at the same time, I am doing a much less involved install, and this isn't the first time I've done it).

Otherwise, I was able to build and run the client from the AMDS Service that was pulled over from another computer. Now I need to build and run the client from a fresh client downloaded from SVN and configured myself... so I anticipate a few "I am not sure what this property is" issues, but then it will be configuring the client, building the jars that grid viewer needs on my box (gmap-polygon) and figuring out what is needed between those steps.

Metadata niftiness

Grid viewer now has servers which are only selectable if the regions selected are chosen. Drop downs are now more based on metadata and not old quicksilver facets.

The next steps will be expanding the conditional handling, futhering the migration from a single-pane to two-panes with the top focusing on metadata and selections and the bottom focusing on the map and repeated queries.... and dealing with new servers and services and making sure all the metadata performs as expected (and probably a few paradigm shifts to allow changes in what 'expected' is).

A few things...

I have been busy with a few things over the past couple of days:

I upgraded GridViewer and Quicksilver so that they used timers (so that one can see which proportion of waiting lies in service calls or data rendering) and had an option for plotting pinpoints instead of polygons (pinpoints being that much faster and clear-cut, polygons looking neater and showing better spatial orientation). Quicksilver also has a link to metadata for servers so you can see what data is avialable from a particular server.

Right now I am working on making grid viewer behave based on what the servers can provide, this means tailoring the lists (states, zip3s, zip5s, conditions) so that one can only select regions that a server can provide data-for, and not everything in the region-relations table or conditions properties. I am hoping to have all the regions and the conditions selecting appropriately tomorrow, that way it will become much less confusing which regions have been asked-for.

After that, more User Interface niftiness is planned. Including hover-over for metadata stats, and a server list that changes based on the selected regions and conditions, but first I want to get the service pulling back data more reliably.

Cheers!

Pinpoints and Polygons

It has been requested to have the ability to display both pinpoints (located centrally) and polygons... namely because some zipcodes are a bit confusing and have multiple polygons.

I have spent a good part of yesterday and most of today enabling that functionality. It involved modifying several handlers and creating new classes to execute interfaces based on what was selected... Luckily, I did not have too many incidences of "oops, not OO enough", so it went smoother than I thought it would.

Otherwise, this is part of a host of speed-increasing tweaks. One of the other adjustments I am planning is to place timers throughout the application structure that both Quicksilver and Grid-Viewer use so that execution times can be displayed and give everyone a better idea of where time is spent between data loads.

After that, it's enabling the metadata in grid-viewer, and beginning the transition to an "available versus selected" framework which will better show that only certain regions will be selectable.

Planned updates.

Getting data into gridviewer has been a bit cantankerous... but now that it's there I know what I want to happen over next week.

- Update the structure of the grid-viewer to deal with items on an available versus selected. Having all the states show up when the service only allows for the Southeastern ones is going to be confusing... Thus, limit the possible selections to what is available. The same goes for services (if metadata could not be fetched for a service, don't let it show up as an option). The other added bonus is allowing for multi selection of regions... thus, you can pull data back for zip3s in two different states.

- Update the graph popups to populate zero values in a time series, as opposed to having them populated upon a service return. It's a bit complicated and behind the scenes, but it essentially moves some workload around and should speed up the client by having it do less work upon return.

- Start working with diversified clients. Things on different computers. Things behind different servers. Things in different states.

- Add bullets, and server return versus view render times.

- Have a histogram view, and update the time series to allow for differentiation of data source.

Either way, I am excited.

Client data in grid viewer

I managed to get AMDS client data in grid viewer after importing several libraries and manufacturing a local temporary client of my own. I am feeling much better now.

The next steps are to get that data on a map... and to get the client out of the "localtempclient" into a cool multi-client that adaptively uses different connection methods depending on the URL provided.

Also, I anticipate a lot of mavening and a lot of jar hunting over the next couple of days.

Metadata Madness Continued

Today, I got the code for the metadata capture executing through tests. I also re-learned a few things.

- Quick little stubs for imitating real services are neither quick nor little, but are always necessary if data is to be provided and you want to test your structures and plans.

- Use the types your given. Creating new types because they seem a bit clearer in your mind means you have to convert them.

- Draw things out. While what you code will rarely look like what you ended up drawing... it gets that tossable first attempt out of the way without having to do a lot of SVN adjustment.

Tomorrow, I try and show the metadata in the grid-viewer pane, and then I try and replicate data.

gridviewer is up

Greetings all,

So, this morning I posted gridviewer to the training node. Right now it is an empty shell being prepared for data and metadata, so it behaves a lot like the gmap-poly-web application, in that there is no data so the polygons will always show zero counts.

you can see it at http://ncphi.phgrid.net:8080/gridviewer/gmap-pane.jsp

The next steps are to set up a framework for handling multiple servers and metadata, reflect that capability in the framework by showing a changing server-list depending on which regions/time-periods were selected, and prepare for integrating Vaughn's AMDS client so that data can be fetched.

I am hoping to get to a "rodsadai" like state (click server A, get some data, click server B, get other data, click both, get combined data) for next Friday's release.

Cheers, and have a good weekend!