Gridviewer improvements

We were able to complete a few substantial improvements to the Gridviewer application. The changes listed below were deployed on the training node (http://ncphi.phgrid.net:8080/gridviewer/):

1. Added download csv extract functionality per query request. (Click on Download Data to get an extract)
   
2. Added HTML data table display with sortable columns per individual query request as well as a combined data set . This feature is currently needing improvements in speed. Further improvements could contain subtotaling by regions or dates. ( Click on Display data link to see control. )
   
3. Corrected bug in the console log with the time displays.
4. Appended build version on the title and header.

PHMap and GIPSEPoison

So, I went away for vacation last week and when I got back, a lot of stuff had changed, but I feel it will make for better progress.

First up, I am supporting Chris in making PHMap... which may combine some of my service wrangling code and geo-fetching code with Chris' web code. This is fortuitous because Chris has a much better hang of actual web technologies like CSS, Spring, Javascript, and AJAX... and the concepts that make those all work smoothly together like REST and JSON. Thus, the code should be relatively clean and followable and adjustable, which is much harder when you have things like Java Objects which spitting out Javascript/HTML strings (like in my code).

This also means some of my service-wrangling and geo-fetching logic will probably be broken out or otherwise added-to to better enable the paradigm Chris wants to support . So essentially, refactors are upon us, and it will pretty much be the good old process of splitting function A into functions B and C since Chris will only need functon B.... and updating libraries and setting up property-less jar files for easy downloads instead of complicated builds.

Part of me feels a bit sheepish about how all the code sort of ended up mashed together and a lot of MVC lines were blurred... but then I remember that grid-viewer is essentially an evolution of RODSA-dai, which was a teeny little mashup that would have been overkill to load up with separate controllers and lots of different views. Then, suddenly, we needed polygons, and we needed to interact with the NPDS service, then we needed lots more UI tweaks and the ability to hit different services with different metadata.... and I must have been doing something right because I was able to make those modifications without too many large refactors, it's just now I get to deal with someone else actually needing to use it and not just install it, and their always runs the chance that the two paradigms for how something should work will be a bit obtuse, and larger refactors will be needed to accomodate both of them.

The other thing I will be doing mainly is getting GIPSEPoison working. I implemented a client and several tests and found that it is indeed getting data, but the aggregation needs to be tweaked, and some of the zip3 translations need to be taken into account. I think for that I will just split out some of the code I made for npdsgmaps (so it doesn't have any axis components) and make a new library that can be used by both npdsgmaps and GIPSEPoison.

Otherwise, lots of changes, lots of work, lots of progress, one hopes.

Cheers,
Peter

GIPSEPoison getting closer

Today I spent some time playing with GIPSEPoison trying to see if I could invoke the service... it seems I could but I kept getting caught up in classpath issues... first needing to import Dr. Jeremy Espino's CXF client for accessing the NPDS service... and then talking a bit with Brian about how he set up the build scripts to include data (properties files) from a "resources" directory in the classpath. So tomorrow I have a good list of things to tinker-with that will hopefully successfully punt data from npds through GIPSE and to the grid viewer.

Then I need to clean up a bit of the fun javascript logic around the magical autopopulating drop-downs (there are some strange inital load issues).

Otherwise, I have gone ahead and made a service registry entry for gridviewer to help aid people who want to install and set up their own gridviewers. I have also added some database definitions to the database loader project to help aid in geodata creation and described their use in that entry. Right now one needs to build and install the gmap-polygon project before they are able to build and use gridviewer, and it has pointed out to me that it's an annoying step that could probably be replaced with a no-props version of the gridviewer jar... so I am going to look into doing that soon as well.

Cheers,
Peter

Training updated

The training node was updated with the new caching gridviewer and the listed schemas were also updated.

In gridviewer, Dallas has been disabled for the time being because it seems to be acting rather flakey (will load fine for a half hour, and then times out). If anything it means I need to work in a good timer function to abandon a load and notify the user after a set amount of time.

Otherwise, gridviewer next week will be given some UI tweaks. I am hoping to get better server allow/disallow logic, multiple loads on one map (with different pushpins depending on the load), and autopopulated classifier/indicator drop-downs that don't require server hits.

Cheers everyone, have a good weekend,
Peter

Data is caching

Now the data for gridviewer is caching using OS cache. I may have to tweak it a bit to behave better as a singleton, but otherwise it is efficient and simple and seems to make everything quite snappy. Especially now that gridviewer is not referencing servers for metadata in each load.

Otherwise, the server url's are now pulling from the wiki page (or any page you wish to configure yourself). Thus, less database configuration is needed just for gridviewer.

more services on gridviewer

There are now some more services on Grid-Viewer, and there are some visual tweaks.

Unfortunately, the data from multiple servers is not combining because of some back-end restrictions that I am going to work on next week, some additional refactors include separating polygons from pinpoints (thus, coastal states won't have really dark shadows (the result of multiple pinpoints being drawn in the same spot)), and storing data from multiple states in the markers (thus, flotplots should be able to distinguish which data came from which service in future versions).

Also, caching (with the help of Brian), fun Javascript and layout changes (with the help of Chris), and new multi-plotting and regional overlays.

Cheers everyone, have a good weekend!

poicondai-2.0-noprops now available in the repository

Poicondai-2.0-noprops is sitting happily in the repository section of the website. Complete with a POM… and maven is actually nifty enough that when npdsgmaps references poicondai-2.0-noprops, mvn will read the pom and download all the extra dependencies needed for poicondai in addition to those needed specifically for npdsgmaps.

Simpler… if Project A needs Jar B… but Jar B relies also on jars C, D, and E… maven will read the pom associate with Jar B in the repository and also go fetch C, D, and E, even though C, D, and E, aren't listed as dependencies in Project A's pom file. At least it seems that way from experimentation.

So, if you want to use this jar, you will need to create a properties file with the url, systemuser, system name, and password (look into the poicondai project to see what the filter is filling), but it will possibly make it much easier for people needing to access poison to include this.

More importantly, it got me comfortable with moving things into a non-local repository (also many thanks to Felicia's posts), because I anticipate I will have to move more things into the remote repository for the sake of fixing a RODSA-DAI bug and for people who want to use grid-viewer (it will keep them from having to do a jar hunt like I have so many times).

Now the next step would be finding a cool way to get all the provided jars into the lib container of a webserver. Maybe there is a deploy plugin for that sort of thing.

grid-viewer updated and returning data

Yay! Grid-viewer and the Test Biosense GIPSE service have united on the training node to have grid viewer returning data (most of it is in January right now).

I'm happy because it is connecting over a secure connection to get both metadata and data, and it's relatively speedy, and allows me to move on with future plans for grid viewer and how it will behave.

First, I will be working with Chris to get a second GIPSE service testing, which will allow me to finish the server-selection logic (IE, you cannot select servers if the data you are looking for is out of their range. You won't be able to scan the North Carolina AMDS source for results in California or try to scan a BioSense node for Poison-based indicators).

Then, I am going to finish separating the selection bits from the mapping bits... which will require some generification: Right now, because it's based off of quicksilver, the map is expecting a series of states, a series of zip3s in a state, or a series of zip5s in a zip3... while it is already flexible enough to allow only a handful of states as opposed to all of them, I hope to expand it to allow for collections of zip3 (that may cross state borders) or zip5s (that may cross zip3 borders) that don't necessarily have to be related. It should also be possible to select states and zip3's at the same time (but I am not sure if that would be terribly helpful or just confusing).

Finally, I am hoping to allow for cumulative data loads. Thus, you can run one query, and then run another query with different pinpoints, thus you can click and compare the data between the two queries.

In addition to all this and in the process, I am hoping to clean up some of how gmap-polygon behaves so that less data has to be stored in databases and things aren't repeated as often.

It's going to be a lot of work, but I think the end results will be rather nifty, and I'm just happy to have this version all wired together and working so I can move on.

old code running on new box

A lot of strange issues knocked down, some new ones cropped up.

Issues knocked down:

- Some of the m2eclipse issues: jar projects will find and eat other jar projects just fine, war projects still go "I can't copy this" and give errors that force you to go to the command line where it works just fine.

- Quicksilver, gmap-poly-web showing on my new box: Meaning I got all of the geodata and user data transferred to the new database and connecting okay. I found some rows that got omitted in the transfer, added them, and replaced the CSVs.

- gmap-polygon and gmap-polyweb version 1.0 have been updated to reflect their proper version in their pom files (before they were considered 1.1, not 1.0).

New issues:

- For some reason M2Eclipse will look up maven artifacts in brians version of eclipse, but not the one I installed.

- IE on my dev box is super-duper-secure. Meaning it doesn't like downloading little things like jQuery or google-map javascript IDEs (but I can look at it from other sources of javascript)

- The GIPSE spec, and resulting client, have changed a lot from prior loads. This means lots and lots of code needs to be updated in gridviewer not just to use all the peices, but to reflect all the metadata options and the like.

new box, old code.

So, today has been a lot of "Try and get the code from the old system onto the new box"

And while it has not been particularly difficult, it has been rather tedious.

Most of the morning has been spent getting SQL Server Management Studio and figuring out that the data export for the CSV's of locational data from Postgres was done in some weird format (IE, dump from the terminal client into a text file) which caused all sorts of padding issues which needed to be repaired, and then updating the data back to the repository.

A lot of the afternoon was spent figuring out that there is a strange M2Eclipse behavior with our projects that keeps installed jars from being found (so, in the eclipse instance I would build gmap-polygon, install it, and then try to build gmap-poly-web, only to have it blow up because it couldn't find gmap-polygon.jar, even though it was installed... and when I did it from the command line, it worked just fine). So that is something I am going to have to debug because really, it is much easier to just right click and get the pretty gui to do it for you.

Another portion of the afternoon was spent fixing a vise-versa error where I thought that a branch was the spot for new code, and not the trunk... thus, I was moving recent code from branch locations back into the trunks (luckily, the eclipse/subclipse SVN browser works rather well for that sort of thing... but sourceforge's SVN is rather slow regardless)

By the end of today, I had some small issue that keeps me from being able to run any cool web-code on this new box:

Gmap-Poly-Web is not set to work with gmap-polygon version 1.1. I either need to upgrade gmap-poly-web to deal with the 1.1 version of gmap-polygon, or I need to download the 1.0 version of gmap-polygon and install it long enough to build gmap-poly-web (I will probably do the latter as gmap-poly-web functionality can be seen in both quicksilver and grid-viewer).

I don't have the username/password tables for poicondai on this box. Thus, I will need to find them and import them before I can log into poicondai.

Finally, AMDSCore has been completely replaced with a new service which will need to be integrated, the main deploy target is going to be having gridviewer working with this new service and thus providing more reliable data pulls.

This, if anything, has illuminated several points where we needed to update our "this is how you get/build/install this software" entries and shown us a few spots where we need to work to make things more automatic and user friendly if we want non-experienced programmers to have an easy time with it.

Gar issues...

Today Vaughn and I tried to get the client for the amds service on my machine so we could develop against it. It didn't work.

First we tried to build his code on my machine, which failed because my machine is not his machine and therefore had a slightly different version of globus... or ant... or user setup... or some other variable that is yet to be discovered that usually pops up during the first couple of attempts like this.

Then we tried to bring over the gar he had built on his machine... and deploy it... only to have the client blow up whenever I tried to run it citing missing class exceptions... despite the fact that all the classes that should have been in the gar.

So, I am still wanting for an AMDS client, and the resulting checklist of code and build and configurion points is long and mildly upsetting. I'm wondering if making a quick little AXIS or CXF service would be worth it just so I could develop against the resulting classes and client examples while the Globus connections were ironed out.

I'm sure it will get sorted out eventually, but when I was working with Quicksilver, I had the benefit of having the service from day one. This time, I have build out a lot of the grid viewer on assumptions of how the client and service will behave, and I worry about increasing the chance of needing to refactor things once it gets booting and connected because my assumptions were off.

Oh well, growing pains. We'll get through it... and I'm sure what we make will be rather cool. It's just "that phase" in the project where things are going wacky all over the place when one was hoping they would just fall into place and it's extra difficult to take the calming breaths you need to collectedly analyze the situation.

The guts are there, the data isn't.

I spent a good portion of today wiring up all of the behavior for taking AMDS service data and populating it into polygons, but I have had issues with the client so I cannot get data, and I didn't have enough time to make a data-faking routine. So there isn't going to be anything for me to deploy tomorrow.

I am guessing tomorrow I will get the client working and then will spend a lot of time debugging the behavior, and then I will spend a lot of time actually flattening and splitting the google map and the server selector next week.

The flattening will allow for much better multi-selection of regions and conditions... I just hope the usability isn't too impacted. But I figure it will be rather obvious: first pane is to select the servers you want to pull-from... second pane is to navigate the data with the given charts/maps and better highlight the data from the chosen servers.

Metadata madness.

Today, I took the first step in getting to Rodsadai-with-grid-viewer settings, setting up how all the metadata would be stored, and discussing it with other people.

For now, we have settled on storing the server names and URL's in a database, and then populating the metadata from the service(s) dynamically. Thus, most of today has been spent filling out the infrastructure for a one-time database load in gridviewer.

Brian and I also discussed how the app would behave. Instead of the Quicksilver-esque selection and then map... there will be an intro page that will probably need the real-estate formerly taken by the map to show a server-list. The given list will change depending on what date ranges, regions, and conditions are picked (based on whether or not the servers support said items) and then, after the ranges and servers are chosen, the map will be loaded and viewed. Another nifty discussed was to have metadata popups over the available servers, so people would get an idea of which limitations each server had.

Otherwise, the big next step will be integrating with the client app Vaughn is developing. It will be needed to pull the metadata for given services and set up the loading pane.