new box, old code.

So, today has been a lot of "Try and get the code from the old system onto the new box"

And while it has not been particularly difficult, it has been rather tedious.

Most of the morning has been spent getting SQL Server Management Studio and figuring out that the data export for the CSV's of locational data from Postgres was done in some weird format (IE, dump from the terminal client into a text file) which caused all sorts of padding issues which needed to be repaired, and then updating the data back to the repository.

A lot of the afternoon was spent figuring out that there is a strange M2Eclipse behavior with our projects that keeps installed jars from being found (so, in the eclipse instance I would build gmap-polygon, install it, and then try to build gmap-poly-web, only to have it blow up because it couldn't find gmap-polygon.jar, even though it was installed... and when I did it from the command line, it worked just fine). So that is something I am going to have to debug because really, it is much easier to just right click and get the pretty gui to do it for you.

Another portion of the afternoon was spent fixing a vise-versa error where I thought that a branch was the spot for new code, and not the trunk... thus, I was moving recent code from branch locations back into the trunks (luckily, the eclipse/subclipse SVN browser works rather well for that sort of thing... but sourceforge's SVN is rather slow regardless)

By the end of today, I had some small issue that keeps me from being able to run any cool web-code on this new box:

Gmap-Poly-Web is not set to work with gmap-polygon version 1.1. I either need to upgrade gmap-poly-web to deal with the 1.1 version of gmap-polygon, or I need to download the 1.0 version of gmap-polygon and install it long enough to build gmap-poly-web (I will probably do the latter as gmap-poly-web functionality can be seen in both quicksilver and grid-viewer).

I don't have the username/password tables for poicondai on this box. Thus, I will need to find them and import them before I can log into poicondai.

Finally, AMDSCore has been completely replaced with a new service which will need to be integrated, the main deploy target is going to be having gridviewer working with this new service and thus providing more reliable data pulls.

This, if anything, has illuminated several points where we needed to update our "this is how you get/build/install this software" entries and shown us a few spots where we need to work to make things more automatic and user friendly if we want non-experienced programmers to have an easy time with it.

Quicksilver Code Complete

The deploy of Quicksilver to training has been complete. I have verified that the other applications are still working, and tested quicksilver on IE, Firefox, and Safari.

It can be reached here

The passwords have been changed to be more complex. Please contact Brian Lee at fya1@cdc.gov for your new password or login if you haven't gotten it.

Quicksilver now uses encoded gmap polygons. This reduces the size of the page sent back to the browser and greatly improves the performance. Internet Explorer no longer throws the "this script is taking a very long time" errors, and the polygons get more complex as you zoom in and less complex as you zoom out. They were encoded using the Douglas-Peucker algorithm described on this site.

The other improvements are tinkerings with the submit button (thus, it becomes disabled and says "loading..." when clicked) positioning improvements and a link to the Wiki help page.

This should be the Code Complete version of Quicksilver that will be deployed to the CDC SDN, and it has been labeled as version 1.0 and branched. New Features and behavior changes will be done to the 1.x branch and deployed to training, while bug fixes will be done to both the 1.x branch and the 1.0 branch and deployed to training and (if allowed) CDC SDN.

Thanks for your patience, I am very excited about Quicksilver and I think it has grown by leaps and bounds in functionality and niftiness from it's poicondai roots.

Otherwise, it looks like the next steps for me are to turn the service end of Quicksilver into an AMDS secure service, and to modify the visual end of Quicksilver (Gmap-polygon) to be flexible enough to combine and display data from multiple AMDS sources.

More immediately, I am going to merge some of the improvements I made for Quicksilver into the gmap-poly-web example project, and work with Felicia to learn the rudiments of a secure AMDS service and to work on making secure service clients coexist happily on the same server.

Flot more flot!

Graph two lines (check)
Have a zoom-able overview (check)
Have the x-axis with dates (check)
Have float-over tool-tips that reveal data (check)
Have a legend to distinguish the data points from the average points (check)
highlight points that are outliers in some way. (check!)

So, got them all working! Now the next step is to get the charts building inside gmap-polygon popup windows, and of course tackling what I am sure will be a fun exercise in getting Java dates converted to JavaScript dates... Then comes the codifying of the C2 algorythm.

Bug Fix Monday.

Over the weekend, I started just googling ways to deal with the Operation Aborted error in GoogleMaps for Internet Explorer... I found this and have implemented it in the gmap-polygon classes, and it seems to be working (at least I wasn't able to get IE on the semi-local box to throw the error... might be a completely different case on boxes not in the next room over)

Otherwise, I have added a lot more parameter checking to the Gmap-pane and the login.jsp.

Tomorrow, Flot. I hope to have a Flot-ted chart made (perhaps with fake data) and maybe even integrated by tomorrow evening.

Not only are there new calendar controls...

But now YOU get to play with them.

The new version of Quicksilver is on the staging node, Read about it and find the link to the app by scrolling through the QuickSilver Service Registry Entry in the PHGrid Wiki.

The service should no longer crash on the Java end... and if it does, I would very much appreciate a quick note about what you were doing to crash it. Otherwise, I suggest poking the service with something not Internet Explorer because Internet Explorer is still a bit testy about the large amounts of javascript processing and might throw an intermittent "Operation aborted" error.

Next week will be working on password-protection capabilities and cool new graphing software. I will also be trying to squish the IE bug and reduce load times. I am looking forward to it!

Cheers!

We have calendar controls, we have clinical effects.

Tomorrow, Should everything go well, a much more functional version of Quicksilver should be deployed.

It will be able to select for dates and clinical effects, will display all the states from the national level, and will be able to zoom in on states (which will show a collection of zip3s) and on zip3s (which will show a collection of zip5s) and on zip5s.

There will be Google charts for all the regions that have a color (which means they have more than one instance) and I have not been able to get a nasty error thrown after doing a lot of odd and strange tests.

Next week will be the adding of cooler charts (with scroll and zoom and hopefully much more nifty statistical significance) and some security layer stuff so that it will be easier to password protect access to the page.

Cheers!

We now have zip5s. Next stop, Dates and Clinical Effects

Today I managed to get zip5s working on the demo node. I ran some pretty thorough tests and I don't think there are any bugs (which is sort of scary, I am sure I will find some tomorrow).

Tomorrow, I am going to pull over code for date selection and clinical effect selection from the old poicondai screen, make sure the values are going into the right places, and probably improve the clinical effect selector to allow for multiple selections (and maybe even use something from jQuery to allow for searching the set of possible clinical effects).

Friday, it should be posted to the web and able for people to play-with.. and I'll start drafting all the changes for making Gmaps-polygon and Quicksilver that much cooler, namely by adding more security and niftier graphs. All while testing Windows and SQL-Server compatibility.

Speaking of cool new graphs... check out some of the blog entries on the various statistical engines and AJAX visualization possibilities. I am probably going to pick one of the more AJAX (as opposed to flash or java-applet) friendly ones and tinker with it on Friday after the deploy. I would be looking at them more but I have been busy with just getting zip3's and zip5's to work without throwing big nasty errors. Many thanks to Brian Lee and William Duck for helping me look into the statistics engine pieces (and discussing the various benefits and drawbacks of each over email).

Also, a big thanks goes out to Dale, our grid-vmware-engineer as it were. He has been a real help and very considerate of our issues with the lab/demo environment, usually fixing problems within a day and working with us on planned outages and upgrades to minimize downtime and the need to restart things. He has been doing it for many, many months and I just realized I hadn't said anything about how hard he works to make sure we're good in terms of networks and Virtual Machines.

Bug smacking.

Continued smacking some bugs I was finding during my development tests... the more memorable one being crashing upon empty zip3 data sets.

I also started getting ready for zip5 collection, and continued discussions about the best way to get state or zip-code limited geocentric data back from the NPDS Service.

I have also started talking with Geographic Information Specialist named Julie from the Maryland Poison Center, and learned that some CDC NCPHI residents are also going to be around to learn more about Globus and the applications we are researching and piloting.

Otherwise, I am also running over, in the back of my mind, the upgrades of Quicksilver and NPDS-Gmaps. After Zip5s and User Interface controls are added, I feel like the next step is some subtle re-architecture of the apps so that they can deal with things like Clean error propagation and better value checking for security needs. It will also tie in nicely with implementation of the new statistical and charting methods so that the information returned by Poicondai can be highlighted with the benefit of actual statistical analysis and not just "hey, look, a graph."

Bugs fixed, but were breeding.

So, I fixed the Quicksilver bug about data replicating on the state-level on dev. Thus, after the next deploy, one will be able to load the states as many times as possible for a given date range and the data will stay the same.

Otherwise, I spent the rest of the day trying to find a fix for the "if I try and load California zip3's, the service crashes" problem... the root of the cause seems to be that I am generating a list of all the zipcodes in California and sending it to the Service, and the list is rather long, longer than the comfy limits of the service.

Luckily, I found a way around it, I can just send a request for the state and not all the zips in said state and it will still aggregate them by zip code.

The only problem is that I can't specify a state with the "main" aggregated national service. So I either have to pull back ALL the zip codes for the entire country (which would take a minute or two) or limited data from only the Colorado poison center. I have sent an email to the lovely poison people to see if there is something I am missing that will work for my needs.

But otherwise, yay. Less horrible crashes and strange behavior while we get the data sorted out.

The other cool thing is I had a chat with William Duck and he is pretty much a statistical Guru. He is helping me look for statistical and web-chart libraries that will help provide better and niftier information via-chart than the nifty, but not as functional google charts (or maybe even see if google charts can do more complicated stuff). That will be a massive help.

New stuff, and a bunch to do.

The updated demos for Gmap-Polygon and and Quicksilver are now on the web, read the Service Registry entries in the wiki to learn more.

In the meantime, there is still a bunch of stuff to be done on Quicksilver and a few things to tweak on Gmap-Polygon, so here comes the list.

1. Gmap-Polygon


1. Bugs
   
   1. Internet Explorer sometimes throws an operation aborted error. I am not sure what to do to fix this as it is sporadic, comes with no extra information, and is related to Javascript.
      
   2. Focus is maintained between different sessions, so someone on one computer has picked California, the other person logging in will see California.
      
   
   
2. Speed
   
   1. Need to get smaller (less complicated, fewer vertices) zip3 polygons so that zip3s of states like California and Florida take less time to transfer and load
      
   
   
3. Security
   
   1. Need to bubble-up errors and make it so that simple, non detailed error notifications are given in a production setting
      
   2. Need to check parameters for validity, warding off various injection attacks
      
   
   


2. Quicksilver


1. Bugs
   
   1. Data is aggregating, not updating, in state counts. Need to make sure that time series replace in new polygons
      
   2. States with a lot of zip-codes are causing errors upon attempted zip3 loading, the string is overloading the service. Need to mitigate by seeing if ziplist length can be extended or by breaking up call into multiple lengths.
      
   3. State data is being loaded from the Colorado NPDS Service and not the global NPDS Service
      
2. Unimplemented features
   
   1. Zip5s need to be loaded and displayed
      
   2. The ability to select dates and conditions need to be added in accordance with the use case
      
   3. The graph-viewer needs to be updated to display a much more statistically useful graph (outliers flagged by different colors, possibly zoom-able)
      
3. Security
   
   1. Need to add security so that login/pass must be supplied before any map information is viewable.
      
   2. Need to double check values and propagate friendly/non-detailed error should values not be invalid.
      
   
   

There are probably a few things I am forgetting, and in addition to all these feature add-ons, I will be checking compatibility and configuration compatibility with different databases and the like.

Either way, I am looking forward to making things niftier and more robust. I hope you all have a good weekend!

Cheers,
Peter

Oh, Internet Explorer... why do you not love GMap Polygon?

So, Internet Explorer seems to have a fickle relationship with GmapPolygon.

Sometimes, when it is loading a set of polygons, it will throw an error saying "The page at [address] cannot be loaded. Operation Aborted". This is a thoroughly unhelpful error, because it doesn't offer to show me the part of the code that broke or otherwise debug the issue. Furthermore, it's not re-create-able on demand. It tends to do it whenever it feels like it, for any given set of polygons... polygons which loaded fine not three seconds before.

Some of the blogs I checked (example) indicate that such errors arise because IE was processing some javascript, freaked out, and cancelled the page load... Usually because some spurious parent element is being referenced. The problem is that in the javascript I don't think any parent DOM elements are being referenced, just maps which are created inside the head script.

I'll admit, I am not that familiar with javascript, so nothing really stands out to me that makes me go "oh, that's what is making it blow up sometimes." So I invite anyone to look at the source of polygon pulls (I suggest poking the site with firefox) to see if anything stands out.

Otherwise, I imagine a lot of the problems involve the fact that a polygon is represented by a metric ton of text... especially polygons that border rivers and coastlines... such sweeping, beautiful crenulations that only nature can carve out are represented digitally by thousands of vertices. which all need to be pulled back from a database somewhere and then provided to google maps.

Meanwhile, I am attempting to make Gmap-Polygon get those back end database connections to be more robust and reliable, and improve caching so that the database doesn't even have to be hit that often... but we are also just looking at getting simpler polygons so that there is less data to pull, transfer, and store, which should have the biggest impact.

Lots of things, little and big

Sorry I haven't posted in a while, I have been dealing with a whole bunch of issues with Gmap-Polygon and the new Poicondai which has been dubbed "Quicksilver"

In short, there are bugs in both, gmap-polygon will be upgraded with more robust data persistence, we need to find ways to mitigate the issues that come with slightly different environments needing slightly different runtime variables (can we force them to be static? Is there a way to set them with an external config file?), and there is a lot more coding to be done, namely getting the zip3 graphs repaired and the zip5s displaying in Quicksilver... and also replacing the old style google graphs with niftier graphs that have cooler statistical analytics and coded outliers and the like.

I'll get the big list posted as soon as I find out a good way to get ordered list code posted into blogspot.

Gmap-Polygon now demonstrable

Greetings, if you check out the new service registry entry you will find a description of the gmap-polygon project I have been working on and a link to a demo.

Now, I will work on taking the revamped NPDS service and getting it to populate the polygons so that little line graphs and colorations occur instead of just "0 cases" showing up all the time.

The other idea is that anyone can use this project if they need a google maps polygon mapper, especially if you need it to work in a secure environment and don't think you would be able to publicly expose a KML file.

Either way, I am happy with the results, I think it is nifty, and the more you poke things the more they cache and the faster they should (hopefully) load.

Cheers!

Polygon Centroid Updates.

So, I have gotten the table and the logic set up for centroids so that the map will automagically zoom in on selected states and zip3s, but I need to tweak a few things and ran out of time today. I am hoping to get things set tomorrow morning, move some of the java-in-jsp code I have made into the actual helper java bean, and then plan out a migration of the sample to the staging node for a Thursday demo (so other people can tinker with the gmap polygon drawer).

After that, it is time to sit down with all the NPDS sample services and see if I can't get the NPDS-service that Dr. Jeremy Espino put together (who has already suggested a bevy of new technologies to look at) working with some of the new constructs and creating new time-series to put into the polygons.

Then it is a hop-skip and debug session from the new NPDS demo, which will be the marriage of the gmap-polygon tech with the new NPDS service.

I'm hoping it shall be sufficiently nifty.

Gmap Polygon coming along nicely

I now have states, zip3s and zip5s all loading and caching nicely in GmapPolygon. The next step is to make it prettier on the back end and easier for other applications to load data.

Also, while the polygons are all in the database, multiplied tho they may be, I need to get the centroids into the database so that the map zooms in better for when individual states or zip3s are picked.

Another thing is that the NPDS service will be updating, so I am finishing up some of the tweaks to the gmapping service just in time.

I am going to go ahead and schedule a deploy early next week so people can start playing with it and seeing how the mapper behaves.

Zip3's now much better.

Well, the zip3's are now all showing up. But, as you can see from the wonderful comment that Jim left, , zip3's are so-so geographical locators and spurious populational references. As Jim pointed out, having one contiguous zip3 for 'Manhattan' pretty much makes epedemiology by zip3 useless in New York City. Then again I don't think we're using Zip3's for epedepiology, they are "things which are bigger than zip5s but smaller than states".

But this all brings up a good point: Our system is going to have to be all sorts of flexible. This is because we are dealing with GeoData, and in addition to GeoData changing all the time (zipcodes move, census numbers are updated), everyone will want to use a different set.

I don't think anyone will ever agree on a good way to geographically handle reporting systems like RODS or Poicondai, and the requirements will be completely different depending on whether things are being veiwed based on geography or population density, so I feel like the easiest thing we can do is allow for our systems to easily start using whatever new-fangled (or old-fangled) visualization technique is preferred at the time by whomever wants to use our system.

I feel like Gmap-Polygon is a good step in that direction. It basically is a way to put polygons on the map for whatever region you want to draw, so as long as you have the polygons put into the database before hand, and you know how you want to shade them, you should be able to use the software easily. This is of course assuming that you wanted to put it in a secure portal and didn't just want to expose KML (because for drawing polygons with nifty shading and effects, KML is a bit more straightforward).

I feel like the other thing that PHGrid products will need to start focusing on are ways to securely associate data, because a lot of people will want to focus on things like the Census Tracks that Jim mentioned, and no-one will know what census track they live in when they go to a hospital, and pulling up address data for a count item is going to be a very big no-no for things outside the secure network. I wonder if one of the applications of the Natural Language Processor can be backfilling the AMDS regional info with census track data and the like.

A build Point

I have reached a point where I was able to hand the code for gmap-polygon to Felicia so she could use it to draw polygons for data that her services were returning. Many thanks to Felicia and Brian. Felicia for her patience for all the "oh crap, that's not doing what I thought, sync and try again" moments. Brian for doing some textedit niftiness to turn the KML files into easier to parse flat-files.

Otherwise, I am going to be off on vacation for the next couple of days, and some of them Felicia will still be coding, so I am glad I got a chance to explain the inner workings of the code and that I spent a lot of time trying to make it as agnostic as possible.

Otherwise, that being said, I really need to step up my Javadoc documentation. I'm committing to spending the last hour of each day documenting all my new stuff (and old stuff I forgot to document) and checking it in. That way, people who AREN'T sitting right next to me will have a much easier time figuring out how to do stuff and why I did the things I did the way I did them.

Cheers, and Happy Holidays.

A plague of ticks (and by ticks, I mean little truncating errors)

So, I am sort of excited because I think that I am ready with Gmap Polygon, but on Friday I noticed that some of the polygons weren't getting drawn well at all, namely because the polygon strings (long lists of coordinates separated by spaces) were way too short and were getting cut off.

So today I spent a lot of time monkeying around with the little SAX engine I was using, and now I am sure my issue is that I need to just not use the sax engine anymore, because while it was a quick class and rather fast, it is really screwing up some of the stuff I am trying to do and some of the cases I am trying to handle and I am no further along figuring out how to make it stop doing that.

So tonight, I am going to read up on JAXB and maybe work a tutorial, and tomorrow, I hope to get this fixed.

Otherwise, there are some other fun questions about what everything will end up looking-like and how crazy some borders are. It seems that zip3s can be inside other zip3s and it also might seem there are more than one area covered by a zip3. I guess that's what happens when you base your centers on mailing routes. Zip3s have a lot of 'outer boundaries', that make me wonder about the best way to store the polygons, we might have to go to a many to one relationship.

I'll keep ya'll posted, and hopefully there will be a cool little demo app soon next year, or even cooler, my mapper on top of Felicia's RODS service.

Cheers!

Gmap Polygon objectier.

I basically have the test I was running before working. Ironically, there are still about 5 lines of java code in the JSP when there should be more like 2 or 3.

But, this is because I need to go ahead and implement the framework that will provide the polygons rather than coding some into the test JSP. This means the two courses of action are to reformat poicondai to use the new system, or to make a set of forms and controllers that are more generic and allow for the automation of a lot of the map controls.

After some discussions with Brian and Felicia, it appears I will be working on the latter: A set of generic map controls... thus making it easier for anyone else to use the map controls for their own display needs.

I think it will also make it even easier to revamp poicondai and integrate with Felicias RodsHDS.

ColorHandler updated, and boy is it huge.

Well, it seems that having a range-conscious coloring object is a bit more difficult than even I was expecting, especially considering that I am allowing the use of nulls to mean "there is no limit, therefore it is infinite" for things like "all cases equal to or less than 0 need to have white shading" and "all cases above 20 need to have red shading".

But, I got it to build, which means I wasn't blowing up something obvious. Tomorrow I will saddle up the test harness for the color processor and run it through it's paces.

Then, it is building a very simple popup handler with a simple template. Here's to hoping it is simple.

Alas, this means it might be until friday before I have another good tester of the GMap Polygon jsp (but hopefully, it will be very small and just sort of go 'create grid map object, draw grid map object'.