old code running on new box

A lot of strange issues knocked down, some new ones cropped up.

Issues knocked down:

- Some of the m2eclipse issues: jar projects will find and eat other jar projects just fine, war projects still go "I can't copy this" and give errors that force you to go to the command line where it works just fine.

- Quicksilver, gmap-poly-web showing on my new box: Meaning I got all of the geodata and user data transferred to the new database and connecting okay. I found some rows that got omitted in the transfer, added them, and replaced the CSVs.

- gmap-polygon and gmap-polyweb version 1.0 have been updated to reflect their proper version in their pom files (before they were considered 1.1, not 1.0).

New issues:

- For some reason M2Eclipse will look up maven artifacts in brians version of eclipse, but not the one I installed.

- IE on my dev box is super-duper-secure. Meaning it doesn't like downloading little things like jQuery or google-map javascript IDEs (but I can look at it from other sources of javascript)

- The GIPSE spec, and resulting client, have changed a lot from prior loads. This means lots and lots of code needs to be updated in gridviewer not just to use all the peices, but to reflect all the metadata options and the like.

new box, old code.

So, today has been a lot of "Try and get the code from the old system onto the new box"

And while it has not been particularly difficult, it has been rather tedious.

Most of the morning has been spent getting SQL Server Management Studio and figuring out that the data export for the CSV's of locational data from Postgres was done in some weird format (IE, dump from the terminal client into a text file) which caused all sorts of padding issues which needed to be repaired, and then updating the data back to the repository.

A lot of the afternoon was spent figuring out that there is a strange M2Eclipse behavior with our projects that keeps installed jars from being found (so, in the eclipse instance I would build gmap-polygon, install it, and then try to build gmap-poly-web, only to have it blow up because it couldn't find gmap-polygon.jar, even though it was installed... and when I did it from the command line, it worked just fine). So that is something I am going to have to debug because really, it is much easier to just right click and get the pretty gui to do it for you.

Another portion of the afternoon was spent fixing a vise-versa error where I thought that a branch was the spot for new code, and not the trunk... thus, I was moving recent code from branch locations back into the trunks (luckily, the eclipse/subclipse SVN browser works rather well for that sort of thing... but sourceforge's SVN is rather slow regardless)

By the end of today, I had some small issue that keeps me from being able to run any cool web-code on this new box:

Gmap-Poly-Web is not set to work with gmap-polygon version 1.1. I either need to upgrade gmap-poly-web to deal with the 1.1 version of gmap-polygon, or I need to download the 1.0 version of gmap-polygon and install it long enough to build gmap-poly-web (I will probably do the latter as gmap-poly-web functionality can be seen in both quicksilver and grid-viewer).

I don't have the username/password tables for poicondai on this box. Thus, I will need to find them and import them before I can log into poicondai.

Finally, AMDSCore has been completely replaced with a new service which will need to be integrated, the main deploy target is going to be having gridviewer working with this new service and thus providing more reliable data pulls.

This, if anything, has illuminated several points where we needed to update our "this is how you get/build/install this software" entries and shown us a few spots where we need to work to make things more automatic and user friendly if we want non-experienced programmers to have an easy time with it.

A few things...

I have been busy with a few things over the past couple of days:

I upgraded GridViewer and Quicksilver so that they used timers (so that one can see which proportion of waiting lies in service calls or data rendering) and had an option for plotting pinpoints instead of polygons (pinpoints being that much faster and clear-cut, polygons looking neater and showing better spatial orientation). Quicksilver also has a link to metadata for servers so you can see what data is avialable from a particular server.

Right now I am working on making grid viewer behave based on what the servers can provide, this means tailoring the lists (states, zip3s, zip5s, conditions) so that one can only select regions that a server can provide data-for, and not everything in the region-relations table or conditions properties. I am hoping to have all the regions and the conditions selecting appropriately tomorrow, that way it will become much less confusing which regions have been asked-for.

After that, more User Interface niftiness is planned. Including hover-over for metadata stats, and a server list that changes based on the selected regions and conditions, but first I want to get the service pulling back data more reliably.

Cheers!

Pinpoints and Polygons

It has been requested to have the ability to display both pinpoints (located centrally) and polygons... namely because some zipcodes are a bit confusing and have multiple polygons.

I have spent a good part of yesterday and most of today enabling that functionality. It involved modifying several handlers and creating new classes to execute interfaces based on what was selected... Luckily, I did not have too many incidences of "oops, not OO enough", so it went smoother than I thought it would.

Otherwise, this is part of a host of speed-increasing tweaks. One of the other adjustments I am planning is to place timers throughout the application structure that both Quicksilver and Grid-Viewer use so that execution times can be displayed and give everyone a better idea of where time is spent between data loads.

After that, it's enabling the metadata in grid-viewer, and beginning the transition to an "available versus selected" framework which will better show that only certain regions will be selectable.

more Quicksilver fixes, some progress on NPDSAMDS

This morning I dealt with some more issues raised in security scans of Quicksilver, as well as fixing an error they had revealed and improving the error handling.

This afternoon I went to the Grid kickoff meeting, and after that I was focusing on cleanup and implementing some more of the service implementation for NPDSAMDS

Updated Quicksilver

So, Quicksilver is being deployed to a version of CDC production, and there are scans being made, and few new possible vulnerabilities were made requiring some adjustments and some repackaging.

On the plus side, I am learning a lot more about how to secure a JSP website, and how to manage multiple builds at the same time.

Tomorrow, svn merging :D

Transitioning

Today I spent a some time transitioning with Felicia, learning where the docs are for dealing with library conflicts, that sort of thing.

Otherwise, I contributed a bit Brian and Vaughn's discussions about how grid services will deal with different security schemes and how they should be loaded, and I have a much better idea for how to design my NPDS-AMDS service and the new AMDS viewer client.

Most of today was spent helping the SDN team get Quicksilver ready for deployment. I helped debug a database issue, and wrote the "these are the properties files you need to change to deploy to a new server" guide.

Cheers!

learning grid services.

So right now I am essentially reading the grid book, learning how to make grid services, and learning how to secure grid services. Then, I will be playing with Introduce to see how it makes all this stuff much easier and much more automatic with the help of Felicia.

I also hope to pick Felicia's brain about implementing security using the client.

Either way, Globus services are a lot more than I thought they were, they have a lot more involved for maintaining state and essentially having all of their interactions marshaled and grouped, and it is just crying out for a shiny (if not leaky) abstraction to make it all pretty and less scary to people who aren't programmers used to fighting with Webservices and some of the gooey "I was generated by a machine" code that can come out of them. I look forward to those abstractions, but until then, I am going to be learning a lot of transport layer fun to get secure/amds-based quicksilver running.

Cheers

Quicksilver Code Complete

The deploy of Quicksilver to training has been complete. I have verified that the other applications are still working, and tested quicksilver on IE, Firefox, and Safari.

It can be reached here

The passwords have been changed to be more complex. Please contact Brian Lee at fya1@cdc.gov for your new password or login if you haven't gotten it.

Quicksilver now uses encoded gmap polygons. This reduces the size of the page sent back to the browser and greatly improves the performance. Internet Explorer no longer throws the "this script is taking a very long time" errors, and the polygons get more complex as you zoom in and less complex as you zoom out. They were encoded using the Douglas-Peucker algorithm described on this site.

The other improvements are tinkerings with the submit button (thus, it becomes disabled and says "loading..." when clicked) positioning improvements and a link to the Wiki help page.

This should be the Code Complete version of Quicksilver that will be deployed to the CDC SDN, and it has been labeled as version 1.0 and branched. New Features and behavior changes will be done to the 1.x branch and deployed to training, while bug fixes will be done to both the 1.x branch and the 1.0 branch and deployed to training and (if allowed) CDC SDN.

Thanks for your patience, I am very excited about Quicksilver and I think it has grown by leaps and bounds in functionality and niftiness from it's poicondai roots.

Otherwise, it looks like the next steps for me are to turn the service end of Quicksilver into an AMDS secure service, and to modify the visual end of Quicksilver (Gmap-polygon) to be flexible enough to combine and display data from multiple AMDS sources.

More immediately, I am going to merge some of the improvements I made for Quicksilver into the gmap-poly-web example project, and work with Felicia to learn the rudiments of a secure AMDS service and to work on making secure service clients coexist happily on the same server.

Yay for the Douglas-Peucker algorithm

So, encoding the polygons worked. The map now displays much faster and without any of the nasty "this script is taking a long time" errors that were plaguing it before, even in IE.

Firefox (and especially Chrome) are still faster than IE, usually with little things like scroll-zoom, but IE is no longer the annoying almost unusable experience it was before, which is good considering a lot more people use IE.

Otherwise, we are about to code-wrap version 1 for a deploy to the SDN, so the most recent dev build should be ported to training tomorrow morning. In addition to the polygon improvements, the passwords will be updated (and you'll have to contact Brian Lee at fya1@cdc.gov to get a new one) and outliers will be much harder to find considering I fixed the logic with the evaluation part of the C2 algorithm. There are some other small UI tweaks and I can't wait to show them to people.

This project has come a long way, and all the things I thought were near impossible turned out to be rather easy (consequently, all the stuff I thought would be easy turned out to be much more annoying than anticipated). Either way, I am excited.

Cheers!

Polygon encoding; or how to make your map of the US load faster

So, the problem we have right now that is causing a lot of slowdown (and annoying "this script is taking forever... continue?" errors) on a lot of browsers is that the polygons we have are too complex.

Things like Colorado or some city zipcode are simple enough... but states with long coastlines (California, Florida) or bordering rivers (Mississippi, Illinois) tend to have polygons with hundreds or thousands of vertices because of all the little crenelations that nature happens to draw on the country. Furthermore, even things like Colorado have zips that border rivers, and the end result is a browser having to download and process a LOT of javascript. At least 95% of the massive page draw is arrays of longitude/latitudes.

Google, however, represents these lovely little things called encoded polygons that take these thousands of points and turn them into two simple lines of text. I think they are the key to removing errors and speeding everything up. There are very in-depth summaries of encoded polygons and how to make them from lists of points here. I will try and explain my perspective anyways.

First off, the encoding is two dimensional in the data they store (hence two strings). The first string is a compressed representation of the points that make up the polygon, which saves lots of space because numbers are very easy for computers to de/compress. The second string is an indicator of which points should be displayed at what zoom levels. Thus, if you are zoomed way out (viewing all of the US) you don't need all the individual points on a river because, well, there could be 15 of them in one pixel of your monitor.

There is also a very neat algorithm that automatically determines the levels, and it is demonstrated here.

The best part, Google understands encoded polygons. Thus, less javascript processing for the browser, and since it's simple compression and algorithms, the server shouldn't have to spend many milliseconds converting the data on the fly. (and even if it does, it will only need to be done once and then cached).

I am hoping I can get this implemented Monday. They even have java ports of the encoding algorythm. I just hope it's rather straightforward, there are a lot of little extra usability tweaks to be made too and the deadline for this version of Quicksilver is relatively soon. And who knows, it might not help speed or errors that much.

But I have great hopes, and I think it will.

Cheers

Quicksilver updates

This afternoon we had a very productive session with the BioSense BIC/Epi team to review the latest build of Quicksilver.

They gave a lot of good feedback that fell into two categories: easy changes (cosmetic) and more complicated. The complicated changes include modifying the map to color the cloropleth based on number of outliers per period (rather than the arbitrary count ranges currently used). We're going to work on this after Dr. Tokars' team is able to analyze the data to find the correct break points for what is significant and what is not.

The easy changes have been made into a tracker item. Here's the list as submitted by Peter Hicks and Steve Benoit:

• Script error issues for every query requested


• For clinical effects, listing in alphabetical order would be helpful


• In legend scale, first option should be 0 (not less than 0), 1 to 10, 11 to 20, above 21. Currently, the categories overlap


• Label what these numbers represent in the legend (visits, calls, etc. ?)


• Adjustable legend option should be called “customize breakpoints or classification”. How to enter the customizable option is not clear or self-evident.


• Queries are extremely slow


• Calendar allows you to pick future dates – should have date limits


• Colors in legend don’t align with colors on map.


• A help section that describes the module and it’s components would be useful


• Call out box when mousing over state shows a time series – the average line is not an average for the time period selected. The outlier box is not clear. How do you define an outlier?


• Unable to understand this visualization and chart. We selected a region and got the call out box below. Was there 1 call or 797?

Deploy is complete. Come revel in new features

So, I have completed another deploy (well, two deploys actually) of Quicksilver.

You can reach it here, and you can read more about how it was built or how to view the code here

One of the new features is the "remembered zoom"... where the zoom for a selected region will be maintained if only the legend, timespan, or conditions change. Changing the region (from MD to IN, from MD to the 208 zip 3, the 208 zip3 to a view of all states) will revert to the default zoom level for that region type. But now, if you had to zoom in on Rhode Island... you won't have to zoom in again after selecting for a different condition or widening your search.

Another new feature is the adjustable legend... meaning that you can change the difference in the count numbers that determine the colors for polygons.

Finally the search dates default to the current week. This is not going to be that useful on the training node because we are using test data that only goes up to about October 2008, but when Quicksilver gets deployed to a production setting and starts getting access to more recent data, it will be much more helpful than a always starting on a random week in February 2008.

Meanwhile, lots of people had a good long look at the app today and came in with lots of feedback. It was both wonderful (because lots of people liked the application and thought it was neat and I got some ooohs and aaahs from things like flot) and terrifying (because I was worried it would break, people found ways to make the app do strange things, and because people tried things and thought of features that would be insanely cool to install that I never would have dreamed of). Having a bunch of users that are not that familiar with the application generates a LOT of very good feedback and questions. People were getting confused over things that, in retrospect, are not very clear at all. Today literally involved a large explosion of possibilities and potential, and it's as paralyzing as it is motivating. If anything because it's difficult to triage what should be done first.

So, next week will be a lot of implementations of little and big fixes. There are some very salient UI tweaks to be made (Like labeling more clearly, having "zip3: 208 // Total Count: 350" is a lot more handy than just "208 // 350") lots of little help pages and legend explanation (namely how the C2 algorythm means the blue average line and outlier status is based only on the average of the preceding 30 days (minus the closest two)) and finally, an attempt to make the polygon drawing much more streamlined to get rid of the really-quite-irritating "This script is taking a long time, do you want it to continue" error thrown by IE, which is exacerbated by having a not-bleeding-edge computer.

I think I have found a way to do that, and it's called polygon encoding, and I'll be detailing that in the next post.

Either way, I am elated and looking-forward to how nifty we can make this application.

Cheers,
Peter

Don't have memory zooms yet, but we do have caching and default start week.

I spent a long amount of today finding various ways that trying to selectively save a user's zoom wouldn't work.

The basics are there, I managed to get the user zoom saved to the page, so the app knows what the user chose before clicking submit time, it's just getting everything lined up so that the app only defaults to the previously selected zip when everything locationally stays the same... throw in the fact that I never really designed the application for that much state awareness from the get go (because I never anticipated someone wanting to save their zoom level) and it's being more difficult than it should and taking much more time than I feel people thought it would.

But, I took a break from that, and got some caching of the NPDS service values working. Now, the service doesn't have to be hit if a call was already made previously, that should help speed things up.

The other thing I implemented was a "default to current week" feature that was requested.

Tomorrow is the deploy. I might brainstorm some nifty way to enable saved zooms before the deploy tomorrow morning, but I can't make any promises. After the deploy I have blocked out some time with other developers to start finding ways to speed up the Quicksilver experience in IE. We have some ideas, we just need to go ahead and try them.

Cheers!

Little Quicksilver Usability updates.

So, now that there is an adjustable legend, I have added a real legend to better explain what the colors mean, and hidden the adjustable bit.

Otherwise, I spent the rest of the day figuring out a way for the app to save the users selected zoom level. That way if they just changed the dates or the search condition, but were zoomed in on a particularly tiny zip3 or zip5, the app would "remember" the last zoom level rather than forcing the user to zoom in from the default again.

Otherwise, I have been talking with Brian about ways to make the application faster. One is to cache averages and standard deviations... and the other is to have a servlet that returns polygon details and eval the polygon additions in a loop eval style. Hopefully this will allow IE users relief from the "This script is taking a long time" error.

adjusting legend

I moved a most of the logic of the legend adjustment and added much more error checking today.

Tomorrow I am going to try and get a more colorful legend, play with the placement, get click-through from the info panels (so you can click an IN link in the popup window and get it to reload focusing on IN) and then brainstorm with Brian for new search types and speedy loading.

Lots of little usability improvements to Quicksilver

Today I managed to get the scroll-wheel zooming and small map overview working on google maps. I also externalized the legend API so one can set new boundaries for what turns a particular region red or yellow or green. Tomorrow will be some debugging of that API, then some discussions of how to implement cool new features and reduce annoyingly long load times and "this script is taking forever" type errors.

C2 Flot deployed to training

Quicksilver with Flot showing counts, with C2 based averages and outliers has been deployed to the training node.

Quicksilver can be reached at: http://ncphi.phgrid.net:8080/npdsgmaps-web/
If you need a user/pass please let someone at PHGrid know.

Next week will be trying to add some small tweaks for the Google maps bit (like zooming and a small overview window) and a few larger features like multiple condition and region selects, and different options for data aggregation (merge all the counts for all the zip3s in Atlanta, for example).

C2 in popup, attached to Quicksilver

It looks like I have flot powered by the C2 algorithm popping up for states, zip3s and zip5s.

Tomorrow I anticipate some debugging, perhaps changing polygon shading/zooming, and perhaps trying some fixes to see if we can get Internet explorer loading more reliably.

Either way, I am happy that the new graphs are working. They are really neat.

c2 in a popup

So, I got all the mechanics working (I think, I will need Will Duck to go over the mechanics with me to make sure I am doing it right) for the C2 algorithm.

Furthermore, I have tested them and put together the mechanics to deploy javascript arrays for flot... including making sure I have GMT-centered timestamps with my counts (much more annoying than it seems it should be).

I pasted the arrays generated into my flot tester, and the graphs worked :D

Tomorrow, I will work on getting the flotplot.jsp working on fetching the appropriate polygon and forwarding it to the flot JSP to build the arrays.

Then, it's plug it all together, import it into NPDS and modify how the dates are pulled and hopefully.. new charts in Quicksilver.

Yeah, this has been a sentence-paragraph kind of day. It means I have been thinking like a machine for most of the day.

Cheers