So, encoding the polygons worked. The map now displays much faster and without any of the nasty "this script is taking a long time" errors that were plaguing it before, even in IE.
Firefox (and especially Chrome) are still faster than IE, usually with little things like scroll-zoom, but IE is no longer the annoying almost unusable experience it was before, which is good considering a lot more people use IE.
Otherwise, we are about to code-wrap version 1 for a deploy to the SDN, so the most recent dev build should be ported to training tomorrow morning. In addition to the polygon improvements, the passwords will be updated (and you'll have to contact Brian Lee at fya1@cdc.gov to get a new one) and outliers will be much harder to find considering I fixed the logic with the evaluation part of the C2 algorithm. There are some other small UI tweaks and I can't wait to show them to people.
This project has come a long way, and all the things I thought were near impossible turned out to be rather easy (consequently, all the stuff I thought would be easy turned out to be much more annoying than anticipated). Either way, I am excited.
Cheers!
Tuesday, March 31, 2009
Updated AMDS Draft Schema
I went ahead and formalized a lot of the talks around the AMDS schema we've been having this winter and spring and updated the schemas on the wiki. Although this namespace is 20090330, it really has been around since January and isn't a major change from the v1 that was posted back on December 15.
The amds schema boils down to just two messages for the service:
This is much smaller than the earlier version since we may as well start with a single operation.
The MetadataQuery is built into the service spec so that a user or registry can programatically check for what capabilities a service provides. Thanks Jeremy for this idea as it's a lot easier than trying to keep the service registry up to date manually.
The amds schema boils down to just two messages for the service:
- MetadataQuery/Response - Returns the appropriate metadata for each service. This exists so that clients and registries can determine metadata through a runtime API rather than rely on an administrator to manually enter.
- AMDSQueryRequest/Response- This takes in a query of conditions, regions and a date range and returns an array of counts by condition, region and day.
This is much smaller than the earlier version since we may as well start with a single operation.
The MetadataQuery is built into the service spec so that a user or registry can programatically check for what capabilities a service provides. Thanks Jeremy for this idea as it's a lot easier than trying to keep the service registry up to date manually.
Monday, March 30, 2009
BioSense AMDS Extract
John, Tom and I met with the BioSense data team to discuss the best way to provide aggregate BioSense data for AMDS. Until we can prepare an automation routine to determine aggregate data, the data team will generate a trailing 30 day report for the 11 syndromes (using the Chief Complaint bucket for Real-Time sources and the Outpatient Final Diagnosis bucket for VA and DoD).
This report will be parsed and loaded into the BioSense extract database for querying using the AMDS service for BioSense.
All this is near future (before June) kind of stuff. So we are getting rather close to being able to pilot AMDS-BioSense securely in a proper production/staging environment.
The release of AMDS-BioSense would be the second step in releasing the BioSense Grid Publisher (the first being the planned release of the Grid Publisher node).
This report will be parsed and loaded into the BioSense extract database for querying using the AMDS service for BioSense.
All this is near future (before June) kind of stuff. So we are getting rather close to being able to pilot AMDS-BioSense securely in a proper production/staging environment.
The release of AMDS-BioSense would be the second step in releasing the BioSense Grid Publisher (the first being the planned release of the Grid Publisher node).
Encoded Polygons draw and popup
Most of today was spent making a series of UI tweaks that were requested. Now the legends are adjusted a bit, some extra labels were added, and a link to the help page is on the development version.
I wanted to play with the encoded GMapPolygons before I went home, so I made sure I could replicate the functionality that I already had, and made thus, with the help of Alabama, I now know I can place an encoded gmap polygon with the same click-for-popup and color/shading properties.
Tomorrow will be spent enacting this. I am probably going to try setting up on-the-fly translation, and then work on storing encoded polygons in the database (namely because I can think in my head how to do an on-the-fly translation and there are some tricky bits with database storage of dual strings that I don't want to start there...
Either way, here's to hoping for a vast improvement in performance by tomorrow afternoon.
I wanted to play with the encoded GMapPolygons before I went home, so I made sure I could replicate the functionality that I already had, and made thus, with the help of Alabama, I now know I can place an encoded gmap polygon with the same click-for-popup and color/shading properties.
Tomorrow will be spent enacting this. I am probably going to try setting up on-the-fly translation, and then work on storing encoded polygons in the database (namely because I can think in my head how to do an on-the-fly translation and there are some tricky bits with database storage of dual strings that I don't want to start there...
Either way, here's to hoping for a vast improvement in performance by tomorrow afternoon.
BioGrid Australia - Health through information - New Site
BioGrid is a unique platform for life science research teams to access and share genetic and clinical research data across multiple organisations in an ethically approved and secure way, using the World Wide Web.
http://www.biogrid.org.au/wps/portal
Overview (Video)
http://au.youtube.com/watch?v=9US57ZhGxeo
Sunday, March 29, 2009
Service, method and input parameter authorization using GAARDS
At the Real-time Outbreak and Disease Surveillance Laboratory we've completed our first sprint to demonstrate input parameter authorization using the GAARDS infrastructure. In lieu of the security requirements of our Pennsylvania Ohio Biosurveillance Grid (PA-OH BiG) project when sharing notifiable disease data between health departments, we implemented input parameter authorization into our notifiable disease data grid application using Dorian, Grid Grouper and Introduce. This was motivated by our belief that it would be infeasible to create an additional service or service method (i.e., programming instead of configuring) every time a different set of valid input parameters for a different person/group were to be authorized.
GAARDS (i.e., Dorian and Grid Grouper) already provide authentication, service authorization and method level authorization but what we have done allows health departments to maintain extremely fine grained authorization using the same infrastructure. For example, we can now define a security group that is only allowed to make queries for data generated by bordering counties of a state. This group is defined by Grid Grouper and authentication is maintained using Dorian. Local mappings of user common names or groups to valid input parameters are maintained in the application using the RODS 6 data model.
We are really liking Dorian because local organizations need not maintain the credentials of foreign users if they have a trust relationship. Local nodes always know who (by common name) is accessing their services and the local nodes maintain local access control to their own data.
BTW: In the process of architecting this we spoke to the Justin Permar and the other caGrid folks over at Ohio State University who built Introduce. We thought we needed to modify the Introduce code but they were able to clearly explain why things are setup the way they are. Thanks to OSU.
GAARDS (i.e., Dorian and Grid Grouper) already provide authentication, service authorization and method level authorization but what we have done allows health departments to maintain extremely fine grained authorization using the same infrastructure. For example, we can now define a security group that is only allowed to make queries for data generated by bordering counties of a state. This group is defined by Grid Grouper and authentication is maintained using Dorian. Local mappings of user common names or groups to valid input parameters are maintained in the application using the RODS 6 data model.
We are really liking Dorian because local organizations need not maintain the credentials of foreign users if they have a trust relationship. Local nodes always know who (by common name) is accessing their services and the local nodes maintain local access control to their own data.
BTW: In the process of architecting this we spoke to the Justin Permar and the other caGrid folks over at Ohio State University who built Introduce. We thought we needed to modify the Introduce code but they were able to clearly explain why things are setup the way they are. Thanks to OSU.
Friday, March 27, 2009
PHGrid Architecture
Moses, Charlie, Vaughn and I met again to revise the PHGrid Architecture models. We're now up to 0.5 and the good news is we've reached consensus on these four models.
I'll schedule some time with NCPHI leadership next week to present our models. But we're still looking for any feedback on the models.
I'll schedule some time with NCPHI leadership next week to present our models. But we're still looking for any feedback on the models.
Polygon encoding; or how to make your map of the US load faster
So, the problem we have right now that is causing a lot of slowdown (and annoying "this script is taking forever... continue?" errors) on a lot of browsers is that the polygons we have are too complex.
Things like Colorado or some city zipcode are simple enough... but states with long coastlines (California, Florida) or bordering rivers (Mississippi, Illinois) tend to have polygons with hundreds or thousands of vertices because of all the little crenelations that nature happens to draw on the country. Furthermore, even things like Colorado have zips that border rivers, and the end result is a browser having to download and process a LOT of javascript. At least 95% of the massive page draw is arrays of longitude/latitudes.
Google, however, represents these lovely little things called encoded polygons that take these thousands of points and turn them into two simple lines of text. I think they are the key to removing errors and speeding everything up. There are very in-depth summaries of encoded polygons and how to make them from lists of points here. I will try and explain my perspective anyways.
First off, the encoding is two dimensional in the data they store (hence two strings). The first string is a compressed representation of the points that make up the polygon, which saves lots of space because numbers are very easy for computers to de/compress. The second string is an indicator of which points should be displayed at what zoom levels. Thus, if you are zoomed way out (viewing all of the US) you don't need all the individual points on a river because, well, there could be 15 of them in one pixel of your monitor.
There is also a very neat algorithm that automatically determines the levels, and it is demonstrated here.
The best part, Google understands encoded polygons. Thus, less javascript processing for the browser, and since it's simple compression and algorithms, the server shouldn't have to spend many milliseconds converting the data on the fly. (and even if it does, it will only need to be done once and then cached).
I am hoping I can get this implemented Monday. They even have java ports of the encoding algorythm. I just hope it's rather straightforward, there are a lot of little extra usability tweaks to be made too and the deadline for this version of Quicksilver is relatively soon. And who knows, it might not help speed or errors that much.
But I have great hopes, and I think it will.
Cheers
Things like Colorado or some city zipcode are simple enough... but states with long coastlines (California, Florida) or bordering rivers (Mississippi, Illinois) tend to have polygons with hundreds or thousands of vertices because of all the little crenelations that nature happens to draw on the country. Furthermore, even things like Colorado have zips that border rivers, and the end result is a browser having to download and process a LOT of javascript. At least 95% of the massive page draw is arrays of longitude/latitudes.
Google, however, represents these lovely little things called encoded polygons that take these thousands of points and turn them into two simple lines of text. I think they are the key to removing errors and speeding everything up. There are very in-depth summaries of encoded polygons and how to make them from lists of points here. I will try and explain my perspective anyways.
First off, the encoding is two dimensional in the data they store (hence two strings). The first string is a compressed representation of the points that make up the polygon, which saves lots of space because numbers are very easy for computers to de/compress. The second string is an indicator of which points should be displayed at what zoom levels. Thus, if you are zoomed way out (viewing all of the US) you don't need all the individual points on a river because, well, there could be 15 of them in one pixel of your monitor.
There is also a very neat algorithm that automatically determines the levels, and it is demonstrated here.
The best part, Google understands encoded polygons. Thus, less javascript processing for the browser, and since it's simple compression and algorithms, the server shouldn't have to spend many milliseconds converting the data on the fly. (and even if it does, it will only need to be done once and then cached).
I am hoping I can get this implemented Monday. They even have java ports of the encoding algorythm. I just hope it's rather straightforward, there are a lot of little extra usability tweaks to be made too and the deadline for this version of Quicksilver is relatively soon. And who knows, it might not help speed or errors that much.
But I have great hopes, and I think it will.
Cheers
Quicksilver updates
This afternoon we had a very productive session with the BioSense BIC/Epi team to review the latest build of Quicksilver.
They gave a lot of good feedback that fell into two categories: easy changes (cosmetic) and more complicated. The complicated changes include modifying the map to color the cloropleth based on number of outliers per period (rather than the arbitrary count ranges currently used). We're going to work on this after Dr. Tokars' team is able to analyze the data to find the correct break points for what is significant and what is not.
The easy changes have been made into a tracker item. Here's the list as submitted by Peter Hicks and Steve Benoit:
They gave a lot of good feedback that fell into two categories: easy changes (cosmetic) and more complicated. The complicated changes include modifying the map to color the cloropleth based on number of outliers per period (rather than the arbitrary count ranges currently used). We're going to work on this after Dr. Tokars' team is able to analyze the data to find the correct break points for what is significant and what is not.
The easy changes have been made into a tracker item. Here's the list as submitted by Peter Hicks and Steve Benoit:
- Script error issues for every query requested
- For clinical effects, listing in alphabetical order would be helpful
- In legend scale, first option should be 0 (not less than 0), 1 to 10, 11 to 20, above 21. Currently, the categories overlap
- Label what these numbers represent in the legend (visits, calls, etc. ?)
- Adjustable legend option should be called “customize breakpoints or classification”. How to enter the customizable option is not clear or self-evident.
- Queries are extremely slow
- Calendar allows you to pick future dates – should have date limits
- Colors in legend don’t align with colors on map.
- A help section that describes the module and it’s components would be useful
- Call out box when mousing over state shows a time series – the average line is not an average for the time period selected. The outlier box is not clear. How do you define an outlier?
- Unable to understand this visualization and chart. We selected a region and got the call out box below. Was there 1 call or 797?
Deploy is complete. Come revel in new features
So, I have completed another deploy (well, two deploys actually) of Quicksilver.
You can reach it here, and you can read more about how it was built or how to view the code here
One of the new features is the "remembered zoom"... where the zoom for a selected region will be maintained if only the legend, timespan, or conditions change. Changing the region (from MD to IN, from MD to the 208 zip 3, the 208 zip3 to a view of all states) will revert to the default zoom level for that region type. But now, if you had to zoom in on Rhode Island... you won't have to zoom in again after selecting for a different condition or widening your search.
Another new feature is the adjustable legend... meaning that you can change the difference in the count numbers that determine the colors for polygons.
Finally the search dates default to the current week. This is not going to be that useful on the training node because we are using test data that only goes up to about October 2008, but when Quicksilver gets deployed to a production setting and starts getting access to more recent data, it will be much more helpful than a always starting on a random week in February 2008.
Meanwhile, lots of people had a good long look at the app today and came in with lots of feedback. It was both wonderful (because lots of people liked the application and thought it was neat and I got some ooohs and aaahs from things like flot) and terrifying (because I was worried it would break, people found ways to make the app do strange things, and because people tried things and thought of features that would be insanely cool to install that I never would have dreamed of). Having a bunch of users that are not that familiar with the application generates a LOT of very good feedback and questions. People were getting confused over things that, in retrospect, are not very clear at all. Today literally involved a large explosion of possibilities and potential, and it's as paralyzing as it is motivating. If anything because it's difficult to triage what should be done first.
So, next week will be a lot of implementations of little and big fixes. There are some very salient UI tweaks to be made (Like labeling more clearly, having "zip3: 208 // Total Count: 350" is a lot more handy than just "208 // 350") lots of little help pages and legend explanation (namely how the C2 algorythm means the blue average line and outlier status is based only on the average of the preceding 30 days (minus the closest two)) and finally, an attempt to make the polygon drawing much more streamlined to get rid of the really-quite-irritating "This script is taking a long time, do you want it to continue" error thrown by IE, which is exacerbated by having a not-bleeding-edge computer.
I think I have found a way to do that, and it's called polygon encoding, and I'll be detailing that in the next post.
Either way, I am elated and looking-forward to how nifty we can make this application.
Cheers,
Peter
You can reach it here, and you can read more about how it was built or how to view the code here
One of the new features is the "remembered zoom"... where the zoom for a selected region will be maintained if only the legend, timespan, or conditions change. Changing the region (from MD to IN, from MD to the 208 zip 3, the 208 zip3 to a view of all states) will revert to the default zoom level for that region type. But now, if you had to zoom in on Rhode Island... you won't have to zoom in again after selecting for a different condition or widening your search.
Another new feature is the adjustable legend... meaning that you can change the difference in the count numbers that determine the colors for polygons.
Finally the search dates default to the current week. This is not going to be that useful on the training node because we are using test data that only goes up to about October 2008, but when Quicksilver gets deployed to a production setting and starts getting access to more recent data, it will be much more helpful than a always starting on a random week in February 2008.
Meanwhile, lots of people had a good long look at the app today and came in with lots of feedback. It was both wonderful (because lots of people liked the application and thought it was neat and I got some ooohs and aaahs from things like flot) and terrifying (because I was worried it would break, people found ways to make the app do strange things, and because people tried things and thought of features that would be insanely cool to install that I never would have dreamed of). Having a bunch of users that are not that familiar with the application generates a LOT of very good feedback and questions. People were getting confused over things that, in retrospect, are not very clear at all. Today literally involved a large explosion of possibilities and potential, and it's as paralyzing as it is motivating. If anything because it's difficult to triage what should be done first.
So, next week will be a lot of implementations of little and big fixes. There are some very salient UI tweaks to be made (Like labeling more clearly, having "zip3: 208 // Total Count: 350" is a lot more handy than just "208 // 350") lots of little help pages and legend explanation (namely how the C2 algorythm means the blue average line and outlier status is based only on the average of the preceding 30 days (minus the closest two)) and finally, an attempt to make the polygon drawing much more streamlined to get rid of the really-quite-irritating "This script is taking a long time, do you want it to continue" error thrown by IE, which is exacerbated by having a not-bleeding-edge computer.
I think I have found a way to do that, and it's called polygon encoding, and I'll be detailing that in the next post.
Either way, I am elated and looking-forward to how nifty we can make this application.
Cheers,
Peter
Subscribe to:
Posts (Atom)
Posts
