Bugs fixed, but were breeding.

So, I fixed the Quicksilver bug about data replicating on the state-level on dev. Thus, after the next deploy, one will be able to load the states as many times as possible for a given date range and the data will stay the same.

Otherwise, I spent the rest of the day trying to find a fix for the "if I try and load California zip3's, the service crashes" problem... the root of the cause seems to be that I am generating a list of all the zipcodes in California and sending it to the Service, and the list is rather long, longer than the comfy limits of the service.

Luckily, I found a way around it, I can just send a request for the state and not all the zips in said state and it will still aggregate them by zip code.

The only problem is that I can't specify a state with the "main" aggregated national service. So I either have to pull back ALL the zip codes for the entire country (which would take a minute or two) or limited data from only the Colorado poison center. I have sent an email to the lovely poison people to see if there is something I am missing that will work for my needs.

But otherwise, yay. Less horrible crashes and strange behavior while we get the data sorted out.

The other cool thing is I had a chat with William Duck and he is pretty much a statistical Guru. He is helping me look for statistical and web-chart libraries that will help provide better and niftier information via-chart than the nifty, but not as functional google charts (or maybe even see if google charts can do more complicated stuff). That will be a massive help.

Windows Node Errors

The SSTS service has been installed on the Windows node, but I don't think it is compatible with Windows in it's current stage of development. I think the path names need to be modified within the code in order for it to be compatible with Windows. Below is an example of the error message received while trying to connect to the SSTS hosted on a Windows node.

Current Error:
An error has occurred, error message = java.net.ConnectException: Connection timed out: connect

Here's an excerpt from the logs:

Feb 23, 2009 2:48:26 PM org.apache.catalina.loader.WebappClassLoader validateJarFileINFO: validateJarFile(C:\Program Files\Apache Software Foundation\Tomcat 5.5\webapps\caGridTransfer\WEB-INF\lib\servlet.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
Feb 23, 2009 2:48:28 PM org.apache.axis.utils.JavaUtils isAttachmentSupportedWARNING: Unable to find required classes (javax.activation.DataHandler and javax.mail.internet.MimeMultipart). Attachment support is disabled.
Feb 23, 2009 2:48:29 PM org.cagrid.transfer.service.TransferServiceImpl INFO: Starting the TransferService ImplementationFeb 23, 2009 2:48:29 PM org.cagrid.transfer.service.TransferServiceImpl INFO: etc path directory = C:\Program Files\Apache Software Foundation\Tomcat 5.5\webapps\wsrf\WEB-INF\etc/cagrid_TransferService
Feb 23, 2009 2:48:29 PM org.cagrid.transfer.service.TransferServiceImpl

The bold portion of the log is the path that should be corrected in the code. Going forward, I think we should test code on both Linux and Windows.

ArcGIS Server 9.3 Flex API; example app

Hello all.
The ESRI ArcGIS Server blog has an article about the Flex API for synchronizing
maps and data grids; The links below are the example application and the article.
If you mouse over a geographic unit...the items on the data table are linked and react to the area moused-over.....

ArcGIS Server 9.3 Flex API example application:
http://serverapps.esri.com/flex/sync_gr_dg/sync_gr_dg.html

ArcGIS Server Blog article:
http://blogs.esri.com/Dev/blogs/arcgisserver/archive/2009/02/19/Sychronizing-map-and-datagrid-interaction-with-the-ArcGIS-API-for-Flex.aspx

Ubuntu now has 'cloud computing inside'

http://news.cnet.com/8301-19413_3-10168951-240.html

New stuff, and a bunch to do.

The updated demos for Gmap-Polygon and and Quicksilver are now on the web, read the Service Registry entries in the wiki to learn more.

In the meantime, there is still a bunch of stuff to be done on Quicksilver and a few things to tweak on Gmap-Polygon, so here comes the list.

1. Gmap-Polygon


1. Bugs
   
   1. Internet Explorer sometimes throws an operation aborted error. I am not sure what to do to fix this as it is sporadic, comes with no extra information, and is related to Javascript.
      
   2. Focus is maintained between different sessions, so someone on one computer has picked California, the other person logging in will see California.
      
   
   
2. Speed
   
   1. Need to get smaller (less complicated, fewer vertices) zip3 polygons so that zip3s of states like California and Florida take less time to transfer and load
      
   
   
3. Security
   
   1. Need to bubble-up errors and make it so that simple, non detailed error notifications are given in a production setting
      
   2. Need to check parameters for validity, warding off various injection attacks
      
   
   


2. Quicksilver


1. Bugs
   
   1. Data is aggregating, not updating, in state counts. Need to make sure that time series replace in new polygons
      
   2. States with a lot of zip-codes are causing errors upon attempted zip3 loading, the string is overloading the service. Need to mitigate by seeing if ziplist length can be extended or by breaking up call into multiple lengths.
      
   3. State data is being loaded from the Colorado NPDS Service and not the global NPDS Service
      
2. Unimplemented features
   
   1. Zip5s need to be loaded and displayed
      
   2. The ability to select dates and conditions need to be added in accordance with the use case
      
   3. The graph-viewer needs to be updated to display a much more statistically useful graph (outliers flagged by different colors, possibly zoom-able)
      
3. Security
   
   1. Need to add security so that login/pass must be supplied before any map information is viewable.
      
   2. Need to double check values and propagate friendly/non-detailed error should values not be invalid.
      
   
   

There are probably a few things I am forgetting, and in addition to all these feature add-ons, I will be checking compatibility and configuration compatibility with different databases and the like.

Either way, I am looking forward to making things niftier and more robust. I hope you all have a good weekend!

Cheers,
Peter

Those thursday nights

Greetings all from the lab.

Tomorrow tends to be our "regular deploy day", which of course means I went "I am going to Get This Bug Fixed before I go home"

So, it's about 8:00PM, but I got that bug fixed, even if in testing I discovered about 3 other bugs.

But tomorrow, should the deploy go well, the zip3s in Quicksilver will behave that much better.

But woo-ey it was a doozy to get those zip3 loads behaving... and a bunch of the other bugs fixed over this last week. Unfortunately, it was a lot of stuff on the back end, so the front end probably won't change that much to the end user.

I guess that is just one of the facts of life sometimes, you can work for a week on a bunch of cool new improvements, and the user will never know it (But ooooh, do they notice it when something goes wacky).

Cheers

AMDS BioSense Extract

I loaded Dr. Tokar's synthetic BioSense data into the Lab's staging database. This gives us AMDS data for 2008 covering GA, SC, NC and IN for BioSense's 10 syndrome classifications.

I'll put the SQLServer DDL up somewhere as soon as we determine a proper way to share DDL.

Some may ask why we're using SQLServer when everything else we use is open source. The answer is because the CDC maintains a farm of SQLServer databases that are paid for at the enterprise level (i.e. "free" to the program). Everything we write will also be available for use in an OSS RDBMS (such as PostgreSQL or MySQL). So although we use SQLServer, nothing is hard wired to SQLServer.

Windows Node Progress

I installed Globus 4.0.8 on the Windows node to avoid the potential 4.0x and 4.2x compatibility issues. Eventually we will have to roll out an upgrade to 4.2x across the grid.

I was able to used the host certificates from staging to start the Windows container. The good thing is, Windows can use the certificates generated by the Linux nodes. Later we will have to solve the issue of generating correct subject names from Windows nodes. Until then, the work around is to initiate the user request on the Linux node and copy the certificates to Windows.

I also installed Apache Tomcat 5.5.27 and Apache Ant 1.7.1 in order to match the versions on staging. When I deployed Globus to Tomcat I received a couple minor errors.

Error 1:
C:\gt4>ant -f share\globus_wsrf_common\tomcat\tomcat.xml redeploySecureTomcat -D
tomcat.dir="C:\Program Files\Apache Software Foundation\Tomcat 5.5"
Unable to locate tools.jar. Expected to find it in C:\Program Files\Java\jre6\li
b\tools.jar
Buildfile: share\globus_wsrf_common\tomcat\tomcat.xml

The Fix: I copied tools.jar from the JDK directory to JRE directory it was looking for it in. This can also be fixed by changing the JAVA_HOME variable.

Error 2:
[copy] Copying 1 file to C:\Program Files\Apache Software Foundation\Tomcat
5.5\webapps\wsrf\WEB-INF
[copy] Warning: Could not find file C:\gt4\lib\xalan.jar to copy.
[copy] Copying 50 files to C:\Program Files\Apache Software Foundation\Tomc
at 5.5\webapps\wsrf\WEB-INF\lib

The Fix: I renamed xalan-2.6.jar to xalan.jar

This resulted in a successful deployment of the Globus container to Tomcat.
BUILD SUCCESSFUL
Total time: 1 second

The next test is to deploy one of Felicia's grid services to the Windows node and verify that it can be called remotely.

Oh, Internet Explorer... why do you not love GMap Polygon?

So, Internet Explorer seems to have a fickle relationship with GmapPolygon.

Sometimes, when it is loading a set of polygons, it will throw an error saying "The page at [address] cannot be loaded. Operation Aborted". This is a thoroughly unhelpful error, because it doesn't offer to show me the part of the code that broke or otherwise debug the issue. Furthermore, it's not re-create-able on demand. It tends to do it whenever it feels like it, for any given set of polygons... polygons which loaded fine not three seconds before.

Some of the blogs I checked (example) indicate that such errors arise because IE was processing some javascript, freaked out, and cancelled the page load... Usually because some spurious parent element is being referenced. The problem is that in the javascript I don't think any parent DOM elements are being referenced, just maps which are created inside the head script.

I'll admit, I am not that familiar with javascript, so nothing really stands out to me that makes me go "oh, that's what is making it blow up sometimes." So I invite anyone to look at the source of polygon pulls (I suggest poking the site with firefox) to see if anything stands out.

Otherwise, I imagine a lot of the problems involve the fact that a polygon is represented by a metric ton of text... especially polygons that border rivers and coastlines... such sweeping, beautiful crenulations that only nature can carve out are represented digitally by thousands of vertices. which all need to be pulled back from a database somewhere and then provided to google maps.

Meanwhile, I am attempting to make Gmap-Polygon get those back end database connections to be more robust and reliable, and improve caching so that the database doesn't even have to be hit that often... but we are also just looking at getting simpler polygons so that there is less data to pull, transfer, and store, which should have the biggest impact.

A non technology, non health blog that you may find interesting

Lately, I've been following reading a book and following a blog, each called 'Wikinomics: Exploring How Mass Collaboration Changes Everything' http://www.wikinomics.com/blog/. The points of view of the contributors to the book / blog are stunningly like those that contribute to this space. In fact, in the book, the authors make some bold proclamations that commercial interests that are closed, hierarchical, and essentially 'old-school' are now the 'losers' in the market. I have yet to find any consideration of how governments and / or health entities fare in these approaches. I wonder if we should consider ourselves a case study?