poicondai-2.0-noprops now available in the repository

Poicondai-2.0-noprops is sitting happily in the repository section of the website. Complete with a POM… and maven is actually nifty enough that when npdsgmaps references poicondai-2.0-noprops, mvn will read the pom and download all the extra dependencies needed for poicondai in addition to those needed specifically for npdsgmaps.

Simpler… if Project A needs Jar B… but Jar B relies also on jars C, D, and E… maven will read the pom associate with Jar B in the repository and also go fetch C, D, and E, even though C, D, and E, aren't listed as dependencies in Project A's pom file. At least it seems that way from experimentation.

So, if you want to use this jar, you will need to create a properties file with the url, systemuser, system name, and password (look into the poicondai project to see what the filter is filling), but it will possibly make it much easier for people needing to access poison to include this.

More importantly, it got me comfortable with moving things into a non-local repository (also many thanks to Felicia's posts), because I anticipate I will have to move more things into the remote repository for the sake of fixing a RODSA-DAI bug and for people who want to use grid-viewer (it will keep them from having to do a jar hunt like I have so many times).

Now the next step would be finding a cool way to get all the provided jars into the lib container of a webserver. Maybe there is a deploy plugin for that sort of thing.

Dueling for Certificates.

So, It appears that the problem between Rodsadai-web and Amds-web that they are playing musical chairs for the client-certificate.

Felecia (who is developing AMDS-Web) has been helping me debug. We were able to replicate the issues outside of staging on my dev node (handy, since debugging these involves a lot of jboss-restarts). It appears that it is just a mutual exclusion issue: If you started AMDS-web first, it will be able to marshal the needed security credentials, and when Rodsadai-web tries to marshal the credentials, it will not be able-to and the connection to a remote globus server will fail. The behavior is the same in reverse. If the rodsadai-web client is called first, it works fine and then AMDS-web will throw a connection error.

Now, we are not sure if the failure to marshal is because of slightly different needed credentials that are being cached (apparently Rodsadai-web is using tramsaction layer security (TLS) with only privacy and AMDS-web is using TLS with privacy and integrity) or whether there is some sort of explicit locking behavior that prevents different applications from using the same set of proxies.

Either way, it reveals that we need to find a better way for clients to make secure connections to Globus services. Whether it involves some sort of Server-level pooling or having the apps load application-specific certificates, the current method would not allow client applications with slightly different needs to coexist on the same server (or worse, not even allow client applications with the exact same needs to coexist).

Thus, we plan to delve into the innards of globus security to see whether the certs need to change or how they are being marshalled. (Probably both).

Deploy Log

Deployed new Quicksilver to staging this morning. The Zip3 and Zip5 counts are now geocentric from the Allied Center, and not just the Colorado Poison Center. Thus, you no longer log into Tennessee which was supposed to have 400 cases and then only see about 30. It also has security, so you will have to email someone on the PHGrid team to get a userid and password. The installation notes are here and the direct link is here.

The AMDS-UI was also deployed, but it was discovered that the AMDS-UI and Rodsadai-web are not playing nicely with the Globus Security Interface. Thus, when one is run first, it prevents the other from running, because the other cannot create a secure connection. Thus, AMDS-UI and Rodsadai-web seem to be mutually exclusive for the time being, and since rodsadai-web is still being demonstrated-on, we are going to try and debug and hopefully alleviate the conflict. Failing that, we will probably create a "switchover" where AMDS-Web replaces Rodsadai. After all, AMDS is capable of doing the same thing Rodsadai did, but better with more functionality and outside of Pittsburgh.

Zipcodes polygons in the database

So, I have updated the poicondai-loader project and the poicondai-util project and now we have zipcode polygons in the polygon database.

Also, the deploy of tomcat to the staging node went swimmingly. Dan switched the tomcat server over to port 8443, and rodsadai and poicondai worked like nothing had changed.

So, tomorrow is starting to do some testing with the zipcode polygons, and then updating the map application to have zipcode polygons in addition to the county polygons.

Demos are pretty

Sorry for the lack of updates on my part. I was spending most of the week improving the already nifty Poicondai demo to have variable y-axises and start grouping by week if queries over 180 days (about 6 months) are chosen.

You can check it out by going to http://ncphi.phgrid.net:8080/poicondai-web/poicondaiMap.jsp

The other thing I did was get ogsadai running on secure-tomcat and verified that rodsadai could connect to it. Many many many thanks to Felicia for helping me with that yesterday. She had already found out all the crazy things that had to be done to get tomcat working and was able to help me get a similar setup working in about a half hour.

Next up is zipcodes. I will need to convert the zipcodes into a geolocational database for poicondai, and then I will be updating poicondai-web to select zipcodes, updating rodsadai-web to use zipcode polygons (which I hope will turn out pretty cool), all while hopefully implementing some new filtering so I can pull all the different configuration options into one file.

Planning and such.

So, RODSAdai and Poicondai are documented, fleshed out, and there ready for people to consume and ask-questions about if people want to recreate them or install them in new places.

The next step is to essentially re-create the OGSA-DAI end of RODSAdai using CaBIG's Introduce framework. Introduce, for all intents and purposes, seems to be really really good at generating a full-out service skeleton and having the tools to package up everything into a .gar file (which is the deployable for a globus toolkit server).

Thus, instead of OGSA-DAI at a given location accepting a query, this will be parameters passed into a webservice. Whether those parameters will be a query or just some filter values will be a discussion for Jeremy (because if they are just filter values, it becomes that much easier to prevent against SQL Injection attacks by using a prepared statement). Also, it will be interesting to see how well parameterized the connection logic could be and whether it would be as universally installable as ogsadai (it probably would not be as versatile).

Otherwise, we haven't really investigated the Dallas slowness at this point, I will also have to ask Jeremy if that Pitt node has been set up or not. I have a few ideas for what to try, but we will see.

DEMODEMODEMO(again)

So there is a big demo tomorrow, and I have been preparing for that by making sure that the RODSA-DAI and Poicondai demos have been polished and a few of their quirks have been ironed out. I also was working on making all the items a lot more portable.

Furthermore, I have been ironing out some issues with a prototype public demo, and doing a lot of documentation. And that documentation is linked-to from this shiny new wiki page that Brian set up and I have filled out: here.

Meanwhile, I have started to look into the caGrid CQL and DCQL, and that stuff looks interesting.

Another demo done.

We had another demo today. This time it was not rushed into a short amount of time and involved a lot more discussion of "Okay, but how do we make something we actually want to use as opposed to something that shows off what we can do?"

The next steps seem to be collecting lots of use cases and then going to NPDS (National Poison Data Service) and seeing if more filters and server-side aggregation can be had.

In the meantime, I cleaned up the demos a bit more, so now I can transfer them to different machines with fewer issues, but I am sort of along with the demo people in that I want to start morphing these things into actual products that people will want to use. It's just that a lot of it involves several discussions and typing out a lot of ideas on paper and then having people question the ideas because the way it looks in your head wasn't transferred adequately (thus you have to explain it 5 different ways until you find the one that sticks... you might also have to just repeat yourself a lot).

In other news, I have seen this pretty thing made by ESRI called Digi-Net, and it looks like I might be getting some training in how to do neat things with Digi-net. But until then it's lots of thought-cud to paper.

Need almost coded.

So I got the PoiconDai-web piece coded... so now it is being gracious enough to put the chart above the table of data.

In Rodsadai-web, I hit a few snags service-enabling, so I will probably have to troubleshoot. Luckily, I took a snapshot of the working code and put it into a tag of SVN and just ported it back for a demo that is tomorrow.

I am really liking MVN and SVN.

Code the need.

So, I am excited to say that I got the JBoss server unbound to localhost. The best guide I found for opening up the server for production and removing all of the dangerous programs was here: http://www.jayson.in/programming/how-to-configure-jboss-as-production-settings-and-tuning-tips.html

So, I managed to get JBoss unlocked, and then found out that there is a lot of location-specific stuff that is not sitting in a properties file, but is instead sitting in hard code.

This is bad.

Thus, in addition to massaging the PoiConDai server to be nicer, I am going to be massaging RODSadai and RODSAdai-web to better handle being placed on different servers... because trying to remember the immediate coding environment specifics is really annoying and should just be able to sit in the given JBoss environment.

Otherwise, more demos soon, yay.

Demo Results.

So, the first demo of the day on Wednesday went really well. Everything worked, everyone had their talking points ready, everyone was impressed. Then we had the second demo which no-one knew would be happening... and luckily everything worked, everyone had their talking points ready, and everyone was impressed.

One of the major developments that came from the demo has already been described by Ken: try and make minimum data set WDSLs and public health appliances. Tom, Jeremy, Ken and I had a big discussion today and we were pretty much focusing on three things:

- An instructional toolkit for web-services
- "Here is how you can use any number of these webservice tools (AXIS, .NET, Introduce, etc) to create a service, here is how you can secure them (globus, caBIG, etc)
- The ideal WSDL
- "here is the WSDL for summarized data you should use or be very-close to"
- A black-box implementation, with open code, so not actually very black box
- "here is an example service implemented completely. Just set it up to point to your databases and open up the ports"

In addition to that, I have been slowly building the ideal RODSAdai description document so that we can start coding next steps for that application and get some more focused feedback from the guys at OGSA-DAI. I imagine that OGSA-DAI and RODSAdai will form a backbone for the black-box application described above.

RODSAdai, PoiConDai, and NPDS-ws

Today I started off by having a productive discussion with the folks who helped me with RODSAdai: Dr. Jeremy Espino from Pitt (who brings the RODS) and Mario Antonioletti and Alastair Grant from University of Edinburgh (who bring the OGSA-DAI).

We discussed future developments, concerns about security, scalability, and automating installation and configuration as much as possible. We also talked about our availability being reduced for the next phases... But mainly we talked about the fact that one of the things that got lost in the making of RODSAdai 1.0 was a quick and descriptive design document that explained what we were all trying to do and how. Something so that Alastair, Mario, or Jeremy would be able to go "oh, well to connect it that way you should try configuring it like this..." Furthermore it is much more likely that the whole Axis/Globus conflict would have been found earlier and mitigated sooner since Alastair was already familiar with that particular limitation. Thus, the first step for the next phase of RODSAdai is to take the ideal-state and current-state diagrams of RODSAdai that I made, and more information about what they mean, how they currently work, and how I hope to get them working to focus advice and let anyone search for red flags.

But I won't be able to do that until Thursday of next week, because right now I am in demodemodemo mode to get PoiConDai in some form of pretty graphical display so we can show that yes, we are pulling Poison Center call data into the CDC.

This was greatly moved forward this morning with the help of Dr. Espino, namely because he already wrote code that used AXIS to consume the NPDS webservice a couple of weeks ago and had already incorporated all of the JAXB libraries needed for parsing the "any" response returned by the web service. The codebase is called npds-ws-client, and I am going to use it because it is pretty and is along the lines of what I was going to do, only with the properties file and return serializing I was planning on doing already implemented (and thank goodness for it).

Let me pause here to say how much I love maven again. It has extensions for generating axis stubs and jaxb code, so all you need to do is have the WSDLs handy and you don't have to worry about constantly regenerating the parsing bits and movig them about manually. How cool is that!?

Meanwhile, my next step is to incorporate a few of those unit tests I posted about earlier, and then set up a web-client front end that I am going to call PoiConDai web for showing the data in an ever-snazzier JSP. I think I am going to get something really nifty ready by Tuesday Evening.

PoiConDAI.

Today I have been working on the use cases for a new little demo project I am dubbing PoiConDAI.

It is basically the consumption of a service that Dr. Jeremy has already created for listing Poison Control Center call volume data. I finalized some of the rough cases (as shown in the linked document) and tomorrow I will probably start playing more with AXIS and perhaps Introduce for making a quick client in anticipation of receiving the WSDL for the service.

Otherwise, I am working on next steps for RODSADai, including scalability improvements, security improvements, and a general design and design-process review with the folks of OGSA-DAI.

Demodemodemo

Greetings all.

I was out last week, and this week I was putting my nose to the grindstone... and I got some pretty neat things done with the RODSAdai demo.

I cleaned it up, so it has a simple look with only intuitive controls.
I managed to get a server-merge in the demo app, thus allowing for data to be spread across different servers and aggregated when multiple servers are selected.
I managed to get some geocaching set up so that the page loaded a lot faster after the initial runs.

I owe many thanks to Brian for being a great sounding board. He was the one that turned major redesigns in my head into minor UI tweaks.

Different Databases

Today, I got the RODSAdai test data split across different databases. I still have a few things to work on before the next demo... and I am going to be on vacation next week so I spent a lot of today documenting steps and locations of things and fearing the fact that people would be looking at my "gotta get this working" code.

Otherwise, everyone seemed to like the demo's we did this morning. So that makes me happy... and I will still have a week to polish things up and help externalize the demo before the PHIN conference... and I feel good that I got the basis of tests I wanted working before I left.

See you in a week!

We've got dots, and they are indicating levels.

After a lot of tinkering with the google maps API and a lot of things that theoretically should work but didn't... I managed to get dots showing up on a google map and color-code them based on the level of incidents indicated by the query. They also load a bit slowly because of how I had to encode the data series, but it works, and that is better than before. I also have lots of ideas of how to make things faster and prettier given time thanks to Alastair and Mario of Ogsa-Dai, who helped me kick around ideas.

Next steps are to bisect the databases and put them on different database servers... wire up such nifty bisections into the demo app, and if time allows, start integrating the results.

I am happy. I got dots.

Building Demos

I got some very cool news from Dr. Espino today, that he had gotten RODSAdai building and happy on his end, and will probably be able to integrate all the code within a week. Furthermore, he added much better logging code (log4j complete with properties files), some null checking, and some defaults. I have successfully updated everything and adjusted to fit my own environment.

On the demo end, I have discovered many things that will not work with google maps as I envisioned them. I was thinking of doing real-time geocoding of zip codes, but that would turn one javascript request to google into about 30, and there are already free databases of zipcode geospatial locations. Furthermore, I am not sure of the best way to just lay out all the data given google's finicky timeouts (thus, if your application takes a while to generate a KML file, you will need to generate the file and place it somewhere in an accessible spot and then tell google to find it there).

Considering I am not exactly working on an accessible node, I am beginning to think it might just be easier to use JSP loops to build an array and then place that array into one of googles marker manager objects. Perhaps there is a way to pass KML into a google map directly? I will probably tinker with this at home.

Preppin' for Demo's

So, today we had the initial big discussion of how we want to demonstrate the neat things that have been coded for the upcoming PHIN conference.

My first priority is to Dr. Espino to try and get RODSAdai working with RODS so he can show multiple data source RODS info pulls.

Otherwise, while I am not helping him, I have the following bits in my stable:

• RODSAdai through google maps
• Split data sources for to demonstrate different data in different places.
• Set up a check-box "one or many" database location selection.
• Enable a spatial series that can display through google maps and/or google earth.
• Server Location Admin Screen
• Have the ability to declare a new server/resource location and modify existing locations.

I'm looking forward to it. I have also reached out to the OGSA-DAI folks for help with the Google Maps portion of the demo since they already did it once with their demo.

Got the time form working

I cleaned up the spatial series test form and moved the code over to make a time series test form. I also talked with Jeremy a bit about his installations of globus and ogsadai.

Tonight I block out the changes for the admin screen and start coding it tomorrow.

Form is coming along

I have managed to get the spatial series test JSP form updated so now dynamic inputs can be entered and different servers (that were stored in the properties file) can be picked from a drop down.

The next step is to replicate the changes for a time-series form, and then I will start modifying RODSAdai to have some administration classes that allow for editing of the server lists and the like using the properties files.