AMDS Data Store

I checked in the first batch of database definitions for the AMDS Data Store to SVN. This is basically the DDL for all the tables, views, triggers, etc. that we're using for the AMDS Store that will be populated with, initially, synthetic BioSense data. This will help out anyone else who is looking to build their own aggregate store.

I'm also starting a list of conditions and classifiers that we'll use internall in the AMDS Store but will be useful to any others developing AMDS services.

Eventually this will have DDL for postgres, any other dbs we can create scripts for by request.

North Carolina is changing colors.

So, I finally got data pulling into the map.

It's just North Carolina, and it doesn't show a graph, but it pulls data.

I had to do a lot of debugging and found that the service is very particular about regions within the boundaries being passed in, and I had to build the list of available conditions... and I know there is a lot of tweaking and changing to be made in order for things to start behaving... This helps map out the ways the application will need to act as well (no selecting regions that aren't available, etc)

But hey, AMDS data in polygon(s). Yay!

AMDS Architecture clarification

After talking with Tom, Barry, John and Ken I thought it would be useful to clarify some of the ideas we have around the AMDS architecture. You can view the full page on the wiki, but I'll describe the concepts a little here.

Option #1 for AMDS is what we've typically be dealing with. A grid node is installed as a publisher and the publisher runs an AMDS service to query the biosurveillance database and return the counts based on the query. This requires hosting a service and getting your IT organization to open up port 443 for internal connections.


Option #2 for AMDS is what we're calling the producer-collector scenario. Some partners cannot/will not host a service. To accomodate this, they generate AMDS reports containing only aggregate data and transfer it to a collector node. CDC doesn't want to be the collector node, but they may need to be to demonstrate the capability. These periodic reports are received and loaded into an AMDS store.


And if we put these both together, we see that producers send data to collectors, collectors then store the data in a database where a publisher can access it. Of course, there will be publishers that connect directly to biosurveillance data sources.

Client data in grid viewer

I managed to get AMDS client data in grid viewer after importing several libraries and manufacturing a local temporary client of my own. I am feeling much better now.

The next steps are to get that data on a map... and to get the client out of the "localtempclient" into a cool multi-client that adaptively uses different connection methods depending on the URL provided.

Also, I anticipate a lot of mavening and a lot of jar hunting over the next couple of days.

Ok it’s been a month now since I have been on the team and I have completed my first service. Following my philosophy of service development, I have now checked in the code under AMDSCore and can be seen here. I have created the complete service implementation that is based on the common or core components. Additionally I have provided a client implementation that is compatible with the provided service. Detailed instructions for building, configuring and running the service are here. Currently, the service is not using security. I will add the security this week and be ready to deploy to the training node this week. For those who just want to try out the service without compiling you can go here to download the GAR.
This service was developed for several types of scenarios in mind and are:

· Connecting to the service while on the same machine
· Connection to the service over the internet

Since the Grid Viewer, the component that Peter is developing, will be running on the same machine, it make more sense for him to use the components geared for connecting to the core service implementation so that it does not have to incur the overhead of the full client implementation. Yesterday we were trying out the client but not having all the environmental variable set correctly on his machine, caused us to run into issues. I am hoping Dan will work with me to get this straighten out. Over all this stresses a need to get a staging internal grid node set up. I am hoping to work with Dan to get this going this week. The result will make our efforts more efficient.
For others who wish to connect to the service directly via the Grid infrastructure, the client implementation is best used scenario. Again this component is included in the distribution. All of these descriptions are captured in the AMDS Biosense Wiki. More to come....

National Biosurveillance Model (Summary / Grid Perspective)

Please review and comment on this 1st draft. Add your comments directly to this post. We are attempting to build a conceptual framework around national biosurveillance from a summary and grid perspective to help us and others understand how and why we think distributed models scale and work for public health. (Version .01)

CAGrid Transfer Service has been redeployed

The CAgrid Transfer service has been redeployed and tested on the training node. I had to rebuild the Globus 4.0.5 container from source in order to restore the missing library files. A backup copy of this new build is stored in the /usr/local/clean-4.0.5 directory. Currently all Globus services are functioning within normal operating parameters.

Upgrading Globus on Training

As you look at the training node, you will notice two new Globus directories in /usr/local. These directories are globus-4.0.8 and globus-4.2.1. Both are upgraded builds of Globus that are currently being tested on the training node.

The Globus 4.0.5 installation will remain intact until testing has been completed. When testing is completed, the system wide profiles will be updated to run the upgraded Globus installation by default.

In other news, the CA Grid Transfer service has been undeployed from the current installation of Globus for troubleshooting purposes. This service will be redeployed to the Tomcat container after the issue is resolved.

Current Error:
avax.servlet.ServletException: Failed to initialize 'cagrid/TransferServiceContext' service [Caused by: gov.nih.nci.cagrid.introduce.security.service.globus.ServiceSecurityProviderImpl]

Gar issues...

Today Vaughn and I tried to get the client for the amds service on my machine so we could develop against it. It didn't work.

First we tried to build his code on my machine, which failed because my machine is not his machine and therefore had a slightly different version of globus... or ant... or user setup... or some other variable that is yet to be discovered that usually pops up during the first couple of attempts like this.

Then we tried to bring over the gar he had built on his machine... and deploy it... only to have the client blow up whenever I tried to run it citing missing class exceptions... despite the fact that all the classes that should have been in the gar.

So, I am still wanting for an AMDS client, and the resulting checklist of code and build and configurion points is long and mildly upsetting. I'm wondering if making a quick little AXIS or CXF service would be worth it just so I could develop against the resulting classes and client examples while the Globus connections were ironed out.

I'm sure it will get sorted out eventually, but when I was working with Quicksilver, I had the benefit of having the service from day one. This time, I have build out a lot of the grid viewer on assumptions of how the client and service will behave, and I worry about increasing the chance of needing to refactor things once it gets booting and connected because my assumptions were off.

Oh well, growing pains. We'll get through it... and I'm sure what we make will be rather cool. It's just "that phase" in the project where things are going wacky all over the place when one was hoping they would just fall into place and it's extra difficult to take the calming breaths you need to collectedly analyze the situation.

International Grid Accreditation

An international consortium called the Americas Grid Policy Management Authority (TAGPMA) has released an updated set of standards for international trust within grid communities in north, central, and south America.

TAGPMA has established several grid 'profiles' that establish accreditation criteria for participating institutions. And as recently as this past weekend, they released their Minimum Integrated X.509 Credential Services (MICS) which defines the backbone of what CA's can be trusted. Think of it as an accrediations for CA's.

Information on the authentication profiles can be found at http://www.tagpma.org/authn_profiles .

Also of interest are links to the EUGridPMA and APGridPMA which are located on the main page at http://www.tagpma.org/. All of these participate in the International Grid Trust Federation located at http://www.igtf.net/ .

So many links so little time. Start here if you only click one: http://www.igtf.net/ .