Client data in grid viewer

I managed to get AMDS client data in grid viewer after importing several libraries and manufacturing a local temporary client of my own. I am feeling much better now.

The next steps are to get that data on a map... and to get the client out of the "localtempclient" into a cool multi-client that adaptively uses different connection methods depending on the URL provided.

Also, I anticipate a lot of mavening and a lot of jar hunting over the next couple of days.

GridViewer is working as a package

So I got gridviewer packaged up and most of the code I wanted transferred into it, and I am happy to say that code and web-code can happily coexist in a maven project and behaves as you think it would.

Next steps are to pull in AMDS data and start modifying the gmap-pane to handle different sets of metadata and different services.

AMDS-UI

So, AMDS-UI is going to be broken into three parts, and the structure will be very similar to Quicksilver but will be replicated because the two apps will be doing two very different things.

Amdsmulticlient will be the main model generator. It will contact AMDS clients with queries(securely or non-securely, depending on the URL), and deal with marshalling data into objects which can be understood by...

Amdsgmap which is the main controller. It will take the data objects and convert them into data better understood by gmap-polygons (and will be using the gmap-polygon jar). It will pack up the data into gmap-polygons through a suite of JSP backing code for...

Amdsgmap-UI, which is the main view. This will take data from amdsgmap and display it and handle the requests and responses to the actual viewer. It will also hopefully have some admin pages for adding and removing AMDS sites and configuring some of the properties.

The look should be a lot like quicksilver, but the options will have shifted, and some of the features wanted for quicksilver will be provided by for amds-web.

I have created all these projects, and tomorrow I will extend the code and set mockups for handling data. Soon after that will be integrating Vaughns client, and then many cycles of testing and coding.

My hope is by the end of the week to have Quicksilver-esque functionality coming from AMDS test data.

Creating a new maven project: npdsamds

I have created a new maven project to hold the service end of the npds-to-amds code, and it is appropriately named npdsamds. It will essentially be a gridified version of the npds service that should be served up via a globus container (Thus requiring security and all that jazz).

Maven solves many worlds of hurt with its dependency management, and the repositories where you can store jars so that others may use them. It also makes other tools pale in comparison and seem much more annoying.

One of them is SVN, which for some reason is "old" on my copy of ubuntu and cannot be easily upgraded using the standard ubuntu tools. Thus, whenever I try and add new projects to the SVN repository or move files, I always have to create new folders and re-check-out the code because otherwise the version of subclipse I use with (you guessed it) eclipse touches the svn special secret hidden files and makes it so that lowly command-line subversion is eschewed from adding the files. I also have to do this every time that subclipse botches something.

The other tool is ant. Don't get me wrong, maven /uses/ ant. But maven is to ant as a chocolate cake is to baker's chocolate. The combination maven of which ant is a part, is much better than the bitter chunks of ant by itself. Ant offers all sorts of commands that rely on all sorts of duck-aligning and basically anyone who writes an ant script also has to write a very long doc about what each part of the script can do and how to invoke it. Maven, you can usually just tell someone "type 'mvn package'" and it will produce the thingy they need, and it is a LOT easier for the person who coded the stuff inside the maven package to set up the project that way.

Don't get me wrong, both tools are very powerful and useful in their own right. I just have a small personal bias in that maven hasn't made me curse it's name yet.
But, we might write a plugin that allows maven to build globus services. So, I'm sure I will be cursing it's name soon enough.

Introduction to Introduce

So, I read much more about Globus and got introduce up and running, more importantly I figured out what I want to try and do with it in the coming weeks: Make a service that will comply with latest AMDS Spec and a client that will allow me to ping it.

The part of converting Poison Center data into AMDS Responses and AMDS Queries into Poison Center queries, while tedious, is something I can envision doing. I can also envision doing the skeleton from scratch, but that would be absolutely horrible to try and do if I don't have any really powerful tools like XML-spy... or Introduce.

The only problems with Introduce for what we need to do is that it is a bit geared towards caGrid, and likes to package up a few special libraries that end up causing some interesting deploy behaviors using globus tools when it is NOT part of caGrid. It also doesn't produce particularly maven-able results from the get go, so there is some footwork to be done playing with packaging and building and deploy tools.

There is also some work to be done with the client and making sure that it's security libraries don't conflict with existing secure clients (like RODSAdai) or how to place libraries so they don't interfere.

Thus, this week will be devoted to getting the service skeleton set up for a secure globus service.

The other cool thing might be that once this is done, it can be saved off as a shell for any given skeleton that needs to wrap other services, since AMDS is supposed to be a repeatable standard.

Need almost coded.

So I got the PoiconDai-web piece coded... so now it is being gracious enough to put the chart above the table of data.

In Rodsadai-web, I hit a few snags service-enabling, so I will probably have to troubleshoot. Luckily, I took a snapshot of the working code and put it into a tag of SVN and just ported it back for a demo that is tomorrow.

I am really liking MVN and SVN.

RODSAdai, PoiConDai, and NPDS-ws

Today I started off by having a productive discussion with the folks who helped me with RODSAdai: Dr. Jeremy Espino from Pitt (who brings the RODS) and Mario Antonioletti and Alastair Grant from University of Edinburgh (who bring the OGSA-DAI).

We discussed future developments, concerns about security, scalability, and automating installation and configuration as much as possible. We also talked about our availability being reduced for the next phases... But mainly we talked about the fact that one of the things that got lost in the making of RODSAdai 1.0 was a quick and descriptive design document that explained what we were all trying to do and how. Something so that Alastair, Mario, or Jeremy would be able to go "oh, well to connect it that way you should try configuring it like this..." Furthermore it is much more likely that the whole Axis/Globus conflict would have been found earlier and mitigated sooner since Alastair was already familiar with that particular limitation. Thus, the first step for the next phase of RODSAdai is to take the ideal-state and current-state diagrams of RODSAdai that I made, and more information about what they mean, how they currently work, and how I hope to get them working to focus advice and let anyone search for red flags.

But I won't be able to do that until Thursday of next week, because right now I am in demodemodemo mode to get PoiConDai in some form of pretty graphical display so we can show that yes, we are pulling Poison Center call data into the CDC.

This was greatly moved forward this morning with the help of Dr. Espino, namely because he already wrote code that used AXIS to consume the NPDS webservice a couple of weeks ago and had already incorporated all of the JAXB libraries needed for parsing the "any" response returned by the web service. The codebase is called npds-ws-client, and I am going to use it because it is pretty and is along the lines of what I was going to do, only with the properties file and return serializing I was planning on doing already implemented (and thank goodness for it).

Let me pause here to say how much I love maven again. It has extensions for generating axis stubs and jaxb code, so all you need to do is have the WSDLs handy and you don't have to worry about constantly regenerating the parsing bits and movig them about manually. How cool is that!?

Meanwhile, my next step is to incorporate a few of those unit tests I posted about earlier, and then set up a web-client front end that I am going to call PoiConDai web for showing the data in an ever-snazzier JSP. I think I am going to get something really nifty ready by Tuesday Evening.

13 new jars

So... 13 new jars were added to the RODSAdai project in order to get the simple client test working (as opposed to just compiling). 2 of them were extra Ogsadai jars... and the other 11 were based in globus, and it took me all day to ferret them out.

I tried deploying the jars to our repository, but it seems that sourceforge doesn't like something because I kept getting 405 (method not allowed) errors, thus I have sent an email to Anurag to start the process with him since he set up the repository initially...

The other reason I sent him the jars and dependency info is because my main workstation is in the process of being cloned and I wanted to make an email-based backup.

Tomorrow will most likely be spent creating a working Ubuntu globus node, as my current one got corrupted.

Now loading server lists from properties files

I hunkered down to do some serious coding of Rodsadai, and I managed to learn a lot more about how maven likes to test things. The most important thing being this: if you are running a test, the properties files (and any other resources you put under the resource or META-INF directory) apparently need to be duplicated in a resource directory under the test peice too.

It makes sense when you think about it, that way you can put in test data you know is there and not risk the configurable data being changed or having reserved names just for the sake of testing, but it was one of those painful "what is going on!?" types of learning.

That being said, the RODSAdai class can now load a server list from the properties file in the format I laid out. Tomorrow, comes the setting up of the rods database, and architecting of how GTSecureClient will be modified to best pass to TimeSeries and SpatialSeries.

Why Maven?

Dr. Savel asked me a very valid question (offline) of why we are looking at Maven. I'm posting my answer here as it affects our programming method and others may ask the same question.

Maven is a build and configuration tool that lets programmers describe a software project in xml in such a way that any programmer anywhere can use the maven tool to compile, build, deploy and document a project in a single common, automated manner.

Without maven, Peter (until recently the only programmer working on the PHIRG) would need to spend hours helping you configure your environment, compile your changes and deploy your changes for testing. This is difficult as new programmers trickle onto the project, but impossible in a distributed & open source project.

So maven makes open source, distributed projects possible. Now that we have a build and a repository (thanks Anurag and Peter), any developer in the world can access the source code, make some changes and test them out. I hope that this will lead developers to send in change candidate snippets of code for a committer to evaluate and commit, but my hopes aren't too high.

Repositories and Builds and Tomcat, Oh My

Yesterday I was here for a while trying to get SVN to behave. I found that the "gov/cdc/phgrid/ogsadai/example" path was stuck in the root directory, and since I was trying to create a the same path inside the maven directory structure (under /main/src/java/, I think it was) it refused to add it there. Even after I removed the old gov structure. I was thinking about scouring the old repository, had Brian try a few things, and still hit roadblocks



Then I had the late arriving epiphany to just move the tree, and that worked, and it was good.



Then I gave Anurag a zip of my local repository files, so that he could create a public repository.



I also had a wonderful discussion with Jeremy about a RODS-OGSADAI interface. I had a bit of a paradigm shift, but in the meantime, I have a good idea for what sorts of use cases, test cases, and code I will need to develop. I have also taken up the task of finding out why Tomcat doesn't find the same security credentials that were find for command line operation, as that hurdle will need to be crossed eventually for happy JSP-based items pulling data from the grid.



Cheers, I hope everyone has a wonderful weekend!

Maven and it's lovely repository

So, I spent a good part of the day paring down how many libraries my example client code actually needed from globus and ogsadai respectively. The list dropped from 40 to about 8.

Then I spent a lot of time putting one of those 8 into the local maven repository in a way that maven seems to recognize... and adjusting the pom file to point to it as a dependency. I hit "mvn package" (ie. compile) and I got a long, thoughtful message about how the dependency wasn't found and how I can add a file with the helpful "mvn install:install-file" command. They also had a "mvn deploy" command for deploying code to a commonly owned repository... showing that deploying to a central and accessible location is

Then proceeded to add all 8 files in about 30 minutes, along with dependencies to the pom file, and then ran mvn compile successfully.

I am really beginning to like maven!

Tomorrow, I am going to put some of the interface and repository plans for the RODS<->OD interface to paper... perhaps start some of the code for simple things like resource discovery. Then, I will test to see whether the jar produced runs properly when put into the proper environment... then I will start reading up on where to put the JSP files. I will also make an eclipse project file and import it into eclipse.

There is also a meeting to discuss package layouts and plans and the like. I am looking forward to it.

SVN and Maven

I have checked in most of my code into the Sourceforge SVN, and I just got the note that we should be using the package gov.cdc.ncphi..... and not org.cdc...

I guess I get to learn about subversion directory changes next :).

Otherwise, I have hit a challenge on the maven-ifcation of the code. Maven prefers jar files stored in remote repositories somewhere. All the OGSA-DAI and Globus code is set up with a series of build files (especially globus considering it has a lot of C++ code in addition to Java code) but there is no maven repository. I experimented with moving jar files into a resource location, but I think I am just going to have to store the jar files in the local repository with the appropriate maven metadata. I am not sure of the best way to do that, whether there is some sort of tool, or whether I should try and get maven building the OGSA-DAI code.

I guess what I am looking for now is a resource for taking projects that already exist and getting it to a point where maven is now managing it, because a lot of the stuff that was handled by changing the classpath before now is going to be handled by formatting the myriad dependencies into an xml file.

I feel like in the end we will have the local repository and we will be able to publish it to our own repository so that other development groups can reference our files with the appropriate sets of Ogsa-Dai libraries and globus libraries.

I also need to start researching how to implement a globus WS extension and looking into the best ways to get data back and forth between RODS and Ogsa-Dai.

Also working with Maven

Having just read the post below mine, I see that Anurag and I are on the same track.

I have created a "ncphi-examples" project to store all the little scripts and pages I have created already, and I now have moved in one of the peices of my modified OGSA-DAI code and already have it throwing compilation errors. I will probably fire off a quick email to Anurag about how best to import the OGSA-DAI code... at this point I am leaning towards just including it in the source tree, so that it will compile it locally and bundle it all into one big OGSA-DAI jar along with NCPHI-Specific modifications. By the end I imagine this particular Maven project will have the OGSA-DAI code, references to the jar repositories needed for builds provided by the maven site, and all the web code too.

As I play with Maven more and more, I find it perfect for just forcing you to have a sensible, realistic build structure. It will make you put your code in one place, your resources in another place, and show you the beauty of unit tests. In the end you get the deployable Jars and Wars that just make life that much easier... and I get the impression that your work will just be that much more legitimate when you go "oh, just install this maven tool, sync to the SVN, and then run 'mvn package' and you should be able to verify the compilation."

If anything this initial build will be a wonderful starting package for any future major NCPHI Globus OGSA-DAI code collaborations, and that is fortuitous considering I am also looking into defining the interface between RODS and OGSA-DAI for a outbreak detection solution that would use remote database access. Some of the interesting use of the extensible functionality that I can think of adding include the ability to deploy OGSA-DAI resources on the fly after filling out a simple form.

Otherwise, my vacation was lovely and personally productive, and it seems that a lot of documentation was completed after memorial day, but this gives me new solid directions and a lot of excitement.

New people and new things.

Today I talked a lot with Peter Casey (the new developer), planned a meeting with Anurag Chawla (the second new developer) sat in a meeting with Brian Lai (as detailed below by) and Brian Lee.

One of the things that is being heavily discussed is setting up all the archetypes that will make this a Real Project. This means setting up some sort of code repository and build scripts for making something that can be deployed somewhere.

In preparation for all this, I have been researching maven... and honestly it's really neat how it will just build out a full project with two inputs... complete with test scripts and apache style documentation pages.