A thousand little libraries.

I am, I think, very close to getting the first of several little JSP applications working, but two things keep holding me back...

The first is that I keep getting cryptic OD errors and for some reason my setup is immune to logging... and the second is that I cannot try and debug using my favorite IDE (Eclipse) because it just seems to not want to run any OD queries from the workspace.

In both cases I get cryptic errors which seem like "Ogsa-dai-" followed by a timestamp.. which are thoroughly unhelpful.

I am guessing this all boils down to some sort of classpath error, libraries not being found somewhere... especially because all this code seems to work but only from one directory and only from the command line. If I try and run a modified class that worked perfectly from one directory in the servlet engine or eclipse... *bam* cryptic error.

I am going to have to sit with the Edinburgh guys for some insights methinks.

Cheers,
Peter.

Grid Questions

Here are the answers to the Johns-Hopkins the questions.

What services will be used on the grid?

We are currently using the following Globus services:

GridFTP - A file transfer protocol that provides secure, robust, fast and efficient transfer of data.

Grid Security Infrastructure OpenSSH (GSI-OpenSSH) - A Globus replacement for OpenSSH
Open Grid Services Architecture Data Access and Integration (OGSA-DAI) - A middleware product which supports the exposure of data resources, such as relational or XML databases, on to grids.

Reliable File Transfer Service (RFT) - A web service that provides “job scheduler"-like functionality for data movement. RFT should be used for extremely large data transfers.

Replica Location Service (RLS) - A registry that keeps track of where one or more copies, or replicas, of files exists on physical storage systems in a Grid environment.

Simple Certificate Authority (SimpleCA) - A package that provides a simplified certification authority for the purpose of issuing credentials to Globus Toolkit users and services. This package is often used for testing Globus installations.

Web Services Grid Resource and Allocation Management (WS GRAM) - A Unix server suite that enables users to submit, monitor, and cancel jobs on Grid computing resources.

Web Monitoring and Discovery System (WebMDS) - A web-based interface to WSRF resource property information that is available as a user-friendly front-end to the Index Service.


Coming Soon:

Data Replication Service (DRS) - A web service that combines RFT and RLS to provide a pull-based replication capability that ensures a specified set of files exists on a storage site.

Java Commodity Grid Kit (Java CoG kit) - A Jglobus library that provides a client-side API and limited server side functionality to the GT2-based services such as GRAM and MDS. It also provides a client-side API for GridFTP, MyProxy and has extensive GSI support.

MyProxy - An online credential repository used to eliminate the need for manually copying private key and certificate files between machines.


How will you demonstrate grid functionality at the PHIN conference? The method for demonstrating grid functionality is still being decided.

Will multiple grid user accounts be created for the PHIN conference? No, we will create a single demo account that will have controlled access to grid data.

Tomcat in code alley

I have started to actually write some JSP code. The first one is going to be a simple query that can be run through a webpage... then I am going to start trying more dynamic pieces where different resources are selected and polled.

I am also discussing the best way to discover OGSA-DAI resources on the grid. We were thinking of perhaps building a MonALISA extension to poll the OD on a given box and then sending that back to a repository, which would reduce the hits on a network since it would be passive polling.

Tomorrow, I hope to get to complete a query through a webpage... but I am anticipating a lot of classpath resolution errors will hold me up (as I have very rarely worked on JSP and Servlet projects where this didn't occur)

Using an existing Certificate Authority (CA) within Globus

Question:

Hi Dan,

I was wondering if there was any support for using existing
authentication sources with the PHGRID. I am interested in using the
University of Washington's Kerberos and Shibboleth services to
authenticate against PHGRID services.

Answer:
Yes, Globus can be configured to trust x.509 certificates issued by a 3rd party CA. This is accomplished by copying the 3rd party CA's certificate hash file and signing policy to the /etc/grid-security/certificates directory.

Example Hash Files:
31f15ec4.0
31f15ec4.signing_policy

Note: The certificate hash is located by running the following command:
$GLOBUS_LOCATION/bin/openssl x509 -hash -noout < ca_certificate_file

The certificate's distinguished name must match the pattern found in the signing policy file.

Daily Lab / POC Activities

Extramural:

Configured Myproxy server on lab1002. Further testing needs to be done in order to determine the best practices for implementing security and scalability within Myproxy.
(I.E Accepted Credentials, Renewal Policy, Passprase Enforcement, Certificate Map, etc.)

The CaBIG developers are currently working on installing a CaBIG grid node on lab1004. I provided them with a history of the previous configuration efforts. The current developers ended the day by configuring /cacoresdk/conf/deploy.properties. Progress will resume on Monday.

OGSA-DAI over large sets

I had a few discussions with Alastair over how we think OGSA-DAI would work over large sets... like hundreds of thousands of distributed nodes

Some queries would be able to just crawl... namely things like aggregations (counts of an infection by zip code) where the set wouldn't get large just the counts would increase... but if you were doing joins on national or global data... it would probably need some sort of tiered architecture.

In my mind I see a MonALISAextension being built to monitor and manage OGSA-DAI instances... handling the delegation of what collects what data from 100 or so nodes and then propogates it up... Alastair pointed out that there were already plans for OD to manage itself in a assymetrical tree concept. It would probably be merged.. with MonALISA providing feedback about which nodes were reachable and their lag times... and OD then selecting to pull the queries through it's more efficient channels.

Otherwise, I have since started to focus on discovery concepts for my demonstrations, being able to use the APIs of globus and OGSA-DAI to figure out what data resources are available and at what locations.

More complex data

I have started using more complex data (sample data from OpenMRS) in my tuple merge and join tests... I might supplement that with RODS test data soon too.

I am also starting to consider the scalability and distributions of the grid when it starts to get Big. I think that OGSA-DAI will oversee most of the data pulls and merging, but only across about a hundred nodes which would then store their results in repositories. That way any polling algorithms will only have to pull from about a hundred "super" nodes instead of tens of thousands of regular nodes. The same would go for resource discovery.

OGSA-DAI update

I have installed the extensions for OGSA-DAI that allow more complicated joins and such... unfortunately when trying to modify the example client I start getting an error about not being able to find the newly installed resource.

After some research it appears that the new resources are being installed on the Globus container but not the tomcat container and I am testing with the tomcat container.

Thus, tomorrow morning will either be help in getting the new bits deployed to the tomcat container, or help debugging the Globus container connection.

Tomorrow I will probably start pulling from similar schemas on different computers and trying more complicated things like aggrigation. At the same time branching out to dynamically built workflows and more secure communications, assuming we get this bug out of the way quickly

Daily Lab / POC Activities

Extramural:

• VM Appliance DVD created for Pittsburgh
• Updated MonALISA installation document
• Researched exchange network
• Modified MonALISA module config on lab 1002

Update on NAPHIT Conversation

Following up the conversation held with NAPHIT, the following is a proposed approach to collaborate.

Pre-Public Health Information Network (PHIN) conference

• Target a webinar for July / Aug for the NAPHIT membership to more concretely describe PH Research Grid activities to date, and possibilities for the future.

PHIN

• Hold meeting to scope potential pilots, (e.g. for the security model/ administration).

Post-PHIN

• Conduct pilot project with NAPHIT membership.