Grid UI Update

Here are couple of more UI's for the Grid, I found
1) http://wiki.arcs.org.au/bin/view/Main/FileTransferSGGCUsageGuide
from Australian research collaboration service which has the capabililty for gridFTP, third party transfers, and also a cool feature which is multi replica transfers. Its sort of like bittorrent with the variation like when we have a large file whose copy exists in several locations, we can specify multiple locations from where we want to get the file which will potentially create a local copy of the file faster.

2) The other one is http://commons-vfs-ui.sourceforge.net/ which is a Commons VFS implementation from Apache. It provide support for Storage Resource Broker and GridFTP.

Remaining SRGM Test Cases

The latest version of SRGM_POC test case document is on the wiki right now with test case 4G to be revisited with either a little more elaboration or a change of approach. I am currently working with Vaughn Mcmullin on that. Apart from that, the case 4J ) Evaluate user interface projects for RFT management and operation (including, but not limited to, job starting/stopping/resuming, route management) ( Req. No 16) is what I am looking into. Early in our work we came across vbrowser which covered a lot of ground, in terms of functionality, as one of the possible UI's. So before we move further in this area, upon Brian's recommendation, I am posting a couple of features that we would want to zero in on while evaluating various UI's
1) Support for manual/interactive as well as automatic transfers. Although, as discussed ,automatic transfers could be more of an added feature rather than a basic requirement.
2) Ability to see the status of jobs going on at any particular time by being able to provide a way to query the persistent storage.
3) Capability to stop the jobs manually in the middle of a transfer, which could remove parts of payload already transferred.
4) As we know,Vbrowser is applet based so making a decision whether we would like to have a web based UI for our purpose.
5) Raj(from Globus) mentioned in one of the calls that they do have a UI implementation also so I will be following up with him whether their implementation covers some of these already.

I also found another UI implementation for GridFTP of WISENT, which is an eclipse based, cross-platform, open-source GridFTP client which allows to transfer, view progress and delete files. Although it would be interesting if they provide support for RFT. Please feel free to check out the url (a flash presentation of the application)
http://bi.offis.de/gridftp/screens/gridftp.htm
Please feel free to add what you would think, could be a feature of the UI for GridFTP.

SRGM Progress

Here's the update on SRGM test cases. With the combined efforts in the past couple of weeks we have been able to cover a lot of ground on these test cases. The ones which are still left are :
1) Transfer HL7 file from partner node (node B) to CDC lab node (node A) using PHINMS issued digital certificates.
2) Evaluate user interface projects for RFT management and operation (including, but not limited to, job starting/stopping/resuming, route management).
and the ones which are knocked off the list are :
1) Evaluate reliable GRIDftp and WS-RM against OCISO guidelines and requirements
2) Evaluate reliable GRIDftp and WS-RM against NIST guidelines and requirements (specifically, but not limited to, the FIPS 140-2 Cryptographic Module Validation Program; FIPS 200 Minimum Security Requirements for Federal Information and Information Systems).
3) Configure lab node and partner node to use PHINMS digital certificate.
4) Transfer HL7 file from partner node B to partner node C using stored at CDC lab node (node A).
5) Test ability of Globus components to integrate into existing PHINMS infrastructure (transfer of program payloads through both PHINMS and Globus).
6) Evaluate capabilities for unattended retries/delivery of payload file.
7) Evaluate capabilities for end to end payload level encryption (Done thanks to efforts of Vaughn Mcmullin, the document still needs to be updated with teminal snapshots)
8) Evaluate capabilities for guaranteed once and only once delivery of file and the robustness of its duplicate file detection
9) Evaluate capabilities for large payload (> 100 MB) reliable delivery.
10) Evaluate capabilities for file payload encryption for strength and validity.
11) Evaluate capabilities for reliable data exchange for once-and-only-once transport of payload data.
12) Evaluate reliable GRIDftp for reliable messaging for reliability due to node availability

So all in all of the 14 cases we have finished 12, 2 more to go, the completion of which we will be on working on next week.

SRGM Progress

I talked to Raj(at Globus) regarding providing more details regarding the RFT Flow diagram provided. I will be upgrading the version on the wiki as soon as I get it.

On the test cases, the PHINMS certificate has been installed on the nodes. So no a successful transfer will knock off another test case 4E : Transfer HL7 file from partner node (node B) to CDC lab node (node A) using PHINMS issued digital certificates (Req. No. 05 & 06).
The test case document on the wiki will be updated as soon as the test is finished.

Also as Brian, Dan,Vaughn and me talked, one of the enhancements mentioned was having the option to set flags for optional parameters regarding the file being transferred. One criteria which came up was having a flag to set for Time Sensitive Data(as in if the file does not reach within specified time frame, discard it). Globus folks agreed that it would be a good feature to have and they can include it in the next version. Also, please feel free to add if you think of some more enhancements and we can convey them to Globus folks further.

Progress on SRGM and call with GridFTP guys

We had a productive call with the GridFTP guys (Raj, to be specific) and they were extremely helpful and prompt in responding. The couple of questions posed to them were
1) Was there any way they were achieving encryption for the payload before putting it on the transport layer. Raj told us about Globus XIO Driver which can extended to do so. However since it not an inbuilt feature, we have decided to go with the same encryption used by PHINMS, with the help of Vaughn, since that algorithm is already approved by CDC standards. This will help us satisfy the test case 4G and 4I

2) Raj also provided us a transactional flow diagram for RFT to which some more details need to be added. I will be calling Raj to get more of that information

Also, it came to light that the version of Globus installed and our nodes is 4.0.5 whereas they have already rolled out 4.2. Now 4.2 already contains inbuilt persistent storage which can maintain the state of files being transferred through RFT. Dan mentioned that an upgrade of the toolkit version is on the agenda which will definitely help our case.

Right now the test case left, which Vaughn and Dan are working on, entails installation of PHINMS digital certificate on the node and then doing file transfer, details of which are in the blog post below.

Questions to GridFTP guys

After having a discussion today with Vaughn, these are some of the questions we agreed on to ask to GridFTP guys, pertaining to OCISO guidelines.
* What are they using for payload level encryption as in right now, we do double encryption i.e. we encrypt data first then place it in the PHINMS queue where there is transport level encryption. Taking the same scenario, we know that for transport level, encryption is achieved through SSL however what is used for encryption before that. Now that could be outside the scope of GriFTP however its worth talking to them as to whether they have been involved in similar initiative like that.

* At present, PHINMS works through Request / Acknowledgement. Now RFT helps us in achieve the similar funtionality in terms of the fact that we can track the status of transfer as in Active/Pending/retrying/completed. So in this case, if they can provide us with some transactional diagram, sequence diagram which can help us explain this in our documentation

* What version of SSL is being used.

* What metadata fields can be configured

* As far as tranmission integrity is concerned, GridFTP is built as an extension of FTP which uses TCP/IP protocol for data transmission. Since TCP uses a checksum computed over the whole packet to verify that the protocol header and the data in each received packet have not been corrupted, this ensures data integrity on the whole packet. We can reconfirm with them on that.

Next step would be to set up a call with GridFTP to discuss that.

More on SRGM Test cases

After talking to Brian, we decided to define some kind of timeline we're looking at, in terms of the test cases defined in the test cases document for SRGM_PoC and also identifying tasks vs research items. So here it goes

* Evaluate the GridFTP functionality against OCISO guidelines and requirements (Req. No 01)
* Evaluate reliable GRIDftp and WS-RM against NIST guidelines and requirements (specifically, but not limited to, the FIPS 140-2 Cryptographic Module Validation Program; FIPS 200 Minimum Security Requirements for Federal Information and Information Systems) (Req No.02)
These research items need to be looked into by involving the GridFTP folks, maybe on a call but before that, I will get together with Vaughn and see how we can formulate specific questions which we would want to pose to them.

* Transfer file from partner node (node B) to CDC lab node (node A) using PHINMS issued digital certificates (Req. No. 04,05 & 06)
We will get hold of a PHINMS issued digital certificate and do a transfer and since the transfer is using SSL, we would not have to explicitly encrypt and decrypt the file.

* Test ability of Globus components to integrate into existing PHINMS infrastructure (transfer of program payloads through both PHINMS and Globus). (Req No.08)
To be further looked into

* Evaluate capabilities for end to end payload level encryption (Req. No 10)
Confirm if there is a switch to enable payload level encryption. This is different than TLS/SSL

* Evaluate capabilities for file payload encryption for strength and validity (Req. No 13)
If previous test is valid, then which algorithm is used to complete the encryption

* Evaluate capabilities for guaranteed once and only once delivery of file and the robustness of its duplicate file detection (Req No 11)
Confirm that a message Transport State is tracked by both Sender and Receiver (Request-Ack)

* Evaluate capabilities for reliable data exchange for once-and-only-once transport of payload data ( Req No 14)
Application Specific

* Evaluate reliable GridFTP for reliable messaging for reliability due to node availability ( Req No. 15)
Same as the test case where we brought down the node multiple no. of times to check automatic restart. Need to check whether we can control transfer of time sensitive material.

* Evaluate user interface projects for RFT management and operation (including, but not limited to, job starting/stopping/resuming, route management) ( Req. No 16)
To be further looked into.

I will also be updating the test case doc with these additions.

Progress on SRGM_PoC test cases

We (me and Dan) finished up on 1 more test case. One which required us to observe the behavior on multiple restarts and system behaved as expected. Results are updated in the test document (available on the wiki). Another thing we observed was that right now if try to re transfer a file by the same name, once it has already been transferred, it overwrites the old file. That is not something we are looking to do. So the idea right now is to find a switch which may allow us to get a warning saying that this filename already exists. Also looking forward to being on the call with the GridFTP guys which will definitely help us knock out couple more test cases.

Grid Messaging Test Cases

I worked on finishing up the first presentable draft of the Secure Reliable Grid Messaging PoC test cases. Its right now located at
http://sites.google.com/site/phgrid/active-projects
Please feel free to browse through the document and provide your feedback. Right now we have tried to explain the tests that we have performed so far, with others outlined in the document. After we created the first draft, Brian recommended some additions, which have been incorporated. If you think of a way where we can be more clear in explaining what has been tested so far, please let us know.

File Transfer on the grid

Today, with the help of Dan we tried to recreate some functionality which the grid promises. It would be safe to say, we got pretty acceptable results. First we tried to transfer a file >100mb over the grid using just normal gridftp, it went through fine. The next step was to test the reliable file transfer. The way we decided to check the reliability was to power down one node while the file was transferring (to simulate a test case scenario). So Dan started the transfer, pulled the cord out for a minute and as soon as he plugged it back in, voila, it resumed as if nothing happened. Another thing, Dan informed me about was, that the timeout can be set by us as to how long we want to leave the file in the queue before it terminates the transfer. Right now it's set to 60 minutes, so we decided to increase the time and Dan will pull the cord before he leaves and plug it back in tomorrow morning. Now that would be a real extreme test of one facet of reliability. Another thing which we noticed was that upon powering back the node, it did not ask us to whether we want to resume or not, it just picked up automatically, which works for us because that ensures that as a user, once the transfer starts, you don't have to monitor whether the file reached or not and I am assuming that's the way PHINMS functions too.
Currently I am working on creating a test case document, to record all these cases and to outline what all are we going to achieve based on the requirements laid down by the project charter.

Postgres Errors

The Postgres database on the staging node was failing to start the postmaster due to a recent IP address change. I updated the postgresql.conf file to reflect the recent IP change. Postgres started fine after the configuration file was modified.
Postgres version: 8.1
Original Postgres Error: autovacuum not started because of misconfiguration
Fix: Modified the /var/lib/pgsql/data/postgresql.conf


Tested 100+ MB file GridFTP and RFT file transfers with Anurag.

Project Management update

We held the a project kickoff meeting for the CDC and Contractor resources who will be working on the Secure Reliable Grid Messaging PoC. We'll be meeting over the following few days to get together a set of detailed requirements and tasks based on the project charter. Following from this will be timelines and resources on the tasks so we can establish more fine grained expectations.

I met with Tom Brinks this afternoon and we started working on a work breakdown structure to define the functional areas within the project. We'll post a draft on the wiki as soon as something digital exists.

Security Evaluations

I had a call with a NCPHI security steward to ask about some of the security evaluations that we'll be performing with the Secure Reliable Grid Messaging PoC project.

She pointed us specifically to the FIPS 140-2, NIST SP 800-53 and eAuthentication guidelines.

It's a lot of reading, but we'll need to evaluate the algorithm type and strength used by Reliable GridFTP, integrity checking of the message payload, and certificate usage.

Tagging Posts by Proof of Concept Project

During today's PHGrid call, someone suggested (I cannot remember who precicely) that we add tags for each of the Proof of Concept Projects running through the summer. This will allow a reader to quickly filter and sort posts by the area they are interested in. Some posters are already tagging their writings to allow for this. I want to propose the following tags that writers will assign whenever they write regarding a PoC:

• Grid PoC Phase II PoC- CoE PoC


• Secure Reliable Grid Messaging- SRGM PoC


• Leveraging Grid to Enhance Biosurveillance Capacity - GBC PoC

I've linked to the searches in the list above. In the future you can click on these tags to search for posts specific to a PoC project.