Daily Lab / POC Activities

Extramural:

• Created a draft version of the Virtual Data Toolkit install document

Note: The Trusted authority procedures need to be modified. We may need to manually extract the trusted authority certificates because the VDT installation does not contain $GLOBUS_LOCATION/sbin/gpt-build. This method was used in the previously successful test of VDT.
The files below will have to be extracted from the Certificate Authority file:
globus-host-ssl.conf.hash
globus-user-ssl.conf.hash
grid-security.conf.hash

• Configured the Globus Container Service on lab 1002.

bubba@gump:/etc/grid-security> counter-client -s https://some.host.net:8443/wsrf/services/CounterService
Got notification with value: 3
Counter has value: 3
Got notification with value: 13

• Configured RFT Service service on lab 1002

• Contacted Ashley from CaBig

He mentioned the issue they are experiencing
is due to hard-coded absolute paths within the software. Instead of using variables, the software has defined absolute paths which causes it to have difficulty locating files if they are not install in a specific location. Ashley said he will try to modify the software to use variables instead of absolute paths.

The Mysql database was also reinstalled during the troubleshooting process.

Daily Lab / POC Activities

Extramural:

• Installed GCC on lab server 1001
• Performed a successful installation of the Globus Toolkit using VDT 1.8.1.

Special Note:

In addition to using the “pacman -pretend-platform linux-suse-9” workaround, the following variables should be set to avoid potential error messages:

export VDTSETUP_ACCEPT_PLATFORM=y

export VDT_ALLOW_UNSUPPORTED=1

• Tested the new Globus installation using the certificates form lab server 1002. Was able to perform data transfers between all sites.

Daily Lab / POC Activities

Extramural:

• Installed a new grid node using the Virtual Data Toolkit. (VDT) The software does not support SUSE 10. Note: Use the following workaround to complete the install on SUSE 10: pacman -pretend-platform linux-suse-9
• First attempt to install VDT resulted in a "exit status 1 code: unable to execute gcc." Will install GCC and retry installation.
  
• Performed successful GridFTP data transfers to all gridnodes.Discovered route cause to hanging file transfers, details below:
  

Problem: GridFTP trying to use private IP to transfer data to a node behind a NAT.
Symptom: The file transfer displays a “NO ROUTE TO HOST” error or hanging file transfers

Test Procedure:
1.Start a proxy using: grid-proxy-init -debug -verify
2.Run the command: globus-url-copy -dbg file:///tmp/foo gsiftp://any.remotehost.net/tmp/foo
3.Pay attention to the debug output and look for the line: Entering Passive Mode (192,168,0,5,195,80) The first four sets of numbers is the IP Address GridFTP is trying to establish a connection with. In this case it is 192.168.0.5.
4.If the address is an internal address, edit the /etc/xinet.d/gridftp file to include the following: server_args = -i -data-interface
5.Restart xinetd using the following command: /etc/init.d/xinetd reload
6.Retry the file transfer

After the above procedure was performed all transfers ran successfully.

• Updated Grid Node Installation Document to include the newly discovered fix for the hanging transfers

Daily Lab / POC Activities

Extramural:


Renamed and rebuilt Globus install using the following tip from HP:

#################################################
Build the Globus Toolkit from source or binary bundles following the instructions available from Globus. After the build completes:

* Set the $GLOBUS_HOSTNAME environment variable to the GCHN (as determined above):

(sh) $ GLOBUS_HOSTNAME=host.dom.ain; export GLOBUS_HOSTNAME
(csh) % setenv GLOBUS_HOSTNAME host.dom.ain

Then run either "make install" (for GT4) or the Globus post-install script (for GT2). Setting this environment variable is necessary because many configuration files created during the post-install process have hardcoded host names (e.g., some job managers and MDS configuration files). By setting $GLOBUS_HOSTNAME before the installation step, these hardcoded host names will be correctly initialized to the GCHN.
* Request the Host and LDAP certificates, being sure to use the GCHN.
* Use env(1) to set $GLOBUS_HOSTNAME for Globus services launched out of inetd(8). For example:

gsiftp stream tcp nowait root /usr/bin/env \
env GLOBUS_HOSTNAME=host.dom.ain LD_LIBRARY_PATH=/globus/active/gtk/lib \
/globus/active/gtk/sbin/globus-gridftp-server -i
####################################################

Updated Grid node installation document to include troubleshooting information

Requested new user and host certificates from host grid

Daily Lab / POC Activities

Extramural:

• Discovered that the connectivity issues were due to a firewall configuration in the ADSL Router. The firewall was configured to drop packets in the 50000-51000 range.

• The ADSL router also has a limitation when it comes to dealing with dynamic ports. The router engineer was able to find a workaround that allowed dynamic ports.

The problem was uncovered by running the following tests:

Connectivity Test From Tarrant to Atlanta:
Start the server
/usr/local/globus-4.0.5/sbin/globus-gridftp-server -s

Server listening at node.hostname.com:50000

TEST THE PORT FROM TARRANT
bubba@gump:~> telnet node.hostname.com 50000
Trying yyy.yyy.yyy.yyy...
telnet: connect to address nnn.nnn.nnn.nnn: Connection refused
bubba@gump:~>

NOW WE SHUTDOWN THE SERVER AND TRY ANOTHER PORT
[5802] Mon Feb 25 13:42:37 2008 :: Server is shutting down...
LLAB-INF-1002:/etc/xinetd.d #

NOW WE START THE SERVER ON 2811
LLAB-INF-1002:/etc/xinetd.d # /usr/local/globus-4.0.5/sbin/globus-gridftp-server -s -p 2811

TEST FROM TARRANT
bubba@gump:~> telnet node.hostname.com 2811
Trying yyy.yyy.yyy.yyy...
Connected to node.hostname.com.
Escape character is '^]'.
220 node.hostname.com GridFTP Server 2.5 (gcc32dbg, 1182369948-63) ready.

NOW WE USE THE SAME EXACT COMMAND TO START THE SERVER ON PORT 50000

LLAB-INF-1002:/etc/xinetd.d # /usr/local/globus-4.0.5/sbin/globus-gridftp-server -s -p 50000

NOW WE TEST FROM TARRANT
bubba@gump:~> telnet node.hostname.com 50000
Trying yyy.yyy.yyy.yyy...
telnet: connect to address yyy.yyy.yyy.yyy: Connection refused
bubba@gump:~>

THIS CLEARLY PROVES THAT PORT 50000 IS NOT OPEN IN ATLANTA.


On the Atlanta server side we are currently able to run gridftp from Dallas and Tarrant to Atlanta.

On the client side we are able to run gridFTP both ways using: globus-url-copy -dbg file:///tmp/foo gsiftp://node.hostname.com:2811/tmp/got-it (Command works in both directions)

Daily Lab / POC Activities

Extramural:

• Continued to research possible fixes for the Internet connectivity issue between Atlanta and Dallas.
• Moved certificates from Lab Node 1002 to Lab Node 1003 and tested the connectivity between Atlanta and Dallas. (Same result, the file hangs with no errors)
• Currently awaiting new host certificates and user certificates for the new 1003 grid node.

Daily Lab / POC Activities

Extramural:

• Installed Globus Toolkit version 4.0.5 on a second Atlanta lab machine

• Updated install instructions based on lab machine install

• Requested new host and user certificates for the new grid node

• Posted install instructions to the project's FTP site.

caBig

• CDC Concept mapping completed
• caArray model was loaded successfully to the sandbox environment
• Evaluated model in sandbox browser at this link
  

Daily Lab / POC Activities

Extramural:

• Worked on Grid Node installation document

• Re-built Lab second grid node with SUSE Linux Enterprise SP1

• Tested possible fixes between Dallas and Atlanta

Daily Lab / POC Activities

Extramural:

• Configured ntp to synchronize time between grid servers

• Worked with Dallas network engineers to open TCP ports

• Able to perform a third party transfer between Atlanta and Tarrant from Dallas's server.

• Researching the SSL handshake error when transfer is performed from Atlanta.

• Researching Globus NAT issues: Globus documention mentions the software has problems running behind NAT firewalls. ( Defining $GLOBUS_HOSTNAME in the etc/xinetd.d/gsiftp file may solve the issue.
• System connects on a public IP, but returns a private address to the remote server. The remote sessions then sends a "NO ROUTE TO 192.168...." error.
  

• Started install of 2nd Atlanta Grid Node.
• Will try an ATL to ATL transfer on current ATL server with no changes to the config.