Daily Lab / POC Activities

Extramural:

• Created a draft version of the Virtual Data Toolkit install document

Note: The Trusted authority procedures need to be modified. We may need to manually extract the trusted authority certificates because the VDT installation does not contain $GLOBUS_LOCATION/sbin/gpt-build. This method was used in the previously successful test of VDT.
The files below will have to be extracted from the Certificate Authority file:
globus-host-ssl.conf.hash
globus-user-ssl.conf.hash
grid-security.conf.hash

• Configured the Globus Container Service on lab 1002.

bubba@gump:/etc/grid-security> counter-client -s https://some.host.net:8443/wsrf/services/CounterService
Got notification with value: 3
Counter has value: 3
Got notification with value: 13

• Configured RFT Service service on lab 1002

• Contacted Ashley from CaBig

He mentioned the issue they are experiencing
is due to hard-coded absolute paths within the software. Instead of using variables, the software has defined absolute paths which causes it to have difficulty locating files if they are not install in a specific location. Ashley said he will try to modify the software to use variables instead of absolute paths.

The Mysql database was also reinstalled during the troubleshooting process.

Daily Lab / POC Activities

Extramural:

• Installed GCC on lab server 1001
• Performed a successful installation of the Globus Toolkit using VDT 1.8.1.

Special Note:

In addition to using the “pacman -pretend-platform linux-suse-9” workaround, the following variables should be set to avoid potential error messages:

export VDTSETUP_ACCEPT_PLATFORM=y

export VDT_ALLOW_UNSUPPORTED=1

• Tested the new Globus installation using the certificates form lab server 1002. Was able to perform data transfers between all sites.

Daily Lab / POC Activities

Extramural:

• Installed a new grid node using the Virtual Data Toolkit. (VDT) The software does not support SUSE 10. Note: Use the following workaround to complete the install on SUSE 10: pacman -pretend-platform linux-suse-9
• First attempt to install VDT resulted in a "exit status 1 code: unable to execute gcc." Will install GCC and retry installation.
  
• Performed successful GridFTP data transfers to all gridnodes.Discovered route cause to hanging file transfers, details below:
  

Problem: GridFTP trying to use private IP to transfer data to a node behind a NAT.
Symptom: The file transfer displays a “NO ROUTE TO HOST” error or hanging file transfers

Test Procedure:
1.Start a proxy using: grid-proxy-init -debug -verify
2.Run the command: globus-url-copy -dbg file:///tmp/foo gsiftp://any.remotehost.net/tmp/foo
3.Pay attention to the debug output and look for the line: Entering Passive Mode (192,168,0,5,195,80) The first four sets of numbers is the IP Address GridFTP is trying to establish a connection with. In this case it is 192.168.0.5.
4.If the address is an internal address, edit the /etc/xinet.d/gridftp file to include the following: server_args = -i -data-interface
5.Restart xinetd using the following command: /etc/init.d/xinetd reload
6.Retry the file transfer

After the above procedure was performed all transfers ran successfully.

• Updated Grid Node Installation Document to include the newly discovered fix for the hanging transfers

Daily Lab / POC Activities

Extramural:


Renamed and rebuilt Globus install using the following tip from HP:

#################################################
Build the Globus Toolkit from source or binary bundles following the instructions available from Globus. After the build completes:

* Set the $GLOBUS_HOSTNAME environment variable to the GCHN (as determined above):

(sh) $ GLOBUS_HOSTNAME=host.dom.ain; export GLOBUS_HOSTNAME
(csh) % setenv GLOBUS_HOSTNAME host.dom.ain

Then run either "make install" (for GT4) or the Globus post-install script (for GT2). Setting this environment variable is necessary because many configuration files created during the post-install process have hardcoded host names (e.g., some job managers and MDS configuration files). By setting $GLOBUS_HOSTNAME before the installation step, these hardcoded host names will be correctly initialized to the GCHN.
* Request the Host and LDAP certificates, being sure to use the GCHN.
* Use env(1) to set $GLOBUS_HOSTNAME for Globus services launched out of inetd(8). For example:

gsiftp stream tcp nowait root /usr/bin/env \
env GLOBUS_HOSTNAME=host.dom.ain LD_LIBRARY_PATH=/globus/active/gtk/lib \
/globus/active/gtk/sbin/globus-gridftp-server -i
####################################################

Updated Grid node installation document to include troubleshooting information

Requested new user and host certificates from host grid

Daily Lab / POC Activities

Extramural:

• Discovered that the connectivity issues were due to a firewall configuration in the ADSL Router. The firewall was configured to drop packets in the 50000-51000 range.

• The ADSL router also has a limitation when it comes to dealing with dynamic ports. The router engineer was able to find a workaround that allowed dynamic ports.

The problem was uncovered by running the following tests:

Connectivity Test From Tarrant to Atlanta:
Start the server
/usr/local/globus-4.0.5/sbin/globus-gridftp-server -s

Server listening at node.hostname.com:50000

TEST THE PORT FROM TARRANT
bubba@gump:~> telnet node.hostname.com 50000
Trying yyy.yyy.yyy.yyy...
telnet: connect to address nnn.nnn.nnn.nnn: Connection refused
bubba@gump:~>

NOW WE SHUTDOWN THE SERVER AND TRY ANOTHER PORT
[5802] Mon Feb 25 13:42:37 2008 :: Server is shutting down...
LLAB-INF-1002:/etc/xinetd.d #

NOW WE START THE SERVER ON 2811
LLAB-INF-1002:/etc/xinetd.d # /usr/local/globus-4.0.5/sbin/globus-gridftp-server -s -p 2811

TEST FROM TARRANT
bubba@gump:~> telnet node.hostname.com 2811
Trying yyy.yyy.yyy.yyy...
Connected to node.hostname.com.
Escape character is '^]'.
220 node.hostname.com GridFTP Server 2.5 (gcc32dbg, 1182369948-63) ready.

NOW WE USE THE SAME EXACT COMMAND TO START THE SERVER ON PORT 50000

LLAB-INF-1002:/etc/xinetd.d # /usr/local/globus-4.0.5/sbin/globus-gridftp-server -s -p 50000

NOW WE TEST FROM TARRANT
bubba@gump:~> telnet node.hostname.com 50000
Trying yyy.yyy.yyy.yyy...
telnet: connect to address yyy.yyy.yyy.yyy: Connection refused
bubba@gump:~>

THIS CLEARLY PROVES THAT PORT 50000 IS NOT OPEN IN ATLANTA.


On the Atlanta server side we are currently able to run gridftp from Dallas and Tarrant to Atlanta.

On the client side we are able to run gridFTP both ways using: globus-url-copy -dbg file:///tmp/foo gsiftp://node.hostname.com:2811/tmp/got-it (Command works in both directions)

Daily Lab / POC Activities

Extramural:

• Continued to research possible fixes for the Internet connectivity issue between Atlanta and Dallas.
• Moved certificates from Lab Node 1002 to Lab Node 1003 and tested the connectivity between Atlanta and Dallas. (Same result, the file hangs with no errors)
• Currently awaiting new host certificates and user certificates for the new 1003 grid node.

Daily Lab / POC Activities

Extramural:

• Installed Globus Toolkit version 4.0.5 on a second Atlanta lab machine

• Updated install instructions based on lab machine install

• Requested new host and user certificates for the new grid node

• Posted install instructions to the project's FTP site.

caBig

• CDC Concept mapping completed
• caArray model was loaded successfully to the sandbox environment
• Evaluated model in sandbox browser at this link
  

Daily Lab / POC Activities

Extramural:

• Worked on Grid Node installation document

• Re-built Lab second grid node with SUSE Linux Enterprise SP1

• Tested possible fixes between Dallas and Atlanta

Daily Lab / POC Activities

Extramural:

• Configured ntp to synchronize time between grid servers

• Worked with Dallas network engineers to open TCP ports

• Able to perform a third party transfer between Atlanta and Tarrant from Dallas's server.

• Researching the SSL handshake error when transfer is performed from Atlanta.

• Researching Globus NAT issues: Globus documention mentions the software has problems running behind NAT firewalls. ( Defining $GLOBUS_HOSTNAME in the etc/xinetd.d/gsiftp file may solve the issue.
• System connects on a public IP, but returns a private address to the remote server. The remote sessions then sends a "NO ROUTE TO 192.168...." error.
  

• Started install of 2nd Atlanta Grid Node.
• Will try an ATL to ATL transfer on current ATL server with no changes to the config.
  

Daily Lab / POC Activities

Extramural:

• Performed a successful transfer of larger files between Dallas, Tarrant, and Atlanta sites.
• Meeting setup with Tarrant network engineers to resolve port issues.
• ***Issue that will be discussed: A server not allowing incoming connections in the ephemeral port range will have the following restrictions:
  
• While it will be possible to make GridFTP control connections if port 2811/tcp is open, it will not possible to actually get or put files. In other words the globus-url-copy will hang until CTRL+C is pressed.
  

• Added $GLOBUS_TCP_PORT_RANGE and $GLOBUS_SOURCE_PORT_RANGE variables to gridftp clients.

Daily Lab / POC Activities

Extramural:

• Updated /etc/xinetd.d/gsiftp on each server to run on a specific port added:
  

env += GLOBUS_TCP_PORT_RANGE=Removed Number,Removed Number
env += GLOBUS_TCP_SOURCE_RANGE=Removed Number,Removed Number

• Installed new usercert.pem on Tarrant

• Added each location's DN to the /etc/grid-security/grid-mapfile and mapped it back a valid local account.
  

Successful globus-url-copy tests:
Dallas to Tarrant
Tarrant to Dallas
Atlanta to Tarrant (While connected to vpn)
Initiated from Atlanta - Dallas to Tarrant Copy
Initiated from Atlanta - Tarrant to Dallas

Failed globus-url-copy tests:
Atlanta to Dallas
Atlanta to Tarrant (No VPN)
Tarrant to Atlanta
Dallas to Atlanta

• Current Error from failed tests:

debug: response from gsiftp://some.server.net/etc/hosts:
500-Command failed. : globus_gridftp_server_register_write failed.
500-globus_xio: Unable to connect to 127.0.0.1:number removed
500-globus_xio: System error in connect: Connection refused
500-globus_xio: A system call failed: Connection refused
500 End.

debug: fault on connection to gsiftp://some.server.net/etc/hosts: globus_ftp_client: the server responded with an error
debug: operation complete

error: globus_ftp_client: the server responded with an error
500 500-Command failed. : globus_gridftp_server_register_write failed.
500-globus_xio: Unable to connect to 127.0.0.1:number removed
500-globus_xio: System error in connect: Connection refused
500-globus_xio: A system call failed: Connection refused
500 End.

Daily Lab / POC Activities

Extramural:

• Installed usercert.pem in the .globus directory on the Dallas server.
• Successfully performed a gridftp on the Dallas localhost
• Discovered connectivity issues between Dallas and Atlanta. Port 2811 is being blocked on the Atlanta side. The lab engineer is currently working to resolve the issue.
• Discovered connectivity issues between Dallas and Tarrant servers. Contacted Tarrant county engineer about the issue.
• Contacted Laura about Public and Private key mismatch issues on the Tarrant server. Resolution: Will generate new keys on the Tarrant server.
  

Daily Lab / POC Activities

Extramural:

• Added user account entries to /etc/grid-security/grid-mapfile to Tarrant server

Command Used: $GLOBUS_LOCATION/sbin/grid-mapfile-add-entry \
-dn "USER DN info removed" \
-ln user_account_removed

• Added new usercert.pem to gridftp users on Tarrant server

Command Used: cp username_usercert.pem to /home/username_removed/.globus/usercert.pem

• I'm having issues with the current usercerts. The problem is the certs don't match the userkey.pem information and errors when a grid-proxy-init command is issued.

ERROR: Could not verify the signature of the generated proxy certificate
This is likely due to a non-matching user key and cert

Use -debug for further information.

• Created a start script in /etc/init.d for the globus container. Script below:

#!/bin/sh -e
case "$1" in
start)
su - globus /usr/local/globus-4.0.5/start-stop start
;;
stop)
su - globus /usr/local/globus-4.0.5/start-stop stop
;;
restart)
$0 stop
sleep 1
$0 start
;;
*)
printf "Usage: $0 {start|stop|restart}\n" >&2
exit 1
;;
esac
exit 0

• Tested gridftp from Atlanta Lab to the Tarrant server. The test failed with the following error message:

globus_xio: Unable to connect to server.name-removed.net:2811
globus_xio: Operation was canceled
globus_xio: Operation timed out

• Attempted test while connected and disconnected from VPN. I was unable to telnet to 2811 so I assume the port is being blocked by a firewall. I verified the server was listening on port 2811 using the following command:

netstat -an |grep 2811
tcp 0 0 0.0.0.0:2811 0.0.0.0:* LISTEN

Daily Lab / POC Activities

Extramural:

• Successfully completed a globus-url-copy on lab the machine
• Corrected certificate error, details below. Note: Will need to submit a new certificate request
  

Renamed the server to match the name in the host certificate (command used: hostname new-servername.some.com)
to correct the following error:
#####Begin Error Message#######
error: globus_ftp_control: gss_init_sec_context failed
GSS Major Status: Unexpected Gatekeeper or Service Name
globus_gsi_gssapi: Authorization denied: The name of the remote host (hostname-removed.net), and the expected name for the remote host (some-other.hostname-removed.com) do not match. This happens when the name in the host certificate does not match the information obtained

globus-url-copy -vb -dbg gsiftp://host.server,running.on/dev/zero file:///dev/nullfrom DNS and is often a DNS configuration problem.
#######End of Error############################

• Command entered to simulate a large file transfer from a remote host to the local machine:

globus-url-copy -vb -dbg gsiftp://localhost/dev/zero file:///dev/null
Note: This transfer will run until you press ctrl+C

Expected Output in Debug Mode:
##############Start of output############
ken@LLAB-INF-1002:~> globus-url-copy -vb -dbg gsiftp://localhost/dev/zero file:///dev/null
Source: gsiftp://localhost/dev/
Dest: file:///dev/
zero -> null
debug: starting to get gsiftp://localhost/dev/zero
debug: connecting to gsiftp://localhost/dev/zero
debug: response from gsiftp://localhost/dev/zero:
220 host_name_removed GridFTP Server 2.5 (gcc32dbg, 1182369948-63) ready.

debug: authenticating with gsiftp://localhost/dev/zero
debug: response from gsiftp://localhost/dev/zero:
230 User ken logged in.

debug: sending command:
SITE HELP

debug: response from gsiftp://localhost/dev/zero:
214-The following commands are recognized:
ALLO ESTO RNTO APPE DCAU MODE SIZE STRU
TYPE DELE SITE CWD ERET FEAT LIST NLST
MLSD MLST PORT PROT EPRT PWD QUIT REST
STAT SYST MKD RMD CDUP HELP NOOP EPSV
PASV TREV SBUF MDTM CKSM OPTS PASS SPAS
PBSZ SPOR RETR STOR USER RNFR LANG
214 End

debug: sending command:
FEAT

debug: response from gsiftp://localhost/dev/zero:
211-Extensions supported
UTF8
LANG EN
DCAU
PARALLEL
SIZE
MLST Type*;Size*;Modify*;Perm*;Charset;UNIX.mode*;UNIX.owner*;UNIX.group*;Unique*;UNIX.slink*;
ERET
ESTO
SPAS
SPOR
REST STREAM
MDTM
211 End.

debug: sending command:
TYPE I
debug: response from gsiftp://localhost/dev/zero:
200 Type set to I.

debug: sending command:
PBSZ 1048576

debug: response from gsiftp://localhost/dev/zero:
200 PBSZ=1048576

debug: sending command:
PASV

debug: response from gsiftp://localhost/dev/zero:
227 Entering Passive Mode (127,0,0,1,140,144)

debug: sending command:
RETR /dev/zero

debug: response from gsiftp://localhost/dev/zero:
150 Begining transfer.

debug: reading into data buffer 0xb79c6008, maximum length 1048576
debug: data callback, no error, buffer 0xb79c6008, length 1048576, offset=0, eof=false
debug: reading into data buffer 0xb78a3008, maximum length 1048576
debug: data callback, no error, buffer 0xb78a3008, length 1048576, offset=1048576, eof=false
debug: reading into data buffer 0xb79c6008, maximum length 1048576
debug: data callback, no error, buffer 0xb79c6008, length 1048576, offset=2097152, eof=false
debug: reading into data buffer 0xb78a3008, maximum length 1048576
debug: data callback, no error, buffer 0xb78a3008, length 1048576, offset=3145728, eof=false
debug: reading into data buffer 0xb79c6008, maximum length 1048576

Cancelling copy...
debug: aborting current operation
debug: data callback, no error, buffer 0xb79c6008, length 0, offset=4194304, eof=false
debug: operation complete
##############End of Output#####################

• Command entered to simulate a large file transfer from local host to remote host:

globus-url-copy -vb -dbg file:///dev/zero gsiftp://localhost/dev/null
Expected Output in Debug Mode:
Similar to the above Output

• Command entered to simulate a small file transfer from remote host to local host /tmp

Expected Output in Debug Mode:
##########Start of Output#################
ken@LLAB-INF-1002:~> globus-url-copy -dbg gsiftp://localhost/home/ken/gridtesting file:///tmp/ken.grid.copy
debug: starting to get gsiftp://localhost/home/ken/gridtesting
debug: connecting to gsiftp://localhost/home/ken/gridtesting
debug: response from gsiftp://localhost/home/ken/gridtesting:
220 host_name_removed GridFTP Server 2.5 (gcc32dbg, 1182369948-63) ready.

debug: authenticating with gsiftp://localhost/home/ken/gridtesting
debug: response from gsiftp://localhost/home/ken/gridtesting:
230 User ken logged in.

debug: sending command:
SITE HELP

debug: response from gsiftp://localhost/home/ken/gridtesting:
214-The following commands are recognized:
ALLO ESTO RNTO APPE DCAU MODE SIZE STRU
TYPE DELE SITE CWD ERET FEAT LIST NLST
MLSD MLST PORT PROT EPRT PWD QUIT REST
STAT SYST MKD RMD CDUP HELP NOOP EPSV
PASV TREV SBUF MDTM CKSM OPTS PASS SPAS
PBSZ SPOR RETR STOR USER RNFR LANG
214 End

debug: sending command:
FEAT

debug: response from gsiftp://localhost/home/ken/gridtesting:
211-Extensions supported
UTF8
LANG EN
DCAU
PARALLEL
SIZE
MLST Type*;Size*;Modify*;Perm*;Charset;UNIX.mode*;UNIX.owner*;UNIX.group*;Unique*;UNIX.slink*;
ERET
ESTO
SPAS
SPOR
REST STREAM
MDTM
211 End.

debug: sending command:
TYPE I
debug: response from gsiftp://localhost/home/ken/gridtesting:
200 Type set to I.

debug: sending command:
PBSZ 1048576

debug: response from gsiftp://localhost/home/ken/gridtesting:
200 PBSZ=1048576

debug: sending command:
PASV

debug: response from gsiftp://localhost/home/ken/gridtesting:
227 Entering Passive Mode (127,0,0,1,128,199)

debug: sending command:
RETR /home/ken/gridtesting

debug: response from gsiftp://localhost/home/ken/gridtesting:
150 Begining transfer.

debug: reading into data buffer 0xb798b008, maximum length 1048576
debug: data callback, no error, buffer 0xb798b008, length 37, offset=0, eof=true
debug: response from gsiftp://localhost/home/ken/gridtesting:
226 Transfer Complete.

debug: operation complete
##########End of Output######################

• Corrected Mapfile issue by adding a user name to the end of the mapfile and using that same user name to run a globus-url-copy.
  

Example of what it would look like if /etc/grid-security/grid-mapfile did not contain the correct information for a user attempting to run a globus-url-copy. I generated this error by removing the username from the end of the line in the /etc/grid-security/grid-mapfile
Expected Output:
########Start of Output#####################
ken@LLAB-INF-1002:~> globus-url-copy -dbg gsiftp://localhost/home/ken/gridtesting file:///tmp/ken.grid.copy
debug: starting to get gsiftp://localhost/home/ken/gridtesting
debug: connecting to gsiftp://localhost/home/ken/gridtesting
debug: response from gsiftp://localhost/home/ken/gridtesting:
220 hostname_removed GridFTP Server 2.5 (gcc32dbg, 1182369948-63) ready.

debug: authenticating with gsiftp://localhost/home/ken/gridtesting
debug: response from gsiftp://localhost/home/ken/gridtesting:
530-Login incorrect. : globus_gss_assist: Gridmap lookup failure: Could not map /O=info remoed/OU=Globus Toolkit/OU=Globus changed info/OU=your.domain.info/OU=another.domain.whatever/CN=Firstname Lastname
530-
530 End.

debug: fault on connection to gsiftp://localhost/home/ken/gridtesting
debug: data callback, error globus_ftp_client: the server responded with an error, buffer 0xb79a2008, length 0, offset=0, eof=true
debug: operation complete
#############End of Output####################

Daily Lab / POC Activities

Extramural:

• Verified and installed prerequisite software packages on Tarrant server
• Updated globus environmental variables based on Quick Start guide requirements
• Added gridftp to /etc/xinetd.d and reloaded xinetd
• Attempted to start grid-proxy-init, but it needs a valid certificate to function
• Note: user certs in $HOME/.globus are zero length and need to be re-requested
• Created a Globus Container start-stop script in $GLOBUS_LOCATION
• Created a globus-4.0.5 startup in /etc/init.d
• Began verifying prerequisite software packages on Dallas County Server

Daily Lab / POC Activities

Extramural:

• Installed and configured the Cisco Systems VPN Client Version 4.8.00 (0490) for Linux in the lab
• Successfully tested VPN connection to Tarrant County

Daily Lab / POC Activities

Extramural:
Updated the /etc/profile.local to include the following:

• export PATH=/usr/local/apache-ant-1.6.5/bin:$PATH
  export PATH=/usr/java/jdk1.5.0_14/bin:$PATH
  export JAVA_HOME=/usr/java/jdk1.5.0_14
  export GLOBUS_LOCATION=/usr/local/globus-4.0.5
  export PATH=$GLOBUS_LOCATION/sbin:$GLOBUS_LOCATION/bin:$PATH