Today I managed to finish sprucing up the poicondai demo with better logging, and better handling of the new zipcode lists. I also started my research of the JQuery show/hide toggle so I could show/hide the raw zip-code list unless data was to be specified.
Tomorrow I hope to complete the show/hide, and get a prelim set of geographical outlays in poicondai with count bubbles. It will look a lot like rodsadai.
Otherwise, today we had a cool meeting on security, and we are going to doing a lot of data and application classifications and start laying out all of the different standards and policies... and just looking upon our data with more of a "security" eye. Thus, I am meeting a lot of new security focused folks who ask the important, retrospectively obvious questions about what services are present and what sorts of fields are used and whether they might be dangerous when boxed up and shipped somewhere to someone who hasn't seen them before.
Tuesday, October 21, 2008
DiSTRIBuTE's Aggregate Data Model
John Stinn reminded me of the DiSTRIBuTE's Aggregate Data Model. This summer, the PHGrid team met with Ross Lazarus and the BioSense epis about the potential structure of an Aggregate Minimum Data Set (written about earlier on this blog).
The International Society for Disease Surveillance has a project called DiSTRIBuTE (aside to future project namers- name your project something that is easily googlable) that seeks to collect aggregate data on ILI. DiSTRIBuTE uses a minimal aggregate data structure of:
| date | zip3 | age group | fever count | denominator |
This is similar to that proposed by Dr. Lazarus for use at Harvard ESP.
The International Society for Disease Surveillance has a project called DiSTRIBuTE (aside to future project namers- name your project something that is easily googlable) that seeks to collect aggregate data on ILI. DiSTRIBuTE uses a minimal aggregate data structure of:
| date | zip3 | age group | fever count | denominator |
This is similar to that proposed by Dr. Lazarus for use at Harvard ESP.
Tracking Changes
The phgrid sourceforge project is now configured to use sourceforge tracking to track changes to the PHGrid projects. So far, there are two active trackers: Feature Requests and Bugs.
The group has been very active in developing out services and demo apps, now that activity will be even more transparent by tracking the changes as they are submitted, prioritized, assigned, developed and tested.
The sourceforge tracker has rather obvious limitations (fixed workflow, no assigned tester field, on and on) but it's free. Any suggestions are appreciated.
This can also be used by the community to suggest new features, services or bugfixes.
To support this tracking, I'm recapping our development workflow below (which is available as a pretty graphic on the wiki):
The group has been very active in developing out services and demo apps, now that activity will be even more transparent by tracking the changes as they are submitted, prioritized, assigned, developed and tested.
The sourceforge tracker has rather obvious limitations (fixed workflow, no assigned tester field, on and on) but it's free. Any suggestions are appreciated.
This can also be used by the community to suggest new features, services or bugfixes.
To support this tracking, I'm recapping our development workflow below (which is available as a pretty graphic on the wiki):
- User Enter Feature/Bug with description (with or without use case)
- Admin Prioritize/Assign change
- Developer creates (or updates if use case exists) and posts use case, use case is reviewed
- Developer codes change in new branch with automated unit testing (JUnit, etc)
- Developer assigns change to tester (other than developer)
- Tester reviews code and either approves (go to #7) or notes required changes (go to #3)
- Developer committs change to trunk/release
- Admin closes Feature/Bug
Friday, October 17, 2008
Change, Test, Repeat
Today I have a wee bit more functionality in the advanced search, and I am getting some good leads on how to get and modify GIS polygonal data. It's neat stuff!
But, something else was brought to my attention today: The plans to get a tracker system behind our changes. This is a very good and cool thing because it means that the people interested in our projects will have a well documented and leading practice way to suggest features and report bugs. Also, it means we'll start being a bit more formal about what we are working on, and invite other developers and users to test our stuff, making it that much more robust and usable.
This means it's becoming more real.
But, something else was brought to my attention today: The plans to get a tracker system behind our changes. This is a very good and cool thing because it means that the people interested in our projects will have a well documented and leading practice way to suggest features and report bugs. Also, it means we'll start being a bit more formal about what we are working on, and invite other developers and users to test our stuff, making it that much more robust and usable.
This means it's becoming more real.
Thursday, October 16, 2008
Troubleshooting Northrop Grumman Node
Troubleshooting the Northrop Grumman Node. I will be on site at 10am working with Marcelo on solving this issue.
530-globus_xio_gssapi_ftp.c:globus_l_xio_gssapi_ftp_decode_adat:846:
530-Authentication Error
530-GSS Major Status: Authentication Failed
530-accept_sec_context.c:gss_accept_sec_context:183:
530-SSLv3 handshake problems
530-globus_i_gsi_gss_utils.c:globus_i_gsi_gss_handshake:890:
530-Unable to verify remote side's credentials
530-globus_i_gsi_gss_utils.c:globus_i_gsi_gss_handshake:863:
530-SSLv3 handshake problems: Couldn't do ssl handshake
530-OpenSSL Error: s3_srvr.c:2010: in library: SSL routines, function SSL3_GET_CLIENT_CERTIFICATE: no certificate returned
530-globus_gsi_callback.c:globus_gsi_callback_handshake_callback:531:
530-Could not verify credential
530-globus_gsi_callback.c:globus_i_gsi_callback_cred_verify:681:
530-Can't get the local trusted CA certificate: Cannot find issuer certificate for local credential with subject: /O=HealthGrid/OU=Globus Toolkit/OUxxxx/OU=xxxxx/CN=xxxx
530 End.
530-globus_xio_gssapi_ftp.c:globus_l_xio_gssapi_ftp_decode_adat:846:
530-Authentication Error
530-GSS Major Status: Authentication Failed
530-accept_sec_context.c:gss_accept_sec_context:183:
530-SSLv3 handshake problems
530-globus_i_gsi_gss_utils.c:globus_i_gsi_gss_handshake:890:
530-Unable to verify remote side's credentials
530-globus_i_gsi_gss_utils.c:globus_i_gsi_gss_handshake:863:
530-SSLv3 handshake problems: Couldn't do ssl handshake
530-OpenSSL Error: s3_srvr.c:2010: in library: SSL routines, function SSL3_GET_CLIENT_CERTIFICATE: no certificate returned
530-globus_gsi_callback.c:globus_gsi_callback_handshake_callback:531:
530-Could not verify credential
530-globus_gsi_callback.c:globus_i_gsi_callback_cred_verify:681:
530-Can't get the local trusted CA certificate: Cannot find issuer certificate for local credential with subject: /O=HealthGrid/OU=Globus Toolkit/OUxxxx/OU=xxxxx/CN=xxxx
530 End.
Taking shapes.
Things are starting to precipitate here in and around the NCPHI Grid Lab. People are meeting and starting to talk about moving from a research stance to a production stance and actually making things to be used and seen by the world. I think it is awesome, and I also think it will be a lot of work and will be a bit tricky.
When I think of what I want when I think of public facing production GRID products... I think of what I would like to turn out, and I keep thinking of my dream/killer application: Something with the availability of OpenOffice and the simple functionality of Google Apps.
Open Office is pretty much available for everything. It is also deliciously packaged for everything. You can get them through any given *NIX package manager as RPM or a debian package files, it can be had for solaris and windows and mac, and it just seems to be there after the double-click.
Google Apps are simple and ridiculously functional. They also get to every computing platform out there. You can get Google Earth for *nix, Windows, mac, and it has just the same sort of "drop and unbox" functionality. The only drawback is they are closed source and don't like redistribution... but you wouldn't know it from all the APIs they post. There is little question to how to use their apps, and if you want to get into some of the really complex functionality you can easily Google (tee hee) how to do it.
I want our apps to be like that.
I want a NCPHI node to be one of those very simple and concise packages that only asks for what it needs (maybe the appropriate X509.11 certificate files) and then just installs. Whether it be a package (with a lot of attached packages) in the package manager of your given *NIX, or the installer on windows or mac. Right now it is a multi-step download-n-build-n-massage process that takes a new person multiple days and a seasoned veteran the better part of 3 hours depending on how the downloads go.
In the meantime, I want the apps that can come with a node to be much like Google apps. Easy to download and include, easy to find, perhaps even a few checkboxes from an admin portal that just lets you include and configure the pieces from the get-go, but otherwise just something you can nab and drop-in and have it Just Work.
Doing this is going to be a bit difficult, a lot of the limitations we are facing have to do with Other People's Code. But, most of that code is open sourced, and can be modified, and we are having an okay time talking with other people and they have been gracious and enthusiastic about working with us, and it might not get to that point due to a bunch of constraints that are unforeseen or just unknown at the moment... but that is my target. It's what I want to work-for.
When I think of what I want when I think of public facing production GRID products... I think of what I would like to turn out, and I keep thinking of my dream/killer application: Something with the availability of OpenOffice and the simple functionality of Google Apps.
Open Office is pretty much available for everything. It is also deliciously packaged for everything. You can get them through any given *NIX package manager as RPM or a debian package files, it can be had for solaris and windows and mac, and it just seems to be there after the double-click.
Google Apps are simple and ridiculously functional. They also get to every computing platform out there. You can get Google Earth for *nix, Windows, mac, and it has just the same sort of "drop and unbox" functionality. The only drawback is they are closed source and don't like redistribution... but you wouldn't know it from all the APIs they post. There is little question to how to use their apps, and if you want to get into some of the really complex functionality you can easily Google (tee hee) how to do it.
I want our apps to be like that.
I want a NCPHI node to be one of those very simple and concise packages that only asks for what it needs (maybe the appropriate X509.11 certificate files) and then just installs. Whether it be a package (with a lot of attached packages) in the package manager of your given *NIX, or the installer on windows or mac. Right now it is a multi-step download-n-build-n-massage process that takes a new person multiple days and a seasoned veteran the better part of 3 hours depending on how the downloads go.
In the meantime, I want the apps that can come with a node to be much like Google apps. Easy to download and include, easy to find, perhaps even a few checkboxes from an admin portal that just lets you include and configure the pieces from the get-go, but otherwise just something you can nab and drop-in and have it Just Work.
Doing this is going to be a bit difficult, a lot of the limitations we are facing have to do with Other People's Code. But, most of that code is open sourced, and can be modified, and we are having an okay time talking with other people and they have been gracious and enthusiastic about working with us, and it might not get to that point due to a bunch of constraints that are unforeseen or just unknown at the moment... but that is my target. It's what I want to work-for.
Wednesday, October 15, 2008
Updates
- Updated the MonaLisa configuration on Dallas, Tarrant, and NCPHI
- Contacted the MonaLisa support about adding PHgrid to the Global configuration. This will show connectivity between the grid nodes.
- Creating an expect script that will support OGSADAI proxy.
- Removed all unessential software from the sandbox node in preparation for the VM appliance.
RODS, poicondai-web, and GIS.
Yesterday I got the filtering completed... today I started answering the question "so how do you get google maps to show polygons for zip codes" and I learned quite a few things.
What I am trying to do will use KML as a last resort. KML generation means you have to stick a KML document at some public URL and tell google maps to find it, and the CDC end of the grid is all sorts of locked down... and as far as I can tell, Google Maps regrettably does not have a "send me a string of KML" function.
But, there are ways to generate polygons and overlays with just the javascript commands, just like there are ways to drop points on a map with javascript commands, it's just a matter of teasing out coordinates for the borders of the polygons, and Jeremy helped me discover the appropriate PostGIS and postgres tools that are supposed to let me get a zip-code/border database... and he has shown me the way to the RODS code that makes the appropriate queries to get the coordinates that are usually sent to KML. I'll just have to make it so they are sent to javascript arrays instead.
It'll be a bit kludgey, and will probably need some caching, but it should work, and then you know the only thing you need to see to deploy the POIConDai web visualization is access to the service and the appropriate GIS database.
Also, tomorrow will probably be spent focusing on just getting the zip code centroids working and teasing the data out of the NPDS service appropriately. But it's nice to have some part of my mind working on how to turn those dots into polygons for the time being.
Otherwise, we had a meeting with the ESRI/DGI-net guys. Their browser is mega-posh.
- Lots of other people have done this before me, as there are all sorts of cool websites (with code access for pay) that have the overlays for zip codes and phone area codes on top of a google map.
- Lots of these sites seem to work by building a KML that google can read.
- RODS is one of these sites, and at least that code is free.
What I am trying to do will use KML as a last resort. KML generation means you have to stick a KML document at some public URL and tell google maps to find it, and the CDC end of the grid is all sorts of locked down... and as far as I can tell, Google Maps regrettably does not have a "send me a string of KML" function.
But, there are ways to generate polygons and overlays with just the javascript commands, just like there are ways to drop points on a map with javascript commands, it's just a matter of teasing out coordinates for the borders of the polygons, and Jeremy helped me discover the appropriate PostGIS and postgres tools that are supposed to let me get a zip-code/border database... and he has shown me the way to the RODS code that makes the appropriate queries to get the coordinates that are usually sent to KML. I'll just have to make it so they are sent to javascript arrays instead.
It'll be a bit kludgey, and will probably need some caching, but it should work, and then you know the only thing you need to see to deploy the POIConDai web visualization is access to the service and the appropriate GIS database.
Also, tomorrow will probably be spent focusing on just getting the zip code centroids working and teasing the data out of the NPDS service appropriately. But it's nice to have some part of my mind working on how to turn those dots into polygons for the time being.
Otherwise, we had a meeting with the ESRI/DGI-net guys. Their browser is mega-posh.
International Science Grid This Week
Issue 96: iSGTW 15 October 2008
Opportunistic storage increases grid job success rate
The DZero high-energy physics experiment at Fermilab, an Open Science Grid user, typically submits 60,000-100,000 jobs per week at 23 sites. The experiment’s application executables make many requests for input data in quick succession. Due to the lack of storage local to the processing sites, up until recently much of DZero’s data had to transfer in real-time over the wide area network, leading to high latencies, job timeouts and job failures.OSG worked with member institutions to allow DZero to use opportunistic storage, that is, idle storage on shared machines, at several sites. This represents the first successful deployment of opportunistic storage on OSG, and opens the door for other OSG Virtual Organizations. With allocations of up to 1 TB at sites where it processes jobs, DZero has increased its job success rate from roughly 30% to upwards of 85%.
Read more
Opportunistic storage increases grid job success rate
The DZero high-energy physics experiment at Fermilab, an Open Science Grid user, typically submits 60,000-100,000 jobs per week at 23 sites. The experiment’s application executables make many requests for input data in quick succession. Due to the lack of storage local to the processing sites, up until recently much of DZero’s data had to transfer in real-time over the wide area network, leading to high latencies, job timeouts and job failures.OSG worked with member institutions to allow DZero to use opportunistic storage, that is, idle storage on shared machines, at several sites. This represents the first successful deployment of opportunistic storage on OSG, and opens the door for other OSG Virtual Organizations. With allocations of up to 1 TB at sites where it processes jobs, DZero has increased its job success rate from roughly 30% to upwards of 85%.
Read more
Tuesday, October 14, 2008
poicondai moves forward
Yesterday and today, I managed to do a few updates to poicondai, and basically have it to the point where it has a drop-down for ClinicalEffect to filter as needed.
Tomorrow, I will be building up the GIS databases to support making polygons, and hopefully I will be able to get polygonal data on a google map sometime tomorrow or Thursday.
Otherwise, we had a meeting with someone who is a bit better at web design than me, so hopefully the demos will be a lot prettier.
And finally, we got the Dallas problem solved, looked like there was a box in need of a restart and some bottlenecks that needed to be sorted out.
Tomorrow, I will be building up the GIS databases to support making polygons, and hopefully I will be able to get polygonal data on a google map sometime tomorrow or Thursday.
Otherwise, we had a meeting with someone who is a bit better at web design than me, so hopefully the demos will be a lot prettier.
And finally, we got the Dallas problem solved, looked like there was a box in need of a restart and some bottlenecks that needed to be sorted out.
Subscribe to:
Posts (Atom)
Posts
