So, my next task for the coming months is to learn, tinker-with, and hopefully implement some cool bits of the GAARDS service as made by the folks up at Ohio State and their work with CaBIG and CaGrid. After a few preliminary readings of white papers and discussions with other people who have investigated various security models, I'm going to try and summarize things as I understand them, and invite people to correct my summarizations...
Globus works with X.509 certificates. To save a lot of complicated two-stepping, I'd say the easiest way to think of a certificate is as a licence with a special key embedded in them. Two nodes wanting to talk to each other have to present their certificates order to access services and establish secure communication, and the nodes have to "trust" each others certificates.
The way to get "automatic" trust without having to add keys into individual trust stores would be to have all the certificates issued by a trusted third party like Verisign or Thawte. This is like getting a passport or a drivers license as ID instead of having a business card with your name on it. It is also expensive, and to have to do it for every node on the grid beyond 5-node grids is pretty much unscalable.
Enter Dorian. Dorian is a GAARDS component and is essentially a Grid Service that allows other authentication methods to be used to access the grid. On one hand, it allows for someone to say "people authenticated by [method] at [node] are allowed to access these grid services". Thus, instead of having to have a certificate, one might just need to enter a username and password, or use their operating system credentials, or use a certificate issued by the Node itself instead of a larger third party.
The other critical component is Grid Trust Services (GTS) which allows for grids with different certificate sets to talk to each other and delegate which services on each grid are available to others. It also performs important syncing functions so that updates to access and authentication chains are propagated through the different grids.
There are other bits too, like GridGrouper which allows for simpler group paradigms (members of the group 'Gridviewer' would be able to access various gridviewers on different nodes... ) and Web Service Single Sign On which would allow an easy port for web applications to gain access to grid services... and you can read about it at the GAARDS website
Either way, I am at the periphery of understanding right now. I hope within a couple of days to have a really good grip on how security works now (and it's limitations) use cases for what we need, and a stronger correlation to how GAARDS will answer those use cases and which components are needed to do it.
Then, over the next couple of months, I'll need to implement those pieces and see what service modifications are needed to use them.
Showing posts with label GBC PoC. Show all posts
Showing posts with label GBC PoC. Show all posts
Thursday, October 8, 2009
Wednesday, October 7, 2009
GIPSEPoison service move, and GIPSE Service Installers
The GIPSEPoison Service has now moved over to the google code repository (you can check out a read-only copy from http://phgrid.googlecode.com/svn/GIPSEPoisonService/trunk).
But, one of the things that I have been doing has been making an ant-fueled bundle that will install both the GIPSEPoisonService and the GIPSEService with a single ant command. You can read about that here: http://sites.google.com/site/phgrid/Home/service-registry/gipseserviceinstaller.
It essentially uses a properties file to download code from a repository, deploys other service-specific properties files to the downloaded code, and then calls the downloaded code's build and deploy scripts.
The idea is that for future service deploys, I can just email Dan a zip with the appropriate properties files and say "unzip this as [user] and then run 'ant all'". My next hope is to try and see if I can make a mvn based download script so that I can make similar installers for things like GIPSEService. The other cool thing is that both ant and mvn should be easily callable by the NSIS installer that Dan found.
But, one of the things that I have been doing has been making an ant-fueled bundle that will install both the GIPSEPoisonService and the GIPSEService with a single ant command. You can read about that here: http://sites.google.com/site/phgrid/Home/service-registry/gipseserviceinstaller.
It essentially uses a properties file to download code from a repository, deploys other service-specific properties files to the downloaded code, and then calls the downloaded code's build and deploy scripts.
The idea is that for future service deploys, I can just email Dan a zip with the appropriate properties files and say "unzip this as [user] and then run 'ant all'". My next hope is to try and see if I can make a mvn based download script so that I can make similar installers for things like GIPSEService. The other cool thing is that both ant and mvn should be easily callable by the NSIS installer that Dan found.
Tuesday, September 29, 2009
GIPSEPoison supporting zipcodes, now onto easier installers
So, after a lot of planning, I finally managed to get general zip5 and zip3 support working in the GIPSEPoison service. I got it so that the service would only ever have to make two calls to the NPDS-poison service (one for states, one for zips) and that it could return zip3 and zip5 and state results simultaneously, and checked in the code after testing it.
The next thing I am planning to work on is some ant scripting for building and deploying the service in one simple command. The hope is that it will turn the install process for this service into a simple command thus turning about ten steps into about a two step process (one if you are repeating the process or just installing a patch).
Otherwise, it seems the farther future involves the generation of more services, more intricate installers, portals and other service/viewer aggregators. My hope is to just start creating things that make fetching and installing these items as easy as possible.
Cheers
The next thing I am planning to work on is some ant scripting for building and deploying the service in one simple command. The hope is that it will turn the install process for this service into a simple command thus turning about ten steps into about a two step process (one if you are repeating the process or just installing a patch).
Otherwise, it seems the farther future involves the generation of more services, more intricate installers, portals and other service/viewer aggregators. My hope is to just start creating things that make fetching and installing these items as easy as possible.
Cheers
Wednesday, September 23, 2009
Many SDN deploys and GIPSEPoison tweaks
So, I finally got the "All Clinical Effects" element working in GIPSEPison... which effectively, if selected, tells the poison service to return all the human call volume for a given region instead of a particular clinical effect.
The next step is to get zip5 and zip3 working. I plan to do it the same way I did it in quicksilver: ask for all the counts of a state and then filter out the zipcodes I don't need. If zip5 is selected, just copy into the observations, but if zip3 is selected, bucket aggregate the zip5s. Ironically, having the whole state return grouped by zipcode is a lot easier than trying to list a long set of zip5s for an entire state (as would be the case with most gridviewer zip3 searches).
I also made a little zip3 to state properties file... in the hopes of not needing the geolocation database (and saving installation steps).
Otherwise, I have also been helping a lot with deploys and scans for various targets... which has caused lots of discussions about ways of automating things so that less time is taken up by these things which are going to be more frequent. The hope is that we can just tell someone about a build label and have someone hit "go" with that variable set and whatever needs to be deployed will get deployed automagically.
Either way, that is not the only improvement being discussed. I get the feeling that soon next week a whole lot of cool new services and visualizations are going to start getting hammered onto and into various design tables.
Cheers,
Peter
The next step is to get zip5 and zip3 working. I plan to do it the same way I did it in quicksilver: ask for all the counts of a state and then filter out the zipcodes I don't need. If zip5 is selected, just copy into the observations, but if zip3 is selected, bucket aggregate the zip5s. Ironically, having the whole state return grouped by zipcode is a lot easier than trying to list a long set of zip5s for an entire state (as would be the case with most gridviewer zip3 searches).
I also made a little zip3 to state properties file... in the hopes of not needing the geolocation database (and saving installation steps).
Otherwise, I have also been helping a lot with deploys and scans for various targets... which has caused lots of discussions about ways of automating things so that less time is taken up by these things which are going to be more frequent. The hope is that we can just tell someone about a build label and have someone hit "go" with that variable set and whatever needs to be deployed will get deployed automagically.
Either way, that is not the only improvement being discussed. I get the feeling that soon next week a whole lot of cool new services and visualizations are going to start getting hammered onto and into various design tables.
Cheers,
Peter
Friday, September 11, 2009
Got GIPSEPoison working on the training node.
So, after many months of on-again, off-again trying... I finally managed to figure out the library conflict on the training node that was preventing GIPSEPoison from loading up on the training node.
The library ended up being WSDL4J. The CXF libraries being used by Jeremy Espino's awesome NPDS Client needed a newer version of wsdl4j than the one that was provided with Globus. Luckily, the newer version seems backwards compatible.
Thus, GIPSEPoison can now be seen as an option in http://ncphi.phgrid.net:8080/gridviewer/
If you read some of my earlier posts, you'll know that the first step was to import the libraries separately (rather than as a part of a Jar-with-dependencies). Otherwise, the error seemed to indicate a WSDL or some other parsing error, and WSDL4J was the first duplicate jar I tried.
The next couple of days are going to be spent fixing up GIPSE Poison. Giving it an "all clinical effects" option, and zip3 and zip5.
Cheers,
Peter
The library ended up being WSDL4J. The CXF libraries being used by Jeremy Espino's awesome NPDS Client needed a newer version of wsdl4j than the one that was provided with Globus. Luckily, the newer version seems backwards compatible.
Thus, GIPSEPoison can now be seen as an option in http://ncphi.phgrid.net:8080/gridviewer/
If you read some of my earlier posts, you'll know that the first step was to import the libraries separately (rather than as a part of a Jar-with-dependencies). Otherwise, the error seemed to indicate a WSDL or some other parsing error, and WSDL4J was the first duplicate jar I tried.
The next couple of days are going to be spent fixing up GIPSE Poison. Giving it an "all clinical effects" option, and zip3 and zip5.
Cheers,
Peter
Tuesday, September 8, 2009
PHIN Conference Thoughts.
Greetings all,
It has been a week since the PHIN conference started with a long weekend in-between, and today I wanted to list out the impressions I got from the crowd and the panels and the discussions I had there... Then I hope to expand a bit with my notes and more thoughts in future blog entries.
- Impression the first(e): Where are the users?
At least two of the panels I attended could be summarized as "this/those informatician(s) used that/those cool grid technolog(y/ies) and find it helpful in doing there work, now if we could only get more of those technologies and work on making them talk to each other". The panels showed me several things. The most important is that informaticians, doctors, and users in general knew that there was stuff out there and they were using it to make their lives easier (IE, "Peter feels validated"). The next coolest thing we learned is that the users had very good opinions and suggestions about how the products could be made to better suit their needs... and the final point I realized is that while many of the fine-grained steps in the process were different (each health department has a different end data format and a different way of describing what they consider to be 'flu'), a lot of the general needs were the same (speed, ease of use, customizability, ease of interaction with other services). I feel that this shows the need to find as many of the users as possible and to try and create a very good space for them to have their opinions and needs voiced, and hopefully help each other install and adjust GRID-like products to suit those needs.
The other user impression I got is that PHGrid, as users of things like Globus and CAGrid (Introduce), are getting a lot of things needed from those programmers and communities. Globus is coming out with cool new things that solve a lot of the old problems, CAGrid is improving Introduce to match the new version(s) of Globus and answering the "how do we create a grid without having to buy lots of expensive third party certificates and how do we simplify registration/addition of new nodes" problems. CAGrid is also looking into ways to remote-deploy services to user boxes so they can run analysis or other functions on data that cannot leave their organization.
- Impression the second(e): We should do the things to make it easy for users even if it makes our lives more difficult.
Several of the panels included the phrase "at our local health department... we tend to see flu like this... but next door, they see flu like this...". In short: "Two health departments, three classifications". In addition, local health departments are wary of attempts to take large chunks of their data so that someone else can re-classify the data. But, they are fine with setting up a service that only gives summary data (no patient info, just counts). Furthermore, they don't have much trouble going "when the service asks for an aggregate count of flu, we'll give them an aggregate count of what we think flu is". Furthermore, they like the idea of having services that allow them to re-organize or re-classify data so it matches a standard, so long as they don't have to email large files or mail DVDs to some place outside of their control.
This means, the biggest impediment to using these types of services is probably going to be the major learning curve required to install all the various toolkits (globus, CABig, tomcat, certificates) and the various configurational complexities to all the applications. The more time we spend making an installer, the less time we have to spend on the phone with interested health departments telling them how to set it up. That also goes for how easy we make it for health departments to write the queries into their own database(s) or datasets. Finally, we need to focus on allowing for customizable outputs from the service viewers we create. If someone can go to GridViewer and get a sample of data to make sure that the CSV we spit out will be able to be read by their analytics... that will help save them some time in a word-processor.
Thus, installers, view panes, things that centralize and simplify configuration and installation with the obvious stuff up front and the complicated stuff defaulted but in an obvious place for modification. Think firefox. Think google.
- Impression the Third(e): Everyone is really happy that all the stuff is relatively open, free (as in speech), and everyone is acquainted with each other and thinking of ways to collaborate and suggest improvements.
If we make a service builder, we will probably be extending CAGrid's introduce. Dr. Jeremy Espino is tweaking globus service projects with Ivy so one doesn't have to upload various libraries into a repository. Globus is new and probably going to be using CXF instead of Axis which will probably improve everything and make us impromptu beta-testers, and a lot of people seem to be marvelling at Quicksilver and Gridviewer and looking forward to their improvements.
So generally, the PHIN conference helped me focus on what I think we need to be doing for the next year, namely getting ready and making our stuff flexible enough for a lot of users to do a lot of different things to make their statistical and analysis efforts easier.
It has been a week since the PHIN conference started with a long weekend in-between, and today I wanted to list out the impressions I got from the crowd and the panels and the discussions I had there... Then I hope to expand a bit with my notes and more thoughts in future blog entries.
- Impression the first(e): Where are the users?
At least two of the panels I attended could be summarized as "this/those informatician(s) used that/those cool grid technolog(y/ies) and find it helpful in doing there work, now if we could only get more of those technologies and work on making them talk to each other". The panels showed me several things. The most important is that informaticians, doctors, and users in general knew that there was stuff out there and they were using it to make their lives easier (IE, "Peter feels validated"). The next coolest thing we learned is that the users had very good opinions and suggestions about how the products could be made to better suit their needs... and the final point I realized is that while many of the fine-grained steps in the process were different (each health department has a different end data format and a different way of describing what they consider to be 'flu'), a lot of the general needs were the same (speed, ease of use, customizability, ease of interaction with other services). I feel that this shows the need to find as many of the users as possible and to try and create a very good space for them to have their opinions and needs voiced, and hopefully help each other install and adjust GRID-like products to suit those needs.
The other user impression I got is that PHGrid, as users of things like Globus and CAGrid (Introduce), are getting a lot of things needed from those programmers and communities. Globus is coming out with cool new things that solve a lot of the old problems, CAGrid is improving Introduce to match the new version(s) of Globus and answering the "how do we create a grid without having to buy lots of expensive third party certificates and how do we simplify registration/addition of new nodes" problems. CAGrid is also looking into ways to remote-deploy services to user boxes so they can run analysis or other functions on data that cannot leave their organization.
- Impression the second(e): We should do the things to make it easy for users even if it makes our lives more difficult.
Several of the panels included the phrase "at our local health department... we tend to see flu like this... but next door, they see flu like this...". In short: "Two health departments, three classifications". In addition, local health departments are wary of attempts to take large chunks of their data so that someone else can re-classify the data. But, they are fine with setting up a service that only gives summary data (no patient info, just counts). Furthermore, they don't have much trouble going "when the service asks for an aggregate count of flu, we'll give them an aggregate count of what we think flu is". Furthermore, they like the idea of having services that allow them to re-organize or re-classify data so it matches a standard, so long as they don't have to email large files or mail DVDs to some place outside of their control.
This means, the biggest impediment to using these types of services is probably going to be the major learning curve required to install all the various toolkits (globus, CABig, tomcat, certificates) and the various configurational complexities to all the applications. The more time we spend making an installer, the less time we have to spend on the phone with interested health departments telling them how to set it up. That also goes for how easy we make it for health departments to write the queries into their own database(s) or datasets. Finally, we need to focus on allowing for customizable outputs from the service viewers we create. If someone can go to GridViewer and get a sample of data to make sure that the CSV we spit out will be able to be read by their analytics... that will help save them some time in a word-processor.
Thus, installers, view panes, things that centralize and simplify configuration and installation with the obvious stuff up front and the complicated stuff defaulted but in an obvious place for modification. Think firefox. Think google.
- Impression the Third(e): Everyone is really happy that all the stuff is relatively open, free (as in speech), and everyone is acquainted with each other and thinking of ways to collaborate and suggest improvements.
If we make a service builder, we will probably be extending CAGrid's introduce. Dr. Jeremy Espino is tweaking globus service projects with Ivy so one doesn't have to upload various libraries into a repository. Globus is new and probably going to be using CXF instead of Axis which will probably improve everything and make us impromptu beta-testers, and a lot of people seem to be marvelling at Quicksilver and Gridviewer and looking forward to their improvements.
So generally, the PHIN conference helped me focus on what I think we need to be doing for the next year, namely getting ready and making our stuff flexible enough for a lot of users to do a lot of different things to make their statistical and analysis efforts easier.
Sunday, August 16, 2009
Prepping for Monday.
Greetings!
Over the past couple of weeks I have been dealing with two fronts: On one front, GridViewer is being modified heavily with lots of cool new javascript and AJAX type functionality, so I have been making simple data-fetching interfaces as needed to support said functionality. On the other front, I have found out that GIPSEPoison works everywhere but the training node and have been investigating why.
There are now two new methods in the new gridviewer class PHMapper and a bunch of new classes, code, and even some new bits in gmap-polygon which will help data be pulled relatively easily. I'm basically mapping out better ways to associate and pull data (things like keeping the regional data separate from the time-series data so that one can be adjusted without messing with the other) but maintaining all the old structures to keep the old grid-viewer working as an example. Thus there has been some function splitting, but luckily I have managed to avoid duplicating lots of code.
Meanwhile, one of the foibles of GIPSEPoison is that it relied on the maven-built NPDS-Client-jar-with-dependencies... The "Jar with dependencies" is something maven does in case you want to run a client from the command line so that you don't have to mess with classpaths or worry about conflicts (shove the jar with dependencies in a directory and hit go). The dangerous temptation is to include it in something that is not maven-related that still needs the code (like an introduce project, more specifically, GIPSEPoison) because it's one jar rather than 20 (NPDS-client uses a lot of CXF and xml/jaxb jars).
Thus, one of the things I hope to complete early next week is to wrangle together all the jars needed separately and include them in the lib of the GIPSEPoison project. Right now GIPSEPoison does not work on the training node... and while it seems like a library conflict it could be something else. If it's a library conflict, having separate jars will be necessary to fix the issue. Otherwise, having separate jars is good form (it allows other potential conflicts to be discovered easier) and we might just run GIPSEPoison off a different globus container.
So, next week should be some adventures in jar hunting and library tweaking. If GIPSEPoison gets sorted out relatively early and the various installs I need to support go well, I hope to clean up some of the packaging of gridviewer (pull out the java code into a 'ServiceUtils class' leaving grid viewer a pure web project) and gmap-polygon (try and see if more of the data can be cleaned up and made simpler to download/sync) and brush up the documentation.
Cheers,
Peter
Over the past couple of weeks I have been dealing with two fronts: On one front, GridViewer is being modified heavily with lots of cool new javascript and AJAX type functionality, so I have been making simple data-fetching interfaces as needed to support said functionality. On the other front, I have found out that GIPSEPoison works everywhere but the training node and have been investigating why.
There are now two new methods in the new gridviewer class PHMapper and a bunch of new classes, code, and even some new bits in gmap-polygon which will help data be pulled relatively easily. I'm basically mapping out better ways to associate and pull data (things like keeping the regional data separate from the time-series data so that one can be adjusted without messing with the other) but maintaining all the old structures to keep the old grid-viewer working as an example. Thus there has been some function splitting, but luckily I have managed to avoid duplicating lots of code.
Meanwhile, one of the foibles of GIPSEPoison is that it relied on the maven-built NPDS-Client-jar-with-dependencies... The "Jar with dependencies" is something maven does in case you want to run a client from the command line so that you don't have to mess with classpaths or worry about conflicts (shove the jar with dependencies in a directory and hit go). The dangerous temptation is to include it in something that is not maven-related that still needs the code (like an introduce project, more specifically, GIPSEPoison) because it's one jar rather than 20 (NPDS-client uses a lot of CXF and xml/jaxb jars).
Thus, one of the things I hope to complete early next week is to wrangle together all the jars needed separately and include them in the lib of the GIPSEPoison project. Right now GIPSEPoison does not work on the training node... and while it seems like a library conflict it could be something else. If it's a library conflict, having separate jars will be necessary to fix the issue. Otherwise, having separate jars is good form (it allows other potential conflicts to be discovered easier) and we might just run GIPSEPoison off a different globus container.
So, next week should be some adventures in jar hunting and library tweaking. If GIPSEPoison gets sorted out relatively early and the various installs I need to support go well, I hope to clean up some of the packaging of gridviewer (pull out the java code into a 'ServiceUtils class' leaving grid viewer a pure web project) and gmap-polygon (try and see if more of the data can be cleaned up and made simpler to download/sync) and brush up the documentation.
Cheers,
Peter
Thursday, July 23, 2009
GIPSEPoison getting closer
Today I spent some time playing with GIPSEPoison trying to see if I could invoke the service... it seems I could but I kept getting caught up in classpath issues... first needing to import Dr. Jeremy Espino's CXF client for accessing the NPDS service... and then talking a bit with Brian about how he set up the build scripts to include data (properties files) from a "resources" directory in the classpath. So tomorrow I have a good list of things to tinker-with that will hopefully successfully punt data from npds through GIPSE and to the grid viewer.
Then I need to clean up a bit of the fun javascript logic around the magical autopopulating drop-downs (there are some strange inital load issues).
Otherwise, I have gone ahead and made a service registry entry for gridviewer to help aid people who want to install and set up their own gridviewers. I have also added some database definitions to the database loader project to help aid in geodata creation and described their use in that entry. Right now one needs to build and install the gmap-polygon project before they are able to build and use gridviewer, and it has pointed out to me that it's an annoying step that could probably be replaced with a no-props version of the gridviewer jar... so I am going to look into doing that soon as well.
Cheers,
Peter
Then I need to clean up a bit of the fun javascript logic around the magical autopopulating drop-downs (there are some strange inital load issues).
Otherwise, I have gone ahead and made a service registry entry for gridviewer to help aid people who want to install and set up their own gridviewers. I have also added some database definitions to the database loader project to help aid in geodata creation and described their use in that entry. Right now one needs to build and install the gmap-polygon project before they are able to build and use gridviewer, and it has pointed out to me that it's an annoying step that could probably be replaced with a no-props version of the gridviewer jar... so I am going to look into doing that soon as well.
Cheers,
Peter
Tuesday, July 21, 2009
Introduce Metadata working.
So, with some help from Dan and Brian, I got GIPSEPoison deployed to my local dev box... and then I pointed grid-viewer to it... and got metadata back in the standard metadata format (which means that I properly imported some of the metadata code Chris already wrote).
Thus, tomorrow will be sorting out the client side of the code and then, hopefully, GIPSEPoison.
Cheers,
Peter
Thus, tomorrow will be sorting out the client side of the code and then, hopefully, GIPSEPoison.
Cheers,
Peter
Introduce Quirks
I am noticing a few strange differences between my GIPSEPoison and the GIPSEServices written by Brian and Chris....
First, when I created my project, the prefix that showed up for the service ([service url]/[prefix]/[ServiceName]) was "cagrid" instead of something like "gipse". Brian and I found that the property "service.deployment.prefix" in the deploy.properties file defines the prefix. Might be handy for anyone else who wants to change their service prefix from cagrid.
Another strange thing I have noticed is that my autogenerated code throws "RemoteExceptions" while Brian and Chris' autogenerated code throw "Exceptions"... I am not sure what quirk caused that or why they are different... Perhaps I got a slightly different version of introduce. But, it all compiles together, and I hope that it ends up working.
First, when I created my project, the prefix that showed up for the service ([service url]/[prefix]/[ServiceName]) was "cagrid" instead of something like "gipse". Brian and I found that the property "service.deployment.prefix" in the deploy.properties file defines the prefix. Might be handy for anyone else who wants to change their service prefix from cagrid.
Another strange thing I have noticed is that my autogenerated code throws "RemoteExceptions" while Brian and Chris' autogenerated code throw "Exceptions"... I am not sure what quirk caused that or why they are different... Perhaps I got a slightly different version of introduce. But, it all compiles together, and I hope that it ends up working.
Monday, July 13, 2009
Client flexibility.
So, one of the little issues that cropped up over the last couple of days is that if you do not really watch how you set up your service in Introduce... there is a very good chance that it will use new packages and namespaces and use an entirely different service in the generated service code.
Right now I am using the client half of the client/server code that was generated by introduce for GIPSE-Biosense. Meanwhile, the client half of the client/server code that was generated by introduce for GIPSE-Poison was different.
Really different. Different packages different.
This means I would have to import both the clients of GIPSE-Poison and GIPSE-Biosense and have logic in my code to differentiate between the servers being called to get data from both... and the idea of having to do that for every service someone wants to generate makes the solution not scalable.
Thus, a big focus over this upcoming week is going to be making a very flexible client. I was chatting with Jeremy about things like Yahoo Pipes and he also just pointed out that the data is XML and I could probably just run things through an XSLT to get data from one point to another... and it was a really good idea. Hopefully, it means that all one would have to do is create an XSLT in case their schema didn't match ours exactly and drop somewhere in the classpath of gridviewer and set a few properties to say "this might need a bit of extra processing" and then gridviewer will be able to display the data.
Also, I am going to be thinking of pinwheels and set-the-URL pushpin styles... but I am not sure if I am going to be able to get to them this week.
Cheers,
- Peter
Right now I am using the client half of the client/server code that was generated by introduce for GIPSE-Biosense. Meanwhile, the client half of the client/server code that was generated by introduce for GIPSE-Poison was different.
Really different. Different packages different.
This means I would have to import both the clients of GIPSE-Poison and GIPSE-Biosense and have logic in my code to differentiate between the servers being called to get data from both... and the idea of having to do that for every service someone wants to generate makes the solution not scalable.
Thus, a big focus over this upcoming week is going to be making a very flexible client. I was chatting with Jeremy about things like Yahoo Pipes and he also just pointed out that the data is XML and I could probably just run things through an XSLT to get data from one point to another... and it was a really good idea. Hopefully, it means that all one would have to do is create an XSLT in case their schema didn't match ours exactly and drop somewhere in the classpath of gridviewer and set a few properties to say "this might need a bit of extra processing" and then gridviewer will be able to display the data.
Also, I am going to be thinking of pinwheels and set-the-URL pushpin styles... but I am not sure if I am going to be able to get to them this week.
Cheers,
- Peter
Thursday, July 9, 2009
Got multiple region searches.
Part of the improvements that will hopefully deploy successfully tomorrow is the ability to have multi-region searches.
So, there will be a new button that says "add to current" which, when pressed, will add data without clearing off the old data. This means you can have all the zip3s in Florida post next to all the zip3s in Georgia in case you want to compare between the two.
Currently, there is no way to differentiate between two searches on the same region (so if you select for all the zip3s in Georgia for one condition, and then add all the zip3s in Georgia for another condition... it will just overwrite the first set)... this is going to be part of a suite of improvements that allow for different sorts of pinpoints (pinwheel) and probably for selecting regions and legends.
But, I think it's neat and I hope you will too (and I actually got it working yesterday but spent a lot of yesterday and today debugging it, as I essentially had to rip apart the gmap-polygon project to get it working (and I branched before I did, thank goodness))
Cheers!
So, there will be a new button that says "add to current" which, when pressed, will add data without clearing off the old data. This means you can have all the zip3s in Florida post next to all the zip3s in Georgia in case you want to compare between the two.
Currently, there is no way to differentiate between two searches on the same region (so if you select for all the zip3s in Georgia for one condition, and then add all the zip3s in Georgia for another condition... it will just overwrite the first set)... this is going to be part of a suite of improvements that allow for different sorts of pinpoints (pinwheel) and probably for selecting regions and legends.
But, I think it's neat and I hope you will too (and I actually got it working yesterday but spent a lot of yesterday and today debugging it, as I essentially had to rip apart the gmap-polygon project to get it working (and I branched before I did, thank goodness))
Cheers!
Tuesday, July 7, 2009
multisets
Gmap Polygon and Grid Viewer are now essentially doing cumulative searches... it's just not displaying them in any particular order... Thus tomorrow will be spent creating some updated pinpoint libraries (pinwheel), modifying the flot-plot screens, adding a "clear map" function... and filtering to make sure that empty data sets aren't added.
Also, there is a distinction for "empty set". If servers are queried and no data comes back, I'll show the pinpoints/polygons as empty. If, however, a search is run without any servers selected.... it will be considered an empty set and not added to the search tree.
Thus, when you first load the page, you'll get a map of the US, but there will be no pinpoints until you actually search for data. This behavior can change, of course, but for now it will help debug the behavior of how data is added.
Then, it's testing to see how it all behaves, and seeing whether a collection of collections will suffice or whether I will have to upgrade it to a map of some sort (a bit more complex to navigate, but probably much better "make sure these are in order" behavior)
Also, there is a distinction for "empty set". If servers are queried and no data comes back, I'll show the pinpoints/polygons as empty. If, however, a search is run without any servers selected.... it will be considered an empty set and not added to the search tree.
Thus, when you first load the page, you'll get a map of the US, but there will be no pinpoints until you actually search for data. This behavior can change, of course, but for now it will help debug the behavior of how data is added.
Then, it's testing to see how it all behaves, and seeing whether a collection of collections will suffice or whether I will have to upgrade it to a map of some sort (a bit more complex to navigate, but probably much better "make sure these are in order" behavior)
Enterprise...
So, I was chatting with a friend yesterday and indicated that I was a bit sad because the next set of changes (modifying GmapPolygon to hold multiple sets of polygons/indicators and writing a new polygon color handler that allows for URL's to be set) will sort of make things ugly and a bit more confusing and threatens to mottle the design patterns even more...
The friend spake (and I paraphrase): "Don't worry about it, it's enterprise code. You go in expecting A... and you end up making A, which eventually needs B after the user sees it and decides it wants B-esque features... but you don't have the time to completely remake it into B, so you make it into a mashup of A and C which works just as good but is not as pretty. So just be happy that A-C is doing good enough and make it into B if you have time before the next deadline." We then went on to discuss how the whole idea that code as a start-do-end thing is more myth than reality... as specifications change and user wishes change and sometimes you change from linux to windows or the tool you need only works with a certain version &c &c.
So yes, The next step is going to be changing the GmapPolygon and the Grid Viewer to work with indexed collections of polygons instead of just one collection of polygons. This will allow for sequential searches (search for all the fever in the US... now search for all the nausea in the US... now search for all the poison-related cellulitis in the US... now do those again for all of Texas...) to be maintained and shown simultaneously on the map until they are explicitly cleared. This means the pinpoint visualizations will need to be changed so that later searches don't show up on top of the older searches (right now I am thinking of using rotating pins... so sequential searches start filling in a pin-wheel). The flot-plotting mechanics are also going to need to change a bit to search on region type, name, and index (thus, it knows to look for the 3rd search or the 2nd search). Eventually the selection mechanisms might also be tweaked to become more intuitive or allow for more specific searches... but right now it's just about getting multiple searches on a map.
I'm sure it will actually not be as ugly or unintuitive as I fear when I'm done, but it is a lot of changes in the name of niftiness,
Cheers!
The friend spake (and I paraphrase): "Don't worry about it, it's enterprise code. You go in expecting A... and you end up making A, which eventually needs B after the user sees it and decides it wants B-esque features... but you don't have the time to completely remake it into B, so you make it into a mashup of A and C which works just as good but is not as pretty. So just be happy that A-C is doing good enough and make it into B if you have time before the next deadline." We then went on to discuss how the whole idea that code as a start-do-end thing is more myth than reality... as specifications change and user wishes change and sometimes you change from linux to windows or the tool you need only works with a certain version &c &c.
So yes, The next step is going to be changing the GmapPolygon and the Grid Viewer to work with indexed collections of polygons instead of just one collection of polygons. This will allow for sequential searches (search for all the fever in the US... now search for all the nausea in the US... now search for all the poison-related cellulitis in the US... now do those again for all of Texas...) to be maintained and shown simultaneously on the map until they are explicitly cleared. This means the pinpoint visualizations will need to be changed so that later searches don't show up on top of the older searches (right now I am thinking of using rotating pins... so sequential searches start filling in a pin-wheel). The flot-plotting mechanics are also going to need to change a bit to search on region type, name, and index (thus, it knows to look for the 3rd search or the 2nd search). Eventually the selection mechanisms might also be tweaked to become more intuitive or allow for more specific searches... but right now it's just about getting multiple searches on a map.
I'm sure it will actually not be as ugly or unintuitive as I fear when I'm done, but it is a lot of changes in the name of niftiness,
Cheers!
Wednesday, July 1, 2009
Enabling and disabling servers.
I spent most of last week ironing out some of the ways that the server disabling and enabling will work.
Thus, as servers get added, if they only have some regions, or (more likely) completely different classifiers and indicators... the drop-downs should change and servers should become enabled or disabled.
Otherwise, tomorrow and next week are going to be a paradigm shift... namely the ability to cumulatively add searches with different indicators, thus being able to see differences in data.
Thus, as servers get added, if they only have some regions, or (more likely) completely different classifiers and indicators... the drop-downs should change and servers should become enabled or disabled.
Otherwise, tomorrow and next week are going to be a paradigm shift... namely the ability to cumulatively add searches with different indicators, thus being able to see differences in data.
Friday, June 26, 2009
Training updated
The training node was updated with the new caching gridviewer and the listed schemas were also updated.
In gridviewer, Dallas has been disabled for the time being because it seems to be acting rather flakey (will load fine for a half hour, and then times out). If anything it means I need to work in a good timer function to abandon a load and notify the user after a set amount of time.
Otherwise, gridviewer next week will be given some UI tweaks. I am hoping to get better server allow/disallow logic, multiple loads on one map (with different pushpins depending on the load), and autopopulated classifier/indicator drop-downs that don't require server hits.
Cheers everyone, have a good weekend,
Peter
In gridviewer, Dallas has been disabled for the time being because it seems to be acting rather flakey (will load fine for a half hour, and then times out). If anything it means I need to work in a good timer function to abandon a load and notify the user after a set amount of time.
Otherwise, gridviewer next week will be given some UI tweaks. I am hoping to get better server allow/disallow logic, multiple loads on one map (with different pushpins depending on the load), and autopopulated classifier/indicator drop-downs that don't require server hits.
Cheers everyone, have a good weekend,
Peter
Thursday, June 25, 2009
Data is caching
Now the data for gridviewer is caching using OS cache. I may have to tweak it a bit to behave better as a singleton, but otherwise it is efficient and simple and seems to make everything quite snappy. Especially now that gridviewer is not referencing servers for metadata in each load.
Otherwise, the server url's are now pulling from the wiki page (or any page you wish to configure yourself). Thus, less database configuration is needed just for gridviewer.
Otherwise, the server url's are now pulling from the wiki page (or any page you wish to configure yourself). Thus, less database configuration is needed just for gridviewer.
Wednesday, June 24, 2009
Data is combining, now to get it caching.
The data is now combining in the data model... and it seems to be doing it correctly.
Now, I am hoping to get a majority of it caching... and instead of trying to write my own caching, I am going to use a caching mechanism suggested by Chris: OSCache from the OpenSymphony project.
It seems to have generic back-end caching with configurable levels of intelligence and persistence (which is all I really needed)... but it also has the most potential for helping on the front end (like request caching: if a request looks like the exact same request that was sent a minute ago, it will return the exact same HTML that was sent back rather than hitting the server) It also allows for better and more fine-grained error handling.
After that, it's some UI clean up, some javascript niftiness, and testing to see if GridViewer will run in the same area as an AMDS service.
Now, I am hoping to get a majority of it caching... and instead of trying to write my own caching, I am going to use a caching mechanism suggested by Chris: OSCache from the OpenSymphony project.
It seems to have generic back-end caching with configurable levels of intelligence and persistence (which is all I really needed)... but it also has the most potential for helping on the front end (like request caching: if a request looks like the exact same request that was sent a minute ago, it will return the exact same HTML that was sent back rather than hitting the server) It also allows for better and more fine-grained error handling.
After that, it's some UI clean up, some javascript niftiness, and testing to see if GridViewer will run in the same area as an AMDS service.
Tuesday, June 23, 2009
New time series handling.
I have adjusted the time series handling of gmap-polygon and grid-viewer so that regional collections hold multiple time series lists (differentiated by server).
Right now, it seems that it is building, but when you open the flot plot, all the data is somehow being tagged with one date, and it is not immediately apparent where or how that data is being set like that. I think I might have run into this problem before and it might have to do with how Java Calendars increments... I hope to figure it out later today or tomorrow.
Otherwise, after that is finished tomorrow, I will implement the new caching structure we discussed with Brian, and start publishing arrays so that the drop-downs for indicators and classifiers can be dynamically populated (as discussed with Chris).
After that, I am going to start allowing for multiple polygon loading... thus you can do one search, then another, and click and compare graphs. I might also look into making histograms in addition to line charts (that way, one can see the different data brought back by the different servers)
Right now, it seems that it is building, but when you open the flot plot, all the data is somehow being tagged with one date, and it is not immediately apparent where or how that data is being set like that. I think I might have run into this problem before and it might have to do with how Java Calendars increments... I hope to figure it out later today or tomorrow.
Otherwise, after that is finished tomorrow, I will implement the new caching structure we discussed with Brian, and start publishing arrays so that the drop-downs for indicators and classifiers can be dynamically populated (as discussed with Chris).
After that, I am going to start allowing for multiple polygon loading... thus you can do one search, then another, and click and compare graphs. I might also look into making histograms in addition to line charts (that way, one can see the different data brought back by the different servers)
Friday, June 19, 2009
more services on gridviewer
There are now some more services on Grid-Viewer, and there are some visual tweaks.
Unfortunately, the data from multiple servers is not combining because of some back-end restrictions that I am going to work on next week, some additional refactors include separating polygons from pinpoints (thus, coastal states won't have really dark shadows (the result of multiple pinpoints being drawn in the same spot)), and storing data from multiple states in the markers (thus, flotplots should be able to distinguish which data came from which service in future versions).
Also, caching (with the help of Brian), fun Javascript and layout changes (with the help of Chris), and new multi-plotting and regional overlays.
Cheers everyone, have a good weekend!
Unfortunately, the data from multiple servers is not combining because of some back-end restrictions that I am going to work on next week, some additional refactors include separating polygons from pinpoints (thus, coastal states won't have really dark shadows (the result of multiple pinpoints being drawn in the same spot)), and storing data from multiple states in the markers (thus, flotplots should be able to distinguish which data came from which service in future versions).
Also, caching (with the help of Brian), fun Javascript and layout changes (with the help of Chris), and new multi-plotting and regional overlays.
Cheers everyone, have a good weekend!
Subscribe to:
Posts (Atom)
Posts
