2007SPDF
From ViRBO
Contents |
1. August 31 Meeting at GMU
Background: A few months ago Bob McGuire suggested that we meet at GMU to discuss our VxO efforts now that we have completed approximately one year of development. The goal of this meeting to communicate what projects we are working on, where we could use help, and ways in which we can work together.
In attendance:
- GMU: Bob Weigel and Brian Curtis
- SPDF: Bob McGuire, Aaron Roberts, David Han, Tami Kovalick (telecon), Dieter Blitza
- Aerospace: Paul O'Brien and Tim Guild
- Cottage Systems: Jeremy Faden (telecon)
1.1. Telecon call-in instructions
USA Toll Free Number: 877-446-3024 USA Toll Number: +1-203-310-6788 PASSCODE: (See email)
1.2. McGuire 11:00-11:30 Eastern
Overview of how SPDF fits in with the VxO effort. Slides: ppt
Things we would like to see covered and questions we have:
- Recent developments
- New services
- How are orbit data served? Only through CGI call?
- Does SPDF intend to have a unified API to all of their data holding. Or should SPDF be perceived as a collection of many data centers, each with their own API to get data? What is the role of VSPO's "get data" function with regard to this?
- What is the source code status of orbit viewer (great for 3-D orbit viewing)? ViSBARD (great for orbit + 3D views of data + basic 2D plots)? Our Autoplot (great for 2-D interactive plotting, reading arbitrary data files from a URL) program is available from the vxoware sourceforge web page. I would love to see some enterprising Java hacker take all of these codes and create a "mash-up".
- Experience with SPASE. What will be the difference between CDF master files and SPASE master records? Will you maintain two "master" metadata lists, or is the SPASE auto-generated from CDF?
1.3. Weigel 11:30-12:00 Eastern
Overview of our development efforts
1.3.1. Data Summary
1.3.2. VxOWare
An implementation of VxOware is in development for ViRBO: html
Draft abstract for AGU:
The recent Heliophysics Virtual Observatory (VxO) effort has emphasized the development of separate observatories with a low overlap in physical or scientific specialization domains and a high degree of overlap in their software needs. VxOWare is a content and data management system that is intended for use by a VxO or an entity that manages scientific data. In analogy to the VxO concept, in which data and services are united, VxOWare unites software and tools for building an instance of a VxO in the Virtual Observatory network. VxOWare contains such features as system and user administration, interactive visualization tools, user-editable content, version tracking, and an integrated OPeNDAP server for data delivery. The VxO package will be available either as a stand-alone software package or as part of a full operating system distributed as a disk image that may be used in an operating system virtualization environment. Besides a formal VxO, the intended audience of this package includes a group or an instrument team that plans to develop a directory structure for distributing their data, and is interested in the possibility of having it served and having their internal and external documentation managed without having a full-time developer to develop and integrate the software to do this.
1.3.3. Autoplot
Motivation:
- We want to do better than static gif or png images
Spec:
- Enter a URL, get a plot
- Platform independent
- Interactive
- URL can be to CDF, and other formats (netCDF, Excel, etc.)
- Can create images that are publication quality (vector PDF) or ready for the future (SVG).
1.3.4. OPeNDAP/Hyrax
Discussion of our experience
1.3.5. CDF Readers
Discussion of file format issues and Matlab CDF reader experience. Thoughts:
- Vendors do dumb things and create terrible implementations of scientific file format readers
- Recent Matlab releases will use HDF5 as one of their "native" file formats
- HDF5 and netCDF are merging (merged?) html, and differences between HDF5 and CDF link.
- Does HDF5 allow random access to compressed data?
- So much data is time on a uniform time grid. CDF is a complex format - it seems to heavy-duty at times for just simple time series. Is there a CDFlite that is easier to implement and less difficult for vendors to screw up?
1.3.6. General Discussion: Services
In development. Have access to SPIDR data now, will expose our data via FTP, OPeNDAP, and ?. Any advice?
1.3.7. General Discussion: HPSE?
(Heliophysics Software Environment?). In the next year the VxOs will begin to interface their API's. There are so many tools out there that are of general interest to VxOs:
- Autoplot
- XSLT templates of VMO (by Merka)
- Java editor of SPASE group
- CDF JINI software
- Java API to ViRBO
- etc.
The easy solution is to have Aaron create section on http://hpde.gsfc.nasa.gov that has links to "known VxO-related software". Is there a better alternative? Should there be a central place where developers can go to find current information on VxO related software?
1.4. Faden 1:00-1:30 Eastern
- Autoplot technical details
- Hyrax / BES
- CDF issues
I was taking a careful look at another cdf-based webstart application (http://sscweb.gsfc.nasa.gov/skteditor/) yesterday, and I discovered they are now using a webstart mechanism I wasn't aware of, where a service provider (such as the CDF people) can publish libraries. So instead of mimicking what they do in their cdf applications (such as the skteditor), I can simply include the library they publish, and they become responsible for the mechanics of distributing the library. So the good news is it's more trivial for me to use their library in a webstart context. For example, I was able to make my cdf-plotting application work on an intel mac, when I wasn't before. Looking at the jnlp for the library (http://sscweb.gsfc.nasa.gov/skteditor/cdf/cdf.jnlp), you can see that quite a few platforms are supported, such as SunOS on an amd64. The bad news is Linux is not supported presently, it's commented out with the note "doesn't work." Also I'm having some trouble getting it to work on Windows, some opaque problem in webstart. Maybe I'm looking at the wrong place. Regardless, this is a very promising option...
2. Telecon on Friday, September 28th, 2007
2.1. Status updates
- New version of CDF (McGuire) A number of performance improvments, especially regarding reading when file is openend in read only mode.
- Interactions with MatLab (Han) David is working on generalizing the reader for Epoch16 and is adding other improvements.
- David noted that the java docs are available html.
- Nand's Java code (Jeremy and Nand) Nand is essentially done. Jeremy has not been in contact with him since last meeting because he was working on other commitments in the interim.
- Still open question on jnlps (Jeremy)
- OPeNDAP status and notes on easy Hyrax installation
2.2. Other
- Scheduling the next telecon and/or visit. Set for October 26th at noon.
- Reconfirm that continued SPDF support of OPeNDAP access and a migration to Hyrax will be highly valuable to ViRBO. SPDF OPeNDAP support is not priority - what is important is that we minimize the number of web services code we need to write to access SPDF data
- What else should SPDF know or pay special attention to wrt ViRBO architecture and immediate issues beyond the materials posted on http://virbo.org (Weigel, Jeremy). Continued support for Java CDF reader and continued advice on the best way to interface with SPDF data holdings.
- A better dialogue in how ViRBO sees SPDF services enabling ViRBO. What is important is that we minimize the number of web services code we need to write to access SPDF data. See action items for next meeting.
- a more systematic joint list of specific ViRBO requests
- they'd like to get >200 days of data in a request and/or create an ability to run "batch" jobs. Yes
- orbit data in standard cdf directory tree. Not desired based on our new understanding that we will need to access SPDF data with only two web services interfaces.
- Contiued dialog about availability of source code. Weigel always wants to encourage this ...
- Also an issue that we have so much updated metadata in our master CDFs that simple file download via FTP or OPeNDAP won't pick up
3. Telecon on Friday, October 26th, 2007 at noon Eastern Time
3.1. Bob Weigel
Summarize study of CDAS web services.
3.2. Bob McGuire
Report on possibility of having CDAS and SSCWeb available from one web service.
SPDF will specifically focus to a "createCDF" like view of SSCweb parameters from within CDAS. We will also focus to an enhanced capability (possibly for select users or clients) to run more files and longer time spans (probably as lower priority batch jobs internally within the server) to the createCDF function of CDAS (including presumably SSCWeb connection if/when we do that).
3.3. Jeremy (with David, Nand, Bernie inputs)
Bernie delivered a reference to the cdf/jnlp support page. Jeremy will try this, and provide feedback if problem persists. (Note since the meeting I've discovered I was not using Bernie's latest version link.)
Nand will deliver cdf-reading java source code, Jeremy will review and try it out, working interactively with Nand to mature the code.
4. Telecon on Friday, November 16th, 2007 at 2:00 Eastern Time
4.1. Weigel Summary of CDAS web service
- Weigel was asked about ViRBOs plans for access to SPDF data holdings. He will report on his findings and progress. Previously we had planned on using local mirrors because the of the access speed to CDAWeb holdings was too slow. With the new server at CDAWeb, we are starting to re-think this. In the short term we may use a hybrid approach. A discussion ensued about how the CDAWeb server-side software works and there was a discussion about mirror sites.
4.2. Discussion and Demo of Autoplot
- Brief summary of Autoplot and its relationship to ViRBO. As discussed in previous telecons, we are working on a plotting tool that renders data from many file formats in the same way that your broswer renders text and image files in many formats.
- Demo of Autoplot. Start Autoplot. A web page with draft user's manual is located at autoplot.org.
- Discussion of ideas for collaboration/code sharing/code re-using.
4.3. Follow-up discussion
McGuire:
Overnight I've realized what you were trying to suggest, namely that distribute the CDAWeb system (core IDL engine and the webservices specifically) to multiple sites with distinct data holdings, so then e.g. the ViRBO interface or Aaron's VSPO invoke could use a common calling process to retrieve not only data but also higher-level services such as plotting or listing directly (as URLs for the results to supply directly to users or to be used e.g. to define an input CDF for autoplot) from those distributed systems (their local server CPUs, their local data stores). I think this is very interesting and potentially promising concept, assuming I've got it right now. We don't now distribute the webservices to our "mirror" sites and the programmers continue to be concerned these webservices were implemented in the older RPC standard, which they'd like to rewrite but could be a large job. So I have some work here internally to sort this out and advise you back in what sensible from our perspective. Presumably we aim for a test installation of the s/w at GMU with webservices to prove out and test the concept before you go out to your investigator sites.
Weigel:
Indeed, this is what I was getting at. And I think it is a big job. My line of reasoning is that we (ViRBO) are developing a web interface software for a generic virtual observatory (VxOware). It has lots of modern features like a wiki, a commenting system, the ability for a user to edit metadata, and other buzzwords. We have included software that allows one to hook into a number of data bases (SPIDR, etc). Our thought is that eventually we would like someone who has an ISTP directory of their own cdf files (with associated SPASE records) to be able to install VxOware and start serving data. At present, they can serve data from a ftp drive, from OPeNDAP, or from Autoplot. Now it would make sense for us to include more advanced web services software for such a user. We currently have software that can do part of the job of what CDAWeb does. Your group has have spent years optimizing and improving your codebase. I don't think we will be successful in matching abilities. This was my reason for asking if you had stand-alone versions of some of your server-side code. The other motivation is that there are currently many people thinking about web services. If your codebase is not out there for others to see (and, of course, complain about), N groups will develop N incompatible approaches to the same problem. If your codebase is out there, the N groups will probably develop fewer approaches that are more compatible as they will have been influenced by your code. I doubt anyone will use your approach as-is, for the same reasons your programmers want to improve it. Let's pick this discussion up again in January. I don't expect we will have a pre-packaged version of VxOware until next year. At AGU our main "releases" will be Autoplot and ViRBO. Wrapping up the codebase that ViRBO uses (VxOware) will take some time.
