From ViRBO

Welcome to the home page of 2007 ViRBO All-Hands Meeting Wiki

The meeting will be at LASP in Boulder, CO at LASP in room A201. Visitors please sign in and get a badge at the reception desk.

Contents Pre-meeting email New ViRBO email list Pre-meeting assignments Monday, July 23 09:00-10:00 Introduction (Weigel) What is a VxO? State of ViRBO activities How to contribute to ViRBO Developments in the VxO world Data Policy Issues Goals for Fall AGU 2008 and 2009 Efforts 10:00-10:45 ONERA (Bourdarie) pdf 10:45-11:00 Break 11:00-12:00 Features VxOWare (Zhizhin and Kihn) ppt Autoplot with DAS2 (Faden) ppt Brief on eGY VxO activities (Baker) User feedback (All) 12:00-01:00 Lunch 01:00-03:00 Data Sets 03:00-03:15 Break 03:15-04:30 Wrap-up Tuesday, July 24 09:00 Hyrax work 01:30 NCAR Hyrax Meeting Wednesday, July 25

1. Pre-meeting email

The meeting will be at LASP in Boulder, CO at LASP in room A201. Visitors please sign in and get a badge at the reception desk.

• Meeting Agenda: http://virbo.org/wiki/index.php?title=2007AllHands (and listed below)
• Telecon call-in number for U.S. participants is 800-503-2899, passcode = 4926412. Bob's cell phone number is 571-230-3233. Global Crossing support number is 1-800-252-5150.
• Telecon call-in number for France: 0800 902 561. Access code: 4926412.
• VNC Session IP=http://virbo.org:X, password and integer "X" will be given over phone.

1.1. New ViRBO email list

Because the management of junk emails is a difficult task, we will not host our own email list. Instead, we will use Google Groups. In the next few hours, you will receive an invitation to join this email list. Your email address will not be visible to search engines. If you reply to the invitation email, you will be able to post messages to this forum without obtaining a Google account.

The message archive will be visible at http://groups.google.com/group/virbo

1.2. Pre-meeting assignments

The "All-Hands" meeting will last one day. Although there will be other meetings on Tuesday and Wednesday, Monday is the only day that I think everyone needs to attend, so some preparation is needed in order to cover everything in one day. Here are some things that everyone can do to speed things along:

• Read the data list section http://virbo.org/wiki/index.php/Main_Page#Data_List. This is the master list of data that I want to create metadata records for. Much of this data will also be available in CDF files or (eventually) downloadable from a web form on the ViRBO data portal. At the All-Hands meeting we will discuss what data sets should get priority and begin to create metadata records for these data sets. At the meeting, I will also discuss SPASE (http://spase-group.org), which can be thought of as the Space Physics metadata markup format.

• Create an account at http://spidrd.ngdc.noaa.gov/virbo. Upload a document or presentation to http://spidrd.ngdc.noaa.gov/virbo. Also, download one. Write down your thoughts. I would like ViRBO to not only be for data. I would like it to be a central place to go for information about radiation belt research. To do this, we need to populate the ViRBO web page with research content (in addition to data, which will be added soon!).

• Play around on Wikipedia. Edit a page (you do not need an account). Here is a page I recently created: http://en.wikipedia.org/wiki/Solar_Anomalous_and_Magnetospheric_Particle_Explorer. Look at the history of the page. Motivation: the ViRBO data portal (http://spidrd.ngdc.noaa.gov/virbo) will have many wiki-like features, and the ViRBO "notes" wiki (http://virbo.org/wiki) actually uses the same wiki engine as that used by Wikipedia. (Much of the material on the ViRBO "notes" wiki will be eventually migrated to the ViRBO data portal. This is an on-going development.)

• Read the preliminary agenda at http://virbo.org/wiki/index.php/2007AllHands. In the "Requested For Discussion" section, write down things (on the page or send me an email) that you want to make sure are covered or discussed on Monday. This could be anything, for example things that came up at other meetings or questions that came up upon completion of the above assignments.

2. Monday, July 23

• Expected participants: Bob Weigel, Eric Kihn, Misha Zhizhin (Telecon), Dmitry Kokovin, Shri Kanekal, Dan Baker, Josh Rigler, Jeremy Faden, Sebastien Bourdarie (Telecon), Reiner Friedel (Telecon), Dimitris Vassiliadis (Telecon), Paul O'Brien (Telecon), Tim Guild (Telecon).
• Invited by not available: Janet Green, Terry Onsager, Howard Singer
• Invited: Reeves
• Time Zone Conversion html

A few logistics

1. I will be posting PDF and PPT files on the agenda. You may download them and follow along on your screen or, we will also try to broadcast the slides on VNC in case you want to have someone advance the slides for you.
2. The live demos will be run over VNC. Make sure you do not kick others off of the session! Specify a Shared Connection (This is the default on some clients, but not all.)
3. I have asked a few people to keep notes and I will post my notes on this page as well.

2.1. 09:00-10:00 Introduction (Weigel)

Recall that ViRBO has two parts (1) To develop a virtual radiation belt observatory, and (2) To develop the software required for developing such an observatory. To date, much has been done on (2) and you have all been hearing from me recently on getting the data to satisfy (1). I think this meeting is a great opportunity for the developers to be in contact with the scientists to start to discuss features and issues.

The state of our effort with (2) may be summarized as the following. All of the pieces are there, and it is a matter of the integration that must take place. I hope to have all of the pieces of in place by the end of summer so that we can actually start using all of these advanced features. What is the point of these advanced features? Suppose you have a directory of CDF files that adhere to one of a few standards. What we will have developed will allow searching subsetting, and all of these VxO type activities. The scientist just needs to bring the directory and the rest is taken care of.

First, to set the stage for this meeting, let me remind you of what a virtual observatory is. In many cases what we are doing includes aspects of

• Data Restoration
• Creation of an active archive
• The development of a user community

2.1.1. What is a VxO?

• To a Programmer: A VO is a service that unites services and/or multiple data providers, with a "VxO" doing this for community "x."
• It depends on what the meaning of "is" is. [Bill Clinton]
• It depends on what the meaning of "unite" is.

• To a Scientist
  • Zeroth level (we are here)
    • A place to find collections of data for x. "List all known radiation belt data from all known data providers." or "Do a search, get a list URLs to files." or "Do a search and get a zipped file of all the data pointed to by the URLs". This is not a trivial task.
    • User downloads data and writes their own file readers for each file format.
  • Level 1
    • A place to find synthesized collections of data for x. "Give me data from GOES at this time. Give me data file that holds 3-D simulation results at the same time."
    • User downloads a data file and uses a single file reader to get at data from the time interval of interest.
  • Level 2
    • A place to mine synthesized data for x. While watching a presentation by a modeler, a user visits the VxO and asks for the MeV electron flux measured at the GOES satellite on this date. Also, he asks for measurements that would be obtained by flying GOES through the data files from the modeler's simulation.
    • User downloads and plots two-column time series file. User shows plot in the talk immediately following the modeler.

2.1.2. State of ViRBO activities

• Primary effort for this summer and fall is to prepare metadata and put 100% of data into CDF so that it is available from DODS server.
• Our development efforts has been kept as public as possible (through wiki, http://virbo.org/wiki). Code, information, links to data (most of it), and metadata is added to this page as I get it.
• Hosted two meetings in an effort to drive traffic and get people familiar with ViRBO.
• Familiar enough with SPASE and Metadata requirements to start assigning tasks to scientists
• 25% of data in archaic forms converted to CDF
• Tested a number of server technologies. Settled on DODS/THREDDS.
  • To the end use this means this will be a yet another way to access remote data

http://virbo.org/datafile.cdf?Variable?TimeRange

You can do this from a web browser, IDL, Matlab, Perl, DAS2, etc.

• • DAP = data access protocol (Hyrax is an implementation of DAP protocol, DODS is another implementation). THREDDS Is used to for creating inventory of metadata. DODS: Appears to be the predecessor to OPeNDAP. Much of the documentation uses OPeNDAP and DODS interchangeably. http://www.opendap.org: "OPeNDAP provides software which makes local data accessible to remote locations regardless of local storage format. OPeNDAP also provides tools for transforming existing applications into OPeNDAP clients (i.e., enabling them to remotely access OPeNDAP served data)." http://www.unidata.ucar.edu/projects/THREDDS "... middleware to bridge the gap between data providers and data users. The goal is to simplify the discovery and use of scientific data and to allow scientific publications and educational materials to reference scientific data."
• Integrated DODS/THREDDS into VxOware codebase. Started VxOWare Sourceforge project http://sourceforge.net/projects/vxoware.
  • The new word for this is "data spaces": Data Spaces are a new database management technology frontier that deals with the virtualization of heterogeneous data and data sources via a plethora of data access protocols. As Unified Data Stores, Data Spaces also provide solid foundation for knowledge creation, processing, and dissemination; a making them a natural foundation platform for the emerging Data-Web (Semantic Web - Layer 1). See http://virtuoso.openlinksw.com/wiki/main/Main/OdsIndex http://www.openlinksw.com/blog/~kidehen/?id=1030
• Plotting software is 80% feature complete. Metadata reading and interpreting software is in the process of being developed (hold-up is the completion of Metadata records)
• VxOware2LiveCD is in development this summer. Goal is for a developer to boot, inject some of their own data, and evaluate if it is a project they want to build upon or use

2.1.3. How to contribute to ViRBO

In this section, I will show users how to report a problem. Get feedback from people that have used it (this afternoon).

The most important question to keep in mind during this demo is this: What would it take to actually use the features in the ViRBO data portal? What annoyances need to be addressed? What features need to be added?

The "belly-up" issue.

First let me try to convince you that developing a community like this is useful. I am confident that it will be useful based on the GEM radiation belt climatology meeting. Quite often issues came up such as "how will we communicate" how will exchange data?, etc. After a while many people were responding with "it seems like this is something that would be useful for ViRBO to be a central point for. Before I do this, I must remind you about Web 2.0 and why you will eventually come around to realize that this is a better communication mechanism that email+attachments.

2.1.4. Developments in the VxO world

• SPASE

• VSPO

2.1.5. Data Policy Issues

In this section I want to continue a dialog about data policies. In short, we (the VxOs) have promised open access to data and now we may be in the position where we need to retrofit this open access system with something that satisfies data providers that are not bound to the NASA data policy ([1], [2]). Without it, the holdings and data available through a VxO may stagnate if data providers do not feel that they are getting enough out of their contributions.

2.1.6. Goals for Fall AGU

• Our first major public release will not be until the Fall AGU
  • This required much thought and deliberation. This is the main reason we are not following the rule "release early, release often". After the first release we will follow this rule, but we want to avoid a bad first impression, which usually results in someone not returning to a codebase for another year after a first impression. I'll often see a project that looks promising, but is not ready for non-hard-core developers. I usually do not return for another year (or major release).
• DODS/THREDDS/VxOware/Autoplot access to 80% of our data.
• Ability for users to correct "typos" in metadata for such things as powerpoints, software, etc.
• Ability for a user to "inject" data. I am thinking of someone like D. Vassiliadis, who wants to archive in ViRBO his predictions of daily-averaged flux measured by SAMPEX. He should be able to upload the file and metadata and then see it in the catalog.
• VxOWare LiveCD
• Shopping Cart features of saving and sharing both data queries and data views generated by VxOware Autoplot.

2.1.7. 2008 and 2009 Efforts

2.2. 10:00-10:45 ONERA (Bourdarie) pdf

I thought that it would be useful to have Sebastien show us some of the features at ONERA. Many of his efforts parallel what we want to do. Sebastien has agreed to give us access to a number of data sets that he has put much effort into preparing. In the next few days, I would like to work out the details of what we can do to add value to the data sets that we have planned on preparing ourself, but will instead be contributed by Sebastien.

• The ONERA data portal and software suite (Bourdarie)

• During Sebastien's talk, we discussed the COSPAR/ISTP standard (CDF) file format guidelines pdf. O'Brien and Weigel will suggest additions to this based on recent developments.

2.3. 10:45-11:00 Break

2.4. 11:00-12:00 Features

2.4.1. VxOWare (Zhizhin and Kihn) ppt

Eric Kihn and Misha Zhizhin are the lead developers on the (newly named) VxOWare software package. Based on my interaction with him, I have come to appreciate the software infrastructure that they have set up. It is quite amazing how easily things are changed and new features are added. I don't know the code base and dependencies extensively, but I know that when a feature is requested, it usually happens quickly.

2.4.2. Autoplot with DAS2 (Faden) ppt

One of the advanced features that ViRBO will have is on-line plotting. Now, you are all familiar with gif plots at CDAWeb or image preview galleries that exist in many data sites. Using the DAS2 framework developed primarily by Jeremy Faden, we have designed a plotting program called ViRBO Autoplot (VAP). It is essentially DAS2 with UI modifications that have been designed to "automatically" do the right thing when passed a URI (i.e., file://C:/Documents and Settings/Me/MyFile.cdf or http://virbo.org/ImportantData.cdf, or http://virbo.org/ImportantData.dat). VAP has many advanced interactive features that may be familiar if you have used Google Maps or the time series tool at Google Finance [link]. I suspect that many users will want to download and use the data in their own programs. Possibly one of the features that will be most used is the "data dump" feature. Besides exporting a pdf, png, svg image of the plot, you may dump data from VAP to an ASCII file.

video1: http://www.cottagesystems.com/virbo/video/das2FirstLook.html

video2: http://www.cottagesystems.com/virbo/video/das2inputOutput.html

video3: http://www.cottagesystems.com/virbo/video/autoplot.html

2.4.3. Brief on eGY VxO activities (Baker)

2.4.4. User feedback (All)

2.5. 12:00-01:00 Lunch

We had a quick discussion of the GOES SEP list: [3].

2.6. 01:00-03:00 Data Sets

• Summary of data set status (Weigel). We will go over this list: Main_Page#Data_List and add more specifics about the plan and timeline for dealing with each data set.
• States of Data with respect to ViRBO

1. Not listed on wiki - let me know
2. Raw dumps of all known info. We create metadata for raw files and directory and then decide if it is worth ingesting.
3. Converted to daily CDF and available via FTP and eventuall web form. SPASE record for CDF files
4. Value added to 3.

• Discussion of assignments (All)

2.7. 03:00-03:15 Break

2.8. 03:15-04:30 Wrap-up

• Finish introductory material
• Wanted Features Discussion (All)
• Work plan for Tuesday and Wednesday

3. Tuesday, July 24

3.1. 09:00 Hyrax work

Work on Hyrax installation in preparation for meeting (Weigel, Faden)

3.2. 01:30 NCAR Hyrax Meeting

Meet with Patrick West at NCAR about Hyrax (Weigel, Faden, Kihn, Zhizhin (telecon))

4. Wednesday, July 25

Data and/or software development work meetings (TBD based on Monday/Tuesday results)