Overview and benefits of the GEO-IDE UAF Grid Project

From NOAA Environmental Data Management Wiki
Jump to: navigation, search

Welcome to the Unified Access Framework (UAF) wiki page. The UAF project involves NOAA (all NOAA Line Offices are involved!) and NOAA partners in an effort to develop an interoperable dataset integration capability.

The capability is being developing using several de facto standards: netCDF, which provides the abstract data model, software libraries and a persistent binary format; the Climate and Forecast (CF) metadata conventions; the OPeNDAP protocol for web transport of data subsets; THREDDS XML catalogs which provide a distributed topology connecting data suppliers; and an OGC compatibility layer that provides access to the grids through WMS and WCS.

The focus is to provide a NOAA-wide and NOAA partner catalog of CF-compliant datasets (e.g. model outputs, satellite products, High Frequency radar observations, in situ observations, etc.) and to connect the catalog to several popular client tools (e.g. Matlab, LAS, ERDDAP, Google Earth, etc.) to enable direct access and use of the datasets.

For more information about the project, how to get involved, and what datasets are currently available please see the links below...

Description of the problem/issue being addressed:

  • NOAA's data systems are insufficiently integrated. This situation is a reflection of technology and management and decision-making strategies of the past that have tended to fragment data management, rather than to unify it. Lines of funding have traditionally been matched to observing systems -- satellites, ships, profilers, etc.-- and data life cycle points -- measurement, real-time applications, climate analysis, archive, etc. Data management has been considered to be "owned" by the observing system element or the function. Each observing system element has therefore developed individualized approaches to data management, often involving the development of unique (and non-interoperable) data formats and protocols. Real-time data management strategies were devised with little thought to analysis or archive, and so on. Predictably these traditions have hindered the development integrated data management.
  • Communities of interest within data management are most naturally organized by structural type of data. The lines between these communities are drawn from the answers to key data management questions such as, what techniques are appropriate for searching for these data? for transporting (interchanging) these data? for visualizing or analyzing these data? for storing or archiving these data?
  • Communities of interest defined by structural data types provide a natural way to organize data management efforts and specify standards required for interoperability. For example, the kinds of standards, best practices, metadata, and access interfaces required for time-series data collections are similar for atmospheric, oceanic, hydrological, biological, or climate data.
  • Traditional communities of interest defined by pattern of usage will continue to thrive of course, based upon scientific and societal goals. These communities will provide the requirements to an increasingly integrated data management community. For example, weather forecasters will continue to require synoptic access to observations; climate modelers will continue to view the same observations as time series. The role of the data management community will be to find unified solutions that address both of these usage patterns.
  1. Engages data providers, users, and IT folks
  2. Leverages stable, proven solutions (one meaning of "standards")
  3. Has a high probability of demonstrable successes.

Current efforts:

UAF Overview
  • NOAA is focused on a dataset integration capability that leverages several de facto standards: netCDF, which provides the abstract data model, software libraries and a persistent binary format; the Climate and Forecast (CF) metadata conventions; the OPeNDAP protocol for web transport of data subsets; THREDDS XML catalogs which provide a distributed topology connecting data suppliers; and an OGC compatibility layer that provides access to the grids through WMS and WCS.
  • The focus is to provide a NOAA-wide THREDDS catalog of CF-compliant datasets (e.g. model outputs, satellite products, HF radar observations, etc.) and to enable the preceding web services ( DAP, WMS, WCS).
  • A parallel activity to harvest, repair, and extend metadata for the datasets will improve users' ability to discover and make use of these gridded datasets.

What are some of the benefits of this effort:

UAF available via LAS
  • Direct access to a variety of datasets through a single top-level URL
  • The ability to view/explore/analyze a variety of datasets, including gridded and in situ data, in a variety of clients (e.g. LAS, Matlab, ERDDAP, etc.)
  • Standard metadata pushed to several existing discovery mechanisms (e.g. NODC Geoportal, NNMR, data.gov, etc.)
  • The ability to compare different model output products with actual observations and to then use that knowledge to improve the models.

What datasets are currently available:

The table below summarizes the current data holdings available via the UAF catalog. As data is continually added, the list may not contain a complete description of all the datasets available. More detailed searches for UAF data can be done via the NODC UAF Geoportal.

NOAA Line Office Center/Office/Lab Name Description of Datasets Served
OAR Pacific Marine Environmental Laboratory Comprehensive Ocean-Atmosphere Data Set (COADS), World Ocean Atlas, others…
OAR Geophysical Fluid Dynamics Laboratory Intergovernmental Panel on Climate Change (IPCC) CM2.0 Runs 1 - 3
OAR Atlantic Oceanographic and Meteorological Laboratory Caribbean and Gulf of Mexico Operational Data
OAR Office of Climate Observation Observing System Monitoring Centering gridded obsevation count summaries
OAR Earth System Research Lab Climate datasets (COADS, NCEP, etc) and CarbonTracker data
NMFS Pacific Fisheries Environmental Laboratory Satellite datasets and Simple Ocean Data Assimilation (SODA) POP model runs
NESDIS National Geophysical Data Center Aerosol and global Sea Surface Temperature (SST) data
NESDIS Ocean Nomads Navy Forecast Model Run Collections
NESDIS National Ocean Data Center High Resolution SST data
NWS National Data Buoy Center High Frequency (HF) radar data
IOOS Mid-Atlantic Coastal Ocean Observing RA (MARACOOS) NYHOPS, ESPRESSO and COAWST model results
IOOS Pacific Coast Ocean Observing System RA (PacIOOS) ROMS model for Hawaii region
IOOS Great Lakes Observing System (GLOS) Great Lakes Forecast and Nowcast data

What technologies and standards are being utilized:

The following standards and technologies are being utilized in the UAF Grid Project:

Who is doing this:

How to get involved: