Overview and benefits of the GEO-IDE UAF Grid Project
Welcome to the Unified Access Framework (UAF) wiki page. The UAF project involves NOAA (all NOAA Line Offices are involved!) and NOAA partners in an effort to develop an interoperable dataset integration capability.
The capability is being developing using several de facto standards: netCDF, which provides the abstract data model, software libraries and a persistent binary format; the Climate and Forecast (CF) metadata conventions; the OPeNDAP protocol for web transport of data subsets; THREDDS XML catalogs which provide a distributed topology connecting data suppliers; and an OGC compatibility layer that provides access to the grids through WMS and WCS.
The focus is to provide a NOAA-wide and NOAA partner catalog of CF-compliant datasets (e.g. model outputs, satellite products, High Frequency radar observations, in situ observations, etc.) and to connect the catalog to several popular client tools (e.g. Matlab, LAS, ERDDAP, Google Earth, etc.) to enable direct access and use of the datasets.
For more information about the project, how to get involved, and what datasets are currently available please see the links below...
Description of the problem/issue being addressed:
- NOAA's data systems are insufficiently integrated. This situation is a reflection of technology and management and decision-making strategies of the past that have tended to fragment data management, rather than to unify it. Lines of funding have traditionally been matched to observing systems -- satellites, ships, profilers, etc.-- and data life cycle points -- measurement, real-time applications, climate analysis, archive, etc. Data management has been considered to be "owned" by the observing system element or the function. Each observing system element has therefore developed individualized approaches to data management, often involving the development of unique (and non-interoperable) data formats and protocols. Real-time data management strategies were devised with little thought to analysis or archive, and so on. Predictably these traditions have hindered the development integrated data management.
- Communities of interest within data management are most naturally organized by structural type of data. The lines between these communities are drawn from the answers to key data management questions such as, what techniques are appropriate for searching for these data? for transporting (interchanging) these data? for visualizing or analyzing these data? for storing or archiving these data?
- Communities of interest defined by structural data types provide a natural way to organize data management efforts and specify standards required for interoperability. For example, the kinds of standards, best practices, metadata, and access interfaces required for time-series data collections are similar for atmospheric, oceanic, hydrological, biological, or climate data.
- Traditional communities of interest defined by pattern of usage will continue to thrive of course, based upon scientific and societal goals. These communities will provide the requirements to an increasingly integrated data management community. For example, weather forecasters will continue to require synoptic access to observations; climate modelers will continue to view the same observations as time series. The role of the data management community will be to find unified solutions that address both of these usage patterns.
- The goal of the Unified Access Framework project is to build momentum for implementing NOAA Global Earth Observation -- Integrated Data Environment (GEO-IDE) through a phased approach to data interoperability that:
- Engages data providers, users, and IT folks
- Leverages stable, proven solutions (one meaning of "standards")
- Has a high probability of demonstrable successes.
- NOAA is focused on a dataset integration capability that leverages several de facto standards: netCDF, which provides the abstract data model, software libraries and a persistent binary format; the Climate and Forecast (CF) metadata conventions; the OPeNDAP protocol for web transport of data subsets; THREDDS XML catalogs which provide a distributed topology connecting data suppliers; and an OGC compatibility layer that provides access to the grids through WMS and WCS.
- The focus is to provide a NOAA-wide THREDDS catalog of CF-compliant datasets (e.g. model outputs, satellite products, HF radar observations, etc.) and to enable the preceding web services ( DAP, WMS, WCS).
- A parallel activity to harvest, repair, and extend metadata for the datasets will improve users' ability to discover and make use of these gridded datasets.
What are some of the benefits of this effort:
- Direct access to a variety of datasets through a single top-level URL
- The ability to view/explore/analyze a variety of datasets, including gridded and in situ data, in a variety of clients (e.g. LAS, Matlab, ERDDAP, etc.)
- Standard metadata pushed to several existing discovery mechanisms (e.g. NODC Geoportal, NNMR, data.gov, etc.)
- The ability to compare different model output products with actual observations and to then use that knowledge to improve the models.
What datasets are currently available:
The table below summarizes the current data holdings available via the UAF catalog. As data is continually added, the list may not contain a complete description of all the datasets available. More detailed searches for UAF data can be done via the NODC UAF Geoportal.
|NOAA Line Office||Center/Office/Lab Name||Description of Datasets Served|
|OAR||Pacific Marine Environmental Laboratory||Comprehensive Ocean-Atmosphere Data Set (COADS), World Ocean Atlas, others…|
|OAR||Geophysical Fluid Dynamics Laboratory||Intergovernmental Panel on Climate Change (IPCC) CM2.0 Runs 1 - 3|
|OAR||Atlantic Oceanographic and Meteorological Laboratory||Caribbean and Gulf of Mexico Operational Data|
|OAR||Office of Climate Observation||Observing System Monitoring Centering gridded obsevation count summaries|
|OAR||Earth System Research Lab||Climate datasets (COADS, NCEP, etc) and CarbonTracker data|
|NMFS||Pacific Fisheries Environmental Laboratory||Satellite datasets and Simple Ocean Data Assimilation (SODA) POP model runs|
|NESDIS||National Geophysical Data Center||Aerosol and global Sea Surface Temperature (SST) data|
|NESDIS||Ocean Nomads||Navy Forecast Model Run Collections|
|NESDIS||National Ocean Data Center||High Resolution SST data|
|NWS||National Data Buoy Center||High Frequency (HF) radar data|
|IOOS||Mid-Atlantic Coastal Ocean Observing RA (MARACOOS)||NYHOPS, ESPRESSO and COAWST model results|
|IOOS||Pacific Coast Ocean Observing System RA (PacIOOS)||ROMS model for Hawaii region|
|IOOS||Great Lakes Observing System (GLOS)||Great Lakes Forecast and Nowcast data|
What technologies and standards are being utilized:
The following standards and technologies are being utilized in the UAF Grid Project:
- THREDDS Data Server
- Data Access Protocol
- NetCDF - A set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data.
- NetCDF Climate and Forecast Metadata Convention
- Web Map Service
- Web Coverage Service
- ISO 19115
- NcML - The NetCDF Markup Language (NcML) is an XML dialect that allows you to create Common Data Model (CDM) datasets. The purpose of NcML is to allow: 1) metadata to be added, deleted, and changed, 2) variables to be renamed, added, deleted and restructured and 3) data from multiple CDM files to be combined.
Who is doing this:
- The project is being coordinated by NOAA's Data Management Integration Team
- Technical development is being led by UAF Technical Team