Standards White Paper
NOAA Data Management Integration Team (DMIT) Standards Whitepaper (2006)
NOTE: The current version of this document is version 4.2, issued 15 February 2006. Some of the recommendations should be considered out of date.
(Downloadable version: Word).
Summary of standards recommendations
Recommendation 1.1 - Discovery-level metadata content standards Full Recommendation
Recommendation 1.2 - Discovery-level keyword lexicon Full Recommendation
Recommendation 1.3 - Discovery-level metadata representation/exchange standard Full Recommendation
Recommendation 1.4 - Catalogue search protocol specification Full Recommendation
Recommendation 1.5 - Comprehensive use-level metadata Full Recommendation
Recommendation 2.1 - File transfer protocols and APIs Full Recommendation
Recommendation 3.1 – Database access methods Full Recommendation
Recommendation 4.1 – Web Services Full Recommendation
Recommendation 5 – Data and product format standards for delivery Full Recommendation
Recommendation 5.5 – XML schemas for NOAA data Full Recommendation
Recommendation 6 – Standards for accuracy and content of geospatial data Full Recommendation
Metadata and keyword/terminology standards
To ensure that maximum value can be obtained from NOAA data and products it is essential that comprehensive metadata and documentation be provided that are sufficient for both specialists and non specialists to be able to understand how and where the data were obtained, to evaluate the quality of the data and to determine if the data or products are applicable to their specific requirements.
Discovery Level Metadata Metadata refers to a wide range of information that describes data. At the highest level, discovery level metadata describe an entire dataset in general terms. As the name implies, this provides information to help a user discover if data of interest exist and where they might be obtained.
- Discovery-level metadata content standards
- Discovery-level keyword lexicon
- Discovery-level metadata representation/exchange standard
- Catalogue search protocol specification
Comprehensive use-level metadata Some existing metadata standards (e.g. FGDC CSDGM and ISO 19115) provide a good starting point for defining comprehensive use-level metadata. However, given the wide range of data and products created and managed by NOAA programs, in their present form these standards are not sufficient to meet NOAA requirements for comprehensive use-level metadata. Therefore, any standard adopted by NOAA must be rigorously evaluated and extensions and additional elements added to describe all of the types of data managed by NOAA (especially biological data).
- Parameter Usage Vocabularies
File transfer protocols and Application Program Interfaces (APIs)
Database access methods
The Web is being increasingly used for automated application to application communications. This activity is being supported by the growing development and use of Web services. Web services are Web-based applications that use open, XML-based standards and transport protocols to exchange data with calling clients. Specifically, a Web service is a software system designed to support interoperable machine-to-machine interaction over a network. It has an interface described in a machine-processable format and other systems interact with the Web service in a manner prescribed by its description using SOAP-messages, typically conveyed using HTTP. Software systems designed as Web services are said to follow a Services-Oriented Architecture (SOA).
As noted in the NOAA GEO-IDE Plan, the vision for GEO-IDE is one of cooperative integration. The goal is to retain existing systems as much as possible while building a software infrastructure that links these systems together. It is proposed to implement the GEO-IDE vision through Web services.
Although a Web services approach to software development has been enthusiastically embraced by the commercial Internet community, it is still in its infancy. Thus, although several standards have been developed, they are continuing to evolve and many holes remain. To successfully build and implement GEO-IDE, it will be necessary for NOAA to develop several standards, building upon the Web services standards that have already be approved by the World Wide Web Consortium.
Standards for Web services are currently divided into two categories: those based on SOAP and those based on REST (Representational State Transfer). SOAP and REST are described in more detail in the paragraphs that follow. In general, standards based on SOAP specify tightly coupled designs similar to remote procedure calls while standards based on REST specify designs that are loosely coupled, similar to navigating hypertext links. Although these alternate approaches are not mutually exclusive (REST is an architectural style while SOAP is a protocol that can be used with many architectures), they do represent different design viewpoints: a navigational style versus a procedural style. Recently, a compromise of sorts has been worked out between the REST and SOAP communities. REST principles have been incorporated into the standards and guidelines for the latest version of SOAP (1.2). Thus, it should be possible to use SOAP and its associated standards to develop systems in either procedural or navigational styles.
Data and product format standards
Different standards are most suitable for different types of data. Furthermore, a variety of software is used by customers to analyze and visualize data. As a consequence, no single format could possibly satisfy the requirements of all users. However, a proliferation of formats is costly and inefficient for both NOAA and its users. To balance the needs for consistency and flexibility, users should be given the option to select (from a short list of standards) the format that best meets their requirements. Note: the following discussion pertains to delivery of information to users (or exchange with other agencies). Standards for archival of information are not discussed in this document.
While most of the document formats listed below are well-known, a few merit an introduction. ANSI is listed as the basic text format rather than the more limited ASCII. OpenDocument is an OASIS standard XML-based file format suitable for office applications. It covers the features required by text, spreadsheets, charts, and graphical documents. It is vendor and application-independent but is not currently supported by any Microsoft product.
A brief comparison of formats, grouped by their intended use, is given below. Recommendations on which formats NOAA should support are provided at the end of each section.
- Formats for delivery of text and documents
- Formats for delivery of images, charts, graphs, and maps
- Formats for delivery of movies, video and animated image loops
- Formats for delivery of scientific/environmental data