EUMETSAT utilises a variety of formats for product dissemination and distribution. The most commonly used types are listed below.
High Rate Information Transmission (HRIT) is a standard agreed upon by the members of the Coordination Group for Meteorological Satellites (CGMS) for the dissemination of data originating from geostationary satellites to users via direct broadcast. HRIT data are typically disseminated at speeds up to 10 Mbps.
The general formats are defined in the HRIT Global Specification, whereas individual satellite operators may customise the format in order to fulfil the needs of a specific mission.
EUMETSAT uses HRIT to encode Level 1.5 SEVIRI image data. These data have been corrected to remove all unwanted radiometric and geometric effects, geolocated using a standardised projection, calibrated, and radiance-linearized. Ancillary information is provided in the data’s header and trailer.
The full description of the implementation of HRIT for Meteosat Second Generation (MSG) is provided in the MSG Ground Segment HRIT Mission Specific Implementation. Specific details concerning the image data are described in the MSG Level 1.5 Image Data Format Description.
NetCDF (network Common Data Form) is a machine-independent, self-describing, binary data format standard for exchanging scientific data. It is designed for platform independent, self-describing data storage, with the intent that users can understand the data without the need for external resources and access it efficiently in its entirety or in portions.
The NetCDF format is governed by the Unidata programme at the University Corporation for Atmospheric Research (UCAR). Because the format is very flexible, several conventions have arisen with the goal of ensuring that data can be easily exchanged between different members of the same community. One notable example can be found in the collected the Climate and Forecast (CF) Conventions and Metadata. Where possible, EUMETSAT adheres to these conventions when producing data encoded in NetCDF.
Further information on the use of NetCDF at EUMETSAT can be found on the NetCDF format for Data Centre products web page.
Sentinel data products are distributed using a Sentinel-specific variation of the Standard Archive Format for Europe (SAFE) format specification. All the information relevant to the product is gathered into a single package. Inside this package, the specific objects containing measurement data are encoded in NetCDF format.
Naming convention: The prefix of the product name is the concatenated product type code, e.g., 'SR_2_WAT', and timeliness code, e.g., 'NRT'.
SENTINEL-SAFE is designed to manage any kind of data and data can be contained in a single file or in multiple files.
Using the SENTINEL-SAFE format, information can be categorised in two main categories:
- Data: the information contained in the product which causes the existence of the product itself.
- Metadata: stored information which is not data.
A SENTINEL-SAFE format product contains the following components:
- Manifest file: an XML document conforming to the XFDU manifest file specifications. It contains the definition of the information package map, wrapped metadata objects, wrapped data objects and references to the files containing the metadata and data objects.
- Binary or XML files: the data or metadata object contents. Currently, in the SAFE core specifications, only two types of files have been identified, i.e. binary matching MIME octet stream definition and XML documents.
- XML schema files: the representation information of the data held by a SAFE format product. To represent the binary information, the SAFE format also defines specific mark-ups to annotate XML schema documents, providing information on the physical structure (SDF mark-ups).
The size of an SRAL/MWR product will depend on the type of measurements it contains. The Altimetry L2 product includes three data files — Reduced, Standard and Enhanced — which can be ordered/delivered together ('all') or separately.
The Sentinel-3 Toolbox consists of a rich set of visualisation, analysis and processing tools for the exploitation of OLCI and SLSTR data from the Sentinel-3 mission.
Hierarchical Data Format (HDF) is a format designed for use with large amounts of hierarchically organised data. It is commonly used by producers of large amounts of environmental data, especially when this data is gridded.
The Binary Universal Form for the Representation of meteorological data (BUFR) is the World Meteorological Organization’s (WMO) standard binary code for observational data. It is designed for efficient exchange and storage.
The BUFR format is governed by WMO and defined in the WMO Manual on Codes (WMO Publication Number 306, Volume 1, Part B, 1995 Edition, plus Supplements). The Manual on Codes defines the basic structure of the BUFR format. Data encoded in BUFR can be decoded through the use of tables which map binary encodings to their meanings. The Manual on Codes and tables defined by WMO can be found on the WMO website.
Data producers can define proprietary table entries for local usage. The local tables used by EUMETSAT can be found in these documents:
- Cloud analysis (CLA), global instability index (GII), and climate data set products
These local descriptors are no longer used to encode new products produced at EUMETSAT, but may be encountered in historical data.
- IASI Level 2 products
These local entries are mostly related to atmospheric composition and quality metadata.
Products encoded in BUFR and disseminated by EUMETSAT using BUFR Master Table Version 31 and higher no longer use local tables. Instead, only entries governed by WMO are used. This should make it easier for users to make use of EUMETSAT products. A Migration Guide is provided in order to help users transition from products encoded in BUFR using Master Table 30 and below to products encoded using Master Table 31 and above: Migration guide: Local EUMETSAT descriptors to WMO descriptors for IASI Level 2 BUFR products.
Several software libraries are capable of decoding BUFR. Currently, ecCodes by ECMWF is the one which is most actively maintained.
GRIB is WMO’s standard binary format for exchanging gridded data. It exists in multiple editions: GRIB Edition 1 (GRIdded Binary) is still in existence, whereas this edition of GRIB has been superseded by the more general GRIB Edition 2 (General Regularly-distributed Information in Binary Form). Both editions of GRIB are designed for the timely dissemination of large volumes of gridded data.
Many weather centres use GRIB to encode the outputs of their numerical weather prediction models.
Like BUFR, GRIB is governed by WMO and defined in the Manual on Codes. As well as the table entries defined by WMO, EUMETSAT uses additional entries to Table 4.2, Product Discipline 3 (space products), parameter category 2 (cloud properties), and to Table 4.218 (pixel scene type). These entries are detailed in are used in the Optical Cloud Analysis (OCA) product and are detailed in the OCA Local GRIB Descriptors document.
The most actively developed library for encoding and decoding GRIB is ecCodes by ECMWF.
Extensible Markup Language (XML) is a general-purpose language designed to be easily readable for both humans and machines. Its primary purpose is to facilitate the exchange of data across different information systems, particularly via the internet. The XML specification is governed by the World Wide Web Consortium (W3C), further details are on the W3C website.
XML can be customised to meet the demands of a specific user community by the use of schemata, which define a limited vocabulary which can be used when encoding data. One such schema is the Common Alerting Protocol (CAP).
CAP is a general alerting protocol adopted by several communities who produce and distribute warnings. It is used by EUMETSAT to alert users to possible threats to life or property due to volcanic ash or fire.
CAP is maintained by the Organization for the Advancement of Structured Information Standards (OASIS) and is recommended for use by the International Telecommunications Union. The CAP specification can be found on the OASIS website.
The Man computer Interactive Data Access System (McIDAS) uses a proprietary data format for the analysis and visualisation of meteorological data, including satellite data. McIDAS is maintained by the Space Science and Engineering Centre at the University of Wisconsin-Madison.
Advanced High Rate Picture Transmission (AHRPT) is a standard agreed upon by the members of CGMS, for the dissemination of data originating from low earth orbit satellites to users via direct broadcast. AHRPT data are typically disseminated at speeds greater than 500 Kbps.
The AHRPT global definition can be found in the Global Specification by CGMS. Information on the mission-specific implementation for Metop can be found in the Metop Direct Readout AHRPT Technical Description.