Formats

Formats

HRIT

High Rate Information Transmission (HRIT) is a standard agreed upon by the members of the Coordination Group for Meteorological Satellites (CGMS) for the dissemination of data originating from geostationary satellites to users via direct broadcast. HRIT data are typically disseminated at speeds up to 10Mbps.

The general formats are defined in the HRIT Global Specification, whereas individual satellite operators may customise the format in order to fulfil the needs of a specific mission.

EUMETSAT uses HRIT to encode Level 1.5 SEVIRI image data. These data have been corrected to remove all unwanted radiometric and geometric effects, geolocated using a standardised projection, calibrated, and radiance-linearized. Ancillary information is provided in the data’s header and trailer.

The full description of the implementation of HRIT for Meteosat Second Generation (MSG) is provided in MSG ground segment HRIT mission specific implementation. Specific details concerning the image data are described in the MSG Level 1.5 image data format description.

netCDF

netCDF (network Common Data Form) is a machine-independent, self-describing, binary data format standard for exchanging scientific data. It is designed for platform independent, self-describing data storage, with the intent that users can understand the data without the need for external resources and access it efficiently in its entirety or in portions.

The netCDF format is governed by the Unidata programme at the University Corporation for Atmospheric Research (UCAR). Because the format is very flexible, several conventions have arisen with the goal of ensuring that data can be easily exchanged between different members of the same community. One notable example can be found in the collected the Climate and Forecast (CF) conventions and metadata. Where possible, EUMETSAT adheres to these conventions when producing data encoded in netCDF.

Further information on the use of netCDF at EUMETSAT can be found on the netCDF format for data centre products page.

Unidata software for reading and writing data netCDF.

HDF

Hierarchical Data Format (HDF) is a format designed for use with large amounts of hierarchically organised data. It is commonly used by producers of large amounts of environmental data, especially when this data is gridded.

HDF is maintained by the HDF Group. Information on the HDF format can be found on their website. The HDF Group also provides software for reading and writing HDF files.

BUFR

The Binary Universal Form for the Representation of meteorological data (BUFR) is the World Meteorological Organization’s (WMO) standard binary code for observational data. It is designed for efficient exchange and storage.

The BUFR format is governed by WMO and defined in the WMO Manual on Codes (WMO Publication Number 306, Volume 1, Part B, 1995 Edition, plus Supplements). The Manual on Codes defines the basic structure of the BUFR format. Data encoded in BUFR can be decoded through the use of tables which map binary encodings to their meanings. The Manual on Codes and tables defined by WMO can be found on the WMO website.

Data producers can define proprietary table entries for local usage. This capability was used in some EUMETSAT cloud analysis (CLA), global instability index (GII), climate dataset products, and IASI level 2 products using BUFR Master Tables version 21 and below. Since migrating to use Master Tables version 31 and above, only entries governed by WMO are used. This makes it possible for users to use EUMETSAT BUFR products without requiring local configuration changes. Mappings between the old and new ways of encoding data are set forth in detail in Migration guide: Local EUMETSAT descriptors to WMO descriptors for IASI Level 2 BUFR products. EUMETSAT BUFR tables can be downloaded in various common formats from the Vocabulary Manager.

Several software libraries are capable of decoding BUFR. These include ecCodes, pybufrkit, wreport, trollbufr, and BUFRdisplay and BUFRextract.

GRIB

GRIB is WMO’s standard binary format for exchanging gridded data. It exists in multiple editions: GRIB Edition 1 (GRIdded Binary) is still in existence, whereas this edition of GRIB has been superseded by the more general GRIB Edition 2 (General Regularly-distributed Information in Binary Form). Both editions of GRIB are designed for the timely dissemination of large volumes of gridded data.

Many weather centres use GRIB to encode the outputs of their numerical weather prediction models.

Like BUFR, GRIB is governed by WMO and defined in the Manual on Codes. As well as the table entries defined by WMO, EUMETSAT uses or has used additional entries specific to EUMETSAT. These entries are detailed in Local GRIB tables used at EUMETSAT. EUMETSAT GRIB tables can be downloaded in various common formats from the Vocabulary Manager.

The most actively developed library for encoding and decoding GRIB is ecCodes by ECMWF.

XML/CAP

Extensible Markup Language (XML) is a general-purpose language designed to be easily readable for both humans and machines. Its primary purpose is to facilitate the exchange of data across different information systems, particularly via the internet. The XML specification is governed by the World Wide Web Consortium (W3C), further details are on the W3C website.

XML can be customised to meet the demands of a specific user community by the use of schemata, which define a limited vocabulary which can be used when encoding data. One such schema is the Common Alerting Protocol (CAP).

CAP is a general alerting protocol adopted by several communities who produce and distribute warnings. It is used by EUMETSAT to alert users to possible threats to life or property due to volcanic ash or fire.

CAP is maintained by the Organization for the Advancement of Structured Information Standards (OASIS) and is recommended for use by the International Telecommunications Union. The CAP specification can be found on the OASIS website.

McIDAS

Package formats

SENTINEL-SAFE

SENTINEL-SAFE is designed to manage any kind of data and data can be contained in a single file or in multiple files.

Using the SENTINEL-SAFE format, information can be categorised in two main categories:

1. Data: the information contained in the product which causes the existence of the product itself.
2. Metadata: stored information which is not data.

A SENTINEL-SAFE format product contains the following components:

• Manifest file: an XML document conforming to the XFDU manifest file specifications. It contains the definition of the information package map, wrapped metadata objects, wrapped data objects and references to the files containing the metadata and data objects.
• Binary or XML files: the data or metadata object contents. Currently, in the SAFE core specifications, only two types of files have been identified, i.e. binary matching MIME octet stream definition and XML documents.
• XML schema files: the representation information of the data held by a SAFE format product. To represent the binary information, the SAFE format also defines specific mark-ups to annotate XML schema documents, providing information on the physical structure (SDF mark-ups).

The size of a Sentinel product will depend on the type of measurements it contains. Several of the Sentinel products contain multiple netCDF files which can be selected individually for download.

The netCDF files, contained within the Sentinel products, can be read using libraries in many commonly used programming languages. Further, the Sentinel-3 Toolbox can be used to open, visualise, process, and analyse OLCI and SLSTR data from the Sentinel-3 mission.

Submission Information Package (SIP)

The Submission Information Package (SIP) is a compressed ZIP file containing:

• one manifest file (manifest.xml) containing further information / metadata about the SIP especially the relationships among the different items constituting the package. The corresponding XML schema can be requested at EUMETSAT.
• EOP metadata (a file “EOPMetadata.xml”) containing metadata about the product based on the OGC Earth Observation Metadata Profile of Observations and Measurements (OGC document 10-157r4). For EUMETSAT specific metadata attributes this is extended in a manner which is conformant to the OGC 10-157r4 principles. The corresponding XML schema can be requested at EUMETSAT.
• Data Object(s) containing the product information: 1 to n data objects (which can be of type Data, Browse, Geolocation, Quicklook, Thumbnail, Other).