EUMETSAT utilises a variety of formats for product storage and distribution. The most commonly used types are listed below.
BUFR is the World Meteorological Organization (WMO) standard binary code for the representation and exchange of observational data. BUFR is defined in the WMO Manual on Codes (WMO Publication Number 306, Volume 1, Part B, 1995 Edition, plus Supplements).
The acronym stands for 'Binary Universal Form for the Representation of meteorological data'. The BUFR tables used at EUMETSAT are the WMO Standard BUFR tables and are available on the WMO codes webpage.
To decode BUFR messages a decoding program and a set of tables are required. Three different versions of the BUFR tables are relevant to MPEF users.
Templates describe the order and type of data within a BUFR message. This data is also contained in Section 3 of the messages themselves. The templates currently used for encoding the MPEF BUFR products are shown here:
Local BUFR Descriptors
For certain meteorological products, namely the Cloud Analysis (CLA), Global Instability Index (GII) and the Climate Data Set (CDS), there are some changes in the WMO BUFR description. These changes are described in the Local Descriptors (PDF, 64 KB). There are two additional code table entries for the Clear Sky Radiance (CSR) (PDF, 10 KB). There are also BUFR descriptors for the All Sky Radiance (ASR) (PDF, 14 KB) product.
GRIB (GRIdded Binary) is WMO's standard binary format for exchanging gridded data. GRIB Edition 2 is an extension of GRIB, with a much higher degree of flexibility and expandability. The expanded name of this code form is 'General Regularly-distributed Information in Binary form'. It is an efficient vehicle for transmitting large volumes of gridded data to automated centres over high-speed telecommunication lines using modern protocols. GRIB 2 can serve well as a data storage format, generating the same efficiencies relative to information storage and retrieval devices.
For a definition of the GRIB 2 format see the WMO GRIB2 description on the WMO codes page.
Local GRIB Descriptors
For certain meteorological products, namely the Optimal Cloud Analysis (OCA), there are some changes in Code table 4.2 (PDF, 93 KB) of the WMO description. From Spring 2014, Version 12 of GRIB Master Tables will be implemented at EUMETSAT. The implication is that the OCA products will be described using only global descriptors and no local descriptors will be required. The other GRIB products (CLM, FIR, CRM, CLAI, MPE, DIV and CTH) will not be impacted.
High Rate Information Transmission (HRIT) and Low Rate Information Transmission (LRIT) are the CGMS standards, agreed upon by satellite operators, for the dissemination of digital data, originating from geostationary satellites to users, via direct broadcast. The distinction between the two standards, as their names suggest, is the data rate (bandwidth) necessary to convey the data content. LRIT data are typically disseminated at speeds up to around 256 Kbps, while HRIT data are typically disseminated at speeds up to 10 Mbps. Commonly the content of LRIT data streams are subsets of the equivalent HRIT data, with subsampling and lossy compression applied.
Global format specifications for these digital data transmission standards, based on the application and presentation layers of the OSI reference model, are described in LRIT/HRIT Global Format Specifications. Individual satellite operators have discretion to make mission-specific implementations of the lower layers of the OSI reference (to implement such elements as data compression methods, data encryption, etc) and these are described for EUMETSAT's Meteosat Second Generation (MSG) and for JMA's Multifunction Transport Satellite (MTSAT) for JMA-HRIT and JMA-LRIT.
Level 1.5 High Rate SEVIRI image data is the result of the processing of the satellite raw data (designated as Level 1.0 data). These data correspond to image data that has been corrected for all unwanted radiometric and geometric effects, has been geolocated using a standardised projection and has been calibrated and radiance-linearised. Ancillary information is also available for Level 1.5 data in the form of a Header and a Trailer to the imagery data. The MSG Level 1.5 Image Data Format Description, EUM/MSG/ICD/105, provides details on the ancillary information.
The MSG Ground Segment LRIT/HRIT Mission Specific Implementation, EUM/MSG/SPE/057, provides information on the implementation of the global HRIT/LRIT format within the MSG System.
It should be noted that the file structure of HRIT/LRIT data are sometimes used to distribute data via other mechanisms, for example in the EUMETCast data dissemination service, and as retrieval formats for archived data.
High Rate Picture Transmission (HRPT) and Low Rate Picture Transmission (LRPT) are the CGMS standards, agreed upon by satellite operators, for the dissemination of digital data, originating from low earth orbit satellites to users, via direct broadcast. In a very similar way to HRIT/LRIT, described above, the distinction between the two standards is the data rate (bandwidth) necessary to convey the data content. LRPT data are typically disseminated at speeds less than 150 Kbps while HRPT data are typically disseminated at speeds greater than 0.5 Mbps.
Global format specifications for these digital data transmission standards, based on the application and presentation layers of the OSI reference model, are described in HRPT/LRPT Global Format Specifications.
WMO binary data exchange formats — BUFR, GRIB
The WMO Binary Universal Form for the Representation of meteorological data (BUFR) is a binary code designed to represent any meteorological dataset employing a continuous binary stream. It has been designed to achieve efficient exchange and storage of meteorological and oceanographic data. It is self defining, table driven and very flexible data representation system, especially for huge volumes of data.
Similarly, another widely used bit-oriented data exchange scheme is the WMO GRIddedBinary (GRIB) format. GRIB is an efficient vehicle for transmitting large volumes of gridded data to automated centers over high-speed telecommunication lines, using modern protocols. An updated version of GRIB, commonly abbreviated to GRIB-2, is currently being introduced and is most relevant for use with satellite data.
These two WMO Table Driven Code Forms have been widely adopted for the distribution of meteorological satellite products, especially those processed to level 2 or beyond (see the Imagery and Derived Products section). They are described in the Operational Codes and Manual on Codes. By packing information into the BUFR or GRIB code, data records can be made more compact than character-oriented bulletins, resulting in faster computer-to-computer transmissions. The formats can equally well serve as a data storage formats, generating the same efficiencies relative to information storage and retrieval devices.
Software for encoding and decoding data in the BUFR and GRIB formats is freely available for download from the ECMWF software.
The Man computer Interactive Data Access System (McIDAS) is not simply a satellite data format, it is a suite of applications for analysing and displaying meteorological data for research and education. McIDAS has been in use, and under continual development, by the University of Wisconsin-Madison Space Science and Engineering Center (SSEC) since 1972. The Unidata McIDAS software (a superset of SSEC McIDAS) has been under development since 1985 and in distribution since 1988. The software can be used with conventional observational, satellite, and grid-point data. Unidata distributes a version of McIDAS (McIDAS-X) for a variety of platforms running Unix, Linux, and MacOS-X.
NetCDF (network Common Data Form) is a machine-independent, self-describing, binary data format standard for exchanging scientific data. The project homepage is hosted by the Unidata program at the University Corporation for Atmospheric Research (UCAR). They are also the chief source of netCDF software, standards development, updates etc. The format is an open standard. The data format is 'self-describing'. This means that there is a header which describes the layout of the rest of the file, in particular the data arrays, as well as arbitrary file metadata in the form of name/value attributes. The format is platform independent, with issues such as endianness being addressed in the software libraries. The data arrays are rectangular, not ragged, and stored in a simple and regular fashion that allows efficient subsetting.
Hierarchical Data Format, commonly abbreviated HDF, HDF4, or HDF5, is a library and multi-object file format for the transfer of graphical and numerical data between computers. It is created and maintained by the NCSA. The freely available HDF distribution consists of the library; command-line utilities; test suite source; Java interface, and the Java-based HDF Viewer (HDFView). HDF supports several different data models, including multidimensional arrays, raster images, and tables. Each defines a specific aggregate data type and provides an API for reading, writing, and organising the data and metadata. New data models can be added by the HDF developers or users. HDF is self-describing, allowing an application to interpret the structure and contents of a file without any outside information. One HDF file can hold a mixture of related objects, which can be accessed as a group or as individual objects.
The Extensible Markup Language (XML) is a general-purpose markup language. Its primary purpose is to facilitate the sharing of data across different information systems, particularly via the Internet. More details of XML and its potential role in the exchange of binary data may be found on the W3C website.
The Common Alerting Protocol (CAP) provides an open, non-proprietary digital message format for all types of alerts and notifications. CAP allows a consistent warning message to be disseminated simultaneously over many different warning systems, increasing warning effectiveness while simplifying the warning task. CAP also aids the detection of emerging patterns in local warnings of various kinds, such as indications of an undetected hazard or hostile act. More details about CAP.