Changes to PC compressed IASI L1C data
Principal Component (PC) compressed IASI L1C data will be updated on 30 March, as part of the new IASI L2 PPF v6.7 release.
18 September 2023
22 March 2023
The PC compression (PCC) basis files will be updated and at the same time the so-called hybrid approach, which will supplement the global PC bases with five additional local PC basis vectors in each of the three spectral bands based on the residuals of each individual data granule, will be introduced.
The new IASI PC compression basis v2.01
The PCC basis is applicable to both Metop-B and C IASI instruments in routine production, and is also applicable to historical IASI-A L1c records. It was generated in a staged approach, where groups of new directions were added successively to an initial set of basis directions, which were based on a large set of training “base” spectra. IASI-C spectra were deliberately excluded from the base spectra made up by 74 days of reprocessed IASI-A L1C measurement data (chosen in the period from 9 August 2008 to 1 July 2019) and 48 days of IASI-B L1C measurement data (chosen in the period from 1 March 2013 to 1 July 2019) for a total of 152911164 base spectra.
The three (one for each band) PCC basis files are:
- IASI_EV1_xx_Mxx_20200131000000Z_xxxxxxxxxxxxxxZ_20203101000201Z_xxxx_xxxxxxxxxx
- IASI_EV2_xx_Mxx_20200131000000Z_xxxxxxxxxxxxxxZ_20203101000201Z_xxxx_xxxxxxxxxx
- IASI_EV3_xx_Mxx_20200131000000Z_xxxxxxxxxxxxxxZ_20203101000201Z_xxxx_xxxxxxxxxx
The contents are summarised in this table:
| Attribute/Dataset | Description | Value/Size for each band/file |
|---|---|---|
| FirstChannel | Channel number of the first channel in the band | 1; 1998; 5117 |
| NbrChannels | Number of channels in the band (m) | 1997; 3119; 3345 |
| Eigenvectors | The PCC basis vectors (ET) | (90×m); (120×m); (90×m) |
| Nedr | The noise normalisation matrix (N) | (m×m) |
| ReconstructionOperator | (NE)T | (90×m); (120×m); (90×m) |
The ReconstructionOperator is redundant as it can be generated by multiplying N and E, but has been provided for convenience.
The noise normalisation matrix is now a full matrix (instead of a diagonal matrix as it has been the case so far). This means that the noise normalisation will also decorrelate the noise (instead of just homogenising it), which leads to better noise filtering properties.
There is no longer any field 'Mean' in the PCC basis files. The disseminated PC scores are generated as p=ETN -1y (with y being the original radiances) and the reconstructed radiances are obtained as y ̃=NEp.
EUMETSAT is currently running two parallel external studies to evaluate the performance of the v2.01 PCC bases for atmospheric composition and air quality applications and to identify any directions of atmospheric signal associated with rare situations which might not be included in this basis. As a result, we foresee to issue a slightly modified version (v2.02) of the PCC basis files, taking such directions into account.
The hybrid approach
The hybrid approach captures spectral features orthogonal to the subspace spanned by the global PC basis vectors and can accommodate long term trends of the atmosphere as well as features originating from eventual rare situations not fully captured by the global PC basis.
Users who wish to take the additional local PC scores into account can reconstruct radiances as y ̃=NEp+ Rlocalplocal where plocal and Rlocal are the local PC scores and their associated reconstruction operator, which are both provided in the L1C hybrid PC compressed products.
To disseminate the local PC scores and associated reconstruction operators of the hybrid approach a new format is needed for the IASI L1C PCC products. We propose the HDF5 format with the following contents in each of the three spectral bands (where n is the number of lines in the granule, s is the number of global PC scores in the band and m is the number of channels).
| Dataset | Description | Size |
|---|---|---|
| p | The PC scores associated with the global PC basis | (n×120×s ) |
| rsum | Sum of all radiances in the band | (n×120) |
| rs | Reconstruction score (root mean square of the reconstruction residual) | (n×120) |
| p_local | Local PC scores | (n×120×5 ) |
| R_local | (transpose of the) Reconstruction operator associated with the local PC scores. | (5×m) |
| rs_local | Reconstruction score after application of local PC scores | (n×120) |
| energy | Trace of the residual covariance (and mean) over the full granule | |
| V_local | First five eigenvalues of the residual covariance (and mean) | (5) |
| quality_flag | L1C quality flag (zero means good) | (n×120) |
Further information about the hybrid approach and the rationale behind it are available in:
- Tim Hultberg, Thomas August, Flavia Lenti, Local or global? How to choose the training set for principal component compression of hyperspectral satellite measurements: a hybrid approach, Proc. SPIE 10423, Sensors, Systems and Next-Generation Satellites XXI, 104231G (29 September 2017); doi: 10.1117/12.2278349
In preparation of the operational roll-out of PC-hybrid products, a live test data stream has been put in place on a rolling FTP archive. Details on how to access the data and how to use them are provided in the README file. Note: The distribution on EUMETCast and the archiving of the PC-hybrid products are planned to start at a later date (to be communicated soon).
For more information, contact our User Service Helpdesk.
