<span style='font-family:"Times New Roman"'>Knorr IMET Data Quality Control Report

Knorr IMET Data Quality Control Report

 

 

 

 

 

 

 

 

Jennifer Lovell and Shawn R. Smith

 

 

 

 

 

 

 

Research Vessel Surface Meteorological Data Center

 

Center for Ocean Atmospheric Prediction Studies

 

Florida State University

 

 

March 27, 2001

 

Report MET 01-04

 

Version 1.0

 

 

       


 

 

INTRODUCTION:

 

This report summarizes the quality of surface meteorological data collected by the research vessel Knorr (identifier: KCEJ) during ten cruises completed in 1993 and 1994.  The data were provided to the Florida State University Data Assembly Center (DAC) in electronic format by B. Walden (WHOI) and were converted to standard DAC netCDF format.  The data were then processed using an automated screening program, which added quality control flags to the data, highlighting potential problems.  Finally, the Data Quality Evaluator (DQE) reviewed the data and current flags, whereby flags were added, removed, or modified according to the judgment of the DQE and other DAC personnel.  Details of the quality control procedures can be found in Smith et al. (1994).  The data quality control report summarizes the flags for the Knorr meteorological data, including those added by both the preprocessor and the DQE.


 

 

DATA VARIABLES:

 

The Knorr data are expected to include observations averaged once every minute on these cruises.  Values for the following variables were collected:

 

Time

Latitude

Longitude

Platform Course

Platform Speed Over Ground

Atmospheric Pressure

Air Temperature

Humidity Temperature

Sea Temperature

Relative Humidity

Precipitation

Atmospheric Radiation

(TIME)

(LAT)

(LON)

(PL_CRS)

(PL_SPD)

(P)

(T)

(T2)

(TS)

(RH)

(PRECIP)

(RAD)


 

 

REMOVED DATA:

 

The winds for the 1993 and 1994 Knorr data, wind vane direction (W_VANE), compass direction (W_COMP), platform relative wind direction (PL_WDIR), platform relative wind speed (PL_WSPD), earth relative wind direction (DIR), and earth relative wind speed (SPD) were not included in the public release of the data.  The quality of these winds were extremely poor. 


 

 

1993 FLAG SUMMARY

 

Statistical Information:

 

Details of each 1993 cruise are listed in Table 1 and include the cruise dates, number of records, number of values, number of flags, and total percentage of data flagged.  A total of 1,924,332 values were evaluated with 147,356 flags added by both the preprocessor and the DQE resulting in a total of 7.66% of the values being flagged.

 

Table 1: Statistical Cruise Information

 

Cruise Identifier

Cruise Dates

Number of Records

Number of Values

Number of Flags

Percent Flagged

93-A

93-B

93-C

93-D

93-E

93-F

93-G

01/28/93 – 02/01/93

02/03/93 – 02/18/93

03/18/93 – 03/28/93

07/25/93 – 08/18/93

09/27/93 – 10/21/93

11/01/93 – 11/19/93

12/08/93 – 12/21/93

6,788

21,697

15,138

35,290

35,525

27,097

18,826

81,456

260,364

181,656

423,480

426,300

325,164

225,912

1,890

15,719

5,017

23,292

57,084

18,240

26,114

2.32

6.04

2.76

5.50

13.39

5.61

11.56

 

Summary:

 

The 1993 IMET data from the Knorr proves to be of fair quality with 7.66% of the reported values flagged for potential problems.  The distribution of flags for the remaining variables are detailed in Table 2.

 

Table 2: Number of Flags and Percentage Flagged for Each Variable

 

Variable

B

C

F

G

J

K

S

Total Number of Flags

Percentage of Variable Flagged

TIME

LAT

LON

PL_CRS

PL_SPD

P

T

T2

TS

RH

PRECIP

RAD

 

 

 

3

47

1

1

1

 

59

 

54,267

1

 

160

159

 

 

 

 

 

 

76

35

1

1

 

3,101

3,623

5,624

5,982

8,457

25,097

6,338

3,997

3,746

8,885

6,223

 

 

 

 

 

 

1,040

1,103

519

6,742

545

69

 

71

73

154

253

93

111

21

446

21

203

7

1

3,332

3,855

5,781

6,282

8,551

26,325

7,498

4,963

10,569

9,633

60,566

 0.00*

2.08

2.40

3.6

3.92

5.33

16.42

4.68

3.09

6.59

6.01

37.77

Total

Number Of

Flags

54,379

1

319

113

81,073

10,018

1,453

147,356

 

Percent Of All Values Flagged

2.83

0.00*

1.66

5.87

4.21

5.21

7.55

7.66

 

                       *Percentages < 0.01

   

C-Flags:

 

During the 93-E cruise, one data value was assessed a C-flag by the preprocessor on 1993 October 2.  C-flags are given to data times that are non sequential.  All flags which were assessed on 1993 Jan 1 2:16, include "CFFZZBBBGGZZ".  All data in the C-flagged record should not be used.  The DQE felt these data values should remain in the Statistical Information for 1993 Knorr data, as the rest of the day's data are useful.

 

B-Flags:

 

Three unrealistic negative data values, –0.01, were recorded on platform course (PL_CRS).

 

Platform speed over ground (PL_SPD) was given 47 bounds flags to negative data values during the 93-A, 93-B, and 93-C cruises.  Negative ship speed is possible when the vessel is moving at slow speeds.  Currents, waves, and wind can cause realistic negative values.  Negative values will also occur when the ship is in reverse.

 

During the 93-B cruise, relative humidity (RH) received 59 B-flags on values that were between 100% and 100.2%.  This is likely due to the sensor not tuned to high relative humidity values.

 

Radiation (RAD) received 54,267 B-flags during the 1993 cruises.  The values were between zero and negative one Wm-2.  These physically unrealistic negative radiation values are likely the result of the instrument not tuned to low radiation values.

 

F-flags:

 

Latitude (LAT) and longitude (LON) were assessed a total of 319 F-flags by the preprocessor during all of the 1993 Knorr cruises.  These F-flags show that the platform speed computed by the preprocessor exceeds the realistic speed (15 ms-1).  This may have been caused by uncertainties or truncation error in the navigation data.

 

G-flags: 

 

Note:  During the 93-A cruise, the ship traversed south of 40 degrees South Latitude.  In this region of the globe, little information is known about the climatology, as the data is sparse.  Consequently, the G-flagged data values may be realistic, though extreme observations. 

 

Temperature (T) was assessed seven G-flags by the preprocessor during the 93-A cruise.  The DQE felt these flagged values were realistic, as they were approximately four to five degrees Celsius lower than the climatological data value and were left in place to highlight extreme temperatures.  During the 93-C cruise, temperature had 69 G-flags that were approximately five to six degrees Celsius higher than the climatological value; therefore, the DQE felt these were realistic, though extreme temperatures.

 

Humidity temperature (i.e., temperature in the relative humidity sensor) (T2) received 8 G-flags during the 93-A cruise and 27 G-flags during the 93-C cruise.  The flagged values on the 93-A cruise were approximately four to five degrees Celsius lower than the climatological value.  During the 93-C cruise, the G-flagged values were approximately five to six degrees Celsius higher than the climatological data value and were left in place to highlight extreme humidity temperatures.

 

All G-flags were left in place to highlight values that are greater than four standard deviations from the climatological mean (da Silva et al. 1994).

 

J-flags:

 

All of the J-flags assessed to the 1993 Knorr cruises were associated with measurements holding at a constant value (often zero) for an unrealistic period of time.  On numerous occasions, all variables would flat-line at the same time.

 

K-flags: 

 

The K-flag represents suspect data and should be used with caution.  K-flags were used to highlight many different suspect situations.   For example, on the last day of the 93-A cruise, temperature (T) had a three degree Celsius increase in one minute and five minutes later returned back to its previous data trend.  A signature to support such an increase was not found in humidity temperature (T2) or relative humidity (RH).  Consequently, the increase was flagged as suspect and should be used with caution.

 

Temperature (T), humidity temperature (T2), and relative humidity (RH) received K-flags on the 93-E, F, and G cruises.  These K-flags emphasize periods when the three variables did not adhere to expected meteorological relationships (e.g., RH generally increases when T decreases).  The exact cause of the variables' discontinuities is unknown as the DQE did not have enough meteorological (e.g., winds) or ship (e.g., heading) data to thoroughly assess the situations.

 

Sea temperature (TS) and precipitation (PRECIP) were given K-flags to draw attention to some type of interference that would occur on both variables at the same time.  (See Fig. 1).

 

Fig. 1  Illustration of interference on the sea temperature and precipitation data from the 93-G Knorr cruise.

 

     

Precipitation received K-flags for values recorded above 50 mm and for data which increased rapidly and then dropped back down to the previous data trend. (See Fig. 2).

 

   
Fig. 2  Shows precipitation data that increases rapidly and then drops back to the previous data trend.

 

 

Radiation (RAD) received 69 B-flags on highly unrealistic data values of –600 to –400 Wm-2.  These erroneous data values were changed to K-flags by the DQE. 

 

Spikes:

 

Isolated spikes occurred in most of the variables throughout the data.  Spikes are a relatively common occurrence with automated data, caused by various factors (e.g. electrical interference, ship movement, etc.).  These individual points were assigned the S-flag.


 

 

1994 FLAG SUMMARY

 

 

Statistical Information:

 

Details of each 1994 cruise are listed in Table 3 and include cruise dates, number of records, number of values, number of flags, and total percentage of data flagged.  A total of 541,220 values were evaluated with 24,341 flags added by the preprocessor and the DQE resulting in 4.50% of the values being flagged.

 

Table 3: Statistical Cruise Information

 

Cruise Identifier

Cruise Dates

Number of Records

Number of Values

Number of Flags

Percent Flagged

94-A

94-B

94-C

02/19/94 – 02/28/94

03/01/94 – 03/19/94

03/20/94 – 03/28/94

14,221

27,057

12,844

142,210

270,570

128,440

10,784

10,037

3,520

7.58

3.71

2.74

 

Summary:

 

The 1994 IMET data from the Knorr proves to be of good quality with 4.50% of the reported values flagged for potential problems.  The distribution of flags for the remaining variables are detailed in Table 4.

 

Table 4: Number of Flags and Percentage Flagged for Each Variable

 

Variable

B

F

J

S

Total Number of Flags

Percentage of Variable Flagged

TIME

LAT

LON

PL_CRS

PL_SPD

T

T2

RH

PRECIP

RAD

 

 

 

 

 

 

 

469

 

17,152

 

44

44

 

3,097

3,096

160

163

 

7

7

40

53

2

 

 

7

0

3,148

3,147

200

216

2

0

469

7

17,152

0.00

5.82

5.81

0.37

0.40

 0.00*

0.00

0.87

0.01

31.69

Total Number Of Flags

17,621

88

6,516

116

24,341

 

Percent Of All Values Flagged

1.81

0.01

1.20

0.01

4.50

 

                              *Percentages < 0.01

   

B-flags:

 

During the 94-A cruise, relative humidity (RH) received 469 B-flags on values that were between 100% and 101.4%.  This is likely due to the sensor not tuned to high relative humidity values.

 

Radiation (RAD) received a total 17,152 B-flags on the 1994 cruises.  These values were between zero and negative one Wm-2.  These physically unrealistic negative radiation values are likely the result of the instrument not tuned to low radiation values.

 

F-flags:

 

Over the course of the 1994 cruises, latitude (LAT) and longitude (LON) received 88 F-flags.  These flags reveal that the platform speed computed by the preprocessor exceeds the realistic speed (15 ms-1).  This may have been caused by uncertainties or truncation error in the navigation data.

 

J-flags:

 

J-flags, erroneous flags, were assigned to data for various reasons.  First, J-flags were given to data that were associated with measurements holding at a constant value (often zero) for an unrealistic period of time.   For example, the platform course (PL_CRS) and platform speed over ground (PL_SPD) fell out of character with the surrounding data and were consequently J-flagged.

 

Second, J-flags were used to highlight stair-steps in the data that were erroneous.  For example, LAT, LON, PL_CRS, and PL_SPD were J-flagged when a large stair-step in the data was recorded.  (See Fig. 3).

 

Fig. 3  Erroneous latitude, longitude, platform course, and platform speed data for the 94-C Knorr cruise.

 

Spikes:

 

Isolated spikes occurred in all of the variables in the data.  Spikes are a relatively common occurrence with automated data, caused by various factors (e.g. electrical interference, ship movement, etc.).  These individual points were assigned the S-flag.


 

 

Removed Data:

 

Pressure (P) and Sea Temperature (TS) were not included in the public release of the 1994 Knorr data, because the data values were flat-lined on zero for each cruise.


 

 

Final Discussions:

 

Precipitation (PRECIP) data were flat-lined on zero for almost all of the 1994 Knorr cruises, except for a few spikes periodically.  On the last two days of the 94-C cruise, PRECIP recorded useful data.  The flat-lined data was not removed from the data set, as it was unknown to the DQE if it truly did not rain or if a possible instrument malfunction occurred and was then fixed before the end of the last cruise.


 

 

References:

 

Smith, S.R., C. Harvey, and D.M. Legler, 1994: Handbook of Quality Control

     Procedures and Methods for Surface Meteorology Data.  Report No. 141/96,

     Report MET 96-1, Center for Ocean-Atmospheric Prediction Studies Florida

     State University, Tallahassee FL 32306-2840

 

da Silva, A.M., C.C. Young and S. Levitus, 1994: Atlas of Surface Marine Data 1994,

     Volume 1: Algorithms and Procedures.  NOAA Atlas Series.