Knorr IMET Data Quality Control Report
Jennifer Lovell and Shawn R. Smith
Research Vessel Surface Meteorological Data Center
Center for Ocean Atmospheric Prediction Studies
Florida State University
December 20, 2000
Report 01-03
Version 1.0
1.0 INTRODUCTION:
This report summarizes the quality of surface meteorological data collected by the research vessel Knorr (identifier: KCEJ) IMET system during twelve cruises which occurred over a two year span, beginning 28 January 1996 and ending 24 March 1997. The data sets were provided to the Florida State University Research Vessel Surface Meteorological Data Center (RVSMDC) in electronic format by M. Lamont (WHOI) and were converted to standard RVSMDC netCDF format. The data were then processed using an automated screening program, which added quality control flags to the data, highlighting potential problems. Finally, the Data Quality Evaluator (DQE) reviewed the data and current flags, whereby flags were added, removed, or modified according to the judgment of the DQE and other RVSMDC personnel. Details of the RVSMDC quality control procedures can be found in Smith et al. (1996). The data quality control report summarizes the flags for the Knorr IMET surface meteorological data, including those added by both the preprocessor and the DQE.
2.0 DATA VARIABLES:
The Knorr IMET data are expected to include observations averaged every minute on these cruises. Values for the following variables were collected:
Time |
(TIME) |
Latitude |
(LAT) |
Longitude |
(LON) |
Platform Heading (Gyrocompass) |
(PL_HD) |
Platform Course |
(PL_CRS) |
Platform Speed Over Ground |
(PL_SPD) |
Platform Speed Over Water |
(PL_SPD2) |
Platform Relative Wind Direction (IMET) |
(PL_WDIR) |
Platform Relative Wind Speed (IMET) |
(PL_WSPD) |
Ocean Relative Wind Direction |
(DIR) |
Ocean Relative Wind Speed |
(SPD) |
Earth Relative Wind Direction (IMET) |
(DIR2) |
Earth Relative Wind Speed (IMET) |
(SPD2) |
Atmospheric Pressure |
(P) |
Air Temperature |
(T) |
Sea Temperature |
(TS) |
Relative Humidity |
(RH) |
Atmospheric Radiation |
(RAD) |
Precipitation |
(PRECIP) |
3.0 CRUISE IDENTIFIERS AND DATES:
Note: The Cruise Identifiers and Cruise Dates were assigned to the Knorr cruises by the DQE for cruise identification in the quality control report. The beginning and ending dates of each cruise were determined by the Knorr's departure and return dates to port successively.
Cruise Identifiers Cruise Dates
1996
96-A 96-B 96-C 96-D 96-E 96-F 96-G 96-H 96-I |
01/28/96 – 01/30/96 02/09/96 – 03/11/96 03/13/96 – 03/26/96 04/03/96 – 05/08/96 05/21/96 – 06/19/96 06/20/96 – 06/22/96 06/27/96 – 08/08/96 10/24/96 – 11/01/96 12/13/96 – 01/27/97 |
1997
97-A |
01/30/97 – 02/01/97 |
4.0 1996 FLAG SUMMARY
Statistical Information:
Details of each 1996 cruise are listed in Table 1 and include cruise dates, number of records, number of values, number of flags, and total percentage of data flagged. A total of 4,954,790 values were evaluated with 306,147 flags added by both the preprocessor and the DQE resulting in 6.18% of the values being flagged.
Table 1: Statistical Cruise Information
Cruise Identifier |
Cruise Dates |
Number of Records |
Number of Values |
Number of Flags |
Percent Flagged |
96-A 96-B 96-C 96-D 96-E 96-F 96-G 96-H 96-I |
01/28/96 – 01/30/96 02/09/96 – 03/11/96 03/13/96 – 03/26/96 04/03/96 – 05/08/96 05/21/96 – 06/19/96 06/20/96 – 06/22/96 06/27/96 – 08/08/96 10/24/96 – 11/01/96 12/13/96 – 01/27/97 |
4,084 45,361 20,159 51,329 38,766 3,200 59,796 10,424 64,989 |
69,428 589,693 262,067 667,277 736,554 60,800 1,136,124 198,056 1,234,791 |
1,331 107,196 45,455 60,968 17,029 373 33,246 24,323 16,226 |
1.92 18.18 17.34 9.14 2.31 0.61 2.93 12.28 1.31 |
Summary:
The 1996 IMET data from the Knorr proves to be of fair quality with 6.18% of the reported values flagged for potential problems. The distributions of flags for the remaining variables are detailed in Table 2.
Table 2: Number of Flags and Percentage Flagged for Each Variable
Variable |
B |
G |
H |
J |
K |
S |
Total Number of Flags |
Percentage of Variable Flagged |
LAT LON PL_HD PL_CRS PL_SPD PL_SPD2 PL_WDIR PL_WSPD DIR SPD DIR2 SPD2 P T TS RH RAD PRECIP |
1
15,701
595
11,589 29,612 |
88
801 3,687
|
2
10 |
1,119 85,019 84,899 7,476 7,476 52 30 1,276 1,241 58 1,235 929 1,387 |
113 5,696 5,696 7,577 6,511 3,015 1,896 6,330 5,087 5,971
159 |
44 9 7 34 42 283 230 116 301 147 185 185 1,352 61 26 791 |
0 1 44 9 7 16,967 90,757 90,878 15,283 14,191 3,368 2,073 7,793 7,909 11,078 12,885 30,567 2,337 |
0 0.00* 0.01 0.00* 0.00* 5.54 29.65 29.68 4.99 4.64 1.10 0.68 2.55 2.58 3.62 4.21 9.98 0.76 |
Total Number Of Flags |
57,498 |
4,576 |
12 |
192,197 |
48,051 |
3,813 |
306,147 |
|
Percent Of All Values Flagged |
1.16 |
0.09 |
0.00* |
3.88 |
0.97 |
0.08 |
6.18 |
*Percentages<0.01
B-flags:
Longitude was assessed one B-flag during the 96-D cruise. This flag occurs on a data value when the Knorr crossed the Prime Meridian. This longitude was incorrectly coded.
Platform speed over water (PL_SPD2) received 15,701 B-flags for negative data values. The sensor on this vessel will record negative values when the ship is moving astern relative to the water surface. B-flags retained to mark negative PL_SPD2.
Temperature (T) received 595 B-flags for very low temperatures. The Knorr was near the coast of Newfoundland and was experiencing low temperatures ranging from –10 degrees Celsius to –16 degrees Celsius.
Relative humidity (RH) was assessed 11,589 B-flags over five cruises. The flagged values were above 100%, but below 102%. This is likely due to the sensor not being tuned to high relative humidity values.
Radiation (RAD) received 29,612 B-flags over eight different cruises. These values were between zero and negative one Wm-2. These physically unrealistic negative radiation values are likely the result of the instrument not tuned to low radiation values.
G-flags:
Earth relative wind speed (SPD) received 88 G-flags during the 96-E cruise. The DQE felt these values were realistic, as they were approximately one to five ms -1 greater than the climatological value and were left in place to highlight extreme wind speed values.
Temperature (T) was assessed 801 G-flags over three different cruises. In each cruise, the flagged temperature did not exceed one degree Celsius greater or less than the climatological value; therefore, the DQE feels these are realistic, though extreme, temperatures.
Sea Temperature (TS) had 3,687 G-flags over five different cruises. During the 96-B cruise, the flagged sea temperatures were approximately one degree Celsius greater than the climatological value; the DQE feels these are realistic, though extreme, sea temperatures. The other four cruises, 96-C, G, H, and J, the flagged sea temperatures were approximately one to four degrees Celsius lower than the climatological value and were left in place to highlight extreme values.
The G-flags were left in place to highlight values that are greater than four standard deviations from the climatological mean (da Silva et al. 1994).
H-flags:
Pressure (P) was given two H-flags during the 96-E cruise. The data were relatively normal when a sudden drop in pressure, approximately three millibars, occurred. At this point, an H-flag was placed to show the beginning of the discontinuity. After this event, the values recorded did not resemble normal pressure data and were subsequently K-flagged. When the pressure increased, approximately three millibars, and resumed to the normal data trend, a second H-flag was placed to highlight the end of the discontinuity.
Sea temperature (TS) received ten H-flags over four cruises. Many times when the Knorr was either arriving or leaving port, the sea temperature would have large decreases or increases respectively. If data values were recorded between two H-flags, they were assessed K-flags and should be used with caution.
J-flags:
Platform speed over water (PL_SPD2) received 1,119 J-flags over four different cruises. During each of the four cruises, the J-flags were placed on data values that flat-lined on one value for a period of time.
Platform relative wind direction (PL_WDIR) and platform relative wind speed (PL_SPD) received a total of 169,918 J-flags. Two major problems occurred with these two variables. First, most of the J-flags highlight when the variables were very similar in data trend, almost exact. The second problem was that the variables flat-lined on zero many times. These data values were J-flagged and should not be used.
Earth relative wind direction (DIR) calculated by Knorr and earth relative wind speed (SPD) calculated by Knorr were assessed a total of 14,952 J-flags during the 96-H cruise. These J-flags were given to highlight stair stepping with ship motion and were related to a change in platform heading (PL_HD).
Earth relative wind direction (DIR2) calculated by DAC and earth relative wind speed (SPD2) calculated by DAC were assessed a total of 82 J-flags during the 96-G, H, and I cruises. During the 96-G cruise, the data values flat-lined on zero and were subsequently J-flagged. In the 96-H cruise, (DIR2) and (SPD2) were given 61 J-flags to highlight a large discontinuity in the data. During the 96-I cruise, (PL_WDIR) and (PL_WSPD) flat-lined on zero. Platform relative wind direction and platform relative wind speed are used to calculate the DIR2 and SPD2. When these values hold constant at zero (see J-flags PL_WDIR and PL_WSPD) the earth relative winds are then incorrectly calculated. Consequently, the DIR2 and SPD2 were J-flagged and should not be used.
During the 96-B, C, D, E, and I cruises, J-flags were given to pressure (P), temperature (T), relative humidity (RH), and atmospheric radiation (RAD) since the data values flat-lined on one value for a period of time. Note: RH and RAD had missing data during the 96-I cruise, so they were not J-flagged for the above reason.
Pressure was given 129 J-flags during the 96-H cruise to highlight large spikes that were possibly the result of instrument malfunction.
Sea temperature (TS) was assessed 58 J-flags during the 96-H cruise. Occasionally, when the ship speed was very low, the sea temperature would increase very rapidly, approximately four to nine degrees Celsius in one minute. The problem may be related to poor water flow into the seawater intake. The sea temperature would resume to previous data trend when the ship's speed increased.
K-flags:
K-flags were used to note signatures of ship motion in certain meteorological variables. Variables such as earth relative wind direction (DIR), earth relative wind speed (SPD), atmospheric pressure (P), and temperature (T) showed stair steps in the data. These stair steps were related to a change in platform course (PL_CRS), heading (PL_HD), and/or platform speed (PL_SPD) and should not exist in earth relative data. Subsequently, the data were flagged as suspect.
The platform speed over water (PL_SPD2) received 113 K-flags during the 96-F cruise. These K-flags highlight values that were extremely noisy compared to the surrounding data. Platform speed over water also received K-flags during the 96-H cruise since the data did were slower than the platform speed over ground (PL_SPD).
During the 96-B, C, and D cruise, the platform relative wind direction (PLWDIR) and platform relative wind speed (PL_WSPD) were almost identical as stated in the previous J-flags section. During the 96-D cruise, the two variables started to show a normal data trend, but were K-flagged as cautionary because the variables had been erroneous for many months.
The ocean relative wind direction (DIR) calculated by Knorr, ocean relative wind speed (SPD) calculated by Knorr, earth relative wind direction (DIR2) calculated by DAC, and the earth relative wind speed (SPD2) calculated by DAC had stair steps occurring throughout the data sets. The cause was likely due to flow distortion. Flow distortion is the disturbance of airflow from other objects or instruments upstream from the anemometer. The significance of the stair stepping varied throughout the data set; therefore, the earth relative winds should be used with caution.
Pressure (P) continued stair steps throughout the data sets. There were some stair steps in the pressure data that were a result of a change in either forward speed or direction. These stair steps were associated with approximately a ½ millibar (mb) increase in pressure relative to both the forward speed and direction change of the ship. Other K-flags were given to highlight pressure values that increased approximately ½ millibar but did not have any meteorological or ship relative data to prove or disprove the increase. These data should be use with caution.
Temperature (T) was assessed several K-flags due to radiational heating of the ship. When the platform relative wind speed was low, ~3 ms -1 or less, significant increases in temperature were occurring during daylight hours. The second problem in the temperature (T) data was a ventilation problem, which occurred when the platform wind direction (PL_WDIR) was from around 180 degrees. This likely affected the flow of the air before reaching the bow-mounted thermometer. In these instances, significant increases in temperature were flagged as cautionary.
Sea temperature (TS) was given 5,971 K-flags to highlight large increases. When the platform speed over water (PL_SPD2) would either suddenly decrease/increase, the sea temperature would increase/decrease respectively. This might be due to a malfunctioning intake valve, making it possible that the water is not being measured correctly.
Precipitation (PRECIP) received 159 K-flags during the 96-E and 96-G cruises. The data contained small increases that did not resemble spikes, so the values were flagged as cautionary.
Spikes:
Isolated spikes occurred in most of the variables throughout the data. Spikes are a relatively common occurrence with automated data, caused by various factors (e.g. electrical interference, ship movement, etc.). These individual points were assigned the S-flag.
Missing Data:
Missing data were not a serious concern during the 1996 cruises, but in the 96-I cruise, precipitation and radiation were missing for the entire cruise.
5.0 1997 FLAG SUMMARY
Statistical Information:
Details of 1997 cruise are listed in Table 3 and include cruise dates, number of records, number of values, number of flags, and total percentage of data flagged. A total of 78,869 values were evaluated with 1,972 flags added by the preprocessor and the DQE resulting in 2.50% of the values being flagged.
Table 3: Statistical Cruise Information
CTC |
Cruise Dates |
Number of Records |
Number of Values |
Number of Flags |
Percent Flagged |
97-A |
01/30/97 – 02/01/97 |
4151 |
78,869 |
1,972 |
2.50 |
Summary:
The 1997 IMET data from the Knorr proves to be of good quality with 2.50% of the reported values flagged for potential problems. Table 4 details the distribution of flags among the variables.
Table 4: Number of Flags and Percentage Flagged for Each Variable
Variable |
B |
S |
Total Number Of Flags |
Percentage of Variables Flagged |
LAT LON PL_HD PL_CRS PL_SPD PL_SPD2 PL_WDIR PL_WSPD DIR SPD DIR2 SPD2 P T TS RH RAD PRECIP |
29
1,925 |
1 2 5 4 3
3 |
0 0 0 0 0 29 1 2 5 4 3 0 0 1,925 0 3 0 0 |
0.00 0.00 0.00 0.00 0.00 0.70 0.02 0.05 0.12 0.10 0.07 0.00 0.00 46.37 0.00 0.07 0.00 0.00 |
Total Number of Flags |
1,954 |
18 |
1,972 |
|
Percentage of All Values Flagged |
2.48 |
0.02 |
2.52 |
*Percentages < 0.01
B-flags:
Platform speed over water (PL_SPD2) received 29 B-flags during the 97-A cruise for negative data values. This occurs when ship speed is very low, close to zero, and the water is flowing in the opposite direction of the ship's forward motion. During this time of slow speed, the ship will appear to have negative ship speed.
Temperature (T) received 1,925 B-flags for very low temperatures. The Knorr was near the coast of Newfoundland and was experiencing low temperatures ranging from –10 degrees Celsius to –16 degrees Celsius.
Spikes:
Isolated spikes occurred in all of the variables in the data. Spikes are a relatively common occurrence with automated data, caused by various factors (e.g. electrical interference, ship movement, etc.). These individual points were assigned the S-flag.
6.0 FINAL DISSCUSSION:
The Knorr's platform speed over ground (PL_SPD) data was extremely noisy. The DQE recommends the final user to apply a smoother on the data.
REFERENCES:
Smith, S.R., C. Harvey, and D.M. Legler, 1996: Handbook of Quality Control Procedures
and Methods for Surface Meteorology Data.WOCE Report No. 141/96,Report WOCEMET
96-1, Center for Ocean-Atmospheric Prediction Studies Florida State University,
Tallahassee FL 32306-2840
da Silva, A.M., C.C. Young and S. Levitus, 1994: Atlas of Surface Marine Data 1994, Volume 1:
Algorithms and Procedures. NOAA Atlas Series.
Slutz, R.J., S.J. Lubker, J.D. Hiscox, S.D. Woodruff, R.L. Jenne, D.H. Joseph, P.M. Seurer and
J.D. Elms, 1985: COADS - Comprehensive Ocean Atmosphere Data Set,
CIRES/ERL/NCAR/NCDC, Boulder, Colorado.