3.2 Deleted Data

The DQE determined that a large amount of the 1997 and 1998 Revelle IMET data was unusable due to extensive missing, highly suspect, or erroneous data. As a result, these data were removed from the final quality controlled data set

Sea Temperature (TS) data were missing from all of the 1997 and 1998 Revelle data sets and the TS variable was removed from the final quality controlled data set.

The Longwave Atmospheric Radiation (RAD2) data from the 1997 and 1998 cruises were also deleted due to highly suspect and erroneous values recorded by the RAD2 sensor. During the 1997 cruises, the average recorded RAD2 values were approximately 1400 Watts per meter squared with some values exceeding 2000 Watts per meter squared. These RAD2 values were approximately three to five times above the expected range for longwave radiation data. During the 1998 cruises, the RAD2 sensor reported only a slight variance between maximum and minimum RAD2 values each day. The average range from the highest to lowest RAD2 values each day was approximately 5 Watts per meter squared. This small variance in RAD2 data was determined to be unrealistic. A RAD2 sensor malfunction was the most likely cause of the extensive amount erroneous RAD2 data from the 1997 and 1998 cruises. Therefore, all RAD2 data were excluded from the final data set.

The overall quality of the Precipitation (PRECIP) data was very poor. There were extensive problems associated with the PRECIP data recorded by the self-siphoning rain gauge. The self-siphoning rain gauge is designed to fill to 50 mm, and then rapidly drain back to zero mm. During the 97-C cruise and all the 1998 cruises, the precipitation gauge experienced extensive leakage and/or sensor malfunction problems. The problems with the PRECIP data varied, from slow steady leaks to rapid drops in the data. At other times, the PRECIP data trend was erratic with rapid fluctuations. An example of these types of PRECIP data problems is shown in Figure 1. Precipitation amounts and accumulation rates from these cruises could not be determined. As a result, the PRECIP data from the 97-C cruise and all 1998 cruises were removed from the final data set.

Figure 1. Four day time series of erroneous precipitation data.

The Relative Humidity (RH) data from the 98-A and 98-B cruises were highly erratic and erroneous. The RH values ranged from below zero percent to over 100 percent with erratic trends that were uncharacteristic of RH data. For example, during the 98-A cruise on 98/02/02, the RH data reported approximately -8.4 percent for two days then rapidly rose to 108.1 percent on 98/02/04. The RH level remained at 108.1 percent through 98/02/07. During the 98-B cruise, the RH values were less than zero percent for over half the cruise and the remaining RH values were determined to be unreliable. The RH data from the 98-A and 98-B cruises were deleted from the final data set.

During portions of the 97-B and 98-F cruises, several variables including Latitude (LAT), Longitude (LON), Platform Course (PL_CRS), Platform Speed Over Ground (PL_SPD), and Earth Relative Wind Direction (DIR) and Speed (SPD) reported erroneous flat-line data. These flat-line data were characterized by hours of successive data points reporting the same values. These erroneous data were apparently caused by a sensor malfunction and flagged J by the DQE.

There were numerous G flags assigned by the preprocessor to data that were greater than four standard deviations from the climatological mean (da Silva et al. 1994). The majority of the G flags were assigned while the Revelle was located in the Antarctic region. Climatology for this region is questionable and in most cases, the data appeared to be valid. At times, the G flag was overwritten with a more appropriate flag by the DQE.

Data spikes occurred in the majority of the variables during the 1997 and 1998 cruises and were flagged S by the DQE. Spikes are common to electronic data and may be associated with power surges or ship movement.

4.0 Variable Flagging

The Latitude (LAT) and Longitude (LON) data were very reliable except for flat-line errors reported during portions of the 97-B and 98-F cruises as noted in Section 3.3. Over 95 percent of the K flags assigned to the position data account for data that were not reviewed by the DQE while the ship was in port (see Section 3.1).

The Platform Heading (PL_HD) variable displayed noise problems throughout all the 1997 and 1998 Revelle cruises. The noisy PL_HD data occurred randomly, at times when the ship was either stationary or moving. The cause of this problem may have been due to a sensor malfunction or rough seas though neither could be confirmed with the available data. The noisy periods were flagged K by the DQE and caution is advised when utilizing PL_HD data from these periods. PL_HD was one of the parameters used to calculate the True Winds. Therefore, the corresponding DIR and SPD data were flagged K when the PL_HD data were noisy as shown in Figure 2.

Figure 2. Platform Heading and True Wind Noise.

The Platform Course (PL_CRS) data were generally reliable except when the ship was nearly stationary. The PL_CRS data were derived from a GPS sensor. PL_CRS data from this type of GPS sensor are expected to fluctuate greatly when the platform speed is less than 2 m/s. However, there were a number of occurrences throughout the 1997 and 1998 cruises where the PL_CRS data contained several successive data points with the same value. These PL_CRS data were highly suspect and assigned the K flag by the DQE. Caution is advised when utilizing PL_CRS data from these flagged periods. PL_CRS was one of the parameters used to calculate the True Winds. Therefore, the corresponding DIR and SPD data were flagged K during these occurrences. Another problem with the PL_CRS variable was associated with the flat-line errors that occurred during portions of the 97-B and 98-F cruises (see Section 3.3). The erroneous flat-line data were flagged J by the DQE.

The Platform Speed Over Ground (PL_SPD) data from the 1997 and 1998 Revelle cruises were extremely noisy at times and the DQE recommends these data be smoothed or filtered. During portions of the 97-B and 98-F cruises, flat-line errors occurred with the PL_SPD data as noted in Section 3.3. The erroneous flat-line data were flagged J by the DQE.

The overall quality of the Platform Relative Wind Direction (PL_WDIR) and Platform Wind Speed (PL_WSPD) data was good. One notable problem associated with the PL_WDIR data occurred when the PL_WSPD, DIR and SPD variables reported missing data. When missing data were reported by these variables, the corresponding PL_WDIR consistently reported 180 degrees. The PL_WDIR data were highly suspect during these occurrences and flagged K by the DQE. A problem identified with the PL_SPD data was associated with flow distortion. Flow distortion occurred when the winds were blocked by the ship's superstructure before reaching the bow-mounted wind instruments. When the PL_WDIR was from the stern (~180 degrees), the PL_SPD values dropped an average of 3 to 10 m/s before reaching the bow-mounted anemometer as shown in Figure 3. The suspect PL_WSPD values during these occurrences were flagged K by the DQE. PL_WDIR and PL_WSPD were two of the parameters used to calculate True Winds. Therefore, the corresponding DIR and SPD data were flagged K during these occurrences.

Figure 3. Flow Distortion (between 17 UTC and 18 UTC).

Overall, the Earth Relative Wind Direction and Speed (DIR and SPD) data were highly suspect with several major problems identified by the DQE. The true wind variables, DIR and SPD were calculated by the Data Center. Details of the true wind calculation can be found in Smith et al. (1999). The first problem was directly related to the PL_CRS data reported by the GPS sensor at low ship speeds as noted in Section 4.3. Platform Course was a parameter involved in calculating the true wind. Therefore, when the PL_CRS data reported suspect data, the corresponding DIR and SPD data were assigned the K flag. Additional problems related to DIR and SPD flat-line errors occurred during portions of the 97-B and 98-F cruises as noted in Section 3.3. The erroneous DIR and SPD data were assigned the J flag by the DQE.

The third problem occurred at various times when the Revelle changed course and heading. The ship movement was reflected in the DIR and SPD data. This may have been caused by a distortion of the wind flow around the various structures on the ship or by the hull of the ship itself. When this occurred, K flags were assigned to the DIR and SPD data. Flow distortion was clearly identifiable when the Platform Relative Wind Direction (PL_WDIR) was from the stern (~180 degrees). During these occurrences, the SPD values dropped approximately 3 to10 m/s from the data trend. The anemometer was mounted on a mast at the bow of the ship, although the height of the instrument was not included in the metadata furnished by the data provider. The decreased wind speeds were caused by flow distortion as the winds were diverted around the ship's superstructure before reaching the bow-mounted anemometer. These SPD values were suspect and assigned K by the DQE. An additional problem with the true wind data was associated with noisy platform heading data as described in Section 4.2 of this report. Platform heading was one of the parameters involved in calculating the true winds. Therefore, when the noisy PL_HD data were flagged K the corresponding DIR and SPD data were also flagged K.

During the end of the 97-A and 98-B cruises, the true winds became highly erratic as the Revelle neared port. The erratic trend was likely due to ship's close proximity to land. These values were suspect and flagged K by the DQE. The preprocessor assigned several E flags to DIR values that failed the resultant wind computation check. This check involved comparing the Data Center computed true winds to the reported true winds. A failed test occurred when the wind direction difference was greater than 10 degrees or the wind speed difference was greater than 5 m/s. During these occurrences, the corresponding SPD values were less than 1 m/s. Caution is advised when utilizing DIR and SPD data during these periods. Lastly, ship movement involving a sudden change in speed or direction caused numerous acceleration spikes in the DIR and SPD data (Smith et al.1999). These acceleration spikes were flagged S by the DQE. The overall quality on the SPD and DIR data from the 1997 and 1998 Revelle cruises was highly suspect and the user is advised to use extreme caution when utilizing DIR and SPD data.

The overall quality of the Pressure (P) data was generally good despite the high percentage of G flags applied by the preprocessor. The preprocessor assigned the G flag to data that were more than four standard deviations from the climatological mean. The majority of G flags were applied to pressure data while the Revelle was located in the Antarctic region. Climatology for this region is questionable and these pressure data were believed to be valid. The majority pressure data throughout all the 1997 and 1998 Revelle cruises were very noisy and the data spikes were too numerous to flag all of them. The most significant spikes were flagged S and the more extensive areas of noisy data were flagged K by the DQE. It is recommended that the user filter the pressure data.

Another problem identified with the pressure data occurred when the pressure values deviated 0.5 to 1.5mb from the average pressure trend for a short period, generally less than 10 minutes. There were several instances when the pressure data dropped slightly during PL_WSPD gusts and returned to the normal trend following the gust. At other times, the pressure values slightly increased from the average pressure trend when the PL_WSPD values dropped to 1 m/s or less. Occasions when the pressure values exhibited these deviations from the average pressure trend were few and these suspect areas were flagged K by the DQE.

During the 97-B cruise, the Revelle encountered a deep low pressure system on 97/10/24. At this time, the ship was located near 60 degrees S latitude and 160 degrees W longitude. The pressure dropped from approximately 970 to 950 mb in 10 hours with wind gusts in excess of 30 meters per second during this event. The I flag was assigned to the lowest pressure (~950mb) and peak wind values to note this interesting event as indicated in Figure 4.

Figure 4. Deep low pressure system encountered by the Revelle during the 97-B cruise.

The overall quality of the Air Temperature (T) data and Humidity Sensor Air Temperature (T2) data was generally good despite the high percentage G flags applied by the preprocessor. The majority of G flags were applied to the T and T2 data while the Revelle was located in the Antarctic region. The climatology for this region is questionable and in most cases, the data appeared to be valid.

A specific problem identified with the air temperature data occurred when the PL_WDIR was from the stern of the ship (~180 degrees). The air temperature sensors were located on a mast at the bow ship. When the PL_WDIR was from the stern, the air temperature data (T and T2) were warmed approximately 0.5 to 1.5 degrees Celsius as shown in Figure 5. The exact height of the temperature sensors was not included in the metadata furnished by the data provider. However, the DQE determined that the most likely cause of the rise in T and T2 data was the ship exhaust modifying the air temperatures when the PL_WDIR was from the stern. These suspect T and T2 values were flagged K by the DQE. There was also some evidence of radiational warming when the PL_WSPD dropped to less than 1 m/s. During these brief occurrences, the air temperature increased ~0.4 degrees Celsius. These T and T2 values were flagged K by the DQE. The associated RH values were also flagged K during these periods.

During all the 1997 and 1998 Revelle cruises, the T and T2 data were expected to be nearly identical, generally within plus or minus 1.0 degree Celsius. However, there were several occasions when the T and T2 data deviated up to 4.0 degrees Celsius from one another. The cause of these small deviations in temperature data was not determined and both T and T2 values were flagged K.

Under normal conditions, the air temperature data and RH data were expected to display inverse data trends. However, during the 98-F cruise, there were numerous occasions where the T2 data trend was nearly identical to the RH trend as indicated in Figure 5. There appeared to be a humidity sensor malfunction and these data were flagged J by the DQE.

Figure 5. Ship exhaust effects on temperature and relative humidity data.

The Relative Humidity (RH) data were affected by the same radiational warming and ship exhaust problems as noted in Section 4.8. The RH values dropped slightly below the surrounding RH trend when the T and T2 values rose as a result of ship exhaust or radiational warming. The RH values were flagged K during these occurrences.

During the 97-C cruise, the RH data were highly suspect and erroneous. On 97/12/16, the RH sensor recorded an uncharacteristic smooth trend of approximately 90 percent between 00 UTC and 06 UTC then the RH values steadily dropped to near 18 percent by 2359 UTC. For the remainder of the 97-C cruise, the RH trends were erratic and unrealistic with RH values ranging from below zero percent to over 100 percent. Highly suspect RH values were flagged K while erroneous values were flagged J by the DQE. There was an apparent RH sensor malfunction and the user is advised to utilize extreme caution when utilizing RH data from the 97-C cruise. The serious RH sensor problems continued during the 98-A and 98-B cruises and the RH data from these cruises were deleted from the final data set.

Under normal conditions, the RH and temperature data were expected to display inverse data trends as noted in Section 4.8. However, during portions of the 98-F cruise, the RH data trend was nearly identical T2 data trend shown in Figure 6. These erroneous RH trends were apparently caused by a RH sensor malfunction and the RH values were assigned the J flag by the DQE.

Figure 6. Erroneous T2 and RH data.

The overall quality of the Precipitation (PRECIP) data was very poor. There were extensive problems associated with the PRECIP data recorded by the self-siphoning rain gauge. The self-siphoning rain gauge is designed to fill to 50 mm, and then rapidly drain back to zero mm. However, there were numerous occasions when the PRECIP levels dropped erratically before reaching the 50 mm level. During the 97-B cruise on 97/10/24, the Revelle encountered a deep low pressure system with strong winds in excess of 30 meters per second. During this event, the precipitation trend became noisy and erratic. The K flag was assigned to PRECIP data that were noisy or decreased at a rate of 2 mm or less per day. When the drop in PRECIP levels exceeded this rate, the J flag was assigned to the PRECIP data. Throughout the remainder of the 97-B cruise, there were numerous data spikes and highly suspect data. The user is advised to utilize 97-B PRECIP data with extreme caution.

The overall quality of the PRECIP data from the 97-C cruise and all 1998 cruises was very poor. The precipitation gauge experienced extensive leakage and/or sensor malfunction problems (see Figure 1). The problems with the PRECIP data varied, from slow steady leaks to rapid drops in the data. At other times, the PRECIP data trend was erratic with rapid fluctuations. Accurate precipitation rates and accumulation amounts could not be determined due to the extensive amount of highly suspect and erroneous PRECIP data. Therefore, the PRECIP data from the 97-C and all 1998 cruises were removed from the final data set.

The overall quality of the Atmospheric Radiation (RAD) data was generally good. The main problem with the RAD data was the extensive amount of negative values reported by the RAD sensor during the overnight hours. The negative RAD values were outside of realistic range bounds and flagged B by the preprocessor. There were also several RAD values above 1400 Watts per meter squared recorded during the daylight hours. These extreme RAD values were also assigned the B flag. In some cases, the B flag was overwritten with the S (spike) flag by the DQE.

A large amount of data was removed from the final 1997 and 1998 Revelle data sets due to extensive missing, highly suspect, or erroneous data. These data included: all Sea Temperature (TS), Longwave Atmospheric Radiation (RAD2), and portions of the Precipitation (PRECIP) and Relative Humidity (RH) data. The overall quality of the Revelle IMET data ranged from good to very poor depending on the variable. The Latitude (LAT) and Longitude (LON) data were generally reliable except for portions of the 97-B and 98-F cruises. The Platform Heading (PL_HD), Platform Speed (PL_SPD), and Pressure (P) variables exhibited noise problems throughout all the 1997 and 1998 cruises. The user is advised to smooth or filter the data from these variables. The Platform Course (PL_CRS) data were generally reliable except when the ship was close to stationary. The user is advised to utilize PL_CRS data with caution during these occurrences. The true wind variables, Earth Relative Wind Direction (DIR) and Earth Relative Wind Speed (SPD), were highly questionable with approximately fourteen percent of the true wind data being flagged. The true wind data exhibited a variety of problems as detailed above and should be used with extreme caution. Overall, the quality of the Air Temperature (T) data and Humidity Sensor Air Temperature (T2) data were generally good except for suspect data caused by ship exhaust or radiational warming. During the 98-F cruise, the T2 data trend significantly deviated from the T trend and was nearly identical to the RH data trend. The erroneous T2 and RH data were apparently caused by a humidity sensor malfunction. The quality of the RH data ranged from good to very poor depending on the cruise. There were major problems with RH data from the 97-C cruise as the sensor reported highly suspect and erroneous values, and erratic trends. The user is advised to utilize RH data from the 97-C cruise with extreme caution. The quality of the RH data from the 98-A and 98-B cruises was extremely poor and the RH data from these two cruises were deleted from the final data set. The overall quality of the Precipitation (PRECIP) data from the Revelle was very poor. There were extensive problems with the self-siphoning rain gauge and accurate precipitation rates and accumulation amounts could not be determined. Therefore, all PRECIP data were removed from the final data set except for the PRECIP data from the 97-A and 97-B cruises. The overall quality of the Atmospheric Radiation (RAD) was generally good with no serious problems identified. All Longwave Atmospheric Radiation (RAD2) data were removed from the final data set due to the extensive amount of highly suspect and erroneous values recorded by the RAD2 sensor.

The quality of surface meteorological data analysis provided by the RVSMDC would be greatly improved if the raw data furnished by the data provider were more complete in regards to the instruments that measure and record the meteorological variables. Instrument information such as instrument type, location, and height are necessary for a more accurate analysis of surface meteorological data provided by the RVSMDC. We recommend adding this metadata to the header portion of the Revelle IMET data files.

References:

Smith, S.R., C. Harvey, and Legler, D.M., 1996: Handbook of Quality Control Procedures
     and Methods for Surface Meteorology Data. WOCE Report No. 141/96, Report WOCEMET
     96-1, Center for Ocean-Atmospheric Prediction Studies, Florida State University,
     Tallahassee FL 32306-2840

Smith, S.R., M.A. Bourassa, and Sharp, R.J., 1999: Establishing More Truth in True Winds.
     J. Atmos. Oceanic Technol., 16, 939-952.