IHOP 2002 Surface Miscellaneous Composite

1.0 General Description

This dataset contains "Special" observation surface data in University Corporation for Atmospheric Research/Joint Office for Science Support (UCAR/JOSS) Quality Control (QC) format for the following networks:

as well as select stations stripped from the International H2O Project 2002 (IHOP 2002) Mesonet: Hourly Surface Meteorological Composite for the following networks: The "Nominal" hourly surface data for the IHOP 2002 domain and time period are included in the IHOP 2002 Mesonet: Hourly Surface Meteorological Composite.

Data for the IHOP 2002 domain (32N to 42N latitude and 90W to 105W longitude) and time period (13 May 2002 through 25 June 2002) are contained within this dataset. This IHOP 2002 Surface Miscellaneous Composite dataset contains data from 456 stations and is in the same format as the data in the IHOP 2002 Mesonet: Hourly Surface Meteorological Composite. No additional QC was performed on this miscellaneous surface composite by the University Corporation for Atmospheric Research/Joint Office for Science Support (UCAR/JOSS).

Section 2.0 contains a detailed description of the instrumentation, siting, and algorithms used by the source network to collect the data. Section 2.1 contains a detailed description of the format of the composite dataset. See Section 2.2 for information on data processing, and Section 3.0 below for the quality control processing performed by UCAR/JOSS on this dataset. Section 4.0 contains references.

2.0 Detailed Data Description

2.0.1 Automated Weather Observing System (AWOS) Algorithms

The Automated Weather Observing System (AWOS) is a suite of sensors, which measure, collect and disseminate weather data to help meteorologists, pilots and flight dispatchers prepare and monitor weather forecasts, plan flight routes, and provide necessary information for correct takeoffs and landings. The sensors measure weather parameters such as wind speed and direction, temperature and dew point, visibility, cloud heights and types, precipitation, and barometric pressure. The AWOS stations are operated by the Federal Aviation Administration (FAA) and/or state Department of Transportation offices.

All data contained in this composite dataset were collected on stations manufactured by Artais of Columbus, OH, a division of Vaisala. Each station records data at 20-minute frequencies. All data are included in this miscellaneous composite. Precipitation is the accumulated precipitation for the 20 minute period. Hourly AWOS data, including accumulated precipitation for the hour, can be found in the IHOP 2002 Mesonet: Hourly Surface Meteorological Composite. Present weather and Sea level pressure are not reported for the Artais AWOS data.

The following are descriptions of the AWOS station data. Further details can be found in the AWOS Operations Manual (USDOT, 1988).


AWOS takes at least 1-min measurements and computes a 5-min running average. A minimum of four 1-min averages are required to compute a valid 5-min average. 5-min averages are rounded to the nearest degree F. AWOS will report the latest valid 5-min average during the previous 15-min period. If one is not available, the data are reported as "missing". If the 5-min average dew point is 1 or 2 degrees higher than the 5-min average temperature, then the dew point is reported equal to temperature. If the 5-min average dew point exceeds the 5-min average temperature by more than 2 degrees, the dew point is reported as "missing".

Station Pressure and Derived Pressure Elements

AWOS takes 10-sec measurements from at least two independent pressure sensors and computes respective 1-min averages. A minimum of 5 measurements are required to compute a 1-min average. The 1-min averages from each sensor are compared to verify that differences do not exceed 0.04" Hg. If the sensors are in agreement, the lowest pressure reading from all sensors is reported. If the sensor differences exceed 0.04" Hg, the data are reported as "missing". The reported pressure is then used by AWOS in the computation of derived parameters (e.g., altimeter reading). UCAR/JOSS takes the altimeter reading and converts it back to station pressure using the algorithms found in the Smithsonian Meteorological Tables. Sea Level Pressure is then calculated from station pressure using standard GEMPAK algorithms (Unidata, 2003).


AWOS takes 1-sec measurements of wind speed and direction and computes a 2-min running average every 5-sec. Wind direction is rounded to the nearest 10 degrees (magnetic north) and wind speed is rounded to the nearest knot. Note that AWOS makes no correction to true north in the archived data. If the 2-min running average is 2 knots or less, the wind is reported as calm. The gust is computed using the highest 5-sec average wind speed during the past 10-min period. A gust is computed only when the 2-min running average exceeds 9 knots and the highest 5- sec measurement exceeds the 2-min running average by 5 knots (during the past minute). For Artais AWOS only, the raw data are reported with respect to magnetic north. No corrections were applied to these Artais AWOS winds.


AWOS takes 1-min accumulated measurements and computes total precipitation over the period specified in the AWOS selected archival interval (usually 5-min or 20-min). The total accumulation counter is automatically reset each hour.

Present Weather

Present weather is not reported in the AWOS data.

For more information see the Federal Aviation Administration (FAA) AWOS website ( FAA, 2003)

2.0.2 Unidata Local Data Manager (LDM) World Meteorological Organization (WMO) Surface METAR (LDMSFCMETR) Algorithms

The Unidata Local Data Manager (LDM) (Unidata, 2002) distributes World Meteorological Organization (WMO) Surface data. These data are ingested by UCAR/JOSS in ASCII WMO meteorological message structure format (NOAA/NWS, 2002). The primary feedset name is "WMO" which includes Public Product Service (PPS), Domestic Data Service (DDS), High resolution Data Service (HDS), and International Data Service (IDS) feedtypes. Only products that match the patterns ^S[AP].* .... ([0-3][0-9])([0-2][0-9]) and ^SX..81 .... ([0-3][0-9])([0-2][0-9]) are collected. In these patterns, S stands for surface, A for Aviation Routine Reports (FM 15 - METAR), P for Special aviation weather reports (FM 16 - SPECI), and X for miscellaneous text records. For information on the METAR format see the ASOS User's Guide ( NOAA, 2003). Only special data and METAR data that do not fall on the hour are included in this dataset. Hourly METAR data are available in the dataset 'IHOP 2002 Mesonet: Hourly Surface Meteorological Composite'. For 20 minute METAR stations, the observation that falls between 15 minutes before the hour and the hour, inclusive, is included in the hourly dataset. If there is no observation in this time period, then the observation closest to the hour and falling between 1 minute and 15 minutes after the hour is included in the hourly dataset. All other observations are included in this miscellaneous dataset.

This dataset contains ASOS, AWOS, and MANUAL stations. Station ID's are 3 characters long. Some networks use the 4-character ID to refer to these stations. To obtain the 4-character id, prepend a "K". For example, station ABR could also be referred to as station KABR.

Visibility values greater than 99999.99 m (~60SM) do not fit in the data format so they were reset to -999.99 C.

On 10 April 2003, UCAR/JOSS was notified by Daryl Herzmann (akrherz@iastate.edu), program assistant at the Iowa Environmental Mesonet (http://mesonet.agron.iastate.edu), of a problem with precipitation observations from 1-minute Iowa AWOS stations. The 1-minute Iowa AWOS data are available in their "as is" format from UCAR/JOSS for the IHOP period at IHOP_2002 Mesonet: Automated Weather Observing System (AWOS) Iowa 1-min Data . Note that the following 20 1-minute Iowa AWOS stations had erroneous precipitation values as described in the email extract below: ADU, AIO, AWG, CBF, CNC, CSQ, DNS, EOK, FFL, FSW, HNR, ICL, IKV, MPZ, MUT, OXV, PEA, RDK, SDA, TNU.

Message from Daryl Herzmann (akrherz@iastate.edu)
* Program Assistant -- Iowa Environmental Mesonet
* http://mesonet.agron.iastate.edu
Sent 10 April 2003:

Anyway, most (not all) of these stations are outfitted with tipping buckets. All of you have been receiving the AWOS precipitation amounts from the raw METARs, but up until today, there has been a fundamental flaw with the observations. The site resets the hourly precipitation counter at :55 after the hour, but METARs were only relayed at :05, :25 and :45 after the hour. This would result in 10 minutes of missing potential rainfall data every hour.

The Iowa Department of Transportation sends us their 1 minute interval data archive from the network at the end of every month. While looking at this dataset, I noticed this timing problem last summer. The Des Moines WFO and the Iowa Mesonet have been working closely with the DOT to get this problem resolved. And today it has been. The Iowa AWOS sites are now reporting at :15, :35 and :55 after the hour, which eliminates the timing issue.

2.0.3 National Oceanic and Atmospheric Administration (NOAA) Forecast Systems Laboratory (FSL) Meteorological Assimilation Data Ingest System (MADIS)

This dataset is a collection of data from many networks. The data are fed to JOSS over the LDM by the FSL Meteorological Assimilation Data Ingest System (MADIS). Some of these networks are themselves collections of data from unrelated sources. Each of these networks/subnetworks can contain different frequency data and different parameters. This code has been written to process all of this data at once and convert it to JOSS QCF format.


This composite contains "special" and off-hour records. The hourly parameter value, which can be found in the IHOP 2002 Mesonet: Surface Meteorological Composite, is the value closest to the hour that falls at or after :45 minutes and before (but not at) 15 minutes after the hour. All other records are included in this composite. MADIS QC flags are not carried forward to JOSS format. However, the MADIS QC summary value X "Failed QC stage 1", which is a gross limit check, has been used to mask out extremely bad data. If the data fails this check, it is set to missing. Precipitation data has not been included in this composite.

The following networks are represented in this dataset. Latitude/longitude accuracy included in the station list that accompanies this dataset is a best guess from observation of the raw data:

        Network         accuracy
        ==========      ============
        ms:APRSWXN      5
        ms:AWX          2
        ms:GLDNWS       2
        ms:GPSMET       2
        ms:IADOT        4
        ms:KSDOT        4
        ms:MesoWes      2
        ms:RAWS         5
        ms:UDFCD        3

For more information on these networks, see the FSL/MADIS Surface Network Information webpage (FSL, 2003).

2.1 Detailed Format Description

The IHOP 2002 Surface Miscellaneous observation data contains ten metadata parameters and 38 data parameters and flags. The metadata parameters describe the station location and time at which the data were collected. The time of observation is reported both in Universal Time Coordinated (UTC) Nominal and UTC actual time. For the IHOP 2002 Miscellaneous data, the UTC Nominal and UTC actual times are equal to the time of observation and are always identical. Days begin at UTC 0100 and end at UTC 0000 the following day. The table below details the data parameters in each record. Several data parameters have an associated Quality Control (QC) Flag Code which are assigned by the Joint Office for Science Support (JOSS). For a list of possible QC Flag values see the Quality Control section 3.0. No additional QC was performed by UCAR/JOSS.
     Parameters                              Units 
     ----------                              -----
     Date of Observation                     UTC Nominal 
     Time of Observation                     UTC Nominal 
     Date of Observation                     UTC Actual
     Time of Observation                     UTC Actual
     Network Identifier                      Abbreviation of platform name 
     Station Identifier                      Network Dependent 
     Latitude                                Decimal degrees, South is negative
     Longitude                               Decimal degrees, West is negative
     Station Occurrence                      Unitless
     Station Elevation                       Meters 
     Station Pressure, QC flag               Hectopascals (mb) 
     Reported Sea Level Pressure, QC flag    Hectopascals (mb) 
     Computed Sea Level Pressure, QC flag    Hectopascals (mb) 
     Dry Bulb Temperature, QC flag           Celsius 
     Dew Point, QC flag                      Celsius 
     Wind Speed, QC flag                     m/s
     Wind Direction, QC flag                 Degrees 
     Total Precipitation, QC flag            mm
     Squall/Gust Indicator                   Code Value
     Squall/Gust Value, QC flag              m/s 
     Present Weather, QC flag                Code Value 
     Visibility, QC flag                     Meters 
     Ceiling Height (first layer)            Hundreds of feet 
     Ceiling Flag (first layer), QC flag     Code Value 
     Cloud Amount (first layer), QC flag     Code Value
     Ceiling Height (second layer)           Hundreds of feet 
     Ceiling Flag (second layer), QC flag    Code Value
     Cloud Amount (second layer), QC flag    Code Value
     Ceiling Height (third layer)            Hundreds of feet 
     Ceiling Flag (third layer), QC flag     Code Value
     Cloud Amount (third layer), QC flag     Code Value
The list of code values for the Present Weather is too large to reproduce in this document. Refer to WMO, 1988 for a complete list of Present Weather codes.

The code values for the Squall/Gust Indicator are:

     Code      Definition
     ----      ----------
     blank     No Squall or Gust
     S         Squall
     G         Gust
The code values for the ceiling flag Indicator are:
     Code      Definition
     ----      ----------
     0         None
     1         Thin
     2         Clear below 12,000 feet
     3         Estimated
     4         Measured
     5         Indefinite
     6         Balloon
     7         Aircraft
     8         Measured/Variable
     9         Clear below 6,000 feet (AUTOB)
     10        Estimated / Variable
     11        Indefinite / Variable
     12        12-14 reserved
     15        Missing
The code values for the Cloud Amount Indicator are:
     Code      Definition
     ----      ----------
     0         0 ( or clear)
     1         1 okta or less, but not zero or 1/10 or less, but not zero
     2         2 oktas or 2/10-3/10 
     3         3 oktas or 4/10
     4         4 oktas or 5/10
     5         5 oktas or 6/10
     6         6 oktas or 7/10-8/10
     7         7 oktas or more, but no 8 oktas or 9/10 or more, but not 10/10
     8         8 oktas or 10/10 (or overcast)
     9         Sky obscured by fog and/or other meteorological phenomena
     10        Sky partially obscured by fog and/or other meteorological 
     11        Scattered
     12        Broken
     13        13-14 Reserved
     15        Cloud cover is indiscernible for reasons other than fog or
                other meteorological phenomena, or observation is not made.

2.2 Data Remarks

This dataset contains the IHOP 2002 Surface "Special" observation data as well as records stripped from the IHOP 2002 Mesonet: Hourly Surface Meteorological Composite. The IHOP 2002 Mesonet: Hourly Surface Meteorological Composite contains only the "Nominal" hourly observations for the IHOP 2002 domain and time period.

3.0 Quality Control Processing

No quality control (QC) was performed by UCAR/JOSS. The possible quality control flags are listed in Table 3.1 below.

Table 3.1 - Quality Control Flags

     QC Code   Description
     -------   -----------
     U         Unchecked
     G         Good
     M         Normally recorded but missing.
     D         Questionable
     B         Unlikely
     N         Not available or Not observed
     X         Glitch
     E         Estimated
     C         Reported value exceeds output format field size
                or was negative precipitation.
     T         Trace precipitation amount recorded
     I         Derived parameter can not be computed due to
                insufficient data.

4.0 References

Federal Aviation Administration (FAA), Cited 2003: Automated Weather Observing System website [Available online from http://www1.faa.gov/asos/awosinfo.htm]

FSL, cited 2003: Meteorological Assimilation Data Ingest System (MADIS) Surface Network Information [Available online from http://www-sdd.fsl.noaa.gov/MADIS/network_info.html]

NOAA, National Weather Service, Automated Surface Observing System (ASOS), cited 2003: ASOS User's Guide [Available online from http://www.nws.noaa.gov/asos]

NOAA/NWS, cited 2002: WMO Message structure 2000 Paraphrased Version [Available from http://www.nws.noaa.gov/tg/head.html]

Smithsonian Meteorological Tables, Table No. 65, p.269. Smithsonian Institution Press, Washington, D.C., September, 1949.

Unidata, Cited 2002: Unidata LDM [Available online from http://www.unidata.ucar.edu/packages/ldm/]

Unidata, Cited 2003: Unidata GEMPAK/N-AWIPS [Available online from http://www.unidata.ucar.edu/packages/gempak/].

United States Department of Transportation (USDOT), 1988. AWOS Operations Manual, Federal Aviation Administration.

World Meteorological Organization (WMO), 1988: Manual on Codes Volume I, Part B - Binary Codes. WMO, Geneva, Switzerland.