STORM Sounding Composite Dataset This document contains the following sections: I. General Information II. How to Access the Data III. Composite Format Description IV. Processing of Data Included in the Composite Datasets V. Quality Control VI. Instrumentation VII. References I. GENERAL INFORMATION ------------------- The second release (version 2) of the STORM-FEST composite dataset for rawinsonde observations is now available. The composite consists of 10mb vertical resolution soundings appended into daily files. Each file contains all soundings for one day and are sorted by time, station latitude and station longitude. Changes from release 1 to release 2 of the composite: ----------------------------------------------------- - Additional soundings from other upper air stations are included in release 2. These include the NCAR/NSSL special CLASS stations. Rawinsonde data from the NPS "Picket Fence", Military sites, Canadian sites and Dropsonde data (both military and NCAR). - Nominal time of the sounding is included in the class format. The sounding composite dataset contains data from: -------------------------------------------------- NCAR CLASS Soundings Canadian Soundings (AES) NCAR L2D2 Dropsondes NWS Rawinsonde Data Pacific Military Dropwinondes Naval Post Graduate School (NPGS) Picket Fence Soundings Ft. Sill Soundings Flatlands Soundings Data were subject to the following quality control procedures: -------------------------------------------------------------- All soundings were manually checked by inspecting the Skew-T for the sounding. Data points were judged to be either good, unlikely or questionable and were flagged accordingly. Limitations of the composites include: ------------------------------------------------- - The ACARS Soundings were not included. II. HOW TO ACCESS THE DATA ---------------------- These datasets and appropriate documentation may be accessed online via the STORM Data Management System over the Internet at: http://www.joss.ucar.edu/codiac/ III. Composite Format Description ---------------------------- The composite dataset is archived in the standard WMO FM-94 BUFR code format enhanced with a fixed size headers preceeding each BUFR data record. This enhanced BUFR format is referred to as Enhanced-BUFR, or E-BUFR. The headers on BUFR data records consist of date/time and location information to allow for easy sort and extraction of BUFR records without the need to decode the binary BUFR data. BUFR (and E-BUFR) has been designed to be machine independent. The sounding composite dataset may be requested in a variety of formats through the STORM Data Management system known as CODIAC. Supported formats include CLASS, a tabular ASCII format designed for easy readability, and E-BUFR. The following is a description of the sounding composite with specifics for the E-BUFR and CLASS formats. A. Parameters in the Composite datasets. The Composite Format contains the parameters listed below. This format applies to both the hourly and 5-minute composites with one variation. The hourly composite dataset contains the nominal date and time of observation, whereas the 5-minute composite does not. The nominal date and time is the nearest top of the hour time for the observation as compared to the actual time of the observation. Most networks actually take the observation about 5-minutes before the hour, but this varies from network to network and station to station in a network. Parameter Units Nominal Date of Launch UTC (YY/MM/DD) Nominal Time of Launch UTC (HH:MM) Actual Date of Launch UTC (YY/MM/DD) Actual Time of Launch UTC (HH:MM) Network Identifier Abbreviation of platform name Station Identifier Network Dependent Latitude Decimal degrees, South is negative Longitude Decimal degrees, West is negative Station Occurrence Unitless Station Elevation Meters Time Seconds Pressure Millibars Dry-bulb Temperature Degrees C Dew Point Degrees C Relative Humidity Percent U Wind Component Meters / Second V Wind Component Meters / Second Wind Speed Meters / Second Wind Direction Degrees Ascension Rate (dZ) Meters / Second Longitude Degrees Latitude Degrees (depends on the network) (depends on the network) Altitude Meters QC for Pressure Code (see below) QC for Temperature Code (see below) QC for Humidity Code (see below) QC for U Component Code (see below) QC for V Component Code (see below) QC for Ascension Rate Code (see below) Parameter notes: 1. The nominal date and time differs from the actual time of a sounding launch. The nominal data and time is the actual time rounded to the nearest hour. Actual times may vary a few minutes before or after nominal time. 2. The station occurance is a uniqueness code to separate two different stations that may be co-located in the same latitude/longitude point. 3. Not all platforms report these parameters. If not available for a network the value was set to missing and the QC flag to "Not Observed". 4. Currently, QC for Ascension Rate and the two fields between Latitude and Altitude are not used. B. E-BUFR File Structure Every E-BUFR file contains header records at the beginning of the file and data records which follow the header records. The header records contain the information necessary to understand the BUFR encoded data contained in the data records. It includes the BUFR tables for the BUFR descriptors used in the file as well as BUFR code and flag tables used. Also included in the header records is the layout of the data in the data records. This layout consists of a sequence of BUFR descriptors in the order they will be found in the data records. The data records follow the header records in the file. There is one sounding per data record. These records contain a header portion and the data portion. The header portion precedes the data portion and gives the nominal time of the launch and the launch site location. Note that it is the header portion of the data record which is the enhanced portion of BUFR turning BUFR into E-BUFR. The data portion contains the BUFR encoded values of the sounding in the layout descibed in the header records in the beginning of the file. Note that no BUFR descriptors are stored in the data records. For a full description of the BUFR and E-BUFR formats and to understand how BUFR descriptors work see the BUFR and E-BUFR reference manuals. The following describes the BUFR descriptors used in the sounding composite and summarizes the E-BUFR format of the sounding records. The BUFR Descriptors used: -------------------------- # bits F XX YYY Width Scale Ref Unit Description - -- --- ----- ----- ------ ----- ------------------------------------ 0 04 248 12 0 0 yr Nominal year 0 04 249 4 0 0 mon Nominal month 0 04 250 6 0 0 day Nominal day 0 04 251 5 0 0 hr Nominal hour 0 04 252 6 0 0 min Nominal minute 0 05 001 6 5 -9000000 deg Launch Location - Latitude 0 06 001 6 5-18000000 deg Launch Location - Longitude 0 01 252 8 0 0 num Station Occurance 0 01 254 80 0 0 char Network ID 0 01 253 80 0 0 char Site ID 0 07 001 15 0 0 m Launch Altitude 0 04 001 12 0 0 yr Year 0 04 002 4 0 0 mon Month 0 04 003 6 0 0 day Day 0 04 004 5 0 0 hr Hour 0 04 005 6 0 0 min Minute 0 04 006 6 0 0 sec Second 1 13 000 0 0 0 - Delayed replication descriptor 0 31 001 8 0 0 num Delayed descriptor replication factor 0 04 247 17 1 -6000 sec Time displacement from Launch time 0 08 021 5 0 0 sec Time significance qualifier (=3) 0 10 004 14 -1 0 Pa Pressure 0 33 255 4 0 0 code QC flag for pressure 0 07 253 19 1 -400 m Height or altitude 0 12 001 12 1 0 K Dry bulb temperature 0 33 255 4 0 0 code QC flag for temperature 0 12 003 12 1 0 K Dew point 0 33 255 4 0 0 code QC flag for dew point 0 11 002 12 1 0 m/s Wind speed 0 11 001 9 0 0 Deg Wind direction true 0 33 255 4 0 0 code QC flag for winds 0 06 001 26 5-18000000 Deg Longitude 0 05 001 25 5 -9000000 Deg Latitude E-BUFR encoded header records: ------------------------------ The E-BUFR header records, located in the beginning of the E-BUFR file, contain information which describes the format and layout of the file. It also contains the BUFR descriptor tables along with the BUFR code and flag tables that specifically apply to data in the file. It is in these header records where the layout of the data records is described. The layout is described using a sequential list of BUFR codes (as shown above). The sounding composite uses BUFR's delayed replication feature. The descriptor 1 13 000 (listed above) means that there is a replication of 13 BUFR elements. The descriptor 0 31 001 is the replication factor (the value of this field is also in the header records) and tells how many times the replication occurs. The next 13 descriptors are the BUFR elements which are replicated. Note that the EBUFR data records will contain no mention of 1 13 000 or 0 31 001, but instead have the 13 elements repeated N times, where N is the value contained in 0 31 001 in the header records. Also used is the time significance qualifier 0 08 021. This appears only in the header and qualifies the time displacement (0 04 247). It has a value of 3 to mean the time displacement is an accumulated time offset from, for the sounding composite, the launch time. E-BUFR encoded data record: --------------------------- All E-BUFR data records contain a header portion and a data portion encoded in binary. The header portion is unique to E-BUFR. The data portion is standard BUFR format except that the descriptors are described only once in the header records at the beginning of the E-BUFR file. This saves space in the data portion of the file. Every data record in the file has the same fixed sequence of parameters and is of fixed length. Header portion contains: Nominal Launch Date (GMT) Nominal Launch Time (GMT) Launch Site Latitude Launch Site Longitude Launch Site Station Occurance (always 0) BUFR encoded portion contains: Network ID Site ID Time displacement in tenths of seconds Pressure QC flag for pressure Height or altitude Dry bulb temperature QC flag for temperature Dew point QC flag for dew point Wind speed Wind direction QC flag for winds Longitude (at this height) Latitude (at this height) Each of these parameters are repeated the number of times recorded in BUFR parameter 0 31 000 in the header records at the beginning of the file. Each set of these parameters represents the value of the parameters at the next 10mb level. The first set is at the surface level. The second set is at the next 10mb level. It could be less than 10mb from the surface. All other sets are at 10mb increments. The number of sets equals the number of 10mb increments in the sounding plus the surface set of values. Note that all data for a single sounding is in one E-BUFR data record. C. Description of the CLASS Format The 10mb sounding composites for STORM-FEST are optionally available in a format called STORM Class Format, or SCF, derived from the variations of "CLASS Format" used at NCAR and by the National Weather Service. This format is created upon request from the archived E-BUFR file. SCF is an ASCII format consisting of 15 header records for each sounding followed by the data and its QC information at 10 mb increments. Header Records -------------- The Header records (15 total records) contain data type, project ID, site ID, site location, release time, ascension number, surface conditions, termination reason/altitude, and comments. The first 5 header lines contain information identifying the sounding, and have a rigidly defined form. The following 6 header lines are used for auxiliary information and comments about the sounding, and may vary from dataset to dataset. The next line (line 12) contains the Nominal date and time of the launch. The last 3 header lines contain headers for the data columns, to enhance readability. Line 13 holds the field names, line 14 the field units, and line 15 dashes ('-' characters) delineating the extent of the field. The six standard header lines are as follows: Line Label (padded to 35 char) Contents 1 Data Type: Description of type and resolution of data. 2 Project ID: ID of weather project. 3 Launch Site Type/Site ID: Description of launch site. 4 Launch Location (lon,lat,alt): Position of launch site, in format described below. 5 GMT Launch Time (y,m,d,h,m,s): Time of launch, in format: yyyy, mm, dd, hh:mm:ss 12 GMT Nominal Launch Time (y,m,d,h,m,s): Nominal launch time. The launch location is given as: lon (deg/min), lat (deg/min), lon (dec. deg), lat (dec. deg), alt (m) Longitude in deg/min is in the format: ddd mm.mm'W where ddd is the number of degrees (with leading zeros if need be), mm.mm is the decimal number of minutes, and W represents W or E for west or east longitude. Latitude has the same format, except there are only two digits for degrees and W is replaced by N or S for north/south latitude. An example of line 4 would thus be: Launch Location (lon,lat,alt): 096 06.60'W, 39 49.80'N, -96.11, 39.83, 384 The seven non-standard header lines may contain any label and contents. The label is be padded to 35 characters to match the standard header lines. As mentioned above line 12 has been used in the sounding composite for the nominal launch date and time. Currently, the other 6 lines are unused. The three column header lines which describe the data follow the other header lines and preceed the data. Data Records ------------ The data records (1 record per 10-mb level including surface) each contain time from release, pressure, temperature, dewpoint, relative humidity, U and V wind components, wind speed and direc- tion, ascent rate, balloon position data, altitude, and quality control flags (see QC description). Each data line contains 21 fields, separated by spaces, with a total width of 130 characters. The data are right-justified within the fields. All fields have one decimal place of precision, with the exception of latitude and longitude, which have three decimal places of precision. The contents and sizes of the fields are detailed below. The 21 fields that appear in each data line are as follows: Field Format No. Width Parameter Unit Missing Flag 1 6 F6.1 Time Seconds 9999.0 2 6 F6.1 Pressure Millibars 9999.0 3 5 F5.1 Dry-bulb Temperature Degrees C 999.0 4 5 F5.1 Dew Point Degrees C 999.0 5 5 F5.1 Relative Humidity Percent 999.0 6 6 F6.1 U Wind Component Meters / Second 9999.0 7 6 F6.1 V Wind Component Meters / Second 9999.0 8 5 F5.1 Wind Speed Meters / Second 999.0 9 5 F5.1 Wind Direction Degrees 999.0 10 5 F5.1 Ascension Rate (dZ) Meters / Second 999.0 11 8 F8.3 Longitude Degrees 9999.0 12 7 F7.3 Latitude Degrees 999.0 13 5 F5.1 Variable (see below) 999.0 14 5 F5.1 Variable (see below) 999.0 15 7 F7.1 Altitude Meters 99999.0 16 4 F4.1 QC for Pressure Code (see below) 99.0 17 4 F4.1 QC for Temperature Code (see below) 99.0 18 4 F4.1 QC for Humidity Code (see below) 99.0 19 4 F4.1 QC for U Component Code (see below) 99.0 20 4 F4.1 QC for V Component Code (see below) 99.0 21 4 F4.1 QC for Ascension Rate Code (see below) 99.0 where Format is the FORTRAN format that could be used to write the field, and Missing Flag is the missing data flag for that field. Note that the missing data flag consists of just enough 9s to fill the field. Note also that there is a space (FORTRAN format 1X) between each field. A FORTRAN 77 FORMAT statement that conforms to the above would be: 100 format(2(2(F6.1,1X),3(F5.1,1X)),F8.3,1X,F7.3,2(1X,F5.1),1X, + F7.1,6(1X,F4.1)) Fields 13 and 14 are "variable" because true CLASS soundings use these fields for range in km and angle in degrees, respectively, which is information unavailable for NWS soundings. NWS soundings do, however, have elevation in degrees and azimuth in degrees, which are used to fill these fields. Processing program should simply copy the contents of these fields. Currently, these fields will always be missing. Fields 16 through 21 contain the Quality Control information derived at the STORM Project Office. Any QC information from the original sounding is ignored. Currently, field 21 will always be missing. The codes used are as follows: Code Meaning 99.0 Unchecked (QC information is "missing.") ("UNCHECKED") 1.0 Checked, datum seems physically reasonable. ("GOOD") 2.0 Checked, datum seems questionable on physical basis. ("MAYBE") 3.0 Checked, datum seems to be in error. ("BAD") 4.0 Checked, datum is interpolated. ("ESTIMATED") 9.0 Checked, datum was missing in original file. ("MISSING") Example File ------------ The following is a sample portion of a class format file. The data portion is much longer than 80 characters and, therefore, wraps around to a second line. Data Type: CLASS 10 SECOND DATA Project ID: STORMFEST -- BURLINGTON, CO Launch Site Type/Site ID: FIXED, 3V1 Launch Location (lon,lat,alt): 102 17.40'W, 39 14.40'N, -102.29, 39.24, 1286 GMT Launch Time (y,m,d,h,m,s): 1992, 02, 01, 23:00:47 Sonde Type/ID/Sensor ID/Tx Freq: VAISALA RS80-15L 0, 0, 403.05 Met Processor/Met Smoothing: VAISALA PP-11, 20 SECONDS Winds Type/Processor/Smoothing: LORAN-C, ANI-7000, 60 SECONDS Pre-launch Met Obs Source: CAMPBELL SCIENTIFIC CR10 System Operator/Comments: L.MACK, (REPROCESSED),GOOD FLIGHT, NEED TO PUT MORE HEL. IN BAL. / GMT Nominal Launch Time (y,m,d,h,m,s): Nominal launch time. Time Press Temp Dewpt RH Uwind Vwind Wspd Dir dZ Lon Lat Rng Ang Alt Qp Qt Qh Qu Qv Quv sec mb C C % m/s m/s m/s deg m/s deg deg km deg m mb C % m/s m/s m/s ------ ------ ----- ----- ----- ------ ------ ----- ----- ----- -------- ------- ----- ----- ------- ---- ---- ---- ---- ---- ---- -43.0 869.3 12.6 1.1 45.2 -0.2 2.2 2.2 174.5 0.0 -102.290 39.240 999.0 999.0 1286.0 2.0 2.0 2.0 2.0 2.0 2.0 22.7 860.0 15.7 -6.5 21.2 3.6 7.7 8.5 205.1 5.2 -102.288 39.242 999.0 999.0 1377.1 1.0 1.0 1.0 2.0 2.0 99.0 41.9 850.0 15.1 -7.7 20.0 -0.5 9.1 9.1 177.0 4.8 -102.286 39.245 999.0 999.0 1476.0 1.0 1.0 1.0 1.0 1.0 99.0 62.6 840.0 14.2 -8.1 20.6 -1.2 9.2 9.2 172.4 4.9 -102.285 39.247 999.0 999.0 1576.1 1.0 1.0 1.0 1.0 1.0 99.0 . . . etc. D. Special Values 1. Missing Values A value that is missing, or is not observed by the given network is indicated by the value filled with nines, e.g. 999.9. The quality control flag is set to 15 (E-BUFR) or '9.0' (CLASS). 2. Quality-Control Flag Values The quality-control flag is a coded value. Codes and definitions are listed briefly here and are expanded in the section on quality control. BUFR CLASS Code Code Definition 0 99.0 Unchecked 1 1.0 Checked and Good 2 2.0 Checked and Questionable 3 3.0 Checked and Unlikely 8 4.0 Interpolated Value 15 9.0 Missing Value Upon initial conversion of the data all QC flags were initialized to 0 or 99.0 to indicate that the data have not been checked, except for data that is missing (15 or 9.0). 3. Network Identifiers ID Network CLASS NCAR CLASS Soundings AES Canadian Soundings (AES) L2D2 NCAR L2D2 Dropsondes NWS NWS Rawinsonde Data PACIFIC Pacific Military Dropwinondes PICKET Naval Post Graduate School (NPGS) Picket Fence Soundings FT. SILL Ft. Sill Soundings FLATLANDS Flatlands Soundings 4. Station Identifiers The stations and their names and identifiers are listed in a separate file called fest_sites. IV. PROCESSING OF DATA INCLUDED IN THE COMPOSITE DATASETS ----------------------------------------------------- 1. NCAR CLASS data 2. Canadian Sounding data 3. L2D2 data 4. NWS Rawinsone 6-second data 5. Picket Fence Soundings 6. Pacific Military Dropwindsondes 7. Ft. Sill Soundings 8. Flatlands Soundings 9. Conversion of 10 second data to 10mb data Conversion to 10 mb data from 10 second data did not handle a balloon's descent properly. If the balloon bursts and descends continuously, then the conversion handled it. But if the balloon descends some and then rises and falls and rises ... then the conversion output bad data after the bursting point. What this all boils down to is that the conversion expected the pressures in the input to be non-decreasing. (As it turns out, the conversion accepted input with pressures which are non-decreasing until a certain level, and then non-increasing until the end of the file.) But, the descent should never appear as part of the input to the conversion. All data was cut off at the 100 mb level. V. QUALITY CONTROL --------------- A. Description of QC Procedures ---------------------------- Each sounding was quality controlled (QC) by a combination of computer (existing flags) and manual review of the data (both Skew-T and constant pressure analysis). The QC procedure (and resultant flags) included only temperature, dewpoint, wind speed and direction. Data observations for these parameters were flagged as either "good", "questionable", "unlikely", "missing", or "estimated". In many cases the manual QC flag procedure was a subjective decision, so these flags should be treated as a guideline only (with the exception of "missing" ). B. Quality control flags --------------------- Quality control flags are stored with each parameter in the composite dataset. The codes have the following meanings: BUFR CLASS Code Code Definition 0 99.0 Unchecked Datum; QC information is missing. 1 1.0 Checked, datum seems physically reasonable. No physical reason observed to question datum. 2 2.0 Checked, datum seems questionable on a physical basis. The user should exercise caution when using data with this flag. 3 3.0 Checked, datum appears to be in error. The user is strongly encouraged not to use data with this flag. 8 4.0 Checked (Wind data only), wind datum was computed or interpolated from bad or outlier original position data. A least squares fit was applied to a running 2-minute window of original position data, and positions which exceeded 2.5 standard deviations were removed and replaced with the least squares curve fit. Users should exercise caution when using this datum. 15 9.0 Missing VI. INSTRUMENTATION --------------- To be added at a later date. VII. REFERENCES ---------- "E-BUFR Version 0 Technical Reference Manual", December 1991, UCAR/OFPS, Boulder, CO. This document explains the details of the E-BUFR format. A digital version of this document may be retrieved via anonymous FTP. Telnet to 128.117.90.53. The document is named "ebufr.doc" and is in directory "documentation". Manual on Codes, 1988, World Meteorological Organization, Geneva, Switzerland. WMO Publication 306, Supplement No. 3 (VIII.1991) Section FM 94-IX Ext., pages I-Bi--43 to I-Bi--174. Format description, code definitions, network conversion details, quality controls details, etc. are documented in this "readme.sfc" file.