DYCOMS-II Satellite: GOES-10 Five Channel 4KM Sectored Data in Hierarchical Data Format (HDF) 1.0 General Information The GOES-10 Five Channel 4KM Sectored Data in Hierarchical Data Format (HDF) is one of several satellite data sets collected as part of the Dynamics and Chemistry of Marine Stratocumulus Phase II: Entrainment Studies (DYCOMS-II) project by the University Corporation for Atmospheric Research/Joint Office for Science Support (UCAR/JOSS). UCAR/JOSS utilized its SeaSpace system to retrieve data from the Geostationary Operational Environmental Satellite 10 (GOES-10) and developed sectors over the DYCOMS-II region. This data set consists of the sectored 4-km data from Channel 1 (visible; 0.55-0.75 um), Channel 2 (Shortwave IR;3.8-4.0um), Channel 3 (Water vapor;6.5-7.0 um), Channel 4 (IR1; 10.2-11.2 um), and Channel 5 (IR2;11.5-12.5 um) of the GOES-10 satellite. The data cover the period from 1 - 30 July 2001 and cover the DYCOMS-II sector (27-37N and 114-126W). During the times the NSF/NCAR C-130 aircraft was in flight (see the web page http://www.joss.ucar.edu/dycoms/catalog/missions/ for flight times) the data are typically available every 15 minutes, but rapid scan imagery are also often available (most often during the afternoon and evening hours). For the times that the NSF/NCAR C-130 was not in flight data are available every three hours (00, 03, 06, ... UTC). All images are in HDF format. A brief description of HDF is given in Section 3.1. 2.0 Data Contact Scot Loehrer (loehrer@ucar.edu) 3.0 Product Information UCAR/JOSS utilized its SeaSpace system to retrieve data from the Geostationary Operational Environmental Satellite 10 (GOES-10) and developed sectors over the DYCOMS-II region. This data set consists of the sectored imagery from Channels 1, 2, 3, 4, and 5 (visible, shortwave IR, water vapor, IR1, and IR2 respectively) of the GOES-10 satellite converted to HDF format. During DYCOMS-II field operations, GOES-10 imagery ingested by the SeaSpace system was archived in real time in Terascan Data Format (TDF) for later processing. From the archived GOES-10 imagery (TDF), sectors covering the DYCOMS-II region were generated using the SeaSpace "gsubset" utility. These sectors were converted to HDF format using the SeaSpace system's "tdftohdf" utility. Each HDF file consists of data for every line and sample pixel contained in original TDF file. In order to geo-reference the image data, two data fields containing latitude and longitude for every line and sample were added to the HDF files using the SeaSpace utility "angles". 3.1 Brief Description of Hierarchical Data Format (HDF) The following sections provide a brief description of the Hierarchical Data Format or HDF. A more complete summary and technical information regarding HDF is available from The National Center for Supercomputing Applications (NCSA) HDF homepage at "http://hdf.ncsa.uiuc.edu". Hierarchical Data Format (HDF) is a file format developed by The National Center for Supercomputing Applications (NCSA) at the University of Illinois Urbana-Champaign Campus (UIUC) and was designed to accommodate scientific data sets. HDF facilitates the sharing of data in a distributed environment and is widely used within the scientific community. ("A Short introduction to HDF given at the Workshop on Multidisciplinary Applications and Interoperability, Wright Patterson Air Force Base, May 1997. retrieved February 4,2002 from http://hdf.ncsa.uiuc.edu/training/hdfintro_mapint97/sld002.htm). HDF can accommodate large data sets, many different data types and structures, and numerous metadata types. HDF binaries and libraires are portable, it's data storage is fast and efficient, and additional features can be added as needed. Furthermore, HDf is freely available and uses open standards. ("A Short introduction to HDF given at the Workshop on Multidisciplinary Applications and Interoperability, Wright Patterson Air Force Base, May 1997. retrieved February 4,2002 from http://hdf.ncsa.uiuc.edu/training/hdfintro_mapint97/sld003.htm). Many NetCDF utilities (eg ncdump) will work on HDF files if compiled and linked to the HDF library. Precompiled HDF utilities and binaries for various platforms are available for download from NCSA's HDF homepage at http://hdf.ncsa.uiuc.edu. The following examples used ncdump compiled for Solaris and downloaded from NCSA's HDF web site.HDF files can also be read by the Interactive Data Language (IDL). Moreover, the HDF packages includes C and Fortran 77 routines which can read and write HDF files as well as perform mathematical operations using the HDF data fields. 3.1 Sample HDF file Below are samples from the header and data records contained in a HDF satellite image file. The header describes the image dimensions, the type and number of variables contained in the HDF file, the units foreach variable, and any scaling, offset, or error factors needed when rederiving calibrated variables. 3.2 DYCOMS-II GOES-10 4KM Multi-Channel HDF file format netcdf g10.2001193.2100.4KM.dycoms { This section details dimensions and channels for each image contained in the HDF file. For the DYCOMS-II 4KM imagery, each image consists of 252 lines by 551 samples (except for the water vapor channel which consists of 126 lines by 551 samples). Visible data is denoted by the "gvar_ch1" descriptor, Shortwave IR by "gvar_ch2", Water vapor by "gvar_ch3", IR1 by "gvar_ch4", and IR2 is denoted by "gvar_ch5". dimensions: gvar_ch1_line = 252 ; gvar_ch1_samp = 551 ; gvar_ch2_line = 252 ; gvar_ch2_samp = 551 ; gvar_ch3_line = 126 ; gvar_ch3_samp = 551 ; gvar_ch4_line = 252 ; gvar_ch4_samp = 551 ; gvar_ch5_line = 252 ; gvar_ch5_samp = 551 ; This section describes how the image data for the visible (gvar_ch1), shortwave IR (gvar_ch2), water vapor (gvar_ch3), IR1 (gvar_ch4), and IR2 (gvar_ch5) channels are stored in the data section of the HDF file. The image data is in units of "albedo*100%" for the visible channel (gvar_ch1) and "temp_deg_c" for the remaining channels (gvar_ch2, gcar_ch3, gvar_ch4, and gvar_ch5). The relationship between a value iy stored in the data set and the actual value y is defined as: y = scale_factor * (iy - add_offset) The variables add_offset_err and scale_factor_err contain the potential errors in add_offset and scale_factor respectively. variables: Visible Channel short gvar_ch1(gvar_ch1_line, gvar_ch1_samp) ; gvar_ch1:long_name = "gvar_ch1" ; gvar_ch1:units = "albedo*100%" ; gvar_ch1:valid_range = -32768s, 32767s ; gvar_ch1:_FillValue = -32768s ; gvar_ch1:scale_factor = 0.009999999776482582 ; gvar_ch1:scale_factor_err = 0. ; gvar_ch1:add_offset = 0. ; gvar_ch1:add_offset_err = 0. ; gvar_ch1:calibrated_nt = 22 ; short gvar_ch2(gvar_ch2_line, gvar_ch2_samp) ; gvar_ch2:long_name = "gvar_ch2" ; gvar_ch2:units = "temp_deg_c" ; gvar_ch2:valid_range = -32768s, 32767s ; gvar_ch2:_FillValue = -32768s ; gvar_ch2:scale_factor = 0.009999999776482582 ; gvar_ch2:scale_factor_err = 0. ; gvar_ch2:add_offset = 27315. ; gvar_ch2:add_offset_err = 0. ; gvar_ch2:calibrated_nt = 22 ; short gvar_ch3(gvar_ch3_line, gvar_ch3_samp) ; gvar_ch3:long_name = "gvar_ch3" ; gvar_ch3:units = "temp_deg_c" ; gvar_ch3:valid_range = -32768s, 32767s ; gvar_ch3:_FillValue = -32768s ; gvar_ch3:scale_factor = 0.009999999776482582 ; gvar_ch3:scale_factor_err = 0. ; gvar_ch3:add_offset = 27315. ; gvar_ch3:add_offset_err = 0. ; gvar_ch3:calibrated_nt = 22 ; short gvar_ch4(gvar_ch4_line, gvar_ch4_samp) ; gvar_ch4:long_name = "gvar_ch4" ; gvar_ch4:units = "temp_deg_c" ; gvar_ch4:valid_range = -32768s, 32767s ; gvar_ch4:_FillValue = -32768s ; gvar_ch4:scale_factor = 0.009999999776482582 ; gvar_ch4:scale_factor_err = 0. ; gvar_ch4:add_offset = 27315. ; gvar_ch4:add_offset_err = 0. ; gvar_ch4:calibrated_nt = 22 ; short gvar_ch5(gvar_ch5_line, gvar_ch5_samp) ; gvar_ch5:long_name = "gvar_ch5" ; gvar_ch5:units = "temp_deg_c" ; gvar_ch5:valid_range = -32768s, 32767s ; gvar_ch5:_FillValue = -32768s ; gvar_ch5:scale_factor = 0.009999999776482582 ; gvar_ch5:scale_factor_err = 0. ; gvar_ch5:add_offset = 27315. ; gvar_ch5:add_offset_err = 0. ; gvar_ch5:calibrated_nt = 22 ; The following used .../4.1r3_solaris/bin/ncdump -v gvar_ch2 g10.2001182.0000.4KM.dycoms.hdf . This is a sample of the gvar_ch2 (shortwave IR) data field. data: gvar_ch2 = 28933, 28933, 28916, 28883, 28883, 28850, 28833, 28816, 28799, 28799, 28781, 28764, 28746, 28764, 28746, 28746, 28746, 28729, 28746, 28746, 28729, 28711, 28675, 28675, 28675, 28693, 28675, 28675, 28657, 28657, 28639, 28639, 28621, 28639, 28621, 28584, 28639, 28602, 28621, 28584, 28584, 28621, 28621, 28621, 28621, 28621, 28602, 28602, 28621, 28584, 28584, 28602, 28639, 28639, 28565, 28584, 28602, 28584, 28565, 28565, 28584, 28584, 28584, 28584, 28565, 28546, 28565, 28546, 28565, 28565, 28584, 28584, 28565, 28546, 28565, 28527, 28508, 28508, 28508, 28527, 28527, 28527, 28527, 28508, 28527, 28508, 28546, 28508, 28527, 28527, 28565, 28565, 28546, 28546, 28546, 28546, 28546, 28546, 28527, 28527, 28508, 28489, 28489, 28489, 28489, 28489, 28489, 28470, 28470, 28430, 28430, 28430, 28430, 28411, 28430, 28430, 28391, 28371, 28371, 28430, 28430, 28391, 28350, 28371, 28350, 28350, 28350, 28350, 28371, 28371, 28391, 28391, 28371, 28371, 28371, 28371, 28371, 28350, 28330, 28350, 28371, 28371, 28371, 28350, 28371, 28371, 28371, 28371, 28371, 28391, 28391, 28350, 28330, 28246, 28268, 28268, 28268, 28268, 28246, 28225, 28225, 28268, 28268, 28246, 28225, 28246, 28246, 28288, 28309, 28330, The following used .../4.1r3_solaris/bin/ncdump -h g10.2001182.0000.4KM.dycoms.hdf. float latitude(gvar_ch4_line, gvar_ch4_samp) ; latitude:long_name = "latitude" ; latitude:units = "degrees" ; latitude:valid_range = -3.4028235e+38f, 3.4028235e+38f ; latitude:_FillValue = -3.4028235e+38f ; latitude:scale_factor = 1. ; latitude:scale_factor_err = 0. ; latitude:add_offset = 0. ; latitude:add_offset_err = 0. ; latitude:calibrated_nt = 5 ; The following used .../4.1r3_solaris/bin/ncdump -v latitude g10.2001182.0000.4KM.dycoms.hdf . This is a sample of the latitude data field. data: latitude = 38.27996 , 38.28038 , 38.2808 , 38.28123 , 38.28166 , 38.28209 , 38.28252 , 38.28296 , 38.28339 , 38.28382 , 38.28426 , 38.2847 , 38.28514 , 38.28559 , 38.28603 , 38.28648 , 38.28692 , 38.28737 , 38.28782 , 38.28828 , 38.28873 , 38.28918 , 38.28964 , 38.2901 , 38.29056 , 38.29102 , 38.29148 , 38.29195 , 38.29242 , 38.29288 , 38.29336 , 38.29383 , 38.2943 , 38.29477 , 38.29525 , 38.29573 , 38.29621 , 38.29669 , 38.29717 , 38.29766 , 38.29814 , 38.29863 , 38.29912 , 38.29961 , 38.3001 , 38.30059 , 38.30109 , 38.30159 , 38.30209 , 38.30259 , 38.30309 , 38.30359 , 38.3041 , 38.3046 , 38.30511 , 38.30562 , 38.30614 , 38.30665 , 38.30716 , 38.30768 , 38.3082 , 38.30872 , 38.30924 , 38.30976 , 38.31029 , 38.31081 , 38.31134 , 38.31187 , 38.3124 , 38.31293 , 38.31347 , 38.314 , 38.31454 , 38.31508 , The following used .../4.1r3_solaris/bin/ncdump -h g10.2001182.0000.4KM.dycoms.hdf. float longitude(gvar_ch4_line, gvar_ch4_samp) ; longitude:long_name = "longitude" ; longitude:units = "degrees" ; longitude:valid_range = -3.4028235e+38f, 3.4028235e+38f ; longitude:_FillValue = -3.4028235e+38f ; longitude:scale_factor = 1. ; longitude:scale_factor_err = 0. ; longitude:add_offset = 0. ; longitude:add_offset_err = 0. ; longitude:calibrated_nt = 5 ; The following used .../4.1r3_solaris/bin/ncdump -v longitude g10.2001182.0000.4KM.dycoms.hdf . This is a sample of the longitude data field. data: longitude = -128.1229 , -128.0952 , -128.0675 , -128.0397 , -128.012 , -127.9843 , -127.9565 , -127.9288 , -127.901 , -127.8733 , -127.8455 , -127.8178 , -127.79 , -127.7623 , -127.7345 , -127.7067 , -127.679 , -127.6512 , -127.6234 , -127.5956 , -127.5678 , -127.54 , -127.5122 , -127.4845 , -127.4566 , -127.4288 , -127.401 , -127.3732 , -127.3454 , -127.3176 , -127.2898 , -127.262 , -127.2341 , -127.2063 , -127.1785 , -127.1506 , -127.1228 , -127.0949 , -127.0671 , -127.0392 , -127.0114 , -126.9835 , -126.9557 , -126.9278 , -126.8999 , -126.8721 , -126.8442 , -126.8163 , -126.7884 , -126.7605 , -126.7326 , -126.7047 , -126.6768 , -126.6489 , -126.621 , -126.5931 , -126.5652 , -126.5373 , -126.5094 , -126.4814 , -126.4535 , -126.4256 , -126.3976 , -126.3697 , -126.3417 , -126.3138 , -126.2858 , -126.2579 , -126.2299 , -126.2019 , -126.174 , -126.146 , -126.118 , -126.09 , -126.0621 , -126.0341 , -126.0061 , -125.9781 , The HDF files in this data set contain data from GOES-10 channels 1,2,3,4 and 5 or from the visible, shortwave IR, water vapor, IR1, and IR2 channels respectively in addition to data fields containing latitudes and longitudes for every image value. 4.0 Quality Control Procedures UCAR/JOSS conducted no quality checks on these data. 5.0 File Naming Convention The file names are structured as follows: goes-10.yyyyddd.hhmm.4KM.dycoms.hdf Where goes-10 is the satellite used yyyy is the four digit year ddd is the three digit julian date hh is the hour (UTC) mm is the minute 4KM is the product's resolution The files on the NCAR Mass Store are gzipped and so have a .gz extention on the end of the file name. 6.0 References HDF Reference Manual Version 4.1r2,June 1998, NCSA University of Illinois at Urbana-Champaign National Center for Supercomputing Applications (NCSA) HDF Home page at http://hdf.ncsa.uiuc.edu. 6.1 Links to HDF documentation and sources National Center for Supercomputing Applications (NCSA) HDF Home page at http://hdf.ncsa.uiuc.edu is the best place to begin searching for HDF utilities, binaries, and documentation. This url located at the NCSA's HDF home page contains numerous links to online HDF documentation and tutorials. http://hdf.ncsa.uiuc.edu/present.html This url located contains links to HDF4 documentation, binaries, utilities, etc. http://hdf.ncsa.uiuc.edu/hdf4.html