29 OCT 1992 ADM NCAR/RAF C Software for Terascan Satellite Image Files ----------------------------------------------------- The image files are in a format called Terascan Data Format (TDF). TDF is a variation of Common Data Format (TDF). Terascan software was used to "register" the original satellite image. See the APPENDIX below for more information about the registration process. One set if software provided is source code required to generate the program textract.exe. This program extracts a satellite channel's data array from a TDF image file and places the array in an output file. Another set of software provided is source code required to convert the TDF files to network Common Data Format (netCDF). This program is convert.exe. netCDF is more widespread than TDF and software to convert netCDF files to other formats is more likely to exist. See OBTAINING NETCDF below for information on acquiring netCDF software. It is assumed that the user is proficient with the C programming language as well as the UNIX operating system, or can consult with someone who is. 1. CONTENTS ------------ The sample source code contained in extract.tar and convert.tar was developed on a Sun Sparcstation (using the UNIX operating system) with a Sun C Compiler (version 4.1 - non-ANSI). The source code in GP.TAR was provided by SeaSpace. The files needed to compile and link the source code are: image.readme - this file makefile - used to install the software set GP.TAR - source code provided by SeaSpace to generate the Terascan library. Please leave this code as is. Users who tamper with this code (other than to compile with their C compiler) do so at their own risk. extract.tar - source code provided by KuDA to create the program textract.exe. This program extracts a channel's data array from a Terascan Satellite Image File. This program does not scale the data. See the section under Scaling The Output below. Users are encouraged to modify this code to suit their own purposes. convert.tar - source code provide by KuDA to create the program convert.exe. This program converts a file from Terascan data format (TDF) to network Common Data Format (netCDF). The netCDF software library is required to link this code. Users are encouraged to modify this code to suit their own purposes. GP.TAR, extract.tar, and convert.tar are Tape Archive (tar) files that contain C source code and corresponding makefiles. 2. CUSTOMIZING --------------- Depending on what program(s) you need, you will need to edit the corresponding makefiles. You must edit the file makefile to specify the directory path for your system. Edit texmake if you want the program textract.exe. Edit cvtmake if you want the program convert.exe. makefile: You will need to edit a line in the file "makefile". This is the line that specifies in which directory to place the programs. INSTALLDIR = /home/put_your/directory_path/here texmake: you may remove the compiler switch "-DSUN4" from texmake if you are compiling on a workstation other than a Sun Sparcstation: CFLAGS = -I../include -DSUN4 cvtmake: A line in the convert.tar makefile, cvtmake, will need to be edited as well. This is the complete path and file name of the netCDF library libnetcdf.a. If you want the conversion program, you must first make the netCDF library. See OBTAINING NETCDF. ## this is where the netcdf library is: LIBS=/usr/local/netcdf/ldm/lib/libnetcdf.a It is possible that the source code needs some minor modifications to work with your C compiler and/or linker. The #include statements in the code may specify a ".h" file that has a different name than the one you would use for your C compiler. Also, if you have an ANSI C compiler, you may need to edit the forward function prototype lines at the beginning of the modules that make up the extract program. The README file from GP.TAR has additional information concerning the Terascan library. 3. INSTRUCTIONS ---------------- It is strongly recommended that you first make a copy of the disk or tape for backup purposes. Then, copy the files to your working directory. 1. Make a temporary compile/link directory and move to it: amy 1% mkdir tmp; cd tmp 2. If you have acquired the software as tar files (via ftp), go to 2a. Or, if you are acquiring the data from an 8mm (Exabyte) tar tape, go to 2b. 2a. tar files acquired via ftp make sure GP.TAR, convert.tar, extract.tar, and makefile are in the directory: amy/tmp 2% ls -al total 308 drwxrwxr-x 2 amy 512 Sep 24 14:19 ./ drwxr-xr-x 26 amy 2048 Sep 28 10:44 ../ -rw-r--r-- 1 amy 262144 Sep 23 14:30 GP.TAR -rw-rw-r-- 1 amy 16384 Sep 24 12:58 convert.tar -rw-rw-r-- 1 amy 24576 Sep 23 14:30 extract.tar -rw-rw-r-- 1 amy 872 Sep 24 14:18 makefile 2b. 8mm (Exabyte) tar tape To retrieve the first tar set, place the 8mm cartridge into the the 8mm drive and enter (replace /dev/rex0 with the appropriate 8mm device name): tar xvf /dev/rex0 Among the files retrieved from the tape, you should have tape.sh and tape.in. Before you run the shell script, make sure the tape.sh file has execute permissions set. If it doesn't, enter chmod +x tape.sh Run the shell script by entering tape.sh The first files to retrieve are the software. Retrieve all of the software files (the shell script will guide you through this process), then quit the shell script. Now you should have the software spread out in subdirectories. amy/tmp 2% ll drwxrwxr-x 10 amy 512 Oct 29 11:25 ./ drwxrwxrwx 15 root 1024 Oct 29 11:15 ../ -rw-rw-r-- 1 amy 960 Aug 11 13:02 README drwxrwxr-x 2 amy 512 Oct 29 11:16 convert/ drwxrwxr-x 2 amy 512 Oct 29 11:16 examples/ drwxrwxr-x 2 amy 512 Oct 29 11:16 extract/ -rw-r--r-- 1 amy 23226 Oct 29 11:18 image.readme drwxrwxr-x 2 amy 512 Aug 11 11:08 include/ drwxrwxr-x 2 amy 512 Oct 29 11:16 lib/ -rw-rw-r-- 1 amy 1266 Oct 29 11:17 makefile drwxrwxr-x 4 amy 512 Aug 11 10:33 man/ drwxrwxr-x 2 amy 1024 Oct 29 11:16 source/ -rw-rw-r-- 1 amy 7214 Oct 28 15:21 tape.in -rw-rw-r-- 1 amy 7922 Oct 28 16:14 tape.sh -rw-r--r-- 1 amy 6498 Oct 28 16:11 tapelist 3. Edit the makefiles as described in CUSTOMIZING (above). 4. to generate textract.exe (see CONVERT.EXE below if you want to generate the TDF to netCDF conversion program): amy/tmp 5% make install This is what the directory contains after "make install": amy/tmp 6% ls -al total 316 drwxrwxr-x 9 amy 512 Sep 28 10:53 ./ drwxr-xr-x 26 amy 2048 Sep 28 11:00 ../ -rw-r--r-- 1 amy 262144 Sep 23 14:30 GP.TAR -rw-rw-r-- 1 amy 960 Aug 11 13:02 README drwxrwxr-x 2 amy 512 Sep 28 10:53 convert/ -rw-rw-r-- 1 amy 16384 Sep 24 12:58 convert.tar drwxrwxr-x 2 amy 512 Sep 28 10:53 examples/ drwxrwxr-x 2 amy 512 Sep 28 10:53 extract/ -rw-rw-r-- 1 amy 24576 Sep 23 14:30 extract.tar drwxrwxr-x 2 amy 512 Aug 11 11:08 include/ drwxrwxr-x 2 amy 512 Sep 28 10:53 lib/ -rw-rw-r-- 1 amy 872 Sep 24 14:18 makefile drwxrwxr-x 4 amy 512 Aug 11 10:33 man/ drwxrwxr-x 2 amy 1024 Sep 28 10:53 source/ The program textract.exe is in your INSTALLDIR: amy/tmp 7% ls -al ../images total 171 drwxrwxr-x 2 amy 512 Sep 28 10:53 ./ drwxr-xr-x 26 amy 2048 Sep 28 11:00 ../ -rwxrwxr-x 1 amy 57344 Sep 28 10:53 textract.exe* 5. OPTIONAL: To remove subdirectories and recover disk space, leaving textract.exe intact in the INSTALLDIR, enter amy/tmp 7% make clean rm -fr source rm -fr include rm -fr man rm -fr lib rm -fr extract rm -fr examples rm -fr convert rm -f README amy/tmp 8% ls -al total 308 drwxrwxr-x 2 amy 512 Sep 24 14:19 ./ drwxr-xr-x 26 amy 2048 Sep 28 10:44 ../ -rw-r--r-- 1 amy 262144 Sep 23 14:30 GP.TAR -rw-rw-r-- 1 amy 16384 Sep 24 12:58 convert.tar -rw-rw-r-- 1 amy 24576 Sep 23 14:30 extract.tar -rw-rw-r-- 1 amy 872 Sep 24 14:18 makefile 4. TEXTRACT.EXE --------------- Your DMSP or NOAA file may reside anywhere in your filesystem. In the example below, the file is in the images directory. You may include the full path name when you enter the image filename. For example: Enter image filename or quit: /home/amy/image/data/n10.91157.0438r ------------------------------------------------------------------- amy/images 20% textract.exe Enter image filename or quit: n10.91157.0438r DATASET ATTRIBUTES: name: satellite noaa-10 name: pass_date 6 Jun 1991 name: start_time 04:34 UCT name: center_lat 27.525 name: center_lon 50.5 DIMENSIONS: name: line size: 1200 name: sample size: 1200 CHANNELS: 1) name: avhrr_ch1 units: albedo*100% scale: 0.01 offset: 0 2) name: avhrr_ch2 units: albedo*100% scale: 0.01 offset: 0 3) name: avhrr_ch3 units: temp_deg_c scale: 0.01 offset: 0 4) name: avhrr_ch4 units: temp_deg_c scale: 0.01 offset: 0 5) name: avhrr_ch5 units: temp_deg_c scale: 0.01 offset: 0 Enter channel number (1 to 5) or quit: 1 Press RETURN to set output filename to noaa-10.ch1.06Jun91.0434, or enter output filename, or quit: Enter image filename or quit: quit amy/images 21% ls -al noaa-10* -rw-rw-r-- 1 amy 2880000 Aug 25 16:54 noaa-10.ch1.06Jun91.0434 amy/images 22% ------------------------------------------------------------------- 4. SCALE THE TEXTRACT.EXE OUTPUT --------------------------------- Now that you have a file containing the channel's data array, you will probably write a program yourself to use this data. The data in the array may need to be scaled. You must include the code to scale the data in your own program. Or, if you know C, you may edit the module tex_main.c to have it write the data to the file already scaled. In tex_main.c, you would do this in the areas with the text: /************************************************* * extract the array * NE Corner: 57 deg 15.0 min E, 33 deg 30.0 min N * SW Corner: 43 deg 45.0 min E, 21 deg 30.0 min N */ and /* * WARNING: This does not scale the data!!! */ For the NOAA images, multiply each 16-bit integer by 0.01. For the F-series images, if the channel is OLS visible, no scaling is needed. For the OLS infrared, use the formula T = (I - 176.69) / 2.125 where I is the value in the file. In other words, for any value, multiply by the scale and then subtract the offset. Both the scale and the offset are displayed when the program is run: For the NOAA series: CHANNELS: 1) name: avhrr_ch1 units: albedo*100% scale: 0.01 offset: 0 2) name: avhrr_ch2 units: albedo*100% scale: 0.01 offset: 0 3) name: avhrr_ch3 units: temp_deg_c scale: 0.01 offset: 0 4) name: avhrr_ch4 units: temp_deg_c scale: 0.01 offset: 0 5) name: avhrr_ch5 units: temp_deg_c scale: 0.01 offset: 0 For the F series (sometimes the order of the ols_infrared and ols_visible channels are reversed): CHANNELS: 1) name: ols_infrared units: temp_deg_c scale: 0.470588 offset: -83.15 2) name: ols_visible units: scale: 1 offset: 0 5. CONVERT.EXE --------------- You must have the netCDF library from UNIDATA before you can compile and link this program (see OBTAINING NETCDF). To make this code, enter amy/tmp 53% make cvt2netcdf You should find convert.exe in your INSTALLDIR. convert.exe uses command-line arguments. The first argument is the path/filename of the Terascan Data Format file. The second argument is the path/filename of the new converted netCDF file. The converted netCDF file will be about the same size as the TDF file, so make sure there is enough room in your filesystem or you may exceed your disk quota. In the following example, the TDF file is in the directory /home/data/NOAA, and the converted file is to be placed in the same directory as the program convert.exe: ------------------------------------------------------------------- amy/images 10% convert.exe /home/data/NOAA/n9.91067.0248r n9.91067.02484.cdf dimensions: variables: attributes: amy/images 11% ls -l convert.exe* n9.91067.02484.cdf -rwxrwxr-x 1 amy 106496 Sep 29 10:46 convert.exe* -rw-rw-r-- 1 amy 14402620 Sep 29 10:52 n9.91067.0248r.cdf ------------------------------------------------------------------- 6. OBTAINING NETCDF -------------------- The current version of netCDF software is available using anonymous FTP. The netCDF C and FORTRAN interfaces have been tested successfully on the following platforms: - Sun 3, SPARCstation (SunOS) - DEC VAX (VMS, Ultrix) - DECstation (Ultrix) - IBM RISC System 6000 (AIX) - CRAY YMP (UNICOS) - IBM PS/2 (MSDOS, OS/2) UNIX systems can access a compressed tar file (in binary mode) from the file pub/netcdf.tar.Z in the anonymous FTP directory of unidata.ucar.edu. You or your system administrator may want to do this: amy 54% ftp unidata.ucar.edu Connected to unidata.ucar.edu. 220 groucho FTP server (SunOS 4.1) ready. Name (unidata.ucar.edu:amy): anonymous 331 Guest login ok, send ident as password. Password: guest 230 Guest login ok, access restrictions apply. ftp> cd pub ftp> bi ftp> get netcdf.tar.Z ftp> quit amy 55% mkdir /usr/local/netcdf amy 56% mv netcdf.tar.Z /usr/local/netcdf amy 57% cd /usr/local/netcdf amy 58% uncompress netcdf.tar.Z VMS sites can get a backup saveset of the same software from the anonymous FTP directory of laurel.ucar.edu. The software distribution includes the netCDF User's Guide in PostScript form, ready to be printed. 7. QUESTIONS ------------- Please direct any questions to Amy Michaelis, NCAR/RAF, (303)497-1050, or e-mail to amy@chinook.atd.ucar.edu (omnet: ncar.raf attn: Amy Michaelis). APPENDIX ------------------------------------------------------------------- ---------------------- NOTES ---------------------- ------------------------------------------------------------------- FILE NOMENCLATURE: ----------------- For now, file names are composed of the satellite name (e.g. n9, n10, n11, n12, f8, f9, f10, and f11), date (2 digit year and 3 digit day number -- e.g. day 222 = August 10), and GMT. The small "r" on the end just indicated that the image was reinterpolated to a Cartesian lat-long grid. FILE NAMES WILL BE CHANGED IN THE FUTURE to conform with the metadata database. All the imagery being staged to the Kuwait Data Archive (KuDA) is in the process of being incorporated into an overall data base system which includes a wide variety of Persian Gulf data (aircraft obs, soundings, surface reports, etc.). PRE-PROCESSING: -------------- All NOAA AVHRR and DMSP OLS image files in the Kuwait Data Archive have undergone pre-processing. This consists of (1) hand navigation of the image using the Sea Space Terascan software (see ABOUT HAND NAVIGATION below), (2) calibration of the AVHRR IR data to brightness temperatures, (3) possible "patching" of missing data lines -- not done very often, and (4) interpolation to a lat-long grid. The images were all reinterpolated to a Cartesian lat-long grid. The corner points of our geographical analysis area are: NE Corner: 57 deg 15.0 min E, 33 deg 33.0 min N ? (was 30.0) SW Corner: 43 deg 45.0 min E, 21 deg 30.0 min N Within this domain the NOAA AVHRR data has been processed to a 1200 x 1200 grid, and the DMSP OLS data (BOTH visible and IR) has been processed to a 2400 x 2400 grid. This is a "rectangular" map projection, essentially a simple lat-long grid, but with the grid spacings adjusted so that the relative grid increments correspond to equal spacings in km at the center of the image (roughly 550 m for the dmsp data, and 1 or 1.1 km for the noaa data). NOTE: The sensor resolution for the dmsp data is typically 500 m (at nadir) for the visible ols data, but only 2.5 km for the IR data. The sensor resolution varies as a function of the distance from nadir. In both cases, however, the data has been gridded (or over-gridded) to the same 500 m grid. The center coordinate is at the center of the four adjacent pixels (since there are an even number of samples). The nominal grid spacing in both the E-W direction (longitude difference between adjacent pixels) and in the N-S direction (latitude difference between adjacent pixels) is given below. The grid spacing for the NOAA data is: 13.50 degrees / (1200-1 grid intervals) = 0.011259 degrees of longitude 12.05 degrees / (1200-1 grid intervals) = 0.010050 degrees of latitude The grid spacing for the DMSP data is: 0.004023 degrees of latitude 0.005627 degrees of longitude These can be converted to kilometer intervals using any earth model or geoid of choice. Using the Smithsonian Meteorological Tables (Tables 162 and 163) for the International Ellipsoid of 1924, the grid spacings are nominally: NOAA: N-S "y" interval 1.113 km at 20-21 degrees N 1.114 km at 27-28 degrees N 1.115 km at 32-33 degrees N E-W "x" interval 1.171 km at 21 degrees N 1.118 km at 27 degrees N 1.052 km at 33 degrees N DMSP: N-S "y" interval 0.556 km at 20-21 degrees N 0.557 km at 27-28 degrees N 0.557 km at 32-33 degrees N E-W "x" interval 0.585 km at 21 degrees N 0.558 km at 27 degrees N 0.526 km at 33 degrees N Users are cautioned that in general, the inaccuracies in the orbital elements, satellite reference time standards, roll-pitch-yaw estimates, sensor alignment, and the interactive hand navigation steps are likely to be much more significant than inaccuracies in an earth model. The following Fortran subroutine provided by SeaSpace may be used to compute Cartesian coordinates (model unknown). > Earth radius is 6378.145 km > > C ll_xyz.f 1.2 1/30/90 > SUBROUTINE ll_xyz(rlat, rlon, x, y, z) > C > C Converts geodetic latitude,longitude to Cartesian > C coordinates. (Earth radius at equator assumed = 1) > C > C Input > C rlat - double precision, latitude (radians) > C rlon - double precision, longitude (radians) > C > C Output > C x,y,z - double precision, (x,y,z) coordinates > C > DOUBLE PRECISION rlat, rlon, x, y, z > C > DOUBLE PRECISION flat, flat2, temp, denom > PARAMETER (flat = 1.D0/298.25D0) > PARAMETER (flat2 = 2.*flat - flat**2) > C > temp = DSIN(rlat) > denom = DSQRT(1.D0 - flat2*temp**2) > z = ((1.D0 - flat)**2)*temp/denom > temp = DCOS(rlat) > x = temp*DCOS(rlon)/denom > y = temp*DSIN(rlon)/denom > C > RETURN > END DATA FORMATS: ------------ The data on the MSS has been kept in its native TDF (i.e. Sea Space) format. This is essentially a variety of CDF. The format consists of a 644 byte header, the data itself, and then a trailer (all in a single file). The data can be accessed by simply skiping the header, reading the data (all channels sequential), and ignoring the CDF stuff at the end. This is essentially what textract.exe does, but one channel at a time. The NOAA data is stored as "short int" variables (2 bytes or 16 bits) per pixel, in units of albedo x 100 and/or brightness temp x 100. Given a 1200 x 1200 array, this means that the channel 1 data will occupy 2,880,000 bytes (2 x 1200 x 1200 = 2,880,000), followed by channel 2 data, 3, 4, and 5 (for a total of 14,400,000 bytes). The overall file sizes will be a bit bigger, since you have to add in the 644 byte header and the variable length TDF trailer. The same pattern applies to the DMSP data, with the exception that the data is stored as "byte" variables (8 bits). The high resolution visible data only maintains 64 gray levels (with a variable gain, hence not calibrated rigorously) in the data, but is still stored in byte format. The low resolution IR data uses all 256 levels available to it, and can be converted to brightness temperature via the relation: T = (I - 176.69)/2.125 where T is in degrees C and I is the (integer) byte value. For the DMSP files, this means that the file sizes will be 11,520,000 (2400 x 2400 x 2 = 11,520,000) and a bit more counting the header and TDF trailer. ------------------------------------------------------------------- --------------------- ABOUT RESOLUTION ---------------------- ------------------------------------------------------------------- QUESTION: -------- Why is the IR data LOW RESOLUTION when it uses 256 levels, and the visible OLS data HIGH RESOLUTION when it uses 64 levels? Shouldn't this be reversed? ANSWER: ------ High/low resolution is used in the context of sensor SPATIAL resolution. The 64 gray-level data has a ground resolution of 500 meters, while the 256 level IR data has a reduced resolution of 2.5 kilometers. The number of data bits is in an inverse relationship because of two factors (1) data transmission rates -- the 500 m resolution data generates a lot of data to transmit and just couldn't keep up with more data bits per pixel, and (2) the more rapidly scanning sensor taking the 500 m resolution data just doesn't have the dwell time to justify more significant digits. ------------------------------------------------------------------- ------------------ ABOUT HAND NAVIGATION -------------------- ------------------------------------------------------------------- The nominal image positioning using the transmitted position, time, and attitude values is often in error by as much as 10 km. The "hand" navigation step involves displaying the image in "satellite-sensor" coordinates (lines and samples), with a geometrical transformation of a coast-line data base as an overlay on the image. In an iterative series of steps, the reported time and attitude parameters can be adjusted to improve the registration of the image relative to the coast-line data base. There are many different ways to do this step. In general, however, the final result is clearly better than the unadjusted registration, with accuracy often quoted as good to 1 (or more honestly 1-2) km. Not all areas of the image will be registered "perfectly". Cloud cover, sensor quality, and position of the satellite track will all affect the quality of the navigation. After the image is navigated, it is regridded into the "standard" lat-long grid used in the data base. This permits scientists not interested in the preliminary processing steps, or not having the appropiate software, to have access to the "full" resolution multi-spectral data sets for all available imagery.