$Id: README.processing,v 1.2 2003/06/05 23:28:18 mankoff Exp $ ------------------------------------------------------------------------ FILE: README.processing AUTHOR: Ken Mankoff (mankoff (at) lasp (dot) colorado (dot) edu) LOCATION: LASP, University of Colorado DATE: 2002-05-28 ------------------------------------------------------------------------ Introduction This file attempts to describe the steps taken to produce the Reflectance Science Product from the PDS Tape Archives (EDR and SEDR records). The end user should not necessarily use the information contained in this file, they should read README.usage. This file is geared towards a developer, maintainer, or improver of the dataset. This file and the software discussed here are available at: http://lasp.colorado.edu/albatross/processing/ The raw data used in the processing is available here: http://lasp.colorado.edu/mariner_9_data/ The refined data and visualization software for it are here: http://lasp.colorado.edu/albatross/ ------------------------------------------------------------------------ Initial Reflectance DB This is the first step in the processing. The input is the EDR files (available at http://lasp.colorado.edu/mariner_9_data/). The output is a database of reflectance (from 2107 to 3497 angstroms). Each output record has a DAS timestamp. The Initial Reflectance DB is created by typing the following commands at an IDL command prompt: IDL> @mm71_setup IDL> mm71_reflectance The mm71_reflectance opens each EDR file in the directory, and performs the following loop: a) returns which records in the file are "zenith" orbit records. All non-zenith records are discarded. a) NEW method: return all records that are from an even orbit. This is because the list of "zenith" is really just a list of "mapping sequence", not the whole orbit. b) throws out records that fail the "zero check" of the F Fiducial. This occurs if the data is not bit-aligned in the file c) returns a gainstate multiplier for each record. All records with a gainstate greater than 90 are discarded. d) calibrates each record to the 2800 magII dip. All records that are shifted more than 5 positions are discarded. e) subtracts the background signal from the sensor data. Background is indices 68:117 (? to ? angstroms) of the F_channel in the EDR files. All records where the background is greater than 10 are discarded. f) Calculates the reflectance from the instrument counts (see mm71_calib_solar.txt for a detailed explanation of this step). g) writes each record to a database that contains reflectance and DAS time. The database is as follows: The name of the database is mm71_reflectance_DAS.dat. The database is a FLTARR( 320, n_recs ). It is made of 4-byte floats and written on SUN hardware. Records 0 to 316 are the reflectance. Records 317 and 318 are the DAS time (broken long word). Record 319 is empty (zero). The DAS can be reconstructed with this command: IDL> DAS = LONG(317)+LONG(318)+4096 SUN hardware means that users with a PC or Mac may have trouble with the byte ordering (Sun == 'big endian', PC == 'little endian', Mac = ?). Refer to IDL help about "BYTE ORDER" if the data appears to be garbage. ------------------------------------------------------------------------ Augmented Reflectance DB The second step is to take the Initial Reflectance Database (see above section) and augment it with auxillary data. Auxillary, augmented, bookkeepping, orbital parameters, etc. all refer data such as latitude, longitude, time (Earth), mu [cos(solar_lighting_angle)], mu0 [cos(view_angle)], etc. All this information comes from the SEDR files. The inputs are the SEDR files and the mm71_reflectance_DAS.dat. The output is the file 'reflectanceDB.dat' The Augmented Reflectance DataBase is generated in IDL with the following commands: IDL> @mm71_setup IDL> reflectanceDB The format of the output (reflectanceDB.dat) is a binary database of IDL structures. The format of the struct is available in the "reflectanceDB.pro" files top procedure, named "reflectanceDB_struct". While the file can be decoded as a flat BYTE() array, it is strongly recommended that the pre-supplied struct format is used. To gain familiarity with the struct, either read the code of "reflectanceDB_struct", or type the following commands at the IDL prompt: IDL> @mm71_setup IDL> .com reflectanceDB IDL> reflectanceDB_struct, s IDL> help, s, /st ------------------------------------------------------------------------ LookUp Table The final step is not necessary in the processing, but aids in the usability of the Reflectance Science Product. Navigating the "reflectanceDB.dat" database is slow and painful (and impossible on some systems) due to the size of the database. So, LUT_make.pro needs to be run, which takes the reflectance DB and generates a LookUp Table with DAS, lat/lon, and a few other useful and/or critical pieces of information, in the LUT.dat file The LUT can be opened quickly in its entirity on most systems. WHERE() queries can then be performed on the LUT. The indices returned by WHERE() can then be used in a for-loop to read out of the larger reflectanceDB.dat database (see README.usage for more details) The input is the reflectanceDB.dat database. The output is the LUT.dat database. The LUT is generated by typing the following commands at the IDL command prompt: IDL> @mm71_setup IDL> LUT_make Read LUT_make, LUT_load, and LUT_struct to find out exactly what is in the LUT.dat file. ADDITION: 2003-06-06 by KDM: The LUT should be sorted (by DAS) for speed optimization. This can be done with the following code: IDL> @mm71_setup IDL> mm71_sort ------------------------------------------------------------------------ EOF