TEXT_DESCRIBING_THE_EICS_SATF_SYSTEM.TXT Dynamics Explorer 1 --- Energetic Ion Composition Spectrometer (EICS) Format for submission of data to the NSSDC Submitted by W.K. Peterson Lockheed Palo Alto Research Labs, July 1989 This file describes the EICS Stand Alone Telemetry File System. The system consists of compressed data files, two index files, a general purpose code to read the files and produce meaningful geophysical parameters, and documentation files: The documentation describing the system is contained in four files, copies of which are on each volume of EICS Stand Alone Telemetry Files. This is the first of the four files. Its name is: OPTICAL_DEVICE:[EICS]TEXT_DESCRIBING_THE_EICS_SATF_SYSTEM.TXT The other documentation files are OPTICAL_DEVICE:[EICS]TEXT_DESCRIBING_THE_INDEX_FILE.TXT OPTICAL_DEVICE:[EICS]TEXT_DESCRIBING_THE_ANALYSIS_SOFTWARE.TXT OPTICAL_DEVICE:[EICS]ANNOTATED_OUTPUT_LISTING.TXT The Structure of an EICS Optical Volume is as follows : ______________________________________________________________________ | Optical Drive Volume - ROOT - contains 9 system files, a | | Master Directory, and the EICS | | _______________________________________________________________| | | MasterVolumeDirectory actually will have numerical name | | |_____________________________________________________________ | | | Backup.SYS | | |_____________________________________________________________ | | | Badblk.SYS | | |_____________________________________________________________ | | | Badlog.SYS | | |_____________________________________________________________ | | | Bitmap.SYS | | |_____________________________________________________________ | | | Config.SYS | | |_____________________________________________________________ | | | Contin.SYS | | |_____________________________________________________________ | | | Index f.SYS | | |_____________________________________________________________ | | | Volset.SYS | | |_____________________________________________________________ | | | EICS.DIR This is the directory containing what we use | | | _______________________________________________ | | | | VolDesc.SFD | | | |_______________________________________________| | | | Format.SFD | | | |_______________________________________________| | | | T*.DIR SATF subdirectories -about 50 of | | | about | these. Each contains about 15 SATF files:| | | 50 of these| _______________________________________ | | | >| about | C*.DAT SATF file - about 15/dir.; | | | | 15 > | -each contains many 8 sec. blocks | | | |_______|_______________________________________| | | | EICSDATA.LIS | | | |_______________________________________________| | | | Text_Describing_the_Index_File.TXT | | | |_______________________________________________| | | | Optical_Volume_Index.LIS | | | |_______________________________________________| | | | Text_Describing_Optical_Volume_Index.TXT | | | |_______________________________________________| | | | Text_Describing_the_EICS_SATF_System.TXT | | | |_______________________________________________| | | | EICS.EXE | | | |_______________________________________________| | | | Text_Describing_the_Analysis_Software.TXT | | | |_______________________________________________| | | | Annotated_Output_Listing.TXT | |____ |______________|_______________________________________________| The Descriptive Relations between between the above Files follow : Text_Describing_the_EICS_SATF_System.TXT describes all the other files Text_Describing_the_Index_File.TXT describes EICSData.Lis EICSData.Lis describes SATF files Text_Describing_the_Analysis_Software.TXT describes EICS.EXE Annotated_Output_Listing.TXT describes the data (supplemenatary) Format.SFD describes SATF files Voldesc.SFD describes SATF files Optical_Volume_Index.Lis describes SATF files Text_Describing_Optical_Volume_Index.TXT describes Optical_Volume_Index.Lis The general purpose code that reads and outputs data is written in FORTRAN77 with VAX extensions and VAX PASCAL. It has been provided on magnetic tape to the National Space Sciences Data Center at NASA/Goddard. The standard VOLDESC.SFD and FORMAT.SFD files associated with each volume provides the rest of the information to completely describe the EICS stand alone telemetry system. The basic outline of this documentation is that this file describes the structure and organization on the optical platter of the EICS stand alone telemetry files. An index file OPTICAL_DEVICE:[EICS]EICSDATA.lis is an index to the EICS stand alone telemetry files on the complete set of optical platters. A description of the content of the index including the summary data quality flags is in the file OPTICAL_DEVICE:[EICS]TEXT_DESCRIBING_THE_INDEX_FILE.TXT A second index file OPTICAL_DEVICE:[EICS]OPTICAL_VOLUME_INDEX.LIS contains two tables, Table 1 giving the range of SATF archive tapes on each optical volume and Table 2 giving the volume on which the SATF from each tape are archived. The Optical_Volume_Index.LIS file on each optical volume is cumulative and describes the data on all previously created volumes. It is important to use the index on the latest optical volume in your possession to avoid missing files that have been archived recently. Text describing the Optical_Volume_Index.LIS file is contained in the Text_Describing_Optical_Volume_Index.TXT file. The general purpose program that reads the SATF and outputs geophysical information is in the file OPTICAL_DEVICE:[EICS]EICS.EXE Instructions on the use of this program are in the file OPTICAL_DEVICE:[EICS]TEXT_DESCRIBING_THE_ANALYSIS_SOFTWARE.TXT and an annotated output listing produced by the general purpose program is in the file OPTICAL_DEVICE:[EICS]ANNOTATED_OUTPUT_LISTING.TXT ----------------------------------- TEXT_DESCRIBING_THE_EICS_SATF_SYSTEM.TXT Dynamics Explorer 1 --- Energetic Ion Composition Spectrometer (EICS) Format for submission of data to the NSSDC Submitted by W.K. Peterson Lockheed Palo Alto Research Labs, July 1989 BACKGROUND INFORMATION ABOUT EICS DATA PROCESSING: There are four levels of EICS data processing. The first level generates a low resolution summary data base and associated summary graphics on the Goddard Sigma Nine Computer. The second level generates a library of selected stand-alone telemetry files that are transferred by magnetic tape to VAX computers at the Lockheed Palo Alto Research Laboratories. The third level updates a local telemetry data base on Lockheed computers and also generates a complementary high resolution summary data base and associated graphics product. The fourth level consists of a suite of analysis programs which have been developed to use this library of stand-alone telemetry files to produce a wide variety of data products including listings of observed fluxes. While the low resolution summary data base produced on the Goddard Sigma Nine computer has been used as the basis for a series of research reports, it is too incomplete and difficult to interpret for general distribution. Second level processing, the creation of stand-alone telemetry files on the Goddard Sigma Nine computer, is done by both EICS and DE project personnel. In early 1991 about half of the telemetry acquired since launch has been converted to stand-alone telemetry files. The rate of production of these files depends on the availability of project Sigma Nine computer resources. Our goal is to create stand-alone telemetry files for all intervals identified as interesting by members of the DE science team. For example we are creating stand-alone telemetry files for all of the intervals identified by the PROMIS and GISMOS campaigns. No one data product is appropriate for all scientific problems. The EICS science team does not use or supply a 'standard' data product to other members of the DE science team. The scientific goals of individual projects determine the type and volume of data products required and we select the most appropriate from those available. In some cases the high resolution summary graphics output is adequate. In other special cases we have developed specific data products to support joint projects. We have developed a general purpose program that provides detailed EICS fluxes for most EICS operational modes. A single screen input allows a user to select the temporal and angular resolution of mass and energy spectra which are listed in a standard output format. We also have made a data catalog that gives summary data quality information so that a user can identify data intervals where, because of the EICS mode or data quality, codes other than the general purpose one must be used to analyze the data. The EICS stand alone telemetry files, the general purpose program discussed above and the associated data catalog are the three types of data files archived on this optical disk. Overview of the WORM optical disk format for EICS stand-alone telemetry files. This optical disk contains EICS stand-alone telemetry files (SATF), a program to read them and return fluxes or velocity space densities of measured ions at selectable time and angular resolution, and indexes (catalogs) of the data intervals that provides summary data quality information. Each type of file (SATF, PROGRAM, and INDEX) has an associated documentation file(s) in the top level directory. The PROGRAM and INDEX files also reside in the top level directory. Individual SATF files reside in subdirectories determined and identified by the physical tape on which they are stored in Palo Alto. An outline of the disk structure and the content of the documentation files are given below. A copy of this set of optical disks is used by EICS personnel in the Lockheed Space Sciences Laboratory for data analysis. The disk format described here was originally proposed to the National Space Science Data Center in March 1987, updated in the summer of 1987, and modified in April 1988, and July 1989 to conform with the "Final Draft" version of the Document "DE Submission of Data on Optical Disk to NSSDC: Procedures and Content" supplied by Dr. J.H. King of the NSSDC. Helpful discussions with NSSDC personnel are gratefully acknowledged. CONTENT OF THE EICS SATF FILES AND INDEX (CATALOG) The EICS stand-alone telemetry files (SATF) contain a variable number of eight second 'blocks' of telemetry, selected orbit/attitude data, and data quality flags. The time boundaries for EICS SATF files are chosen to be exactly the same as the time boundaries in the DE-1 telemetry data files. Most files contain less than two hours of data. The SATF format retains the full temporal resolution of the instrument, which is essential for studies of auroral phenomena and leads to significant costs savings because the extremely large (more than 700 magnetic tapes in 1991). Additionally the SATF format includes the necessary DE-1 orbit/attitude data. Because the Goddard Sigma Nine computer was unable to process all of the EICS telemetry, data was not converted to the SATF format in time order. Non-sequential groups of about 20 SATF files are then written to EICS_SATF archive tapes. Physical files are identified by three items: 1) the physical tape number on which they reside, 2) the year and day the data they contain was acquired (telemetry segments crossing day boundaries are broken into two files) and 3) a letter A,B,... to identify files from the same day on the same physical tape. The reconstruction of the 'logical' data file from its individual physical pieces is done using an index file. The most current copy of the index file for all EICS data is accessible through the NSSDC supported central on-line data directory (CODD) facility and is also is readable over SPAN as specified on the CODD. Each optical volume contains the most current copy of the index file available at the time the volume is closed. This file will generally include reference to all of the files on the current, prior volumes and to parts of volumes not yet closed. This file will have the name [EICS]EICSDATA.LIS. Thus if the optical device name is QSA0:, the complete index file specification is QSA0:[EICS]EICSDATA.LIS. The documentation file describing the contents of the EICSDATA.LIS file is then OPTICAL_DEVICE:[EICS]TEXT_DESCRIBING_THE_INDEX_FILE.TXT This file defines the summary data quality flags for each data interval listed in the EICSDATA.LIS file. For example, the catalog information presented in OPTICAL_DEVICE:[EICS]TEXT_DESCRIBING_THE_INDEX_FILE.TXT shows that the EICS stand alone telemetry data from October 8, 1984 (day 282 of 1984) at 02:00 universal time is on physical tape #248 in a physical file with the name 'C84282A.DAT' (the 'C' prefix is implicit), and that the instrumental mode and data quality are normal. It is important to note that this index is updated several times a week as new data is added to the data base. MORE DETAILS ABOUT THE WORM OPTICAL DISK FORMAT The WORM optical disk format is indicated in the simulated directory listings below. QSAO: is the optical drive on one of the Lockheed VAXes where this platter was created. At other sites the optical disk could have a different logical or physical name. QSA0:[0,0] is the top level directory which contains the standard DEC FILES11 systems files and one data directory. The command dir/size=all/date/protection qsa0:[0,0] returns the following information Directory QSA0:[000,000] 000000.DIR;1 1/8 9-MAR-1988 15:34 (RWED,RWED,RE,E) BACKUP.SYS;1 0/0 9-MAR-1988 15:34 (RWED,RWED,RE,) BADBLK.SYS;1 8/8 9-MAR-1988 15:34 (RWED,RWED,RE,) BADLOG.SYS;1 1/8 9-MAR-1988 15:34 (RWED,RWED,RE,) BITMAP.SYS;1 63/64 9-MAR-1988 15:34 (RWED,RWED,RE,) CONFIG.SYS;1 1/8 9-MAR-1988 15:39 (RWED,RWED,RWE,R) CONTIN.SYS;1 0/0 9-MAR-1988 15:34 (RWED,RWED,RE,) CORIMG.SYS;1 0/0 9-MAR-1988 15:34 (RWED,RWED,RE,) EICS.DIR;1 36/56 9-MAR-1988 15:34 (RWED,RWED,RWE,R) INDEXF.SYS;1 11047/20952 9-MAR-1988 15:34 (RWED,RWED,RE,) VOLSET.SYS;1 0/0 9-MAR-1988 15:34 (RWED,RWED,RE,) Total of 11 files, 11157/21104 blocks. Archived EICS data are all included in the QSA0:[EICS...] directory tree. The QSA0:[EICS] directory contains Directory QSA0:[EICS] FORMAT.SFD VOLDESC.SFD TEXT_DESCRIBING_THE_EICS_SATF_SYSTEM.TXT TEXT_DESCRIBING_THE_INDEX_FILE.TXT TEXT_DESCRIBING_THE_ANALYSIS_SOFTWARE.TXT TEXT_DESCRIBING_OPTICAL_VOLUME_INDEX.TXT ANNOTATED_OUTPUT_LISTING.TXT EICSDATA.LIS;223 1458/1458 19-APR-1988 19:12 (RWED,RWED,RE,RWED) <> OPTICAL_VOLUME_INDEX.LIS <> EICS.EXE;27 169/171 9-OCT-1987 12:00 <> T00001.DIR;1 5/8 9-MAR-1988 15:39 (RWED,RWED,RWE,R) T00002.DIR;1 3/8 9-MAR-1988 15:39 (RWED,RWED,RWE,R) T00003.DIR;1 2/8 9-MAR-1988 15:39 (RWED,RWED,RWE,R) T00004.DIR;1 3/8 9-MAR-1988 15:39 (RWED,RWED,RWE,R) T00005.DIR;1 3/8 9-MAR-1988 15:39 (RWED,RWED,RWE,R) .... .... T00398.DIR;1 1/8 9-MAR-1988 15:35 (RWED,RWED,RWE,R) T00399.DIR;1 1/8 9-MAR-1988 15:35 (RWED,RWED,RWE,R) T00400.DIR;1 1/8 9-MAR-1988 15:35 (RWED,RWED,RWE,R) <> <<<>>>> Directory QSA0:[EICS.T00010] C81277A.DAT C81282A.DAT C81283B.DAT C81283C.DAT C81284E.DAT C81329E.DAT C81329F.DAT C81331A.DAT C81331C.DAT C81331B.DAT C82075A.DAT C82126A.DAT C82126B.DAT C82145B.DAT C82145A.DAT C82229B.DAT C82229A.DAT Establishing a subdirectory (with the name 'Txxxxx') for each tape of existing EICS stand-alone telemetry data is the key feature of this format that makes it possible to transfer existing data structures to the optical disk using the SOAR OCOPY command. Existing VMS code accesses these files with no modifications. For example the file 'C84282A.DAT', mentioned above, would be addressed as QSA0:[EICS.T00248]C84282A.DAT when the appropriate volume is mounted on the VAX in Palo Alto. To obtain data from the EICS archive platter you must run the a general purpose program against the EICS data file for the appropriate interval. This program creates outputs of geophysical interest. This File OPTICAL_DEVICE:[EICS]TEXT_DESCRIBING_THE_EICS_SATF_SYSTEM.TXT describes the structure and organization on this optical platter of the EICS stand alone telemetry files. As noted above an index of the data available is in the file, QSA0:[EICS]EICSDATA.LIS A description of the content of the index including the summary data quality flags is in the file OPTICAL_DEVICE:[EICS]TEXT_DESCRIBING_THE_INDEX_FILE.TXT The general purpose program to read and output geophysical information from the data files is in the file QSA0:[EICS]EICS.EXE Instructions on the use of this program are in the file OPTICAL_DEVICE:[EICS]TEXT_DESCRIBING_THE_ANALYSIS_SOFTWARE.TXT and an annotated output listing produced by the general purpose program is in the file OPTICAL_DEVICE:[EICS]ANNOTATED_OUTPUT_LISTING.TXT