TIMAS NRT Monitoring
***************************************************************************** /timas/local/operations/doc/timas_nrt_monitor_doc DLCARR 17 Oct 98 TIMAS_NRT_MONITOR_DOC ---------------------- TIMAS Operations phone numbers/email: ------------------------------------- Daryl Carr (Lockheed) 650-424-2044 (W Bldg 255) 415-271-5113 (H cell phone) 970-229-0726 (Fort Collins, Colorado) 415-454-6742 (San Rafael, CA Lucy) 706-265-1307 (Atlanta, GA - daughter Suzy) 770-271-4438 (Atlanta, GA - daughter Shannon) email: carr@timas.spasci.com Dave Chenette (Lockheed) 650-424-3449 (W Bldg 255) 560-854-7789 (H Palo Alto) email: chenette@agena.spasci.com Harry Collin (Lockheed) 650-424-3445 (W Bldg 255) 650-856-3891 (H Palo Alto) email: collin@agena.spasci.com Jerry Drake (Lockheed) 650-424-3404 (W Bldg 252) 650-424-3333 (W FAX Bldg 252) 408-730-4824 (H Cupertino, CA) 209-795-6320 (H BlueLake Springs, CA) email: drake@agena.spasci.com Gary Heyman (Lockheed) 650-424-3063 (W Bldg 255) 650-424-3206 (W Bldg 252) 650-965-1777 (H Mtn View) 800-SKY-PAGE, PIN=255-0217# (pager) 800-759-7243 2550217# YourNumber# email: gary@nimbus.spasci.com Tony Magoncelli (Lockheed) 650-424-3438 (W Bldg 255) 510-643-9515 (W Berkeley) 415-431-9793 (H San Francisco) 415-921-1156 (H Parents) email: magoncelli@sierra.spasci.com Bill Peterson (Lockheed) 650-424-3269 (W Bldg 255) 650-322-5872 (H Palo Alto) 303-492-0686 (W Boulder,CO) 303-333-5609 (H Denver, CO) email: pete@willow.Colorado.EDU Ed Shelley (Lockheed) 541-552-1491 (H Ashland, OR) email: egshelley@aol.com Ron Black (SwRI) 210-522-3562 (W) 210-235-5487 (pager) 210-647-8686 (H) email: POCC 301-286-9313,7 (GSFC Bldg 14) Polar Operations Control Center Steve: telemetry questions, upload realtime commands, nrt data TIMAS Ground Station (POCC) 301-286-0726 TIMAS modem (POCC) 301-286-3100 x-60726 (modem on B, while ringing switch to A on A/B switch box) TIMAS PC at POCC samit.gsfc.nasa.gov (IP 128.183.40.143) SCAMA 301-286-5473 (SCAMA 158) or 8737 TIMAS GSE Rm. OF-121, B. 255 650-424-3302 650-424-2333 Gerry Blackwell (GSFC) 301-794-2318 email: gblackwell@istp1.gsfc.nasa.gov Carol (GSFC Y2K) 301-286-8675 email: Mike Cassidy (GSFC) 301-286-7163 cassidy@istp1.gsfc.nasa.gov email: cassidy@istp7.gsfc.nasa.gov John Galloway (GSFC IM) 301-286-4874 800-SKY-PAGE, PIN=517-0777 301-294-2542 (H) 301-286-1736 (FAX at GSFC) email: Ed Macy (GSFC OPS Manager) 301-286-0762 email: Bill Mish (GSFC) 301-286-5444 email: u2whm@lepvax.gsfc.nasa.gov WMISH@istp1.gsfc.nasa.gov PAC (GSFC) 301-286-9453 email: pac@istp1.gsfc.nasa.gov Chris Raymond (SPOF) 301-286-7864 301-303-9078 (pager) email: raymond@spof01.gsfc.nasa.gov Ann Strickling (GSFC) 301-286-9453 email: pac@istp1.gsfc.nasa.gov Ken Tom (POCC Manager) 301-286-0666 email: John Wainwright (GSFC OPS) 301-286-0724 email: wainwright@istp1.gsfc.nasa.gov Sheila Wroe (GSFC) 301-286-2487,-9453 email: sheila.a.wroe.1@gsfc.nasa.gov SAGNET dial in: 800-457-0998 408-734-8952 650-855-9064 ------------------------------------------------------------------------------- nrt monitoring procedure: ----------------------------- NOTE: Use "go_nrt" instead of "nrt" : The nrt program has been upgraded and renamed as "go_nrt" If there is any doubt about the "go_nrt" connection, just re-run "go_nrt". It will automatically kill any old "go_nrt" that may be running. A command script "nrt_check" has been added to give status of NRT data processing. It will tell if you are collecting data during an active pass time. telnet timas.spasci.com The nrt code is now on timas type "nrt_check" Verifies that nrt process is running. If not running the response message will tell you to type "go_nrt" to restart nrt process If nrt process not running then: type "go_nrt" and do "returns" for default values for login process. Any old "go_nrt" that may be running will automatically be killed. If go_nrt login process (to istp6 or istp7) fails with read socket error, then call PAC Ann Strickling and ensure that stip1, istp6, istp7 are up. After confirming that "go_nrt" is running then run following processes: - get_spins data-nrt date-3/6/97 - dps fec - hk - hkstat ********************* If I failed to collect the nrt data on realtime passes, I can get it from CDHF: cd /timas/data/nrt fis name pword ftp> cd polar_nrttl:[polar_nrt.tim] 250 Connected to POLAR_NRTTL:[POLAR_NRT.TIM]. ftp> ls 200 Port 190.156 at Host 192.68.162.50 accepted. 150 List started. po_lz_tim_20000704_v01.0014 po_lz_tim_20000704_v01.0604 po_lz_tim_20000704_v01.0945 get po_lz_tim_20000704_v01.0945 quit Now, modify name to po_lz_tim_20000704_v01.dat then run it either for survey plots, or thru get_spins, hk, etc ********************* An efficient way to make the processes run simultaneously is to open a new cmdtool window for each process. If you are running from a remote computer and want to display windows: Before you telnet to TIMAS type "xhost timas" After you have logged onto TIMAS, type"setenv DISPLAY YourComputerName:0" Now you can create cmdtool windows for each process. Some useful commands: process open process Ctrl z suspend process bg process restarts and put process into background fg process bring process to foreground Ctrl c kill process zapmem clears get_spins buffer if problem zap go_nrt kills the process nrt_check See if go_nrt is running. use it after you start go_nrt to make sure that 2 go_nrt's are not running Ctrl H Use for backspace on the MAC ps -ef |grep nrt If zapmem cannot clear memory, log on as superuser (root), then: /timas/code/gse/release/zapmem This should clear any memory that was owned by root... If you cant acquire the data stream: - Verify the pass times - zapmem - Restart go_nrt - nrt_check - Call POC at 301-286-9313, ask if pass is acquired and downloadin nrt data - Call Carr, peterson, Drake, or Heyman for help. GaryH suggested a way to monitor on a mac when you have only one screen: the trick is to first make sure go_nrt running in bkgd, then start the get_spins program with output directed to null, so that it is not in the way when you run hk, hkstat, dps progrmas serially: get_spins data-nrt date-04/18/97 >! /dev/null & dps fec hk hkstat fg fg# (bring get_spins to foreground then can terminate with a Ctrl C or ps -ef | grep get_spins, then kill proc# Another hint: to set a key: stty function 'hit a key' eg stty erase Note: I can do monitoring even if I miss a pass by replaying nrt data using: get_spins play step data-lz dir-/timas/data/nrt date-05/25/99 time-00:04:00 then: hk play now back to get_spins to step thru. hkstat play now back to get_spins to step thru. dps fec play now back to get_spins to step thru. ============================================================================ Run dps: dps needs to have get_spins running before it will work: you can start dps, but must then start get_spins in order to collect data.. This is because get_spins provides the args to identify the data, sets up a shared memory link for dps to find the specified data when it is called. e.g.: get_spins play step data-lz dir-/timas/data/lz date-02/05/99 time-00:04:00 then: dps esweep play now back to get_spins to step thru. or without stepping:: dps esweep play then get_spins play data-lz dir-/timas/data/lz date-02/05/99 time-00:04:00 ---------------- Here is an example of what I did on 5/26/99 to look at possible bad files: Check playlist code answers to monitor problem I saved one of the offending nrt files: po_lz_tim_19990525_v99.dat and did playlist run: get_spins play step data-lz dir-/timas/data/nrt date-05/25/99 time-00:04:00 then: hk play now back to get_spins to step thru. hkstat play now back to get_spins to step thru. dps fec play now back to get_spins to step thru. Result: could not read file, same error as before I next tried next day file po_lz_tim_19990525_v99.dat, and successfully stepped through the data, got monitor values.. --------------- dps help Running dps ( %Z% %M% v%I% - %G% ) Parameters enclosed in [] are optional. If [] parameter is not present all members of group are displayed. Command Summary: Spin Filters: odd Turn off display of even spins even Turn off display of odd spins Instrument imode filters: all-mode Display ALL Spin headers (default) no-mode Display No Spin Headers sci-mode Display Science Spin headers scical-mode Display Science Cal-On Spin headers mcp-mode Display MCP CAL Spin headers event-mode Display EVENT Spin headers dump-mode Display DUMP Spin headers engr-mode Display ENGINEERING Spin headers Data Product filters: all-dp Display all DP data no-dp Display no DP data (default) ms[n] Display mass spectra data products, n=1-8 lrdf[nm] Display lrdf data product n=1-4, m=a,b mrdf[nm] Display mrdf data product n=1-4, m=a,b pad[nm] Display pad data product n= 1-4, m= a,b fec Display fast event count data product pec Display processed event count data product bcr Display background count rate data product mcp Display MCP PHA data product event diplay event data product dump[-nnn] Display dump data product nnn= dp, imp or mlut engr[-nnn] Display engineering data product nnn= hv, ea1, ea2, cmd Misc: play Playback mode enabled, read data from playlink/playlist -bell Disable console bell at error conditions -dp Disable DP headers -sh Disable Spin headers DPS only: address=0xnnnnnn start address of DUMP length=nnnnn length of DUMP checksum=nnnnnn expected checksum of DUMP Special Data product filters (dps only): esweep Generate a sweeping ps, monitor report- all esweep-pa Generate a sweeping ps, monitor report- pa esweep-ea[n] Generate a sweeping ps, monitor report- ea raw Supress data product unique display sum Generate a summary data product display -decomp Supress De-precompression of DP data Engineering Monitor Filters(dps only): pa-imon vac-imon pa1-ref pa2-ref pa1-vfd pa2-vfd ea1-hvref ea1-mvf ea1-pvf ea2-hvref ea2-mvf ea2-pvf MISC (dps only): coeffs Display analog conversion coeffs., then exit dps sh Print full spin header carr@timas% ============================================================================ READING THE NRT DATA FILE: -------------------------- If you want to review an nrt data, they are same format as any tim_lz data file, so you can use the same diagnostic tools, noting however that you do not have the scr or spha or or_pre files available which are required by some of the programs. But, you can always run "view_edp: e.g. view_edp ms1 dir-/timas/data/nrt date-09/29/98 time-00:04:00 dsecs-600 > ms1_980929.dat or if you dont know the times of the data: view_edp ms1 dir-/timas/data/nrt date-09/29/98 time-00:04:00 > ms1_980929.dat Example of a monitor session: ---------------------------- telnet timas.spasci.com login: carr Password: timas% nrt_check <<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>> go_nrt running since Aug_05 Process ID: 12081 %cpu ussage: 0.0 Last MaF received at UT= 17:38:05:0 08/06/1997 MaF=70 Current UT is: 17:41:16 08/06/97 timas% get_spins data-nrt date-3/6/97 start get_spins Start at: 06-Mar-1997 00:00:00 Stop at: 06-Mar-1997 23:59:59 Running get_spins (@(#) get_spins.c v1.20 - 09/12/96) HOME= /export/home/carr Creating shared memory for spins Starting control process Creating data product spin files in /export/home/timas/timas/spins (getdatafile) Opened /timas/data/nrt/po_lz_tim_19970305_v99 Skip to end of file Awaiting next MaF ...+ spin: 08/1/040 16:58:04.602 03/05/97 UT length: 3768 SCIENCE 9995959595 elut e_range reduced_e stim: OFF mode_change: NO state/cmd: ROM/ROM STD/LOW NO /NO dtc: ENA ICP_table#: 006 dp: R=1 ID=20 E=0 A=0 length=00F5 FEC dp: R=1 ID=21 E=0 A=0 length=00F5 PEC dp: R=1 ID=22 E=0 A=0 length=0063 BCR dp: R=1 ID=10 E=1 A=0 length=033D MRDF1A dp: R=1 ID=12 E=1 A=0 length=01FE MRDF2A dp: R=1 ID=16 E=1 A=0 length=0248 MRDF4A dp: R=1 ID=13 E=0 A=0 length=0146 MRDF2B dp: R=1 ID=17 E=0 A=0 length=016E MRDF4B dp: R=1 ID=11 E=0 A=0 length=01EC MRDF1B spin: 09/1/041 16:58:04.602 03/05/97 UT length: 2589 SCIENCE 9595959595 elut e_range reduced_e stim: OFF mode_change: NO state/cmd: ROM/ROM STD/LOW NO /NO dtc: ENA ICP_table#: 006 dp: R=1 ID=01 E=0 A=0 length=0187 MS2 . . . Now put it in bkgnd or switch to a new screen for dps, hk, and hkstat: timas@cis% dps fec look at fec outputs Running dps ( @(#) display_product.c v1.40 - 09/10/96 ) FEC DP data filter enabled /* conversions.txt v1.6 03/07/96 conversion coefficients */ *spin_index= 16Waiting for NEXT spin ... spin: 24/1/056 3185 bytes 16:59:36.602 03/05/97 UT v1.42 mode: SCIENCE elut e_range reduced_e stim: OFF mode_change: NO state/cmd: ROM/ROM STD/LOW NO /NO dtc: ENA ICP_table#: 006 >>>>>>>>>>>>> Data product: length rice energy angle id 0245 YES -- -- 20 FEC WARNING: Compressed data(245) >= expected data(240) RICE decompressed from 245 to 240 bytes Spin De-precompressed Counts (DEC) Sector Energy Sequence\Step --> 0\14 1\12 2\10 3\8 4\6 5\4 6\2 7 8\BG 9\26 10\24 11\22 12\20 13\18 14\16 00: 308 308 297 278 308 341 391 570 287 353 330 365 341 308 297 01: 341 297 287 278 287 319 308 551 260 330 341 391 353 330 308 02: 297 308 297 269 260 341 365 551 226 330 365 353 330 278 287 03: 278 251 251 278 269 297 378 551 251 308 365 330 341 269 260 04: 287 260 287 278 287 287 353 532 260 297 341 341 330 260 260 05: 251 278 269 319 278 353 365 551 242 330 319 378 341 308 278 06: 319 251 319 297 269 353 378 610 251 341 341 378 287 297 319 07: 278 297 269 297 287 341 341 631 269 319 378 391 319 269 278 08: 278 287 297 297 278 341 391 653 260 319 330 365 341 287 308 09: 234 278 308 260 278 330 391 676 287 297 319 330 297 260 278 10: 287 278 278 278 269 278 341 551 234 287 341 297 297 260 251 11: 278 269 269 242 242 251 330 570 226 297 319 341 319 242 278 12: 269 269 287 287 260 297 330 551 260 297 341 365 308 287 234 13: 278 278 287 260 251 278 378 532 260 297 365 365 308 251 269 14: 278 260 269 260 287 308 330 449 260 319 341 365 319 319 251 15: 251 278 278 260 269 319 330 497 278 297 330 308 287 287 287 Counts/spin: 76416 Counts/accum: 318.4 Spin: 24/1/56 16:59:36.602 03/05/97 UT v1.42 FEC ^C timas@cis% hk look at housekeeping parameter values Running hk (v1.18 ) /* conversions.txt v1.6 03/07/96 conversion coefficients */ getenv"TZ")= US/Pacific... mode: NORMAL/SCIENCE TLM errors: TIMAS power: ON Temps (Deg C) Maj Fr: 060 INP Current (mA): 410 ICDPS: 36.4 EA1: 56.8 SuperSpin: 1 +5V (V): 4.95 uP: 31.6 EA2: 66.3 Cmd Cnt: 181 +15V (V): 11.9 ME: 49.8 PA: 41.0 VAC IMON (mA): 93.2 SEN: 34.9 PS: 37.9 V BIAS (V): 25.3 Overcurrent Status +10 : ON EA1: OFF EA2: OFF DTPS: OFF HV Status: ENA ROM STANDARD HV PWR IMON HVMON DAC PWR CTRL +VF -VF REF VSEL DTPS1: ON 33.0 2607 BC EA1: ON ENA -0.77 0.73 4175 0 DTPS2: ON 34.4 2607 BC EA2: ON ENA -0.77 0.80 4195 0 PABS: ON 12.4 PA1: ON ENA -0.69 3225 PS_settle: 8 PA2: ON ENA -0.69 3198 DET Elect. CTRL DAC STIM : DIS ARM Status: ARM FULL ANODE: ENA -0.53 VF C0 Disc Adj: 2 TIMAS ARM : ARM MCP: ENA -3580 HV 2 PHA Mode: DIS ^^^^^^^^^^^^^^^^ 16:59:55.002 03/05/97 UT (hk v1.18 ) ^^^^^^^^^^^^^^^^^ ^C exit hk timas@cis% hkstat look at housekeeping status values Running hkstat v1.13 getenv("TZ")= US/Pacific... mode: NORMAL/SCIENCE TLM errors: TIMAS power: ON Command/TLM Status Maj Fr: 089 Cmd Receipt: OK SuperSpin: 3 Instrument Mode: SCIENCE Cmd Executed: OK Cmd Cnt: 181 Mode Change: NO DP Command: OK Data Load: OK TLM Queue: OK Esweep Control ICP Control: OK LUT Source: ROM ICP Parameter: OK Mode: STD PAD: OK Reduced: NO Initialization Status LUT Status IMP DP MLUT: OK RAM: OK OK Operations Esweep: OK ROM: OK OK DP Health: OK Shared RAM: OK OK Data Acq Mode: SUN Data/MS Acq: OK Debug Initiated: NO TLM Valid: YES ^^^^^^^^^^^^^^^^ 17:04:21.802 03/05/97 UT (hk v1.13 ) ^^^^^^^^^^^^^^^^^ ^C timas% exit Connection closed. -------------------------------------------------------------------