FILE:TORUS1:[GLL_RAW.INFO]HOW_TO_SETUP_IPF_PROGS.DOC 12-Jun-1997 jjg 30-Jun-1998 wds 21-Apr-1999 ps This file explains how to set up the various programs on glsdt1.jpl.nasa.gov and ansa.colorado.edu that are used for querying and retrieving instrument packet files. 0. Overview: JPL maintains 2 databases of IPF telemetry data. We use A combination of several programs to query these databases and send the data back to ansa. XXXXX They are mostly in the /ansa2/gll_raw/socket/query_wid or /ansa2/gll_raw/socket/ directory. Here is a brief summary of what each program does. query_wid.pro -user inteface for getting files. run this. queryserver.pro -takes input and writes a queryfile that the database accepts. ipf_socket - csh script that uses queryfile to contact the data base. qpkt - a C program that actually makes a socket connection to send/recieve the data. qpkt_d - the program that runs on glsdt1, it talks to both qpkt and the database query program, stot. This program MUST be running on glsdt1 for this system to work. stot - the data base query program. accepts a queryfile and returns the data. 1. query_wid.pro This program prompts the user for which server to use, what data types to get, and which times or observations to get. You should always be able to query by time, but to query by observation, a file must be created for the orbit, listing the observations and their start and end RIM times. These files should be kept in /ansa2/gll_raw/lrsp/pkt_times/ and named: orbi_euv.lis OR orbi_uvs_rt.lis OR orbi_uvs_pb.lis where orbi is replaced by the full orbit name in lowercase, (e.g. 'g08a') The file should be have this structure: Observation Start End ----------------------------------------- 1) GTORUS01 3941139 3941468 2) GTORUS02 3941802 3942071 3) CUPHAS87 3942106 3942124 It is important for there to be no leading spaces before the '1)' column. Information about running this program is in HOW_TO_GET_IPF.DOC See HOW_TO_MAKE_ANSA_OBSERVATION_LIST_FILES.DOC for details creating the list files. 2. queryserver.pro This idl file contains a procedure that builds a queryfile, called ./tmp_query.pvl based on parameters it recieves from QUERY_WID. The format of the file is described in the comments of this program. 3. ipf_socket This is a shell script that is called by QUERY_WID to actually do the query, using SPAWN (idl). It takes 3 parameters: queryfile - the name of the file made by QUERYSERVER.PRO, usually tmp_query.pvl server - the name of the database server to query from. outfile - the output filename A typical call looks like this: /ansa2/gll_raw/socket/ipf_socket tmp_query.pvl GLL_RT_QueryServer \ /ansa2/gll_raw/lrsp/uvs/uvs_3893489_3895000.rt_bin This shell script is currently configured to start up the qpkt program, but a different version exists to start up an older program. (Note: ipf_socket is actually a symbolic link to the ipf_socket.qpkt script. This is done so that you can change between versions of this program by simply making a different symbolic link) It is also important to know that the hostname of the computer that is used to process the request is hard coded into this script. 4. qpkt This is the C program that actually does all of the data transfer. It is called by the ipf_socket script. It does the following things: 1. makes a connection to the qpkt_d program on the JPL computer. 2. sends the query request information (the parameters to ipf_socket) to qpkt_d. 3. recieves an acknowledgment from qpkt_d and opens another socket connection to recieve the data. 4. waits for data transfer to finish. Recieves a status message from qpkt_d, that tells if the query was successful. 5. deletes the file it wrote if something went wrong in the query. This is a rather complicated program. The source is well documented, (and in /ansa2/gll_raw/socket/qpkt_dir/qpkt_clnt.c ), but there is a lot of essential information can be found on the man pages. Some good topics to look up are make, socket, ntohl, connect, fork, exec, and gethostbyname. It may also be a good idea to find a basic book on C programming and possibly one on Network programming. There is probably some good information available on the www as well. (or you can just bug me) qpkt takes the following parameters: queryfile - same as for ipf_socket server - same as for ipf_socket hostname - name of machine running qpkt_d can be: glsdt1.jpl.nasa.gov or glsdt2... or sdtss10... it is important to give the full name of the computer. output_file- same as for ipf_socket timeout - # of seconds to wait before killing the query. This parameter is used to clean things up if nothing is working. A good value is about 300 seconds. 5. qpkt_d This is the server, or daemon, that runs on glsdt1 and waits for queries to come in. It is intended to run all the time, but does nothing until a connection is made. To start it you need to: 5.1. Make you are on the computer that is named in the ipf_socket script and logged in as ksimmons. (you can use glsdt1, glsdt2, or sdtss10) 5.2. Make sure that qpkt_d is not already running. The best way to do this is by typing: ps -aux | grep qpkt_d This command displays the list of running processes, and then searches for the ones named qpkt_d. You should see exactly 1 line that is similar to this one, if qpkt_d is running: ksimmons 2166 0.0 0.0 16 0 ? IW Jun 10 0:00 ./qpkt_d Note: there may be another line for the grep (search) process: ksimmons 5896 0.0 0.2 32 196 p2 S 09:00 0:00 grep qpkt_d If it is running, you will need to kill it before you can restart it. To do this you want to send a kill signal to its PID (the second column of the ps output, in this example it is 2166) To do this you would type: kill -9 2166 5.3. Now you can start qpkt_d. to do this you want to type: cd ~/qpkt/ ; ./start_qpkt_d You must be in the directory ~/qpkt/ for this program to work. start_qpkt_d is a script that will start the qpkt in the background, so that it can keep running after you log out. If you just type: cd ~/qpkt ; ./qpkt_d the program will run in your current shell, and you will not be able to do anything else until you type Ctrl-C. This is a good way to see if anything is wrong with any part of the system, as all of the error messages, which are normally discarded, will be sent to your screen. The source for this program is in /ansa2/gll_raw/socket/qpkt_srvr.c and qpkt_handler.c qpkt_d does the following things: 1. Waits for a query connection from qpkt 2. When a connection is made, it reads in the query information and starts stot. It also sends a reply to qpkt, informing it to get ready to recieve data. 3. Waits for stot to finish and gets its return value, if it doesn't finish before the timeout time (parameter to qpkt), the query is aborted. 4. Sends a 'done' message to qpkt, cleans up and goes back to waiting. 6.0 stot This program talks to the JPL data bases, retrieves data, and sends it through a socket to another program. We use qpkt_d to control this program, it tells it what data to look for, and where to send it. This program was written by someone at JPL. I got it from Dianne Conner at JPL, and really don't know much else about it. It takes the following parameters: server - name of the query server queryfile - the query data host - the host name and port to connect to. eg. ansa.colorado.edu:1800 -s - this flag says to make a socket connection. We usually call it like this: stot Gll_RT_QueryServer tmp.file ansa.jpl.nasa.gov:7779 -s 6. Common problems There are several common error messages that are reported while running do_query.pro 6.1 Connection refused: this message usually means that qpkt_d is not running, or not running correctly on the machine that is in the ipf_socket script. (Re)starting qpkt_d on the appropriate computer is the best way to fix this. A second possible explanation is software updates to the JPL machines. Make certain you are running the current version of stot! 6.2 host xxxx not found : this means that an IP address could not be found for the hostname in ipf_socket. Check that it is correct, and contains the full hostname (glsdt1.jpl.nasa.gov not glsdt1 ). It could also mean that the computer you are trying to connect to is not working at all. You should try to telnet there, if that works, try restarting qpkt_d... 6.3 exit status other than 1, or exit info other than 0 i.e: ipf_socket: Starting IPF receive program PID 20050 started as read_data_port() connected PID 20050 exited exit status=1, exit info = 1 etc.... This means that stot crashed. This could be because the queryfile isn't formated correctly, or because the database is down. Try the same query on another database. If it doesn't work (exit info should be 0, even if no data is found), then you should kill qpkt_d and restart it in your current shell (see section 5.3) and watch for any error messages while you do the query again. Usually you will see an error from stot that should make some sense. 7. Running the older version. There is an older, less complicated version of this software. It still uses stot, so switching to it may not solve all problems, but it could help. To do this you need to: 1. telnet to glsdt1,glsdt2, or sdtss10 as ksimmons and edit the ~/.rhosts file to contain an entry that looks like this: ansa.colorado.edu your_user_name_on_ansa If there isn't a line with your user name in the file already. 2. On ansa, change the ipf_socket script to be ipf_socket.rcvpkt. First you should make sure ipf_socket is really a symbolic link: cd /ansa2/gll_raw/socket ls -l ipf_socket.* lrwxrwxrwx 1 gebben 40 15 Jun 10 08:58 ipf_socket -> ipf_socket.qpkt -rwxr-xr-x 1 gebben gllraw 608 Jun 13 15:25 ipf_socket.qpkt -rwxr-xr-x 1 gebben gllraw 1007 Jun 13 15:49 ipf_socket.rcvpkt The lrwxrwxrwx and the -> mean that ipf_socket is a symbolic link to ipf_socket.qpkt. If ipf_socket is a symbolic link, you can rm it. Otherwise you should mv it to another name. 3. Make a link to ipf_socket.rcvpkt: ln -s ipf_socket.rcvpkt ipf_socket 4. You should now be able to run do_query using the old version of the software. This version uses rcp and rsh (see man pages) to copy the query info and start stot. It is much slower than the qpkt way, but it is very simple. The data from stot is read to a file by a C program called rcvpkt (source is in /ansa2/gll_raw/socket/rcvpkt_dir/ ).