5 November 2002:     The SRE2003 API

This technical document discusses features of SRE2003 that may be of
interest to those interested in writing their own filters or scripts.
You may also want to examine SRE2PRC.HTM (describes the SRE2003 procedure
libarary) and SRE_DMN.TXT (describes how to use SRE2003 daemons, queues,
and flags).


Contents:

   I. Data structures supported by SRE2003
     I.a. System Variables
     I.b. Global variables:
     I.c. Counter variables:
     I.d. Event Variables:
     I.e. Request specific variables.
     I.f. Request id variables.
     I.g. Caches
     1.h. Queues
     1.i  Data series
     1.j  Repetitive tasks

   2. Using the SRE2003 TEST Mode

      -------------------------------------------------

I. Data structures supported by SRE2003


SRE2003 supports the following data structures:
   System Variables:  static variables set by SRE2003 at startup
   Global variables:  dynamic variables that are available to all threads
                      under SRE2003
   Counter variables: counter variables are used to maintain running counts. 
   Event variables:   event variables change frequently, the most recent value
                      (the event) is retained.
 Request specific variables:
                     request specific variables are unique to a given request.
                     There are  two sets of request specific variables: 
                     pre-defined and custom.
 Request id variables: 
                     Request id variables are a small set of pre-defined request-
                     specifc "identifiers". 

          Caches:   Caches are arrays of variables and their associated 
                    values. These variables can be randomly accessed.
                    Caches have a finite size, when filled a LRU (least
                    recently used) algorithim is used to free up space.

            Lists:  Lists are lists of values. Values can be added or removed
                   from the front or back of this hist.  Lists have a finite
                   size. When filled, a LIFO (or a FIFO) algorithim is used
                   to free up space.

          Queues:  Queues are arrays of values. Values can be added & removed 
                   from the front or back of queues, but only removed from the
                   front.  Queues have no size limit.




I.a. System Variables

 Description:
   This is a set of about 50 variables that contain information on basic SRE2003
   parameters. Essentially, these are the variables set in SRE2003.CFG. 
   Although these parameters can not be changed on the fly, if SRE2003.CFG is 
   changed these variables will be updated (it may take a few minutes).

 Reading & Writing:
   Use the SRE_VALUE function:
      aval=sre_value(varname,,'SRESYS')

   As noted in the description, these are semi-permanent; hence they can not
   be changed on the fly. To modify them, you have to edit SRE2003.CFG.

 List of system variables:

Name                         Example
----------------------------------------------------------------------------
 SRE2003_INI                 F:\test\temp\sre80.ini
 The SRE80_INI file is an OS/2 .INI file. System variables, and other 
 information on flags and daemons, is stored there. This .INI file is
 not used by SRE2003, it is created as a convenience for other programs.

 LASTSTART                   2001.12.10 12:57:53
 The day and time that SRE2003 was started (local time).

 LISTEN_SOCK                 699
 The socket that SRE2003 is monitoring.

 PORT                        80
 The http port that SRE2003 is accepting requests on.

 LIMITBODY                   500000
 Max size of request body.

 LIMITCLIENTS                30
 Max number of simultaneous clients.

 LIMITHEADER                 10000
 Max size of request header.

 LIMITTIMETOTAL              20
 Max seconds per request

 LIMITTIMEWAIT               5
 Max seconds to maintain a connection

 LIMITTIMEINACTIVE           10
 Max seconds of connection inactivity (nothing sent or recieved)

 ENTITY_HEADERS_IGNORE
 A list of entity-headers that SRE2003 should ignore when automatically
 creating Etags.
  
 FILTER                      srelite2\srelite2.rxx
 The filter to use.

 FAST_FILTER                 0
 The fast-filter to use

 DEFAULT_HOSTS               *
 List of hosts that will be assigned to the DEFAULT host-nickname.
 This complements the canonical host name.

 EXIT_PROC                   srelite2\srelexit.rxx
 List of procedures to run when exiting SRE2003.

 INIT_PROC                   bin\inithost.rxx,srelite2\SREL_INI.RXX
 List of procedures to run when starting SRE2003.

 DEFAULT_SOCKET_TIMEOUT      15
 TIme out a socket after this many seconds of waiting for data

 SERVER_SOFTWARE             SRE2003 for OS/2 ver 1.12d
What SRE2003 reports as it's name.

 STATUS_ITEMS                ERRORS LIMITS CLIENTS STAT CONNECTIONS PEAK
 Items to list on the status screen.

 ERRORFILE                   log\error.log
 Where to write errors.

 AUDITFILE                   log\audit.log
 Where to write audit information.

 AUDIT_ITEMS                 CLIENT RESET
 What types of event to audit.

 SERVER_SOFTWARE_SHORT       SRE2003 1.12d
 Short descriptive name for SRE2003

 UNALLOWED_FILE              F:\test\cfg\BADIPS.IN
 File listing "unallowed IP addresses"

 NO_MODIFY_ONTHEFLY          0
 Flag to suppress the ALT-M  option for "on the fly" modification of SRE2003.

 AUDIT_DELAY                 0.25
 MInutes to delay writing to audit file (a "lazywrite" stall)

 MAXMD5S                     500
 Maximum entries in the MD5 cache.

 MAXCOUNTERS                 200
 Maximum number of counters.

 MAXCLIENTNAMES              120
 Maximum entries on the clientnames cache

 MAXSTATVALS                 100
 Number of "most recent" requests to use in calculating performance statistics.

 REQ_CACHE_SIZE              500  
 Maximum number of request cache (the proxy-like cache) entries.

 REQ_CACHE_CALL_ANYWAYS      0
 Call filter even on request cache hit.

 REQ_CACHE_ENABLE            1
 Enable use of the request cache.

 REQ_CACHE_VERIFY            2
 Check no-cache and pragma request headers before using the request cache.

 REQ_CACHE_RECORD            1
 Record request cache hits in cachehit.log file.

 STATUS_MESSAGE              Sre2 1.12
 Short message written on side of status screen.

 TRACKING_ON                 0
 Enable SRE2003's request tracking mode.

 FILTER_VERSION              sreLite2 ver 1.10a
 Descriptive name of the SRE2003 filter being used.

 SECURITY_LEVEL              1
 Controls the ability to use on-line tools to modify SRE2003 parameters.

 GMTOFFSET                   18000
 GMT offset in seconds (add to local time to get GMT time)

 SERVERADDR_CANONICAL       113.12.55.36
 The "canonical" IP address (as reported by TCP/IP).
 SRE2003 will report this value, unless overridden by a non-0 value
 of the SERVERADDR_USE parameter.

 SERVERNAME_CANONICAL       www.foobar.net
 The "canonical" domain name (as reported by TCP/IP).
 SRE2003 will report this value, unless overridden by a non-0 value
 of the SERVERNAME_USE parameter.


 SERVDIR                     F:\test
 The directory containing SRE2003.

 SRV_DIR                     F:\test
 The directory containing SRE2003 (same as SERVDIR)

 SERVDRIVE                   F:
 The drive ontaining SRE2003.

 TEMPDIR                     F:\test\TEMP
 A temporary files directory.

 TMP_DIR                     F:\test\TEMP
 Same as TMP_DIR

 TEMP_DIR                    F:\test\TEMP
 Same as TMP_DIR

 SRE2003_CFG                 F:\test\cfg\SRE2003.CFG
 The SRE2003 configuration file.

 BIN_DIR                     F:\test\BIN
 Directory containing SRE2003 code.

 CFG_DIR                     F:\test\CFG
 Directory containing SRE2003 configuration files (SRE2003.CFG, BADIPS.IN, HOSTINFO.CFG).

 LOG_DIR                     F:\test\LOG
 Directory containing SRE201 log files.

 STORED_DATA_DIR               F:\test\DATA
 Directory for storing miscellaneous data files (for use by filters).

 REQ_CACHE_HITFILE           F:\test\log\CACHEHIT.LOG
 File containing request cache hit log.

 DATADIR                     F:\WWW
 The datadir -- requests for files are assumed to be relative to this directory.

 FILTER_DIR                  F:\test\srelite2
 The directory containing the SRE2003 filter you are using.

 HOSTFILE                    F:\test\cfg\hostinfo.cfg
 The file containing host definitions.

For more details on these "system" variables, see SRE2003.HTM




I.b. SRE "internal", and "user", global  variables:

 Description:
    Similar to system variables in that they are available to all threads, but 
    they are set dynamically (that is, they can be changed on the fly).  

    There are two kinds of "global" variables: SRE (internal)
    and GLOBAL (user).  They are stored in seperate locations, which
    means that each can have variables with the same name (and
    different values).

    SRE2003 makes some use of the "internal" global variables -- 
    for storing temporary  counters, event variables, and small caches. 
    In addition, several filters (such as sreLite2 and SREhttp/2) use
    global variables to store a number of configuration parameters.


 Reading & Writing:
    Use the SRE_VALUE function:
        avalue=sre_value(varname,newval,'SRE')   -- for "internal globals"
        avalue=sre_value(varname,newval,'GLOBAL')   -- for "user globals"
    or 
        avalue=sre_value(varname,newval,'SRELARGE',[ownid])
     
    The first and second are used for quick retrieval of small "textual" 
    variables. In general, casual programmers (such as script authors)
    should use GLOBALS, while filter and IM-module authors can (carefully)
    use SRE.

    The third form (SRELARGE) is used for not-as-quick retrieval of
    any-sized, any-content variables.

    If newval is NOT specified, then read current value of varname.
        newval=''  means "read current value"
        newval=' ' means "set varname equal to ' '"

    Ownid is strictly optional. It is the "id" of the request (say, as provided
    by the sre_dmn_own procedure). Including it will speed up throughput a bit.

 Notes:
    * Global variables are accessible from all threads (daemons) in the 
      SRE2003 process. However, we recommend that the writing of global
      variables be limited to only one thread; although there are no
      limits on who can read global values.
    * They are not easily accessible from other processes.
    * Servicing (reading and writing) SRELARGE global variables is done via a 
      dedicated daeamon (the SRE_VAR.RXX daemon). That means it's not very fast.
    * SRE2003 reserves several "internal" global variables for use as 
      dynamically changing counters, and for transaction and request 
      specific information.
      These are:
         TRAN_INFO_CT, REQ_INFO_CT,
         CLIENTNAME_CACHE, HOSTNAME_CACHE, 
         several versions of REQ_nn_INFO  where nn are integer "thread ids"
         several CT_varname variables, where varname is one of: 
            BYTESREADTOTAL  BYTESSENTTOTAL TRANSACTIONS 
            REQUESTS LIMITS ERRORS BROKEN CACHEHITS) </em> </td>
         several EVENT_varname variables, where varname is one of: 
            LASTACCEPT LASTIDLE LASTRESET LASTADDRESS PEAKCLIENTS 

I.c. Counter variables:
 Description:
    Counter variables are a special kind of global variable.
    They are used to maintain running counts. 
    Morever, for a "quicklist", of several variable, a speed optimization 
    is supported.

 Reading & Writing:
   Use the SRE_COUNTVARS function: 
           a_count=SRE_COUNTVARS(varname,avalue,ownid)
   where:
       varname: the counter variable. If you begin varname with a "=" (without
                the quotes) then "force" an accurate read
       avalue:  the value to add 
                If avalue is not specified, return current value of varname
        ownid:  optional, the clients "own_id".

 The Counter Quicklist:
   The "counter quicklist" contains the following variables:
       BYTESREADTOTAL BYTESSENTTOTAL TRANSACTIONS  REQUESTS LIMITS ERRORS  
   [see SRE2003.HTM for a decription of these counter variables]

   Variables that are in this quicklist are read using a speed optimization.
   However, use of this speed optimization may yield inaccurate results, if the
   count is changing rapidly. In particular, the value returned may not be 
   up-to-date.
   If you require an accurate (up-to-date) value for a quicklist variable,
   use the "prepend varname with an  = sign" (as noted above).

       For example:
           accurate_ct=SRE_COUNTVARS('=TRANSACTIONS')

 Notes:
      *  For further details on SRE_COUNTVARS, see SRE2PRC.HTM
      *  By "up-to-date", we mean "within the last few seconds"

I.d. Event Variables:
 Description:
    Event variables are global variables that change frequently.  They are
     similar to count variables (they change frequently), but instead of
     augmenting a counter, the most recent value (the event) is retained.

   As with counter variables, an "optimized for speed quicklist" of event
   variables is maintained.

 Reading & Writing:
   Use the SRE_EVENTVARS function:
      oldval=sre_eventvars(varname,newvalue,ownid)
   where:
      varname: the event variable. If you begin varname with a "=" (without
                 the ") then "force" an up to date read.
      newvalue: the value to store 
                 If avalue is not specified, then return current value
                 of varname
       ownid: optional, the clients "own_id". This is used ONLY if you              
                 are reading the value of a non "quick list" event. 
                 If not specified, an own_id will be generated automatically.

       For further details, see SRE2PRC.HTM

 The Event Quicklist:
    The "event quicklist" contains the following variables:
            LASTACCEPT LASTIDLE LASTRESET LASTADDRESS
    [see SRE2003.HTM for a decription of these event variables]

 Notes:
      *  For further details on SRE_EVENTVARS, see SRE2PRC.HTM

    
I.e. Request specific variables.

  Description
     Request specific variables are unique to a given request. There are 
     two sets of request specific variables: pre-defined and custom.

  Reading & Writing:
    The pre-defined request specific variables can be read using the EXTRACT or
    SRE_COMMAND functions -- see SRE2003.HTM for the details.

    The custom variables can be read and written with the SRE_VALUE function.
      aval=sre_value(varname,newval,'REQ')  
    As defined above, if newval is not specified, the value of varname is read.
    If newval is specified, varname will be given the value of newval.

    Each request has it's own custom variables. That is: custom variables are
    defined for requests, not for transactions. Hence, each request in a
    "maintained connection" will have it's own values (that is, values do
     not carry over to the next request read from a given connection).

I.f. Request id variables.

     Request id variables are unique to a given request. They are a small
     set of pre-defined request-specifc "identifiers".  In particular:

      CLIENT - client id of this request
        HOST - host name
         NTH  - the transaction number (which will be the same for 
                maintained-connection requests)
         OWN - "own id" of this reqeust (used to "ask" for info)
   PARENTID - the thread id of this requests "parent" (a transaction daemon)
          SEL - selector
        SOCK - socket of this request
       START - starttime (date & second)
        TRAN - return transaction id of this request (the transaction of 
               this request)
  
   Read & Write
       aval=sre_request_info(avar,tid,ownid)
    where:
        avar - variable to lookup
         tid  - optional. The request identifier. 
                If not specified, lookup info for the current request.
                Or, the tid can be the "id_info" (provided as the 5th argument
                to the filter) -- using this can speed up processing a bit. 
      ownid - optional. The caller's "own id". This can speed up some lookups a
              little bit.          
    
      See SRE2003.HTM for the details.      

I.g. Caches

 Description:
     Caches are arrays of variables and their associated values. 
     Each entry (array & variable) in a cache can be read independently 
     (using the variable name).  
     One can freely add new entries to a cache, up until the cache is full. 
     When the cache is full, new entries will cause old entries (in the 
     least recently used sense) to be removed (so as to make room for
     new variables).
        
 Reading & Writing:
   Use the SRE_CACHE function.
        val=sre_cache(cache_name,action,varname,varvalue,time_to_live,ownid)
   where
        cache_name:  The cache to work with. Each cache is seperately 
                     maintained -- each cache has it's own size limit.

        action: What to do. Supported actions include:
                  CREATE RESET WRITE ADD READ PRUNE REMOVE STATUS LIST

        varname: The variable to read, write, etc.

        varvalue: The value of the variable (used with WRITE and ADD)

        time_to_live: The lifespan of this cache entry. After this many 
                      minutes, the entry is removed

        ownid: Optional. The ID of the daemon that is calling sre_cache.

   For a complete description of SRE_CACHE, sre SRE2PRC.HTM.
         
 SRE2003 internal caches.
    SRE2003 creates and uses the following caches for internal purposes:
         REQUEST: the request cache (for quickly matching URIs to files)
         IPNAME:  the IP name cache (for matching ip addresses to domain names)
         MD5:     the MD5 cache     (for matching files to their MD5 hashes)
         COUNTS:  for counters (such as BYTESREADTOTAL and LIMITS)
         EVENTS:  for retaining events (such as LASTACCEPT and PEAKCLIENTS)
         REQ_INFO: queue and semaphore information for request daemons
         TRAN_INFO: queue and semaphore information for transaction daemons

 Notes:
    * Caches are implemented through the SRE_CSH.RXX daemon.
    * If cachename is "!LIST" (without the quotes), a list of currently active 
      caches is returned.

1.h. Lists
 
 Description
    Lists are arrays of values. Values can be added to the front or back 
    of the list, and can also be read or removed from either end. 
    Values can be added until the size of the list is reached, at which point
    when a new value is added, an old value is removed (just one value is
    remove). 

    A list differs from queues and caches in several ways:
        i) Compare to a cache; there is no variable name associated with each 
           entry in a list.
           There is a position (defined from the bottom of the list), though
           this position can change as new items are added
  
       ii) There is no least recently used, or time to live, mechanism for 
           selecting which items to remove when more space is needed. Instead,
            the  "oldest" element is dropped; where "old" simply means a value
            on the other end of the list!

      iii) A queue can only have it's top element read. 

       if) The size of a queue is unlimited.

     Basically --
           Caches are random access with a dynamic LRU algorithim used to
           remove old  entries. 
           List are sequential, with a simple overflow mechanism used to
           remove old entries.
           Queues are sequential with no internal readsd, and have no size limit.



 Reading & Writing:
   Use the SRE_LIST function:
      val=sre_list(list_name,action,avalue,ownid)
   where:
        list_name:  The list to work with. Each list is seperately 
                     maintained --  each list has it's own size limit.
        action: What to do. Supported actions include:
                CREATE RESET PUSH QUEUE POP PEEK READ STATUS LIST
        avalue: A value to add to the queue
        ownid: Optional. The ID of the daemon that is calling sre_list

 SRE2003 internal queues:
    SRE2003 creates and uses the following lists for internal purposes --
    they retain information for the last hundred or so requests:
        BYTES :  bytes transmitted 
        TOTAL_TIME : time required to complete request
        START_TRANSMIT_TIME : time required before first byte was sent
        CLIENTADDR : client's IP address
        REQUEST : the requst line
         
    For details on these "status" variables, see SRE2003.HTM.

 Notes:
    * Queues are implemented through the SRE_LIST.RXX daemon.
    * For further details on SRE_LIST, see SRE2003.HTM.
    

1.i.  Data Series

    SRE2003's "data series" are simple databases that are designed to be quickly
    accessed.  Compared to "caches", data-series are permanent -- they are 
    stored in fixed-length text files. Data series do have some cache-like features; 
    with recently read or modified values retained in memory.

    Features of data series:
        * You can have several data series open at the same time.
        * Each record in a data series is referenced by an arbitrarily long 
         identifier string.  
        * Random access, using a memory resident index, is used to lookup records.
        *  Each record can have up to 15 unique variables assigned to it. 
        *  A few simple "increment" functions are easily implemented with data series.
        *  SRE2003 data series are designed to be used with small to mid sized
           database. Large databases (say, greater the 10,000 records), or wide
           database (say, greater then 1000 bytes per record) are not efficiently
           supported as an SRE2003 data series.  
        *  SRE2003 data series are fairly robust, but they should not be used with
           crucial transaction data. That is, on occassion SRE2003 may fail to
           record a change (say, when the server prematurely closes).

  Reading and writing:

       Use the SRE_CACHE_DATA function to create, write, read, and delete SRE2003
       data series. The basic syntax of SRE_CACHE_DATA is:

          aresult=sre_cache_date(series,action [options],identifier,val1,..,val10)

       where:

        series:  a data series.  
        action: what to do.  The following actions are available:
            DEFINE : create a data series.  
            OPEN   : open a currently existing data series
            ADD    : add a  new record
            MODIFY : modify a record
            EXTRACT : Extract one or several records, by record number
            FIND   : find a record, or records
            REMOVE : remove a record
            CLOSE  : close the data series (save cached information)
            INFO   : info about a data series
            SAVE   : save any cached information
            CLEANUP : cleanup data
            REWRITE : rewrite the data file (fix odd records, remove old junk)
            DELETE  : delete a data series
        
          [options]  Space delimited list of optional action modifiers.

       identifier : an identifier 

       val1 .. val10 : up to 10 values, whose used depends on action
 
       aresult: depends on action.


  For the details on working with SRE2003 data series, and on how to use the 
  SRE_CACHE_DATA procedure, see SRE_DCSH.TXT.


1.j.  Tasks

   The SRE2003  task manager is a facility that simplifies the execution of non-request 
   related tasks. For example, the task manager can be used to schedule recurring 
   housekeeping tasks (such as cleaning up temporary storage), or to produce output 
   that a client will obtain at a later date.

   Running tasks is straightforward process of defining the task, and then launching 
   instances of this task as needed (or scheduling instances to run a number of times). 
   The main advantage of using the SRE2003 task manager, as opposed to a CRON 
   like system utility, is that tasks can be easily launched by daemons, or by 
   web based scripts.


   Using tasks:

     The SRE_TASK procedure is used to specify, invoke, and other work with tasks.

     Syntax:    
        astat=sre_task(action,name,controls,data)

     where:

       ACTION The action to take. Four actions are recognized:

         * FAMILY = define a family (and set some parameters)
         * DEFINE = define a task (assign a task NAME to a rexx procedure), or 
                    remove an already defined task
         * LAUNCH = launch a task
         * STATS = compute some statistics

      NAME is a task name or a family name. The task name can also include
      a family name.

      CONTROLS is optional control information.
      DATA is used to pass ACTION specific parameters.

      ASTAT depends on action.

  For more details on tasks, and on how to use SRE_TASK, see SRE_TASK.HTM.

      -------------------------------------------------

2. Using the SRE2003 TEST Mode

To facilitate quick tests of a running server, SRE2003 supports a simple TEST
mode. The test mode uses a special TEST verb (which is NOT a standard http/1.1
protocol).

When a TEST request is recieved by SRE2003, it looks for a modifier (in the 
selector portion of the request), and sends back a simple 1 line response. 
That is,  only a response line is returned -- no headers, and no body, 
are returned to the client.

Several "modifiers" are supported:

    ADDRESS == Returns the server's dotted numeric IP address
    CLIENT  == Return's the client's dotted numeric IP address
    NAME    == Returns the server's IP name (domain name)
    PING    == Returns a "1"
    PORT    == Returns the server's port (typically, 80)
    TIME    == Returns the time and date    
   TRANSACTION == Returns the current transaction number 
Examples:
  Request:  TEST NAME HTTP/1.1
  Response: 200 foo.bar.net http/1.1

  Request:  TEST ADDRESS HTTP/1.1
  Response: 200 121.151.2.5 http/1.1

  Request:  TEST PING HTTP/1.1
  Response: 200 1 http/1.1

  Request: TEST time HTTP/1.1
  Response:  200 02:58:07 21 Jul 2001 http/1.1

Notes:
  * The DOTEST.CMD program demonstrates how the TEST mode is used.
  * The TEST mode is used by the SREWATCH program
  * SRE2003 will count TEST mode requests as transactions, but will
     i) NOT record information to any AUDIT file. 
    ii) NOT call the filer
 
