23 Sept. 2002

      Denying access to probing attacks using the CHECK_404 plugin 

I.    Introduction
II.   Installation 
II.a.     Installing the initialization procedure
II.b.     Installing the pre-request procedure
II.c.     Installing the post-response procedure
III.  How does CHECK_404 work.
IV.   Specifying response options
IV.a.     Parameters in CHK_PRE.REX
IV.b.     Parameters in CHK_POST.REX
V.    The penalty addon

                        ------------------

I. Introduction

The CHECK_404 plugin for SREhttp/2 is used to deny access to clients
who've committed too many "404" requests (requests for non-existent
resources) in the past several minutes. The reason for 
doing this is to stop hackers who are randomly searching for
system files.

The basic idea is:

     If an excessive number of "404" (resource not found) responses
     have been recently sent to an ip address, then this client is 
     assumed to be an attacker, and no further requests from
     him will be acceted..  This assumption is maintained for several
     minutes, after which the slate is wiped clean (the client is 
     assumed to now be innocent, until the next set of suspicious
     requests).

                        ------------------

II. Installation: 

The CHECK_404 plugin consists of 3 components:
  a) A SRE2003 initialization procedure
  b) A SRE2003 pre-request procedure
  c) A SREhttp/2 post-response procedure

Installing these components is simple: all you have to do is make
a few changes to parameters in SRE2003.CFG and in SREHTTP2.CFG.
No files need to be copied or modified (SREhttp/2 installs all
the needed files in the appropriate directories).


*** Quick installation   ***

   Run the SREhttp/2 configurator, and choose the 
       Enable/Disable the Check-404 facility
   link.

   Make sure you restart SRE2003 -- a link is provided, 
   in the response to "Enable/Disable the Check-404 facility",
   that will automatically restart SRE2003.

*** Sort of quick installation ***

   i) Change INIT_PROCS, in SRE2003.CFG, to
          init_procs=srehttp2\sreh2ini.rxx, chk_init.rex
  ii) Change PRE_REQUEST, in SRE2003.CFG, to
          pre_request=chk_pre.rex
 iii) Change POST_RESPONSE, in PRELOADS.CFG, to
          post_response=procs\CHK_POST.REX

After making these modifications, restart SRE2003  -- CHECK_404 will be on 
the job (using default parameters that in most cases need never be changed). 
Note that this installation is permanent -- every time you start SRE2003 & 
SREhttp/2, CHECK_404 is enabled. Of course, you can always uninstall
CHECK_404 (by undoing steps i to iii).

The next three sections describe steps i to iii in greater detail ...

                        ------------------

IIa. Installing the initialization procedure

Installing the initialization procedure is done by modifying the 
SRE2003 init_procs configuration parameter (in SRE2003.CFG).
   
For example, the default SRE2003.CFG used by SREhttp/2 has:
    init_proc=srehttp2\SREH2INI.RXX
You should change that to:
    init_proc=srehttp2\SREH2INI.RXX, chk_init.rex

Note that CHK_INIT.REX should be in the SRE2003\BIN directory.

You can also change the default values of several parameters when
launching CHK_INIT.REX. To do this, just enter the parameters 
immediately after the CHK_INIT.REX  

The syntax is:

   INIT_PROC = 'CHK_INIT.REX  max404 duration '

 Where:
        max404 == threshold number of 404 responses (default=8)
      duration == duration (in minutes) this threshold is computed over
                  (default=5).
                  If more then max404 "404 responses" have been sent in
                  the last "duration" minutes, deny access.
                  
                    

Notes:
   * to specify duration you must also specify  max404

   * SRE2003.CFG should be in your SRE2003\CFG directory.

   * Note that the use of duration is liberal in an unsophisticated
     kind of way  -- as soon as a client slips beneath the threshold,
     access is allowed. This means that a slow and steady stream 
     of probes will not be thwarted.

     You can set a high value for duration to deal with this,
     but this may end up penalizing sloppy-but-honest clients.

     If this becomes a problem, let us know (danielh@crosslink.net)
     and a more sophisticated algorithim (say, one that credits
     non-404 responses) may be developed.

Example:
   init_procs=chk_init.rex 5 10 , srehttp2\sreh2ini.rxx

Examples (abstracting from other initialization procedures that
might also be listed in INIT_PROCS):

  a) Use the default parameters:
       init_procs='CHK_INIT.REX '
 
  b) Set max404=10, and use the default value of duration
       INIT_PROC ='CHK_INIT.REX 10 ' 

                        ------------------

IIb. Installing the pre-request procedure

Installing the initialization procedure is done by modifying the 
SRE2003 pre_request configuration parameter (in SRE2003.CFG).
   
For example, the default SRE2003.CFG used by SREhttp/2 has:
    pre_request=0
You should change that to:
    pre_request=chk_pre.rex

Notes:
  *  more then one pre-request procedure can be specified.
     For example, if you are using the simple load balancer, then  
     you could use..
       pre_request=balance1.rxx , chk_pre.rex

   * CHK_PRE.REX should be in the SRE2003\BIN directory.

   * SRE2003.CFG should be in your SRE2003\CFG directory.

   * CHK_PRE.REX contains a few user configurable parameters (see
     section IV for the details).

                        ------------------

IIc. Installing the post-response procedure

Installing the post-response procedure is done by modifying the 
SREhttp/2 post_response configuration parameter (in PRELOADS.CFG).
   
For example, the default PRELOADS.CFG used by SREhttp/2 has:
    post_response=0
You should change that to:
    post_response=procs\chk_post.rex


Notes:

   * CHK_POST.REX should be in the SREHTTP2\PROCS directory.

   * PRELOADS.CFG should be in your SREHTTP2\CFG directory.

   * CHK_POST.REX contains a few user configurable parameters (see
     section IV for the details).

                        ------------------

III. How does CHECK_404 work.

The basic algorithim used by CHECK_404 is:

a) When SRE2003 starts, a "CHK_404" cache is created.

   This step can be customized -- you can set the threshold number of "404" 
   responses, and the timespan to consider when counting these responses,
   and a few other variables (see section II.a for details).

b) On all requests, SRE2003 examines this cache (using a pre-request
   procedure) to see if "this client been sent too many 404 responses in 
   the last several minutes".

   If this threshold has been exceeded, SRE2003 will immediately deny
   access to this client.   

   This denial of access consists of either (see section IV for details):
     i) a simple 404 response is sent to the client (that can be modestly
        customize), or
    ii) a redirection to a "custom procedure"
 
c) Assuming b is not binding, SREhttp/2 will process the request.
   If it turns out that the request yields a "404" response (a Not Found
   response), SREhttp/2 will do it's usual thing (such as use MTCHFILE
   to find possible alternatives). 

   However, after responding, the CHECK_404 "post response" procedure
   will note, in the cache setup in step a, that this client has
   been sent a 404 response.

   Note that if the response was NOT a 404 response, the CHECK_404 post-
   response procedure does nothing.

   Exceptions:   

      *  If the client is one of the SUPERUSERS the post-response
         procedure will do nothing.

         That is ... SUPERUSERS are NOT subject to this 404 limit.

         Important detail:

            If the client's IP address is listed in the possibly 
            host-specific SUPERUSER parameter of SREHTTP2.CFG,
            then he is NOT subject to a 404 limit.

            This is NOT THE SAME as having a SUPERUSER privileges 
            assigned  via an entry in USERS.CFG.


      *  As described in section IV, you can specify a set of host_nickname 
         for which this 404 tracking is disabled (that is, you specify hosts
         that will NOT be subject to 404 checking).

      These two exceptions are especially useful when constructing
      a site, a condition where missing links will probably be
      common.

In short, CHECK_404 monitors every request, keeping track of the "404 
response" status on a client specific basis.  This does add extra overhead,
and may slow down throughput.

                        ------------------

IV. Specifying response options

Both the pre-request (CHK_PRE.REX) and the post-response procedures 
(CHK_POST.REX) contain a few user-configurable options. They can be
modified if you want to get fancier about what kind of responses
you send to malfeasant clients, and if you want to specify 
hosts that don't need this protection


IV.a.  Parameters in CHK_PRE.REX

CHK_PRE.REX (the pre-request procedure file) contains two parameters that 
control CHECK_404's response to a "repetitive 404" attack.

REDIRECT_RESPONSE:

   A URL that the client will be redirected to.  Typically, this will be
   an addon on this site, but it could be a simple HTML document (or even
   a URL on a different site).

   A "query string" will be appended to the end of this URL. The
   query string consits of:
        ?url_encoded_selector&client_ip_address

   Thus, if ....
        REDIRECT_RESPONSE=/TOO_MANY404.CMD
   and a client, who has exceeded his max404 threshold, at
        135.22.15.22
   requests
        /FOO/USER.DATA
   Then he will get a redirection response (302) with a location of
         /MANY404.CMD?foo/user.data&135.22.15.22

   (actually, the selector is url-encoded, so the location will be
   /TOO_MANY404.CMD?foo%2fuser.data&135.22.15.22)
             

  If you do NOT want to redirect, set redirect_response='' -- a simple
  response will be sent.

  Example: 
        redirect_response='/system/CHK_RED.cmd'

  Notes:

     * CHK_RED.CMD, in the SREHTTP2\ADDON\SYSTEM directory, 
       is a simple example of an addon that will respond to clients
       who've gone over the limit.

     * Check_404 will allow unlimited accesss to the REDIRECT_RESPONSE
       selector. That is, when a client is over his MAX404 limit, he
       still can request the REDIRECT_RESPONSE selector (say, through a 
       request after a 302 redirection response).


MESSAGE_404:

   A short message to use when directly responding to a request from an 
   attacker (when REDIRECT_RESPONSE=''). 

   This string will be included in a 404 response.
  
   Example: 
     message_404='Resource not available'

   Note that (assuming that REDIRECT_RESPONSE='') a 404 response is sent to 
   attackers EVEN IF THE RESOURCE does exist. Actually, given that a 
   client's max404 threshold has been exceeded, this response is sent by 
   the pre-request component of CHECK_404, which doesn't even try to check 
   on the existence of a resource corresponding to the request.


IV.b.  Parameters in CHK_POST.REX

CHK_POST.REX (the post-response procedure file) contains one parameter that 
controls CHECK_404's response to a "repetitive 404" attack.

NO_CHECK_HOSTS:
  
   A space delimited list of host-nicknames. All request to these hosts
   will NOT be checked for 404s. 

                        ------------------

V. The penalty addon

Some resources are obviously off-limits, and clients who ask for it
are quite likely to be ill-intentioned.  You can use the "penalty
addon" (CHK_PENL.CMD) to "penalize" such clients.

In particular, one would createselector-specific entries in the 
(possibly host-specific) ATTRIBS.CFG file that identify resources that 
are liable to be asked for by attackers, and assign such requests a 
"404 request penalty".

That is, each such request can be treated as a fixed number of 404
responses. One could be stringent, and set this penalty to be
greater then the MAX404 threshold, or one could be more tolerant and
set the penalty to two or three.

Syntax: 

   Realm: chk404_a
   Rule: /usr.dat
   Redirect: internal: /SYSTEM/chk_penl.cmd?penalty&amessage

where
  usr.dat: an example of a file that only an attacker would look for
  penalty: an integer number
  amessage: an optional, url encoded message

Another example:

   realm: chk404_b
   Rule: password.lst 
   Alias: /SYSTEM/CHK_PENL.CMD?500&Unknown+resource

In this first example,  a request for PASSWORD.LST is treated harshly 
(since 500 is likely to be greater then any threshold you might set).  

Notes:

   * amessage is strictly optional -- it is used in a simple
     response to the client. Note that if included, it must be url 
     encoded (i.e.; use + for spaces).

   * By default, CHK_PENL.CMD is located in SREHTTTP2\ADDON\SYSTEM




