NSQ - Name Server Queries - version 0.36 - 25 October 2011
==========================================================

Contents
--------
Introduction
Variables
Settings
Cache
Commands
SWIs
Details


Introduction
------------
This module came about when I wanted to implement the use of DNS Block Lists
for AntiSpam and discovered that the default Resolver module was useless for
that purpose, because it assumes that any query that has a request field
starting with a dotted quad (e.g. 12.34.56.78) is a reverse lookup using the
in-addr.arpa zone. Which is incredibly stupid but probably unavoidable
because of the broken design of that module: you can't TELL it to perform a
reverse lookup - it has to decide you want one by itself.

NSQ only queries the in-addr.apra zone if you use the ByAddress SWI.

The module currently provides four *-commands and nine SWIs:

  *NSQCache
  *NSQGet
  *NSQParse
  *NSQSet

  SWI_ByName
  SWI_ByAddress
  SWI_CheckBL
  SWI_TestZones
  SWI_Cache
  SWI_State
  SWI_IPAddress
  SWI_Results
  SWI_Country

Its primary use is to query one or more DNSBLs, but it can also be used to
perform a normal lookup. The module uses a dynamic area to store settings and
cached lookups. Expired entries are removed from the cache at regular
intervals and can also be removed manually. The cache can be initialised with
IP addresses using a file with entries similar to those in the Hosts file.


Variables
---------
Aside from the usual NSQ$Dir variable pointing to the application directory
for NSQ, the following environment variables are defined:

  NSQ$Settings         The full pathname of NSQ's settings file. By default
                       this will be <NSQ$Dir>.Settings.
  NSQ$CleanUpInterval  The time in seconds between automatic clean ups of the
                       cache. Default value is 300.
  NSQ$DefaultTTL       The default 'time to live' for cache entries in
                       seconds. This is used if the query reply doesn't
                       contain a sensible value. Default is 300.
  NSQ$Cache            The full pathname of a Hosts like file with IP
                       addresses to be read in every time the module is
                       loaded or the cache is reset.

The module also uses the values of the Inet$Resolvers, Inet$ResolverDelay and
Inet$ResolverRetries variables, but these are usually controlled by the OS.
If the Hosts file is to be loaded into the cache (see below), the variable
InetDBase$Path must point to the directory containing it.


Settings
--------
NSQ's settings file specifies the block lists to use. It's generally made up
of four sections:

  DNSBL   - zones used for IP address based block list lookups
  RHSBL   - zones used for 'right hand side' - name based - lookups
  Private - IP addresses or ranges considered private
  Country - a zone used for country lookups

Sections start with a 'token' in square brackets. The opening bracket and the
character following it are significant (i.e. you can use [D, [R, [P and [C if
you're in a minimalist mood...).

A valid entry MUST have a non space/tab character in the first column. Any
fields belonging to that entry MUST have a tab or space in the first column
and MUST be seperated by tabs or spaces.

An entry's zone name can be followed by a space or tab and a number or IP
address code. This has to do with the availability check for that zone. Such
a check is generally done by querying the zone with 127.0.0.2 (DNS) or
example.com (RHS). If a block list service does not support this, a number of
1 should be used. This will mark the zone as 'always available', which can
have a negative effect on performance in case the zone does go off line.
If the service uses a different code (IP address only), this should be used.
For country code zones, a 2 can be used to indicate the check should use the
IP address of the first nameserver, which is bound to be listed in the
country zone.

The module can be used with mailservers accepting SMTP connections from other
hosts as well as programs parsing trace fields of messages received from POP3
servers. The responses from block list services must be interpreted
accordingly.

The fields in the DNSBL and RHSBL section entries are:
  the IP address code returned
  a weight value (0-255) for direct connections
  a weight value (0-255) for deep parsing
  a textual remark
The weight value is used to 'weigh' a response. Policy block lists will
return a positive result if the address queried is in a range for users that
cannot connect to mailservers directly. However, if such an address is taken
from an early trace field of a message (i.e. when doing deep parsing) this is
irrelevant and the positive result should be ignored.

The Private section doesn't have zone entries, but the fields must still be
preceded by a tab or a space to be recognised. The fields are:
  the IP address or range
  a bitmask determining the range
Anyone connecting to a mailserver with a private address is suspect, so the
module won't even bother checking and return a weight of 10 straight away.
Private addresses found in trace fields while doing deep parsing are ignored
(weight = zero).

Querying the zone mentioned in the Country section with an IP address returns
a code that indicates in which country the address has been issued. The
weight is optional and defaults to zero.
Be aware that although the two letter codes used here may look like country
code TLDs - and most can be seen as such - they are actually the official ISO
3166-1 alpha-2 codes.
The ISO 3166-1 numeric code is incorporated in the two bottom bytes of the IP
address code that is returned (i.e. United Kingdom = gb = 826 = 127.0.3.58).


Cache
-----
The cache is a dynamic area containing both the settings and the actual
cached query data. The first part - the settings - is built whenever the
module is loaded and the second - the cache - is constructed entry by entry
when queries are performed, but can be initialised using a file specified
with the NSQ$Cache variable.

At regular intervals (5 minutes by default), the cache is purged of entries
which have reached their expiry time. The command *NSQCache -c 0  performs a
manual purge, as does the SWI NSQ_Cache 0. If you use a Cache file you will
need to reset the cache whenever you've modified it. Use *NSQCache -c 1  or
NSQ_Cache 1 for that purpose.

The Cache file is similar to the Hosts file, but uses some extra bits to be
able to add a weight and TTL. Part of it could look like this:

  127.0.0.1       loopback localhost
  192.168.1.1     router
  8.40.57.152
  15.1.3.0/24
  58.40.00.00/16  |   40   2000

The first two lines could be taken straight out of a Hosts file. Their
function is the same as well: they tell NSQ what hostnames belong to certain
(fixed) IP addresses, just like the Hosts file does for the Resolver module.
The third line doesn't have a hostname and NSQ interprets it as a block list
entry. Weight and TTL aren't specified, so the defaults of 200 and
indefinite are used.
The fourth line is similar, but blocks an entire range of IP addresses
(15.1.3.0 to 15.1.3.255, i.e. only the top 24 bits are relevant) and the
fifth line does the same for 58.40.0.0 to 58.40.255.255 but with a weight of
40 and a TTL of 2000 seconds. The pipe symbol | is significant here. If it is
present, the entry is a block list one, even if there is a hostname.

Be careful with this manual blocking functionality. It may look useful to
block spammers by IP address, but many ISPs issue addresses dynamically, so
one of your friends or relatives may inadvertently end up in your private
block list.

If you want to include your entire Hosts file, you can add a line to the
Cache file starting with the word Hosts (may be followed with spaces or tabs
and a comment as usual).


Commands
--------
The commands are mainly for debugging purposes.

*NSQCache displays or manipulates the cache.
Syntax: *NSQCache [-abdersvwx] [-c 0|1|2] [<host>]

  a        Display all cached data in hexadecimal format.
  b ...    Show blocklist entry for the specified host (IP address).
  c 0|1|2  Cache clean up: 0 = expired data, 1 = complete reset (also reloads
           Settings and Cache), 2 = force expiry.
  d        Detailed. Only with option b e s v.
  e        Show main entry data.
  r        Show reply datagram data.
  s        Show settings area.
  v        Show variables area.
  w        Show buffer area.
  x        Show auxiliary reply datagram data

Force expiry will not remove any entries that never expire.


*NSQGet tries to look up an Internet host or IP address.
Syntax: *NSQGet [-abcdfgimrvx] | [-z 0|1] [-s <srvr>] [-t <name>|<nmbr>] <host>

Without options, the host will be looked up using the normal DNS.

  a        Insist on authoritative answers (report failure otherwise).
  b        Blocklist lookup using the zones specified in Settings.
  c        Continue querying other zones after an item has been found.
  d        Return the weight related to deep parsing.
  f        Force a query and ignore any cached data.
  g        Country lookup using the zone specified in Settings.
  i        Present data from the cache even if it has expired.
  m        Display datagram contents in master file format.
  r        Don't recurse.
  s ...    Lookup using the specified server.
  t ...    Lookup a particular type of resource record (name or number).
  v        Be verbose.
  x        Be extra verbose (implies v).
  z 0|1    Check whether the block list zones are available (0=DNS, 1=RHS).


*NSQParse displays DNS information from a datagram stored with a cached entry
or in a binary or textual dump file.
Syntax: *NSQParse [-bmvx] -f <filespec>|<host>

  b        Blocklist lookup.
  f ...    Read from file specified.
  m        Display datagram contents in master file format.
  v        Be verbose.
  x        Be extra verbose (implies v).


*NSQSet displays, sets or clears some internal variables.
Syntax: *NSQSet [-bks on|off] [-i <clean up interval>] [-t <default TTL>]

  b on|off Set or clear blocking of scheduled clean ups.
  k on|off Set or clear keeping stored replies.
  s on|off Set or clear not locking the socket during queries.
  i        Set clean up interval.
  t        Set default TTL.

) don't switch this on unless you like to break things...


SWIs
----
The SWIs working on block lists need those lists to be set up using the
Settings file. Use *NSQCache -c 1  or NSQ_Cache 1 to reload a modified file.

NSQ_ByName (&58200)

 => R0 = pointer to hostname
    R2 = query type or 0 for default

 <= R0 = pointer to cache entry or pointer to standard error block
    R1 = query state or -1 on error
    R2 = IP address (or zero) if query finished
    R3 = corrupted (zero on error)
    V=1 on error - other flags undefined

Performs a normal DNS lookup with the hostname as source.
Be aware that NSQ's cache entries DO NOT have the standard hostent structure
format!


NSQ_ByAddress (&58201)

 => R0 = pointer to IP address (dotted quad) or IP address itself (network
         byte order - i.e. big endian)
    R1 = flags: bit 2 = R0 contains an address, not a pointer
                bit 3 = force query even if cache entry hasn't expired yet

 <= R0 = pointer to cache entry or pointer to standard error block
    R1 = query state or -1 on error
    R2 = IP address (even if query still running)
    R3 = corrupted (zero on error)
    V=1 on error - other flags undefined

Finds a hostname using an IP address.


NSQ_CheckBL (&58202)

 => R0 = pointer to hostname or IP address (dotted quad) or IP address itself
         (network byte order)
    R1 = flags: bit 0 = return weight for deep parsing
                bit 1 = continue querying other zones after item is found
                bit 2 = R0 contains an address, not a pointer
                bit 3 = force query even if cache entry hasn't expired yet
                bit 4 = check country list
    R2 = query type or 0 for default

 <= R0 = pointer to cache entry or pointer to standard error block
    R1 = query state or -1 on error
    R2 = IP address
    R3 = weight (zero on error)
    V=1 on error - other flags undefined

Queries one or more DNS Block Lists which have to be specified in NSQ's
settings file.


NSQ_TestZones (&58203)

 => R0 = zero if testing DNSBLs - else RHSBLs
    R1 = flags: bit 3 = force query even if cache entry hasn't expired yet

 <= R0 = pointer to cache entry or pointer to standard error block
    R1 = query state or -1 on error
    R2 = corrupted
    R3 = number of zones available (zero on error)
    V=1 on error - other flags undefined

Good block list services have some means of determining whether they are
available or not. This SWI performs the necessary checks.


NSQ_Cache (&58204)

 => R0 = 0 - clean up of expired entries
         1 - full reset
         2 - force expiry

 <= all registers unchanged
    flags undefined

This SWI currently only provides actions similar to the *NSQCache command
with the -c option.


NSQ_State (&58205)

 => R0 = query state value

 <= R0 = pointer to the text associated with the state value

Gets a pointer to the textual representation of a state (zero terminated).
Negative values are error conditions (more or less), positive ones processing
states:

  -9 Private
  -8 Deleted
  -7 Expired
  -6 Secondary query failed
  -5 Server failed
  -4 Host not found - try other server
  -3 Host not found
  -2 Host exists - no data
  -1 Unspecified fatal error
   0 OK (succesfully completed)
   1 Processing answer
   2 Incoming data read
   3 Waiting for answer
   4 Sending query
   5 Relaunching query
   6 Timed out
   7 Launching query


NSQ_IPAddress (&58206)

 => R0 = IP address

 <= R0 = pointer to buffer with dotted quad

Creates a dotted IP address string from a network order 32-bit word.
The result is zero terminated and must be copied because the buffer is
transient and will not retain its contents.


NSQ_Results (&58207)

 => R0 = pointer to cache entry
    R1 = null pointer or pointer to buffer for results
    R2 = size of buffer if r1 is not null

 <= R0 = pointer to results buffer or -1 or null pointer if not valid (in
         which case the other registers aren't valid either)
    R1 = query state
    R2 = IP address
    R3 = weight
    R4 = pointer to hostname
    Z=1 on error (i.e. entry invalid) - other flags undefined

Offers an interface to cache entries, both directly and by copying them to a
buffer (determined by the value in R1).


NSQ_Country (&58208)

 => R0 = 0, -1 or pointer to string of country codes

 <= R0 = unchanged if 0 on entry, pointer to output string if -1 and number
         of modified entries otherwise (zero on error)
    flags undefined

Takes a string of two letter country codes, without spaces or tabs but
optionally separated by commas (i.e. 'cn,kr,tw' and 'cnkrtw' will have the
same effect), gives them a weight of 100 in the Country section's zone and
links this zone into the block list chain.
Using zero resets all weights back to zero and removes the link.
Both these actions imply a forced purge of the cache (*NSQCache -c 2) because
cached entries may not be consistent with the changes.

If R0 is -1 on entry, this SWI returns a null terminated string of country
codes (separated by commas), indicating which countries are currently marked.
The result must be copied because the buffer is transient and will not retain
its contents.

Unfortunately, the default Country zone (zz.countries.nerd.dk) does not
support the usual availability check of block list services.


Details
-------
Detailed information about entry formats and such will be added later.
