The Swish-e program is controlled by command line arguments (called
switches). Often, it is run manually from a shell (command prompt), or from a
program such as a CGI script that passes the command line arguments to
swish.
Note: A number of the command line switches may be specified in the Swish-e
configuration file specified with the -c command line argument. Please see SWISH-CONFIG for a complete description of available configuration file directives.
There are two basic operating modes of Swish-e: indexing and searching.
There are command line arguments that are unique to each mode, and others
that apply to both (yet may have different meaning depending on the
operating mode). These command line arguments are listed below, grouped by:
INDEXING -- describes the command line arguments used while indexing.
SEARCHING -- lists the command line arguments used while searching.
OTHER SWITCHES -- lists switches that don't apply to searching or indexing.
Beginning with Swish-e version 2.1, you may embed its search engine into
your applications. Please see SWISH-LIBRARY.
Swish-e indexing is initiated by passing command line arguments to swish. The command line arguments used for searching are described in SEARCHING. Also, see SWISH-SEARCH
for examples of searching with Swish-e.
The -h switch (help) will list the available Swish-e command line arguments:
swish-e -h
Typically, most if not all indexing settings are placed in a configuration
file (specified with the -c switch). Once the configuration file is setup indexing is initiated as:
swish-e -c /path/to/config/file
See SWISH-CONFIG for information on the configuration file.
Security Note: If the swish binary is named swish-search then swish will not allow any operation that would cause swish to write to
the index file.
When indexing it may be advisable to index to a temporary file, and then
after indexing has successfully completed rename the file to the final
location. This is especially important when replacing an index that is
currently in use.
swish-e -c swish.config -f index.tmp
[check return code from swish or look for err: output]
mv index.tmp index.swish-e
This specifies the directories and/or files to index. Directories will be
indexed recursively. This is typically specified in the configuration file with the IndexDir directive instead of on the command line. Use of this switch overrides the
configuration file settings.
This specifies the method to use for accessing documents to index. Can be
either fs for local indexing via the file system (the default),
http for spidering, or prog for reading documents from an external program.
Located in the conf directory are example configuration files that demonstrate indexing with
the different document source methods.
See the SWISH-FAQ for a discussion on the different indexing methods, and the difference
between spidering with the http method vs. using the file system method.
The fs method simply reads files from a local (or networked) drive. This is the
default method if the -S switch is not specified. See SWISH-CONFIG for configuration directives specific to the fs method.
The http method is used to spider web servers. It uses an included helper program
called swishspider located in the src directory. Swish needs to be able to locate this program when using the http method. See SWISH-CONFIG for configuration directives specific to the http method.
By default, swish looks in the current directory for the swishspider program, or in the directory specified by the SwishSpiderDir directive. The first line of the swishspider program (the ``shebang'' line) must point to the location of the Perl
program (if your operating system uses it).
Security Note: Under Windows swish passes the URLs fetched from remote
documents through the shell (swish uses the system() command
for running swishspider under Windows), and this may be considered an additional security risk.
The http method is depreciated (or at least not very well appreciated). Consider
using the prog method described below for spidering. There's a spider program available in
the
prog-bin directory for use with the prog method.
By default, this method of spidering only indexes files that have a content
The Swish-e distribution includes perl modules to make converting non-text
documents into a format that Swish-e can parse easy. The helper script swishspider will use these modules if installed. These modules only provide an
interface to programs that do the conversion. For example, you will need to
download and install the ``catdoc'' program to convert MSWord documents
into text for indexing. Please see filters/README to see how to use this filter system.
The prog method is new to Swish-e version 2.2. It's designed as a general purpose
method to feed documents to swish from an external program.
For example, the external program can read a database (e.g. MySQL), spider
a web server, or convert documents from one format to another (e.g. pdf to
html). Or, you can simply use it to read the files of the file system (like -S fs), yet provide you with full control of what files are indexed.
The external program name to run is passed to swish either by the IndexDir directive, or via the -i option. Additional parameters may be passed to the external program via the SwishProgParameters directive.
A special name ``stdin'' may be used with -i or IndexDir
which tells swish to read from standard input instead of from an external
program. See example below.
The external program prints to standard output (which swish captures) a set
of headers followed by the content of the file to index. The output looks
similar to an email message or a HTTP document returned by a web server in
that it includes name/value pairs of headers, a blank line, and the
content.
The content length is determined by a content-length header supplied to
swish by the program; there is no ``end of record'' character or flag sent
between documents. Therefore, it is critical that the content-length header
is correct. This is a common source of errors.
One advantage of this method (over using filters, for example) is that the
external program is run only once for the entire indexing job, instead of
once for every document. This avoids forking and creating a new process for
every document, and makes a huge difference when your external program is
something like perl that has a large startup cost.
Here's a simple example written in Perl:
#!/usr/local/bin/perl -w
use strict;
# Build a document
my $doc = <<EOF;
<html>
<head>
<title>Document Title</title>
</head>
<body>
This is the text.
</body>
</html>
EOF
# Prepare the headers for swish
my $path = 'Example.file';
my $size = length $doc;
my $mtime = time;
# Output the document (to swish)
print <<EOF;
Path-Name: $path
Content-Length: $size
Last-Mtime: $mtime
Document-Type: HTML*
EOF
print $doc;
The external program passes to swish a header. The header is separated from
the body of the document with a blank line. The available headers are:
This is the name of the file you are indexing. This can be any string, so
for example it could be an ID of a record in a database, a URL or a simple
file name.
This header specifies the length in bytes of the document that follows the
header. This length must be exactly the length of the document -- do not
make the mistake of adding an extra line feed at the end of the document.
You may override swish's determination of document type (Indexcontents) by using the Document-Type: header. The document type is used to select which parser Swish-e uses to
parse the document's contents.
For example, a spider program might map the content-type returned from a
web server to one of the types Swish-e understands. For example,
my $doc_type = 'HTML*' if $response->content_type =~ m!text/html!'
This header is not required.
The above example program only returns one document and exits, which is not
very useful. Normally, your program would read data from some source, such
as files or a database, format as XML, HTML, or text, and pass them to
swish, one after another. The Content-Length: header tells swish where each document ends -- there is not any special
``end of record'' character or marker.
To index with the above example you need to make sure that the program is
executable (and that the path to perl is correct), and then call swish
telling to run in prog
mode, and the name of the program to use for input.
This gives an easy way to run swish without a configuration file with a -S prog program that requires parameters.
Using ``stdin'' might also be useful for programs that call swish (instead
of swish calling the program).
(The reason ``stdin'' is used instead of the more common ``-'' dash is due
to the rotten way swish parses the command line. This should be fixed in
the future.)
The prog method bypasses some of the configuration parameters available to the file
system method -- settings such as
IndexOnly, FileRules, FileMatch and FollowSymLinks
are ignored when using the prog method. It's expected that these operations are better accomplished in the
external program before passing the document onto swish. In other words,
when using the prog method, only send the documents to swish that you want indexed.
You may use swish's filter feature with the prog method, but performance will be better if you run filtering programs from
within your external program. See also filters/README for an example how to easily add document converstion and filtering into
your Perl-based programs.
Notes when using -S prog on MS Windows
Windows does not use the shebang (#!) line of a program to determine the
program to run. So, when running, for example, a perl program you will need
to specify the perl.exe binary as the program, and use the
SwishProgParameters to name the file.
Swish will replace the forward slashes with backslashes before running the
command specified with
IndexDir. Swish uses the popen(3) command which passes the command
through the shell.
If you are indexing, this specifies the file to save the generated index
in, and you can only specify one file. See also IndexFile in the configuration file.
If you are searching, this specifies the index files (one or more) to
search from. The default index file is index.swish-e in the current
directory.
Specify the configuration file(s) to use for indexing. This
file contains many directives that control how Swish-e proceeds. See SWISH-CONFIG for a complete listing of configuration file directives.
Example:
swish-e -c docs.conf
If you specify a directory to index, an index file, or the verbose option
on the command-line, these values will override any specified in the
configuration file.
You can specify multiple configuration files. For example, you may have one
configuration file that has common site-wide settings, and another for a
specific index.
The settings in the configuration file will be used to index a site.
These command-line options will override anything in the configuration
file.
The variables in swish-e.conf will be read, then the variable in
stopwords.conf will be read. Note that if the same variables occur in both
files, older values may be written over.
For large sites indexing may require more RAM than is available. The -e switch tells swish to use disk space to store data structures while
indexing, saving memory. This option is recommended if swish uses so much
RAM that the computer begins to swap excessively, and you cannot increase
available memory. The trade-off is slightly longer indexing times, and a
busy disk drive.
Specifying this option tells swish to follow symbolic links when indexing.
The configuration file value FollowSymLinks will override the command-line value.
The default is not to follow symlinks. A small improvement in indexing time
my result from enabling FollowSymLinks since swish does not need to stat
every directory and file processed to determine if it is a symbolic link.
The -N option takes a path to a file, and only files newer than the specified file will be indexed. This is helpful for creating
incremental indexes -- that is, indexes that contain just files added since
the last full index was created of all files.
Example (bad example)
swish-e -c config.file -N index.swish-e -f index.new
This will index as normal, but only files with a modified date newer
than F<index.swish-e> will be indexed.
This is a bad example because it uses index.swish-e which one might assume was the date of last indexing. The problem is that
files might have been added between the time indexing read the directory
and when the index.swish-e file was created -- which can be quite a bit of time for very large
indexing jobs.
The only solution is to prevent any new file additions while full indexing
is running. If this is impossible then it will be slightly better to do
this:
The -v option can take a numerical value from 0 to 3. Specify 0 for completely
silent operation and 3 for detailed reports.
If no value is given then 1 is assumed. See also IndexReport in the configuration file.
Warnings and errors are reported regardless of the verbosity level. In
addition, all error and warnings are written to standard out. This is for
historical reasons (many scripts exist that parse standard out for error
messages).
The following command line arguments are available when searching with
Swish-e. These switches are used to select the index to search, what fields
to search, and how and what to print as results.
This section just lists the available command line arguments and their
usage. Please see SWISH-SEARCH for detailed searching instructions.
Warning: If using Swish-e via a CGI interface, please see CGI Danger!
Security Note: If the swish binary is named swish-search then swish will not allow any operation that would cause swish to write to
the index file.
This performs a case-insensitive search using a number of keywords. If no
index file to search is specified (via the -f switch), swish-e will try to search a file called index.swish-e in the
current directory.
swish-e -w word
Phrase searching is accomplished by placing the quote delimiter (a
double-quote by default) around the search phrase.
swish-e -w 'word or "this phrase"'
Search would should be protected from the shell by quotes. Typically, this
is single quotes when running under Unix.
Under Windows command.com you may not need to use quotes, but you will need to backslash the quotes
used to delimit phrases:
swish-e -w \"a phrase\"
The phrase delimiter can be set with the -P switch.
The search may be limited to a MetaName. For example:
swish-e -w meta1=(foo or baz)
will only search within the meta1 tag.
Please see SWISH-SEARCH for a description of MetaNames.
-f *file1 file2 ...* (index files)
Specifies the index file(s) used while searching. More than
one file may be listed, and each file will be searched. If no -f switch is specified then the file index.swish-e in the current directory will be used as the index file.
Sets the begining search result to return (records are numbered from 1). This switch can be
used with the -m switch to return results in groups or pages.
Example:
swish-e -w 'word' -b 1 -m 20 # first 'page'
swish-e -w 'word' -b 21 -m 20 # second 'page'
The -t option allows you to search for words that exist only in specific HTML
tags. Each character in the string you specify in the argument to this
option represents a different tag in which to search for the word. H means
all HEAD tags, B stands for BODY tags, t is all TITLE tags, h is H1 to H6
(header) tags, e is emphasized tags (this may be B, I, EM, or STRONG), and
c is HTML comment tags
Set the delimiter used when printing results. By default, Swish-e separates
the output fields by a space, and places double-quotes around the document
title. This output may be hard to parse, so it is recommended to use -d to specify a character or string used as a separator between fields.
The string dq means ``double-quotes''.
swish-e -w word -d , # single char
swish-e -w word -d :: # string
swish-e -w word -d '"' # double quotes under Unix
swish-e -w word -d \" # double quotes under Windows
swish-e -w word -d dq # double quotes
The following control characters may also be specified: \t \r \n \f.
This causes swish to print the listed property in the search results. The
properties are returned in the order they are listed in the -p argument.
Properties are defined by the ProperNames directive in the configuration file (see SWISH-CONFIG) and properties must also be defined in MetaNames. Swish stores the text of the meta name as a property, and then will return this text while searching if this option is used.
Properties are very useful for returning data included in a source documnet
without having to re-read the source document while searching. For example,
this could be used to return a short document description. See also see Document Summeries and PropertyNames in SWISH-CONFIG.
To return the subject and category properties while indexing.
swish-e -w word -p subject category
Properties are returned in double quotes. If a property contains a double
quote it is HTML escaped ("). See the -x switch for a more advanced method of returning a list of properties.
NOTE: it is necessary to have indexed with the proper PropertyNames
directive in the user config file in order to use this option.
Normally, search results are printed out in order of relevancy, with the
most relevant listed first. The -s sort switch allows you to sort results in order of a specified property, where a property
was defined using the MetaNames and PropertyNames directives during indexing (see SWISH-CONFIG).
The string passed can include the strings asc and desc to specify the sort order, and more than one property may be specified to
sort on more than one key.
The -L switch can be used to limit search results to a range of property values
Example:
swish-e -w foo -L swishtitle a m
finds all documents that contain the word foo, and where the document's title is in the range of a to m, inclusive. By default, the case of the property is ignored, but this can
be changed by using PropertyNamesCompareCase
configuation directive.
Limiting may be done with user-defined properties, as well.
For example, if you indexed documents that contain a created timestamp in a
meta tag:
<meta name="created_on" content="982648324">
Then you tell Swish that you have a property called created_on, and that it's a timestamp.
PropertyNamesDate created_on
After indexing you will be able to limit documents to a range of
timestamps:
-w foo -L created_on 946684800 949363199
will find documents containing the word foo and that have a created_on date
from the start of Jan 1, 2000 to the end of Jan 31, 2000.
Note: swish currently does not parse dates; Unix timestamps must be used.
Two special formats can be used:
-L swishtitle <= m
-L swishtitle >= m
Finds titles less than or equal, or grater than or equal to the letter m.
This feature will not work with swishrank or swishdbfile properties.
This feature takes advantages of the pre-sorted tables built by swish
during indexing to make this feature fast while searching. You should see
in the indexing output a line such as:
6 properties sorted.
That indicates that six pre-sorted tables were built during indexing. By
default, all properties are presorted while indexing. What properties are
pre-sorted can be controlled by the configuration parameter PreSortedIndex.
Using the -L switch on a property that was not pre-sorted will still work, but may be much
slower during searching.
This is an experimental feature, and its use and interface are subject to
change.
The -x switch defines the output format string. The format string can contain
plain text and property names (including swish-defined internal property
names) and is used to generate the output for every result. In addition,
the output format of the property name can be controlled with C-like printf
format strings. This feature overrides the cmdline switches -d and -p, and a warning will be generated if -d or -p are used with -x.
For example, to return just the title, one per line, in the search results:
swish-e -w ... -x '<swishtitle>\n' ...
Note: the \n may need to be protected from your shell.
See also ResultExtFormatName for a way to define named
format strings in the swish configuration file.
the name of a user property as specified with the config file directive
``PropertyNames''
the name of a swish Auto property (see below). These properties are defined
automatically by swish -- you do not need to specify them with
PropertyNames directive. (This may change in the future.)
propertynames must be placed within ``<'' and ``>''.
User properties:
Swish-e allows you to specify certain META tags within your documents that
can be used as document properties. The contents of any META tag that has been identified as a document
property can be returned as part of the search results. Doucment properties
must be defined while indexing using the PropertyNames
configuration directive (see SWISH-CONFIG).
Swish defines a number of ``Auto'' properties for each document indexed.
These are available for output when using the -x format.
Name Type Contents
-------------- ------- ----------------------------------------------
swishreccount Integer Result record counter
swishtitle String Document title
swishrank Integer Result rank for this hit
swishdocpath String URL or filepath to document
swishdocsize Integer Document size in bytes
swishlastmodified Date Last modified date of document
swishdescription String Description of document (see:StoreDescription)
swishdbfile String Path of swish database indexfile
The Auto properties can also be specified using shortcuts:
Use a double percent sign ``%%'' to enter a literal percent sign in the
output.
Formatstrings of properties:
Properties listed in an -x format string can include format control strings. These ``propertyformats''
are used to control how the contents of the associated property are
printed. Property formats are used like C-language printf formats. The
property format is specified by including the attribute ``fmt'' within the
property tag.
Format strings cannot be used with the ``%'' shortcuts described above.
General syntax:
-x '<propertyname fmt="propfmtstr">'
where subfmt controls the output format of propertyname.
Please see the manual pages for strftime(3) and
sprintf(3) for an explanation of format strings. Note: some
versions of strftime do not offer the %s format string (number
of seconds since the Epoch), so swish provides a special format string
``%ld'' to display the number of seconds since the Epoch.
The first character of a property format string defines the delimiter for
the format string. For example,
Text will be output as-is in format strings (and property format strings).
Special characters can be escaped with a backslash. To get a new line for
each result hit, you have to include the Newline-Character ``\n'' at the
end of ``fmtstr''.
The -H n switch generates extened header output. This is most useful when searching more than one index file at a
time by specifying more than one index file with the -f switch.
-H 2 will generate a set of headers specific to each index file. This gives
access to the settings used to generate each index file.
Even when searching a single index file, -H n will provided additional information about the index file, how it was
indexed, and how swish is interperting the query.
-H 0 : print no header information, output only search result entries.
-H 1 : print standard result header (default).
-H 2 : print additional header information for each searched index file.
-H 3 : enhanced header output (e.g. print stopwords).
-H 9 : print diagnostic information in the header of the results (changed from: C<-v 4>)
The -k switch is used for testing and will cause swish to print out all keywords
in the index beginning with that letter. You may enter -k '*' to generate a list of all words indexed by swish.
The -T option is used to print out information that may be helpful when
debugging swish-e's operation. This option replaced the -D option of previous versions.
Running -T help will print out a list of available *options*
In previous versions of Swish-e indexing would require a very large amount
of memory and the indexing process could be very slow. Merging provided a
way to index in chunks and then combine the indexes together into a single
index.
Indexing is much faster now and uses much less memory, and with the -e switch very little memory is needed to index a large site.
Still, at times it can be useful to merge different index files into one
file for searching. This could be because you want to keep separate site
indexes and a common one for a global search, or you have separate
collections of documents that you wish to search all at one time, but
manage separately.
Merges the indexes specified on the command line -- the last file name
entered is the output file. The output index must not exist (otherwise
merge will not proceed).
Only indexes that were indexed with common settings may be merged. (e.g.
don't mix stemming and non-stemming indexes, or indexes with different
WordCharacter settings, etc.).
Use the -e switch while merging to reduce memory usage.
Merge generates progress messages regardless of the setting of -v.
-c *configuration file*
Specify a configuration file while indexing to add administrative
information to the output index file.