3.13. Storing compressed document copies

In DataparkSearch it is possible to store compressed copies of indexed documents. Copies are stored and retrieved by the new daemon - stored, that is installed into sbin directory of DataparkSearch installation (default: /usr/local/dpsearch/sbin).

To enable documents copies archiving without stored usage, place DoStore yes command in your indexer.conf file instead of stored daemon configuration.

Stored document copies are retrieved by means of storedoc.cgi CGI script. It requests a saved copy of a documents from stored, then a copy is displayed with user's web browser with search keywords highlighted.

To enable stored support, compile DataparkSearch with zlib support:

./configure --with-zlib <other arguments>

You may use the Store and NoStore commands to allow or disallow storing several files by pattern. For arguments of those commands are exactly the same as for the Allow command (see Section 3.10.14>). All documents are stores by defaults, if support for stored is enabled.

3.13.1. Configure stored

To start using stored, please do the following:

Copy /usr/local/dpsearch/etc/stored.conf-dist to /usr/local/dpsearch/etc/stored.conf.
Edit /usr/local/dpsearch/etc/stored.conf
There are several commands available for using with stored:
- Listen tells stored which address and/or port to bind to. By default stored listens to port 7004 and any address. It is possible to specify port only:
  Listen 7004
  Or address only:
  Listen 127.0.0.2
  Or both address and port:
  Listen 127.0.0.2:7004
- VarDir command specifies an alternate var/ working directory, e.g.
  VarDir /mnt/d/dpsearch/var/
- StoredFiles command specifies number of stored data files created in var/stored/ directory, e.g.
  StoredFiles 256
- OptimizeInterval command specifies interval in seconds between attempts to optimize a stored datafile, e.g.
  OptimizeInterval 300
- OptimizeRatio command specifies defragmentation threshold for a stored datafile optimization.
  OptimizeRatio 3
Run stored:
/usr/local/dpsearch/sbin/stored &
Configure indexer.conf and search.htm (or searchd.conf, if searchd is used). Specify address and port that indexer will use to connect to stored. Use stored parameter for DBAddr command, e.g.:
DBAddr mysql://localhost/search/?dbmode=cache&stored=localhost:7004

3.13.2. How stored works

After you have successfully configured stored, the indexer pass downloaded documents to stored daemon. After that, stored compress the received documents and save them.

3.13.3. Using stored during search

To enable displaying stored documents during search, do the following:

Configure storedoc.htm (storedoc.cgi template) if necessary.
Add a storedoc.cgi link to search.htm  section: e.g. <A HREF="$(stored_href)">Cached copy</A>
Specify storedoc.cgi CGI script URL in search.htm (by default $(stored_href) will return /cgi-bin/storedoc.cgi). In case you have to specify other URL, add to search.htm variables section the following line:
StoredocURL /path/to/storedoc.cgi
Or an absolute path:
StoredocURL http://servername/path/to/storedoc.cgi

This is how stored works during search, if everything configured correctly:

search.htm displays a link to storedoc.cgi;
When user clicks the link, storedoc.cgi sends a query to stored daemon to the address, specified in storedoc.htm with the Stored parameter of DBAddr command;
After the query, stored will decompress the requested saved copy and send it to storedoc.cgi;
storedoc.cgi parses the received document and apply highlighting of search keywords. Highlighting method is specified with storedoc.htm HlBeg and HlEnd commands;

3.13.4. Document excerpts

stored is also used to make documents excerpts for search results.

You can use ExcerptSize command in search.htm template to specify average excerpt size in characters; value by default: 256.

With ExcerptPadding command you can specify average number of characters is taken before and after a search word in excerpts; value by default: 40.

With ExcerptMark command you can alter the marking character sequence which delimits excerpt chunks; value by default: " ... " (a space, a dots, a space).

You may switch off document excerpts (but retain ability to show stored copies) with DoExcerpt no command in your search template.