- The HarvestMan Web Crawler

1. project.url    : The starting url of your project; This has to be a valid url. It might or might not start with a protocol specification like 'http://'. If the url does not have a protocol specification, HarvestMan tries to figure out the protocol on its own. The default protocol is 'http://'.

2. project.basedir    : The download directory for all your projects. This directory has to be writeable by the user running HarvestMan. If this directory does not exit, HarvestMan will attempt to create it. All files are downloaded to project directories under this directory (explained below).

3. project.name    : HarvestMan identifies your project by this name. All project files for a project goes to a directory with this name inside the main download directory (see above). The project & cache files created for your project also share this name.

4. project.verbosity    : This variable determines the degree and amount of console messages printed by the program while it is running. The default value of 2 prints general project information and basic information about each url, while suppressing errors. Each level adds a certain degree of verbosity over its previous level, as explained below.

   0 -> No console messages are printed
   1 -> Just a starting message about the project & HarvestMan is printed
   2 -> Above plus, messages for each url and its write status are printed
   3 -> Above plus, download filter messages, parser messages and critical 
        error messages are printed.
   4 -> This is the first debug level. It prints more download filter 
        messages and most of the error messages.
   5 -> This is final debug level. This provides maximum verbosity including 
        developer tracebacks.
   

   For most projects, you should use a value of either 2 or 3.

1. network.proxyserver    : If you are connecting to the Internet via a proxy, specify it's name or ip address here.

2. network.proxyport    : The port to which the proxy server listens (if any).

3. network.proxyuser    : If the proxy requires authentication, specify the username here.

4. network.proxypasswd    : The password of the user accessing the authenticated proxy.

5. network.urlserver    : From 1.4 version, HarvestMan allows the user to run a 'url server', which collects and serves new urls for HarvestMan. If this variable is enabled (set to 1), HarvestMan will run a url server (based on the asyncore module ) on port number 3081 (default), and use this for storing & retrieving new urls, instead of the standard Queue. This can sometimes offer a higher performance and speed, since the urlserver operates by using the 'select' multiplex call on your OS.

6. network.urlhost    : The host on which the url server runs. Right now, this is a dummy option, since the url server is always run on the localhost in its own thread.

7. network.urlport    : The port number on which the url server listens for connections. This can be modified. The default is 3081.

Note on proxy settings

For creating the proxy settings, you have to use the utility named 'proxyobfuscator.py' in the 'genutils' subdirectory of HarvestMan installation. HarvestMan obfuscates your proxy entries namely server, username and password (if any) during configuration file creation, so that these sensitive information is not visible to others. Hence the program assumes that the entries in the config file are obfuscated; if they are not, the program will fail to read them.

The 'genconfig.py' script automatically obfuscates the proxy entries for you so there is no need to use 'proxyobfuscator.py' if you have created your config file using 'genconfig.py'.

If you have not given any network proxy options, HarvestMan assumes a direct connection to the Internet.

1. download.fetchlevel    : This option sets the fetch level for HarvestMan, which allows you to control how the program behaves when it encounters links belonging to various servers and directories with respect to the starting url.

   The default value is 0, which permits downloads only below the directory (on the server) of your starting url. For example, if your starting url is

   http://www.foo.com/bar/index.html

   only the links belonging to the directory on the server

   http://www.foo.com/bar

   and its sub-directories will be downloaded.
   
   A value of 1 limits the downloads to the starting server, but does not limit it to the starting url's directory. In the above case, it will download all links from the starting url in the server

   http://www.foo.com .

   
   The values of 0 and 1 do not do any crawling of external servers. The values of 2 & 3 control crawling of external domains.
   
   A value of 2 lets the program crawl all urls in the starting server, and also any first level links in external servers. A first level link is any url linked directly from any of the pages in the starting server that the crawler encounters. It does not let the program to crawl second and higher level links, i.e urls linked further away, on external servers.
   
   A value of 3 behaves like a combination of levels 0 and 2 minus level 1. Thus it will let the program crawl any first level external links and all links below the starting url's directory, but will not allow to crawl other links on the starting server.
   
   A value of 4 gives the program a literal free-ride. This lets the program to go wherever it feels like on the Internet. Be careful to use this level, since you will not be able to restrict the crawling in a meaningful manner. If used, it should be combined with other download control options like depth limits, server/url filters etc.

   For most projects, a fetch level of 0 or 1 would suffice. If you want the program to crawl external servers, choose 2 or 3, while remembering that 2 has less restrictions on the starting server when compared to 3.

2. download.html    : This controls download of html files. If you don't want to download & save html files, set this to 0. This does not mean that the program won't work. It will still download & parse web pages for new links, but the difference is that those links (web pages) won't be saved to the disk.

3. download.images    : This controls download of image files. If you don't want to download images, set this to zero.

4. download.cookies    : Controls processing of cookies on websites. If set to 1 (default), this will save the cookies in a file named 'cookies.dat' in your project directory. This option is not of much consequence in most projects.
5. download.javascript    : Download of server-side javascript files is controlled by this option. If enabled (default), it will save server-side javascript files (they have a ".js" extension) embeddded in web pages to the disk.
6. download.javaapplet    : Control download of server-side java applet class files using this option. If enabled (default), it will download java class files to the disk. The parser knows to parse the

   '<applet>'

   tags and the

   'codebase'

   param intelligently to figure out the location of the java class files.
7. download.checkfiles    : If enabled (default), this will ask the program to perform an md5 checksum of the downloaded files. After the file is saved to the disk, it is read again and a checksum of its contents is compared to the checksum of the original downloaded data. If they don't match, the program signals an error.
8. download.rename    : This option tells the program to try to rename content generated by server-side queries originating from php, jsp and similar cgi calls. Websites can generate different kinds of content by using server-side scripts. HarvestMan assumes that these generate web-pages, by default. This option will trigger an algorithm which will try to find out the actual type (mime-type) by reading the signature of the file and renames it with the appropriate extension. Currently this works well only for BMP, JPEG and GIF image files, hence it is disabled by default.
9. download.linkedimages    : This option, if set to 1 (default), will force the program to download image files linked from web pages by overriding fetch-level and other external server-related rules. If set to 0, download of images will be controlled by the download rules. It is a good idea to enable it.
10. download.linkedstylesheets    : This works similar to above option, the difference being that it controls the download of cascading style sheet files of webpages. If set, css files are always downloaded regardless of download rules. It is a good idea to enable this also.

Url Control Variables

1. control.extpagelinks    : This can be used to configure the parsing & fetching of urls external to the starting server's directory. This is set to 1 (enabled) by default. Since fetch levels offer much finer control and have precedence over this option, it is advised to skip this option and control your fetchlevel instead.

2. control.maxextdirs    : A limit on the maximum number of external directories (paths outside the directory of the starting url) on your project. By default, there is no limit.

3. control.skipqueryforms    : An option that tells the program to skip query forms of the format

   http://www.foo.com/cgi-bin/query?item=val .

   This is enabled by default.

4. control.urlpriority    : A priority algorithm by which you can specify priorities for urls based on file extensions. A priority specification takes the form of "extension+/-priority".
   
   For example, a url priority string of "png+5" gives maximum priority for PNG image files. Another one of "jpg-5" gives the least priority for JPEG image files. Urls with higher priority are downloaded first and those with lower priority, later. The priority algorithm works only if the default Queue algorithm is enabled. If the url server option is enabled, then priorities won't work.
   
   Priorities can be chained together using commas to create a priority string.
   Example: control.urlpriority    png+5,html-2,jpg+4,pdf-3
   
   Priorities are in the range of (-5,+5). +5 gives maximum priority while -5 gives least priority.

5. control.urlfilter    : This allows to create a regular expression like url filter for filtering specific urls. The match can be based on file extensions, directory components and wildcards. For more information see FAQ.

6. control.junkfilter    : This is a new feature which will be available in 1.4 final release. It will add advertisement & banner filtering to HarvestMan. The design of this is similar to Internet Junkbuster.

Server control variables

1. control.extserverlinks    : This tells the program to crawl external server links. Again a better option is to set fetch levels, since fetch level setting overrides this option.

2. control.maxextservers    : This allows to set a limit to the number of external servers crawled. By default, there is no limit.

3. control.serverfiler    : This is an option similar to control.urlfilter, only that it allows you to filter external servers using a regular expression like syntax. Fore more information, see FAQ.

4. control.serverpriority    : This is similar to control.urlpriority, only that it sets priorities for servers. This makes sense only when crawling multiple servers (fetchlevels 2, 3 & 4). The priority syntax is same as that for url priorities.

5. control.subdomain    : This option allows to differentiate between servers in the same base domain. By default, it is set to 0 (False), which makes the program to treat servers belonging to the same domain (shopping.yahoo.com & mail.yahoo.com for example), to be the same server. If set to 1 (True), it will treat the servers as different. This makes a difference when creating server priorities, filters and in controlling requests to the same server.

Connection control

1. control.connections    : This is a setting which allows you to control the number of total open network connections (sockets) to servers at a given time. This is set to 5 by default. If you have a high network bandwidth, feel free to increase it!

2. control.requests    : This setting allows much finer control than 'control.connections'. It controls the number of simultaneous requests to the same server. By default it is set to 5. Again, you should set a higher value to allow speedier downloads, if you got a higher bandwidth!

Depth control

1. control.depth    : This allows to control the depth of a url relative to the starting url. The depth is measured in terms of the number of path components separating a url from the starting url. For example the depth of the url

   http://www.foo.com/images/holidays/hawaii/me.jpg

   is 2, if its starting url is

   http://www.foo.com/images/index.html

   . By default this is set to 10. If a url's depth is more than this limit, it is skipped.

2. control.extdepth    : This controls the depth of urls belonging to external servers. It works the same way as 'control.depth' with the difference that the depth is measured from the root of the server. For example, the depth of the url

   http://www.foo.com/images/holidays/hawaii/me.jpg

   is 3, if it is an external server url.
   
   By default, this setting is 0, which means that there is no limit to the depth.

Parser control

1. control.tidyhtml    : This is the only parser control variable. If set (default), it will try to invoke the Tidy html cleaner library to clean up downloaded html, before parsing it and extracting links. The tidy library is supplied with HarvestMan. It is also available as a separate download in the download page.

File control

1. control.maxfiles    : This option can be used to control the download by setting a limit on the maximum number of files downloaded. By default this is 3000. If the number of downloaded files exceeds this limit, a controller thread automatically brings the program to end.

2. control.pagecache    : Set this to 1 to enable caching of downloaded files. If caching is enabled, HarvestMan need not re-download files from a server when the project is re-run, if the cache is upto date. Caching is done by checking the timestamp of the file at the server and the local timestamp which is stored in a specially designed cache file. This option is enabled by default.

3. control.datacache    : This option allows finer control of caching. If set to 1 (the default), the program will save the data of each file downloaded and write it to the cache file of the project.
   
   This allows the program to regenerate the files from the cache if the downloaded files of a project got deleted with the cache file intact. In such cases, if the cache is upto date, HarvestMan will generate the files from the cache.
   
   The downside of this is that the cache file becomes huge for projects which download a lot of files. Hence if you are downloading a huge number of files (say > 5000) in your project, it is sometimes better to disable data caching.

Time-based control

1. control.projtimeout    : This option provides a safe exit sentinel for HarvestMan, if the threads don't terminate due to some reason, thereby hanging a project. The main program thread monitors the download threads. If the program hangs (no write/read from queues) for a time period more than this limit, the main thread signals the other threads to terminate and forces the program to a halt. By default, this is set to five minutes.

2. control.timelimit    : This provides a way of running a HarvestMan project for a stipulated time period. The time limit is specified in seconds. A controller thread keeps track of the program's running time. If the time limit expires, this thread signals other threads to stop and brings the program to a halt. By default, there is no time limit for a project.

Robots Exclusion Protocol based control

HarvestMan obeys the Robots Exclusion Protocol used by certain web-sites to block certain sections of the web server or certain files to crawlers like HarvestMan. A single configuration option lets you to configure HarvestMan to obey or disobey REP.

1. control.robots    : If enabled, HarvestMan will parse the 'robots.txt' file on each web server and obey the rules specified in it. These rules are applied to each url to find out if the 'robots.txt' file allows fetching of that url. If fetching is disallowed, HarvestMan outputs a console message (for verbosity >= 3) and skips that url.
   
   If you want to disble 'robots.txt' file parsing, set this variable to 0. Neglecting of REP is not very good Internet etiquette. However, HarvestMan gives you the freedom to make that choice.