HOWTO install Enkive 1.4 on Ubuntu 12.04 LTS

This HOWTO documents step-by-step how to install and configure Enkive 1.4 on Ubuntu 12.04 LTS (Precise Pangolin).

Before continuing, you should read the official, and generic, installation instructions for Enkive 1.4 at [http://wiki.enkive.org/index.php/1.4_Installation_Instructions http://wiki.enkive.org/index.php/1.4_Installation_Instructions].


'''Assumptions:''' This will be a standalone server that only runs the Enkive email archiver. Any email sent to this machine's postfix (port 25) gets archived. In a production environment you will have to configure your existing mail server(s) to send a copy of each email that comes in to and goes out of your organization to this Enkive server. ''Configuring your existing mail server(s) to send duplicate emails to Enkive is not covered in this HOWTO''.



==== Prerequisites ====


The files used to install Enkive 1.4. These were the latest available versions at the time of this writing (Feb. 2014):
* enkive-1.4.tar.gz
* jdk-7u51-linux-x64.tar.gz
* mongodb-linux-x86_64-2.4.9.tgz
* indri-5.6.tar.gz
* commons-daemon-1.0.15-native-src.tar.gz (do not use an earlier version than 1.0.12 with Enkive 1.4)
The download links are available at [http://wiki.enkive.org/index.php/1.4_Installation_Instructions http://wiki.enkive.org/index.php/1.4_Installation_Instructions].

Linux operating system:
* Ubuntu 12.04 LTS Server default install.


Make sure you have at least 10GB (and preferably a lot more) of free space on your machine. MongoDB won't start properly if there's not enough free space.

<br>

==== Preliminary steps ====


===== Postfix =====

Ubuntu 12.04 does not have any mail transfer agent installed. So, install the Postfix MTA. Enkive only works with postfix at this time.
 sudo apt-get install postfix
After you type this, you will get a pop-up asking what for the kind of mail configuration. Select "Internet Site". Next it will ask for the "system mail name", which you can leave as is -- whatever you already selected as your server's hostname.

Postfix will now be running, and listening on your network interface (not just localhost).

You will also need to modify the "mynetworks" line in /etc/postfix/main.cf. By default, the Ubuntu postfix will only accept mail from itself, not even the local network. So find this line:
 mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128

and add your local subnet. For example, if your local network is in 192.168.1.0/24, change the line to read: 
 mynetworks = 127.0.0.0/8 192.168.1.0/24 [::ffff:127.0.0.0]/104 [::1]/128

The above assumes that the production mail server that will be sending copies of email to Enkive is on the same local network. It may be on a different subnet; for example, your mail server might have an IP address of 172.16.1.5 and the Enkive server an IP address of 10.1.1.20. Then the "mynetworks" line in main.cf might look like:
 mynetworks = 127.0.0.0/8 172.16.1.0/16 10.1.1.0/24 [::ffff:127.0.0.0]/104 [::1]/128
which would accept mail from anyone on the 172.16.1.0 and 10.1.1.0 subnets.

When Enkive is finally up and running you can change the "mynetworks" line to restrict mail, allowing it from just a certain IP, for example.

After you have saved /etc/postfix/main.cf, reload postfix:
 sudo /etc/init.d/postfix reload

===== Development environment =====

Some of the Enkive-related packages need to be compiled, so get a development environment installed. At minimum, the packages you'll need are:
* gcc
* g++
* make
* libcap-dev
 sudo apt-get install gcc g++ make libcap-dev

Alternatively, installing the "build-essential" package will also get you most packages needed for compiling:
 sudo apt-get install build-essential libcap-dev

<br>

==== Java ====

Unpack the tarball in /opt:
 sudo tar zxvf jdk-7u51-linux-x64.tar.gz -C /opt/

Make a symbolic link:
 cd /opt/
 sudo ln -s jdk1.7.0_51/ java

So, per the official Enkive instructions, $JAVA_HOME is: /opt/jdk1.7.0_51 or /opt/java. We'll use /opt/java as the $JAVA_HOME from now on.

You may want to put /opt/java/bin in your path. There are various ways to do this. You can modify your .bashrc file, or even the /etc/environment file (which applies system-wide). Remember to log out and log back in for the changes to take effect.

<br>

==== Jsvc ====

<font color="red">You must use Jsvc version 1.0.12 (or later) with Enkive 1.4. (Version 1.0.15 is recommended.) Prior versions of Jsvc (1.0.11 and earlier) are not compatible with Enkive 1.4. Do not use the Jsvc from the Ubuntu 12.04 repository.</font>

You can download previous versions of Jsvc at [http://archive.apache.org/dist/commons/daemon/source/ http://archive.apache.org/dist/commons/daemon/source/].

For Jsvc, we will compile with the java option and then copy the resulting binary to another directory so that it's in our path. Note that $JAVA_HOME is /opt/java:
 tar zxvf commons-daemon-1.0.15-native-src.tar.gz
 cd commons-daemon-1.0.15-native-src/
 cd unix/
 ./configure --with-java=/opt/java
 make
 sudo cp jsvc /usr/local/bin/
Now "jsvc" is in your path. This is also where the enkive initscript expects to find the jsvc binary.


<br>

==== Swig ====

For swig:
 sudo apt-get install swig zlib1g zlib1g-dev
 

<br>

==== Indri ====

For indri, we're going to install it in /opt/indri, as the official documentation suggests.
 tar zxvf indri-5.6.tar.gz
 cd indri-5.6/
 ./configure --prefix=/opt/indri-5.6 --enable-java --with-javahome=/opt/java
 make
 sudo make install

Create a symbolic link for /opt/indri:
 cd /opt/
 ln -s indri-5.6/ indri

<br>

==== MongoDB ====

There are several different ways to install MongoDB.

The easiest may be to use the custom MongoDB repository that the MongoDB maintainers offer. See the following link for how to install and configure:

[http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/ http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/]

(The MongoDB that is part of the Ubuntu 12.04 repository is slightly older, version 2.0.4. This will work, but we recommend using a 2.4.x MongoDB.)

If you just use the tarball provided on the downloads page -- mongodb-linux-x86_64-2.4.9.tgz -- keep in mind that it does not come with an initscript or upstart script.

An upstart script can be found in the Enkive tarball in the scripts/init/ubuntu/ subdirectory. Or you can use the SysV initscript below.

If you're using the mongodb-linux-x86_64-2.4.9.tgz tarball, unpack it in /opt. Create a symbolic link for it. Create a directory within it (ie, data/db) for the database files.
 sudo tar zxvf mongodb-linux-x86_64-2.4.9.tgz -C /opt/
 cd /opt/
 sudo ln -s mongodb-linux-x86_64-2.4.9/ mongodb
 cd mongodb/
 sudo mkdir -p data/db


Create the initscript for mongodb, and put the following in /etc/init.d/mongodb:
<pre>
#! /bin/sh
### BEGIN INIT INFO
# Provides:          mongodb
# Required-Start:    $remote_fs $syslog $network
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: MongoDB initscript
# Description:       This file should be used to construct scripts to be
#                    placed in /etc/init.d.
### END INIT INFO

# Author: Foo Bar <foobar@baz.org>
#
# Please remove the "Author" lines above and replace them
# with your own name if you copy and modify this script.

# Do NOT "set -e"

# PATH should only include /usr/* if it runs after the mountnfs.sh script
#PATH=/sbin:/usr/sbin:/bin:/usr/bin
PATH=/opt/mongodb/bin:/sbin:/usr/sbin:/bin:/usr/bin
DESC="MongoDB"
NAME=mongod
DAEMON=/opt/mongodb/bin/$NAME
#DAEMON_ARGS="--options args"
DAEMON_ARGS="--fork --dbpath /opt/mongodb/data/db --logpath /var/log/mongodb.log --logappend"
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/mongodb

# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0

# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions

#
# Function that starts the daemon/service
#
do_start()
{
	# Return
	#   0 if daemon has been started
	#   1 if daemon was already running
	#   2 if daemon could not be started
	start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null \
		|| return 1
	start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON -- \
		$DAEMON_ARGS 2>&1 >>/var/log/mongodb.log \
		|| return 2
	# Add code here, if necessary, that waits for the process to be ready
	# to handle requests from services started subsequently which depend
	# on this one.  As a last resort, sleep for some time.
}

#
# Function that stops the daemon/service
#
do_stop()
{
	# Return
	#   0 if daemon has been stopped
	#   1 if daemon was already stopped
	#   2 if daemon could not be stopped
	#   other if a failure occurred
	start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME
	RETVAL="$?"
	[ "$RETVAL" = 2 ] && return 2
	# Wait for children to finish too if this is a daemon that forks
	# and if the daemon is only ever run from this initscript.
	# If the above conditions are not satisfied then add some other code
	# that waits for the process to drop all resources that could be
	# needed by services started subsequently.  A last resort is to
	# sleep for some time.
	start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
	[ "$?" = 2 ] && return 2
	# Many daemons don't delete their pidfiles when they exit.
	rm -f $PIDFILE
	return "$RETVAL"
}

#
# Function that sends a SIGHUP to the daemon/service
#
do_reload() {
	#
	# If the daemon can reload its configuration without
	# restarting (for example, when it is sent a SIGHUP),
	# then implement that here.
	#
	start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME
	return 0
}

case "$1" in
  start)
	[ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
	do_start
	case "$?" in
		0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
		2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
	esac
	;;
  stop)
	[ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
	do_stop
	case "$?" in
		0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
		2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
	esac
	;;
  status)
       status_of_proc "$DAEMON" "$NAME" && exit 0 || exit $?
       ;;
  #reload|force-reload)
	#
	# If do_reload() is not implemented then leave this commented out
	# and leave 'force-reload' as an alias for 'restart'.
	#
	#log_daemon_msg "Reloading $DESC" "$NAME"
	#do_reload
	#log_end_msg $?
	#;;
  restart|force-reload)
	#
	# If the "reload" option is implemented then remove the
	# 'force-reload' alias
	#
	log_daemon_msg "Restarting $DESC" "$NAME"
	do_stop
	case "$?" in
	  0|1)
		do_start
		case "$?" in
			0) log_end_msg 0 ;;
			1) log_end_msg 1 ;; # Old process is still running
			*) log_end_msg 1 ;; # Failed to start
		esac
		;;
	  *)
	  	# Failed to stop
		log_end_msg 1
		;;
	esac
	;;
  *)
	#echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
	echo "Usage: $SCRIPTNAME {start|stop|status|restart|force-reload}" >&2
	exit 3
	;;
esac
</pre>


Make the mongodb initscript executable:
 sudo chmod 755 /etc/init.d/mongodb

Start mongodb. It may take a minute to start on slower machines:
 sudo /etc/init.d/mongodb start

<br>

==== Enkive (and Postfix) ====

Now unpack the Enkive tarball and put it in /opt. We also create a symlink here.
 sudo tar zxvf enkive-1.4.tar.gz -C /opt/
 cd /opt/
 sudo ln -s enkive-1.4/ enkive


Create the Enkive socket filter:
 cd /opt/enkive/support/filters/postfix/enkive-socket-filter/
 sudo make
This socket filter is now at:
 /opt/enkive/support/filters/postfix/enkive-socket-filter/dist/enkive-socket-filter 
We'll need the path to it for Postfix.

Edit the /etc/postfix/master.cf file as follows. There's already a line at the top for "smtp". You will add a second line below it for the content filter, and then 3-line filter section.

The top of the master.cf file should look something like this:

<pre>
#
# Postfix master process configuration file.  For details on the format
# of the file, see the master(5) manual page (command: "man 5 master").
#
# Do not forget to execute "postfix reload" after editing this file.
#
# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
smtp      inet  n       -       n       -       -       smtpd
   -o content_filter=filter:dummy
filter    unix  -       n       n       -       -      pipe
          flags=Rq user=nobody null_sender=
          argv=/opt/enkive/support/filters/postfix/enkive-socket-filter/dist/enkive-socket-filter localhost 2527 ${sender} ${recipient}
#smtp      inet  n       -       -       -       1       postscreen
#smtpd     pass  -       -       -       -       -       smtpd
#dnsblog   unix  -       -       -       -       0       dnsblog
#tlsproxy  unix  -       -       -       -       0       tlsproxy
...
</pre>

Restart Postfix:
 sudo /etc/init.d/postfix restart


Create an "enkive" user. The -r option will make a system user, rather than a regular user:
 sudo useradd -r enkive


Allow the new "enkive" user to read and write to Enkive (don't forget the trailing slash!):
 sudo chown -R enkive.enkive /opt/enkive/

Prepare the enkive initscript.

If you are using the Upstart script:
 sudo cp /opt/enkive/scripts/init/ubuntu/enkive.conf /etc/init/
then edit the environment variables at the top of the script.

Otherwise, you can use the SysV initscript:
 sudo cp /opt/enkive/scripts/init/debian/enkive /etc/init.d/
 sudo chmod 755 /etc/init.d/enkive
At the top of the enkive initscript you need to edit several environment variables.


Now start enkive. The Upstart way:
 sudo service enkive start
The SysV way:
 sudo /etc/init.d/enkive start

Give it a few seconds. Then log in to a graphical desktop and use the web browser to go to http://localhost:8888/ediscovery/. Even better: from another computer on your network go to http://enkiveIPaddress:8888/ediscovery/ in a web browser.

Log in as user "enkive" with the password "enkive". There won't be any archived email yet to search. Log out.

<br>

==== Add email to Enkive ====

Sent a test email to the Enkive using the telnet client. If you've never done this before, you might want to read the following:
* [http://workaround.org/ispmail/lenny/test-mail-through-telnet http://workaround.org/ispmail/lenny/test-mail-through-telnet]
* [http://www.expta.com/2010/03/how-to-use-telnet-to-send-smtp-email-to.html http://www.expta.com/2010/03/how-to-use-telnet-to-send-smtp-email-to.html] (Windows-centric, but useful)

Follow the session below. You can only escape from the telnet session with CTRL-] (not CTRL-C!).

The actual lines you type after telnetting to port 25 are:
 ehlo localhost
 mail from:<bill@example.com>
 rcpt to:<ted@example.org>
 data
 From: Bill <bill@example.com>
 To: Ted <ted@example.org>
 Subject: my first archived email
 
 Archive me, please!
 
 .


Note that you finally "send" the email by typing a single period by itself on a line, then hit enter. After which, postfix will queue the message.

<pre>
user@localhost:~$ telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 localhost ESMTP Postfix (Ubuntu)
ehlo localhost
250-localhost
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
mail from:<bill@example.com>
250 2.1.0 Ok
rcpt to:<ted@example.org>
250 2.1.5 Ok
data
354 End data with <CR><LF>.<CR><LF>
From: Bill <bill@example.com>
To: Ted <ted@example.org>
Subject: my first archived email

Archive me, please!

.

250 2.0.0 Ok: queued as D002D28D02
^]    
telnet> quit
</pre>

Now go back to http://localhost:8888/discovery and log in. You will be able to search for the email.

<br>

==== Final clean up and reboot ====

The Upstart scripts provided in the Enkive tarball are configured so that Enkive starts when it detects that MongoDB is running.

If you ues the old style, these update-rc.d commands make sure that MongoDB starts first, and that Enkive starts well after it. See the note below re: problems with Enkive
 sudo update-rc.d mongodb start 60 2 3 4 5 . stop 40 0 1 6 .
 sudo update-rc.d enkive start 90 2 3 4 5 . stop 10 0 1 6 .


Reboot the machine for good measure.

Note: MongoDB should start before Enkive. If you're having issues with Enkive not starting on boot, it could be that MongoDB has not finished starting when Enkive starts. If that's the case, you need to modify the sequence numbers in the mongodb and enkive update-rc.d commands. The current settings used here should work, however.


<br>

==== You're mostly done ====

At this point, if you want to start archiving email you will have configure your mail server (and there may be more than one in your organization) to send a duplicate of each email to this Enkive server.

Postfix (including Zimbra), sendmail and exim have different ways of doing this; however, this is not covered here yet.