modules/og2list/exim.INSTALL

$Id: $

Pre-Requisites for Installing OG2list with Exim

1. Perl and Perl Modules
    - FindBin
    - Mail::Verp
    - MIME::Parser
    - MIME::Words
    - DBI
    - Getopt::Std
    - Encode

2. Exim as your MTA.

3. Configuration of your MTA so that the OG2list router and transport only handle the mail they are supposed to. Otherwise, the OG2list router and transport will take in and send out all mail sent to the server. Which will effectively render email that wasnt meant for OG2list useless.
 
4. A dedicated domain or sub-domain (localhost will do)

5. CVS and SVN are recommended

6. Shell Access to your server with sudo/root privileges

7. Drupal 4.7

8. If your on OS X. You will need to install Wget for the drupal cronjobs. You will also need to install lynx for CPAN. You can use Fink (http://fink.sourceforge.net/) to do this

This was written as a guide to installing the OG2list module on OS X using the Exim MTA.
The instructions should be relevant for any *nix based OS though.

Step 1 : Install Perl Pre requisites

A. Install CPAN

To get CPAN do this:
OS X : Install Xcode

After you have the Xcode package you will have access to perl and its package management system, CPAN.
CPAN allows you to install extra packages and modules for Perl to use. OG2list relies on 3 more than come default.

To access CPAN, as root, type:

        'cpan'

you should see

        cpan shell -- CPAN exploration and modules installation (v1.7601)
        ReadLine support available (try 'install Bundle::CPAN')

        cpan>

you can perform the installs by typing each of the install lines below, each install will have its own set of prompts and questions - agree to everything:

B. Install Perl Modules
From within CPAN, do...

        install DBI
        install DBD::mysql
        install Mail::Verp
        install FindBin
        install Getopt::Std
        install MIME::Tools
        install MIME::Words
        install Encode

Integration into MTA setup (Exim 4.x)
    Global
        The global section of Exim's configuration file is the place to
        define variables and installation-specific parameters such as the
        SQL location below.

        See Section 9.21 of the Exim specification for details.

        The user/group ID and the home directory need not match any other
        user account on the system, in particular not the webserver's or the
        web application's user. mail-in.pl only needs to access its
        configuration file and the og2list_* tables in the SQL database for
        which the credentials are stored in the configuration file.

          OG2LIST_HOME = /home/og2list
          OG2LIST_UID  = og2list
          OG2LIST_GID  = og2list

          hide mysql_servers = host/dbname/dbuser/dbpass

    Router
        Routers are rules that determine how a destination address should be
        handled. The following piece of code assigns addresses to a
        transport rule called og2list_transport which is defined below.

        Note that the since Routers are evaluated one after anotehr, their
        order in the configuration file will most likely affect overall
        behavior.

        See section 3.8 of the Exim specification for details.

          og2list:
            debug_print = "R: og2list from $return_path \
              for $local_part@$domain"
            driver = accept
            transport = og2list_transport
            condition = ${lookup mysql{SELECT 1 FROM og2list_groups \
                                       WHERE recipient \
                                       LIKE '$local_part'} \
                                 {$value} \
                                 fail}

        If Exim has been built without MySQL support, the following
        condition statement should work:

            condition = ${run{/path/to/valid-recipient.pl $local_part}{1}{0}}

        It is possible to use the "domains" statement to restrict the
        domains for which the og2list is actually tried (see Exim
        specification, chapter 15 for details).

        "abuse"(RfC2142), "postmaster"(RfC2821) and other essential
        local parts should be handled by a router which precedes the
        og2list router. The system_aliases router that is part of
        Exim's default configuration file is a good place for this.

    Transport
        Transports are the rules that determine by what means a message
        should be delivered that has been assigned to this transport by a
        router.

          og2list_transport:
            debug_print "T: og2list_transport from \
              $return_path for $local_part@$domain"
            driver = pipe
            freeze_exec_fail
            log_output
            command = OG2LIST_HOME/mail-in.pl \
              -f $return_path -t $local_part@$domain
            current_directory = OG2LIST_HOME
            user = OG2LIST_UID
            group = OG2LIST_GID
            
Debugging - or what to do if something does not work
----------------------------------------------------


A) No mail is recieved by the group

If you have sent mail to your groups address and it is not appearing inside your groups forum. Try these debugging steps.

0) Make sure Exim is running and is the MTA for the domain you are trying to send to.

TODO: Add link to run this script http://cvs.drupal.org/viewcvs/drupal/contributions/modules/devel/generate/generate-og-users.php

1) Enable your mysql query log and check to make sure the messages are getting inserted into the og2list tables.

The mysql configuration files, called my.cnf, are located in different places depending on your setup. Most popular place is /etc/my.cnf
If you absolutely cant find your my.cnf file. Find all instances by doing
    
  sudo find / -iname 'my.cnf'

mysql 4.1 - add
        log=/path/to/logfile
under [mysqld] in your my.cnf and then restart mysql

mysql 5.1 - add
       log=/path/to/logfile
under [mysqld] in your my.cnf and then restart mysql with
       sudo /path/to/mysql/bin/mysqld_safe --user=mysql --log --log-output=FILE &

In order to recreate the situation
a) send mail to a group address
        
        echo "some sort of text" >> ~/test-mail.txt

There are testmails located under /path/to/civicspace/modules/og2list/testmails
Youll need to edit them and change the from address to an address of a member in your group with posting permissions.
You can use them by executing this command

mail -s "This is the subject" group_addy@www.example.com < /path/to/civicspace/modules/og2list/testmails/killes/tb-arabic

To check to see what your groups address, or if your groups address is being properly created. Enter your mysql DB and

        mysql> select * from og2list_groups;
        +-----+-----------+--------+
        | nid | recipient | status |
        +-----+-----------+--------+
        |   2 | test      |      1 |
        +-----+-----------+--------+

        mail -s "subject line" test@www.example.com < ~/test-mail.txt

In this case the address for that group would be test@www.example.com
So double check and make sure the group your sending mail to actually has an address.

If your group does not have an address, make sure you added the address upon initial creation of a group. Group mailing addresses cannot be connected to a group after the group has been created.
        
b) check your Exim log to make sure it got there and that the relay is equal to "drupal" - to confirm it was handed to the correct transport. Exim logs are generally located at '/var/log/mail.log'

If the message was succesfully recieved and relayed you should have an entry that looks somewhat like. The 3 parts you should look at.

        to= this should be the email address of your group
        relay = this has to say "drupal", otherwise, your email is being spat out into the ether, never to return.
        status = this should say sent. if it says deferred or bounced. Make sure the address of your group is correct. see #6, if the address is correct. Check your Exim configuration.

        Jun  8 18:32:16 punlc exim/pipe[13474]: 6C89A9A1035: to=<mailin@www.example.com>, relay=drupal, delay=16, status=sent (john.localdomain)
        Jun  8 18:32:16 punlc exim/qmgr[8473]: 6C89A9A1035: removed

c) run cron - log in as an administrator on your site and then surf to: http://www.example.com/cron.php

d) check your mysql query log for INSERTS into og2list_incoming_content and og2list_incoming_groups

        mysql> select * from og2list_incoming_content;
        mysql> select * from og2list_incoming_groups;

2) Make sure you created an organic group with a mailing list.
        mysql> select * from og2list_groups;

3) Make sure you are an approved member of the group you are trying to send to. That means the mail address you are sending from has to match the one used for your Drupal account.

4) Check your Exim log (mail.log) for anomalities.

use this command to continually scroll additions to the log onto the screen.

        tail -f /var/log/mail.log

Once you have the file tailed. Send a message or two into one of your groups address, or Post a few topics in your groups forums.
In a few seconds you should see the messages get processed by Exim and be spit out into the log.
Look at the specific values. Make sure to look at status,relay, and the to and from addresses. See (b) above to see specifics on analyzing Exim log entries.

5) Try to debug from the commandline according to the documentation in
mail-in.pl (-d switch):

TODO: give an exact example with sender and recipient values

        ./mail-in.pl -d -f sender -t recipient < drupal/modules/og2list/testmails/killes/tb-arabic

sender is an email address of a member of the group which has the
local email 'recipient'. Test mails can be found in the testmails
directory.

Make sure to check your Exim log afterwards.

6) Check to make sure your group has an address. You can do this by using this query in mysql.

        select * from og2list_groups.

You should see something like

        mysql> select * from og2list_groups;
        +-----+-----------+--------+
        | nid | recipient | status |
        +-----+-----------+--------+
        | 1   | mailin    | 1      |
        +-----+-----------+--------+
        1 row in set (0.01 sec)

In this case the address for that group would be mailin@www.example.com
So double check and make sure the group your sending mail to actually has an address.

the group you are sending mail to should have its address in the recipient column.

If your group does not have an address, make sure you added the address upon initial creation of a group. Group mailing addresses cannot be connected to a group after the group has been created.

B) No mail is sent to subscribers

0) Check that your Drupal cronjob is running. Try accessing your installation's cron.php file from your browser.

a) log in as admin
b) goto www.example.com/cron.php in order to manually run your cronjob
c) make sure there is an entry in your cronjob that looks something like this:

        */5 * * * * /sw/bin/wget -O - -q http://www.example.com/cron.php

This cronjob will run every 5 minutes, as denoted by the */5. If yours is not running frequently. This could be the cause of why you havent seen your messages appear in your group forums in a timely manner. On the other hand, you should be wary of setting your cronjob to run any more frequently than every 2 minutes. As two cronjobs running at the same time could cause duplicate messages to be sent out or imported.

1) Make sure that the group's mailing list is not disabled.

a)Log in as your groups administrator.
b)Under the "My Groups" block, select your group.
c)Goto the "Mailing List" Tab on your groups homepage.
d)Make sure it is set to "Enabled".

3) Do the settings in mail.conf match your database and Exim
configuration?

Your $mail_domain is simply the domain of the site where you are running og2list. if your sites address is "www.example.com" your $mail_domain should look something like:
        # The domain that is appended to the ogroup-specific localparts.
        $mail_domain = "example.com";

Your $dbtype will not change. it should be "mysql".
        # Connection parameters to the Drupal database
        # type of database connection
        $dbtype = "mysql";

These are your DB settings. See below for how to test them.
        # name of your database server
        $dbase  = "og2list";
        # name of your Drupal database
        $dbhost = "localhost";
        # name of your database user
        $dbuser = "og2listuser";
        # the user's password for your database server
        $dbauth = "og2listpass";

To test your db connection fill in the the parameter name with its value from your mail.conf. If you successfully make a connection its correct, otherwise. its wrong. pay careful attention to the -p. theres no space between the -p and the value for a reason.
        mysql -u og2listuser -h localhost -pog2listpass og2list

        # Path to sendmail binary
        $sendmail_binary = "/usr/sbin/sendmail";

to find sendmail do
        whereis sendmail
and you should get a result back saying the location of sendmail.
        /usr/sbin/sendmail

4) Try to run mail-out.pl from the commandline. An error message should be printed. Show it to your system administrator.
        
Switch to your og2list user and run mail-out.pl
        sudo su og2list
        ./mail-out.pl
        
5) Check your Exim log (mail.log) for anomalities.

C) I sent mail to my group but cant see it!

1) Check under admin >> content to make sure your posts are coming in 'unpublished'

2) See above for checking your mail.log to make sure that the mail is being recieved and processed correctly.


1	$Id: $
2
3	Pre-Requisites for Installing OG2list with Exim
4
5	1. Perl and Perl Modules
6	- FindBin
7	- Mail::Verp
8	- MIME::Parser
9	- MIME::Words
10	- DBI
11	- Getopt::Std
12	- Encode
13
14	2. Exim as your MTA.
15
16	3. Configuration of your MTA so that the OG2list router and transport only handle the mail they are supposed to. Otherwise, the OG2list router and transport will take in and send out all mail sent to the server. Which will effectively render email that wasnt meant for OG2list useless.
17
18	4. A dedicated domain or sub-domain (localhost will do)
19
20	5. CVS and SVN are recommended
21
22	6. Shell Access to your server with sudo/root privileges
23
24	7. Drupal 4.7
25
26	8. If your on OS X. You will need to install Wget for the drupal cronjobs. You will also need to install lynx for CPAN. You can use Fink (http://fink.sourceforge.net/) to do this
27
28	This was written as a guide to installing the OG2list module on OS X using the Exim MTA.
29	The instructions should be relevant for any *nix based OS though.
30
31	Step 1 : Install Perl Pre requisites
32
33	A. Install CPAN
34
35	To get CPAN do this:
36	OS X : Install Xcode
37
38	After you have the Xcode package you will have access to perl and its package management system, CPAN.
39	CPAN allows you to install extra packages and modules for Perl to use. OG2list relies on 3 more than come default.
40
41	To access CPAN, as root, type:
42
43	'cpan'
44
45	you should see
46
47	cpan shell -- CPAN exploration and modules installation (v1.7601)
48	ReadLine support available (try 'install Bundle::CPAN')
49
50	cpan>
51
52	you can perform the installs by typing each of the install lines below, each install will have its own set of prompts and questions - agree to everything:
53
54	B. Install Perl Modules
55	From within CPAN, do...
56
57	install DBI
58	install DBD::mysql
59	install Mail::Verp
60	install FindBin
61	install Getopt::Std
62	install MIME::Tools
63	install MIME::Words
64	install Encode
65
66	Integration into MTA setup (Exim 4.x)
67	Global
68	The global section of Exim's configuration file is the place to
69	define variables and installation-specific parameters such as the
70	SQL location below.
71
72	See Section 9.21 of the Exim specification for details.
73
74	The user/group ID and the home directory need not match any other
75	user account on the system, in particular not the webserver's or the
76	web application's user. mail-in.pl only needs to access its
77	configuration file and the og2list_* tables in the SQL database for
78	which the credentials are stored in the configuration file.
79
80	OG2LIST_HOME = /home/og2list
81	OG2LIST_UID = og2list
82	OG2LIST_GID = og2list
83
84	hide mysql_servers = host/dbname/dbuser/dbpass
85
86	Router
87	Routers are rules that determine how a destination address should be
88	handled. The following piece of code assigns addresses to a
89	transport rule called og2list_transport which is defined below.
90
91	Note that the since Routers are evaluated one after anotehr, their
92	order in the configuration file will most likely affect overall
93	behavior.
94
95	See section 3.8 of the Exim specification for details.
96
97	og2list:
98	debug_print = "R: og2list from $return_path \
99	for $local_part@$domain"
100	driver = accept
101	transport = og2list_transport
102	condition = ${lookup mysql{SELECT 1 FROM og2list_groups \
103	WHERE recipient \
104	LIKE '$local_part'} \
105	{$value} \
106	fail}
107
108	If Exim has been built without MySQL support, the following
109	condition statement should work:
110
111	condition = ${run{/path/to/valid-recipient.pl $local_part}{1}{0}}
112
113	It is possible to use the "domains" statement to restrict the
114	domains for which the og2list is actually tried (see Exim
115	specification, chapter 15 for details).
116
117	"abuse"(RfC2142), "postmaster"(RfC2821) and other essential
118	local parts should be handled by a router which precedes the
119	og2list router. The system_aliases router that is part of
120	Exim's default configuration file is a good place for this.
121
122	Transport
123	Transports are the rules that determine by what means a message
124	should be delivered that has been assigned to this transport by a
125	router.
126
127	og2list_transport:
128	debug_print "T: og2list_transport from \
129	$return_path for $local_part@$domain"
130	driver = pipe
131	freeze_exec_fail
132	log_output
133	command = OG2LIST_HOME/mail-in.pl \
134	-f $return_path -t $local_part@$domain
135	current_directory = OG2LIST_HOME
136	user = OG2LIST_UID
137	group = OG2LIST_GID
138
139	Debugging - or what to do if something does not work
140	----------------------------------------------------
141
142
143	A) No mail is recieved by the group
144
145	If you have sent mail to your groups address and it is not appearing inside your groups forum. Try these debugging steps.
146
147	0) Make sure Exim is running and is the MTA for the domain you are trying to send to.
148
149	TODO: Add link to run this script http://cvs.drupal.org/viewcvs/drupal/contributions/modules/devel/generate/generate-og-users.php
150
151	1) Enable your mysql query log and check to make sure the messages are getting inserted into the og2list tables.
152
153	The mysql configuration files, called my.cnf, are located in different places depending on your setup. Most popular place is /etc/my.cnf
154	If you absolutely cant find your my.cnf file. Find all instances by doing
155
156	sudo find / -iname 'my.cnf'
157
158	mysql 4.1 - add
159	log=/path/to/logfile
160	under [mysqld] in your my.cnf and then restart mysql
161
162	mysql 5.1 - add
163	log=/path/to/logfile
164	under [mysqld] in your my.cnf and then restart mysql with
165	sudo /path/to/mysql/bin/mysqld_safe --user=mysql --log --log-output=FILE &
166
167	In order to recreate the situation
168	a) send mail to a group address
169
170	echo "some sort of text" >> ~/test-mail.txt
171
172	There are testmails located under /path/to/civicspace/modules/og2list/testmails
173	Youll need to edit them and change the from address to an address of a member in your group with posting permissions.
174	You can use them by executing this command
175
176	mail -s "This is the subject" group_addy@www.example.com < /path/to/civicspace/modules/og2list/testmails/killes/tb-arabic
177
178	To check to see what your groups address, or if your groups address is being properly created. Enter your mysql DB and
179
180	mysql> select * from og2list_groups;
181	+-----+-----------+--------+
182	\| nid \| recipient \| status \|
183	+-----+-----------+--------+
184	\| 2 \| test \| 1 \|
185	+-----+-----------+--------+
186
187	mail -s "subject line" test@www.example.com < ~/test-mail.txt
188
189	In this case the address for that group would be test@www.example.com
190	So double check and make sure the group your sending mail to actually has an address.
191
192	If your group does not have an address, make sure you added the address upon initial creation of a group. Group mailing addresses cannot be connected to a group after the group has been created.
193
194	b) check your Exim log to make sure it got there and that the relay is equal to "drupal" - to confirm it was handed to the correct transport. Exim logs are generally located at '/var/log/mail.log'
195
196	If the message was succesfully recieved and relayed you should have an entry that looks somewhat like. The 3 parts you should look at.
197
198	to= this should be the email address of your group
199	relay = this has to say "drupal", otherwise, your email is being spat out into the ether, never to return.
200	status = this should say sent. if it says deferred or bounced. Make sure the address of your group is correct. see #6, if the address is correct. Check your Exim configuration.
201
202	Jun 8 18:32:16 punlc exim/pipe[13474]: 6C89A9A1035: to=<mailin@www.example.com>, relay=drupal, delay=16, status=sent (john.localdomain)
203	Jun 8 18:32:16 punlc exim/qmgr[8473]: 6C89A9A1035: removed
204
205	c) run cron - log in as an administrator on your site and then surf to: http://www.example.com/cron.php
206
207	d) check your mysql query log for INSERTS into og2list_incoming_content and og2list_incoming_groups
208
209	mysql> select * from og2list_incoming_content;
210	mysql> select * from og2list_incoming_groups;
211
212	2) Make sure you created an organic group with a mailing list.
213	mysql> select * from og2list_groups;
214
215	3) Make sure you are an approved member of the group you are trying to send to. That means the mail address you are sending from has to match the one used for your Drupal account.
216
217	4) Check your Exim log (mail.log) for anomalities.
218
219	use this command to continually scroll additions to the log onto the screen.
220
221	tail -f /var/log/mail.log
222
223	Once you have the file tailed. Send a message or two into one of your groups address, or Post a few topics in your groups forums.
224	In a few seconds you should see the messages get processed by Exim and be spit out into the log.
225	Look at the specific values. Make sure to look at status,relay, and the to and from addresses. See (b) above to see specifics on analyzing Exim log entries.
226
227	5) Try to debug from the commandline according to the documentation in
228	mail-in.pl (-d switch):
229
230	TODO: give an exact example with sender and recipient values
231
232	./mail-in.pl -d -f sender -t recipient < drupal/modules/og2list/testmails/killes/tb-arabic
233
234	sender is an email address of a member of the group which has the
235	local email 'recipient'. Test mails can be found in the testmails
236	directory.
237
238	Make sure to check your Exim log afterwards.
239
240	6) Check to make sure your group has an address. You can do this by using this query in mysql.
241
242	select * from og2list_groups.
243
244	You should see something like
245
246	mysql> select * from og2list_groups;
247	+-----+-----------+--------+
248	\| nid \| recipient \| status \|
249	+-----+-----------+--------+
250	\| 1 \| mailin \| 1 \|
251	+-----+-----------+--------+
252	1 row in set (0.01 sec)
253
254	In this case the address for that group would be mailin@www.example.com
255	So double check and make sure the group your sending mail to actually has an address.
256
257	the group you are sending mail to should have its address in the recipient column.
258
259	If your group does not have an address, make sure you added the address upon initial creation of a group. Group mailing addresses cannot be connected to a group after the group has been created.
260
261	B) No mail is sent to subscribers
262
263	0) Check that your Drupal cronjob is running. Try accessing your installation's cron.php file from your browser.
264
265	a) log in as admin
266	b) goto www.example.com/cron.php in order to manually run your cronjob
267	c) make sure there is an entry in your cronjob that looks something like this:
268
269	/5 * * * /sw/bin/wget -O - -q http://www.example.com/cron.php
270
271	This cronjob will run every 5 minutes, as denoted by the */5. If yours is not running frequently. This could be the cause of why you havent seen your messages appear in your group forums in a timely manner. On the other hand, you should be wary of setting your cronjob to run any more frequently than every 2 minutes. As two cronjobs running at the same time could cause duplicate messages to be sent out or imported.
272
273	1) Make sure that the group's mailing list is not disabled.
274
275	a)Log in as your groups administrator.
276	b)Under the "My Groups" block, select your group.
277	c)Goto the "Mailing List" Tab on your groups homepage.
278	d)Make sure it is set to "Enabled".
279
280	3) Do the settings in mail.conf match your database and Exim
281	configuration?
282
283	Your $mail_domain is simply the domain of the site where you are running og2list. if your sites address is "www.example.com" your $mail_domain should look something like:
284	# The domain that is appended to the ogroup-specific localparts.
285	$mail_domain = "example.com";
286
287	Your $dbtype will not change. it should be "mysql".
288	# Connection parameters to the Drupal database
289	# type of database connection
290	$dbtype = "mysql";
291
292	These are your DB settings. See below for how to test them.
293	# name of your database server
294	$dbase = "og2list";
295	# name of your Drupal database
296	$dbhost = "localhost";
297	# name of your database user
298	$dbuser = "og2listuser";
299	# the user's password for your database server
300	$dbauth = "og2listpass";
301
302	To test your db connection fill in the the parameter name with its value from your mail.conf. If you successfully make a connection its correct, otherwise. its wrong. pay careful attention to the -p. theres no space between the -p and the value for a reason.
303	mysql -u og2listuser -h localhost -pog2listpass og2list
304
305	# Path to sendmail binary
306	$sendmail_binary = "/usr/sbin/sendmail";
307
308	to find sendmail do
309	whereis sendmail
310	and you should get a result back saying the location of sendmail.
311	/usr/sbin/sendmail
312
313	4) Try to run mail-out.pl from the commandline. An error message should be printed. Show it to your system administrator.
314
315	Switch to your og2list user and run mail-out.pl
316	sudo su og2list
317	./mail-out.pl
318
319	5) Check your Exim log (mail.log) for anomalities.
320
321	C) I sent mail to my group but cant see it!
322
323	1) Check under admin >> content to make sure your posts are coming in 'unpublished'
324
325	2) See above for checking your mail.log to make sure that the mail is being recieved and processed correctly.
326
327