mailman3 not working for mailman21 lists
I installed mailman3 on a server where mailman21 was installed before. Newly created mailinglists work fine. But if I create mailinglists with a name that was previously used on mailman21 nothing works (same behaviour with and without import of old configs and archives; no logs). I purged all old mailman21 config and data and also old postfix aliases (or at least I do think so).
If I send a mail to old list list1@a.b and new list list2@a.b I get the following logs
/var/log/mail.log Aug 21 22:17:40 basilisgg postfix/lmtp[1728]: ED2171120BF: to=list2@c.a.b, orig_to=list2@a.b, relay=127.0.0.1[127.0.0.1]:8024, delay=0.31, delays=0.17/0.01/0/0.13, dsn=2.0.0, status=sent (250 Ok) Aug 21 22:17:40 basilisgg postfix/lmtp[1728]: ED2171120BF: to=list1@c.a.b, orig_to=list1@a.b, relay=127.0.0.1[127.0.0.1]:8024, delay=0.31, delays=0.17/0.01/0/0.13, dsn=2.0.0, status=sent (250 Ok)
/var/log/mailman3/smtp.log ug 21 22:17:40 2022 (934) Available AUTH mechanisms: LOGIN(builtin) PLAIN(builtin) Aug 21 22:17:40 2022 (934) Peer: ('127.0.0.1', 44746) Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) handling connection Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'LHLO c.a.b' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'MAIL FROM:<....>' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) sender: .... Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'RCPT TO:list2@c.a.b' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) recip: list2@c.a.b Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'RCPT TO:list1@c.a.b' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) recip: list1@c.a.b Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'DATA' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'QUIT' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) connection lost Aug 21 22:17:40 2022 (934) Connection lost during _handle_client() Aug 21 22:17:46 2022 (936) <166111306097.933.1775779871266718007@...> smtp to list2@a.b for 1 recips, completed in 3.1437485218048096 seconds
How can I get more info on why the mail for list1 isn't processed? Where could old mailman21 or postfix info on list1 be stored that causes mailman3 to not handle the list properly? How can I diff the config between list1 and list2 from the command line?
Regards Diana
blup@fortytwo.ch writes:
I installed mailman3 on a server where mailman21 was installed before. Newly created mailinglists work fine. But if I create mailinglists with a name that was previously used on mailman21 nothing works
Your logs below show that *something* is working because Mailman 3 is logging the incoming message.
(same behaviour with and without import of old configs and archives; no logs).
Archives should have nothing to do with it (except that if there are no members with delivery enabled, you might see posts delivered to the archives, but nothing in outgoing SMTP logs).
I don't understand what you mean by "without import of old configs". Do you mean you just create the list with the old name? Did you do any further configuration and set up of that list?
I purged all old mailman21 config and data
This should not be necessary. Mailman 2 and Mailman 3 can even happily coexist on the same host as long as you don't try to reuse /etc/mailman, /usr/lib/mailman and /var/lib/mailman (or whatever they happen to be) from Mailman 2 for Mailman 3.
and also old postfix aliases (or at least I do think so).
Since Mailman 3's SMTP logs below show incoming LMTP for both lists, I don't think there's a problem with Postfix.
If I send a mail to old list list1@a.b and new list list2@a.b I get the following logs
Logs edited for brevity:
/var/log/mail.log 22:17:40 basilisgg postfix/lmtp[1728]: ED2171120BF: to=list2@c.a.b, status=sent (250 Ok) 22:17:40 basilisgg postfix/lmtp[1728]: ED2171120BF: to=list1@c.a.b, status=sent (250 Ok)
So this is one message with two lists as addressees.
/var/log/mailman3/smtp.log Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'LHLO c.a.b' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'MAIL FROM:<....>' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'RCPT TO:list2@c.a.b' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'RCPT TO:list1@c.a.b'
Postfix has the right alias, since it's sending messages for list1 to Mailman's LMTP port. It also appears list1 was accepted here.
Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'DATA' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'QUIT' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) connection lost Aug 21 22:17:40 2022 (934) Connection lost during _handle_client() Aug 21 22:17:46 2022 (936) smtp to list2@a.b for 1 recips, completed in 3.14 seconds
I don't see any good reason to believe either Postfix or Mailman 3 is misbehaving from the above.
How can I get more info on why the mail for list1 isn't processed?
First check the queues (normally /var/lib/mailman/qfiles/*/* where the first * is a wildcard for list names and the second is a wildcard for message objects, the '/var/lib' part may be different in your installation) for a "stuck" message. Second, have you checked the other Mailman logs for information? Third, are both lists visible in the message header as addressees?
Is the 'outgoing' qrunner still running? (ps ax | grep outgoing) Are there any subscribers with delivery enabled on 'list1'? Is list1 configured to accept only member posts? If so, is the sender a member whose posts are accepted?
Is there a sibling list setting that would prevent the message from being sent to the same address if it is on both lists? Are the spam filter settings the same?
How can I diff the config between list1 and list2 from the command line?
This doesn't seem to be implemented. I need to prep for some meetings, but here's the basic strategy I'd follow:
- Use "mailman withlist -i -l list1". This puts you in a Python interpreter command line.
- Next "attrs = dir(m)" to get the list attributes as a list.
- Check "attrs" for things like the roster of subscribers that you're not interested in printing out, and delete them.
- Open a file as 'f', then loop over attr in attrs using "print(f'{attr}: {getattr(m, attr)}', file=f)".
Then do the same for list2 (sorry, I don't know if there's a way to change lists in the middle of withlist), and diff the two files. No promises that will work exactly as written above, but it should be close. If you don't get it quickly, feel free to come back and ask for more help, I just am out of time and wanted to at least give you the basics.
Steve
Stephen J. Turnbull wrote:
blup@fortytwo.ch writes:
I installed mailman3 on a server where mailman21 was installed before. Newly created mailinglists work fine. But if I create mailinglists with a name that was previously used on mailman21 nothing works If I send a mail to old list list1@a.b and new list list2@a.b [..] Your logs below show that *something* is working because Mailman 3 is logging the incoming message.
I didn't mean nothing literally, what I meant is that the mail to list1 is neither being sent out nor shown in the list archives nor ... I.e. all general stuff is working, but things specific for list1 aren't (whereas they do for list2).
(same behaviour with and without import of old configs and archives; no logs). I don't understand what you mean by "without import of old configs". Do you mean you just create the list with the old name? Did you do any further configuration and set up of that list?
First I did create the list with the old name and imported the old configs with mailman import21. As the list did not work then, I deleted it and created it again (with the same manual configuration as for the new list that's working). If you think the problem is neither mailman3 nor postfix, the only issue could be that the first import set some wrong setting for the old list which was not removed when deleting the list.
How can I get more info on why the mail for list1 isn't processed? First check the queues (normally /var/lib/mailman/qfiles/*/* where the first * is a wildcard for list names and the second is a wildcard for message objects, the '/var/lib' part may be different in your installation) for a "stuck" message.
I don't have a qfiles directory. In /var/lib/mailman3/ there are archives cache data lists locks messages queue templates web
Second, have you checked the other Mailman logs for information?
Sure -- nothing there...
Third, are both lists visible in the message header as addressees?
You mean in the header of the received message via list2? Yes it is.
Is the 'outgoing' qrunner still running? (ps ax | grep outgoing)
no
Are there any subscribers with delivery enabled on 'list1'?
yes
Is list1 configured to accept only member posts? If so, is the sender a member whose posts are accepted?
Yes. Anyway: there should be a reaction if there is a post from a non-member (i.e. mail shown in held messages, sender and admin being informed of it).
Is there a sibling list setting that would prevent the message from being sent to the same address if it is on both lists?
I don't think so. But also if it would be that couldn't be the problem, as my previous tests all only included list1.
Are the spam filter settings the same?
No clue -- where do you set them? But once more: list1 was working fine as long as mailman21 was used and stopped working when switching to mailman3. I don't expect spam filters that worked with mailman21 to stop working with mailman3.
How can I diff the config between list1 and list2 from the command line? This doesn't seem to be implemented. I need to prep for some meetings, but here's the basic strategy I'd follow:
Use "mailman withlist -i -l list1". This puts you in a Python interpreter command line. Next "attrs = dir(m)" to get the list attributes as a list.
Should this give an output? I get none...
Check "attrs" for things like the roster of subscribers that you're not interested in printing out, and delete them.
How?
Open a file as 'f', then loop over attr in attrs using "print(f'{attr}: {getattr(m, attr)}', file=f)".
I have more or less no Python knowledge (i.e. I can read and understand Python but can't write it). Could you please give me detailed command by command instructions. Thanks.
Regards Diana
blup@fortytwo.ch writes:
First I did create the list with the old name and imported the old configs with mailman import21. As the list did not work then, I deleted it and created it again (with the same manual configuration as for the new list that's working).
OK. So as far as you know the configurations of list1 and list2 on Mailman 3 are identical, at least in the case where you have deleted list1 and recreated by hand.
I don't have a qfiles directory.
Try queue/, especially look for the out[going], pipeline, and shunt subdirectories.
Is the 'outgoing' qrunner still running? (ps ax | grep outgoing)
no
Sorry, I got that wrong. Mark recommends this:
ps -fwwA|grep runner=|sed s/.*runner=//and gave this as sample output:
archive:0:1
bounces:0:1
command:0:1
in:0:1
lmtp:0:1
nntp:0:1
out:0:1
pipeline:0:1
rest:0:1
retry:0:1
task:0:1
virgin:0:1
digest:0:1
rest:0:1
rest:0:1so it looks like the process name of the runner is "out", not "outgoing". Besides checking for the "out" runner, also check for "pipeline", which has been known to fail in the past.
But once more: list1 was working fine as long as mailman21 was used and stopped working when switching to mailman3. I don't expect spam filters that worked with mailman21 to stop working with mailman3.
Mailman 3 is a complete rewrite, essentially no Mailman 2 code was left untouched because of the pervasiveness of the change of the fundamental string type str from encoded bytes to abstract characters. I don't expect behavior to change, but I certainly am not going to rule anything out when trying to diagnose problems based on so little information.
I have more or less no Python knowledge (i.e. I can read and understand Python but can't write it). Could you please give me detailed command by command instructions. Thanks.
I'll take a look at it tomorrow, but I can't promise results until the weekend if I can't write a script in 30 minutes. Normally Mark Sapiro would handle this kind of thing, but he's offline for a couple weeks so you've got me. My knowledge is mostly on the email protocols side of things (Internet message format, authentication protocols), not the internal processing. I'll do what I can, but if I can't do it real quick it's probably going to take me several hours, which I'm not going to have until the weekend.
Steve
Hi Stephen
On Mon, 2022-08-22 at 23:51 +0900, Stephen J. Turnbull wrote:
blup@fortytwo.ch writes:
Try queue/, especially look for the out[going], pipeline, and shunt subdirectories.
I have several entries in /var/lib/mailman3/queue/shunt/ with Aug 21 timestamps, and several entries in /var/lib/mailman3/queue/shunt/ with current timestamp (i.e. everytime I ls -l there is a new timestamp with the current time).
Is the 'outgoing' qrunner still running? (ps ax | grep outgoing)
no
Sorry, I got that wrong. Mark recommends this:
ps -fwwA|grep runner=|sed s/.*runner=//
and gave this as sample output: [..]
My output looks exactly the same.
> I have more or less no Python knowledge (i.e. I can read and > understand Python but can't write it). Could you please give me > detailed command by command instructions. Thanks.
I'll take a look at it tomorrow, but I can't promise results until the weekend if I can't write a script in 30 minutes. Normally Mark Sapiro would handle this kind of thing, but he's offline for a couple weeks so you've got me. My knowledge is mostly on the email protocols side of things (Internet message format, authentication protocols), not the internal processing. I'll do what I can, but if I can't do it real quick it's probably going to take me several hours, which I'm not going to have until the weekend.
Having something by the weekend would be perfectly fine. Thanks for your time!
Regards Diana
Diana von Bidder writes:
I have several entries in /var/lib/mailman3/queue/shunt/ with Aug 21 timestamps,
Evidence! Preserve it!
You can look at those using "mailman qfile <file>", where the specification for <file> needs to be a regular path. the 'qfile' command does not assume a prefix of /var/lib/mailman3/queue/, unfortunately. Look for non-ASCII characters in headers, very long lines, a line with no indentation that doesn't match the "name: content" pattern that header follow (indented lines are considered to be "logically" part of the previous line so that's not a problem), and empty lines in the header. (There is always an empty line between the message header and the message body, but an empty line in the middle of the header will often cause problems.)
Check the size of the files; probably all these messages were short test messages and it won't matter but in real life you might have a huge attachment which I think would be printed out. Start with the short ones. :-)
and several entries in /var/lib/mailman3/queue/shunt/ with current timestamp (i.e. everytime I ls -l there is a new timestamp with the current time).
I'm not sure what's going on with that. The point of the shunt queue is that Mailman doesn't know what to do and it's putting the messages there to wait for human intervention. So timestamps shouldn't be changing.
My output looks exactly the same.
Good!
If you want you can send me one of those entries in shunt (assuming you're satisfied with my assurance of privacy of any identifying information such as IP addresses, domain names, and mailbox names).
Steve
Stephen J. Turnbull writes:
Diana von Bidder writes:
I have several entries in /var/lib/mailman3/queue/shunt/ with Aug 21 timestamps,
Evidence! Preserve it!
You can look at those using "mailman qfile <file>", where the specification for <file> needs to be a regular path. the 'qfile' command does not assume a prefix of /var/lib/mailman3/queue/, unfortunately.
You can also try the "mailman unshunt" command (which takes no further arguments). Then "mailman unshunt" just moves all the files in "shunt" back to the queues they came from for normal processing. Most likely the same error will happen and they end up in shunt again, but sometimes the problem was on Mailman's side and unshunting causes them to be sent as normal.
Of course, even if the messages get sent, that doesn't mean "everything's fixed", but at least it's a clue to help us figure out why the original messages got shunted (ie, probably not a major defect in the messages themselves).
Regards, Steve
Hi Stephen
Stephen J. Turnbull wrote:
Diana von Bidder writes:
I have several entries in /var/lib/mailman3/queue/shunt/ with Aug 21 timestamps,
Evidence! Preserve it!
You can look at those using "mailman qfile <file>" [..]
These files looked pretty normal...
and several entries in /var/lib/mailman3/queue/shunt/ with current timestamp (i.e. everytime I ls -l there is a new timestamp with the current time).
I'm not sure what's going on with that. The point of the shunt queue is that Mailman doesn't know what to do and it's putting the messages there to wait for human intervention. So timestamps shouldn't be changing.
I just realised that I made a mistake here while typing: it should read "in ... out ...".
You can also try the "mailman unshunt" command (which takes no further arguments). Then "mailman unshunt" just moves all the files in "shunt" back to the queues they came from for normal processing.
As I didn't want to spam the recipients of the mailinglist with a lot of testmails (once they all would be sent out), I created a tar.gz of all the files in shunt and out, then deleted all contents and resent a single new mail (also because in the meantime a mail to another old mailinglist just worked fine, so the problem seems to be connected to a specific list and not lists imported from mailman2 in general).
What then happens is the following:
- I see the connection in /var/log/mailman3/smtp.log and a bak-file in /var/lib/mailman3/in.
- I see an ACCEPT in /var/log/mailman3/mailman.log and the bak-file in /var/lib/mailman3/pipeline
- I get the following error once a second in /var/log/mailman3/mailman.log
Aug 29 11:35:06 2022 (936) HyperKitty failure on http://localhost/mailman3/hyperkitty/api/mailman/urls: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL was not found on this server.</p> <hr> <address>Apache/2.4.54 (Debian) Server at localhost Port 80</address> </body></html> (404) ------ and see the bak-file and a .pck.tmp-file in /var/lib/mailman3/out (always current timestamp). The files always vanish and reoccur with new names (I assume timestamps are part of the names). Doing a mailman qfile on the .pck.tmp file shows nothing special. Do you have any idea which configuration could cause this behaviour for this specific list but not for others?
Regards Diana
Diana von Bidder writes:
You can look at those using "mailman qfile <file>" [..]
These files looked pretty normal...
Are they addressed to the problematic list? What's in them?
Do you actually know what's normal for email, that is, RFC 822 conformance? Do you know how to find out what characters are producing the whitespace you see? Do you know how to find non-printing characters?
What then happens is the following:
- I see the connection in /var/log/mailman3/smtp.log and a bak-file in /var/lib/mailman3/in.
- I see an ACCEPT in /var/log/mailman3/mailman.log and the bak-file in /var/lib/mailman3/pipeline
- I get the following error once a second in
The only reason I can think of at the moment for this to happen so frequently is that you have the list owner set to the list's posting address, or perhaps one of the list's administrative addresses, and each error sends a mail which triggers another error, etc. Please check that.
/var/log/mailman3/mailman.log
Aug 29 11:35:06 2022 (936) HyperKitty failure on http://localhost/mailman3/hyperkitty/api/mailman/urls: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL was not found on this server.</p> <hr> <address>Apache/2.4.54 (Debian) Server at localhost Port 80</address> </body></html> (404)
suggests you have mailman-hyperkitty installed but HyperKitty is either not running, or running on a different host, or Apache doesn't know how to proxy for HyperKitty, or HyperKitty is configured to serve a different base URL (ie, other than http://localhost/mailman3/hyperkitty/).
Are the other lists that seem to be working configured to archive posts? Can you access the archives of the working lists from that URL? If not, mailman-hyperkitty is misconfigured. If you can, I can't think of a reason for this offhand. The relevant configurations are
Hi Stephen Archive wasn't working for any list (I fixed that in the meantime and found debug logging to be quite helpful once I found that option in the config) but all other lists did send mails out.
Anyway: I decided to stop debugging why this one list isn't working, as revenue and expense don't fit. Thanks for your help anyway.
Regards Diana
Diana von Bidder writes:
Hi Stephen Archive wasn't working for any list (I fixed that in the meantime and found debug logging to be quite helpful once I found that option in the config) but all other lists did send mails out.
Anyway: I decided to stop debugging why this one list isn't working, as revenue and expense don't fit. Thanks for your help anyway.
Here's something Mark pointed out in another thread: frequently if one list isn't working while everything else does, it's because list configuration data is inaccessible due to permissions or ownership. Check if the ownership or permissions are different for particular files, especially if they're related to the non-working list(s). This will only take a few minutes to find and fix if it's the problem, I think.
Steve
On 8/24/22 9:25 AM, Diana von Bidder wrote:
I have several entries in /var/lib/mailman3/queue/shunt/ with Aug 21 timestamps, and several entries in /var/lib/mailman3/queue/shunt/ with current timestamp (i.e. everytime I ls -l there is a new timestamp with the current time).
For each of the /var/lib/mailman3/queue/shunt/ entries there should be a
corresponding log entry in mailman.log (guessing
/var/log/mailman3/mailman.log) with the same timestamp logging the
exception and traceback and a SHUNTING message. These exceptions and
tracebacks will indicate what's happening.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Mon, Aug 22, 2022 at 7:04 AM blup@fortytwo.ch wrote:
I installed mailman3 on a server where mailman21 was installed before. Newly created mailinglists work fine. But if I create mailinglists with a name that was previously used on mailman21 nothing works (same behaviour with and without import of old configs and archives; no logs). I purged all old mailman21 config and data and also old postfix aliases (or at least I do think so).
I recently installed (and got it working! Yayi!) MM3 on a server that had MM21. A few things are still fresh in my mind. But I use Exim as the MTA, not Postfix.
If I send a mail to old list list1@a.b and new list list2@a.b I get the
following logs
/var/log/mail.log Aug 21 22:17:40 basilisgg postfix/lmtp[1728]: ED2171120BF: to=list2@c.a.b, orig_to=list2@a.b, relay=127.0.0.1[127.0.0.1]:8024, delay=0.31, delays=0.17/0.01/0/0.13, dsn=2.0.0, status=sent (250 Ok) Aug 21 22:17:40 basilisgg postfix/lmtp[1728]: ED2171120BF: to=list1@c.a.b, orig_to=list1@a.b, relay=127.0.0.1[127.0.0.1]:8024, delay=0.31, delays=0.17/0.01/0/0.13, dsn=2.0.0, status=sent (250 Ok)
/var/log/mailman3/smtp.log ug 21 22:17:40 2022 (934) Available AUTH mechanisms: LOGIN(builtin) PLAIN(builtin) Aug 21 22:17:40 2022 (934) Peer: ('127.0.0.1', 44746) Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) handling connection Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'LHLO c.a.b' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'MAIL FROM:<....>' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) sender: .... Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'RCPT TO:list2@c.a.b' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) recip: list2@c.a.b Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'RCPT TO:list1@c.a.b' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) recip: list1@c.a.b Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'DATA' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'QUIT' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) connection lost Aug 21 22:17:40 2022 (934) Connection lost during _handle_client() Aug 21 22:17:46 2022 (936) <166111306097.933.1775779871266718007@...> smtp to list2@a.b for 1 recips, completed in 3.1437485218048096 seconds
How can I get more info on why the mail for list1 isn't processed? Where could old mailman21 or postfix info on list1 be stored that causes mailman3 to not handle the list properly? How can I diff the config between list1 and list2 from the command line?
In Exim parlance, we usually talk about "routers" and "transports". Routers decide who the mail belongs to and hands the mail to a transport for delivery. In a server where you are running MM3 and MM21, you really need to let ML mails to be processed _first_ by a "router" handling MM3. I don't know so well how you do that on Postfix. It might be possible that rules for MM21 are still being applied in your Postfix config.
I used the following HOWTO - https://wiki.list.org/DOC/Howto_Install_Mailman3_On_Debian10 - for my setup. There is a section that talks about Postfix which you can refer to. However, the whole shebang is here: https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.ht...
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-)
参加者 (5)
-
blup＠fortytwo.ch -
Diana von Bidder
-
Mark Sapiro -
Odhiambo Washington -
Stephen J. Turnbull