Mark Sapiro writes:

In Postorius go to the Templates tab for the list, click New Template, then in the Name dropdown select [list:member:regular:footer], leave the Data box empty and Save.

Does Django empty out the "spacetebasket" if it's all white space, or does spurious whitespace in that box lead to an (apparently) empty footer being added?

Anway, I recommend to the OP that they make sure there's no whitespace in the Data box.

I didn't touch the data box at all, and it looks like it's working, so thanks for those instructions!

On 12/8/21 6:05 PM, Philippe B wrote:

Update: After disabling the footer on a few lists (as described here a week ago), we started seeing "Cache content lost" in most (but not all) messages from those lists.

In some cases, that phrase is an attachment to the "real" body of the message. In others, it's the opposite (which is more problematic for responding). That seems to depend on whether the poster sent in plain text.

The difference between inline and attachment has to do with how footers are added. They are only added inline if the message after content filtering is a simple text/plain message.

One puzzling aspect is that it doesn't happen consistently. One hourly automated message will have it; but exactly the same message one hour later won't have it; but will have it again 3 hours later for example.

Is that a known problem, or would it be possible to troubleshoot? Thanks.

The issue may be a bug or a race condition. What is going on is the 'empty' footer you created in Postorius is retrieved from Postorius by a URL. In order to reduce the number of http transfers to retrieve such templates, they are cached in the local file system. Now, when we want a template that is accessed via a non-mailman: URI, we call the cache managers get() method at https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/model/cache.py#... to get it. This should return either None if there's no cached file or the contents of the file, but if there is an entry but no corresponding file, it returns 'Cache content lost'.

This could occur if some asynchronous process expunged the entry between getting the entry and getting the file. A process that could do that is the task runner that evicts expired cache entries.

Look in your mailman.log. There will be ACCEPT: <message_id> entries for each accepted message and there will be hourly Task runner evicted expired cache entries entries. Do the ACCEPTs for the affected messages occur at the same time as the evictions?

One way to avoid this is to not use Postorius to set the template but rather, create an empty templates/lists/<list_id>/en/list:member:regular:footer.txt in mailman's var/ directory for each such list and delete the Postorius templates. Of course, this is just a work around and not a solution to the issue. A proper fix is probably to implement a lock on the get(), evict(), evict_expired() and clear() methods so they can't execute concurrently.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 12/10/21 10:03 AM, dancab@caltech.edu wrote:

Mark,

Is there a mailman-web command that could be used on a specific list to expire cached entries?

The cache is a Mailman core function, not a mailman-web function.

If you have Mailman core >= 3.3.5, the task runner deletes expired cache entries every hour or at intervals defined by the run_tasks_every setting in the [mailman] section of mailman.cfg.

For older versions you could run in an interactive mailman shell

getUtility(ICacheManager).evict_expired()

in a script, you'd also need

from mailman.interfaces.cache import ICacheManager
from zope.component import getUtility

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 12/10/21 10:07 AM, dancab@caltech.edu wrote:

Also, just to clarify we're storing all list templates in MySQL. Would the cacheing to local file system still be necessary in that case?

How is Mailman accessing these templates? If by URL and the URL's scheme is other than mailman templates are cached. See https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/model/template....

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 12/10/21 3:45 PM, dancab@caltech.edu wrote:

Thanks Mark. We're using https:// for the URLs. Mailman 3.3.5 is what we're running.

(mailmanve) mailman@14ea21f6d4b1:/var/www/mailman-www$ mailman conf |grep tasks [mailman] run_tasks_every: 1h

We're using two containers behind a load balancer for Mailman so I suspect this is where the lost cache comes into play.

Yes. The cache entry is stored in the database which presumably is shared, but the actual cached data may only be stored in the file system of the server that cached it so the other server will think it's lost. Are you sharing Mailman's var/ directory between the two servers? I think you need to do that.

Would it be safe to increase the task frequency to every 15 minutes via the setting below?

run_tasks_every: 15m

I don't think that will do what you hope. It won't hurt, but the task runner only evicts expired entries, and they have a life set by cache_life in the [mailman] section of mailman.cfg (not to be confused with cache_lifetime set in the [dmarc] section) with a default of 7 days.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 12/10/21 3:49 PM, dancab@caltech.edu wrote:

Also, just out of curiosity where is this local cache stored on the file system?

In Mailman's var directory in var/cache/. The files are named with a hash of the URL and in a hierarchy based on that hash.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 12/16/21 10:35 AM, dancab@caltech.edu wrote:

Mark,

I updated mailman.cfg to point at our shared EFS mount.

cache_dir: /mnt/var-mailman3/cache

But I'm still getting reports of this issue. Is there somewhere else I need to set that location?

If your layout is other than fhs, I would set

var_dir: /mnt/var-mailman3/

For fhs layout, I would ensure that at a minimum, /var/lib/mailman, /var/spool/mailman and /var/lock/mailman are all shared.

However for your issue with shared cache, you need to wait 7 days for all the old cache entries to expire or do

getUtility(ICacheManager).clear()

in mailman shell to clear the cache.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 12/17/21 2:15 PM, dancab@caltech.edu wrote:

Thank you. Should I be able to see the specific cache creation and expiration dates in the file_cache table of the database? Once I run the command you mentioned, should I then see that table cleared?

MySQL [mailman3_prod]> select id, created_on, expires_on from file_cache;

Yes. Also, you can see the file_id, and the key is the URL the template was retrieved from. The file_id is a 64 byte hex hash of the key and points to the cached content. If the file_id is, e.g., e4ff6a68a78b7ca911d5b16271980fa0cb816938fa187ec0f62bae707e541de0, the content it stored at /e4/ff/e4ff6a68a78b7ca911d5b16271980fa0cb816938fa187ec0f62bae707e541de0 relative to the cache_dir setting (or whatever it was when the content was stored).

+----+---------------------+---------------------+ | id | created_on | expires_on | +----+---------------------+---------------------+ | 1 | 2021-03-03 23:02:48 | 2021-03-10 23:02:48 | | 2 | 2021-03-03 23:42:49 | 2021-03-10 23:42:49 | | 4 | 2021-03-05 21:13:07 | 2021-03-12 21:13:07 | | 6 | 2021-03-05 22:30:33 | 2021-03-12 22:30:33 | | 7 | 2021-11-23 00:11:23 | 2021-11-30 00:11:23 | | 11 | 2021-11-23 23:09:36 | 2021-11-30 23:09:36 | | 12 | 2021-11-23 23:23:28 | 2021-11-30 23:23:28 | | 13 | 2021-11-23 23:48:51 | 2021-11-30 23:48:51 | | 14 | 2021-11-23 23:53:30 | 2021-11-30 23:53:30 | | 15 | 2021-11-24 00:03:40 | 2021-12-01 00:03:40 | | 16 | 2021-11-24 00:14:03 | 2021-12-01 00:14:03 | | 17 | 2021-11-30 22:50:14 | 2021-12-07 22:50:14 | | 18 | 2021-12-01 15:57:48 | 2021-12-08 15:57:48 | | 19 | 2021-12-01 15:57:48 | 2021-12-08 15:57:48 | | 20 | 2021-12-01 17:01:09 | 2021-12-08 17:01:09 | | 21 | 2021-12-01 17:41:42 | 2021-12-08 17:41:42 | | 22 | 2021-12-02 23:09:58 | 2021-12-09 23:09:58 | | 23 | 2021-12-03 22:52:10 | 2021-12-10 22:52:10 | | 24 | 2021-12-04 02:54:06 | 2021-12-11 02:54:06 | | 27 | 2021-12-08 14:28:41 | 2021-12-15 14:28:41 | | 29 | 2021-12-10 17:43:16 | 2021-12-17 17:43:16 | | 30 | 2021-12-11 02:00:04 | 2021-12-18 02:00:04 | | 31 | 2021-12-13 09:02:44 | 2021-12-20 09:02:44 | | 33 | 2021-12-16 20:07:11 | 2021-12-17 20:07:11 | | 35 | 2021-12-17 19:21:19 | 2021-12-18 19:21:19 | +----+---------------------+---------------------+ 25 rows in set (0.001 sec)

It is not clear to me why you have long expired entries in the file_cache table. The task runner should have expunged those.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 12/17/21 3:06 PM, dancab@caltech.edu wrote:

I don't know why either. The mailman shell command seems to have had no effect either.

Would it be safe to remove those expired rows directly via MySQL commands? Is the id column keyed or joined to another table where I would also have to delete corresponding rows?

Yes it should be fine. Just

DELETE FROM file_cache;

I'm assuming your MySQL database is shared between your two Mailman servers or that both are accessing the same MySQL server. Is it somehow possible that there are two MySQL databases and Mailman is using one and you are looking at another?

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan