Caching problem with templates
Caching problem with templates.
GNU Mailman 3.3.10 Python 3.11.2 Postorius Version 1.3.13 Venv installation per docs
Hi All,
A footer template is created and initially it works as expected.
Editing the footer template sees the new content written to the database,
In addition to viewing the edited content in the database, the new template can also be viewed in Lynx on localhost at:
http://localhost:8000/mailman3/api/templates/list/<IDENTIFIER>/list:member:regular:footer
The problem is that the original template is being cached somewhere and the old footer persists in new emails.
In trying to get the edited footer to show in the emails I've restarted mailman3/mailmanweb, rebooted the server - but no joy.
Any advice on how to flush the cache or hammer this gremlin greatly appreciated.
Thanks, Mark
On 6/26/25 04:25, Mark wrote:
A footer template is created and initially it works as expected.
Editing the footer template sees the new content written to the database,
In addition to viewing the edited content in the database, the new template can also be viewed in Lynx on localhost at:
http://localhost:8000/mailman3/api/templates/list/<IDENTIFIER>/list:member:regular:footer
The problem is that the original template is being cached somewhere and the old footer persists in new emails.
What does the database query
SELECT uri FROM template WHERE name = 'list:member:regular:footer' AND
context = '<IDENTIFIER>';
return?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 6/26/25 04:25, Mark wrote:
A footer template is created and initially it works as expected.
Editing the footer template sees the new content written to the database,
In addition to viewing the edited content in the database, the new template can also be viewed in Lynx on localhost at:
http://localhost:8000/mailman3/api/templates/list/<IDENTIFIER>/list:member:regular:footer
The problem is that the original template is being cached somewhere and the old footer persists in new emails.
=================================
On 2025-06-26 16:10, Mark Sapiro wrote:
What does the database query
SELECT uri FROM template WHERE name = 'list:member:regular:footer' AND context = '<IDENTIFIER>';return?
=================================
The result of the query is: +--- uri +---- http://localhost:8000/mailman3/api/templates/list/<IDENTIFIER>/list:member:regular:footer +----
I should have also said, if I delete the template, emails still use the old footer -- not the Mailman default as I would have expected.
And I had a look in /opt/mailman/mm/var/templates/lists/ -- nothing there.
On 6/26/25 15:12, Mark wrote:
On 2025-06-26 16:10, Mark Sapiro wrote:
What does the database query
SELECT uri FROM template WHERE name = 'list:member:regular:footer' AND context = '<IDENTIFIER>';return?
=================================
The result of the query is: +--- uri +---- http://localhost:8000/mailman3/api/templates/list/<IDENTIFIER>/list:member:regular:footer +----
I should have also said, if I delete the template, emails still use the old footer -- not the Mailman default as I would have expected.
As long as there is an entry in the template table with a uri which is
not a mailman:// scheme, that entry is cached in the file_cache table
in the database with a key = the uri and a lifetime set by cache_life in
the [mailman] section of mailman.cfg - default 7 days.
You can delete the cached entry from the database via `mailman shell as in
$ mailman shell
Welcome to the GNU Mailman shell
Use commit() to commit changes.
Use abort() to discard changes since the last commit.
Exit with ctrl+D does an implicit commit() but exit() does not.
>>> cm = getUtility(ICacheManager)
>>> uri = 'the_uri_from_the_template_table'
>>> cm.get(uri, expunge=True)
>>> commit()
You can also delete the entry from the file_cache table, but if you do
that, the actual file containing the cached data will remain. To remove
it, note that the file_cache table entry has a file_id. This will be a
hex digest of the sha256 hash of the uri, something like
c85210810f8824cf9227c1689817e1d081046ff132bf8abe56a7c08a89016e4e and
the file is stored in the cache/ directory in Mailman's var/ directory
with a path of file_id[0:2]/file_id[2:4]/file_id. I.e. for the above
example file_id, the file will be
var/cache/c8/52/c85210810f8824cf9227c1689817e1d081046ff132bf8abe56a7c08a89016e4e.
If you remove the entry from the file_cache table without removing the file, the file will remain forever but will be overwritten if the same uri is cached again.
Rather than doing any of the above, if you want to manipulate the database, just set the expires_on column in the file_cache table entry to a date in the past.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Thu, 2025-06-26 at 18:36 -0700, Mark Sapiro wrote:
You can delete the cached entry from the database via `mailman shell as in
>>> cm = getUtility(ICacheManager) >>> uri = 'the_uri_from_the_template_table' >>> cm.get(uri, expunge=True) >>> commit()
You can also delete the entry from the file_cache table, but if you do that, the actual file containing the cached data will remain. To remove it, Rather than doing any of the above, if you want to manipulate the database, just set the expires_on column in the file_cache table entry to a date in the past.
Hi Mark,
Thank you for this information. I've given the options a good workout over the last couple of days and found that:
Deleting the cache via the mailman shell does the job, but is beyond me to script. There are numerous lists and list owners updating their templates.
Changing the expires_on date (to a date in the past) in the file_cache table didn't prevent the cached entry from being used;
What did work was removing the records in the file_cache table PLUS removing the corresponding files in var/cache/*/*/* -- for example: /opt/mailman/mm/var/cache/3a/2f/*
This I was able to script in bash and cron to run every minute.
A hack I know, but it's working for now.
I've attached the script in case it's of use to anyone else. (mailman_clear_template_cache.txt) Note: change <USER> in the script to what suits.
Does Mailman really need to cache the templates? It seems to be an unnecessary complication.
Anyways it happy days again. Thank you again, very much. Your guidance was appreciated immensely as always.
Best, Mark
On 6/28/25 20:22, Mark wrote:
- What did work was removing the records in the file_cache table PLUS removing the corresponding files in var/cache/*/*/* -- for example: /opt/mailman/mm/var/cache/3a/2f/*
This I was able to script in bash and cron to run every minute.
A much simpler way to accomplish the same thing is to set
cache_life: 1m
in the [mailman] section of mailman.cfg.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Sat, 2025-06-28 at 20:56 -0700, Mark Sapiro wrote:
A much simpler way to accomplish the same thing is to set
cache_life: 1m
in the [mailman] section of mailman.cfg.
Yes, I thought so too. I also tried 0d -- which wrote the expiry date the same as the created date (to the second).
# /etc/mailman3/mailman.cfg [mailman] cache_life: 0d
But, just as with writing in a past date, it didn't solve the problem. I also spun up a brand new server to thrash it out there, but no joy there either.
On 6/28/25 21:50, Mark wrote:
Yes, I thought so too. I also tried 0d -- which wrote the expiry date the same as the created date (to the second).
# /etc/mailman3/mailman.cfg [mailman] cache_life: 0d
But, just as with writing in a past date, it didn't solve the problem. I also spun up a brand new server to thrash it out there, but no joy there either.
This is a different issue. See https://gitlab.com/mailman/mailman/-/issues/1231
The template is not removed from the cache until the next run of the task runner which defaults to running every hour.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 6/28/25 22:55, Mark Sapiro wrote:
This is a different issue. See https://gitlab.com/mailman/mailman/-/issues/1231
Both this and https://gitlab.com/mailman/mailman/-/issues/1230 are fixed by https://gitlab.com/mailman/mailman/-/merge_requests/1368 and https://gitlab.com/mailman/mailman/-/merge_requests/1367 respectively. The fixes will be in Mailman 3.3.11. This patch ``` --- a/src/mailman/model/cache.py +++ b/src/mailman/model/cache.py @@ -27,7 +27,7 @@ from mailman.database.model import Model from mailman.database.transaction import dbconnection from mailman.database.types import SAUnicode from mailman.interfaces.cache import ICacheManager -from mailman.utilities.datetime import now +from mailman.utilities.datetime import now, ZERO from mailman.utilities.filesystem import safe_remove from public import public from sqlalchemy import Boolean, Column, DateTime, Integer @@ -104,6 +104,8 @@ class CacheManager: """See `ICacheManager`.""" if lifetime is None: lifetime = as_timedelta(config.mailman.cache_life) + if lifetime == ZERO: + return None is_bytes = isinstance(contents, bytes) file_id = self._key_to_file_id(key) # Is there already an unexpired entry under this id in the database? @@ -124,7 +126,7 @@ class CacheManager: """See `ICacheManager`.""" entry = store.query(CacheEntry).filter( CacheEntry.key == key).one_or_none() - if entry is None: + if entry is None or entry.expires_on <= now(): return None file_path, dir_path = self._id_to_path(entry.file_id) with ExitStack() as resources: ``` implements both fixes. -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 6/28/25 21:50, Mark wrote:
Yes, I thought so too. I also tried 0d -- which wrote the expiry date the same as the created date (to the second).
# /etc/mailman3/mailman.cfg [mailman] cache_life: 0d
But, just as with writing in a past date, it didn't solve the problem. I also spun up a brand new server to thrash it out there, but no joy there either.
On Sat, 2025-06-28 at 22:55 -0700, Mark Sapiro wrote:
This is a different issue. See https://gitlab.com/mailman/mailman/-/issues/1231
The template is not removed from the cache until the next run of the task runner which defaults to running every hour.
Dang! If you know, you know. Thank you for that, and the patch information. I'll leave my hack in place for now (it works) until the next release. My brain is fried from the last couple of days getting to grips with this.
Best, Mark
Mark Sapiro wrote:
A much simpler way to accomplish the same thing is to set
cache_life: 1m
in the [mailman] section of mailman.cfg.
See https://gitlab.com/mailman/mailman/-/issues/1230
I intend to implement disabling the cache entirely by setting cache_life to any zero value, i.e. 0s, 0m, 0h, etc.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Sat, 2025-06-28 at 21:55 -0700, Mark Sapiro wrote:
See https://gitlab.com/mailman/mailman/-/issues/1230
I intend to implement disabling the cache entirely by setting cache_life to any zero value, i.e. 0s, 0m, 0h, etc.
Setting the cache_life to a zero value only writes the expiry date as "now". The cached file is still there. And it doesn't appear to be getting overwritten (as expected) when the template is edited.
On 6/28/25 22:05, Mark wrote:
Setting the cache_life to a zero value only writes the expiry date as "now". The cached file is still there. And it doesn't appear to be getting overwritten (as expected) when the template is edited.
The template caching occurs on retrieval. Editing the template in Postorius does not change the URI in the template table, so a subsequent retrieval still gets the old cached value.
The task runner will delete the expired template the next time it runs but with default settings that's up to an hour later.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (2)
-
Mark -
Mark Sapiro