Tobias Diekershoff writes:
On 22.11.24 07:59, Tobias Diekershoff wrote:
so far we could not find the reason behind the performance problem.
Since you're already running two allegedly identical systems, have you tried hitting the experimental one with the "do what the developers recommend" hammer? That is, blow away the Debian packages in favor of a vanilla installation from source in a virtual environment (and without --system-site-packages!)? It's impossible to be sure what's going on in a Debian package environment because not only Mailman, but any of the scores of packages that Mailman 3 depends on may have been patched or reconfigured by Debian.
To rule out virtualization problems
Do you mean a VM, which should not cause a problem, or a container, which is known to be tricky to configure? (Note, I've not heard of it causing the kinds of problems you describe.)
in our setup, we tried to migrate MM3 to a bare metal server in the data center. After importing the database for MM3 and Hyperkitty to the new postgresql server MM3 wcomplained about
MultipleResultsFound: Multiple rows were found when one or none was required [in the listarchiver table]
I've seen that in a migration from a Mailman 2 site to Mailman 3. Once I removed the duplicate rows in the archiver table, it stopped. I don't know why it happened, but it was a small number of lists (4 or 5 out of 20k), and at most one extra per (list, archiver) pair. I suspect it carried over from the Mailman 2 instance. I no longer have access to that system to confirm, sadly.
However the duplicates are added back to the listarchiver table for all archivers (mhonarc, hyperkitty, mail-archive and prototype) even though only hyperkitty is activated.
By "duplicates are added back" do you mean something like this:
mailman=> select * from listarchiver; id | mailing_list_id | name | _is_enabled ----+-----------------+--------------+------------- 1 | 1 | mhonarc | f 2 | 1 | mail-archive | f 3 | 1 | prototype | t 4 | 1 | hyperkitty | t 5 | 2 | mhonarc | f 6 | 2 | mail-archive | f 7 | 2 | prototype | t 8 | 2 | hyperkitty | t
where each list has one of each archiver whether or not that archiver is active? This is correct behavior, as long as the archiving/*.py and mailman_hyperkitty.py files are importable by Mailman core. Or by "duplicates are added back" do you mean this?
mailman=> select * from listarchiver; id | mailing_list_id | name | _is_enabled ----+-----------------+--------------+------------- 1 | 1 | mhonarc | f 2 | 1 | mhonarc | f 3 | 1 | mail-archive | f 4 | 1 | mail-archive | f 5 | 1 | prototype | t 6 | 1 | prototype | t 7 | 1 | hyperkitty | t 8 | 1 | hyperkitty | t 9 | 2 | mhonarc | f 10 | 2 | mhonarc | f 11 | 2 | mail-archive | f 12 | 2 | mail-archive | f 13 | 2 | prototype | t 14 | 2 | prototype | t 15 | 2 | hyperkitty | t 16 | 2 | hyperkitty | t
(don't take the id -> (list_id, archiver) mapping too seriously). In the multiple instances of an archiver per list case, which are enabled?
We tried to add a constrain on the table to only allow one entry per list and archiver, but that caused the runner to fail and the message to be shunt.
Uncaught runner exception: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "workaround" DETAIL: Key (mailing_list_id, name)=(98, mhonarc) already exists.
This doesn't make any sense. That appears to be the listarchiver table. As far as I can tell in a quick grep, the only place that is mutated is in mailman/models/mailinglist.py:643 (line number is current HEAD), which happens whenever the list checks for its active archivers. It's initialized lazily for each list in that place.
@public @implementer(IListArchiverSet) class ListArchiverSet: @dbconnection def __init__(self, store, mailing_list): self._mailing_list = mailing_list system_archivers = {} for archiver in config.archivers: system_archivers[archiver.name] = archiver # Add any system enabled archivers which aren't already associated # with the mailing list. for archiver_name in system_archivers: exists = store.query(ListArchiver).filter( ListArchiver.mailing_list == mailing_list, ListArchiver.name == archiver_name).one_or_none() if exists is None: store.add(ListArchiver(mailing_list, archiver_name, system_archivers[archiver_name]))
As you can see, the .add is protected by "if exists". That code has not changed since v3.3.8, which is what is in Bookworm according to "apt list mailman3".
If that is so in your installation, I'm pretty sure the problem is somewhere between the SQLAlchemy API and the RDBMS, not in Mailman itself. And if that's true, that might explain the weirdness in archiving as well.
What version of PostgreSQL are you using? of SQLAlchemy? Are both the original system with performance problems and the new one accessing the same PostgreSQL server, or is there are separate PostgreSQL instance for each?
Steve