Error while mass operation, no progress display while mass operation is running
Dear List,
At the University of Konstanz we have mailing list administrators who frequently add and remove up to 10000 (and sometimes more) list members. I have therefore tested the “Mass subscription” and “Mass removal” operations and found that both the “Remove ALL members” operation and the “Mass subscription” operation report an internal server error on the website, but continue to run in the background until all new members have been successfully added or deleted. Unfortunately, I have not found any error message in the log files. I also tested that mass operations “Remove listed users” and “Remove Selected” ran successfully and without any errors.
Regardless of whether a mass operation process runs without errors or continues to run despite an error, I find it hard that Postorius does not display the progress of such a mass operation in order to monitor the process, as mass operations with more than 1000 members are very time-consuming. Perhaps this could be changed in a future version of Postorius?
Best Markus
-- Markus Grandpré Universität Konstanz Kommunikations-, Informations-, Medienzentrum (KIM) Abteilung IT-Dienste Forschung, Lehre und Infrastruktur, Tel: ++49 7531 88 4342
On 2/6/25 04:53, Markus Grandpré wrote:
At the University of Konstanz we have mailing list administrators who frequently add and remove up to 10000 (and sometimes more) list members. I have therefore tested the “Mass subscription” and “Mass removal” operations and found that both the “Remove ALL members” operation and the “Mass subscription” operation report an internal server error on the website, but continue to run in the background until all new members have been successfully added or deleted.
The internal server error is probably due to the wsgi server timing out waiting for a response. See https://docs.gunicorn.org/en/stable/settings.html#timeout for Gunicorn or https://uwsgi-docs.readthedocs.io/en/latest/Options.html#harakiri for uWSGI
Unfortunately, I have not found any error message in the log files.
If you are using uWSGI, a harakiri timeout should be logged in uWSGI's log.
I also tested that mass operations “Remove listed users” and “Remove Selected” ran successfully and without any errors.
You may want to increase the appropriate timeout and see if that helps.
Regardless of whether a mass operation process runs without errors or continues to run despite an error, I find it hard that Postorius does not display the progress of such a mass operation in order to monitor the process, as mass operations with more than 1000 members are very time-consuming. Perhaps this could be changed in a future version of Postorius?
This is a reasonable request. The best place for such a request is https://gitlab.com/mailman/postorius/-/issues/new
Requests and suggestions on the mailing list, no matter how reasonable, get lost over time.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro writes:
On 2/6/25 04:53, Markus Grandpré wrote:
Unfortunately, I have not found any error message in the log files.
If you are using uWSGI, a harakiri timeout should be logged in uWSGI's log.
I think gunicorn logs to access.log and error.log in the Mailman logs directory. I would think it would log 500 level events, that indicates server trouble.
Regardless of whether a mass operation process runs without errors or continues to run despite an error, I find it hard that Postorius does not display the progress of such a mass operation in order to monitor the process, as mass operations with more than 1000 members are very time-consuming. Perhaps this could be changed in a future version of Postorius?
This is a reasonable request. The best place for such a request is https://gitlab.com/mailman/postorius/-/issues/new
It's reasonable, but for unsubscribe Postorius invokes the mass remove REST API. So to report progress there it would require new code in Mailman and mailmanclient to report progress, or Postorius could use an explicit loop over addresses as mass subscribe does -- which would be noticably slower, I suspect.
Periodic heavy use of mass operations is a common use case at universities. It's not uncommon for each term to see thousands of list averaging scores of subscriptions or unsubscriptions, with both number of lists and average roster size spike at academic year transitions when departmental and university-wide lists turn over. We should look at speeding up mass operations. I suspect internally Mailman is looping over individual addresses, probably creating a separate RDBMS transaction for each one. It should be possible to batch these, allowing the RDBMS to optimize its own table locking. Perhaps for the university use case it would make sense to even have Mailman multiplex these transactions across multiple lists being accessed at the same time.
Steve
participants (3)
-
Mark Sapiro -
Markus Grandpré -
Stephen J. Turnbull