Restore/export existing Mailman lists + hyperkitty archives on a new server
Hello! My plan is to install fresh Mailman3 host, which means that i need to migrate/transfer my existing mailman mailing lists + hyperkitty archives to new host. Unfortunately i didnt find any good documentation about this. I tried myself to export and import mailmanweb/mailman databases (from old PostgreSQL to new PostgreSQL) and recreate indexes(mailman-web update_index). Now i can see and search hyperkitty archives, but if try to open one of the mailing list, i get error - HTTP Error 500: {"title": "500 Internal Server Error"} So im pretty sure, that I'm gonna have to transport some more stuff over, which I have no idea. Maybe existing users and user rights.
If its not a big trouble, can someone kindly share, what exactly should i need to export/import and what would be the right steps/commands/moves.
Br, Kimmo
On 7/5/22 12:32 PM, Kimmo L wrote:
Hello! My plan is to install fresh Mailman3 host, which means that i need to migrate/transfer my existing mailman mailing lists + hyperkitty archives to new host. Unfortunately i didnt find any good documentation about this. I tried myself to export and import mailmanweb/mailman databases (from old PostgreSQL to new PostgreSQL) and recreate indexes(mailman-web update_index). Now i can see and search hyperkitty archives, but if try to open one of the mailing list, i get error - HTTP Error 500: {"title": "500 Internal Server Error"} So im pretty sure, that I'm gonna have to transport some more stuff over, which I have no idea. Maybe existing users and user rights.
Dumping the databases from the old serverr and loading them on the new server should work. There should be an exception and traceback logged in Mailman core's mailman.log or possibly in the Django (mailmanweb) log. What is logged?
If its not a big trouble, can someone kindly share, what exactly should i need to export/import and what would be the right steps/commands/moves.
I thing just dumping and loading the databases and copying mailman.cfg and the Django settings should do it. Are the domains the same on the new server? If not there may be issues with that. I'd need to see the exception and traceback to say more.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
After checking the logs i found out, that i forgot to create mailman-hyperkitty.cfg file. Now it seems mailman and hyperkitty works fine. I only noticed, that this "Post volume over the past 30 days" column diagram chart does not update. Even if new mails come in. Any idea, how to fix it ?
Also i noticed, that if a member of the list tries to send an email to the same list, it will go to held messages and admin must approved. Our list setting is the following: Default action to take when a member posts to the list : Default processing Default action to take when a non-member posts to the list: Hold for moderation Could it be some kind indexing issue ? Because importing databases to the new host and enabled xapian-haystack , i rebuild indexes (mailman-web rebuild_index).
I also noticed, that after i approved that held message, it was archived, but not forwarded.
Br, Kimmo
I was able to solve the first issue - it seems that if user belongs member list and non-member list. And your non-member rule is hold for moderation, then the message go directly to the held messages.
On 8/2/22 04:15, Kimmo L wrote:
I was able to solve the first issue - it seems that if user belongs member list and non-member list. And your non-member rule is hold for moderation, then the message go directly to the held messages.
This should not be the case. The member rule should take precedence.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 8/2/22 03:07, Kimmo L wrote:
I also noticed, that after i approved that held message, it was archived, but not forwarded.
What's in Mailman core's var/logs/mailman.log and var/queue/shunt directory.
Presumably the message is shunted and there is an error and traceback in the log.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 8/1/22 22:33, Kimmo L wrote:
I only noticed, that this "Post volume over the past 30 days" column diagram chart does not update. Even if new mails come in. Any idea, how to fix it ?
If you are running the Django periodic cron jobs, I think this will
eventually clear up. I think it is the cache_cleanup job that will do it.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hey Mark! Diagram chart thing works fine. I had to give the new server time to get him running :) Also about held message - it seems to working now. At least i have run couple of new tests and no issues so far. Everything seems to work smoothly after migration from one server to another.
The only concern I've observed and I'm trying to solve right now is with memory. Not sure whats consumes the memory gradually, so that it will start consume also swap memory and eventually you can find"OSError: write error" entries in uwsgi-error.log or "django.request Internal Server Error: /hyperkitty/" in mailmanweb.log Do i need to modify my uwsgi.ini configuration maybe ? Server has 2CPU and 5GB RAM.
On 8/5/22 07:02, Kimmo L wrote:
The only concern I've observed and I'm trying to solve right now is with memory. Not sure whats consumes the memory gradually, so that it will start consume also swap memory and eventually you can find"OSError: write error" entries in uwsgi-error.log or "django.request Internal Server Error: /hyperkitty/" in mailmanweb.log Do i need to modify my uwsgi.ini configuration maybe ? Server has 2CPU and 5GB RAM.
We have seen reports of memory leaks, but no solutions. We do not see these issues on any of the production servers we manage. The server that hosts this lists has 4GB RAM and that seems fine for a server doing just Mailman.
Our servers use gunicorn, not uwsgi, so this could be a uwsgi issue.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Okay. Maybe i should to the same and replace my existing configuration with gunicorn. Is it complicated to do that change ? I ask because this gunicorn term is quite new for me.
On 8/5/22 10:13, Kimmo L wrote:
Okay. Maybe i should to the same and replace my existing configuration with gunicorn. Is it complicated to do that change ? I ask because this gunicorn term is quite new for me.
You already have gunicorn installed because Mailman core uses it to support the REST interface.
For configuration see
https://wiki.list.org/DOC/Mailman%203%20installation%20experience. In
particular, see the Recommended Configuration section items labeled
if using Gunicorn: for sample gunicorn.conf and Script to start Gunicorn.
Also, the Web Server and Running Things sections.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
So i tried to replace uwsgi with unicorn, but still i have some wierd memory leak issue. It consumes memory slowly time by time and if all the memory has been consumed, it will bite some swap memory and restart the workers and it will repeat the same until there is no memory and swap left. Any suggestions ?
On 8/14/22 14:18, Kimmo L wrote:
So i tried to replace uwsgi with unicorn, but still i have some wierd memory leak issue. It consumes memory slowly time by time and if all the memory has been consumed, it will bite some swap memory and restart the workers and it will repeat the same until there is no memory and swap left. Any suggestions ?
What Mailman process(es) are consuming the memory?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Here is the list of processes. Not sure yet, which stores the most memory in the cache. For example at the moment i have 6GB ram - 250mb free and 3gb cached.
● postgresql.service - PostgreSQL database server Main PID: 881 (postmaster) Tasks: 28 (limit: 37223) Memory: 689.2M
● mailman-qcluster.service - HyperKitty async tasks runner Main PID: 852 (django-admin) Tasks: 10 (limit: 37223) Memory: 272.0M
● gunicorn.service - GNU Mailman web interfaces Main PID: 868 (gunicorn) Tasks: 6 (limit: 37223) Memory: 575.8M
● mailman3.service - GNU Mailing List Manager Main PID: 1446 (python3) Tasks: 18 (limit: 37223) Memory: 904.2M
Kimmo L writes:
Here is the list of processes.
That's an unfamiliar format to me. Is this a Linux system running systemd?
Not sure yet, which stores the most memory in the cache. For example at the moment i have 6GB ram - 250mb free and 3gb cached.
"Cache" in the context of Linux kernel operation normally means "data file contents we keep around for possible future use, but if you need memory we will evict some stuff and give you some memory". It's not memory committed to running processes. So a couple hundred MB of memory with lots of cache is to be expected after a Linux system has been running for a while. So if you have 6GB RAM and 3GB cached, that's consistent with a machine that's running Mailman, using about 2.5 GB of RAM and not much else is running except the kernel. This report does not indicate a machine on the edge of OOM-killing processes, and it's quite consistent with Mark's experience that the whole Mailman suite runs comfortably in 4 GB.
How much swap do you have, and how much is in use?
What other daemons are running on the host besides Mailman? I guess you must have an MTA (Postfix, Exim) running and filtering software, as well as a front-end webserver (you shouldn't be running gnuicorn directly on the web). Systemd, dbus, I guess. How much memory are they using?
Have you modified any web pages associated with Mailman, including any of the Django templates? Are there any pages with unusually long load times?
I checked, and neither of the errors that you report for the uwsgi configuration ("OSError: write error" and "django.request Internal Server Error") seems to be related to a memory problem, but rather to difficulties serving certain resources, typically non-existent ones, or a user leaning on the "refresh" function in their browser. (Information from checking a few StackOverflow articles.)
By the way, Mark is going to be mostly unavailable for about 4 weeks. I'll do what I can to help, but Mark is probably a lot more expert on this than I am, so bear with me.
Steve
Hi Stephen! Thanks for trying to help me out!
That's an unfamiliar format to me. Is this a Linux system running systemd? Our mailman + hyperkitty runs on RHEL 8 and yes, all the services are systemd format.
How much swap do you have, and how much is in use? What other daemons are running on the host besides Mailman?
Host has 8GB swap at the moment ,which decreases in small steps if main memory is consumed. But if this happens, then something happens and there is again couple of hundreds free memory.
In addition to the above mentioned services, there is also NGINX and postfix services as well.
Have you modified any web pages associated with Mailman, including any of the Django templates? Are there any pages with unusually long load times? No and usually all pages are very smooth. Only if i open archives, then sometimes it will load like sec or three.
I also noticed that after midnight, if one of the mailman-web cronjob was finished, the free memory jumped from 100mb to 1gb. And then it will start to decrease again. I will try to investigate little bit more and play little bit with the services to be sure, that the issue is related with mailman or gunicorn or with something else.
Kimmo L writes:
Our mailman + hyperkitty runs on RHEL 8 and yes, all the services are systemd format. Host has 8GB swap at the moment In addition to the above mentioned services, there is also NGINX and postfix services as well.
OK. That's helpful to know.
Have you modified any web pages associated with Mailman, including any of the Django templates? Are there any pages with unusually long load times?
No and usually all pages are very smooth. Only if i open archives, then sometimes it will load like sec or three.
That's what I would expect.
I also noticed that after midnight, if one of the mailman-web cronjob was finished, the free memory jumped from 100mb to 1gb.
Search indexes are quite large, and constructing them takes a lot of concurrently active data. I'm not at all surprised that some of the cronjobs take ~1GB.
The thing about Linux is that it hates free memory. It thinks that memory should be used. If you open a data file, then close it, Linux will leave it in memory along with the metadata needed to find that memory without reading the file again. (Consider the case of a script where the first process writes a temporary file, closes it, and exits, then the second process opens the file and reads it. Big win, and this is not an uncommon case.) This is what is meant by "cache" in Linux kernel memory statistics.
And then it will start to decrease again.
Because of caching, free memory is not the right statistic to look at to check for memory leaks when you're running Linux. It's free + cache. If that total consistently decreases, you have either a leak or a process that's building a large data structure. For example, when the cronjobs go on their midnight run, I'm sure you see not only free memory decrease, but cache as well. Then when they're done, boom! you see free memory jump by ~1GB.
When a program builds a large structure in memory, it does so because it needs to write to it a lot. It's definitely volatile during the run, and normally it's variable from run to run. So there's no provision for caching it: when a program the creates a large data structure exits, that memory is returned to the free pool. In fact, how do programs cache such structures? They write them to files! So the first run doesn't use the kernel cache for that data, but the second run checks for the file, reads it, and after that it will be in the kernel's cache until some process demands the memory.
I will try to investigate little bit more and play little bit with the services to be sure, that the issue is related with mailman or gunicorn or with something else.
Write a script that prints free + cache and the top ten processes from ps, sorted by VM size and also by RSS size. Run this every 15 minutes or so, it should help catch the culprit.
Steve
Sorry for late response (i was little bit away from keyboard). I double checked the logs one more time to see, that maybe i can find something from there and i did found something:
Aug 29 08:41:27 2022 (1571056) Exception in the HyperKitty archiver: 'NoneType' object has no attribute 'strip' Aug 29 08:41:27 2022 (1571056) Traceback (most recent call last): File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman_hyperkitty/__init__.py", line 151, in _archive_message url = self._send_message(mlist, msg) File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman_hyperkitty/__init__.py", line 185, in _send_message self.name, msg['Message-Id'].strip()) AttributeError: 'NoneType' object has no attribute 'strip' Aug 29 08:41:27 2022 (1571056) HyperKitty failure on http://127.0.0.1:8000/hyperkitty/api/mailman/archive: {"error": "invalid arguments; address parts cannot contain CR or LF"} (400) Aug 29 08:41:27 2022 (1571056) Exception in the HyperKitty archiver: {"error": "invalid arguments; address parts cannot contain CR or LF"} Aug 29 08:41:27 2022 (1571056) Traceback (most recent call last): File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman_hyperkitty/__init__.py", line 151, in _archive_message url = self._send_message(mlist, msg) File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman_hyperkitty/__init__.py", line 207, in _send_message raise ValueError(result.text) ValueError: {"error": "invalid arguments; address parts cannot contain CR or LF"}
Very wierd error messages. Not sure if it will affect some how memory consuming, but i think it needs to be fixed. Any ideas ?
Kimmo L writes:
File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman_hyperkitty/__init__.py", line 207, in _send_message raise ValueError(result.text) ValueError: {"error": "invalid arguments; address parts cannot contain CR or LF"}
This is a problem with a particular message, and it means you've got a line break in the middle of an address. Not sure how it got there, possibly by using a text editor on the file.
Not sure if it will affect some how memory consuming, but i think it needs to be fixed.
No, it shouldn't affect memory consumption.
I know its out of topic, but is there possibility to somehow delete that one message, which causing this ? I see a lot of these messages with same message-ID number: hyperkitty.views.mailman Could not archive the email with message-id ..
Kimmo L writes:
I know its out of topic, but is there possibility to somehow delete that one message, which causing this ? I see a lot of these messages with same message-ID number: hyperkitty.views.mailman Could not archive the email with message-id ..
If this is process of importing an mbox into HyperKitty, you can just look at the mbox file with a mail client. Any mail client should treat it as a mail folder.
On 8/29/22 4:54 AM, Kimmo L wrote:
I know its out of topic, but is there possibility to somehow delete that one message, which causing this ? I see a lot of these messages with same message-ID number: hyperkitty.views.mailman Could not archive the email with message-id ..
Look in Mailman's var/archives/hyperkitty/spool/ directory for the
message and delete it or move it aside. You can also examine it with
mailman qfile.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Thanks guys! I got that solved by deleting that message. Unfortunately i discovered a new issue - one mailing list is not able to send email out to members. All the other mailing lists works fine after migrating from old to new server, but if i try to send a message to this one specific list and accept it under "Held messages", then the message will never reach to the members.
Log from mailman.log:
Sep 05 14:17:49 2022 (1331) Uncaught runner exception: 'list' object has no attribute 'encode' Sep 05 14:17:49 2022 (1331) Traceback (most recent call last): File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman/core/runner.py", line 176, in _one_iteration self._process_one_file(msg, msgdata) File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman/core/runner.py", line 269, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman/runners/pipeline.py", line 37, in _dispose process(mlist, msg, msgdata, pipeline) File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman/core/pipelines.py", line 50, in process handler.process(mlist, msg, msgdata) File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman/handlers/to_digest.py", line 51, in process mbox.add(msg) File "/usr/lib64/python3.9/mailbox.py", line 606, in add self._toc[self._next_key] = self._append_message(message) File "/usr/lib64/python3.9/mailbox.py", line 761, in _append_message offsets = self._install_message(message) File "/usr/lib64/python3.9/mailbox.py", line 833, in _install_message self._dump_message(message, self._file, self._mangle_from_) File "/usr/lib64/python3.9/mailbox.py", line 216, in _dump_message gen.flatten(message) File "/usr/lib64/python3.9/email/generator.py", line 116, in flatten self._write(msg) File "/usr/lib64/python3.9/email/generator.py", line 181, in _write self._dispatch(msg) File "/usr/lib64/python3.9/email/generator.py", line 218, in _dispatch meth(msg) File "/usr/lib64/python3.9/email/generator.py", line 431, in _handle_text if _has_surrogates(msg._payload) and not self.policy.cte_type=='7bit': File "/usr/lib64/python3.9/email/utils.py", line 57, in _has_surrogates s.encode() AttributeError: 'list' object has no attribute 'encode'
Sep 05 14:17:49 2022 (1331) SHUNTING: 1662376669.3194733+25b74de3b7cd8a28d0c3cf5b6cc6fc582726efec Sep 05 14:17:50 2022 (1324) Exception in the HyperKitty archiver: 'NoneType' object has no attribute 'strip' Sep 05 14:17:50 2022 (1324) Traceback (most recent call last): File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman_hyperkitty/__init__.py", line 151, in _archive_message url = self._send_message(mlist, msg) File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman_hyperkitty/__init__.py", line 185, in _send_message self.name, msg['Message-Id'].strip()) AttributeError: 'NoneType' object has no attribute 'strip'
Any suggestions ? And apologies for bothering you so much.
Kimmo L writes:
Unfortunately i discovered a new issue - one mailing list is not able to send email out to members. All the other mailing lists works fine after migrating from old to new server, but if i try to send a message to this one specific list and accept it under "Held messages", then the message will never reach to the members.
[...]
Log from mailman.log:
Sep 05 14:17:49 2022 (1331) Uncaught runner exception: 'list' object has no attribute 'encode' Sep 05 14:17:49 2022 (1331) Traceback (most recent call last): File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman/core/runner.py", line 176, in _one_iteration self._process_one_file(msg, msgdata) File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman/core/runner.py", line 269, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman/runners/pipeline.py", line 37, in _dispose process(mlist, msg, msgdata, pipeline) File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman/core/pipelines.py", line 50, in process handler.process(mlist, msg, msgdata) File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman/handlers/to_digest.py", line 51, in process mbox.add(msg) File "/usr/lib64/python3.9/mailbox.py", line 606, in add self._toc[self._next_key] = self._append_message(message) File "/usr/lib64/python3.9/mailbox.py", line 761, in _append_message offsets = self._install_message(message) File "/usr/lib64/python3.9/mailbox.py", line 833, in _install_message self._dump_message(message, self._file, self._mangle_from_) File "/usr/lib64/python3.9/mailbox.py", line 216, in _dump_message gen.flatten(message) File "/usr/lib64/python3.9/email/generator.py", line 116, in flatten self._write(msg) File "/usr/lib64/python3.9/email/generator.py", line 181, in _write self._dispatch(msg) File "/usr/lib64/python3.9/email/generator.py", line 218, in _dispatch meth(msg) File "/usr/lib64/python3.9/email/generator.py", line 431, in _handle_text if _has_surrogates(msg._payload) and not self.policy.cte_type=='7bit': File "/usr/lib64/python3.9/email/utils.py", line 57, in _has_surrogates s.encode() AttributeError: 'list' object has no attribute 'encode'
Sep 05 14:17:49 2022 (1331) SHUNTING: 1662376669.3194733+25b74de3b7cd8a28d0c3cf5b6cc6fc582726efec
I can't guess what's happening without access to at least one message which is getting shunted. You can find them in queue/shunt in Mailman's data directory. The exception occurs because that an email can be a header plus a single string payload, or it can be a list of partial payloads (eg, text + attached file), and Mailman is trying to process a list as a string. This could be caused by a problem with the message, and it could be a Mailman or Python bug. It's hard to guess which is more likely because this code is quite old, and this exception is rarely if ever encountered.
Sep 05 14:17:50 2022 (1324) Exception in the HyperKitty archiver: 'NoneType' object has no attribute 'strip' Sep 05 14:17:50 2022 (1324) Traceback (most recent call last): File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman_hyperkitty/__init__.py", line 151, in _archive_message url = self._send_message(mlist, msg) File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman_hyperkitty/__init__.py", line 185, in _send_message self.name, msg['Message-Id'].strip()) AttributeError: 'NoneType' object has no attribute 'strip'
This is saying that some message in HyperKitty has no message-id, which is a required header field in email. For that reason this email is not being normally archived. I think it's unrelated to the shunt error above (different process ID, 1324 v. 1331) because this is about the message header, not the payload.
- I did check the shunt folder and the output is like this: [----- start pickle -----] <----- start object 1 -----> Traceback (most recent call last): File "/opt/mailman/venv/bin/mailman", line 33, in <module> sys.exit(load_entry_point('mailman==3.3.5', 'console_scripts', 'mailman')()) File "/opt/mailman/venv/lib64/python3.9/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "/opt/mailman/venv/lib64/python3.9/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman/bin/mailman.py", line 69, in invoke return super().invoke(ctx) File "/opt/mailman/venv/lib64/python3.9/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/mailman/venv/lib64/python3.9/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/mailman/venv/lib64/python3.9/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman/commands/cli_qfile.py", line 74, in qfile printer.pprint(obj) File "/usr/lib64/python3.9/pprint.py", line 148, in pprint self._format(object, self._stream, 0, 0, {}, 0) File "/usr/lib64/python3.9/pprint.py", line 170, in _format rep = self._repr(object, context, level) File "/usr/lib64/python3.9/pprint.py", line 431, in _repr repr, readable, recursive = self.format(object, context.copy(), File "/usr/lib64/python3.9/pprint.py", line 444, in format return _safe_repr(object, context, maxlevels, level, self._sort_dicts) File "/usr/lib64/python3.9/pprint.py", line 596, in _safe_repr rep = repr(object) File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman/email/message.py", line 46, in __repr__ return self.__str__() File "/usr/lib64/python3.9/email/message.py", line 135, in __str__ return self.as_string() File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman/email/message.py", line 55, in as_string value = email.message.Message.as_string(self) File "/usr/lib64/python3.9/email/message.py", line 158, in as_string g.flatten(self, unixfrom=unixfrom) File "/usr/lib64/python3.9/email/generator.py", line 116, in flatten self._write(msg) File "/usr/lib64/python3.9/email/generator.py", line 181, in _write self._dispatch(msg) File "/usr/lib64/python3.9/email/generator.py", line 218, in _dispatch meth(msg) File "/usr/lib64/python3.9/email/generator.py", line 239, in _handle_text raise TypeError('string payload expected: %s' % type(payload)) TypeError: string payload expected: <class 'list'>
This is simple test email with content like "Test test test" to test out this specific mailing list, which somehow does not sent emails to the list of members. Wierd that other lists works fine, but this one is bugged. Maybe i should try to delete it and recreate it ?
- This second error seems indeed not related with that first one. "AttributeError: 'NoneType' object has no attribute 'strip'" occurs quite many times in mailman.log file. Should i check some specific list settings?
Kimmo L writes:
This is simple test email with content like "Test test test" to test out this specific mailing list, which somehow does not sent emails to the list of members. Wierd that other lists works fine, but this one is bugged. Maybe i should try to delete it and recreate it ?
That might work.
- This second error seems indeed not related with that first one. "AttributeError: 'NoneType' object has no attribute 'strip'" occurs quite many times in mailman.log file. Should i check some specific list settings?
I don't think so. That usually indicates mal-formed mail, missing a required header field. In the case you posted earlier, the mail had no Message-ID that Mailman could find. I suspect you have a configuration where your MUA doesn't add the Message-ID header (this is common, because all internet-facing MTAs do add one if there isn't one already in the message, so many MUAs let the MTA do it), and for some reason local mail doesn't go through a proper MTA and never does get one.
Recreating the faulty list didnt solve my issue. Somehow the new one with same name still gives the same error logs and does not send email out.
I was able to found the cause of the issue. Under list settings > Alter Messages there is a option called "anonymous list". It seems that if i set this to YES, im not able to send email out. If i set this to NO - everything works fine. Is there some other option, that might affect anonymous option effect ?
Kimmo L writes:
I was able to found the cause of the issue. Under list settings > Alter Messages there is a option called "anonymous list". It seems that if i set this to YES, im not able to send email out. If i set this to NO - everything works fine. Is there some other option, that might affect anonymous option effect ?
No, it has to be something about the message, because looks like Mailman is throwing exceptions when it tries to turn the anonymized email.Message Python object into an RFC 822 message it can send to the MTA.
Can you send a test message to the list, also CC: me at "stephen@xemacs.org" (do not copy-paste my Gmail address, I will never see it, that's where I sign up for vendor spam and read it once a month)? Also send me the pickle (.pck object from the shunt queue).
Steve
I sent you an email with with the shunt pck file and also i did send one test email by adding you under CC.
Hi, Kimmo!
I don't know how to explain this, but the message you sent me as CC has a Content-Type header field:
Content-Type: multipart/alternative; boundary="_000_a352a6421f0240e0bc795ac1c4747d3celisafi_"
which is correct, the content is provided as both text/plain and text/html in separate parts with that boundary delimiting the parts. However in the shunted message object the Content-Type field isn't present although somehow it recognizes the message body as multipart, and content-type as text/plain.
I don't understand how this can happen. I've checked all the places where Mailman manipulates headers destructively, and all of those that might touch Content-Type immediately replace it.
On the other hand the absence of a Content-Type field but the content type is recognized a text/plain is consistent with there being no Content-Type field in the message header when Mailman receives it. I don't know how that can happen, either. What mail client and MTA are you using?
Regards, Steve
Im using Outlook and MTA is Postfix. Postfix has the same configuration that i used in old server and wierd is that there is no issue with other lists. But ofc other lists dont have this "Anonymous list" option on.
Kimmo L writes:
Im using Outlook and MTA is Postfix. Postfix has the same configuration that i used in old server and
Outlook has a looong history of nonconformance to standards. Postfix, on the other hand was designed on the principle of security through standard conformance (as well as a certain degree of paranoia ;-). But Outlook is very common, so <shrug emoji>.
wierd is that there is no issue with other lists. But ofc other lists dont have this "Anonymous list" option on.
Ah, thanks for the reminder! I forgot that detail, but it is likely to really localize the problem.
Will let you know if I find anything. Mark will be back soon, too.
Steve
Big thanks Steve! Let me know, if there is anything else i can check or share. I will try to do some testing myself as well, but im out of ideas as well. I even tried the same thing with other lists and the same issue occurs. (Switched anonymous list to ON)
Br, Kimmo
On 9/10/22 22:22, Stephen J. Turnbull wrote:
Will let you know if I find anything. Mark will be back soon, too.
Yes, I'm back and am looking at this. I'm looking at the traceback at https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/... and I would like to get a copy of the shunted 1662376669.3194733+25b74de3b7cd8a28d0c3cf5b6cc6fc582726efec.pck file.
If that is not available, any of the shunted files from the same
Uncaught runner exception: 'list' object has no attribute 'encode'
will do.
Also, the same post contains a traceback from mailman_hyperkitty indicating the message has no Message-ID header. This is possibly related to the fact this is an anonymous list. For anonymous lists, the original Message-ID is removed because it can reveal the sender's domain, but it should always be replaced by a generated one. See https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/handlers/cleans...
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi Mark!
I forwarded the same shunted file to you as well, which i sent to Steve.
About that no "Message-ID" error - Basically it still occurs in mailman.log file even if some other list has been used. And our other lists does not use this "anonymous list" option.
On 9/15/22 01:22, Kimmo L wrote:
Hi Mark!
I forwarded the same shunted file to you as well, which i sent to Steve.
Thanks. I got it and will look further.
About that no "Message-ID" error - Basically it still occurs in mailman.log file even if some other list has been used. And our other lists does not use this "anonymous list" option.
Those errors should result in the affected messages being saved as *.pck
files in Mailman's var/archives/hyperkitty/spool/ directory. Possibly
there is only one such 'bad message' which is retried each time there is
a new post resulting in the same error. How many such files are there?
If only one, please send me that one too. Also, these can be examined
with mailman qfile.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 9/15/22 10:20, Mark Sapiro wrote:
On 9/15/22 01:22, Kimmo L wrote:
Hi Mark!
I forwarded the same shunted file to you as well, which i sent to Steve.
Thanks. I got it and will look further.
This is very strange. The message object in the shunted pickle is missing several important headers. In particular, there is no Message-ID: and no MIME-Version: and no Content-Type: while the body contains subpart headers like
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
and
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
but no boundaries.
This apparently confuses the Python email package so that the get_payload() method returns a list of strings rather than a list of message objects.
The question is how did this happen.
What does
mailman conf -k anonymous_list_keep_headers
show, and what is the content of mailman/handlers/cleanse.py?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro writes:
This apparently confuses the Python email package so that the get_payload() method returns a list of strings rather than a list of message objects.
Yes, I got that far too. I'm guessing that in the absence of a Content-Type, which then *should* default to text/plain, the email package goes ahead and detects the MIME headers anyway (although I don't think it should do that) and breaks the message into MIME parts. I would think we'd raise an Exception here.
If there's a reason for the email package to do that detection (*shrug*, it seems to work), then the dispatching in _handle_text should be done on the Python type rather than the MIME Content-Type.
I forget exactly how the code is structured, the relevant dispatching may be in _dispatch or in _dispatch_meth.
Steve
Oops, premature send.
I did grep through the whole mailman core code base for "del", and didn't come up with anything that looks like it would delete fields like Content-Type and Message-ID (assuming the default for anonymous_list_keep_headers, I wish I had thought to ask about that, but who changes it?) I assumed Content-Type is modified by direct assignment in handlers that delete parts or transform them to a different MIME type.
I wonder if his Python is old enough to not recognize negated regexps? That should be logged in errors. Or (as you guessed), the '(?!x-)' regexp (or the whole content of anonymous_list_keep_headers) has been removed?
Stephen J. Turnbull writes:
Mark Sapiro writes:
This apparently confuses the Python email package so that the get_payload() method returns a list of strings rather than a list of message objects.
Yes, I got that far too. I'm guessing that in the absence of a Content-Type, which then *should* default to text/plain, the email package goes ahead and detects the MIME headers anyway (although I don't think it should do that) and breaks the message into MIME parts. I would think we'd raise an Exception here.
If there's a reason for the email package to do that detection (*shrug*, it seems to work), then the dispatching in _handle_text should be done on the Python type rather than the MIME Content-Type.
I forget exactly how the code is structured, the relevant dispatching may be in _dispatch or in _dispatch_meth.
Steve
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
On 9/15/22 12:09, Mark Sapiro wrote:
This apparently confuses the Python email package so that the get_payload() method returns a list of strings rather than a list of message objects.
Actually, that's not correct. The get_payload() method is returning a list of message objects. It only appears to be a list of strings because this is a mailman.email.message.Message object whose __repr__() method returns the content as a string.
The real issue is the missing Content-Type: header which implies the content is text/plain.
What I think is happening is the original message is parsed into a multipart/alternative Message object with payload of the text/plain and text/html subparts. Then something deletes the top level Content-Type: header which makes the message appear to be text/plain, yet get_payload() still returns the list of two subparts.
So the real issue is what's removing all those headers from the anonymized message. I think there must be some issue with the configured anonymous_list_keep_headers or the code in mailman/handlers/cleanse.py is somehow corrupted.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 9/15/22 20:39, Mark Sapiro wrote:
What I think is happening is the original message is parsed into a multipart/alternative Message object with payload of the text/plain and text/html subparts. Then something deletes the top level Content-Type: header which makes the message appear to be text/plain, yet get_payload() still returns the list of two subparts.
As an experiment, I did the following
$ /opt/mailman/mm/bin/mailman shell
Welcome to the GNU Mailman shell
Use commit() to commit changes.
Use abort() to discard changes since the last commit.
Exit with ctrl+D does an implicit commit() but exit() does not.
>>> import pickle
>>> with open('path/to/shunted/pickle', 'rb') as fp:
... msg = pickle.load(fp)
... msgdata = pickle.load(fp)
...
>>> msg
Traceback (most recent call last):
<snip>
File "/usr/local/lib/python3.9/email/generator.py", line 239, in
_handle_text
raise TypeError('string payload expected: %s' % type(payload))
TypeError: string payload expected: <class 'list'>
>>> msg['MIME-Version'] = '1.0'
>>> msg['Content-Type'] = 'multipart/alternative; boundary="aaaabbbb"'
>>> msg
<displays the message as a string with no exception>
>>>
I.e., restoring the missing headers allows the message to be properly flattened. I don't think we can blame email.generator:_handle_text for this as we have corrupted the message object by removing the Content-Type: header.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
What does mailman conf -k anonymous_list_keep_headers show, and what is the content of mailman/handlers/cleanse.py? Command output: [mailman] anonymous_list_keep_headers:
And content of mailman/handlers/cleanse.py:
# Copyright (C) 1998-2021 by the Free Software Foundation, Inc. # # This file is part of GNU Mailman. # # GNU Mailman is free software: you can redistribute it and/or modify it under # the terms of the GNU General Public License as published by the Free # Software Foundation, either version 3 of the License, or (at your option) # any later version. # # GNU Mailman is distributed in the hope that it will be useful, but WITHOUT # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for # more details. # # You should have received a copy of the GNU General Public License along with # GNU Mailman. If not, see <https://www.gnu.org/licenses/>.
"""Cleanse certain headers from all messages."""
import re import logging
from email.utils import formataddr, make_msgid from mailman.config import config from mailman.core.i18n import _ from mailman.handlers.cook_headers import uheader from mailman.interfaces.handler import IHandler from public import public from zope.interface import implementer
log = logging.getLogger('mailman.smtp') elog = logging.getLogger('mailman.error')
@public @implementer(IHandler) class Cleanse: """Cleanse certain headers from all messages."""
name = 'cleanse'
description = _('Cleanse certain headers from all messages.')
def remove_nonkeepers(self, msg):
cres = []
for regexp in config.mailman.anonymous_list_keep_headers.split():
try:
if regexp.endswith(':'):
regexp = regexp[:-1] + '$'
cres.append(re.compile(regexp, re.IGNORECASE))
except re.error as e:
elog.error(
'Ignored bad anonymous_list_keep_headers regexp %s: %s',
regexp, e)
for hdr in msg.keys():
keep = False
for cre in cres:
if cre.search(hdr):
keep = True
break
if not keep:
del msg[hdr]
def process(self, mlist, msg, msgdata):
"""See `IHandler`."""
# Remove headers that could contain passwords.
del msg['approved']
del msg['approve']
del msg['x-approved']
del msg['x-approve']
del msg['urgent']
# We remove other headers from anonymous lists.
if mlist.anonymous_list:
log.info('post to %s from %s anonymized',
mlist.fqdn_listname, msg.get('from'))
del msg['from']
del msg['reply-to']
del msg['sender']
del msg['organization']
del msg['return-path']
# Hotmail sets this one
del msg['x-originating-email']
# And these can reveal the sender too
del msg['received']
# And so can the message-id so replace it.
del msg['message-id']
msg['Message-ID'] = make_msgid()
# And something sets these
del msg['x-mailfrom']
del msg['x-envelope-from']
# And now remove all but the keepers.
self.remove_nonkeepers(msg)
i18ndesc = str(uheader(mlist, mlist.description, 'From'))
msg['From'] = formataddr((i18ndesc, mlist.posting_address))
msg['Reply-To'] = mlist.posting_address
# Some headers can be used to fish for membership.
del msg['return-receipt-to']
del msg['disposition-notification-to']
del msg['x-confirm-reading-to']
# Pegasus mail uses this one... sigh.
del msg['x-pmrqc']
# Don't let this header be spoofed. See RFC 5064.
del msg['archived-at']Kimmo L writes:
What does mailman conf -k anonymous_list_keep_headers show, and what is the content of mailman/handlers/cleanse.py?
Command output: [mailman] anonymous_list_keep_headers:
Mark may have more suggestions, but this is almost certainly the cause of the problem. The default for this parameter is
anonymous_list_keep_headers: ^(?!x-) ^x-mailman- ^x-content-filtered-by: ^x-topics: ^x-ack: ^x-beenthere: ^x-list-administrivia: ^x-spam-
which is a list of strings separated by spaces, and I believe the leading space on the second line is significant (meaning a continuation of the list on the first line).
There are a number of possibilities that cause this. (1) Check whether anonymous_list_keep_headers is set in mailman.cfg. If it is, it may be all you need to do is remove a spurious newline after the keyword anonymous_list_keep_headers. Otherwise, you can restore the default by copying the setting above. (2) If not, schema.cfg is corrupt or missing, and the default setting for anonymous_list_keep_headers is empty. In that case you should get a pristine copy of schema.cfg to compare to and check for any other undesired changes.
If you actually intended to change the value of this parameter, (1) you must keep the ^(?!x-) because this protects all the standard fields like To, From, Content-Type and Message-ID that are essential for any mail system; (2) you should keep the following: ^x-mailman- ^x-topics: ^x-ack: ^x-beenthere: ^x-list-administrivia: because they are used internally by Mailman, and do not contain any identifying information; (3) you probably should keep ^x-content-filtered-by: because it is used by Mailman, and should not contain identifying information; and (4) you may want to remove x-spam- because they occasionally identify MTAs upstream of you, but this degree of anonymization is rarely needed.
And content of mailman/handlers/cleanse.py:
This appears to be unchanged from the distribution, as expected.
Mark may have more suggestions, but this is almost certainly the cause of the problem. The default for this parameter is anonymous_list_keep_headers: ^(?!x-) ^x-mailman- ^x-content-filtered-by: ^x-topics: ^x-ack: ^x-beenthere: ^x-list-administrivia: ^x-spam- which is a list of strings separated by spaces, and I believe the leading space on the second line is significant (meaning a continuation of the list on the first line). There are a number of possibilities that cause this. (1) Check whether anonymous_list_keep_headers is set in mailman.cfg. If it is, it may be all you need to do is remove a spurious newline after the keyword anonymous_list_keep_headers. Otherwise, you can restore the default by copying the setting above.
Thanks to this i found the cause of concern - my mailman.cfg had "empty anonymous_list_keep_headers:" option defined. I commented out and now test email went through by using anonymous option successfully. One small thing, but so much trouble. My bad that i didint notice it before. Big thanks again Steve and Mark!
Kimmo L writes:
Thanks to this i found the cause of concern - my mailman.cfg had "empty anonymous_list_keep_headers:" option defined. I commented out and now test email went through by using anonymous option successfully. One small thing, but so much trouble.
Great!
My bad that i didint notice it before.
Please, don't say that. It's very difficult to judge what a user should notice in many cases. Especially in a case like this, where in fact much of the processing is hard-coded, you might assume that not only do we hard-code the fields to delete but we also hard code the RFC822-required fields. And we probably should: https://gitlab.com/mailman/mailman/-/issues/1031
Steve
Those errors should result in the affected messages being saved as *.pck files in Mailman's var/archives/hyperkitty/spool/ directory. Possibly there is only one such 'bad message' which is retried each time there is a new post resulting in the same error. How many such files are there? If only one, please send me that one too. Also, these can be examined with mailman qfile. There was 3 file. I used the mailman qfile command and the output was this: [----- start pickle -----] <----- start object 1 -----> Traceback (most recent call last): File "/opt/mailman/venv/bin/mailman", line 33, in <module> sys.exit(load_entry_point('mailman==3.3.5', 'console_scripts', 'mailman')()) File "/opt/mailman/venv/lib64/python3.9/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "/opt/mailman/venv/lib64/python3.9/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman/bin/mailman.py", line 69, in invoke return super().invoke(ctx) File "/opt/mailman/venv/lib64/python3.9/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/mailman/venv/lib64/python3.9/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/mailman/venv/lib64/python3.9/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman/commands/cli_qfile.py", line 74, in qfile printer.pprint(obj) File "/usr/lib64/python3.9/pprint.py", line 148, in pprint self._format(object, self._stream, 0, 0, {}, 0) File "/usr/lib64/python3.9/pprint.py", line 170, in _format rep = self._repr(object, context, level) File "/usr/lib64/python3.9/pprint.py", line 431, in _repr repr, readable, recursive = self.format(object, context.copy(), File "/usr/lib64/python3.9/pprint.py", line 444, in format return _safe_repr(object, context, maxlevels, level, self._sort_dicts) File "/usr/lib64/python3.9/pprint.py", line 596, in _safe_repr rep = repr(object) File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman/email/message.py", line 46, in __repr__ return self.__str__() File "/usr/lib64/python3.9/email/message.py", line 135, in __str__ return self.as_string() File "/opt/mailman/venv/lib64/python3.9/site-packages/mailman/email/message.py", line 55, in as_string value = email.message.Message.as_string(self) File "/usr/lib64/python3.9/email/message.py", line 158, in as_string g.flatten(self, unixfrom=unixfrom) File "/usr/lib64/python3.9/email/generator.py", line 116, in flatten self._write(msg) File "/usr/lib64/python3.9/email/generator.py", line 181, in _write self._dispatch(msg) File "/usr/lib64/python3.9/email/generator.py", line 218, in _dispatch meth(msg) File "/usr/lib64/python3.9/email/generator.py", line 239, in _handle_text raise TypeError('string payload expected: %s' % type(payload)) TypeError: string payload expected: <class 'list'>
After i cleaned up /var/archives/hyperkitty/spool folder, mailman.log file is much cleaner and i dont see any new no "Message-ID" error at the moment.
On 9/15/22 23:10, Kimmo L wrote:
Those errors should result in the affected messages being saved as *.pck files in Mailman's var/archives/hyperkitty/spool/ directory. Possibly there is only one such 'bad message' which is retried each time there is a new post resulting in the same error. How many such files are there? If only one, please send me that one too. Also, these can be examined with mailman qfile. There was 3 file. I used the mailman qfile command and the output was this: [----- start pickle -----] <----- start object 1 -----> Traceback (most recent call last): ...> File "/usr/lib64/python3.9/email/generator.py", line 239, in _handle_text raise TypeError('string payload expected: %s' % type(payload)) TypeError: string payload expected: <class 'list'>
These again are anonymized posts that had important headers removed because of the incorrect anonymous_list_keep_headers setting.
After i cleaned up /var/archives/hyperkitty/spool folder, mailman.log file is much cleaner and i dont see any new no "Message-ID" error at the moment.
Yes. Now that you have the default anonymous_list_keep_headers setting things should be OK going forward.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro wrote: ...
I thing just dumping and loading the databases and copying mailman.cfg and the Django settings should do it. Are the domains the same on the new server? If not there may be issues with that. I'd need to see the exception and traceback to say more.
I am migrating from mailman 3.2.1 & mailman3-web 0+20180916-8 on Debian Buster to mailman 3.3.8 & mailman3-web 0+20200530-2.1 on Debian bookworm, have mailman3 and mailman3-web configured (and running on bookworm) from the initial install. But when I use the mailman3-web 0+20180916-8 database (after stopping mailman3 and mailman3-web) on bookworm, unfortunately migrate fails:
# mailman-web showmigrations account [X] 0001_initial [X] 0002_email_max_length admin [X] 0001_initial [X] 0002_logentry_remove_auto_add [X] 0003_logentry_add_action_flag_choices auth [X] 0001_initial [X] 0002_alter_permission_name_max_length [X] 0003_alter_user_email_max_length [X] 0004_alter_user_username_opts [X] 0005_alter_user_last_login_null [X] 0006_require_contenttypes_0002 [X] 0007_alter_validators_add_error_messages [X] 0008_alter_user_username_max_length [X] 0009_alter_user_last_name_max_length [X] 0010_alter_group_name_max_length [X] 0011_update_proxy_permissions [X] 0012_alter_user_first_name_max_length contenttypes [X] 0001_initial [X] 0002_remove_content_type_name ...
# mailman-web migrate --plan Planned operations: contenttypes.0001_initial Create model ContentType Alter unique_together for contenttype (1 constraint(s)) auth.0001_initial Create model Permission Create model Group Create model User account.0001_initial Create model EmailAddress Create model EmailConfirmation account.0002_email_max_length Alter field email on emailaddress admin.0001_initial Create model LogEntry admin.0002_logentry_remove_auto_add Alter field action_time on logentry admin.0003_logentry_add_action_flag_choices Alter field action_flag on logentry contenttypes.0002_remove_content_type_name Change Meta options on contenttype Alter field name on contenttype Raw Python operation Remove field name from contenttype ...
But the migrate fails at the first operation: # mailman-web migrate Operations to perform: Apply all migrations: account, admin, auth, contenttypes, django_mailman3, django_q, hyperkitty, postorius, sessions, sites, socialaccount Running migrations: Applying contenttypes.0001_initial...Traceback (most recent call last): File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 82, in _execute return self.cursor.execute(sql) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/django/db/backends/mysql/base.py", line 73, in execute return self.cursor.execute(query, args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 209, in execute res = self._query(query) ^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 315, in _query db.query(q) File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 239, in query _mysql.connection.query(self, query) MySQLdb._exceptions.OperationalError: (1050, "Table 'django_content_type' already exists") ...
Indeed the table django_content_type already exists on 0+20180916-8 . Apart from fields changes, there are an additional 4 tables in mailman3-web 0+20200530-2.1
I managed to import the mailman 3.2.1 database to the mailman 3.3.8 database, and all lists, subscriptions and settings looks fine.
So how can I upgrade my existing mailman3-web database?
On 9/14/23 2:09 AM, Alex Schuilenburg via Mailman-users wrote:
I am migrating from mailman 3.2.1 & mailman3-web 0+20180916-8 on Debian Buster to mailman 3.3.8 & mailman3-web 0+20200530-2.1 on Debian bookworm, have mailman3 and mailman3-web configured (and running on bookworm) from the initial install. But when I use the mailman3-web 0+20180916-8 database (after stopping mailman3 and mailman3-web) on bookworm, unfortunately migrate fails:
...
But the migrate fails at the first operation: # mailman-web migrate Operations to perform: Apply all migrations: account, admin, auth, contenttypes, django_mailman3, django_q, hyperkitty, postorius, sessions, sites, socialaccount Running migrations: Applying contenttypes.0001_initial...Traceback (most recent call last): ... MySQLdb._exceptions.OperationalError: (1050, "Table 'django_content_type' already exists")
You are attempting to apply migrations that have already been applied. Apparently the django_migrations table is inconsistent with the state of the database.
Indeed the table django_content_type already exists on 0+20180916-8 . Apart from fields changes, there are an additional 4 tables in mailman3-web 0+20200530-2.1
I managed to import the mailman 3.2.1 database to the mailman 3.3.8 database, and all lists, subscriptions and settings looks fine.
So how can I upgrade my existing mailman3-web database?
This appears to be a Debian package issue. I suggest you ask Debian.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Teilnehmer (4)
-
alexs＠ecoscentric.com -
Kimmo L -
Mark Sapiro -
Stephen J. Turnbull