Mailman REST API not available. Please start Mailman core.
Okay, so I got the webservice of Mailman3 up and running, but when I go to mailman3/postorius/lists/, I get the following error:
Something went wrong Mailman REST API not available. Please start Mailman core.
When I logged in to the admin user, it threw a server 500 error, but I could refresh and see myself logged in. If I try to go to lists or mailman settings, however, I get the error above.
joseverde@gmail.com writes:
Okay, so I got the webservice of Mailman3 up and running,
It would help to know how you did that. Did you wipe the server and start over? Did you just purge Mailman3 and its configuration files? Something else?
but when I go to mailman3/postorius/lists/, I get the following error:
Something went wrong Mailman REST API not available. Please start Mailman core.
When I logged in to the admin user, it threw a server 500 error, but I could refresh and see myself logged in.
The admin user is a Django concept. There isn't really one for Mailman core; it assumes you have direct access to the REST API. If you do have access to the REST API, you're the admin. So this doesn't prove anything about what core is doing or if it's running.
If I try to go to lists or mailman settings, however, I get the error above.
The webservice is two front ends, Postorius for administration and HyperKitty for list traffic archives. You mention only Postorius. Is HyperKitty running?
The problem here could be several different things. Mailman core, Postorius, and HyperKitty are separate processes, communicating over TCP sockets. If Postorius says core isn't running, that just means it was unable to open a socket to connect to core. Core might be running but listening on a different port than the one Postorius expects, or it might not be running, and there are probably arcane reasons it could fail even if it's running and they have the same port configured.
You can can check whether Mailman core is running in several ways, depending on what Ubuntu 18.04 uses to control daemons. If it's systemd, then "systemctl status mailman" (or maybe "... mailman3") should do the trick.
If core is not running, you need to start it, and figure out how to get that to happen automatically since it didn't happen this time.
If core is running, most likely either the webservice(s) or the core are misconfigured as to the port where core is providing the REST API. But we'll cross that bridge when we get to it, as there are a number of configuration files involved.
It would help to know how you did that. Did you wipe the server and start over? Did you just purge Mailman3 and its configuration files? Something else?
It's a bit hard to say, my brain got mushy after a few hours of staring at bash. What was happening was the install was never completing, there was always an error blocking it. One of the main ones I found was pymysql for python2 wasn't installed. Once I installed that, it started to be happy. I also had to run dpkg-reconfigure for mailman3-web and after that I had a working web service.
As far as Hyper Kitty, the web service is up (the webpage runs, at least).
And I did check systemctl status mailman3, it was inactive. So I did systemctl start mailman3 and it is now running. However, the same error still shows up on postorius.
Something went wrong Mailman REST API not available. Please start Mailman core.
On 7/26/20 12:00 AM, joseverde@gmail.com wrote:
Okay, so I got the webservice of Mailman3 up and running, but when I go to mailman3/postorius/lists/, I get the following error:
Something went wrong Mailman REST API not available. Please start Mailman core.
Do your settings in the webservice section of mailman.cfg for things like:
admin_pass: restpass admin_user: restadmin hostname: localhost port: 8001 (Defaults shoun)
match your Django settings
MAILMAN_REST_API_USER = 'restadmin' MAILMAN_REST_API_PASS = 'restpass' MAILMAN_REST_API_URL = 'http://localhost:8001'
Also, is Mailman's rest runner running
ps -fwwA|grep runner=rest
should show 3 processes. The runner and 2 workers.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
The settings match the Django settings, yes. As far as the rest runner, the following shows on that command:
root 25868 25475 0 21:50 pts/0 00:00:00 grep --color=auto runner=rest
Just curious, I did some searching for the REST server and ran these commands from the documentation:
from mailman.testing import helpers master = helpers.TestableMaster(helpers.wait_for_webservice) master.start('rest')
It returned the following error:
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3/dist-packages/mailman/testing/helpers.py", line 169, in start self.start_runners(runners) File "/usr/lib/python3/dist-packages/mailman/bin/master.py", line 405, in start_runners runner_config = getattr(config, section_name) File "/usr/lib/python3/dist-packages/mailman/config/config.py", line 88, in __getattr__ return getattr(self._config, name) AttributeError: 'NoneType' object has no attribute 'runner.rest'
Also, looking closely at DJANGO, I noticed this was set to MAILMAN_REST_API_URL = 'http://localhost:8001'
but in the mailman info command it is listed as http://localhost:8001/3.1 . I tried changing the URL to that and no change. Not sure if I need to run some command for it to become effective.
And one last infopoint for tonight, I checked on a whim to see if mailman3 was still running, it wasn't since the last time I checked. (the command mailman status, on the other hand, shows it is up and running. The following error was shown when starting to start:
Jul 27 22:30:40 lamp-s-1vcpu-1gb-sfo2-01 mailman3[27388]: File "/usr/lib/python3/dist-packages/mailman/commands/cli_control.py", line 88, in process Jul 27 22:30:40 lamp-s-1vcpu-1gb-sfo2-01 mailman3[27388]: status, lock = master_state() Jul 27 22:30:40 lamp-s-1vcpu-1gb-sfo2-01 mailman3[27388]: File "/usr/lib/python3/dist-packages/mailman/bin/master.py", line 133, in master_state Jul 27 22:30:40 lamp-s-1vcpu-1gb-sfo2-01 mailman3[27388]: hostname, pid, tempfile = lock.details Jul 27 22:30:40 lamp-s-1vcpu-1gb-sfo2-01 mailman3[27388]: File "/usr/lib/python3/dist-packages/flufl/lock/_lockfile.py", line 145, in details Jul 27 22:30:40 lamp-s-1vcpu-1gb-sfo2-01 mailman3[27388]: with open(self._lockfile) as fp: Jul 27 22:30:40 lamp-s-1vcpu-1gb-sfo2-01 mailman3[27388]: PermissionError: [Errno 13] Permission denied: '/var/lib/mailman3/locks/master.lck' Jul 27 22:30:40 lamp-s-1vcpu-1gb-sfo2-01 systemd[1]: mailman3.service: Control process exited, code=exited status=1 Jul 27 22:30:40 lamp-s-1vcpu-1gb-sfo2-01 systemd[1]: mailman3.service: Failed with result 'exit-code'. Jul 27 22:30:40 lamp-s-1vcpu-1gb-sfo2-01 systemd[1]: Failed to start Mailman3 server.
Efforts to restart give this error:
Job for mailman3.service failed because the control process exited with error code.
On 7/27/20 9:52 PM, joseverde@gmail.com wrote:
The settings match the Django settings, yes. As far as the rest runner, the following shows on that command:
root 25868 25475 0 21:50 pts/0 00:00:00 grep --color=auto runner=rest
So the rest runner is not running as all you see is your grep command.
On 7/27/20 10:34 PM, joseverde@gmail.com wrote:
On 7/27/20 10:34 PM, joseverde@gmail.com wrote:
And one last infopoint for tonight, I checked on a whim to see if mailman3 was still running, it wasn't since the last time I checked. (the command mailman status, on the other hand, shows it is up and running. The following error was shown when starting to start:
Jul 27 22:30:40 lamp-s-1vcpu-1gb-sfo2-01 mailman3[27388]: File "/usr/lib/python3/dist-packages/mailman/commands/cli_control.py", line 88, in process Jul 27 22:30:40 lamp-s-1vcpu-1gb-sfo2-01 mailman3[27388]: status, lock = master_state() Jul 27 22:30:40 lamp-s-1vcpu-1gb-sfo2-01 mailman3[27388]: File "/usr/lib/python3/dist-packages/mailman/bin/master.py", line 133, in master_state Jul 27 22:30:40 lamp-s-1vcpu-1gb-sfo2-01 mailman3[27388]: hostname, pid, tempfile = lock.details Jul 27 22:30:40 lamp-s-1vcpu-1gb-sfo2-01 mailman3[27388]: File "/usr/lib/python3/dist-packages/flufl/lock/_lockfile.py", line 145, in details Jul 27 22:30:40 lamp-s-1vcpu-1gb-sfo2-01 mailman3[27388]: with open(self._lockfile) as fp: Jul 27 22:30:40 lamp-s-1vcpu-1gb-sfo2-01 mailman3[27388]: PermissionError: [Errno 13] Permission denied: '/var/lib/mailman3/locks/master.lck' Jul 27 22:30:40 lamp-s-1vcpu-1gb-sfo2-01 systemd[1]: mailman3.service: Control process exited, code=exited status=1 Jul 27 22:30:40 lamp-s-1vcpu-1gb-sfo2-01 systemd[1]: mailman3.service: Failed with result 'exit-code'. Jul 27 22:30:40 lamp-s-1vcpu-1gb-sfo2-01 systemd[1]: Failed to start Mailman3 server.
It appears there is a permissions issue in trying to read the locks.
Does your systemd script set the user to the Mailman user, and does the Mailman user have read/write permission on Mailman's var/locks/?
Also, is any of Mailman core running? Try
ps -fwwA|egrep 'master|runner'
If a master is running, try
service mailman stop
If that doesn't stop it, send it a sigterm or, only if necessary, a sigkill.
Then if there are an runners left, sigterm them.
Finally if there's anything in var/locks, remove it.
Then try to start Mailman.
Note that if you've previously started Mailman as root, there may be
locks and or other files owned by root that are now inaccessible to
mailman. You may need to chown -R mailman in Mailman's var directory.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
[Does your systemd script set the user to the Mailman user, and does the Mailman user have read/write permission on Mailman's var/locks/?]
How do I check for that?
Also, what's the difference between the mailman command and the systemctl mailman3? Are they the same processes?
Oh and does the /3.1 matter at the end of the Rest url for Django?
On 7/28/20 1:07 PM, joseverde@gmail.com wrote:
[Does your systemd script set the user to the Mailman user, and does the Mailman user have read/write permission on Mailman's var/locks/?]
How do I check for that?
ls -la /path/to/mailman/var/locks
Also, what's the difference between the mailman command and the systemctl mailman3? Are they the same processes?
Presumably your mailman3.service actually uses the mailman start
command to start Mailman. The mailman command is a shell with many
subcommands for various purposes including starting and stopping Mailman.
Oh and does the /3.1 matter at the end of the Rest url for Django?
Do not put the 3.1 at the end of your MAILMAN_REST_API_URL. Postorius adds it. See <https://gitlab.com/mailman/postorius/-/blob/master/src/postorius/utils.py#L5...>.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Okay, I *think* I killed the processes. When I run
ps -fwwA|egrep 'master|runner'
I just get
root 7595 1 0 Jul26 ? 00:00:04 /usr/lib/postfix/sbin/master -w mailer 21851 21001 0 22:45 pts/1 00:00:00 grep -E --color=auto master|runner
As far as the permissions, you asked does your systemd script set the user to the Mailman user? I don't know how to check that, but I saw many files in /var/lib/mailman3 were owned by a user named list. Last time I tried to run mailman status, it threw a permission error on mailman.cfg, which was owned by root. I changed owner to list, and it produced a different error. So I'm guessing we are on the right path? There was no mention of a user named mailman in the system before, but maybe I need to check something.
Lastly, I deleted everything in /var/lib/mailman3/locks.
However, running mailman status now throws this permission error
File "/usr/bin/mailman", line 11, in <module> load_entry_point('mailman==3.1.1', 'console_scripts', 'mailman')() File "/usr/lib/python3/dist-packages/mailman/bin/mailman.py", line 94, in main initialize(config_path) File "/usr/lib/python3/dist-packages/mailman/core/initialize.py", line 188, in initialize initialize_1(config_path) File "/usr/lib/python3/dist-packages/mailman/core/initialize.py", line 123, in initialize_1 mailman.config.config.load(config_path) File "/usr/lib/python3/dist-packages/mailman/config/config.py", line 107, in load self.push(filename, user_config.read()) File "/usr/lib/python3/dist-packages/mailman/config/config.py", line 113, in push self._post_process() File "/usr/lib/python3/dist-packages/mailman/config/config.py", line 125, in _post_process self.ensure_directories_exist() File "/usr/lib/python3/dist-packages/mailman/config/config.py", line 227, in ensure_directories_exist with Lock(lock_file): File "/usr/lib/python3/dist-packages/flufl/lock/_lockfile.py", line 334, in __enter__ self.lock() File "/usr/lib/python3/dist-packages/flufl/lock/_lockfile.py", line 203, in lock self._write() File "/usr/lib/python3/dist-packages/flufl/lock/_lockfile.py", line 417, in _write with open(self._claimfile, 'w') as fp: PermissionError: [Errno 13] Permission denied: '/var/lib/mailman3/locks/mailman-cfg.lck|[redacted server name]|21787|3720843219351355903'
That file was removed when you told me to clear the locks folder (I save a copy, however). So I'm stuck again. Sorry for all the questions and thanks very much for the help.
On 7/28/20 10:51 PM, joseverde@gmail.com wrote:
Okay, I *think* I killed the processes. When I run
ps -fwwA|egrep 'master|runner'
I just get
root 7595 1 0 Jul26 ? 00:00:04 /usr/lib/postfix/sbin/master -w mailer 21851 21001 0 22:45 pts/1 00:00:00 grep -E --color=auto master|runner
Yes, there are no mailman processes running.
As far as the permissions, you asked does your systemd script set the user to the Mailman user? I don't know how to check that, but I saw many files in /var/lib/mailman3 were owned by a user named list. Last time I tried to run mailman status, it threw a permission error on mailman.cfg, which was owned by root. I changed owner to list, and it produced a different error. So I'm guessing we are on the right path? There was no mention of a user named mailman in the system before, but maybe I need to check something.
You are apparently using the Debian/Ubuntu packages. In those, the
Mailman user is list. There is no mailman user.
As far as what user the systemd script sets, just look at the script, but if it was installed by the Debian/Ubuntu package, it's probably OK. The script is named xxx.service where 'xxx' is the name you might use in systemctl commands (probably 'mailman3'). It should be at /etc/systemd/service/ or one of the subdirectories.
Lastly, I deleted everything in /var/lib/mailman3/locks.
However, running mailman status now throws this permission error
File "/usr/bin/mailman", line 11, in <module> load_entry_point('mailman==3.1.1', 'console_scripts', 'mailman')() File "/usr/lib/python3/dist-packages/mailman/bin/mailman.py", line 94, in main initialize(config_path) File "/usr/lib/python3/dist-packages/mailman/core/initialize.py", line 188, in initialize initialize_1(config_path) File "/usr/lib/python3/dist-packages/mailman/core/initialize.py", line 123, in initialize_1 mailman.config.config.load(config_path) File "/usr/lib/python3/dist-packages/mailman/config/config.py", line 107, in load self.push(filename, user_config.read()) File "/usr/lib/python3/dist-packages/mailman/config/config.py", line 113, in push self._post_process() File "/usr/lib/python3/dist-packages/mailman/config/config.py", line 125, in _post_process self.ensure_directories_exist() File "/usr/lib/python3/dist-packages/mailman/config/config.py", line 227, in ensure_directories_exist with Lock(lock_file): File "/usr/lib/python3/dist-packages/flufl/lock/_lockfile.py", line 334, in __enter__ self.lock() File "/usr/lib/python3/dist-packages/flufl/lock/_lockfile.py", line 203, in lock self._write() File "/usr/lib/python3/dist-packages/flufl/lock/_lockfile.py", line 417, in _write with open(self._claimfile, 'w') as fp: PermissionError: [Errno 13] Permission denied: '/var/lib/mailman3/locks/mailman-cfg.lck|[redacted server name]|21787|3720843219351355903'
That file was removed when you told me to clear the locks folder (I save a copy, however). So I'm stuck again. Sorry for all the questions and thanks very much for the help.
Does the list user own /var/lib/mailman3/ and
/var/lib/mailman3/locks/? Are you running mailman commands as user list?
Also see <https://wiki.list.org/x/12812344>.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Alrighty, so resetting the permissions and running as list got things running. I was able to start mailman, although when it started I got 4 pages of exceptions. Here's one shortened example (most seemed to follow the same pattern):
File "/usr/lib/python3/dist-packages/zope/configuration/config.py", line 794, in toargs args[str(name)] = field.fromUnicode(s) zope.configuration.xmlconfig.ZopeXMLConfigurationError: File "<string>", line 19.2-23.6 MemoryError: File "/usr/lib/python3/dist-packages/zope/configuration/fields.py", line 99, in fromUnicode v = vt.fromUnicode(s)
However, I do get these positive signs:
GNU Mailman is running (master pid: 4434)
and on running the grep, I get these two:
list 4434 1 0 10:53 ? 00:00:01 /usr/bin/python3 /usr/lib/mailman3/bin/master -C /etc/mailman3/mailman.cfg list 4440 4434 0 10:53 ? 00:00:01 /usr/bin/python3 /usr/lib/mailman3/bin/runner --runner=bounces:0:1 -C /etc/mailman3/mailman.cfg
So that seems promising, however Mailman REST API not available. Please start Mailman core still remains. Also, not sure if this matters, but doing a systemctl status for mailman3 yielded this:
mailman3.service - Mailman3 server Loaded: loaded (/lib/systemd/system/mailman3.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Mon 2020-07-27 22:35:34 PDT; 1 day 12h ago Docs: man:mailman(1) https://mailman.readthedocs.io/ Main PID: 16834 (code=exited, status=0/SUCCESS)
Trying to start that yields:
Job for mailman3.service failed because the control process exited with error code. See "systemctl status mailman3.service" and "journalctl -xe" for details.
The log for mailman3.service seems to also be giving a memory error from when it last tried to run:
25 21:45:12 systemd[1]: Started Mailman3 server. Jul 25 21:45:56 mailman3[30357]: Traceback (most recent call last): Jul 25 21:45:56 mailman3[30357]: File "/usr/lib/python3/dist-packages/zope/configuration/xmlconfig.py", line 272, in endElementNS Jul 25 21:45:56 mailman3[30357]: self.context.end() Jul 25 21:45:56 mailman3[30357]: File "/usr/lib/python3/dist-packages/zope/configuration/config.py", line 345, in end Jul 25 21:45:56 mailman3[30357]: self.stack.pop().finish() Jul 25 21:45:56 mailman3[30357]: File "/usr/lib/python3/dist-packages/zope/configuration/config.py", line 452, in finish Jul 25 21:45:56 mailman3[30357]: args = toargs(context, *self.argdata) Jul 25 21:45:56 mailman3[30357]: File "/usr/lib/python3/dist-packages/zope/configuration/config.py", line 794, in toargs Jul 25 21:45:56 mailman3[30357]: args[str(name)] = field.fromUnicode(s) Jul 25 21:45:56 mailman3[30357]: File "/usr/lib/python3/dist-packages/zope/configuration/fields.py", line 72, in fromUnicode Jul 25 21:45:56 mailman3[30357]: value = self.context.resolve(name) Jul 25 21:45:56 mailman3[30357]: File "/usr/lib/python3/dist-packages/zope/configuration/config.py", line 151, in resolve Jul 25 21:45:56 mailman3[30357]: mod = __import__(mname, *_import_chickens) Jul 25 21:45:56 mailman3[30357]: File "/usr/lib/python3/dist-packages/mailman/email/validate.py", line 30, in <module> Jul 25 21:45:56 mailman3[30357]: _valid_local = re.compile("[-0-9a-z!#$%&'*+./=?@_`{}~]", re.IGNORECASE) Jul 25 21:45:56 mailman3[30357]: File "/usr/lib/python3.6/re.py", line 233, in compile Jul 25 21:45:56 mailman3[30357]: return _compile(pattern, flags) Jul 25 21:45:56 mailman3[30357]: File "/usr/lib/python3.6/re.py", line 301, in _compile Jul 25 21:45:56 mailman3[30357]: p = sre_compile.compile(pattern, flags) Jul 25 21:45:56 mailman3[30357]: File "/usr/lib/python3.6/sre_compile.py", line 566, in compile Jul 25 21:45:56 mailman3[30357]: code = _code(p, flags) Jul 25 21:45:56 mailman3[30357]: File "/usr/lib/python3.6/sre_compile.py", line 551, in _code Jul 25 21:45:56 mailman3[30357]: _compile(code, p.data, flags) Jul 25 21:45:56 mailman3[30357]: File "/usr/lib/python3.6/sre_compile.py", line 108, in _compile Jul 25 21:45:56 mailman3[30357]: _compile_charset(av, flags, code, fixup, fixes) Jul 25 21:45:56 mailman3[30357]: File "/usr/lib/python3.6/sre_compile.py", line 226, in _compile_charset Jul 25 21:45:56 mailman3[30357]: for op, av in _optimize_charset(charset, fixup, fixes): Jul 25 21:45:56 mailman3[30357]: File "/usr/lib/python3.6/sre_compile.py", line 354, in _optimize_charset Jul 25 21:45:56 mailman3[30357]: charmap = bytes(charmap) # should be hashable Jul 25 21:45:56 mailman3[30357]: MemoryError
On 7/29/20 11:28 AM, joseverde@gmail.com wrote:
Alrighty, so resetting the permissions and running as list got things running. I was able to start mailman, although when it started I got 4 pages of exceptions. Here's one shortened example (most seemed to follow the same pattern):
Your issues all seem to be MemoryError This is a Python detected out of memory condition. You need more RAM or you need to configure or increase the size of a swap file.
What does free show?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Well that was it! Upped from 1GB to 2GB, restarted and started mailman service. It seems to be up and running! Now I think I just need to configure some things and start learning! Thanks for the help! I'm sure I'll be back with config questions later!
Hi there, hope i can enrich the discussion with my (at the end easy) solution after 6hours of trying and reading. I needed to reboot my server with well running lists. I run into the same problem and solved it by
sudo -u mailman /opt/mailman/venv/bin/mailman start -f
Kind regards
participants (4)
-
joseverde＠gmail.com -
Mark Sapiro -
markus.laspeyres＠nerdline.de -
Stephen J. Turnbull