On Oct 26, 2019, at 12:38 AM, Franklin Weng franklin@goodhorse.idv.tw wrote:

In the mailman.cfg database is set to mysql and in the mailman-web.py database is set to sqlite3. Does it matter? I tried to search but couldn't find any further information or discussion about this.

This seems like it would matter but I am no mailman3 expert (yet).

There is no systemd service for Hyperkitty. How can I check if Hyperkitty is running or not?

How did you install mailman3?

I updated from ubuntu 19.04 to ubuntu 19.10 last week using the .deb packages for mailman3 and I had a ton of problems with hyperkitty. On a whim I tried to re-install the mailman3-full package and lo and behold, instead of telling me it was all installed, it told me that it was installing the hyperkitty component. So somehow it got skipped silently during the distro upgrade.

Worth a check.

• Mark

mark@pdc-racing.net | 408-348-2878

Hi,

Tamas HOLCZER via Mailman-users mailman-users@mailman3.org 於 2019年10月28日 週一 20:32 寫道：

Hi Franklin,

it seems you have a brand new install, right? By default (at least in Debian) the archivers are not enabled. You should add something like this to your mailman.cfg:

[archiver.hyperkitty] class: mailman_hyperkitty.Archiver enable: yes configuration: /etc/mailman3/mailman-hyperkitty.cfg

Please check this site:

https://docs.mailman3.org/en/latest/devsetup.html#set-up-and-run-hyperkitty

Best regards,

Tamás

Yes, I did follow the document and add it in the mailman.cfg, but seems not worked. How to check if it really is working or not?

Regards, Franklin

Hi Stephen,

Thanks for your explanation.

Stephen J. Turnbull turnbull.stephen.fw@u.tsukuba.ac.jp 於 2019年10月29日 週二 下午1:35寫道：

Franklin Weng writes:

...

However in the Archive section I couldn't see anything and all the lists were "inactive" and hence were hidden by default.

If you can see that much, HyperKitty is running.

Have you sent mail to the lists? Has it been delivered to subscribers?

Yes, mails to subscribers were delivered successfully.

...

I confirmed that api_key set in the /etc/mailman3/mailman-hyperkitty.cfg and /etc/mailman3/mailman-web.py is the same. I also confirmed that the apache2 wsgi module has been loaded.

In the mailman.cfg database is set to mysql and in the mailman-web.py database is set to sqlite3. Does it matter?

I don't think they need to be the same. The HyperKitty and Postorius databases are a Django database backed by an RDBMS (typically sqlite3, PostgreSQL, or MySQL). The databases used by the Mailman core and the Django webservices HyperKitty and Postorius are conceptually separate, although they can be backed by the same RDBMS server.

As Tamas has said in the previous reply, my mailman is a fresh new install on Debian buster. I followed Debian.README and didn't change much. In Debian's setting, under /usr/share/mailman3-web/ are these files:

manage.py settings_local.py -> /etc/mailman3/mailman-web.py settings_local.py.sample settings.py urls.py wsgi.py

In the top of settings_locas.py there are comments saying that this file is to override default settings in /usr/share/mailman3-web/settings.py, so I suppose hyperkitty database should define here, though it is different from /usr/share/mailman3-web/settings.py.

However, sqlite3 is likely to result in poor performance compared to PostgreSQL or MySQL.

You have to configure HyperKitty's database. In my sqlite3 test installation it looks like this:

# Top of settings.py. BASE_DIR is the directory containing settings.py. BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

# 30-50 lines later: DATABASES = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', # This database file is in the same directory as settings.py. 'NAME': os.path.join(BASE_DIR, 'hyperkitty.db'), # Not used with sqlite3: 'USER': '', 'PASSWORD': '', 'HOST': '', 'PORT': '', } }

Does the database file named by 'NAME' exist?

In the settings_local.py, it defines sqliet3 and the file of 'NAME' /var/lib/mailman3/web/mailman3web.db exists and is a sqlite3 format file. But in settings.py (which is supposed to be overridden by setting_local.py) it defines using postgresql, which doesn't exist in my Debian system.

There is no systemd service for Hyperkitty. systemd runs Apache, which loads mod_wsgi, which runs Python, which imports the HyperKitty package, which responds to web requests by building pages dynamically using Django from information from Django's database, which may be backed by MySQL, PostgreSQL, or sqlite3. Sounds complicated, but it performs well once you get it properly configured.

Is there any log I can trace the executing path? It looks not complicated but I didn't know and couldn't find out how to check if it works correctly. For Apache2 mod_wsgi, the wsgi.conf in mods-enabled does not enable any configuration (all commented out) and when I use ps to check, uwsgi runs as:

14233 ? Ss 0:01 /usr/bin/uwsgi --plugin python3 --ini /etc/mailman3/uwsgi.ini 14236 ? Sl 0:00 /usr/bin/uwsgi --plugin python3 --ini /etc/mailman3/uwsgi.ini

The /var/log/mailman3/web/mailman-web.log is at https://pastebin.com/U3jHR6JS There seems to be something running but in the interface I still didn't see any results.

Franklin

Yes, it is checked.

Mark Sapiro mark@msapiro.net 於 2019年10月30日 週三 11:19 寫道：

I haven't read this thread in detail. I've been off line for 3 days due to a Public Safety Power Shutoff, but my first question is if you go the the Pustorius web UI for the list(s) and look at Settings -> Archiving -> Active archivers, is hyperkitty checked?

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

---

Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/

Franklin Weng writes:

In Debian's setting, under /usr/share/mailman3-web/ are these files:

manage.py settings_local.py -> /etc/mailman3/mailman-web.py settings_local.py.sample settings.py urls.py wsgi.py

That looks right. This is a typical Debian approach to preserving local configuration automatically when Debian is upgraded.

In the top of settings_locas.py there are comments saying that this file is to override default settings in /usr/share/mailman3-web/settings.py, so I suppose hyperkitty database should define here,

Yes.

though it is different from /usr/share/mailman3-web/settings.py.

In Debian, you can't control the contents of that file. It may be overwritten in any upgrade. That's why the settings_local.py approach is used.

In the settings_local.py, it defines sqliet3 and the file of 'NAME' /var/lib/mailman3/web/mailman3web.db exists and is a sqlite3 format file.

That's OK (until you need higher performance).

But in settings.py (which is supposed to be overridden by setting_local.py) it defines using postgresql, which doesn't exist in my Debian system.

I assume it is overridden, unless Debian really messed up its packaging. But it seem to be working for other users (at least, no complaints I can recall).

Is there any log I can trace the executing path?

You really don't want to do that.

14233 ? Ss 0:01 /usr/bin/uwsgi --plugin python3 --ini /etc/mailman3/uwsgi.ini 14236 ? Sl 0:00 /usr/bin/uwsgi --plugin python3 --ini /etc/mailman3/uwsgi.ini

I don't know much about uswgi, so I can't say whether multiple processes with the same invocation is a problem. It's something to keep in mind.

Recently uwsgi is problematic. There are multiple threads in the last few weeks. Unfortunately, Mailman devs are mostly using Apache + mod_swgi or gunicorn, so we can't be terribly helpful if there is a problem with uwsgi. Apparently the Apache process with mod_swgi is irrelevant.

The /var/log/mailman3/web/mailman-web.log is at https://pastebin.com/U3jHR6JS

That log looks perfectly normal to me. Since the log looks normal, you don't need to worry about uwsgi yet, but you should review the threads in our archives.

The only thing that strikes me about the log is that there are no accesses to ".../hyperkitty/api/...", which is how HyperKitty receives messages to archive. Perhaps you simply didn't send any messages during the period of the captured log, but if you did, it should be there. Three questions:

1. Do you have HyperKitty configured as the Mailman archiver in Mailman?

2. In mailman.cfg do you have

[archiver.hyperkitty] class: mailman_hyperkitty.Archiver enable: yes configuration: /path/to/here/mailman-hyperkitty.cfg

where the "configuration" value should give the full path to a mailman-hyperkitty.cfg file?

3. Do your lists have archiving configured on in Postorius?

Steve

Hi Franklin,

when I installed mailman3 on Debian I had some issues. One was about MAILMAN_ARCHIVER_FROM. It turned out, that the request to Hyperkitty was sent from one of the real IPs of the machine (not from localhost). This depends on the interfaces and the routing of that machine. In my case the solution was something like:

MAILMAN_ARCHIVER_FROM = ('127.0.0.1', '::1', '1.2.3.4', '1.2.3.5') , where 1.2.3.4 and 1.2.3.5 are the IP addresses of the machine running mailman3 (I have not debugged if it is using .4 or .5 in reality I left both of them there). Give it a try with your IPs.

Tamás

2019. 10. 30. 8:52 keltezéssel, Franklin Weng írta:

Hi,

It looks like to have some clues now.

Mark Sapiro mark@msapiro.net 於 2019年10月30日 週三 13:07 寫道：

...

...

Yes, it is checked. Have you checked Mailman's logs - Mailman's var/logs/mailman.log in

On 10/29/19 8:21 PM, Franklin Weng wrote: particular?

I found some hyperkitty URL error:

The default settings of base_url in mailman-hyperkitty.cfg is defined as

base_url: http://localhost/mailman/hyperkitty/

In the mailman.log there are messages like:

Oct 30 13:53:43 2019 (14259) HyperKitty failure on http://localhost/mailman3/hyperkitty/api/mailman/archive: <!DOCTYPE HTML PUBLIC "-//IETF// DTD HTML 2.0//EN">

<html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL was not found on this server.</p>

I changed it to

base_url: https://localhost/mailman3/hyperkitty/

then the log became

File "/usr/lib/python3/dist-packages/requests /adapters.py", line 514, in send raise SSLError(e, request=request) requests.exceptions.SSLError: HTTPSConnectionPool(host='localhost', port=443): Max retries exceeded with url: /mailman3/hyperkitty/api/mailman/archive?key=xxx (Caused by SSLError(SSLCertVerification Error("hostname 'localhost' doesn't match ' lists.slat.org'")))

which makes sense because the SSL certificate applied from Letsencrypt is for lists.slat.org.

If I use

base_url: https://lists.slat.org/mailman3/hyperkitty/

The log became

Oct 30 15:25:50 2019 (6256) HyperKitty failure on https://lists.slat.org/mailman3/hyperkitty/api/mailman/urls:

<html><title>Auth required</title><body>

            <h1>Authorization Required</h1>

</body></html> (401)

In my mailman-web.py:

MAILMAN_ARCHIVER_KEY = 'my-key,same as in mailman-hyperkitty.cfg' MAILMAN_ARCHIVER_FROM = ('127.0.0.1', '::1')

Not sure what to change here.

...

Do messages received from the list have an Archived-At: header?

Yes, but it's empty

List-Id: 測試論壇 Archived-At: <> List-Archive: <> List-Help: mailto:slat-list-request@lists.slat.org?subject=help List-Post: mailto:slat-list@lists.slat.org List-Subscribe: mailto:slat-list-join@lists.slat.org List-Unsubscribe: mailto:slat-list-leave@lists.slat.org

...

What's in /etc/mailman3/mailman-hyperkitty.cfg? In particular does the [general] setting base_url define a URL that accesses hyperkitty

It seems to be the question…

and

...

does the [general] setting api_key match the MAILMAN_ARCHIVER_KEY setting in /etc/mailman3/mailman-web.py, but note that the MAILMAN_ARCHIVER_KEY setting is quotes because it is a Python string assignment and the api_key setting is not quoted

Yes, it's single quoted.

Thanks for all your help!

Franklin

---

Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/

Hi,

Then I found that

MAILMAN_REST_API_URL = 'http://localhost:8001'

the URL does not work either (404 Page not found), but I didn't find any error log about that.

My apache setting is: 000-default.conf: port 80, didn't change the default setting (/var/www/html/index.html replaced) 001-lists.conf: server name "lists.slat.org", port 80 Redirect "/" to " https://lists.slat.org/". Port 443 has all the SSL and mailman alias set.

In this case how should I set the API_URL and hyperkitty URL?

Thanks, Franklin

Tamas HOLCZER via Mailman-users mailman-users@mailman3.org 於 2019年10月30日 週三 下午8:09寫道：

Hi Franklin,

when I installed mailman3 on Debian I had some issues. One was about MAILMAN_ARCHIVER_FROM. It turned out, that the request to Hyperkitty was sent from one of the real IPs of the machine (not from localhost). This depends on the interfaces and the routing of that machine. In my case the solution was something like:

MAILMAN_ARCHIVER_FROM = ('127.0.0.1', '::1', '1.2.3.4', '1.2.3.5') , where 1.2.3.4 and 1.2.3.5 are the IP addresses of the machine running mailman3 (I have not debugged if it is using .4 or .5 in reality I left both of them there). Give it a try with your IPs.

Tamás

On 10/30/19 12:52 AM, Franklin Weng wrote:

If I use

base_url: https://lists.slat.org/mailman3/hyperkitty/

The log became

Oct 30 15:25:50 2019 (6256) HyperKitty failure on https://lists.slat.org/mailman3/hyperkitty/api/mailman/urls:

<html><title>Auth required</title><body>

            <h1>Authorization Required</h1>

</body></html> (401)

In my mailman-web.py:

MAILMAN_ARCHIVER_KEY = 'my-key,same as in mailman-hyperkitty.cfg' MAILMAN_ARCHIVER_FROM = ('127.0.0.1', '::1')

Not sure what to change here.

There have been other reports of authorization failures even though the keys match. See the last few posts at https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/M.... The solution there was to change both the api_key setting in /etc/mailman3/mailman-hyperkitty.cfg and the MAILMAN_ARCHIVER_KEY setting in /etc/mailman3/mailman-web.py to new values.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

I have a similar problem with mailman 3.2.1 (debian package):

# mailman info GNU Mailman 3.2.1 (La Villa Strangiato) Python 3.7.3 (default, Jul 25 2020, 13:03:44) [GCC 8.3.0]

the messages are not archived even if the key is set and hyperkitty is checked as active archiver.

To analyze the problem I set DEBUG = True in /etc/mailman3/mailman-web.py. In debug mode the problem does not occur, but this second problem appears: https://gitlab.com/mailman/hyperkitty/-/issues/84

How can I fix it?