Search results for "sapiro" - lists.mailman3.org

[MM3-users] Re: import21 detailed usage instructions in 2024

by Mark Sapiro

On 6/2/24 4:28 PM, Keith Christian wrote: > 1 Soon, I will be executing the first test imports from > Mailman2.1 to Mailman3.9. > > 2 Planning to copy over the entire production Mailman2.1 to a new > directory structure on the Mailman 3.9 machine. > > 3 Expecting that all .mbox and .pkl files will be present and no > special work has to be done to bring them over will a "full" > mailman2.1 copy. That's correct. > 4 Can the import21 program find the mailman2.1 .mbox and .pkl > files on its own and process them? No. `mailman import21` needs to be given the path to the config.pck to import and hyperkitty_import needs to be given the path to the .mbox. > 5 Other than the mailman2.1 instance being shut down, any other > prep that needs to be done? Ideally there are no held messages or other moderator requests for the MM 2.1 list, i.e. deal with these before shutting down the list. If there is a MM 2.1 digest.mbox it needs to be moved and converted for MM 3 I have attached a bash script I use on mail.python.org to migrate lists. Paths and some other things are specific to that installation, but you may find it useful. It calls a move_digest script which I've also attached > 6 Any special command line parameters to be aware of for import21? Usage: mailman import21 [OPTIONS] LISTSPEC PICKLE_FILE Import Mailman 2.1 list data. Requires the fully-qualified name of the list to import and the path to the Mailman 2.1 pickle file. Options: -c, --charset TEXT Specify the encoding of strings in PICKLE_FILE if not utf-8 or a subset thereof. This will normally be the Mailman 2.1 charset of the list's preferred_language. -h, --help Show this message and exit. > 7 Is there a list of troubleshooting steps for commonly > encountered migration issues? Not that I'm aware of. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

1 year, 3 months

[MM3-users] Re: Splitting hosts: mta&core vs postorius

by Mark Sapiro

On 6/22/20 12:21 PM, Ruth Ivimey-Cook wrote: > > The localhost address is coming in via the hosts file, which is a > relatively standard practice IMO. I could change it, but then perhaps > other things would not work. I would prefer to be able to specify what I > need. There are also things like systemd-resolvd which will do the > localhost-substitution for you even if /etc/hosts doesn't. I have done further testing. See my comment at <https://gitlab.com/mailman/mailman/-/issues/735#note_366074014>. You need to set hostname: in mailman.cfg to the actual external IP address of the Mailman host machine or to a name which resolves to that IP both internally and externally, and also set that host name/IP in the MAILMAN_REST_API_URL setting on the Postorius machine. > I did try with the gunicorn patch Mark mentioned (host = "0.0.0.0") and > changing the cfg file hostname back to the full name of the host, and > that now works!! -- in that the problem I showed earlier is gone. Remove that patch and do the above. > However I then get another issue. From the mailman.log file I now see in > the last few lines. For clarity I've numbered them like this "{1}": > > {1} [22/Jun/2020:00:58:24 +0000] "GET /3.1/domains HTTP/1.1" 200 320 "-" > "GNU Mailman REST client v3.3.1" > {2} [22/Jun/2020:00:58:24 +0000] "GET /3.1/domains/ch-bc.org.uk > HTTP/1.1" 200 215 "-" "GNU Mailman REST client v3.3.1" > {3} [22/Jun/2020:00:58:24 +0000] "GET > /3.1/domains/greyarea-web1.cam.ivimey.org/lists?advertised=true&count=0&page=1 > HTTP/1.1" 404 26 "-" "GNU Mailman REST client v3.3.1" > > At this point the browser screen reports "Something went wrong" and the > message "HTTP Error 404: {"title": "404 Not Found"}" Make the above changes and see if that solves this. If not, let us know and we'll try to help. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

5 years, 2 months

[MM3-users] Re: Hyperkitty 403 Forbidden Error

by Mark Sapiro

On 1/23/22 07:41, Josef Dietl wrote: >> >> The issue here is accessing the hyperkitty API. The base URL for these >> accesses is configured in mailman-hyperkitty.cfg. From what I see here, >> that is something like `http(s)://host.example.com/hyperkitty/` If you >> append `api/` to that base URL and go there in a web browser do you get >> a page of documentation of the HyperKitty API? If so, then figure out >> why it works from a web browser and not when Mailman core does it. If >> you get the 403, figure out why. This is probably an nginx issue. > > Seems like an nginx issue indeed. However, I'm an apache guy, trying to re-educate myself on nginx. > OK, here's the result of changing mailman-hyperkitty.cfg from `http(s)://host.example.com/hyperkitty/` to `http(s)://host.example.com/hyperkitty/api/` Do not make this change in mailman-hyperkitty.cfg. That will produce 404's for sure. This was only a suggestion to try from a browser. > Yes, if I open things from the web browser, I do get the HyperKitty API doc. OK. That means you can access HyperKitty's API from a browser at `http(s)://host.example.com/hyperkitty/api/`. This is the same thing Mailman core is doing if `base_url` is set to `http(s)://host.example.com/hyperkitty/`, so what's the difference between Mailman core's access via mailman-hyperkitty and your's via the web browser. For further diagnosis, you might try `wget -O- http(s)://host.example.com/hyperkitty/api/` on the Mailman server. > With what follows below, I guess it's a timeout: Processing a message takes longer than rendering a HTML document. > > Jan 22 20:26:27 2022 (149435) ACCEPT: > <AM0PR04MB4785365CA1BB735884E149CEC25C9(a)AM0PR04MB4785.eurprd04.prod.outlook.com> > Jan 22 20:26:28 2022 (149439) HyperKitty failure on https://[FQDN]/hyperkitty/api/api/mailman/urls: Do not include the api/ in base_url. That was just for testing with a web browser or wget/curl. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

3 years, 7 months

[MM3-users] Re: Pd CRON mailman gatenews

by Mark Sapiro

On 3/15/23 23:31, Pierre Malard via Mailman-users wrote: > File "/usr/lib/python3/dist-packages/flufl/lock/_lockfile.py", line 502, in _read > with open(self._lockfile) as fp: > PermissionError: [Errno 13] Permission denied: '/var/lib/mailman3/locks/dbcreate.lck' > —————————————————————————————————————————————————— > > CRON content: > —————————————————————————————————————————————————— ... > # Every 15 minutes, gate messages from usenet to those lists which have the gateway configured > */15 * * * * list if [ -x /usr/bin/mailman ]; then /usr/bin/mailman gatenews; fi Do you have any list's for which Gateway to mail is set to yes or for which there is a Linked Newsgroup?. If not, running gatenews is superfluous. > I thought, given the last line, a problem of rights. It is not the case ! The directory and the files contained in /var/lib/mailman3/locks belong to the user "list". > When I run "/usr/bin/mailman gatenews" by hand everything is correct. Can you do `sudo -u list cat /var/lib/mailman3/locks/dbcreate.lck`. If that succeeds, this may be a SELinux or other security manager issue. > The questions I have are these: > We have 8000 files "dbcreate.lck..." in the directory "/var/lib/mailman3/locks". > What are they for? > Can we delete them? They are probably all orphaned lock files due the the gatenews failure. To be certain that none are current you could stop Mailman core. Then with Mailman core stopped, any files in /var/lib/mailman3/locks are orphaned and should be removed. > We have 2 files "master.lck..." dated from the day after tomorrow ! > What is this date? > Aren't these files the problem? Those files are normal. They are the master lock that prevents Mailman core from being started when it's already running. The future date is the way flufl.lock handles lock lifetimes. Locks when set have a lifetime, after which they can be broken. This is implemented in flufl.lock by setting the file's date that far into the future. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

2 years, 5 months

[MM3-users] Re: Fwd: [Django] ERROR (EXTERNAL IP): Internal Server Error: /mailman3/hyperkitty/api/mailman/archive

by Mark Sapiro

On 3/2/21 12:33 PM, David Partain via Mailman-users wrote: > On Tue, 2021-03-02 at 20:07 +0000, David Partain via Mailman-users > wrote: >> On Tue, 2021-03-02 at 19:56 +0000, David Partain via Mailman-users >> wrote: >> >> I assume this is because /usr/share/mailman3-web/manage.py, which >> has: >> >> '#!/usr/bin/env python' >> >> resolves to Python 2.7.17. >> >> Should I could try changing that to '#!/usr/bin/env python3' and see >> what happens? :) I could also uninstall python2 but that becomes >> more of a nail-biter... NO! Do not do that. >> I can report that the above doesn't help. Neither does just making >> /usr/bin/python link to /usr/bin/python3.6 >> >> I still get errors in mailman-web.log pointing to 2.7 as shown in my >> previous mail. > > If I force the issue by linking /usr/bin/python to /usr/bin/python3.6, > I start seeing: > > ModuleNotFoundError: No module named 'hyperkitty > > in mailman-web.log. > > I know next to nothing about python. Is > > pip install hyperkitty > > the right answer? No. > I have the following installed now: > > # dpkg -l | grep hyperkitty > python-django-hyperkitty 1.1.4-4 > python3-mailman-hyperkitty 1.1.0-4 > > but this version of Ubuntu has no python3-django-hyperkitty or I'd just > try installing that... This is all because you are using the Ubuntu 18.04 package which is about 4 years out of date and is from the era where HyperKitty, Postorius and django_mailman3 still ran under Python 2.1.7. Unless you want to remove the Ubuntu package and install everything with pip, you need to just stick with what you have that at least partly works and we'll try to determine how to fix the utf8mb4 issue. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

4 years, 6 months

[MM3-users] Re: import21: ModuleNotFoundError: No module named 'Mailman'

by Mark Sapiro

On 7/28/19 4:18 PM, Christian Schwarz wrote: > > Most mailing lists can be imported successfully using the `import21` subcommand, as described in the docs. Good. > Some mailing lists fail with the following error: > > sudo -u list mailman import21 REDACTEDLIST(a)example.org /var/lib/mailman/lists/REDACTEDLIST/config.pck > Traceback (most recent call last): ... > File "/usr/lib/python3/dist-packages/mailman/commands/cli_import.py", line 64, in import21 > pickle_file, encoding='utf-8', errors='ignore') > ModuleNotFoundError: No module named 'Mailman' I know what's happening, but I need more detail to do more than guess at how to fix it. The issue is there is a Mailman 2.1 Mailman.xxx.yyy object in the config.pck file. My best guess is that the list's bounce_info is the culprit. The list's bounce_info is a dictionary with keys equal to member addresses and values equal to the Mailman.Bouncer._BounceInfo object representing that member's bounce info. So when import21 calls pickle.load to load the config.pck, there is a reference to one or more Mailman.Bouncer._BounceInfo objects, but pickle.load can't find the definition of that object and that raises the exception. As to how to proceed, there are a couple of ways. One is to fix import21 so it ignores this issue. I just reported <https://gitlab.com/mailman/mailman/issues/618> on this, but I'm not sure how to fix it. I don't think it will work to try the pickle.load and pass the ModuleNotFoundError exception because I think the rest of the pickled data won't be returned. It looks like you still have the MM 2.1 installation, so you could use either MM 2.1's bin/withlist or bin/config_list to set the list's bounce_info to an empty dictionary. and then use the updated config.pck for the import. This assumes that bounce_info is the issue. You can verify that there is a non-empty bounce_info dictionary with either bin/withlist or bin/dumpdb. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

6 years, 1 month

[MM3-users] Re: Pb with archive display

by Mark Sapiro

On 5/5/22 08:02, Pierre Malard via Mailman-users wrote: > > Unfortunately we still don't have any archives... > > Here are the logs retrieved from the mailman.log logs: > May 05 15:59:42 2022 (579075) Exception in the HyperKitty archiver: {"error": "character with byte sequence 0xe2 0x80 0x99 in encoding \"UTF8\" has no equivalent in encoding \"LATIN1\"\n"} > raise ValueError(result.text) > ValueError: {"error": "character with byte sequence 0xe2 0x80 0x99 in encoding \"UTF8\" has no equivalent in encoding \"LATIN1\"\n"} > > And in mailman-web.log: > WARNING 2022-05-05 13:58:52,996 579167 hyperkitty.views.mailman Could not archive the email with message-id '<C7E61653-D399-466A-BEC3-CE61D567E870(a)teledetection.fr>': character with byte sequence 0xe2 0x80 0x99 in encoding "UTF8" has no equivalent in encoding "LATIN1" > > Is it a way? > > I don't understand the message about LATIN1. > It seems to me that UTF-8 was the encoding retained in Mailman. I too don't understand what's trying to encode as LATIN1. The particular byte sequence is a utf-8 encoding of right single quote (’) which has no LATIN1 equivalent, but there's not enough info in the logs for me to determine exactly what's going on. There is an issue with MySQL and derivatives if that's the database you are using. You need the MySQL encoding to be UTF8MB4. in the DATABASES definition in your django settings you want ``` 'OPTIONS': {'charset': 'utf8mb4'} # Enable utf8 4-byte encodings. ``` but that doesn't seem to be the issue here. I suspect this issue has to do with the particular message. When mailman-hyperkitty processes the queue and a queued message throws an exception, it just dies. You might try moving aside the first (in alphanumeric sequence) message in /var/lib/mailman3/archives/hyperkitty/spool and see if the remaining messages can be processed. If that works I would like to see that message's .pck file. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

3 years, 4 months

[MM3-users] Re: problems installing mailman3 under gentoo and upgrading mailman2 to mailman3

by Mark Sapiro

On 3/12/21 7:36 PM, John Covici wrote: > Hi. I am having a great deal of difficulty installing mailman3 using > gentoo and sendmail. Also, I want to upgrade my old mailman2 data to > mailman3. > > What I have done so far is to install the mailman3 packages from my > repository and got this: > django-mailman3 version 1.3.4, mailmanclient version 3.3.1, mailman > version 3.3.2 (apparently the core), postorius version 1.3.3 and > hyperkitty version 1.3.3. I also have mariadb and postgresql already installed. Are these gentoo packages? If so, I have no idea how they are configured. > Now trying to read your documentation, it seems it wants me to > reinstall some packages using pip in the home directory of the mailman > user in a python virtual environment, that was the first thing. But I > would like to avoid multiple package managers wherever possible -- > gentoo is already a lot of maintainance without dealing with pip's > maintainance. The binaries also seem to be different from my gentoo packages. Our documantation generally covers installing from source. If you are installing gentoo packages, you should be looking at gentoo docs and going to gentoo for help. We recommend installing Mailman and the web components in a virtualenv, but it is not a requirement. We also suggest that postgresql is a better choice than MariaDB. > So, how do I proceed with getting mailman3 up and working, configuring > the database, configs, etc and I guess after doing that upgrading > mailman2 and getting sendmail to interface properly with mailman3? Importing lists from Mailman 2 is done with the `mailman import21` command. See <https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/commands/d…>. Importing archives is done with the Django management `hyperkitty_import` command. See <https://docs.mailman3.org/projects/hyperkitty/en/latest/install.html#import…> For sendmail, see <https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.h…> -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

4 years, 6 months

[MM3-users] Re: Multi-domain oddities in Hyperkitty and Postorius

by Mark Sapiro

On 12/14/22 13:40, Jeremy Stanley wrote: > > Once these entries are automatically added by visiting that page, > they're all associated with the one "web host" ("site" in Django) we > have for the server, and now the list filtering functionality in > Postorius breaks and any of the vhosts all display the full set of > mailing lists on the server rather than only those associated with a > particular vhost (note that this seems to only be confusing > Postorius, Hyperkitty has them correctly filtered still). See https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message… and also below. > Okay, so it looks like this is because the mail hosts are all tied > to the one web host. Yes. > We can create additional "sites" in the Django > admin UI and then switch the association of the mail hosts to those > distinct web hosts. But after doing so, the first vhost still seems > to show an unfiltered list of all lists in Postorius (Hyperkitty > continues to be fine), while Postorius on the other vhosts correctly > shows their filtered sets of lists instead. > > I also noticed when mailman-web is starting up, its log claims the > first site is being set as the "default domain," so have to wonder > if that's related to this behavior. Should we create an extra web > host to serve as the "default" so that all our typical vhosts get > the expected list filtering behavior, or am I completely off-base > and there's a much better approach I've missed? I'm not really sure what's going on here. You could try creating a dummy default host to see if that fixes the issue for you. There are subtle differences in the implementation between HyperKitty and Postorius that allow HyperKitty to work better, but it too will exhibit some anomalous behaviors. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

2 years, 8 months

[MM3-users] Re: Postorius no connection to REST API

by Mark Sapiro

On 8/12/21 12:01 AM, Richard Rosner wrote: > Mark Sapiro wrote: >>> But whatever is listening on port 8001 is apparently not Mailman's rest >> server. What does >> ps -fww 14080 >> or whatever PID is currently listening on port 8001 show. > > UID PID PPID C STIME TTY STAT TIME CMD > list 15972 15963 0 Aug11 ? Sl 0:13 /usr/bin/python3 /usr/lib/mailman3/bin/runner -C /etc/mailman3/mailman.cfg --runner=rest:0:1 I had asked "What does ps -fwwa|grep rest show?" and it didn't find this. That was my error. I meant 'ps -fwwA|grep rest' >>> OK. >> Perhaps instead you should have >> uid: list >> gid: list >> in your uwsgi configuration if you don't already. > > Tried that while also switching mailman3-web.service to list:list, exactly the same error. Also, I have /run/mailman3-web/uwsgi.sock owned by list:list now instead of www-data:www-data. All the mailman processes should run as `list` and all mailman related things should be readable and writable by `list`. >>> I'm not familiar enough with this form of ProxyPass using sockets to >> understand what the localhost:8001 does in this context, but uwsgi >> should not be doing anything with port 8001. In a configuration using >> TCP, it would listen on port 8000. Port 8001 is where Mailman's REST API >> server listens. uwsgi should receive connects in your case via the unix >> socket and then pass them to Django via the application in wsgi.py >> Also, you may want other paths proxied to uwsgi, namely at least some of >> hyperkitty, postorius, archives, accounts, admin and user-profile. > > Ok, I removed that port again, now it's exctly like in the automatically generated config file. > And you mean like > > <IfModule mod_proxy_uwsgi.c> > ProxyPass /mailman3/favicon.ico ! > ProxyPass /mailman3/static ! > ProxyPass /mailman3 unix:/run/mailman3-web/uwsgi.sock|uwsgi://localhost/ > ProxyPass /hyperkitty unix:/run/mailman3-web/uwsgi.sock|uwsgi://localhost/ > ProxyPass /postorius unix:/run/mailman3-web/uwsgi.sock|uwsgi://localhost/ > ProxyPass /admin unix:/run/mailman3-web/uwsgi.sock|uwsgi://localhost/ > ProxyPass /archives unix:/run/mailman3-web/uwsgi.sock|uwsgi://localhost/ > ProxyPass /accounts unix:/run/mailman3-web/uwsgi.sock|uwsgi://localhost/ > ProxyPass /user-profile unix:/run/mailman3-web/uwsgi.sock|uwsgi://localhost/ > </IfModule> Yes, I meant if you want those URLs to work, you have to pass them to uwsgi. You don't need all of them. `mailman2` and `postorius` are synonyms as are `archives` and `hyperkitty` However, you could try using TCP to communicate with uwsgi rather than a Unix socket. This means replacing uwsgi-socket = /run/mailman3-web/uwsgi.sock with uwsgi-socket = 0.0.0.0:8000 in your uwsgi.ini file and using directives like ProxyPass "/postorius" "http://127.0.0.1:8000/postorius" ProxyPass "/hyperkitty" "http://127.0.0.1:8000/hyperkitty" ProxyPass "/accounts" "http://127.0.0.1:8000/accounts" ProxyPass "/django" "http://127.0.0.1:8000/django" ProxyPass "/user-profile" "http://127.0.0.1:8000/user-profile" ProxyPass "/mailman3" "http://127.0.0.1:8000/mailman3" ProxyPass "/archives" "http://127.0.0.1:8000/archives" in apache. I am not familiar enough with apache/uwsgi configuration to be of more help. Have you seen https://wiki.list.org/x/12812344 and have you asked Debian for help with their package. They are the ones who should be supporting it <https://www.debian.org/support>. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

4 years, 1 month

Search results for query "sapiro"