Search results for "sapiro" - lists.mailman3.org

[MM3-users] Re: digests are created but not sent out

by "Jörg M. Schulz"

When I remove the offending special character from the footer, digests are sent out perfectly. So, for international use, we have to wait for the language template system to be ready … By the way: Thank you for your continuing work with this marvellous piece of software. > Am 09.01.2019 um 19:16 schrieb Mark Sapiro <mark(a)msapiro.net>: > > On 1/9/19 9:33 AM, "Jörg M. Schulz" wrote: >> But, also in mailman log. >> Uncaught runner exception: 'ascii' codec can't encode character '\xdf' in position 157: ordinal not in range(128) >> Jan 09 12:24:06 2019 (17491) Traceback (most recent call last): >> File "/usr/lib/python3/dist-packages/mailman/core/runner.py", line 173, in _one_iteration >> self._process_one_file(msg, msgdata) >> File "/usr/lib/python3/dist-packages/mailman/core/runner.py", line 266, in _process_one_file >> keepqueued = self._dispose(mlist, msg, msgdata) >> File "/usr/lib/python3/dist-packages/mailman/runners/digest.py", line 324, in _dispose >> mime = mime_digest.finish() >> File "/usr/lib/python3/dist-packages/mailman/runners/digest.py", line 185, in finish >> footer = MIMEText(footer_text.encode(self._charset), >> so the footers may not contain special characters. >> Will review this one. > > > in the above, self._charset is the character set of the lists > preferred_language. The footer can't contain characters not in the > character set of the list's preferred_language. For English (en) this is > us-ascii. For German (de) it is iso-8859-1. These are artifacts from > Mailman 2.1 > > You can change this for English by putting > > [language.en] > charset: utf-8 > > in your mailman.cfg. This works because us-ascii is a proper subset of > utf-8. > > For other languages, a change like this might require recoding of > templates and/or a message catalog, but since we don't currently ship > any non-English templates or message catalogs, that is moot at the moment. > > -- > Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, > San Francisco Bay Area, California better use your sense - B. Dylan > _______________________________________________ > Mailman-users mailing list -- mailman-users(a)mailman3.org > To unsubscribe send an email to mailman-users-leave(a)mailman3.org > https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/

6 years, 8 months

[MM3-users] Re: Hyperkitty on Debian 12

by Markus Grandpré

Dear Mr Sapiro, thank you very much for your help. The required command is: # mailman-web hyperkitty_import \ -l mailman21-migration-test-04(a)mailman.uni-konstanz.de \ /var/mailman21/testlist.mbox Due to the following warnings, I still have to customize Hyperkitty: django_mailman3.MailDomain: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'. HINT: Configure the DEFAULT_AUTO_FIELD setting or the DjangoMailman3Config.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'. ... Can you give me another hint? As Linux admin I installed the packages mailman and mailman-web that were offered to me under Debian 12. Although during the installation of the packages user "list" was created, I cannot use it: # su - list This account is currently not available As with Mailman 2.1, I will probably have to continue working with Mailman3 as root. Too bad that there is no documentation for Debian admins like me, who don't want to dive too deep into the python framework. Or have I overlooked something? Best regards, Markus Am 25.03.24 um 18:56 schrieb Mark Sapiro: > On 3/25/24 07:58, Markus Grandpre wrote: >> Dear list members, >> >> when following the instructions to migrate list archives from >> Mailman21 to Mailman3 (see [1]), I face the following problem: >> >> # python3 manage.py hyperkitty_import -l mailman21-migration-test-04 >> /var/mailman21/testlist.mbox >> python3: can't open file '/var/mailman21/manage.py': [Errno 2] No such >> file or directory >> >> and have no clue, where to find the right <manage.py> file. The command > > > This depends on how you installed Mailman 3. You may or may not want > `python3 manage.py` at all. The required command could be something like > `django-admin` or `mailman-web`. > > Also the `#` prompt indicates you may be running as root. Mailman > related commands should be run as the Mailman user (could be `mailman` > or `list` or ... depending again on how you installed Mailman). Running > as root can create files owned by root that later can't be read by the > Mailman user. > -- Markus Ludwig Grandpré Universität Konstanz Kommunikations-, Informations-, Medienzentrum (KIM) Abteilung IT-Dienste Forschung und Lehre, B803, Tel: ++49 7531 88 4342

1 year, 5 months

[MM3-users] Re: config incoming email in my domain cpanel

by Rafael Mora

Hello Mark, Ok I Will ask hostgator if they provide mailman. Thank you very much for all your help! On Thu, Aug 3, 2017, 18:12 Mark Sapiro <mark(a)msapiro.net> wrote: > On 08/03/2017 02:14 PM, Rafael Mora wrote: > > El jue., 3 ago. 2017 a las 16:07, tlhackque via Mailman-users (< > > mailman-users(a)mailman3.org>) escribió: > > > >> You've said you're working with labbrands.com. > >> > >> labbrands.com has address 192.185.51.89 > >> mail.labbrands.com has address 192.185.51.89 > >> www.labbrands.com is an alias for labbrands.com. > >> > >> telnet proves that Exim is running on that IP address. Either you're > >> running it, or hostgator is redirecting the SMTP ports. > >> > >> You added: > >> lists.labbrands.com has address 190.145.27.66 > >> > >> Presumably that's a different machine. > >> > >> So that now needs an MX record, and a firewall routing. > > > > > > I added the MX record on our hostgator hosting like this: > > [image: image.png] > > > > Is it correct??? > > > Your screen shot, along with the excessive quoting, only caused your > post to be "too big". > > All the screen shot says is you have two MX records. The higher > priority, 0, is mail.labbrands.com and the lower priority , 1, is > lists.labbrands.com. > > What this means is mail to the domain will be delivered to > mail.labbrands.com unless that server is somehow unreachable in which > case lists.labbrands.com will be tried. > > This is probably not what you want. > > I hate to say this at this point, but you are on a cPanel host. cPanel > provides a functional Mailman 2.1 service. You might save yourself a lot > of grief by just using that. > > -- > Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, > San Francisco Bay Area, California better use your sense - B. Dylan > _______________________________________________ > Mailman-users mailing list > mailman-users(a)mailman3.org > https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ > -- Atentamente / Best Regards Ing. Rafael Mora

8 years, 1 month

[MM3-users] Re: Please help - Held message is crashing mailman.

by Mark Sapiro

On 4/5/19 10:27 PM, Alex King via Mailman-users wrote: > Hi, > > I've used mailman a lot in the past, and decided to install mailman3 in > a recent install. I'm unimpressed so far, it seems buggy. > > Running on Ubuntu 18.04.2 LTS from packages, mailman3 3.1.1-9, > mailman3-web 0+20170523-14, python-django-postorius 1.1.2-3. > > After a few messages, I now can't go to the "Held Messages" in > postorius, I get a 500 error. > > I tried to interact on the command line, but could not work out how. > Reading > https://mailman.readthedocs.io/en/latest/src/mailman/docs/install.html > and > https://mailman.readthedocs.io/en/latest/src/mailman/commands/docs/shell.ht…, > I tried: > > # mailman shell > Welcome to the GNU Mailman shell > >>>> command = cli('mailman.commands.cli_withlist.shell') > Traceback (most recent call last): > File "/usr/lib/python3.6/code.py", line 91, in runcode > exec(code, self.locals) > File "<console>", line 1, in <module> > NameError: name 'cli' is not defined > > I did a lot of reading of the manual and googled, but I have no idea > where the cli object is supposed to come from.....? The documentation > is confusing. The line command = cli('mailman.commands.cli_withlist.shell') in <https://mailman.readthedocs.io/en/latest/src/mailman/commands/docs/shell.ht…> is dependent upon some behind the scenes setup to make the doctests work. Granted this is confusing, but basically all that does is set things up for the doctests to run the 'mailman shell' command. It is not intended to be run from within 'mailman shell'. > Anyway, after some trial and error, i found: > > # mailman shell > Welcome to the GNU Mailman shell >>>> list_manager = getUtility(IListManager) >>>> m=list_manager.get("committee(a)[redacted].org") >>>> from mailman.interfaces.requests import IListRequests >>>> requests = IListRequests(m) >>>> [x for x in requests.held_requests][0].id > 8 >>>> requests.get_request(8) > Traceback (most recent call last): > File "/usr/lib/python3.6/code.py", line 91, in runcode > exec(code, self.locals) > File "<console>", line 1, in <module> > File "/usr/lib/python3/dist-packages/mailman/database/transaction.py", > line 85, in wrapper > return function(args[0], config.db.store, *args[1:], **kws) > File "/usr/lib/python3/dist-packages/mailman/model/requests.py", line > 120, in get_request > result.data_hash, expunge=False) > File "/usr/lib/python3/dist-packages/mailman/database/transaction.py", > line 85, in wrapper > return function(args[0], config.db.store, *args[1:], **kws) > File "/usr/lib/python3/dist-packages/mailman/model/pending.py", line > 138, in confirm > value = json.loads(keyvalue.value) > File "/usr/lib/python3.6/json/__init__.py", line 354, in loads > return _default_decoder.decode(s) > File "/usr/lib/python3.6/json/decoder.py", line 339, in decode > obj, end = self.raw_decode(s, idx=_w(s, 0).end()) > File "/usr/lib/python3.6/json/decoder.py", line 355, in raw_decode > obj, end = self.scan_once(s, idx) > json.decoder.JSONDecodeError: Unterminated string starting at: line 1 > column 1 (char 0) > > It seems something was written out to disk that can't be read in for > some reason (assuming a json object on disk?) This is <https://gitlab.com/mailman/mailman/issues/385> which should be fixed by <https://gitlab.com/mailman/mailman/merge_requests/333>. The fix in in Mailman core version 3.2.0 > How do I debug this further? How do I find the json being decoded? Any > help would be appreciated. (I found mailman2 just worked, I was happy > with that.) The issue is in storing a pickled representation of the json encoding of 'rule_misses'. This gets very long if you have lots of header filters, and with at least the mysql backend gets truncated in the database which causes the Unterminated string error. I'm not sure what your options are for upgrading. The current Debian packages are all >= 3.2.0 [1], but Ubuntu doesn't get 3.2.0 until cosmic (18.10) [2]. [1] https://packages.debian.org/search?keywords=mailman3 [2] https://packages.ubuntu.com/search?keywords=mailman3 > (My colleague posted a bug about this I believe but I don't have the > link to that.) I haven't seen it. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

6 years, 5 months

[MM3-users] Re: [EXTERNAL] Re: Database deadlock when trying to delete a member.

by Simon Coggins

I’ve been running with this change as you suggested for a few weeks now in our test environment and I’ve not hit any issues. So this fixes my problem. Will this change be added to mailman core or should I look at manually patching during my upgrades? -Simon From: Mark Sapiro <mark(a)msapiro.net> Date: Wednesday, 24 March 2021 at 4:10 am To: "mailman-users(a)mailman3.org" <mailman-users(a)mailman3.org> Subject: [MM3-users] Re: [EXTERNAL] Re: Database deadlock when trying to delete a member. On 3/22/21 9:53 PM, Simon Coggins wrote: > Ok after bit tracking I’ve worked out that the bouncer is dangling transactions when it’s sending warning messages. When https://gitlab.com/mailman/mailman/-/blob/3.3.4/src/mailman/model/bounce.py…<https://gitlab.com/mailman/mailman/-/blob/3.3.4/src/mailman/model/bounce.py…> is called and it get to update last_warn_sent it does the following in the DB: > > 2021-03-23 14:42:50.622 AEST [11693] LOG: statement: UPDATE member SET last_warning_sent='2021-03-23T04:42:50.622015'::timestamp, total_warnings_sent=1 WHERE member.id = 169331 > > But it never COMMITs this transaction. So all write operations are now hung waiting for the transction to be closed as the member record has been modified. As a test I added config.db.commit() after https://gitlab.com/mailman/mailman/-/blob/3.3.4/src/mailman/model/bounce.py…<https://gitlab.com/mailman/mailman/-/blob/3.3.4/src/mailman/model/bounce.py…> and restarted. And my mailman syncmembers worked perfectly and I didn’t have any hangs it correctly removed everyone it should have. In the database logs I can see the COMMIT. > > I’m pretty sure this is not the right spot for a config.db.commit() but hopefully this helps point to what might be going wrong. Thank you for all the research on this. It is extremely helpful. Instead of adding a commit after line 277 please try adding the @transactional decorator to the send_warnings_and_remove method at line 210 @transactional @dbconnection def send_warnings_and_remove(self, store): I think that is the proper fix. Let us know if that works. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan _______________________________________________ Mailman-users mailing list -- mailman-users(a)mailman3.org To unsubscribe send an email to mailman-users-leave(a)mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/<https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org>

4 years, 5 months

[MM3-users] Re: Single-Sign-out for List member

by Abhilash Raj

On Thu, Aug 13, 2020, at 10:04 AM, Mark Sapiro wrote: > On 8/13/20 6:21 AM, w.schoener(a)alp.dillingen.de wrote: > > > > So I would like to ad a link in the footer of every message, with which every receiver/member can sign out of the newsletter with one click. > > It does not seam to be possible, to put this into effect with mailman 3. > > Does anybody have any hints, how to create a link, and when this link is called, the user is signed out of the newsletter. > > > You can set the list's unsubscription_policy to open and set the link as > a mailto: to the list-leave address, but that still requires a second > click to send the mail. > > The solution is to create a web based CGI script which is called with a > URL like > <https://example.com/unsub_script/list.example.com/user@example.net> and > have that script invoke the appropriate `mailman` command to remove the > member from the list. Beginning with Mailman 3.3.2, this would be > > mailman delmembers -l list.example.com -m user(a)example.net -g > > to remove the member and send a goodbye message. In older Mailman, it > would be something like > > echo user(a)example.net | mailman members -x - list.example.com > > I understand you may have a legal requirement do do this, but these > "one-click - no user confirmation" links are a bad idea because usera > will reply to a list post or forward it and not remove the link and > userb will receive usera's reply and click usera's unsubscribe link > either maliciously or thinking it will unsubscribe userb. There is also the issue that these days some Mail clients will try to create a preview of all the URLs if they can and it would inadvertently GET the URL resulting in the un-subscription. > > -- > Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, > San Francisco Bay Area, California better use your sense - B. Dylan > _______________________________________________ > Mailman-users mailing list -- mailman-users(a)mailman3.org > To unsubscribe send an email to mailman-users-leave(a)mailman3.org > https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ > -- thanks, Abhilash Raj (maxking)

5 years

[MM3-users] Re: Trouble Installing Mailman3

by Mark Sapiro

On 6/12/25 12:44, Henry Hartley via Mailman-users wrote: > > Here's what I mean when I say I'm doing this "in venv" > > I've become the mailman user: sudo su - mailman > The .bashrc for that user includes: source /opt/mailman/venv/bin/activate > The prompt has (venv) at the left end of it: (venv) mailman@wz-prd-tpmd-1:~$ So your venv is active. That is good. > All pip commands have been run in this state. I did first install setuptools with apt install python3-setuptools and that installed version 68.1.2-2ubuntu1.2. Which installed it in some /usr/<maybe local here>/lib/pythonxx/dist-packages or maybe site-packages directory which is probably not goint to be consulted by the python in your venv. > Note that I'm still not sure the firewall is completely out of the picture, since I do have to include the --trusted-host and --proxy options. Some things install, though, so I think that's OK. > > When I do this at the (venv) prompt, asking to install both setuptools and mailman-web: > > pip install --proxy http://webproxyta.westat.com:3128 --trusted-host pypi.org setuptools mailman-web > > The first line of the output is: > > Requirement already satisfied: setuptools in ./venv/lib/python3.12/site-packages (80.9.0) So setuptools 80.9.0 is installed in your venv which is good. > And that's followed by downloading (or using a cached version) of 11 items before I get the subprocess error. > > (venv) mailman@wz-prd-tpmd-1:~$ /opt/mailman/venv/bin/pip install --proxy http://webproxyta.westat.com:3128 --trusted-host pypi.org setuptools mailman-web > Requirement already satisfied: setuptools in ./venv/lib/python3.12/site-packages (80.9.0) > Collecting mailman-web > Downloading mailman_web-0.0.9-py3-none-any.whl.metadata (3.2 kB) > Collecting django<4.3,>=4.0 (from mailman-web) > Downloading django-4.2.23-py3-none-any.whl.metadata (4.2 kB) > Collecting hyperkitty (from mailman-web) > Downloading hyperkitty-1.3.12-py3-none-any.whl.metadata (3.6 kB) > Collecting postorius (from mailman-web) > Downloading postorius-1.3.13-py3-none-any.whl.metadata (2.9 kB) > Collecting whoosh (from mailman-web) > Downloading Whoosh-2.7.4-py2.py3-none-any.whl.metadata (3.1 kB) > Collecting asgiref<4,>=3.6.0 (from django<4.3,>=4.0->mailman-web) > Downloading asgiref-3.8.1-py3-none-any.whl.metadata (9.3 kB) > Collecting sqlparse>=0.3.1 (from django<4.3,>=4.0->mailman-web) > Downloading sqlparse-0.5.3-py3-none-any.whl.metadata (3.9 kB) > Collecting django-compressor>=1.3 (from hyperkitty->mailman-web) > Downloading django_compressor-4.5.1-py2.py3-none-any.whl.metadata (5.0 kB) > Collecting django-extensions>=1.3.7 (from hyperkitty->mailman-web) > Downloading django_extensions-4.1-py3-none-any.whl.metadata (6.1 kB) > Collecting django-gravatar2>=1.0.6 (from hyperkitty->mailman-web) > Downloading django_gravatar2-1.4.5-py2.py3-none-any.whl.metadata (4.3 kB) > Collecting django-haystack>=2.8.0 (from hyperkitty->mailman-web) > Downloading django_haystack-3.3.0.tar.gz (467 kB) > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 467.3/467.3 kB 22.3 MB/s eta 0:00:00 The above is collecting mailman-web and its dependencies. > Installing build dependencies ... error > error: subprocess-exited-with-error > > × pip subprocess to install build dependencies did not run successfully. > │ exit code: 1 > ╰─> [2 lines of output] > ERROR: Could not find a version that satisfies the requirement setuptools>=61.2 (from versions: none) > ERROR: No matching distribution found for setuptools>=61.2 > [end of output] > > note: This error originates from a subprocess, and is likely not a problem with pip. > error: subprocess-exited-with-error > > × pip subprocess to install build dependencies did not run successfully. > │ exit code: 1 > ╰─> See above for output. > > note: This error originates from a subprocess, and is likely not a problem with pip. And this is all from a pip subprocess which is trying to build install build dependencies. I.e., this error is not necessarily related to django-haystack or any other specific package, but is saying it can't find a setuptools package to satisfy the build requirements. This in turn may be that the subprocess doesn't see your --proxy option. I think the issue may be that normally pip installs build requirements in an isolated temporary directory so as not to create potential conflicts with installed packages. See <https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#build-…>. Specifying --no-build-isolation on the pip command will disable this. See <https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-no-build-isolation>. If you specify --no-build-isolation on the pip command, it should use setuptools and wheel from your venv. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

2 months, 4 weeks

[MM3-users] Re: About to Install MM3 -- Seeking clarification

by Onyeibo Oku

On Wed, 12 Oct 2022 17:52:47 -0700 Mark Sapiro <mark(a)msapiro.net> wrote: Thanks Eggert and Sapiro. > > As Eggert suggests, see > https://docs.mailman3.org/en/latest/install/virtualenv.html and > follow that. I already started out on the above route. Its an adaptation because I am running Fedora. I had no problem with system-wide installations via dnf package management. However, some parts of the documentation does not add up. > You don't need to download the source from GitLab. > Following that doc will get the latest releases from PyPI via pip. > Those are still somewhat out of date - we really should make new > releases - so if you want the latest from GitLab, there are a couple > of ways to do it. Perhaps the easiest is to just run the various `pip > install` commands as documented and then upgrade Mailman core in your > venv with ``` > pip install --upgrade git+https://gitlab.com/mailman/mailman@master > ``` > and update the various mailman web packages with > ``` > pip install --upgrade git+https://gitlab.com/mailman/xxx@master > ``` > where xxx is mailman-web, mailman-hyperkitty, django-mailman3, > postorius, hyperkitty. I.e., run it 5 times, once for each value of > xxx. > I will come to this eventually (assuming that does not break things later). I aim to get it all working with regular pip packages first. I have installed mailman core already. Fedora expects mailman files to reside at /usr/lib/mailman. It appears to ship with a mailman unix user. I adapted to those system defaults. Now I observe that "su - mailman" gives me the proper prompt but .bashrc doesn't kick in. I still have to manually change to the virtual environment despite setting the automatic change in .bashrc. However, "su mailman" runs .bashrc but the prompt says "bash X.XX $". Is this normal? Subsequent parts of the doc show the path "/opt/mailman/mm/var/*" which should correspond to "/usr/lib/mailman/mm/var/*" on my instance. However, I don't see any instructions on how to get the "/mm" folder and the associated sub-folders. I just have "/usr/lib/mailman" I am lost "pip list" within the venv environment confirms that I have "mailman 3.3.5". What am I missing? Regards Onyeibo

2 years, 10 months

[MM3-users] Re: How to fix lists with characters errors?

by Helio Loureiro

Hi, it is python 3.10. Problem is... some mailing lists were created long ago, with python2 something. Most probably 2.5. And the tables on mysql were set to latin1. That I know. Question now is how to fix this. Any suggestion? What is the table? I probably can run some script for these specific lists that have the failure, import data as latim-1, and update back to the table as utf-8. Best Regards, Helio Loureiro https://helio.loureiro.eng.br https://github.com/helioloureiro https://mastodon.social/@helioloureiro On Fri, 22 Mar 2024 at 20:53, Mark Sapiro <mark(a)msapiro.net> wrote: > On 3/22/24 05:37, Helio Loureiro wrote: > > Hi, > > > > I just found several lists have the issue like this regarding charset: > > > > (venv) mailman@new-mailman3 ~ (v3.1.1)> mailman members > ... > > Traceback (most recent call last): > ... > > line 109, in display_members > > print(f'{address.display_name} <{address.original_email}>', > > UnicodeEncodeError: 'latin-1' codec can't encode character '\u0142' in > > position 9: ordinal not in range(256) > > > There is something in your python that has set the encoding for stdout > to latin1 instead of the default utf-8. E.g. > ``` > $ python3.10 > Python 3.10.9 (main, Dec 11 2022, 12:59:24) [GCC 9.4.0] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>> name = 'abc\u0142def' > >>> print (f'{name}') > abcłdef > >>> import sys > >>> sys.stdout.reconfigure(encoding='latin1') > >>> print (f'{name}') > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > UnicodeEncodeError: 'latin-1' codec can't encode character '\u0142' in > position 3: ordinal not in range(256) > >>> > ``` > > -- > Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, > San Francisco Bay Area, California better use your sense - B. Dylan > > _______________________________________________ > Mailman-users mailing list -- mailman-users(a)mailman3.org > To unsubscribe send an email to mailman-users-leave(a)mailman3.org > https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ > Archived at: > https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message… > > This message sent to helio(a)loureiro.eng.br >

1 year, 5 months

[MM3-users] Re: Speed/progress of update_index_one_list?

by Philip Colmer

We've now completed the indexing of our lists, some of which are quite large (100K+ messages per list). I wanted to share our findings of using Xapian to do this. Firstly, using "-v 2", e.g. "mailman-web update_index_one_list -v 2 <email address>", causes the command to print out the progress being made in batches of 1,000 messages. Secondly, Xapian is single-threaded. It only allows one writer at a time. Therefore, we focussed on speed of the hard drives and, on the advice received from the Xapian list, memory. Since we were using AWS EC2 with EBS storage, I elected to move the EBS storage from gp2 to gp3 - which has a higher base I/O figure - and I selected a r6i.8xlarge as this guaranteed 10Gb throughput to the EBS storage. It also delivered 32 vCPU and 256GB RAM. It was possibly overkill but it did the job (see below) and we've now switched back to a t3a.xlarge. Speed-wise, with the above configuration, the system indexed a list of nearly 85K messages in 46 minutes. I hope that is helpful information to anyone else who finds themselves migrating large Mailman 2 archives. Regards Philip On Fri, 14 Jan 2022 at 17:33, Mark Sapiro <mark(a)msapiro.net> wrote: > On 1/14/22 12:38 AM, Philip Colmer wrote: > > I have now switched to Xapian but, in a way, my original questions still > > stand. Is there a way of monitoring the progress of > > "update_index_one_list"? What can I do to the specification of the server > > to make that process go (much) faster? > > > > HyperKitty does the indexing by calling the Haystack update_index > command. See > > https://django-haystack.readthedocs.io/en/master/management_commands.html#u… > > This command has options such as --verbosity and --workers. I think > verbosity is set from the option provided to update_index_one_list and > if set to 2 will give some progress info. > > workers could increase parallelism but is unconditionally passed by > hyperkitty as 0 > > > -- > Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, > San Francisco Bay Area, California better use your sense - B. Dylan > _______________________________________________ > Mailman-users mailing list -- mailman-users(a)mailman3.org > To unsubscribe send an email to mailman-users-leave(a)mailman3.org > https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ >

3 years, 7 months

Search results for query "sapiro"