Search results for "sapiro" - lists.mailman3.org

[MM3-users] Change to content filtering collapse alternatives - was: Re: Mailman shutting down Hyperkitty: Request too large

by Stephen J. Turnbull

Mark Sapiro writes: > That said, the effect on the OP's case, which may be more common, may be > harmful depending on list and Mailman config. > > The original message from the OP (edited) is > ``` > Content-Type: multipart/related; boundary="000000000000da2dcc062cb38aa1" Ouch, that's a mess. First, unless you elided it, it's missing the required type parameter. Despite being required, it's basically an optimization and in most cases will not be much of one. So that's not really a big deal. Second, the Content-IDs are malformed. They are supposed to have the form of message IDs. Also not a big deal, they're close enough for government work. The big problem is I have no idea what the semantics of multipart/related text/plain image/jpeg should be. Throwing away the image data seems like it would be the right thing to do according to RFC 2387, which delegates rendering of the entire multipart/related entity to the root entity. But if that were the desired result of the author of the original message, it's straightforward to achieve by inverting the related and alternative parts. And (at least for image content) I would think that would be a rare intention. I don't see an obvious way to resolve this without checking what popular MUAs do with the various possibilities. So as long as your patch does the expected thing, it's the minimal solution. Steve

5 months

[MM3-users] Re: Weird hourly indexing error

by Mark Dadgar

On May 5, 2020, at 9:21 AM, Mark Sapiro <mark(a)msapiro.net> wrote: > > On 5/5/20 8:55 AM, Mark Dadgar wrote: >> I moved my mail server to a Digital Ocean droplet. I basically installed all the appropriate packages and then rsync’d everything that matters from my metal server up to the cloud instance. Everything more or less works fine. >> >> However, I am now getting this hourly error - see below. > ... >> [ERROR/MainProcess] Failed indexing 1 - 1000 (retry 5/5): Term too long (> 245): XSUBJECT95251413 (pid 36301): Term too long (> 245): XSUBJECT95251413 >> Traceback (most recent call last): >> File "/usr/lib/python3/dist-packages/haystack/management/commands/update_index.py", line 97, in do_update >> backend.update(index, current_qs, commit=commit) >> File "/usr/local/lib/python3.8/dist-packages/xapian_backend.py", line 488, in update >> database.replace_document(document_id, document) >> xapian.InvalidArgumentError: Term too long (> 245): XSUBJECT95251413 > > > This error comes from Xapian. It is due to a (new) message in your > archive that contains a 'word' longer than 245 characters. > > There is a PR at > <https://github.com/notanumber/xapian-haystack/pull/181> that addresses > this plus another suggested fix in that PR's comment thread. Thanks. I guess I wait for them to roll the fix in and the ubuntu maintainers to pick it up. > Note: I got there by googling "xapian term too long" which gave > <https://github.com/notanumber/xapian-haystack/issues/153> as the first > hit and that led me to the above. Yeah, that would have been smarter. - Mark ----- mark(a)pdc-racing.net | 408-348-2878

5 years, 1 month

[MM3-users] Re: Cron Error: <mailman@gw> /opt/mailman/mm/bin/django-admin runjobs monthly

by Odhiambo Washington

On Wed, Feb 1, 2023 at 8:31 PM Mark Sapiro <mark(a)msapiro.net> wrote: > On 1/31/23 23:33, Odhiambo Washington wrote: > > Is this a race condition? > > > > <CUT> > > ERROR OCCURED IN JOB: update_and_clean_index (APP: hyperkitty) > > Traceback (most recent call last): > > File > > > "/opt/mailman/mm/venv/lib/python3.9/site-packages/django_extensions/management/commands/runjobs.py", > > line 40, in runjobs > > job().execute() > > File > > > "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/jobs/update_and_clean_index.py", > > line 37, in execute > > run_with_lock(update_index, remove=True) > > File > > > "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/lib/utils.py", > > line 181, in run_with_lock > > log.exception("Failed to update the fulltext index: %s", e) > > File > > > "/opt/mailman/mm/venv/lib/python3.9/site-packages/flufl/lock/_lockfile.py", > > line 470, in __exit__ > > self.unlock() > > File > > > "/opt/mailman/mm/venv/lib/python3.9/site-packages/flufl/lock/_lockfile.py", > > line 420, in unlock > > raise NotLockedError('Already unlocked') > > flufl.lock._lockfile.NotLockedError: Already unlocked > > <CUT> > > > Yes, it is a race between the monthly hyperkitty update_and_clean_index > job and the hourly hyperkitty update_index job > Okay. This is a bit off from my understanding. What is the cause and how do I mitigate it? -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-)

2 years, 5 months

[MM3-users] Re: Verification emails 'helo' rejected

by Greg Newby

Success! Below: On Wed, Oct 16, 2024 at 10:52 AM Mark Sapiro <mark(a)msapiro.net> wrote: > On 10/14/24 12:42, Greg Newby via Mailman-users wrote: > > > > Here are the corresponding logfile entries from mailmanweb.log: > > smtplib.SMTPRecipientsRefused: {'redacted(a)gmail.com': (504, b'5.5.2 > <mail>: > > Helo command rejected: need fully-qualified hostname')} > > > > and from mail.log: > > Oct 14 12:06:26 domain postfix/smtpd[722764]: NOQUEUE: reject: RCPT from > > localhost[127.0.0.1]: 504 5.5.2 <mail>: Helo command rejected: need > > fully-qualified hostname; from=<root(a)lists.domain.tld> > to=<root@localhost> > > proto=ESMTP helo=<mail> > > > > lists.domain.tld is the domain for Mailman3. I don't know why it's trying > > to send email from root@. > > > I think the mail.log entry is for the DSN for the failure. The issue is > the HELO/EHLO command. Try the following in your venv > ``` > (venv) ...:~$ python > Python 3... > Type "help", "copyright", "credits" or "license" for more information. > >>> import socket > >>> socket.getfqdn() > ``` > If that returns `localhost`, that's the issue. I think you can fix it by > putting a fully qualified domain name in /etc/hostname. > > That was the answer. I had the short hostname, not the fully qualified domain name. Thank you! The logfile messages were pretty confusing since they did not mention the hostname at all. Postfix was not the problem, as you wrote separately. ~ Greg

8 months, 2 weeks

[MM3-users] Re: Hyperkitty search on Message-ID?

by Bryan Fields

On 12/5/24 2:02 PM, Mark Sapiro wrote: > Message-ID is not one of the fields indexed. These are defined at > https://gitlab.com/mailman/hyperkitty/-/blob/master/hyperkitty/search_index… Looks like it would trivial to add there. Think anyone would be opposed to adding support for this? > A bit obscure for laypersons, but one can compute the message_id_hash > from the Message-ID > ``` > from base64 import b32encode > from hashlib import sha1 > > def get_message_id_hash(msg_id): > return b32encode(sha1(msg_id).digest()).decode() > ``` This is great to know. I assume the thread ID is generated the same way? > I can't help but wonder though how one has the Message-ID without having > the message. I'm wondering in general how often if ever the layperson > would be searching the archive for a Message-ID. Well I've been working with an import of 34 years of mail archives (200k messages or so), and we have several people spot checking what we have and fixing threading based on their archive. Since they have the message already, it would be easy to search for the message ID for them. The older archives predate RFC2822 by a decade or so and in-reply-to is missing or just wrong on quite a few of them. From what I can see it was migrated from LISTSERV to majordomo in 1993, then to mailman2 in 2008, and what I have for an archive was from the mail to NNTP spool, at least for the pre-1993 list. Thanks, -- Bryan Fields 727-409-1194 - Voice http://bryanfields.net

6 months, 4 weeks

[MM3-users] Re: Mails are stuck in Mailman Pipeline Queue

by Shashikanth Komandoor

Thank you Mark. This solution worked fine for me. On Sun, Mar 27, 2022 at 9:58 AM Mark Sapiro <mark(a)msapiro.net> wrote: > On 3/26/22 21:01, Shashikanth Komandoor wrote: > > > > But the below trace back is observed during mailman service stopping: > > > This doesn't say much ... > > > ...> Mar 27 09:01:22 2022 (5864) Uncaught runner exception: > > Mar 27 09:01:22 2022 (5864) Traceback (most recent call last): > > File > "/var/lib/mailman/venv/lib64/python3.6/site-packages/mailman/core/runner.py", > > line 173, in _one_iteration > ... > > File > "/var/lib/mailman/venv/lib64/python3.6/site-packages/mailman/core/runner.py", > > line 114, in signal_handler > > raise RunnerInterrupt > > mailman.interfaces.runner.RunnerInterrupt > > Mar 27 09:01:22 2022 (5864) SHUNTING: 1648351882.6693938+ > > c2e6ed709646a7680c73059b52d2f0451836e591 > > Mar 27 09:01:22 2022 (5864) digest runner exiting. > > Above, digest runner was interrupted in processing and shunted the > message it was working on. > > and below, pipeline runner was interrupted in processing and shunted the > message it was working on. > > > Mar 27 09:01:22 2022 (5860) Uncaught runner exception: > > Mar 27 09:01:22 2022 (5860) Traceback (most recent call last): > > File > "/var/lib/mailman/venv/lib64/python3.6/site-packages/mailman/core/runner.py", > > line 173, in _one_iteration > > self._process_one_file(msg, msgdata) > > File > "/var/lib/mailman/venv/lib64/python3.6/site-packages/mailman/core/runner.py", > > line 266, in _process_one_file > > keepqueued = self._dispose(mlist, msg, msgdata) > > File > "/var/lib/mailman/venv/lib64/python3.6/site-packages/mailman/runners/pipeline.py", > > line 37, in _dispose > > process(mlist, msg, msgdata, pipeline) > > File > "/var/lib/mailman/venv/lib64/python3.6/site-packages/mailman/core/pipelines.py", > > line 50, in process > > handler.process(mlist, msg, msgdata) > > File "/var/lib/mailman/venv/lib64/python3.6/site-packages/ > > mailman/handlers/member_recipients.py", line 84, in process > > for member in mlist.regular_members.members > > File "/var/lib/mailman/venv/lib64/python3.6/site-packages/ > > mailman/handlers/member_recipients.py", line 83, in <genexpr> > > recipients = set(member.address.email > > File > "/var/lib/mailman/venv/lib64/python3.6/site-packages/mailman/model/roster.py", > > line 239, in members > > yield from self._get_members(DeliveryMode.regular) > > File > "/var/lib/mailman/venv/lib64/python3.6/site-packages/mailman/model/roster.py", > > line 226, in _get_members > > if member.delivery_mode in delivery_modes: > > File > "/var/lib/mailman/venv/lib64/python3.6/site-packages/mailman/model/member.py", > > line 205, in delivery_mode > > return self._lookup('delivery_mode') > > File > "/var/lib/mailman/venv/lib64/python3.6/site-packages/mailman/model/member.py", > > line 167, in _lookup > > pref = getattr(self.address.preferences, preference) > > File > "/var/lib/mailman/venv/lib64/python3.6/site-packages/sqlalchemy/orm/attributes.py", > > line 287, in __get__ > > return self.impl.get(instance_state(instance), dict_) > > File > "/var/lib/mailman/venv/lib64/python3.6/site-packages/sqlalchemy/orm/attributes.py", > > line 723, in get > > value = self.callable_(state, passive) > > File > "/var/lib/mailman/venv/lib64/python3.6/site-packages/sqlalchemy/orm/strategies.py", > > line 760, in _load_for_state > > session, state, primary_key_identity, passive > > File "<string>", line 1, in <lambda> > > File > "/var/lib/mailman/venv/lib64/python3.6/site-packages/sqlalchemy/orm/strategies.py", > > line 850, in _emit_lazyload > > session.query(self.mapper), primary_key_identity > > File > "/var/lib/mailman/venv/lib64/python3.6/site-packages/sqlalchemy/ext/baked.py", > > line 603, in _load_on_pk_identity > > setup, tuple(elem is None for elem in primary_key_identity) > > The above says the runner was doing a database lookup. This may be a > coincidence or there may be some database locking issue that underlies > the issue with the pipeline runner. > > > > File > "/var/lib/mailman/venv/lib64/python3.6/site-packages/mailman/core/runner.py", > > line 114, in signal_handler > > raise RunnerInterrupt > > mailman.interfaces.runner.RunnerInterrupt > > Mar 27 09:01:22 2022 (5860) SHUNTING: 1648351882.8537986+ > > 9bf59cb23a71a4bae65536bfdbec0abb261adaa0 > > Mar 27 09:01:22 2022 (5860) pipeline runner exiting. > > > I suggest doing the following assuming your Mailman database is not > sqlite3 (if it is, just do step 2): > > 1) Stop mailman core and maybe other mailman services too. > 2) Assuming the two shunted messages are still in Mailman's shunt queue, > run `mailman unshunt` to unshunt them. If there are more shunted > messages, inspect them with `mailman qfile` to see if you want them, > They may be spam. > 3) Restart your database server. > 4) Start Mailman core and any other stopped services. > > Also, look in the database server logs for any issues. > > -- > Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, > San Francisco Bay Area, California better use your sense - B. Dylan > _______________________________________________ > Mailman-users mailing list -- mailman-users(a)mailman3.org > To unsubscribe send an email to mailman-users-leave(a)mailman3.org > https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ > -- Thanks & Regards, Shashi Kanth.K 9052671936

3 years, 3 months

[MM3-users] Re: Sudden increase in "***UNCHECKED***" in email subjects

by Dave Clements

Hi Peter and Mark, Thanks for the insight and fixes. We (we being Nate who is also on this list) think it was a lack of virtual memory on the server. This crashed ClamAV and it could not restart. Thus our mail really was unchecked. Restarting processes and cleaning up some things has fixed this. Cheers, Dave C On Sun, Jul 19, 2020 at 4:07 PM Peter Adebahr via Mailman-users < mailman-users(a)mailman3.org> wrote: > On 7/18/20 7:46 PM, Mark Sapiro wrote: > > On 7/18/20 9:30 AM, Dave Clements wrote: > >> Hi all, > >> > >> I have just noticed a big jump in the number of emails going out with > >> "***UNCHECKED***" added to the subject line. > > > > > > This does not come from Mailman. Some milter or other process in your > > MTA or elsewhere in the delivery chain is adding this to the Subject: > > > > Hi Dave, > > I had experienced the same symptom a while ago. For me, it was amavis > doing this. This behaviour could be corrected by adding this line to > amavisd.conf: > > > # $undecipherable_subject_tag = '[UNCHECKED] '; # undef disables it > $undecipherable_subject_tag = undef; > # > > Hope it helps :) > > Best regards, > Peter > > _______________________________________________ > Mailman-users mailing list -- mailman-users(a)mailman3.org > To unsubscribe send an email to mailman-users-leave(a)mailman3.org > https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ > -- https://bcc2020.github.io/ https://galaxyproject.org/

4 years, 11 months

[MM3-users] Re: 404 on fresh install (Debian11)

by Jeremy Stanley

On 2023-03-23 09:07:02 -0700 (-0700), Mark Sapiro wrote: [...] > bookworm (testing) and sid (unstable) do have current versions of > Mailman core and django-mailman3 and possibly mailman3-web > 0+20200530-2.1 has current Postorius and HyperKitty, but those are > generally not the Debian packages that most users would install. And those will also become out of date over time, because Debian is a stable server distribution so intentionally freezes the versions of software it supplies and backports only critical/security fixes to those increasingly outdated versions when warranted. Basically it's a question of whether users value stability with a bit of security support over newest versions of software and new features. The upstream Mailman packages on PyPI (and container images) cater to users who want the latest versions of the software, packages in stable server distros like Debian cater to users who want to safely go years between behavior changes in software. Both are valid choices, it just comes down to the user's specific situation. That said, when you have trouble with the packaged versions of software in Debian, the expectation is that you ask on Debian mailing lists and possibly report bugs in the Debian bug tracker, rather than burdening upstream software maintainers with details that might be specific to choices or mistakes in Debian's packages. The package maintainers in Debian generally forward bug reports upstream if they determine the issue you've encountered isn't related to how it's been packaged, isn't already fixed upstream (in which case they might backport the fix), et cetera. -- Jeremy Stanley

2 years, 3 months

[MM3-users] Re: Member Issue Discovered

by Stephen J. Turnbull

Victoriano Giralt writes: > El vie, 23-10-2020 a las 08:19 -0700, Mark Sapiro escribió: > > > > This would be more difficult for Postorius/Hyperkitty because those > > users are Django users, but that isn't an issue for Affinity. > > > > All of a sudden, an idea has dawned on me ... Wouldn't it be possible for > Postorius/Hyperkitty, both Django applications, to use Mailman core user > model as their User model? It is just a setting for Django > (AUTH_USER_MODEL). I find the "just a setting" claim a bit implausible. I suspect a certain amount of replumbing of Postorius and HyperKitty will be necessary to get it to work. Most important, the "AUTH_" is pretty scary. Mailman core has *no* authentication built in. It is assumed to be firewalled off from the rest of the world with only access from Postorius and HyperKitty permitted. Can you expand on how hard it would be to do this? Remember that some Mailman instances are either big targets (eg, Apple) or require good security for content reasons (lists for doctors' patient support groups, domestic violence support groups). This is not a great authn/authz model in the modern Internet, but as far as I can see changing it is going to be quite a lot of work, and none of us are authentication specialists (although Abhilash did study security, I don't think he specialized in authentication tech) -- providing reasonably reliable and secure authentication services, especially "social auth" (OpenAuth, OpenID etc) is an important reason we went with Django in the first place. Steve

4 years, 8 months

[MM3-users] Re: hyperkitty threads by month question

by Bryan Fields

On 7/25/24 7:04 PM, Mark Sapiro wrote: > On 7/25/24 11:20, Bryan Fields wrote: >> When this happens, I get an email error below. > > > This is really wierd. See below. > >> line 88, in reverse >> return resolver._reverse_with_prefix(view, prefix, *args, **kwargs) >> File >> "/opt/mailman/venv/lib/python3.10/site-packages/django/urls/resolvers.py", line 828, in _reverse_with_prefix >> raise NoReverseMatch(msg) >> >> Exception Type: NoReverseMatch at >> /archives/list/nanog(a)nanog-mm3.keekles.org/ >> Exception Value: Reverse for 'hk_archives_with_month' with keyword >> arguments '{'year': 999, 'mlist_fqdn': 'nanog(a)nanog-mm3.keekles.org', >> 'month': 12}' not found. 2 pattern(s) tried: >> ['archives/list/(?P<mlist_fqdn>[^/@]+@[^/@]+)/(?P<year>\\d{4})/(?P<month>\\d\\d?)/$', 'hyperkitty/list/(?P<mlist_fqdn>[^/@]+@[^/@]+)/(?P<year>\\d{4})/(?P<month>\\d\\d?)/$'] >> Raised during: hyperkitty.views.mlist.overview > > > It appears to be trying to get the threads for December, 999. > > What do these queries show? > > SELECT * FROM hyperkitty_thread WHERE EXTRACT(year FROM date_active) < 1000; > > SELECT id FROM hyperkitty_email WHERE EXTRACT(year FROM date) < 1000; mailman=> \c mailmanweb You are now connected to database "mailmanweb" as user "mailman". mailmanweb=> SELECT * FROM hyperkitty_thread WHERE EXTRACT(year FROM date_active) < 1000; id | thread_id | date_active | category_id | mailinglist_id | starting_email_id -------+----------------------------------+------------------------+-------------+----------------+------------------- 35670 | GHLCKUC7TDEBAI27SKRQN5PWX3TGB26Q | 0102-04-15 20:06:12+00 | | 3 | 150555 (1 row) mailmanweb=> SELECT id FROM hyperkitty_email WHERE EXTRACT(year FROM date) < 1000; id -------- 136872 150555 (2 rows) Does this mean it imported archives with an invalid date somehow? Thanks, -- Bryan Fields 727-409-1194 - Voice http://bryanfields.net

11 months, 1 week

Search results for query "sapiro"