Source Installation Assistance

newer
HyperKitty async tasks runner:...

older
Cannot start lmtp runner due to...

Brian Carpenter

Oct. 31, 2019

8:11 p.m.

I am trying to install Mailman 3 from source using:

https://docs.mailman3.org/en/latest/index.html

My first question is where is the best place to get assistance or ask questions regarding the installation process? I am not sure if mailman-users is that place anymore.

Thanks, Brian

Show replies by date

Mark Sapiro

October 2019

8:38 p.m.

On 10/31/19 1:11 PM, Brian Carpenter wrote:

...

My first question is where is the best place to get assistance or ask questions regarding the installation process? I am not sure if mailman-users is that place anymore.

Either this list or mailman-developers@python.org. Also see <https://wiki.list.org/x/17891998> documenting some of my installation decisions, although it doesn't cover my dropping Whoosh in favor of ElasticSearch. Whoosh is painfully slow at indexing large archives. ElasticSearch is a lot better, but Xapian may be better still. See the thread at <https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/C...>

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Brian Carpenter

9:07 p.m.

The first thing is my server environment:

Debian 10 Postgres for the database Nginx for the webserver

I was able to create the virtual environment, install Mailman core and I have the Django installed and the project setup. I have Postorius and Hyperkitty setup. Everything is looking good. The problem I am having is when I am trying to migrate the data into the Postgres database. The migration process worked fine with SQLite but I decided to go with Postgres. Here is what I have in the settings.py for the database: DATABASES = { 'default': { 'ENGINE': 'django.db.backends.postgresql_psycopg2', 'NAME': '*************', 'USER': '**********', 'PASSWORD': '************', 'HOST': 'localhost', 'PORT': '', }

When I run python manage.py migrate, I get the following exception:

Applying hyperkitty.0007_allauth_20160808_1604...Traceback (most recent call last): File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 82, in _execute return self.cursor.execute(sql) psycopg2.errors.InFailedSqlTransaction: current transaction is aborted, commands ignored until end of transaction block

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "manage.py", line 10, in <module> execute_from_command_line(sys.argv) File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line utility.execute() File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/core/management/base.py", line 323, in run_from_argv self.execute(*args, **cmd_options) File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/core/management/base.py", line 364, in execute output = self.handle(*args, **options) File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/core/management/base.py", line 83, in wrapped res = handle_func(*args, **kwargs) File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/core/management/commands/migrate.py", line 234, in handle fake_initial=fake_initial, File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/db/migrations/executor.py", line 117, in migrate state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial) File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial) File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/db/migrations/executor.py", line 246, in apply_migration self.record_migration(migration) File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/db/migrations/executor.py", line 261, in record_migration self.recorder.record_applied(migration.app_label, migration.name) File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/db/migrations/recorder.py", line 82, in record_applied self.ensure_schema() File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/db/migrations/recorder.py", line 62, in ensure_schema if self.has_table(): File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/db/migrations/recorder.py", line 56, in has_table return self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor()) File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/db/backends/base/introspection.py", line 48, in table_names return get_names(cursor) File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/db/backends/base/introspection.py", line 43, in get_names return sorted(ti.name for ti in self.get_table_list(cursor) File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/db/backends/postgresql/introspection.py", line 52, in get_table_list """.format('c.relispartition' if self.connection.features.supports_table_partitions else 'FALSE')) File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 99, in execute return super().execute(sql, params) File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/opt/mailman/mailman-web/mailman_venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 82, in _execute return self.cursor.execute(sql) django.db.utils.InternalError: current transaction is aborted, commands ignored until end of transaction block

So at this point I am stuck. Brian

mark.van.holsteijn＠gmail.com

November 2019

2:51 p.m.

I ran into this problem too. It is due to the fact that the migration 0007_allauth_20160808_1604 performs a conditional upgrade if the table social_auth_usersocialauth exists.

The test executes a select 1 from social_auth_usersocialauth, which results in a SQL error and the transaction is marked abort only.

When replaced with: select 1 from information_schema.tables where table_name = 'social_auth_usersocialauth' and to_regclass('social_auth_usersocialauth') is not null it works fine.

actionmystique＠gmail.com

3:37 p.m.

mark.van.holsteijn＠gmail.com I stumbled across the same issue as Brian Carpenter within a slightly different environment: Ubuntu focal 20.04 PostgreSQL 12.1-1

I tried to replace the following line in /usr/lib/python3/dist-packages/hyperkitty/migrations/0007_allauth_20160808_1604.py from python3-django-hyperkitty (1.3.0-1ubuntu1):

        cursor.execute("SELECT 1 from social_auth_usersocialauth")

with

        cursor.execute("SELECT 1 from information_schema.tables where table_name ='social_auth_usersocialauth' and to_regclass('social_auth_usersocialauth') is not null")

then

python3 manage.py makemigrations

leads to a different issue:

        "Traceback (most recent call last):", 
        "  File \"/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 84, in _execute", 
        "    return self.cursor.execute(sql, params)", 
        "psycopg2.errors.UndefinedTable: relation \"social_auth_usersocialauth\" does not exist", 
        "LINE 3:             FROM social_auth_usersocialauth usa", 
        "                         ^", 
        "", 
        "", 
        "The above exception was the direct cause of the following exception:", 
        "", 
        "Traceback (most recent call last):", 
        "  File \"manage.py\", line 10, in &lt;module>", 
        "    execute_from_command_line(sys.argv)", 
        "  File \"/usr/lib/python3/dist-packages/django/core/management/__init__.py\", line 381, in execute_from_command_line", 
        "    utility.execute()", 
        "  File \"/usr/lib/python3/dist-packages/django/core/management/__init__.py\", line 375, in execute", 
        "    self.fetch_command(subcommand).run_from_argv(self.argv)", 
        "  File \"/usr/lib/python3/dist-packages/django/core/management/base.py\", line 323, in run_from_argv", 
        "    self.execute(*args, **cmd_options)", 
        "  File \"/usr/lib/python3/dist-packages/django/core/management/base.py\", line 364, in execute", 
        "    output = self.handle(*args, **options)", 
        "  File \"/usr/lib/python3/dist-packages/django/core/management/base.py\", line 83, in wrapped", 
        "    res = handle_func(*args, **kwargs)", 
        "  File \"/usr/lib/python3/dist-packages/django/core/management/commands/migrate.py\", line 234, in handle", 
        "    fake_initial=fake_initial,", 
        "  File \"/usr/lib/python3/dist-packages/django/db/migrations/executor.py\", line 117, in migrate", 
        "    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)", 
        "  File \"/usr/lib/python3/dist-packages/django/db/migrations/executor.py\", line 147, in _migrate_all_forwards", 
        "    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)", 
        "  File \"/usr/lib/python3/dist-packages/django/db/migrations/executor.py\", line 245, in apply_migration", 
        "    state = migration.apply(state, schema_editor)", 
        "  File \"/usr/lib/python3/dist-packages/django/db/migrations/migration.py\", line 124, in apply", 
        "    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)", 
        "  File \"/usr/lib/python3/dist-packages/django/db/migrations/operations/special.py\", line 190, in database_forwards", 
        "    self.code(from_state.apps, schema_editor)", 
        "  File \"/usr/lib/python3/dist-packages/hyperkitty/migrations/0007_allauth_20160808_1604.py\", line 43, in migrate_social_users", 
        "    \"\"\", (provider_old,))", 
        "  File \"/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 67, in execute", 
        "    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)", 
        "  File \"/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 76, in _execute_with_wrappers", 
        "    return executor(sql, params, many, context)", 
        "  File \"/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 84, in _execute", 
        "    return self.cursor.execute(sql, params)", 
        "  File \"/usr/lib/python3/dist-packages/django/db/utils.py\", line 89, in __exit__", 
        "    raise dj_exc_value.with_traceback(traceback) from exc_value", 
        "  File \"/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 84, in _execute", 
        "    return self.cursor.execute(sql, params)", 
        "django.db.utils.ProgrammingError: relation \"social_auth_usersocialauth\" does not exist", 
        "LINE 3:             FROM social_auth_usersocialauth usa", 
        "                         ^"
    ], 
    "stdout_lines": [
        "No changes detected", 
        "Operations to perform:", 
        "  Apply all migrations: account, admin, auth, contenttypes, django_mailman3, django_q, hyperkitty, postorius, sessions, sites, socialaccount", 
        "Running migrations:", 
        "  Applying contenttypes.0001_initial... OK", 
        "  Applying auth.0001_initial... OK", 
        "  Applying account.0001_initial... OK", 
        "  Applying account.0002_email_max_length... OK", 
        "  Applying admin.0001_initial... OK", 
        "  Applying admin.0002_logentry_remove_auto_add... OK", 
        "  Applying admin.0003_logentry_add_action_flag_choices... OK", 
        "  Applying contenttypes.0002_remove_content_type_name... OK", 
        "  Applying auth.0002_alter_permission_name_max_length... OK", 
        "  Applying auth.0003_alter_user_email_max_length... OK", 
        "  Applying auth.0004_alter_user_username_opts... OK", 
        "  Applying auth.0005_alter_user_last_login_null... OK", 
        "  Applying auth.0006_require_contenttypes_0002... OK", 
        "  Applying auth.0007_alter_validators_add_error_messages... OK", 
        "  Applying auth.0008_alter_user_username_max_length... OK", 
        "  Applying auth.0009_alter_user_last_name_max_length... OK", 
        "  Applying auth.0010_alter_group_name_max_length... OK", 
        "  Applying auth.0011_update_proxy_permissions... OK", 
        "  Applying sites.0001_initial... OK", 
        "  Applying django_mailman3.0001_initial... OK", 
        "  Applying django_mailman3.0002_maildomain... OK", 
        "  Applying django_q.0001_initial... OK", 
        "  Applying django_q.0002_auto_20150630_1624... OK", 
        "  Applying django_q.0003_auto_20150708_1326... OK", 
        "  Applying django_q.0004_auto_20150710_1043... OK", 
        "  Applying django_q.0005_auto_20150718_1506... OK", 
        "  Applying django_q.0006_auto_20150805_1817... OK", 
        "  Applying django_q.0007_ormq... OK", 
        "  Applying django_q.0008_auto_20160224_1026... OK", 
        "  Applying django_q.0009_auto_20171009_0915... OK", 
        "  Applying socialaccount.0001_initial... OK", 
        "  Applying socialaccount.0002_token_max_lengths... OK", 
        "  Applying socialaccount.0003_extra_data_default_dict... OK", 
        "  Applying hyperkitty.0001_initial... OK", 
        "  Applying hyperkitty.0002_auto_20150311_0913... OK", 
        "  Applying hyperkitty.0003_thread_starting_email... OK", 
        "  Applying hyperkitty.0004_archived_date_and_subject... OK", 
        "  Applying hyperkitty.0005_MailingList_list_id... OK", 
        "  Applying hyperkitty.0006_thread_on_delete... OK", 
        "  Applying hyperkitty.0007_allauth_20160808_1604..."
    ]

Mark Sapiro

4:24 p.m.

On 11/29/19 7:37 AM, actionmystique@gmail.com wrote:

...

mark.van.holsteijn＠gmail.com I stumbled across the same issue as Brian Carpenter within a slightly different environment: Ubuntu focal 20.04 PostgreSQL 12.1-1

I tried to replace the following line in /usr/lib/python3/dist-packages/hyperkitty/migrations/0007_allauth_20160808_1604.py from python3-django-hyperkitty (1.3.0-1ubuntu1):
        cursor.execute("SELECT 1 from social_auth_usersocialauth")
with
        cursor.execute("SELECT 1 from information_schema.tables where table_name ='social_auth_usersocialauth' and to_regclass('social_auth_usersocialauth') is not null")
then
python3 manage.py makemigrations
leads to a different issue:

We are working on a proper fix for this issue. See <https://gitlab.com/mailman/hyperkitty/merge_requests/200>.

In the mean time, to work around this issue on a new install, the simplest change is to just delete the second to last line

        migrations.RunPython(migrate_social_users),

from hyperkitty/migrations/0007_allauth_20160808_1604.py

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

actionmystique＠gmail.com

December 2019

11:26 a.m.

I have built python3-django-hyperkitty for Ubuntu/Debian distributions from the gitlab hyperkitty sources with the commit that fixes the issue(266): use an errorless test to determine the existence of the table social_auth_usersocialauth. It passes the test on Ubuntu focal with flying colors:

# python3 manage.py makemigrations
# python3 manage.py migrate
        "No changes detected", 
        "Operations to perform:", 
        "  Apply all migrations: account, admin, auth, contenttypes, django_mailman3, django_q, hyperkitty, postorius, sessions, sites, socialaccount", 
        "Running migrations:", 
        "  Applying contenttypes.0001_initial... OK", 
        "  Applying auth.0001_initial... OK", 
        "  Applying account.0001_initial... OK", 
        "  Applying account.0002_email_max_length... OK", 
        "  Applying admin.0001_initial... OK", 
        "  Applying admin.0002_logentry_remove_auto_add... OK", 
        "  Applying admin.0003_logentry_add_action_flag_choices... OK", 
        "  Applying contenttypes.0002_remove_content_type_name... OK", 
        "  Applying auth.0002_alter_permission_name_max_length... OK", 
        "  Applying auth.0003_alter_user_email_max_length... OK", 
        "  Applying auth.0004_alter_user_username_opts... OK", 
        "  Applying auth.0005_alter_user_last_login_null... OK", 
        "  Applying auth.0006_require_contenttypes_0002... OK", 
        "  Applying auth.0007_alter_validators_add_error_messages... OK", 
        "  Applying auth.0008_alter_user_username_max_length... OK", 
        "  Applying auth.0009_alter_user_last_name_max_length... OK", 
        "  Applying auth.0010_alter_group_name_max_length... OK", 
        "  Applying auth.0011_update_proxy_permissions... OK", 
        "  Applying sites.0001_initial... OK", 
        "  Applying django_mailman3.0001_initial... OK", 
        "  Applying django_mailman3.0002_maildomain... OK", 
        "  Applying django_q.0001_initial... OK", 
        "  Applying django_q.0002_auto_20150630_1624... OK", 
        "  Applying django_q.0003_auto_20150708_1326... OK", 
        "  Applying django_q.0004_auto_20150710_1043... OK", 
        "  Applying django_q.0005_auto_20150718_1506... OK", 
        "  Applying django_q.0006_auto_20150805_1817... OK", 
        "  Applying django_q.0007_ormq... OK", 
        "  Applying django_q.0008_auto_20160224_1026... OK", 
        "  Applying django_q.0009_auto_20171009_0915... OK", 
        "  Applying socialaccount.0001_initial... OK", 
        "  Applying socialaccount.0002_token_max_lengths... OK", 
        "  Applying socialaccount.0003_extra_data_default_dict... OK", 
        "  Applying hyperkitty.0001_initial... OK", 
        "  Applying hyperkitty.0002_auto_20150311_0913... OK", 
        "  Applying hyperkitty.0003_thread_starting_email... OK", 
        "  Applying hyperkitty.0004_archived_date_and_subject... OK", 
        "  Applying hyperkitty.0005_MailingList_list_id... OK", 
        "  Applying hyperkitty.0006_thread_on_delete... OK", 
        "  Applying hyperkitty.0007_allauth_20160808_1604... OK", 
        "  Applying hyperkitty.0008_django_mailman3_profile... OK", 
        "  Applying hyperkitty.0009_duplicate_persona_users... OK", 
        "  Applying hyperkitty.0010_email_sender_name... OK", 
        "  Applying hyperkitty.0011_email_parent... OK", 
        "  Applying hyperkitty.0012_thread_order_null... OK", 
        "  Applying hyperkitty.0013_mailinglist_id_1... OK", 
        "  Applying hyperkitty.0014_mailinglist_id_2... OK", 
        "  Applying hyperkitty.0015_mailinglist_id_3... OK", 
        "  Applying hyperkitty.0016_auto_20180309_0056... OK", 
        "  Applying hyperkitty.0017_file_attachments... OK", 
        "  Applying hyperkitty.0018_threadcategory_color... OK", 
        "  Applying hyperkitty.0019_auto_20190127_null_description... OK", 
        "  Applying hyperkitty.0020_auto_20190907_1927... OK", 
        "  Applying postorius.0001_initial... OK", 
        "  Applying postorius.0002_auto_20160210_0721... OK", 
        "  Applying postorius.0003_drop_addressconfirmationprofile... OK", 
        "  Applying postorius.0004_create_email_template... OK", 
        "  Applying postorius.0005_auto_20180707_1107... OK", 
        "  Applying postorius.0006_auto_20180711_1359... OK", 
        "  Applying postorius.0007_auto_20180712_0536... OK", 
        "  Applying sessions.0001_initial... OK", 
        "  Applying sites.0002_alter_domain_unique... OK", 
        "  Applying socialaccount.0004_openidnonce_openidstore... OK"

Anyone can grab the binary package on my Ubuntu Focal PPA or Debian Bullseye Repository. python3-django-hyperkitty has been built for the latest Ubuntu & Debian development distributions (focal & bullseye), but it should be possible to use them on earlier distributions since there is no specific dependencies.

1888

Age (days ago)

1924

Last active (days ago)

List overview

Download

6 comments

4 participants

participants (4)

actionmystique＠gmail.com
Brian Carpenter
Mark Sapiro
mark.van.holsteijn＠gmail.com