Hirayama, Pat writes:
I finally looked at the host (rather than the logs of each container) and realized that the oom-killer was killing django-admin.
I don't have a django-admin process in my installation as far as I know. (I didn't check to see if some process renames itself 'django-admin' though.) Are you sure that's what got killed?
As a temporary workaround, I've limited memory on the mailman-django-uwsgi container to 4 GiB (though, since I just did that this morning, I won't know until after midnight (presumably) if it worked or not), and set the RestartPolicy to always.
What do you mean by "work"? If you've got a process blowing past 4GB that's going probably going to die of ENOMEM. I hope it doesn't manage to try to allocate 10GB.
I'm thinking that this points to a memory leak or some kind?
I would think not. Something's allocating gobs of memory and it's not getting collected, but I doubt it's a process forgetting to delete garbage. I think it's just a runaway.
I did have to add a bunch of packages to my Dockerfile when I rebuilt the container, since the last build was like 5 years ago, so I suppose it could be that django is just using more memory than it used to, though at the moment, I see that it is around 1 GiB:
I've run a Debian system in 1GB, including the kernel, cloud admin stuff, PostgreSQL, Postfix, nginx, Xapian, and the Mailman suite. It always OOM'd during a Xapian reindexing run. :-) Upped the droplet to 2GB, and it's fine on memory.
I have seen reports that uwsgi systems use a lot more memory than gunicorn systems. I don't have hands on to confirm or analyze why, though. I'm not sure using Whoosh (instead of Xapian, Elastic Search, or SOLR) is a good idea -- I found it to be *extremely* slow on initial indexing of a system with lots of archives migrated from Mailman 2, and I wouldn't be surprised if that uses a lot of memory (since then I have stuck to Xapian, so no confirmation or analysis).
-- GNU Mailman consultant (installation, migration, customization) Sirius Open Source https://www.siriusopensource.com/ Software systems consulting in Europe, North America, and Japan