Clearing webserver cache - Nginx
I am having an ugly-looking https://mm3-lists.kictanet.or.ke/archives/ page because of cached content:
2024/08/05 17:00:31 [error] 694#694: *9562 open()
"/opt/mailman/mm/static/CACHE/css/output.6dab123e4897.css" failed (2: No
such file or directory), client: 172.70.111.39, server:
mm3-lists.kictanet.or.ke, request: "GET
/static/CACHE/css/output.6dab123e4897.css HTTP/2.0", host: "
mm3-lists.kictanet.or.ke", referrer: "
https://mm3-lists.kictanet.or.ke/archives/"
2024/08/05 17:00:31 [error] 694#694: *9564 open()
"/opt/mailman/mm/static/CACHE/css/output.158acc288604.css" failed (2: No
such file or directory), client: 162.158.159.31, server:
mm3-lists.kictanet.or.ke, request: "GET
/static/CACHE/css/output.158acc288604.css HTTP/2.0", host: "
mm3-lists.kictanet.or.ke", referrer: "
https://mm3-lists.kictanet.or.ke/archives/"
2024/08/05 17:00:31 [error] 705#705: *9566 open()
"/opt/mailman/mm/static/CACHE/js/output.3aaa7705d68a.js" failed (2: No such
file or directory), client: 162.158.63.166, server: mm3-lists.kictanet.or.ke,
request: "GET /static/CACHE/js/output.3aaa7705d68a.js HTTP/2.0", host: "
mm3-lists.kictanet.or.ke", referrer: "
https://mm3-lists.kictanet.or.ke/archives/"
That IP address appearing there is not my actual IP, but Cloudflare's, which is acting as proxy for the site.
Having cleared MM3-web cache, is there anything else I can easily do (apart from disabling CF proxying) to help deal with the above issue? Or am I barking the wrong tree?
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
On 8/5/24 07:04, Odhiambo Washington via Mailman-users wrote:
I am having an ugly-looking https://mm3-lists.kictanet.or.ke/archives/ page because of cached content:
2024/08/05 17:00:31 [error] 694#694: *9562 open() "/opt/mailman/mm/static/CACHE/css/output.6dab123e4897.css" failed (2: No such file or directory), client: 172.70.111.39, server: mm3-lists.kictanet.or.ke, request: "GET /static/CACHE/css/output.6dab123e4897.css HTTP/2.0", host: " mm3-lists.kictanet.or.ke", referrer: " https://mm3-lists.kictanet.or.ke/archives/" 2024/08/05 17:00:31 [error] 694#694: *9564 open() ... That IP address appearing there is not my actual IP, but Cloudflare's, which is acting as proxy for the site.
I **think** you may have something like
location /static/ {
alias /opt/mailman/web/static/;
}in nginx on cloudflare, but since cloudflare is not the Mailman server, /opt/mailman/web/static/ doesn't exist there.
I can't be sure on how to configure this without knowing the details of how requests get from cloudflare to mm3-lists.kictanet.or.ke and from there to the wsgi server.
However, if cloudflare just proxies the request for /static/ to the
Mailman server and nginx on that server has the above location
directive, that may do it.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Mon, Aug 5, 2024 at 8:19 PM Mark Sapiro <mark@msapiro.net> wrote:
I am having an ugly-looking https://mm3-lists.kictanet.or.ke/archives/
On 8/5/24 07:04, Odhiambo Washington via Mailman-users wrote: page
because of cached content:
2024/08/05 17:00:31 [error] 694#694: *9562 open() "/opt/mailman/mm/static/CACHE/css/output.6dab123e4897.css" failed (2: No such file or directory), client: 172.70.111.39, server: mm3-lists.kictanet.or.ke, request: "GET /static/CACHE/css/output.6dab123e4897.css HTTP/2.0", host: " mm3-lists.kictanet.or.ke", referrer: " https://mm3-lists.kictanet.or.ke/archives/" 2024/08/05 17:00:31 [error] 694#694: *9564 open() ... That IP address appearing there is not my actual IP, but Cloudflare's, which is acting as proxy for the site.I **think** you may have something like
location /static/ { alias /opt/mailman/web/static/; }in nginx on cloudflare, but since cloudflare is not the Mailman server, /opt/mailman/web/static/ doesn't exist there.
Actually, what I mean is that CF is proxying requests to the host named m3-lists.kictanet.or.ke. I do not have any configuration other than DNS proxying at CF.
I can't be sure on how to configure this without knowing the details of
how requests get from cloudflare to mm3-lists.kictanet.or.ke and from there to the wsgi server.
When you ping m3-lists.kictanet.or.ke, the IP you get is CFs, not the actual IP of the host. So when you talk to m3-lists.kictanet.or.ke via a browser, it's CF IP who you talk to and it in turn talks to the host directly. So the IP address you see in the logs is CF's IP address, not yours, not mine and CF are doing some caching. That's why they are requesting for expired files.
However, if cloudflare just proxies the request for /static/ to the Mailman server and nginx on that server has the above
locationdirective, that may do it.
I suppose it's possible to configure Nginx to NOT ALLOW caching of content from the /static/ location. Being new to Nginx, time to say hi to uncle G again.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
On 8/5/24 10:30, Odhiambo Washington via Mailman-users wrote:
Actually, what I mean is that CF is proxying requests to the host named m3-lists.kictanet.or.ke. I do not have any configuration other than DNS proxying at CF.
dig a mm3-lists.kictanet.or.ke
...
mm3-lists.kictanet.or.ke. 300 IN A 172.67.167.73
mm3-lists.kictanet.or.ke. 300 IN A 104.21.11.217
Both of those are cloudflare IPs, and why 2?
So when you talk to m3-lists.kictanet.or.ke via a browser, it's CF IP who you talk to and it in turn talks to the host directly.
And how is that configured?
So the IP address you see in the logs is CF's IP address, not yours, not mine and CF are doing some caching. That's why they are requesting for expired files.
Why is cloudfare trying to access /opt/mailman/mm/static/CACHE/css/output.6dab123e4897.css locally if all it is doing is proxying the request to the actual mailman server?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Mon, Aug 5, 2024 at 9:34 PM Mark Sapiro <mark@msapiro.net> wrote:
On 8/5/24 10:30, Odhiambo Washington via Mailman-users wrote:
Actually, what I mean is that CF is proxying requests to the host named m3-lists.kictanet.or.ke. I do not have any configuration other than DNS proxying at CF.
dig a mm3-lists.kictanet.or.ke ... mm3-lists.kictanet.or.ke. 300 IN A 172.67.167.73 mm3-lists.kictanet.or.ke. 300 IN A 104.21.11.217Both of those are cloudflare IPs, and why 2?
Yes, those are Cloudflare IPs and I don't know why 2! All I know is that in DNS, I created an A record for the hostname and chose the "proxy" option in the cloudflare DNS management interface.
So when you talk to m3-lists.kictanet.or.ke via a browser, it's CF IP who you talk to and it in turn talks to the host directly.
And how is that configured?
You add the domain - kictanet.or.ke to CF's control panel. They give you TWO name servers. You substitute the NSes your registrar gave to your domain for those two. Subsequently you create A|CNAME|MX\TXT records on the CF's control panel. You decide whether an A or CNAME record you create is proxy-ed or not. If you proxy the record, no one will ever get to know its real IP, but if you don't proxy, then the dig tool above will tell you the real IP address.
So the IP address you see in the logs is CF's IP address, not yours, not
mine and CF are doing some caching. That's why they are requesting for expired files.
Why is cloudfare trying to access /opt/mailman/mm/static/CACHE/css/output.6dab123e4897.css locally if all it is doing is proxying the request to the actual mailman server?
I don't know. However, those IPs you see in the Nginx logs, none belong to me! I have two possible IPs - one domiciled in KE and another domiciled in the US (when I use VPN).
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
On 8/5/24 12:45, Odhiambo Washington via Mailman-users wrote:
Yes, those are Cloudflare IPs and I don't know why 2! All I know is that in DNS, I created an A record for the hostname and chose the "proxy" option in the cloudflare DNS management interface.
OK, I know nothing about how cloudflare proxying works, so ignoring that and assuming it's OK, ... ...
I don't know. However, those IPs you see in the Nginx logs, none belong to me! I have two possible IPs - one domiciled in KE and another domiciled in the US (when I use VPN).
I think I've misunderstood. Is it the case that the messages you report like /opt/mailman/mm/static/CACHE/css/output.6dab123e4897.css" failed (2: No such file or directory) ... are in the nginx log on the mailman server. If so, see https://docs.mailman3.org/en/latest/install/virtualenv.html#nginx-configurat.... Is that what you have?
If not, post your nginx configuration.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Mon, Aug 5, 2024 at 11:55 PM Mark Sapiro <mark@msapiro.net> wrote:
On 8/5/24 12:45, Odhiambo Washington via Mailman-users wrote:
Yes, those are Cloudflare IPs and I don't know why 2! All I know is that in DNS, I created an A record for the hostname and
chose
the "proxy" option in the cloudflare DNS management interface.
OK, I know nothing about how cloudflare proxying works, so ignoring that and assuming it's OK, ... ...
I don't know. However, those IPs you see in the Nginx logs, none belong to me! I have two possible IPs - one domiciled in KE and another domiciled in the US (when I use VPN).
I think I've misunderstood. Is it the case that the messages you report like /opt/mailman/mm/static/CACHE/css/output.6dab123e4897.css" failed (2: No such file or directory) ... are in the nginx log on the mailman server.
That's correct.
If so, see https://docs.mailman3.org/en/latest/install/virtualenv.html#nginx-configurat....
Is that what you have?
Yes.
If not, post your nginx configuration.
In any case, here is my virtualhost config:
server {
if ($host = mm3-lists.kictanet.or.ke) {
return 301 https://$host$request_uri;
listen 80;
server_name mm3-lists.kictanet.or.ke;
return 301 https://mm3-lists.kictanet.or.ke$request_uri;
include snippets/letsencrypt.conf;
}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name mm3-lists.kictanet.or.ke;
ssl_certificate /etc/letsencrypt/live/
mm3-lists.kictanet.or.ke/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/
mm3-lists.kictanet.or.ke/privkey.pem;
ssl_trusted_certificate /etc/letsencrypt/live/
mm3-lists.kictanet.or.ke/chain.pem;
include snippets/ssl.conf;
include snippets/letsencrypt.conf;
access_log /var/log/nginx/mm3-lists_access.log;
error_log /var/log/nginx/mm3-lists_error.log;
location = /favicon.ico {
log_not_found off;
access_log off;
}
location = /robots.txt {
allow all;
log_not_found off;
access_log off;
}
location /static/ {
alias /opt/mailman/mm/static/;
}
location / {
proxy_pass http://127.0.0.1:8010;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
}
}
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
On 8/6/24 00:24, Odhiambo Washington via Mailman-users wrote:
In any case, here is my virtualhost config:
It looks OK.
I suggest you run
mailman-web collectstatic --clear mailman-web compress
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Tue, Aug 6, 2024 at 7:18 PM Mark Sapiro <mark@msapiro.net> wrote:
On 8/6/24 00:24, Odhiambo Washington via Mailman-users wrote:
In any case, here is my virtualhost config:
It looks OK.
I suggest you run
mailman-web collectstatic --clear mailman-web compress
Done.
However, I still see the errors in Nginx logs:
root@eu:~# tail -f /var/log/nginx/mm3-lists_error.log
2024/08/06 18:52:54 [error] 37221#37221: *53355 open()
"/opt/mailman/mm/static/CACHE/css/output.158acc288604.css" failed (2: No
such file or directory), client: 108.162.238.164, server:
mm3-lists.kictanet.or.ke, request: "GET
/static/CACHE/css/output.158acc288604.css HTTP/2.0", host: "
mm3-lists.kictanet.or.ke", referrer: "
https://mm3-lists.kictanet.or.ke/archives/list/kictanet@lists.kictanet.or.ke/message/MU6OAP73MH3HZMHIGHS5YYBKO26MBEU7/
"
2024/08/06 19:11:24 [error] 37221#37221: *54872 open()
"/opt/mailman/mm/static/CACHE/css/output.6dab123e4897.css" failed (2: No
such file or directory), client: 162.158.170.169, server:
mm3-lists.kictanet.or.ke, request: "GET
/static/CACHE/css/output.6dab123e4897.css HTTP/2.0", host: "
mm3-lists.kictanet.or.ke", referrer: "
https://mm3-lists.kictanet.or.ke/archives/list/kictanet@lists.kictanet.or.ke/2014/12/?count=25%27nvOpzp%3B+AND+1=1+OR+%28%3C%27%22%3EiKO%29%29%2C&page=4
"
2024/08/06 19:11:24 [error] 37224#37224: *54875 open()
"/opt/mailman/mm/static/CACHE/css/output.158acc288604.css" failed (2: No
such file or directory), client: 162.158.171.16, server:
mm3-lists.kictanet.or.ke, request: "GET
/static/CACHE/css/output.158acc288604.css HTTP/2.0", host: "
mm3-lists.kictanet.or.ke", referrer: "
https://mm3-lists.kictanet.or.ke/archives/list/kictanet@lists.kictanet.or.ke/2014/12/?count=25%27nvOpzp%3B+AND+1=1+OR+%28%3C%27%22%3EiKO%29%29%2C&page=4
"
2024/08/06 19:12:27 [error] 37228#37228: *54974 open()
"/opt/mailman/mm/static/CACHE/css/output.6dab123e4897.css" failed (2: No
such file or directory), client: 162.158.106.81, server:
mm3-lists.kictanet.or.ke, request: "GET
/static/CACHE/css/output.6dab123e4897.css HTTP/2.0", host: "
mm3-lists.kictanet.or.ke", referrer: "
https://mm3-lists.kictanet.or.ke/archives/list/kictanet@lists.kictanet.or.ke/latest?count=100%27nvOpzp%3B+AND+1=1+OR+%28%3C%27%22%3EiKO%29%29%2C&page=75
"
2024/08/06 19:12:27 [error] 37228#37228: *54975 open()
"/opt/mailman/mm/static/CACHE/css/output.158acc288604.css" failed (2: No
such file or directory), client: 162.158.106.14, server:
mm3-lists.kictanet.or.ke, request: "GET
/static/CACHE/css/output.158acc288604.css HTTP/2.0", host: "
mm3-lists.kictanet.or.ke", referrer: "
https://mm3-lists.kictanet.or.ke/archives/list/kictanet@lists.kictanet.or.ke/latest?count=100%27nvOpzp%3B+AND+1=1+OR+%28%3C%27%22%3EiKO%29%29%2C&page=75
"
2024/08/06 19:27:27 [error] 37228#37228: *56500 open()
"/opt/mailman/mm/static/CACHE/css/output.6dab123e4897.css" failed (2: No
such file or directory), client: 172.70.188.20, server:
mm3-lists.kictanet.or.ke, request: "GET
/static/CACHE/css/output.6dab123e4897.css HTTP/2.0", host: "
mm3-lists.kictanet.or.ke", referrer: "
https://mm3-lists.kictanet.or.ke/archives/"
2024/08/06 19:27:27 [error] 37228#37228: *56501 open()
"/opt/mailman/mm/static/CACHE/css/output.158acc288604.css" failed (2: No
such file or directory), client: 172.70.188.84, server:
mm3-lists.kictanet.or.ke, request: "GET
/static/CACHE/css/output.158acc288604.css HTTP/2.0", host: "
mm3-lists.kictanet.or.ke", referrer: "
https://mm3-lists.kictanet.or.ke/archives/"
2024/08/06 19:29:18 [error] 37221#37221: *56722 open()
"/opt/mailman/mm/static/CACHE/css/output.6dab123e4897.css" failed (2: No
such file or directory), client: 172.68.1.156, server:
mm3-lists.kictanet.or.ke, request: "GET
/static/CACHE/css/output.6dab123e4897.css HTTP/2.0", host: "
mm3-lists.kictanet.or.ke", referrer: "
https://mm3-lists.kictanet.or.ke/archives/"
2024/08/06 19:29:18 [error] 37224#37224: *56724 open()
"/opt/mailman/mm/static/CACHE/css/output.158acc288604.css" failed (2: No
such file or directory), client: 172.68.3.81, server:
mm3-lists.kictanet.or.ke, request: "GET
/static/CACHE/css/output.158acc288604.css HTTP/2.0", host: "
mm3-lists.kictanet.or.ke", referrer: "
https://mm3-lists.kictanet.or.ke/archives/"
2024/08/06 19:29:19 [error] 37221#37221: *56728 open()
"/opt/mailman/mm/static/CACHE/js/output.3aaa7705d68a.js" failed (2: No such
file or directory), client: 172.68.2.67, server: mm3-lists.kictanet.or.ke,
request: "GET /static/CACHE/js/output.3aaa7705d68a.js HTTP/2.0", host: "
mm3-lists.kictanet.or.ke", referrer: "
https://mm3-lists.kictanet.or.ke/archives/"
And the archives page is still br0ken,
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
On 8/6/24 09:33, Odhiambo Washington via Mailman-users wrote:
On Tue, Aug 6, 2024 at 7:18 PM Mark Sapiro <mark@msapiro.net> wrote:
On 8/6/24 00:24, Odhiambo Washington via Mailman-users wrote:
In any case, here is my virtualhost config:
It looks OK.
I suggest you run
mailman-web collectstatic --clear mailman-web compress
Done.
However, I still see the errors in Nginx logs:
root@eu:~# tail -f /var/log/nginx/mm3-lists_error.log 2024/08/06 18:52:54 [error] 37221#37221: *53355 open() "/opt/mailman/mm/static/CACHE/css/output.158acc288604.css" failed (2: No such file or directory), ...
Is this with only one browser? It's a long shot, but clearing browser cache might work.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 8/6/24 14:27, Mark Sapiro wrote:
Is this with only one browser? It's a long shot, but clearing browser cache might work.
Actually, I see the same issues from my browsers, so it's not a browser caching issue. The only thing left is nginx caching, but, while I know very little about nginx, I think it doesn't cache by default and I don't see where in your posted config it's enabled. Perhaps there's more in your config that you didn't post.
If there is a proxy_cache_path directive in your config try
find /the/proxy/cache/path -type f -delete
to clear the cache.
As a last resort, your /opt/mailman/mm/static/CACHE/ directory should
contain a css/ directory with 3 files and a js directory containing one
file and a manifest.json file with a template for the link directives
referencing those files.
The js/ directory is easy as its only one file, so your server is looking for /static/CACHE/js/output.3aaa7705d68a.js and you could create a symlink in the js/ directory from output.3aaa7705d68a.js to the actual file. For the css directory, your server is looking for output.158acc288604.css and output.6dab123e4897.css, and I don't see the third file in the logs you posted, but if I go to pages in your archive that I see as unstyled and look at the stylesheet references, they are to output.44ea6c55e917.css, output.e68c4908b3de.css and output.9efeb5f3d52b.css so you could just symlink those to the actual files, it doesn't matter which as all three are loaded on each page.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Wed, Aug 7, 2024 at 2:08 AM Mark Sapiro <mark@msapiro.net> wrote:
On 8/6/24 14:27, Mark Sapiro wrote:
Is this with only one browser? It's a long shot, but clearing browser cache might work.
Actually, I see the same issues from my browsers, so it's not a browser caching issue. The only thing left is nginx caching, but, while I know very little about nginx, I think it doesn't cache by default and I don't see where in your posted config it's enabled. Perhaps there's more in your config that you didn't post.
There is nothing I didn't post :) I finally found a way to purge the files cached by Cloudflare. It's in their Dashboard. But not before I had migrated everything to apache2! So now I have all my sites configured in both webservers and I can use whichever I like.
Right now I am using apache2, but I am seeing something odd: https://mm3-lists.kictanet.or.ke/archives/ is empty. Is this normal? I am expecting my most active list to appear there or am I overthinking? Like this: https://mm3-lists.kictanet.or.ke/archives/list/kictanet@lists.kictanet.or.ke...
If there is a proxy_cache_path directive in your config try
find /the/proxy/cache/path -type f -delete
to clear the cache.
There isn't. All the caching mess was coming from Cloudflare proxy servers. If you host your DNS with CF, you will realize they proxy/cache your traffic by default unless you turn that off when creating an A record.
As a last resort, your /opt/mailman/mm/static/CACHE/ directory should
contain a css/ directory with 3 files and a js directory containing one file and a manifest.json file with a template for the
linkdirectives referencing those files.
I have seen those files.
The js/ directory is easy as its only one file, so your server is
looking for /static/CACHE/js/output.3aaa7705d68a.js and you could create a symlink in the js/ directory from output.3aaa7705d68a.js to the actual file. For the css directory, your server is looking for output.158acc288604.css and output.6dab123e4897.css, and I don't see the third file in the logs you posted, but if I go to pages in your archive that I see as unstyled and look at the stylesheet references, they are to output.44ea6c55e917.css, output.e68c4908b3de.css and output.9efeb5f3d52b.css so you could just symlink those to the actual files, it doesn't matter which as all three are loaded on each page
I did the symlinks and they helped with nginx, but because I have never had to do that before, I did not like it. That's why I switched to apache2 to see what the effects are and apache2 had/has no problem.
Because I have purged the CF cache, I reverted back to nginx and to see from the logs what they are looking for:
root@eu:/opt/mailman/mm/static/CACHE/js# tail -f
/var/log/nginx/mm3-lists_error.log
2024/08/16 17:19:55 [error] 2403078#2403078: *62 open()
"/usr/share/nginx/html/static/CACHE/css/output.44ea6c55e917.css" failed (2:
No such file or directory), client: 172.70.111.111, server:
mm3-lists.kictanet.or.ke, request: "GET
/static/CACHE/css/output.44ea6c55e917.css HTTP/2.0", host: "
mm3-lists.kictanet.or.ke", referrer: "
https://mm3-lists.kictanet.or.ke/archives/"
^C
root@eu:/opt/mailman/mm/static/CACHE/js# cd ../css/
root@eu:/opt/mailman/mm/static/CACHE/css# ls -al
total 196
drwxr-xr-x 2 mailman mailman 4096 Aug 16 16:55 .
drwxr-xr-x 4 mailman mailman 4096 Mar 21 15:55 ..
-rw-r--r-- 1 mailman mailman 1209 Mar 21 15:55 output.d29798365ddb.css
-rw-r--r-- 1 mailman mailman 569 Mar 21 15:55 output.e68c4908b3de.css
-rw-r--r-- 1 mailman mailman 180904 Mar 21 15:55 output.e69dbb26da3b.css
root@eu:/opt/mailman/mm/static/CACHE/css#
So I am left wondering why nginx is prefixing "usr/share/nginx/html" to /static.
I then decided to specify a root directive in the vhost pointing to /opt/mailman/mm/static, and I saw nginx prefixing that path to the /static/ again:
root@eu:/opt/mailman/mm/static/CACHE/css# cp /dev/null
/var/log/nginx/mm3-lists_error.log
root@eu:/opt/mailman/mm/static/CACHE/css# tail -f
/var/log/nginx/mm3-lists_error.log
2024/08/16 17:28:11 [error] 2404382#2404382: *239 open()
"/opt/mailman/mm/static/static/CACHE/css/output.44ea6c55e917.css" failed
(2: No such file or directory), client: 172.70.111.112, server:
mm3-lists.kictanet.or.ke, request: "GET
/static/CACHE/css/output.44ea6c55e917.css HTTP/2.0", host: "
mm3-lists.kictanet.or.ke", referrer: "
https://mm3-lists.kictanet.or.ke/archives/list/kictanet@lists.kictanet.or.ke/
"
Finally, I changed the root directory to /opt/mailman/mm and this time, I see nginx getting the path correctly, but still looking for the same nonexistent file from above and another file:
root@eu:/opt/mailman/mm/static/CACHE/css# tail -f
/var/log/nginx/mm3-lists_error.log
2024/08/16 17:32:25 [error] 2405076#2405076: *38 open()
"/opt/mailman/mm/static/CACHE/css/output.44ea6c55e917.css" failed (2: No
such file or directory), client: 172.70.111.112, server:
mm3-lists.kictanet.or.ke, request: "GET
/static/CACHE/css/output.44ea6c55e917.css HTTP/2.0", host: "
mm3-lists.kictanet.or.ke", referrer: "
https://mm3-lists.kictanet.or.ke/archives/list/kictanet@lists.kictanet.or.ke/
"
2024/08/16 17:36:17 [error] 2405080#2405080: *841 open()
"/opt/mailman/mm/static/CACHE/css/output.9efeb5f3d52b.css" failed (2: No
such file or directory), client: 108.162.238.56, server:
mm3-lists.kictanet.or.ke, request: "GET
/static/CACHE/css/output.9efeb5f3d52b.css HTTP/2.0", host: "
mm3-lists.kictanet.or.ke", referrer: "
https://mm3-lists.kictanet.or.ke/archives/list/kictanet@lists.kictanet.or.ke/message/YVEMMPACZNRMCFVSAMJBACO5RQKYJ6V2/
"
From the above logs, that IP making requests is not my IP! It's the CF proxies.
I then disabled the A record proxying for the site and accessed it from a browser that I havven't used in months, and there was still some errors: The IP showing is my IP.
root@eu:/opt/mailman/mm/static/CACHE/css# systemctl restart nginx
root@eu:/opt/mailman/mm/static/CACHE/css# tail -f
/var/log/nginx/mm3-lists_error.log
2024/08/16 17:41:11 [error] 2406218#2406218: *267 open()
"/opt/mailman/mm/static/CACHE/css/output.44ea6c55e917.css" failed (2: No
such file or directory), client: 185.199.102.24, server:
mm3-lists.kictanet.or.ke, request: "GET
/static/CACHE/css/output.44ea6c55e917.css HTTP/2.0", host: "
mm3-lists.kictanet.or.ke", referrer: "
https://mm3-lists.kictanet.or.ke/archives/list/kictanet@lists.kictanet.or.ke/
"
2024/08/16 17:41:11 [error] 2406218#2406218: *267 open()
"/opt/mailman/mm/static/CACHE/css/output.9efeb5f3d52b.css" failed (2: No
such file or directory), client: 185.199.102.24, server:
mm3-lists.kictanet.or.ke, request: "GET
/static/CACHE/css/output.9efeb5f3d52b.css HTTP/2.0", host: "
mm3-lists.kictanet.or.ke", referrer: "
https://mm3-lists.kictanet.or.ke/archives/list/kictanet@lists.kictanet.or.ke/
"
2024/08/16 17:41:12 [error] 2406218#2406218: *267 open()
"/opt/mailman/mm/static/CACHE/js/output.3aaa7705d68a.js" failed (2: No such
file or directory), client: 185.199.102.24, server: mm3-lists.kictanet.or.ke,
request: "GET /static/CACHE/js/output.3aaa7705d68a.js HTTP/2.0", host: "
mm3-lists.kictanet.or.ke", referrer: "
https://mm3-lists.kictanet.or.ke/archives/list/kictanet@lists.kictanet.or.ke/
"
And the issue is something about accessing the archives. Not sure if there is something to do for nginx to behave.
Apache does not display any single error.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
On 8/16/24 7:58 AM, Odhiambo Washington via Mailman-users wrote:
Right now I am using apache2, but I am seeing something odd: https://mm3-lists.kictanet.or.ke/archives/ is empty. Is this normal?
No, it should be similar to https://lists.mailman3.org/archives/
So I am left wondering why nginx is prefixing "usr/share/nginx/html" to /static.
Do you have
location /static/ {
alias /opt/mailman/web/static/;
}
in your nginx config?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Wed, Aug 7, 2024 at 2:08 AM Mark Sapiro <mark@msapiro.net> wrote:
On 8/6/24 14:27, Mark Sapiro wrote:
Is this with only one browser? It's a long shot, but clearing browser cache might work.
Actually, I see the same issues from my browsers, so it's not a browser caching issue. The only thing left is nginx caching, but, while I know very little about nginx, I think it doesn't cache by default and I don't see where in your posted config it's enabled. Perhaps there's more in your config that you didn't post.
If there is a proxy_cache_path directive in your config try
find /the/proxy/cache/path -type f -delete
to clear the cache.
As a last resort, your /opt/mailman/mm/static/CACHE/ directory should contain a css/ directory with 3 files and a js directory containing one file and a manifest.json file with a template for the
linkdirectives referencing those files.The js/ directory is easy as its only one file, so your server is looking for /static/CACHE/js/output.3aaa7705d68a.js and you could create a symlink in the js/ directory from output.3aaa7705d68a.js to the actual file. For the css directory, your server is looking for output.158acc288604.css and output.6dab123e4897.css, and I don't see the third file in the logs you posted, but if I go to pages in your archive that I see as unstyled and look at the stylesheet references, they are to output.44ea6c55e917.css, output.e68c4908b3de.css and output.9efeb5f3d52b.css so you could just symlink those to the actual files, it doesn't matter which as all three are loaded on each page.
Could you allow me to revisit this issue since I haven't solved it? First, I bypassed CloudFlare DNS proxying just to ensure I have nothing is coming in between my browser and the web server. That did not help with the issue of the "missing" CSS files.
Ideally, I should not be creating these symlinks, right? On my old server the symlinks weren't necessary.
I have created some symlinks and that appears to help with Apache. I will test with Nginx and see how it goes.
root@eu:/opt/mailman/mm/static/CACHE/css# ls -al
total 196
drwxr-xr-x 2 mailman mailman 4096 Sep 29 10:40 .
drwxr-xr-x 4 mailman mailman 4096 Mar 21 2024 ..
lrwxrwxrwx 1 mailman mailman 23 Sep 29 10:40 output.ac23d5f5127c.css ->
output.d29798365ddb.css
lrwxrwxrwx 1 mailman mailman 23 Sep 29 10:31 output.bacec526ec98.css ->
output.e69dbb26da3b.css
-rw-r--r-- 1 mailman mailman 1209 Mar 21 2024 output.d29798365ddb.css
-rw-r--r-- 1 mailman mailman 569 Mar 21 2024 output.e68c4908b3de.css
-rw-r--r-- 1 mailman mailman 180904 Mar 21 2024 output.e69dbb26da3b.css
root@eu:/opt/mailman/mm/static/CACHE/css#
My question though: What is the source of truth for the files that should be in /opt/mailman/mm/static/CACHE/css/ ?? Maybe that is what I need to focus on. If need be, then I should *fix* the contents of that file so that it doesn't contain the nonexistent files, no?
Lastly, why is it that my https://mm3-lists.kictanet.or.ke/archives/ does not display content similar to what https://lists.mailman3.org/archives/ does? What change should I make in which settings.py?
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
On 9/29/24 00:59, Odhiambo Washington via Mailman-users wrote:
Could you allow me to revisit this issue since I haven't solved it? First, I bypassed CloudFlare DNS proxying just to ensure I have nothing is coming in between my browser and the web server. That did not help with the issue of the "missing" CSS files.
Then you have som other issue in your configuration.
Ideally, I should not be creating these symlinks, right? On my old server the symlinks weren't necessary.
That is correct.
I have created some symlinks and that appears to help with Apache. I will test with Nginx and see how it goes.
root@eu:/opt/mailman/mm/static/CACHE/css# ls -al total 196 drwxr-xr-x 2 mailman mailman 4096 Sep 29 10:40 . drwxr-xr-x 4 mailman mailman 4096 Mar 21 2024 .. lrwxrwxrwx 1 mailman mailman 23 Sep 29 10:40 output.ac23d5f5127c.css -> output.d29798365ddb.css lrwxrwxrwx 1 mailman mailman 23 Sep 29 10:31 output.bacec526ec98.css -> output.e69dbb26da3b.css -rw-r--r-- 1 mailman mailman 1209 Mar 21 2024 output.d29798365ddb.css -rw-r--r-- 1 mailman mailman 569 Mar 21 2024 output.e68c4908b3de.css -rw-r--r-- 1 mailman mailman 180904 Mar 21 2024 output.e69dbb26da3b.css root@eu:/opt/mailman/mm/static/CACHE/css#My question though: What is the source of truth for the files that should be in /opt/mailman/mm/static/CACHE/css/ ??
When you run django-admin collectstatic --clear ... it removes all the
old files from the static/ directory and installs new ones. At this
point static/CACHE/ contains no files. Then when you run django-admin compress it creates three output.*.css files in static/CACHE/css and
one output.*.js file in static/CACHE/js and creates a
static/CACHE/manifest.json file that points to those four files and
which is what Django uses for the paths to put in the relevant <script
...> and <link rel="stylesheet" ...> tags in the pages it serves. I.e.
static/CACHE/manifest.json is the source of truth.
Maybe that is what I need to focus on. If need be, then I should *fix* the contents of that file so that it doesn't contain the nonexistent files, no?
I very much doubt that it does contain the nonexistent files since it is
built by the same django-admin compress that creates the output* files.
Lastly, why is it that my https://mm3-lists.kictanet.or.ke/archives/ does not display content similar to what https://lists.mailman3.org/archives/ does? What change should I make in which settings.py?
This is a clue. I presume you actually have archived lists. There is something very wrong in your configuration if HyperKitty is not finding those lists. Possibly you have more than one WSGI server running and you aren't always proxying to the correct one. That seems far fetched to me, but without seeing all your web server, WSGI server and Django configurations, I have no idea what the issue could be.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (2)
-
Mark Sapiro -
Odhiambo Washington