I am running into a new error were my php server is reporting in php8.2-fpm.log
FastCGI sent in stderr: “scoreboard: failed to lock (already locked); status: unable to find or access status shared memory”
I have googled this, and have not come up with good answers for what might be causing this. I was running 8.3 without issue until 3-4 days ago. And have now reverted to 8.2, just in case this issue is related to 8.3. It is not. The logs are showing the error for php8.2 and php8.3.
System is php 8.2/8.3, Nginx.
Using sshfs to access remote drives for storage
Web server on an HP DL360 Gen9. 384GB Ram, 2 Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz
At the moment, the reason I am interested in solving this, is that over time (perhaps 2-3 hours), the php processes essentially stop responding, and hang. And the whole server becomes unresponsive.
Ideally, I’d love to get some information about what the error means, and what settings might be related to it so that I can determine why it suddenly started occuring.
Thank you
more details:
nginx error log for the site in question.
2024/07/22 12:20:10 [error] 2675#2675: *1439 FastCGI sent in stderr: “scoreboard: failed to lock (already locked); status: unable to find or access status shared memory” while reading response header from upstream, client: 192.168.16.241, server: X.X.com, request: “GET /status?json&full HTTP/1.1”, upstream: “fastcgi://unix:/var/run/php/php8.2-fpm.sock:”, host: “192.168.16.241”
2024/07/22 12:20:10 [error] 2675#2675: *1470 FastCGI sent in stderr: “scoreboard: failed to lock (already locked); status: unable to find or access status shared memory” while reading response header from upstream, client: 192.168.16.241, server: X.X.com, request: “GET /status?json&full HTTP/1.1”, upstream: “fastcgi://unix:/var/run/php/php8.2-fpm.sock:”, host: “192.168.16.241”
2024/07/22 12:20:10 [error] 2663#2663: *1482 FastCGI sent in stderr: “scoreboard: failed to lock (already locked); status: unable to find or access status shared memory” while reading response header from upstream, client: 192.168.16.241, server: X.X.com, request: “GET /status?json&full HTTP/1.1”, upstream: “fastcgi://unix:/var/run/php/php8.2-fpm.sock:”, host: “192.168.16.241”
2024/07/22 12:20:10 [error] 2663#2663: *1494 FastCGI sent in stderr: “scoreboard: failed to lock (already locked); status: unable to find or access status shared memory” while reading response header from upstream, client: 192.168.16.241, server: X.X.com, request: “GET /status?json&full HTTP/1.1”, upstream: “fastcgi://unix:/var/run/php/php8.2-fpm.sock:”, host: “192.168.16.241”
php8.2-fpm.log shows normal messaging of increasing server children.
I am seeing this in relation to the hung processes:
INFO: task php-fpm8.2:2905 blocked for more than 122 seconds.
[Mon Jul 22 12:46:46 2024] Not tainted 6.8.0-38-generic #38-Ubuntu
[Mon Jul 22 12:46:46 2024] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[Mon Jul 22 12:46:46 2024] task:php-fpm8.2 state:D stack:0 pid:2905 tgid:2905 ppid:1826 flags:0x00004006
[Mon Jul 22 12:46:46 2024] Call Trace:
[Mon Jul 22 12:46:46 2024]
[Mon Jul 22 12:46:46 2024] __schedule+0x27c/0x6b0
[Mon Jul 22 12:46:46 2024] schedule+0x33/0x110
[Mon Jul 22 12:46:46 2024] request_wait_answer+0x1be/0x2a0
[Mon Jul 22 12:46:46 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Jul 22 12:46:46 2024] fuse_simple_request+0x18d/0x2f0
[Mon Jul 22 12:46:46 2024] fuse_flush+0x1d2/0x210
[Mon Jul 22 12:46:46 2024] filp_flush+0x38/0x90
[Mon Jul 22 12:46:46 2024] __x64_sys_close+0x34/0x90
[Mon Jul 22 12:46:46 2024] x64_sys_call+0x19bd/0x25c0
[Mon Jul 22 12:46:46 2024] do_syscall_64+0x7f/0x180
[Mon Jul 22 12:46:46 2024] ? do_syscall_64+0x8c/0x180
[Mon Jul 22 12:46:46 2024] ? irqentry_exit+0x43/0x50
[Mon Jul 22 12:46:46 2024] entry_SYSCALL_64_after_hwframe+0x78/0x80
[Mon Jul 22 12:46:46 2024] RIP: 0033:0x71abb971671c
[Mon Jul 22 12:46:46 2024] RSP: 002b:00007fffbacbd8d0 EFLAGS: 00000293 ORIG_RAX:
0000000000000003
[Mon Jul 22 12:46:46 2024] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 000071abb971671c
[Mon Jul 22 12:46:46 2024] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000007
[Mon Jul 22 12:46:46 2024] RBP: 00007fffbacbd8e0 R08: 0000000000000000 R09: 01fffffffffffffe
[Mon Jul 22 12:46:46 2024] R10: 00005d04ad4182b0 R11: 0000000000000293 R12: 000071abb6ff6900
[Mon Jul 22 12:46:46 2024] R13: 000071abb6e771c0 R14: 0000000000000001 R15: 0000000000000000
[Mon Jul 22 12:46:46 2024]