GNOME 40.4: gvfsd-trash spins on CPU, clogs syslog with "Too many open files"
Opening a new issue based on comment from @oholy in #400 (comment 1393258).
Software Versions
- OS: PopOS 21.10
- Gnome: 40.4.0
- Kernel: 5.15.23-76051523-generic
Additional System Info
- Number of files in trash: zero
- Network mounts: read-only SMB share mounted via cifs
Description
I recently noticed that storage on my root file system had been completely exhausted by massive syslog and journal files (30+ GB for syslog in one instance). The syslog is almost entirely filled with the messages below from gvfsd-trash:
Feb 23 12:17:50 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 17:17:50.575: fail: Error accepting connection: Too many open files
Feb 23 12:17:50 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 17:17:50.575: fail: Error accepting connection: Too many open files
Feb 23 12:17:50 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 17:17:50.576: fail: Error accepting connection: Too many open files
Feb 23 12:17:50 gvfsd[16832]: message repeated 5 times: [ (process:16832): GLib-GIO-WARNING **: 17:17:50.576: fail: Error accepting connection: Too many open files]
Feb 23 12:17:50 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 17:17:50.577: fail: Error accepting connection: Too many open files
Feb 23 12:17:50 gvfsd[16832]: message repeated 4 times: [ (process:16832): GLib-GIO-WARNING **: 17:17:50.577: fail: Error accepting connection: Too many open files]
Feb 23 12:17:50 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 17:17:50.578: fail: Error accepting connection: Too many open files
Feb 23 12:17:50 gvfsd[16832]: message repeated 6 times: [ (process:16832): GLib-GIO-WARNING **: 17:17:50.578: fail: Error accepting connection: Too many open files]
Feb 23 12:17:50 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 17:17:50.579: fail: Error accepting connection: Too many open files
Feb 23 12:17:50 gvfsd[16832]: message repeated 5 times: [ (process:16832): GLib-GIO-WARNING **: 17:17:50.579: fail: Error accepting connection: Too many open files]
Feb 23 12:17:50 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 17:17:50.580: fail: Error accepting connection: Too many open files
Feb 23 12:17:50 gvfsd[16832]: message repeated 5 times: [ (process:16832): GLib-GIO-WARNING **: 17:17:50.580: fail: Error accepting connection: Too many open files]
Feb 23 12:17:50 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 17:17:50.581: fail: Error accepting connection: Too many open files
These messages are pushed to the syslog hundreds of times per second. While these logs are generated, gvfsd-trash runs at 100% on the CPU core it's scheduled on. This, along with the volume of messages, indicates to me that it's stuck in a tight loop trying to open connections with no back off after failure.
Unfortunately while I was in the process of collecting information for this bug report the gvfsd-trash process terminated, preventing me from temporarily grabbing additional information. I did however find several slightly different messages in the syslog from right before it stopped appending new lines:
Feb 23 21:33:46 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 02:33:46.650: fail: Error accepting connection: Too many open files
Feb 23 21:33:46 nautilus[987537]: The peer-to-peer connection failed: Unable to create socket: Too many open files. Falling back to the session bus. Your application is probably missing -
-filesystem=xdg-run/gvfsd privileges.
Feb 23 21:33:46 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 02:33:46.650: fail: Error accepting connection: Too many open files
Feb 23 21:33:46 gvfsd[16832]: message repeated 3 times: [ (process:16832): GLib-GIO-WARNING **: 02:33:46.650: fail: Error accepting connection: Too many open files]
Feb 23 21:33:46 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 02:33:46.651: fail: Error accepting connection: Too many open files
Feb 23 21:33:46 gvfsd[16832]: message repeated 5 times: [ (process:16832): GLib-GIO-WARNING **: 02:33:46.651: fail: Error accepting connection: Too many open files]
Feb 23 21:33:46 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 02:33:46.652: fail: Error accepting connection: Too many open files
Feb 23 21:33:46 gvfsd[16832]: message repeated 3 times: [ (process:16832): GLib-GIO-WARNING **: 02:33:46.652: fail: Error accepting connection: Too many open files]
Feb 23 21:33:46 gvfsd[16832]: daemon: Error creating server at address unix:path=/run/user/1000/gvfsd/socket-z2hdWPWv: Unable to create socket: Too many open files
Feb 23 21:33:46 nautilus[987537]: The peer-to-peer connection failed: Unable to create socket: Too many open files. Falling back to the session bus. Your application is probably missing --filesystem=xdg-run/gvfsd privileges.
Feb 23 21:33:46 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 02:33:46.652: fail: Error accepting connection: Too many open files
Feb 23 21:33:46 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 02:33:46.653: fail: Error accepting connection: Too many open files
Feb 23 21:33:46 gvfsd[16832]: message repeated 5 times: [ (process:16832): GLib-GIO-WARNING **: 02:33:46.653: fail: Error accepting connection: Too many open files]
Feb 23 21:33:46 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 02:33:46.654: fail: Error accepting connection: Too many open files
Feb 23 21:33:46 gvfsd[16832]: message repeated 5 times: [ (process:16832): GLib-GIO-WARNING **: 02:33:46.654: fail: Error accepting connection: Too many open files]
Feb 23 21:33:46 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 02:33:46.655: fail: Error accepting connection: Too many open files
Feb 23 21:33:46 gvfsd[16832]: message repeated 5 times: [ (process:16832): GLib-GIO-WARNING **: 02:33:46.655: fail: Error accepting connection: Too many open files]
Feb 23 21:33:46 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 02:33:46.656: fail: Error accepting connection: Too many open files
Feb 23 21:33:46 gvfsd[16832]: message repeated 5 times: [ (process:16832): GLib-GIO-WARNING **: 02:33:46.656: fail: Error accepting connection: Too many open files]
Feb 23 21:33:46 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 02:33:46.657: fail: Error accepting connection: Too many open files
Feb 23 21:33:46 gvfsd[16832]: message repeated 5 times: [ (process:16832): GLib-GIO-WARNING **: 02:33:46.657: fail: Error accepting connection: Too many open files]
Feb 23 21:33:46 gvfsd[16832]: daemon: Error creating server at address unix:path=/run/user/1000/gvfsd/socket-ORCj1eWS: Unable to create socket: Too many open files
Feb 23 21:33:46 gvfsd[16832]: daemon: Error creating server at address unix:path=/run/user/1000/gvfsd/socket-Klv9gIsw: Unable to create socket: Too many open files
Feb 23 21:33:46 gvfsd[16832]: daemon: Error creating server at address unix:path=/run/user/1000/gvfsd/socket-pRH4E5t4: Unable to create socket: Too many open files
Feb 23 21:33:46 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 02:33:46.658: fail: Error accepting connection: Too many open files
Feb 23 21:33:46 gvfsd[16832]: message repeated 4 times: [ (process:16832): GLib-GIO-WARNING **: 02:33:46.658: fail: Error accepting connection: Too many open files]
Feb 23 21:33:46 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 02:33:46.659: fail: Error accepting connection: Too many open files
Feb 23 21:33:46 gvfsd[16832]: message repeated 5 times: [ (process:16832): GLib-GIO-WARNING **: 02:33:46.659: fail: Error accepting connection: Too many open files]
Feb 23 21:33:46 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 02:33:46.660: fail: Error accepting connection: Too many open files
Feb 23 21:33:46 gvfsd[16832]: message repeated 5 times: [ (process:16832): GLib-GIO-WARNING **: 02:33:46.660: fail: Error accepting connection: Too many open files]
Feb 23 21:33:46 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 02:33:46.661: fail: Error accepting connection: Too many open files
Feb 23 21:33:46 gvfsd[16832]: message repeated 5 times: [ (process:16832): GLib-GIO-WARNING **: 02:33:46.661: fail: Error accepting connection: Too many open files]
Feb 23 21:33:46 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 02:33:46.662: fail: Error accepting connection: Too many open files
Feb 23 21:33:46 gvfsd[16832]: message repeated 3 times: [ (process:16832): GLib-GIO-WARNING **: 02:33:46.662: fail: Error accepting connection: Too many open files]
Feb 23 21:33:46 nautilus[987537]: The peer-to-peer connection failed: Unable to create socket: Too many open files. Falling back to the session bus. Your application is probably missing --filesystem=xdg-run/gvfsd privileges.
Feb 23 21:33:46 nautilus[987537]: The peer-to-peer connection failed: Unable to create socket: Too many open files. Falling back to the session bus. Your application is probably missing --filesystem=xdg-run/gvfsd privileges.
Feb 23 21:33:46 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 02:33:46.662: fail: Error accepting connection: Too many open files
Feb 23 21:33:46 nautilus[987537]: The peer-to-peer connection failed: Unable to create socket: Too many open files. Falling back to the session bus. Your application is probably missing --filesystem=xdg-run/gvfsd privileges.
Feb 23 21:33:46 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 02:33:46.662: fail: Error accepting connection: Too many open files
Feb 23 21:33:46 gvfsd[16832]: (process:16832): GLib-GIO-WARNING **: 02:33:46.663: fail: Error accepting connection: Too many open files
Feb 23 21:33:46 gvfsd[16832]: (process:16832): GLib-ERROR **: 02:33:46.663: Creating pipes for GWakeup: Too many open files
Feb 23 21:33:46 kernel: [43521.156296] traps: gvfsd-trash[16832] trap int3 ip:7f360845e66f sp:7ffe6d35b650 error:0 in libglib-2.0.so.0.6800.4[7f360841f000+8d000]
Feb 23 21:33:46 nautilus[209496]: The peer-to-peer connection failed: Error while getting peer-to-peer dbus connection: Error receiving data: Connection reset by peer. Falling back to the session bus. Your application is probably missing --filesystem=xdg-run/gvfsd privileges.
I was also able to grab the output of lsof -c gvfsd-trash
, which shows
roughly 1,000 open unix domain stream sockets. Here's a truncated list:
gvfsd-tra 16832 evan 9u unix 0xffff943727111dc0 0t0 68648 /run/user/1000/gvfsd/socket-z695cL9X type=STREAM
gvfsd-tra 16832 evan 10u unix 0xffff943784fd7b40 0t0 250978 /run/user/1000/gvfsd/socket-ABP6wuKY type=STREAM
gvfsd-tra 16832 evan 11u unix 0xffff9437c8a2aec0 0t0 236373 /run/user/1000/gvfsd/socket-7cENbCWh type=STREAM
gvfsd-tra 16832 evan 12u unix 0xffff943784fd5500 0t0 250983 /run/user/1000/gvfsd/socket-ptcfRuG6 type=STREAM
gvfsd-tra 16832 evan 13u unix 0xffff9437d6d98440 0t0 247274 /run/user/1000/gvfsd/socket-MMVbFz5u type=STREAM
gvfsd-tra 16832 evan 14u unix 0xffff9436de6e2a80 0t0 244291 /run/user/1000/gvfsd/socket-JqZxJiY2 type=STREAM
gvfsd-tra 16832 evan 15u unix 0xffff9438bc289980 0t0 252302 /run/user/1000/gvfsd/socket-wU7TyyOc type=STREAM
gvfsd-tra 16832 evan 16u unix 0xffff9436de6e0880 0t0 244296 /run/user/1000/gvfsd/socket-F4rJRUdC type=STREAM
gvfsd-tra 16832 evan 17u unix 0xffff9436de6e0440 0t0 244301 /run/user/1000/gvfsd/socket-oWQluaW8 type=STREAM
gvfsd-tra 16832 evan 18u unix 0xffff9436de6e5940 0t0 244310 /run/user/1000/gvfsd/socket-erHV35XA type=STREAM
gvfsd-tra 16832 evan 19u unix 0xffff9438bc289100 0t0 252159 /run/user/1000/gvfsd/socket-nWqY8EAY type=STREAM
gvfsd-tra 16832 evan 20u unix 0xffff9438bc28c840 0t0 252163 /run/user/1000/gvfsd/socket-UK7YcLPu type=STREAM
gvfsd-tra 16832 evan 21u unix 0xffff9438bc28d940 0t0 252174 /run/user/1000/gvfsd/socket-ktxR0uOC type=STREAM
gvfsd-tra 16832 evan 22u unix 0xffff9438bc28f700 0t0 252178 /run/user/1000/gvfsd/socket-8Bj2YNmL type=STREAM
gvfsd-tra 16832 evan 23u unix 0xffff9436dbb5e1c0 0t0 250368 /run/user/1000/gvfsd/socket-u073a72L type=STREAM
gvfsd-tra 16832 evan 24u unix 0xffff943728ea8cc0 0t0 251007 /run/user/1000/gvfsd/socket-PEtTqps4 type=STREAM
Reproducing
Restarting seems to temporarily relieve the issue. It also appears to spontaneously resolve, which is what happend while I was in the process of writing this bug report. That said, gvfsd-trash seems to have been spamming my syslog for ~9 hours before it spontanously stopped.
Additional Information
I will try to grab a core dump the next time I see this issue pop up, however at a certain point I might need to re-image my system since I'm not sure there's a reliable way to fix/prevent this, and it impacts system stablitily.
I'll also try to collect debug output using kill -SIGUSR2 $(pidof gvfsd-trash)
and journalctl -f --user-unit=gvfs-daemon.service
.
Let me know if you have other suggestions for collecting more information.