pass_fd function now fails on Solaris
Submitted by Brian Cameron
Link to original bug (#575183)
Description
I notice that in GNOME 2.25 (with vte 0.19.4) that gnome-pty-helper does not seem to be working properly.
With the older GNOME 2.24 version (with vte 0.17.4), I would see this output when I built VTE with debug enabled and setting VTE_DEBUG=pty:
Sent request to helper. Received response from helper. Helper returns success. Tag = 80682b0. Got master pty 21 and slave pty 22. Setting up child pty: name = /dev/pts/3, fd = 22 Starting new session Setting size on fd 22 to (80,24). Returning ptyfd = 21, child = 13860. Setting size on fd 21 to (80,24). Size on fd 21 is (80,24).
However, with 2.25 (with vte 0.19.4) I see this output (with some comments added to explain the difference):
Sent request to helper. Received response from helper. Helper returns success. Tag = 8068c18.
Note that there was a failure here and instead of getting the "Got master
pty # and slave pty #" message, it instead falls back and uses the
_vte_pty_open_unix98 function.
Allocated pty on fd 19. PTY slave is `/dev/pts/3'. Setting up child pty: name = /dev/pts/3, fd = -1 Starting new session Setting size on fd 19 to (80,24). Returning ptyfd = 19, child = 127. Setting size on fd 19 to (80,24). Size on fd 19 is (80,24).
Digging into the code, I found that gnome-pty-helper was exiting when the open_ptys() function was called pass_fd() and this function returned a -1. This was happening because the pass_fd function has these lines:
if (sendmsg (client_fd, &msg, 0) != 1)
return -1;
I notice that on Solaris it is falling into the "return -1" and errno is 9 (EBADF).
If I revert the pass_fd code so it is the same as an older 0.17.4 version of VTE, then the problem goes away.
I'm unsure what the right fix is here, though. The newer code seems cleaner so it would be nicer to fix it than to revert to the older code. I am hoping the maintainers might have some suggestions how to approach fixing this.
Version: 0.19.x
Resolution: RESOLVED NOTGNOME