We've had several reports of Xvnc crashing with this backtrace:
> (EE) Backtrace:
> (EE) 0: /opt/thinlinc/libexec/Xvnc (xorg_backtrace+0x36) [0x5dac96]
> (EE) 1: /opt/thinlinc/libexec/Xvnc (0x400000+0x1deb69) [0x5deb69]
> (EE) 2: /lib64/libpthread.so.0 (0x7f0195916000+0xf710) [0x7f0195925710]
> (EE) 3: /opt/thinlinc/libexec/Xvnc (_ZN3rfb13EncodeManager11analyseRectEPKNS_11PixelBufferEPNS_8RectInfoEi+0x387) [0x61aed7]
> (EE) 4: /opt/thinlinc/libexec/Xvnc (_ZN3rfb13EncodeManager12writeSubRectERKNS_4RectEPKNS_11PixelBufferE+0xef) [0x61bdcf]
> (EE) 5: /opt/thinlinc/libexec/Xvnc (_ZN3rfb13EncodeManager11writeUpdateERKNS_10UpdateInfoEPKNS_11PixelBufferEPKNS_14RenderedCursorE+0xfc) [0x61c96c]
> (EE) 6: /opt/thinlinc/libexec/Xvnc (_ZN3rfb16VNCSConnectionST22writeFramebufferUpdateEv+0x4eb) [0x61600b]
> (EE) 7: /opt/thinlinc/libexec/Xvnc (_ZN3rfb16VNCSConnectionST15processMessagesEv+0xdc) [0x61676c]
> (EE) 8: /opt/thinlinc/libexec/Xvnc (_ZN14XserverDesktop13wakeupHandlerEP6fd_seti+0xe8) [0x5f7858]
> (EE) 9: /opt/thinlinc/libexec/Xvnc (0x400000+0x1ee1b4) [0x5ee1b4]
> (EE) 10: /opt/thinlinc/libexec/Xvnc (WakeupHandler+0x5b) [0x57f9fb]
> (EE) 11: /opt/thinlinc/libexec/Xvnc (WaitForSomething+0x4b6) [0x5d7976]
> (EE) 12: /opt/thinlinc/libexec/Xvnc (Dispatch+0xb2) [0x57ae92]
> (EE) 13: /opt/thinlinc/libexec/Xvnc (main+0x3da) [0x56888a]
> (EE) 14: /lib64/libc.so.6 (__libc_start_main+0xfd) [0x7f01955a0d5d]
> (EE) 15: /opt/thinlinc/libexec/Xvnc (0x400000+0x57529) [0x457529]
> (EE) Segmentation fault at address 0x0
> Fatal server error:
> Caught signal 11 (Segmentation fault). Server aborting
We can't see any obvious cause for this in the code, and we have yet to find a way to reproduce it. For now we'll just have to keep an eye on it and see if more information surfaces.
We've gotten a decent core dump with proper debug info now.
The problem is that we're feeding an empty rect into the encoder. And it is caused by an empty server side cursor rect. This shouldn't really happen, but the logic for this is very messy so there is probably a bug in the handling. No idea how to provoke it though.
Hopefully fixed in r30365 by cleaning up the logic.
I couldn't figure out a way to provoke this, so the tester will have to make due with merely testing server side cursor rendering. This is most easily done by exploiting bug 2251.
This vendor drop causes Xvnc to consume 100% CPU whilst a client is connected.
Refixed in r30484.