Bugzilla – Bug 5492
crash in Xvnc analyseRect()
Last modified: 2017-01-04 16:24:31
You need to
before you can comment on or make changes to this bug.
We've had several reports of Xvnc crashing with this backtrace:
> (EE) Backtrace:
> (EE) 0: /opt/thinlinc/libexec/Xvnc (xorg_backtrace+0x36) [0x5dac96]
> (EE) 1: /opt/thinlinc/libexec/Xvnc (0x400000+0x1deb69) [0x5deb69]
> (EE) 2: /lib64/libpthread.so.0 (0x7f0195916000+0xf710) [0x7f0195925710]
> (EE) 3: /opt/thinlinc/libexec/Xvnc (_ZN3rfb13EncodeManager11analyseRectEPKNS_11PixelBufferEPNS_8RectInfoEi+0x387) [0x61aed7]
> (EE) 4: /opt/thinlinc/libexec/Xvnc (_ZN3rfb13EncodeManager12writeSubRectERKNS_4RectEPKNS_11PixelBufferE+0xef) [0x61bdcf]
> (EE) 5: /opt/thinlinc/libexec/Xvnc (_ZN3rfb13EncodeManager11writeUpdateERKNS_10UpdateInfoEPKNS_11PixelBufferEPKNS_14RenderedCursorE+0xfc) [0x61c96c]
> (EE) 6: /opt/thinlinc/libexec/Xvnc (_ZN3rfb16VNCSConnectionST22writeFramebufferUpdateEv+0x4eb) [0x61600b]
> (EE) 7: /opt/thinlinc/libexec/Xvnc (_ZN3rfb16VNCSConnectionST15processMessagesEv+0xdc) [0x61676c]
> (EE) 8: /opt/thinlinc/libexec/Xvnc (_ZN14XserverDesktop13wakeupHandlerEP6fd_seti+0xe8) [0x5f7858]
> (EE) 9: /opt/thinlinc/libexec/Xvnc (0x400000+0x1ee1b4) [0x5ee1b4]
> (EE) 10: /opt/thinlinc/libexec/Xvnc (WakeupHandler+0x5b) [0x57f9fb]
> (EE) 11: /opt/thinlinc/libexec/Xvnc (WaitForSomething+0x4b6) [0x5d7976]
> (EE) 12: /opt/thinlinc/libexec/Xvnc (Dispatch+0xb2) [0x57ae92]
> (EE) 13: /opt/thinlinc/libexec/Xvnc (main+0x3da) [0x56888a]
> (EE) 14: /lib64/libc.so.6 (__libc_start_main+0xfd) [0x7f01955a0d5d]
> (EE) 15: /opt/thinlinc/libexec/Xvnc (0x400000+0x57529) [0x457529]
> (EE) Segmentation fault at address 0x0
> Fatal server error:
> Caught signal 11 (Segmentation fault). Server aborting
We can't see any obvious cause for this in the code, and we have yet to find a
way to reproduce it. For now we'll just have to keep an eye on it and see if
more information surfaces.
We've gotten a decent core dump with proper debug info now.
The problem is that we're feeding an empty rect into the encoder. And it is
caused by an empty server side cursor rect. This shouldn't really happen, but
the logic for this is very messy so there is probably a bug in the handling. No
idea how to provoke it though.
Hopefully fixed in r30365 by cleaning up the logic.
I couldn't figure out a way to provoke this, so the tester will have to make
due with merely testing server side cursor rendering. This is most easily done
by exploiting bug 2251.
This vendor drop causes Xvnc to consume 100% CPU whilst a client is connected.
Refixed in r30484.