www.cendio.com
Bug 5492 - crash in Xvnc analyseRect()
: crash in Xvnc analyseRect()
Status: CLOSED FIXED
: ThinLinc
VNC
: pre-1.0
: PC Unknown
: P2 Normal
: 4.5.0
Assigned To:
:
:
:
:
  Show dependency treegraph
 
Reported: 2015-04-13 09:48 by
Modified: 2017-01-04 16:24 (History)
Acceptance Criteria:


Attachments


Note

You need to log in before you can comment on or make changes to this bug.


Description From cendio 2015-04-13 09:48:40
We've had several reports of Xvnc crashing with this backtrace:

> (EE) 
> (EE) Backtrace:
> (EE) 0: /opt/thinlinc/libexec/Xvnc (xorg_backtrace+0x36) [0x5dac96]
> (EE) 1: /opt/thinlinc/libexec/Xvnc (0x400000+0x1deb69) [0x5deb69]
> (EE) 2: /lib64/libpthread.so.0 (0x7f0195916000+0xf710) [0x7f0195925710]
> (EE) 3: /opt/thinlinc/libexec/Xvnc (_ZN3rfb13EncodeManager11analyseRectEPKNS_11PixelBufferEPNS_8RectInfoEi+0x387) [0x61aed7]
> (EE) 4: /opt/thinlinc/libexec/Xvnc (_ZN3rfb13EncodeManager12writeSubRectERKNS_4RectEPKNS_11PixelBufferE+0xef) [0x61bdcf]
> (EE) 5: /opt/thinlinc/libexec/Xvnc (_ZN3rfb13EncodeManager11writeUpdateERKNS_10UpdateInfoEPKNS_11PixelBufferEPKNS_14RenderedCursorE+0xfc) [0x61c96c]
> (EE) 6: /opt/thinlinc/libexec/Xvnc (_ZN3rfb16VNCSConnectionST22writeFramebufferUpdateEv+0x4eb) [0x61600b]
> (EE) 7: /opt/thinlinc/libexec/Xvnc (_ZN3rfb16VNCSConnectionST15processMessagesEv+0xdc) [0x61676c]
> (EE) 8: /opt/thinlinc/libexec/Xvnc (_ZN14XserverDesktop13wakeupHandlerEP6fd_seti+0xe8) [0x5f7858]
> (EE) 9: /opt/thinlinc/libexec/Xvnc (0x400000+0x1ee1b4) [0x5ee1b4]
> (EE) 10: /opt/thinlinc/libexec/Xvnc (WakeupHandler+0x5b) [0x57f9fb]
> (EE) 11: /opt/thinlinc/libexec/Xvnc (WaitForSomething+0x4b6) [0x5d7976]
> (EE) 12: /opt/thinlinc/libexec/Xvnc (Dispatch+0xb2) [0x57ae92]
> (EE) 13: /opt/thinlinc/libexec/Xvnc (main+0x3da) [0x56888a]
> (EE) 14: /lib64/libc.so.6 (__libc_start_main+0xfd) [0x7f01955a0d5d]
> (EE) 15: /opt/thinlinc/libexec/Xvnc (0x400000+0x57529) [0x457529]
> (EE) 
> (EE) Segmentation fault at address 0x0
> 
> Fatal server error:
> Caught signal 11 (Segmentation fault). Server aborting

We can't see any obvious cause for this in the code, and we have yet to find a
way to reproduce it. For now we'll just have to keep an eye on it and see if
more information surfaces.
------- Comment #2 From cendio 2015-06-08 12:18:55 -------
We've gotten a decent core dump with proper debug info now.

The problem is that we're feeding an empty rect into the encoder. And it is
caused by an empty server side cursor rect. This shouldn't really happen, but
the logic for this is very messy so there is probably a bug in the handling. No
idea how to provoke it though.
------- Comment #3 From cendio 2015-06-09 16:16:00 -------
Hopefully fixed in r30365 by cleaning up the logic.

I couldn't figure out a way to provoke this, so the tester will have to make
due with merely testing server side cursor rendering. This is most easily done
by exploiting bug 2251.
------- Comment #5 From cendio 2015-06-23 15:29:01 -------
This vendor drop causes Xvnc to consume 100% CPU whilst a client is connected.
------- Comment #6 From cendio 2015-06-23 15:36:27 -------
Refixed in r30484.