Bug 5492

Summary: crash in Xvnc analyseRect()
Product: ThinLinc Reporter: Pierre Ossman <ossman@cendio.se>
Component: VNCAssignee: Pierre Ossman <ossman@cendio.se>
Status: CLOSED FIXED QA Contact: Bugzilla mail exporter <bugzilla-qa@cendio.se>
Severity: Normal    
Priority: P2 CC: hean01@cendio.se, samuel@cendio.se
Version: pre-1.0Keywords: relnotes, samuel_tester
Target Milestone: 4.5.0   
Hardware: PC   
OS: Unknown   
Acceptance Criteria:

Description From cendio 2015-04-13 09:48:40
We've had several reports of Xvnc crashing with this backtrace:

> (EE) 
> (EE) Backtrace:
> (EE) 0: /opt/thinlinc/libexec/Xvnc (xorg_backtrace+0x36) [0x5dac96]
> (EE) 1: /opt/thinlinc/libexec/Xvnc (0x400000+0x1deb69) [0x5deb69]
> (EE) 2: /lib64/libpthread.so.0 (0x7f0195916000+0xf710) [0x7f0195925710]
> (EE) 3: /opt/thinlinc/libexec/Xvnc (_ZN3rfb13EncodeManager11analyseRectEPKNS_11PixelBufferEPNS_8RectInfoEi+0x387) [0x61aed7]
> (EE) 4: /opt/thinlinc/libexec/Xvnc (_ZN3rfb13EncodeManager12writeSubRectERKNS_4RectEPKNS_11PixelBufferE+0xef) [0x61bdcf]
> (EE) 5: /opt/thinlinc/libexec/Xvnc (_ZN3rfb13EncodeManager11writeUpdateERKNS_10UpdateInfoEPKNS_11PixelBufferEPKNS_14RenderedCursorE+0xfc) [0x61c96c]
> (EE) 6: /opt/thinlinc/libexec/Xvnc (_ZN3rfb16VNCSConnectionST22writeFramebufferUpdateEv+0x4eb) [0x61600b]
> (EE) 7: /opt/thinlinc/libexec/Xvnc (_ZN3rfb16VNCSConnectionST15processMessagesEv+0xdc) [0x61676c]
> (EE) 8: /opt/thinlinc/libexec/Xvnc (_ZN14XserverDesktop13wakeupHandlerEP6fd_seti+0xe8) [0x5f7858]
> (EE) 9: /opt/thinlinc/libexec/Xvnc (0x400000+0x1ee1b4) [0x5ee1b4]
> (EE) 10: /opt/thinlinc/libexec/Xvnc (WakeupHandler+0x5b) [0x57f9fb]
> (EE) 11: /opt/thinlinc/libexec/Xvnc (WaitForSomething+0x4b6) [0x5d7976]
> (EE) 12: /opt/thinlinc/libexec/Xvnc (Dispatch+0xb2) [0x57ae92]
> (EE) 13: /opt/thinlinc/libexec/Xvnc (main+0x3da) [0x56888a]
> (EE) 14: /lib64/libc.so.6 (__libc_start_main+0xfd) [0x7f01955a0d5d]
> (EE) 15: /opt/thinlinc/libexec/Xvnc (0x400000+0x57529) [0x457529]
> (EE) 
> (EE) Segmentation fault at address 0x0
> Fatal server error:
> Caught signal 11 (Segmentation fault). Server aborting

We can't see any obvious cause for this in the code, and we have yet to find a
way to reproduce it. For now we'll just have to keep an eye on it and see if
more information surfaces.
------- Comment #2 From cendio 2015-06-08 12:18:55 -------
We've gotten a decent core dump with proper debug info now.

The problem is that we're feeding an empty rect into the encoder. And it is
caused by an empty server side cursor rect. This shouldn't really happen, but
the logic for this is very messy so there is probably a bug in the handling. No
idea how to provoke it though.
------- Comment #3 From cendio 2015-06-09 16:16:00 -------
Hopefully fixed in r30365 by cleaning up the logic.

I couldn't figure out a way to provoke this, so the tester will have to make
due with merely testing server side cursor rendering. This is most easily done
by exploiting bug 2251.
------- Comment #5 From cendio 2015-06-23 15:29:01 -------
This vendor drop causes Xvnc to consume 100% CPU whilst a client is connected.
------- Comment #6 From cendio 2015-06-23 15:36:27 -------
Refixed in r30484.