Bug 2928 - automatic lossless refresh
Summary: automatic lossless refresh
Status: CLOSED FIXED
Alias: None
Product: ThinLinc
Classification: Unclassified
Component: VNC (show other bugs)
Version: trunk
Hardware: PC Unknown
: P2 Enhancement
Target Milestone: 4.10.0
Assignee: Pierre Ossman
URL:
Keywords: relnotes
: 4350 (view as bug list)
Depends on: 4915 7158
Blocks: 5106 2751
  Show dependency treegraph
 
Reported: 2008-10-02 09:31 CEST by Peter Åstrand
Modified: 2018-09-24 11:00 CEST (History)
1 user (show)

See Also:
Acceptance Criteria:
* Screen areas damaged by JPEG compression should automatically be replaced with copies with no noticeable damage * "Real" updates should get priority over refresh updates (i.e. you should not get worse update rate with this change) * When regions are changed very often, for example with video playback, the server should not send lossless updates for that region until it has stopped changing


Attachments
Prototype patch which avoids ALR if regions are constantly changing (5.03 KB, patch)
2018-09-18 14:53 CEST, Peter Åstrand
Details

Description Peter Åstrand cendio 2008-10-02 09:31:20 CEST
We do not want to use JPEG unless necessary; we don't want JPEG artefacts with office applications, for example. One idea is to do a lossless refresh automatically after a while. Ideally, the server should only use JPEG compression on suitable subsets of the screen.
Comment 1 Pierre Ossman cendio 2008-10-13 16:25:06 CEST
Bug 2927 covers automatic selection of encoding (and JPEG is already automatically selected).

What we need here is:

- Automatic detection of areas with lots of updates

- Automatic lossless refresh once it settles down


I have no idea what the detection code looks like, so the time est. is a very wild guess.
Comment 2 Pierre Ossman cendio 2013-11-25 13:16:16 CET
*** Bug 4350 has been marked as a duplicate of this bug. ***
Comment 3 Karl Mikaelsson cendio 2016-06-08 13:04:46 CEST
This is a server-side-only change.
Comment 4 Pierre Ossman cendio 2016-12-01 13:32:10 CET
First attempt available here:

https://github.com/CendioOssman/tigervnc/tree/alr

Probably needs more work with regard to tuning the update rate. The work done on bug 5719 and bug 4735 could probably be helpful.
Comment 5 Pierre Ossman cendio 2018-08-27 13:36:26 CEST
Latest upstream code now respects available bandwidth.
Comment 6 Peter Åstrand cendio 2018-09-18 14:45:18 CEST
The current implementation does not work very well in the following scenario: 

* High bandwidth (~300 MB/s)
* Video playback

In this case, the server will send lossless updates for every video frame, which causes the CPU and network usage to rise. The playback experience is also affected (more choppy).
Comment 7 Peter Åstrand cendio 2018-09-18 14:53:34 CEST
Created attachment 886 [details]
Prototype patch which avoids ALR if regions are constantly changing
Comment 8 Pierre Ossman cendio 2018-09-19 14:06:47 CEST
I've added some diagnostics and determined that the problem is CPU resources and not bandwidth (as suspected). 

In my test case it determines that it has 15 ms before the next update. However it spends between 40 and 45 ms sending the lossless refresh. Of those only 1-2 ms is spent waiting for things and the rest is using the CPU.

Not sure how to best adjust the update size so we don't overshoot here. Perhaps hard code a conservative value for CPU throughput?
Comment 9 Pierre Ossman cendio 2018-09-21 10:49:22 CEST
A hard coded throttling to avoid CPU issues seem to do the trick. Some other issues have also been found and fixed:

Refresh size miscalculation
---------------------------

A mixup between pixels and bytes caused the system to over-estimate the appropriate size. That was also part of the reason we got fps drops.

Attempts to send even if there was no room
------------------------------------------

The system tried to send a refresh even if it determined that it was 0 ms to the next "real" updated. As each update has a minimum size this could cause some issues. At least 1 ms needs to be available now.

Avoid unnecessary refreshes
---------------------------

A final version of the attached patch. The idea is to avoid sending a refresh that will be immediately overwritten as the user will not see it anyway. As we can't tell the future we approximate this by assuming that an area recently updated will soon see another update.

We do the refresh once the area has been stable for a short while (100ms). That's long enough to catch common cases like video, but short enough to feel "immediate".

Consider high quality JPEG to be "lossless"
-------------------------------------------

It's technically not "lossless" as some bits will vary from the "real" image. But users should not be able to see the difference, so let's try to make use of the extra compression that gives us.

More aggressive refresh when probing bandwidth
----------------------------------------------

The probing of bandwidth relies on the principle of overshooting a bit and seeing if we get delays. But the refresh tries to stay under the current bandwidth estimation so it never sends enough data to improve the probe. That also means it gets stuck at an estimate much lower than what's actually available.

Improve this by telling the refresh system there is more bandwidth as long as we are in the probing stage. Once the bandwidth has been properly determined we go back to a conservative guess.
Comment 12 Pierre Ossman cendio 2018-09-24 10:59:07 CEST
I've tested this in different scenarios and I cannot see any real difference compared to 4.9.0 in either bandwidth, CPU usage or perceived performance. I've tried:

 * Unlimited bandwidth, no latency
 * 4 Mbps, 50 ms
 * Unlimited bandwidth, 150 ms

I tested glxgears, some different movies and general UI interaction (e.g. dragging a window around). Some positive effects from bug 4735 and bug 5719 could be seen, but in no case was the experience any worse.

As for positive effects from this bug, it could only really be seen once I configured a lower JPEG quality. At that point the experience got much better on the networks with low bandwidth or high latency. Unlike 4.9.0 though I got a nice, clear picture once the updates calmed down.
Comment 13 Pierre Ossman cendio 2018-09-24 11:00:34 CEST
> * Screen areas damaged by JPEG compression should automatically be replaced with copies with no noticeable damage

Yup.

> * "Real" updates should get priority over refresh updates (i.e. you should not get worse update rate with this change)

I can not see any difference compared to 4.9.0, so seems to work fine.

> * When regions are changed very often, for example with video playback, the server should not send lossless updates for that region until it has stopped changing

CPU and bandwidth are similar to 4.9.0, so seems to work.

Note You need to log in before you can comment on or make changes to this bug.