[sync screen] Top/Bottom colors are confused by movies's black borders about hue-lights HOT 14 CLOSED

Hi @cm-t , it is a known issue to me and I am not sure how to resolve it yet.

The problem is the limitation of the Gnome shell. Taking a screenshot and analyze it is a very expensive operation. I tried it. It is much cheaper to get just a few pixels and make an average color based on the few pixels, but it is not suitable for detecting the black borders.

My idea here is to add a new sync mode - 'sync selected area'. You could select a rectangle at the beginning of synchronization manually with your mouse. Maybe this could be also activated using some key shortcut.

What do you think?

from hue-lights.

I understand your point @cm-t . There are a lot of different use cases to consider.

from hue-lights.

Since version 13, you can use a shortcut SUPER+ATL+a during full-screen video playing and select a screen area for syncing, but first, you need to set an entertainment area for the synchronization in the menu.

Do you consider this solution sufficient?

from hue-lights.

Ohh, I did not thought about this way

I thought about some options, but they are only ideas, not especially good or bad, just a brainstorm i had and didnt post because i didn't want to pollute the 1st post (because I think issues should explain the need, not the solution; explain the need, then talk about solution⋅s ^^):

• Dont scan the presence of black borders on each frame but only each n milliseconds for example (maybe 10 000 by default in settings) so people could adjust regarding their hardware capacity.
• Use a whilist such as [firefox, vlc, totem, spotify, netflix, etc] (should be editable) and add a setting to enable the auto-detection of the black borders (select input) such as:
  • no detection (current behavior)
  • detection (using the n time settings)
  • "detection only if the focused app is in the whitelist" (using the n time settings)
• I forgot others ideas to be inserted here, sorry, maybe it will comes back to me later ^^
• Edit (other ideas):
  • read color of 1st pixel (corner) and check if it changes after n time, if no, apply black border detection and apply it until next n timing, if yes, disable black border detection
  • tbc

But most of those ideas are not especially needed with your idea to start a screen area-selection sync. I think I really like this idea.
I think I could even fix some wrong behaviors issues with multi screen ? (I didn't tried with multi-screen, so I'm just supposing)

So now, yep, I'm spitted. I know both way of doing (detection or area selection) are not incompatible (without double negative: they are compatible !), but since you are mostly the only one to commit, I guess we should select one of these way; and maybe think later about the other one, I don't know, more discussion needed i think :)

from hue-lights.

Thank you for your ideas. I would prefer the autodetection over the whitelist. I believe it would be more universal.

I think detecting the black borders only in a configurable time could work well. The detection stresses the CPU, and due to gnome-shell limitation, there are no threads or parallelism. So people with powerful CPU could detect more often than others. Also, I believe the span of configurable times could be, let's say, from 1s to 30s. The default value of 10s seems reasonable.

It could also prolong the detecting interval after detecting the black borders. Typically, the movie takes more than one hour with the same borders. It could work like this: Once the border is detected, the interval for the next detecting would be the same (10 seconds), then 30 seconds, and finally 1 minute.

Multi-display requires the screen to be a solid rectangle. Otherwise, it will not start (with the notification displayed).

area-selection sync would work on the non-solid (rectangle) screens. That would be a solution for people with various displays.

I think about implementing both solutions, though it could take some time. I am quite busy recently. I would start with the autodetection.

from hue-lights.

I like the idea of extending the time detection after it has been detected first !

Just a though about the rectangle: before finding the rectangle dimension, the every 10 sec scan could just be on the 1st pixel (top left corner?) and the last pixel (bottom right corner?), and if it is same after the 10 sec, you can double check 1 sec (or some millisecond) later to confirm, and then finally launch the detection of the size of the top and bottom rectangles to adjust the color sent to the hub's API.
I think scanning 2 pixels cost not so much than detect rectangle at every scan.

from hue-lights.

Please note that light position is set relative to your screen, not your content.
For this reason if the average color of your top area is almost black (for whatever reason) the lights on that position should be almost off. That's how it is supposed to work and how the official sync app work.

E.g.: If you try to play a 32:9 video on a 16:9 screen there will be huge black borders and the official Philips Hue Sync app will almost turn off the lights on those positions.

There is a reason behind this choice! Imagine a huge screen attached to the wall with some lights around it.
The lights must have a color similar to their position "in real life". This means that light just above the screen should have the same color of the screen top area even if it is black.

Also note that dynamic autodetection is a double-edged sword. What happens if a content (e.g. a movie) has some scenes with some completely black areas? Will the extension suddenly change its configuration making weird glitches?

As stated before, the default behavior of the official app is to match the sync area to the whole screen and I suggest to keep this choice as default in this extension. Any other behavior should be a setting.

from hue-lights.

Hi @RiccardoMaffei

There is a reason behind this choice! Imagine a huge screen attached to the wall with some lights around it.
The lights must have a color similar to their position "in real life". This means that light just above the screen should have the same color of the screen top area even if it is black.

I am in the case you are mentioning: Hudge screen (video projector) + led stripe (mounted below on the top of the furniture below and facing the wall, so no direct eyes contact but wall reflection, and is set at minimum brightness when using the video projector), the LED stripe light override the black borders of the video projector beam on the wall and provide the immersion wished. Pictures are better than words, but sadly my attempt to take a decent picture of my setup is a fail since my old phone cam is terribly bad at dark shoot with screen facing the cam...

Also note that dynamic autodetection is a double-edged sword. What happens if a content (e.g. a movie) has some scenes with some completely black areas? Will the extension suddenly change its configuration making weird glitches?

This should be anyway something to be tested; and maybe we can even have a look at other projects (i'm naming one below) to see how they implemented it and handle it well.

Once the border is detected, the interval for the next detecting would be the same (10 seconds), then 30 seconds, and finally 1 minute.

This won't be an issue when the movie gets darker. But, this could maybe add a glitch if the movie stay dark too long, and then resume lighter (means we could wait 60 secondes to have LED lighting again.
If the black border detection expend to the full size, then keep last working size, or if it was the first second of the sync screen mode and no size was found before, then keep scanning for it every second (or every n setting). something like this should be working without glitches. As said before, it should be tested, and we could even see at others projects too (even if they are not gnome-shell extension, what matters here is screen scanning methods) .

NB: my coworker has a DIY led stripe, and the project he uses to sync screen with LED (hyperion.ng) has the option to provide a working well (no glitches) black borders detection without lag issue, (as far as i remember when talking about it with him). If the feature is requested, and implemented, and used, that's not just "because we can do it", but also because it matches some uses cases.

NBB: when you already have large hardware screen borders, or not, there is already a gap of the immersion between users anyway, but I understand your call and I can only agree with you: this feature should obviously be an option that users can enable/disable.

NBBB: sorry for my glitchy english

from hue-lights.

Hi @cm-t,
a projector in a dark room is a completely different scenario than a physical screen.
In the former case, the black borders can be "painted" by the light, in the latter they can't.

Also, things get more complicated when you have several lights (10, 20, 50 etc..)! If you skip the black borders you are "virtually stretching" the image to fill the whole screen (e.g. vertically) and if you have several lights on the stretched dimension (e.g. on the right/left border) the color of the light won't match the actual color of the image on the screen.

Take a look at the following example:

      Actual screen              Stretched screen    
+-----------------------+   +-----------------------+
|         Black         |L1 |                       |
+-----------------------+   |          Red          |
|          Red          |L2 +-----------------------+
+-----------------------+   |                       |
|         Green         |L3 |         Green         |
+-----------------------+   |                       |
|          Red          |L4 +-----------------------+
+-----------------------+   |          Red          |
|         Black         |L5 |                       |
+-----------------------+   +-----------------------+

The light color should try to match as closely as possible the average color of the area next to the light (just like the Philips Ambilight technology). The color of that area on the actual screen may be different than the one on the stretched virtual version.

For the aforementioned reasons, in my opinion, the standard (and default) behavior should be to use the whole screen.

from hue-lights.

Yes @RiccardoMaffei , video-projector is a "different" use case than the most people's usecase, but it is still a valid usecase, i mean, i'm not the only one with no tv screen but a projector, am i?

Also, this is my entertainment zone (named "cinema") 's settings:

• A: the LED stripe i've mentionned, just below the screen, will softly pain the wall near the screen
• B: One mid screen tall, behing user's head when sit on sofa, will very softly paint wall near the screen
• C: One at the roof/top screen, behing user's head when sit on sofa, will very softly paint roof/wall near the screen

Using full screen color in a movie use case (black border, but no detection of them):

• A: switched off
• B: middle screen color
• C: switched off

immersion: only middle screen color (could be the red of a thirst of a character, wile everything except that on screen is the green from the golf ground); only 1 color.

Using full screen color in a movie use case (black border, but detection of them):

• A: bottom screen color
• B: middle screen color
• C: top screen color

immersion: more color from the screen will be represented in the room

In most usecase, people have between 1 and 5 light max in a room, and most of them are not close to the TV (usually 1 or a pair) but could paint a part of the wall close to it.
The user enabled his lights to the sync experience by adding them manually one after one in the official phone app as a "entertainment area".

In the usecase of TV, having all those light color could be breaking the experience, because your eyes see a, for example 10 cm black hole rectangle between bottom screen color and wall color, right ?

Do i got your point if I say this ?:

Black border detection

• prevent you to have less selected light of your entertainment zone than what you actually selected
• can break a black border but
• provide more colors to the immersion (while prevent monocolor/nocolor)

No black border detection

• can force the number of light less than what you selected, even to 1 or 0
• might provide less or zero color to the immersion but
• doesn't break any dark area of the screen (in your example, it wont do redScreen/blackScreen/redWall at the bottom for eg)

This is probably not an objective point of view, but a subjective one since i focus a lot on my own usecase; but, since we already mentioned it should be an option, not enabled by default, can be enabled manually in the panel settings, do you see any more risk ?:

• can have laggyness (perf)
  • we can solve with: test + check at others projects
• can have glitches
  • we can solve with: test + check at others projects
• can have a divergent experience than the desktop official app (such as the black hole between screen's color and LED's color; without mentioning permanent black hardware screen borders)
  • we can solve with: not enabled by default, user can switch it on manually

edit: typos

from hue-lights.

Thank you @RiccardoMaffei and @cm-t for the discussion.

I am sorry for not being active recently. I was busy. Hopefully, it will be better.

I understand the autodetection could be a problem... Now, I think the 'sync selected area' could be maybe an sutiable option.

Or...

The current algorithm checks only four pixels in the area dedicated for one light and then it determines the average color for the appropriate light. This is because I would like to spare the CPU power. With a powerful CPU, it could check definitely more pixels. The rectangle dedicated to one particular light on the screen is also quite small. I could make it bigger and use more pixels. This could be configurable (both the light rectangle size and the number of checked pixels) so anybody could adjust it for his/her needs and use cases. This could be closer to the original hue app... I have the sync box, but I do not use the original sync app because o do not use Windows or Mac. Not sure how the app works, but the sync box seems to deal with black borders very well.

Your thoughts?

I have 10 lights in my living room and one of the lights is the gradient light strip (it has 7 light zones), which means I have 16 lights for calculating the average color... It is a lot for my old i5 CPU. That is why I try to spare the CPU power.

from hue-lights.

I have added support for sync screen selection mode. After selecting the mode and starting the sync, you can draw a rectangle on the screen and this rectangle will be synchronized.

I will also improve the sync mode selection. It is a bit uncomfortable now.

from hue-lights.

Hi, I think this solution is great for performance, and also allow users to select the right screen to sync (in case this usecase happen)

In case you want to improve the experience by adding an autodetection (or any other method), I beleive it is ok to keep the selection area mode (and add another) because it just works :)

Also, thank you again 🤟

from hue-lights.

Autodetection seems to be unreliable - mostly because of dark scenes. It would be a source of issues and not a reliable solution.

I consider this issue to be resolved with the "selection sync" and the key shortcut. In case of other ideas let me know. I will close this issue now.

from hue-lights.