Application-controlled window dragging

[Christian Hammond-3]

Hey everyone.

I'm working on an experimental project at VMware that will require us to have greater control over a window's position while the window manager is dragging it. We'd need to basically give the application more control over the movement of the window, yet allow the the window manager to apply whatever effects and conditions it would typically apply while moving a window. We'd need to let the application, for example, prevent the window from going past a certain part of the screen, or to stay within a bounding box, or to just prevent it from moving over another window.

We need this for custom shaped windows without standard window decorations. Right now, we go the xmms, etc. route and listen to mouse events and reposition the window, but 1) we won't be able to move off the left side of the screen on Metacity (as Metacity seems to only allow that when it's moving the window) and 2) we won't get any window manager effects when moving (such as wobbly windows).

Ideally, to allow the greatest flexibility, I'd like to see messages similar to _NET_WM_MOVERESIZE that performs any effects the window manager would normally use with a moving window, except instead of grabbing the mouse, the application would have the responsibility of listening to mouse events and moving the window. To the user, it would look like the window is moving normally, but the application would be in full control of the move operation.

There would be two messages, one for beginning the operation, and one for ending it. The _BEGIN operation would be fairly similar to _NET_WM_MOVERESIZE, though without the button parameter (I don't think it'd be needed)? The _END would merely finalize the _BEGIN operation. It'd be the equivalent of releasing the mouse button used to drag the window.

Another option would be to introduce two new flags for the existing _NET_WM_MOVERESIZE that would initiate and finalize this operation. Maybe this is the right thing to do, so long as there's a way we can detect whether this is supported from the application.

Thoughts?

Christian

--
Christian Hammond - chipx86@...
VMware, Inc.
_______________________________________________
wm-spec-list mailing list
wm-spec-list@...
http://mail.gnome.org/mailman/listinfo/wm-spec-list

[Havoc Pennington]

Hi,

Christian Hammond wrote:
> I'm working on an experimental project at VMware that will require us to
> have greater control over a window's position while the window manager
> is dragging it. We'd need to basically give the application more control
> over the movement of the window, yet allow the the window manager to
> apply whatever effects and conditions it would typically apply while
> moving a window. We'd need to let the application, for example, prevent
> the window from going past a certain part of the screen, or to stay
> within a bounding box, or to just prevent it from moving over another
> window.

It would seem more logical to me to convey these constraints to the WM,
rather than adding an "escape hatch" that potentially leaves the WM with
no clue what's going on or why.

I'd guess metacity would have a really hard time knowing how to
implement the "escape hatch" - most of the time, the move constraints in
the metacity code apply even to moves originated by the WM itself.

Certainly the codepaths related to this are "interesting" enough already.

Is it possible to develop a more semantic hint instead of the escape
hatch? (without more detail on the project, it's tough for me to suggest
anything.)

Havoc

_______________________________________________
wm-spec-list mailing list
wm-spec-list@...
http://mail.gnome.org/mailman/listinfo/wm-spec-list

[Christian Hammond-3]

On 10/5/07, Havoc Pennington <hp@...> wrote:

Hi,

Christian Hammond wrote:
> I'm working on an experimental project at VMware that will require us to
> have greater control over a window's position while the window manager
> is dragging it. We'd need to basically give the application more control
> over the movement of the window, yet allow the the window manager to
> apply whatever effects and conditions it would typically apply while
> moving a window. We'd need to let the application, for example, prevent
> the window from going past a certain part of the screen, or to stay
> within a bounding box, or to just prevent it from moving over another
> window.

It would seem more logical to me to convey these constraints to the WM,
rather than adding an "escape hatch" that potentially leaves the WM with
no clue what's going on or why.

I'd guess metacity would have a really hard time knowing how to
implement the "escape hatch" - most of the time, the move constraints in
the metacity code apply even to moves originated by the WM itself.

Certainly the codepaths related to this are "interesting" enough already.

Is it possible to develop a more semantic hint instead of the escape
hatch? (without more detail on the project, it's tough for me to suggest
anything.)

Havoc

The problem is, we won't necessarily know these restrictions before we're dragging. We only have some idea of them as we drag, as we hit them.

If we go to a more traditional example, let's look at XMMS. It consists of multiple, separate windows that stick together while dragging. They have to handle move events themselves in order to guarantee the windows will stick together to bind them, and to move all the windows at the same time. They wouldn't be able to do this with _NET_WM_MOVERESIZE, and are forced to act in such a way where the window manager will have no idea that they're dragging.

We have this problem to a much higher degree. Not only do we need to support this case, but more complex cases where windows can't reach positions on the screen. And we just don't know until the user tries to drag it there.

Christian

--
Christian Hammond - chipx86@...
VMware, Inc.
_______________________________________________
wm-spec-list mailing list
wm-spec-list@...
http://mail.gnome.org/mailman/listinfo/wm-spec-list

[Christian Hammond-3]

The piece of code I'm working on now is an experiment at the moment, but it will be part of a full-blown feature we're working on, one which I'm sure will be very popular. I can't talk about the feature in detail (unless either of you want to sign an NDA, which I can probably arrange), as it hasn't been announced yet, but not having this support will have a serious impact on this feature's usability.

We are planning to put some support in window managers for now just to test it. That won't do our end users any good though when the feature's released unless we can get it standardized and into several main window managers.

Christian

On 10/5/07, Matthias Clasen <mclasen@...> wrote:

On Fri, 2007-10-05 at 15:13 -0700, Christian Hammond wrote:

>
>
>
> The problem is, we won't necessarily know these restrictions before
> we're dragging. We only have some idea of them as we drag, as we hit
> them.
>
> If we go to a more traditional example, let's look at XMMS. It
> consists of multiple, separate windows that stick together while
> dragging. They have to handle move events themselves in order to
> guarantee the windows will stick together to bind them, and to move
> all the windows at the same time. They wouldn't be able to do this
> with _NET_WM_MOVERESIZE, and are forced to act in such a way where the
> window manager will have no idea that they're dragging.
>
> We have this problem to a much higher degree. Not only do we need to
> support this case, but more complex cases where windows can't reach
> positions on the screen. And we just don't know until the user tries
> to drag it there.

Before discussing mechanisms for doing this kind of stuff, how about we
start with a discussion of the usefulness.

What is the desired user experience here ?

Why should window managers implement very complicated protocols to
support something that is ultimatively not helping the user ?

Also, considering that you called this initially an experimental
project, I don't think this should this be incorporated in the spec
before we have even seen the outcome of the experiment. Wouldn't it be
much more natural to just patch some window manager to fit the needs of
your experiment ?

--
Christian Hammond - chipx86@...
VMware, Inc.
_______________________________________________
wm-spec-list mailing list
wm-spec-list@...
http://mail.gnome.org/mailman/listinfo/wm-spec-list

[Nathaniel Smith]

On Fri, Oct 05, 2007 at 04:13:41PM -0700, Christian Hammond wrote:
>    The piece of code I'm working on now is an experiment at the moment, but
>    it will be part of a full-blown feature we're working on, one which I'm
>    sure will be very popular. I can't talk about the feature in detail
>    (unless either of you want to sign an NDA, which I can probably arrange),
>    as it hasn't been announced yet, but not having this support will have a
>    serious impact on this feature's usability.

I appreciate these issues, but I also hope you realize there are some
fundamental difficulties with convincing a free standards group to
mandate some behavior in N different window manager projects, just on
your word that it will allow one proprietary app to have a totally
cool feature that is best implemented and set in stone in this way
that I came up with and no-one has reviewed, trust me.

I mean, these might be all true... but I'm not sure how, as
hopefully-responsible engineers, we can just accept them without
checking for ourselves, or convince projects that they should bother
implementing/maintaining this feature with no more to go on than your
word.  Do you have any ideas?

If the problem is just that it needs to remain a secret for a while
longer, you may have more luck adding support for a custom (not
_NET_WM_*) extension to a few important window managers, convincing
their maintainers to take the patch, and then worry about pushing it
up into EWMH once the feature is out and there is some real-world
experience with it.

(Also, umm... is "lack of wobbly windows" really a "serious impact"?
Though I guess you can't really explain why things would affect
usability, either.)

-- Nathaniel

--
The Universe may  /  Be as large as they say
But it wouldn't be missed  /  If it didn't exist.
  -- Piet Hein
_______________________________________________
wm-spec-list mailing list
wm-spec-list@...
http://mail.gnome.org/mailman/listinfo/wm-spec-list

[Dana Jansens-2]

On 10/5/07, Nathaniel Smith <njs@...> wrote:

> On Fri, Oct 05, 2007 at 04:13:41PM -0700, Christian Hammond wrote:
> >    The piece of code I'm working on now is an experiment at the moment, but
> >    it will be part of a full-blown feature we're working on, one which I'm
> >    sure will be very popular. I can't talk about the feature in detail
> >    (unless either of you want to sign an NDA, which I can probably arrange),
> >    as it hasn't been announced yet, but not having this support will have a
> >    serious impact on this feature's usability.
>
> I appreciate these issues, but I also hope you realize there are some
> fundamental difficulties with convincing a free standards group to
> mandate some behavior in N different window manager projects, just on
> your word that it will allow one proprietary app to have a totally
> cool feature that is best implemented and set in stone in this way
> that I came up with and no-one has reviewed, trust me.

Given that it would seem to finally allow for professional quality
user interfaces outside of the "one square window" realm, I think it's
a very welcome proposition.

Dana
_______________________________________________
wm-spec-list mailing list
wm-spec-list@...
http://mail.gnome.org/mailman/listinfo/wm-spec-list

[Christian Hammond-3]

If I could describe the whole feature, I would. This is just how things work in competitive businesses, but I'm not proposing something that would only be useful in this case. This is beneficial for any application that needs to do something special as the window moves.

The thing to keep in mind is that there are several applications that already bypass the window manager's moving logic and implement it themselves. XMMS is a good example because of how it deals with moving multiple windows simultaneously and binds windows together when they come in contact. What I'm proposing lets the window manager know that the window is being dragged, allows it to feel more "real," and yet still gives the application some control over its own movement logic.

In the past at VMware, we've had to use X in ways that most people just haven't needed so far. Our multimonitor support in VMware Workstation wasn't possible without either a spec extension (which hasn't been accepted last I heard) or some hacks. This new feature would benefit greatly from the proposed spec extension, as would some existing programs out there. Sometimes you have to push the envelope to make really useful features and do things that haven't been required from apps until now. If we can't get this implemented, we'll live without it, but it will really reduce the experience and I don't want to see that happen.

For the time being, yes, we can make a proprietary extension and try to get some window managers to use it. I don't think it's the best option long-term, but perhaps that'll do until we at the least have a demo or beta out. It'll be easier to describe at that point why it's essential.

Christian

On 10/5/07, Nathaniel Smith <njs@...> wrote:

On Fri, Oct 05, 2007 at 04:13:41PM -0700, Christian Hammond wrote:
>    The piece of code I'm working on now is an experiment at the moment, but
>    it will be part of a full-blown feature we're working on, one which I'm
>    sure will be very popular. I can't talk about the feature in detail
>    (unless either of you want to sign an NDA, which I can probably arrange),
>    as it hasn't been announced yet, but not having this support will have a
>    serious impact on this feature's usability.

I appreciate these issues, but I also hope you realize there are some
fundamental difficulties with convincing a free standards group to
mandate some behavior in N different window manager projects, just on
your word that it will allow one proprietary app to have a totally
cool feature that is best implemented and set in stone in this way
that I came up with and no-one has reviewed, trust me.

I mean, these might be all true... but I'm not sure how, as
hopefully-responsible engineers, we can just accept them without
checking for ourselves, or convince projects that they should bother
implementing/maintaining this feature with no more to go on than your
word.  Do you have any ideas?

If the problem is just that it needs to remain a secret for a while
longer, you may have more luck adding support for a custom (not
_NET_WM_*) extension to a few important window managers, convincing
their maintainers to take the patch, and then worry about pushing it
up into EWMH once the feature is out and there is some real-world
experience with it.

(Also, umm... is "lack of wobbly windows" really a "serious impact"?
Though I guess you can't really explain why things would affect
usability, either.)

-- Nathaniel

--
The Universe may  /  Be as large as they say
But it wouldn't be missed  /  If it didn't exist.
  -- Piet Hein
_______________________________________________
wm-spec-list mailing list
wm-spec-list@...
http://mail.gnome.org/mailman/listinfo/wm-spec-list

--
Christian Hammond - chipx86@...
VMware, Inc.
_______________________________________________
wm-spec-list mailing list
wm-spec-list@...
http://mail.gnome.org/mailman/listinfo/wm-spec-list

[Carsten Haitzler (The Rasterman)]

On Fri, 5 Oct 2007 14:56:15 -0700 "Christian Hammond" <chipx86@...>
babbled:

I'm more interested in why you want to limit the movement/resizes etc.? Are you
sure this is the right thing to do? why do you need to bound the moves/resizes
etc.? Maybe there is a better solution waiting here?

> Hey everyone.
>
> I'm working on an experimental project at VMware that will require us to
> have greater control over a window's position while the window manager is
> dragging it. We'd need to basically give the application more control over
> the movement of the window, yet allow the the window manager to apply
> whatever effects and conditions it would typically apply while moving a
> window. We'd need to let the application, for example, prevent the window
> from going past a certain part of the screen, or to stay within a bounding
> box, or to just prevent it from moving over another window.
>
> We need this for custom shaped windows without standard window decorations.
> Right now, we go the xmms, etc. route and listen to mouse events and
> reposition the window, but 1) we won't be able to move off the left side of
> the screen on Metacity (as Metacity seems to only allow that when it's
> moving the window) and 2) we won't get any window manager effects when
> moving (such as wobbly windows).
>
> Ideally, to allow the greatest flexibility, I'd like to see messages similar
> to _NET_WM_MOVERESIZE that performs any effects the window manager would
> normally use with a moving window, except instead of grabbing the mouse, the
> application would have the responsibility of listening to mouse events and
> moving the window. To the user, it would look like the window is moving
> normally, but the application would be in full control of the move
> operation.
>
> There would be two messages, one for beginning the operation, and one for
> ending it. The _BEGIN operation would be fairly similar to
> _NET_WM_MOVERESIZE, though without the button parameter (I don't think it'd
> be needed)? The _END would merely finalize the _BEGIN operation. It'd be the
> equivalent of releasing the mouse button used to drag the window.
>
> Another option would be to introduce two new flags for the existing
> _NET_WM_MOVERESIZE that would initiate and finalize this operation. Maybe
> this is the right thing to do, so long as there's a way we can detect
> whether this is supported from the application.
>
> Thoughts?
>
> Christian
>
> --
> Christian Hammond - chipx86@...
> VMware, Inc.
>

--
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
裸好多
Tokyo, Japan (東京 日本)
_______________________________________________
wm-spec-list mailing list
wm-spec-list@...
http://mail.gnome.org/mailman/listinfo/wm-spec-list

[Carsten Haitzler (The Rasterman)]

On Fri, 5 Oct 2007 17:22:49 -0700 Nathaniel Smith <njs@...> babbled:

> On Fri, Oct 05, 2007 at 04:13:41PM -0700, Christian Hammond wrote:
> >    The piece of code I'm working on now is an experiment at the moment, but
> >    it will be part of a full-blown feature we're working on, one which I'm
> >    sure will be very popular. I can't talk about the feature in detail
> >    (unless either of you want to sign an NDA, which I can probably arrange),
> >    as it hasn't been announced yet, but not having this support will have a
> >    serious impact on this feature's usability.
>
> I appreciate these issues, but I also hope you realize there are some
> fundamental difficulties with convincing a free standards group to
> mandate some behavior in N different window manager projects, just on
> your word that it will allow one proprietary app to have a totally
> cool feature that is best implemented and set in stone in this way
> that I came up with and no-one has reviewed, trust me.
>
> I mean, these might be all true... but I'm not sure how, as
> hopefully-responsible engineers, we can just accept them without
> checking for ourselves, or convince projects that they should bother
> implementing/maintaining this feature with no more to go on than your
> word.  Do you have any ideas?

agreed. i am skeptical of the original need for this - why is it needed? is
there a better way? is there just some fundamental mistake/mis-understanding
behind all of this? can we get some of the why's without an nda being violated?

> If the problem is just that it needs to remain a secret for a while
> longer, you may have more luck adding support for a custom (not
> _NET_WM_*) extension to a few important window managers, convincing
> their maintainers to take the patch, and then worry about pushing it
> up into EWMH once the feature is out and there is some real-world
> experience with it.
>
> (Also, umm... is "lack of wobbly windows" really a "serious impact"?
> Though I guess you can't really explain why things would affect
> usability, either.)

consistency is the problem there. xmms and friends create no end of issues form
users and when they move them and they don't quite behave like other windows
(don't resist, display a co-ords overlay, etc. etc.)

> -- Nathaniel
>
> --
> The Universe may  /  Be as large as they say
> But it wouldn't be missed  /  If it didn't exist.
>   -- Piet Hein
> _______________________________________________
> wm-spec-list mailing list
> wm-spec-list@...
> http://mail.gnome.org/mailman/listinfo/wm-spec-list
>

--
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
裸好多
Tokyo, Japan (東京 日本)
_______________________________________________
wm-spec-list mailing list
wm-spec-list@...
http://mail.gnome.org/mailman/listinfo/wm-spec-list

[Carsten Haitzler (The Rasterman)]

On Fri, 5 Oct 2007 15:13:00 -0700 "Christian Hammond" <chipx86@...>
babbled:

> On 10/5/07, Havoc Pennington <hp@...> wrote:
> >
> > Hi,
> >
> > Christian Hammond wrote:
> > > I'm working on an experimental project at VMware that will require us to
> > > have greater control over a window's position while the window manager
> > > is dragging it. We'd need to basically give the application more control
> > > over the movement of the window, yet allow the the window manager to
> > > apply whatever effects and conditions it would typically apply while
> > > moving a window. We'd need to let the application, for example, prevent
> > > the window from going past a certain part of the screen, or to stay
> > > within a bounding box, or to just prevent it from moving over another
> > > window.
> >
> > It would seem more logical to me to convey these constraints to the WM,
> > rather than adding an "escape hatch" that potentially leaves the WM with
> > no clue what's going on or why.
> >
> > I'd guess metacity would have a really hard time knowing how to
> > implement the "escape hatch" - most of the time, the move constraints in
> > the metacity code apply even to moves originated by the WM itself.
> >
> > Certainly the codepaths related to this are "interesting" enough already.
> >
> > Is it possible to develop a more semantic hint instead of the escape
> > hatch? (without more detail on the project, it's tough for me to suggest
> > anything.)
> >
> > Havoc
> >
> >
> The problem is, we won't necessarily know these restrictions before we're
> dragging. We only have some idea of them as we drag, as we hit them.
>
> If we go to a more traditional example, let's look at XMMS. It consists of
> multiple, separate windows that stick together while dragging. They have to
> handle move events themselves in order to guarantee the windows will stick

to GUARANTEE - yes. but with sufficiently smart wm's and some hints the wm
could do this too.

> together to bind them, and to move all the windows at the same time. They
> wouldn't be able to do this with _NET_WM_MOVERESIZE, and are forced to act
> in such a way where the window manager will have no idea that they're
> dragging.

correct. all the wm knows is that the app requests to be in a new location, and
nicely obliges. sometimes it doesn't oblige depending on the wm, policies and
user settings.

> We have this problem to a much higher degree. Not only do we need to support
> this case, but more complex cases where windows can't reach positions on the
> screen. And we just don't know until the user tries to drag it there.

we have a race condition here too. if you let the wm to do the move - there
will be a delay between you updating where it can and can't move things, the wm
getting such a message and then enforcing it. so it would be possible to just
set a property on your window that is a list of "allowed rect regions" (you
will need to list all the allowed regions for the window to be moved within,
and re-set this property with a new list whenever that changes, and hope the wm
enforces it. there will be a delay between a mouse move, then window move and
it could be a window will violate a non-move region before you update to say it
shouldn't go there. are such violations (or race conditions/temporary
violations) acceptable? or are they a hard limit? in which case you are talking
a round trip via x to the wm and back per mouse motion event where the wm sends
you a new location and your app ok's it or not. ugly imho.... :(

so i see 2 possible ways this could work - taking it on faith that you really
do need to limit a window's location on screen to a non-rectangular subset of
the screen:

1. set property that is a list of rectangles where the window is allowed to go.
the wm' must listen for updates to this property and never set it itself, and
keep the window within the region defined by that list of rectangles at all
times, even during moves and resizes. we have a race condition here - but that's
probably better than method 2:

2. wm WANTS window to go to x,y - it sends request to client, client responds
with yes or no. if yes, wm can move there. round trips to x and then over to
client, then back to x and back to wm per move. latency heaven. not pretty at
all.

oh yeah - does the region include the window frame or not? can the frame go
outside of it?

> Christian
>
> --
> Christian Hammond - chipx86@...
> VMware, Inc.
>


--
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
裸好多
Tokyo, Japan (東京 日本)
_______________________________________________
wm-spec-list mailing list
wm-spec-list@...
http://mail.gnome.org/mailman/listinfo/wm-spec-list

[Havoc Pennington]

Hi,

Christian Hammond wrote:
> The thing to keep in mind is that there are several applications that
> already bypass the window manager's moving logic and implement it
> themselves. XMMS is a good example because of how it deals with moving
> multiple windows simultaneously and binds windows together when they
> come in contact.

FWIW I think there was some past discussion (here, or maybe in bugzilla?
  it was years ago) of XMMS. One suggestion I remember was some kind of
"glue" hints - something like "stick this window to this other window on
these edges"

In any case, if you develop a prototype patch for some WMs and post the
patches, that might help illustrate the impact of this proposal in terms
of code complexity and maintenance headaches.

I'd strongly prefer some kind of semantic hints to an "escape hatch"
though. Escape hatches are both very open to abuse, and hard for WM
authors to maintain since it's unclear when to honor them or what the
apps using them are trying to accomplish, so smart default behavior (or
even sane behavior at all) becomes a big headache to figure out. When
making a behavior decision for an "escape hatched" window, it's never
clear whether it will break some apps or what, because you just don't
know what those apps are doing.

App authors always want an escape hatch so they can just proceed to do
their thing with no further WM involvement. But the problem is that the
screen is a shared resource. The WM and the X server and associated
specs are needed, along with some amount of restrictive policy, because
multiple apps need to be coordinated.

So ultimately apps just can't do arbitrary things. UI decisions have
been made about how the desktop as a whole works, and often if an app
wants to do something really original, the desktop as a whole needs to
be modified or revisited.

It's just like global keybindings; some resources (and decisions) are
inherently global and can't be made by apps in isolation.

My suspicion is that an app like VMWare would strongly benefit from
being designed together with the whole desktop; it is effectively an
idea similar to workspaces, and is in part allocating a shared global
resource (screen and keyboard) between the virtual and physical machine.
In my usage of vmware, I might like to essentially have the VM be an
extra workspace, for example.

Of course, since VMWare is cross-platform and not part of the desktop,
this is challenging. But perhaps it could still be done - essentially
hardcode the idea of a "VM manager" and "VM app" in the panel, WM, etc.
and do smart things there. It could be pluggable for VMWare vs.
alternatives.

I would rather have virtual-machine-specific hints than "escape hatch"
hints.

Sorry to speak abstractly and philosophically, but since I don't know
the concrete details in this case ;-)

Havoc
_______________________________________________
wm-spec-list mailing list
wm-spec-list@...
http://mail.gnome.org/mailman/listinfo/wm-spec-list