Touch and gestures events

[Olli Pettay]

Hi all,

there seems to be some interest to standardize touch and gesture events.
They are becoming more common on OS level: Windows 7 has them, iPhone
has them, OSX has at least gestures, X is getting support for them etc.

So perhaps there should be a new spec for touch and gesture events.
Hopefully a pretty simple spec.

It is not yet quite clear what the events should look like, but maybe
something like this:

interface PointerEvent : UIEvent {
         readonly attribute long streamId;

         readonly attribute long screenX;
         readonly attribute long screenY;

         readonly attribute long clientX;
         readonly attribute long clientY;

         readonly attribute boolean ctrlKey;
         readonly attribute boolean shiftKey;
         readonly attribute boolean altKey;
         readonly attribute boolean metaKey;
         void initPointerEvent(...);
};

interface TouchEvent : PointerEvent {
         readonly attribute float pressure;
         readonly attribute long size;
         void initTouchEvent(...);
};
Touch Event types: touchdown, touchmove, touchup, touchover, touchout


interface GestureEvent : PointerEvent {
          // degrees
         readonly attribute float direction;
         readonly attribute float delta;
         void initGestureEvent(...);
};
Gesture Event types: gestureStart, gestureEnd, pan, rotate, zoom, swipe...


Touch Event handlers might want to know all the current touch positions,
but even then, all the events should be dispatched, so that
events propagate properly also for example through (XBL2) anonymous
content. Perhaps defaultView (window) could have a property which lists
current touches. (I don't like iPhone's touch events which contain all
sort of list of touches.)

Gesture Events have the problem where to dispatch the events. Should
gestureStart be dispatched to the same node as gestureEnd? What if the
node is removed, or moved to another document? User is still doing the
gesture, so the event should be dispatched to somewhere. Where should
other gesture events be dispatched? Whatever is "under the gesture"?

Any comments, ideas?

-Olli

[Garrett]

On Wed, Jun 17, 2009 at 1:22 PM, Olli Pettay <Olli.Pettay@...> wrote:
> Hi all,
>
> there seems to be some interest to standardize touch and gesture events.

Please provide links to where interest was indicated.

Especially valuable are where mobile companies have provided
information about the hardware technology and how it relates to the
touch evetns API.

> They are becoming more common on OS level: Windows 7 has them, iPhone has
> them, OSX has at least gestures, X is getting support for them etc.
>

Can Touch and Gesture events be separated, or are they necessarily coupled?

> So perhaps there should be a new spec for touch and gesture events.
> Hopefully a pretty simple spec.

Would it be simpler to have a touch events as a stand-alone standard?

>
> It is not yet quite clear what the events should look like, but maybe
> something like this:
>
> interface PointerEvent : UIEvent {
>        readonly attribute long streamId;
>
>        readonly attribute long screenX;
>        readonly attribute long screenY;
>
>        readonly attribute long clientX;
>        readonly attribute long clientY;
>
>        readonly attribute boolean ctrlKey;
>        readonly attribute boolean shiftKey;
>        readonly attribute boolean altKey;
>        readonly attribute boolean metaKey;
>        void initPointerEvent(...);
> };
>
> interface TouchEvent : PointerEvent {
>        readonly attribute float pressure;
>        readonly attribute long size;
>        void initTouchEvent(...);
> };
> Touch Event types: touchdown, touchmove, touchup, touchover, touchout
>

A blatant violation of US Patent: US20090225039.


>
> Touch Event handlers might want to know all the current touch positions, but

The Event instance does not need to know the current touch positions,
but the callback does. The current event positions could easily be
stored in a place where they can be polled, either on the Event, a
View, or some other object.

> even then, all the events should be dispatched, so that
> events propagate properly also for example through (XBL2) anonymous content.

For those of us who are unfamiliar with how events are propagated with
XBL2 anonymous content, can you please explain?

> Perhaps defaultView (window) could have a property which lists
> current touches.

This allows the callback to check that property (currentTouches, or
whatever), on window.

(I don't like iPhone's touch events which contain all sort
> of list of touches.)
>
No, fortunately that API is patented so it won't be usable (and hence,
will not be used as a public standard).

The legality of that Patent seems questionable, however. Itself
borrows from existing D3E API and idiom, so it is not completely
original.

> Gesture Events have the problem where to dispatch the events. Should
> gestureStart be dispatched to the same node as gestureEnd? What if the node
> is removed, or moved to another document? User is still doing the gesture,
> so the event should be dispatched to somewhere. Where should other gesture
> events be dispatched? Whatever is "under the gesture"?
>
> Any comments, ideas?
>
Apple has patented Touch events under US Patent: US20090225039.

I do not mind the innovation so much, but the fact that it is
proprietary leads to difficulty for cross-device scripting. If the
proliferation of touch screen devices were matched 1:1 with
proprietary touch event API, the result will be greater divergence
than was seen during the Great Browser Wars.

(I see Palm now has constant Event.MOUSE_DRAG).

Has it been so long since the Browser Wars of IE and Netscape that the
lesson has been forgotten? Were the consequences not understood deeply
enough?

Perhaps Maciej or other Apple developers can comment on some of the following.

* initTouchEvent takes 18 parameter variables. Three of these
variables are "TouchList" variables. Each TouchList is comprised of
one or more Touch instances. Was a simpler API considered?

* the TouchEvent payload for the eventListener does not have the
interesting data itself. Instead, the interesting data is on the
"changedTouches" TouchList. The callback must address this complexity
by hanling not only a touch event, but a touch event which is not
directly useful (the information the program needs is in the touches
or changedTouches).

I would like to see input from Palm, RIM, Opera, and other developers
of mobile browsers and touch screen devices on this thread.

Garrett

[Olli Pettay]

On 10/14/09 3:40 AM, Garrett Smith wrote:

>
> A blatant violation of US Patent: US20090225039.
>


Thanks for this information! I was a bit afraid of that Apple
might have done something evil like this.
[1] seems to be dated Sept. 10 2009.



-Olli

[1]
http://www.patents.com/TOUCH-EVENT-MODEL-PROGRAMMING-INTERFACE/US20090225039/en-US/

[Maciej Stachowiak]

On Oct 14, 2009, at 2:00 AM, Olli Pettay wrote:

> On 10/14/09 3:40 AM, Garrett Smith wrote:
>
>>
>> A blatant violation of US Patent: US20090225039.
>>
>
>
> Thanks for this information! I was a bit afraid of that Apple
> might have done something evil like this.
> [1] seems to be dated Sept. 10 2009.
>
>
>
> -Olli
>
> [1] http://www.patents.com/TOUCH-EVENT-MODEL-PROGRAMMING-INTERFACE/US20090225039/en-US/
>

I note that Apple is a member of the Web Apps Working Group and thus  
subject to the W3C Patent Policy with respect to Web Apps WG  
deliverables. I can't comment on whether Apple would exclude any  
patent claims relating to this area, if they turn out to be essential  
claims.

Regards,
Maciej

[kari.hiitola]

Hi,

I suppose that the interest Olli mentioned was ours (Nokia). Unfortunately there was a long delay before we were able to participate the discussion and release our implementation, but yes, we have previously discussed touch events with Olli. We would be interested in participating standardization of touch and gesture events.

We have implemented touch and manipulate (including pan, scale and rotate) events on top of Qt 4.6 and WebKit, and the open-sourced results can be seen in http://opensource.nokia.com/Starlight . Working binaries for Windows 7 (on Dell Latitude XT/XT2, some HP TouchSmarts), source, videos and some documentation can be found by following the links there. For the period when there is no existing standard, we also funded an event abstraction library in Script.aculo.us. Its release post is here: http://mir.aculo.us/2009/10/15/pinch-twist-zoom-bringing-multitouch-to-a-browser-near-you/

The approach that we had was a bit different from Olli’s, but I see a lot of merit in his proposal, especially in extracting the common parameters to PointerEvent. Naturally there are also some improvement ideas:

In TouchEvent, instead of size there should be separate height and width, because touch area is not necessarily circle or square. Of course that is also an approximation, but in real life mostly accurate enough. Having a bitmap of the touch area is a natural next step, but I think it isn’t needed for now, and futureproofing usually fails miserably.

It might require a bit too much violence to fit the attributes of pan, zoom, rotate, and swipe all to parameters direction and delta. We have panX/Y delta coordinates, scale factor, rotation angle, and the speed for each of them. Our event types are only manipulate start/move/end instead of telling in the event type what kind of change happened. I think that most of the time all of these happen in parallel, but I see a point in being able to be notified e.g. only when pan coordinates change.

The manipulate/gesture naming is of course a matter of taste, but we wanted to give a name according to what is done (manipulate) instead of what secret handshake is needed to do it (gesture). This way the events are mentally less tightly bound to any UI technology. For instance, our implementation starts panning with one finger moving instead of requiring two fingers to be present to start what can be called a gesture. The same manipulate stream can be continued by inserting a second finger and rotating. In another UI style pan and rotate could be as well done by touching the screen and rotating a jog wheel.

To mention some omissions of our implementation, there can only be one manipulate event stream happening any given time, and there is no touchover/touchout implemented.

Let’s hope that now there could be a real discussion about the topic. Let’s not make premature conclusions about anyone’s patent applications.

 - Kari Hiitola
The architect of Nokia’s Starlight project



On 10/14/09 3:40, "ext Garrett Smith" <dhtmlkitchen@...> wrote:

On Wed, Jun 17, 2009 at 1:22 PM, Olli Pettay <Olli.Pettay@...> wrote:
> Hi all,
>
> there seems to be some interest to standardize touch and gesture events.

Please provide links to where interest was indicated.

Especially valuable are where mobile companies have provided
information about the hardware technology and how it relates to the
touch evetns API.

> They are becoming more common on OS level: Windows 7 has them, iPhone has
> them, OSX has at least gestures, X is getting support for them etc.
>

Can Touch and Gesture events be separated, or are they necessarily coupled?

> So perhaps there should be a new spec for touch and gesture events.
> Hopefully a pretty simple spec.

Would it be simpler to have a touch events as a stand-alone standard?

>
> It is not yet quite clear what the events should look like, but maybe
> something like this:
>
> interface PointerEvent : UIEvent {
>        readonly attribute long streamId;
>
>        readonly attribute long screenX;
>        readonly attribute long screenY;
>
>        readonly attribute long clientX;
>        readonly attribute long clientY;
>
>        readonly attribute boolean ctrlKey;
>        readonly attribute boolean shiftKey;
>        readonly attribute boolean altKey;
>        readonly attribute boolean metaKey;
>        void initPointerEvent(...);
> };
>
> interface TouchEvent : PointerEvent {
>        readonly attribute float pressure;
>        readonly attribute long size;
>        void initTouchEvent(...);
> };
> Touch Event types: touchdown, touchmove, touchup, touchover, touchout
>

A blatant violation of US Patent: US20090225039.


>
> Touch Event handlers might want to know all the current touch positions, but

The Event instance does not need to know the current touch positions,
but the callback does. The current event positions could easily be
stored in a place where they can be polled, either on the Event, a
View, or some other object.

> even then, all the events should be dispatched, so that
> events propagate properly also for example through (XBL2) anonymous content.

For those of us who are unfamiliar with how events are propagated with
XBL2 anonymous content, can you please explain?

> Perhaps defaultView (window) could have a property which lists
> current touches.

This allows the callback to check that property (currentTouches, or
whatever), on window.

(I don't like iPhone's touch events which contain all sort
> of list of touches.)
>
No, fortunately that API is patented so it won't be usable (and hence,
will not be used as a public standard).

The legality of that Patent seems questionable, however. Itself
borrows from existing D3E API and idiom, so it is not completely
original.

> Gesture Events have the problem where to dispatch the events. Should
> gestureStart be dispatched to the same node as gestureEnd? What if the node
> is removed, or moved to another document? User is still doing the gesture,
> so the event should be dispatched to somewhere. Where should other gesture
> events be dispatched? Whatever is "under the gesture"?
>
> Any comments, ideas?
>
Apple has patented Touch events under US Patent: US20090225039.

I do not mind the innovation so much, but the fact that it is
proprietary leads to difficulty for cross-device scripting. If the
proliferation of touch screen devices were matched 1:1 with
proprietary touch event API, the result will be greater divergence
than was seen during the Great Browser Wars.

(I see Palm now has constant Event.MOUSE_DRAG).

Has it been so long since the Browser Wars of IE and Netscape that the
lesson has been forgotten? Were the consequences not understood deeply
enough?

Perhaps Maciej or other Apple developers can comment on some of the following.

* initTouchEvent takes 18 parameter variables. Three of these
variables are "TouchList" variables. Each TouchList is comprised of
one or more Touch instances. Was a simpler API considered?

* the TouchEvent payload for the eventListener does not have the
interesting data itself. Instead, the interesting data is on the
"changedTouches" TouchList. The callback must address this complexity
by hanling not only a touch event, but a touch event which is not
directly useful (the information the program needs is in the touches
or changedTouches).

I would like to see input from Palm, RIM, Opera, and other developers
of mobile browsers and touch screen devices on this thread.

Garrett

[Garrett]

On Thu, Oct 15, 2009 at 7:33 AM,  <kari.hiitola@...> wrote:

>
> Hi,
>
> I suppose that the interest Olli mentioned was ours (Nokia). Unfortunately
> there was a long delay before we were able to participate the discussion and
> release our implementation, but yes, we have previously discussed touch
> events with Olli. We would be interested in participating standardization of
> touch and gesture events.
>
> We have implemented touch and manipulate (including pan, scale and rotate)
> events on top of Qt 4.6 and WebKit, and the open-sourced results can be seen
> in http://opensource.nokia.com/Starlight .

What is that?

| Directly manipulated multitouch user interfaces, combined
| with rich visual content, are today's trend in mobile devices

Trend based programming is all the rage.

Working binaries for Windows 7
> (on Dell Latitude XT/XT2, some HP TouchSmarts), source, videos and some
> documentation can be found by following the links there. For the period when
> there is no existing standard, we also funded an event abstraction library

The decision came from (misguided individuals in) Nokia USA?

> in Script.aculo.us. Its release post is here:
> http://mir.aculo.us/2009/10/15/pinch-twist-zoom-bringing-multitouch-to-a-browser-near-you/

What problem does this solve?

>
> The approach that we had was a bit different from Olli’s, but I see a lot of
> merit in his proposal, especially in extracting the common parameters to
> PointerEvent. Naturally there are also some improvement ideas:
>
> In TouchEvent, instead of size there should be separate height and width,
> because touch area is not necessarily circle or square. Of course that is
> also an approximation, but in real life mostly accurate enough. Having a
> bitmap of the touch area is a natural next step, but I think it isn’t needed
> for now, and futureproofing usually fails miserably.
>
> It might require a bit too much violence to fit the attributes of pan, zoom,
> rotate, and swipe all to parameters direction and delta. We have panX/Y
> delta coordinates, scale factor, rotation angle, and the speed for each of
> them. Our event types are only manipulate start/move/end instead of telling
> in the event type what kind of change happened. I think that most of the
> time all of these happen in parallel, but I see a point in being able to be
> notified e.g. only when pan coordinates change.
>
> The manipulate/gesture naming is of course a matter of taste, but we wanted
> to give a name according to what is done (manipulate) instead of what secret
> handshake is needed to do it (gesture). This way the events are mentally
> less tightly bound to any UI technology. For instance, our implementation
> starts panning with one finger moving instead of requiring two fingers to be
> present to start what can be called a gesture. The same manipulate stream
> can be continued by inserting a second finger and rotating. In another UI
> style pan and rotate could be as well done by touching the screen and
> rotating a jog wheel.
>
> To mention some omissions of our implementation, there can only be one
> manipulate event stream happening any given time, and there is no
> touchover/touchout implemented.
>
> Let’s hope that now there could be a real discussion about the topic.

[snip]

A "real discussion"?

I would think that would include identifing problems and have them
post the problems with a few alternatives.

A productive discussion would include reasons and insight to the
design of touch event, including problem scenario and proposed
solution.

The one known existing api (Apple's) for touch event has not been
justified (and justificaiton for specific parts was directly asked, by
me, above, in this trhead).

Garrett

[Doug Schepers-3]

Hi, Kari-

That's very cool stuff.  Thanks for letting us know about that... I hope
that if we do indeed standardize multitouch and gesture events, you will
lend us your practical experience in specifying them.

It's great that you've opened the source on the multitouch stuff... I
don't suppose you have Mac binaries?  What sort of hardware is needed to
support it?  Are there any restrictions on the OS level, or can you
enable multitouch on any OS?

Regards-
-Doug Schepers
W3C Team Contact, SVG and WebApps WGs


kari.hiitola@... wrote (on 10/15/09 10:33 AM):

>
> Hi,
>
> I suppose that the interest Olli mentioned was ours (Nokia).
> Unfortunately there was a long delay before we were able to participate
> the discussion and release our implementation, but yes, we have
> previously discussed touch events with Olli. We would be interested in
> participating standardization of touch and gesture events.
>
> We have implemented touch and manipulate (including pan, scale and
> rotate) events on top of Qt 4.6 and WebKit, and the open-sourced results
> can be seen in http://opensource.nokia.com/Starlight . Working binaries
> for Windows 7 (on Dell Latitude XT/XT2, some HP TouchSmarts), source,
> videos and some documentation can be found by following the links there.
> For the period when there is no existing standard, we also funded an
> event abstraction library in Script.aculo.us. Its release post is here:
> http://mir.aculo.us/2009/10/15/pinch-twist-zoom-bringing-multitouch-to-a-browser-near-you/
>
>
> The approach that we had was a bit different from Olli’s, but I see a
> lot of merit in his proposal, especially in extracting the common
> parameters to PointerEvent. Naturally there are also some improvement ideas:
>
> In TouchEvent, instead of size there should be separate height and
> width, because touch area is not necessarily circle or square. Of course
> that is also an approximation, but in real life mostly accurate enough.
> Having a bitmap of the touch area is a natural next step, but I think it
> isn’t needed for now, and futureproofing usually fails miserably.
>
> It might require a bit too much violence to fit the attributes of pan,
> zoom, rotate, and swipe all to parameters direction and delta. We have
> panX/Y delta coordinates, scale factor, rotation angle, and the speed
> for each of them. Our event types are only manipulate start/move/end
> instead of telling in the event type what kind of change happened. I
> think that most of the time all of these happen in parallel, but I see a
> point in being able to be notified e.g. only when pan coordinates change.
>
> The manipulate/gesture naming is of course a matter of taste, but we
> wanted to give a name according to what is done (manipulate) instead of
> what secret handshake is needed to do it (gesture). This way the events
> are mentally less tightly bound to any UI technology. For instance, our
> implementation starts panning with one finger moving instead of
> requiring two fingers to be present to start what can be called a
> gesture. The same manipulate stream can be continued by inserting a
> second finger and rotating. In another UI style pan and rotate could be
> as well done by touching the screen and rotating a jog wheel.
>
> To mention some omissions of our implementation, there can only be one
> manipulate event stream happening any given time, and there is no
> touchover/touchout implemented.
>
> Let’s hope that now there could be a real discussion about the topic.
> Let’s not make premature conclusions about anyone’s patent applications.
>
> - Kari Hiitola
> The architect of Nokia’s Starlight project

[kari.hiitola]

Hi,

Sorry, no mac binaries, as there is no touch screen Mac hardware available. Multi-touch trackpad support would be well possible, but it wouldn’t be quite the same.

Our implementation should work on any hardware where Windows 7 touch events are supported. We have been developing on Dell Latitude XT & XT2, and HP TouchSmart iq512. TouchSmart supports 2 fingers, Latitudes as many as you need.

 - Kari Hiitola


On 10/16/09 23:21, "ext Doug Schepers" <schepers@...> wrote:

Hi, Kari-

That's very cool stuff.  Thanks for letting us know about that... I hope
that if we do indeed standardize multitouch and gesture events, you will
lend us your practical experience in specifying them.

It's great that you've opened the source on the multitouch stuff... I
don't suppose you have Mac binaries?  What sort of hardware is needed to
support it?  Are there any restrictions on the OS level, or can you
enable multitouch on any OS?

Regards-
-Doug Schepers
W3C Team Contact, SVG and WebApps WGs


kari.hiitola@... wrote (on 10/15/09 10:33 AM):
>
> Hi,
>
> I suppose that the interest Olli mentioned was ours (Nokia).
> Unfortunately there was a long delay before we were able to participate
> the discussion and release our implementation, but yes, we have
> previously discussed touch events with Olli. We would be interested in
> participating standardization of touch and gesture events.
>
> We have implemented touch and manipulate (including pan, scale and
> rotate) events on top of Qt 4.6 and WebKit, and the open-sourced results
> can be seen in http://opensource.nokia.com/Starlight . Working binaries
> for Windows 7 (on Dell Latitude XT/XT2, some HP TouchSmarts), source,
> videos and some documentation can be found by following the links there.
> For the period when there is no existing standard, we also funded an
> event abstraction library in Script.aculo.us. Its release post is here:
> http://mir.aculo.us/2009/10/15/pinch-twist-zoom-bringing-multitouch-to-a-browser-near-you/
>
>
> The approach that we had was a bit different from Olli’s, but I see a
> lot of merit in his proposal, especially in extracting the common
> parameters to PointerEvent. Naturally there are also some improvement ideas:
>
> In TouchEvent, instead of size there should be separate height and
> width, because touch area is not necessarily circle or square. Of course
> that is also an approximation, but in real life mostly accurate enough.
> Having a bitmap of the touch area is a natural next step, but I think it
> isn’t needed for now, and futureproofing usually fails miserably.
>
> It might require a bit too much violence to fit the attributes of pan,
> zoom, rotate, and swipe all to parameters direction and delta. We have
> panX/Y delta coordinates, scale factor, rotation angle, and the speed
> for each of them. Our event types are only manipulate start/move/end
> instead of telling in the event type what kind of change happened. I
> think that most of the time all of these happen in parallel, but I see a
> point in being able to be notified e.g. only when pan coordinates change.
>
> The manipulate/gesture naming is of course a matter of taste, but we
> wanted to give a name according to what is done (manipulate) instead of
> what secret handshake is needed to do it (gesture). This way the events
> are mentally less tightly bound to any UI technology. For instance, our
> implementation starts panning with one finger moving instead of
> requiring two fingers to be present to start what can be called a
> gesture. The same manipulate stream can be continued by inserting a
> second finger and rotating. In another UI style pan and rotate could be
> as well done by touching the screen and rotating a jog wheel.
>
> To mention some omissions of our implementation, there can only be one
> manipulate event stream happening any given time, and there is no
> touchover/touchout implemented.
>
> Let’s hope that now there could be a real discussion about the topic.
> Let’s not make premature conclusions about anyone’s patent applications.
>
> - Kari Hiitola
> The architect of Nokia’s Starlight project