Quasar

Hey,

as some of you might have noticed it's a beautiful day (where I'm at right
now, I don't care about you). Something else you might have noticed is that
Matthias and I have been working on Quasar a bit lately and I feel we're
slowly moving towards a "kinda working" stage which is good (not "sexy" but
I'm sexy and bad, so clearly one doesn't have to imply the other).

Simply put Quasar is a presentation layer. What that means is that is handles
display and intermediate manipulation of the rendered data - be it effects
(like blur, saturation...) or transformations (e.g. scaling, rotation,
waving...). Because of a huge interest in animation and a number of code that
tries to handle it (neat things in Plasma and the new code Qt) Quasar doesn't
handle animations itself. The framework to do them can be built either on top
of it or alongside it. So again, Quasar is a presentation layer - how it
should present it is something that things on top of it should decide.

Quasar is a purely OpenGL creation. I thought about software filters for a
while, then I decided that I was dumb for even entertaining the thought of
using software filters. There is no software fallbacks in Quasar - absolutely
everything goes through OpenGL.

Filters in Quasar are simply GL shaders.

The way Quasar works internally is that everything is a Node that has an
execute method. Nodes can be connected much in a similar way to the way
QObject's signals/slots can be connected. A graph of those connections is
created and upon execution Quasar goes over the graph, allocates surfaces
(framebuffer objects) necessary to perform actions and executes the nodes.
The output of the graph can be automatically displayed (by adding
Quasar::RenderOutputNode to the graph) or saved to a QImage (by adding
Quasar::ImageOutputNode to the graph).


There's a lot of things missing right now in Quasar, in particular:

- API is utter crap - I was mostly adding GL code and because I know GL code
in and out I'm a rather bad person to design it. My knowledge of graphics
hardware from the bottom up means that whatever graphics API I'll design will
likely unnecessarily expose the internals.

- Graph scheduling/iteration algorithm are non-existant - currently Quasar
will assert if the graph is not linear

- No cleanups - tons of stuff isn't being deleted right now

- No application to show off Quasar - it'd be awesome if we had an application
that simply allowed loading of an image, displayed a list of filters in one
of the corners and a very simple property editor to manipulate properties of
the filters and allowed people to play with those.

- Very few filters - I just wrote things that I needed for testing

- Some GL code still missing, in particular texture transfers aren't using
pixel buffer objects, which especially for video is a bottleneck. I'll
address that next.

- Lack of ability to load a Quasar::Texture from a QPixmap. We need code that
uses GLX_EXT_texture_from_pixmap to create a gl texture from a x pixmap for
that.

If you have any questions, comments or requests it'd be best to send them to
kde-graphics-devel@....
Also if you'll have any patches (which would be wicked awesome) I'd appreciate
if I could get a chance to review them before they go in.

z
_______________________________________________
Kde-graphics-devel mailing list
Kde-graphics-devel@...
https://mail.kde.org/mailman/listinfo/kde-graphics-devel

On Monday 09 June 2008, Zack Rusin wrote:

> The way Quasar works internally is that everything is a Node that has an
> execute method. Nodes can be connected much in a similar way to the way
> QObject's signals/slots can be connected. A graph of those connections is
> created and upon execution Quasar goes over the graph, allocates surfaces
> (framebuffer objects) necessary to perform actions and executes the nodes.
> The output of the graph can be automatically displayed (by adding
> Quasar::RenderOutputNode to the graph) or saved to a QImage (by adding
> Quasar::ImageOutputNode to the graph).
>
>
> There's a lot of things missing right now in Quasar, in particular:
>
> - API is utter crap - I was mostly adding GL code and because I know GL
> code in and out I'm a rather bad person to design it. My knowledge of
> graphics hardware from the bottom up means that whatever graphics API I'll
> design will likely unnecessarily expose the internals.

Well, for the users of Quasar GL should probably be hidden. But for node
implementors there won't be a way around GL anyway. So perhaps it helps to
keep in mind that there are two Quasar APIs.

The main problem with using the API I have at the moment is that the
application, if it wants to use the RenderOutputNode, needs to create a
QGLWidget and is solely responsible for redraws.
suggestions:
- add a Quasar::Composition::graphChanged() signal. Then
Node::setPropertyValue would trigger that signal as could a node
implementation by itself (PhononInputNode would need to trigger that signal
with every new frame).
- add a Quasar::RenderWidget which redraws and resizes/sets the size hint
automatically and which hides OpenGL for the ignorant user.

Regarding FilterNode implementations:
a) FilterNode is a very handy class and I'd like to use it where possible, but
sometimes it would make sense to not hide the Mesh to the FilterNode impl but
let it render the vertices itself. I was thinking of e.g. a rotation filter
where you probably don't want to calculate the rotation of every vertex in
the shader but calculate it once when the rotation angle is set and then
every reevaluation of the filter is as fast as it can get. Also, an impl
might want to use different texture coordinates (flip the image,
stretch, ...) or pass other vertex parameters (i.e. not uniforms).

b) did you fix the viewport and projection matrix for fbos? I.e. per default
every fbo should use a viewport of the dimensions of its attached texture and
a glOrtho projection of the same dimensions. Alternatively the projection
could also be glOrtho(0, 1, 0, 1, -1, 1). Depending on the node this might
make sense, so it would be good if the node can override the projection
matrix easily.

Ok, that's what I came up with for now, hope it's useful. :-)

Hmm, thinking about UI overlays on videos (like most flash players do these
days). Would it be in scope of Quasar to add event handling to the nodes?
Like let the RenderWidget pass mouse and key events to the composition which
would somehow handle it/let the right nodes handle it? Would it then make
sense to create UI elements as Quasar nodes?

--
________________________________________________________
Matthias Kretz (Germany)                            <><
http://Vir.homelinux.org/
MatthiasKretz@..., kretz@...,
Matthias.Kretz@...


_______________________________________________
Kde-graphics-devel mailing list
Kde-graphics-devel@...
https://mail.kde.org/mailman/listinfo/kde-graphics-devel

On Monday 09 June 2008 08:04:38 am Matthias Kretz wrote:
> Well, for the users of Quasar GL should probably be hidden. But for node
> implementors there won't be a way around GL anyway. So perhaps it helps to
> keep in mind that there are two Quasar APIs.
>
> The main problem with using the API I have at the moment is that the
> application, if it wants to use the RenderOutputNode, needs to create a
> QGLWidget and is solely responsible for redraws.

Yea, that's a good point.

> suggestions:
> - add a Quasar::Composition::graphChanged() signal. Then
> Node::setPropertyValue would trigger that signal as could a node
> implementation by itself (PhononInputNode would need to trigger that signal
> with every new frame).

Yea, definitely. The graphChanged signal should definitely be there.

> - add a Quasar::RenderWidget which redraws and resizes/sets the size hint
> automatically and which hides OpenGL for the ignorant user.

Ideally I'd like to see Quasar widget independent. So I'd much rather have
SomeWidget::SomeWidget(QWidget *parent)
{
  composition.setOutput(this);
}
void SomeWidget::paintEvent(QPainteEvent *e)
{
   composition.execute();
}
and setOutput basically would do:
connect(composition, SIGNAL(graphChanged()),
        widget, SLOT(update()));

To completely avoid any custom widget's. And allow rendering to both QWidget's
and QGraphicsItem's (the latter would be trivial if QGLWidget would be the
viewport for the QGraphicsView on which a given QGraphicsItem resides. For
the latter we'll need some magic in RenderOutputNode);

> Regarding FilterNode implementations:
> a) FilterNode is a very handy class and I'd like to use it where possible,
> but sometimes it would make sense to not hide the Mesh to the FilterNode
> impl but let it render the vertices itself. I was thinking of e.g. a
> rotation filter where you probably don't want to calculate the rotation of
> every vertex in the shader but calculate it once when the rotation angle is
> set and then every reevaluation of the filter is as fast as it can get.

I wanted to do geometrical transformations as a vertex shader. Currently for
all filters we're always using a vertex shader with:
gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;
for any other transformation we'd either use:
gl_Position = userPassedMatrix * gl_Vertex;
or
gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex * someTransformation;
GPU will execute the vertex shader anyway, it makes sense to use that.

> Also, an impl might want to use different texture coordinates (flip the
> image, stretch, ...)

Realistically speaking that's also something that should be done in the vertex
shader. Currently we do:
gl_TexCoord[0] = gl_MultiTexCoord0;
which is passing default texture coords. We can do anything we want with them
there.

> or pass other vertex parameters (i.e. not uniforms).

Yea, definitely. I wanted to add a nicer api for custom parameters. But aside
of uniforms probably only varying would make sense for us.


> b) did you fix the viewport and projection matrix for fbos? I.e. per
> default every fbo should use a viewport of the dimensions of its attached
> texture and a glOrtho projection of the same dimensions.

Ah, no! The fbos shouldn't have dimensions of the master texture, they should
have dimensions of the domainOfDefinition and such coordinates.
For example:
ImageInputNode reads image 600x400
GaussianBlurNode filters that image

For gaussian-blur to be correct the dimensions of the produced texture need to
be 600+filter radius / 400 + filter radius. So the master texture will be
600x400 but it will be rendered in the middle of 600+filter radius/400 +
filter radius fbo.
I probably should write some documentation about the point of domain of
definition and region of interest. It was described in detail in this paper:
http://portal.acm.org/citation.cfm?id=192191
(currently it's not really implemented too well, or at all, in Quasar but this
is where we want to get)

> Alternatively the projection could also be glOrtho(0, 1, 0, 1, -1, 1).
> Depending on the node this might make sense, so it would be good if the node
> can override the projection matrix easily.

Any particular reason for it?

> Hmm, thinking about UI overlays on videos (like most flash players do these
> days). Would it be in scope of Quasar to add event handling to the nodes?
> Like let the RenderWidget pass mouse and key events to the composition
> which would somehow handle it/let the right nodes handle it? Would it then
> make sense to create UI elements as Quasar nodes?

I think ideally we'd do this on top of Quasar. Basically to keep it as simple
as possible.
But yea, it would be great to have nodes that do some kind of event handling.
It's just that personally I didn't have a good idea on how to do that yet =)

z
_______________________________________________
Kde-graphics-devel mailing list
Kde-graphics-devel@...
https://mail.kde.org/mailman/listinfo/kde-graphics-devel

On Tuesday 10 June 2008, Zack Rusin wrote:
> On Monday 09 June 2008 08:04:38 am Matthias Kretz wrote:
> > Well, for the users of Quasar GL should probably be hidden. But for node
> > implementors there won't be a way around GL anyway. So perhaps it helps
> > to keep in mind that there are two Quasar APIs.

It occured to me that one is an API and the other an NPI (Node Programmer
Interface) ;-)

> > - add a Quasar::RenderWidget which redraws and resizes/sets the size hint
> > automatically and which hides OpenGL for the ignorant user.
>
> Ideally I'd like to see Quasar widget independent. So I'd much rather have
> SomeWidget::SomeWidget(QWidget *parent)
> {
>   composition.setOutput(this);
> }
> void SomeWidget::paintEvent(QPainteEvent *e)
> {
>    composition.execute();
> }
> and setOutput basically would do:
> connect(composition, SIGNAL(graphChanged()),
> widget, SLOT(update()));
>
> To completely avoid any custom widget's. And allow rendering to both
> QWidget's and QGraphicsItem's (the latter would be trivial if QGLWidget
> would be the viewport for the QGraphicsView on which a given QGraphicsItem
> resides. For the latter we'll need some magic in RenderOutputNode);

Ah, then Composition::setOutput(QWidget *) would check for
qobject_cast<QGLWidget *> and render the textures directly, or if it's not a
GL widget convert the texture to a pixmap and draw that, right?

Things to consider:
- a composition should allow more than one output
- split processing and drawing between Composition::execute() and
Composition::paint(QWidget / QImage / QPixmap / QGLWidget / QPainter). Then
the app could call execute once and paint the result as often as it wants to.

> > Regarding FilterNode implementations:
> > a) FilterNode is a very handy class and I'd like to use it where
> > possible, but sometimes it would make sense to not hide the Mesh to the
> > FilterNode impl but let it render the vertices itself. I was thinking of
> > e.g. a rotation filter where you probably don't want to calculate the
> > rotation of every vertex in the shader but calculate it once when the
> > rotation angle is set and then every reevaluation of the filter is as
> > fast as it can get.
>
> I wanted to do geometrical transformations as a vertex shader. Currently
> for all filters we're always using a vertex shader with:
> gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;
> for any other transformation we'd either use:
> gl_Position = userPassedMatrix * gl_Vertex;
> or
> gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex *
> someTransformation; GPU will execute the vertex shader anyway, it makes
> sense to use that.

Sidenote: One could try to make the vertex shader use less instructions. I
assume the performance gain will be negligible, so feel free to ignore this.
If one were to use an identity matrix for both projection and modelview - the
latter is normal for most filters, the former is the same as using
glOrtho(-1, 1, -1, 1, 1, -1) - one would be able to do "gl_Position =
gl_Vertex;".

Anyway. Here's what I had in mind for rotation:
glMatrixMode(GL_MODELVIEW);
glPushMatrix();
glTranslatef(-width * 0.5, -height * 0.5, 0);
glRotatef(angle, 0, 0, 1);
glTranslatef(width * 0.5, height * 0.5, 0);
mesh->render();
glPopMatrix();

And that's what's currently not easily possible with a FilterNode subclass.

> > b) did you fix the viewport and projection matrix for fbos? I.e. per
> > default every fbo should use a viewport of the dimensions of its attached
> > texture and a glOrtho projection of the same dimensions.
>
> Ah, no! The fbos shouldn't have dimensions of the master texture,

I didn't mean the input texture, but the resulting texture. I.e. the one that
is written into when rendered onto the fbo. The problem was visible with
recent Quasar when you had an image that was greater than the QGLWidget. Then
only the lower left part of that image got processed, the rest of the texture
was a dark grey.

> they
> should have dimensions of the domainOfDefinition and such coordinates.
> For example:
> ImageInputNode reads image 600x400
> GaussianBlurNode filters that image
>
> For gaussian-blur to be correct the dimensions of the produced texture need
> to be 600+filter radius / 400 + filter radius. So the master texture will
> be 600x400 but it will be rendered in the middle of 600+filter radius/400 +
> filter radius fbo.
> I probably should write some documentation about the point of domain of
> definition and region of interest. It was described in detail in this
> paper: http://portal.acm.org/citation.cfm?id=192191
> (currently it's not really implemented too well, or at all, in Quasar but
> this is where we want to get)

I know your plans wrt this, which is why I'm keeping some of my hacks for
myself. :-)

> > Alternatively the projection could also be glOrtho(0, 1, 0, 1, -1, 1).
> > Depending on the node this might make sense, so it would be good if the
> > node can override the projection matrix easily.
>
> Any particular reason for it?

Say you want to do a translation in the vertex shader. If the projection
matrix is glOrtho(0, width, 0, height, ...) then the vertex shader doesn't
know how far it has to move the vertex to do e.g. a 50% translation to the
left. The shader would need to know the image dimensions - those could be
passed as uniforms. But if the vertices are "normalized" to the 1x1 rectangle
the uniforms become unnecessary.

> > Hmm, thinking about UI overlays on videos (like most flash players do
> > these days). Would it be in scope of Quasar to add event handling to the
> > nodes? Like let the RenderWidget pass mouse and key events to the
> > composition which would somehow handle it/let the right nodes handle it?
> > Would it then make sense to create UI elements as Quasar nodes?
>
> I think ideally we'd do this on top of Quasar. Basically to keep it as
> simple as possible.
> But yea, it would be great to have nodes that do some kind of event
> handling. It's just that personally I didn't have a good idea on how to do
> that yet =)

Probably QGraphicsView would be the best framework to integrate with?

--
________________________________________________________
Matthias Kretz (Germany)                            <><
http://Vir.homelinux.org/
MatthiasKretz@..., kretz@...,
Matthias.Kretz@...


_______________________________________________
Kde-graphics-devel mailing list
Kde-graphics-devel@...
https://mail.kde.org/mailman/listinfo/kde-graphics-devel

On Tuesday 10 June 2008 05:34:52 pm Matthias Kretz wrote:
> Ah, then Composition::setOutput(QWidget *) would check for
> qobject_cast<QGLWidget *> and render the textures directly, or if it's not
> a GL widget convert the texture to a pixmap and draw that, right?

Yes, (except that I'd like to see RenderOutputNode do that selection, but
based on what was passed to setOutput) :)

> Things to consider:
> - a composition should allow more than one output

Definitely.

> - split processing and drawing between Composition::execute() and
> Composition::paint(QWidget / QImage / QPixmap / QGLWidget / QPainter). Then
> the app could call execute once and paint the result as often as it wants
> to.

I think this binds with the one above, right? One would basically create
multiple RenderOutputNode's with different targets as output.

> Sidenote: One could try to make the vertex shader use less instructions.

I don't think we should worry about that. The vertex shader constructed for
the "famous" GL gears example has 15 instructions (due to lightning) and it's
running in multiple of hundreds of fps with software GL implementations.

> Anyway. Here's what I had in mind for rotation:
> glMatrixMode(GL_MODELVIEW);
> glPushMatrix();
> glTranslatef(-width * 0.5, -height * 0.5, 0);
> glRotatef(angle, 0, 0, 1);
> glTranslatef(width * 0.5, height * 0.5, 0);
> mesh->render();
> glPopMatrix();
>
> And that's what's currently not easily possible with a FilterNode subclass.

Yea, I'll implement geometrical transformations filter sometime this week.
Hopefully it's gonna be clearer by then.


> I didn't mean the input texture, but the resulting texture. I.e. the one
> that is written into when rendered onto the fbo. The problem was visible
> with recent Quasar when you had an image that was greater than the
> QGLWidget. Then only the lower left part of that image got processed, the
> rest of the texture was a dark grey.

Ah, interesting, we'll need a test for that. I haven't seen it. The resulting
texture should be of the size of the fbo.
Of the top of my head I don't remember anything in the EXT_framebuffer_object
spec that would make it impossible to have fbo's bigger than the surface
direct rendering context was constructed for. I'll look into it. If you have
a simple example, that would help as well :)

> > > Alternatively the projection could also be glOrtho(0, 1, 0, 1, -1, 1).
> > > Depending on the node this might make sense, so it would be good if the
> > > node can override the projection matrix easily.
> >
> > Any particular reason for it?
>
> Say you want to do a translation in the vertex shader. If the projection
> matrix is glOrtho(0, width, 0, height, ...) then the vertex shader doesn't
> know how far it has to move the vertex to do e.g. a 50% translation to the
> left.

Yea, I'm going to pass dimensions as a vec4 to the shader. So then the shader
would simply do
uniform vec4 dimensions;
gl_Position = gl_ModelViewProjectionMatrix * (gl_Vertex + dimensions.xy*0.5);
or such.

> The shader would need to know the image dimensions - those could be
> passed as uniforms. But if the vertices are "normalized" to the 1x1
> rectangle the uniforms become unnecessary.

But then the vertices in the mesh have to mapped within the 0-1 viewport for
all the textures.
So it's really a question of where you want to be doing the math. I also
wanted to keep a more natural coordinate mode 0-width/0-height. I never liked
the fact that in Qt we did upper-left 0,0, but I guess at somepoint we'll
need to switch to the same coordinate system Qt uses by default to integrate
better.

> Probably QGraphicsView would be the best framework to integrate with?

Yea, I think so.

z
_______________________________________________
Kde-graphics-devel mailing list
Kde-graphics-devel@...
https://mail.kde.org/mailman/listinfo/kde-graphics-devel