using fribidi and Pango

[Tomas Frydrych]

>> Date: Mon, 21 Nov 2005 02:56:37 +0330
>> From: Hossein Noorikhah <address@hidden>
>>
>> I want to know more about using Pango for text rendering that can
>> solve these kind of problems. I read in mailing list achieves that
>> Emacs developers have tried Pango for text rendering, but they found
>> it not suitable for Emcas. I want to know why?

> I once wrote here the reason: Pango, like fribidi, is a batch-mode
> reordering library: you hand it a buffer with text and it returns it
> reordered for display.

This is not really to do with Pango or fribidi, but with the nature of
the bidirectional problem. The Unicode bidirectional algorithm requires
text of entire paragraph as an input, and you will not get satisfactory
results otherwise.

> By contrast, Emacs needs a reordering function that could be called
> repeatedly, and will on every call return the next character in the
> visual order.  This is because the Emacs display engine walks the
> buffer one character at a time and decides how to display it based on
> data structures built by the application.

You can do that easily with fribidi; the fribidi_log2vis() function
generates a map that allows you to translate logical index into a visual
one, and you can use it to walk over your buffer. (AbiWord, which too
has complex data structures associated with its text buffer, uses
fribidi without any great difficulties.).

Pango is a slightly different story; its primary api is not suited to an
application that needs to control the layout more precisely; you can use
the low level (engine specific) api with bit more work, but as said
elsewhere in the original thread, you would probably need to
significantly redesign how things are handled internally; again, this
might not be simply because of Pango, but because of the nature of the
problem that Pango addresses.

It really boils down to how much i18n matters, but you will not get
decent support for 'exotic' languages unless you do use something like
Pango.

Tomas


               
___________________________________________________________
Yahoo! Model Search 2005 - Find the next catwalk superstars - http://uk.news.yahoo.com/hot/model-search/



_______________________________________________
emacs-bidi mailing list
emacs-bidi@...
http://lists.gnu.org/mailman/listinfo/emacs-bidi

[Eli Zaretskii]

> Date: Thu, 01 Dec 2005 20:00:50 +0000
> From: Tomas Frydrych <tomasfrydrych@...>
>
> > I once wrote here the reason: Pango, like fribidi, is a batch-mode
> > reordering library: you hand it a buffer with text and it returns it
> > reordered for display.
>
> This is not really to do with Pango or fribidi, but with the nature of
> the bidirectional problem. The Unicode bidirectional algorithm requires
> text of entire paragraph as an input, and you will not get satisfactory
> results otherwise.

Well, you may wish to look at the code I wrote for Emacs: modulo any
remaining bugs, it gives satisfactory results and implements the full
UAX#9 algorithm.  It is true that the Unicode algorithm's description
is heavily biased in favor of batch reordering, but that doesn't mean
it couldn't be serialized.  The fact is that I succeeded doing that.

> > By contrast, Emacs needs a reordering function that could be called
> > repeatedly, and will on every call return the next character in the
> > visual order.  This is because the Emacs display engine walks the
> > buffer one character at a time and decides how to display it based on
> > data structures built by the application.
>
> You can do that easily with fribidi; the fribidi_log2vis() function
> generates a map that allows you to translate logical index into a visual
> one, and you can use it to walk over your buffer.

The initial code to support bidi, written by Handa-san and others at
m17n.org, did something like that.  However, Gerd Moellmann, the
person who wrote the current Emacs display engine, objected to using
such code, since it in effect caches (e.g., in the map you mentioned)
portions of the buffer, and that caching adds a significant overhead
to display engine code.  Emacs has so many display-related features
that walking all the characters in the portion of the buffer that is
shown on the screen makes redisplay unbearably slow.  That is why the
display code has many special cases that optimize it in most practical
situations (the simplest ones being cursor movement and insert/delete
a single character).  Any code that needs to traverse large portions
of the buffer to create a cache of some kind will slow down these
simple cases too much.

> (AbiWord, which too
> has complex data structures associated with its text buffer, uses
> fribidi without any great difficulties.).

I doubt that any other editor out there comes close to Emacs in the
amount of display-related features.  Plus, Emacs needs to run over
remote logins with slow speed and still be reasonably fast.

> It really boils down to how much i18n matters, but you will not get
> decent support for 'exotic' languages unless you do use something like
> Pango.

I don't know what is your value of ``exotic'', but Emacs supports many
languages for several years now, since Emacs 20.1.


_______________________________________________
emacs-bidi mailing list
emacs-bidi@...
http://lists.gnu.org/mailman/listinfo/emacs-bidi

[Tomas Frydrych]

> Well, you may wish to look at the code I wrote for Emacs:

I do indeed; could you please point me to the relevant file(s) in the
repository?

> modulo any remaining bugs, it gives satisfactory results and
> implements the full UAX#9 algorithm.  It is true that the Unicode
> algorithm's description is heavily biased in favor of batch
> reordering, but that doesn't mean it couldn't be serialized.  The
> fact is that I succeeded doing that.

If that is the case, it would be shame if your work was simply confined
to emacs -- there are many other apps that could benefit from that, and
I think the fribidi maintainer would be interested in incorporating that
into fribidi.

> Emacs has so many display-related features that walking all the
> characters in the portion of the buffer that is shown on the screen
> makes redisplay unbearably slow.  That is why the display code has
> many special cases that optimize it in most practical situations (the
> simplest ones being cursor movement and insert/delete a single
> character).  Any code that needs to traverse large portions of the
> buffer to create a cache of some kind will slow down these simple
> cases too much.

There is undoubtedly overhead for bidi, but you do not have to walk all
the characters all the time, and the cache does not have to be there for
the entire buffer (I have got very satisfactory results with caching
single line at a time), and more importantly, the cache does not have to
be fully refreshed on every operation on the buffer, as many (perhaps
most) of the normal use cases can be shortcircuted.

>> It really boils down to how much i18n matters, but you will not get
>>  decent support for 'exotic' languages unless you do use something
>> like Pango.
>
> I don't know what is your value of ``exotic'', but Emacs supports
> many languages for several years now, since Emacs 20.1.

In which ever way you will achieve that, you will have to do something
like Pango does. It is relatively easy to do until you hit languages
that require shaping; even then it might seem easier to reinvent the
wheel until you come to languages for which contextual glyphs do not
have separated Unicode code points, then it gets bit more interesting
(and you will probably wish you did not set down that road in the first
place).

Whatever you do, if you do not use external script processor you will be
creating one internally and in the process caching significant amounts
of Unicode character properties somewhere in your code. Pango will not
provide out of the box answer for you, I am fully aware of that, but
there might be quite a lot of value to the community in helping to
improve Pango, rather than reinventing the whole thing for emacs alone.

Tomas



       
       
               
___________________________________________________________
Yahoo! Messenger - NEW crystal clear PC to PC calling worldwide with voicemail http://uk.messenger.yahoo.com



_______________________________________________
emacs-bidi mailing list
emacs-bidi@...
http://lists.gnu.org/mailman/listinfo/emacs-bidi

[Eli Zaretskii]

> Date: Fri, 02 Dec 2005 11:24:23 +0000
> From: Tomas Frydrych <tomasfrydrych@...>
>
> > Well, you may wish to look at the code I wrote for Emacs:
>
> I do indeed; could you please point me to the relevant file(s) in the
> repository?

In the emacs-bidi branch of the Emacs CVS, look for src/bidi.c.

> I think the fribidi maintainer would be interested in incorporating that
> into fribidi.

The fribidi maintainer, or at least one of the main contributors to
fribidi, is well aware of this code (he should be reading this list as
we speak).

> There is undoubtedly overhead for bidi, but you do not have to walk all
> the characters all the time, and the cache does not have to be there for
> the entire buffer (I have got very satisfactory results with caching
> single line at a time)

Even a single line could be too much to cache, as lines can be very
long in Emacs.

> and more importantly, the cache does not have to
> be fully refreshed on every operation on the buffer, as many (perhaps
> most) of the normal use cases can be shortcircuted.

Maybe so, but the code I wrote is free from any need to cache
characters.  This makes its interface with the rest of display code
simpler, since any smart caching will need to know a lot about
redisplay optimizations in Emacs (and there's an awful lot of them) to
support them.

> >> It really boils down to how much i18n matters, but you will not get
> >>  decent support for 'exotic' languages unless you do use something
> >> like Pango.
> >
> > I don't know what is your value of ``exotic'', but Emacs supports
> > many languages for several years now, since Emacs 20.1.
>
> In which ever way you will achieve that, you will have to do something
> like Pango does.

Sure, but Emacs has so many unique requirements that it's improbable
that Pango (or any other similar package) supports them out of the
box.

> It is relatively easy to do until you hit languages that require
> shaping

I think someone already wrote the code for Arabic to cover this.  It's
not yet in mainstream Emacs, though.

> Whatever you do, if you do not use external script processor you will be
> creating one internally and in the process caching significant amounts
> of Unicode character properties somewhere in your code.

Yes.  But I don't see any reasonable way to avoid that.

> Pango will not provide out of the box answer for you, I am fully
> aware of that, but there might be quite a lot of value to the
> community in helping to improve Pango, rather than reinventing the
> whole thing for emacs alone.

For this, some of the Pango maintainers need to volunteer to learn the
Emacs unique requirements and discuss the various design issues that
will undoubtfully appear once we begin looking into some kind of
mutual support.  Until now, IIRC, no Pango developers expressed any
interest in the Emacs display code, at least not on the Emacs
developers' mailing list, although Emacs supports editing multiple
languages in the same buffer since September 1997, when version 20.1
was released.


_______________________________________________
emacs-bidi mailing list
emacs-bidi@...
http://lists.gnu.org/mailman/listinfo/emacs-bidi

[Behdad Esfahbod]

On Fri, 2 Dec 2005, Eli Zaretskii wrote:

> > Date: Fri, 02 Dec 2005 11:24:23 +0000
> > From: Tomas Frydrych <tomasfrydrych@...>
> >
> > > Well, you may wish to look at the code I wrote for Emacs:
> >
> > I do indeed; could you please point me to the relevant file(s) in the
> > repository?
>
> In the emacs-bidi branch of the Emacs CVS, look for src/bidi.c.
>
> > I think the fribidi maintainer would be interested in incorporating that
> > into fribidi.
>
> The fribidi maintainer, or at least one of the main contributors to
> fribidi, is well aware of this code (he should be reading this list as
> we speak).

Yes, I'm reading here.  No, I didn't know about src/bidi.c.
Until a few seconds ago, I was under the impression that your
work was all implemented in Emacs Lisp.  I will look into it, but
generally, if you think your code suites project X, doens't hurt
if you propose its inclusion, instead of waiting for project X
maintainer to say "hey, your code looks good, would you mind I
include it in X?" which kinda sounds like they are trying to get
your cookies...


> For this, some of the Pango maintainers need to volunteer to learn the
> Emacs unique requirements and discuss the various design issues that
> will undoubtfully appear once we begin looking into some kind of
> mutual support.  Until now, IIRC, no Pango developers expressed any
> interest in the Emacs display code, at least not on the Emacs
> developers' mailing list, although Emacs supports editing multiple
> languages in the same buffer since September 1997, when version 20.1
> was released.

Same here.  Moreover, Pango developers are limited to one or two.
The priorities of Pango are rather different, but Pango has been
accepting patches all the time.  Or to being with, you can open a
bug in gnome bugzilla, stating that to use Pango in Emacs, we
need it to support this and this and this and this...

Regards,

--behdad
http://behdad.org/

"Commandment Three says Do Not Kill, Amendment Two says Blood Will Spill"
        -- Dan Bern, "New American Language"


_______________________________________________
emacs-bidi mailing list
emacs-bidi@...
http://lists.gnu.org/mailman/listinfo/emacs-bidi

[Tomas Frydrych]

>> I do indeed; could you please point me to the relevant file(s) in
>> the repository?
> In the emacs-bidi branch of the Emacs CVS, look for src/bidi.c.

That is very nice piece of code; for what it is worth, I think this
would be worth incorporating into fribidi.

> Sure, but Emacs has so many unique requirements that it's improbable
> that Pango (or any other similar package) supports them out of the
> box.

The Pango high level code will not do that (there is only that much that
can be generalised into a library), but in this regard Emacs is not
unique; there are other applications that have complex demands which can
only be satisfied by their own layout engine. However, scripting engine
can provide a low-level api that makes it possible to plug it into that
engine. Pango has a low level api that is quite usable and if it does
not entirely suit your needs as is, Behdad is a very reasonable guy :).
I am going to shut up now.

Tomas

               
___________________________________________________________
To help you stay safe and secure online, we've developed the all new Yahoo! Security Centre. http://uk.security.yahoo.com



_______________________________________________
emacs-bidi mailing list
emacs-bidi@...
http://lists.gnu.org/mailman/listinfo/emacs-bidi

[Eli Zaretskii]

> Date: Sun, 04 Dec 2005 16:06:49 +0000
> From: Tomas Frydrych <tomasfrydrych@...>
>
> > In the emacs-bidi branch of the Emacs CVS, look for src/bidi.c.
>
> That is very nice piece of code;

Thanks.

> for what it is worth, I think this would be worth incorporating into
> fribidi.

Feel free; it is Free Software.

However, as found in the Emacs CVS, the code needs some tweaking to
become a part of another package: some necessary data structure
definitions are in the header dispextern.h (they will probably need to
be put on a separate header for other packages), and some of the
constants and macros in bidi.c itself are specific to Emacs and its
character representation in buffers and strings.


_______________________________________________
emacs-bidi mailing list
emacs-bidi@...
http://lists.gnu.org/mailman/listinfo/emacs-bidi

[Behdad Esfahbod]

On Sun, 4 Dec 2005, Tomas Frydrych wrote:

> >> I do indeed; could you please point me to the relevant file(s) in
> >> the repository?
> > In the emacs-bidi branch of the Emacs CVS, look for src/bidi.c.
>
> That is very nice piece of code; for what it is worth, I think this
> would be worth incorporating into fribidi.

Agreed.  I looked into the code too.  Looks very complex, but
then it is expected to be.  Fortunately a lot of data tables and
other stuff can be replaced by FriBidi's structures, chopping
some 500 lines of it I guess.  The rest needs some restructuring.

But then the question is, is the API usable in more than Emacs
only?  I'm personally interested in having a bidi  API that does
not process paragraph at a time, but you give it the range you
are interested in (the range that fits in the screen), and it
will only lookahead as much as needed only.

My recent work in Pango made me believe that writing general
purpose libraries for algorithms is just not easy to get right.
You cannot have an stable general API and high performance
together.  Algorithms to consume low memory and perform fast,
need to integrate into eachother at levels not quite possible by
library APIs.  Even an array containing the input text is being
problematic.  There's always someone asking for UTF-8, one for
UTF-16, another for UTF-32.  Then there are libraries that keep
some kind of struct Char [], that contains more than the text
itself.  You just cannot write a fast low-overhead library to
handle them all.  And you cannot use glib, since too many
projects don't use it, so you need to define all the UTF-8 and
UTF-16 functions yourself, should you go that route.  Not talking
about how hard just finding a 32-bit integer can be...
</rant>

> > Sure, but Emacs has so many unique requirements that it's improbable
> > that Pango (or any other similar package) supports them out of the
> > box.
>
> The Pango high level code will not do that (there is only that much that
> can be generalised into a library), but in this regard Emacs is not
> unique; there are other applications that have complex demands which can
> only be satisfied by their own layout engine. However, scripting engine
> can provide a low-level api that makes it possible to plug it into that
> engine. Pango has a low level api that is quite usable and if it does
> not entirely suit your needs as is, Behdad is a very reasonable guy :).
> I am going to shut up now.

Heh,
I'm more interested to see the highlevel PangoLayout object
extended to cover needs of applications like Emacs and AbiWord.
And I'm willing to hear more about those needs :).

> Tomas

--behdad
http://behdad.org/

"Commandment Three says Do Not Kill, Amendment Two says Blood Will Spill"
        -- Dan Bern, "New American Language"


_______________________________________________
emacs-bidi mailing list
emacs-bidi@...
http://lists.gnu.org/mailman/listinfo/emacs-bidi

[Tomas Frydrych]

Behdad Esfahbod wrote:
> My recent work in Pango made me believe that writing general
> purpose libraries for algorithms is just not easy to get right.
> You cannot have an stable general API and high performance
> together.  Algorithms to consume low memory and perform fast,
> need to integrate into eachother at levels not quite possible by
> library APIs.

Yes, this is why I have been very resistant to use Pango in the win32
version of AbiWord -- perhaps Pango should limit itself to being a *nix
library, but this is probably not of great interest to this list.

> Even an array containing the input text is being
> problematic.  There's always someone asking for UTF-8, one for
> UTF-16, another for UTF-32.  Then there are libraries that keep
> some kind of struct Char [], that contains more than the text
> itself.  You just cannot write a fast low-overhead library to
> handle them all.  And you cannot use glib, since too many
> projects don't use it, so you need to define all the UTF-8 and
> UTF-16 functions yourself, should you go that route.  Not talking
> about how hard just finding a 32-bit integer can be...

Yep, the utf8 api of Pango is such a pain when you work with ucs4 ;)

> I'm more interested to see the highlevel PangoLayout object
> extended to cover needs of applications like Emacs and AbiWord.
> And I'm willing to hear more about those needs :).

My gut instinct is that I am not sure whether that is practicable but I
would be very much interested in pursuing this further -- not sure what
would be the best forum where to throw ideas around.

Tomas

               
___________________________________________________________
Yahoo! Model Search 2005 - Find the next catwalk superstars - http://uk.news.yahoo.com/hot/model-search/



_______________________________________________
emacs-bidi mailing list
emacs-bidi@...
http://lists.gnu.org/mailman/listinfo/emacs-bidi

[Eli Zaretskii]

> Date: Sun, 4 Dec 2005 16:24:28 -0500 (EST)
> From: Behdad Esfahbod <behdad@...>
> Cc: emacs-bidi@...
>
> Fortunately a lot of data tables and other stuff can be replaced by
> FriBidi's structures, chopping some 500 lines of it I guess.

If you mean the tables of character properties, then these will
eventually go away in Emacs as well; see the FIXMEs there.

> The rest needs some restructuring.

What kind of restructuring, and why is it needed?

> But then the question is, is the API usable in more than Emacs
> only?

Potentially, any display engine that needs to examine text one
character at a time.

> Even an array containing the input text is being problematic.
> There's always someone asking for UTF-8, one for UTF-16, another for
> UTF-32.

You will see that my code calls a bunch of macros to fetch characters.
Any general-purpose library needs to give these or similar
macros/functions to the caller as part of the API, because each caller
will have its own way of walking the characters.  Those functions will
also take care of converting from whatever representation they use for
characters into Unicode codepoints.

> I'm more interested to see the highlevel PangoLayout object
> extended to cover needs of applications like Emacs and AbiWord.

I doubt that heavyweight display monsters like Emacs will ever
delegate the layout job to libraries.

> And I'm willing to hear more about those needs :).

What do you want to know?

In general, reading xdisp.c and dispnew.c in Emacs sources will give
you some ideas about what's involved.


_______________________________________________
emacs-bidi mailing list
emacs-bidi@...
http://lists.gnu.org/mailman/listinfo/emacs-bidi

[Omer Zak]

On Sun, 2005-12-04 at 22:06 +0000, Tomas Frydrych wrote:

>
> Behdad Esfahbod wrote:
> > My recent work in Pango made me believe that writing general
> > purpose libraries for algorithms is just not easy to get right.
> > You cannot have an stable general API and high performance
> > together.  Algorithms to consume low memory and perform fast,
> > need to integrate into eachother at levels not quite possible by
> > library APIs.
>
> Yes, this is why I have been very resistant to use Pango in the win32
> version of AbiWord -- perhaps Pango should limit itself to being a *nix
> library, but this is probably not of great interest to this list.
>
> > Even an array containing the input text is being
> > problematic.  There's always someone asking for UTF-8, one for
> > UTF-16, another for UTF-32.  Then there are libraries that keep
> > some kind of struct Char [], that contains more than the text
> > itself.  You just cannot write a fast low-overhead library to
> > handle them all.  And you cannot use glib, since too many
> > projects don't use it, so you need to define all the UTF-8 and
> > UTF-16 functions yourself, should you go that route.  Not talking
> > about how hard just finding a 32-bit integer can be...
>
> Yep, the utf8 api of Pango is such a pain when you work with ucs4 ;)

It may be a good idea to review how C++ implements templates and works
together with the linker (ld in *nix) not to link in functions, which
have been instantiated but are not actually used.

Then have the standard Pango codebase build several variants of the
libraries, each library with different API details.
Details which could be different:
- Data type name of 32-bit integer.
- UTF-8 vs. UCS-32 etc.
- Different ways of integration of the BiDi algorithm with the client
application.

Once this is implemented, we open the way to possibilities of factoring
in the future, so that the final number of variants can be reduced and
each variant of the library be made capable of handling different APIs
in efficient manner.
                                               --- Omer
--
Sent from a PC running a top secret test version of Windows 97.
My own blog is at http://www.livejournal.com/users/tddpirate/

My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS:  at http://www.zak.co.il/spamwarning.html



_______________________________________________
emacs-bidi mailing list
emacs-bidi@...
http://lists.gnu.org/mailman/listinfo/emacs-bidi

[Behdad Esfahbod]

On Sun, 4 Dec 2005, Eli Zaretskii wrote:

> > Fortunately a lot of data tables and other stuff can be replaced by
> > FriBidi's structures, chopping some 500 lines of it I guess.
>
> If you mean the tables of character properties, then these will
> eventually go away in Emacs as well; see the FIXMEs there.

I was talking from the adopting your code in GNU FriBidi point of
view.

> > The rest needs some restructuring.
>
> What kind of restructuring, and why is it needed?

Well, to be honest, the code is far more complex than what I've
ever seen in any Free Software project.  I think I know how
complex what you are trying to accomplish is, but I don't think
the extra complexity of the incremental algorithm is worth the
effort.  As one can guess, debugging it is a nightmare, and I
think there's a real-world proof of that.

> > But then the question is, is the API usable in more than Emacs
> > only?
>
> Potentially, any display engine that needs to examine text one
> character at a time.
>
> > Even an array containing the input text is being problematic.
> > There's always someone asking for UTF-8, one for UTF-16, another for
> > UTF-32.
>
> You will see that my code calls a bunch of macros to fetch characters.
> Any general-purpose library needs to give these or similar
> macros/functions to the caller as part of the API, because each caller
> will have its own way of walking the characters.  Those functions will
> also take care of converting from whatever representation they use for
> characters into Unicode codepoints.

Yeah, having a callback is a good-enough solution. Having
different functions is the other good (and more widespread)
solution.  I don't really like any of them :(.  But if I am to
choose one, I go multiple-functions way.  I easily get fooled by
the overhead of the callback.


> > I'm more interested to see the highlevel PangoLayout object
> > extended to cover needs of applications like Emacs and AbiWord.
>
> I doubt that heavyweight display monsters like Emacs will ever
> delegate the layout job to libraries.

Ok, I'm not an Emacs user/developer at all, but I really don't
understand why Emacs should be called a heavyweight display
monster.  Most of what Emacs does happens on the logical stream
of text AFAIU.  Theoretically, getting Emacs to use Pango should
be easier than, say, AbiWord.  Practically is the other way
around.

> > And I'm willing to hear more about those needs :).
>
> What do you want to know?
>
> In general, reading xdisp.c and dispnew.c in Emacs sources will give
> you some ideas about what's involved.

Reading.

Anyway, the only reason I spoke up was to tell you that your
contributions to whatever project I'm involved in is always
welcome and appreciated.


Regards,

--behdad
http://behdad.org/

"Commandment Three says Do Not Kill, Amendment Two says Blood Will Spill"
        -- Dan Bern, "New American Language"


_______________________________________________
emacs-bidi mailing list
emacs-bidi@...
http://lists.gnu.org/mailman/listinfo/emacs-bidi

[Tomas Frydrych]

Behdad Esfahbod wrote:
> On Sun, 4 Dec 2005, Eli Zaretskii wrote:
> Yeah, having a callback is a good-enough solution. Having
> different functions is the other good (and more widespread)
> solution.  I don't really like any of them :(.  But if I am to
> choose one, I go multiple-functions way.  I easily get fooled by
> the overhead of the callback.

I think callbacks is the better solution. In AbiWord (I guess in other
applications) the text lives in a piecetable, and the only way to
provide you with a continuous string of textual data is for me to walk
over the piecetable and make a physical copy of the text into some
temporary buffer. Providing a callback means that you can directly store
the data in whatever format is most efficient for Pango, rather than
having to go through a multiple process of converting things into some
other (temporary) things. Moreover, with the multiple functions you will
never be able to examine text past the end of the temporary buffer I
give you, while with callback you can examine as much as is there,
should you need it.

Tomas

               
___________________________________________________________
To help you stay safe and secure online, we've developed the all new Yahoo! Security Centre. http://uk.security.yahoo.com



_______________________________________________
emacs-bidi mailing list
emacs-bidi@...
http://lists.gnu.org/mailman/listinfo/emacs-bidi

[Eli Zaretskii]

> Date: Mon, 5 Dec 2005 01:10:51 -0500 (EST)
> From: Behdad Esfahbod <behdad@...>
> Cc: Emacs Bidirectional Mailing List <emacs-bidi@...>
>
> On Sun, 4 Dec 2005, Eli Zaretskii wrote:
>
> > > Fortunately a lot of data tables and other stuff can be replaced by
> > > FriBidi's structures, chopping some 500 lines of it I guess.
> >
> > If you mean the tables of character properties, then these will
> > eventually go away in Emacs as well; see the FIXMEs there.
>
> I was talking from the adopting your code in GNU FriBidi point of
> view.

Yes, me too.  What I meant to say was that, eventually, I will have to
refactor the code so that it uses external tables instead of private
ones.  When that happens, the interface to those tables will present
an API that other packages, like FriBidi, could use to reference their
own tables or data structures.

> > > The rest needs some restructuring.
> >
> > What kind of restructuring, and why is it needed?
>
> Well, to be honest, the code is far more complex than what I've
> ever seen in any Free Software project.

???  You may wish to have another look at regex.c, for example, it's
no less complex, IMHO.

Not that I consider myself the best software designer out there, but
the structure of the code follows the description in UAX#9 so closely
that I didn't think anyone who has the document handy could have any
difficulty understanding it.  Roughly speaking, there's a function for
each one of the 5 steps of the Unicode algorithm listed in paragraph
3.3 of UAX#9, and then there's a couple of higher-level functions that
know how to find the next visual-order character given the resolved
levels of the current and the next character.  That's about all there
is to it; the rest of the complexity can be mapped directly to what
the Unicode algorithm mandates, and the comments in the code make this
mapping very simple.

> I think I know how complex what you are trying to accomplish is, but
> I don't think the extra complexity of the incremental algorithm is
> worth the effort.

I'm not sure I understand: in what sense is it not ``worth the
effort''?  The decision to code an incremental implementation was made
by discussing possible designs with Gerd Moellmann, the designer and
main implementor of the Emacs 21 redisplay code.  Gerd felt that any
batch-type implementation, and the necessary caching that it brings
with it, would slow down Emacs display for bidi scripts to an
unbearable level.  So writing such an incremental implementation was a
necessity for Emacs, thus clearly ``worth the effort''.

> As one can guess, debugging it is a nightmare

Actually, I discovered that it isn't very hard to debug the code.  The
function that dumps the resolved levels makes it easy to spot
problems, and the calls to `abort' where unexpected things happen
reveals problems very early.

> Ok, I'm not an Emacs user/developer at all, but I really don't
> understand why Emacs should be called a heavyweight display
> monster.

Because each buffer position can have very complex data structured
associated with them that affect various aspects of display in
arbitrary complex ways.  Just to scratch the surface: Text properties
and overlays can specify that, instead of displaying the current
character in the default font and color, Emacs should display some
other text, or use some other font/color, or display an image, or
evaluate arbitrary Lisp expression and use the result for display, or
make the text invisible, etc.  Each character can have entries in
various tables which modify the glyph (or a string of glyphs) that
need to be produced to display that character.  Every possible type of
glyph consults many user options and other Lisp-level structures and
functions to determine how the glyphs need to be produced and laid
out.

In other words, the layout engine has so many hooks into the
surrounding Emacs features that I doubt any external layout
widget/library could do the same job, and do it efficiently.

> Anyway, the only reason I spoke up was to tell you that your
> contributions to whatever project I'm involved in is always
> welcome and appreciated.

Thanks, I had no doubt in that.


_______________________________________________
emacs-bidi mailing list
emacs-bidi@...
http://lists.gnu.org/mailman/listinfo/emacs-bidi

["Martin J. Dürst"]

At 05:05 05/12/05, Eli Zaretskii wrote:
 >> Date: Sun, 04 Dec 2005 16:06:49 +0000
 >> From: Tomas Frydrych <tomasfrydrych@...>
 >>
 >> > In the emacs-bidi branch of the Emacs CVS, look for src/bidi.c.

Or for direct access, just use
http://savannah.gnu.org/cgi-bin/viewcvs/emacs/emacs/src/Attic/bidi.c
(I didn't know you can commit updates to the Attic (which is
the place where deleted files go in CVS), but it seems possible :-).

Regards,   Martin.



_______________________________________________
emacs-bidi mailing list
emacs-bidi@...
http://lists.gnu.org/mailman/listinfo/emacs-bidi

[Eli Zaretskii]

> Date: Tue, 06 Dec 2005 15:32:06 +0900
> From: Martin Duerst <duerst@...>
> Cc: emacs-bidi@...
>
> (I didn't know you can commit updates to the Attic (which is
> the place where deleted files go in CVS), but it seems possible :-).

I noticed that it was in the Attic, but I have no clue why: that file
is certainly _not_ deleted, as "cvs co" checks it out.


_______________________________________________
emacs-bidi mailing list
emacs-bidi@...
http://lists.gnu.org/mailman/listinfo/emacs-bidi

[Behdad Esfahbod]

On Tue, 6 Dec 2005, Eli Zaretskii wrote:

> > (I didn't know you can commit updates to the Attic (which is
> > the place where deleted files go in CVS), but it seems possible :-).
>
> I noticed that it was in the Attic, but I have no clue why: that file
> is certainly _not_ deleted, as "cvs co" checks it out.

Seems like it was first committed to HEAD, and then deleted from
HEAD, so there is one in the Attic that corresponds to branches
that don't have it.  Something like that.


--behdad
http://behdad.org/

"Commandment Three says Do Not Kill, Amendment Two says Blood Will Spill"
        -- Dan Bern, "New American Language"


_______________________________________________
emacs-bidi mailing list
emacs-bidi@...
http://lists.gnu.org/mailman/listinfo/emacs-bidi