terminfo implementation

[Roy Marples-2]

Hi list

I've been working on a terminfo implementation and this part is now
pretty much done, although there is one encoding type that needs to be
finished and the tputs needs to work with ncurses values as well. You
can see the source here. Basic tic and infocmp utilities are included.
http://www.netbsd.org/~roy/terminfo.tar.bz2

I've not included a terminal description file as I propose we use the
ncurses one.

This is compatible with the terminfo x/open standard with respect to
tparm. tparmv is provided as well, and if you #define TPARM_VARARGS then
tparm is mapped to tparmv to provide source compatibility with default
ncurses.

tmux-0.8 needs a tiny patch to work with this, basically adding the
#define, removing the #include <ncurses.h> line and replacing OK with 0.

Does anyone have any ideas about merging this into the tree? I think we
can add it as a new library and slowly move code over to using
libterminfo instead of libtermcap, although this does mean mainting two
different databases until completed as my libterminfo does not have any
libtermcap capabilities.

Thanks

Roy

[Adam Hoka]

On Sat, 27 Jun 2009 11:02:11 +0100
Roy Marples <roy@...> wrote:

> Hi list
>
> I've been working on a terminfo implementation and this part is now
> pretty much done, although there is one encoding type that needs to be
> finished and the tputs needs to work with ncurses values as well. You
> can see the source here. Basic tic and infocmp utilities are included.
> http://www.netbsd.org/~roy/terminfo.tar.bz2
>
> I've not included a terminal description file as I propose we use the
> ncurses one.
>
> This is compatible with the terminfo x/open standard with respect to
> tparm. tparmv is provided as well, and if you #define TPARM_VARARGS then
> tparm is mapped to tparmv to provide source compatibility with default
> ncurses.
>
> tmux-0.8 needs a tiny patch to work with this, basically adding the
> #define, removing the #include <ncurses.h> line and replacing OK with 0.
>
> Does anyone have any ideas about merging this into the tree? I think we
> can add it as a new library and slowly move code over to using
> libterminfo instead of libtermcap, although this does mean mainting two
> different databases until completed as my libterminfo does not have any
> libtermcap capabilities.

Hi!

Sounds nice! Can you still ensure binary compatibility with old applications
using libtermcap after that proposed future removal?

--
Adam Hoka <Adam.Hoka@...>
Adam Hoka <ahoka@...>
Adam Hoka <ahoka@...>

[Roy Marples-2]

Adam Hoka wrote:
> Sounds nice! Can you still ensure binary compatibility with old applications
> using libtermcap after that proposed future removal?

No, we would have to keep libtermcap around in compat.

Thanks

Roy

[Thor Lancelot Simon]

On Sat, Jun 27, 2009 at 11:02:11AM +0100, Roy Marples wrote:
>
> Does anyone have any ideas about merging this into the tree? I think we  
> can add it as a new library and slowly move code over to using  
> libterminfo instead of libtermcap, although this does mean mainting two  
> different databases until completed as my libterminfo does not have any  
> libtermcap capabilities.

I think there's been significant difference of opinion about whether this
is a desirable thing to add to the tree, and that it should not be added
unless we either achieve at least rough consensus, or core decides they
want it.

--
Thor Lancelot Simon                                   tls@...
    "Even experienced UNIX users occasionally enter rm *.* at the UNIX
     prompt only to realize too late that they have removed the wrong
     segment of the directory structure." - Microsoft WSS whitepaper

[Greg A. Woods-3]

At Sat, 27 Jun 2009 11:02:11 +0100, Roy Marples <roy@...> wrote:
Subject: terminfo implementation
>
> I've been working on a terminfo implementation and this part is now
> pretty much done, although there is one encoding type that needs to be
> finished and the tputs needs to work with ncurses values as well. You
> can see the source here. Basic tic and infocmp utilities are included.
> http://www.netbsd.org/~roy/terminfo.tar.bz2

Awsome!  This is something I've been longing for -- termcap just sucks
horribly for most everything, IMO.


> I've not included a terminal description file as I propose we use the
> ncurses one.

Sounds good enough for starters.

> This is compatible with the terminfo x/open standard with respect to
> tparm. tparmv is provided as well, and if you #define TPARM_VARARGS then
> tparm is mapped to tparmv to provide source compatibility with default
> ncurses.

very nice.

--
                                                Greg A. Woods
                                                Planix, Inc.

<woods@...>       [phone]        http://www.planix.com/

[Roy Marples-2]

Here's a new snapshot.
http://www.netbsd.org/~roy/terminfo-20090709.tar.bz2

It should now implement all documented ncurses terminfo capabilities.
The tic(1) implementation is a lot faster than before and is 3 times
quicker than ncurses tic :)
Both tic(1) and infocmp(1) now sport the -x flag to control acceptance
of non standard capabilities. If you have tried a previous snapshot,
you'll need to rebuild your databases.

In addition to the standard cap_long_name macros to access a cap quickly
on cur_term, we have t_cap_long_name macros to access a cap quickly on
any TERMINAL struct which would make any merge with our curses library a
little less painless.

It also has a termcap compatibility layer and is source and binary
compatible with our current libtermcap provided that NO NetBSD
extensions have been used. Only ul(1) and curses(3) use this in the tree
according to my grep search. Whilst curses(3) does have an excuse to use
them (see prior discussions for why) ul(1) does not so I'm pretty sure
we can just drop them.

On the whole, most people are generally positive about this.
I've taken on board some comments by Joerg and changed some code
accordingly, but more eyes would be better.

Thanks

Roy

[Christos Zoulas-2]

In article <4A55BB7B.3040902@...>,
Roy Marples  <roy@...> wrote:

>Here's a new snapshot.
>http://www.netbsd.org/~roy/terminfo-20090709.tar.bz2
>
>It should now implement all documented ncurses terminfo capabilities.
>The tic(1) implementation is a lot faster than before and is 3 times
>quicker than ncurses tic :)
>Both tic(1) and infocmp(1) now sport the -x flag to control acceptance
>of non standard capabilities. If you have tried a previous snapshot,
>you'll need to rebuild your databases.
>
>In addition to the standard cap_long_name macros to access a cap quickly
>on cur_term, we have t_cap_long_name macros to access a cap quickly on
>any TERMINAL struct which would make any merge with our curses library a
>little less painless.
>
>It also has a termcap compatibility layer and is source and binary
>compatible with our current libtermcap provided that NO NetBSD
>extensions have been used. Only ul(1) and curses(3) use this in the tree
>according to my grep search. Whilst curses(3) does have an excuse to use
>them (see prior discussions for why) ul(1) does not so I'm pretty sure
>we can just drop them.
>
>On the whole, most people are generally positive about this.
>I've taken on board some comments by Joerg and changed some code
>accordingly, but more eyes would be better.
>

curterm.c:
        - there are more baud rates defined on NetBSD.
defs.c:
        - the arrays of constants should be sorted so the index functions
          can use binary search instead of of linear.
setupterm.c:
        - use errx(3) in the exit case so that the program name gets printed.
term.c:
        - what frees tp->_extras?
termcap.c:
        - sort arrays and use binary search.
tputs.c:
        - to support multiple terminals outc should be passed a second
          void * argument that can be used from the application to determine
          the terminal context. ie. the signature for ti_puts should be
          int ti_puts(const TERMINAL *term, const char *str, int affcnt,
              int (*outc)(int, void *), void *arg)

christos

[Thomas Dickey-3]

On Thu, Jul 09, 2009 at 10:42:19AM +0100, Roy Marples wrote:
> Here's a new snapshot.
> http://www.netbsd.org/~roy/terminfo-20090709.tar.bz2
>
> It should now implement all documented ncurses terminfo capabilities.
     ^might

(it's missing a lot of functionality, e.g., the ability to compare
extended entries, which I implemented in ncurses more than 10 years ago ;-)

> The tic(1) implementation is a lot faster than before and is 3 times
> quicker than ncurses tic :)

(perhaps - see above).

--
Thomas E. Dickey
http://invisible-island.net
ftp://invisible-island.net

[Roy Marples-2]

Christos Zoulas wrote:

> curterm.c:
>         - there are more baud rates defined on NetBSD.
> defs.c:
>         - the arrays of constants should be sorted so the index functions
>           can use binary search instead of of linear.
> setupterm.c:
>         - use errx(3) in the exit case so that the program name gets printed.
> term.c:
>         - what frees tp->_extras?
> termcap.c:
>         - sort arrays and use binary search.
> tputs.c:
>         - to support multiple terminals outc should be passed a second
>           void * argument that can be used from the application to determine
>           the terminal context. ie. the signature for ti_puts should be
>           int ti_puts(const TERMINAL *term, const char *str, int affcnt,
>               int (*outc)(int, void *), void *arg)

OK, new snapshot with all the above changes done.
http://www.netbsd.org/~roy/terminfo-20090710.tar.bz2

I generate hash tables for terminfo and termcap strings now.

Thanks

Roy

[Roy Marples-2]

Thomas Dickey wrote:
> On Thu, Jul 09, 2009 at 10:42:19AM +0100, Roy Marples wrote:
>> Here's a new snapshot.
>> http://www.netbsd.org/~roy/terminfo-20090709.tar.bz2
>>
>> It should now implement all documented ncurses terminfo capabilities.
>      ^might
>
> (it's missing a lot of functionality, e.g., the ability to compare
> extended entries, which I implemented in ncurses more than 10 years ago ;-)

Yes, the userland tool infocmp I provide doesn't compare entries yet, it
just prints them to the screen. It's purpose to date is to ensure that a
tic compiled entry matches a ncurses tic compiled entry.

However, the terminfo library and and compiler have all the
functionality required to be standards compliant.

Thanks

Roy

[Christos Zoulas-2]

In article <4A569299.8030601@...>,
Roy Marples  <roy@...> wrote:

>Christos Zoulas wrote:
>> curterm.c:
>>         - there are more baud rates defined on NetBSD.
>> defs.c:
>>         - the arrays of constants should be sorted so the index functions
>>           can use binary search instead of of linear.
>> setupterm.c:
>>         - use errx(3) in the exit case so that the program name gets printed.
>> term.c:
>>         - what frees tp->_extras?
>> termcap.c:
>>         - sort arrays and use binary search.
>> tputs.c:
>>         - to support multiple terminals outc should be passed a second
>>           void * argument that can be used from the application to determine
>>           the terminal context. ie. the signature for ti_puts should be
>>           int ti_puts(const TERMINAL *term, const char *str, int affcnt,
>>               int (*outc)(int, void *), void *arg)
>
>OK, new snapshot with all the above changes done.
>http://www.netbsd.org/~roy/terminfo-20090710.tar.bz2
>
>I generate hash tables for terminfo and termcap strings now.

Looks good. I would not bother duplicating the functions in tputs.c. I would
just cast putchar to (int (*)(int, void *)) and pass NULL as the argument.

christos

[Roy Marples-3]

Christos Zoulas wrote:

> In article <4A569299.8030601@...>,
> Roy Marples  <roy@...> wrote:
>> Christos Zoulas wrote:
>>> curterm.c:
>>>         - there are more baud rates defined on NetBSD.
>>> defs.c:
>>>         - the arrays of constants should be sorted so the index functions
>>>           can use binary search instead of of linear.
>>> setupterm.c:
>>>         - use errx(3) in the exit case so that the program name gets printed.
>>> term.c:
>>>         - what frees tp->_extras?
>>> termcap.c:
>>>         - sort arrays and use binary search.
>>> tputs.c:
>>>         - to support multiple terminals outc should be passed a second
>>>           void * argument that can be used from the application to determine
>>>           the terminal context. ie. the signature for ti_puts should be
>>>           int ti_puts(const TERMINAL *term, const char *str, int affcnt,
>>>               int (*outc)(int, void *), void *arg)
>> OK, new snapshot with all the above changes done.
>> http://www.netbsd.org/~roy/terminfo-20090710.tar.bz2
>>
>> I generate hash tables for terminfo and termcap strings now.
>
> Looks good. I would not bother duplicating the functions in tputs.c. I would
> just cast putchar to (int (*)(int, void *)) and pass NULL as the argument.

Done. I've also renamed tparmv to vtparm so the naming follows other
variadic functions.
I've refreshed the tarball.

I've also tested this against tmux on Sparc64 where it works equally
well and compiles without warnings, so looking good :)

Thanks

Roy

[Roy Marples-2]

Here's a new snapshot, using the jenkins_hash and nbperf Joerg posted in
another thread. Changing from mph to nbperf reduced the binary of
libterminfo size by nearly 10k :)
http://www.netbsd.org/~roy/terminfo-20090718.tar.bz2

The only thing I have left on my todo list is to stop exposing the
termcap hash directly. After this has been done, and hopefully a
consensus on jenkins_hash in libc has been reached I'll request a merge
into -current. So now's a good time for more comments or any missing
infocmp/tic that you think you really need.

Thanks

Roy

[Roy Marples-2]

Thomas Dickey wrote:
> On Thu, Jul 09, 2009 at 10:42:19AM +0100, Roy Marples wrote:
>> Here's a new snapshot.
>> http://www.netbsd.org/~roy/terminfo-20090709.tar.bz2
>>
>> It should now implement all documented ncurses terminfo capabilities.
>      ^might
>
> (it's missing a lot of functionality, e.g., the ability to compare
> extended entries, which I implemented in ncurses more than 10 years ago ;-)

My infocmp implementation now compares entries to each other in the same
way as ncurses. A few comparsion and output options have not been
implemented as I don't support termcap -> terminfo translation and I've
not entirely sure some output options are needed.

>> The tic(1) implementation is a lot faster than before and is 3 times
>> quicker than ncurses tic :)
>
> (perhaps - see above).

Still 3x quicker ;)
Maybe this is because I just write one database whereas ncurses spits
out one per terminal?

Thanks

Roy

[Thomas Dickey-3]

On Sat, Jul 18, 2009 at 11:22:56AM +0100, Roy Marples wrote:
> Still 3x quicker ;)
> Maybe this is because I just write one database whereas ncurses spits
> out one per terminal?

ncurses has the option of writing to a hashed database rather than
to a directory-tree.

(NetBSD code tends to build only in NetBSD - I've not looked to see
if your code ports to other systems where I might measure it ;-)

--
Thomas E. Dickey
http://invisible-island.net
ftp://invisible-island.net

[Roy Marples-2]

On Saturday 18 July 2009 16:06:58 Thomas Dickey wrote:
> On Sat, Jul 18, 2009 at 11:22:56AM +0100, Roy Marples wrote:
> > Still 3x quicker ;)
> > Maybe this is because I just write one database whereas ncurses spits
> > out one per terminal?
>
> ncurses has the option of writing to a hashed database rather than
> to a directory-tree.

True. I'll try turning it on and re-test.

> (NetBSD code tends to build only in NetBSD - I've not looked to see
> if your code ports to other systems where I might measure it ;-)

These are the BSD only functions/macros I think I use

__RCSID
__UNCONST
__arraycount
DEFFILEMODE
le16dec
le16enc
fgetln (I've recoded around getline(3) as NetBSD-current now has it)
strlcpy (this really should be in glibc)

The jenkins hash I use for fast string lookup uses these additional functions.
le32dec

Other BSDisms such as errx are available in glibc.

My Gentoo box appears to lack the dbm_open(3) interface even though it's a
standard (no header, not in libc - but man page exists which is odd).

So porting it is non trivial. Also I'm not sure it's that worthwhile beyond
the half hour I've spent working the above out as World + Dog seems to be
using ncurses and happy with it.

You are also welcome to install NetBSD ;)

Thanks

Roy

[Thomas Dickey-3]

On Sat, Jul 18, 2009 at 07:55:45PM +0100, Roy Marples wrote:

> On Saturday 18 July 2009 16:06:58 Thomas Dickey wrote:
> > On Sat, Jul 18, 2009 at 11:22:56AM +0100, Roy Marples wrote:
> > > Still 3x quicker ;)
> > > Maybe this is because I just write one database whereas ncurses spits
> > > out one per terminal?
> >
> > ncurses has the option of writing to a hashed database rather than
> > to a directory-tree.
>
> True. I'll try turning it on and re-test.

it might be an improvement...
 

> > (NetBSD code tends to build only in NetBSD - I've not looked to see
> > if your code ports to other systems where I might measure it ;-)
>
> These are the BSD only functions/macros I think I use
>
> __RCSID
> __UNCONST
> __arraycount
> DEFFILEMODE
> le16dec
> le16enc
> fgetln (I've recoded around getline(3) as NetBSD-current now has it)
> strlcpy (this really should be in glibc)
>
> The jenkins hash I use for fast string lookup uses these additional functions.
> le32dec
>
> Other BSDisms such as errx are available in glibc.
>
> My Gentoo box appears to lack the dbm_open(3) interface even though it's a
> standard (no header, not in libc - but man page exists which is odd).
>
> So porting it is non trivial. Also I'm not sure it's that worthwhile beyond
> the half hour I've spent working the above out as World + Dog seems to be
> using ncurses and happy with it.
>
> You are also welcome to install NetBSD ;)

yes (but there's only so much time to work)

--
Thomas E. Dickey
http://invisible-island.net
ftp://invisible-island.net

["C. Bergström"-2]

Roy Marples wrote:

> On Saturday 18 July 2009 16:06:58 Thomas Dickey wrote:
>  
>> On Sat, Jul 18, 2009 at 11:22:56AM +0100, Roy Marples wrote:
>>    
>>> Still 3x quicker ;)
>>> Maybe this is because I just write one database whereas ncurses spits
>>> out one per terminal?
>>>      
>> ncurses has the option of writing to a hashed database rather than
>> to a directory-tree.
>>    
>
> True. I'll try turning it on and re-test.
>
>  
>> (NetBSD code tends to build only in NetBSD - I've not looked to see
>> if your code ports to other systems where I might measure it ;-)
>>    
>
> These are the BSD only functions/macros I think I use
>
> __RCSID
> __UNCONST
> __arraycount
> DEFFILEMODE
> le16dec
> le16enc
> fgetln (I've recoded around getline(3) as NetBSD-current now has it)
> strlcpy (this really should be in glibc)
>
> The jenkins hash I use for fast string lookup uses these additional functions.
> le32dec
>
> Other BSDisms such as errx are available in glibc.
>  

Hi Roy,

If you have time would you be kind enough to test on OpenSolaris.  It
would benefit us if we could replace what we're currently using.  If you
need access to an OpenSolaris box just send me a private email and I can
easily set that up.

Thanks

./Christopher