crash when parsing boost c++ headers

[Alex Ott-2]

Hello all

I found, that if run gtags against the directory, containing Boost C++
library headers, then gtags will crash after it creates 2Gb GTAGS files.
It seems, that in some place, it enters into endless loop (may be expanding
one of the macroses), and generate lot of unneeded information

--
With best wishes, Alex Ott, MBA
http://alexott.blogspot.com/        http://xtalk.msk.su/~ott/
http://alexott-ru.blogspot.com/


_______________________________________________
Bug-global mailing list
Bug-global@...
http://lists.gnu.org/mailman/listinfo/bug-global

[Shigio YAMAGUCHI-2]

Hi,
Thank you for your bug report.

> I found, that if run gtags against the directory, containing Boost C++
> library headers, then gtags will crash after it creates 2Gb GTAGS files.
> It seems, that in some place, it enters into endless loop (may be expanding
> one of the macroses), and generate lot of unneeded information

What do you mean with the word 'crash'?
If you tell me the situation by example, it will be a help for me to fix the problem.

I couldn't reproduce any problem with the following environment:

OS: FreeBSD 6.1-RELEASE
global: global-5.7.4.tar.gz
Boost C++ library: boost_1_37_0.tar.gz

        % pwd
        % /usr/local/src/boost_1_37_0
        % gtags -v
        ...
         [11809/11810] extracting tags of tools/wave/cpp.hpp
         [11810/11810] extracting tags of tools/common/time_string.hpp
        [Thu Jan 22 00:30:31 JST 2009] Done.
        % ls -l G*
        -rw-r--r--  1 shigio  wheel     4112384 Jan 22 00:28 GPATH
        -rw-r--r--  1 shigio  wheel    16572416 Jan 22 00:29 GRTAGS
        -rw-r--r--  1 shigio  wheel    11010048 Jan 22 00:30 GSYMS
        -rw-r--r--  1 shigio  wheel  1337466880 Jan 22 00:28 GTAGS
        % _

Could you please tell me your environment?
Thank you in advance.
--
Shigio YAMAGUCHI <shigio@...>
PGP fingerprint: D1CB 0B89 B346 4AB6 5663  C4B6 3CA5 BBB3 57BE DDA3


_______________________________________________
Bug-global mailing list
Bug-global@...
http://lists.gnu.org/mailman/listinfo/bug-global

[Alex Ott-2]

Hello

I able to reproduce this on our project, that includes boost includes.
When i generate gtags for this project, GTAGS  files is increased to 2Gb,
and then gtags stops working.

According to sloccount, our project (together with boost) has ~1.4 million
lines of code - may be this is a reason, that gtags isn't designed to work
with such big projects? Why GTAGS files are so big? Is any tips, how i can
compact it?

>>>>> "SY" == Shigio YAMAGUCHI writes:
 SY> Hi, Thank you for your bug report.

 >> I found, that if run gtags against the directory, containing Boost C++
 >> library headers, then gtags will crash after it creates 2Gb GTAGS
 >> files.  It seems, that in some place, it enters into endless loop (may
 >> be expanding one of the macroses), and generate lot of unneeded
 >> information

 SY> What do you mean with the word 'crash'?  If you tell me the situation
 SY> by example, it will be a help for me to fix the problem.

 SY> I couldn't reproduce any problem with the following environment:


--
With best wishes, Alex Ott, MBA
http://alexott.blogspot.com/           http://xtalk.msk.su/~ott/
http://alexott-ru.blogspot.com/


_______________________________________________
Bug-global mailing list
Bug-global@...
http://lists.gnu.org/mailman/listinfo/bug-global

[Shigio YAMAGUCHI-2]

> I able to reproduce this on our project, that includes boost includes.
> When i generate gtags for this project, GTAGS  files is increased to 2Gb,
> and then gtags stops working.

What do you mean with the word 'stop'?
Could you explain it as you see?

If possible, could you please show me the output of top(1) or ps(1)
when gtags 'stopped'?

        % ps u | grep gtags (In BSD system)
--
Shigio YAMAGUCHI <shigio@...>
PGP fingerprint: D1CB 0B89 B346 4AB6 5663  C4B6 3CA5 BBB3 57BE DDA3


_______________________________________________
Bug-global mailing list
Bug-global@...
http://lists.gnu.org/mailman/listinfo/bug-global

[Alex Ott-2]

>>>>> "SY" == Shigio YAMAGUCHI writes:
 >> I able to reproduce this on our project, that includes boost includes.
 >> When i generate gtags for this project, GTAGS files is increased to
 >> 2Gb, and then gtags stops working.

 SY> What do you mean with the word 'stop'?  Could you explain it as you
 SY> see?

It report, that it couldn't write to file (i think, that it doesn't support
files with size more than 2Gb)

here is log of strace (last lines)

read(5, "ass P13 , class P14 , class P15 ,"..., 4096) = 4096
read(5, " item113; typedef mpl::int_<1> it"..., 4096) = 4096
lseek(3, 2147467264, SEEK_SET)          = 2147467264
write(3, "\376\377\3\0\0\0\0\0\0\0\0\0\4\0\0\0\0\0\0\00010985 @n 314 "..., 8192) = 8192
lseek(3, 1723179008, SEEK_SET)          = 1723179008
write(3, "\2555\3\0u\230\2\0\6\232\2\0\2\0\0\0x\2P\3\350\37\320\37\270\37\240\37\210\37p\37X"..., 8192) = 8192
lseek(3, 1899298816, SEEK_SET)          = 1899298816
read(3, "\250\211\3\0S\210\3\0B\376\3\0\2\0\0\0.\0\10\20(\21\20\21\370\20\340\20\310\20\260\20\10"..., 8192) = 8192
read(5, "               315 ./ExtLibs/buil"..., 4096) = 4096
read(5, "int_<1> item105; typedef mpl::int"..., 4096) = 4096
read(5, " typedef mpl::int_<1> item242; ty"..., 4096) = 4096
lseek(3, 2147475456, SEEK_SET)          = 2147475456
write(3, "\377\377\3\0\0\0\0\0\0\0\0\0\4\0\0\0\0\0\0\00010985 @n 314 "..., 8192) = 8191
write(2, "gtags: "..., 7gtags: )               = 7
write(2, "cannot write to database."..., 25cannot write to database.) = 25
write(2, "\n"..., 1
)                    = 1


gtags is 5.7.1 from ubuntu repository

 SY> If possible, could you please show me the output of top(1) or ps(1)
 SY> when gtags 'stopped'?

 SY> % ps u | grep gtags (In BSD system)

--
With best wishes, Alex Ott, MBA
http://alexott.blogspot.com/           http://xtalk.msk.su/~ott/
http://alexott-ru.blogspot.com/


_______________________________________________
Bug-global mailing list
Bug-global@...
http://lists.gnu.org/mailman/listinfo/bug-global

[Shigio YAMAGUCHI-2]

> It report, that it couldn't write to file (i think, that it doesn't support
> files with size more than 2Gb)

Could you please explain what you saw?
Did not gtags display any message?

Please refer to this page:
http://www.gnu.org/software/global/maillist.html
--
Shigio YAMAGUCHI <shigio@...>
PGP fingerprint: D1CB 0B89 B346 4AB6 5663  C4B6 3CA5 BBB3 57BE DDA3


_______________________________________________
Bug-global mailing list
Bug-global@...
http://lists.gnu.org/mailman/listinfo/bug-global

[Alex Ott-2]

>>>>> "SY" == Shigio YAMAGUCHI writes:
 >> It report, that it couldn't write to file (i think, that it doesn't
 >> support files with size more than 2Gb)

 SY> Could you please explain what you saw?  Did not gtags display any
 SY> message?

gtags just write "Cann't write to file" and exits

 SY> Please refer to this page:
 SY> http://www.gnu.org/software/global/maillist.html


--
With best wishes, Alex Ott, MBA
http://alexott.blogspot.com/           http://xtalk.msk.su/~ott/
http://alexott-ru.blogspot.com/


_______________________________________________
Bug-global mailing list
Bug-global@...
http://lists.gnu.org/mailman/listinfo/bug-global

[Shigio YAMAGUCHI-2]

> gtags just write "Cann't write to file" and exits

Summarizing information, maybe, you saw the following output.
Right?

% gtags
gtags: cannot write to database.
% _

If so, write(2) system call failed.
There may be a limitation in your file system or
file system is full.
--
Shigio YAMAGUCHI <shigio@...>
PGP fingerprint: D1CB 0B89 B346 4AB6 5663  C4B6 3CA5 BBB3 57BE DDA3


_______________________________________________
Bug-global mailing list
Bug-global@...
http://lists.gnu.org/mailman/listinfo/bug-global

[Alex Ott-2]

>>>>> "SY" == Shigio YAMAGUCHI writes:
 >> gtags just write "Cann't write to file" and exits
 SY> Summarizing information, maybe, you saw the following output.  Right?

 SY> % gtags gtags: cannot write to database.  % _

Yes

 SY> If so, write(2) system call failed.  There may be a limitation in your
 SY> file system or file system is full.

I have 15Gb of free disk space, and may programs are working with 4Gb iso
images without any problem

P.S. ok - i'll look to the gtags source, to check the corresponding
support for big files

--
With best wishes, Alex Ott, MBA
http://alexott.blogspot.com/           http://xtalk.msk.su/~ott/
http://alexott-ru.blogspot.com/


_______________________________________________
Bug-global mailing list
Bug-global@...
http://lists.gnu.org/mailman/listinfo/bug-global

[Shigio YAMAGUCHI-2]

> P.S. ok - i'll look to the gtags source, to check the corresponding
> support for big files

When you find the method, please teach me.
I'll take it into the source code or FAQ.
Thank you in advance.
--
Shigio YAMAGUCHI <shigio@...>
PGP fingerprint: D1CB 0B89 B346 4AB6 5663  C4B6 3CA5 BBB3 57BE DDA3


_______________________________________________
Bug-global mailing list
Bug-global@...
http://lists.gnu.org/mailman/listinfo/bug-global

[Alex Ott-2]

Hello

>>>>> "SY" == Shigio YAMAGUCHI writes:
 >> P.S. ok - i'll look to the gtags source, to check the corresponding
 >> support for big files

 SY> When you find the method, please teach me.  I'll take it into the
 SY> source code or FAQ.  Thank you in advance.

You need to add to configure the ckeck for support of 64-bit offsets and if
it exists, then add following flags to CFLAGS -> "-D_LARGEFILE64_SOURCE
-D_FILE_OFFSET_BITS=64"

with these flags, all works fine

P.S. why are you using your own copy of libdb (old version) instead of
system-wide? the same for the sys/queue.h header file

--
With best wishes, Alex Ott, MBA
http://alexott.blogspot.com/           http://xtalk.msk.su/~ott/
http://alexott-ru.blogspot.com/


_______________________________________________
Bug-global mailing list
Bug-global@...
http://lists.gnu.org/mailman/listinfo/bug-global

[Shigio YAMAGUCHI-2]

Hello,
> You need to add to configure the ckeck for support of 64-bit offsets and if
> it exists, then add following flags to CFLAGS -> "-D_LARGEFILE64_SOURCE
> -D_FILE_OFFSET_BITS=64"

Thank you for information.
I wrote a FAQ. What do you think?

---------------------------------------------------------------------
Q99:

I'm using GNU system.
In a large project, gtags fails in making tags like follows:

$ gtags
gtags: cannot write to database.
$ _

It seems that gtags cannot make a file over 2GB.

A99:

If your GNU system supports 64-bit file offset then please try the
following configuration.

$ ./configure CFLAGS='-D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64'
---------------------------------------------------------------------

I have two questions.
o Is there a method of checking for support 64-bit offset?
o Is the _LARGEFILE64_SOURCE requred for GLOBAL?

> P.S. why are you using your own copy of libdb (old version) instead of
> system-wide? the same for the sys/queue.h header file

Because there are many libdbs, and they are not compatible each other.

--
Shigio YAMAGUCHI <shigio@...>
PGP fingerprint: D1CB 0B89 B346 4AB6 5663  C4B6 3CA5 BBB3 57BE DDA3


_______________________________________________
Bug-global mailing list
Bug-global@...
http://lists.gnu.org/mailman/listinfo/bug-global

[Alex Ott-2]

>>>>> "SY" == Shigio YAMAGUCHI writes:
 SY> Hello,
 >> You need to add to configure the ckeck for support of 64-bit offsets
 >> and if it exists, then add following flags to CFLAGS ->
 >> "-D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64"

 SY> Thank you for information.  I wrote a FAQ. What do you think?

 SY> ---------------------------------------------------------------------
 SY> Q99:

 SY> I'm using GNU system.  In a large project, gtags fails in making tags
 SY> like follows:

 SY> $ gtags gtags: cannot write to database.  $ _

 SY> It seems that gtags cannot make a file over 2GB.

 SY> A99:

 SY> If your GNU system supports 64-bit file offset then please try the
 SY> following configuration.

 SY> $ ./configure CFLAGS='-D_LARGEFILE64_SOURCE
 SY> -D_FILE_OFFSET_BITS=64'
 SY> ---------------------------------------------------------------------

For me it seems ok

 SY> I have two questions.  o Is there a method of checking for support
 SY> 64-bit offset?  o Is the _LARGEFILE64_SOURCE requred for GLOBAL?

I'm not sure about global, but it definitely need for db-related code.

For autoconf check you can use for example following macro:

http://autoconf-archive.cryp.to/rssh_check_off64_t.html

 >> P.S. why are you using your own copy of libdb (old version) instead of
 >> system-wide? the same for the sys/queue.h header file

 SY> Because there are many libdbs, and they are not compatible each other.

ok

--
With best wishes, Alex Ott, MBA
http://alexott.blogspot.com/           http://xtalk.msk.su/~ott/
http://alexott-ru.blogspot.com/


_______________________________________________
Bug-global mailing list
Bug-global@...
http://lists.gnu.org/mailman/listinfo/bug-global

[Jean-Marc Saffroy-2]

Hi,

On Fri, 23 Jan 2009, Shigio YAMAGUCHI wrote:

> I have two questions.
> o Is there a method of checking for support 64-bit offset?

You can try using the ouput of "getconf LFS_CFLAGS" which AFAIK works on
Linux and Solaris (maybe others).

> o Is the _LARGEFILE64_SOURCE requred for GLOBAL?

Not sure, but I think it's harmless to use it, and on Linux the command
mentioned above defines it anyway.

Maybe some useful background info:
http://en.wikipedia.org/wiki/Large_file_support
http://www.suse.de/~aj/linux_lfs.html


Cheers,
JM

--
saffroy@...


_______________________________________________
Bug-global mailing list
Bug-global@...
http://lists.gnu.org/mailman/listinfo/bug-global

[Shigio YAMAGUCHI-2]

Hi,
> > o Is the _LARGEFILE64_SOURCE requred for GLOBAL?
>
> Not sure, but I think it's harmless to use it, and on Linux the command
> mentioned above defines it anyway.

OK. I'll introduce above variable in the FAQ.

>
> Maybe some useful background info:
> http://en.wikipedia.org/wiki/Large_file_support
> http://www.suse.de/~aj/linux_lfs.html

I'll introduce them in the FAQ too.
Thank you for profitable information.
--
Shigio YAMAGUCHI <shigio@...>
PGP fingerprint: D1CB 0B89 B346 4AB6 5663  C4B6 3CA5 BBB3 57BE DDA3


_______________________________________________
Bug-global mailing list
Bug-global@...
http://lists.gnu.org/mailman/listinfo/bug-global

[Shigio YAMAGUCHI-2]

Hi,

>  SY> $ ./configure CFLAGS='-D_LARGEFILE64_SOURCE
>  SY> -D_FILE_OFFSET_BITS=64'
>  SY> ---------------------------------------------------------------------
>
> For me it seems ok
>
>  SY> I have two questions.  o Is there a method of checking for support
>  SY> 64-bit offset?  o Is the _LARGEFILE64_SOURCE requred for GLOBAL?
>
> I'm not sure about global, but it definitely need for db-related code.
>
> For autoconf check you can use for example following macro:
>
> http://autoconf-archive.cryp.to/rssh_check_off64_t.html

I postpone implementing automatic recognition of LFS this time
to avoid causing new troubles. Instead, I will do two things:

1. Adding a Q and A about LFS to the FAQ.

   I will introduce two definitions _LARGEFILE64_SOURCE and _FILE_OFFSET_BITS=64.

2. Reviving the compact format of GTAGS.

   The compact format reduce the size of GTAGS.

Thanks to Alex and Jean-Marc for profitable information and support.
Cheers,
--
Shigio YAMAGUCHI <shigio@...>
PGP fingerprint: D1CB 0B89 B346 4AB6 5663  C4B6 3CA5 BBB3 57BE DDA3


_______________________________________________
Bug-global mailing list
Bug-global@...
http://lists.gnu.org/mailman/listinfo/bug-global