gtk-gnutella-devel Digest, Vol 18, Issue 2

View: New views

1 Messages — Rating Filter: Alert me

gtk-gnutella-devel Digest, Vol 18, Issue 2

by gtk-gnutella-devel-request :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Send gtk-gnutella-devel mailing list submissions to
gtk-gnutella-devel@...

To subscribe or unsubscribe via the World Wide Web, visit
https://lists.sourceforge.net/lists/listinfo/gtk-gnutella-devel
or, via email, send a message with subject or body 'help' to
gtk-gnutella-devel-request@...

You can reach the person managing the list at
gtk-gnutella-devel-owner@...

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gtk-gnutella-devel digest..."

Today's Topics:

1. Re: Dtrace and memory leaks, a first effort. (Christian Biere)
2. Re: Dtrace and memory leaks, a first effort. (Matthew Lye)
3. Re: Dtrace and memory leaks, a first effort. (Christian Biere)
4. Re: Dtrace and memory leaks, a first effort. (Matthew Lye)
5. Re: Dtrace and memory leaks, a first effort. (Christian Biere)
6. Re: Dtrace and memory leaks, a first effort. (Alex Bennee)
7. Re: Unexpected message ratio? [moved from ..-users] (Matthew Lye)

----------------------------------------------------------------------

Message: 1
Date: Sat, 1 Dec 2007 20:20:01 +0100
From: Christian Biere <christianbiere@...>
Subject: Re: [gtk-gnutella-devel] Dtrace and memory leaks, a first
effort.
To: gtk-gnutella-devel@...
Message-ID: <20071201192001.GB22607@cyclonus>
Content-Type: text/plain; charset=us-ascii

Matthew Lye wrote:
> I've been messing around a little bit with the new development tools
> in Leopard. If I've managed to use the memory leak tool properly
> (based on Dtrace, with a GUI), and do in fact understand what I'm
> doing, you should find that the logic of the following functions
> allows a return without deallocating allocated memory, i.e., they can
> (and sometimes do) leak.

> If this proves to be correct, and useful, then I'll do further testing.
[...]

Thanks. These leaks should be plugged in current SVN now.

--
1000 octets = 1 ko = 1 kilooctet; 1024 octets = 1 Kio = 1 kibioctet
1000^2 octets = 1 Mo = 1 megaoctet; 1024^2 octets = 1 Mio = 1 mebioctet
1000^3 octets = 1 Go = 1 gigaoctet; 1024^3 octets = 1 Gio = 1 gibioctet

------------------------------

Message: 2
Date: Sun, 2 Dec 2007 11:24:31 -0500
From: Matthew Lye <mlye@...>
Subject: Re: [gtk-gnutella-devel] Dtrace and memory leaks, a first
effort.
To: gtk-gnutella-devel@...
Message-ID: <47E8F40D-A7CA-4CA6-8D79-B36072CFF212@...>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes

The following also seems to be a source of several memory leaks, about
144 bytes each. I'd found 1k-sized leaks earlier, but unfortunately
the "Instruments" program crashed before they could be recorded. I
believe they involved a call to TLS as well, though, so they could
have been from the same place. I need to find a way of invoking the
Dtrace instruments with a lot less overhead, in order to run it for
more extended periods of time. As it is, it tends to bog things down.

0 libSystem.B.dylib malloc
1 libgnutls.13.dylib _gnutls_mpi_dprint_lz
2 libgnutls.13.dylib _gnutls_dh_set_peer_public
3 libgnutls.13.dylib _gnutls_proc_dh_common_server_kx
4 libgnutls.13.dylib proc_anon_server_kx
5 libgnutls.13.dylib _gnutls_recv_server_kx_message
6 libgnutls.13.dylib _gnutls_handshake_client
7 libgnutls.13.dylib gnutls_handshake
8 gtk-gnutella tls_handshake
9 gtk-gnutella socket_tls_setup
10 gtk-gnutella socket_connected
11 gtk-gnutella dispatch_poll
12 libglib-1.2.0.dylib g_main_dispatch
13 libglib-1.2.0.dylib g_main_iterate
14 libglib-1.2.0.dylib g_main_run
15 libgtk-1.2.0.dylib gtk_main
16 gtk-gnutella main_gui_run
17 gtk-gnutella main
18 gtk-gnutella start
19 0x0

I also ran across three other leaks, the third of which is probably
covered by the leak already mentioned. It seems to be the same, just
coming from a callback or something.

probably "main_gui_early_init" as it calls "gtk_init":
0 libSystem.B.dylib malloc
1 libX11.6.dylib _X11TransParseAddress
2 libX11.6.dylib _X11TransConnect
3 libX11.6.dylib _X11TransConnectDisplay
4 libX11.6.dylib XOpenDisplay
5 libgdk-1.2.0.dylib gdk_init_check
6 libgtk-1.2.0.dylib gtk_init_check
7 libgtk-1.2.0.dylib gtk_init
8 gtk-gnutella main_gui_early_init
9 gtk-gnutella main
10 gtk-gnutella start
11 0x0

probably "create pixmap" as it calls "gtk_pixmap_new":
0 libSystem.B.dylib calloc
1 libglib-1.2.0.dylib g_malloc0
2 libgtk-1.2.0.dylib gtk_type_new
3 libgtk-1.2.0.dylib gtk_pixmap_new
4 gtk-gnutella create_pixmap
5 gtk-gnutella icon_init
6 gtk-gnutella main_gui_run
7 gtk-gnutella main
8 gtk-gnutella start
9 0x0

probably "tls_handshake" as it calls "gnutls_handshake":
0 libSystem.B.dylib malloc
1 libgnutls.13.dylib _gnutls_mpi_dprint_lz
2 libgnutls.13.dylib _gnutls_dh_set_peer_public
3 libgnutls.13.dylib _gnutls_proc_dh_common_server_kx
4 libgnutls.13.dylib proc_anon_server_kx
5 libgnutls.13.dylib _gnutls_recv_server_kx_message
6 libgnutls.13.dylib _gnutls_handshake_client
7 libgnutls.13.dylib gnutls_handshake
8 gtk-gnutella tls_handshake
9 gtk-gnutella socket_tls_setup
10 gtk-gnutella socket_connected
11 gtk-gnutella dispatch_poll
12 libglib-1.2.0.dylib g_main_dispatch
13 libglib-1.2.0.dylib g_main_iterate
14 libgtk-1.2.0.dylib gtk_main_iteration_do
15 gtk-gnutella gtk_main_flush
16 gtk-gnutella recurse_scan_intern
17 gtk-gnutella recurse_scan_intern
18 gtk-gnutella recurse_scan_intern
19 gtk-gnutella share_scan
20 gtk-gnutella scan_files_once
21 libglib-1.2.0.dylib g_timeout_dispatch
22 libglib-1.2.0.dylib g_main_dispatch
23 libglib-1.2.0.dylib g_main_iterate
24 libglib-1.2.0.dylib g_main_run
25 libgtk-1.2.0.dylib gtk_main
26 gtk-gnutella main_gui_run
27 gtk-gnutella main
28 gtk-gnutella start
29 0x0

------------------------------

Message: 3
Date: Sun, 2 Dec 2007 18:03:32 +0100
From: Christian Biere <christianbiere@...>
Subject: Re: [gtk-gnutella-devel] Dtrace and memory leaks, a first
effort.
To: gtk-gnutella-devel@...
Message-ID: <20071202170332.GE27668@cyclonus>
Content-Type: text/plain; charset=us-ascii

Matthew Lye wrote:

> The following also seems to be a source of several memory leaks, about
> 144 bytes each. I'd found 1k-sized leaks earlier, but unfortunately
> the "Instruments" program crashed before they could be recorded. I
> believe they involved a call to TLS as well, though, so they could
> have been from the same place. I need to find a way of invoking the
> Dtrace instruments with a lot less overhead, in order to run it for
> more extended periods of time. As it is, it tends to bog things down.
>
> 0 libSystem.B.dylib malloc
> 1 libgnutls.13.dylib _gnutls_mpi_dprint_lz
> 2 libgnutls.13.dylib _gnutls_dh_set_peer_public
> 3 libgnutls.13.dylib _gnutls_proc_dh_common_server_kx
> 4 libgnutls.13.dylib proc_anon_server_kx
> 5 libgnutls.13.dylib _gnutls_recv_server_kx_message
> 6 libgnutls.13.dylib _gnutls_handshake_client
> 7 libgnutls.13.dylib gnutls_handshake
> 8 gtk-gnutella tls_handshake
> 9 gtk-gnutella socket_tls_setup
> 10 gtk-gnutella socket_connected
> 11 gtk-gnutella dispatch_poll

Isn't there a counter which shows how often this leak occurs? If there's only
one, there's probably some initialization the first time one of these functions
is called and it's not really a leak. If this happens for each handshake, that
would be something to look at.

> probably "main_gui_early_init" as it calls "gtk_init":
[...]
> probably "create pixmap" as it calls "gtk_pixmap_new":
[...]

These two are not really memory leaks. They are not accumulating. It's
some memory allocated at startup used for the whole run time. It's only
a "leak" in so far that the memory isn't released on exit.

--
1000 octets = 1 ko = 1 kilooctet; 1024 octets = 1 Kio = 1 kibioctet
1000^2 octets = 1 Mo = 1 megaoctet; 1024^2 octets = 1 Mio = 1 mebioctet
1000^3 octets = 1 Go = 1 gigaoctet; 1024^3 octets = 1 Gio = 1 gibioctet

------------------------------

Message: 4
Date: Sun, 2 Dec 2007 13:52:20 -0500
From: Matthew Lye <mlye@...>
Subject: Re: [gtk-gnutella-devel] Dtrace and memory leaks, a first
effort.
To: gtk-gnutella-devel@...
Message-ID: <F649C1A2-C55C-498C-BE87-65CC4AEB8D9F@...>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes

On 2-Dec-07, at 12:03 PM, Christian Biere wrote:
> Isn't there a counter which shows how often this leak occurs? If
> there's only
> one, there's probably some initialization the first time one of
> these functions
> is called and it's not really a leak. If this happens for each
> handshake, that
> would be something to look at.

It happens more than once, yeah. I think there were 20+ instances
that accumulated over an hour or so. It's visible in the screenshots
I sent you directly, to the left of the . So far, though, I don't
have any easy way to export this information as text. Apple's command-
line 'leak' tool might be an option, I'm not sure yet. so far I
haven't gotten it to cough up a stack trace, which it's supposed to
do. I'm not entirely sure the leak tracing relies on Dtrace, after
all; it may just be the object history aspect of the 'Instrument' run
tracing thing.

> These two are not really memory leaks. They are not accumulating. It's
> some memory allocated at startup used for the whole run time. It's
> only
> a "leak" in so far that the memory isn't released on exit.

If I understand correctly, the leak reporting relies on whether or not
there's a pointer to the memory block, or some such thing. It also
relies on the use (ultimately) of the system's standard malloc, I
think. The documentation is a bit above my comprehension level for
regular casual reading, which is all I've done so far.

I accept your assertion, but can't disambiguate. I'll ignore them in
the future.

------------------------------

Message: 5
Date: Sun, 2 Dec 2007 22:33:53 +0100
From: Christian Biere <christianbiere@...>
Subject: Re: [gtk-gnutella-devel] Dtrace and memory leaks, a first
effort.
To: gtk-gnutella-devel@...
Message-ID: <20071202213353.GA11288@cyclonus>
Content-Type: text/plain; charset=us-ascii

Matthew Lye wrote:
> If I understand correctly, the leak reporting relies on whether or not
> there's a pointer to the memory block, or some such thing. It also
> relies on the use (ultimately) of the system's standard malloc, I
> think. The documentation is a bit above my comprehension level for
> regular casual reading, which is all I've done so far.

You should compile gtk-gnutella with --cflags=-DMALLOC when looking for
memory leaks.

--
1000 octets = 1 ko = 1 kilooctet; 1024 octets = 1 Kio = 1 kibioctet
1000^2 octets = 1 Mo = 1 megaoctet; 1024^2 octets = 1 Mio = 1 mebioctet
1000^3 octets = 1 Go = 1 gigaoctet; 1024^3 octets = 1 Gio = 1 gibioctet

------------------------------

Message: 6
Date: Mon, 3 Dec 2007 10:20:40 +0000
From: Alex Bennee <alex@...>
Subject: Re: [gtk-gnutella-devel] Dtrace and memory leaks, a first
effort.
To: gtk-gnutella-devel@...
Message-ID: <20071203102040.GA15663@...>
Content-Type: text/plain; charset=us-ascii

On Sun, Dec 02, 2007 at 01:52:20PM -0500, Matthew Lye wrote:
>
> On 2-Dec-07, at 12:03 PM, Christian Biere wrote:
> > These two are not really memory leaks. They are not accumulating. It's
> > some memory allocated at startup used for the whole run time. It's
> > only
> > a "leak" in so far that the memory isn't released on exit.
>
> If I understand correctly, the leak reporting relies on whether or not
> there's a pointer to the memory block, or some such thing.

I'm assuming it means if there is still a accessible pointer to the
block of memory. It's quite usual for Unix program to allocate
memory and never free it as the OS will claim it all back when the
program ends. However you can check to see if there are still
accessible pointers to the allocated memory at the end of your run. Unfortunately you can't
just check memory for them as they may be orphaned.

------- ------- ---
|A *--|--->|B *--|--->|C|
------- ------- ---

If A get free() without cleaning up B and C then a pointer to C will
still exist in memory but the program will never be able to find it as
it is orphaned. The situation is complicated if you have many cross
linked structures where multiple things may point at them.

I wouldn't rule out those reported leaks without looking at the cases
where they are free'ed and structures that point to them are free'd.

--
Alex, homepage: http://www.bennee.com/~alex/
Presidency: The greased pig in the field game of American politics.
-- Ambrose Bierce

------------------------------

Message: 7
Date: Wed, 5 Dec 2007 00:44:24 -0500
From: Matthew Lye <mlye@...>
Subject: Re: [gtk-gnutella-devel] Unexpected message ratio? [moved
from ..-users]
To: gtk-gnutella-devel@...
Message-ID: <CD306B59-A8AB-48B2-8044-2026C88CAF26@...>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes

On 4-Dec-07, at 9:07 PM, Christian Biere wrote:
> If you run multiple peers at the same time, do NOT copy the
> configuration files. At the very least, remove "guid" and
> "servent_kuid"
> from config_gnet because these might be unique per peer by all
> means. In
> fact, they are no longer persistent in current SVN because I've
> noticed
> a few peers sharing a servent ID ("guid").

This is a possible oops. If internally indexed only by guid/kuid,
then subsequently the IP:port is looked up... right.
In my case, the guids were different, but not the kademlia
identifiers. They've been regenerated with unique values by following
your suggestion.

> The hostcache is not managed in any smart way. If it uses any strategy
> it must be garbage in, garbage out. Are you sure, both are using the
> same hostiles.txt?

They should be, as they're both updated at the same time.
'use_global_hostiles_txt' is commented out. No 'hostiles.txt' appears
in any gtk-gnutella directory. I'd always taken this to mean that the
hostiles data was stored internally in the gtk-gnutella binary
itself. I certainly don't have overwhelming spam problems.

If the hostcache is not managed in any way to prevent gaming, isn't X-
try a wee bit of a vulnerability? A 'full' malware node could advise
the client to go try twenty other malware nodes. The nefarious
benefit of this is... well, far too diabolical for me to comprehend.

>> I seem to be getting an extremely high number of 'unexpected message'
>> packet drops, according to the statistics panel. For instance, right
>> now, I've gotten 2267 local DB searches, 81,000+ query hits for local
>> queries, and 288,500+ (that's correct) packets dropped as "Unexpected
>> message."
>
>> Is this in any way unusual?
>
> This looks rather unusual. You could check the sources for
> occurences of
> MSG_DROP_UNEXPECTED and add some additional debug output to see where
> most of these derive from.

Right now, the problem is acting ephemeral. It looks to me like
there's supposed to be some reporting if node_debug is set, and a bit
more for 3+ on the search_debug. MSG_DROP_UNEXPECTED occurs in
"nodes.c", "search.c", and "udp.c". The situations are:

nodes.c:
unexpected QRP message
unexpected HSEP message

search.c:
unexpected OOB hit indication

udp.c:
Queries not yet processed from UDP [this refers to GUESS queries.]
Gnutella message not processed from UDP [see quote below.]

> /*
> * We only support a subset of Gnutella message from UDP. In
> particular,
> * messages like HSEP data, BYE or QRP are not expected!
> */

>> My peers are predominantly BearShare, in the version 5.X range, in
>> Canada or the United States, with a handful from Poland, and a few
>>> from various other countries.
>
> I've seen quite a few "BearShare (Polska)" peers too, a bit too many
> for
> my taste. I suspected it's because Poland is geographically next to
> Germany but they are noticeable to you as well, that's somewhat odd.

I've noticed a significant number of BearShare (Polska) servents for
several months.

Matt

------------------------------

-------------------------------------------------------------------------
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell. From the desktop to the data center, Linux is going
mainstream. Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4

------------------------------

_______________________________________________
gtk-gnutella-devel mailing list
gtk-gnutella-devel@...
https://lists.sourceforge.net/lists/listinfo/gtk-gnutella-devel

End of gtk-gnutella-devel Digest, Vol 18, Issue 2
*************************************************