Running a program through gdb without "interfering"

[Mel Flynn-6]

Hi,

is there a way to have a program run through gdb and gdb only record a
segfault, but otherwise let the program run?

Why I'd like this is the following:
I've got a i386 jail on an amd64 box, running 7.2-p4. UNAME_p and UNAME_m have
been set to i386 as well as ARCH in /etc/make.conf. Running portmaster[1] to
build ports under my uid and PM_SU_CMD, sudo *sometimes* segfaults. It's only
sudo, so at present I don't have a reason to doubt memory. However, it doesn't
dump core, so I'm at a loss what the culprit could be.

[1] In order to get this working I had to put a statically compiled ps in the
jail, or the uid test would fail. It has the downside that it lists both jail
and host processes, but it is acceptable to me as the jail is only accessible
from the host (pf enforced). I suspect sudo to have a similar problem or even
related to ps returning processes from a uid that doesn't exist in the jail,
but without a backtrace I don't know what to fix.
--
Mel
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Mel Flynn-6]

On Friday 09 October 2009 00:38:32 Paul B Mahol wrote:

> On 10/9/09, Mel Flynn <mel.flynn+fbsd.hackers@...> wrote:
> > Hi,
> >
> > is there a way to have a program run through gdb and gdb only record a
> > segfault, but otherwise let the program run?
> >
> > Why I'd like this is the following:
> > I've got a i386 jail on an amd64 box, running 7.2-p4. UNAME_p and UNAME_m
> > have
> > been set to i386 as well as ARCH in /etc/make.conf. Running portmaster[1]
> > to build ports under my uid and PM_SU_CMD, sudo *sometimes* segfaults.
> > It's only
> > sudo, so at present I don't have a reason to doubt memory. However, it
> > doesn't
> > dump core, so I'm at a loss what the culprit could be.
>
> Tried 'sysctl kern.sugid_coredump=1' ?

Hmm, no. Enabled now and waiting for the next segfault.
I actually looked at the sysctl -d, but it didn't register that this could be
the main cause.
Perhaps that sentence could be more clear:
-kern.sugid_coredump: Enable coredumping set user/group ID processes
+kenr.sugid_coredump: Allow setuid/setgid processes to dump core

--
Mel
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Gary Jennejohn-3]

On Fri, 9 Oct 2009 01:16:59 +0200
Mel Flynn <mel.flynn+fbsd.hackers@...> wrote:

> On Friday 09 October 2009 00:38:32 Paul B Mahol wrote:
> > On 10/9/09, Mel Flynn <mel.flynn+fbsd.hackers@...> wrote:
> > > Hi,
> > >
> > > is there a way to have a program run through gdb and gdb only record a
> > > segfault, but otherwise let the program run?
> > >
> > > Why I'd like this is the following:
> > > I've got a i386 jail on an amd64 box, running 7.2-p4. UNAME_p and UNAME_m
> > > have
> > > been set to i386 as well as ARCH in /etc/make.conf. Running portmaster[1]
> > > to build ports under my uid and PM_SU_CMD, sudo *sometimes* segfaults.
> > > It's only
> > > sudo, so at present I don't have a reason to doubt memory. However, it
> > > doesn't
> > > dump core, so I'm at a loss what the culprit could be.
> >
> > Tried 'sysctl kern.sugid_coredump=1' ?
>
> Hmm, no. Enabled now and waiting for the next segfault.
> I actually looked at the sysctl -d, but it didn't register that this could be
> the main cause.
> Perhaps that sentence could be more clear:
> -kern.sugid_coredump: Enable coredumping set user/group ID processes
> +kenr.sugid_coredump: Allow setuid/setgid processes to dump core
>

See the info file for gdb, section 5.3 Signals.  It's possible to tell
gdb how to handle signals, e.g. stop vs. nostop, etc.

---
Gary Jennejohn
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Dag-Erling Smørgrav]

Mel Flynn <mel.flynn+fbsd.hackers@...> writes:
> is there a way to have a program run through gdb and gdb only record a
> segfault, but otherwise let the program run?

Yes, just run "gdb /path/to/program" and type "run".

> [...] sudo *sometimes* segfaults [...] However, it doesn't dump core

sudo(1) is setuid root.  You need to set kern.sugid_coredump to get it
to dump core.

> [1] In order to get this working I had to put a statically compiled ps in the
> jail, or the uid test would fail. It has the downside that it lists both jail
> and host processes, [...]

Uh, no.  Processes outside the jail are not visible inside it, no matter
what version of ps(1) or top(1) or any other such program you use.

DES
--
Dag-Erling Smørgrav - des@...
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Stef Walter]

Mel Flynn wrote:
> [1] In order to get this working I had to put a statically compiled ps in the
> jail

This is a pretty standard practice. I always put these statically built
into any jails that don't match the outside system. I use the following
crunchgen config to accomplish that.

Cheers,

Stef



# Commands to build in
progs ps ipcs netstat pkill top w killall
progs systat iostat
progs jkill
progs kldstat

# Link these programs to each other
ln pkill pgrep
ln w uptime

# Libraries which we need
libs -lutil -lkvm -ldevstat -lncurses -ldevstat -lm -lnetgraph -lmemstat
-lipx
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Mel Flynn-6]

On Friday 09 October 2009 11:38:29 Dag-Erling Smørgrav wrote:
> Mel Flynn <mel.flynn+fbsd.hackers@...> writes:
> > is there a way to have a program run through gdb and gdb only record a
> > segfault, but otherwise let the program run?
>
> Yes, just run "gdb /path/to/program" and type "run".

Not what I was looking for. The segfaults are random and the only way to
somewhat reliably reproduce it is to have portmaster invoke it as it's
PM_SU_CMD. And no, running that same command again doesn't trigger the
segfault, so it's "something environmental". Hence I'm looking for something
like:
gdb -batch -x script_with_run_cmd.gdb -exec /usr/local/bin/sudo $argv

where somehow I need $argv to be passed as arguments to sudo. I'm thinking i
should just wrap it and mktemp(1) a new command script for gdb to use with set
args $*, but if anyone has a more clever idea, I'd love to hear it.

> > [...] sudo *sometimes* segfaults [...] However, it doesn't dump core
>
> sudo(1) is setuid root.  You need to set kern.sugid_coredump to get it
> to dump core.

It still segfaults and doesn't dump:
Oct  9 04:34:18 smell kernel: pid 39476 (sudo), uid 0: exited on signal 11
Oct  9 04:36:32 smell kernel: pid 79657 (sudo), uid 0: exited on signal 11
Oct  9 04:36:43 smell kernel: pid 82390 (sudo), uid 0: exited on signal 11
Oct  9 04:51:46 smell kernel: pid 3601 (sudo), uid 0: exited on signal 11

find / -name '*.core' in the jail does not yield anything.

> > [1] In order to get this working I had to put a statically compiled ps in
> > the jail, or the uid test would fail. It has the downside that it lists
> > both jail and host processes, [...]
>
> Uh, no.  Processes outside the jail are not visible inside it, no matter
> what version of ps(1) or top(1) or any other such program you use.

I'll write this off as pilot error, cause I cannot reproduce it. I saw bash as
one of the processes listed in a blank ps run, which isn't installed in the
jail, but since I don't have the terminal history anymore, it's entirely
possible I ran ps on the host.
--
Mel
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Mel Flynn-6]

On Friday 09 October 2009 16:50:04 Mel Flynn wrote:
> On Friday 09 October 2009 11:38:29 Dag-Erling Smørgrav wrote:
> > Mel Flynn <mel.flynn+fbsd.hackers@...> writes:

> > > [...] sudo *sometimes* segfaults [...] However, it doesn't dump core
> >
> > sudo(1) is setuid root.  You need to set kern.sugid_coredump to get it
> > to dump core.
>
> It still segfaults and doesn't dump:
> Oct  9 04:34:18 smell kernel: pid 39476 (sudo), uid 0: exited on signal 11
> Oct  9 04:36:32 smell kernel: pid 79657 (sudo), uid 0: exited on signal 11
> Oct  9 04:36:43 smell kernel: pid 82390 (sudo), uid 0: exited on signal 11
> Oct  9 04:51:46 smell kernel: pid 3601 (sudo), uid 0: exited on signal 11
>
> find / -name '*.core' in the jail does not yield anything.

FYI, there's one read-only mount into the jail, being /usr/src. I don't see a
reason given the commands it segfaults on, for $cwd to be below that.For
example it segfaulted on sudo pkg_delete glproto2.
Thought I'd mention it to rule it out.
--
Mel
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Nate Eldredge-4]

On Fri, 9 Oct 2009, Mel Flynn wrote:

> On Friday 09 October 2009 11:38:29 Dag-Erling Smørgrav wrote:
>> Mel Flynn <mel.flynn+fbsd.hackers@...> writes:
>>> is there a way to have a program run through gdb and gdb only record a
>>> segfault, but otherwise let the program run?
>>
>> Yes, just run "gdb /path/to/program" and type "run".
>
> Not what I was looking for. The segfaults are random and the only way to
> somewhat reliably reproduce it is to have portmaster invoke it as it's
> PM_SU_CMD. And no, running that same command again doesn't trigger the
> segfault, so it's "something environmental". Hence I'm looking for something
> like:
> gdb -batch -x script_with_run_cmd.gdb -exec /usr/local/bin/sudo $argv
>
> where somehow I need $argv to be passed as arguments to sudo. I'm thinking i
> should just wrap it and mktemp(1) a new command script for gdb to use with set
> args $*, but if anyone has a more clever idea, I'd love to hear it.

This won't work.  You can't debug setuid programs (for reasons which
should be obvious).  You could do it if you ran everything as root, but it
sounds like the bug doesn't occur in that case.

--

Nate Eldredge
nate@...
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Mel Flynn-6]

On Friday 09 October 2009 16:50:04 Mel Flynn wrote:

> On Friday 09 October 2009 11:38:29 Dag-Erling Smørgrav wrote:
> > Mel Flynn <mel.flynn+fbsd.hackers@...> writes:
> > > is there a way to have a program run through gdb and gdb only record a
> > > segfault, but otherwise let the program run?
> >
> > Yes, just run "gdb /path/to/program" and type "run".
>
> Not what I was looking for. The segfaults are random and the only way to
> somewhat reliably reproduce it is to have portmaster invoke it as it's
> PM_SU_CMD. And no, running that same command again doesn't trigger the
> segfault, so it's "something environmental". Hence I'm looking for
>  something like:
> gdb -batch -x script_with_run_cmd.gdb -exec /usr/local/bin/sudo $argv
>
> where somehow I need $argv to be passed as arguments to sudo. I'm thinking
>  i should just wrap it and mktemp(1) a new command script for gdb to use
>  with set args $*, but if anyone has a more clever idea, I'd love to hear
>  it.

Dead end path :/
% bin/gdbsudo echo hi
/tmp/gdbsudo.F3kdwJ:1: Error in sourced command file:
/usr/local/bin/sudo: Permission denied.

% ls -l /usr/local/bin/sudo
---s--x--x  2 root  wheel  116380 Oct  8 18:31 /usr/local/bin/sudo

% sudo chmod g+r /usr/local/bin/sudo

% bin/gdbsudo echo hi

(no debugging symbols found)...(no debugging symbols found)...(no debugging
symbols found)...(no debugging symbols found)...sudo: must be setuid root

Program exited with code 01.

Perhaps the cause of it not dumping core either. Would've been nice to know
why it segfaults, but not nice enough to keep digging.
--
Mel
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Dag-Erling Smørgrav]

Mel Flynn <mel.flynn+fbsd.hackers@...> writes:

> Dag-Erling Smørgrav <des@...> writes:
> > Yes, just run "gdb /path/to/program" and type "run".
> Not what I was looking for. The segfaults are random and the only way to
> somewhat reliably reproduce it is to have portmaster invoke it as it's
> PM_SU_CMD. And no, running that same command again doesn't trigger the
> segfault, so it's "something environmental". Hence I'm looking for something
> like:
> gdb -batch -x script_with_run_cmd.gdb -exec /usr/local/bin/sudo $argv
>
> where somehow I need $argv to be passed as arguments to sudo. I'm thinking i
> should just wrap it and mktemp(1) a new command script for gdb to use with set
> args $*, but if anyone has a more clever idea, I'd love to hear it.

Why look for a clever option, when the simple one will do just fine?

:>gdb-script-$$
echo "set args $@" >>gdb-script-$$
echo "run" >>gdb-script-$$
gdb -batch -x gdb-script-$$ /usr/local/bin/sudo

> It still segfaults and doesn't dump:
> Oct  9 04:34:18 smell kernel: pid 39476 (sudo), uid 0: exited on signal 11
> Oct  9 04:36:32 smell kernel: pid 79657 (sudo), uid 0: exited on signal 11
> Oct  9 04:36:43 smell kernel: pid 82390 (sudo), uid 0: exited on signal 11
> Oct  9 04:51:46 smell kernel: pid 3601 (sudo), uid 0: exited on signal 11
>
> find / -name '*.core' in the jail does not yield anything.

Add 'ulimit -c unlimited' somewhere in the script before it invokes sudo.

DES
--
Dag-Erling Smørgrav - des@...
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Dag-Erling Smørgrav]

Nate Eldredge <nate@...> writes:
> This won't work.  You can't debug setuid programs (for reasons which
> should be obvious).

Ah, true, but easily fixable.  Add a sysctl for it (just copy-paste the
declaration for kern.sugid_coredump and change the name) and check its
value in p_candebug() (hint: "if (credentialchanged)").

DES
--
Dag-Erling Smørgrav - des@...
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Mel Flynn-6]

On Friday 09 October 2009 21:27:21 Dag-Erling Smørgrav wrote:

> Mel Flynn <mel.flynn+fbsd.hackers@...> writes:
> > Dag-Erling Smørgrav <des@...> writes:
> > > Yes, just run "gdb /path/to/program" and type "run".
> >
> > Not what I was looking for. The segfaults are random and the only way to
> > somewhat reliably reproduce it is to have portmaster invoke it as it's
> > PM_SU_CMD. And no, running that same command again doesn't trigger the
> > segfault, so it's "something environmental". Hence I'm looking for
> > something like:
> > gdb -batch -x script_with_run_cmd.gdb -exec /usr/local/bin/sudo $argv
> >
> > where somehow I need $argv to be passed as arguments to sudo. I'm
> > thinking i should just wrap it and mktemp(1) a new command script for gdb
> > to use with set args $*, but if anyone has a more clever idea, I'd love
> > to hear it.
>
> Why look for a clever option, when the simple one will do just fine?

Cause I don't know how much of the cause of this bug I'm influencing. Even
though this is now the simple solution, it would be simpler if gdb (or another
debugger) could work similar as sudo, where it would take the first argument
as binary and the rest as arguments to the binary. This would do away with
some extra IO I'm now creating. Though, it's unlikely it is related to IO,
there is no pattern that I've found yet for the segfault, so I'm trying to
limit any "extra stuff".

I'll patch the kernel tomorrow with the new sysctl and see how far that gets
me.


> Add 'ulimit -c unlimited' somewhere in the script before it invokes sudo.

I'll add it.
--
Mel
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."