system() call causes core dump

[PeterSteele]

I have an application running a number of threads. I've had recent instances where the code below is causing a core dump to occur:

char fstatCmd[200];
char *fstatOut = "/tmp/fstat.out";
sprintf(fstatCmd, "fstat | grep -v USER | wc -l >%s", fstatOut);
rc = system(fstatCmd);

The call is simply intended to get a count of the current open handles. The system call though causes a core:

#0 0x0000000801058307 in _spinunlock () from /lib/libthr.so.3
#1 0x00000008011d0afb in _malloc_postfork () from /lib/libc.so.7
#2 0x000000080105c5fb in fork () from /lib/libthr.so.3
#3 0x0000000801191aae in system () from /lib/libc.so.7
#4 0x00000008010553aa in system () from /lib/libthr.so.3
#5 0x000000000040b6f9 in mythread at myapp.c:461
#6 0x0000000801056a88 in pthread_getprio () from /lib/libthr.so.3

There appears to be some kind of thread-safe issue going on. I have a number of threads that are monitoring various items, waking up a differing intervals to do their respective tasks. Do I need to put in a global mutex so that the threads never attempt to make simultaneous system() calls? Curiously, only this particular system() call appears to be causing a core.


_______________________________________________
freebsd-questions@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@..."

[Kris Kennaway-3]

Peter Steele wrote:

> I have an application running a number of threads. I've had recent instances where the code below is causing a core dump to occur:
>
> char fstatCmd[200];
> char *fstatOut = "/tmp/fstat.out";
> sprintf(fstatCmd, "fstat | grep -v USER | wc -l >%s", fstatOut);
> rc = system(fstatCmd);
>
> The call is simply intended to get a count of the current open handles. The system call though causes a core:
>
> #0 0x0000000801058307 in _spinunlock () from /lib/libthr.so.3
> #1 0x00000008011d0afb in _malloc_postfork () from /lib/libc.so.7
> #2 0x000000080105c5fb in fork () from /lib/libthr.so.3
> #3 0x0000000801191aae in system () from /lib/libc.so.7
> #4 0x00000008010553aa in system () from /lib/libthr.so.3
> #5 0x000000000040b6f9 in mythread at myapp.c:461
> #6 0x0000000801056a88 in pthread_getprio () from /lib/libthr.so.3
>
> There appears to be some kind of thread-safe issue going on. I have a number of threads that are monitoring various items, waking up a differing intervals to do their respective tasks. Do I need to put in a global mutex so that the threads never attempt to make simultaneous system() calls? Curiously, only this particular system() call appears to be causing a core.

In UNIX it is not safe to perform arbitrary actions after forking a
multi-threaded process.  You're basically expected to call exec soon
after the fork, although you can do certain other work if you are very
careful.

The reason for this is that after the fork, only one thread will be
running in the child, and if that thread tries to acquire a lock or
other formerly-shared resource it may deadlock or crash, because the
child process is no longer accessing the same memory location as the
threads in the parent process (it gets a separate copy of the address
space at the time of fork, which may not be in a consistent state from
the point of view of the thread library).

Kris
_______________________________________________
freebsd-questions@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@..."

[PeterSteele]

>In UNIX it is not safe to perform arbitrary actions after forking a multi-threaded process.  You're basically expected to call exec soon after the fork, although
>you can do certain other work if you are very careful.

>The reason for this is that after the fork, only one thread will be running in the child, and if that thread tries to acquire a lock or other formerly-shared resource
>it may deadlock or crash, because the child process is no longer accessing the same memory location as the threads in the parent process (it gets a separate copy
>of the address space at the time of fork, which may not be in a consistent state from the point of view of the thread library).

I am not calling fork explicitly. The thread I'm running in was created with pthread_create(). The fork() in the stack trace in my original email is being called by the system() function as it spawns off the process it is supposed want to run. Is there a safe way to call system() within a pthread? The app has several such threads doing various monitoring actions, some calling functions using system(), others invoking various C library routines. The parent process where these threads were spawned from is basically sleeping, waking up only periodically to check for shutdown events.

_______________________________________________
freebsd-questions@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@..."

[usleepless]

On Sat, Oct 31, 2009 at 4:55 PM, Kris Kennaway <kris@...> wrote:

> Peter Steele wrote:
>
>> I have an application running a number of threads. I've had recent
>> instances where the code below is causing a core dump to occur:
>>
>> char fstatCmd[200];
>> char *fstatOut = "/tmp/fstat.out";
>> sprintf(fstatCmd, "fstat | grep -v USER | wc -l >%s", fstatOut);
>> rc = system(fstatCmd);
>>
>> The call is simply intended to get a count of the current open handles.
>> The system call though causes a core:
>>
>> #0 0x0000000801058307 in _spinunlock () from /lib/libthr.so.3
>> #1 0x00000008011d0afb in _malloc_postfork () from /lib/libc.so.7
>> #2 0x000000080105c5fb in fork () from /lib/libthr.so.3
>> #3 0x0000000801191aae in system () from /lib/libc.so.7
>> #4 0x00000008010553aa in system () from /lib/libthr.so.3
>> #5 0x000000000040b6f9 in mythread at myapp.c:461
>> #6 0x0000000801056a88 in pthread_getprio () from /lib/libthr.so.3
>>
>> There appears to be some kind of thread-safe issue going on. I have a
>> number of threads that are monitoring various items, waking up a differing
>> intervals to do their respective tasks. Do I need to put in a global mutex
>> so that the threads never attempt to make simultaneous system() calls?
>> Curiously, only this particular system() call appears to be causing a core.
>>
>
> In UNIX it is not safe to perform arbitrary actions after forking a
> multi-threaded process.  You're basically expected to call exec soon after
> the fork, although you can do certain other work if you are very careful.
>
> The reason for this is that after the fork, only one thread will be running
> in the child, and if that thread tries to acquire a lock or other
> formerly-shared resource it may deadlock or crash, because the child process
> is no longer accessing the same memory location as the threads in the parent
> process (it gets a separate copy of the address space at the time of fork,
> which may not be in a consistent state from the point of view of the thread
> library).
>

Are you  saying system/popen can't be used in threads? Is there a
workaround? ( forking manual and executing exec? )

Would calling 'system("exec fstat | ... > result.txt")' make any difference?

just curious,

kind regards,

usleep
_______________________________________________
freebsd-questions@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@..."

[Mel Flynn-2]

On Saturday 31 October 2009 21:52:37 Peter Steele wrote:

> >In UNIX it is not safe to perform arbitrary actions after forking a
> > multi-threaded process.  You're basically expected to call exec soon
> > after the fork, although you can do certain other work if you are very
> > careful.
> >
> >The reason for this is that after the fork, only one thread will be
> > running in the child, and if that thread tries to acquire a lock or other
> > formerly-shared resource it may deadlock or crash, because the child
> > process is no longer accessing the same memory location as the threads in
> > the parent process (it gets a separate copy of the address space at the
> > time of fork, which may not be in a consistent state from the point of
> > view of the thread library).
>
> I am not calling fork explicitly. The thread I'm running in was created
>  with pthread_create(). The fork() in the stack trace in my original email
>  is being called by the system() function as it spawns off the process it
>  is supposed want to run. Is there a safe way to call system() within a
>  pthread?

Either I'm very lucky, or popen is better suited for this, as I have this
running on various machines, 24/7:

#define PING_CMD \
    "ping -n -c 50 %s 2>/dev/null|egrep 'round-trip|packets received'"

/* worker thread main loop */
void *monitor_host(void *data)
{
...
    if( -1 == asprintf(&cmd, PING_CMD, ip) )
    {
        warnl("Failed to construct command");
        *ex = EX_OSERR;
        return(ex);
    }

....

    while( !signalled )
    {
        if( (cmd_p = popen(cmd, "r")) == NULL )
        {
            warnl("Failed to run command %s", cmd);
            *ex = EX_OSERR;
            return(ex);
        }

        EV_SET(&ch, fileno(cmd_p), EVFILT_READ, EV_ADD|EV_ENABLE,
                0, 0, NULL);
        for( ;; )
        {
            int nev;

            if( signalled ||
                    (nev = kevent(kq, &ch, 1, &ev, 1, &timeout)) == -1 )
            {
                if( signalled == SIGHUP )
                    goto closeproc;
                else
                    goto cleanup;
            }

            if( nev )
                break;
        }
        /* read fp, store in db */
    }
}
--
Mel
_______________________________________________
freebsd-questions@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@..."

[PeterSteele]

> Either I'm very lucky, or popen is better suited for this, as I have this running on various machines, 24/7:
...

I could try popen as a next step. What we tried first was to replace our system calls with a function called vsys() which does exactly the same thing but uses vfork() underneath instead of fork(). If this problem had been due to resource conflicts due to the fork() performed by system(), we figured this would solve the problem. Unfortunately, we are still getting the exact same core with the identical stack trace.

#0 0x0000000801058307 in _spinunlock () from /lib/libthr.so.3
#1 0x00000008011d0afb in _malloc_postfork () from /lib/libc.so.7
#2 0x000000080105c5fb in fork () from /lib/libthr.so.3
#3 0x0000000801191aae in system () from /lib/libc.so.7
#4 0x00000008010553aa in system () from /lib/libthr.so.3
#5 0x000000000040b6f9 in mythread at myapp.c:461
#6 0x0000000801056a88 in pthread_getprio () from /lib/libthr.so.3

I don't know much about the internal implementation of popen but I suspect that at its core it still likely ends up doing a fork(). Still, it's worth a try. Curiously, we have several other threads in our application running in threads that periodically make system calls. For some reason only this call to fstat causes a core to occur. Seems that something fishy is going on here....

_______________________________________________
freebsd-questions@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@..."

[usleepless]

On Fri, Nov 6, 2009 at 4:19 PM, Peter Steele <psteele@...> wrote:

> > Either I'm very lucky, or popen is better suited for this, as I have this
> running on various machines, 24/7:
> ...
>
> I could try popen as a next step. What we tried first was to replace our
> system calls with a function called vsys() which does exactly the same thing
> but uses vfork() underneath instead of fork(). If this problem had been due
> to resource conflicts due to the fork() performed by system(), we figured
> this would solve the problem. Unfortunately, we are still getting the exact
> same core with the identical stack trace.
>
> #0 0x0000000801058307 in _spinunlock () from /lib/libthr.so.3
> #1 0x00000008011d0afb in _malloc_postfork () from /lib/libc.so.7
> #2 0x000000080105c5fb in fork () from /lib/libthr.so.3
> #3 0x0000000801191aae in system () from /lib/libc.so.7
> #4 0x00000008010553aa in system () from /lib/libthr.so.3
> #5 0x000000000040b6f9 in mythread at myapp.c:461
> #6 0x0000000801056a88 in pthread_getprio () from /lib/libthr.so.3
>
> I don't know much about the internal implementation of popen but I suspect
> that at its core it still likely ends up doing a fork(). Still, it's worth a
> try. Curiously, we have several other threads in our application running in
> threads that periodically make system calls. For some reason only this call
> to fstat causes a core to occur. Seems that something fishy is going on
> here....
>
>

I am following this thread since i find it interesting ( playing around with
popen & threads at times ).

I am still trying to understand what Kris said. He says it is not safe to
fork from a multithread proces. And he suggests using execve.

You stacktrace btw is consistent with what Kris says: "and if that thread
tries to acquire a lock or other formerly-shared resource it may deadlock or
crash, because the child process is no longer accessing the same memory
location as the threads in the parent process". -> that's probably the
spinlock in malloc.

I think you need to 1. (v)fork, 2. run execve on your fstat ( make it a sh
script )

Something along the lines of this

<< your current thread code here >>

// rc = system(fstatCmd);

childpid = fork();

    if (childpid >= 0) /* fork succeeded */
    {
        if (childpid == 0) /* fork() returns 0 to the child process */
        {

           execve("/usr/local/app/myscript", NULL, NULL);

        }

    }

    else

    {

      // didn't work, log something

    }

<< rest of your code here >>

regards,

usleep




> _______________________________________________
> freebsd-questions@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "
> freebsd-questions-unsubscribe@..."
>
_______________________________________________
freebsd-questions@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@..."

[PeterSteele]

>I am following this thread since i find it interesting ( playing around with popen & threads at times ).
>
>I am still trying to understand what Kris said. He says it is not safe to fork from a multithread proces. And he suggests using execve.
>
>You stacktrace btw is consistent with what Kris says: "and if that thread tries to acquire a lock or other formerly-shared resource it may deadlock or crash, because the child process is
>no longer accessing the same memory location as the threads in the parent process". -> that's probably the spinlock in malloc.

This is why we switched to vsys/vfork, assuming it would effectively do the same thing as execve. Perhaps not not. I can give it a try. I am still curious though why only this one particular system call is causing a crash. We have many calls running in other threads and none have ever caused a problem. They all do the same thing, run a command and save the output in a file, and then we process the output. The only thing different is the command we run in this case-fstat.

_______________________________________________
freebsd-questions@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@..."

[usleepless]

On Fri, Nov 6, 2009 at 5:52 PM, Peter Steele <psteele@...> wrote:

> >I am following this thread since i find it interesting ( playing around
> with popen & threads at times ).
> >
> >I am still trying to understand what Kris said. He says it is not safe to
> fork from a multithread proces. And he suggests using execve.
> >
> >You stacktrace btw is consistent with what Kris says: "and if that thread
> tries to acquire a lock or other formerly-shared resource it may deadlock or
> crash, because the child process is
> >no longer accessing the same memory location as the threads in the parent
> process". -> that's probably the spinlock in malloc.
>
> This is why we switched to vsys/vfork, assuming it would effectively do the
> same thing as execve. Perhaps not not. I can give it a try. I am still
> curious though why only this one particular system call is causing a crash.
> We have many calls running in other threads and none have ever caused a
> problem. They all do the same thing, run a command and save the output in a
> file, and then we process the output. The only thing different is the
> command we run in this case-fstat.
>

Actually, i have just read the implementation of system() (
/usr/src/lib/libc/system.c ), It does fork() and then execl(). So my
suggestion is not going to work. Then i read some of the malloc code (
/usr/src/lib/libstd/malloc.c ). It uses spinlocks alright, and they need to
be initialized. My guess is that are not reinitialized in your forked
process. I have no clue how to fix that. Another thought was that your
application is compiled as multithreaded, but fstat, grep etc are probably
not. But from what i can see, they use the same malloc implementation.

I think the crash in the malloc is from a malloc in fstat or grep ( there is
no malloc in system() ).

regards,

usleep







> _______________________________________________
> freebsd-questions@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "
> freebsd-questions-unsubscribe@..."
>
_______________________________________________
freebsd-questions@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@..."

[PeterSteele]

> I think the crash in the malloc is from a malloc in fstat or grep ( there is no malloc in system() ).

As a test I changed the 'fstat|wc -l' call we use now to 'ps aux|wc -l'. The problem does not occur all the time but the QA group is seeing it periodically during their test runs. I'm going to let the alternative version got through some days of testing to see if the problem occurs again. If it doesn't, it may very well be something to do with fstat. If it does, well, I guess we have more digging to do...


_______________________________________________
freebsd-questions@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@..."