StringIO and reopen

[rubyhacker]

I've noticed (via  my friend Taylor Carpenter) that IO#reopen
doesn't work for a StringIO object.

Should it?

I haven't dug into the source yet, but my guess is that it would
be easy to implement...


Hal Fulton

[Charles Oliver Nutter-4]

StringIO is not actually an IO object; it just mimics one. Unless
reopen worked with non-descriptor IO-like objects (as in JRuby), it
would not be possible to IO#reopen against a StringIO.

On Monday, November 2, 2009, Hal Fulton <rubyhacker@...> wrote:

> I've noticed (via  my friend Taylor Carpenter) that IO#reopen
> doesn't work for a StringIO object.
>
> Should it?
>
> I haven't dug into the source yet, but my guess is that it would
> be easy to implement...
>
>
> Hal Fulton
>

[rubyhacker]

By "Should it?" I mean "in the future" -- I'm not implying there's
a bug now.

That is, I'm aware of how it works now. I'm just questioning whether
it is worth changing.

I believe it should be possible in theory to make it work -- though
as I said, I haven't dig into the source.

Hal Fulton


On Mon, Nov 2, 2009 at 5:44 PM, Charles Oliver Nutter
<headius@...>wrote:

> StringIO is not actually an IO object; it just mimics one. Unless
> reopen worked with non-descriptor IO-like objects (as in JRuby), it
> would not be possible to IO#reopen against a StringIO.
>
> On Monday, November 2, 2009, Hal Fulton <rubyhacker@...> wrote:
> > I've noticed (via  my friend Taylor Carpenter) that IO#reopen
> > doesn't work for a StringIO object.
> >
> > Should it?
> >
> > I haven't dug into the source yet, but my guess is that it would
> > be easy to implement...
> >
> >
> > Hal Fulton
> >
>
>

[Robert Klemme-2]

2009/11/3 Hal Fulton <rubyhacker@...>:
> By "Should it?" I mean "in the future" -- I'm not implying there's
> a bug now.

I don't think so.

> That is, I'm aware of how it works now. I'm just questioning whether
> it is worth changing.
>
> I believe it should be possible in theory to make it work -- though
> as I said, I haven't dig into the source.

This will be difficult to do because then StringIO would somehow have
to change its type.  Of course you could achieve that with some kind
of proxy pattern but I doubt that this would be a good idea; we all
would always pay the price of the overhead (1 more object per IO /
StringIO) for a feature which seems rarely wanted (I cannot recall
having seen a request like this in years).

The main purpose of reopen is to be able to reuse file descriptors
because there are some well known file descriptors (mainly 0,1 and 2)
which you might want to redirect for a sub process.

11:20:25 tmp$ ruby -e '$stdout.reopen("x"); exec "date"'
11:20:47 tmp$ cat x
Tue Nov  3 11:20:47 WEST 2009
11:20:53 tmp$ ruby -e '$stdout.reopen($stderr); exec "date"' >out 2>err
11:22:45 tmp$ cat out
11:22:47 tmp$ cat err
Tue Nov  3 11:22:45 WEST 2009
11:22:48 tmp$

That does not make much sense for a StringIO which is not connected to
any file descriptor - so you cannot reuse a file descriptor.  So for
StringIO the natural solution would be to just overwrite the variable
with another IO and make sure that there are no aliased variables
(e.g. by always referring to the single variable via a method).

Kind regards

robert

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

[rubyhacker]

>
> This will be difficult to do because then StringIO would somehow have
> to change its type.  Of course you could achieve that with some kind
> of proxy pattern but I doubt that this would be a good idea; we all
> would always pay the price of the overhead (1 more object per IO /
> StringIO) for a feature which seems rarely wanted (I cannot recall
> having seen a request like this in years).
>


Well, reopen also can take a string, I believe. So it already takes
more than one type. But the overhead argument is valid.

[snip]


> So for
> StringIO the natural solution would be to just overwrite the variable
> with another IO and make sure that there are no aliased variables
> (e.g. by always referring to the single variable via a method).
>
>
I'm not sure I understand.

How would I, for example, capture all that went to standard error
in a StringIO object?

Hal

[Robert Klemme-2]

On 11/03/2009 11:32 PM, Hal Fulton wrote:
> [Note:  parts of this message were removed to make it a legal post.]

!

>> This will be difficult to do because then StringIO would somehow have
>> to change its type.  Of course you could achieve that with some kind
>> of proxy pattern but I doubt that this would be a good idea; we all
>> would always pay the price of the overhead (1 more object per IO /
>> StringIO) for a feature which seems rarely wanted (I cannot recall
>> having seen a request like this in years).

> Well, reopen also can take a string, I believe. So it already takes
> more than one type. But the overhead argument is valid.

The string is interpreted as a file name.  With changing type I meant
that a StringIO after #reopen (either with number, IO object or string
as file name) must behave like an IO or File object which is a different
type.  I was not talking about the argument types to method #reopen but
the type of the object itself.

>> So for
>> StringIO the natural solution would be to just overwrite the variable
>> with another IO and make sure that there are no aliased variables
>> (e.g. by always referring to the single variable via a method).
>>
>>
> I'm not sure I understand.

@x = StringIO.new "foo"

def io; @x; end

io.puts "foo"

io.puts "another line"

# later
@x = File.open "log.txt"

io.puts "this line goes to a completely different location"

> How would I, for example, capture all that went to standard error
> in a StringIO object?

You can try to do

$stderr = StringIO.new

but this works only within the same process, i.e. not for sub processes.
  There is no way I am aware of to redirect a file descriptor to an in
memory structure.  If you want the redirection to work for a child
process and get the data in the parent you need to use one of the popen
family of methods.

Kind regards

        robert

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

[rubyhacker]

>
>
>  This will be difficult to do because then StringIO would somehow have
>>> to change its type.  Of course you could achieve that with some kind
>>> of proxy pattern but I doubt that this would be a good idea; we all
>>> would always pay the price of the overhead (1 more object per IO /
>>> StringIO) for a feature which seems rarely wanted (I cannot recall
>>> having seen a request like this in years).
>>>
>>
>  Well, reopen also can take a string, I believe. So it already takes
>> more than one type. But the overhead argument is valid.
>>
>
> The string is interpreted as a file name.  With changing type I meant that
> a StringIO after #reopen (either with number, IO object or string as file
> name) must behave like an IO or File object which is a different type.  I
> was not talking about the argument types to method #reopen but the type of
> the object itself.
>
>

Yes - but what I meant was, reopen already checks its argument and
behaves differently based on what is passed in, so it *could* of course
check for a StringIO as well.


[snip]

Yes, this obviously works, but it is irrelevant to what I am trying
to do.

>  How would I, for example, capture all that went to standard error
>> in a StringIO object?
>>
>
> You can try to do
>
> $stderr = StringIO.new
>
> but this works only within the same process, i.e. not for sub processes.
>  There is no way I am aware of to redirect a file descriptor to an in memory
> structure.  If you want the redirection to work for a child process and get
> the data in the parent you need to use one of the popen family of methods.

I am not interested in subprocesses actually.

The problem with what you do here is that it changes the variable $stderr
to a different object (leaving the original object of course unaffected).

So it depends, I think, on code referencing $stderr rather than STDERR. (The
code whose stderr I am catching is not necessarily my own.)

Maybe I could also do const_set(STDERR, my_io) before any requires.  That is

something I have not tried.

Another ugly trick, of course, would be something like doing a popen, a
reopen,
then read from the pipe periodically. Or perhaps even redirect stderr to a
file
and do a File::Tail on that...

Ah, well. It is not something that will come up often.


Hal

[Caleb Clausen]

On 11/3/09, Robert Klemme <shortcutter@...> wrote:
> This will be difficult to do because then StringIO would somehow have
> to change its type.  Of course you could achieve that with some kind

IO subclasses can already change their type (and even class!) when
#reopen is called:

irb(main):001:0> f=File.open "endless.rb"
=> #<File:endless.rb>
irb(main):002:0> f.reopen STDIN
=> #<IO:0xb7d267cc>
irb(main):003:0> f
=> #<IO:0xb7d267cc>

Clearly StringIO is not a subclass of IO, but maybe this is why it should be?

[Robert Klemme-2]

On 04.11.2009 00:26, Hal Fulton wrote:

> [Note:  parts of this message were removed to make it a legal post.]
>
>>
>>  This will be difficult to do because then StringIO would somehow have
>>>> to change its type.  Of course you could achieve that with some kind
>>>> of proxy pattern but I doubt that this would be a good idea; we all
>>>> would always pay the price of the overhead (1 more object per IO /
>>>> StringIO) for a feature which seems rarely wanted (I cannot recall
>>>> having seen a request like this in years).
>>>>
>>  Well, reopen also can take a string, I believe. So it already takes
>>> more than one type. But the overhead argument is valid.
>>>
>> The string is interpreted as a file name.  With changing type I meant that
>> a StringIO after #reopen (either with number, IO object or string as file
>> name) must behave like an IO or File object which is a different type.  I
>> was not talking about the argument types to method #reopen but the type of
>> the object itself.
>>
> Yes - but what I meant was, reopen already checks its argument and
> behaves differently based on what is passed in, so it *could* of course
> check for a StringIO as well.

Oh, that wasn't clear to me from your posting.  I thought you were
referring to the receiver being a StringIO object.  Actually, when
rereading your original posting both interpretations seem valid...

> Yes, this obviously works, but it is irrelevant to what I am trying
> to do.

Well, _now_ I know. :-)

>>  How would I, for example, capture all that went to standard error
>>> in a StringIO object?
>>>
>> You can try to do
>>
>> $stderr = StringIO.new
>>
>> but this works only within the same process, i.e. not for sub processes.
>>  There is no way I am aware of to redirect a file descriptor to an in memory
>> structure.  If you want the redirection to work for a child process and get
>> the data in the parent you need to use one of the popen family of methods.
>
>
> I am not interested in subprocesses actually.
>
> The problem with what you do here is that it changes the variable $stderr
> to a different object (leaving the original object of course unaffected).
>
> So it depends, I think, on code referencing $stderr rather than STDERR. (The
> code whose stderr I am catching is not necessarily my own.)
>
> Maybe I could also do const_set(STDERR, my_io) before any requires.  That is
>
> something I have not tried.
>
> Another ugly trick, of course, would be something like doing a popen, a
> reopen,
> then read from the pipe periodically. Or perhaps even redirect stderr to a
> file
> and do a File::Tail on that...
>
> Ah, well. It is not something that will come up often.

So you are trying to catch output of Ruby code run within the same
process and of which you are not sure that it actually always references
$stderr during write operations.  You suspect it might store $stderr in
some local variable and thus be unaffected by your change to the global
variable.

I'd say the risk is rather low since it does not really make sense to
replace $stderr with something else (unless someone is a lazy typer).

The problem remains the same: StringIO and IO / File are two different
things and I believe there is no reasonable way to add functionality to
them that they can mutate.

The safest seems indeed to redirect the IO to a pipe, have a background
thread read from the pipe and store data in a String or StringIO and
then execute the code whose output you want to catch.  As a nice side
effect that would also work for subprocesses.

Kind regards

        robert

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

[Robert Klemme-2]

On 04.11.2009 02:05, Caleb Clausen wrote:

> On 11/3/09, Robert Klemme <shortcutter@...> wrote:
>> This will be difficult to do because then StringIO would somehow have
>> to change its type.  Of course you could achieve that with some kind
>
> IO subclasses can already change their type (and even class!) when
> #reopen is called:
>
> irb(main):001:0> f=File.open "endless.rb"
> => #<File:endless.rb>
> irb(main):002:0> f.reopen STDIN
> => #<IO:0xb7d267cc>
> irb(main):003:0> f
> => #<IO:0xb7d267cc>
>
> Clearly StringIO is not a subclass of IO, but maybe this is why it should be?

Amazing, I wasn't aware of that.  Not sure though whether a change from
IO / File to StringIO or reverse would be possible since IO and File are
much more similar...

Thanks for the update!

Kind regards

        robert

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

[rubyhacker]

>
> Yes - but what I meant was, reopen already checks its argument and
>> behaves differently based on what is passed in, so it *could* of course
>> check for a StringIO as well.
>>
>
> Oh, that wasn't clear to me from your posting.  I thought you were
> referring to the receiver being a StringIO object.  Actually, when rereading
> your original posting both interpretations seem valid...


Yes, I did not think of the other interpretation at first.

 Yes, this obviously works, but it is irrelevant to what I am trying
> to do.
>

Well, _now_ I know. :-)


Hehe... I did not specify my problem well enough.   ;)


>
> So you are trying to catch output of Ruby code run within the same process
> and of which you are not sure that it actually always references $stderr
> during write operations.  You suspect it might store $stderr in some local
> variable and thus be unaffected by your change to the global variable.
>
> I'd say the risk is rather low since it does not really make sense to
> replace $stderr with something else (unless someone is a lazy typer).
>

Maybe I am the only one who does this, but I habitually use STDERR
instead of $stderr.

> The problem remains the same: StringIO and IO / File are two different
> things and I believe there is no reasonable way to add functionality to them
> that they can mutate.
>
> The safest seems indeed to redirect the IO to a pipe, have a background
> thread read from the pipe and store data in a String or StringIO and then
> execute the code whose output you want to catch.  As a nice side effect that
> would also work for subprocesses.
>
>

Thanks for discussion!

Hal

>
> Kind regards
>
>        robert
>
> --
> remember.guy do |as, often| as.you_can - without end
> http://blog.rubybestpractices.com/
>
>