Source.fromFile(file).getLines

[Gilles SCOUVART]

Hello all,

 

I was surprised to notice that using Source.fromFile(file).getLines on a file with empty lines, those were added to the previous non empty line.

I.e. if the file contains this:

------

test

 

 

------

then getLines.next would return "test\n\n" (written java-echap-style).

I expected it would iterate on three lines instead: "test", "", and "".

This is a little bit annoying for pattern matching on full lines...

 

Is this expected behaviour?

 

Gilles Scouvart.

[Ricky Clarkson]

I'd guess not.  You might want to use java.util.Scanner until this is fixed.

2009/7/9 Gilles SCOUVART <Gilles.SCOUVART@...>:

> Hello all,
>
> I was surprised to notice that using Source.fromFile(file).getLines on a
> file with empty lines, those were added to the previous non empty line.
> I.e. if the file contains this:
> ------
> test
>
>
> ------
> then getLines.next would return "test\n\n" (written java-echap-style).
> I expected it would iterate on three lines instead: "test", "", and "".
> This is a little bit annoying for pattern matching on full lines...
>
> Is this expected behaviour?
>
> Gilles Scouvart.

--
Ricky Clarkson
Java Programmer, AD Holdings
[phone]
Skype: ricky_clarkson
Google Talk: ricky.clarkson@...

[Daniel Sobral]

No, not expected. I don't get that either. Try doing map(_ map(_ toInt)) to the result of getLines. Maybe you are getting "\r\n" and confusing it with "\n\n".

On Thu, Jul 9, 2009 at 1:38 PM, Gilles SCOUVART <Gilles.SCOUVART@...> wrote:

Hello all,

 

I was surprised to notice that using Source.fromFile(file).getLines on a file with empty lines, those were added to the previous non empty line.

I.e. if the file contains this:

------

test

 

 

------

then getLines.next would return "test\n\n" (written java-echap-style).

I expected it would iterate on three lines instead: "test", "", and "".

This is a little bit annoying for pattern matching on full lines...

 

Is this expected behaviour?

 

Gilles Scouvart.

--
Daniel C. Sobral

Something I learned in academia: there are three kinds of academic reviews: review by name, review by reference and review by value.

[Ricky Clarkson]

ricky@ricky-desktop:~$ cat test.scala
import scala.io.Source
import java.io.File

object Main {
        def main(args: Array[String]) {
                for (line <- Source.fromFile(new File("test.scala")).getLines)
                        print("("+line+")")
        }
}


ricky@ricky-desktop:~$ scalac test.scala
ricky@ricky-desktop:~$ scala Main
(import scala.io.Source
)(import java.io.File
)(
)(object Main {
)(      def main(args: Array[String]) {
)(              for (line <- Source.fromFile(new File("test.scala")).getLines)
)(                      print("("+line+")")
)(      }
)(}
)(
)(
)

Note that I used print instead of println.  I can't imagine it's a
good thing to have line contain a line terminator of any kind.

2009/7/9 Daniel Sobral <dcsobral@...>:

> No, not expected. I don't get that either. Try doing map(_ map(_ toInt)) to
> the result of getLines. Maybe you are getting "\r\n" and confusing it with
> "\n\n".
>
> On Thu, Jul 9, 2009 at 1:38 PM, Gilles SCOUVART <Gilles.SCOUVART@...>
> wrote:
>>
>> Hello all,
>>
>> I was surprised to notice that using Source.fromFile(file).getLines on a
>> file with empty lines, those were added to the previous non empty line.
>> I.e. if the file contains this:
>> ------
>> test
>>
>>
>> ------
>> then getLines.next would return "test\n\n" (written java-echap-style).
>> I expected it would iterate on three lines instead: "test", "", and "".
>> This is a little bit annoying for pattern matching on full lines...
>>
>> Is this expected behaviour?
>>
>> Gilles Scouvart.
>
>
> --
> Daniel C. Sobral
>
> Something I learned in academia: there are three kinds of academic reviews:
> review by name, review by reference and review by value.
>

--
Ricky Clarkson
Java Programmer, AD Holdings
[phone]
Skype: ricky_clarkson
Google Talk: ricky.clarkson@...

[Daniel Sobral]

How would you tell apart a line that ends in "\n" with one that doesn't? The method getLines doesn't strip away any characters present in the file. Which, by the way, is pretty standard behavior. In Perl, for instance, you need to "chomp" away newlines if you don't want them. Likewise, in Scala, you need to stripLineEnd newlines if you don't want them.

On Thu, Jul 9, 2009 at 2:07 PM, Ricky Clarkson <ricky.clarkson@...> wrote:

ricky@ricky-desktop:~$ cat test.scala
import scala.io.Source
import java.io.File

object Main {
       def main(args: Array[String]) {
               for (line <- Source.fromFile(new File("test.scala")).getLines)
                       print("("+line+")")
       }
}


ricky@ricky-desktop:~$ scalac test.scala
ricky@ricky-desktop:~$ scala Main
(import scala.io.Source
)(import java.io.File
)(
)(object Main {
)(      def main(args: Array[String]) {
)(              for (line <- Source.fromFile(new File("test.scala")).getLines)
)(                      print("("+line+")")
)(      }
)(}
)(
)(
)

Note that I used print instead of println.  I can't imagine it's a
good thing to have line contain a line terminator of any kind.

2009/7/9 Daniel Sobral <dcsobral@...>:

> No, not expected. I don't get that either. Try doing map(_ map(_ toInt)) to
> the result of getLines. Maybe you are getting "\r\n" and confusing it with
> "\n\n".
>
> On Thu, Jul 9, 2009 at 1:38 PM, Gilles SCOUVART <Gilles.SCOUVART@...>
> wrote:
>>
>> Hello all,
>>
>> I was surprised to notice that using Source.fromFile(file).getLines on a
>> file with empty lines, those were added to the previous non empty line.
>> I.e. if the file contains this:
>> ------
>> test
>>
>>
>> ------
>> then getLines.next would return "test\n\n" (written java-echap-style).
>> I expected it would iterate on three lines instead: "test", "", and "".
>> This is a little bit annoying for pattern matching on full lines...
>>
>> Is this expected behaviour?
>>
>> Gilles Scouvart.
>
>
> --
> Daniel C. Sobral
>
> Something I learned in academia: there are three kinds of academic reviews:
> review by name, review by reference and review by value.
>

--
Ricky Clarkson
Java Programmer, AD Holdings
[phone]
Skype: ricky_clarkson
Google Talk: ricky.clarkson@...

--
Daniel C. Sobral

Something I learned in academia: there are three kinds of academic reviews: review by name, review by reference and review by value.

[Josh Suereth]

We're actively working on an I/O layer for scala in the Scalax project.  The 2.8.0 branch (on github) contains a method for handling different types of line endings.

You should see these test files:
http://github.com/eengbrec/Scalax.IO/blob/7dd08918223e7044bdd218fa075196160ae67b28/src/test/scala/scalax/io/TestStreams.scala
http://github.com/eengbrec/Scalax.IO/blob/7dd08918223e7044bdd218fa075196160ae67b28/src/test/scala/scalax/io/TestManagedStreams.scala

For an Idea on usage.   You should be able to use the File abstraction to pull in lines/chars/bytes/objects.

We're in the process of development now, but any feedback on the approach would be greatly appreciated!  We hope to have releasable code with the scala 2.8.0 release.


- Josh

On Thu, Jul 9, 2009 at 1:55 PM, Daniel Sobral <dcsobral@...> wrote:

How would you tell apart a line that ends in "\n" with one that doesn't? The method getLines doesn't strip away any characters present in the file. Which, by the way, is pretty standard behavior. In Perl, for instance, you need to "chomp" away newlines if you don't want them. Likewise, in Scala, you need to stripLineEnd newlines if you don't want them.

On Thu, Jul 9, 2009 at 2:07 PM, Ricky Clarkson <ricky.clarkson@...> wrote:

ricky@ricky-desktop:~$ cat test.scala
import scala.io.Source
import java.io.File

object Main {
       def main(args: Array[String]) {
               for (line <- Source.fromFile(new File("test.scala")).getLines)
                       print("("+line+")")
       }
}


ricky@ricky-desktop:~$ scalac test.scala
ricky@ricky-desktop:~$ scala Main
(import scala.io.Source
)(import java.io.File
)(
)(object Main {
)(      def main(args: Array[String]) {
)(              for (line <- Source.fromFile(new File("test.scala")).getLines)
)(                      print("("+line+")")
)(      }
)(}
)(
)(
)

Note that I used print instead of println.  I can't imagine it's a
good thing to have line contain a line terminator of any kind.

2009/7/9 Daniel Sobral <dcsobral@...>:

> No, not expected. I don't get that either. Try doing map(_ map(_ toInt)) to
> the result of getLines. Maybe you are getting "\r\n" and confusing it with
> "\n\n".
>
> On Thu, Jul 9, 2009 at 1:38 PM, Gilles SCOUVART <Gilles.SCOUVART@...>
> wrote:
>>
>> Hello all,
>>
>> I was surprised to notice that using Source.fromFile(file).getLines on a
>> file with empty lines, those were added to the previous non empty line.
>> I.e. if the file contains this:
>> ------
>> test
>>
>>
>> ------
>> then getLines.next would return "test\n\n" (written java-echap-style).
>> I expected it would iterate on three lines instead: "test", "", and "".
>> This is a little bit annoying for pattern matching on full lines...
>>
>> Is this expected behaviour?
>>
>> Gilles Scouvart.
>
>
> --
> Daniel C. Sobral
>
> Something I learned in academia: there are three kinds of academic reviews:
> review by name, review by reference and review by value.
>

--
Ricky Clarkson
Java Programmer, AD Holdings
[phone]
Skype: ricky_clarkson
Google Talk: ricky.clarkson@...

--

Daniel C. Sobral

Something I learned in academia: there are three kinds of academic reviews: review by name, review by reference and review by value.

[Kieron.Wilkinson]

Daniel Sobral <dcsobral@...> wrote on 09/07/2009 18:55:47:
> How would you tell apart a line that ends in "\n" with one that
> doesn't? The method getLines doesn't strip away any characters
> present in the file. Which, by the way, is pretty standard behavior.
> In Perl, for instance, you need to "chomp" away newlines if you
> don't want them. Likewise, in Scala, you need to stripLineEnd
> newlines if you don't want them.

I thought this behaviour was odd because I couldn't think of any use-cases
of why you would actually be interested in the line endings. Though I
suppose that if you were, it is at least possible to remove them, whereas
getting whatever the original ones were back wouldn't be...

Kieron


This message may contain confidential and privileged information and is intended solely for the use of the named addressee. Access, copying or re-use of the e-mail or any information contained therein by any other person is not authorised. If you are not the intended recipient please notify us immediately by returning the e-mail to the originator and then immediately delete this message. Although we attempt to sweep e-mail and attachments for viruses, we do not guarantee that either are virus-free and accept no liability for any damage sustained as a result of viruses.

Please refer to http://www.bnymellon.com/disclaimer/piml.html for certain disclosures.

[Ricky Clarkson]

Java strips them away in Scanner and BufferedReader.  Forgetting chomp
(or using chop instead) is one of the most common newbie mistakes in
Perl, and I don't think that's worth propagating to Scala.  Out of
interest, if it's pretty standard, what else does it that way?

2009/7/9 Daniel Sobral <dcsobral@...>:

> How would you tell apart a line that ends in "\n" with one that doesn't? The
> method getLines doesn't strip away any characters present in the file.
> Which, by the way, is pretty standard behavior. In Perl, for instance, you
> need to "chomp" away newlines if you don't want them. Likewise, in Scala,
> you need to stripLineEnd newlines if you don't want them.
>
> On Thu, Jul 9, 2009 at 2:07 PM, Ricky Clarkson <ricky.clarkson@...>
> wrote:
>>
>> ricky@ricky-desktop:~$ cat test.scala
>> import scala.io.Source
>> import java.io.File
>>
>> object Main {
>>        def main(args: Array[String]) {
>>                for (line <- Source.fromFile(new
>> File("test.scala")).getLines)
>>                        print("("+line+")")
>>        }
>> }
>>
>>
>> ricky@ricky-desktop:~$ scalac test.scala
>> ricky@ricky-desktop:~$ scala Main
>> (import scala.io.Source
>> )(import java.io.File
>> )(
>> )(object Main {
>> )(      def main(args: Array[String]) {
>> )(              for (line <- Source.fromFile(new
>> File("test.scala")).getLines)
>> )(                      print("("+line+")")
>> )(      }
>> )(}
>> )(
>> )(
>> )
>>
>> Note that I used print instead of println.  I can't imagine it's a
>> good thing to have line contain a line terminator of any kind.
>>
>> 2009/7/9 Daniel Sobral <dcsobral@...>:
>> > No, not expected. I don't get that either. Try doing map(_ map(_
>> > toInt)) to
>> > the result of getLines. Maybe you are getting "\r\n" and confusing it
>> > with
>> > "\n\n".
>> >
>> > On Thu, Jul 9, 2009 at 1:38 PM, Gilles SCOUVART
>> > <Gilles.SCOUVART@...>
>> > wrote:
>> >>
>> >> Hello all,
>> >>
>> >> I was surprised to notice that using Source.fromFile(file).getLines on
>> >> a
>> >> file with empty lines, those were added to the previous non empty line.
>> >> I.e. if the file contains this:
>> >> ------
>> >> test
>> >>
>> >>
>> >> ------
>> >> then getLines.next would return "test\n\n" (written java-echap-style).
>> >> I expected it would iterate on three lines instead: "test", "", and "".
>> >> This is a little bit annoying for pattern matching on full lines...
>> >>
>> >> Is this expected behaviour?
>> >>
>> >> Gilles Scouvart.
>> >
>> >
>> > --
>> > Daniel C. Sobral
>> >
>> > Something I learned in academia: there are three kinds of academic
>> > reviews:
>> > review by name, review by reference and review by value.
>> >
>>
>>
>>
>> --
>> Ricky Clarkson
>> Java Programmer, AD Holdings
>> [phone]
>> Skype: ricky_clarkson
>> Google Talk: ricky.clarkson@...
>
>
>
> --
> Daniel C. Sobral
>
> Something I learned in academia: there are three kinds of academic reviews:
> review by name, review by reference and review by value.
>

--
Ricky Clarkson
Java Programmer, AD Holdings
[phone]
Skype: ricky_clarkson
Google Talk: ricky.clarkson@...

[Detering Dirk-2]

Sorry, I'm feeling a bit confused now ...

I thought that "\n" is a "line terminator", (at least the
one on *nix based systems).

So I would translate your sentence

> How would you tell apart a line that ends in "\n" with one
> that doesn't?

Into "How would you tell apart a line that ends in a line end
with one that doesn't?"

I now ask: How would tell that a stream of characters is a "line"
if not by the line end marker? (Beside eof/eos of course).

> The method getLines doesn't strip away any characters present in
> the file. Which, by the way, is pretty  standard behavior.

So far ...

> In Perl, for instance, you need to "chomp"
> away newlines if you don't want them. Likewise, in Scala, you
> need to stripLineEnd newlines if you don't want them.

... so bad. Think of the regex pattern match problem discussed earlier
here. (You have to strip the line before matching with "^some$" or
use "(?s)^some$").

From a domain progger's view:

When reading a CSV file into excel I get the data into cells,
without the cell separator (the C in CSV).

So when calling getLine() I would expect to get the "line" without
the technical EOL detail. This may be added again if I write
"line" out to some file with write[Line|ln] or print[Line|ln],
like I get the comma again when exporting the data as CSV.

Same is true for every envelope data on underlying transport
layers. They are only present on that layer.

Looking at a stream from a char perspective I would
expect to see it.
Looking at it from a line perspective, it makes no sense to me.

Just my 2ct.

>
>
> On Thu, Jul 9, 2009 at 2:07 PM, Ricky Clarkson
> <ricky.clarkson@...> wrote:
>
>
> ricky@ricky-desktop:~$ cat test.scala
> import scala.io.Source
> import java.io.File
>
> object Main {
>       def main(args: Array[String]) {
>               for (line <- Source.fromFile(new
> File("test.scala")).getLines)
>                       print("("+line+")")
>       }
> }
>
>
> ricky@ricky-desktop:~$ scalac test.scala
> ricky@ricky-desktop:~$ scala Main
> (import scala.io.Source
> )(import java.io.File
> )(
> )(object Main {
> )(      def main(args: Array[String]) {
> )(              for (line <- Source.fromFile(new
> File("test.scala")).getLines)
> )(                      print("("+line+")")
> )(      }
> )(}
> )(
> )(
> )
>
> Note that I used print instead of println.  I can't
> imagine it's a
> good thing to have line contain a line terminator of any kind.
>
> 2009/7/9 Daniel Sobral <dcsobral@...>:
>
> > No, not expected. I don't get that either. Try doing
> map(_ map(_ toInt)) to
> > the result of getLines. Maybe you are getting "\r\n"
> and confusing it with
> > "\n\n".
> >
> > On Thu, Jul 9, 2009 at 1:38 PM, Gilles SCOUVART
> <Gilles.SCOUVART@...>
> > wrote:
> >>
> >> Hello all,
> >>
> >> I was surprised to notice that using
> Source.fromFile(file).getLines on a
> >> file with empty lines, those were added to the
> previous non empty line.
> >> I.e. if the file contains this:
> >> ------
> >> test
> >>
> >>
> >> ------
> >> then getLines.next would return "test\n\n" (written
> java-echap-style).
> >> I expected it would iterate on three lines instead:
> "test", "", and "".
> >> This is a little bit annoying for pattern matching
> on full lines...
> >>
> >> Is this expected behaviour?
> >>
> >> Gilles Scouvart.
> >
> >
> > --
> > Daniel C. Sobral
> >
> > Something I learned in academia: there are three
> kinds of academic reviews:
> > review by name, review by reference and review by value.
> >
>
>
>
>
> --
> Ricky Clarkson
> Java Programmer, AD Holdings
> [phone]
> Skype: ricky_clarkson
> Google Talk: ricky.clarkson@...
>
>
>
>
>
> --
> Daniel C. Sobral
>
> Something I learned in academia: there are three kinds of
> academic reviews: review by name, review by reference and
> review by value.
>
>

[Josh Suereth]

Line endings are different on different platforms.  Either '\n', "\r\n" or just '\r'.  Java's BufferedReader.getLine actually parses all three without distinguishing between them.  I think Scala's Source (last I checked) only supports 'nix based files (and dos as they have a '\n' after a '\r').

As I mentioned before, we're trying something in the Scalax.IO-2.8.0 branch to allow specifying line endings, but using default parameters to "handle the most usual case"

- Josh

On Fri, Jul 10, 2009 at 5:21 AM, Detering Dirk <Dirk.Detering@...> wrote:

Sorry, I'm feeling a bit confused now ...

I thought that "\n" is a "line terminator", (at least the
one on *nix based systems).

So I would translate your sentence

> How would you tell apart a line that ends in "\n" with one
> that doesn't?

Into "How would you tell apart a line that ends in a line end

with one that doesn't?"

I now ask: How would tell that a stream of characters is a "line"
if not by the line end marker? (Beside eof/eos of course).

> The method getLines doesn't strip away any characters present in
> the file. Which, by the way, is pretty  standard behavior.

So far ...

> In Perl, for instance, you need to "chomp"
> away newlines if you don't want them. Likewise, in Scala, you
> need to stripLineEnd newlines if you don't want them.

... so bad. Think of the regex pattern match problem discussed earlier
here. (You have to strip the line before matching with "^some$" or
use "(?s)^some$").

From a domain progger's view:

When reading a CSV file into excel I get the data into cells,
without the cell separator (the C in CSV).

So when calling getLine() I would expect to get the "line" without
the technical EOL detail. This may be added again if I write
"line" out to some file with write[Line|ln] or print[Line|ln],
like I get the comma again when exporting the data as CSV.

Same is true for every envelope data on underlying transport
layers. They are only present on that layer.

Looking at a stream from a char perspective I would
expect to see it.
Looking at it from a line perspective, it makes no sense to me.

Just my 2ct.

>
>
> On Thu, Jul 9, 2009 at 2:07 PM, Ricky Clarkson
> <ricky.clarkson@...> wrote:
>
>
>       ricky@ricky-desktop:~$ cat test.scala
>       import scala.io.Source
>       import java.io.File
>
>       object Main {
>              def main(args: Array[String]) {
>                      for (line <- Source.fromFile(new
> File("test.scala")).getLines)
>                              print("("+line+")")
>              }
>       }
>
>
>       ricky@ricky-desktop:~$ scalac test.scala
>       ricky@ricky-desktop:~$ scala Main
>       (import scala.io.Source
>       )(import java.io.File
>       )(
>       )(object Main {
>       )(      def main(args: Array[String]) {
>       )(              for (line <- Source.fromFile(new
> File("test.scala")).getLines)
>       )(                      print("("+line+")")
>       )(      }
>       )(}
>       )(
>       )(
>       )
>
>       Note that I used print instead of println.  I can't
> imagine it's a
>       good thing to have line contain a line terminator of any kind.
>
>       2009/7/9 Daniel Sobral <dcsobral@...>:
>
>       > No, not expected. I don't get that either. Try doing
> map(_ map(_ toInt)) to
>       > the result of getLines. Maybe you are getting "\r\n"
> and confusing it with
>       > "\n\n".
>       >
>       > On Thu, Jul 9, 2009 at 1:38 PM, Gilles SCOUVART
> <Gilles.SCOUVART@...>
>       > wrote:
>       >>
>       >> Hello all,
>       >>
>       >> I was surprised to notice that using
> Source.fromFile(file).getLines on a
>       >> file with empty lines, those were added to the
> previous non empty line.
>       >> I.e. if the file contains this:
>       >> ------
>       >> test
>       >>
>       >>
>       >> ------
>       >> then getLines.next would return "test\n\n" (written
> java-echap-style).
>       >> I expected it would iterate on three lines instead:
> "test", "", and "".
>       >> This is a little bit annoying for pattern matching
> on full lines...
>       >>
>       >> Is this expected behaviour?
>       >>
>       >> Gilles Scouvart.
>       >
>       >
>       > --
>       > Daniel C. Sobral
>       >
>       > Something I learned in academia: there are three
> kinds of academic reviews:
>       > review by name, review by reference and review by value.
>       >
>
>
>
>
>       --
>       Ricky Clarkson
>       Java Programmer, AD Holdings
>       [phone]
>       Skype: ricky_clarkson
>       Google Talk: ricky.clarkson@...
>
>
>
>
>
> --
> Daniel C. Sobral
>
> Something I learned in academia: there are three kinds of
> academic reviews: review by name, review by reference and
> review by value.
>
>

[Detering Dirk-2]

Thanks Josh,

> As I mentioned before, we're trying something in the
> Scalax.IO-2.8.0 branch to allow specifying line endings, but
> using default parameters to "handle the most usual case"

BTW: I felt annoyed by Java's File notion and thought about
an alternative based on DirEntry, Dir and File.

Did a quick look into the repos and was glad to find
Location, Directory and File there, which seems to fit
into the same paradigm.

KR
Det

>
> - Josh
>
>
>
>
> On Fri, Jul 10, 2009 at 5:21 AM, Detering Dirk
> <Dirk.Detering@...> wrote:
>
>
> Sorry, I'm feeling a bit confused now ...
>
> I thought that "\n" is a "line terminator", (at least the
> one on *nix based systems).
>
> So I would translate your sentence
>
>
> > How would you tell apart a line that ends in "\n" with one
> > that doesn't?
>
>
> Into "How would you tell apart a line that ends in a line end
>
> with one that doesn't?"
>
>
> I now ask: How would tell that a stream of characters
> is a "line"
> if not by the line end marker? (Beside eof/eos of course).
>
>
> > The method getLines doesn't strip away any characters
> present in
> > the file. Which, by the way, is pretty  standard behavior.
>
>
> So far ...
>
>
> > In Perl, for instance, you need to "chomp"
> > away newlines if you don't want them. Likewise, in Scala, you
> > need to stripLineEnd newlines if you don't want them.
>
>
> ... so bad. Think of the regex pattern match problem
> discussed earlier
> here. (You have to strip the line before matching with
> "^some$" or
> use "(?s)^some$").
>
> From a domain progger's view:
>
> When reading a CSV file into excel I get the data into cells,
> without the cell separator (the C in CSV).
>
> So when calling getLine() I would expect to get the
> "line" without
> the technical EOL detail. This may be added again if I write
> "line" out to some file with write[Line|ln] or print[Line|ln],
> like I get the comma again when exporting the data as CSV.
>
> Same is true for every envelope data on underlying transport
> layers. They are only present on that layer.
>
> Looking at a stream from a char perspective I would
> expect to see it.
> Looking at it from a line perspective, it makes no sense to me.
>
> Just my 2ct.
>
>
> >
> >
> > On Thu, Jul 9, 2009 at 2:07 PM, Ricky Clarkson
> > <ricky.clarkson@...> wrote:
> >
> >
> >       ricky@ricky-desktop:~$ cat test.scala
> >       import scala.io.Source
> >       import java.io.File
> >
> >       object Main {
> >              def main(args: Array[String]) {
> >                      for (line <- Source.fromFile(new
> > File("test.scala")).getLines)
> >                              print("("+line+")")
> >              }
> >       }
> >
> >
> >       ricky@ricky-desktop:~$ scalac test.scala
> >       ricky@ricky-desktop:~$ scala Main
> >       (import scala.io.Source
> >       )(import java.io.File
> >       )(
> >       )(object Main {
> >       )(      def main(args: Array[String]) {
> >       )(              for (line <- Source.fromFile(new
> > File("test.scala")).getLines)
> >       )(                      print("("+line+")")
> >       )(      }
> >       )(}
> >       )(
> >       )(
> >       )
> >
> >       Note that I used print instead of println.  I can't
> > imagine it's a
> >       good thing to have line contain a line
> terminator of any kind.
> >
> >       2009/7/9 Daniel Sobral <dcsobral@...>:
> >
> >       > No, not expected. I don't get that either. Try doing
> > map(_ map(_ toInt)) to
> >       > the result of getLines. Maybe you are getting "\r\n"
> > and confusing it with
> >       > "\n\n".
> >       >
> >       > On Thu, Jul 9, 2009 at 1:38 PM, Gilles SCOUVART
> > <Gilles.SCOUVART@...>
> >       > wrote:
> >       >>
> >       >> Hello all,
> >       >>
> >       >> I was surprised to notice that using
> > Source.fromFile(file).getLines on a
> >       >> file with empty lines, those were added to the
> > previous non empty line.
> >       >> I.e. if the file contains this:
> >       >> ------
> >       >> test
> >       >>
> >       >>
> >       >> ------
> >       >> then getLines.next would return "test\n\n" (written
> > java-echap-style).
> >       >> I expected it would iterate on three lines instead:
> > "test", "", and "".
> >       >> This is a little bit annoying for pattern matching
> > on full lines...
> >       >>
> >       >> Is this expected behaviour?
> >       >>
> >       >> Gilles Scouvart.
> >       >
> >       >
> >       > --
> >       > Daniel C. Sobral
> >       >
> >       > Something I learned in academia: there are three
> > kinds of academic reviews:
> >       > review by name, review by reference and
> review by value.
> >       >
> >
> >
> >
> >
> >       --
> >       Ricky Clarkson
> >       Java Programmer, AD Holdings
> >       [phone]
> >       Skype: ricky_clarkson
> >       Google Talk: ricky.clarkson@...
> >
> >
> >
> >
> >
> > --
> > Daniel C. Sobral
> >
> > Something I learned in academia: there are three kinds of
> > academic reviews: review by name, review by reference and
> > review by value.
> >
> >
>
>
>
>

[Josh Suereth]

Erik has been focusing on that and doing a great job.  Any other  
feedback is highly appreciated

Thanks
Sent from my iPhone

On Jul 10, 2009, at 8:13 AM, "Detering Dirk"  
<Dirk.Detering@...> wrote:

> Thanks Josh,
>
>> As I mentioned before, we're trying something in the
>> Scalax.IO-2.8.0 branch to allow specifying line endings, but
>> using default parameters to "handle the most usual case"
>
> BTW: I felt annoyed by Java's File notion and thought about
> an alternative based on DirEntry, Dir and File.
>
> Did a quick look into the repos and was glad to find
> Location, Directory and File there, which seems to fit
> into the same paradigm.
>
> KR
> Det
>
>
>>
>> - Josh
>>
>>
>>
>>
>> On Fri, Jul 10, 2009 at 5:21 AM, Detering Dirk
>> <Dirk.Detering@...> wrote:
>>
>>
>>    Sorry, I'm feeling a bit confused now ...
>>
>>    I thought that "\n" is a "line terminator", (at least the
>>    one on *nix based systems).
>>
>>    So I would translate your sentence
>>
>>
>>    > How would you tell apart a line that ends in "\n" with one
>>    > that doesn't?
>>
>>
>>    Into "How would you tell apart a line that ends in a line end
>>
>>    with one that doesn't?"
>>
>>
>>    I now ask: How would tell that a stream of characters
>> is a "line"
>>    if not by the line end marker? (Beside eof/eos of course).
>>
>>
>>    > The method getLines doesn't strip away any characters
>> present in
>>    > the file. Which, by the way, is pretty  standard behavior.
>>
>>
>>    So far ...
>>
>>
>>    > In Perl, for instance, you need to "chomp"
>>    > away newlines if you don't want them. Likewise, in Scala, you
>>    > need to stripLineEnd newlines if you don't want them.
>>
>>
>>    ... so bad. Think of the regex pattern match problem
>> discussed earlier
>>    here. (You have to strip the line before matching with
>> "^some$" or
>>    use "(?s)^some$").
>>
>>    From a domain progger's view:
>>
>>    When reading a CSV file into excel I get the data into cells,
>>    without the cell separator (the C in CSV).
>>
>>    So when calling getLine() I would expect to get the
>> "line" without
>>    the technical EOL detail. This may be added again if I write
>>    "line" out to some file with write[Line|ln] or print[Line|ln],
>>    like I get the comma again when exporting the data as CSV.
>>
>>    Same is true for every envelope data on underlying transport
>>    layers. They are only present on that layer.
>>
>>    Looking at a stream from a char perspective I would
>>    expect to see it.
>>    Looking at it from a line perspective, it makes no sense to me.
>>
>>    Just my 2ct.
>>
>>
>>    >
>>    >
>>    > On Thu, Jul 9, 2009 at 2:07 PM, Ricky Clarkson
>>    > <ricky.clarkson@...> wrote:
>>    >
>>    >
>>    >       ricky@ricky-desktop:~$ cat test.scala
>>    >       import scala.io.Source
>>    >       import java.io.File
>>    >
>>    >       object Main {
>>    >              def main(args: Array[String]) {
>>    >                      for (line <- Source.fromFile(new
>>    > File("test.scala")).getLines)
>>    >                              print("("+line+")")
>>    >              }
>>    >       }
>>    >
>>    >
>>    >       ricky@ricky-desktop:~$ scalac test.scala
>>    >       ricky@ricky-desktop:~$ scala Main
>>    >       (import scala.io.Source
>>    >       )(import java.io.File
>>    >       )(
>>    >       )(object Main {
>>    >       )(      def main(args: Array[String]) {
>>    >       )(              for (line <- Source.fromFile(new
>>    > File("test.scala")).getLines)
>>    >       )(                      print("("+line+")")
>>    >       )(      }
>>    >       )(}
>>    >       )(
>>    >       )(
>>    >       )
>>    >
>>    >       Note that I used print instead of println.  I can't
>>    > imagine it's a
>>    >       good thing to have line contain a line
>> terminator of any kind.
>>    >
>>    >       2009/7/9 Daniel Sobral <dcsobral@...>:
>>    >
>>    >       > No, not expected. I don't get that either. Try doing
>>    > map(_ map(_ toInt)) to
>>    >       > the result of getLines. Maybe you are getting "\r\n"
>>    > and confusing it with
>>    >       > "\n\n".
>>    >       >
>>    >       > On Thu, Jul 9, 2009 at 1:38 PM, Gilles SCOUVART
>>    > <Gilles.SCOUVART@...>
>>    >       > wrote:
>>    >       >>
>>    >       >> Hello all,
>>    >       >>
>>    >       >> I was surprised to notice that using
>>    > Source.fromFile(file).getLines on a
>>    >       >> file with empty lines, those were added to the
>>    > previous non empty line.
>>    >       >> I.e. if the file contains this:
>>    >       >> ------
>>    >       >> test
>>    >       >>
>>    >       >>
>>    >       >> ------
>>    >       >> then getLines.next would return "test\n\n" (written
>>    > java-echap-style).
>>    >       >> I expected it would iterate on three lines instead:
>>    > "test", "", and "".
>>    >       >> This is a little bit annoying for pattern matching
>>    > on full lines...
>>    >       >>
>>    >       >> Is this expected behaviour?
>>    >       >>
>>    >       >> Gilles Scouvart.
>>    >       >
>>    >       >
>>    >       > --
>>    >       > Daniel C. Sobral
>>    >       >
>>    >       > Something I learned in academia: there are three
>>    > kinds of academic reviews:
>>    >       > review by name, review by reference and
>> review by value.
>>    >       >
>>    >
>>    >
>>    >
>>    >
>>    >       --
>>    >       Ricky Clarkson
>>    >       Java Programmer, AD Holdings
>>    >       [phone]
>>    >       Skype: ricky_clarkson
>>    >       Google Talk: ricky.clarkson@...
>>    >
>>    >
>>    >
>>    >
>>    >
>>    > --
>>    > Daniel C. Sobral
>>    >
>>    > Something I learned in academia: there are three kinds of
>>    > academic reviews: review by name, review by reference and
>>    > review by value.
>>    >
>>    >
>>
>>
>>
>>

[Daniel Sobral]

Well, I'm not going to defend preserving the new line characters. I just don't think it surprising (as opposed to what Scala does with regex, which I do think is very suprising).

I'd be perfectly happy with a getLines that stripped the new line characters.

On Fri, Jul 10, 2009 at 6:21 AM, Detering Dirk <Dirk.Detering@...> wrote:

Sorry, I'm feeling a bit confused now ...

I thought that "\n" is a "line terminator", (at least the
one on *nix based systems).

So I would translate your sentence

> How would you tell apart a line that ends in "\n" with one
> that doesn't?

Into "How would you tell apart a line that ends in a line end

with one that doesn't?"

I now ask: How would tell that a stream of characters is a "line"
if not by the line end marker? (Beside eof/eos of course).

> The method getLines doesn't strip away any characters present in
> the file. Which, by the way, is pretty  standard behavior.

So far ...

> In Perl, for instance, you need to "chomp"
> away newlines if you don't want them. Likewise, in Scala, you
> need to stripLineEnd newlines if you don't want them.

... so bad. Think of the regex pattern match problem discussed earlier
here. (You have to strip the line before matching with "^some$" or
use "(?s)^some$").

From a domain progger's view:

When reading a CSV file into excel I get the data into cells,
without the cell separator (the C in CSV).

So when calling getLine() I would expect to get the "line" without
the technical EOL detail. This may be added again if I write
"line" out to some file with write[Line|ln] or print[Line|ln],
like I get the comma again when exporting the data as CSV.

Same is true for every envelope data on underlying transport
layers. They are only present on that layer.

Looking at a stream from a char perspective I would
expect to see it.
Looking at it from a line perspective, it makes no sense to me.

Just my 2ct.

>
>
> On Thu, Jul 9, 2009 at 2:07 PM, Ricky Clarkson
> <ricky.clarkson@...> wrote:
>
>
>       ricky@ricky-desktop:~$ cat test.scala
>       import scala.io.Source
>       import java.io.File
>
>       object Main {
>              def main(args: Array[String]) {
>                      for (line <- Source.fromFile(new
> File("test.scala")).getLines)
>                              print("("+line+")")
>              }
>       }
>
>
>       ricky@ricky-desktop:~$ scalac test.scala
>       ricky@ricky-desktop:~$ scala Main
>       (import scala.io.Source
>       )(import java.io.File
>       )(
>       )(object Main {
>       )(      def main(args: Array[String]) {
>       )(              for (line <- Source.fromFile(new
> File("test.scala")).getLines)
>       )(                      print("("+line+")")
>       )(      }
>       )(}
>       )(
>       )(
>       )
>
>       Note that I used print instead of println.  I can't
> imagine it's a
>       good thing to have line contain a line terminator of any kind.
>
>       2009/7/9 Daniel Sobral <dcsobral@...>:
>
>       > No, not expected. I don't get that either. Try doing
> map(_ map(_ toInt)) to
>       > the result of getLines. Maybe you are getting "\r\n"
> and confusing it with
>       > "\n\n".
>       >
>       > On Thu, Jul 9, 2009 at 1:38 PM, Gilles SCOUVART
> <Gilles.SCOUVART@...>
>       > wrote:
>       >>
>       >> Hello all,
>       >>
>       >> I was surprised to notice that using
> Source.fromFile(file).getLines on a
>       >> file with empty lines, those were added to the
> previous non empty line.
>       >> I.e. if the file contains this:
>       >> ------
>       >> test
>       >>
>       >>
>       >> ------
>       >> then getLines.next would return "test\n\n" (written
> java-echap-style).
>       >> I expected it would iterate on three lines instead:
> "test", "", and "".
>       >> This is a little bit annoying for pattern matching
> on full lines...
>       >>
>       >> Is this expected behaviour?
>       >>
>       >> Gilles Scouvart.
>       >
>       >
>       > --
>       > Daniel C. Sobral
>       >
>       > Something I learned in academia: there are three
> kinds of academic reviews:
>       > review by name, review by reference and review by value.
>       >
>
>
>
>
>       --
>       Ricky Clarkson
>       Java Programmer, AD Holdings
>       [phone]
>       Skype: ricky_clarkson
>       Google Talk: ricky.clarkson@...
>
>
>
>
>
> --
> Daniel C. Sobral
>
> Something I learned in academia: there are three kinds of
> academic reviews: review by name, review by reference and
> review by value.
>
>

--
Daniel C. Sobral

Something I learned in academia: there are three kinds of academic reviews: review by name, review by reference and review by value.

[Josh Suereth]

What about a getLines that allowed you to specify your expected line ending (or defaulted to "any")?

On Fri, Jul 10, 2009 at 11:00 AM, Daniel Sobral <dcsobral@...> wrote:

Well, I'm not going to defend preserving the new line characters. I just don't think it surprising (as opposed to what Scala does with regex, which I do think is very suprising).

I'd be perfectly happy with a getLines that stripped the new line characters.

On Fri, Jul 10, 2009 at 6:21 AM, Detering Dirk <Dirk.Detering@...> wrote:

Sorry, I'm feeling a bit confused now ...

I thought that "\n" is a "line terminator", (at least the
one on *nix based systems).

So I would translate your sentence

> How would you tell apart a line that ends in "\n" with one
> that doesn't?

Into "How would you tell apart a line that ends in a line end

with one that doesn't?"

I now ask: How would tell that a stream of characters is a "line"
if not by the line end marker? (Beside eof/eos of course).

> The method getLines doesn't strip away any characters present in
> the file. Which, by the way, is pretty  standard behavior.

So far ...

> In Perl, for instance, you need to "chomp"
> away newlines if you don't want them. Likewise, in Scala, you
> need to stripLineEnd newlines if you don't want them.

... so bad. Think of the regex pattern match problem discussed earlier
here. (You have to strip the line before matching with "^some$" or
use "(?s)^some$").

From a domain progger's view:

When reading a CSV file into excel I get the data into cells,
without the cell separator (the C in CSV).

So when calling getLine() I would expect to get the "line" without
the technical EOL detail. This may be added again if I write
"line" out to some file with write[Line|ln] or print[Line|ln],
like I get the comma again when exporting the data as CSV.

Same is true for every envelope data on underlying transport
layers. They are only present on that layer.

Looking at a stream from a char perspective I would
expect to see it.
Looking at it from a line perspective, it makes no sense to me.

Just my 2ct.

>
>
> On Thu, Jul 9, 2009 at 2:07 PM, Ricky Clarkson
> <ricky.clarkson@...> wrote:
>
>
>       ricky@ricky-desktop:~$ cat test.scala
>       import scala.io.Source
>       import java.io.File
>
>       object Main {
>              def main(args: Array[String]) {
>                      for (line <- Source.fromFile(new
> File("test.scala")).getLines)
>                              print("("+line+")")
>              }
>       }
>
>
>       ricky@ricky-desktop:~$ scalac test.scala
>       ricky@ricky-desktop:~$ scala Main
>       (import scala.io.Source
>       )(import java.io.File
>       )(
>       )(object Main {
>       )(      def main(args: Array[String]) {
>       )(              for (line <- Source.fromFile(new
> File("test.scala")).getLines)
>       )(                      print("("+line+")")
>       )(      }
>       )(}
>       )(
>       )(
>       )
>
>       Note that I used print instead of println.  I can't
> imagine it's a
>       good thing to have line contain a line terminator of any kind.
>
>       2009/7/9 Daniel Sobral <dcsobral@...>:
>
>       > No, not expected. I don't get that either. Try doing
> map(_ map(_ toInt)) to
>       > the result of getLines. Maybe you are getting "\r\n"
> and confusing it with
>       > "\n\n".
>       >
>       > On Thu, Jul 9, 2009 at 1:38 PM, Gilles SCOUVART
> <Gilles.SCOUVART@...>
>       > wrote:
>       >>
>       >> Hello all,
>       >>
>       >> I was surprised to notice that using
> Source.fromFile(file).getLines on a
>       >> file with empty lines, those were added to the
> previous non empty line.
>       >> I.e. if the file contains this:
>       >> ------
>       >> test
>       >>
>       >>
>       >> ------
>       >> then getLines.next would return "test\n\n" (written
> java-echap-style).
>       >> I expected it would iterate on three lines instead:
> "test", "", and "".
>       >> This is a little bit annoying for pattern matching
> on full lines...
>       >>
>       >> Is this expected behaviour?
>       >>
>       >> Gilles Scouvart.
>       >
>       >
>       > --
>       > Daniel C. Sobral
>       >
>       > Something I learned in academia: there are three
> kinds of academic reviews:
>       > review by name, review by reference and review by value.
>       >
>
>
>
>
>       --
>       Ricky Clarkson
>       Java Programmer, AD Holdings
>       [phone]
>       Skype: ricky_clarkson
>       Google Talk: ricky.clarkson@...
>
>
>
>
>
> --
> Daniel C. Sobral
>
> Something I learned in academia: there are three kinds of
> academic reviews: review by name, review by reference and
> review by value.
>
>

--

Daniel C. Sobral

Something I learned in academia: there are three kinds of academic reviews: review by name, review by reference and review by value.

[Detering Dirk-2]

> -----Original Message-----
> From: Josh Suereth [mailto:joshua.suereth@...]

> What about a getLines that allowed you to specify your
> expected line ending (or defaulted to "any")?

Hm. I would not like to specify the ending itsself on the
getLines method. It has too much of a configuration character
(think: "file mode") to be done on access.

OTOH I thought about two methods:
getLine(s) -- "higher" level, without line ending
              ideal for scripters
getRawLine(s) -- "lower" level, keeping the EOL characters
              ideal for byte shifters.

I like it when you can look at a code and immediately
get an idea of the level at which the snippet works.

[Ricky Clarkson]

I think it'd be reasonable to have it as an optional parameter on getLines.

2009/7/10 Detering Dirk <Dirk.Detering@...>:

>> -----Original Message-----
>> From: Josh Suereth [mailto:joshua.suereth@...]
>
>> What about a getLines that allowed you to specify your
>> expected line ending (or defaulted to "any")?
>
> Hm. I would not like to specify the ending itsself on the
> getLines method. It has too much of a configuration character
> (think: "file mode") to be done on access.
>
> OTOH I thought about two methods:
> getLine(s) -- "higher" level, without line ending
>              ideal for scripters
> getRawLine(s) -- "lower" level, keeping the EOL characters
>              ideal for byte shifters.
>
> I like it when you can look at a code and immediately
> get an idea of the level at which the snippet works.
>
>
>
>
>
>

--
Ricky Clarkson
Java Programmer, AD Holdings
[phone]
Skype: ricky_clarkson
Google Talk: ricky.clarkson@...

[Josh Suereth]

The current design for Scalax.Io is to have them as defaulted-implicit parameters.  The method looks somewhat as follows:

def getLine(implicit lineEndingStyle : LineEndingStyle = LineEndingStyle.default)

This allows you to type: 

stream.getLine  //for default ending (which tries to match any style)

or

stream.getLine(LineEndingStyle.UNIX)  //for unix line endings on this line

or

implicit val lineEndingStyle = LineEndingStyle.MAC
stream.getLine  //So all get line calls in this scope use MAC line ending.


If you wanted raw bytes, you should use stream.bytes instead.  Perhaps we could provide some sort of default parameter in additon that lets you specify whether or not to include the line ending characters in the returned string.


- Josh

On Fri, Jul 10, 2009 at 1:01 PM, Ricky Clarkson <ricky.clarkson@...> wrote:

I think it'd be reasonable to have it as an optional parameter on getLines.

2009/7/10 Detering Dirk <Dirk.Detering@...>:

>> -----Original Message-----
>> From: Josh Suereth [mailto:joshua.suereth@...]
>
>> What about a getLines that allowed you to specify your
>> expected line ending (or defaulted to "any")?
>
> Hm. I would not like to specify the ending itsself on the
> getLines method. It has too much of a configuration character
> (think: "file mode") to be done on access.
>
> OTOH I thought about two methods:
> getLine(s) -- "higher" level, without line ending
>              ideal for scripters
> getRawLine(s) -- "lower" level, keeping the EOL characters
>              ideal for byte shifters.
>
> I like it when you can look at a code and immediately
> get an idea of the level at which the snippet works.
>
>
>
>
>
>

--

Ricky Clarkson
Java Programmer, AD Holdings
[phone]
Skype: ricky_clarkson
Google Talk: ricky.clarkson@...

[Daniel Sobral]

On Fri, Jul 10, 2009 at 6:21 AM, Detering Dirk <Dirk.Detering@...> wrote:

Sorry, I'm feeling a bit confused now ...

I thought that "\n" is a "line terminator", (at least the
one on *nix based systems).

So I would translate your sentence

> How would you tell apart a line that ends in "\n" with one
> that doesn't?

Into "How would you tell apart a line that ends in a line end

with one that doesn't?"

I now ask: How would tell that a stream of characters is a "line"
if not by the line end marker? (Beside eof/eos of course).

Well, "beside eof/eos" is the whole point, actually.

 

I have lots of scripts which process files, doing what is, for all purposes, advanced search&replace. Having something change in the file that was outside what is expected would break a lot of things for me.

 

These are all in Perl, though, so I don't care about this issue Scala-wise. But a stream of bytes can't be regex-matched, so that is not a solution to getting "\n".

--
Daniel C. Sobral

Something I learned in academia: there are three kinds of academic reviews: review by name, review by reference and review by value.