Abstraction leak

[Donald Bruce Stewart]

drtomc:

> On 7/4/07, Donald Bruce Stewart <dons@...> wrote:
> >Can we do a cheap bytestring binding to libxml, to avoid any initial
> >String processing?
>
> For my part, it's not too big an issue. A version of HaXml or at least
> Parsec built on top of ByteString would be a good start. I know there
> was a SoC for the latter, though I have not looked to see where it
> ended up.
>
> Actually, if you were looking for a good bit of abstraction to build
> how's this? It would be *really* nice to do all my IO with mmap so my
> program isn't hit by the buffer duplication problem[*].  The kind of
> API I have in mind is something like:
>
> data Mapping -- abstract
>
> mmap :: Handle {- or Fd, perhaps -} -> Offset -> Length -> IO Mapping
>
> read :: Mapping -> Offset -> Length -> IO ByteString
>
> write :: Mapping -> Offset -> ByteString -> IO ()
>
> munmap :: Mapping -> IO () -- maybe just use a finalizer

Oh, we should really restore the mmapFile interface in Data.ByteString.
Currently its commented out to help out windows people.

And the current implementation does indeed use finalisers to handle the
unmapping.

> This API has the problem that read in particular still has to do
> copying. If you think about the binary XML stuff I mentioned before,
> you'll see that it would be really nice if I could mmap in a record
> and parse it without having to do any copying, or at least to defer
> any copying with a copy-on-write scheme. Doing a simple implementation
> of read that just put a ByteString wrapper around the mmapped memory
> would be nice and efficient, but would suffer from the problem that if
> something changed that bit of the underlying file, things would break.
> Maybe it's just not possible to finesse this one.

Yep. The current impl is:

    mmapFile :: FilePath -> IO ByteString
    mmapFile f = mmap f >>= \(fp,l) -> return $! PS fp 0 l

    mmap :: FilePath -> IO (ForeignPtr Word8, Int)
    mmap = do
         ...
         p  <- mmap l fd
         fp <- newForeignPtr p unmap -- attach unmap finaliser
         return fp

Maybe I should just stick this in the unix package.

-- Don
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Thomas Conway]

On 7/5/07, Donald Bruce Stewart <dons@...> wrote:

> Yep. The current impl is:
>
>     mmapFile :: FilePath -> IO ByteString
>     mmapFile f = mmap f >>= \(fp,l) -> return $! PS fp 0 l
>
>     mmap :: FilePath -> IO (ForeignPtr Word8, Int)
>     mmap = do
>          ...
>          p  <- mmap l fd
>          fp <- newForeignPtr p unmap -- attach unmap finaliser
>          return fp
>

Which, if I read it correctly is not safe in a concurrent/multitasking
environment, since it wraps the underlying mmapped region. In many
programs, I'm sure this won't be a problem. Unfortunately, the system
I'm working on is multi-threaded, and we definitely want to update
regions. Perhaps I'll have to bite the bullet and implement the
Mapping thing I described. The really unfortunate thing is that I'd
really like to be able to do it within the STM monad, with rollback,
etc - escaping to the IO monad is annoying.

FWIW, the technique I use to handle this kind of situation may be of
general interest. Consider a cache of structures reconstituted from an
external file.  If a requested item is not in the cache, then we throw
an exception which is caught in a wrapper function which is in the IO
monad, read the requested structure, stick it in the cache, then rerun
the transaction. There are a few details you have to get right,
including making sure none of the items you require to complete the
operation get evicted by another thread, but it works very nicely.

T.
--
Dr Thomas Conway
drtomc@...

Silence is the perfectest herald of joy:
I were but little happy, if I could say how much.
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Philip Armstrong]

On Thu, Jul 05, 2007 at 08:50:42AM +1000, Donald Bruce Stewart wrote:
[useful stuff]

So, in fact pretty much everything I was looking for exists, in some
form or other!

It's just a bit hard to find at the moment, perhaps because none of
this stuff is regarded as 'core Haskell' by any of the tutorials,
books etc etc.

I'll have a play with some of the libraries mentioned upthread
anyway. Thanks everyone.

Phil

--
http://www.kantaka.co.uk/ .oOo. public key: http://www.kantaka.co.uk/gpg.txt
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Thomas Conway]

I was explaining Haskell to a perl/python hacking friend recently and
characterized things thus:

Perl is a horrible language with fantastic libraries.
Haskell is a fantastic language with horrible libraries.

Actually, many of the libraries that exist for Haskell *are*
fantastic, it's just that Haskell lacks the *coverage* that Perl or
Python have.

cheers,
T.
--
Dr Thomas Conway
drtomc@...

Silence is the perfectest herald of joy:
I were but little happy, if I could say how much.
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Donald Bruce Stewart]

drtomc:
> I was explaining Haskell to a perl/python hacking friend recently and
> characterized things thus:
>
> Perl is a horrible language with fantastic libraries.
> Haskell is a fantastic language with horrible libraries.
>
> Actually, many of the libraries that exist for Haskell *are*
> fantastic, it's just that Haskell lacks the *coverage* that Perl or
> Python have.

Yes, and we know exactly what to do about this. hackage.haskell.org is
growing by a few packages a week -- and anyone who binds to any C lib
should just upload their stuff.

So ... if you're reading this message -- please upload a library today,
and more than just a few of us might have nice Haskell jobs tomorrow!

-- Don

P.S.  Maybe we should run 'bindathons' or have CPAN-style contests to
get new libraries written? If you're bored, write a binding to some C
lib, that python, ruby or perl have already got a binding to.  Do it!
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Jonathan Cast]

On Thursday 05 July 2007, Thomas Conway wrote:
> I was explaining Haskell to a perl/python hacking friend recently and
> characterized things thus:
>
> Perl is a horrible language with fantastic libraries.
> Haskell is a fantastic language with horrible libraries.
>
> Actually, many of the libraries that exist for Haskell *are*
> fantastic, it's just that Haskell lacks the *coverage* that Perl or
> Python have.

Can't say I agree.  I've been learning Python, and have been very un-impressed
so far with its library coverage, which I would rate no better than (in terms
of the POSIX bindings, worse than) Haskell.

The one thing off the top of my head that Python had was Base64, but that's 20
lines of Haskell tops.  Aside from that, nothing.

Jonathan Cast
http://sourceforge.net/projects/fid-core
http://sourceforge.net/projects/fid-emacs
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Paul Moore-2]

On 05/07/07, Jonathan Cast <jcast@...> wrote:
> Can't say I agree.  I've been learning Python, and have been very un-impressed
> so far with its library coverage, which I would rate no better than (in terms
> of the POSIX bindings, worse than) Haskell.

It probably depends on your perspective. I've found lots of tasks that
would be a simple library call in Python, but which require me to
write the code myself in Haskell. Examples:

* Send an email
* Parse an ini file
* Gzip compress a data stream
* Calculate the MD5 checksum of a file

(Of course, I may just not have found the relevant library - that says
something about discoverability rather than coverage, I guess).

For bindings, Python's Windows bindings (pywin32) are superb, where
Haskell's are minimal and unmaintained. Of course, that won't matter
to you if you use POSIX...

> The one thing off the top of my head that Python had was Base64, but that's 20
> lines of Haskell tops.  Aside from that, nothing.

But that's 20 lines of code I don't want to write, and more, I don't
know how to write (without looking up the definition of Base64).
Having lots of these seemingly trivial helpers available "out of the
box" is what library coverage means to me. (And Python does have lots
of these - I don't know how Haskell fares in practice).

I'm not trying to start (or fan) a flamewar, but it's interesting how
different people's perspectives on libraries can be...

Paul.
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Bulat Ziganshin-2]

Hello Paul,

Thursday, July 5, 2007, 7:00:46 PM, you wrote:
> * Gzip compress a data stream

zlib

> * Send an email
> * Parse an ini file
>> The one thing off the top of my head that Python had was Base64, but that's 20

MissingH

> * Calculate the MD5 checksum of a file

crypto


--
Best regards,
 Bulat                            mailto:Bulat.Ziganshin@...

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Paul Moore-2]

On 05/07/07, Bulat Ziganshin <bulat.ziganshin@...> wrote:

> > * Gzip compress a data stream
> zlib
>
> > * Send an email
> > * Parse an ini file
> >> The one thing off the top of my head that Python had was Base64, but that's
> MissingH
>
> > * Calculate the MD5 checksum of a file
> crypto

Thanks.

The need I had for these is no longer current, but sometime I'll try
an experiment and see how easy it is, on a relatively clean Windows
box with just GHC installed, to grab and use these libraries. (Side
note: with Python, I'm used to 3rd party modules being available as
Windows installer packages - does the concept of an installable binary
for something like MissingH, which I can just install and use, make
sense for a compiled language like Haskell? The build, find I'm
missing a C library, get it, compile it, fix bugs, try again cycle I
used to hit before Python Windows installers became common isn't
something I'd like to repeat...)

As I mentioned in passing, it may well be that the library issue with
Haskell is more of a perception (or maybe organisational) issue than a
technical one...

Paul.
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Duncan Coutts]

On Thu, 2007-07-05 at 17:07 +0100, Paul Moore wrote:

> On 05/07/07, Bulat Ziganshin <bulat.ziganshin@...> wrote:
> > > * Gzip compress a data stream
> > zlib
> >
> > > * Send an email
> > > * Parse an ini file
> > >> The one thing off the top of my head that Python had was Base64, but that's
> > MissingH
> >
> > > * Calculate the MD5 checksum of a file
> > crypto
>
> Thanks.
>
> The need I had for these is no longer current, but sometime I'll try
> an experiment and see how easy it is, on a relatively clean Windows
> box with just GHC installed, to grab and use these libraries.

Just to warn you, a lot of haskell packages that bind to C libs are much
harder to get working on Windows, the zlib package for example.

This is because on all other platforms zlib comes with the system and is
installed in a location where any application can link to it. On Windows
there is no equivalent of /usr/lib you cannot easily install a C lib
somewhere that it can be used by any .exe on the system.

To make things easier yuo could avoid using -fvia-C and then at least
the zlib header files would not need to be installed, but to run a
program that uses the zlib package you'd still have to copy the zlib.dll
into the same dir as your .exe.

There is a mechanism in newer versions of Windows that allows
installing .dlls systemwide where any .exe can use them, however ghc and
the gcc toolchain do not support them yet. It requires embeding xml
manifests into the .dll and .exe files and you have to be the admin user
to install one of these systemwide .dll things. It's all a bit of a
pain.

Duncan

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Bulat Ziganshin-2]

Hello Paul,

Thursday, July 5, 2007, 8:07:34 PM, you wrote:

> note: with Python, I'm used to 3rd party modules being available as
> Windows installer packages - does the concept of an installable binary
> for something like MissingH, which I can just install and use, make
> sense for a compiled language like Haskell? The build, find I'm

all cabalized libraries are installed using the same commands:

runghc setup.hs configure
runghc setup.hs build
runghc setup.hs install

after this, you just "import" modules you need in your program and
when you build your program, ghc automatically links in all the
library functions you used

> As I mentioned in passing, it may well be that the library issue with
> Haskell is more of a perception (or maybe organisational) issue than a
> technical one...

the problem exists and for me, it's most important Haskell
insfrastructure problem. but things you are mentioned are already
implemented

i personally implemented several libs required for my own program; and
still miss ease-to-use and maintained GUI library


--
Best regards,
 Bulat                            mailto:Bulat.Ziganshin@...

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Paul Moore-2]

On 05/07/07, Paul Moore <p.f.moore@...> wrote:
> The need I had for these is no longer current, but sometime I'll try
> an experiment and see how easy it is, on a relatively clean Windows
> box with just GHC installed, to grab and use these libraries.

Just for fun I had a go with crypto:

- Found crypto 3.0.3 on hackage.
- Tried to build, it depends on NewBinary
- Found that on hackage, downloaded and built OK. Lots of scary
warnings about happy, greencard etc, not being found during configure,
but let's go on.
- Installed NewBinary as I didn't know how to make crypto find it
without installing it. I'm a bit nervous, as I don't know how to
*un*install it after I've finished. And it installed to my C drive,
where I'd really rather it went somewhere else. There is probably
documentation on how to do this, but remember, all I really want to do
is to write a tiny program to get the MD5 checksum of a file.

Ah, well. Carry on.

- crypto builds and installs OK. But where are the docs? Not installed
anywhere obvious, not on hackage. Try google.
- Found the crypto website, but aargh! It looks like 3.0.3 is out of
date and there's a 4.x available. Never mind, that's not likely to
have changed.
- But no simple examples, and the haddoc docs show APIs, but not usage examples!

Going for the obvious approach:

import System.IO
import Data.Digest.MD5

main = do
    h <- openBinaryFile "md5.hs" ReadMode
    s <- hGetContents h
    hClose h
    md5 <- hash s

No surprise, this doesn't work. After all, hash wants [Octet], not String.

OK, I know this is now getting beyond library availability. But it
doesn't compare well to Python's "I want an md5 checksum of a file -
check the docs, there's a library function built in, use it, no
problems".

I see you've already responded, and we're in broad agreement. So I
won't labour the point. It's an infrastructure issue rather than a
technical one, and it *will* improve. What will be interesting is how
much the generally lousy Windows experience can be improved - as
Duncan points out, installing development libraries on Windows,
whether Haskell or C, is a hugely irritating pain. If no-one has found
a good answer for C in all these years, it would be great if Haskell
could even do slightly better :-)

Regards,
Paul.
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Dave Bayer]

On Jul 5, 2007, at 8:00 AM, Paul Moore wrote:

> It probably depends on your perspective. I've found lots of tasks that
> would be a simple library call in Python, but which require me to
> write the code myself in Haskell. Examples:
>
> * Calculate the MD5 checksum of a file

How's this, only one line is specific to your problem:

> import System.Process
> import IO
>
> doShell :: String -> IO String
> doShell cmd = do
>     (_,out,_,_) <- runInteractiveCommand cmd
>     hGetContents out
>
> main :: IO ()
> main = do
>     md5 <- doShell "md5 -q md5.hs"
>     putStrLn md5

It's not like you'll be kicked out of the tree house for leaving the  
Haskell world to get things done. For example, ghostscript and pdf2ps  
are well-supported open source tools for converting PS to PDF, that  
can be called from most languages. What's the deal with everyone  
rewriting PDF handling in their pet language, when it's so much  
easier to generate Postscript? I'd call that Balkanization; if I were  
managing a software group, I'd never let that happen.

The true problem isn't adequate libraries in each language, it's  
interoperability so great open-source tools can get written once and  
then be supported by a cast of thousands.

There are people who claim with a straight face that they migrated to  
OS X primarily to use TextMate

        http://www.textmate.com

which is a GUI editor getting Emacs-like buzz, making Emacs seem by  
comparison like your grandfather's razor. It's as much a text-based  
operating system as an editor, and the whole thing is glued together  
with hundreds of snippets of code one can hack, written in every  
scripting language imaginable. Polyglots feel right at home...

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Neil Mitchell]

Hi

It's not a great experience now, but hopefully things are moving in
the right direction.

> - Found crypto 3.0.3 on hackage.
> - Tried to build, it depends on NewBinary

Cabal-install is intended to remove this problem, so that you can say
"i want crypto" and it gets everything that requires.

> - Found that on hackage, downloaded and built OK. Lots of scary
> warnings about happy, greencard etc, not being found during configure,
> but let's go on.

I've complained about these before, although I don't think anyone
considered doing anything about it.

> - Installed NewBinary as I didn't know how to make crypto find it
> without installing it. I'm a bit nervous, as I don't know how to
> *un*install it after I've finished. And it installed to my C drive,
> where I'd really rather it went somewhere else. There is probably
> documentation on how to do this, but remember, all I really want to do
> is to write a tiny program to get the MD5 checksum of a file.

--prefix should put it where you want. What you really want is a
Windows user interface, which is what I want too.

> - crypto builds and installs OK. But where are the docs? Not installed
> anywhere obvious, not on hackage. Try google.

You can install them with runhaskell Setup haddock, but I have no idea
where they end up. They will be on hackage at some point, with
cross-indexing etc.

> - But no simple examples, and the haddoc docs show APIs, but not usage examples!

Complain to the author. I always try and include a manual with at
least one short example of how to use the library to do something
interesting. Unfortunately all these things take time, something that
not everyone has.

> I see you've already responded, and we're in broad agreement. So I
> won't labour the point. It's an infrastructure issue rather than a
> technical one, and it *will* improve. What will be interesting is how
> much the generally lousy Windows experience can be improved

Part of the problem is that the number of Windows Haskell developers
is low. Another part of the problem is that some people have scorn and
contempt for Windows. Alas, those things are hard to change.

Thanks

Neil
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Paul Moore-2]

On 05/07/07, Dave Bayer <bayer@...> wrote:
> How's this, only one line is specific to your problem:
[...]
> >     md5 <- doShell "md5 -q md5.hs"

Doesn't work on my (Windows) PC, where I have no md5 command
available. While I agree in theory with the idea of combining focused
tools, it's a potential portability nightmare, particularly on Windows
where decent command line tools are almost non-existent on a clean
install.

You're changing the problem from finding a Haskell library (which only
needs to be installed on the development machine at compile time) to
finding a 3rd party utility, which has to be installed at runtime on
any machine using the compiled Haskell program. Not a good trade-off.

And I'm not going to get into Windows/Linux arguments... :-)

Paul.
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Philip Armstrong]

On Thu, Jul 05, 2007 at 09:41:23AM -0700, Dave Bayer wrote:
> There are people who claim with a straight face that they migrated to OS X
> primarily to use TextMate
>
> http://www.textmate.com

Presumably you mean http://macromates.com/ ?

Phil

--
http://www.kantaka.co.uk/ .oOo. public key: http://www.kantaka.co.uk/gpg.txt
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Paul Moore-2]

On 05/07/07, Neil Mitchell <ndmitchell@...> wrote:
> > - But no simple examples, and the haddoc docs show APIs, but not usage examples!
>
> Complain to the author.

Yes, that's completely unrelated to library availability issues. I got
off the topic, in all my ranting. Sorry.

> Part of the problem is that the number of Windows Haskell developers
> is low. Another part of the problem is that some people have scorn and
> contempt for Windows. Alas, those things are hard to change.

Agreed. I'm extremely grateful for all the contributions people *do*
make, even where they don't fit my environment as well as I might
like.

Anyway, I'm running out of time for today, so I'll leave this thread
now. Thanks to all for some interesting comments and pointers!

Paul.
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Duncan Coutts]

On Thu, 2007-07-05 at 17:39 +0100, Paul Moore wrote:
> I see you've already responded, and we're in broad agreement. So I
> won't labour the point. It's an infrastructure issue rather than a
> technical one, and it *will* improve. What will be interesting is how
> much the generally lousy Windows experience can be improved -

We have this slightly odd problem where half our user base use Windows
(according to the GHC user survey) but almost every active developer
uses Linux or OSX (or a few other BSD/Unix OSs). So we could enormously
improve the Windows user experience (and a few heroes work hard on doing
just that) but basically there just aren't enough developers who use
windows to give it a satisfactory level of support.

This is of course a slightly circular problem, since using/developing
Haskell on Windows is a pain, developers avoids it and so there are not
enough developers irritated by how difficult it is to motivate
developers to fix it!

Duncan

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Duncan Coutts]

On Thu, 2007-07-05 at 17:51 +0100, Neil Mitchell wrote:

> Hi
>
> It's not a great experience now, but hopefully things are moving in
> the right direction.
>
> > - Found crypto 3.0.3 on hackage.
> > - Tried to build, it depends on NewBinary
>
> Cabal-install is intended to remove this problem, so that you can say
> "i want crypto" and it gets everything that requires.
>
> > - Found that on hackage, downloaded and built OK. Lots of scary
> > warnings about happy, greencard etc, not being found during configure,
> > but let's go on.
>
> I've complained about these before, although I don't think anyone
> considered doing anything about it.

We know what needs to change, but it's not a trivial change.

The problem is that currently .cabal files to not specify what build
tools they need, so Cabal has to look for all the possible tools it
knows about before it finds out if any of them will be needed. This is
because at the moment it has to check for these tools in the configure
step, but currently it only finds out if it needs the tools in the build
phase.

The right thing to do is to have proper dep resolution that works out
what tools are needed and it should tell the developer to record this in
the .cabal file in a new build tools dependency field. They do have to
be recorded in the .cabal file because it's impossible for the dep
resolution to discover the build tools required in the configure step
without actually running some of those build tools, which obviously
should not happen in the configure step. So the best we can do is tell
developers when they've missed a tool.

Then once they're recorded in the .cabal file it'll be easy to avoid
looking for and warning about build tools that are not required.

Duncan

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Ian Lynagh]

On Thu, Jul 05, 2007 at 06:08:45PM +0100, Duncan Coutts wrote:

> On Thu, 2007-07-05 at 17:51 +0100, Neil Mitchell wrote:
> >
> > > - Found that on hackage, downloaded and built OK. Lots of scary
> > > warnings about happy, greencard etc, not being found during configure,
> > > but let's go on.
> >
> > I've complained about these before, although I don't think anyone
> > considered doing anything about it.
>
> We know what needs to change, but it's not a trivial change.

If anyone's interested, this is the Cabal bug for it:
    http://hackage.haskell.org/trac/hackage/ticket/132


Thanks
Ian

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@...
http://www.haskell.org/mailman/listinfo/haskell-cafe