Presentation: "Publisher-grade LilyPond" in Ottawa

[Boris Shingarov]

I will be giving a presentation called "Publisher-grade LilyPond"
at the next meeting of the Ottawa Linux Users Group
(this coming Thursday), in Ottawa, Canada.
The presentation is about our experiences using
LilyPond to publish a critical edition of 16th century
Gregorian chant.
Everyone is welcome to attend.



_______________________________________________
lilypond-user mailing list
lilypond-user@...
http://lists.gnu.org/mailman/listinfo/lilypond-user

[Jan Nieuwenhuizen-2]

Op woensdag 02-06-2010 om 22:50 uur [tijdzone +0000], schreef Boris
Shingarov:

> I will be giving a presentation called "Publisher-grade LilyPond"
> The presentation is about our experiences using
> LilyPond to publish a critical edition of 16th century
> Gregorian chant.
> Everyone is welcome to attend.

Nice!

And for those who cannot make it or are too lazy
to travel to Canada, will your presentation or
paper be available online?

Good luck,
Jan.

--
Jan Nieuwenhuizen <janneke@...> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyOfSource.com | Avatar®  http://AvatarAcademy.nl 



_______________________________________________
lilypond-user mailing list
lilypond-user@...
http://lists.gnu.org/mailman/listinfo/lilypond-user

[Boris Shingarov]

Hi Jan,

Thanks for the kind words, -- it's good to have the presentation is
blessed by the Founding Father.

> And for those who cannot make it or are too lazy
> to travel to Canada, will your presentation or
> paper be available online?
>    

Yes.  Of course, I will put the slides up for everyone, not just those
in the room.  And I will post a summary of any interesting discussion
that might develop (in the Q&A section possibly).

Also, we are preparing to have a paper published on the subject; so it
will have actual flowing language, not just bullet points.

Boris


_______________________________________________
lilypond-user mailing list
lilypond-user@...
http://lists.gnu.org/mailman/listinfo/lilypond-user

[Boris Shingarov]

> And for those who cannot make it or are too lazy
> to travel to Canada, will your presentation or
> paper be available online?

The slides are now online at
http://www.shingarov.com/lily


_______________________________________________
lilypond-user mailing list
lilypond-user@...
http://lists.gnu.org/mailman/listinfo/lilypond-user

[Graham Percival-3]

On Fri, Jun 11, 2010 at 9:00 PM, Boris Shingarov <b1@...> wrote:
>> And for those who cannot make it or are too lazy
>> to travel to Canada, will your presentation or
>> paper be available online?
>
> The slides are now online at
> http://www.shingarov.com/lily

Are you seriously going to submit a paper to CMJ saying "a volunteer
open-source project has limited resources for fixing bugs" ???

That's classy.  Sometimes I wonder why I bother trying to improve lilypond.

- Graham

_______________________________________________
lilypond-user mailing list
lilypond-user@...
http://lists.gnu.org/mailman/listinfo/lilypond-user

[Graham Percival-3]

On Sat, Jun 12, 2010 at 3:33 PM, Graham Percival
<graham@...> wrote:

> On Fri, Jun 11, 2010 at 9:00 PM, Boris Shingarov <b1@...> wrote:
>>> And for those who cannot make it or are too lazy
>>> to travel to Canada, will your presentation or
>>> paper be available online?
>>
>> The slides are now online at
>> http://www.shingarov.com/lily
>
> Are you seriously going to submit a paper to CMJ saying "a volunteer
> open-source project has limited resources for fixing bugs" ???

Oh, and I hope that this "paper to appear" includes an excellent
reason why you didn't use lilypond-book and LaTeX, which unlike
lilypond itself is *designed* to mix music and text.

- Graham

_______________________________________________
lilypond-user mailing list
lilypond-user@...
http://lists.gnu.org/mailman/listinfo/lilypond-user

Am Samstag, 12. Juni 2010, um 16:39:45 schrieb Graham Percival:
> Oh, and I hope that this "paper to appear" includes an excellent
> reason why you didn't use lilypond-book and LaTeX, which unlike
> lilypond itself is *designed* to mix music and text.

Ahm, I completely understand a decision NOT to use lilypond-book for this kind
of stuff. Actually, I chose to use pure latex over lilypond-book, simply
because lilypond-book is a PITA to use with latex:

-) It takes two steps to generate the PDF
-) It takes ages to do the pre-processing
-) Latex will be run on a intermediate file, so all latex problems will not
display the correct line number from your input file
-) You can't use any latex IDE (like kile), because of the above points
-) it still does not solve the problem of line breaking INSIDE music examples
-) The image file names cannot be easily deduced. In particular, if you want
to use an image somewhere else, you first need to find the correct file.
-) etc...

I chose to use latex and include the lilypond-generated music snippets
manually (they are generated by a proper make file, so I have no hassle; see
the OrchestralLily-generated make files).

Cheers,
Reinhold

--
------------------------------------------------------------------
Reinhold Kainhofer, reinhold@..., http://reinhold.kainhofer.com/
 * Financial & Actuarial Math., Vienna Univ. of Technology, Austria
 * http://www.fam.tuwien.ac.at/, DVR: 0005886
 * LilyPond, Music typesetting, http://www.lilypond.org

_______________________________________________
lilypond-user mailing list
lilypond-user@...
http://lists.gnu.org/mailman/listinfo/lilypond-user

Am Freitag, 11. Juni 2010, um 22:00:21 schrieb Boris Shingarov:
> > And for those who cannot make it or are too lazy
> > to travel to Canada, will your presentation or
> > paper be available online?
>
> The slides are now online at
> http://www.shingarov.com/lily

Some comments:

-) Flowing text with music is simply out of the scope of lilypond. That would
mean that a full text layout system is implemented, too. Otherwise things like
hyphenation or widows/orphans cannot be properly handled. Ideally, of course,
we would have someone / some company working on that, but the resources are
simply limited.
And, as you note, fragment line breaking is extremely complex...

For my critical editions (of classical music), I first tried lilypond, too,
but soon switched to latex. I don't have to mix music with text, but only
insert some figures (not with lilypond-book, but rather using a make file to
(re-)generate the images as needed). The only things that I sometimes need are
dynamics. It's still on my todo-list to make the lilypond dynamics font
available in latex.

-) "Upstream community shows zero interest in addressing these issues"
I don't think there is zero interest in having the problems fixed, rather the
contrary. However, there is zero time to address them ourselves.
The problem is that we are all doing this in our (very limited) spare time, so
of course we concentrate on stuff we need.
And, btw, I'm very interested in footnotes, as I'd need them with my critical
editions, too... But I have even more pressing needs: Fixing lilypond-book for
the MusicXML test suite, implementing cue notes WITH lyrics, finally finishing
the 10 editions I have been working on so far, etc.



BTW, have you fixes for the vertical alignment been pushed yet?

Cheers,
Reinhold


--
------------------------------------------------------------------
Reinhold Kainhofer, reinhold@..., http://reinhold.kainhofer.com/
 * Financial & Actuarial Math., Vienna Univ. of Technology, Austria
 * http://www.fam.tuwien.ac.at/, DVR: 0005886
 * LilyPond, Music typesetting, http://www.lilypond.org

_______________________________________________
lilypond-user mailing list
lilypond-user@...
http://lists.gnu.org/mailman/listinfo/lilypond-user

[Xavier Scheuer]

2010/6/11 Boris Shingarov <b1@...>:

> The slides are now online at
> http://www.shingarov.com/lily

I also found the last slides a bit too "pessimistic"...

But anyway, it's nice that people talk about LilyPond ; if we make
people discover it, maybe in those people we will find some useful
developers to continue improving LilyPond!  ;D

Cheers,
Xavier

--
Xavier Scheuer <x.scheuer@...>

_______________________________________________
lilypond-user mailing list
lilypond-user@...
http://lists.gnu.org/mailman/listinfo/lilypond-user

[Mike Blackstock-3]

I look forward to the article with great anticipation; it'll be
interesting to see how you justify the conclusions in the last few
slides.

Other than that, there isn't much one can say - no argument is put
forward, just a series of statements.

Good luck
-Mike

On Fri, Jun 11, 2010 at 4:00 PM, Boris Shingarov <b1@...> wrote:

>> And for those who cannot make it or are too lazy
>> to travel to Canada, will your presentation or
>> paper be available online?
>
> The slides are now online at
> http://www.shingarov.com/lily
>
>
> _______________________________________________
> lilypond-user mailing list
> lilypond-user@...
> http://lists.gnu.org/mailman/listinfo/lilypond-user
>

_______________________________________________
lilypond-user mailing list
lilypond-user@...
http://lists.gnu.org/mailman/listinfo/lilypond-user

[Graham Percival-3]

On Sat, Jun 12, 2010 at 07:42:24PM +0200, Xavier Scheuer wrote:

> 2010/6/11 Boris Shingarov <b1@...>:
>
> > The slides are now online at
> > http://www.shingarov.com/lily
>
> I also found the last slides a bit too "pessimistic"...
>
> But anyway, it's nice that people talk about LilyPond ; if we make
> people discover it, maybe in those people we will find some useful
> developers to continue improving LilyPond!  ;D

I doubt that -- his slides would discourage any potential
developers from joining; after all, we showed "zero interest in
addressing these issues", he was "unable to get help from the
development community, and patches were "hard or impossible to
contribute back".

... I mean, I'm *already* a developer, and I now have almost no
motivation to continue working on lilypond.  If somebody wasn't
already engaged in the project, there's no way they'd want to join
after reading those slides.

- Graham

_______________________________________________
lilypond-user mailing list
lilypond-user@...
http://lists.gnu.org/mailman/listinfo/lilypond-user

[Boris Shingarov]

Hi Graham and all Lilyponders,

I was quite astonished to hear that my slides were understood to mean anything pessimistic or negative.  If they give people this impression, then it is a defect of the slides which I will fix.

But right now, let me address some of the apparent misunderstandings.

Are you seriously going to submit a paper to CMJ saying "a volunteer
open-source project has limited resources for fixing bugs" ???
  

This is not what we are saying in our presentation and/or paper.
What we are saying is: "We attempted a publication of a major critical edition through a major publishing house, using software from a volunteer open-source project with limited resources.  We first hoped that this software would be immediately (or almost immediately) suitable for critical-edition work.  We found issues blocking this work.  These issues are orthogonal to the main direction of the LilyPond project.  We fixed them, making the book possible.  Future work includes making these solutions useful for the wide LilyPond audience, not just for the immediate needs of this particular book".

Oh, and I hope that this "paper to appear" includes an excellent
reason why you didn't use lilypond-book and LaTeX, which unlike
lilypond itself is *designed* to mix music and text.
  

Two reasons, really.

Reason one, we did investigate that route.  Lilypond-book and LaTeX do not achieve the kind of music/text integration that the editor demanded.  Another issue was integrating with their existing database of variant readings.  And for me, there was the obstacle that it was not what the end user was asked for.

Reason two, the cry for help was heard all over the mailing list starting many, many months ago.  Anyone with a LaTeX-based solution yet?  Anyone even *suggesting* a LaTeX-based solution yet?  After a year and non-trivial (thousands of Euros) bounties offered?

I agree that the presentation has a disproportionately large first (introductory) part, which makes it seem like a "presentation about LilyPond", while it is really the second part which is the essence.  There is a background why I chose to extend this first part to be somewhat detailed.  Before the presentation at OCLUG, there were two "micro-presentations" I did in a more informal setting.  They ended in somewhat curious Q&A discussions.

The first question -- asked by a guy who is well-known to everyone in the digital typography and content engineering field through dozens of papers and monographs -- he couldn't believe that today, in 2010, when everything programmable into a computer had already been programmed, today there are still books that can not be printed using already written software.  He demanded examples of problems (in music typesetting) which still waited for their programming solutions.  So 45 minutes were devoted to these examples.

At another micro-presentation, there was another question.  "I thought music typography was completely solved by Unicode?  That is, all you need to publish any score, is LaTeX and a font with all the music symbols in it?"  So, another 15 minutes to explain why LaTeX+Feta != music engraving system.

So this is the kind of questions typically asked by those in the audience who care at all.  This is why I felt an extended introduction to LilyPond was needed -- although the real subject of the presentation is mostly orthogonal to LilyPond per se.

his slides would discourage any potential
developers from joining

If there is a person who gets this impression from the slides, then the slides are definitely defective and need fixing ASAP.  I am not sure if the presentation had the same defect or it's just the slides (there certainly was a lot more explaining of the reasons and the background), but they are most definitely not supposed to convey any negative ideas.

"The community showed no interest in addressing these issues", well I am certainly not saying it's the community's *fault* -- the point is that there are other dimensions, which are of concern to [at least one] top publisher, in connection with [at least one] type of book, and which are orthogonal to the dimension of the direction of work of the LilyPond project.  After all, none of these issues originated from me; they originated from the musicologist doing the chant research, and the editor of the book -- before I even started looking at LilyPond -- and were rejected as not even being valid LilyPond problems.  The typical answers were,

"I think that you'll be better served by others"
http://www.mail-archive.com/lilypond-user@.../msg52031.html

"Your post is absolutely unnecessary"
http://www.mail-archive.com/lilypond-user@.../msg52334.html

"I recommend magic"
http://www.mail-archive.com/lilypond-user@.../msg52026.html

Does this represent a problem enough to "discourage" people?  I think it does not, for two reasons.  First, people are smart to take such things with a grain of salt.  If you look through the lkml archives through the last decade, you will see a lot darker atmosphere in that community, yet there are plenty of active kernel developers.  Second, and I insist to repeat it over and over, the main LilyPond project focus, and what we are doing in our "Publisher-Grade" project, are different, orthogonal dimensions.

"Patches were hard or impossible to contribute back" -- this is through no fault of the community.  In fact, working with Joe Neeman -- both to initially produce patches, and to make them conform to the LilyPond standards so they can be pushed into the main codebase, -- has been a highest pleasure, and his help and guidance are priceless.  His time and patience are highly appreciated.  The help from other developers, has also been great (Neil, Carl, my sincere thanks!!)  But this is not what I am talking about; all this work together with Joe and others, it's all about "the LilyPond dimension", while the problems blocking the publisher, are in those other dimensions, and LilyPond being a volunteer project, no one has incentives to do that not-very-core, not-very-exciting work.

So it's a question of the proverbial "scratching one's own itch".

Simplest example: a patch fixes a bug (a Blocker for our real-life project).  The fix is used in production for some time, and seems to be working fine.  Code review on Rietveld, patch looks good to the reviewers.  The only problem delaying its push, is the absence of a test case.  Ok, I spend some time adding a test case, but bump into problems.  Joe is so kind to offer his help in debugging; but to get through the debugging of the problem would seem to require half a day, maybe a day.  Do I understand the importance of having a test case?  Yes!  Do I want a test case?  Yes!!!  But it's about customer value. "A defect is anything that leads to customer dissatisfaction".  Well, the end-user does not care about having a test case.  The time is very tight, and we have other pressing priorities (like other Blocker bugs).  He would appreciate the inclusion of the work in the main LilyPond codebase, but not to the extent of stopping work on the other Blocker bug in order to write the test case so that the fix gets pushed to the main repo.  If it's an hour, yes.  But five hours...let it sit in the vendor tree.

There are also patches that require a lot more work to become universally useful to all LilyPond users, not just to our book here.  When a patch takes 3 days from start to deployment in production, and half a day to polish to be pushed into main repo, that's fine, I do that half-day.  When a patch takes the same 3 days, and would need another 5 days to be able to conform to LilyPond's standards, I might, but the customer would not appreciate it, so *having* the patch would in effect constitute a defect.

What's the way out of this?
Well I *hope* that by solving these "other-dimension" problems, we will be able to make LilyPond a platform for serious projects which will support its development.  That way, we will be able to afford real professional development; afford full-blown solutions, to meet all standards, and be useful to the community at large; get everything from my vendor tree into the main tree, and have no need for a vendor tree any more.  This is what I meant when I wrote that last "Future Work" slide.

On 06/12/2010 11:28 AM, Reinhold Kainhofer wrote:

Flowing text with music is simply out of the scope of lilypond. That would 
mean that a full text layout system is implemented, too. Otherwise things like 
hyphenation or widows/orphans cannot be properly handled. Ideally, of course, 
we would have someone / some company working on that, but the resources are 
simply limited.
  

Dear Dr. Kainhofer, your posts on the list have been very reassuring to me, that my work on LilyPond is going in the right direction.  What you observe about your work on your critical editions, agrees very well with my understanding of publisher-grade work.

Widows/orphans are already implemented (and even pushed! back in January I think).  Although the implementation is very limited (for example, the length of the line is not taken into account), but the Editor is pleased with the resulting look of the book, and then again, "a defect is anything that leads to customer dissatisfaction".

Yes, the text layout system is extremely primitive.  Even worse than hyphenation, the line-breaker is a simply greedy word-wrapper, and worst of all, the stretchability of the embedded music does not mingle well with that of text.  But again, the result pleases the Editor, and the first volume of the book will be accepted with this slightly imperfect stretching.  If we get support to continue our work to improve LilyPond, there will be further progress in the text layout system.

And, as you note, fragment line breaking is extremely complex...

The key here is to be very conscious of what limitations we are willing to accept.
If we make our mind that we only have the resources to within the limitations of the greedy line breaking, fragment line breaking suddenly becomes very easy.  We have it working in production since February.

I first tried lilypond, too, 
but soon switched to latex. I don't have to mix music with text, but only 
insert some figures

That's the thing -- we HAVE to mix music with text, and because if the specifics of the plainchant we are dealing with, fragments HAVE to flow seamlessly with the text.

I don't think there is zero interest in having the problems fixed, rather the 
contrary. However, there is zero time to address them ourselves.

What I am trying to do, is create some sort of professional LilyPond ecosystem, where people would be allowed to spend serious amount of time on LilyPond work, but where problems would actually get fixed.  If a publishing project is willing to spend many thousand dollars to fix a certain problem, and the only kind of response they get in a year, is "I recommend magic" and "Your post is absolutely unnecessary", that kind of interest-in-having-the-problems-fixed does not really matter to the publishing project.

And, btw, I'm very interested in footnotes, as I'd need them with my critical 
editions, too...

At this point, we temporarily put them on the back burner.
When the question about footnotes was asked on the list, no one raised their hand saying "I'll do it".  Plus, footnotes would be probably less good in the book we are preparing than endnotes, because of the huge amount of variants.  If we place the critical apparatus in footnotes, it will occupy almost the whole page, making the book difficult to use for practical singing.

BTW, have you fixes for the vertical alignment been pushed yet?

You mean, the vertical space estimation?  There were several bugs screwing height-estimation.  Some of those fixes are pushed, and some are still sitting on Rietveld.  Which ones specifically were you interested in?  Or do you mean the text / embedded score alignment?  The fix to that, requires as a prerequisite the "multi-line embedded score" fix which is sitting on Rietveld.

Boris


_______________________________________________
lilypond-user mailing list
lilypond-user@...
http://lists.gnu.org/mailman/listinfo/lilypond-user

[n.puttock]

On 13 June 2010 07:02, Boris Shingarov <b1@...> wrote:

> "Your post is absolutely unnecessary"
> http://www.mail-archive.com/lilypond-user@.../msg52334.html

That comment wasn't directed at Jiříi; it was part of a reply to David Kastrup.

Regards,
Neil

_______________________________________________
lilypond-user mailing list
lilypond-user@...
http://lists.gnu.org/mailman/listinfo/lilypond-user

[Carl Sorensen-3]

On 6/13/10 12:02 AM, "Boris Shingarov" <b1@...> wrote:

> Hi Graham and all Lilyponders,
>
> I was quite astonished to hear that my slides were understood to mean anything
> pessimistic or negative.  If they give people this impression, then it is a
> defect of the slides which I will fix.
>
> But right now, let me address some of the apparent misunderstandings.
>
>>  
>> Are you seriously going to submit a paper to CMJ saying "a volunteer
>> open-source project has limited resources for fixing bugs" ???
>>  
> This is not what we are saying in our presentation and/or paper.
> What we are saying is: "We attempted a publication of a major critical edition
> through a major publishing house, using software from a volunteer open-source
> project with limited resources.  We first hoped that this software would be
> immediately (or almost immediately) suitable for critical-edition work.  We
> found issues blocking this work.  These issues are orthogonal to the main
> direction of the LilyPond project.  We fixed them, making the book possible. 
> Future work includes making these solutions useful for the wide LilyPond
> audience, not just for the immediate needs of this particular book".
>

I like this message.  However, as others have noted, I didn't get that
message from the slides.

>>  
>> Oh, and I hope that this "paper to appear" includes an excellent
>> reason why you didn't use lilypond-book and LaTeX, which unlike
>> lilypond itself is *designed* to mix music and text.
>>  
> Two reasons, really.
>
> Reason one, we did investigate that route.  Lilypond-book and LaTeX do not
> achieve the kind of music/text integration that the editor demanded.  Another
> issue was integrating with their existing database of variant readings.  And
> for me, there was the obstacle that it was not what the end user was asked
> for.
>
> Reason two, the cry for help was heard all over the mailing list starting
> many, many months ago.  Anyone with a LaTeX-based solution yet?  Anyone even
> *suggesting* a LaTeX-based solution yet?  After a year and non-trivial
> (thousands of Euros) bounties offered?

Where are the thousands of Euros bounties for the LaTeX-based solution?  I'm
not aware of *any* thousands of Euros bounties for anything on LilyPond.
This is a serious question, not a rhetorical question, by the way.  I have
looked at the issues with Bounty tags, and the only one I can find that
seems to be relevant to this discussion is "support for footnotes and/or
endnotes".  That issue makes no mention of a LaTeX-based solution.
 

<snip>

>
> "The community showed no interest in addressing these issues", well I am
> certainly not saying it's the community's *fault* -- the point is that there
> are other dimensions, which are of concern to [at least one] top publisher, in
> connection with [at least one] type of book, and which are orthogonal to the
> dimension of the direction of work of the LilyPond project. 

I think that "these issues are orthogonal to the current work in the
LilyPond project" carries a much different connotation than "the community
has no interest in these issues", even though they may cover the same facts.

> After all, none
> of these issues originated from me; they originated from the musicologist
> doing the chant research, and the editor of the book -- before I even started
> looking at LilyPond -- and were rejected as not even being valid LilyPond
> problems.  The typical answers were,
>
> "I think that you'll be better served by others"
> http://www.mail-archive.com/lilypond-user@.../msg52031.html
>
> "Your post is absolutely unnecessary"
> http://www.mail-archive.com/lilypond-user@.../msg52334.html
>
> "I recommend magic"
> http://www.mail-archive.com/lilypond-user@.../msg52026.html
>
> Does this represent a problem enough to "discourage" people?  I think it does
> not, for two reasons.  First, people are smart to take such things with a
> grain of salt.  If you look through the lkml archives through the last decade,
> you will see a lot darker atmosphere in that community, yet there are plenty
> of active kernel developers.  Second, and I insist to repeat it over and over,
> the main LilyPond project focus, and what we are doing in our
> "Publisher-Grade" project, are different, orthogonal dimensions.

May I suggest that "Publisher-Grade" is perhaps a misnomer?  This makes it
sound like we can't produce *any* publisher-grade music without footnotes,
or endnotes, or widow/orphan control.  Perhaps an adjustment to the title to
"Publisher-Grade Critical Edition" would help clarify things.

>
> "Patches were hard or impossible to contribute back" -- this is through no
> fault of the community.  In fact, working with Joe Neeman -- both to initially
> produce patches, and to make them conform to the LilyPond standards so they
> can be pushed into the main codebase, -- has been a highest pleasure, and his
> help and guidance are priceless.  His time and patience are highly
> appreciated.  The help from other developers, has also been great (Neil, Carl,
> my sincere thanks!!)  But this is not what I am talking about; all this work
> together with Joe and others, it's all about "the LilyPond dimension", while
> the problems blocking the publisher, are in those other dimensions, and
> LilyPond being a volunteer project, no one has incentives to do that
> not-very-core, not-very-exciting work.
>
> So it's a question of the proverbial "scratching one's own itch".
>
> Simplest example: a patch fixes a bug (a Blocker for our real-life project). 
> The fix is used in production for some time, and seems to be working fine. 
> Code review on Rietveld, patch looks good to the reviewers.  The only problem
> delaying its push, is the absence of a test case.  Ok, I spend some time
> adding a test case, but bump into problems.  Joe is so kind to offer his help
> in debugging; but to get through the debugging of the problem would seem to
> require half a day, maybe a day.  Do I understand the importance of having a
> test case?  Yes!  Do I want a test case?  Yes!!!  But it's about customer
> value. "A defect is anything that leads to customer dissatisfaction".  Well,
> the end-user does not care about having a test case.  The time is very tight,
> and we have other pressing priorities (like other Blocker bugs).  He would
> appreciate the inclusion of the work in the main LilyPond codebase, but not to
> the extent of stopping work on the other Blocker bug in order to write the
> test case so that the fix gets pushed to the main repo.  If it's an hour,
> yes.  But five hours...let it sit in the vendor tree.
>
> There are also patches that require a lot more work to become universally
> useful to all LilyPond users, not just to our book here.  When a patch takes 3
> days from start to deployment in production, and half a day to polish to be
> pushed into main repo, that's fine, I do that half-day.  When a patch takes
> the same 3 days, and would need another 5 days to be able to conform to
> LilyPond's standards, I might, but the customer would not appreciate it, so
> *having* the patch would in effect constitute a defect.
>
> What's the way out of this?
> Well I *hope* that by solving these "other-dimension" problems, we will be
> able to make LilyPond a platform for serious projects which will support its
> development.  That way, we will be able to afford real professional
> development; afford full-blown solutions, to meet all standards, and be useful
> to the community at large; get everything from my vendor tree into the main
> tree, and have no need for a vendor tree any more.  This is what I meant when
> I wrote that last "Future Work" slide.

I'd appreciate anything you could do to clarify (for the  -devel list and
for the bug tracker, not necessarily for the paper) the status of the
"vendor tree" relative to inclusion in LilyPond.  For example, if there's a
patch available that is just missing its test case, then we ought to have an
issue about that.  If there are other patches that are not yet up to
LilyPond status, but might be able to get there, it would be nice to have
them listed as issues with patches attached.  Then we'd at least know that
there is work to be done.  As long as the patches are only in your vendor
tree, the community can't really help.

>
> On 06/12/2010 11:28 AM, Reinhold Kainhofer wrote:
<snip>

>>  
>> I first tried lilypond, too,
>> but soon switched to latex. I don't have to mix music with text, but only
>> insert some figures
> That's the thing -- we HAVE to mix music with text, and because if the
> specifics of the plainchant we are dealing with, fragments HAVE to flow
> seamlessly with the text.
>
>>  
>> I don't think there is zero interest in having the problems fixed, rather the
>> contrary. However, there is zero time to address them ourselves.
> What I am trying to do, is create some sort of professional LilyPond
> ecosystem, where people would be allowed to spend serious amount of time on
> LilyPond work, but where problems would actually get fixed.  If a publishing
> project is willing to spend many thousand dollars to fix a certain problem,
> and the only kind of response they get in a year, is "I recommend magic" and
> "Your post is absolutely unnecessary", that kind of
> interest-in-having-the-problems-fixed does not really matter to the publishing
> project.
>

There are two ways to ask for help.  One way is to ask for somebody to
implement a feature.  The other way is to ask for guidance in how to
implement the feature oneself.  I'm not aware of any "guidance" requests
that have gone unanswered.  On the other hand, there are *lots* of feature
requests that go unanswered.

>>  
>> And, btw, I'm very interested in footnotes, as I'd need them with my critical
>> editions, too...
> At this point, we temporarily put them on the back burner.
> When the question about footnotes was asked on the list, no one raised their
> hand saying "I'll do it".  Plus, footnotes would be probably less good in the
> book we are preparing than endnotes, because of the huge amount of variants. 
> If we place the critical apparatus in footnotes, it will occupy almost the
> whole page, making the book difficult to use for practical singing.
>

Nobody *ever* raises their hand and says "I'll do it" for a feature request
in a new area.  If I had just asked for somebody to implement fret diagrams,
I doubt they'd be part of LilyPond yet.
 
>>  
>> BTW, have you fixes for the vertical alignment been pushed yet?
> You mean, the vertical space estimation?  There were several bugs screwing
> height-estimation.  Some of those fixes are pushed, and some are still sitting
> on Rietveld.  Which ones specifically were you interested in?  Or do you mean
> the text / embedded score alignment?  The fix to that, requires as a
> prerequisite the "multi-line embedded score" fix which is sitting on Rietveld.

Can you give us Rietveld issue numbers?

Thanks,

Carl


_______________________________________________
lilypond-user mailing list
lilypond-user@...
http://lists.gnu.org/mailman/listinfo/lilypond-user

Am Sonntag, 13. Juni 2010, um 16:55:01 schrieben Sie:
> May I suggest that "Publisher-Grade" is perhaps a misnomer?  This makes it
> sound like we can't produce *any* publisher-grade music without footnotes,
> or endnotes, or widow/orphan control.  Perhaps an adjustment to the title
> to "Publisher-Grade Critical Edition" would help clarify things.

Well, my talk about OrchestralLily was exactly about creating Publisher-grade
critical editions ;-) They are definitely possible in lilypond (+LaTeX for the
prefatory matter and the critical report).

On the other hand, each critical edition has its own requirements. I have
found LilyPond fairly good for classical music and critical reports where the
sources are clear, so only differences in the dynamics or maybe different
orchestration (and some typos) need to be discussed. In this case, I simply
insert larger images into latex, but don't have to mix text with music.

For ancient music, where the notation is less than clear (like in Boris'
cases), however, the situation is apparently quite different.

Of course, there are still lots of shortcomings for classical editions, too,
but they are no showstoppers, just annoyances (like not having footnotes to
add a comment about a different orchestration, or a note referring to the
critical report).

The only two things that are holding me up still are

-) The vertical reordering of figured bass figures with extenders (currently
<6 5> <5 3> wrongly places the 3 above the extender). I'm still working on the
patch, only some spacing issues are still holding me up...

-) Cues to vocal voices do not contain lyrics, and it is practically
impossible to add lyrics to the quotes at all using workarounds.

> I'd appreciate anything you could do to clarify (for the  -devel list and
> for the bug tracker, not necessarily for the paper) the status of the
> "vendor tree" relative to inclusion in LilyPond.  For example, if there's a
> patch available that is just missing its test case, then we ought to have
> an issue about that.

+1

> If there are other patches that are not yet up to
> LilyPond status, but might be able to get there, it would be nice to have
> them listed as issues with patches attached.  

I suppose a starting point could be to look at the patches already uploaded to
rietveld by Boris:
http://codereview.appspot.com/user/Boris%20Shingarov


> Then we'd at least know that
> there is work to be done.  As long as the patches are only in your vendor
> tree, the community can't really help.

BTW, is your "vendor tree" git repository available somewhere? It might even
be useful to use your own dev/shingarov branch (or dev/shingarov/* branches)
in Lilypond's git...

Cheers,
Reinhold


--
------------------------------------------------------------------
Reinhold Kainhofer, reinhold@..., http://reinhold.kainhofer.com/
 * Financial & Actuarial Math., Vienna Univ. of Technology, Austria
 * http://www.fam.tuwien.ac.at/, DVR: 0005886
 * LilyPond, Music typesetting, http://www.lilypond.org

_______________________________________________
lilypond-user mailing list
lilypond-user@...
http://lists.gnu.org/mailman/listinfo/lilypond-user

[Carl Sorensen-3]

On 6/13/10 9:17 AM, "Reinhold Kainhofer" <reinhold@...> wrote:
>
>> If there are other patches that are not yet up to
>> LilyPond status, but might be able to get there, it would be nice to have
>> them listed as issues with patches attached.
>
> I suppose a starting point could be to look at the patches already uploaded to
> rietveld by Boris:
> http://codereview.appspot.com/user/Boris%20Shingarov
>

Thanks!  I didn't know that you could do that on Rietveld.  I should have
know, because it happens for me all the time!  D'oh!  (Smacks forehead).

Thanks,

Carl


_______________________________________________
lilypond-user mailing list
lilypond-user@...
http://lists.gnu.org/mailman/listinfo/lilypond-user

[A_w_R]

Graham Percival <graham <at> percival-music.ca> writes:

> Are you seriously going to submit a paper to CMJ saying "a volunteer
> open-source project has limited resources for fixing bugs" ???
>
> That's classy.  Sometimes I wonder why I bother trying to improve lilypond.
>
> - Graham

Graham,

There is a difference between perception and fact!
Every participant in a project like this will face incomprehension and ignorance
from end-users; considering the complexity of Lilypond this comes as no surprise.

I myself have moved away from a commercial scorewriter to Lilypond, impressed as
I am about the quality of the output, convinced as I am about it's
architecture/design and having confidence in the support and development of
Lilypond.

Regards,

Arno Rog







_______________________________________________
lilypond-user mailing list
lilypond-user@...
http://lists.gnu.org/mailman/listinfo/lilypond-user

[Graham Percival-3]

On Sun, Jun 13, 2010 at 7:02 AM, Boris Shingarov <b1@...> wrote:

> This is not what we are saying in our presentation and/or paper.
> What we are saying is: "We attempted a publication of a major critical
> edition through a major publishing house, using software from a volunteer
> open-source project with limited resources.  We first hoped that this
> software would be immediately (or almost immediately) suitable for
> critical-edition work.  We found issues blocking this work.  These issues
> are orthogonal to the main direction of the LilyPond project.  We fixed
> them, making the book possible.  Future work includes making these solutions
> useful for the wide LilyPond audience, not just for the immediate needs of
> this particular book".

That message is entirely fair.  Others have mentioned that this isn't
the impression the slides give, so let's drop the subject now.  I hope
that you've looked at Reinhold's LAU talks from last month; one of
them is quite relevant to this topic.

> Reason two, the cry for help was heard all over the mailing list starting
> many, many months ago.  Anyone with a LaTeX-based solution yet?  Anyone even
> *suggesting* a LaTeX-based solution yet?  After a year and non-trivial
> (thousands of Euros) bounties offered?

I really cannot recall seeing any offers of "thousands of Euros".  If
they're still valid offers, please send info to bug-lilypond.


> Simplest example: a patch fixes a bug (a Blocker for our real-life
> project).  The fix is used in production for some time, and seems to be
> working fine.  Code review on Rietveld, patch looks good to the reviewers.
> The only problem delaying its push, is the absence of a test case.

We now (in the past hour) have expanded documentation about test
cases; this might help:
http://lilypond.org/doc/v2.13/Documentation/contributor/regression-tests


> What I am trying to do, is create some sort of professional LilyPond
> ecosystem, where people would be allowed to spend serious amount of time on
> LilyPond work, but where problems would actually get fixed.  If a publishing
> project is willing to spend many thousand dollars to fix a certain problem,

That has been attempted before... hmm, 2007?  Han-Wen tried to work on
lilypond full-time, but there just wasn't enough people offering
bounties to be able to support his family (with a young child).  I
mean, think of what an average software developer earns in a month --
can the lilypond user community really come up with that kind of
money?

In some ways it's a "chicken and egg" problem -- you need to have
highly skilled developer(s) to be able to respond quickly and
efficiently to sponsorship requests, but on the other hand, the best
way to get highly skilled developers is to have them working on code,
and bounties are a good way to motivate some people.
* note: they're only good for _some_ people.  Most of our programmers
already have stable, busy jobs (for example, a professor of mechanical
engineering).  Chasing a few 50-euro bounties often works out to be
less than their "real" job.

OTOH, if there's a consistent stream of many hundred-euro bounties,
this could dramatically change things.

- Graham

_______________________________________________
lilypond-user mailing list
lilypond-user@...
http://lists.gnu.org/mailman/listinfo/lilypond-user

[Jan Nieuwenhuizen-2]

Op dinsdag 15-06-2010 om 16:50 uur [tijdzone +0100], schreef Graham
Percival:

> I really cannot recall seeing any offers of "thousands of Euros".  If
> they're still valid offers, please send info to bug-lilypond.

> That has been attempted before... hmm, 2007?  Han-Wen tried to work on
> lilypond full-time, but there just wasn't enough people offering
> bounties

> OTOH, if there's a consistent stream of many hundred-euro bounties,
> this could dramatically change things.

I am considering to offer commercial support and may be able to do
that on a part-time basis.  However, working on two bounties has
illustrated that bounty work can be quite tricky.  It would be very
nice for someone doing this for a hobby and getting to know LilyPond,
but commercial support requires some level of predictability.

Also, if the amount of work is not consistent but takes the form
of a few thousand euros once a year, you would be very lucky if I
(or whoever else would take this on) would happen to be available
within a reasonable time frame to work on those.


Greetings,
Jan.

--
Jan Nieuwenhuizen <janneke@...> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyOfSource.com | Avatar®  http://AvatarAcademy.nl 



_______________________________________________
lilypond-user mailing list
lilypond-user@...
http://lists.gnu.org/mailman/listinfo/lilypond-user

[Graham Percival-3]

On Tue, Jun 15, 2010 at 08:25:27PM +0200, Jan Nieuwenhuizen wrote:

> Op dinsdag 15-06-2010 om 16:50 uur [tijdzone +0100], schreef Graham
> Percival:
>
> > That has been attempted before... hmm, 2007?  Han-Wen tried to work on
> > lilypond full-time, but there just wasn't enough people offering
> > bounties
>
> I am considering to offer commercial support and may be able to do
> that on a part-time basis.  However, working on two bounties has
> illustrated that bounty work can be quite tricky.

Indeed; there's almost no relationship between the amount of work
required and the amount of money being offered.

> It would be very
> nice for someone doing this for a hobby and getting to know LilyPond,
> but commercial support requires some level of predictability.

Actually, somebody pointed out (privately) that chasing bounties
is less appealing for inexperienced developers: a $100 bounty
could very well take you 50 hours to complete (i.e. if it's your
first time working on spacing code), making the job $2 / hr.

> Also, if the amount of work is not consistent but takes the form
> of a few thousand euros once a year, you would be very lucky if I
> (or whoever else would take this on) would happen to be available
> within a reasonable time frame to work on those.

Yes.


I'm not trying to discourage people from offering bounties -- it's
certainly better than nothing!  However, there's very good reasons
why programmers don't immediately start working on any issue that
has a bounty being offered.

One idea I've toyed with is seeking a grant to work on lilypond.
Various governments and agencies give research grants; I'm pretty
certain that we could get a grant to improve medieval chant
notation or contemporary non-Western scales or whatnot.  However,
this would probably require
- a bunch of grant applications
- collaborating with some musicologists (i.e. a medieval chant
  expert, or John Cage scholar, or whatever)
- overhead of writing reports about deliverables, giving
  presentations to people, etc.
- etc.
In the process of doing the specialized notation, the developer
might fix a few "normal" bugs as well.


If there was a concerted effort, particularly between the European
academics involved with LilyPond, it could be done, and we might
even be able to fund a full-time developer for 6 months or even a
year.  However, I'm not certain the effort would be worth it --
writing grants is a lot of work; we'd probably have to make
multiple attempts; dealing with the administration of the grant
would be a lot of work; etc etc.

Cheers,
- Graham

_______________________________________________
lilypond-user mailing list
lilypond-user@...
http://lists.gnu.org/mailman/listinfo/lilypond-user