TIFF support is coming!

[David Gerard-2]

http://techblog.wikimedia.org/2009/09/full-tiff-support-is-comming/

Funded by WMDE. Nice one :-)


- d.

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Charlotte Webb]

On Fri, Sep 11, 2009 at 1:41 PM, David Gerard <dgerard@...> wrote:
> http://techblog.wikimedia.org/2009/09/full-tiff-support-is-comming/
>
> Funded by WMDE. Nice one :-)

I don't suppose there's any easy way to use an automatic jpg
thumbnail....... but only if the browser appears not to have an
appropriate plug-in for displaying tiffs inline.

—C.W.

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

[David Gerard-2]

2009/9/11 Charlotte Webb <charlottethewebb@...>:
> On Fri, Sep 11, 2009 at 1:41 PM, David Gerard <dgerard@...> wrote:

>> http://techblog.wikimedia.org/2009/09/full-tiff-support-is-comming/
>> Funded by WMDE. Nice one :-)

> I don't suppose there's any easy way to use an automatic jpg
> thumbnail....... but only if the browser appears not to have an
> appropriate plug-in for displaying tiffs inline.


Most wouldn't. But even for browsers with SVG support, MediaWiki
thumbnails to PNG (because the browser SVG rendering is, at present,
still very slow and rather buggy).


- d.

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Charlotte Webb]

On Fri, Sep 11, 2009 at 3:12 PM, David Gerard <dgerard@...> wrote:
> Most wouldn't. But even for browsers with SVG support, MediaWiki
> thumbnails to PNG (because the browser SVG rendering is, at present,
> still very slow and rather buggy).

Could be feature creep but I was just thinking those few whose
browsers do support tiff might for whatever reason prefer thumbnails
which are in fact a scaled tiff.

—C.W.

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

[John Vandenberg]

On Sat, Sep 12, 2009 at 4:41 AM, David Gerard <dgerard@...> wrote:
> http://techblog.wikimedia.org/2009/09/full-tiff-support-is-comming/
>
> Funded by WMDE. Nice one :-)

\o/

This is most relevant for Wikisource (of course..;-) .  Many people
walk away from Wikisource because they cant grapple with this DjVu
format we love.  The PDF handler has recently been installed, after
waiting for two years.  Here is an example of a PDF being transcribed
on German Wikisource.

http://de.wikisource.org/wiki/Index:Geschichte_von_Berthelsdorf

These handlers of commonly used formats help other projects because
many people dont know how to losslessly transcode between different
formats, so it is better that they upload what they have and let
someone else do the conversion.

There are many PDFs on English Wikipedia that have been deleted
because they are "non-media without encyclopedic value" (groan).

For example, here is one I have now undeleted:

https://secure.wikimedia.org/wikipedia/en/wiki/File:2008_Sichuan_Earthquake_aftershockes_through_May_28.pdf

and put it onto the relevant article.

https://secure.wikimedia.org/wikipedia/en/w/index.php?title=2008_Sichuan_earthquake&diff=313274012&oldid=313098595

Hopefully the TIFF handler doesn't take 2 years to install like the PDF handler.

https://bugzilla.wikimedia.org/show_bug.cgi?id=11215

--
John Vandenberg

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

[John Vandenberg]

On Sat, Sep 12, 2009 at 6:08 AM, Charlotte Webb
<charlottethewebb@...> wrote:
> On Fri, Sep 11, 2009 at 1:41 PM, David Gerard <dgerard@...> wrote:
>> http://techblog.wikimedia.org/2009/09/full-tiff-support-is-comming/
>>
>> Funded by WMDE. Nice one :-)
>
> I don't suppose there's any easy way to use an automatic jpg
> thumbnail....... but only if the browser appears not to have an
> appropriate plug-in for displaying tiffs inline.

The DjVu and PDF handlers provides thumbnails as jpgs.

I would expect that the TIFF handler will also do this, otherwise it
will not integrate with the Proofread Page extension.

--
John Vandenberg

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Federico Leva (Nemo)]

John Vandenberg, 12/09/2009 01:23:
> This is most relevant for Wikisource (of course..;-) .  Many people
> walk away from Wikisource because they cant grapple with this DjVu
> format we love.

Even with DjVu, the 100 MB limit is often too low: how can TIFF be
useful with multi-paged huge documents?

Nemo

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Gerard Meijssen-3]

Hoi.
The one reason why we need TIFF documents is because museums and archives typically store their digitised material as TIFF. When we receive digitised material from a partnering organisation the best way to get it is as a TIFF. Usually we get them as JPG but when we have a relation like with the Tropenmuseum where we can request a super high resolution TIFF for restoration. It is important to have this picture as a TIFF on Commons because in that way we maintain a link with the original material. In this way we also prove that the restoration is a best effort practice to make historical material into something that retains its authenticity but is useful as an illustration.

Yes, 100MB is not big even for single paged documents. The biggest file we want to upload but cannot is over 600 Mb. Obviously they are a minority.
Thanks,
      GerardM

2009/9/12 Federico Leva (Nemo) <nemowiki@...>

John Vandenberg, 12/09/2009 01:23:

> This is most relevant for Wikisource (of course..;-) .  Many people
> walk away from Wikisource because they cant grapple with this DjVu
> format we love.

Even with DjVu, the 100 MB limit is often too low: how can TIFF be
useful with multi-paged huge documents?

Nemo

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Liam Wyatt]

On Sat, Sep 12, 2009 at 8:41 AM, Gerard Meijssen <gerard.meijssen@...> wrote:

Yes, 100MB is not big even for single paged documents. The biggest file we want to upload but cannot is over 600 Mb. Obviously they are a minority.
Thanks,
      GerardM

Yes, 600Mb files will indeed be the minority of TIFF files - for a few years yet at least :-) And, even then, TIFF files will be in the minority relative to JPG. *but* what makes supporting the TIFF format so important for the purposes of working with museums and galleries is that it demonstrates that we can do very good work (and take professional care of) their high quality images. If we can only support compressed formats then the museums and galleries can quite legitimately ask us why would we want high-res files. At that point it returns us to arguing over pixel-widths... So, not only is supporting lossless file formats good for WikiSource texts and good for image restorationists, it is also good for our future negotiations with museums and galleries.

So, keep up the good work guys!


--
wittylama.com/blog
Peace, love & metadata

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Federico Leva (Nemo)]

Gerard Meijssen, 12/09/2009 10:41:
> The one reason why we need TIFF documents is because museums and
> archives typically store their digitised material as TIFF.

I know (all of us know), but John pointed out that TIFF is useful also
for Wikisource. :-)

Nemo

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Daniel Kinzler]

Federico Leva (Nemo) schrieb:
> John Vandenberg, 12/09/2009 01:23:
>> This is most relevant for Wikisource (of course..;-) .  Many people
>> walk away from Wikisource because they cant grapple with this DjVu
>> format we love.
>
> Even with DjVu, the 100 MB limit is often too low: how can TIFF be
> useful with multi-paged huge documents?

I don't think we'll see very many huge TIFF files. But generally, uploading very
large files is possible by doing a server side import. Setting up a process for
this, and perhaps some nice tools, is something I have been thinking about.
perhaps we'll offer a contract for that next year :)

-- daniel

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Daniel Kinzler]

Charlotte Webb schrieb:
> On Fri, Sep 11, 2009 at 3:12 PM, David Gerard <dgerard@...> wrote:
>> Most wouldn't. But even for browsers with SVG support, MediaWiki
>> thumbnails to PNG (because the browser SVG rendering is, at present,
>> still very slow and rather buggy).
>
> Could be feature creep but I was just thinking those few whose
> browsers do support tiff might for whatever reason prefer thumbnails
> which are in fact a scaled tiff.

I couldn't think of a good reason for wanting that. Direct access to the full
TIFF, yes - but you have that by clicking through the description page to the
file. That will trigger the browser's native rendering, just as with svg. I
expect that for the things for wich you want native tiff support, you would want
to have full resolution anyway.

-- daniel

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

[peter green-2]

>> I don't suppose there's any easy way to use an automatic jpg
>> thumbnail....... but only if the browser appears not to have an
>> appropriate plug-in for displaying tiffs inline.
>>    
>
> The DjVu and PDF handlers provides thumbnails as jpgs.
>
> I would expect that the TIFF handler will also do this, otherwise it
> will not integrate with the Proofread Page extension.
>  

On a related note is there any chance of getting the ability to specify
thumbnail formats when adding an image to a page and/or determining what
format to use huristically? We already support png which is a lossless
format with wide support and better compression than TIFF but if you
upload a photo as a png at the moment the thumbnails will also be pngs
which is generally bad.

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Gerard Meijssen-3]

Hoi,
The objectivehere  is NOT to have PNG but to  support TIFF. This is what gets us credibility with our partners. This is what gives us the provenance that we care about representing material as it has been given to us. In this urging the use of PNG in stead of TIFF is counter productive.
Thanks,
     GerardM

2009/9/12 peter green <plugwash@...>

>> I don't suppose there's any easy way to use an automatic jpg
>> thumbnail....... but only if the browser appears not to have an
>> appropriate plug-in for displaying tiffs inline.
>>
>
> The DjVu and PDF handlers provides thumbnails as jpgs.
>
> I would expect that the TIFF handler will also do this, otherwise it
> will not integrate with the Proofread Page extension.
>

On a related note is there any chance of getting the ability to specify
thumbnail formats when adding an image to a page and/or determining what
format to use huristically? We already support png which is a lossless
format with wide support and better compression than TIFF but if you
upload a photo as a png at the moment the thumbnails will also be pngs
which is generally bad.

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Yann Forget-2]

Gerard Meijssen wrote:
> Hoi,
> The objectivehere  is NOT to have PNG but to  support TIFF. This is what
> gets us credibility with our partners. This is what gives us the
> provenance that we care about representing material as it has been given
> to us. In this urging the use of PNG in stead of TIFF is counter productive.

Nobody says otherwise. Please read again.
What is suggested, and I think it is a Good Thing (R),
is to render TIFF as JPEG.

> Thanks,
>      GerardM

Regards,

Yann
--
http://www.non-violence.org/ | Site collaboratif sur la non-violence
http://www.forget-me.net/ | Alternatives sur le Net
http://fr.wikisource.org/ | Bibliothèque libre
http://wikilivres.info | Documents libres

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Platonides]

Daniel Kinzler wrote:
> I don't think we'll see very many huge TIFF files. But generally, uploading very
> large files is possible by doing a server side import. Setting up a process for
> this, and perhaps some nice tools, is something I have been thinking about.
> perhaps we'll offer a contract for that next year :)
>
> -- daniel

Process? Tools?
It would just be making a 'bigupload' right for people to bypass file
size restrictions (or have a extremely high one).
Then give it to sysops or a new group.

The only thing is not to get those users uploading giantic files when
storage nodes are getting out of space. It could even be automated so
bigupload right would only be effective if there's at least X or X% free
space on disk.

Of course we would also need an interface able at least to continue
interrupted uploads, to make it really useful. I did a proposal years
ago based on a FTP upload interface. Maybe you are referring to
something similar. Please keep me posted.
Upload from URL and Firefogg should alleviate the issue, though.


_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

[John Vandenberg]

On Sat, Sep 12, 2009 at 7:31 PM, Federico Leva (Nemo)
<nemowiki@...> wrote:
> Gerard Meijssen, 12/09/2009 10:41:
>> The one reason why we need TIFF documents is because museums and
>> archives typically store their digitised material as TIFF.
>
> I know (all of us know), but John pointed out that TIFF is useful also
> for Wikisource. :-)

People say "archives", but do not recognise or care that archives
consist primarily of *text*.

--
John Vandenberg

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

[John Vandenberg]

On Sat, Sep 12, 2009 at 6:52 PM, Liam Wyatt <liamwyatt@...> wrote:

> On Sat, Sep 12, 2009 at 8:41 AM, Gerard Meijssen <gerard.meijssen@...>
> wrote:
>>
>> Yes, 100MB is not big even for single paged documents. The biggest file we
>> want to upload but cannot is over 600 Mb. Obviously they are a minority.
>> Thanks,
>>       GerardM
>>
> Yes, 600Mb files will indeed be the minority of TIFF files - for a few years
> yet at least :-) And, even then, TIFF files will be in the minority relative
> to JPG. *but* what makes supporting the TIFF format so important for the
> purposes of working with museums and galleries is that it demonstrates that
> we can do very good work (and take professional care of) their high quality
> images. If we can only support compressed formats then the museums and
> galleries can quite legitimately ask us why would we want high-res files. At
> that point it returns us to arguing over pixel-widths...

Wrong.  TIFF is a container format.  The contents in a TIFF file can
be compressed or uncompressed, losslessly or lossy.

PNG/MNG is a container format.  It can also have compressed or
uncompressed, losslessly or lossy.

DJVU and PDF can also contain compressed or uncompressed chunks,
losslessly or lossy.

> So, not only is
> supporting lossless file formats good for WikiSource texts and good for
> image restorationists, it is also good for our future negotiations with
> museums and galleries.

TIFF support is good for future negotiations because it means that we
don't need to argue with them about which file formats are better, or
concern ourselves with the feasibility of transcoding large
collections.

--
John Vandenberg

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Gerard Meijssen-3]

Hoi,
John you are right. TIFF can be everything you describe. The question I am left with what is your point to this? Material is scanned without compression by GLAM, we get it per standard as TIFF files, we restore them. When the material is compressed, we do not restore them. We need to retain the original to demonstrate provenance. It is  problematic to have files nobody can see in a standard way. This is why we need TIFF support, because otherwise we are likely find an admin who starts deleting this essential material.
Thanks,
      GerardM

2009/9/13 John Vandenberg <jayvdb@...>

On Sat, Sep 12, 2009 at 6:52 PM, Liam Wyatt <liamwyatt@...> wrote:
> On Sat, Sep 12, 2009 at 8:41 AM, Gerard Meijssen <gerard.meijssen@...>
> wrote:
>>
>> Yes, 100MB is not big even for single paged documents. The biggest file we
>> want to upload but cannot is over 600 Mb. Obviously they are a minority.
>> Thanks,
>>       GerardM
>>
> Yes, 600Mb files will indeed be the minority of TIFF files - for a few years
> yet at least :-) And, even then, TIFF files will be in the minority relative
> to JPG. *but* what makes supporting the TIFF format so important for the
> purposes of working with museums and galleries is that it demonstrates that
> we can do very good work (and take professional care of) their high quality
> images. If we can only support compressed formats then the museums and
> galleries can quite legitimately ask us why would we want high-res files. At
> that point it returns us to arguing over pixel-widths...

Wrong.  TIFF is a container format.  The contents in a TIFF file can
be compressed or uncompressed, losslessly or lossy.

PNG/MNG is a container format.  It can also have compressed or
uncompressed, losslessly or lossy.

DJVU and PDF can also contain compressed or uncompressed chunks,
losslessly or lossy.

> So, not only is
> supporting lossless file formats good for WikiSource texts and good for
> image restorationists, it is also good for our future negotiations with
> museums and galleries.

TIFF support is good for future negotiations because it means that we
don't need to argue with them about which file formats are better, or
concern ourselves with the feasibility of transcoding large
collections.

--
John Vandenberg

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l

[John Vandenberg]

On Sun, Sep 13, 2009 at 4:33 PM, Gerard Meijssen
<gerard.meijssen@...> wrote:
> Hoi,
> John you are right. TIFF can be everything you describe. The question I am
> left with what is your point to this? Material is scanned without
> compression by GLAM, we get it per standard as TIFF files, we restore them.
> When the material is compressed, we do not restore them. We need to retain
> the original to demonstrate provenance. It is  problematic to have files
> nobody can see in a standard way. This is why we need TIFF support, because
> otherwise we are likely find an admin who starts deleting this essential
> material.

My point is that we *can* _losslessly_ transcode TIFF files to PNG/MNG.

Provenance requires that we know where the original digitised copy is
(an identifier), but we don't need to have a copy of the original TIFF
if we have an PNG with the same quality.

TIFF support means we don't need to worry about transcoding, or have
fights about TIFF vs PNG vs PDF/A.  That will be good, as it is a
hurdle with working GLAMs, but it is not preventing high quality
images or working with GLAMs, as transcoding is not a difficult
process.

The main problem at the moment is the upload limit.

--
John Vandenberg

_______________________________________________
Commons-l mailing list
Commons-l@...
https://lists.wikimedia.org/mailman/listinfo/commons-l