Large files upload fails

[Paul-Inge Flakstad]

Hi all

 

I'm pretty sure this question is asked on a regular basis in here, and my apologies for asking it again. I just couldn't find any solution.

 

OpenCms misbehaves when I try to upload large files. One of two scenarios occur: either 1.) The upload applet closes and no file is visible in the workplace, or 2.) The upload applet closes and the file is visible in the workplace, but a CmsVfsResourceNotFoundException is thrown when I try to view the file. 

 

No exception is thrown during the applet upload. If I disable "Use upload applet", a CmsVfsResourceNotFoundException is thrown.

 

Now, I've set/checked some values to try and fix the problem:

 

1.) In my.cnf: max_upload_packet=64M

2.) In opencms-workplace.xml: <maxfileuploadsize>-1</maxfileuploadsize>

3.) In catalina.bat: CATALINA_OPTS="-Xmx256m -Xms8m"

 

After changing to this, an upload of a 2.7MB .pdf file worked (which didn't work prior to the change), but I still can't upload a 16.0MB .pdf file... Is there anything else I need to check?

 

Any solution, tip or idea is most welcome.

 

Cheers,

Paul

_______________________________________________
This mail is sent to you from the opencms-dev mailing list
To change your list options, or to unsubscribe from the list, please visit
http://lists.opencms.org/mailman/listinfo/opencms-dev

[Marc Johnen]

Hi Paul,
yes you could have found the solution by searching in the mailing list first.
This is probably connected to the max_allowed_packet property of MySql.
Also I'd remove the "-Xms8m" from your config, because it potentially create
problems, but not related to uploading.
Greetings
Marc Johnen

Paul-Inge Flakstad wrote:

Hi all

I'm pretty sure this question is asked on a regular basis in here, and my apologies for asking it again. I just couldn't find any solution.

OpenCms misbehaves when I try to upload large files. One of two scenarios occur: either 1.) The upload applet closes and no file is visible in the workplace, or 2.) The upload applet closes and the file is visible in the workplace, but a CmsVfsResourceNotFoundException is thrown when I try to view the file.

No exception is thrown during the applet upload. If I disable "Use upload applet", a CmsVfsResourceNotFoundException is thrown.

Now, I've set/checked some values to try and fix the problem:

1.) In my.cnf: max_upload_packet=64M
2.) In opencms-workplace.xml: <maxfileuploadsize>-1</maxfileuploadsize>
3.) In catalina.bat: CATALINA_OPTS="-Xmx256m -Xms8m"

After changing to this, an upload of a 2.7MB .pdf file worked (which didn't work prior to the change), but I still can't upload a 16.0MB .pdf file... Is there anything else I need to check?

Any solution, tip or idea is most welcome.

Cheers,
Paul


_______________________________________________
This mail is sent to you from the opencms-dev mailing list
To change your list options, or to unsubscribe from the list, please visit
http://lists.opencms.org/mailman/listinfo/opencms-dev

[Paul-Inge Flakstad]

Thanks for the reply, Marc.

First, a correction: My previous mail said "max_upload_packet", which was a typo - it should've said "max_allowed_packet". It is set to 64M so I should have that case covered.

I tried uploading files of various sizes yesterday, and it is pretty clear that there's some kind of ceiling at 16MB. All is well for files <16MB, but files that exceed 16MB (e.g. PDF files) cause problems.

Any other ideas? I read a post somewhere where someone suggested changing a MySQL field definition from (MEDIUM)BLOB to LONGBLOB - could that be it (sounds pretty awkward)?

-Paul

> -----Original Message-----
> From: opencms-dev-bounces@...
> [mailto:opencms-dev-bounces@...] On Behalf Of Marc Johnen
> Sent: 25. juni 2009 12:01
> To: opencms-dev@...
> Subject: Re: [opencms-dev] Large files upload fails
>
>
> Hi Paul,
> yes you could have found the solution by searching in the mailing list
> first.
> This is probably connected to the max_allowed_packet property
> of MySql.
> Also I'd remove the "-Xms8m" from your config, because it
> potentially create
> problems, but not related to uploading.
> Greetings
> Marc Johnen
>
>
>
> Paul-Inge Flakstad wrote:
> >
> > Hi all
> >
> > I'm pretty sure this question is asked on a regular basis
> in here, and my
> > apologies for asking it again. I just couldn't find any solution.
> >
> > OpenCms misbehaves when I try to upload large files. One of
> two scenarios
> > occur: either 1.) The upload applet closes and no file is
> visible in the
> > workplace, or 2.) The upload applet closes and the file is
> visible in the
> > workplace, but a CmsVfsResourceNotFoundException is thrown
> when I try to
> > view the file.
> >
> > No exception is thrown during the applet upload. If I
> disable "Use upload
> > applet", a CmsVfsResourceNotFoundException is thrown.
> >
> > Now, I've set/checked some values to try and fix the problem:
> >
> > 1.) In my.cnf: max_upload_packet=64M
> > 2.) In opencms-workplace.xml:
> <maxfileuploadsize>-1</maxfileuploadsize>
> > 3.) In catalina.bat: CATALINA_OPTS="-Xmx256m -Xms8m"
> >
> > After changing to this, an upload of a 2.7MB .pdf file worked (which
> > didn't work prior to the change), but I still can't upload
> a 16.0MB .pdf
> > file... Is there anything else I need to check?
> >
> > Any solution, tip or idea is most welcome.
> >
> > Cheers,
> > Paul
> >
> >
> > _______________________________________________
> > This mail is sent to you from the opencms-dev mailing list
> > To change your list options, or to unsubscribe from the
> list, please visit
> > http://lists.opencms.org/mailman/listinfo/opencms-dev
> >
>
> --
> View this message in context:
> http://www.nabble.com/Large-files-upload-fails-tp24187053p2420

0400.html
> Sent from the OpenCMS - Dev mailing list archive at Nabble.com.
>
>
> _______________________________________________
> This mail is sent to you from the opencms-dev mailing list
> To change your list options, or to unsubscribe from the list,
> please visit
> http://lists.opencms.org/mailman/listinfo/opencms-dev
>

_______________________________________________
This mail is sent to you from the opencms-dev mailing list
To change your list options, or to unsubscribe from the list, please visit
http://lists.opencms.org/mailman/listinfo/opencms-dev

[Sébastien Bianco-2]

Indeed it could be a solution. I had a size problem when uploading large file and it solved my problem... (But it was not specifically a OpenCms upload even if in OpenCms !)

Sébastien BIANCO

Ingénieur développement

 EURELIS

 AGITATEUR DE TECHNOLOGIES

Tel : [phone]

Fax : [phone]

 

-----Message d'origine-----
De : opencms-dev-bounces@... [mailto:opencms-dev-bounces@...] De la part de Paul-Inge Flakstad
Envoyé : jeudi 25 juin 2009 13:10
À : The OpenCms mailing list
Objet : Re: [opencms-dev] Large files upload fails

Thanks for the reply, Marc.

First, a correction: My previous mail said "max_upload_packet", which was a typo - it should've said "max_allowed_packet". It is set to 64M so I should have that case covered.

I tried uploading files of various sizes yesterday, and it is pretty clear that there's some kind of ceiling at 16MB. All is well for files <16MB, but files that exceed 16MB (e.g. PDF files) cause problems.

Any other ideas? I read a post somewhere where someone suggested changing a MySQL field definition from (MEDIUM)BLOB to LONGBLOB - could that be it (sounds pretty awkward)?

-Paul

> -----Original Message-----
> From: opencms-dev-bounces@...
> [mailto:opencms-dev-bounces@...] On Behalf Of Marc Johnen
> Sent: 25. juni 2009 12:01
> To: opencms-dev@...
> Subject: Re: [opencms-dev] Large files upload fails
>
>
> Hi Paul,
> yes you could have found the solution by searching in the mailing list
> first.
> This is probably connected to the max_allowed_packet property
> of MySql.
> Also I'd remove the "-Xms8m" from your config, because it
> potentially create
> problems, but not related to uploading.
> Greetings
> Marc Johnen
>
>
>
> Paul-Inge Flakstad wrote:
> >
> > Hi all
> >
> > I'm pretty sure this question is asked on a regular basis
> in here, and my
> > apologies for asking it again. I just couldn't find any solution.
> >
> > OpenCms misbehaves when I try to upload large files. One of
> two scenarios
> > occur: either 1.) The upload applet closes and no file is
> visible in the
> > workplace, or 2.) The upload applet closes and the file is
> visible in the
> > workplace, but a CmsVfsResourceNotFoundException is thrown
> when I try to
> > view the file.
> >
> > No exception is thrown during the applet upload. If I
> disable "Use upload
> > applet", a CmsVfsResourceNotFoundException is thrown.
> >
> > Now, I've set/checked some values to try and fix the problem:
> >
> > 1.) In my.cnf: max_upload_packet=64M
> > 2.) In opencms-workplace.xml:
> <maxfileuploadsize>-1</maxfileuploadsize>
> > 3.) In catalina.bat: CATALINA_OPTS="-Xmx256m -Xms8m"
> >
> > After changing to this, an upload of a 2.7MB .pdf file worked (which
> > didn't work prior to the change), but I still can't upload
> a 16.0MB .pdf
> > file... Is there anything else I need to check?
> >
> > Any solution, tip or idea is most welcome.
> >
> > Cheers,
> > Paul
> >
> >
> > _______________________________________________
> > This mail is sent to you from the opencms-dev mailing list
> > To change your list options, or to unsubscribe from the
> list, please visit
> > http://lists.opencms.org/mailman/listinfo/opencms-dev
> >
>
> --
> View this message in context:
> http://www.nabble.com/Large-files-upload-fails-tp24187053p2420

0400.html
> Sent from the OpenCMS - Dev mailing list archive at Nabble.com.
>
>
> _______________________________________________
> This mail is sent to you from the opencms-dev mailing list
> To change your list options, or to unsubscribe from the list,
> please visit
> http://lists.opencms.org/mailman/listinfo/opencms-dev
>

_______________________________________________
This mail is sent to you from the opencms-dev mailing list
To change your list options, or to unsubscribe from the list, please visit
http://lists.opencms.org/mailman/listinfo/opencms-dev

_______________________________________________
This mail is sent to you from the opencms-dev mailing list
To change your list options, or to unsubscribe from the list, please visit
http://lists.opencms.org/mailman/listinfo/opencms-dev

[Marc Johnen]

Hi Paul,
are you sure the changes to MySql are active? Did you edit the
correct config-file and restart MySql? I never had to change
anything more than that property.
Greetings
Marc

Paul-Inge Flakstad wrote:

Thanks for the reply, Marc.

First, a correction: My previous mail said "max_upload_packet", which was a typo - it should've said "max_allowed_packet". It is set to 64M so I should have that case covered.

I tried uploading files of various sizes yesterday, and it is pretty clear that there's some kind of ceiling at 16MB. All is well for files <16MB, but files that exceed 16MB (e.g. PDF files) cause problems.

Any other ideas? I read a post somewhere where someone suggested changing a MySQL field definition from (MEDIUM)BLOB to LONGBLOB - could that be it (sounds pretty awkward)?

-Paul

> -----Original Message-----
> From: [e-mail]
> [mailto:[e-mail]] On Behalf Of Marc Johnen
> Sent: 25. juni 2009 12:01
> To: [e-mail]
> Subject: Re: [opencms-dev] Large files upload fails
>
>
> Hi Paul,
> yes you could have found the solution by searching in the mailing list
> first.
> This is probably connected to the max_allowed_packet property
> of MySql.
> Also I'd remove the "-Xms8m" from your config, because it
> potentially create
> problems, but not related to uploading.
> Greetings
> Marc Johnen
>
>
>
> Paul-Inge Flakstad wrote:
> >
> > Hi all
> >
> > I'm pretty sure this question is asked on a regular basis
> in here, and my
> > apologies for asking it again. I just couldn't find any solution.
> >
> > OpenCms misbehaves when I try to upload large files. One of
> two scenarios
> > occur: either 1.) The upload applet closes and no file is
> visible in the
> > workplace, or 2.) The upload applet closes and the file is
> visible in the
> > workplace, but a CmsVfsResourceNotFoundException is thrown
> when I try to
> > view the file.
> >
> > No exception is thrown during the applet upload. If I
> disable "Use upload
> > applet", a CmsVfsResourceNotFoundException is thrown.
> >
> > Now, I've set/checked some values to try and fix the problem:
> >
> > 1.) In my.cnf: max_upload_packet=64M
> > 2.) In opencms-workplace.xml:
> <maxfileuploadsize>-1</maxfileuploadsize>
> > 3.) In catalina.bat: CATALINA_OPTS="-Xmx256m -Xms8m"
> >
> > After changing to this, an upload of a 2.7MB .pdf file worked (which
> > didn't work prior to the change), but I still can't upload
> a 16.0MB .pdf
> > file... Is there anything else I need to check?
> >
> > Any solution, tip or idea is most welcome.
> >
> > Cheers,
> > Paul
> >
> >
> > _______________________________________________
> > This mail is sent to you from the opencms-dev mailing list
> > To change your list options, or to unsubscribe from the
> list, please visit
> > http://lists.opencms.org/mailman/listinfo/opencms-dev
> >
>
> --
> View this message in context:
> http://www.nabble.com/Large-files-upload-fails-tp24187053p2420
0400.html
> Sent from the OpenCMS - Dev mailing list archive at Nabble.com.
>
>
> _______________________________________________
> This mail is sent to you from the opencms-dev mailing list
> To change your list options, or to unsubscribe from the list,
> please visit
> http://lists.opencms.org/mailman/listinfo/opencms-dev
>

_______________________________________________
This mail is sent to you from the opencms-dev mailing list
To change your list options, or to unsubscribe from the list, please visit
http://lists.opencms.org/mailman/listinfo/opencms-dev

[Shi Yusen]

Perhaps you should use
max_allowed_packet
rather than
max_upload_packet

BTW, I just heard the first ice breaker of China will be designed by
Norway, is this a business of your institute?

Regards,

Shi Yusen/Beijing Langhua Ltd.

在 2009-06-24三的 17:24 +0200，Paul-Inge Flakstad写道：

> Hi all
>  
> I'm pretty sure this question is asked on a regular basis in here, and
> my apologies for asking it again. I just couldn't find any solution.
>  
> OpenCms misbehaves when I try to upload large files. One of two
> scenarios occur: either 1.) The upload applet closes and no file
> is visible in the workplace, or 2.) The upload applet closes and the
> file is visible in the workplace, but a
> CmsVfsResourceNotFoundException is thrown when I try to view the
> file.
>  
> No exception is thrown during the applet upload. If I disable "Use
> upload applet", a CmsVfsResourceNotFoundException is thrown.
>  
> Now, I've set/checked some values to try and fix the problem:
>  
> 1.) In my.cnf: max_upload_packet=64M
> 2.) In opencms-workplace.xml:
> <maxfileuploadsize>-1</maxfileuploadsize>
> 3.) In catalina.bat: CATALINA_OPTS="-Xmx256m -Xms8m"
>  
> After changing to this, an upload of a 2.7MB .pdf file worked (which
> didn't work prior to the change), but I still can't upload a
> 16.0MB .pdf file... Is there anything else I need to check?
>  
> Any solution, tip or idea is most welcome.
>  
> Cheers,
> Paul
> _______________________________________________
> This mail is sent to you from the opencms-dev mailing list
> To change your list options, or to unsubscribe from the list, please visit
> http://lists.opencms.org/mailman/listinfo/opencms-dev

_______________________________________________
This mail is sent to you from the opencms-dev mailing list
To change your list options, or to unsubscribe from the list, please visit
http://lists.opencms.org/mailman/listinfo/opencms-dev

[Paul-Inge Flakstad]

Hi again Marc

I contacted our in-house DB guru, he helped me dig deeper.

There were some strange things going on. Although I had set the max_allowed_packet variable in MySQL config file, this setting was not employed as intended.

The return value of the command (in bash):
mysql --verbose --help | grep max_allowed_packet

was inconsistent with the return value of the query (in mysql shell):
show variables like 'max_allowed_packet';

We had to add the max_allowed_packet variable in a different part of the config file file - specifically, in the [mysql] section instead of the [mysqld] section. After this small change, the return values are consistent, and large file upload is working fine (yay!).

For reference:
1) We ran a describe query on the CMS_OFFLINE_CONTENTS table, which revealed that the data field is in fact a longblob field and thus does not have to be altered.
2) I had a problem with the publishing of large files as well, even though the files got uploaded correctly. Checking the OpenCms log file, I identified the cause to be 'java.lang.OutOfMemoryError: Java heap space'. This was easily solved by increasing the Java heap space.

Many thanks to Marc and Sébastien for useful input. :-)

Cheers,
Paul

> -----Original Message-----
> From: opencms-dev-bounces@...
> [mailto:opencms-dev-bounces@...] On Behalf Of Marc Johnen
> Sent: 25. juni 2009 13:54
> To: opencms-dev@...
> Subject: Re: [opencms-dev] Large files upload fails
>
>
> Hi Paul,
> are you sure the changes to MySql are active? Did you edit the
> correct config-file and restart MySql? I never had to change
> anything more than that property.
> Greetings
> Marc
>
>
> Paul-Inge Flakstad wrote:
> >
> > Thanks for the reply, Marc.
> >
> > First, a correction: My previous mail said
> "max_upload_packet", which was
> > a typo - it should've said "max_allowed_packet". It is set
> to 64M so I
> > should have that case covered.
> >
> > I tried uploading files of various sizes yesterday, and it
> is pretty clear
> > that there's some kind of ceiling at 16MB. All is well for
> files <16MB,
> > but files that exceed 16MB (e.g. PDF files) cause problems.
> >
> > Any other ideas? I read a post somewhere where someone
> suggested changing
> > a MySQL field definition from (MEDIUM)BLOB to LONGBLOB -
> could that be it
> > (sounds pretty awkward)?
> >
> > -Paul
> >
> >> -----Original Message-----
> >> From: opencms-dev-bounces@...
> >> [mailto:opencms-dev-bounces@...] On Behalf Of Marc Johnen
> >> Sent: 25. juni 2009 12:01
> >> To: opencms-dev@...
> >> Subject: Re: [opencms-dev] Large files upload fails
> >>
> >>
> >> Hi Paul,
> >> yes you could have found the solution by searching in the
> mailing list
> >> first.
> >> This is probably connected to the max_allowed_packet property
> >> of MySql.
> >> Also I'd remove the "-Xms8m" from your config, because it
> >> potentially create
> >> problems, but not related to uploading.
> >> Greetings
> >> Marc Johnen
> >>
> >>
> >>
> >> Paul-Inge Flakstad wrote:
> >> >
> >> > Hi all
> >> >
> >> > I'm pretty sure this question is asked on a regular basis
> >> in here, and my
> >> > apologies for asking it again. I just couldn't find any solution.
> >> >
> >> > OpenCms misbehaves when I try to upload large files. One of
> >> two scenarios
> >> > occur: either 1.) The upload applet closes and no file is
> >> visible in the
> >> > workplace, or 2.) The upload applet closes and the file is
> >> visible in the
> >> > workplace, but a CmsVfsResourceNotFoundException is thrown
> >> when I try to
> >> > view the file.
> >> >
> >> > No exception is thrown during the applet upload. If I
> >> disable "Use upload
> >> > applet", a CmsVfsResourceNotFoundException is thrown.
> >> >
> >> > Now, I've set/checked some values to try and fix the problem:
> >> >
> >> > 1.) In my.cnf: max_upload_packet=64M
> >> > 2.) In opencms-workplace.xml:
> >> <maxfileuploadsize>-1</maxfileuploadsize>
> >> > 3.) In catalina.bat: CATALINA_OPTS="-Xmx256m -Xms8m"
> >> >
> >> > After changing to this, an upload of a 2.7MB .pdf file
> worked (which
> >> > didn't work prior to the change), but I still can't upload
> >> a 16.0MB .pdf
> >> > file... Is there anything else I need to check?
> >> >
> >> > Any solution, tip or idea is most welcome.
> >> >
> >> > Cheers,
> >> > Paul
> >> >
> >> >
> >> > _______________________________________________
> >> > This mail is sent to you from the opencms-dev mailing list
> >> > To change your list options, or to unsubscribe from the
> >> list, please visit
> >> > http://lists.opencms.org/mailman/listinfo/opencms-dev
> >> >
> >>
> >> --
> >> View this message in context:
> >> http://www.nabble.com/Large-files-upload-fails-tp24187053p2420
> > 0400.html
> >> Sent from the OpenCMS - Dev mailing list archive at Nabble.com.
> >>
> >>
> >> _______________________________________________
> >> This mail is sent to you from the opencms-dev mailing list
> >> To change your list options, or to unsubscribe from the list,
> >> please visit
> >> http://lists.opencms.org/mailman/listinfo/opencms-dev
> >>
> >
> > _______________________________________________
> > This mail is sent to you from the opencms-dev mailing list
> > To change your list options, or to unsubscribe from the
> list, please visit
> > http://lists.opencms.org/mailman/listinfo/opencms-dev
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Large-files-upload-fails-tp24187053p2420

1834.html
> Sent from the OpenCMS - Dev mailing list archive at Nabble.com.
>
>
> _______________________________________________
> This mail is sent to you from the opencms-dev mailing list
> To change your list options, or to unsubscribe from the list,
> please visit
> http://lists.opencms.org/mailman/listinfo/opencms-dev
>

_______________________________________________
This mail is sent to you from the opencms-dev mailing list
To change your list options, or to unsubscribe from the list, please visit
http://lists.opencms.org/mailman/listinfo/opencms-dev

[Paul-Inge Flakstad]

Yeah, that was a (horrible) typo in the mail to opencms-dev, I corrected it in a later post.

About the ice breaker, it may very well be that we're involved. ;-) (But, being merely a humble techie web-developer, I don't keep track of all the institute's activities, so I cannot say for sure.)

Cheers,
Paul

> -----Original Message-----
> From: opencms-dev-bounces@...
> [mailto:opencms-dev-bounces@...] On Behalf Of Shi Yusen
> Sent: 25. juni 2009 15:06
> To: The OpenCms mailing list
> Subject: Re: [opencms-dev] Large files upload fails
>
> Perhaps you should use
> max_allowed_packet
> rather than
> max_upload_packet
>
> BTW, I just heard the first ice breaker of China will be designed by
> Norway, is this a business of your institute?
>
> Regards,
>
> Shi Yusen/Beijing Langhua Ltd.
>
> 在 2009-06-24三的 17:24 +0200，Paul-Inge Flakstad写道：
> > Hi all
> >  
> > I'm pretty sure this question is asked on a regular basis
> in here, and
> > my apologies for asking it again. I just couldn't find any solution.
> >  
> > OpenCms misbehaves when I try to upload large files. One of two
> > scenarios occur: either 1.) The upload applet closes and no file
> > is visible in the workplace, or 2.) The upload applet closes and the
> > file is visible in the workplace, but a
> > CmsVfsResourceNotFoundException is thrown when I try to view the
> > file.
> >  
> > No exception is thrown during the applet upload. If I disable "Use
> > upload applet", a CmsVfsResourceNotFoundException is thrown.
> >  
> > Now, I've set/checked some values to try and fix the problem:
> >  
> > 1.) In my.cnf: max_upload_packet=64M
> > 2.) In opencms-workplace.xml:
> > <maxfileuploadsize>-1</maxfileuploadsize>
> > 3.) In catalina.bat: CATALINA_OPTS="-Xmx256m -Xms8m"
> >  
> > After changing to this, an upload of a 2.7MB .pdf file worked (which
> > didn't work prior to the change), but I still can't upload a
> > 16.0MB .pdf file... Is there anything else I need to check?
> >  
> > Any solution, tip or idea is most welcome.
> >  
> > Cheers,
> > Paul
> > _______________________________________________
> > This mail is sent to you from the opencms-dev mailing list
> > To change your list options, or to unsubscribe from the
> list, please visit
> > http://lists.opencms.org/mailman/listinfo/opencms-dev
>
>
> _______________________________________________
> This mail is sent to you from the opencms-dev mailing list
> To change your list options, or to unsubscribe from the list,
> please visit
> http://lists.opencms.org/mailman/listinfo/opencms-dev

_______________________________________________
This mail is sent to you from the opencms-dev mailing list
To change your list options, or to unsubscribe from the list, please visit
http://lists.opencms.org/mailman/listinfo/opencms-dev