[jira] [Comment Edited] (OFBIZ-11534) Error in uploading large files

[Nicolas Malin (Jira)]

    [ https://issues.apache.org/jira/browse/OFBIZ-11534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17075092#comment-17075092 ]

Jacques Le Roux edited comment on OFBIZ-11534 at 4/4/20, 8:45 AM:
------------------------------------------------------------------

OK, It's time to write what I did so far.

I hoped that simply using a FileInputStream instead of ByteBuffer would solve the issue
 So I changed in related service definitions (uploadFileInterface, attachUploadToDataResource and createContentFromUploadedFile)
{code:xml}
<attribute mode="IN" name="uploadedFile" type="java.nio.ByteBuffer" optional="true" />
{code}
to
{code:xml}
<attribute mode="IN" name="uploadedFile" type="java.io.FileInputStream" optional="true" />
{code}

After trying 100 KB (instead of 10KB currently), I increased http.upload.max.sizethreshold to 1024000 (almost 1BM). But it did not have much effect, just much faster, I kept it.

The "Stream ended unexpectedly" error which is the gist of this issue is not much documented and/or confusing. After searching a lot and getting many deadends, I finally found the real reason: [https://stackoverflow.com/questions/12476604/processing-of-multipart-form-data-request-failed-read-timed-out] and changed catalina/ofbiz-component.xml accordingly.

It then passes the "Stream ended unexpectedly" error and I guess would work with an image or any file type recognised by Tika. Using Ubuntu iso (2MB) I get:
{noformat}
2020-04-03 14:49:51,956 |jsse-nio-8443-exec-8 |ControlServlet |T| [[[partymgr::uploadPartyContent (Domain:https://localhost)] Request Begun, encoding=[UTF-8]- total:0.0,since last(Begin):0.0]]
 [...]
 2020-04-03 14:50:45,808 |jsse-nio-8443-exec-8 |ServiceDispatcher |T| Sync service [partymgr/genericDataResourcePermission] finished in [227] milliseconds
 2020-04-03 14:50:45,915 |jsse-nio-8443-exec-8 |SequenceUtil |I| Got bank of sequenced IDs for [DataResource]; curSeqId=10160, maxSeqId=10170, bankSize=10
 Apr 03, 2020 2:50:46 PM org.apache.tika.config.InitializableProblemHandler$3 handleInitializableProblem
 WARNING: J2KImageReader not loaded. JPEG2000 files will not be processed.
 See https://pdfbox.apache.org/2.0/dependencies.html#jai-image-io
 for optional dependencies.
Apr 03, 2020 2:50:46 PM org.apache.tika.config.InitializableProblemHandler$3 handleInitializableProblem
 WARNING: org.xerial's sqlite-jdbc is not loaded.
 Please provide the jar on your classpath to parse sqlite files.
 See tika-parsers/pom.xml for the correct version.
 java.io.IOException: Stream Closed
 at java.io.FileInputStream.readBytes(Native Method)
 at java.io.FileInputStream.read(FileInputStream.java:255)
 at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
 at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
 at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
 at org.apache.commons.io.IOUtils.read(IOUtils.java:3077)
 at org.apache.commons.io.IOUtils.read(IOUtils.java:3099)
 at org.apache.tika.parser.pkg.ZipContainerDetector.detect(ZipContainerDetector.java:110)
 at org.apache.tika.detect.CompositeDetector.detect(CompositeDetector.java:84)
 at org.apache.tika.Tika.detect(Tika.java:158)
 at org.apache.tika.Tika.detect(Tika.java:203)
 at org.apache.ofbiz.content.data.DataResourceWorker.getMimeTypeFromFileInputStream(DataResourceWorker.java:412)
 [...]
 at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
 at java.lang.Thread.run(Thread.java:748)
{noformat}
Could someone ([~chandan.khandelwal]?) apply  [^OFBIZ-11534.patch]  and confirm it works with a file recognised by Tika, like an image (but not a JPEG2000)?

�I'm not quite quite sure about the changes in CommonServices::uploadTest and I also noticed that tests in CommonServices class are not used. Of course they can be tested by hand, but for uploadContentTest it's quite hard, if even possible. Or do I miss something?

I also noticed that the purpose for the upload is no longer required. It's because this (no longer?) works [http://svn.apache.org/viewvc?view=revision&revision=1087254]. I wonder why Sascha deleted jquery.validate.pack.js in this commit! But then is not a "required error" CSS class missing?


was (Author: jacques.le.roux):
OK, It's time to write what I did so far.

I hoped that simply using a FileInputStream instead of ByteBuffer would solve the issue
 So I changed in related service definitions (uploadFileInterface, attachUploadToDataResource and createContentFromUploadedFile)
{code:xml}
<attribute mode="IN" name="uploadedFile" type="java.nio.ByteBuffer" optional="true" />
{code}
to
{code:xml}
<attribute mode="IN" name="uploadedFile" type="java.io.FileInputStream" optional="true" />
{code}
Before, for the sake of the history, the error I got was:
{noformat}
2020-03-29 21:58:29,854 |jsse-nio-8443-exec-1 |Converters                    |W| *** No converter found, converting from java.io.FileInputStream to java.nio.ByteBuffer. Please report this message to the developer community so a suitable
converter can be created. ***
2020-03-29 21:58:29,854 |jsse-nio-8443-exec-1 |ObjectType                    |E| null
java.lang.ClassNotFoundException: No converter found for java.io.FileInputStream->java.nio.ByteBuffer
{noformat}
After trying 100 KB (instead of 10KB currently), I increased http.upload.max.sizethreshold to 1024000 (almost 1BM). But it did not have much effect, just much faster, I kept it.

The "Stream ended unexpectedly" error which is the gist of this issue is not much documented and/or confusing. After searching a lot and getting many deadends, I finally found the real reason: [https://stackoverflow.com/questions/12476604/processing-of-multipart-form-data-request-failed-read-timed-out] and changed catalina/ofbiz-component.xml accordingly.

It then passes the "Stream ended unexpectedly" error and I guess would work with an image or any file type recognised by Tika. Using Ubuntu iso (2MB) I get:
{noformat}
2020-04-03 14:49:51,956 |jsse-nio-8443-exec-8 |ControlServlet |T| [[[partymgr::uploadPartyContent (Domain:https://localhost)] Request Begun, encoding=[UTF-8]- total:0.0,since last(Begin):0.0]]
 [...]
 2020-04-03 14:50:45,808 |jsse-nio-8443-exec-8 |ServiceDispatcher |T| Sync service [partymgr/genericDataResourcePermission] finished in [227] milliseconds
 2020-04-03 14:50:45,915 |jsse-nio-8443-exec-8 |SequenceUtil |I| Got bank of sequenced IDs for [DataResource]; curSeqId=10160, maxSeqId=10170, bankSize=10
 Apr 03, 2020 2:50:46 PM org.apache.tika.config.InitializableProblemHandler$3 handleInitializableProblem
 WARNING: J2KImageReader not loaded. JPEG2000 files will not be processed.
 See https://pdfbox.apache.org/2.0/dependencies.html#jai-image-io
 for optional dependencies.
Apr 03, 2020 2:50:46 PM org.apache.tika.config.InitializableProblemHandler$3 handleInitializableProblem
 WARNING: org.xerial's sqlite-jdbc is not loaded.
 Please provide the jar on your classpath to parse sqlite files.
 See tika-parsers/pom.xml for the correct version.
 java.io.IOException: Stream Closed
 at java.io.FileInputStream.readBytes(Native Method)
 at java.io.FileInputStream.read(FileInputStream.java:255)
 at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
 at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
 at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
 at org.apache.commons.io.IOUtils.read(IOUtils.java:3077)
 at org.apache.commons.io.IOUtils.read(IOUtils.java:3099)
 at org.apache.tika.parser.pkg.ZipContainerDetector.detect(ZipContainerDetector.java:110)
 at org.apache.tika.detect.CompositeDetector.detect(CompositeDetector.java:84)
 at org.apache.tika.Tika.detect(Tika.java:158)
 at org.apache.tika.Tika.detect(Tika.java:203)
 at org.apache.ofbiz.content.data.DataResourceWorker.getMimeTypeFromFileInputStream(DataResourceWorker.java:412)
 [...]
 at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
 at java.lang.Thread.run(Thread.java:748)
{noformat}
Could someone ([~chandan.khandelwal]?) apply  [^OFBIZ-11534.patch]  and confirm it works with a file recognised by Tika, like an image (but not a JPEG2000)?

�I'm not quite quite sure about the changes in CommonServices::uploadTest and I also noticed that tests in CommonServices class are not used. Of course they can be tested by hand, but for uploadContentTest it's quite hard, if even possible. Or do I miss something?

I also noticed that the purpose for the upload is no longer required. It's because this (no longer?) works [http://svn.apache.org/viewvc?view=revision&revision=1087254]. I wonder why Sascha deleted jquery.validate.pack.js in this commit! But then is not a "required error" CSS class missing?

> Error in uploading large files
> -------------------------------
>
>                 Key: OFBIZ-11534
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-11534
>             Project: OFBiz
>          Issue Type: Bug
>          Components: ALL APPLICATIONS
>    Affects Versions: Trunk
>            Reporter: Chandan Khandelwal
>            Assignee: Jacques Le Roux
>            Priority: Major
>         Attachments: OFBIZ-11534.patch, OFBIZ-11534.patch
>
>
> There is an issue in uploading large files, I am able to upload files up to 2 GB successfully but getting an error when trying to upload a file larger than 2GB.
> Example - 
> 1. Go to party profile -
> [https://demo-trunk.ofbiz.apache.org/partymgr/control/viewprofile?partyId=admin]
> 2. Try to create a party content with a file larger than 2 GB
> 3. Error on console
> ============================
> java.lang.NegativeArraySizeException: nulljava.lang.NegativeArraySizeException: null at org.apache.commons.fileupload.disk.DiskFileItem.get(DiskFileItem.java:308) ~[commons-fileupload-1.4.jar:1.4] at org.apache.ofbiz.base.util.UtilHttp.getMultiPartParameterMap(UtilHttp.java:275) ~[main/:?] at
> =============================
>  

--
This message was sent by Atlassian Jira
(v8.3.4#803005)