reading entity bytes from a HttpEntityEnclosingRequest in case of large content size

[Brijesh Deo]

Hi,

 

I am implementing an http reverse proxy, somewhat on the lines of the
sample implementation - ElementalReverseProxy with some modifications. I
am using HttpCore classes to implement the http server (using
org.apache.http.protocol.HttpService) and the HttpClient api (instead of
the HttpRequestExecutor) to implement the client side for sending the
Http requests to the target servers and receiving Http Responses from
them. I am using the latest HttpClient 4.0 binaries bundle (with
dependencies).

 

I have a requirement to store Http Post data from some webpages
(accessed through my reverse proxy) when they are submitted and later
Post the stored data to a different target server or the same target
server. For this, I read the entity data into a byte array whenever a
HttpPost request comes into my http server. I can store the byte array
into a file/database table. Then I recreate a ByteArrayEntity from the
saved byte array and re-populate the HttpPost request object and send it
to the intended target server using the HttpClient instance. This works
fine when the content type is "application/x-www-form-urlencoded" or
even "multipart/form-data".

 

However, in case of "multipart/form-data", if the content size is huge
(in cases of larger file uploads > 50MB) I have some doubts/concerns
whether I will be able to read all the entity data completely. I am
reading the entity data in the way shown below: My questions are
mentioned below.

 

if (request instanceof HttpEntityEnclosingRequest) {

      HttpEntity entity =
((HttpEntityEnclosingRequest)request).getEntity();

      byte entityBytes[] = null;          

      if (entity != null) {

ByteArrayOutputStream baos = new ByteArrayOutputStream() ;

            entity.writeTo(baos);

            entityBytes = baos.toByteArray() ;

            baos.flush();

            baos.close();

            entity.consumeContent();

            Question 1: Does calling consumeContent() here close the
underlying socket connection with the client(browser)? In that case, how
would the HttpResponse be sent back to the client(browser) using this
connection?

      }

if (entityBytes != null) {

            ByteArrayEntity byteArrayEntity = new
ByteArrayEntity(entityBytes);

 
((HttpEntityEnclosingRequest)request).setEntity(byteArrayEntity);

      }

}

 

Question 2: As mentioned above, if it is a large file upload request
(multipart/form-data), is it guaranteed that I will be able to read all
the bytes of the file being uploaded with the above approach?

 

Question 3: Is there any possibility that I could receive more than one
HttpRequest object(s) for such a (multipart/form-data) post with the
bytes of the uploaded file contained partially in each of those request
object entities? If that is a possibility, how can I associate/append
all such request entities to form a single piece of byte array?

 

Question 4: Is there a more efficient method to achieve this using some
other classes/methods of the HttpCore and HttpClient apis?

 

I would be thankful for any guidance or help here.

 

Thanks,

Brijesh

 

 

[olegk]

On Sun, Aug 16, 2009 at 11:49:20PM +0530, Brijesh Deo wrote:

> Hi,
>
>  
>
> I am implementing an http reverse proxy, somewhat on the lines of the
> sample implementation - ElementalReverseProxy with some modifications. I
> am using HttpCore classes to implement the http server (using
> org.apache.http.protocol.HttpService) and the HttpClient api (instead of
> the HttpRequestExecutor) to implement the client side for sending the
> Http requests to the target servers and receiving Http Responses from
> them. I am using the latest HttpClient 4.0 binaries bundle (with
> dependencies).
>
>  
>
> I have a requirement to store Http Post data from some webpages
> (accessed through my reverse proxy) when they are submitted and later
> Post the stored data to a different target server or the same target
> server. For this, I read the entity data into a byte array whenever a
> HttpPost request comes into my http server. I can store the byte array
> into a file/database table. Then I recreate a ByteArrayEntity from the
> saved byte array and re-populate the HttpPost request object and send it
> to the intended target server using the HttpClient instance. This works
> fine when the content type is "application/x-www-form-urlencoded" or
> even "multipart/form-data".
>
>  
>
> However, in case of "multipart/form-data", if the content size is huge
> (in cases of larger file uploads > 50MB) I have some doubts/concerns
> whether I will be able to read all the entity data completely. I am
> reading the entity data in the way shown below: My questions are
> mentioned below.
>
>  
>
> if (request instanceof HttpEntityEnclosingRequest) {
>
>       HttpEntity entity =
> ((HttpEntityEnclosingRequest)request).getEntity();
>
>       byte entityBytes[] = null;          
>
>       if (entity != null) {
>
> ByteArrayOutputStream baos = new ByteArrayOutputStream() ;
>
>             entity.writeTo(baos);
>
>             entityBytes = baos.toByteArray() ;
>
>             baos.flush();
>
>             baos.close();
>
>             entity.consumeContent();
>
>             Question 1: Does calling consumeContent() here close the
> underlying socket connection with the client(browser)? In that case, how
> would the HttpResponse be sent back to the client(browser) using this
> connection?
>

No, it does not. Actually in this particular case entity#consumeContent() will
have no effect as entity#writeTo will have fully consumed the entity content.

>       }
>
> if (entityBytes != null) {
>
>             ByteArrayEntity byteArrayEntity = new
> ByteArrayEntity(entityBytes);
>
>  
> ((HttpEntityEnclosingRequest)request).setEntity(byteArrayEntity);
>
>       }
>
> }
>
>  
>
> Question 2: As mentioned above, if it is a large file upload request
> (multipart/form-data), is it guaranteed that I will be able to read all
> the bytes of the file being uploaded with the above approach?
>
>

Yes, it is

>
> Question 3: Is there any possibility that I could receive more than one
> HttpRequest object(s) for such a (multipart/form-data) post with the
> bytes of the uploaded file contained partially in each of those request
> object entities? If that is a possibility, how can I associate/append
> all such request entities to form a single piece of byte array?
>
>

I seriously doubt that, but it may depend on the HTTP client.
 
>
> Question 4: Is there a more efficient method to achieve this using some
> other classes/methods of the HttpCore and HttpClient apis?
>
>

If you need to create a presistent copy of each request you should seriously
consider streaming content directly from the client request to a file or a
database BLOB without buffering it in memory and then streaming it out from the
file or the BLOB to the target server.
 
Hope this helps

Oleg

>
> I would be thankful for any guidance or help here.
>
>  
>
> Thanks,
>
> Brijesh
>
>  
>
>  
>

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@...
For additional commands, e-mail: httpclient-users-help@...

[cllai]

hi, i m working on same type of system as u. i have no idea how i can redirect my uploaded file in HttpClient to target server. And how the target server store the file into hardisk?