Best practise to add SOCKS proxy support?

[Peter Paul]

Hello,
I'm about add SOCKS proxy support to an application that uses
HttpClient and it's HttpProxy capabilities. While Http Proxy support is
given by the HttpClient API, I could not find said thing for SOCKS, so
I found that it would be nessesary to implement it via
ProtocolSocketFactory.
However, the application in question can has several open connections
that even may go through different proxies. There is a HttpClient
object for every connection. So I did a ProxySocketFactory class and
specified a custom proxy protocol

Protocol myProxy = new Protocol("http", new ProxySocketFactory(proxy),
80);

How can I get the HttpClient object to use this protocol by default?
There is a default protocol in HttpClient's HostConfiguration object,
but it only provides a getProtocol method.

So maybe there is a better way on how to get an application support
SOCKS as well? I have not figured out yet how to get authentification
either.

I'm looking foreward hearing from you
Peter


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@...
For additional commands, e-mail: httpclient-users-help@...

[olegk]

On Sun, Aug 23, 2009 at 11:57:56PM +0200, Peter Paul wrote:

> Hello,
> I'm about add SOCKS proxy support to an application that uses
> HttpClient and it's HttpProxy capabilities. While Http Proxy support is
> given by the HttpClient API, I could not find said thing for SOCKS, so
> I found that it would be nessesary to implement it via
> ProtocolSocketFactory.
> However, the application in question can has several open connections
> that even may go through different proxies. There is a HttpClient
> object for every connection. So I did a ProxySocketFactory class and
> specified a custom proxy protocol
>
> Protocol myProxy = new Protocol("http", new ProxySocketFactory(proxy),
> 80);
>
> How can I get the HttpClient object to use this protocol by default?
> There is a default protocol in HttpClient's HostConfiguration object,
> but it only provides a getProtocol method.
>

Use static Protocol#register method


> So maybe there is a better way on how to get an application support
> SOCKS as well? I have not figured out yet how to get authentification
> either.
>

As far as I know SOCK proxying works on the java.net.Socket level and therefore
should be absolutely transparent for the HTTP layer.

Oleg

> I'm looking foreward hearing from you
> Peter
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@...
> For additional commands, e-mail: httpclient-users-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@...
For additional commands, e-mail: httpclient-users-help@...

[Peter Paul]

On Mon, 24 Aug 2009 10:35:09 +0200
Oleg Kalnichevski <olegk@...> wrote:

> On Sun, Aug 23, 2009 at 11:57:56PM +0200, Peter Paul wrote:
> > Hello,
> > I'm about add SOCKS proxy support to an application that uses
> > HttpClient and it's HttpProxy capabilities. While Http Proxy
> > support is given by the HttpClient API, I could not find said thing
> > for SOCKS, so I found that it would be nessesary to implement it via
> > ProtocolSocketFactory.
> > However, the application in question can has several open
> > connections that even may go through different proxies. There is a
> > HttpClient object for every connection. So I did a
> > ProxySocketFactory class and specified a custom proxy protocol
> >
> > Protocol myProxy = new Protocol("http", new
> > ProxySocketFactory(proxy), 80);
> >
> > How can I get the HttpClient object to use this protocol by default?
> > There is a default protocol in HttpClient's HostConfiguration
> > object, but it only provides a getProtocol method.
> >
>
> Use static Protocol#register method

Do you mean I should override the default HTTP protocol with that? But
then all HttpClients will use the same proxy - that's the reason I
can't just set the proxy of the java enviroment.
The idea behind having multiple HttpClients in this application is that
they can use of different proxies (HTTP and SOCKS alike), so there
might be two connections using SOCKS proxy A, a third one to use HTTP
proxy B and another one using SOCKS proxy C at the same time.
I don't see how I can accomplish this using Proxy.registerProtocol()

> > So maybe there is a better way on how to get an application support
> > SOCKS as well? I have not figured out yet how to get
> > authentification either.
> >  
>
> As far as I know SOCK proxying works on the java.net.Socket level and
> therefore should be absolutely transparent for the HTTP layer.

Well, that's what the SocketFactory is for, but how can I get an
HttpClient to always use this SocketFactory?

> Oleg

regards

Peter

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@...
For additional commands, e-mail: httpclient-users-help@...

[olegk]

On Tue, Aug 25, 2009 at 01:25:21AM +0200, Peter Paul wrote:

> On Mon, 24 Aug 2009 10:35:09 +0200
> Oleg Kalnichevski <olegk@...> wrote:
>
> > On Sun, Aug 23, 2009 at 11:57:56PM +0200, Peter Paul wrote:
> > > Hello,
> > > I'm about add SOCKS proxy support to an application that uses
> > > HttpClient and it's HttpProxy capabilities. While Http Proxy
> > > support is given by the HttpClient API, I could not find said thing
> > > for SOCKS, so I found that it would be nessesary to implement it via
> > > ProtocolSocketFactory.
> > > However, the application in question can has several open
> > > connections that even may go through different proxies. There is a
> > > HttpClient object for every connection. So I did a
> > > ProxySocketFactory class and specified a custom proxy protocol
> > >
> > > Protocol myProxy = new Protocol("http", new
> > > ProxySocketFactory(proxy), 80);
> > >
> > > How can I get the HttpClient object to use this protocol by default?
> > > There is a default protocol in HttpClient's HostConfiguration
> > > object, but it only provides a getProtocol method.
> > >
> >
> > Use static Protocol#register method
>
> Do you mean I should override the default HTTP protocol with that? But
> then all HttpClients will use the same proxy - that's the reason I
> can't just set the proxy of the java enviroment.
> The idea behind having multiple HttpClients in this application is that
> they can use of different proxies (HTTP and SOCKS alike), so there
> might be two connections using SOCKS proxy A, a third one to use HTTP
> proxy B and another one using SOCKS proxy C at the same time.
> I don't see how I can accomplish this using Proxy.registerProtocol()
>
> > > So maybe there is a better way on how to get an application support
> > > SOCKS as well? I have not figured out yet how to get
> > > authentification either.
> > >  
> >
> > As far as I know SOCK proxying works on the java.net.Socket level and
> > therefore should be absolutely transparent for the HTTP layer.
>
> Well, that's what the SocketFactory is for, but how can I get an
> HttpClient to always use this SocketFactory?
>
> > Oleg
>
> regards
>
> Peter
>

You should be using just a single instance of HttpClient, multiple
HostConfiguration objects (one per request), and (very important) relative
request URIs.

Alternatively, you should consider upgrading to HttpClient 4.0 which has a
significantly better API.

Hope this helps.

Oleg


> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@...
> For additional commands, e-mail: httpclient-users-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@...
For additional commands, e-mail: httpclient-users-help@...

[Peter Paul]

On Tue, 25 Aug 2009 12:06:04 +0200
Oleg Kalnichevski <olegk@...> wrote:

> On Tue, Aug 25, 2009 at 01:25:21AM +0200, Peter Paul wrote:
>
> You should be using just a single instance of HttpClient, multiple
> HostConfiguration objects (one per request), and (very important)
> relative request URIs.
>
> Alternatively, you should consider upgrading to HttpClient 4.0 which
> has a significantly better API.
>
> Hope this helps.

Thank you, but it's not my project, in fact, it's an OpenSource project
and the application would only be usefull for me if it has SOCKS
support, so I thought of adding it. I have not designed the underlying
architecture, so I intended to change as few as possible about it.
If I had found a way to easily support multiple SOCKS proxies in
HttpClient 4.0, I would have put effort in porting the application to
this, but I have neither found appropriate methods in the 4.0 api nor
in the HostConfiguration object.
Could you please clearify what you meant?

Thank you

Peter
 
> Oleg


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@...
For additional commands, e-mail: httpclient-users-help@...

[olegk]

Peter Paul wrote:

> On Tue, 25 Aug 2009 12:06:04 +0200
> Oleg Kalnichevski <olegk@...> wrote:
>
>> On Tue, Aug 25, 2009 at 01:25:21AM +0200, Peter Paul wrote:
>>
>> You should be using just a single instance of HttpClient, multiple
>> HostConfiguration objects (one per request), and (very important)
>> relative request URIs.
>>
>> Alternatively, you should consider upgrading to HttpClient 4.0 which
>> has a significantly better API.
>>
>> Hope this helps.
>
> Thank you, but it's not my project, in fact, it's an OpenSource project
> and the application would only be usefull for me if it has SOCKS
> support, so I thought of adding it. I have not designed the underlying
> architecture, so I intended to change as few as possible about it.
> If I had found a way to easily support multiple SOCKS proxies in
> HttpClient 4.0, I would have put effort in porting the application to
> this, but I have neither found appropriate methods in the 4.0 api nor
> in the HostConfiguration object.
> Could you please clearify what you meant?
>

(1) HttpClient 3.1

HttpClient client = new HttpClient();

ProtocolSocketFactory mysf = new DefaultProtocolSocketFactory();
Protocol myhttp = new Protocol("http", mysf, 80);
HostConfiguration hostconf = new HostConfiguration();
hostconf.setHost("targethost", 80, myhttp);
HttpState state = new HttpState();

GetMethod httpget = new GetMethod("/use/relative/uris/only");
try {
     client.executeMethod(hostconf, httpget, state);

     if (httpget.getStatusCode() == HttpStatus.SC_OK) {
         System.out.println(httpget.getResponseBodyAsString());
     } else {
       System.out.println("Unexpected failure: " +
httpget.getStatusLine().toString());
     }
} finally {
     httpget.releaseConnection();
}

(2) HttpClient 4.0

Use custom ClientConnectionOperator

http://hc.apache.org/httpcomponents-client/tutorial/html/connmgmt.html#d4e552

Oleg

> Thank you
>
> Peter
>  
>> Oleg
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@...
> For additional commands, e-mail: httpclient-users-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@...
For additional commands, e-mail: httpclient-users-help@...

[Peter Paul]

On Sat, 29 Aug 2009 11:45:50 +0200
Oleg Kalnichevski <olegk@...> wrote:

> (1) HttpClient 3.1
>
> HttpClient client = new HttpClient();
>
> ProtocolSocketFactory mysf = new DefaultProtocolSocketFactory();
> Protocol myhttp = new Protocol("http", mysf, 80);
> HostConfiguration hostconf = new HostConfiguration();
> hostconf.setHost("targethost", 80, myhttp);
> HttpState state = new HttpState();
>
> GetMethod httpget = new GetMethod("/use/relative/uris/only");
> try {
>      client.executeMethod(hostconf, httpget, state);
>
>      if (httpget.getStatusCode() == HttpStatus.SC_OK) {
>          System.out.println(httpget.getResponseBodyAsString());
>      } else {
>        System.out.println("Unexpected failure: " +
> httpget.getStatusLine().toString());
>      }
> } finally {
>      httpget.releaseConnection();
> }

Thank you, to approach your proposed solution, I thought I gould change
the URI of the HttpMethod and use the setHost function of
HostConfiguration - however, I encountered some trouble when changing
absolute URIs to relative ones (I don't want to change the Methods
initial creation as they happen all over the place, so I've just
written a function to replace the client.execute(method) calls:

private void processHttpMethod(HttpMethod method) throws IOException {
    if(protocol != null ) {
        try {
            /* We set host and our custom protocol */
            client.getHostConfiguration().setHost(method.getURI().getHost(),
80, protocol);        

            System.out.println("Path: "+method.getURI().getPath());

            URI relPath = null;

            /* create a relative URI */
            try {
                relPath = new URI(method.getURI().getPath(), false);
            } catch (Exception e) { System.out.println("Can't build
            URI: "+e); }

            if(!relPath.isRelPath())
                System.out.println("URI is not relative (it should
            be)");

            method.setURI(relPath);

            System.out.println("URI: "+method.getURI());

        } catch (URIException e) {
            System.err.println("Failed to parse URI: "+e);
        }
    } else
        System.out.println("Using default protocol");

    System.out.println(client.getHost());
    client.executeMethod(method); // use state and hostconfig of the
    client object
}

This however fails building the relative URI:

Path: /files/131745342/foobar.tar.gz <- should be the URI
URI is not relative (it should be)
URI: http://myhost.example.com/files/131745342/foobar.tar.gz
myhost.example.com <- host is ok, but seems like the absolute method's
URI overwrites it as well as the protocol.
 

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@...
For additional commands, e-mail: httpclient-users-help@...

[olegk]

On Mon, Aug 31, 2009 at 06:59:05PM +0200, Peter Paul wrote:

> On Sat, 29 Aug 2009 11:45:50 +0200
> Oleg Kalnichevski <olegk@...> wrote:
>
> > (1) HttpClient 3.1
> >
> > HttpClient client = new HttpClient();
> >
> > ProtocolSocketFactory mysf = new DefaultProtocolSocketFactory();
> > Protocol myhttp = new Protocol("http", mysf, 80);
> > HostConfiguration hostconf = new HostConfiguration();
> > hostconf.setHost("targethost", 80, myhttp);
> > HttpState state = new HttpState();
> >
> > GetMethod httpget = new GetMethod("/use/relative/uris/only");
> > try {
> >      client.executeMethod(hostconf, httpget, state);
> >
> >      if (httpget.getStatusCode() == HttpStatus.SC_OK) {
> >          System.out.println(httpget.getResponseBodyAsString());
> >      } else {
> >        System.out.println("Unexpected failure: " +
> > httpget.getStatusLine().toString());
> >      }
> > } finally {
> >      httpget.releaseConnection();
> > }
>
>
> Thank you, to approach your proposed solution, I thought I gould change
> the URI of the HttpMethod and use the setHost function of
> HostConfiguration - however, I encountered some trouble when changing
> absolute URIs to relative ones (I don't want to change the Methods
> initial creation as they happen all over the place, so I've just
> written a function to replace the client.execute(method) calls:
>
> private void processHttpMethod(HttpMethod method) throws IOException {
>     if(protocol != null ) {
>         try {
>             /* We set host and our custom protocol */
>             client.getHostConfiguration().setHost(method.getURI().getHost(),
> 80, protocol);        
>
>             System.out.println("Path: "+method.getURI().getPath());
>
>             URI relPath = null;
>
>             /* create a relative URI */
>             try {
>                 relPath = new URI(method.getURI().getPath(), false);
>             } catch (Exception e) { System.out.println("Can't build
>             URI: "+e); }
>
>             if(!relPath.isRelPath())
>                 System.out.println("URI is not relative (it should
>             be)");
>
>             method.setURI(relPath);
>
>             System.out.println("URI: "+method.getURI());
>
>         } catch (URIException e) {
>             System.err.println("Failed to parse URI: "+e);
>         }
>     } else
>         System.out.println("Using default protocol");
>
>     System.out.println(client.getHost());
>     client.executeMethod(method); // use state and hostconfig of the
>     client object
> }
>
> This however fails building the relative URI:
>
> Path: /files/131745342/foobar.tar.gz <- should be the URI
> URI is not relative (it should be)
> URI: http://myhost.example.com/files/131745342/foobar.tar.gz
> myhost.example.com <- host is ok, but seems like the absolute method's
> URI overwrites it as well as the protocol.
>

This is the expected behaviour. You can override it, though, by using this
workaround:

http://svn.apache.org/repos/asf/httpcomponents/oac.hc3x/trunk/src/contrib/org/apache/commons/httpclient/contrib/ssl/HostConfigurationWithStickyProtocol.java

Having said that, consider upgrading to 4.0 which has a massively better /
cleaner API.

Oleg

 
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@...
> For additional commands, e-mail: httpclient-users-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@...
For additional commands, e-mail: httpclient-users-help@...

[Peter Paul]

On Tue, 1 Sep 2009 17:54:42 +0200
Oleg Kalnichevski <olegk@...> wrote:

> This is the expected behaviour. You can override it, though, by using
> this workaround:
>
> http://svn.apache.org/repos/asf/httpcomponents/oac.hc3x/trunk/src/contrib/org/apache/commons/httpclient/contrib/ssl/HostConfigurationWithStickyProtocol.java
>

Thank you, this is exactly what I was looking for.

Peter


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@...
For additional commands, e-mail: httpclient-users-help@...