Java HTTP Client - Configuration

Here is an index to configuration topics, some of which are on this page, others having their own page:

This gives instructions for setting the parameters for the Java HTTP client. All of the methods shown are on com.oaklandsw.http.HttpURLConnection. In general, the setDefault... methods are static and apply to the class, and the set... methods apply to an instance.

Many of the configuration parameters below can also be set using the system properties, see the Javadoc for HttpURLConnection for those details.

Note: All of the timeout values are specified in milliseconds. 1000 milliseconds equals one second.


Connection Timeout - set[Default]ConnectionTimeout()

The connection timeout mechanism allows you to limit the amount of time you wait for the connection to be established with the server. When you make an HTTP request, the HTTP client may immediately attempt to create a TCP connection with the server, or if the maximum number of connections to the given server has been reached, the HTTP client will wait for an existing connection to become available.

This timeout applies both to the time it takes an underlying socket connection to be established, and to the time that you have to wait to get a currently established connection.

If you are using JDE 1.4 or higher, this timeout is the value set as the timeout value when setting up the socket (in the Socket.connect() method). For releases prior to JRE 1.4, the connection timeout is handled using a timeout thread.


Request Timeout - set[Default]RequestTimeout()

The request timeout is the amount of time from when the request is initiated until the response has been read. This includes all of the time to establish the connection.


Idle Connection Timeout - set[Default]IdleTimeout()

Most TCP connections that support HTTP requests service more than one request in sequence and are therefore kept alive after a request is finished. Most HTTP servers kill these idle connections after a specific interval of inactivity. The HTTP client can also terminate the idle connection if no request has been sent on it for a specified period of time.

When an idle connection is terminated by the server, the client does not get any notification of this fact, and it is detected the next time an HTTP request is sent on the connection. This is undesirable, because that HTTP request will either fail or have to be retried.

Because of this, it is generally desirable to kill idle connections in the HTTP client just before they are killed by the server. The default for the idle connection timeout is set to 14 seconds, which is one second less than the default of the idle connection timeout of the Apache web server.


Idle Connection Ping - set[Default]IdlePing()

There are times when you cannot be sure of the idle connection timeout value in the server, and may therefore have to risk sending an HTTP request on a connection that might have been killed by the server. In the case of an HTTP GET, this is not a problem, because if it is sent on a dead connection it can automatically be retried.

However, an HTTP POST can never be retried because the semantics of the HTTP POST require it to be sent only once. To ensure the best chance of success for a POST, we can send out a "ping" message on the connection just before the POST request, which will either detect a dead connection, or keep the connection alive long enough to send the POST successfully.

If you can't reliably use the idle connection timeout parameter (because you don't know the server environment), you should set this value to a relatively small value like 1000 milliseconds (one second).


Request Retry - setTries()

This is the number of times an idempotent HTTP request is attempted to be sent before giving up. An idempotent HTTP request is any HTTP request except for POST. This parameter is also the number of attempts to send the idle connection ping before a POST request.

The default value for this is 3, which is one greater than the default maximum number of connections for a host/port. If you want to guarantee that a request will be sent on a working connection, set this value to one greater than the maximum number of connections per host/port. Doing this insures that even if all of the existing connections between the host/port are dead (and they are all tried), one new more connection will be created to allow the request to be sent.


Maximum Connections per Host/Post - setMaxConnectionsPerHost()

This is the maximum number of connections to be kept alive for any host/port combination. It is a global parameter which applies to all host/port combinations. The default is 2, as this is what is recommended for use on the Internet in RFC 2616 section 8.1.4. However, if you are using the HTTP client in certain high throughput applications, a much higher value might be appropriate. In setting this value, be sure and consider the Tries value.


Proxy - setProxyHost/Port(), setConnectionProxyHost/Port()

In addition to respecting all of the Java APIs and properties (see the HttpURLConnection documentation) related to setting the proxy, the setConnectionProxyHost/Port() allows you to set a proxy for a specific connection.


Authentication

Basic, Digest and NTLM authentication are supported. In order to use these authentication modes, implement an instance of the HttpUserAgent class and call the HttpURLConnection.set[Default]UserAgent() method to tell us about your HttpUserAgent. Your HttpUserAgent will provide the authentication credential (username and password) to be supplied when authentication is required.

If you already know that you will require authentication, it's better to use the HttpURLConnection.setAuthenticationType() method to indicate the authentication type (Basic, Digest or NTLM) that is preferred. If you do this, the authentication process will begin preemptively when the request is sent out, rather than waiting for an authentication required (401/407) response from the server.

When when using the output streaming or pipelining features, setting the authentication type beforehand is essential, as this allows streaming or pipelining to progress only after authentication has completed. This is particularly important for NTLM authentication which is connection based. Once the underlying socket connection is authenticated with NTLM, any HTTP request can be executed on it (using the same credential) and it requires no further authentication.


Authentication Credential Caching

By default, when using authentication for connections, each host/port (or proxy host/port) is associated with a single set of credentials that is requested using the HttpUserAgent only once. After being initially requested (and assuming the credentials are valid), any subsequent HttpURLConnections on that host/port (or proxy host/port) will use the same credentials. Doing this allows the HttpUserAgent to be directly hooked to a user interface that asks the user to specify the credentials for a given host and port.

There are some situations where you may wish to have different sets of authentication credentials for connections going to the same host/port. You can do this by calling HttpURLConnection.setMultiCredentialsPerAddress(true). If you do this, there will be many calls to HttpUserAgent as this is necessary to get the desired credentials each HttpURLConnection in order to find the correct socket connection in the connection pool.


SOCKS Proxy Support

The JRE supports SOCKS socket connections. To use a SOCKS connection with the HTTP client, you must create a custom socket factory which creates the correct type of Socket.

Make a subclass of AbstractSocketFactory to create the SOCKS socket, and associate the socket factory with the HTTP client using the HttpURLConnection.set[Default]SocketFactory() method.

You can set the SOCKS configuration either using the socksProxyHost/Port system properties, or in JRE 1.5 or later, open the socket using a Proxy.SOCKS constant as the first argument of the Socket() constructor.


Streaming the Request

By default, any data sent with the request is buffered prior to being sent, which results in a double copy of the data. This can be avoided by the use of the HttpURLConnection.setChunkedStreamingMode() and setFixedLengthStreamingMode() methods.

If you are using authentication, you should also set the authentication type (Basic, Digest, or NTLM). When this is used with streaming, it will complete the authentication process before streaming the request. If you don't set the authentication type in advance, the request will fail with an HttpRetryException in the same manner as documented with in the JRE, because the request for authentication from the server will interrupt the streaming output.


HTTP Failover

In some applications, it might be critical to get a response right away, and there are many different servers available to handle the request. In this case, the use of the connection timeout, and request timeout are recommended. Set these to relatively small values, and in the event of a timeout, you create a new HTTP connection to a different server. Alternatively, you can use a different proxy, and set the proxy using the setConnectionProxyHost/Port() APIs.


Using the FTP Protocol Through a Proxy

Most proxy servers support the FTP protocol for the purpose of accessing files or directories. You can use this simply by specifying "ftp:" as the protocol instead of "http:" in your URL, and of course specify the proxy server using the usual means.

However, you must use the com.oaklandsw.http.HttpURLConnection.openConnection() method to open the connection. If you don't, the URL provided is selected based on the protocol and you will not get the Oakland Software implementation.


Streaming and Tunneling Through a Proxy using CONNECT

When using a proxy server, you can use the CONNECT method to use the underlying TCP stream for an arbitrary purpose through the proxy. This is called HTTP Tunneling.

When using the CONNECT method, you can stream using both the HttpURLConnection's input and output streams by calling HttpURLConnection.setTunneledStreamingMode(). When you do this, you establish the connection which creates the tunnel. You then read/write from/to the input/output streams associated with the connection. When you close the input stream, the connection is released.

Here is some sample code to illustrate: 

  
        // FTP
        URL url = new URL("http://hostname:21");
        HttpURLConnection urlCon = HttpURLConnection.openConnection(url);
        urlCon.setRequestMethod("CONNECT");
        urlCon.setDoOutput(true);
        urlCon.setDoInput(true);
        urlCon.setTunneledStreamingMode(true);

        OutputStream outStr = urlCon.getOutputStream();

        int response = urlCon.getResponseCode();
        assertEquals(200, response);

        InputStream is = urlCon.getInputStream();

        // Write some stuff
        byte[] output;
        output = "USER anonymous\n".getBytes("ASCII");
        outStr.write(output);
        output = "PASS anonuser\n".getBytes("ASCII");
        outStr.write(output);
        output = "STAT\n".getBytes("ASCII");
        outStr.write(output);
        output = "QUIT\n".getBytes("ASCII");
        outStr.write(output);
        outStr.flush();
        outStr.close();

        // Read the responses, this closes the stream and releases the connection
        byte[] bytes = Util.getBytesFromInputStream(is);
        String response = new String(bytes);
        System.out.println(response);


WebDAV Support

The Java HTTP client supports all of the WebDAV methods defined by the following RFCs:


New Method Support

The Java HTTP client allows any method to be defined in addition to the built-in HTTP and WebDAV methods. This is accomplished by setting the properties associated with a method using the setMethodProperties() method on HttpURLConnection. The method properties are defined in the Javadoc as the fields of HttpURLConnection that begin with METHOD_PROP. As examples, here are the definitions of the HTTP methods: 

  
            setMethodProperties(HTTP_METHOD_GET, METHOD_PROP_RETRY
                | METHOD_PROP_REDIRECT
                | METHOD_PROP_SWITCH_TO_POST
                | METHOD_PROP_REQ_LINE_URL);
            setMethodProperties(HTTP_METHOD_POST, METHOD_PROP_REDIRECT
                | METHOD_PROP_ADD_CL_HEADER
                | METHOD_PROP_REQ_LINE_URL
                | METHOD_PROP_CALCULATE_CONTENT_LEN
                | METHOD_PROP_SEND_CONTENT_TYPE);
            setMethodProperties(HTTP_METHOD_HEAD, METHOD_PROP_RETRY
                | METHOD_PROP_REDIRECT
                | METHOD_PROP_IGNORE_RESPONSE_BODY
                | METHOD_PROP_REQ_LINE_URL);
            setMethodProperties(HTTP_METHOD_PUT, METHOD_PROP_RETRY
                | METHOD_PROP_ADD_CL_HEADER
                | METHOD_PROP_REQ_LINE_URL
                | METHOD_PROP_CALCULATE_CONTENT_LEN);
            setMethodProperties(HTTP_METHOD_OPTIONS, METHOD_PROP_RETRY
                | METHOD_PROP_REQ_LINE_STAR);
            setMethodProperties(HTTP_METHOD_DELETE, METHOD_PROP_RETRY
                | METHOD_PROP_REQ_LINE_URL);
            setMethodProperties(HTTP_METHOD_TRACE, METHOD_PROP_RETRY
                | METHOD_PROP_REQ_LINE_URL);
            setMethodProperties(HTTP_METHOD_CONNECT,
                                METHOD_PROP_IGNORE_RESPONSE_BODY
                                    | METHOD_PROP_REQ_LINE_HOST_PORT
                                    | METHOD_PROP_LEAVE_OPEN);


Support for JRE Versions 1.2 and 1.3

These versions of the JRE are supported by the HTTP client version 1.9.0. Version 1.9.0 does not include the following features:

  • Apache Axis2 (Apache Axis is supported)
  • Pipelining
  • Streaming

If you require this version, please it.