Pipelining HTTP Requests

Pipelining HTTP requests can provide performance improvements by factors of 10 or more under certain conditions. Normally, on a given TCP connection, only a single HTTP request is outstanding. If you have multiple requests, the next request is sent after the response from the previous request. With pipelining, multiple requests are sent without waiting for responses. Under certain (common) conditions, this can cause a significant performance improvement because you don't have to wait the entire round-trip time for each response.

In addition, pipelining is the only way you can batch multiple HTTP requests into fewer TCP/IP packets. This can provide a huge improvement in networking performance.

Usually pipelining is beneficial when you must read a large number of relatively small objects over a network with some overhead. For example, one test we conducted reading 1K objects from an Internet host took 102ms per object without pipelining and 19ms per object with pipelining over a single connection.


Programming

The use of pipelining requires a slightly different programming model than the standard use of the HttpURLConnection. Pipelining is implemented using a callback mechanism, so you must provide a callback object which is notified when it's time to read or write data to the HTTP request and if there is an error condition. Essentially, the code looks like this: 

    public static class SamplePipelineCallback implements Callback
    {

        public void writeRequest(com.oaklandsw.http.HttpURLConnection urlCon,
                                 OutputStream os)
        {
        	// This is used only if you are pipelining POST/PUT, which is rare
            os.write("whatever".getBytes());
            os.close();
        }

        public void readResponse(com.oaklandsw.http.HttpURLConnection urlCon,
                                 InputStream is)
        {
            try
            {
                System.out.println("Response: " + urlCon.getResponseCode());

                // Print the output stream
                InputStream inputStream = urlCon.getInputStream();
                byte[] buffer = new byte[10000];
                int nb = 0;
                while (true)
                {
                    nb = inputStream.read(buffer);
                    if (nb == -1)
                        break;
                    System.out.write(buffer, 0, nb);
                }
            }
            catch (AutomaticHttpRetryException arex)
            {
                System.out.println("Automatic retry: " + urlCon);
                // The read will be redriven
                return;
            }
            catch (IOException e)
            {
            	// This is a problem
                System.out.println("ERROR - IOException: " + urlCon);
                e.printStackTrace();
            }
        }

        public void error(com.oaklandsw.http.HttpURLConnection urlCon,
                          InputStream is,
                          Exception ex)
        {
            System.out.println("ERROR: " + urlCon);
            ex.printStackTrace();
        }
    }

    ...
    
    // The code to create and execute the HttpURLConnections

    com.oaklandsw.http.HttpURLConnection.setDefaultCallback(new SamplePipelineCallback());

    for (int i = 0; i < _times; i++)
    {
        com.oaklandsw.http.HttpURLConnection urlCon = 
        	com.oaklandsw.http.HttpURLConnection).openConnection(url);
        urlCon.setPipelining(true);
        
        // Set any desired headers or other properties on the urlCon
        
        // Queues the pipelined request for execution
        urlCon.pipelineExecute();
    }

    // Blocks until the urlCons are complete
    com.oaklandsw.http.HttpURLConnection.pipelineBlock();

The examples directory in the distribution provides a program that shows how pipelining works.


General Configuration

The basic configuration is to simply turn on pipelining (HttpURLConnection.setPipelining() or setDefaultPipelining()) for each HttpURLConnection and execute the connections using an instance of the Callback class above. The default options for pipelining will work well in most cases. The performance analysis we have done shows that pipelining with 2 connections (the default number of connections) is optimal in many cases. If you are using pipelining with with authentication (Basic, Digest or NTLM), see the authentication configuration section below for additional configuration requirements.

The pipelining mechanism guarantees that the either the Callback.readResponse() or Callback.error() method will be called at least once per executed request. The Callback.writeRequest() will only be called for requests where HttpURLConnection.doOutput() is set to true (usually HTTP POST requests). The order in which the Callback methods is called is not guaranteed to the the same order that the requests were executed. This is because pipelined requests might fail and be automatically retried, or different requests might be executed on different connections.

Pipelining can be used with any type of HTTP request, but it is best used with the idempotent requests (HTTP GET, HTTP PUT) because the likelyhood of retrying a request is greater using pipelining. By default, the number of retries is 3, after which the request fails and the failure is reported at Callback.error(). The number of retries can be changed using the HttpURLConnection.setDefaultMaxTries() method.

Important Note - While reading or writing the data associated with a pipelined request, it is possible that you might receive an AutomaticHttpRetryException. This exception indicates an error while directly reading/writing from the underlying socket stream (all pipelined I/O is done directly on the socket stream, there is no additional buffering). When you get this exception, simply discard your work with this request and exit the Callback method. In the future, the Callback method will be called again with the same request. In the event there is some error and retry is no longer possible, the Callback.error() method will be called.


Authentication Configuration

However, pipelining is easy to use and works well with either Basic or NTLM authentication, as the authentication is either preemptive (in the case of Basic), or the authentication is taken care of before the pipelining starts for the connection (in the case of NTLM). You cannot use pipelining with Digest authentication, because the nature of the authentication exchange precludes the benefits of pipelining (that is, it makes no sense to do so).

Like streaming, if you are using authentication, you must call HttpURLConnection.set[Default]AuthenticationType(). See the authentication configuration section for more details.


Advanced Configuration

In general, the first thing to configure for pipelining (if you are unhappy with the default configuration) is the maximum number of connections it will use. By default, it will create as many connections as allowed (use the HttpURLConnection.setMaxConnectionsPerHost() to control this), and pipeline requests on those connections in a round robin fashion up to the maximum pipeline depth. It is possible to alter the performance by changing the number of allowed connections. Keep in mind that the recommendation in the HTTP RFC is that you use only a maximum of 2 connections per host.

By default, the pipelining mechanism will calculate the maximum pipeline depth based on the observed lifetime of the TCP connection to a given host. You can set the max pipelining depth explicitly using the HttpURLConnection.setPipelineDepth() method.

Additional information about advanced configuration will be provided soon, as we are working on a paper showing pipelining performance in various scenarios