HTTP In Depth

HTTP

11 min readMar 31, 2018

The Hypertext Transfer Protocol (HTTP) is an application protocol for distributed, collaborative, and hypermedia information systems. HTTP is the foundation of data communication for the World Wide Web.

Protocol

a protocol is the special set of rules that end points in a telecommunication connection use when they communicate. Protocols specify interactions between the communicating entities.
Protocols usually consist of three main parts:-
1- Header: contains information such as source, destination, fileType and size.
2- Payload: the actual information transmitted using the protocol .
3- Footer: a control field to ensure the Payload data is transmitted free of errors.

History of HTTP

HTTP 0.9

From 1989, Tim Berners-Lee and his team at CERN , the European Nuclear Research Center in Switzerland, developed the Hypertext Transfer Protocol, together with the concepts of URL and HTML , which created the foundations of the World Wide Web. First results of this effort in 1991 were the version HTTP 0.9.

Methods supported: GET only
Response type: hypertext only
Connection nature: terminated immediately after the response
Headers: No HTTP headers (cannot transfer other content type files), No status/error codes, No URLs, No versioning
Support: HTTP/0.9 requests have been removed.
keep-alive: N/A

RequestGET /Response<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="http://any23.apache.org">here</a>.</p>
</body></html>

HTTP 1.0

In 1996, RFC 1945 became known as HTTP / 1.0. With HTTP / 1.0, a new TCP connection is established before each request and, after the response has been transmitted.

Methods supported: GET , HEAD , POST
Response type: not limited to hypertext (Content-Type header provided ability to transmit files other than plain HTML files — e.g. scripts, stylesheets, media)
Connection nature: terminated immediately after the response
Headers: does not officially required
Support: used by proxies, wget and some IE browsers.
keep-alive: not considered persistent unless a keep-alive header is included.
status code: only define up to 16 status codes.

RequestGET / HTTP/1.0ResponseHTTP/1.1 301 Moved Permanently
Date: Fri, 02 Jan 2015 05:40:17 GMT
Server: Apache/2.4.7 (Ubuntu)
Location: http://any23.apache.org
Content-Length: 231
Connection: close
Content-Type: text/html; charset=iso-8859-1<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="http://any23.apache.org">here</a>.</p>
</body></html>

so we can see HTTP 0.9 doesn’t support headers and HTTP 1.0 supports headers (not officially required).

HTTP Headers

HTTP headers allow the client and the server to pass additional information with the request or the response.
Headers can be grouped according to their contexts or according to how proxies handle them, read more

Major Problem in both HTTP/0.9 and HTTP/1.0

both are required to open new TCP connection per outstanding Request and it closes it immediately after the response was sent.
to open TCP connection a three-way handshake should occur which takes time.

TCP 3-Way Handshake (SYN,SYN-ACK,ACK)

A three-way handshake is primarily used to create a TCP socket connection. It works when:
1- A client node sends a SYN data packet over an IP network to a server on the same or an external network. The objective of this packet is to ask/infer if the server is open for new connections.
2- The target server must have open ports that can accept and initiate new connections. When the server receives the SYN packet from the client node, it responds and returns a confirmation receipt — the ACK packet or SYN/ACK packet.
3- The client node receives the SYN/ACK from the server and responds with an ACK packet.

Upon completion of this process, the connection is created and the host and server can communicate.

HTTP 1.1

In 1999, RFC 2616 , reflects the HTTP / 1.1 standards, With HTTP / 1.1 a client can use an additional header entry ( keep-alive ) to express a desire not to disconnect in order to be able to use the connection again (persistent connection).

Methods supported: GET , HEAD , POST , PUT , DELETE , TRACE , OPTIONS
Connection nature: long-lived
Support: common use until today.
keep-alive: considered persistent unless declared otherwise.
status code: came with 24 status codes.

The Key Features Of HTTP/1.1

1- HTTP pipelining: is a technique in which multiple HTTP requests are sent on a single TCP connection without waiting for the corresponding responses.

HTTP pipelining is not enabled by default in modern browsers due to proxy servers, Head-of-line blocking (HOL) .

2- HTTP persistent connection (HTTP keep-alive) :

is the idea of using a single TCP connection to send and receive multiple HTTP requests/responses, as opposed to opening a new connection for every single request/response pair.

3- Compression/Decompression (Content Encoding) : a capability that can be built into web servers and web clients to improve transfer speed and bandwidth utilization, The most common compression schemes include gzip and Deflate.

How does it work?

The web client advertises which compression schemes it supports by including a list of tokens in the HTTP request.

GET /encrypted-area HTTP/1.1
Host: www.example.com
Accept-Encoding: gzip, deflate

If the server supports compression scheme the server will add a Content-Encoding or Transfer-Encoding field in the HTTP response

HTTP/1.1 200 OK
Date: mon, 26 June 2016 22:38:34 GMT
Server: Apache/1.3.3.7 (Unix)  (Red-Hat/Linux)
Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT
Accept-Ranges: bytes
Content-Length: 438
Connection: close
Content-Type: text/html; charset=UTF-8
Content-Encoding: gzip

4- New Status Code: a new header code 100 — Continue, this helps client to know if a large request will be processed by server and if server responds with status code 100, client sends the actual body request to the server for further processing. Apart from this, there are 409-Conflict and 410-Gone status code also added.

5- Content Negotiation: serve different versions of a document (or more generally, representations of a resource) at the same URI, so that user agents can specify which version fits their capabilities the best. One classical use of this mechanism is to serve an image in GIF or PNG format so that a browser that cannot display PNG images (e.g. MS Internet Explorer 4) will be served the GIF version.

6-Warning Header: used to carry additional information about the status or transformation of a message which might not be reflected in the message. This information is typically used to warn about a possible lack of semantic transparency from caching operations or transformations applied to the entity body of the message.

The Performance Issues Of HTTP/1.1

1-Head-of-line blocking(HOL blocking)

It as in networking is a performance issue that occurs when a bunch of packets is blocked by the first (a large or slow ) packet in line . It can happen especially in input buffered network switches where out-of-order delivery of packets can occur (arrive out of order). A switch can be composed of buffered input ports, a switch fabric and buffered output ports. If first-in-first-out (FIFO) input buffers are used, only the oldest packet is available for forwarding.

2-The repetition of information sent for each request

Each of the requests repeats a lot of the same information. Each includes the same user-agent information and accepted content information, etc.

3-The additional round trips(3-Way Handshake) needed.

SPDY

SPDY is a protocol developed by Google to increase the speed and efficiency of delivering web content. SPDY modifies parts of the HyperText Transfer Protocol (HTTP) to improve web performance.

HTTP Problems that SPDY Solves

Single request per connection: HTTP can only fetch one resource at a time, leading to delays and underutilized bandwidth. SPDY allows for many concurrent downloads.
Client-initiated requests: HTTP requires the user to request content from a server before it can be delivered. With SPDY, the server can “push” data to the client without having to wait for a request, making it possible to load web content before it’s needed.
Redundant Headers: HTTP headers define the behaviour of an HTTP transaction. In some cases, the same headers are repeated over the course of a session. SPDY removes unnecessary headers to reduce the amount of bandwidth required.
Uncompressed Data: Compression, which shrinks the size of data during delivery, is optional for HTTP. SPDY forces compression for all communications including headers.

BENEFITS OF SPDY

Users see faster load times and lower bandwidth usage: Bandwidth is utilized more intelligently, allowing users to stretch data caps while experiencing improved performance.
Enterprises see higher customer satisfaction: Web pages load faster, resulting in a better user experience.
Enterprises also see lower bandwidth costs: SPDY optimizes traffic before it leaves the server, allowing enterprises to utilize more of their network without having to upgrade their hardware.

HTTP/2 replaces SPDY
Chrome drops support in 2016

HTTP/2 — A protocol for greater performance

HTTP/2 was based largely on Google’s own protocol SPDY which had many of the same features found in HTTP/2 and managed to improve data transmission while keeping backwards compatibility. SPDY had already proven many of the concepts used in HTTP/2.

The key Features of HTTP/2

1- Binary, instead of textual

Binary protocols are more efficient to parse, more compact “on the wire”, and most importantly, they are much less error-prone, compared to textual protocols like HTTP/1.x, because they often have a number of affordances to “help” with things like whitespace handling, capitalization, line endings, blank lines and so on.

2- Streams

an independent, bi-directional sequence of frames exchanged between the client and server within an HTTP/2 connection. Streams have several important characteristics:

A single HTTP/2 connection can contain multiple concurrently open streams, with either endpoint interleaving frames from multiple streams.
Streams can be established and used unilaterally or shared by either the client or server.
Streams can be closed by either endpoint.
The order in which frames are sent on a stream is significant. Recipients process frames in the order they are received. In particular, the order of HEADERS and DATA frames is semantically significant.
Streams are identified by an integer. Stream identifiers are assigned to streams by the endpoint initiating the stream.

3- Request and Response Multiplexing

is a method in HTTP/2 by which multiple HTTP requests can be sent and responses can be received asynchronously via a single TCP connection. Multiplexing is the heart of HTTP/2 protocol.

With HTTP/1.x, if the client wants to make multiple parallel requests to improve performance, then multiple TCP connections must be used; see Using Multiple TCP Connections. This behaviour is a direct consequence of the HTTP/1.x delivery model, which ensures that only one response can be delivered at a time (response queuing) per connection. Worse, this also results in head-of-line blocking and inefficient use of the underlying TCP connection.

The new binary framing layer in HTTP/2 removes these limitations, and enables full request and response multiplexing, by allowing the client and server to break down an HTTP message into independent frames, interleave them, and then reassemble them on the other end.

4- Server Push

is the ability of the server to send multiple responses for a single client request. That is, in addition to the response to the original request, the server can push additional resources to the client, without the client having to request each one explicitly!

5- HPACK- Data compression of HTTP headers

HTTP/2 introduces a form of header compression called HPACK. HPACK allows for very efficient header compression, without being vulnerable to compression related attacks such as CRIME. HPACK provides the following:

The ability to encode large headers using a fixed Huffman encoding
The ability to encode commonly used headers as a variable length integer, rather than re-sending the whole header each time.

HTTP/1.1 Upgrade Header

The HTTP Upgrade mechanism is used to establish HTTP/2 starting from plain HTTP. The client starts a HTTP/1.1 connection and sends “Upgrade: h2c” header. If the server supports HTTP/2, it replies with HTTP 101 Switching Protocol status code.

The HTTP Upgrade mechanism is used only for cleartext HTTP2 (h2c). In the case of HTTP2 over TLS (h2), the ALPN TLS protocol extension is used instead.

Example:

curl -vs -o /dev/null — http2 nghttp2.org

* Rebuilt URL to: nghttp2.org/
* Trying 139.162.123.134…
* TCP_NODELAY set
* Connected to nghttp2.org (139.162.123.134) port 80 (#0)
> GET / HTTP/1.1
> Host: nghttp2.org
> User-Agent: curl/7.54.0
> Accept: */*
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c 
> HTTP2-Settings: AAMAAABkAARAAAAAAAIAAAAA
>
< HTTP/1.1 101 Switching Protocols
< Connection: Upgrade
< Upgrade: h2c
* Received 101
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=33
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< date: Fri, 30 Mar 2018 18:33:00 GMT
< content-type: text/html
< last-modified: Tue, 27 Feb 2018 13:31:12 GMT
< etag: “5a955da0–19d8”
< accept-ranges: bytes
< content-length: 6616
< x-backend-header-rtt: 0.001283
< server: nghttpx
< via: 2 nghttpx
< x-frame-options: SAMEORIGIN
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
<
{ [1232 bytes data]
* Connection #0 to host nghttp2.org left intact

References :

More where this came from

This story is published in Noteworthy, where thousands come every day to learn about the people & ideas shaping the products we love.

Follow our publication to see more product & design stories featured by the Journal team.