Extremely big HTTP chunk size

Question

I am working in a project which we must count HTTP response size.

I am getting chunk size like:

 crlf = g_strstr_len(chunkeddata , responceSize - responceHeaderSize, "\r\n")

And than I had a simple check like sizeof chunkdata information should be 9 bytes most .

Than it turns out that wasn't enough , while counting my code crushed because big HTTP chunk size(our product recieves real traffic from a big region )

Now I am curious if that was a attack or just a mistake by server which generated http response header.

My question : Do you know any known attacks with using chunkdata size information in http responce?

Answer 1

From your vague description, I suppose that your code gathers whatever the server sends back as a big sequence of bytes, then interprets the bytes as if they were a nul-terminated character string, and then tries to find a CR+LF sequence in it.

This is wrong on many levels. First off, bytes are not necessarily a character string, and may, in particular, lack the terminating 0. Also, while the HTTP header consists in lines ending with CR+LF, and ends on an empty line (i.e. a CR+LF immediately after the previous CR+LF), nothing prevents the response body from containing CR+LF sequences as well. Finally, there is a thing called content encoding which is how the response contents are encoded into bytes. With HTTP, there are mainly four encodings for the response body:

1. No response body at all. This depends on the response type. For instance, types 204 and 304 do not have a body.

2. Explicit length. A Content-Length header specifies the length of the body, in bytes.

3. Implicit length. The body extends up to the end of the transport connection. When implicit length is used, the response cannot, by definition, be followed on the same connection with another response for another request.

4. Chunked encoding. The body is split into chunks; each chunk has its own size, given as a chunk header. This is described in section 3.6.1. There is no standard maximum size on a chunk. Chunked encoding also supports a "trailer", i.e. additional HTTP header lines after the data.

The bottom-line is that you should not try to find CR+LF sequences directly in some bytes. What you should do is the following:

• Use an HTTP library which yields the response as a stream; e.g. LibSoup (I suppose, from your use of g_strstr_len(), that you use Glib; LibSoup is a GNOME HTTP library designed to work well with Glib).
• Read the response data, as provided by the library, as small chunks (e.g. by chunks of at most 8192 bytes, a very traditional value). Don't bother thinking about "chunks" as seen on the wire; this is the job of the HTTP library, not yours.
• Count the bytes as you read them. I suppose you want to accumulate these bytes somewhere in RAM, e.g. in an array, for further processing. Be sure to enforce a size limit: if the response body happens to be larger than a maximum expected value (which depends on what you want), then discard further bytes and report an error.

This way, you will get the response length, and the response as well, while keeping under controlled memory size: an oversized response will not make your code allocate an oversized chunk of RAM. This will also be compatible with the various encoding types. In any case, you MUST read the complete response at some point: unless an explicit Content-Length header is included, there is no way to know the response length without reading it all; and reading the complete response is important to allow a subsequent response to use the same TCP connection.

In that sense, attacks related to chunk size work only against poorly written software who insist on allocating RAM blocks based on sizes sent by the peer and without any safeguard against "absurd" data. If you want to implement proper, not-too-vulnerable network code, you have to be somewhat defensive about the received data, which must be assumed to be hostile by default until proven benign. You must not expect the server to always send you nice ASCII characters in well-formed messages. You get bytes and you can get any byte, both in quantity and contents. Using a library designed for HTTP will abstract away encoding issues, thus doing half of this defensive job.