Ben Biddington

Whatever it is, it's not about "coding"

Posts Tagged ‘buffer

The Response.OutputStream floater

leave a comment »

Today we had to recall a release due to a message like:

Not enough memory for operation.

The error occured while trying to emit a large (~800MB) zip file via ASP.NET. Even though I knew buffering was on, I figured that because we were writing directly to HttpResponse.OutputStream that we would somehow bypass the buffering enforced by HttpResponse.

Because we were flushing HttpResponse.OutputStream regularly during processing I thought we were okay. I didn’t realise that HttpResponse.OutputStream was bound by the same buffering policy.

My theory is as follows.

Response.OutputStream.Flush does nothing

Here’s what happens when we call HttpResponse.OutputStream.Flush:

[Pseudo Stacktrace]

And HttpWriter.Flush looks like this:

// HttpWriter
public override void Flush() { }

That’s right: it’s empty. Flushing is completely ignored. I guess this makes sense, HttpResponse maintains full control of its underlying stream.

Flushing depends entirely on HttpResponse.BufferOutput

Here’s what happens when we call HttpResponse.OutputStream.Write:

[Pseudo Stacktrace]

And HttpWriter.WriteFromStream looks like:

// HttpWriter
internal void WriteFromStream(byte[] data, int offset, int size)
    if (this._charBufferLength != this._charBufferFree)

    this.BufferData(data, offset, size, true);

    if (!this._responseBufferingOn)

Therefore, flushing depends on the HttpWriter._responseBufferingOn:

[Pseudo stacktrace]

Provided we have HttpResponse.Buffer set to false, we should get flushing every time we write to Response.OutputStream.

That was the theory anyway. In practice we experienced high memory consumption still. This seems very weird to me.  By turning buffering off — we should be flushing the response at every write as shown by the last line of HttpWriter.WriteFromStream above. [TBD: Investigate this properly].


In the end, though, we decided to implement this by explictly calling HttpResponse.Flush whenever Response.OutputStream.Flush is invoked. But how? Passing an HttpResponse into our writer API is just pollution.

Our zip files API are already doing everything correctly — it just happens that one Stream implementation is not working as expected.

We needed to modify what happens when call Stream.Flush. To do this without breaking our abstraction, we created a Stream decorator type.

This type takes a Stream, and an HttpResponse to decorate Stream operations. Mostly this decorator delegates all of its Stream operations, but some are decorated — like Flush. It is here we are invoking HttpResponse.Flush.

The nice thing about this design is that no changes at all were required within the zip API. It is written in terms of an abstraction — Stream. Our decorator is completely transparent, it is just another Stream.

Another option may be to implement an ObservableStream using decoration. This allows a client (our view) to observe a Stream object and respond to changes in state. In this case, we can raise a Flush event, which our view can respond to by flushing the HttpResponse for the request. This was feels cleaner somehow — though we sacrifice some testability.

Related patterns

Are we decorating, or proxying? Well, according to the GoF, we are decorating because we’re adding responsibilities rather than controlling access to an object.

We are decorating a Stream by adding the responsibility of flushing an HTTP response. Or, in the second case, adding the responsibility of notifying observers of state changes on a Stream.


Download dialog is misleading

You may think your download doesn’t start until you confirm — actually it’s already started by this point. You can see this for yourself if you inspect your HTTP connections. Interestingly, Chrome doesn’t seem to offer a dialog at all.

This is because this is just an HTTP response like any other. There is no special callback in place. Once response has been started, your browser is reading it. Cancelling will disconnect though, hence it’s important to check the client is still connected while emitting large files.

[TBD: With that in mind then, what happens to the data if we continue writing but the client has disconnected? Do the bytes just fall out the end? Or doesan error result?]

Response.Buffer == Response.BufferOutput

You can optionally use HttpResponse.Buffer, or HttpResponse.BufferOutput: they do the same thing:

// HttpResponse
public bool Buffer
        return this.BufferOutput;
        this.BufferOutput = value;

[UPDATE, 2009-10-30] We have encountered an issue which has required us to set buffering on. It seems that we get poorer throughput with buffering off for some reason. There has got to be some reason for this — something in HttpResponse.Flush(Boolean finalFlush) is optimizing.


  • Examine .NET assemblies with Reflector
  • KB 812406 — Response.WriteFile cannot download a large file
  • KB 823409 — The hotfix for KB 812406

Written by benbiddington

19 June, 2009 at 08:51