As mentioned last time, there is another behavior that may be a surprise (it certainly was to me when I discovered it). Consider the following minimal TCP server code:

It’s as simple as it gets and seems to be quite correct. The lock on the type is an overkill but should do the trick making sure our buffer is consistent.

Client code is just sending single bytes and flushing the stream every time (200 single byte messages).
What is the output? Here it is (a relevant part of it at least):

Notice lines 3 and 4 – the data is printed in reversed order. How come ?!

Let’s look again at the server code:

Console.Write is protected by the lock and it is guaranteed that only one thread will call it at the time, buffer is already copied, what can be wrong ?

The only thing that may rise an eyebrow is the call to BeginRead happening before the processing of the data read from buffer. It cannot interfere with the data though as there is a lock and our data is already in a copied byte array. “When you have eliminated the impossible, whatever remains, however improbable, must be the truth“. The only remaining option is that the thread that is doing the out-of-order write is the same one that is asynchronously called to process the incoming data, but instead of calling BeginRead and returning it actually synchronously calls the async callback. MSDN documentation fails to mention that, but actually that’s exactly what happens. We can verify that easily by adding the following piece of code:

and here is the relevant output:

Notice lines 1 and 8 – it’s the same thread that is calling the rcv method twice. Since in C# locks are reentrant, it doesn’t protect the EndReceive from executing (if they weren’t it would be a self-deadlock).

In the software where I observed it it didn’t happen a lot – the bigger the load the more likely was it to surface. Though when it did happen, the recursion depth was around 5-6.  The reason is that under the hood an OverlappedAsyncResult is used which can detect that there is another async IO in progress and piggyback on it.

Hope you’ll find it useful, I learned the hard way.