I've been in head-down, job-focused, non-blogging mode for a while now, but this issue has been enough to wake me out of hibernation. I've always been best at learning things on the job, since I tend to run into real-world issues that the examples never hit (happy path, anyone?), and I've had an intermittent memory exception in an app that I've been trying to track down on nights & weekends for a while. It's not severe enough to linger, but it's frequent enough (once a month or two) to nag at me for not fixing it.

Enter Jetbrains Dottrace Memory. I've used it for performance analysis a few times in the past, and discovered it's invaluable at tracking down where your problems are. Seriously, I can't recommend the Jetbrains suite of tools enough. Sometime in the past year or so, they split apart DotTrace performance from DotTrace memory & I'm at about a half-upgrade (DotTrace memory 3.5), with a currently bad integration story with Studio & Resharper 6, but that's a side story.

I figured I would need to run a memory snapshot on a machine that runs my application (x64 machine, .NET 4, 32-bit console application in case your curious), since the network connectivity the application relies on doesn't exist on good-old localhost. (In the back of my mind I'm thinking I should take some of the remote server's connection logs, and mock up the remote side using an interface... I still may go down that path, but for now I need to rule out the TcpStream as a factor in the leak).

So I install DotTrace, and didn't bother putting in my personal product key, since I was hoping to grab the dump and analyze it on my development workstation anyway. I fire it up, start my console app, run the dump, and boom. I get stuck mid-dump, my app crashes, and no snapshot. I'll save hours and days of summary here, but how I figured this out was to put in "Console.ReadLine()"s in my code, doing deeper and deeper dumps until I found the line of code causing the problem.

It turns out that my Linq2Sql code is at fault (or DotTrace is... an argument can be made for either side). I have a repository that on construction stores a reference to the DataContext needed for queries. DotTrace failed to allow a memory dump after the data context was used. Wrapping it in dispose did not resolve the issue. A deeper change resolves the single code issue I had, however I would need to analyze the rest of the code I have that still uses Linq2Sql (I'm in favor of the NHibernate / Fluent NHibernate / Lambda Extensions kit for data access these days, but I still have some retro code lying around). The greater concern here is code changes just to allow tooling to work. Part of me is curious if the memory leaks are in fact due to stale DataContext objects that do not get disposed.

I'd be curious to see disposal patterns for Linq2Sql DataContext objects, however part of me thinks disposal of Linq2Sql altogether is the more prudent option. I had found this article on options for context instantiation (http://blog.stevensanderson.com/2007/11/29/linq-to-sql-the-multi-tier-story/), and my app used one of the options that relied on being run in single-thread mode (names changed to protect the guilty):

public static class DataContextHelper
{
   public static MyRequestModelDataContext RequestContext = 
      new MyRequestModelDataContext(ConnectionStrings.MyConnectionString);
   public static MyResultModelDataContext ResultContext = 
      new MyResultModelDataContext (ConnectionStrings.MyConnectionString);
}

Unwittingly, by putting a constructor on my repo that went around the static property's context construction, and instead putting the context instantiation inside the repository itself, even having the context being used doesn't prevent the memory dump. I know that having a static class means one instance of that class, and in this case its properties as well, around in memory. I'm not sure what that open data context does to DotTrace, but clearly the result is a crash while attempting to dump. So at this point I start trying to eliminate DataContextHelper in favor of some more atomic operations, and explicit lifecycle of the DataContext.

I started with the above, and a call to create a context in a private member in the repository, and moved to:

public class MyRepository
{
   private MyRequestModelDataContext _context = 
      new MyRequestModelDataContext(ConnectionStrings.MyConnectionString);
   public MyResponse GetResponse()
   {
      var query = from x in context.MyResponses
                  where x.Property.Equals(false)
                  orderby x.AnotherProperty
                  select x;
      var result =  query.FirstOrDefault();
      return MyEntityConverter.ConvertFrom(result);
   }
}

Now the interesting thing is that this alone seems to solve the issue... when MyRepository goes out of scope, the garbage collector would need to handle both the repository instance and its own instance of the DataContext.

The only downside to this would be that the instantiation of a context is by nature expensive, however the typical use case would be to create a single repository instance and use that for all of the data calls within a specific operation set or transaction anyway. So I make the change, and see the behavior of crashing memory dumps; clearly this is not the answer. So lets make things a little more interesting. One more change to the repository to make sure the context is cleaned up:

public class MyRepository : IDisposable
{
   ...
   public void Dispose()
   {
      if (_context != null)
      {
         _context.Dispose();
      }
   }
}

And then back a layer. The original code was:

var repo = new MyRepository();
return repo.GetResponse();

And the changed code is:

MyResponse result;
using (var repo = new MyRepository();
{
   //dump 1
   result = repo.GetResponse();
   //dump 2
}
//dump 3
return result;

The dump comments are Console.ReadLine's injected in to track when the dumps fail. While the console application is set up to wait, I can run over to DotTrace's "Control Profiling" window and dump the memory into another snapshot (or try to, anyway). The results of the above change are that the three dump points work fine. The next step of course would be to extend out these changes to any other repositories / calls.  The repositories are easy... the calls are not quite as easy. I find usages of the repository class, and wrap everything in using statements; the next step is verifying at least that the functioning of the application is not impaired, using a test run. So far so good. I then go and run memory dumps at various points throughout the process -- mission succeeded. I'm still unclear as to the nature of the behavior, however the code change allows me to proceed with my diagnostic regiment.

What is a bit unsettling still is that there are points in the application during execution where I am unable to dump out memory without crashing. I need to keep an eye on where there are objects around that depend on open data contexts, and those seem to be directly related to when I am able to successfully generate snapshots.