Storing dynamically generated code to disk

<< Successful ALT.NET Meeting completed | Home | ALT.NET Logos under Creative Commons Licence >>

Storing dynamically generated code to disk

Below I will show what it takes to generate a dynamic method at run-time and persist it to an assembly on disk. The assembly may later, at any time, be loaded normally from disk or referenced from other code and executed any number of times. This post is a minimal example just to practically show the lay of the land of dynamic code generation.

Background

(Skip this section if you just want to quickly review my sample!)

This post is also the result from feedback on my previous posts Dynamic Code Generation in .NET presentation with Anders Hejlsberg and myself online at Viddler and Dynamically Generated Code at Øredev co-presented with Anders Hejlsberg. If you want to know just how important dynamic code generation is, don't take my word for it! Please go to these other posts and listen to Anders Hejlsberg in a video explanation and motivation on the importance of this technique. Also in his recorded demo is a sample of the next generation code generation on the .NET platform. The problem today is that we have a compiler for .NET written in C++ that is a little out of date. It works really well for the purposes it is built for but changing it is a bitch. The next generation compiler for .NET is re-written in managed code. A C# compiler written in C# - talk about the hen and the egg! This new compiler will enable dynamic code generation from scripted code files on the fly - Anders is talking about the "compiler as a service". Until that time we have only the old fashioned way of doing it. Lots of applications are in fact doing it today!

Even though dynamic code generation is error prone and tedious you'd be hard pressed to find an enterprise application that does not use the technique due to the general advantages listed below. One of the most famous examples of usage is the fact that ASP.NET uses it to JIT compile scripted pages into executable code on the IIS so when I say every enterprise solution uses dynamic code generation I really mean it. This stuff is everywhere. But what does it take to really do it?

First I'll list some of the advantages of dynamic code generation (there may be more advantages I did not think of just now):

Decreased maintenance - you don't have to write all of the code instead you script what you intend and have the compiler fill in the blanks.
Reduced errors due to less repetitive tasks - writing the same code over and over again is not work for engineers. A true engineer builds a tool that takes care of the monotonous tasks.
Increased Development speed - generating a piece of code is faster than hand coding it. Multiply this by lots of files and you have lots of time saved.
Fewer compile time dependencies - maintenance is further enhanced by not having to reference every bit of code from every other bit of code. Let the compiler discover your code files and hook up messy interdependencies for you.

On to the actual code of this sample

Scenario: We want to create a method dynamically at run-time and execute it lots of time. If the code generation assembly (and the things it reference) do not change from one execution to another there is no need to re-generate the code since the outcome will be the same! Thus you can persist the generated code to disk and load the assembly back up for execution at any time.

In this post I will create a very simple method in order to show the steps involved. In fact this piece of code is so short that we may use a single test method. The method does three things:

Generates the code.
Persists it to disk.
Loads the generated code assembly and executes it to validate completion.

The method: I will create an assembly having one name space - having one static class - having one static method. The code that is generated is the following:

namespace Hitchhikers.Guide.to.the.Galaxy
{
    /// <summary>
    /// This is the method dynamically generated below and also executed!
    /// Naturally I wanted to do something silly fun and calculated '42'.
    /// Only I do it right as opposed to the result in the famous book by Douglas Adams
    /// <see cref="http://en.wikipedia.org/wiki/Hitchhikers_guide_to_the_galaxy"/>
    /// Note: Just to be funny I send in a question parameter. It is not used and could have been omitted.

    /// </summary>
    public static class ArthurDent
    {
        public static int TheUltimateQuestionOfLifeTheUniverseAndEverything(string question)
        {
            return 6*7;
        }
    }
}

The code generating code: Below is the code that generates, persists, loads and executes the above code:

[Test]
public void WhatItTakesToStoreDynamicCodeToDisc()
{
    var assemblyFileName = "earth.dll";
    var nameSpace = "Hitchhikers.Guide.to.the.Galaxy";
    var typeName = "ArthurDent";
    var methodName = "TheUltimateQuestionOfLifeTheUniverseAndEverything";

    var expectedAnswer = 42;

    if (File.Exists(assemblyFileName))
        File.Delete(assemblyFileName); // Proof that the code does not exist in advance

    AssemblyName assemblyName = new AssemblyName(nameSpace);
    AssemblyBuilder assemblyBuilder =
        AppDomain.CurrentDomain.DefineDynamicAssembly(assemblyName, AssemblyBuilderAccess.RunAndSave); // We want to be able to save the code not just run it in memory
    ModuleBuilder mb = assemblyBuilder.DefineDynamicModule(assemblyName.Name, assemblyFileName);
    TypeBuilder typeBuilder = mb.DefineType(string.Concat(nameSpace, ".", typeName),
                                            (TypeAttributes.Public |
                                             TypeAttributes.Abstract |
                                             TypeAttributes.AutoClass |
                                             TypeAttributes.AnsiClass |
                                             TypeAttributes.Sealed |
                                             TypeAttributes.BeforeFieldInit
                                            ) /* LOTS of flags for creating a static class. 
                                               * There is no flag 'static' however throw enough other flags on it and it will become a static */
                                           );
    MethodBuilder methodBuilder = typeBuilder.DefineMethod(methodName,
                                                           MethodAttributes.Public | MethodAttributes.Static,
                                                           typeof(int),
                                                           new[] { typeof(string) });
    ILGenerator ilGenerator = methodBuilder.GetILGenerator();

    // Use the ILGenerator to create your code here!
    ilGenerator.Emit(OpCodes.Ldc_I4_S, 6);
    ilGenerator.Emit(OpCodes.Ldc_I4_S, 7); // Should really be 9 in HHGG http://en.wikipedia.org/wiki/Hitchhikers_guide_to_the_galaxy
    ilGenerator.Emit(OpCodes.Mul);
    ilGenerator.Emit(OpCodes.Ret);

    typeBuilder.CreateType();

    assemblyBuilder.Save(assemblyFileName);

    // Now load and execute the method
    Assembly generatedAssembly = Assembly.LoadFile(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, assemblyFileName));
    Type generatedType = generatedAssembly.GetType(string.Concat(assemblyName, ".", typeName));
    MethodInfo generatedMethod = generatedType.GetMethod(methodName);
    var theAnswer = (int)generatedMethod.Invoke(null, new[] { "What is six times seven?" });

    Assert.AreEqual(expectedAnswer, theAnswer);
}

The general approach is to define your assembly name and other properties, define a code module, define a type and finally define a method. Then you get an ILGenerator for the method body. This is where the error prone mess starts. ;~) What you have to do is by hand write code to emit the op-codes or IL instructions (Intermediate Language) you want your method to execute. In our simple example we push 6 and 7 to the stack, multiply them and push the result on the stack. Then we return effectively popping value from the top of the stack and returning it. In my seminar demos in the talk linked above I show a much more complex sample that generates code instantiating and dependency injecting code. Please review this for a more in depth scenario.

In summation

My sample shows the basic building blocs of creating and persisting dynamically generated code on the .NET platform. It is only the basic ins and outs of how to actually do generate code. Dynamic code generation is absolutely necessary and it is in fact used everywhere in enterprise scenarios. Today it is difficult, messy and un-modern. But since we still do it and it actually has some really nice benefits everything you need is in the System.Reflection.Emit namespace. Tomorrow - soon, in a Microsoft minute, we will get a much greater tool for this on the .NET Platform thanks to Anders Hejlsberg and the C# team at Microsoft re-writing the compiler to enable "Compiler as a service".

Cheers,