Extending the Lokad DSL tool

Submitted by Vidar on

antlr.jpg

Photo by Al_HikesAZ (CC license)

What, where, why

I need to fast forward past several planned blog topics to establish a context for this one:

  • I'm working on a legacy VB6 app that is about to get some ddd, messaging love in the process of moving off the VB6 platform.
  • I've landed on using the Lokad DSL tool (Github) for defining messages.

The Lokad dsl tool does what it says on the box: It lets you define immutable, datacontract serializeable, message classes. But without all the mind numbing typing normally required to write out such classes in C#.

DSL code:

AddSecurityPassword?(SecurityId id, string displayName, string login, string password)

Becomes c# code:

[DataContract(Namespace = "Sample")]
public partial class AddSecurityPassword : ICommand<SecurityId>
{
    [DataMember(Order = 1)] public SecurityId Id { get; private set; }
    [DataMember(Order = 2)] public string DisplayName { get; private set; }
    [DataMember(Order = 3)] public string Login { get; private set; }
    [DataMember(Order = 4)] public string Password { get; private set; }
 
    AddSecurityPassword () {}
    public AddSecurityPassword (SecurityId id, string displayName, string login, string password)
    {
        Id = id;
        DisplayName = displayName;
        Login = login;
        Password = password;
    }
}

But I needed more

There is another branch of c# coding that requires a lot of typing:
COM interop code.

As might or might not be well known:

  • COM code can only understand interfaces, so an interface must be written for each class that will be consumed by a COM client.
  • COM does not support parameterized constructors. So a class that will be created by a COM client either needs to have public setters on all state or a factory method needs to be created. The factory method can not be a static helper on the class or interface, becuase COM does not recognize static methods. There needs to be a factory class, with its own factory class COM interface definition.

I need to share my message definitions with my legacy COM (VB6) application, so I was looking at a lot of upcoming typing. Unless...

I could extend the DSL tool to generate all my stupid COM stuff

A quick question on the Lokad google group and I get a response from Rinat saying that extending the DSL tool to fit my needs should be trivial.
He also wrote something about antlers, doms and trees.

How the DSL tool is built

The DSL tool uses ANTLR to generate a lexer and a parser for the dsl from a definition stored in a grammar file. Don't panic if this doesn't make sense to you, and your mind wanders to large forrest living animals.
The tokens found by the antlr lexer and parser in the provided dsl code is then put into a crude DOM (document object model) and a code generator writes out c# code based on the content of the generated DOM.

Grammar (MessageContracts.g) -> ANTLR -> MessageContractsLexer.cs, MessageContractsParser.cs.
your dsl text -> lexer / parser -> token tree
token tree -> Crude DOM
DOM -> generated C# code

When extending the tool you need to touch three of these components:

  • The grammar must be extended to recognice new constructs
  • The generated DOM must be modified to include information provided by the new constructs.
  • The code generation component must be modified to write code based on the new content of the DOM.

Extending the grammar

I do not know a lot about lexers, parsers and grammars. That said, it did not take me long from reading the exisiting grammer to understand how to add to it.
The grammar is basically more structured and easy to read regular expressions. It's based on the following super simple syntax:

SYMBOL : {mathing rule} -> ^(data_token);

The matching rules can contain previously defined symbols. There are probably loads of exciting stuff you can restrict and define in the grammar file but you probably won't need it for simple additions.

I needed to provide two pieces of data to trigger the generation of COM interop code:

  1. Guid for the generated class
  2. Guid for the generated COM interface for the class

I needed to define a new datatype and two keywords, and I needed to extend the definition of types:
To keep things simple, I ended up just adding the code for the Guid attribute as part of the dsl. Ugly but it works.

First I defined a symbol to represent the value I was after:

GUID    :       '[Guid("' (HEX_DIGIT|'-')+ '")]'

Then I defined two keywords to that would decide if it was class or interface guids.
(The choice of keywords was based on the dsl tool convention of using c# keywords to enable syntax highlighting.)

CLASS   :       'class';
ALIAS   :       'alias'

Next I defined rules for matching a combination of keyword and guid values that would be stored in corresponding tokens. To define new token names I simply added them to the list of tokens near the top of the grammar file.

comGuid         :       CLASS GUID -> ^(ComGuidToken GUID);
comInterfaceGuid        :       ALIAS GUID -> ^(ComInterfaceGuidToken GUID)

At last I extended the type declaration to include the possibility of class and interface guids. Notice the question mark after the definitons. They mark the preceding definition as optional, works the same as a question mark after a group in regular expressions.

block
    :   lc='('
            (member (',' member)*)?
        ')' representation? comGuid? comInterfaceGuid?
        -> ^(BlockToken[$lc,"Block"] member* representation? comGuid? comInterfaceGuid?

There is a lot going on in that code, but I only added the com stuff.

Next you need to generate new lexer and parser code. This is done with the antlr tool. The only tool I could get to work was the command line tool, and even that emitted a nasty error message

antlr.exe MessageContracts.g

This command should generate the correct code, just ignore the nasty error thing.

Consuming the new tokens

Then the antlr tool has done its job, the next step is to pull the new token values out of the generated token tree, and put them into our DOM model of the message definitions.

The DOM is defined in the file named CodeModel.cs, I added a property for interop data, and defined a class to hold this new information.

 public sealed class Message
    {
        public readonly string Name;
        public readonly IList<Modifier> Modifiers;

        public Message(string name, IList<Modifier> modifiers)
        {
            Name = name;
            Modifiers = modifiers;
        }

        public string StringRepresentation;

        public ComInteropData InteropData;

        public List<Member> Members = new List<Member>();
    }

    public sealed class ComInteropData

    {

        public readonly string ClassGuidAttribute;

        public readonly string ComInterfaceGuidAttribute;

        public ComInteropData(string classGuidAttribute, string comInterfaceGuidAttribute)

        {

            ClassGuidAttribute = classGuidAttribute;

            ComInterfaceGuidAttribute = comInterfaceGuidAttribute;

        }

    }

The DOM is generated by the MessageContractAssembler. It basically iterates through the tree, and builds up the DOM.

This code needed only a couple of tweaks to work:

//In WalkContractMember, recognize Com guid values:
 if (tree.Type == MessageContractsLexer.ComGuidToken)

{

                var text = tree.GetChild(0).Text;

        yield return new Member(null, text, null, Member.Kinds.ComGuid);

        yield break;

}

if (tree.Type == MessageContractsLexer.ComInterfaceGuidToken)

{

        var text = tree.GetChild(0).Text;

        yield return new Member(null, text, null, Member.Kinds.ComInterfaceGuid);

        yield break;

}

//Adding com interop info if both interface and class guid is given
 if (!string.IsNullOrWhiteSpace(comClassGuid) && !string.IsNullOrWhiteSpace(comInterfaceGuid))

{

        message.InteropData = new ComInteropData(comClassGuid, comInterfaceGuid);

        message.Modifiers.Add(new Modifier("",string.Format("I{0}",message.Name)));

        context.Using.Add("System.Runtime.InteropServices");

}

Lets have the code write us some code

The last piece of the puzzle is placed in the TemplatedGenerator class. This code iterates through the DOM and writes code using an IndentedTextWriter.
The changes here was simple enough:

if (contract.InteropData != null)

        WriteInteropData(writer, contract);

private void WriteInteropData(CodeWriter writer, Message contract)

{
        writer.WriteLine("public partial class MessageFactory");
        writer.WriteLine("{");
        writer.Indent += 1;
        writer.Write("public I{0} Create{0} (", contract.Name);
        WriteParameters(contract, writer);
        writer.WriteLine(")");
        writer.WriteLine("{");
        writer.Indent += 1;
        writer.Write("return new {0}(", contract.Name);
        WriteFactoryParamteres(contract, writer);
        writer.WriteLine(");");
        writer.Indent -= 1;
        writer.WriteLine("}");
        writer.Indent -= 1;
        writer.WriteLine("}");
        writer.WriteLine();

        writer.WriteLine(contract.InteropData.ComInterfaceGuidAttribute);
        writer.WriteLine("public partial interface I{0}", contract.Name);
        writer.WriteLine("{");
        writer.Indent += 1;
        if (contract.Members.Count > 0)
        {
                WriteInterfaceMembers(contract, writer);
        }
        writer.Indent -= 1;
        writer.WriteLine("}");
        writer.WriteLine();

        writer.WriteLine(contract.InteropData.ClassGuidAttribute);
        writer.WriteLine("[ClassInterface(ClassInterfaceType.None)]");
}

The DSL can now read this:

MyMessageContract(string myValue)
  explicit "my tostring override"
  class "D0DF1E7A-A99D-4F2C-A5DF-29E0957F2F4E"
  alias "68456B51-CB83-4220-81DC-F2B2256C06CE"

And produce this:

[Guid("68456B51-CB83-4220-81DC-F2B2256C06CE")]
public interface IMyMessageContract
{
  string MyValue { get; }
}
[Guid("D0DF1E7A-A99D-4F2C-A5DF-29E0957F2F4E")]
[ClassInterface(ClassInterfaceType.None)]
[DataContract(Namespace = "Test")]
public partial class MyMessageContract : IMyMessageContract
{
  [DataMember(Order = 1)]  public string MyValue { get; private set; }
  public MyMessageContract(){}
  public MyMessageContract(string myValue)
  {
    MyValue = myValue;
  }
}

//There needs to be an "uniplemented" base file for the factories with the
//required guid and classinteface attributes..
// this could verywell be the responsibility of the user to
// create an empty interface, and an emty class with these attributes
// important that the GUID's are not regenerated
public partial interface IInteropMessageFactory
{
  IMyMessageContract CreateMyMessageContract(string myValue);
}
public partial class InteropMessageFactory : IInteropMessageFactory
{
  public MyMessageContract CreateMyMessageContract(string myValue)
  {
    return new MyMessageContract(myValue);
  }
}

The end

I now have this tool doing most of the grunt work required to share message definitions with a VB6 client. The complete code can be found at my fork of the dsl tool on github.

Categories: