oct. 31

Currently I'm working with finite state machine and I was searching for an easy way of rendering them. I have found the free tool Graphviz and I was wondering how we could use it to render a graph in an ASP.NET environment.

Prerequisites : the tool

For the sake of this example, I will use Graphviz. This is a C++ set of tool that can be used to render many types of graphes. Very interesting and it can be questionnable in command line.

Graphviz works from a "dot file" : a text file that represent the graph vertices and edges.

To be able to generate this dot file, I will use Quickgraph. This is a free .NET library create by Jonathan Halleux. It can be used for creating and manipulating graphs, as for doing calculation on graphes (shortest path, ...). For the rendering, it can be used with MSAGL (Microsoft Automatic Graph Layout - ex Microsoft GLEE). As this tool is NOT free, I will combine the two tool to achieve this work.

Generating the graph

  • Let's create a website and add a new Generic Handler
    • Right-clic on the website
    • Choose Add Item
    • Choose Web and then Generic Handler
    • Name it GraphGeneratorAndRenderer
  • Add a reference to QuickGraph.dll and QuickGraph.Graphviz.dll
  • Modify your handler as follow:
using System.Web;
using QuickGraph;
 
//To ease the writing of our graph, we can use type aliases
using TEdge = QuickGraph.TaggedEdge<string, string>;
using TVertex = System.String;
 
namespace WebApplication1
{
   public class GraphGeneratorAndRenderer : IHttpHandler
   {
      private AdjacencyGraph<TVertex, TEdge> CreateGraph()
      {
         AdjacencyGraph<TVertex, TEdge> graph = new AdjacencyGraph<TVertex, TEdge>();
 
         //1. Let's declare our vertices (ie our states)
         TVertex init = "Initial State";
         TVertex cancelled = "Cancelled";
         TVertex deleted = "Deleted";
         TVertex scheduled = "Scheduled";
         TVertex expected = "Expected";
 
         //2. Let's add them to our graph
         graph.AddVertex(init);
         graph.AddVertex(cancelled);
         graph.AddVertex(deleted);
         graph.AddVertex(scheduled);
         graph.AddVertex(expected);
 
         //3. Let's add the edges between our states
         graph.AddEdge(new TEdge(init, deleted, "Delete"));
         graph.AddEdge(new TEdge(init, scheduled, "Reception of a schedule"));
         graph.AddEdge(new TEdge(scheduled, cancelled, "CNL message"));
         graph.AddEdge(new TEdge(scheduled, expected, "Reception of Flight plan"));
         graph.AddEdge(new TEdge(expected, cancelled, "CNL message"));
         graph.AddEdge(new TEdge(scheduled, init, "Reinitialization"));
         graph.AddEdge(new TEdge(expected, init, "Reinitialization"));
 
         return graph;
      }
 
      public void ProcessRequest(HttpContext context)
      {
      }
 
      public bool IsReusable
      {
         get { return false; }
      }
   }
}

Transforming the graph to a dot file

More precisely we do not want to generate a file, but we want to have the dot structure, so we can use it in the future.

Quickgraph expose a GraphvizAlgorithm that will be responsible of the generation of the graph into a dot structure and it will transfer this to a IDotEngine that can be used for extra processing. By default, Quickgraph expose a FileDotEngine that will generate a file containing the dot structure. As this is not satisfying for us, we'll create our own dot engine.

You should of course create the dot engine in a separate file and DLL but for the sake and quickiness of this example, let's put them all together !

Update your generic handler file to add a new class definition :

using QuickGraph.Graphviz;
using QuickGraph.Graphviz.Dot;
 
public class BitmapGeneratorDotEngine : IDotEngine
{
   #region IDotEngine Members
 
   public string Run(GraphvizImageType imageType, string dot, string outputFileName)
   {
      return "We should return something here !";
   }
 
   #endregion
}

and let's add a method to our generic handler and complete the ProcessRequest method.

public class GraphGeneratorAndRenderer : IHttpHandler
{
   private string GenerateBitmap(AdjacencyGraph<TVertex, TEdge> graph)
   {
      GraphvizAlgorithm<TVertex, TEdge> algo = new GraphvizAlgorithm<TVertex, TEdge>(graph);
      string output = algo.Generate(new BitmapGeneratorDotEngine(), "ignored");
      return output;
   }
 
   public void ProcessRequest(HttpContext context)
   {
      AdjacencyGraph<TVertex, TEdge> graph = CreateGraph();
      string bitmap = GenerateBitmap(graph);
   }
 
   //Other implementation remains unchanged
}

Let's now check the dot structure:

  • Right-Clic on the website and choose Properties
  • Choose Web
  • For the Start Action, set Specific Page, and give the name of your handler : GraphGeneratorAndRenderer.ashx
  • Add a breakpoint in the Run method of you dot engine and press F5

We so have our finite state machine graph rendered in the following dot structure :

digraph G 
{
0 [];
1 [];
2 [];
3 [];
4 [];
0 -> 2 [];
0 -> 3 [];
3 -> 1 [];
3 -> 4 [];
3 -> 0 [];
4 -> 1 [];
4 -> 0 [];
}

Customize the dot structure

It's correct, but we would like to add some labels. Let's simply modify our GraphvizAlgorithm as follows:

private string GenerateBitmap(AdjacencyGraph<TVertex, TEdge> graph)
{
   GraphvizAlgorithm<TVertex, TEdge> algo = new GraphvizAlgorithm<TVertex, TEdge>(graph);
   algo.FormatEdge += delegate(object sender, FormatEdgeEventArgs<TVertex, TEdge> e)
   {
      e.EdgeFormatter.Label.Value = e.Edge.Tag;
   };
   algo.FormatVertex += delegate(object sender, FormatVertexEventArgs<TVertex> e)
   {
      e.VertexFormatter.Label = e.Vertex;
   };
   string output = algo.Generate(new BitmapGeneratorDotEngine(), "ignored");
   return output;
}

If we debug again, we'll have the following dot structure : much better.

digraph G 
{
0 [label="Initial State"];
1 [label="Cancelled"];
2 [label="Deleted"];
3 [label="Scheduled"];
4 [label="Expected"];
0 -> 2 [ label="Delete"];
0 -> 3 [ label="Reception of a schedule"];
3 -> 1 [ label="CNL message"];
3 -> 4 [ label="Reception of Flight plan"];
3 -> 0 [ label="Reinitialization"];
4 -> 1 [ label="CNL message"];
4 -> 0 [ label="Reinitialization"];
}

Convert the dot structure to a bitmap

Let's now go back to the dot engine to improve it in order to generate a bitmap instead. To do so, we'll use the tool dot.exe from Graphviz to do the conversion for us. You will also note that as we do not generate any output file, we do not use the parameter "outputFileName".

public string Run(GraphvizImageType imageType, string dot, string outputFileName)
{
   using ( Process process = new Process() )
   {
      //We'll launch dot.exe in command line
      process.StartInfo.FileName = @"C:\Program Files\Graphviz 2.21\bin\dot.exe";
      //Let's give the type we want to generate to, and a charset
      //to support accent
      process.StartInfo.Arguments = string.Format("-T{0} -Gcharset=latin1", imageType.ToString());
      //We'll receive the bitmap thru the standard output stream
      process.StartInfo.RedirectStandardOutput = true;
      //We'll need to give the dot structure in the standard input stream
      process.StartInfo.RedirectStandardInput = true;
      process.StartInfo.UseShellExecute = false;
      process.Start();
      //Let's sent the dot structure and close the stream to send the data and tell
      //we won't give any more
      process.StandardInput.Write(dot);
      process.StandardInput.Close();
      //Wait the process is finished and get back the image (binary format)
      process.WaitForExit(1000);
      return process.StandardOutput.ReadToEnd();
   }
}

Render the bitmap

What we'll want to do now is to generate an HTML image from this binary bitmap. To do so, we'll use our HTTP Handler as an image source.

  • Add a new webform to the website and call it GraphRenderer.aspx
  • Set this page as the default page
  • Add an image on the aspx page and set the URL to the HTTP Handler
<body>
    <form id="form1" runat="server">
    <div>
      <asp:Image runat="server" ID="imgGraph" ImageUrl="~/GraphGeneratorAndRenderer.ashx" />    
    </div>
    </form>
</body>

We can now update our HttpHandler to render the image:

public void ProcessRequest(HttpContext context)
{
   AdjacencyGraph<TVertex, TEdge> graph = CreateGraph();
   string binaryBitmap = GenerateBitmap(graph);
 
   //Save the bitmap to the response stream
   Stream stream = context.Response.OutputStream;
   using ( StreamWriter sw = new StreamWriter(stream, Encoding.Default) )
      sw.Write(binaryBitmap);
}

What to know ?

There are several things to know / drawback about his solution. As you may have seen, we have specified a timeout in the WaitForExit method. Indeed, there is a bug in the Graphviz tool that may freeze when generating some "large" graphs. And the graph we took in example is falling into this category. Giving a timeout gives us the opportunity to get the hand on the tool after 1 second. This value is arbitrary and we consider this is enough for the tool to generate the graph.

So the generation will take 1 second after what we should get the gif, even if we abort the process.

Is there any way to avoid this ugly trick ?

Generating files

The first solution is to use graphviz to generate output file. To do so, we can just add some command parameters : "-o"c:\temp\MyTempFile.gif"".

This solution is less elegant in my view as it involves some folder security (giving ASPNET / IIS_WPG the right of writing) and cleaning (you should clear this temp folder after a while).

Reading the standard output asynchronously

Another solution is to read the standard output asynchronously. For a reason I cannot explain, when doing so, no error / freeze can be encountered.

We have two solutions to do that : we can work high-level using the Process methods to do asynchronous reading : BeginOutputReadLine and OutputDataReceived. This will not work. Indeed in the OutputDataReceived's event handler, when accessing the e.Data property to get a line of data, we won't get any line ending (\r, \n or \r\n). This highly critical as each line of a gif image will be ended either by \r or by \n. Changing one of this delimiter will produce an unreadable image !

So how can we achieve that ? Going a bit lower level and accessing directly the underlying stream to read it asynchronously. And this will work like a charm ! Let's see some code to see how to get it work :

public class BitmapGeneratorDotEngine : IDotEngine
{
   private Stream standardOutput;
   private MemoryStream memoryStream = new MemoryStream();
   private byte[] buffer = new byte[4096];
 
   public string Run(GraphvizImageType imageType, string dot, string outputFileName)
   {
      using ( Process process = new Process() )
      {
         //We'll launch dot.exe in command line
         process.StartInfo.FileName = @"C:\Program Files\Graphviz 2.21\bin\dot.exe";
         //Let's give the type we want to generate to, and a charset
         //to support accent
         process.StartInfo.Arguments = "-Tgif -Gcharset=latin1";
         process.StartInfo.WindowStyle = ProcessWindowStyle.Minimized;
 
         //We'll receive the bitmap thru the standard output stream
         process.StartInfo.RedirectStandardOutput = true;
         //We'll need to give the dot structure in the standard input stream
         process.StartInfo.RedirectStandardInput = true;
         process.StartInfo.UseShellExecute = false;
         process.Start();
         standardOutput = process.StandardOutput.BaseStream;
         standardOutput.BeginRead(buffer, 0, buffer.Length, StandardOutputReadCallback, null);
         //Let's sent the dot structure and close the stream to send the data and tell
         //we won't give any more
         process.StandardInput.Write(dot);
         process.StandardInput.Close();
         //Wait the process is finished and get back the image (binary format)
         process.WaitForExit();
         return Encoding.Default.GetString(memoryStream.ToArray());
      }
   }
 
   private void StandardOutputReadCallback(IAsyncResult result)
   {
      int numberOfBytesRead = standardOutput.EndRead(result);
      memoryStream.Write(buffer, 0, numberOfBytesRead);
 
      // Read next bytes.   
      standardOutput.BeginRead(buffer, 0, buffer.Length, StandardOutputReadCallback, null);
   }
}

We will store the standard output base stream and use the BeginRead method on it to start getting the output. Each time we'll receive an output - via the read callback - we'll put temporarily the data in a buffer, and then write them in a stream. The read callback will call himself recursively until the process exit.

Tags: | |
oct. 20

The CITCON 2008 (Europe) is now finished and so it's time to give some feedback about this event. First for those who are not aware of the Open Space Discussion, let's see a small resume of the principle of the CITCON.

The principle

The CITCON has been brought by Jeffrey Fredrick and Paul Julius that have created the OIF (Open Information Foundation) in 1996 I think.

Most of the participants arrive on friday evening and are welcome by the two organizers that will launch the conference. Then a microphone is given to all participants (more or less 90 this year) that will briefly present themselves and explain why do they come to the CITCON. Always interesting to be able to target a bit the audience, to have a proportion between people coming to share experience, those that want to learn, the SCRUM or XP guys, the Java or .NET "adepts".

After that, this is the subject presentation time. Jeffrey and Paul will distribute some post-its so anyone can write a subject they want to speak about. Those who have a subject in mind will thus come in the center of the "arena" and will briefly explain its topic (maybe 40 have been proposed this year).

The conference introduction day is so almost finished. Now anyone is free to go and vote on the subject they are interested in. The organizers will then regroup the topics, elect the subjets with the more votes, and assign them a schedule and a room. And so on the saturday, each session will turn in a free discussion where anyone is free to join (or to leave) and where the guy(s) that have proposed this subject are the facilitaters of the talk.

A real return of experience !

The sessions

This year I went to the CITCON with two colleagues : Norman Deschauwer and Thierry Thoua. Thierry should also post soon some notes about the sessions he has seen. Follow his blog!

But let's come back to the sessions I have seen.

  1. At 10h00, there had been an interesting talk about the role of a functional tester. The guys that have proposed this subject were not there but Sai Patel took the animator role to speak about his experience. A bit less than 20 people were present with (among others) Eric JimminkFreddy Mallet, Cirilo Wortel, Paul Julius, Norman Deschauwer and Thierry Thoua.
    Very interesting to compare his implementation of agile methodology with mine. Many questions and comparisons, as its project size and evolution was very similar to mine, but with different streangth and weaknesses ! He was explaining that his team grew (from 3 persones to 10) and that with the time there had been a specialization of the testers. He began with developers volounteers and is now working with dedicated testers. His main problem ? Working with iterations of 6 weeks where the testers can only interect during the two last weeks. We so came back with our idea of integrating the testers during the development phase (and also doing some pairs between functional analysts and developers) and Cirilo (he is working as a functional tester) continued by explaining how he was testing each task after it was finished, simply by getting the sources locally. Interesting comparison with my project as we are now thinking to introduce a kanban to see more precisely where all task is located.
    Paul Julius also spoke a bit of the best XP team he has been working with : a team where all the 6 XP roles had been clearly identified (client, tester, manager, developer, tracker and coach) but where all the developers (let's understood here collaborator or employee) where passing from one role to the other.
    Someone (sorry guys, I could not remember the name of all of you...) also presented the finger chart where we can see the evolution / trend between what is analysed, developped, tested, ...
    One conclusion sentence ? The tester is a proxy to the client.
  2. At 11h15, we had a talk about the Data used for development and testing. Some presents ? Douglas Squirrel and Alexander Snaps. Three subjects have been discussed :
    • What is the source of data during the development phase (ie, can we / shall we use the production data)
    • Shall we create the database from scratch or shall we have some change script to migrate from one version to another one ?
    • What about the DB scripts while branching ?
    The talk was quite interesting even if presenting some fixed positions stating that we should not need (or very rarely) the production data to test the application. Alexander explained that he having in his database a table storing the version number of the database, and the list of all changesets that have been applied from one version to another one, in order to be able to easily compare the state of two production databases.
    We also spoke of Unitils, an open source tool that is dealing with testing in general and DB testing in particular.
  3. At 14h00, Freddy Mallet and myself, we had proposed a subject about the code quality and the metric use. Among other, were present Peter Camfield, Rene Medellin and John Van Krieken. Freddy wanted to briefly present one of his company sofware "Sonar" focusing on code quality, while my goal was more like a brainstorming, stating that a lot of metrics exist (for client, developpers, project managers, ...) but some only are interesting (for a developper / architect point of view) and can provide a quick feedback about the code quality.
    Peter Camfield has briefly presented the six indicators he was using in his project, among the test coverage and the class coupling. We had a debate whether the code coverage was a good indicator or not, stating that if the code coverage is low, it's clearly a warning, but when it's high, we cannnot really conclude as we may only pass in the code, without doing any assertions for example. We also put it in relation with the Cyclomatic complexity of a method to try to give a further interpretation of the code coverage.
    We also spoke about some tools like FxCop and Gendarme (for static code analysis : syntaxic and naming rules, ...), Simian (for the code redundancy), NDepend and its CQL (query language), Crap4J (that has been presented by Jeffrey last year) and another tool to do some mutation tests (can not remember the name. Was it Nester?)
    We spoke also of other metrics like burndown charts, some ratio time (time for correcting defects versus the time for new features for instance), the number of commits of a developper, the number of build failures, the build length, the number of "TODO", "FIXME" (or any related tag we can find in the code).
    We finally spoke about systems like "Team City" or "Gated" to do some deferred check-in.
  4. At 15h15, I have assited partly to the talk of Cirilo Wortel and Jamie Dobson that spoke about the functional testing.
    They mainly spoke about FIT (Framework of Integration Testing) and FITnesss to explain how these tools can help doing functional testing. There has also been a long methodology debate about the iterations in an agile project and whether or not we should allow scope changes during an iteration.
  5. Finally at 16h30, Matt Wynne asked the question Who cares about Javascript ? He was going to work soon in Javascript and he wanted to know how he can / he should work in TDD in this case.
    So we spoke of some problems / difficulties inherent to Javascript like the different browsers and the different supported versions of Javascript, the lack of speed, the habot of writing procedural code for many JS developpers.
    We have also discussed of some testing tools : some working inside of a browser like Watir (and FireWatir or SafariWatir for Firefox and Safari), Watin and Selenium. And some working without any browser like Celerity, HtmlUnit and JsUnit Server.
    We presented some javascript framework like JQuery, Prototype and YUI.
    I also spoke a bit of a difficulty we encounter in .NET as the framework will "change" the ids of the control and how we could, via a static ".js" file, work with the controls of the page, knowing only part of their name.
    Finally Matt give us the reference to the book of Douglas Crockford "Javascript : The good parts". I can also add the reference "Object Oriented Javascript" by Stoyan Stefanov.

Some photos

At least, some photos of the event to share with you !

The IBM premises where the conferences took place

Friday evening : myself presenting my possible talk

"What are the best metrics for you, and which feedback do they give to you ?"

 Jamie Dobson during the voting phase

 Some evening photos !

From right to left : Norman Deschauwer, Thierry Thoua and myself in Amsterdam

One typical flat entry in the center of Amsterdam : three flats, and one a bit underground   One of the very numerous canal in the center of the city
Freddy Mallet (in white) and myself during the talk of Jamie Dobson on functional testing Jeffrey is being the reporter during the session "Is Scrum evil ?" Thr program of the day, and the result of the vote for the CICTON 2009 location

A conclusion ? 

What else to say about the event ? Like last year, we had very interesting talk there, and this is an highly motivating event.

So let's meet again next year (in Barcelona ? Praga ?) and in the meanwhile let's try to share part of this motivation and enthousiasm to our team !

Thanks guys !

oct. 01

Publicize.exe is the VS tool that is responsible for generating the accessors for Visual Studio and I came across a scenario yesterday that was causing a bug. Here is the code we used :

public interface MyGenericInterface<T>
{
   void MyMethod();
}
 
public class MyClass<T> : MyGenericInterface<T>
{
   void MyGenericInterface<T>.MyMethod()
   {
   }
}

So what's happening ? If you have any test DLL that reference this DLL, with an accessor on the DLL you will receive an error like "Method 'MyMethod' in type '...MyClass_Accessor`1' from assembly '...' does not have an implementation". This is quite funny bacause as all the types are public, there is no need for an accessor.

Is there any workaround to that problem ? Implement the interface implicitely and it will work like a charm.

For any news, about that, check this post on MS forums.

Tags: