Exploring Beautiful Languages

Creating a small parser using Parsec

2012-01-23T06:10:00.000-06:00

In this post I'm going to show a small parser for a Scheme-like language written in Haskell using the Parsec parsing combinator library.

We're going to start by defining an AST for Scheme-like expressions:

data Expr = ScSymbol String
            | ScString String
            | ScNumber Integer
            | ScDouble Double 
            | ScCons Expr Expr
            | ScNil
            | ScBool Bool
            | ScQuote Expr
     deriving Show

For this experiment I'm going to use the Text.Parsec.Token and Text.Parsec.Language packages which have useful functions for creating language parsers.

idSymbol = oneOf ":!$%&*+/<=>?@\\^|-~"

schemeLanguageDef :: LanguageDef st
schemeLanguageDef = emptyDef { 
                   commentLine = ";;"
                   , identStart = letter <|> idSymbol
                   , identLetter = alphaNum <|> idSymbol
                   , opStart = parserZero
                   , opLetter = parserZero
                 }

schemeTokenParser = makeTokenParser schemeLanguageDef

TokenParser {
   identifier = idParser,
   reservedOp = opParser,
   stringLiteral = stringLiteralParser,
   parens = parParser,
   lexeme = lexParser,
   naturalOrFloat = naturalOrFloatParser
} = schemeTokenParser

Given these definitions we get parsers for identifiers, numbers, string literals "for free".

Boolean literals are parsed using the following parser:

boolLiteral = lexParser (
                 do 
                   char '#'
                   val <- (char 't') <|> (char 'f')
                   return $ ScBool $ val == 't'
             )

Basic quotations are parsed as follows:

quoteParser = lexParser (
              do 
                char '\''
                val <- expressionParser
                return $ ScQuote val
          )

Atomic expressions are parsed as follows:

atomParser =
   (do 
      id <- idParser
      return $ ScSymbol id)
   <|>
    (do 
       fnumber <- naturalOrFloatParser
       return $ case fnumber of
                Left num -> ScNumber num
                Right num -> ScDouble num)
   <|>        
    (do str <- stringLiteralParser
        return $ ScString str)
   <|> boolLiteral
   <|> quoteParser

Next we need to consider S-Expressions. This kind of expressions are combinationsof the above elements surrounded by parentheses.

For example:

(+ 1 2 3)

dottedSuffixParser =     
   do 
      dotParser
      finalExpr <- expressionParser
      return finalExpr

parExpressionParser = 
   do (exprs, last) <- parParser 
                (do 
                   seq <- many expressionParser
                   dottedSuffix <- optionMaybe dottedSuffixParser 
                   return (case dottedSuffix of
                            Just lastExpr -> (seq, lastExpr)
                            Nothing -> (seq, ScNil)))
      return $ foldr ScCons last exprs

As shown above, the parser need to verify if there was a dot ('.') before the last element for example:

(1 . 2)

More details on the dot syntax can be found here.
Finally we define expressions:

       expressionParser =
            atomParser <|> parExpressionParser


       parseIt input = parse expressionParser "" input

We can use this function to parse this expressions as follows:

*SCParser> parseIt "(- x 1)"
Right (ScCons (ScSymbol "-") (ScCons (ScSymbol "x") (ScCons (ScNumber 1) ScNil)))

There's a great a series of articles on implementing a Scheme interpreter in Haskell called "Write Yourself a Scheme in 48 Hours/Parsing" which is a great source of information on this topic.

A call-with-current-continuation example

2012-01-02T21:21:00.000-06:00

The call-with-current-continuation (or call/cc for short) function in Scheme allows the creation of powerful control structures. It basically allows you to capture the current state of a computation as a first class value. This value could be assigned to a variable and invoked later.

For a great introduction to this function see the call/cc entry in the Community-Scheme-Wiki .

The following toy example shows how this function is used to do an incremental search inside a tree. Say that you have the following tree structure definition:


(define tree-data
    '(html
        (head (title "test"))
        (body
           (p "foo" (i "goo"))
           (ol
            (li "xoo" (i "moo"))
            (li (i "loo")))
           (p))))

We want to define a function to search for elements identified by a predicate. For example, if looking for a "i" element we can write:


(define (is-italic? element)
  (and (list? element) 
       (not (null? element))
       (eq? 'i (car element))))

We want to create a function to perform an incremental search inside a tree structure. That is, we want to find the first element and then request the next element if needed.

Here's a demo of the function we want to create:

> (define hit (search is-italic? tree-data))
                                           
> hit
((i "goo") #<continuation>)
> (set! hit (get-next-result hit))
> hit
((i "moo") #<continuation>)
> (set! hit (get-next-result hit))
> hit
((i "loo") #<continuation>)
> (set! hit (get-next-result hit))
> hit
(() #<continuation>)

This function is defined using call-with-current-continuation as follows:

(define (search pred? tree)
  (call-with-current-continuation
   (lambda (c)
     (searching pred? tree c))))

(define (searching pred? tree return)
  (set! return
        (call-with-current-continuation
         (lambda (continuation)
           (if (pred? tree)
               (return (list tree continuation))
               return))))
  (if (list? tree)
      (for-each
       (lambda (item)
         (set! return (cadr (searching pred? item return))))
       tree))
  (list '() return))

The entry point is the search function which captures the current continuation in the return variable. This continuation value allows us to jump to the point where call-with-current-continuation was called. We pass this value to the searching function which contains the logic to search inside the tree.

The first expression of the searching function seems confusing since it contains another call to call-with-current-continuation and an assignment of return. This code is used to do the following things:

Return a search result (remember that return contains a continuation)
Capture the current state of the tree traversal (this is done by the call-with-current-continuation call)

...
  (set! return
        (call-with-current-continuation
         (lambda (continuation)
           (if (pred? tree)
               (return (list tree continuation))
               return))))
...

Notice that a search result contains two things:

The tree fragment that makes the predicate true
The continuation of the point where the element was found (which can be used to restore the traversal)

One key thing to understand is that we change the value of return to the result of the call/cc call. This is done this way because we will change the return continuation to retrieve new results. This is done using the get-next-result function as follows:


(define (get-next-result result)
  (if (not (null? (car result)))
      (call-with-current-continuation (cadr result))
      result))

Calling call-with-current-continuation with the last captured continuation will result on setting it to the return variable in searching.

A quick look at APL

2011-03-31T05:42:00.042-06:00

In this post I'm going to show an implementation of polynomial multiplication written in APL and the steps to create it.

APL

APL is an array oriented programming language which means it's good for manipulating vectors, matrices and other kinds of arrays.

APL has many interesting characteristics among them is its syntax which uses non ASCII characters .

Here's a very nice video showing how to implement the Conway's Game Of Life in APL:

A very nice APL resource is the APL Wiki it has lots of information. Another very nice resource is The FinnAPL Idiom Library which contains lots little examples of code. Another nice resource is Vector the journal of the British APL Association which has information on the community around APL related languages.

The examples presented in this post were written using NARS2000 a nice open source implementation of APL.

Polynomial multiplication

As with a previous post on J I'm going to show the implementation of polynomial multiplication .

The following example shows the process of multiplying two polynomials:


(4x³ - 23x² + x + 1)   *   (2x² - x - 3) 

= ((4x³ - 23x² + x + 1) * 2x²) + 
  ((4x³ - 23x² + x + 1) * -x) +
  ((4x³ - 23x² + x + 1) * -3)

= (8x⁵ - 46x⁴ + 2x³ + 2x²) + 
  (-4x⁴ + 23x³ - x² - x)  +
  (-12x³ + 69x² - 3x - 3) 

= (8x⁵ - 50x⁴ + 13x³ + 70x² - 4x - 3)

This example will be used to illustrate the parts of the code in the following sections.

The program

The final program to calculate polynomial multiplication looks like this:

Where a and b are the polynomials.

As with the previous post on J I'm sure there's a shorter way to write this program
(the number of parenthesis seems to indicate this!).

One key aspect to understand APL code is to remember that it evaluates expressions from right to left. See the Order of execution section of the APL Wiki for more information.

A description of each part of this program is presented in the following sections.

Polynomial representation

To represent polynomials we're going to use vectors the following way:

For:

(4x³ + 23x² + x + 1)

the array will be:

1 1 23 4

In APL arrays are written as sequences of numbers separated by spaces (as above). You can manipulate this values easily, for example here we create two variables with two polynomials and add them:

In order to multiply each element of the array by a value we can simply write:

Outer product

The outer product operator lets you apply an operation to all the combinations of the elements of the operands. For example if we want to multiply each value of the two arrays representing the polynomials we want to multiply we can write:

Creating a matrix of intermediate polynomials

We need to obtain a matrix with the intermediate polynomials given the result of the outer product. But first I'm going to introduce the rho function. This function can be used to with one argument (monadic) to obtain the dimensions of the given array for example:

Or it can be used with two arguments (dyadic) to create an array of the given shape.

We can use this function combined with comma function to expand the matrix of polynomials:

The iota function lets you, among other things, create arrays of sequences of values. We're going to use it to create the values used to rotate each polynomial in our matrix:

The rotation of each matrix element is performed using the rotate function as follows:

The final step is to sum every vector in this matrix. We can do that by using the reduce operator (/) . This operator lets you apply a dyad operation to every element of an array accumulating the result. For example:

The nice thing about it is that if you apply it to a matrix, it will do the right thing by summing up every 1d vector. In our case we need to sum the rows so we change the axis of the operation.

Index origin

Before defining the final function we need to take an additional consideration. There is a system variable which modifies the origin of the iota function among other things. For example:

In other to make our code independent of the value of IO by using it instead of references to '1' as above.

Defining a function

The final function definition looks like this:

We can use this function as follows:

IronPython & WPF: Data binding with TreeView's selected element

2011-02-02T05:26:00.004-06:00

In this post I'm going to show a small example of using data binding with the selected element of a WPF TreeView with an IronPython class.

A couple of days ago I had the necessity of using data binding to keep track of the selected value of a WPF TreeView . At first it seemed to be an easy task so I wrote:

<TreeView SelectedValue="{Binding selected, Mode=TwoWay}" ... />

Running this code results on the following error:

SystemError: 'Provide value on 'System.Windows.Data.Binding' threw an exception.' Line number '12' and line position '7'.

The problem is that the SelectedValue (and SelectedItem) property is read-only.

There are several ways to deal with this problem. One alternative is to use a technique similar to the one described in the "Forwarding the Result of WPF Validation in MVVM" post. We're going to define an attached property which works as an "output only" property that could be used with data binding.

Attached property definition

I couldn't find a way to wrote the definition of the attached property in IronPython because it needed to be instanciated by XamlReader. So the definition was written using C#:

using System.Windows.Markup;
using System.Windows;
using System.Windows.Controls;
using System.IO;
using System.Collections.ObjectModel;


namespace Langexplr 
{
   public static class TreeViewSelectedBehavior
   {
      public static readonly DependencyProperty MySelectedProperty = 
             System.Windows.DependencyProperty.RegisterAttached(
                                       "MySelected",
                                       typeof(object),
                                       typeof(TreeViewSelectedBehavior)
                                    );
      public static object GetMySelected(TreeView t)
      {
         return t.GetValue(MySelectedProperty);
      }

      public static void SetMySelected(TreeView t, object theValue)
      {
         t.SetValue(MySelectedProperty, theValue);
      }

                                    
      public static readonly DependencyProperty SelectedHelperProperty =
             System.Windows.DependencyProperty.RegisterAttached(
                                       "SelectedHelper",
                                       typeof(TreeViewSelectedHelper),
                                       typeof(TreeViewSelectedBehavior),
                                       new UIPropertyMetadata(null,OnSelectedHelperChanged)
                                    );

      public static  TreeViewSelectedHelper GetSelectedHelper(TreeView t) 
      {
          return (TreeViewSelectedHelper)t.GetValue(SelectedHelperProperty);
      }

      public static void SetSelectedHelper(TreeView t, 
                                           TreeViewSelectedHelper theValue) 
      {
          t.SetValue(SelectedHelperProperty, theValue);
      }
      static void OnSelectedHelperChanged(
                    DependencyObject depObj, 
                    DependencyPropertyChangedEventArgs e)
      {
         ((TreeViewSelectedHelper)e.NewValue).Register((TreeView)depObj);
      }

   }

This class define two properties:

MySelected: the output property that is used to set the selected element in the view model
SelectedHelper: which is used to as an object that modifies the value of MySelected when the selected item changes(see below).

The following helper class is used to subscribe the SelectedItemChanged event and change "MySelected" .

public class TreeViewSelectedHelper 
   {
       public TreeViewSelectedHelper() { }
       void SelectedItemChanged(object sender, 
                                RoutedPropertyChangedEventArgs<object> args)
       {
           (sender as TreeView).SetValue(
                        TreeViewSelectedBehavior.MySelectedProperty,
                        ((sender as TreeView)).SelectedItem);
       }
       public void Register(TreeView t)
       {
           t.SelectedItemChanged += SelectedItemChanged;
       }
   }

}

We can now compile this class:


set NETFX4=c:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\
set WPFPATH=%NETFX4%\WPF
csc /debug /r:%NETFX4%System.Xaml.dll /r:%WPFPATH%\WindowsBase.dll /r:%WPFPATH%\PresentationCore.dll /r:%WPFPATH%\PresentationFramework.dll  /target:library utils.cs

The example

Having defined this attached property and helper class we can now write the following example.

<Window xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:utils="clr-namespace:Langexplr;assembly=utils"
        Title="TreeView selection test" Width="300" Height="300">
   <Window.Resources>
     <utils:TreeViewSelectedHelper x:Key="selHelper" />
   </Window.Resources>
   <StackPanel>

      <TextBlock Text="{Binding selected.label}"/> 
         <TreeView ItemsSource="{Binding roots}"      
                   utils:TreeViewSelectedBehavior.SelectedHelper="{StaticResource selHelper}">
             <utils:TreeViewSelectedBehavior.MySelected>
               <Binding Path="selected" Mode="OneWayToSource"/> 
             </utils:TreeViewSelectedBehavior.MySelected> 
              <TreeView.ItemTemplate>
                  <HierarchicalDataTemplate ItemsSource="{Binding children}">
       <TextBlock Text="{Binding label}"/>
    </HierarchicalDataTemplate>
       </TreeView.ItemTemplate>
         </TreeView>
   </StackPanel>
</Window>

With this XAML definition we can write the following IronPython code:

import clr
clr.AddReference("PresentationCore")  
clr.AddReference("PresentationFramework")  
clr.AddReference("WindowsBase")  
clr.AddReference('GalaSoft.MvvmLight.WPF4.dll')


from System.Windows.Markup import XamlReader
from System.Windows import Application
from System.IO import File
from System.Windows.Controls import TreeView
import System
import clrtype
from GalaSoft.MvvmLight import ViewModelBase
from System.Collections.ObjectModel import ObservableCollection


class TestModel(ViewModelBase):
   __metaclass__ = clrtype.ClrClass
   def __init__(self):
       self.name = 'algo'
       self.root = NodeModel('x1',[NodeModel('y2',[NodeModel('z2',[])]),
                                   NodeModel('y3',[])])

       self.sselected = NodeModel('',[])

   @property
   @clrtype.accepts()
   @clrtype.returns(System.Object)
   def selected(self): 
      result = self.sselected
      return result

   @selected.setter
   @clrtype.accepts(System.Object)
   @clrtype.returns()
   def selected(self, value):
      self.sselected = value
      self.RaisePropertyChanged('selected')

   @property
   @clrtype.accepts()
   @clrtype.returns(System.Object)
   def roots(self): 
      return [self.root]

   @property
   @clrtype.accepts()
   @clrtype.returns(System.String)
   def label(self): 
      return self.name
 
   @label.setter
   @clrtype.accepts(System.String)
   @clrtype.returns()
   def label(self, value):
      self.name = value
      self.RaisePropertyChanged('label')



class NodeModel:
   __metaclass__ = clrtype.ClrClass

   def __init__(self,label, initchildren):
      self.children_collection = ObservableCollection[System.Object](initchildren)
      self.name = label


   @property
   @clrtype.accepts()
   @clrtype.returns(System.Object)
   def label(self): 
      return self.name

   @property
   @clrtype.accepts()
   @clrtype.returns(System.Object)
   def children(self): 
      return self.children_collection
 

xamlFile = File.OpenRead('test.xaml')
window = XamlReader.Load(xamlFile)
window.DataContext = TestModel()
xamlFile.Close()

Application().Run(window)

By running this example we can see how the label of the selected element of the TreeView is reflected in the TextBlock defined above.

IronPython & Silverlight Part VI: Using the TreeView control

2011-01-31T05:31:00.002-06:00

In this post I'm going to show a small example of using the Silverlight 4 TreeView control with IronPython.

Referencing System.Windows.Controls.dll

Before we start using the TreeView control we need to add a reference to System.Windows.Controls.dll . This assembly can be found in the "Microsoft Silverlight 4 Tools for Visual Studio 2010" package.

You need to copy this assembly to the location of chiron.exe (ex. IronPython2.7\Silverlight\bin) and add a reference to it in the AppManifest.xaml file.

<Deployment xmlns="http://schemas.microsoft.com/client/2007/deployment" 
            xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" 
            RuntimeVersion="4.0.50917.0" 
            EntryPointAssembly="Microsoft.Scripting.Silverlight"  
            EntryPointType="Microsoft.Scripting.Silverlight.DynamicApplication" 
            ExternalCallersFromCrossDomain="ScriptableOnly">
  <!-- Add assembly references here -->
  <Deployment.Parts>
... 
    <AssemblyPart Source="System.Windows.Controls.dll" />
  </Deployment.Parts>
...
</Deployment>

Since we're using Silverlight 4 we need to change the runtime version to "4.0.50917.0" (see this post for more details).

Using the TreeView in XAML

Now that we have access to the assembly we can use it in app.xaml.

<UserControl x:Class="System.Windows.Controls.UserControl"
 xmlns="http://schemas.microsoft.com/client/2007"
 xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:sdk="clr-namespace:System.Windows.Controls;assembly=System.Windows.Controls"
        xmlns:wsdk="clr-namespace:System.Windows;assembly=System.Windows.Controls">
  <StackPanel x:Name="layout_root" Background="White">
    <TextBlock x:Name="Message" FontSize="30" Text="TreeView experiment" />
    <sdk:TreeView x:Name="tree">
       <sdk:TreeView.ItemTemplate>
           <wsdk:HierarchicalDataTemplate ItemsSource="{Binding nodes}">
               <TextBlock Foreground="Red" Text="{Binding label}" />
           </wsdk:HierarchicalDataTemplate>
       </sdk:TreeView.ItemTemplate>
    </sdk:TreeView>
  </StackPanel>
</UserControl>

TreeView data binding

The XAML file above shows that we're binding the content of the tree view to the nodes property of the data source object. Here's the definition of the Python class used for this example:

from System.Windows import Application
from System.Windows.Controls import UserControl
from System import Object
from System.Collections.ObjectModel import ObservableCollection
import clrtype
import clr

clr.AddReferenceToFile('GalaSoft.MvvmLight.SL4.dll')

from GalaSoft.MvvmLight.Command import RelayCommand


class Node:
  __metaclass__ = clrtype.ClrClass
  __clrnamespace = "LangexplrExperiments"


  def __init__(self,label,children):
    self.text = label
    self.children = children

  @property 
  @clrtype.accepts()
  @clrtype.returns(str)
  def label(self): 
     return self.text

  @property 
  @clrtype.accepts()
  @clrtype.returns(Object)
  def nodes(self): 
     return self.children



class App:
  def __init__(self):
    root = Application.Current.LoadRootVisual(UserControl(), "app.xaml")
    root.tree.ItemsSource = [Node('level1',
                                 [Node('level11',[]),
                                  Node('level12',[Node('level121',[])])
                                 ])]


App()

With this changes we can run chiron.exe /b and we get:

IronPython & Silverlight Part V: Using the MVVM Light Toolkit

2011-01-29T06:21:00.003-06:00

In this post I'm going to show a small example of using the MVVM Light Toolkit in Silverlight with IronPython.

MVVM Light

According to its website, MVVM Light Toolkit is a:

...set of components helping people to get started in the Model - View - ViewModel pattern in Silverlight and WPF...

It provides useful elements such an implementation of "Relay Command" and a feature to expose events as commands (EventToCommand).

In the context of IronPython it will save us a lot of code.

Adding a reference to MVVM Light

There are several ways to add a reference to the MVVM Light Toolkit assembly. However the first step is to set the Silverlight version to 4 (see this post for more information).

Adding the reference using the manifest

One way to add the reference is to use the manifest AppManifest.xaml. Once we generated this file(using chiron.exe /m and copying it to app/) and changed the Silverlight version to 4 (see here) we can add an entry referencing the GalaSoft.MvvmLight.SL4.dll assembly.

<Deployment xmlns="http://schemas.microsoft.com/client/2007/deployment" 
            xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
            RuntimeVersion="4.0.50401.0" 
            EntryPointAssembly="Microsoft.Scripting.Silverlight" 
            EntryPointType="Microsoft.Scripting.Silverlight.DynamicApplication" 
            ExternalCallersFromCrossDomain="ScriptableOnly">
  <!-- Add assembly references here -->
  <Deployment.Parts>
...    
    <AssemblyPart Source="GalaSoft.MvvmLight.SL4.dll"/>
...    
  </Deployment.Parts>
...
</Deployment>

Also we need to copy of the GalaSoft.MvvmLight.SL4.dll file to the chiron.exe folder (ex. IronPython\Silvelright\bin).

Using it in Python

Once we referenced this file using the manifest we need to reference it from code.

import clr
clr.AddReferenceToFile('GalaSoft.MvvmLight.SL4.dll')

Example

The following example shows the use of RelayCommand and ViewModelBase from IronPython.

app.xaml:

<UserControl x:Class="System.Windows.Controls.UserControl"
 xmlns="http://schemas.microsoft.com/client/2007"
 xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:wbc="clr-namespace:System.Windows.Controls">
  <StackPanel Width="300" x:Name="layout_root" Background="White">
     <TextBox x:Name="my_text_box" 
              Text="{Binding text_to_display, Mode=TwoWay}" />

     <TextBlock Text="{Binding text_to_display}" />
     <Button Content="Reset" Command="{Binding reset_command}"/>
  </StackPanel>
</UserControl>

app.py:

import clr
clr.AddReferenceToFile('GalaSoft.MvvmLight.SL4.dll')
from System.Windows import Application
from System.Windows.Controls import UserControl
import System
import clrtype
from System.Windows import MessageBox
from GalaSoft.MvvmLight.Command import RelayCommand
from GalaSoft.MvvmLight import ViewModelBase

class MyBasicModel(ViewModelBase):
   __metaclass__ = clrtype.ClrClass
   
   def __init__(self):
      self.text = 'Initial text'

   @property
   @clrtype.accepts()
   @clrtype.returns(System.String)
   def text_to_display(self): return self.text


   @text_to_display.setter
   @clrtype.accepts(System.String)
   @clrtype.returns()
   def text_to_display(self, value): 
       self.text = value
       self.RaisePropertyChanged('text_to_display')

   @property
   @clrtype.accepts()
   @clrtype.returns(System.Object)
   def reset_command(self): 
      return RelayCommand(lambda: self.perform_reset_text() )

   def perform_reset_text(self):
      self.text_to_display = ''


class App:

  def __init__(self):

    self.model = MyBasicModel()
    self.root = Application.Current.LoadRootVisual(UserControl(), "app.xaml")
    self.root.DataContext = self.model


theApp = App()

Using this library from IronPython saves us from many thins such as having to declare events to comply with the INotifyPropertyChanged interface.

IronPython & Silverlight Part IV: Using Silverlight 4

2011-01-25T06:00:00.001-06:00

The default Silverlight runtime version of programs created with IronPython 2.7 Beta 1 is "2.0.31005.0". If you want to take advantage of Silverlight 4 features you have to make a small change to AppManifest.xaml.

For example, say that you want to add a RichTextBox control(available on Silverlight 4).

<UserControl x:Class="System.Windows.Controls.UserControl"
 xmlns="http://schemas.microsoft.com/client/2007"
 xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:wbc="clr-namespace:System.Windows.Controls">
  <StackPanel Width="300" x:Name="layout_root" Background="White">
     <wbc:RichTextBox x:Name="rtb" />
     <Button x:Name="my_button" Content="Ok"/>
  </StackPanel>
</UserControl>

And

from System.Windows import Application
from System.Windows.Controls import UserControl
import System


class App:

  def __init__(self):
    self.theText = """
this
is
some
text"""


    self.root = Application.Current.LoadRootVisual(UserControl(), "app.xaml")
    self.root.rtb.Selection.Select(self.root.rtb.ContentStart, self.root.rtb.ContentEnd)
    self.root.rtb.Selection.Text = self.theText

theApp = App()

In order to make this program work you have to add the AppManifest.xaml to the app/ folder of your application. You can get a copy of this file using chiron.exe /m :

By default this file looks like this:

<Deployment xmlns="http://schemas.microsoft.com/client/2007/deployment" 
            xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" 
            RuntimeVersion="2.0.31005.0" 
            EntryPointAssembly="Microsoft.Scripting.Silverlight" 
            EntryPointType="Microsoft.Scripting.Silverlight.DynamicApplication" 
            ExternalCallersFromCrossDomain="ScriptableOnly">
  <!-- Add assembly references here -->
  <Deployment.Parts>
    <!-- In the XAP -->
    <!-- <AssemblyPart Source="Foo.dll" /> -->
    <!-- Outside the XAP, same domain -->
    <!-- <AssemblyPart Source="/Foo.dll" /> -->
    <!-- Outside the XAP, different domain -->
    <!-- <AssemblyPart Source="http://bar.com/Foo.dll" /> -->
    <AssemblyPart Source="Microsoft.Scripting.Silverlight.dll" />
    <AssemblyPart Source="System.Numerics.dll" />
    <AssemblyPart Source="Microsoft.Scripting.dll" />
    <AssemblyPart Source="Microsoft.Dynamic.dll" />
    <AssemblyPart Source="IronPython.dll" />
    <AssemblyPart Source="IronPython.Modules.dll" />
  </Deployment.Parts>
  <!-- Add transparent platform extensions (.slvx) references here -->
  <Deployment.ExternalParts>
    <!-- Example -->
    <!-- <ExtensionPart Source="http://bar.com/v1/Foo.slvx" /> -->
  </Deployment.ExternalParts>
...
</Deployment>

After copying this file to the app/ folder you have to change the RuntimeVersion attribute value to "4.0.50401.0".

Now to can run the application and have access to Silverlight 4 features.

IronPython & Silverlight Part III: Basic Data Binding

2010-12-15T05:58:00.003-06:00

One of the nicest features of Silverlight is data binding. This feature allows you to perform and receive changes on the UI without explicitly adding or removing elements from UI controls. For example:

<TextBox Text="{Binding TextToDisplay}">

This XAML code says that the value of the Text property is bound to the TextToDisplay property of the of the object specified by the DataContext property.

As with other Silverlight features data binding requires you to use a .NET object with properties. We can use clrtype to take advantage of this feature with IronPython.

For example, say that we want to bind a text property to a Python object to a TextBox:

<UserControl x:Class="System.Windows.Controls.UserControl"
 xmlns="http://schemas.microsoft.com/client/2007"
 xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml">
  <StackPanel Width="300" x:Name="layout_root" Background="White">
     <TextBox x:Name="my_text_box" 
              Text="{Binding text_to_display, Mode=TwoWay}" />
     <Button x:Name="my_button" Content="Show text"/>
  </StackPanel>
</UserControl>

The app.py file looks like this:


from System.Windows import Application
from System.Windows.Controls import UserControl
import System
import clrtype
from System.Windows import MessageBox

class MyData:
   __metaclass__ = clrtype.ClrClass
   
   def __init__(self):
      self.text = 'Initial text'

   @property
   @clrtype.accepts()
   @clrtype.returns(System.String)
   def text_to_display(self): return self.text


   @text_to_display.setter
   @clrtype.accepts(System.String)
   @clrtype.returns()
   def text_to_display(self, value): 
       self.text = value

class App:

  def handle_click(self, sender, event_args):
      MessageBox.Show(self.data.text)

  def __init__(self):
    self.data = MyData()
    self.root = Application.Current.LoadRootVisual(UserControl(), "app.xaml")
    self.root.my_text_box.DataContext = self.data
    self.root.my_button.Click += lambda s,ea: self.handle_click(s,ea)

theApp = App()

Here the definition of MyData is decorated with the information on how to generate the .NET class elements to be exposed .

Since the binding is declarated as TwoWay modifications to the TextBox are reflected in the data instance.

As with Part I and Part II of these series, IronPython 2.7 beta 1 is used for all examples.

IronPython & Silverlight Part II: Basic event handling

2010-12-07T06:05:00.000-06:00

There are a couple of ways to add event handlers to Silverlight controls. The common way is to add the event handlers directly in XAML. For example:

<Button x:Name="ButtonGo" Content="Go!" Click="MyClickHandler" />

Given that there's a definition for the MyClickHandler method in your C# code. However for IronPython there's a couple of options:

Directly in Python code

Event handlers can be added as in C# by using the '+=' operator. For example say that you have the following XAML code describing an UserControl:

<UserControl x:Class="System.Windows.Controls.UserControl"
 xmlns="http://schemas.microsoft.com/client/2007"
 xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml">

  <StackPanel Width="300" x:Name="layout_root" Background="White">
    <TextBlock x:Name="Message" FontSize="30" />
    <Button x:Name="ButtonGo" Content="Go!" />
  </StackPanel>
  <UserControl.Resources>
     <Storyboard x:Name="MyStoryboard">
        <DoubleAnimation 
             Storyboard.TargetName="Message"
             Storyboard.TargetProperty="Opacity"
             From="1.0" To="0.0" Duration="0:0:3"
             AutoReverse="True"
             RepeatBehavior="Forever"/>
     </Storyboard>
  </UserControl.Resources>
</UserControl>

Say we want to start the 'MyStoryboard' animation in the event handler for the Click event of the ButtonGo button. We can write:

from System.Windows import Application
from System.Windows.Controls import UserControl

def handle_click(sender, event_args):
   theApp.root.MyStoryboard.Begin()

class App:

  def __init__(self):
    self.root = Application.Current.LoadRootVisual(MyUserControl(), "app.xaml")
    self.root.Message.Text = "Welcome to Python and Silverlight!"
    self.root.ButtonGo.Click += handle_click

theApp = App()

Using `clrtype`

The clrtype module can be used to define .NET classes from (Iron)Python classes . This module can be found in the IronPython samples package or here in the GitHub repository. The GUI Automated Testing blog has some nice tutorials on using clrtype.

To use this module, you have to copy the clrtype.py file to your app/ folder.

We can change the code this way:

from System.Windows import Application
from System.Windows.Controls import UserControl
from System.Windows import MessageBox
import System
import clrtype

class MyUserControl(UserControl):
   __metaclass__ = clrtype.ClrClass

   _clrnamespace = "MyNs"

   @clrtype.accepts(System.Object, System.EventArgs)
   @clrtype.returns(System.Void)
   def my_click_handler(self, sender, event_args):
      theApp.root.MyStoryboard.Begin()

   def __getattr__(self, name):
      return self.FindName(name)

def handle_click(sender,event_args):
   theApp.root.MyStoryboard.Begin()

class App:

  def __init__(self):
    self.root = Application.Current.LoadRootVisual(MyUserControl(), "app.xaml")
    self.root.Message.Text = "Welcome to Python and Silverlight!"

theApp = App()

With these definitions we can change the XAML code for the Button to have be:

<Button x:Name="ButtonGo" Content="Go!" Click="my_click_handler"/>

Notice that the definition of MyUserControl has a definition for __getattr__. This definition is used to still be able to access the definitions of child controls for example theApp.root.MyStoryboard.

Final words

Another way to do event handling is to use Commanding. This approach is preferred for MVVM. For future posts I'll try to cover the use of Commanding with IronPython.

Using IronPython with Silverlight, Part I

2010-11-30T21:42:00.000-06:00

This is the first of series of posts on the topic of using IronPython to create Silverlight programs. Experimenting with these technologies is nice because you only need a text editor and the IronPython distribution.

Getting started

In these series of posts I'll be using IronPython 2.7 (which is in beta right now) and Silverlight 4 .

Once these packages are installed the first step is to copy a basic program template to a your work directory. The template is located in (IronPython path)\Silverlight\script\templates\python.

This template contains the following structure:


C:.
¦   index.html
¦
+---app
¦       app.py
¦       app.xaml
¦
+---css
¦       screen.css
¦
+---js
        error.js

The app.py and app.xaml files contain the code for the entry point of the demo Silverlight application. The index.html file contains the Silverlight control host.

The default app.xaml code looks like this:

<UserControl x:Class="System.Windows.Controls.UserControl"
 xmlns="http://schemas.microsoft.com/client/2007"
 xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml">

  <Grid x:Name="layout_root" Background="White">
    <TextBlock x:Name="Message" FontSize="30" />
  </Grid>

</UserControl>

The default app.py code looks like this:

from System.Windows import Application
from System.Windows.Controls import UserControl

class App:
  def __init__(self):
    root = Application.Current.LoadRootVisual(UserControl(), "app.xaml")
    root.Message.Text = "Welcome to Python and Silverlight!"

App()

As you can see this code loads the XAML file for defining the 'root visual' . The 'root.Message.Text = ...' assignment changes the text displayed by the 'Message' TextBlock instance. Notice that it uses the name of the control as if it were a member of the UserControl instance returned by LoadRootVisual. This is accomplished by using a DLR feature defined in the FrameworkElementExtension class (see ExtensionTypes.cs )

You can easily run this example by going to the command line and running Chiron:

C:\development\blog\ipy\basictest>c:\development\IronPython-2.7\Silverlight\bin\Chiron.exe /b
Chiron - Silverlight Dynamic Language Development Utility. Version 1.0.0.0
Chiron serving 'C:\development\blog\ipy\basictest' as http://localhost:2060/
21:03:45 200     1,185 /
21:03:45 200       792 /style.css!
21:03:45 200     2,492 /sl.png!
21:03:45 200       642 /slx.png!
21:03:49 404       580 /favicon.ico [Resource not found]

Running Chiron this way will start the web server and will open a web browser in its root.

(Note: For IronPython 2.7 Beta you have to copy the System.Numerics.dll assembly in the %IPY_HOME\Silverlight\bin folder, this file could be found in the Silverlight SDK distribution).

When you select the index.html file the Silverlight program is executed.

As you can see an IronPython REPL console is activated by default in this template. This console is activated by using the following tag in the HTML file.

<param name="initParams" value="reportErrors=errorLocation, console=true" />

The nice thing about this is that you can manipulate the Silverlight application while is executing. To see an example of this, we can change the definition of the App class to be like this:

class App:

  def __init__(self):
    self.root = Application.Current.LoadRootVisual(UserControl(), "app.xaml")
    self.root.Message.Text = "Welcome to Python and Silverlight!"

theApp = App()

With this change we can now have access to the UI elements defined in XAML. For example:

For future posts I'm going to talk about specific topics on the use of Silverlight and IronPython.

A nice source of information on this topic is the following page by Michael Foord "Python in your Browser with Silverlight"

Writing Python's groupby in C#

2010-10-25T05:44:00.004-06:00

A couple of days ago, while working on a C# program, I had the necessity of grouping contiguous elements from a sequence given a property. A group needs to be created each time the value of the property changes in a similar way to the uniq Unix utility.

Python has a function called groupby which is part of the nice itertools module which does exactly what I want. For example:


>>> strList = ["abc","ert","bre","sd","ghj","awe","ew","gh"]
>>> [list(g) for i,g in groupby(strList,lambda x: len(x))]
[['abc', 'ert', 'bre'], ['sd'], ['ghj', 'awe'], ['ew', 'gh']]
>>> numList = [1,2,2,2,1,1,1,5,5]
>>> [list(g) for i,g in groupby(numList)]
[[1], [2, 2, 2], [1, 1, 1], [5, 5]]

In .NET a class called Enumerable with lots of extension methods to manipulate with sequences (IEnumerable<T>). This class includes an extension method called GroupBy which groups values according to a key. However it behaves more like SQL's Group by in that it considers all the values of the collection. For example:

csharp> using System.Collections.Generic;
csharp> var strList = new List<string>() {"abc","ert","bre","sd","ghj","awe","ew","gh"}; 

csharp> strList.GroupBy(x => x.Length);
{ { "abc", "ert", "bre", "ghj", "awe" }, { "sd", "ew", "gh" } }

csharp> var numList = new List<int>() {1,2,2,2,1,1,1,5,5}; 

csharp> numList.GroupBy(x => x);
{ { 1, 1, 1, 1 }, { 2, 2, 2 }, { 5, 5 } }

(The C# examples will be presented using the very useful Mono C# REPL)

Writing the equivalent function in C# turned out to be a nice programming excessive. Also trying to write the function using only "yield return" turned out the more challenging than I thought!.

As a first try we can write this function using intermediate collections to store the partial groups:

using System.Collections.Generic;
using System;
namespace Langexplr.Experiments 
{
  public class MyTuple<T,K>
  { 
     public T Item1 { get; set; }
     public K Item2 { get; set; }


     public static MyTuple<T1,K1> Create<T1,K1>(T1 first, K1 second)
     {
        return new MyTuple<T1,K1>() {Item1 = first, Item2 = second};
     }
  }
  public static class GroupByTests
  {
       public static IEnumerable<MyTuple<K,IList<T>>> MyGroupByWithLists<T,K>(this IEnumerable<T> en,Func<T,K> keyExtraction)
       {
          K currentKey = default(K);
          bool firstTime = true;
          IList<T> currentGroup = new List<T>();
          foreach(var aValue in en)
          {
             if (firstTime)
             {
                currentKey = keyExtraction(aValue);
                firstTime = false;
             }
             else
             {
                K tmpKey = keyExtraction(aValue);
                if (!tmpKey.Equals(currentKey))
                {
                   yield return MyTuple<K,IList<T>>.Create(currentKey, currentGroup); 
                   currentGroup = new List<T>();
                   currentKey = tmpKey;
                }

             }
             currentGroup.Add(aValue);
          }
          if (currentGroup.Count > 0)
          {
             yield return MyTuple<K,IList<T>>.Create(currentKey, currentGroup); 
          }
       }

  }
}

Here I'm defining a class called MyTuple which is very similar to .NET 4' Tuple class to store group's key and members.

This function works as expected, for example:

csharp> strList.MyGroupByWithLists(x => x.Length).Select(x => x.Item2).ToList();
{ { "abc", "ert", "bre" }, { "sd" }, { "ghj", "awe" }, { "ew", "gh" } }
csharp> numList.MyGroupByWithLists(x => x).Select(x => x.Item2).ToList();       
{ { 1 }, { 2, 2, 2 }, { 1, 1, 1 }, { 5, 5 } }

One of the interesting things about the Python version of groupby is that it doesn't create an intermediate collections for each group. The itertools module reference has the code for the groupby implementation.

Trying to write a this function with similar characteristics in C# resulted in the following (scary) code:


 public static IEnumerable<MyTuple<K,IEnumerable<T>>> MyGroupBy<T,K>(this IEnumerable<T> en,Func<T,K> keyExtraction)
 {
    K currentGroupKey = default(K);
    bool firstTime = true;
    bool hasMoreElements = false;
    bool yieldNewValue = false;

    IEnumerator<T> enumerator = en.GetEnumerator();
    hasMoreElements = enumerator.MoveNext();
    while (hasMoreElements) 
    {
       if (firstTime)
       {
          firstTime = false;
          yieldNewValue = true;
          currentGroupKey = keyExtraction(enumerator.Current); 
       }
       else
       {
          K lastKey;
          while((lastKey = keyExtraction(enumerator.Current)).Equals( currentGroupKey) &&
                (hasMoreElements = enumerator.MoveNext()))
          {

          }
          if(hasMoreElements &&
             !lastKey.Equals(currentGroupKey))
          {
             currentGroupKey = lastKey;
             yieldNewValue = true;
          }
          else
          {
             yieldNewValue = false;
          }
       }

       if (yieldNewValue) {
          yield return MyTuple<K,IEnumerable<T>>.Create(
                          currentGroupKey, 
                          ReturnSubSequence((x) => { 
                             hasMoreElements = enumerator.MoveNext();
                             return hasMoreElements &&
                                    x.Equals(keyExtraction(enumerator.Current)); },
                            enumerator,
                            currentGroupKey,
                            enumerator.Current));
       }
    }
 }
 static IEnumerable<T> ReturnSubSequence<T,K>(Predicate<K> pred, IEnumerator<T> seq,K currentElement,T first)
 {
    yield return first; 
    while( pred(currentElement)) 
    {
       yield return seq.Current;
    }
 }

Using this function we can write:


csharp> numList.MyGroupBy().Select(x => x.Item1);         
{ 1, 2, 1, 5 }
csharp> numList.MyGroupBy().Select(x => x.Item2.ToList());
{ { 1 }, { 2, 2, 2 }, { 1, 1, 1 }, { 5, 5 } }
csharp> strList.MyGroupBy(s => s.Length).Select(x => x.Item1);
{ 3, 2, 3, 2 }
csharp> strList.MyGroupBy(s => s.Length).Select(x => x.Item2.ToList());
{ { "abc", "ert", "bre" }, { "sd" }, { "ghj", "awe" }, { "ew", "gh" } }

One interesting fact about this way of writing the groupby function is that, you have to be very careful handling the resulting iterator/enumerable. From Python's groupby documentation:

The returned group is itself an iterator that shares the underlying iterable with groupby(). Because the source is shared, when the groupby() object is advanced, the previous group is no longer visible

For example in Python, the following effect occurs if we consume the complete iterator before consuming each group:

>>> l = groupby(strList,lambda s: len(s))
>>> consumed = list(groupby(strList,lambda s: len(s)))
>>> [list(g) for k,g in consumed]
[[], ['gh'], [], []]

In our C# version we have a similar restriction, for example:


csharp> var consumed = strList.MyGroupBy(s => s.Length).ToList();
csharp> consumed.Select(x => x.Item2);
{ { "abc" }, { "sd" }, { "ghj" }, { "ew" } }

Code for this post can be found here.

Using C#'s implicit type conversions from other .NET languages

2010-10-13T21:18:00.010-06:00

One interesting C# feature is the ability to define a method that implements implicit conversion from one type to another. In this post I'm going to show how to use this feature from IronPython, F#, VB.NET and IronRuby.

Example

In order to illustrate the implicit conversion feature we're going to use the following classes:


namespace Langexplr.Experiments
{
   public class Complex 
   {
       public double Real { get; set; }
       public double Img { get; set; }
       
       public static implicit operator Complex(double real) 
       {
           return new Complex() { Real = real };
       }

       public static implicit operator Polar(Complex complex) 
       {
           return new Polar() { Angle = Math.Atan(complex.Img/complex.Real),
                                Length = Math.Sqrt(complex.Real*complex.Real +
                                                   complex.Img*complex.Img) };
       }
    
       public static implicit operator double(Complex complex) 
       {
           return Math.Sqrt(complex.Real*complex.Real +
                            complex.Img*complex.Img);
       }
       
   }

   public class Polar 
   {
       public double Angle { get; set; }
       public double Length { get; set; }
   }
}

The Complex class is a simple definition of a complex number. The Polar class is defined(conveniently) to represent a complex number in polar form. The Complex class defines three implicit conversions:

From double to a complex number

From Complex to Polar

From Complex to double

The following C# code shows a use of this feature:


using Langexplr.Experiments;
using System;

class main 
{ 
  public static void Main(string[] args) 
  {    
       Complex c = 10.3;
       Polar p = new Complex() {Real = 12.3, Img = 5.2};
       double abs = c;

       Console.WriteLine("abs:{0} Polar: {1},{2}", abs ,p.Angle ,p.Length);
    } 
}

By looking at the definitions generated by the compiler for the Complex class, we can see several definitions for the op_Implicit method with different parameters and return types.

...
.method public hidebysig specialname static 
        class Langexplr.Experiments.Complex 
        op_Implicit(float64 real) cil managed
...
.method public hidebysig specialname static 
        class Langexplr.Experiments.Polar 
        op_Implicit(class Langexplr.Experiments.Complex complex) cil managed
...
.method public hidebysig specialname static 
        float64  op_Implicit(class Langexplr.Experiments.Complex complex) cil managed
...

Now these uses of the Complex class will be presented on different .NET languages.

IronPython

As described in "Dark Corners of IronPython" by Michael Foord the clr.Convert function can be used to convert between types using the op_Implicit if necessary.

For example:


import clr
clr.AddReference("ImplicitTest")

from Langexplr.Experiments import *
from System import Double

c = clr.Convert(10.3, Complex)
nC = Complex()
nC.Real = 12.3
nC.Img = 5.2
p = clr.Convert(nC, Polar)
abs = clr.Convert(c, Double)

print 'abs: %(0)f Polar: %(1)f,%(2)f\n' % \
       { '0': abs, '1' : p.Angle, '2' : p.Length }

IronRuby

IronRuby will use the op_Implicit definition if a conversion required at a particular call. I couldn't find a nice way to do this directly as with IronPython's clr.Convert . However the following function definition seems to do the trick:


def dotnet_convert(value,type)
   f =  System::Func[type,type].new {|x| x}
   f.invoke(value)
end

This conversion function works since IronRuby tries to convert the value to the expected .NET type in the call to 'invoke' .

Using this definition we can write:


require 'ImplicitTest.dll'

c = dotnet_convert(10.3,Langexplr::Experiments::Complex)
nC = Langexplr::Experiments::Complex.new
nC.Real = 12.3
nC.Img = 5.2
p = dotnet_convert(nC,Langexplr::Experiments::Polar)
abs = dotnet_convert(c,System::Double)

print "abs: #{abs} Polar: #{p.Angle},#{p.Length} \n"

F#

In F# we can call the op_Implicit method directly and F# will use type inference to determine the correct overload to use.

For example:


open Langexplr.Experiments

let c : Complex = Complex.op_Implicit 10.3
let p : Polar = Complex.op_Implicit (new Complex(Real=12.3, Img=5.2))
let abs : double = Complex.op_Implicit c

System.Console.WriteLine("1. {0} Polar: {1},{2} ", abs, p.Angle, p.Length )

There's a nice post called "F# – Duck Typing and Structural Typing" by Matthew Podwysocki, which describes a nice way to define a generic function to use the op_Implicit operator.


let inline convert (x:^a) : ^b =  ((^a or ^b) : (static member op_Implicit : ^a -> ^b) x )

This function can be used as follows:


let c2:Complex = convert 10.3
let p2:Polar = convert (new Complex(Real=12.3, Img=5.2))
let abs2:float = convert c

System.Console.WriteLine("2. {0} Polar: {1},{2} ",abs2,p2.Angle,p2.Length)

VB.NET

Finally in Visual Basic .NET the implicit conversion is used automatically as in C#. For example:


Imports System
Imports Langexplr.Experiments
Module Test
   Sub Main
      Dim c As Complex = 10.3
      Dim p As Polar = new Complex() With { _
            .Real = 12.3, _
            .Img = 5.2 _
      }
      Dim abs As Double = c
      Console.WriteLine("abs:{0} Polar: {1},{2}",abs,p.Angle,p.Length)
   End Sub
End Module

Extracting elements from Win32 resource files with F#

2010-08-23T05:37:00.001-06:00

Recently I had the necessity of extracting cursors, bitmaps and icons stored in a Win32 compiled resource script file (.res) file. Although there are tools that could do that, this seems like a good excuse to use F# and code from
a previous post: "Using F# computation expressions to read binary files".

Resource file format

The format for the compiled resource file is documented in MSDN "Resource File Formats" . According with this document each entry of the resource file is prefixed by the following header (RESOURCEHEADER) :

  DWORD DataSize;
  DWORD HeaderSize;
  DWORD TYPE;
  DWORD NAME;
  DWORD DataVersion;
  WORD  MemoryFlags;
  WORD  LanguageId;
  DWORD Version;
  DWORD Characteristics;

The DataSize field indicates the size of the resource data following the header. According to "RESOURCEHEADER Structure" The TYPE and NAME fields could be stored in two different ways. It could be a zero terminated unicode string or a int16 identifier prefixed by a -1 int16 value. Also after the type/name and entries and after the data of each resource we need to read extra padding bytes that align the entry to DWORD.

Using the code defined for reading binary files the RESOURCEHEADER structure could be specified as:

type ResourceId =
     | Numeric of int16
     | Name of char[]

let pBuilder = new BinParserBuilder()

let resourceParser = pBuilder {
     let! dataSize = RInt
     let! headerSize = RInt
     let! typePrefix = RShort   
     let! resourceType =
            if typePrefix = -1s then
                wrap(RShort, Numeric)
            else
                wrap(RZString,
                     (fun data -> Name(Array.concat
                                          [[|char(typePrefix)|];
                                           data] )))
     let! namePrefix = RShort   
     let! resourceName =
            if namePrefix = -1s then
                wrap(RShort, Numeric)
            else
                wrap(RZString,
                     (fun data -> Name(Array.concat
                                          [[|char(namePrefix)|];
                                           data] )))
     let typeAndNameLength = sizeOfResourceId resourceType +
                             sizeOfResourceId resourceName 
                             
     let! _ = RDWordAlignData(typeAndNameLength)
     
     let! dataVersion = RInt
     let! memoryFlags = RShort
     let! languageId = RShort
     let! version = RInt
     let! characteristics = RInt
     let! contents = RByteBlock(dataSize)

     let! _ = RDWordAlignData(dataSize)

     return (resourceName,resourceType,contents)
   }

Using this code we can extract each entry of the resource file by writing the following:

let file = new FileStream(fileName, FileMode.Open)
let binaryReader = new BinaryReader(file, System.Text.Encoding.Unicode)

let resources = 
  seq {
     while(file.Position < file.Length) do
       match (resourceParser.Function binaryReader) with
       | Success (t,_) -> yield (Some t)
       | Failure _ -> printf "Error!"
                      yield None
  } |>  List.ofSeq

Extracting Icons

Icons have two entries inside the resource files. The first entry has information about the image and the other contains the actual image data.

The following function writes all the icons in the resource list:

resources |>
   filterMap (fun current ->
                   cursorIconInfo current 14s getIconInfo "ico") |>
   filterMap (fun (originalid, (_, i, planes, bitcount, bytes, iconId)) -> 
                  match (List.find (isEntryWithId iconId) resources) with
                  | Some(_, _, data) ->
                         Some(originalid, i, planes, bitcount, data)
                  | _ -> None) |>
   Seq.iter (fun (name, (w, h, colorcount), planes, bitcount, contents) -> 
                  writeIcon(name, (w, h, colorcount),
                            bitcount, planes, contents))

The information about the icon is extracted using cursorIconInfo which works for the cursor and the icon entry:

let cursorIconInfo resourceInfo resourceType infoExtractFunction extension =
     match resourceInfo with
     | Some(resName,Numeric(rType),data) when rType = resourceType  -> 
            match resName,(infoExtractFunction data) with
            | Numeric(id), Success(t,_) ->
                       Some(sprintf "%s%O.%s" extension  id extension,t)
            | Name(nameChars), Success(t,_) -> 
                       Some(new String(nameChars) + "." + extension,t)
            | _,_ ->
                 printf "WARNING: Skipping resource: %O\n" id
                 None
     | _ -> None

With the data returned from this function we can get the id number of the resource file entry that contains the image data (iconCursorId). We can then get the resource entry of the icon and extract the image info from there using the following code:

let iconResEntry = pBuilder {
    let! reserved = RShort
    let! restype  = RShort
    let! rescount = RShort
    let! iconWidth = RByte
    let! iconHeight = RByte
    let! colorCount = RByte
    let! reserved = RByte
    let! planes = RShort 
    let! bitCount = RShort 
    let! bytesInRes = RInt
    let! iconCursorId = RShort
    return (reserved, restype, rescount),
           (iconWidth, iconHeight, colorCount),
           planes, bitCount, bytesInRes, iconCursorId
}

let getIconInfo (data : byte array) =
  use str = new MemoryStream(data)
  use bReader = new BinaryReader(str)
  iconResEntry.Function bReader

Image information stored in the payload section of the resource is almost the same as a .ICO file . So in order to extract the icon what we need to do is to generate part of the header of a valid ICO file. The following function does that:

let writeIcon(fileName,
              (width : byte, height : byte, bitcount : byte),
              bpp : int16,
              planes : int16,
              contents : byte array) =
    use writer = new FileStream(fileName, FileMode.Create)
    use bwriter = new BinaryWriter(writer)
    bwriter.Write(0s)
    bwriter.Write(1s)
    bwriter.Write(1s)
    bwriter.Write(byte(width))
    bwriter.Write(byte(height))
    bwriter.Write(byte(bitcount))
    bwriter.Write(byte(0))
    bwriter.Write(planes) 
    bwriter.Write(bpp)
    bwriter.Write(contents.Length)
    bwriter.Write(int32(writer.Position) + 4)
    bwriter.Write(contents)

Extracting Cursors

The process of writing the cursors is almost the same as the process with icons. The following code filters the cursor resources and extract them:

resources |>
   filterMap (fun current -> cursorIconInfo current 12s getCursorInfo "cur") |>
   filterMap (fun (name, (_, (w, h), _, bitCount, _, cursorId)) -> 
                   match List.find (isEntryWithId cursorId) resources with
                   | Some(_, _, theData) -> Some(name, bitCount, theData)
                   | _ -> None)  |>
   Seq.iter ( fun (name, bitcount, contents) ->
                 writeCursor(name, byte(bitcount),contents))

The getCursorInfo function extracts data from the cursor specific entry:

let cursorResEntry = pBuilder {
    let! reserved = RShort
    let! restype  = RShort
    let! rescount = RShort
    let! cursorWidth = RShort
    let! cursorHeight = RShort
    let! planes = RShort 
    let! bitCount = RShort 
    let! bytesInRes = RInt
    let! iconCursorId = RShort
    return (reserved, restype, rescount),
           (cursorWidth, cursorHeight),
           planes, bitCount, bytesInRes, iconCursorId
}

let getCursorInfo (data : byte array) =
  use str = new MemoryStream(data)
  use bReader = new BinaryReader(str, System.Text.Encoding.Unicode)
  cursorResEntry.Function bReader

As with the icon entry, the cursor entry has the data in almost the same format as a .CUR file so we need to generate a valid header for this file.

let writeCursor(fileName, bitcount : byte, contents : byte array) =
    use writer = new FileStream(fileName, FileMode.Create)
    use bwriter = new BinaryWriter(writer)
    bwriter.Write(0s)
    bwriter.Write(2s)
    bwriter.Write(1s)
    //Assume 32x32
    bwriter.Write(byte(32))
    bwriter.Write(byte(32))
    bwriter.Write(bitcount)
    bwriter.Write(byte(0))
    let v1 = uint16(contents.[0]) ||| (uint16(contents.[1] <<< 16))
    let v2 = uint16(contents.[2]) ||| (uint16(contents.[3] <<< 16))
    bwriter.Write(v1) 
    bwriter.Write(v2)
    bwriter.Write(contents.Length - 4)
    bwriter.Write(int32(writer.Position) + 4)
    bwriter.Write(contents,4,contents.Length - 4 )

As you can see from the code I had to assume that the resolution of the cursor is 32x32 . This is because the data didn't seem to have the correct information as specified in "CURSORDIR Structure" .

Extracting Bitmaps

Finally writing bitmap file entries is very easy. The bitmap data is stored using the DIB format which is part of the BMP file format. The following fragment shows how we extract these resources.

resources |> 
   filterMap (fun res -> match res with 
                         | Some(Numeric id, Numeric 2s, data) -> 
                                 Some(sprintf "bmp%O.bmp" id, data) 
                         | Some(Name nameChars, Numeric 2s, data) -> 
                                 Some(new String(nameChars)+".bmp", data) 
                         | _ -> None) |> 
   Seq.iter (fun (name, data) -> writeBmp(name, data))

As with the other elements, the writeBmp function writes the appropriate header according to the BMP file format.

let writeBmp(name,data:byte array) =
    use writer = new FileStream(name,FileMode.Create)
    use bwriter = new BinaryWriter(writer)
    bwriter.Write(0x42uy)
    bwriter.Write(0x4Duy)
    bwriter.Write(14  + data.Length)
    bwriter.Write(5s)
    bwriter.Write(5s)
    // According to the DIB header the image color depth
    // is located ad offset 14
    let paletteSize =
         if data.[14] < 24uy then
            int((2.0 ** float(data.[14])) * 4.0)
         else
            0         
    bwriter.Write(14 + 40 + paletteSize)
    bwriter.Write(data)

Code for this post can be found here.

Generating XML with Newspeak

2010-06-19T07:39:00.000-06:00

While experimenting with Newspeak I wrote a small utility class for generating basic XML documents.

An example of its use looks like this:

w:: getTestingWriter: output .
w element: 'foo' 
   attributes: { {'id'. 'goo'.} }
   with: [
    w element: 'foo1'.
    w element: 'foo2'.
    w element: 'foo3'
        with: [
        w element: 'foo4'
            with: [
              w text: 'Hello'.
           ].
        w cdata: ''.
        ].
   ].

This generates:

<foo id="goo"><foo1 /><foo2 /><foo3><foo4>Hello</foo4><![CDATA[<loo>]]></foo3></foo>

The code for generating XML this way is based on a previous experiment of using Python's 'with' statement to wrap the .NET's System.Xml.XmlWriter class to generate XML.

Code for this post can be found here.

Parsing XML documents with namespaces in Newspeak

2010-06-13T08:49:00.004-06:00

The previous post showed a basic grammar for XML written using Newspeak parsing combinators. Although Newspeak already includes XML support, it is a great exercise to explore another interesting parts of the language.

The grammar

The grammar defined in the previous post looks like this:

class XmlGrammar = ExecutableGrammar(
"Xml 1.0 grammar with namespaces according to http://www.w3.org/TR/REC-xml/ "
|
   openAB = char: $<.
   closeAB = char: $>.
   amp = char: $&.
   semicolon = char: $; .
   slash = char: $/.


   topenAB = tokenFromChar: $<.
   tcloseAB = tokenFromChar: $>.
   tslash = tokenFromChar: $/.

   comment = openAB , (char: $-),(char: $-), 
                        ((charExceptFor: $-) | ((char: $-), (charExceptFor: $-))) plus,
                       (char: $-),(char: $-),closeAB .

   letter = (charBetween: $a and: $z) | (charBetween: $A and: $Z).
   digit = charBetween: $0 and: $9.

   colon = char: $:.

   quote = (char:$') .
   dquote = (char:$") .   

   eq = tokenFromChar: $= .
 
   VersionNum = (char:$1), (char: $.) , (charBetween: $0 and: $9) plus.

   VersionInfo = (tokenFromSymbol: #version), eq, ((quote, VersionNum,quote) | (dquote, VersionNum, dquote )).

   EncName = letter, (letter | digit) star.

   EncodingDecl = (tokenFromSymbol: #enconding) , eq , ((quote, EncName ,quote) | (dquote , EncName , dquote  )).

   yesNo = (tokenFromSymbol: #yes) | (tokenFromSymbol: #no).

   SDDecl = (tokenFromSymbol: #standalone), eq, ((quote, yesNo ,quote) | (dquote , yesNo , dquote  )).

   XMLDecl = (char: $<) , (char: $?) ,(tokenFromSymbol: #xml), VersionInfo , EncodingDecl opt, SDDecl opt,
                     (tokenFromChar: $?), (char: $>).

   dprolog = XMLDecl.
 
   NameStartChar = letter | (char: $_) .

   NameChar =  NameStartChar | (char: $-) | (char: $.) | digit.

   Name = NameStartChar, NameChar star.

   NameWithPrefix = Name, colon, Name.

   QName = NameWithPrefix | Name.

   TQName = tokenFor: QName.

   EntityRef = amp, Name ,semicolon .

   CharRef = amp, (char: $#), (((char:$x), (digit | letter) plus) | (digit plus)) ,semicolon .

   Reference = EntityRef | CharRef.

   AttributeContent1 = (charExceptForCharIn: {$< . $". $&. }) | Reference.

 
   AttributeValue = (dquote, AttributeContent1  star,dquote) | 
                              (quote, AttributeContent1 star,quote).

   Attribute = TQName ,eq, AttributeValue.

   EmptyElemTag = topenAB ,QName, Attribute star, tslash , closeAB .
   STag = topenAB ,QName, Attribute star, tcloseAB .
   ETag = topenAB ,slash,QName, tcloseAB .

   CDStart = topenAB ,(char: $!),(char:$[),(tokenFromSymbol: #CDATA),(char:$[).
   CDEnd = (char: $]),(char: $]),(char: $>).

   CDSect = CDStart, (charExceptFor: $]) star ,   CDEnd.

   CharData  = tokenFor: ((charExceptForCharIn: {$&. $<}) plus).

   content = CharData opt, ((element | Reference | CDSect), CharData opt) star.

   ComplexElement = STag, content , ETag.

   element =  EmptyElemTag | ComplexElement  .

 
|
)
( 
  ... 
)

XML nodes

The next step was to add support for the creation of data structures representing the XML document. A technique presented in the Executable Grammars[PDF] paper suggest creating a subclass of the grammar which adds the code for creating the XML node tree.

class XmlParserWithXmlNodes = XmlGrammar (
"Basic XmlParser with XML node AST"
|
   
   
|)
('as yet unclassified'
Attribute = (
   ^ super Attribute wrapper: [:name :eq :value | {name. value.}].
)

AttributeValue = (
   |flattenString|
   ^ super AttributeValue 
               wrapper: [:q1 :chars :q2 | 
                              flattenString:: (chars collect: 
                                                 [:c | (c isString) 
                                                         ifTrue: [c at: 1] 
                                                         ifFalse: [c]]).
                              String withAll:flattenString  ].
)

CDSect = (
   ^ super CDSect wrapper: [:cs :data :ce | String withAll: data].
)

CDStart= (
   ^ super CDStart  wrapper: [:t :e :b1 :cdata :b2 | 'cdatastart'].
)

CharData = (
   ^ super CharData wrapper: [:chars | String withAll: (chars token ) ].

)

CharRef = (
   |numberStr|
   ^ super CharRef 
               wrapper: [:a1 :p :numberChars :sc | 
                         |base| 
                         numberStr:: ((numberChars at: 1) = $x) 
                                          ifTrue: [base:: 16.
                                                   String withAll: (numberChars at: 2)]
                                          ifFalse: [base:: 10.
                                                    String withAll: numberChars].

                           (Unicode value: (Number readFrom: numberStr base: base)) asString
                       ].
)

ComplexElement = (
   |tn attCollection|
   tn:: XmlNodes new.
   attCollection:: tn Attributes new.

   ^ super ComplexElement 
                   wrapper: [:s :childNodes :e | 
                                 (e asString = s name asString) 
                                       ifFalse: [error: 'Open tag different from closing tag'].
                                 s childNodes: childNodes.
                                 s].
)

ETag = (
   ^ super ETag wrapper: [:o :s :name :c | name].
)

EmptyElemTag = (

   ^super EmptyElemTag 
                     wrapper: [:oab :name :atts :s :cab | 
                               |tn attCollection|
                               tn:: XmlNodes new.
                               attCollection:: tn Attributes new.

                               atts do: [:att | attCollection addAttributeWithQName: ((att at: 1) token) value: (att at:2)].
                               tn Element name: name attributes: attCollection childNodes:{} ].
)

EntityRef = (
   ^ super EntityRef 
                   wrapper: [:a :name :sc | 
                                 (entities  valueForName: ((name at: 1) at: 1)) asString ].
)

Name = (
   ^ super Name  wrapper: [:startChar :rest | { { (startChar asString), (String withAll: rest) } } ].
)

NameWithPrefix = (
   |tn|
   tn:: XmlNodes new.
   ^ super NameWithPrefix 
                 wrapper: [ :prefix :c  :name | 
                                   {{(prefix at: 1). (name at: 1).}} ].
)

QName = (
      |tn|
      tn:: XmlNodes new.
   ^ super QName  wrapper: [ :data | ((data size) = 2) 
                                        ifTrue:[tn QualifiedName prefix: ((data at:1) at: 1) 
                                                                 localPart:((data at: 2) at:1)]
                                        ifFalse:[tn QualifiedName prefix: nil localPart: (data at: 1)]]
)

STag = (
   ^super STag 
            wrapper: [:oab :name :atts :cab |                                        
                         |tn attCollection|
                         tn:: XmlNodes new.
                         attCollection:: tn Attributes new.
                         "Add attributes"
                         atts do: [:att | attCollection addAttributeWithQName: ((att at: 1) token) 
                                                        value: (att at:2)].
                         "Create elements"
                         tn Element name: name attributes: attCollection childNodes:{} ].
)

VersionInfo = (
   ^super VersionInfo 
               wrapper: [:v :e :versionTextList | versionTextList at: 2].
)

VersionNum = ( 
   ^super VersionNum  
               wrapper: [:one :dot :num | 
                            Number 
                                readFrom: (String withAll: {one.dot},(String withAll: num))].
)

XMLDecl = (
   |tn|
   tn:: XmlNodes new.
   ^super XMLDecl 
             wrapper: [:c1 :c2 :x :versionInfo :enc :sd :c3 :c4 | 
                           tn TAstXmlDecl version: versionInfo 
                                          encoding: enc
                                          standalone: sd] .
)

content = (
   |result|

   ^ super content wrapper: [:chars :cseq |
                   (chars = nil) 
                       ifTrue: [result:: {}] 
                       ifFalse: [result:: {chars}].
                   cseq inject: result into: [:total :current  |  
                               result::  addCompactingStrings: (current at: 1) to: result  .
                               (current at: 2) ifNotNil: [:c | addCompactingStrings: c to: result   ].
                               result ].
         ].
)

...

)

XML Namespaces

Support for XML namespace resolution was the next step. As described in the Namespaces in XML 1.0 document, XML can have different namespaces for its elements. For example:

<docs:letter docs:xmlns="http://www.foo.com/docs"
             pic:xmlns="http://www.foo.com/docs/links">
   <docs:paragraph>some text1</docs:paragraph> 
   <pic:pictureRef location="goo/moo/loo"/>
   <docs:paragraph>some text2</docs:paragraph> 
</docs:letter>

In this case the 'docs:xmlns' and 'pic:xmlns' attributes associates the 'docs' and 'pics' prefixes to the specified namespaces. So the namespace of the 'letter' element is 'http://www.foo.com/docs'.

Also default namespaces could be specified, like this:

<letter xmlns="http://www.foo.com/docs">
             
   <paragraph>some text1</paragraph> 
   <pictureRef location="goo/moo/loo" 
               xmlns="http://www.foo.com/docs/links"  />
   <paragraph>some text2</paragraph> 
</letter>

The basic idea is that xmlns attributes define an scope where a set of prefix/namepace associations and a default namespace are valid. For each start tag a new scope with new declarations must be in context and each end tag must remove the last scope.

Extending the parsing context

In order to do add this functionality, we need to be able to keep a stack with the scopes of namespace declarations that is modified each time we enter or exit from a element declaration. Fortunately, the parser combinator library includes a ParserContext class which is used for keeping track parsing errors.

One problem to use ParserContext was that an instance of it is created inside the CombinatorialParser parse: method which is part of the parsing library. This instance is configured in certain way to process parsing errors. So in order to create add a extended context we needed to somehow replace the ParserContext class with a new implementation.

Luckly, as described in the Modules as Objects in Newspeak[PDF] paper in Newspeak you can override inner class definitions just like you override methods in any other OO language. This means that I can extend the parser library and since ParserContext is defined as an inner class, override its definition to add my own parsing context. The definition of the extended parsing library looks like this:

class ParserLibraryWithXmlContext = parserLibraryClass usingLib: platform (
"A parser library with overwritten parsing context for Xml namespace resolution"
|
   parserLibContext =  super ParserContext .  
|
)
(

class ParserContext =  parserLibContext(
"An Xml parser context that keeps track of namespaces."
|
   protected prefixes = collections MutableArrayList new:  10.
|
)
('as yet unclassified'
addPrefix: prefix for: namespace = (
   |lastLevel|
   lastLevel:: prefixes at: (prefixes size).
   lastLevel at: prefix put: namespace.
   
)

namespaceFor: prefix = (
   |levelIndex|

   levelIndex:: prefixes findLast: [:lvl | (lvl includesKey: prefix) ].
   ^(levelIndex > 0)
            ifTrue: [(prefixes at: levelIndex) at: prefix]
            ifFalse: [nil].
)

popLevel = (
   prefixes pop.
)

pushLevel = (
   prefixes push: (collections MutableHashedMap new).
)



))

This is pretty nice since it allowed us to inject our own implementation of ParsingContext (which extends the original!). An interesting thing to notice is the definition of the parserLibContext which is bound to the base definition of the ParserContext class which we need in order to inherit from it.

Using the extended context

Now having added the new parsing context the next step is to add the code to manipulate the context. In order to do that, we need to add a some functionality to the STag, EmptyElement and ETag productions of the grammar so we can push and pop namespace scopes for each element. In other to do this without modifying the existing functionality three wrappers were created for each of this productions:


class ParserWithNamespaceForStartTag = CombinatorialParser (
"A parser that takes care of modyfing the parsing context for a start tag"
|
    innerparser
|
)
('as yet unclassified'
forParser: p = (
    innerparser:: p.
)

parse: input inContext: context ifError: blk = (
    |result xmlnsAttributes elementName elementNamespace attName|
    result:: innerparser parse: input inContext:context ifError:blk.
    "First update the context with the newest prefix declarations"
    context pushLevel.
    xmlnsAttributes:: result attributes allAttributesWithLocalName: 'xmlns'.
    xmlnsAttributes 
             do: [:aPair | context addPrefix: ((aPair at: 1) prefix)
                                            for: (aPair at: 2)].
    "Update the element"
    elementName:: result name.
    elementNamespace:: context namespaceFor: elementName prefix.
    elementName namespace: elementNamespace.

    "Update the attributes"
    result attributes 
             do: [:aPair | 
                     attName:: aPair at: 1.
                     attName namespace: 
                              (context namespaceFor: attName prefix)
                  ].

    ^result.
)

class ParserWithNamespaceForStartEndTag =ParserWithNamespaceForStartTag(
"Parser that takes care of modifying the parsing context for namespace declarations for self closing tags."
|

|
)
('as yet unclassified'
parse: input inContext: context ifError: blk = (
    |result|
    result:: super parse: input inContext: context ifError: blk.
    context popLevel.
    ^result
)

)

class ParserWithNamespaceForEndTag = CombinatorialParser (
"A parser that takes care of modyfing the parsing context for a end tag"
|
    protected innerParser = nil.
    
|
)
('as yet unclassified'
forParser: parser = (
    innerParser:: parser.
)

parse: input inContext: context ifError: blk = (
    |result|
    result:: innerParser parse: input inContext: context ifError: blk.
    context popLevel.
    ^result
)

As shown here the start tag parser pushes a new scope with the new namespace declarations, while the end tag parser pops a scope.

Now to add this wrappers I created a new subclass of the parser with nodes to add this functionality.

class XmlParserWithNodesAndNamespaces = XmlParserWithXmlNodes (
"An XML Parser that creates a node tree and that resolves the namespaces."
|
        
|
)
('as yet unclassified'
ETag  = (
    |newWrappingParser|
    newWrappingParser:: ParserWithNamespaceForEndTag new.
    newWrappingParser forParser: (super ETag).
    ^newWrappingParser
)

EmptyElemTag = (
    |newWrappingParser|
    newWrappingParser:: ParserWithNamespaceForStartEndTag new.
    newWrappingParser forParser: (super EmptyElemTag).
    ^newWrappingParser
)

STag = (
    |tn attCollection newWrappingParser|
    newWrappingParser:: ParserWithNamespaceForStartTag new.
    newWrappingParser forParser: (super STag).
    ^newWrappingParser
)

)

Code organization

The code for this experiment is organized as follows:

class  XmlTools withParserLibClass: parserLibraryClass usingLib: platform = 
(
...
) (
   class XmlParsing withParsingLib: parserLibrary = (
   ...
   ) 
   (
       class XmlGrammar = ExecutableGrammar  
           ( ...) ( ... )
       class XmlParserWithXmlNodes = XmlGrammar
           ( ... ) ( ... )
       class XmlParserWithNodesAndNamespaces = XmlParserWithXmlNodes 
           ( ... ) ( ... )
       ... 
   )

   basicParser = (
       |parsingLib xmlparsing|
       parsingLib:: parserLibraryClass usingLib: platform.
       xmlparsing:: XmlParsing withParsingLib: parsingLib.
       ^xmlparsing XmlParserWithXmlNodes new.
   )

   parserWithNamespaceSupport = (
       |parsingLib xmlparsing|
       parsingLib:: ParserLibraryWithXmlContext new.
       xmlparsing:: XmlParsing withParsingLib: parsingLib.
       ^xmlparsing XmlParserWithNodesAndNamespaces new.
   )
   ...
)

Here the XmlTools class definition will represent a module for XML utilities. Its inner definitions include the XmlParsing inner class which defines the grammar along other parsing utilities. The basicParser parserWithNamespaceSupport show an example of how the parser is created.

The following methods show how the XmlTools class is used to create a concrete parser for a basic XML document:

getTestingNsParser = (
    |platform|
    platform:: Platform new.
    ^(XmlTools 
            withParserLibClass: BlocklessCombinatorialParsing  
            usingLib: platform) parserWithNamespaceSupport.
)

testElementWithOneChildWithNamespaces = (
    |parser r ctxt|
    parser:: xmlNsParserWrapper: (getTestingNsParser element) .
    r:: parser parse: (streamFromString: '') .
                     
    assert:[r childNodes size = 1].
    assert:[r name asString = 'myElement'].
    assert:[r name namespace = 'http://foo'].
    assert:[((r childNodes at: 1) name asString) = 'childElement1'].
    assert:[((r childNodes at: 1) name namespace) = 'http://foo'].
)

Final words

The nicest think to notice is that the original class containing the grammar for XML was not modified in order to introduce this feature. In fact the parser with XML nodes without namespaces is also available . Also I really liked the way the ParserContext class was replaced which automatically allowed me to add this new functionality.

Code for this post can be found here.

A simple XML Grammar in Newspeak

2010-05-24T05:29:00.005-06:00

Recently I've been doing some experiments for parsing XML in using Newspeak parser combinators.

Here's the grammar:


class XmlGrammar = ExecutableGrammar(
"Xml 1.0 grammar with namespaces"
|
   openAB = char: $<.
   closeAB = char: $>.
   amp = char: $&.
   semicolon = char: $; .
   slash = char: $/.


   topenAB = tokenFromChar: $<.
   tcloseAB = tokenFromChar: $>.
   tslash = tokenFromChar: $/.

   comment = openAB , (char: $-),(char: $-), 
                        ((charExceptFor: $-) | ((char: $-), (charExceptFor: $-))) plus,
                       (char: $-),(char: $-),closeAB .

   letter = (charBetween: $a and: $z) | (charBetween: $A and: $Z).
   digit = charBetween: $0 and: $9.

   colon = char: $:.

   quote = (char:$') .
   dquote = (char:$") .   

   eq = tokenFromChar: $= .
 
   VersionNum = (char:$1), (char: $.) , (charBetween: $0 and: $9) plus.

   VersionInfo = (tokenFromSymbol: #version), eq, ((quote, VersionNum,quote) | (dquote, VersionNum, dquote )).

   EncName = letter, (letter | digit) star.

   EncodingDecl = (tokenFromSymbol: #enconding) , eq , ((quote, EncName ,quote) | (dquote , EncName , dquote  )).

   yesNo = (tokenFromSymbol: #yes) | (tokenFromSymbol: #no).

   SDDecl = (tokenFromSymbol: #standalone), eq, ((quote, yesNo ,quote) | (dquote , yesNo , dquote  )).

   XMLDecl = (char: $<) , (char: $?) ,(tokenFromSymbol: #xml), VersionInfo , EncodingDecl opt, SDDecl opt,
                     (tokenFromChar: $?), (char: $>).

   dprolog = XMLDecl.
 
   NameStartChar = letter | (char: $_) .

   NameChar =  NameStartChar | (char: $-) | (char: $.) | digit.

   Name = NameStartChar, NameChar star.

   QName = (Name, colon, Name)| Name.

   TQName = tokenFor: QName.

   EntityRef = amp, Name ,semicolon .

   CharRef = amp, (char: $#), (((char:$x), (digit | letter) plus) | (digit plus)) ,semicolon .

   Reference = EntityRef | CharRef.

   AttributeContent1 = (charExceptForCharIn: {$< . $". $&. }) | Reference.

 
   AttributeValue = (dquote, AttributeContent1  star,dquote) | 
                              (quote, AttributeContent1 star,quote).

   Attribute = TQName ,eq, AttributeValue.

   EmptyElemTag = topenAB ,QName, Attribute star, tslash , closeAB .
   STag = topenAB ,QName, Attribute star, tcloseAB .
   ETag = topenAB ,slash,QName, closeAB .

   CDStart = topenAB ,(char: $!),(char:$[),(tokenFromSymbol: #CDATA),(char:$[).
   CDEnd = (char: $]),(char: $]),(char: $>).

   CDSect = CDStart, (charExceptFor: $]) star ,   CDEnd.

   CharData  = tokenFor: ((charExceptForCharIn: {$&. $<}) plus).

   content = CharData opt, ((element | Reference | CDSect), CharData opt) star.

   ComplexElement = STag, content , ETag.

   element =  EmptyElemTag | ComplexElement  .

 
|
)
...

Code for this experiment can be found here.

Some basic image processing operations with F#

2010-03-11T06:02:00.002-06:00

The previous post presented a way to access the image data from the Webcam using DirectShow.Net and F#. We can manipulate this data to do some basic image processing operations with it.

Converting image to gray scale

Many image processing operations and algorithms are defined for grayscale images. A simple way to convert the image to grayscale is the following:


let grayGrabber(transform) = 
     fun (width,height) ->
      {  new ISampleGrabberCB with
             member this.SampleCB(sampleTime:double , pSample:IMediaSample )= 0                     
             member this.BufferCB(sampleTime:double , pBuffer:System.IntPtr , bufferLen:int) =                       
                let grayImage = getGrayImage pBuffer bufferLen
                                
                let resultImage = transform width height grayImage
                
                saveGrayImageToRGBBuffer resultImage  pBuffer bufferLen                                                                   
                0
                }

Given that we have:


let inline getGrayImage (data:IntPtr) (size:int)  = 
    let grayImage = Array.create (size/3) (byte(0))
    let pixelBuffer = Array.create 3 (byte(0))
    let mutable it = 0
    for i in 0..(size - 3 ) do 
       if (i + 1) % 3 = 0 then 
          Marshal.Copy(new System.IntPtr(data.ToInt32()+i),pixelBuffer,0,3) |> ignore
          grayImage.[it] <-  byte(float(pixelBuffer.[0])*0.3 + 
                                  float(pixelBuffer.[1])*0.59 + 
                                  float(pixelBuffer.[2])*0.11)
          it <- it+1
    grayImage

let inline saveGrayImageToRGBBuffer (grayImage:byte array) (data:IntPtr) (size:int) =
    let mutable targetIndex = 0
    for i in 0..(size/3 - 1) do
      let p = grayImage.[i]
      Marshal.WriteByte(data,targetIndex,p)
      Marshal.WriteByte(data,targetIndex+1,p)
      Marshal.WriteByte(data,targetIndex+2,p)
      targetIndex <- targetIndex+3  
    ()

The technique for converting the image to grayscale was taken from here.

Using this new grabber definition we can change the code from the previous post to do this:


let mediaControl,filterGraph = createVideoCaptureWithSampleGrabber 
                                      device 
                                      nullGrayGrabber
                                      None

Given that nullGrayGrabber is defined as:

let nullGrayGrabber = grayGrabber (fun (_:int) (_:int) image -> image)

The original webcam output looks like this:

By applying this filter the webcam output looks like this:

Applying templates

With this definitions we can do a common operation in image processing called template convolution. Basically, this technique consists in applying an operation to each pixel of an image and its surrounding neighborhood . The operation is represented by a matrix, for example:


0.0  1.0 0.0
1.0 -1.0 1.0
0.0  1.0 0.0

To apply this template to the a pixel of the image at x',y' we write:


(0.0*image[x-1,y-1]) + (1.0*image[x,y-1]) + (0.0*image[x+1,y-1]) +
(1.0*image[x-1,y])   + (-1.0*image[x,y]) +  (1.0*image[x+1,y]) +
(0.0*image[x-1,y+1]) + (1.0*image[x,y+1]) + (0.0*image[x+1,y+1])

The function to apply this kind of operation looks like this:

let convolveGray3 w h (image:byte array) (template:float[,]) hhalf whalf =
      
   let result = Array.create (image.Length) (byte(0))
   
   let getPixelGray' = getPixelGray w h image
   let setPixelGray' = setPixelGray w h result
   
   for y in (hhalf + 1) .. (h - ((Array2D.length1 template) - hhalf - 1) - 1 ) do
       for x in (whalf + 1) .. (w - ((Array2D.length2 template) - whalf - 1) - 1 ) do
          
          let mutable r = 0.0
          
          for ty in 0 .. (Array2D.length1 template - 1) do
          for tx in 0 .. (Array2D.length2 template - 1) do
             let ir = getPixelGray' ( x + (tx - whalf)) ( y + (ty - hhalf))
             r <- r + template.[ty ,tx ]*float(ir)  
             
          setPixelGray' x y (byte(Math.Abs r))
      
   result

Given that:

let inline getPixelGray width height (image:byte array) x y  =
   let baseHeightOffset = y*width
   let offset = baseHeightOffset + x
   image.[offset]

let inline setPixelGray width height (image:byte array) x y value1 =
   let baseHeightOffset = y*width
   let offset = baseHeightOffset + x
   image.[offset] <- value1  
   ()

We can represent operations such as edge detection or smoothing.
First order edge detection can be represented using the following template:


let firstOrderEdgeDetectTemplate = 
             (array2D [|[|2.0;-1.0|];
                        [|-1.0;0.0|];|])

Changing the code of the main program to the following:

let convGrayGrabber1 = 
    grayGrabber (fun width height grayImage -> convolveGray3 width height grayImage firstOrderEdgeDetectTemplate 0 0 )

Now the output looks like:

We can also use a template template to represent averaging. This technique removes detail from the image by calculating the average value of pixel given its neighborhood. The definition of a function:

let averagingTemplate (windowSize) =
  Array2D.create windowSize windowSize (1.0/float(windowSize*windowSize))

This function creates a template which looks like this:


> averagingTemplate 3;;
val it : float [,] = [[0.1111111111; 0.1111111111; 0.1111111111]
                      [0.1111111111; 0.1111111111; 0.1111111111]
                      [0.1111111111; 0.1111111111; 0.1111111111]]

We can change again the main program to use this function:

let averagingGrayGrabber = 
    let template = averagingTemplate 3
    grayGrabber (fun width height grayImage -> convolveGray3 width height grayImage template 1 1 )

...

let mediaControl,filterGraph = createVideoCaptureWithSampleGrabber 
                                      device 
                                      averagingGrayGrabber 
                                      None

The result image looks like this:

Final words

As with the previous post I'm using mainly the imperative features of F# . For future posts I'll try change this and also continue the exploration of image processing with F#.

Most of the techniques described here were taken from the "Feature Extraction & Image Processing" book by Mark S. Nixon and Alberto S. Aguado.

Code for this post can be found here.

Using a Webcam with DirectShowNET and F#

2010-02-24T06:00:00.001-06:00

In this post I'm going to show a small F# example of using DirectShowNET to access a webcam and manipulate the image data.

DirectShow

DirectShow is a huge Windows API for video playback and capture. Among many things this API allows flexible access to data captured by video input devices such as a webcam.

DirectShowNET

The DirectShow is COM based API. According to the documentation, it's meant to be used from C++. Luckily there's DirectShowNET which is a very nice library that exposes the DirectShow interfaces to .NET languages .

Accessing the webcam

What I wanted to do for this example is to have access to the data of the image being captured at a given time. The DirectShow API provides the concept of a Sample Grabber which not only gives you access to the captured data, but also allows its modification .

The following example shows the use of a couple of functions defined below:


let device = findCaptureDevice

let mediaControl,filterGraph = createVideoCaptureWithSampleGrabber 
                                      device 
                                      nullGrabber 
                                      None



let form = new Form(Size = new Size(300,300), Visible = true,Text = "Webcam input")


let videoWindow = configureVideoWindow (form.Handle) 300 300 filterGraph

form.Closing.Add (fun _ -> 
                      mediaControl.StopWhenReady() 
                      Marshal.ReleaseComObject(videoWindow) |> ignore
                      Marshal.ReleaseComObject(mediaControl) |> ignore
                      Marshal.ReleaseComObject(filterGraph) |> ignore)
                      

mediaControl.Run() |> ignore

Application.Run(form)

Running this program will show the following window:

Notice that we called the createVideoCaptureWithSampleGrabber function using the nullGrabber parameter. This parameter specifies a sample grabber that does nothing with the current frame. Here's the definition:


let nullGrabber =
    fun (width,height) ->
          {  new ISampleGrabberCB with
             member this.SampleCB(sampleTime:double , pSample:IMediaSample )= 0                     
             member this.BufferCB(sampleTime:double , pBuffer:System.IntPtr , bufferLen:int) = 0 
          }

We can change this sample grabber to something more interesting:


let incrementGrabber = 
     fun (width,height) ->
      {  new ISampleGrabberCB with
             member this.SampleCB(sampleTime:double , pSample:IMediaSample )= 0                     
             member this.BufferCB(sampleTime:double , pBuffer:System.IntPtr , bufferLen:int) =                       
                for i = 0 to (bufferLen - 1) do
                   let c = Marshal.ReadByte(pBuffer,i)
                   Marshal.WriteByte(pBuffer,i ,if c > byte(150) then byte(255) else c+byte(100))
                0 }

We can change the program to use:


let mediaControl,filterGraph = createVideoCaptureWithSampleGrabber 
                                      device 
                                      incrementGrabber 
                                      None

Running the program we can see:

This new sample grabber increments each pixel value by 100 or leaves the maximum value to prevent overflow. As presented here the pixel information is provided as an unmanaged pointer.

Using DirectShowNet

DirectShow also has the concept of filters which are software components that are assembled together to mix various inputs and outputs. For this examples we're going to used only a couple of interfaces. The code for this post is based on the DxLogo sample provided with DirectShowNET.

First we define the module containing these functions:


module LangExplrExperiments.DirectShowCapture

open DirectShowLib
open System.Runtime.InteropServices
open System.Runtime.InteropServices.ComTypes


let private checkResult hresult = DsError.ThrowExceptionForHR( hresult )

The checkResult function is an utility function to check for the HRESULT of some of the calls to COM interfaces. Since DirectShowNET is a thin wrapper for these interfaces we need to use this function to check for errors.

The findCaptureDevice returns the first capture devices detected by DirectShowNET .


let findCaptureDevice = 
   let devices = DsDevice.GetDevicesOfCat(FilterCategory.VideoInputDevice)
   let source : obj ref = ref null
   devices.[0].Mon.BindToObject(null,null,ref typeof<IBaseFilter>.GUID,source)
   devices.[0]

The following functions define the capture filter and sample grabber. The createVideoCaptureWithSampleGrabber function is the entry point. We can specify a file name as the final parameter to save the captured data to a video file.


let private ConfigureSampleGrabber( sampGrabber:ISampleGrabber,callbackobject:ISampleGrabberCB) =
    let media = new AMMediaType()
        
    media.majorType <- MediaType.Video
    media.subType <- MediaSubType.RGB24
    media.formatType <- FormatType.VideoInfo
    
    sampGrabber.SetMediaType( media ) |> checkResult    
    DsUtils.FreeAMMediaType(media);
            
    sampGrabber.SetCallback( callbackobject, 1 ) |> checkResult
    

let getCaptureResolution(capGraph:ICaptureGraphBuilder2 , capFilter:IBaseFilter) =        
    let o : obj ref = ref null
    let media : AMMediaType ref = ref null            
    let videoControl = capFilter :?> IAMVideoControl

    capGraph.FindInterface(new DsGuid( PinCategory.Capture), 
                           new DsGuid( MediaType.Video), 
                           capFilter, 
                           typeof<IAMStreamConfig>.GUID, 
                           o ) |> checkResult    
                           
    let videoStreamConfig = o.Value :?> IAMStreamConfig;
                
    videoStreamConfig.GetFormat(media) |> checkResult    
        
    let v  = new VideoInfoHeader()
    Marshal.PtrToStructure( media.Value.formatPtr, v )
    DsUtils.FreeAMMediaType(media.Value)
    
    v.BmiHeader.Width,v.BmiHeader.Height

let createCaptureFilter (captureDevice:DsDevice) 
                        (sampleGrabberCBCreator: int*int -> ISampleGrabberCB) = 
   let captureGraphBuilder =  box(new CaptureGraphBuilder2()) :?> ICaptureGraphBuilder2
   let sampGrabber = box(new SampleGrabber()) :?> ISampleGrabber;
   let filterGraph = box(new FilterGraph()) :?> IFilterGraph2   
   let capFilter: IBaseFilter ref = ref null
   
   captureGraphBuilder.SetFiltergraph(filterGraph) |> checkResult   
   
   filterGraph.AddSourceFilterForMoniker(
                      captureDevice.Mon,
                      null,
                      captureDevice.Name,
                      capFilter) |> checkResult
                      
   
   let resolution = getCaptureResolution(captureGraphBuilder,capFilter.Value)   
   ConfigureSampleGrabber(sampGrabber, sampleGrabberCBCreator(resolution) )
   
   filterGraph.AddFilter(box(sampGrabber) :?> IBaseFilter , "FSGrabberFilter") |> checkResult
                 
           
   captureGraphBuilder,filterGraph,sampGrabber,capFilter.Value
   
let getMediaControl (captureGraphBuilder :ICaptureGraphBuilder2)
                    (sampGrabber: ISampleGrabber)
                    (capFilter:IBaseFilter)
                    (filterGraph:IFilterGraph2)
                    (fileNameOpt:string option)=
  let muxFilter: IBaseFilter ref = ref null
  let fileWriterFilter : IFileSinkFilter ref = ref null
  try        
      match fileNameOpt with
        | Some filename ->            
            captureGraphBuilder.SetOutputFileName(
                                                    MediaSubType.Avi, 
                                                    filename,                          
                                                    muxFilter,  
                                                    fileWriterFilter) |> checkResult
            
        | None -> ()
        
      captureGraphBuilder.RenderStream(
                                        new DsGuid( PinCategory.Capture),
                                        new DsGuid( MediaType.Video),
                                        capFilter,
                                        sampGrabber :?> IBaseFilter,
                                        muxFilter.Value) |> checkResult
      
  finally
      if fileWriterFilter.Value <> null then
         Marshal.ReleaseComObject(fileWriterFilter.Value) |> ignore
      if muxFilter.Value <> null then
         Marshal.ReleaseComObject(muxFilter.Value) |> ignore     
         
  filterGraph :?>  IMediaControl  
  

let createVideoCaptureWithSampleGrabber (device:DsDevice) 
                                        (sampleGrabberCBCreator: int*int -> ISampleGrabberCB) 
                                        (outputFileName: string option) =
   let capGraphBuilder,filterGraph,sampGrabber,capFilter = createCaptureFilter device sampleGrabberCBCreator
   
   let mediaControl = getMediaControl capGraphBuilder sampGrabber capFilter filterGraph outputFileName
   
   Marshal.ReleaseComObject capGraphBuilder |> ignore
   Marshal.ReleaseComObject capFilter |> ignore
   Marshal.ReleaseComObject sampGrabber |> ignore
   
   mediaControl,filterGraph

Also for the creation of the video window the following function is provided:


let configureVideoWindow windowHandle width height (filterGraph:IFilterGraph2) =
   let videoWindow  = filterGraph :?> IVideoWindow

   videoWindow.put_Owner(windowHandle) |> checkResult   
   videoWindow.put_WindowStyle(WindowStyle.Child ||| WindowStyle.ClipChildren) |> checkResult   
   videoWindow.SetWindowPosition(0,0,width,height) |> checkResult   
   videoWindow.put_Visible(OABool.True)|> checkResult
   
   videoWindow

For future posts I'm going to try some experiments with the pixel data provided by the sample grabber.

Using a Webcam with WIA and F#

2010-02-01T06:42:00.006-06:00

In this post I'm going to show a small example of taking a picture using a Webcam with Windows Image Adquisition 1.0 . This API seems to have changed in Vista and above, the following code only applies to XP.

First step, create the interop assemblies:


C:\test\> TlbImp c:\WINDOWS\system32\wiascr.dll
Microsoft (R) .NET Framework Type Library to Assembly Converter 3.5.30729.1
Copyright (C) Microsoft Corporation.  All rights reserved.

Type library imported to WIALib.dll

C:\test>TlbImp c:\WINDOWS\system32\wiavideo.dll
Microsoft (R) .NET Framework Type Library to Assembly Converter 3.5.30729.1
Copyright (C) Microsoft Corporation.  All rights reserved.

Type library imported to WIAVIDEOLib.dll

Now we launch FSI and load the assemblies:


C:\test> fsi.exe

Microsoft F# Interactive, (c) Microsoft Corporation, All Rights Reserved
F# Version 1.9.7.8, compiling for .NET Framework Version v2.0.50727

For help type #help;;

>  #r "WIALib.dll";;

--> Referenced 'C:\development\blog\wiafs\WIALib.dll'

>  #r "WIAVIDEOLIB.dll";;

--> Referenced 'C:\development\blog\wiafs\WIAVIDEOLIB.dll'

Now we can list the capture devices:


> wia.Devices |> Seq.cast |> Seq.map (fun (x:WIALib.IWiaDeviceInfo) -> x.Name);;

val it : seq<string> = seq ["My iPod"; "Logitech QuickCam Chat"]

Now I'm going to use the QuickCam to take the picture so we get the information for this device (of type 'StreamingVideo'):


> let devInfo = wia.Devices |> Seq.cast |> Seq.filter (fun (x:WIALib.IWiaDeviceInfo) -> x.Type.Contains("StreamingVideo")) |> Seq.head;;

val devInfo : WIALib.IWiaDeviceInfo

> devInfo.Name;;
val it : string = "Logitech QuickCam Chat"

Now we create an instance of the 'WIAVIDEOLib.WiaVideoClass' which will let us use the Webcam.


> let vid = new WIAVIDEOLib.WiaVideoClass();;
Binding session to 'C:\development\blog\wiafs\WIAVIDEOLib.dll'...

val vid : WIAVIDEOLib.WiaVideoClass

>  vid.ImagesDirectory <- "c:\\temp";;
val it : unit = ()
> let h = ref (new WIAVIDEOLib._RemotableHandle());;

val h : WIAVIDEOLib._RemotableHandle ref =
  {contents = WIAVIDEOLib._RemotableHandle;}

> vid.CreateVideoByWiaDevID(devInfo.Id,h,0,1);;
val it : unit = ()

The 'ImagesDirectory' property specifies the location for the captured images.

Now we can take a picture:


> let outFileName = ref "";;

val outFileName : string ref = {contents = "";}

> vid.TakePicture( outFileName );;
val it : unit = ()
> outFileName;;
val it : string ref = {contents = "c:\temp\Imagen 009.jpg";}

The 'outFileName' parameter gets the name of the saved image in the images directory.

Adding automatic semicolon insertion to a Javascript parser

2009-11-23T06:03:00.001-06:00

A couple of weeks ago I wrote a blog post about a Javascript parser written using the Newspeak parsing combinators. As mentioned in that post, no semicolon insertion was supported. This post shows how the feature was added.

Automatic semicolon insertion

As detailed in section 7.9 of the ECMA 262 document[PDF], in Javascript you can use newline as statement separator in some scenarios. For example a semicolon is "implicitly inserted" if expression-statements are separated by line terminators:


if (condition) {
   print("A")
   print("B")
}

This code snippet is equivalent to:


if (condition) {
   print("A");
   print("B");
}

Solution

In the a original post about the parser, espin pointed me out to a paper[PDF] by A. Warth that mentions how the semicolon insertion problem was solved in a Javascript parser written in OMeta. The solution presented is this post is based on the one from the paper.

I wanted to isolate the code that performs this function. So in order to add this functionality I created a subclass that overrides the productions that get involved in this process. This way we can have both a parser with and without the feature. Here's the code:


class JSGrammarWithSemicolonInsertion = JSGrammar (
"Parser features that add automatic semicolon insertion"
|
 
    specialStatementTermination = ((( cr | lf ) not & whitespace ) star,
                                       (semicolon | comment | lf | cr | (peek: $})) ) 
                                       wrapper: [ :ws :terminator | | t | t:: Token new. t token: $;. t].

    returnStatement =  return, (specialStatementTermination | 
                                (expression ,  specialStatementTermination)).

    breakStatement = break,  (specialStatementTermination | 
                              (identifier ,  specialStatementTermination)). 

    continueStatement = continue,  (specialStatementTermination | 
                                    (identifier ,  specialStatementTermination)). 

    whitespaceNoEOL = (( cr | lf ) not & whitespace ) star,
                          (((peek: (Character cr)) | (peek: (Character lf))) not) .

    throwStatement = throw,  whitespaceNoEOL , expression ,  specialStatementTermination. 

    expressionStatement = (((function | leftbrace) not) & expression), specialStatementTermination.

    variableStatement = var, variableDeclarationList, specialStatementTermination.
| 
)

The result of parsing the following code:


var x = 0
while (true) {
   x++
   document.write(x)
   if ( x > 10)
     break
   else continue
}

... is presented using the utility created for the previous post:

Code for this post is available here.

A quick look at J

2009-10-19T06:22:00.002-06:00

In this post I'm going to show a small overview of the J programming language. An example of polynomial multiplication is examined.

J

J is an array programming language derived from APL which means is good for manipulating arrays and matrices. Its syntax and semantics are very different from other languages which makes it an interesting topic for studying.

There's a lot of documentation and examples available from the J software website. Two complete tutorials are "Learning J" by Roger Stokes and "J for C programmers" by Henry Rich.

The J distribution also includes examples and tutorials. The examples will be presented using J's REPL called jconsole.

The example

In order to give an overview of the language I'm going to start from the definition of a function that performs polynomial multiplication and examine how it was constructed.

The function is defined as follows:

polymulti =: dyad : '+/ (((_1 * i. #y) (|. "0 1) ((x (*"_ 0) y) (,"1 1) (((#y) - 1) $ 0))) , 0)'

Writing this example helped me understand some of the J's basic concepts (I'm completely sure there's a better/more efficient way to do this!).

Polynomial multiplication

The basic technique for polynomial multiplication consists on multiplying each term of one of the polynomials by the order and them simply the result. For example:


(4x³ - 23x² + x + 1)   *   (2x² - x - 3) 

= ((4x³ - 23x² + x + 1) * 2x²) + 
  ((4x³ - 23x² + x + 1) * -x) +
  ((4x³ - 23x² + x + 1) * -3)

= (8x⁵ - 46x⁴ + 2x³ + 2x²) + 
  (-4x⁴ + 23x³ - x² - x)  +
  (-12x³ + 69x² - 3x - 3) 

= (8x⁵ - 50x⁴ + 13x³ + 70x² - 4x - 3)

Now I'll start creating polymulti from the bottom up to the definition.

1. Defining polynomials

As mentioned above J is a nice language for manipulating arrays. We're going to use J arrays to define polynomials. In fact J supports some operations on arrays as polynomials which will be described bellow.

In order to write a new array literal containing 1,1, -23 and 4 in J we write (here using jconsole):


   1 1 _23 4
1 1 _23 4

As you can see the elements of the array are separated by space. Also negative numbers are prefixed by underscore '_'. This array is going to be used to represent the coefficients of the "(4x³ - 23x² + x + 1)" polynomial.

We're going to define two variables with the polynomials shown above to use them for examples:


   p1 =: 1 1 _23 4
   p2 =: _3 _1 2

Here the '=:' operator is used to bind the specified arrays with p1 and p2.

2. Multiply each element of one of the polynomials

We can proceed to apply the first step in the process which is multiply each element of the second operand by the first operand.

In J we can operate on arrays easily, for example if we want to multiply each element of the above array by a 2 we write:


   1 1 _23 4 * 2
2 2 _46 8

In J the an operation that receives two arguments is called a dyad and a operation that receives only one is called monad. Here we're using the '*' dyad to perform the multiplication.

We cannot directly go and type "p1 * p2" because we will get the following error:


   p1 * p2
|length error
|   p1    *p2

This happens because the length of the two arrays (4 and 3) could not be used to perform the operation. We can apply '*' to a two same size arrays and J will multiply each element. For example:


   p1 * p1
1 1 529 16

Now what I want is to multiply each element of 'p2' by 'p1'. In order to do this we can change the behavior of the '*' dyad by specifying its rank (more details on how to do this can be found in "Verb Execution -- How Rank Is Used (Dyads)") . To change the rank of '*' and say that we want to multiply the complete array on the left for each of the cells of the right array we write (*"_ 0) .For example:


   p1 (*"_ 0) p2
_3 _3  69 _12
_1 _1  23  _4
 2  2 _46   8

By specifying (*"_ 0) we say that the left operand has "infinite rank" (_) which means it will consider the array as single unit.We also say that for the right argument we will consider every cell (by using the 0 rank). Notice that in order to modify the rank we use double quote (") which, in J, doesn't have to paired with another double quote as in most programming languages.

Also you can notice that the result of this operation is an array of arrays, one for each element of the right argument.

2. Sum each element of one of the polynomials

Now that we have an array of polynomials with the result of multiplying the coefficients, we need to change the degree of each of the polynomials. First we need to make more space to increment the degree of the polynomials. We're going to use the ',' dyad which lets you concatenate two arrays. For example:


   p1 , 0 0
1 1 _23 4 0 0

We need to append an array that is the size of the second polynomial minus one. In order to create a new array of this size we use '$' which lets you create an array or matrix by specifying the size and the value of the elements of the new array. For example:


   10 $ 0
0 0 0 0 0 0 0 0 0 0

The size of the array can be obtained with the '#' monad. For example:


   # p1
4

Now we can combine all these elements to resize each array resulting from the multiplication of the coefficients like this:


   (p1 (*"_ 0) p2)
_3 _3  69 _12
_1 _1  23  _4
 2  2 _46   8
    (((#p2) - 1 ) $ 0)
0 0
   (p1 (*"_ 0) p2) (,"1 1) ( ((#p2) - 1 ) $ 0)
_3 _3  69 _12 0 0
_1 _1  23  _4 0 0
 2  2 _46   8 0 0

Here we use (((#p2) - 1 ) $ 0) to generate an array of zeros of the desired size. Then we use (,"1 1) which is the ',' dyad with a modified rank saying that each element of the left matrix (an array) will be concatenated with the second array (since the second argument is a single dimension array we could have said (,"1 _) ).

With the arrays resized we can change the degree of our polynomial array. In order to do this we're going to use the '|.' dyad which let's you rotate an array an specified amount of positions. For example:

 
   1 |.  1 2 3
2 3 1
   _1 |.  1 2 3
3 1 2

As shown in the example, by specifying a positive number the array will be rotated to the left and a negative number to the right.

Now the question is, how to rotate each element of the polynomial array by a different element count? Before showing that we're going to introduce the 'i.' primitive which lets you create an array of with a sequence of numbers for example, to create an array of 10 numbers (starting with 0) we write:


   i. 10
0 1 2 3 4 5 6 7 8 9

We can use this primitive in conjunction with the rotate primitive to say


   m =:    (p1 (*"_ 0) p2) (,"1 1) ( ((#p2) - 1 ) $ 0)
   i. #p2
0 1 2
   _1 * i. #p2
0 _1 _2
   (_1 * i. #p2) (|."0 1) m
_3 _3 69 _12   0 0
 0 _1 _1  23  _4 0
 0  0  2   2 _46 8

As you can see we use the array generated by (_1 * i. #p2) to specify how many positions are we moving. We apply the change the rank of '|.' by saying (|."0 1) which means apply '|.' for each single cell of the left array to each row of the right array.

The previous step generated an array polynomials to be summed. So the only thing left is to generate the final polynomial. In order to do this we use the '+/' dyad which let's use sum the contents of an array. For example:


   +/ 5 3 4 1
13

We can apply +/ to any array and J will do the operation as expected.


   +/ (_1 * i. #p2) (|."0 1) m
_3 _4 70 13 _50 8

As a work around I'm appending a zero row to this matrix, just in case the p2 array is a 0 degree polynomial.


   +/ ((_1 * i. #p2) (|."0 1) m) , 0
_3 _4 70 13 _50 8

And that's it we have the complete process for multiplying the array.

3. Create the function

Now putting all the elements described above we can put together the polynomial multiplication dyad:

polymulti =: dyad : '+/ (((_1 * i. #y) (|. "0 1) ((x (*"_ 0) y) (,"1 1) (((#y) - 1) $ 0))) , 0)'

The 'x' and 'y' names are the implicit names of the left and right operands.

We can use it with any pair of polynomials. For example:


   2 34 3 polymulti 4 3
8 142 114 9
   2 34 3 polymulti 4 0 3
8 136 18 102 9
   2 34 3 polymulti 1
2 34 3

4. Use the polynomials

J already has support for polynomials inside the library. For example by using the 'p.' we can get an evaluation of a given polynomial.


   3 4 5 p. 34
5919
   3 + (4*34) + (5*34*34)
5919
   _2 p. 34
_2
   (5 3 _2 polymulti 3 0 0 _2)
15 9 _6 _10 _6 4
   (5 3 _2 polymulti 3 0 0 _2) p. 3
204

Final words

It was very interesting to read about J. It offers a different perspective on programming which is definitely worth studying. I have to admit it was difficult at the beginning because of the number of concepts to learn ( only a couple are presented in this post) and the syntax . For future posts I'm going to try to explore more J features.

AS3 Getter/Setter support in AbcExplorationLib

2009-10-06T05:20:00.004-06:00

Recently I added initial support for reading and writing AS3/Avm2 getters and setters to AbcExplorationLib.

Given the following ActionScript class:


class Complex {
   public var radius:Number;
   public var angle:Number;
   public function Complex(ar:Number,aa:Number):void {
      radius = ar;
      angle = aa;
   }

   public function set imaginary(newImaginary:Number):void 
   {
      var oldReal = this.real;
      angle = Math.atan(newImaginary/oldReal);
      radius = Math.sqrt(oldReal*oldReal + newImaginary*newImaginary);
   }
   public function get imaginary():Number
   {
          return radius*Math.sin(angle);
   }

   public function set real(newReal:Number):void 
   {
      var oldImaginary = this.real;
      angle = Math.atan(oldImaginary/newReal);
      radius = Math.sqrt(newReal*newReal + oldImaginary*oldImaginary);
   }
   public function get real():Number
   {
          return radius*Math.cos(angle);
   }
}

We compile it using the Flex SDK:

java -jar c:\flexsdk\lib\asc.jar complexclasstest.as

Then we can load it using the library:


Microsoft F# Interactive, (c) Microsoft Corporation, All Rights Reserved
F# Version 1.9.6.16, compiling for .NET Framework Version v2.0.50727

Please send bug reports to fsbugs@microsoft.com
For help type #help;;

> #r "abcexplorationlib.dll";;

--> Referenced 'abcexplorationlib.dll'

> open System.IO;;
> open Langexplr.Abc;;
> let f = using (new FileStream("complexclasstest.abc",FileMode.Open)) (fun s -> AvmAbcFile.Create(s));;

val f : AvmAbcFile

> let complexClass = List.hd f.Classes;;

val complexClass : AvmClass


> complexClass.Properties |> List.map (fun p -> p.Name);;
val it : QualifiedName list =
  [CQualifiedName (Ns ("",PackageNamespace),"real");
   CQualifiedName (Ns ("",PackageNamespace),"imaginary")]
> let realGetter =  complexClass.Properties |> List.map (fun p -> p.Getter.Value) |> List.hd;;

val realGetter : AvmMemberMethod:
> realGetter.Method.Body.Value.Instructions |> Array.map (fun x -> x.Name);;
val it : string array =
  [|"getlocal_0"; "pushscope"; "getlocal_0"; "getproperty";
    "findpropertystrict"; "getproperty"; "getlocal_0"; "getproperty";
    "callprop"; "multiply"; "returnvalue"|]

The library can be found here.

Parsing Javascript using Newspeak parsing combinators

2009-10-01T05:16:00.007-06:00

I've been working on a parser for Javascript/Ecmascript using Newspeak parsing combinators. The parser is based on the grammar presented in the ECMAScript Language Specification [PDF] document. It is still incomplete, however it can parse simple statements.

The grammar looks like this:


class JSGrammar = ExecutableGrammar (
"Experiment for JS grammar based on the description from http://www.ecma-international.org/publications/standards/Ecma-262.htm"
|
     doubleQuote = (char: $").
     backslash = (char: $\).
     str = doubleQuote,((backslash, ( char: $" ))   | 
                        (backslash, ( char: $/ ))   |
                        (backslash, backslash)      |
                        (backslash, ( char: $r ))   |
                        (backslash, ( char: $n ))   |
                        (backslash, ( char: $t ))   |
                        (charExceptFor: $")) star, doubleQuote.
     string = tokenFor: str.
   
     tilde = char: $~.
     exclamation = char: $!.
     starChar = char: $*.
     slash = char: $/.
     modulo = char: $%.
     pipe = char: $|.
     amp = char: $&.
     cir = char: $^.
     question = char: $?.
     colon = char: $:.
     semicolon = char: $;.
   
     negSign = (char: $-).
     plusSign = (char: $+).
     digit = (charBetween: $0 and: $9).
     dot = (char: $. ) .
     lt = char: $&lt;.
     gt = char: $&gt;.
     eq = char: $=.
     num = negSign opt, digit, digit star, dot opt,digit star, ((char: $e) | (char: $E)) opt, (plusSign | negSign) opt,digit star.
     number = tokenFor: num.

     tQuestion = tokenFor: question.
     tColon = tokenFor: colon.
     tplusSign = tokenFor: plusSign.
     tnegSign = tokenFor: negSign.
     tmodulo = tokenFor: modulo.
     tslash = tokenFor: slash.
     tstarChar = tokenFor: starChar.
     texclamation = tokenFor:exclamation.
     tdot = tokenFor:dot.
     tLt = tokenFor: lt.
     tGt = tokenFor: gt.
     tEq = tokenFor: eq.   
     tAmp = tokenFor: amp.
     tPipe = tokenFor: pipe.
     tCir = tokenFor: cir.
     tSlash = tokenFor: slash.
     tSemicolon = tokenFor: semicolon.

     tStarEq = tstarChar,eq.
     tModEq = tmodulo,eq.
     tSlashEq = tSlash,eq.
     tPlusEq = tplusSign,eq.
     tMinusEq = tnegSign,eq.
     tAmpAmp = tAmp,amp.
     tPipePipe = tPipe,pipe.
     tLtEq = tLt,eq.
     tGtEq = tGt,eq.
     tleftShift = tLt,lt.
     trightShift = tGt,gt.
     tsRightShift = tGt,gt,gt. 
     tEqEq = tEq,eq.
     tEqEqEq = tEq,eq,eq.
     tNotEq = texclamation,eq.
     tNotEqEq = texclamation,eq,eq.
     tleftShiftEq = tleftShift,eq.
     trightShiftEq = trightShift,eq.
     tsRightShiftEq = tsRightShift,eq.
     tAmpEq = tAmp,eq.
     tPipeEq = tPipe,eq.
     tCirEq = tCir,eq.

     lineTerminator = (char: (Character lf)) | (char: (Character cr)).

     regularExpressionLiteral = 
                     tslash, 
                     (  ((backslash, ( charExceptForCharIn: { (Character lf).  (Character cr). })) |
                      (charExceptForCharIn: { (Character lf).  (Character cr). $/.})) plus),
                      slash, (identifierStart star).

     leftparen = tokenFromChar: $(.
     rightparen =tokenFromChar: $).   
   
     leftbrace = tokenFromChar: ${.
     rightbrace =tokenFromChar: $}.
     comma = tokenFromChar: $,.
     propertyName = string | identifier | number.
     propertyNameAndValue = propertyName,tColon,value.
     obj =  leftbrace,  (propertyNameAndValue starSeparatedBy: comma),rightbrace.
     object =  obj.

     leftbracket = tokenFromChar: $[.
     rightbracket = tokenFromChar: $].
     arr = leftbracket, (value starSeparatedBy: comma), rightbracket.
     array = tokenFor: arr.

     comment = (slash,starChar,blockCommentBody,starChar,slash) | (slash,slash, lineCommentBody).


     ttrue = tokenFromSymbol: #true.
     tfalse = tokenFromSymbol: #false.
     null = tokenFromSymbol: #null.
     function = tokenFromSymbol: #function.
     tnew = tokenFromSymbol: #new.
     break = tokenFromSymbol: #break.
     case = tokenFromSymbol: #case.
     catch = tokenFromSymbol: #catch.
     continue = tokenFromSymbol: #continue.
     default = tokenFromSymbol: #default.
     delete = tokenFromSymbol: #delete.
     do = tokenFromSymbol: #do.
     else = tokenFromSymbol: #else.
     finally = tokenFromSymbol: #finally.
     for = tokenFromSymbol: #for.
     if = tokenFromSymbol: #if.
     in = tokenFromSymbol: #in.
     instanceof = tokenFromSymbol: #instanceof.
     return = tokenFromSymbol: #return.
     switch = tokenFromSymbol: #switch.
     this = tokenFromSymbol: #this.
     throw = tokenFromSymbol: #throw.
     try = tokenFromSymbol: #try.
     typeof  = tokenFromSymbol: #typeof.
     var = tokenFromSymbol: #var.
     void = tokenFromSymbol: #void.
     while = tokenFromSymbol: #while.
     with = tokenFromSymbol: #with.

      
 
     letter = (charBetween: $a and: $z) | (charBetween: $A and: $Z).
     identifierStart =  letter | (char: $$) | (char: $_).
     identifier = accept: (tokenFor: (identifierStart), (identifierStart | digit) star) ifNotIn: keywords .

     value =  assignmentExpression .      
  
     literal  = null | ttrue | tfalse | number | string |  regularExpressionLiteral.

     primaryexpression  = this |  literal | identifier | array | object | parenthesized.

     parenthesized = leftparen,expression,rightparen.

     functionexpression = function , identifier opt, 
                                       leftparen,formalParameterList , rightparen ,
                                       leftbrace,sourceElements,rightbrace.
     formalParameterList = identifier starSeparatedBy: comma. 

     memberexpression =   (simplememberexpression  ),
                                         (( leftbracket, expression, rightbracket) | ( tdot, identifier)) star.
     
     simplememberexpression = primaryexpression | 
                                               functionexpression  |                        
                                                simpleNewExpression.
     simpleNewExpression = tnew,memberexpression, arguments.

      
  
     callExpression =  (simplememberexpression  ),
                       ( arguments |
                         ( leftbracket, expression, rightbracket) | 
                         ( tdot, identifier)  ) star.
     simpleCallExpression = memberexpression , arguments.
     arguments = leftparen ,
                 (assignmentExpression, (comma, assignmentExpression) star) opt, 
                 rightparen.

     newExpression = memberexpression | simpleNewMemberExpression.
     simpleNewMemberExpression = tnew, memberexpression.
 
     plusPlus = plusSign,plusSign.
     minusMinus = negSign,negSign.

     leftHandSideExpression = callExpression | newExpression  .
     postfixExpression = leftHandSideExpression , 
                          ((plusPlus | minusMinus) star).


     unaryExpression = postfixExpression | complexUnaryExpression.

     complexUnaryExpression = 
           (typeof, unaryExpression) |
           (delete, unaryExpression) | 
           (void, unaryExpression) | 
           (plusPlus, unaryExpression) |
           (minusMinus, unaryExpression) |
           ((tokenFor: ( plusSign | negSign | tilde | exclamation )), unaryExpression).

     multiplicativeExpression = 
              unaryExpression, ((tstarChar | tslash | tmodulo), unaryExpression) star.     

     additiveExpression = 
              multiplicativeExpression, ((tplusSign | tnegSign), multiplicativeExpression) star. 

     shiftExpression =  
             additiveExpression,  ((tsRightShift | tleftShift |  trightShift), additiveExpression) star.
      
     relationalExpression =
            shiftExpression, ((  tLtEq | tGtEq | tLt | tGt | instanceof | in) , shiftExpression)  star.

     relationalExpressionNoIn =
            shiftExpression, ((  tLtEq | tGtEq | tLt | tGt | instanceof ) , shiftExpression)  star. 

     equalityExpression =
            relationalExpression, ((tEqEqEq | tEqEq | tNotEqEq | tNotEq ), relationalExpression) star.

     equalityExpressionNoIn =
            relationalExpressionNoIn, ((tEqEqEq | tEqEq | tNotEqEq | tNotEq ), relationalExpressionNoIn) star.

     bitwiseANDExpression =
           equalityExpression,(tAmp, equalityExpression) star.

     bitwiseANDExpressionNoIn =
           equalityExpressionNoIn,(tAmp, equalityExpressionNoIn) star.

     bitwiseXORExpression =
           bitwiseANDExpression,(tCir, bitwiseANDExpression) star.

     bitwiseXORExpressionNoIn =
           bitwiseANDExpressionNoIn,(tCir, bitwiseANDExpressionNoIn) star.

     bitwiseORExpression =
           bitwiseXORExpression,(tPipe, bitwiseXORExpression) star.

     bitwiseORExpressionNoIn =
           bitwiseXORExpressionNoIn,(tPipe, bitwiseXORExpressionNoIn) star.

     logicalAndExpression = 
           bitwiseORExpression, (tAmpAmp,bitwiseORExpression) star.

     logicalAndExpressionNoIn = 
           bitwiseORExpressionNoIn, (tAmpAmp,bitwiseORExpressionNoIn) star.

     logicalOrExpression = 
           logicalAndExpression, (tPipePipe,logicalAndExpression) star.

     logicalOrExpressionNoIn = 
           logicalAndExpressionNoIn, (tPipePipe,logicalAndExpressionNoIn) star.

     assignmentOperator = 
              tEq | tStarEq | tSlashEq | tModEq | tPlusEq | tMinusEq |
              tleftShiftEq | tsRightShiftEq | trightShiftEq | tAmpEq | tPipeEq |
              tCirEq.
 
     conditionalExpression =
            logicalOrExpression, (tQuestion, assignmentExpression,tColon,assignmentExpression) opt.

     assignmentExpression =    
           conditionalExpression, (assignmentOperator,conditionalExpression) star.

     conditionalExpressionNoIn =
            logicalOrExpressionNoIn, (tQuestion, assignmentExpressionNoIn,tColon,assignmentExpressionNoIn) opt.

     assignmentExpressionNoIn =    
           conditionalExpressionNoIn, (assignmentOperator,conditionalExpressionNoIn) star.

     expression = assignmentExpression, (comma , assignmentExpression) star.

     expressionNoIn = assignmentExpressionNoIn, (comma , assignmentExpressionNoIn) star.


     statement = block | variableStatement | emptyStatement | expressionStatement |
                          ifStatement | iterationStatement | withStatement | switchStatement |
                          labelledStatement | tryStatement | throwStatement | 
                          breakStatement | returnStatement.
  
     block = leftbrace, statementList , rightbrace.

     statementList = statement star.
     variableStatement = var, variableDeclarationList, tSemicolon.

     variableDeclarationList = (variableDeclaration plusSeparatedBy: comma). 
     variableDeclaration = identifier, (tEq,assignmentExpression) opt.

     variableDeclarationListNoIn = (variableDeclarationNoIn plusSeparatedBy: comma). 
     variableDeclarationNoIn = identifier, (tEq,assignmentExpressionNoIn) opt.
       
     emptyStatement = tSemicolon.

     expressionStatement = (((function | leftbrace) not) & expression), tSemicolon.

     ifStatement = if, leftparen,expression,rightparen,statement,(else,statement) opt.

     iterationStatement = doStatement | forStatement | forStatementNoVar |  
                                        whileStatement | forInStatement | forInStatementNoVar.

     doStatement = do, statement, while, leftparen,expression,rightparen,tSemicolon.

     forStatement = for, leftparen, 
                         var, variableDeclarationListNoIn ,
                         tSemicolon,
                         (expression opt),
                         tSemicolon,
                         (expression opt),
                         rightparen, statement.

      forStatementNoVar = for, leftparen, 
                              (expression opt),
                              tSemicolon,
                              (expression opt),
                              tSemicolon,
                              (expression opt),
                              rightparen, statement.


      forInStatement = for, leftparen, 
                                       var, variableDeclarationListNoIn ,
                                       in,
                                       expression ,
                                       rightparen, statement.

      forInStatementNoVar = for, leftparen, 
                                       leftHandSideExpression ,
                                       in,
                                       expression ,
                                       rightparen, statement.
      whileStatement = while,leftparen,expression,rightparen,statement.

      continueStatement = continue, (identifier opt), tSemicolon.

      breakStatement = break, (identifier opt), tSemicolon.

      returnStatement = return, (expression opt), tSemicolon.

      withStatement = with, leftparen, expression ,rightparen, statement.
  
      switchStatement = switch ,leftparen,expression,rightparen, clauseBlock.

      clauseBlock = leftbrace,(clause star),(defaultClause opt),rightbrace.

      clause = case, expression, tColon,statementList.
         
      defaultClause = default,tColon, statementList.

      labelledStatement = identifier,tColon,statement.
        
      throwStatement = throw,expression,tSemicolon.

      tryStatement = try, block, (catchBlock opt), (finallyBlock opt).

      catchBlock = catch, leftparen, identifier,rightparen, block.

      finallyBlock = finally, block.

      functionDeclaration = function,identifier,
                                     leftparen,formalParameterList,rightparen,
                                     leftbrace,sourceElements,rightbrace.

 

      sourceElements = (statement | functionDeclaration ) star.

      program = sourceElements.
|
)

...

I really like the way you can separate the grammar from the AST creation (as described in the Executable Grammars[PDF] paper). As you can see there's no code specified for this purpose. Along with the source code there's is a 'testing AST' and a parser that inherits from this grammar which is used by the unit tests .

Almost all the grammar was written using the parser combinators from the library. Only charExceptFor: and accept: ifNotIn: were created.

There are still a lot work to do with this parser:

Clean up the code
Work on performance issues
Find a solution for the "Automatic Semicolon Insertion" feature (see section 7.9) of the Ecma document)
Get rid of some repetition (for example the 'NoVar' productions which are also present in the document)
Better AST creation
See if unicode support is possible
More tests!

In order to see the result of using this parser I created a little program to display the 'testing AST'. For example:

The parser with tests and the other code mentioned can be found here .

Creating Dynamic JSON array finders using the DLR

2009-08-03T05:50:00.001-06:00

One of the things that really impressed me while reading about Ruby on Rails was the use of method_missing to implement dynamic finders. The technique is described on the "How dynamic filters work". In this post I'm going to show a little experiment of creating a similar technique for querying JSON arrays using the .NET's Dynamic Language Runtime infrastructure.

Code for this post was created using Visual Studio 2010 Beta 1, IronRuby for .NET 4 beta 1 and IronPython 2.6 beta 4 for .NET 4.

JSON.NET

For this post I'm using the JSON.NET library for loading the JSON data. This library provides several ways of loading JSON data. Here I'll be using a set of predefined classes: JObject for JSON objects, JArray for arrays, JValue for literal values, etc. All these classes inherit from JToken.
Code in this post use the JSON data returned by the Twitter REST API. An example of this data:


[
 {"in_reply_to_screen_name":null,
  "text":"...",
  "user": { "following":null,
            "description":"...",
            "screen_name":"...",
            "utc_offset":0,
            "followers_count":10,
            "time_zone":"...",
            "statuses_count":155,
            "created_at":"...",
            "friends_count":1,
            "url":"...",
            "name":"...",
            "notifications":null,
            "protected":false,
            "verified":false,
            "favourites_count":0,
            "location":"...",
            "id": ...,
            ...
  },
  "truncated":false,
  "created_at":"...",
  "in_reply_to_status_id":null,
  "in_reply_to_user_id":null,
  "favorited":false,
  "id":...,
  "source":"...."
 },
  ...
]

Dynamic queries on JSON data

Finders will be implemented for JSON arrays. As with Rail's dynamic finders the names of the required fields will be encoded in the name of the invoked method.

The following C# 4.0 code shows an example of this wrapper class in conjunction with the dynamic keyword.


JsonTextReader reader = new JsonTextReader(rdr);
JsonSerializer serializer = new JsonSerializer();
JArray o = (JArray)serializer.Deserialize(reader);
dynamic dArray = new FSDynArrayWrapper(o);

string name = "ldfallas";
foreach (var aJObject in dArray.FindAllByFavoritedAlsoByUserWithScreen_Name("false",name))
{
   dynamic tobj = new FSDynJObjectWrapper(aJObject);
   Console.WriteLine("========");
   Console.WriteLine(tobj.user.screen_name);
   Console.Write("\t'{0}'",tobj.text);
}

A small definition of the syntax used for names is the following.


method-name = "FindAllBy" , 
              property-name , ("With", property-name)? ,
             ("AlsoBy" property-name , ("With", property-name)? ) *
property-name = valid json property name

In order to be more flexible the following syntax will also be allowed:


method-name = "find_all_by_" , 
              property-name , ("_with_", property-name)? ,
             ("_also_by" property-name , ("_with_", property-name)? ) *
property-name = valid json property name

A sample name for this query methods look like this:


array.FindAllByFavoritedAlsoByUserWithScreen_Name("true","ldfallas")

This method will accept two parameters and is going to :

Find all the object elements from the array that has a 'Favorited' property equal to 'true' and also has an object with a 'User' property associated with an object which has a 'Screen_Name' property which is equal to the 'ldfallas'

Interoperability

One of the nice things of using the DLR infrastructure to create this feature, is that it can be used by other DLR languages. The following example is an IronRuby snippet:


require 'FsDlrJsonExperiments.dll'
include Langexplr::Experiments

while true do
  print "Another try\n"
  str = System::Net::WebClient.new().download_string("http://twitter.com/statuses/public_timeline.json")
  json = FSDynArrayWrapper.CreateFromReader(System::IO::StringReader.new(str))

  for i in json.find_all_by_user_with_time_zone('Central America') do
     print i.to_string()
  end
  sleep(5)
end

The following IronPython code shows a little example of this wrapper class.


fReader = StreamReader(GetTwitterPublicTimeline())
jReader  = JsonTextReader(fReader)
serializer = JsonSerializer()

json = FSDynArrayWrapper( serializer.Deserialize(jReader) )

for i in json.FindAllByFavoritedAlsoByUserWithScreen_Name("false","ldfallas"):
   print i

Implementation

In order to implement the functionality presented here, the IDynamicMetaObjectProvider interface and the DynamicMetaObject class were used. By using these we can generate the code for the call site as a expression tree. For more information on how to use this interface see
Getting Started with the DLR as a Library Author document (available here) .

The code generated to do the filtering is an expression which uses the Where method from System.Linq.Enumerable
. The generated expression written in source using a pseudo C# looks like this:


{
  object tmp;
  array.Where(c => (((c Is JObject) && 
                     CompareHelper(
                         GetJObjectPropertyCI(((JObject)c), "Favorited"), 
                         "true")) 
                    && 
                    ((((tmp = GetJObjectPropertyCI(((JObject)c), "User")) As JObject) != null) && 
                     CompareHelper(
                         GetJObjectPropertyCI(((JObject)tmp), "Screen_Name"), 
                         "ldfallas"))))

}

Where GetJObjectPropertyCI is a helper method that gets a property from a JObject by case-intensive name . And CompareHelper is a helper method to do the comparison.

The implementation for FSDynArrayWrapper was written in F#. Mainly because it's a nice language to implement this kind of features. However there's no easy way to consume this feature using F# since it doesn't use the DLR.

Here's the definition:


type FSDynArrayWrapper(a:JArray) =   
   member this.array with get() = a
   static member CreateFromReader(stream : System.IO.TextReader) =
     ...
   static member CreateFromFile(fileName:string) =
     ... 
   interface IDynamicMetaObjectProvider with
      member this.GetMetaObject( parameter : Expression) : DynamicMetaObject =
         FSDynArrayWrapperMetaObject(parameter,this)  :> DynamicMetaObject

As you can see, the interesting part is in the implementation of FSDynArrayWrapperMetaObject. The CreateFromReader and CreateFromFile methods are only utility methods to load data from a document.

The implementation of FSDynArrayWrapperMetaObject looks like this:


type FSDynArrayWrapperMetaObject(expression : Expression, value: System.Object) = 
   inherit DynamicMetaObject(expression,BindingRestrictions.Empty,value) 
   
   ...
   
   override this.BindInvokeMember(binder : InvokeMemberBinder, args: DynamicMetaObject array) =
     match QueryInfo.GetQueryElements(binder.Name) with
     | Some( elements ) ->
         (new DynamicMetaObject(
             this.GenerateCodeForBinder(
                 elements, 
                 Array.map 
                   (fun (v:DynamicMetaObject) -> 
                      Expression.Constant(v.Value.ToString()) :> Expression) args),
             binder.FallbackInvokeMember(this,args).Restrictions))
     | None -> base.BindInvokeMember(binder,args)

The BindInvokeMember creates the expression tree for the code that will be executed for a given invocation of a dynamic finder method. Here the QueryInfo.GetQueryElements method is called to extract the elements of the name as described above. The value returned by this method is QueryElement list option where:


type QueryElement =
  | ElementQuery of string
  | SubElementQuery of string * string

ElementQuery specifies the "Favorited" part in FindAllByFavoritedAlsoByUserWithScreen and the SubElementQuery belongs to the "ByUserWithScreen" part in FindAllByFavoritedAlsoByUserWithScreen .

If the name of the invoked method corresponds is a supported name for a finder, the GenerateCodeForBinder is called to generate the expression tree. The last argument of this method is a collection of the arguments provided for this invocation.


member this.GenerateCodeForBinder(elements, arguments : Expression array) =
   let whereParameter = Expression.Parameter(typeof<JToken>, "c") in
   let tmpVar = Expression.Parameter(typeof<JToken>, "tmp") in
   let whereMethodInfo =
         (typeof<System.Linq.Enumerable>).GetMethods() 
                 |> Seq.filter (fun (m:MethodInfo) -> m.Name = "Where" && (m.GetParameters().Length = 2))
                 |> Seq.map (fun (m:MethodInfo) -> m.MakeGenericMethod(typeof<JToken>))
                 |> Seq.hd
   let queryElementsConditions =              
        elements 
           |> Seq.zip arguments
           |> Seq.map 
                  (fun (argument,queryParameter) ->
                       this.GetPropertyExpressionForQueryArgument(queryParameter,argument,whereParameter,tmpVar)) in
   let initialCondition =  Expression.TypeIs(whereParameter,typeof<JObject>)  in
   
   let resultingExpression =
        Expression.Block(
           [tmpVar],
           Expression.Call(
              whereMethodInfo,
              Expression.Property(
                 Expression.Convert(
                    this.Expression,this.LimitType),"array"),
              Expression.Lambda(                     
                  Seq.fold 
                       (fun s c -> Expression.And(s,c) :> Expression) 
                       (initialCondition :> Expression) 
                       queryElementsConditions,
                   whereParameter))) in
   resultingExpression

The most important parts of this method is the definition of queryElementsConditions and resultingExpression. The resulting expression specifies the invocation to the Where method

The queryElementsConditions take each argument extracted from the name of the method and tries to generate the necessary conditions for the value provided as an argument. In order to do this the GetPropertyExpressionForQueryArgument method is used:


member this.GetPropertyExpressionForQueryArgument(parameter:QueryElement,argument,cParam,tmpVar) : Expression =
    match parameter with
    | ElementQuery(propertyName) -> 
        this.CompareExpression(     
           this.GetJObjectPropertyExpression(cParam,propertyName) ,
           argument) 
    | SubElementQuery(propertyName,subPropertyName) ->
        Expression.And(
          Expression.NotEqual(
              Expression.TypeAs(
                 Expression.Assign(
                     tmpVar,
                     this.GetJObjectPropertyExpression(cParam,propertyName)),
                 typeof<JObject>),  
              Expression.Constant(null)),          
          this.CompareExpression(          
              this.GetJObjectPropertyExpression(
                  tmpVar,
                  subPropertyName),argument)) :> Expression

This method generates a different expression depending on the kind of query element that is requested.

Considerations for IronPython

As a curious note, in IronPython the BindInvokeMember method is not called in the FSDynArrayWrapperMetaObject when a method is invoked. It seems that IronPython calls the BindGetMember method and then tries to apply the result of getting the method.

So to make this object work with IronPython a implementation of the BindGetMember method was created that returns a lambda expression tree with the generated Where invocation.


override this.BindGetMember(binder: GetMemberBinder) =
  match QueryInfo.GetQueryElements(binder.Name) with
  | Some( elements ) ->
      let parameters =
        List.mapi ( fun i _ ->
                         Expression.Parameter(
                                     typeof<string>,
                                     sprintf "p%d" i)) elements
      (new DynamicMetaObject(
          Expression.Lambda(
              this.GenerateCodeForBinder(
                  elements, 
                  parameters 
                  |> List.map (fun p -> p :> Expression)
                  |> List.to_array ),
              parameters),
          binder.FallbackGetMember(this).Restrictions))
  | None -> base.BindGetMember(binder)

Accessing using different names

The QueryInfo.GetQueryElements method is used to allow the "FindAllBy..." and "find_all_by..." method names .


module QueryInfo = begin

    ...

    let GetQueryElements(methodName:string) =
       match methodName with
       | str when str.StartsWith("FindAllBy") ->
              Some(ExtractQueryElements(str.Substring("FindAllBy".Length),"AlsoBy","With")) 
       | str when str.StartsWith("find_all_by_") ->
           Some(ExtractQueryElements(str.Substring("find_all_by_".Length),"_also_by_","_with_") )
       | _ -> None
end

Code

Code for this post can be found here.

Using the DynamicObject class

2009-07-26T19:49:00.003-06:00

In this post I'm going to show a little example of using the DynamicObject class from the .NET's Dynamic Language Runtime. This class is an easy way to provide dynamic dispatch to your objects in a DLR language.

Code samples presented here were created using Visual Studio 2010 Beta 1 so there may be differences with the final version.

The experiment

The example is a class that allows access to the properties of JSON objects using the property syntax of a DLR language. In order to do this, a wrapper to JSON.NET's JObject was created. This wrapper is called FSDynJObjectWrapper.

The following code shows an example of using FSDynJObjectWrapper with C# 4.0 :


// Load the document
JsonTextReader reader = new JsonTextReader(GetTwitterPublicTimeLine());
JsonSerializer serializer = new JsonSerializer();
JArray topArray = (JArray)serializer.Deserialize(reader);

// Access the document
dynamic aObj = new FSDynJObjectWrapper((JObject)topArray[0]);
Console.WriteLine("========");
Console.WriteLine(aObj.user.screen_name);
Console.Write("\t'{0}'", aObj.text);

Given that the JSON returned by the Twitter REST api looks like this:


 {"in_reply_to_screen_name":null,
  "text":"...",
  "user": { "following":null,
            "description":"...",
            "screen_name":"...",
            "utc_offset":0,
            "followers_count":10,
            "time_zone":"...",
            "statuses_count":155,
            "created_at":"...",
            "friends_count":1,
            "url":"...",
            "name":"...",
            "notifications":null,
            "protected":false,
            "verified":false,
            "favourites_count":0,
            "location":"...",
            "id": ...,
            ...
  },
  "truncated":false,
  "created_at":"...",
  "in_reply_to_status_id":null,
  "in_reply_to_user_id":null,
  "favorited":false,
  "id":...,
  "source":"...."
 }

DLR and DynamicObject

The Dynamic Language Runtime provides a common infrastructure to build dynamic languages on the CLR. The System.Dynamic.DynamicObject class is an easy way to override the behavior of access to an object from a dynamic language. A partial definition of this class looks like this:


public abstract class DynamicObject : IDynamicMetaObjectProvider {
    public virtual bool TryGetMember(GetMemberBinder binder,
                                     out object result)
    public virtual bool TrySetMember(SetMemberBinder binder,
                                     object value)
    public virtual bool TryDeleteMember(DeleteMemberBinder binder)

    public virtual bool TryConvert(ConvertBinder binder,
                                   out object result)        
    public virtual bool TryUnaryOperation
        (UnaryOperationBinder binder, out object result)
    public virtual bool TryBinaryOperation
        (BinaryOperationBinder binder, object arg,
         out object result)
    public virtual bool TryInvoke
        (InvokeBinder binder, object[] args, out object result)
    public virtual bool TryInvokeMember
        (InvokeMemberBinder binder, object[] args,
         out object result)
...
}

As you can see this class has methods to override things like what happens when a property is access or a method is invoked. A detailed description of this class is available from the Getting Started with the DLR as a Library Author document .

The `FSDynJObjectWrapper` class

As presented above the FSDynJObjectWrapper inherits from DynamicObject and provides the required functionality. For this example the class is implemented using F# (although any .NET language could be used) and looks like this:


open System.Dynamic
open System.Reflection
open System.Linq.Expressions
open Newtonsoft.Json.Linq
open Newtonsoft.Json
 
 
type FSDynJObjectWrapper(theObject:JObject) =
   inherit DynamicObject() with
   override this.TryGetMember(binder : GetMemberBinder, result : obj  byref ) =
      match theObject.[binder.Name] with
      | null -> false
      | :? JObject as aJObject -> 
               result <- FSDynJObjectWrapper(aJObject)
               true
      | theValue -> 
               result <- theValue
               true

The TryGetMember will be called when accessing a property. It tries to lookup the name of the requested property in the JObject instance. A new instance of the wrapper is created if the property value is another JObject instance allowing expressions like: aObj.user.screen_name .

For the next posts I'm going to show the use of other DLR classes and examples using other DLR languages.

Code for this post can be found here.

Creating a calendar using Newspeak and Hopscotch

2009-06-25T21:53:00.002-06:00

For this post I'm going to show the code for a little calendar UI fragment created using the Newspeak programming language and the Hopscotch framework.

Calendar

Here's how the calendar looks:

(As you can see, I'm focusing on the functionality for the moment).

The code

As described in "Hopscotch: Towards User Interface Composition" this framework promotes the separation between data (the subject) and the UI elements (presenter) . For this calendar fragment the data part will be the given date and the UI part will be a series of UI elements that represent the days of a month.

Here's an overview of the definition for the HCalendar class:


class HCalendar usingLib: platform = NewspeakObject (
|
...
|
)
(
 
class CalendarSubject for: date = Subject (
|
...
|
)
(
...
)
 
class CalendarPresenter = Presenter (
|
...
|
)
(
...
))

The subject

As mentioned above the subject only holds a given date. Some operations are added to make it easy to manipulate it.


class CalendarSubject for: date = Subject (
|
   private date = date.
|
)
('as yet unclassified'
changeDayTo: newDay <Number> = (
  date:: Date year: year
                       month: (month name)
                       day: newDay.
)
 
createPresenter = (
  ^CalendarPresenter new subject: self.
)
 
day ^ <Number> = (
  ^date dayOfMonth.
)
 
month ^ <Month>= (
  ^date month.
)
 
moveToNextMonth = (
   date:: date addMonths: 1.
)
 
moveToPreviousMonth = (
   date:: date addMonths: -1.
)
 
year ^ <Number> = (
  ^date year.
))

The presenter

The presenter class is more interesting. It takes the date from the subject and tries to create a representation of the month using Hopscotch fragments. The following snippet shows an overview of the presenter class.


class CalendarPresenter = Presenter (
"Calendar presenter, shows the days of the a month."
|
 
  protected weeksRow
  protected monthHolder
|
)
('as yet unclassified'

 
definition = (
    monthHolder::
        holder: [
           row: {
                link: '<' action: [ subject moveToPreviousMonth.
                                    refreshHolders. ].
                blank: 1.
                column: { header .
                          weeks. }.
                blank: 1.
                link: '>' action: [ subject moveToNextMonth.
                                    refreshHolders. ].
 
       }.].
    ^monthHolder.
)
 
fragmentForDaysNotInCurrentMonth = (
  ^label: ' '.
)
 
header = (
  |headerRow|
  headerRow:: row: {
                    filler.
                    label:: subject month name, ' ' , subject year asString.
                    filler.
                   }.
  ^headerRow.
)


refreshHolders= (  
      monthHolder refresh.
)
 
weekDayFragmentFor: dayNumber = (
    |dayNumberText|
    dayNumberText:: dayNumber asString.
 
    ^link: dayNumberText
        action: [ subject changeDayTo: dayNumber.
                  highlightSelectedDay.
                   ].
)
 
highlightSelectedDay = (
 ... 
)
 
 
weeks = (
 ... 
)

addFirstWeekTo: result withDaysFromPreviousMonth: previousMonthDays = (
   ...
)
 
addLastWeekTo: result weeksToShow: weeksToShow lastDayAdded: lastDayAdded daysInNextMonthToShow: nextMonthDays= (
     ... 
)
 
addMonthWeeksTo: result weeksToShow: weeksToShow lastDayAdded: lastday= (
    ... 
)
 
columnSeparator = (
   ... 
)
 
createColumnsFromWeekArray: weekArray = (
    ... 
)
 
)

The weeks method is where most of the work of creating the calendar is done. For space reasons I'm not including it here, see the link at the end of the post for the complete code.

Reusing the calendar

Now that we have the definition of the calendar presenter and subject we can reuse it to create more interesting fragments. For example the following definition could be used to create a date range picker.


class HDateRange usingLib: platform = (
"Date range selector."
|
...
|
)
(
 
class DateRangePresenter = Presenter (
"Presenter for date range."
|
   dateRangeTextHolder
|
)
(
calendar: dateSubject = (
  |aCalendar|
  aCalendar:: dateSubject createPresenter.
  aCalendar onChange: [ dateRangeTextHolder refresh ].
  ^aCalendar.
)
 
definition = (
     dateRangeTextHolder::
       holder: [label: subject initialDate selectedDate asString, ' - ',
                              subject finalDate selectedDate asString].
     ^heading: dateRangeTextHolder
       details: [
         row: {
          calendar: subject initialDate .
          blank:49.
          calendar: subject finalDate.
          }]
)
 
)
 
class DateRangeSubject from: initial to: final= Subject (
"Data for the date range."
|
     private initialDate = (HCalendar usingLib: platform) CalendarSubject for: initial.
     private finalDate = (HCalendar usingLib: platform) CalendarSubject for: final.
|
)
(
createPresenter = (
  ^ (DateRangePresenter new) subject: self.
)))

Code for this post can be found here.

Using libcurl with Newspeak FFI (continued)

2009-06-21T07:50:00.001-06:00

The previous post presented a small low level interface to libcurl using the Newspeak programming language. In this post I'm going to show the HttpServiceClient class, which was created to give a simple interface to the LibCurlHelper class.

The definition for class looks like this:


Newsqueak2
'LangexplrExperiments'

class HttpServiceClient usingLib: platform withCurlPath: curlLibraryPath = (
"This class is used to access services provided by the HTTP protocol"
|
   LibCurlHelper = platform LibCurlHelper.
   ByteString = platform ByteString.
   platform = platform.
   Transcript = platform Transcript .
   private curlLibraryPath  = curlLibraryPath .
|
)
(

class HttpRequestResult curlErrorCode: curlErrorCode httpResponse: httpResponse data: data= (
...
)
(
...
)

createNewCurlInstance = (
...
)

get: url <String> ^ <HttpRequestResult> = (
 ... 
)

get: url <String> withHeaders: headers <Array> ^ <HttpRequestResult> = (
... 
)

private isHttpsUrl: url <String> ^ <Boolean> = (
...
)

postForm: formData <Dictionary> to: url <String> ^ <HttpRequestResult> = (
...
)

) : (
...
)

The get:, get: withHeaders: and postForm: to: methods provide the functionally to do very simple GET and POST requests.

The HttpRequestResult encapsulates the result of calling these methods which has the result of calling libcurl, the HTTP response code and the text of the requested data if successful.

The code for the GET methods looks like this:


get: url <String> ^ <HttpRequestResult> = (
      | curl data tmpBuffer bufferLength response|
      ^ get: url withHeaders: {}.
)


get: url <String> withHeaders: headers <Array> ^ <HttpRequestResult> = (
      | curl data tmpBuffer bufferLength curlCallResult response|
      data:: ''.
      curl:: createNewCurlInstance.  
      curl writeCallback: 
               [:args :result|
                      bufferLength:: ((args datasize) * (args nmemb)).
                      tmpBuffer:: ByteString new:  bufferLength.
                      args data copyInto: tmpBuffer 
                                     from: 1 to: bufferLength
                                     in: (args data) startingAt: 1.
                      data:: data,tmpBuffer.
                      result returnInteger:  bufferLength.
               ]. 

      headers size > 0 ifTrue: [curl headers: headers].

      (isHttpsUrl: url)  
              ifTrue: [curl noSslVerification.].        
      curl url: url. 

      curlCallResult:: curl performOperation.

      response:: curl responseCode.
      curl cleanup.
      ^HttpRequestResult 
                   curlErrorCode: curlCallResult 
                   httpResponse: response
                   data: data.
)

The code for the POST operation looks like this:


postForm: formData <Dictionary> to: url = (
      | curl data tmpBuffer bufferLength curlFormData response curlCallResult|
      data:: ''.
 curl:: createNewCurlInstance. 

      curl post: formData.
      curl writeCallback: 
               [:args :result|
                      bufferLength:: ((args datasize) * (args nmemb)).
                      tmpBuffer:: ByteString new:  bufferLength.
                      args data copyInto: tmpBuffer 
                                     from: 1 to: bufferLength
                                     in: (args data) startingAt: 1.
                      data:: data,tmpBuffer.
                      result returnInteger:  bufferLength.
               ].

      (isHttpsUrl: url)  
              ifTrue: [curl noSslVerification.].        
         
      curl url: url.
      curlCallResult: curl performOperation.

      response:: curl responseCode.
      curl cleanup.
      ^HttpRequestResult 
                   curlErrorCode: curlCallResult 
                   httpResponse: response
                   data: data.
)

Code for this post can be found here.

Using libcurl with Newspeak FFI

2009-06-13T07:24:00.000-06:00

In this post I'm going to show a little example of using libcurl from the Newspeak programming language .

Newspeak FFI

Newspeak provides a nice mechanism to call C code. This mechanism is described in
Newspeak Foreign Function Interface User Guide document. The AlienDemo example provided with the Newspeak prototype has some nice small examples of the FFI.

The experiment presented in this post was created using the Newspeak prototype from February 2009. Due to some limitations of this release, this code only works with the Windows version of the prototype.

libcurl

libcurl is a C library that provides client access to several networking protocols with a common interface. For this post I'm going to implement a wrapper for very small subset of the functionality provided by libcurl in order to perform simple HTTP/HTTPS GET and POST requests .

The simple.c example shows how to do a simple GET request.


int main(void)
{
  CURL *curl;
  CURLcode res;

  curl = curl_easy_init();
  if(curl) {
    curl_easy_setopt(curl, CURLOPT_URL, "curl.haxx.se");
    res = curl_easy_perform(curl);
    curl_easy_cleanup(curl);
  }
  return 0;
}

The LibCurlHelper class

A class named LibCurlHelper will be used to encapsulate calls to libcurl. As you will notice the interface of this class is pretty low level. For future posts I'll try to create a better interface using more Newspeak features.


class LibCurlHelper usingLib: platform = (
"This class wraps an implementation of the libcurl library"
|
      Transcript = platform Transcript.
      Alien = platform Alien.
      UnsafeAlien = platform UnsafeAlien.
      Callback = platform Callback.
      CurlWriteCallback = platform CurlWriteCallbackNs1.
      CurlDebugCallback = platform CurlDebugCallback.
      ByteString = platform ByteString.
      OrderedCollection = platform OrderedCollection.
      ...
      public libcurlPath = ''.
      public errorBuffer = ''.
      protected CURL_OPT_URL = 10002.
      protected CURLOPT_WRITEFUNCTION = 20011.
      ...
     
      internalDebugCallback = nil.
      internalWriteCallback = nil.
      formPostData = nil.
      private curlInstance = nil.
      private aliensToRelease = nil.  

      
      CURLFORM_NOTHING = 0 .
      CURLFORM_COPYNAME = 1 .
      CURLFORM_PTRNAME = 2.
      ...

libcurl uses a lot of constaints prefixed with "CURL" this class contains definitions for some of them.

Initialization

The initializeCurl method calls the curl_easy_init (as in the simple.c example shown above) and stores the returned pointer in a slot called curlInstance which will be used in further calls.


initializeCurl = (
     |curl|
     ensureLibrariesLoaded .
     (Alien lookup: 'curl_easy_init' inLibrary: curlLibName )
            primFFICallResult: (curl:: Alien new: 4).  
     curlInstance: curl.
)

The ensureLibrariesLoaded and methods.


curlLibName = (
       ^libcurlPath, 'libcurl.dll' 
)
ensureLibrariesLoaded = (
      Alien ensureLoaded: libcurlPath, 'libidn-11.dll'.
      Alien ensureLoaded: libcurlPath, 'libeay32.dll'.
      Alien ensureLoaded: libcurlPath, 'libssl32.dll'.
      Alien ensureLoaded: curlLibName. 
)

Setting the URL

In order to set the URL for the request we need to call the curl_easy_setopt function with the CURL_OPT_URL with the URL string.

The code looks like this:


url: url <String> = (
    |result|
    (Alien lookup: 'curl_easy_setopt' inLibrary: curlLibName )
             primFFICallResult: (result:: Alien new:4)
             withArguments: {  curlInstance.
                               CURL_OPT_URL.
                               (addAlienToRelease: (url asAlien)) pointer. }.
    ^result.
)

The addAlienToRelease: method was added to in order to keep track of resources allocated in the C heap that need to be manually released when not needed. The asAlien method of the String class creates a resource of this kind.

The implementation of this method looks like this:


addAlienToRelease: anAlien = (
   aliensToRelease isNil ifTrue: [ aliensToRelease:: OrderedCollection new. ].
   aliensToRelease add: anAlien.
   ^anAlien.
)

Setting the write callback

Callback functions are used by libcurl to process the data coming from the network. The Newspeak FFI provides a nice way to add this kind of callbacks.


writeCallback:  callback <Block>= (
      |result|
      internalWriteCallback:: Callback
                                    block: callback 
                                    argsClass: CurlWriteCallback. 
 
      (Alien lookup: 'curl_easy_setopt' inLibrary: curlLibName ) 
          primFFICallResult: (result:: Alien new: 4)
          withArguments: {  curlInstance.
                            CURLOPT_WRITEFUNCTION.
                            internalWriteCallback thunk. }.

     ^result.
)

The writeCallback: method sets the block in callback as the libcurl write callback. In order to do this it creates an instance of the Callback class with the block and the arguments class. An instance of this class is used to create a function pointer which is passed to the curl_easy_setopt function.

The "arguments class" is defined using the NS1 Newspeak syntax as follows:


Newsqueak1
'LangexplrExperiments'
CurlWriteCallbackNs1 = Alien (
"Class used to represent arguments of the LibCurl write function."
'as yet unclassified'
data = (
    ^Alien forPointer: (self unsignedLongAt: 1)
)
datasize = (
    ^(self unsignedLongAt: 5)
)
nmemb = (
    ^(self unsignedLongAt: 9)
)
writerData = (
    ^Alien forPointer: (self unsignedLongAt: 13)
)
) : (
'as yet unclassified'
dataSize = (
    ^16
))

An instance of this class is used to represent the arguments of a callback call. An example of the use of this function is presented below.

Performing the request

The curl_easy_perform function is used to start the operation. The following code shows the call to this function:


performOperation = (
   |r|
    (Alien lookup: 'curl_easy_perform' inLibrary: curlLibName )
          primFFICallResult: (r:: Alien new: 4)
           withArguments: {  curlInstance. }.
   ^r signedLongAt: 1.
)

Cleanup

Finally the following method is used to release the resources allocated by libcurl.


cleanup = (
      (Alien lookup: 'curl_easy_cleanup' inLibrary: curlLibName )
          primFFICallResult: nil
             withArguments: { curlInstance }   .

      aliensToRelease do: [:anAlien | anAlien free ].
)

Example of using the library

As mentioned above, the LibCurlHelper class provides a low level interface to libcurl, something needs to be created to encapsulate this functionality.

The following method shows a method that preforms a simple GET request and returns the downloaded data as a string.


class HttpServiceClient usingLib: platform withCurlPath: curlLibraryPath = (
"This class is used to access services provided by the HTTP protocol"
|
   LibCurlHelper = platform LibCurlHelper.
   ByteString = platform ByteString.
   platform = platform.
   Transcript = platform Transcript .
   private curlLibraryPath  = curlLibraryPath .
|
)

(
simpleGet: url  ^  = (
      | curl data tmpBuffer bufferLength response|
      curl:: (LibCurlHelper usingLib: platform).
      curl libcurlPath: curlLibraryPath .
      curl initializeCurl.

      data:: ''.
      curl:: createNewCurlInstance. 

      curl writeCallback: 
               [:args :result|
                      bufferLength:: ((args datasize) * (args nmemb)).
                      tmpBuffer:: ByteString new:  bufferLength.
                      args data copyInto: tmpBuffer 
                                     from: 1 to: bufferLength
                                     in: (args data) startingAt: 1.
                      data:: data,tmpBuffer.
                      result returnInteger:  bufferLength.
               ].
      curl url: url.
      curl performOperation.
      curl cleanup.
      ^data
)
)

Notice that here the callback function modifies a local variable every time the data arrives. Also notice that args is an instance of CurlWriteCallbackNs1.

Final words

The experiment of using libcurl from Newspeak was a nice way to learn about its foreign function interface. Having access to libcurl access to useful things such as HTTPS requests.

There's already a nice Squeak wrapper for libcurl called CurlPlugin .

Code for this post can be found here.

Modifying an AS3 class with AbcExplorationLib

2009-05-17T05:37:00.005-06:00

Recently, I made a couple of changes to AbcExplorationLib to allow the modification of a compiled ActionScript 3 (AS3) class.

The following code will be used to illustrate this feature:


class Shape {
  public function foo() {


    print("Base foo");
 }
 public function paint():void {
    foo();
 } 
}

class Rectangle extends Shape{
 public override function paint():void {
    super.paint();
    print( "Rectangle");
 } 
}

class Circle extends Shape{
 public override function paint():void {
    super.paint();
    print( "Circle");
 } 
}

var shapes = [new Rectangle(),new Circle()];

for each(var s:Shape in shapes) {
  s.paint();
}

Compiling this program using the Flex SDK and running it using the Tamarin binaries shows the following output:


$ java -jar asc.jar -import builtin.abc Shapes.as

Shapes.abc, 678 bytes written
$ avmshell Shapes.abc
Base foo
Rectangle
Base foo
Circle

Say that we want to create a definition of the foo method in the Rectangle class that overrides the definition from Shape.

1. First we load the class file:


let abcFile = using (new FileStream(sourceFile,FileMode.Open)) ( 
                      fun stream -> AvmAbcFile.Create(stream))

2. Then we need a definition for the new foo implementation. The following function creates a foo override that prints a message to the screen:


let newFooMethod(message:string) = 
   AvmMemberMethod( 
     CQualifiedName(Ns("",NamespaceKind.PackageNamespace),"foo"),
     AvmMethod(
        "",
        SQualifiedName("*"),
        [||],
        Some <| 
          AvmMethodBody(
            2,1,4,5,
            [|
              GetLocal0;
              PushScope;
              FindPropertyStrict(
                   MQualifiedName(
                      [|Ns("",
                           NamespaceKind.PackageNamespace)|],
                      "print"));
              PushString message;
              CallProperty(
                   MQualifiedName(
                      [|Ns("",
                           NamespaceKind.PackageNamespace)|],
                      "print"),
                   1);
              Pop;
              ReturnVoid
              |],[||],[||]             
          )          
     ),
     AbcTraitAttribute.Override
   )

3. We need a function to add a method to a existing class. Notice that the modification consists in only creating a new instance of the AvmClass with the same values as the original but adding the new method.


let addClassMethod(aClass,newMethod) =
  match aClass with
  | AvmClass(name,
             superclassname,
             init,
             cinit,
             slots,
             methods,
             pns)  ->
      AvmClass(name,
               superclassname,
               init,cinit,
               slots,
               newMethod::methods,
               pns)

4. The following method is used to locate the rectangle class and apply the modification:


let modifyFileToAddMethod(f:AvmAbcFile) =
 AvmAbcFile(f.Scripts,
            f.Classes |>
               List.map (fun (c:AvmClass) ->
                          
                          match c.Name  with
                          | CQualifiedName(_,"Rectangle") ->
                              addClassMethod(
                                   c,
                                   newFooMethod("New foo for Rectange!!!"))
                          | _ -> c))

5. Finally we write the input file back to disk:


let modifiedAbcFile = modifyFileToAddMethod(abcFile)
let abcFileCreator = AbcFileCreator()
let file = modifiedAbcFile.ToLowerIr(abcFileCreator)
using (new BinaryWriter(new FileStream(targetFileName,FileMode.Create)))
      (fun f -> file.WriteTo(f))

After running this program we can execute the bytecode again to get the new results:


$ mono modify.exe Shapes.abc
...
$ avmshell Shapes_t.abc
New foo for Rectange!!!
Rectangle
Base foo
Circle

One area that really needs works is name handling. A particular challenge is to find a good way to represent multinames ( name references in a set of name spaces ).

In general the way of defining a method from scratch(for example in newFooMethod) needs some work since it is requires lots of details that might not be interesting for the developer.

Finally, another area that really needs improvement is the output file generation. Right now it requires the user to write three instructions to write the file to disc. This will be changed to be similar to the load process.

Code for this program can be found as part of the AbcExplorationLib samples.

Writing a small Twitter client with Newspeak and Hopscotch

2009-04-14T06:22:00.001-06:00

In this post I'm going to show a small Twitter client written using the Newspeak programming language and the Hopscotch framework.

As part of the process of exploring the Newspeak language here I'm going to focus on the Hopscotch UI framework. It will be used to present information from the Twitter REST API which is very easy to use.

The Hopscotch framework and IDE is described in the "Hopscotch: Towards User Interface Composition" paper by Vassili Bykov.

Code for this program was created using the Newspeak prototype from 2009-02-27.

The program

Here's a screenshot of the program:

The program is not a complete client, it just allows to post a new twit and to read the current time line of a the user.

The code from the previous post "Parsing JSON with Newspeak" is used to access the data provided by the Twitter services.

The code

The following screenshot shows the definition of the TwitterGUI class

The following nested classes provide the functionality for the client:

TwitterClient: A class for using the Twitter REST API
TwitPresenter,TwitSubject: The UI piece and information for a single twit
TwitterMainPresenter,TwitterMainSubject: The UI piece and information for a the complete client

As described in the Hopscotch paper a pair of elements is required to create a UI piece. A subject which contains the information being presented and the presenter which defines the UI that shows it.

In the case of a single twit the data is transmitted from the service as a JSON document.

The following code shows an example of the JSON response of a single twit from the Twitter REST API:


[{"in_reply_to_screen_name":null,
  "user":{
     "description":"Father, husband, friend, developer and late night programming language enthusiast.",
     "statuses_count":318,
     "utc_offset":-21600,
     "profile_background_tile":false,
     "profile_background_color":"6E8182",
     "following":null,
     "profile_text_color":"000000",
     "url":"http:\/\/langexplr.blogspot.com",
     "name":"Luis Diego Fallas",
     "protected":false,
     "profile_image_url":"http:\/\/s3.amazonaws.com\/twitter_production\/profile_images\/133285952\/ldnach_normal.png",
     "notifications":null,
     "profile_link_color":"0000ff",
     "profile_background_image_url":"http:\/\/static.twitter.com\/images\/themes\/theme1\/bg.gif",
     "created_at":"Mon Jul 28 13:51:55 +0000 2008",
     "screen_name":"ldfallas",
     "profile_sidebar_fill_color":"e0ff92",
     "followers_count":45,
     "time_zone":"Central America",
     "location":"Costa Rica",
     "id":15631932,
     "favourites_count":9,
     "friends_count":43,
     "profile_sidebar_border_color":"87bc44"},
  "text":"Experimenting with Hopscotch in Newspeak",
  "truncated":false,
  "in_reply_to_status_id":null,
  "created_at":"Tue Apr 07 14:21:57 +0000 2009",
  "in_reply_to_user_id":null,
  "id":1469769323,
  "favorited":false,
  "source":"web"}
...]

A TwitSubject represents one of these pieces of information.


class TwitSubject withTwit: twit images: images= Subject (
"Describe the class in this comment."
|
   theTwit = twit.
   theImages = images.
|
)
(
createPresenter  = (
    ^TwitPresenter new subject: self.
)
)

The definition of the TwitPresenter looks like this:


class TwitPresenter = Presenter (
"Presenter for a single twit."
|   
|
)
(
definition = (
   ^(padded:( column: {

        (row: { link: (subject theTwit user screen_name) action: [] . }) color: ( Color veryVeryLightGray).
         row: { image: (subject theImages at: (subject theTwit user profile_image_url)) .
         blank: 4.
         elastic: twitBody}.

   }) with: {3. 3. 2. 2.}) .
)
...
)

The following screenshot shows an example of the previous definition.

A special treatment need to be applied to the message since we want to be able to click on links or twitter user id's (not supported right now). The definition of twitBody shows this.


twitBody = (
   |  text result |
   text: subject theTwit text.
   ((string: text contains: '@') or: [string: text contains: 'http'] )
   ifTrue: [ result:: flow: ((text subStrings: {Character space}) collect: [:t | componentFor: t])]
   ifFalse: [ result:: textDisplay: text ].
   ^result
)

componentFor: s = (
   |result|
   (s includesSubString: '@') 
   ifTrue: [ result:: link: s action: [] ]
      ifFalse: [
        (string: s contains: 'http')
      ifTrue: [ result:: link: s action: [openLink: s] ]
      ifFalse:[ result:: label: s]
   ].
   ^result.
)

openLink:url  = (
   OSProcess command:  (browser , ' ' , url).
)

What these methods do is to break the string of the message into words separated by spaces. If a word is an URL or and '@' character a link is created if not a 'label' is created . For the future this code needs to be improved with a better technique to identify urls.

The following code shows the definition of TwitterMainSubject:


class TwitterMainSubject user: userName password: pswd  = Subject (
"Main subject."
|
   
   user = userName.
   password = pswd.
   data
   images = Dictionary new.
   twitterClient = TwitterClient user: userName password:pswd.   

|
)
createPresenter ^  = (    
   data:: twitterClient getFriendsTimeline.
   ^TwitterMainPresenter new subject: self
)

twits = (
   data:: twitterClient getFriendsTimeline.
   ^data collect: [:t | TwitSubject withTwit: t images: imagesDictionary].
)

updateStatus: statusText = (
   twitterClient updateStatus: statusText.
)
...

This class receives the used and password of the Twitter account. The TwitterClient class provides the access to the Twitter services.

Here's the definition of the TwitterMainPresenter class.


class TwitterMainPresenter = Presenter (
"Presenter for the main section of the GUI client."
|
   editor
   twitsHolder
   charCountHolder
   
|
)
(

definition ^ <Fragment> = (
   ^column: { 
        row: { label: 'What are you doing?'.} . 
        row: { elastic:twitEditor.} . 
        row: { getCharCountHolder.
               filler.
               button: 'Update' action:[updateStatus: (editor editedText asString)].
               blank: 5.
               button: 'Refresh' action:[twitsHolder refresh].
               blank: 5.
             }. 
        row: { 
               blank: 5.
               elastic:: getTwitsHolder 
             }}
))

The getTwitsHolder method create an instance of an HolderComposer object that allows the content to be recalculated using the refresh method. Here's the definition:


getTwitsHolder = (
   twitsHolder:: holder: [ list:: subject twits collect: [ :i | i presenter  ] ].
   ^twitsHolder.
)

Notice also that here we are requesting the list of twits to the subject, which calls the web service again getting new content.

Final words

The experience of using the Newspeak and the Hopscotch framework to create this program was very nice.

One thing that I need to find out is how to prevent the application from blocking when requesting the data from the services.

Code for this post can be found here.

Parsing JSON with Newspeak

2009-04-02T06:40:00.003-06:00

In this post I'm going to show a JSON parser written using the Newspeak parser combinator library.

Newspeak

Newspeak is a new programmming language. From its webpage http://newspeaklanguage.org/:

Newspeak is a new programming language in the tradition of Self and Smalltalk. Newspeak is highly dynamic and reflective - but designed to support modularity and security. It supports both object-oriented and functional programming.

In The Newspeak Programming Platform the authors give a nice introduction to the language and platform.

The first time I heard about Newspeak was by watching the Lang.NET 2008 symposium presentation video by Gilad Braha (available here). In this video, a nice parser combinator library is presented . This parser combinator library is described in the Executable Grammars in Newspeak paper by Gilad Bracha.

Code in this post was written using the Newspeak prototype released February 27 2009.

JSON

In order to learn about the language and platform I decided to create a little parser for JSON (Javascript Object Notation).

JSON is a simple data-interchange format defined in http://www.json.org/ .An example of it:


[{ "name": "Wiston Smith",
   "description"  :"Protagonist"},
 { "name": "Julia",
   "description"  :"Lover"},
 { "name": "O Brien",
   "description"  :"Goverment agent"}]

Parser structure

The parser is defined as a single class with a couple of nested classes. The following image shows the definition of JSON Parser in the Newspeak environment.

As described in the "Modularity" section of the The Newspeak Programming Platform paper, top-level classes (in this case JSONParser) doesn't have access to its surrounding scope, it only has access to its own or inherited definitions. This is the reason why the JSONParser has the following definitions:


class JSONParser withParserLib: parserLibrary usingLib: platform =  (
"Experiment for JSON parser based on the description from http://www.json.org/fatfree.html "
|
 ExecutableGrammar = parserLibrary ExecutableGrammar.
 CharParser = parserLibrary CharParser.
 PredicateTokenParser = parserLibrary PredicateTokenParser.
 Dictionary = platform Dictionary.
 OrderedCollection = platform OrderedCollection.
 Number = platform Number.
|
)
...

The "withParserLib: parserLibrary usingLib: platform" part defines parameters for the construction of JSONParser. These parameters are used to 'import' classes defined elsewhere.

The following code shows a way to create an instance of the JSONParser class:


    |platform parser|
    platform:: Platform new.
    parser = (JSONParser withParserLib: (BlocklessCombinatorialParsing usingLib: platform) usingLib: platform).
    ...

The JSONParser nested classes are the following:

CharExceptForParser which is a parser that accepts any character except for the one specified (this is for internal use)
JSONGrammar The definition of the JSON grammar
JSONGrammarWithAST which defines the way the AST is created
JSONObject which is used in the representation of the JSON AST

Grammar

The following code shows the JSON grammar defined using the parsing combinators:


class JSONGrammar = ExecutableGrammar (
"Experiment for JSON grammar based on the description from http://www.json.org/fatfree.html "
|
   doubleQuote = (char: $").
   backslash = (char: $\).
   str = doubleQuote,((backslash, ( char: $" ))   | 
                                  (backslash, ( char: $/ ))  |
                                  (backslash, backslash)    |
                                  (backslash, ( char: $r ))   |
                                  (backslash, ( char: $n ))   |
                                  (backslash, ( char: $t ))   |
                                  (charExceptFor: $")) star, doubleQuote.
   string = tokenFor: str.
   
   negSign = (char: $-).
   plusSign = (char: $+).
   digit = (charBetween: $0 and: $9).
   dot = (char: $. ) .
   num = negSign opt, digit, digit star, dot opt,digit star, ((char: $e) | (char: $E)) opt, (plusSign | negSign) opt,digit star.
   number = tokenFor: num.
   
   leftbrace = tokenFromChar: ${.
   rightbrace =tokenFromChar: $}.
   colon = tokenFromChar: $:.
   comma = tokenFromChar: $,.
   definition = string,colon,value.
   obj =  leftbrace,  (definition starSeparatedBy: comma),rightbrace.
   object = tokenFor: obj.

   leftbracket = tokenFromChar: $[.
   rightbracket = tokenFromChar: $].
   arr = leftbracket, (value starSeparatedBy: comma), rightbracket.
   array = tokenFor: arr.

   ttrue = tokenFromSymbol: #true.
   tfalse = tokenFromSymbol: #false.
   null = tokenFromSymbol: #null.

   value = string | number | object | array | ttrue | tfalse | null. 
   
|
)
...

For more information on the how this library works, check the Executable Grammars in Newspeak paper.

AST construction

We need to define a way to represent the tree structure(AST) parsed by JSONGrammar. As described in the "Executable Grammars is Newspeak" paper, one of the nice things about Newspeak is that we don't have the modify the grammar definition to add AST construction code. We can do that by inheriting from the original grammar:


class JSONGrammarWithAST = JSONGrammar(
"Parses a JSON File and generates and Ast"
|
   
|
)
('as yet unclassified'
array = (
   ^super array wrapper: [:a | (a token at: 2)  ].
)


null = (
   ^ super null wrapper: [:o | nil].
)

number = (
   ^super number wrapper: [:o | Number readFrom: (flattenCharCollectionToString: (o token)) ].
)

object  = (
   ^super object wrapper: 
         [:obj | JSONObject withContent: 
                     (Dictionary newFrom: ((obj token at: 2) collect: [:e | (e at: 1) -> (e at: 3)]))].
)

parse: input  = (
   ^super value parse: input.
)

string  = (
   ^super string wrapper: 
          [:t | flattenCollectedString: (t token at: 2)].
)

tfalse  = (
   ^super tfalse wrapper: [:o | false].
)

ttrue = (
   ^super ttrue wrapper: [:o | true].
)

...
)

As shown here (omitting some method definitions) the arrays are converted to Ordered collections, the numbers,strings,booleans to its equivalents and JSON objects to instances of JSONObject(described below).

JSONObject

In order to make it easy to use a JSON object in Newspeak the JSONParser class was defined:


class JSONObject withContent: dContent =  (
"Instances of this class represent JSON objects."
|
   content = dContent.
|
)
('as yet unclassified'
doesNotUnderstand: message = (
      | fieldName |
      fieldName:: message selector string.
      (fieldName beginsWith: 'json_')  
             ifTrue: [fieldName:: fieldName allButFirst: 5].
      ^content  at: fieldName ifAbsent: [nil].
)

)

This class receives a Dicionary as parameter. This dictionary contains all the name/value pairs of the JSON object definition. We create a definition of the doesNotUnderstand method which as in Smalltalk is called when a message sent to an object doesn't have a explicit way to respond it. We take the name of the message being called and check it against the dictionary.

If the message is prefixed by 'json_', the string after it is used as the key in the dictionary. This is defined this way because JSONObject has definitions inherited from Object (such as 'name').


...
| parsed |
parsed:: parserWithAST 
                     parse: (streamFromString: '[{ "name": "Wiston Smith",
                                                   "description"  :"Protagonist"},
                                                 { "name": "Julia",
                                                   "description"  :"Lover"},
                                                 { "name": "O Brien",
                                                   "description"  :"Goverment agent"}]').
assert:[((parsed at: 2) description)  = 'Lover'].
assert:[((parsed at: 3) json_name)  = 'O Brien'].

In the following post I'm going to use this parser to explore the GUI library provided with Newspeak.

Code for this post can be found here.

Support for the LookupSwitch opcode in AbcExplorationLib

2009-03-15T07:55:00.001-06:00

The LookupSwitch AVM2 opcode is used to represent the ActionScript switch statement.

This is an interesting branch instruction because it has multiple targets. It is almost a direct translation of the switch statement because it has two parameters, a default case and an array of possible targets.

Recently support for this opcode was added to AbcExplorationLib.

Given the following ActionScript code:


var x;
for(x = 1;x < 5;x++) {
   switch(x) {
      case 1:
        print("one");
        break;
      case 2:
        print("two");
        break;
      case 3:
        print("three");
        break;
      case 4:
        print("four");
        break;
   }
}

We compile this file to a .abc file using the following command:


$ java -jar /opt/flex3sdk/lib/asc.jar  testswitch.as 

testswitch.abc, 280 bytes written

By using a little IronPython example included with AbcExplorationLib we can see how this library interprets the LookupSwitch opcode:


$ mono /opt/IronPython-2.0/ipy.exe  ../ipyexample/abccontents.py testswitch.abc 
...
Instructions:
   getlocal_0 
   pushscope 
   pushbyte 1
   getglobalscope 
   swap 
   setslot 1
   jump dest177
dest12:
   label 
   jump dest74
dest17:
   label 
   findpropertystrict M.print
   pushstring "one"
   callprop M.print
   pop 
   jump dest165
dest30:
   label 
   findpropertystrict M.print
   pushstring "two"
   callprop M.print
   pop 
   jump dest165
dest43:
   label 
   findpropertystrict M.print
   pushstring "three"
   callprop M.print
   pop 
   jump dest165
dest56:
   label 
   findpropertystrict M.print
   pushstring "four"
   callprop M.print
   pop 
   jump dest165
dest69:
   label 
   jump dest165
dest74:
   getglobalscope 
   getslot 1
   setlocal_1 
   pushbyte 1
   getlocal_1 
   ifstrictneq dest91
   pushshort 0
   jump dest143
dest91:
   pushbyte 2
   getlocal_1 
   ifstrictneq dest104
   pushshort 1
   jump dest143
dest104:
   pushbyte 3
   getlocal_1 
   ifstrictneq dest117
   pushshort 2
   jump dest143
dest117:
   pushbyte 4
   getlocal_1 
   ifstrictneq dest130
   pushshort 3
   jump dest143
dest130:
   pushfalse 
   iffalse SolvedReference dest141
   pushshort 4
   jump dest143
dest141:
   pushshort 4
dest143:
   kill 
   lookupswitch dest69 dest17,dest30,dest43,dest56,dest69
dest165:
   getglobalscope 
   getslot 1
   increment 
   setlocal_1 
   getlocal_1 
   getglobalscope 
   swap 
   setslot 1
   kill 
dest177:
   getglobalscope 
   getslot 1
   pushbyte 5
   iflt dest12
   returnvoid

As described in the documentation the parameters of the LookupSwitch instruction are specified as relative byte offsets that specify the target. In order to give a higher level representation of the code, these relative offsets are converted to symbolic references. This process is detailed in the "Using F# Active Patterns to encapsulate complex conditions" post.

This process starts in the following functions:


static member ReadAndProcessInstructions(aInput:BinaryReader,
                                         count,
                                         constantPool:ConstantPoolInfo) = 
   let instructionsAndOffsets = 
               (AvmMethodBody.ReadingInstructions([],
                                                  aInput,
                                                  count,
                                                  constantPool)) in        
   let destinations = 
               AvmMethodBody.CollectDestinations(instructionsAndOffsets,
                                                 Map.empty,
                                                 instructionsAndOffsets)
   in
      AvmMethodBody.UpdateCodeWithDestinations(
                                    destinations,
                                    instructionsAndOffsets,[]) |> List.to_array

The CollectDestinations method collects all the absolute offsets used by branch instructions and stores them in a dictionary with a generated label. The UpdateCodeWithDestinations method modifies the instruction list use the generated labels.

For example to add support in CollectDestinations the following code was added:


static member CheckLookupSwitchCase  baseOffset
                                      (totalInstructions:(int64*AbcFileInstruction)
                                      list) =
    fun (destinations:Map<int64,string>) target  ->
       match target with
       | UnSolvedReference(relativeOffset) when
           (AvmMethodBody.IsDestinationDefined(int(baseOffset+relativeOffset),
                                               totalInstructions)) ->
                 destinations.Add(int64(baseOffset+relativeOffset),
                                  sprintf "dest%d" (baseOffset+relativeOffset))
       | _ -> destinations

 static member  CollectDestinations(instructions:(int64*AbcFileInstruction) list, 
                                    destinations:Map<int64,string>,
                                    totalInstructions:(int64*AbcFileInstruction) list) =
    match instructions with 
    ...
    | ((offset,(LookupSwitch(defaultBranch,cases)))::rest) ->
           let baseOffset = int(offset) in
            AvmMethodBody.CollectDestinations(rest,
                                    Seq.append [defaultBranch] cases |>                                        
                                    Seq.fold (AvmMethodBody.CheckLookupSwitchCase baseOffset totalInstructions ) destinations,
                                    totalInstructions)
   ...

Then the code is modified in the UpdateCodeWithDestinations method to use the generated labels.


 static member UpdateCodeWithDestinations(destinations:Map<int64,string>,
                                          instructions,
                                          resultingInstructions) =
    let processedInstructions = 
          match instructions with
   ...
          | ((offset,LookupSwitch(defaultCase,cases))::rest) ->                 
             (offset,
                LookupSwitch(AvmMethodBody.SolveSwitchCase defaultCase offset destinations,
                             Array.map (fun c->AvmMethodBody.SolveSwitchCase c offset destinations) cases))::rest
          | _ -> instructions
   ...

Manipulating AVM2 byte code with F#

2009-03-03T07:20:00.002-06:00

In this post I'm going to show an example of using AbcExplorationLib to manipulate simple AVM2 byte code (ActionScript). This example show how load a .ABC file and write it back to disk.

AbcExplorationLib is a library that will allow the manipulation of AVM2 Byte Code(described here). Although it's still incomplete, some basic examples work as the ones presented in this post .

The following ActionScript code will be compiled to byte code .


var i = 0;
for( i = 0;i < 10;i++) {
   print("inside loop");
}
print("Done");

To generate the ".abc" file we type:

c:\test\> java -jar c:\flexsdk\lib\asc.jar test.as

Loading the compiled file

We're going to use the F# REPL(fsi.exe) to manipulate the file. We start by referencing the library.


> #r "abcexplorationlib.dll";;

--> Referenced 'C:\test\abcexplorationlib.dll'

> open Langexplr.Abc;;

Now we load the file:


>   let abcFile = using (new System.IO.FileStream("test.abc",System.IO.FileMode.Open)) ( 
                    fun s -> AvmAbcFile.Create(s));;

Now abcFile contains the code of the compiled program.


> abcFile;;
val it : AvmAbcFile
= Langexplr.Abc.AvmAbcFile {Classes = [];
                            Scripts = [Langexplr.Abc.AvmScript];}

Inspecting the instructions

We're interested in the instructions of the top-level script for this .abc file. By typing the following expression we can get to this section:


> abcFile.Scripts.[0].InitMethod.Body.Value.Instructions;;
val it : AbcFileInstruction array
= [|GetLocal0; PushScope; PushByte 0uy; GetGlobalScope; Swap; SetSlot 1;
    PushByte 0uy; GetGlobalScope; Swap; SetSlot 1;
    Jump (SolvedReference "dest39"); ArtificialCodeBranchLabel "dest18"; Label
    FindPropertyStrict
      (MQualifiedName
         ([|Ns ("",CONSTANT_Namespace); Ns ("test.as$0",CONSTANT_PrivateNs)|],
          "print")); PushString "inside loop";
    CallProperty
      (MQualifiedName
         ([|Ns ("",CONSTANT_Namespace); Ns ("test.as$0",CONSTANT_PrivateNs)|],
          "print"),1); Pop; GetGlobalScope; GetSlot 1; Increment; SetLocal_2;
    GetLocal2; GetGlobalScope; Swap; SetSlot 1; Kill 2;
    ArtificialCodeBranchLabel "dest39"; GetGlobalScope; GetSlot 1;
    PushByte 10uy; IfLt (SolvedReference "dest18");
    FindPropertyStrict
      (MQualifiedName
         ([|Ns ("",CONSTANT_Namespace); Ns ("test.as$0",CONSTANT_PrivateNs)|],
          "print")); PushString "Done";
    CallProperty
      (MQualifiedName
         ([|Ns ("",CONSTANT_Namespace); Ns ("test.as$0",CONSTANT_PrivateNs)|],
          "print"),1); CoerceA; SetLocal_1; GetLocal1; ReturnValue; Kill 1|]

We're going to define the following function to assist in the presentation of instruction listings.


> open Langexplr.Abc.InstructionPatterns;;
> let pr (i:AbcFileInstruction) =
-            match i with
-            | ArtificialCodeBranchLabel t -> printf "%s:\n" <| t.ToString()
-            | i & UnsolvedSingleBranchInstruction(d,_) -> printf " %s %d\n" i.Name d
-            | i & SolvedSingleBranchInstruction(l,_) -> printf " %s %s\n" i.Name l
-            | _ -> printf " %s\n" i.Name;;

val pr : AbcFileInstruction -> unit

Now we can type:


> abcFile.Scripts.[0].InitMethod.Body.Value.Instructions |> Array.iter pr;;
 getlocal_0
 pushscope
 pushbyte
 getglobalscope
 swap
 setslot
 pushbyte
 getglobalscope
 swap
 setslot
 jump dest39
dest18:
 label
 findpropertystrict
 pushstring
 callprop
 pop
 getglobalscope
 getslot
 increment
 setlocal_2
 getlocal_2
 getglobalscope
 swap
 setslot
 kill
dest39:
 getglobalscope
 getslot
 pushbyte
 iflt dest18
 findpropertystrict
 pushstring
 callprop
 coerce_a
 setlocal_1
 getlocal_1
 returnvalue
 kill
val it : unit = ()

A note on branch instructions

In order to make it easy to manipulate and analyze the code AbcExplorationLib adds a non-existing instruction called ArtificialCodeBranchLabel to mark the position where a branch instruction will jump. When these labels are generated the branch instructions are modified to point to the label's name instead of a relative byte offset. Details on how this process is briefly described in "Using F# Active Patterns to encapsulate complex conditions"

Converting from label references to byte offsets is also necessary to write code back to an .abc file. This process is performed by a function called ConvertSymbolicLabelsToByteReferences, for example:


> let c = AbcFileCreator();;

val c : AbcFileCreator

> abcFile.Scripts.[0].InitMethod.Body.Value.Instructions |>
-      InstructionManipulation.ConvertSymbolicLabelsToByteReferences c |>
-      Array.iter pr;;
 getlocal_0
 pushscope
 pushbyte
 getglobalscope
 swap
 setslot
 pushbyte
 getglobalscope
 swap
 setslot
 jump 21
dest18:
 label
 findpropertystrict
 pushstring
 callprop
 pop
 getglobalscope
 getslot
 increment
 setlocal_2
 getlocal_2
 getglobalscope
 swap
 setslot
 kill
dest39:
 getglobalscope
 getslot
 pushbyte
 iflt -30
 findpropertystrict
 pushstring
 callprop
 coerce_a
 setlocal_1
 getlocal_1
 returnvalue
 kill
val it : unit = ()
>

Modifying the code

Values for branch instruction targets are adjusted if new code added, for example, lets add some code to print "Hola!" inside the loop.


> let printName = CQualifiedName(Ns("",NamespaceKind.CONSTANT_Namespace),"print"
- ) ;;

val printName : QualifiedName

> let printCode = [| FindPropertyStrict printName ;
-                    PushString "Hola!" ;
-                    CallProperty(printName,1);
-                    Pop |] ;;

val printCode : AbcFileInstruction array

> Seq.append instr.[0..16] <| Seq.append printCode instr.[17..] |>
-  Seq.to_array |>
-  InstructionManipulation.ConvertSymbolicLabelsToByteReferences c |>
-  Array.iter pr;;
 getlocal_0
 pushscope
 pushbyte
 getglobalscope
 swap
 setslot
 pushbyte
 getglobalscope
 swap
 setslot
 jump 29
dest18:
 label
 findpropertystrict
 pushstring
 callprop
 pop
 findpropertystrict
 pushstring
 callprop
 pop
 getglobalscope
 getslot
 increment
 setlocal_2
 getlocal_2
 getglobalscope
 swap
 setslot
 kill
dest39:
 getglobalscope
 getslot
 pushbyte
 iflt -38
 findpropertystrict
 pushstring
 callprop
 coerce_a
 setlocal_1
 getlocal_1
 returnvalue
 kill
val it : unit = ()

Writing the new file

We can write this code back to a .abc file by doing this:


> let newCode =  Seq.append instr.[0..16] <| Seq.append printCode instr.[17..] |
- > Seq.to_array;;

val newCode : AbcFileInstruction array

> let newBody = AvmMethodBody(oldbody.Method,
-                    oldbody.MaxStack,
-                    oldbody.LocalCount,
-                    oldbody.InitScopeDepth,
-                    oldbody.MaxScopeDepth,
-                    newCode,
-                    oldbody.Exceptions,
-                    oldbody.Traits);;

val newBody : AvmMethodBody

> let newFile = AvmAbcFile( [AvmScript( abcFile.Scripts.[0].InitMethod.CloneWithBody(newBody), abcFile.Scripts.[0].Members)], []);;

val newFile : AvmAbcFile
> open System.IO;;
> let c = AbcFileCreator();;

val c : AbcFileCreator

>
- using (new BinaryWriter(new FileStream("test_modified.abc",FileMode.Create)))
-
-       (fun f -> let file = newFile.ToLowerIr(c) in file.WriteTo(f));;
val it : unit = ()

Running this program using Tamarin shows:


c:\test\>avmplus_sd.exe test_modified.abc
inside loop
Hola!
inside loop
Hola!
inside loop
Hola!
inside loop
Hola!
inside loop
Hola!
inside loop
Hola!
inside loop
Hola!
inside loop
Hola!
inside loop
Hola!
inside loop
Hola!
Done

The AbcExplorationLib library is still pretty incomplete. Also there's a lot to improve, for example name handling and instruction modification. Future posts will present new features/experiments.

Writing Xml with IronPython, XmlWriter and the 'with' statement

2009-02-12T05:23:00.002-06:00

This post shows a little example of wrapping calls to System.Xml.XmlWriter inside a Python's 'with' statement using IronPython.

Writing Xml

While reading some code examples from the XIST HTML/XML generation library I noticed the nice use of Python's 'with' statement to represent the target HTML or XML.

The System.Xml.XmlWriter class provided with .NET already gives you a way to write well formed Xml documents. In this post I'm going to show how to use an XmlWriter instance in conjunction with Python's 'with' statement.

We want to write the following code:


from __future__ import with_statement

...

w = XmlWriter.Create(System.Console.Out,XmlWriterSettings(Indent=True))
x = XWriter(w)

with x.element('tableofcontents'):
    with x.element('section',{'page' : '10'}):
       x.text('Introduction')
    with x.element('section',{'page' : '12'}):
       x.text('Main topic')
    with x.element('section',{'page' : '14'}):
       x.text('Extra topic')

To generate the following Xml file:


<tableofcontents>
  <section page="10">Introduction</section>
  <section page="12">Main topic</section>
  <section page="14">Extra topic</section>
</tableofcontents>

The 'with' statement

The 'with' statement was introduced in Python 2.5 . This statement is used wrap the execution of a series of statements with some special code. For example it is used to implement the try...except...finally pattern.

As described in the documentation the following statement:


with context expression:
   statements...

Will be executed as follows:

Evaluate the context expression to obtain the context manager

Invoke the context manager's __enter__() method

Execute the statements

When the execution of the statements finishes(even with an exception), the context manager's __exit()__ method is called.

Given these steps we're going to implement a context manager that assist in the creation of Xml documents using the System.Xml.XmlWriter .NET class.

The following code shows a class that wraps the XmlWriter instance and helps with the creation of context managers:


class XWriter(object):
   def __init__(self,writer):
      self.writer = writer

   def element(self,name,atts = {}):
       return ElementCtxt(name,atts,self)

   def nselement(self,prefix,name,namespace,atts = {}):
       return NamespaceElementCtxt(prefix,name,namespace,atts,self)


   def text(self,text):
       self.writer.WriteString(text)

   def cdata(self,text):
       self.writer.WriteCData(text)

Notice that the element method creates an instance of the ElementCtxt class using the element name and an optional dictionary with the attributes. As the following listing shows this class performs the calls to WriteStartElement and WriteEndElement in the __enter__ and __exit__ methods.


class ElementCtxt(object):
   def __init__(self,elementName,atts,writer):
      self.elementName = elementName
      self.atts = atts
      self.writer = writer

   def processAttributes(self):
      for att in self.atts:
         self.writer.writer.WriteAttributeString(att,self.atts[att].__str__())
   
   def processStartTag(self):
      self.writer.writer.WriteStartElement(self.elementName)
      self.processAttributes()

   def __enter__(self):
      self.processStartTag()
      return self

   def __exit__(self,t,v,tr):
      self.writer.writer.WriteEndElement()
      return t == None

The XWriter.nselement method is used to write elements with namespace and prefix. This call generates an instance of the following context manager:


class NamespaceElementCtxt(ElementCtxt):
   def __init__(self,prefix,elementName,namespace,atts,writer):
      ElementCtxt.__init__(self,elementName,atts,writer)
      self.namespace = namespace
      self.prefix = prefix
   def processStartTag(self):
      self.writer.writer.WriteStartElement(self.prefix,self.elementName,self.namespace)     
      self.processAttributes()

Final example

The following code shows how to create a little SVG file:


from __future__ import with_statement
from xmlwriterw import XWriter

import clr

clr.AddReference('System.Xml')

from System.Xml import *
import System


w = XmlWriter.Create(System.Console.Out,\
                     XmlWriterSettings(Indent=True))
x = XWriter(w)

svgNs = 'http://www.w3.org/2000/svg'

with x.nselement('s','svg',svgNs,{'version': '1.1',
                                  'viewBox': '0 0 100 100',
                                  'style':'width:100%; height:100%; position:absolute; top:0; left:0; z-index:-1;'}):
   with x.nselement('s','linearGradient',svgNs, { 'id' : 'gradient' }):
       with x.nselement('s','stop',svgNs, {'class' : 'begin',
                                           'offset' : '0%',
                                           'stop-color':'red'}): 
          pass
       with x.nselement('s','stop',svgNs, {'class' : 'end',
                                           'offset' : '100%'}): 
          pass
   with x.nselement('s','rect',svgNs, { 'x':0,
                                        'y':0,
                                        'width':100,
                                        'height':100,
                                        'style':'fill:url(#gradient)'} ):
         pass
   for i in range(1,5):
      with x.nselement('s','circle',svgNs, { 'cx': 50,
                                             'cy': 50,
                                             'r': 30 - i*3,
                                             'style':'fill:url(#gradient)'} ):
          pass
   
w.Close()

Running this program shows:


<s:svg viewBox="0 0 100 100" style="width:100%; height:100%; position:absolute; top:0; left:0; z-index:-1;" version="1.1" xmlns:s="http://www.w3.org/2000/svg">
  <s:linearGradient id="gradient">
    <s:stop offset="0%" class="begin" stop-color="red" />
    <s:stop offset="100%" class="end" />
  </s:linearGradient>
  <s:rect x="0" height="100" width="100" style="fill:url(#gradient)" y="0" />
  <s:circle cx="50" cy="50" style="fill:url(#gradient)" r="27" />
  <s:circle cx="50" cy="50" style="fill:url(#gradient)" r="24" />
  <s:circle cx="50" cy="50" style="fill:url(#gradient)" r="21" />
  <s:circle cx="50" cy="50" style="fill:url(#gradient)" r="18" />
</s:svg>

Some notes on using F# code from IronPython

2009-01-21T06:29:00.001-06:00

This post presents a couple of things I learned about the interaction between IronPython and F#.

Most information about this topic can be found by looking at the code generated by the F# compiler, using tools like ILDASM or Reflector. Also the F# from C# section of the F# 1.1.12 documentation is very helpful.

FSharp.Code library

In order to call some F# basic functionality we need to add a reference to FSharp.Core.

Python


import clr
clr.AddReference('FSharp.Core')

Type parameters

IronPython provides a way to specify type parameters to methods and classes by using square brackets followed by the names of the types to be used. This mechanism can be used to create F# constructs that require them.

The following example shows how to create an F# list:

Python


>>> import clr
>>> clr.AddReference('FSharp.Core')
>>> from Microsoft.FSharp.Collections import List
>>> l1 = List[str].Cons('World',List[str].Empty)
>>> l2 = List[str].Cons('Hello',l1)
>>> " ".join(l2)
'Hello World'

Type parameters, for methods that require them, also can be specified using the same syntax. For example the following code shows the use of Seq.length with an Python sequence.

Python


print Seq.length[int]([45,234,52,345,2,346,657])

Tuples

Tuples in F# are instances of the Microsoft.FSharp.Core.Tuple class (with several definitions with depending on the number of arguments). A tuple instance contains properties Item# where # is the position of the element.

The following example shows how to use a tuple created in F# from IronPython.

F#


module MyFunctions = begin
  ...
  let numberEvenTest x = x,x % 2 = 0
  ...
end

Python


print MyFunctions.numberEvenTest(46)
print MyFunctions.numberEvenTest(46).Item1
print MyFunctions.numberEvenTest(46).Item2

This program will print:


(46,True)
46
True

To create a tuple inside IronPython the Tuple[type1,type2,...](value1,value2,...) constructor must be used.

For example to given the following definition:

F#


module MyFunctions = begin
...
  let movePoints points xdistance ydistance =
     Seq.map (fun (x,y) -> (x+xdistance,y+ydistance)) points
...
end

We can used from IronPython like this:
Python


from Microsoft.FSharp.Core import Tuple 
...
points = [ Tuple[int,int](1,2), Tuple[int,int](5,3) ]

for p in MyFunctions.movePoints(points,20,40):
   print p

It is tempting to use the Python syntax for assigning a list of variables to the values of the tuples. For example:


x,y,z = 2,3,5

However this is not possible since F# tuples are not compatible with IronPython sequences.

Operators

F# operators are created using the same conventions as the C# operators. Thus they can be used directly in IronPython.

For example the following F# definition:


type Complex(real:double,img:double) = class
   member this.real with get() = real
   member this.img with get() = img
   static member (+) (c1:Complex,c2:Complex) = 
        Complex(c1.real+c2.real,c1.img+c2.img)
   override this.ToString() = sprintf "%f + %fi" real img
end

Can be used in IronPython:

Python


c1 = Complex(20,234)
c2 = Complex(34.35,32.2)
c3 = (c1+c2)

Discriminated unions

Discriminated unions in F# provide a compact way to define data structures. The following example shows the definition of simple math expressions that include addition, subtraction and numeric literals.

F#


type MathExpr =
  | Addition of MathExpr * MathExpr
  | Subtraction of MathExpr * MathExpr
  | Literal of double

The following code snippet shows how to create instance of these math expression elements from IronPython.

Python


expr = MathExpr.Addition(MathExpr.Literal(10),\
                         MathExpr.Subtraction(\
                                   MathExpr.Literal(40), \
                                   MathExpr.Literal(24)))

A instance of the discriminated union has methods IsXXXX to verify the kind. Also properties a generated with the name of the constructor followed by a number (for example Addition1) to access values given to the constructor.

The following example shows how to evaluate a math expression.

Python


def evalExpr(expr):
  if expr.IsAddition():
     return evalExpr(expr.Addition1) + evalExpr(expr.Addition2)
  elif expr.IsSubtraction():
     return evalExpr(expr.Subtraction1) - evalExpr(expr.Subtraction2)
  else:
     return expr.Literal1

Functions

Functions passed as parameters need to be converted to a special F# element that represent them. In order to do this the FuncConvert.ToFastFunc function is used.

For example the following code shows how to call Seq.iter to print all the elements of a Python list.


from Microsoft.FSharp.Core import FuncConvert 
...
aList = [5,3,42,6]

def foo(x):
   print(x)

Seq.iter[int](FuncConvert.ToFastFunc[int]( foo),aList)

Since IronPython and F# use IEnumerable<T> types to represent collections and sequences we can use F# functions to manipulate IronPython collections.

For example given the following definition of a Python generator for Fibonacci numbers:


def fibo():
  last1 = 0
  last2 = 1
  while True:    
    yield last1
    next = last1+last2
    last2 = last1
    last1 = next

We can call F# functions to process values generated by fibo. For example the following IronPython code prints the first 15 squared Fibonacci numbers.


for n in Seq.map[int,int]( FuncConvert.ToFastFunc[int,int](lambda x: x*x)  ,\
                           Seq.take[int](15,fibo())):
   print n

Code for this post was created using IronPython 2.0 and F# 1.9.6.2 .

A quick look at MEF with F#

2008-12-01T05:39:00.001-06:00

In this post I'm going to show a little example of using the Managed Extensibility Framework (MEF) from F#.

The Managed Extensibility Framework (MEF) is a very interesting framework for building extensible applications. It is planned to be used in products such as Visual Studio 2010.

Episode .NET Rocks #398: Glenn Block on MEF is a nice source of information the project.

The MEF Codeplex page has documentation on how to start using it. Also the Addicted to MEF series is a nice source of information on how to start using it.

The example

For this example I want to define a little program F# program that plots math functions. Function definitions will be represented as classes that implement the IFunction interface. These definitions will be taken from DLLs stored in a "extensions" directory.

Here's the definition of the IFunction interface:


type IFunction = interface
    abstract Apply : double -> double
end

The following code shows the definition of the Winforms Form that is used to display the function:


open System
open System.Windows.Forms
open System.Drawing
open FunctionDefinitions
open System.ComponentModel.Composition
open System.Collections.Generic

type MyForm() as this = class
   inherit System.Windows.Forms.Form()
              
   let mutable functions : IEnumerable<IFunction> = Seq.empty
                         
   do this.Size <- new System.Drawing.Size(300,300)
   do this.BackColor <- Color.White
   do this.Text <- "Functions"
   do this.Paint.Add(
          fun (pargs:PaintEventArgs) ->
                      using(new Pen(Brushes.Black))
                           (fun(p) -> pargs.Graphics.DrawLine(p,
                                                         new Point(this.Size.Width/2,0),
                                                         new Point(this.Size.Width/2,this.Size.Height))

                                      pargs.Graphics.DrawLine(p,
                                                         new Point(0,this.Size.Height/2),
                                                         new Point(this.Size.Width,this.Size.Height/2))
                                      Seq.iter (fun func -> pargs.Graphics.DrawLines(p,this.CalcPoints(func))) functions
                           ))

   member this.CalcPoints(func:IFunction) : Point array =
       { 0..300} |> Seq.map (fun x -> -1.0/30.0 * double(x) + 5.0) |>
                    Seq.map (fun x -> x,func.Apply(x)) |>
                    Seq.map (fun(x,y) -> new Point(int32(x*30.0+150.0),
                                                   int32(y*(-30.0)+150.0)))  |>
                    Seq.to_array

   [<Import>]
   member this.Functions 
       with get() : IEnumerable<IFunction> = functions
       and  set (f : IEnumerable<IFunction>) = functions <- f

end

In order to say that we want all available functions(IFunction) identified by MEF we declare the Functions property with type IEnumerable<IFunction> and decorated with the Import attribute.

The following code shows the how we use MEF to compose the application and run the winforms app:


let f = new MyForm()
let catalog = new DirectoryPartCatalog("c:\\temp\\Extensions")
let container = new CompositionContainer(catalog,[||])

container.AddPart(f)
container.Compose()

System.Windows.Forms.Application.Run(f)

Notice that by using DirectoryPartCatalog we say that we want to take all extensions(math function definitions) stored in c:\temp\Extensions.

It is important to notice that the program that has the Winforms application is stored in a different assembly from where IFunction is defined. The reason is that the extension DLLs need a reference to it.

Running this program without extensions shows the following form:

The following code shows an extension written in F#:


#light 

namespace MoreFunctions
open FunctionDefinitions
open System.ComponentModel.Composition

[<Export(typeof<IFunction>)>]
type AFunction() = class
    interface IFunction with
       member this.Apply(x:double) = System.Math.Sin(x) 
end

We compile this file and copy the result to the extensions directory.


fsc.exe other.fs -r FunctionDefinitions.dll -r System.ComponentModel.Composition.dll  -a
copy other.dll c:\temp\Extensions

By running the program again the following form is shown

Since we're working with common .NET assemblies we can write a function definition using C#, for example:



using FunctionDefinitions;
using System.ComponentModel.Composition;
namespace Test2 {
   [Export(typeof(IFunction))]
   public class AFunction : IFunction
   {
    public double Apply(double d) 
    {
     return d*d;
    }
   }
}

We compile this file and copy it to the extensions directory:


csc /reference:FunctionDefinitions.dll;System.ComponentModel.Composition.dll  /t:library AFunc.cs
copy AFunc.dll c:\temp\Extensions

Running this program shows the following form:

Final words

MEF provides a nice/simple way to add extensions to .NET applications. I was really impressed by how simple it is to define Exports and Imports and how little MEF-specific code you have to write in order to make it work.

One thing that will be interesting to see is how this framework is going to interact with things such as the DLR. A nice example of this is Intellipad which, as mentioned in the .NET Rocks interview, is used to allow IronPython extensions.

Code for this post uses MEF Preview 3 release.

Using the Canvas Html element

2008-11-21T05:27:00.001-06:00

In this post I'm going to show a little example of using the HTML 5 Canvas element.

The Canvas element provides a way to draw graphics using Javascript. It is currently supported in browsers such as Safari, Firefox, Opera and Chrome. Sadly it is not supported in IE.

There's a lot of documentation on how to use this element. The Mozilla Developer Center provides a nice tutorial: Canvas tutorial. Also the HTML 5 specification document provides a detailed description of the element in The canvas element section.

The Example

In order to play learn about this element I decided to create a toy program that takes an image and creates a little Jigsaw Puzzle of it.

Here's how the page looks:

The running version of this program can be found here.

The Code

The code to create this little program is very simple. First the canvas element is specified as follows:


...
<body onload="init();" >
<p>Canvas Object Tests</p>
<canvas id="myCanvas" width="700" height="600"
        onmousemove="moveHandler(event);"
        onmousedown="mouseDownHandler(event);"
        onmouseup="mouseUpHandler(event);"
        onmouseout="mouseUpHandler(event);"></canvas>
<hr/>
<div id="notSup" ></div>

</body>
...

Four event handlers are added to the canvas element to handle mouse interactions to move the pieces of the puzzle.

The init function is called in the "load" event of the body element.


var imageName = 'http://farm3.static.flickr.com/2161/2543461722_a68f3d1e2e_m.jpg';
...
function init() {
   img = new Image();
        
   img.onload = function() {
      imageLoaded = true;
      draw();
   }
   img.src = imageName;
       
   theObjects = createImageSegments(3,3,{width:240,height:180},{x:5,y:10});
        
   draw();
}

The createImageSegments function creates the pieces of the puzzle.


function createImageSegments(lines,cols,imageSize,basePosition)
{
    var incrY = imageSize.height/lines;
    var incrX = imageSize.width/cols;
    var theX = basePosition.x;
    var theY = basePosition.y;
    var result = new Array();
    for(var i = 0;i < lines;i++) {
        for(var j = 0;j < cols;j++) {
            result.push(new ImageSegment(this.img,
                              {x:theX,y:theY},
                              {width:incrX,height:incrY},
                              {x:(j*incrX),y:(i*incrY)},
                              {width:incrX,height:incrY}));                
            theY += 2 + incrY;
        }
    }
    return shuffleArray(result);
}

The ImageSegment objects are defined by the following function:


function ImageSegment(image,position,size,insidePoint,insideSize){
   this.image = image;
   this.position = position;
   this.size = size;
   this.insidePoint = insidePoint;
   this.insideSize = insideSize;

   this.draw = function(context) {
      context.drawImage(
               this.image,
               this.insidePoint.x,
               this.insidePoint.y,
               this.insideSize.width,
               this.insideSize.height,
               this.position.x,
               this.position.y,
               this.size.width,this.size.height);           
   };

   this.isInside = function(point) {
      return (this.position.x <= point.x &&
          this.position.y <= point.y &&
          this.position.x + this.size.width >= point.x &&
          this.position.y + this.size.height >= point.y);
   }
}

Two methods are defined: one to draw the section of the image and one to determine of a given point hits the current object.

In order to draw the image the following function is defined:


function draw() {
   var myCanvas = document.getElementById("myCanvas");
   if(myCanvas.getContext) {
                     
     var ctxt = myCanvas.getContext('2d');
     ctxt.clearRect(0,0,myCanvas.width,myCanvas.height);
     if (imageLoaded) {
        for(var i = 0;i < theObjects.length;i++) {
           theObjects[i].draw(ctxt);
        } 
     }

    // Draw the frame
    ...

In order to allow the user to drag the pieces of the puzzle across the canvas the following mouse event handlers were added to the canvas.


function mouseDownHandler(e)
{
   var x = e.pageX - e.target.offsetLeft;
   var y = e.pageY - e.target.offsetTop;
   for(var i = 0; i < theObjects.length;i++) {
      var theObject = theObjects[i];
      if (theObject.isInside({x:x,y:y})) {
         dragging = true;
         lastPoint.x = x;
         lastPoint.y = y;
         currentDragObject = theObject;
      }
   }
}

function mouseUpHandler(e)
{
   dragging = false;
   lastPoint.x = -1;
   lastPoint.y = -1;
}

function moveHandler(e) {
   if (dragging) {
       var x = e.pageX - e.target.offsetLeft;
       var y = e.pageY - e.target.offsetTop;
       var deltaX = x - lastPoint.x;
       var deltaY = y - lastPoint.y;

       
       currentDragObject.position.x += deltaX;
       currentDragObject.position.y += deltaY;
 
       
       lastPoint.x = x;
       lastPoint.y = y;
       draw();
   }
}

Basically what these event handlers do is to calculate the new position of the image being dragged and update the canvas.

Conclusion

The Html Canvas object provides a nice way to draw custom graphics in a web page using Javascript. For future posts I'm going to try to rewrite the example from the
JavaFX Script, Silverlight and Flex posts using Javascript and Canvas.

Exploring Beautiful Languages

Creating a small parser using Parsec

A call-with-current-continuation example

A quick look at APL

APL

Polynomial multiplication

The program

Polynomial representation

Outer product

Creating a matrix of intermediate polynomials

Index origin

Defining a function

IronPython & WPF: Data binding with TreeView's selected element

Attached property definition

The example

IronPython & Silverlight Part VI: Using the TreeView control

Referencing System.Windows.Controls.dll

Using the TreeView in XAML

TreeView data binding

IronPython & Silverlight Part V: Using the MVVM Light Toolkit

MVVM Light

Adding a reference to MVVM Light

Adding the reference using the manifest

Using it in Python

Example

IronPython & Silverlight Part IV: Using Silverlight 4

IronPython & Silverlight Part III: Basic Data Binding

IronPython & Silverlight Part II: Basic event handling

Directly in Python code

Using clrtype

Final words

Using IronPython with Silverlight, Part I

Getting started

Writing Python's groupby in C#

Using C#'s implicit type conversions from other .NET languages

Example

IronPython

IronRuby

F#

VB.NET

Extracting elements from Win32 resource files with F#

Resource file format

Extracting Icons

Extracting Cursors

Extracting Bitmaps

Generating XML with Newspeak

Parsing XML documents with namespaces in Newspeak

The grammar

XML nodes

XML Namespaces

Extending the parsing context

Using the extended context

Code organization

Final words

A simple XML Grammar in Newspeak

Some basic image processing operations with F#

Converting image to gray scale

Applying templates

Final words

Using a Webcam with DirectShowNET and F#

DirectShow

DirectShowNET

Accessing the webcam

Using DirectShowNet

Using a Webcam with WIA and F#

Adding automatic semicolon insertion to a Javascript parser

Automatic semicolon insertion

Solution

A quick look at J

J

The example

Polynomial multiplication

1. Defining polynomials

2. Multiply each element of one of the polynomials

Final words

AS3 Getter/Setter support in AbcExplorationLib

Parsing Javascript using Newspeak parsing combinators

Creating Dynamic JSON array finders using the DLR

JSON.NET

Dynamic queries on JSON data

Using `clrtype`

The `FSDynJObjectWrapper` class