Sunday, May 17, 2009

Modifying an AS3 class with AbcExplorationLib

Recently, I made a couple of changes to AbcExplorationLib to allow the modification of a compiled ActionScript 3 (AS3) class.

The following code will be used to illustrate this feature:


class Shape {
public function foo() {


print("Base foo");
}
public function paint():void {
foo();
}
}

class Rectangle extends Shape{
public override function paint():void {
super.paint();
print( "Rectangle");
}
}

class Circle extends Shape{
public override function paint():void {
super.paint();
print( "Circle");
}
}

var shapes = [new Rectangle(),new Circle()];

for each(var s:Shape in shapes) {
s.paint();
}



Compiling this program using the Flex SDK and running it using the Tamarin binaries shows the following output:


$ java -jar asc.jar -import builtin.abc Shapes.as

Shapes.abc, 678 bytes written
$ avmshell Shapes.abc
Base foo
Rectangle
Base foo
Circle


Say that we want to create a definition of the foo method in the Rectangle class that overrides the definition from Shape.

1. First we load the class file:


let abcFile = using (new FileStream(sourceFile,FileMode.Open)) (
fun stream -> AvmAbcFile.Create(stream))


2. Then we need a definition for the new foo implementation. The following function creates a foo override that prints a message to the screen:


let newFooMethod(message:string) =
AvmMemberMethod(
CQualifiedName(Ns("",NamespaceKind.PackageNamespace),"foo"),
AvmMethod(
"",
SQualifiedName("*"),
[||],
Some <|
AvmMethodBody(
2,1,4,5,
[|
GetLocal0;
PushScope;
FindPropertyStrict(
MQualifiedName(
[|Ns("",
NamespaceKind.PackageNamespace)|],
"print"));
PushString message;
CallProperty(
MQualifiedName(
[|Ns("",
NamespaceKind.PackageNamespace)|],
"print"),
1);
Pop;
ReturnVoid
|],[||],[||]
)
),
AbcTraitAttribute.Override
)


3. We need a function to add a method to a existing class. Notice that the modification consists in only creating a new instance of the AvmClass with the same values as the original but adding the new method.


let addClassMethod(aClass,newMethod) =
match aClass with
| AvmClass(name,
superclassname,
init,
cinit,
slots,
methods,
pns) ->
AvmClass(name,
superclassname,
init,cinit,
slots,
newMethod::methods,
pns)



4. The following method is used to locate the rectangle class and apply the modification:


let modifyFileToAddMethod(f:AvmAbcFile) =
AvmAbcFile(f.Scripts,
f.Classes |>
List.map (fun (c:AvmClass) ->

match c.Name with
| CQualifiedName(_,"Rectangle") ->
addClassMethod(
c,
newFooMethod("New foo for Rectange!!!"))
| _ -> c))



5. Finally we write the input file back to disk:


let modifiedAbcFile = modifyFileToAddMethod(abcFile)
let abcFileCreator = AbcFileCreator()
let file = modifiedAbcFile.ToLowerIr(abcFileCreator)
using (new BinaryWriter(new FileStream(targetFileName,FileMode.Create)))
(fun f -> file.WriteTo(f))


After running this program we can execute the bytecode again to get the new results:


$ mono modify.exe Shapes.abc
...
$ avmshell Shapes_t.abc
New foo for Rectange!!!
Rectangle
Base foo
Circle


One area that really needs works is name handling. A particular challenge is to find a good way to represent multinames ( name references in a set of name spaces ).

In general the way of defining a method from scratch(for example in newFooMethod) needs some work since it is requires lots of details that might not be interesting for the developer.

Finally, another area that really needs improvement is the output file generation. Right now it requires the user to write three instructions to write the file to disc. This will be changed to be similar to the load process.


Code for this program can be found as part of the AbcExplorationLib samples.

Tuesday, April 14, 2009

Writing a small Twitter client with Newspeak and Hopscotch

In this post I'm going to show a small Twitter client written using the Newspeak programming language and the Hopscotch framework.

As part of the process of exploring the Newspeak language here I'm going to focus on the Hopscotch UI framework. It will be used to present information from the Twitter REST API which is very easy to use.

The Hopscotch framework and IDE is described in the "Hopscotch: Towards User Interface Composition" paper by Vassili Bykov.

Code for this program was created using the Newspeak prototype from 2009-02-27.

The program



Here's a screenshot of the program:

screenshot fo the twitter client

The program is not a complete client, it just allows to post a new twit and to read the current time line of a the user.

The code from the previous post "Parsing JSON with Newspeak" is used to access the data provided by the Twitter services.


The code



The following screenshot shows the definition of the TwitterGUI class



The following nested classes provide the functionality for the client:

  1. TwitterClient: A class for using the Twitter REST API
  2. TwitPresenter,TwitSubject: The UI piece and information for a single twit
  3. TwitterMainPresenter,TwitterMainSubject: The UI piece and information for a the complete client


As described in the Hopscotch paper a pair of elements is required to create a UI piece. A subject which contains the information being presented and the presenter which defines the UI that shows it.

In the case of a single twit the data is transmitted from the service as a JSON document.

The following code shows an example of the JSON response of a single twit from the Twitter REST API:


[{"in_reply_to_screen_name":null,
"user":{
"description":"Father, husband, friend, developer and late night programming language enthusiast.",
"statuses_count":318,
"utc_offset":-21600,
"profile_background_tile":false,
"profile_background_color":"6E8182",
"following":null,
"profile_text_color":"000000",
"url":"http:\/\/langexplr.blogspot.com",
"name":"Luis Diego Fallas",
"protected":false,
"profile_image_url":"http:\/\/s3.amazonaws.com\/twitter_production\/profile_images\/133285952\/ldnach_normal.png",
"notifications":null,
"profile_link_color":"0000ff",
"profile_background_image_url":"http:\/\/static.twitter.com\/images\/themes\/theme1\/bg.gif",
"created_at":"Mon Jul 28 13:51:55 +0000 2008",
"screen_name":"ldfallas",
"profile_sidebar_fill_color":"e0ff92",
"followers_count":45,
"time_zone":"Central America",
"location":"Costa Rica",
"id":15631932,
"favourites_count":9,
"friends_count":43,
"profile_sidebar_border_color":"87bc44"},
"text":"Experimenting with Hopscotch in Newspeak",
"truncated":false,
"in_reply_to_status_id":null,
"created_at":"Tue Apr 07 14:21:57 +0000 2009",
"in_reply_to_user_id":null,
"id":1469769323,
"favorited":false,
"source":"web"}
...]


A TwitSubject represents one of these pieces of information.


class TwitSubject withTwit: twit images: images= Subject (
"Describe the class in this comment."
|
theTwit = twit.
theImages = images.
|
)
(
createPresenter = (
^TwitPresenter new subject: self.
)
)


The definition of the TwitPresenter looks like this:


class TwitPresenter = Presenter (
"Presenter for a single twit."
|
|
)
(
definition = (
^(padded:( column: {

(row: { link: (subject theTwit user screen_name) action: [] . }) color: ( Color veryVeryLightGray).
row: { image: (subject theImages at: (subject theTwit user profile_image_url)) .
blank: 4.
elastic: twitBody}.

}) with: {3. 3. 2. 2.}) .
)
...
)


The following screenshot shows an example of the previous definition.

Single twit

A special treatment need to be applied to the message since we want to be able to click on links or twitter user id's (not supported right now). The definition of twitBody shows this.


twitBody = (
| text result |
text: subject theTwit text.
((string: text contains: '@') or: [string: text contains: 'http'] )
ifTrue: [ result:: flow: ((text subStrings: {Character space}) collect: [:t | componentFor: t])]
ifFalse: [ result:: textDisplay: text ].
^result
)

componentFor: s = (
|result|
(s includesSubString: '@')
ifTrue: [ result:: link: s action: [] ]
ifFalse: [
(string: s contains: 'http')
ifTrue: [ result:: link: s action: [openLink: s] ]
ifFalse:[ result:: label: s]
].
^result.
)

openLink:url = (
OSProcess command: (browser , ' ' , url).
)


What these methods do is to break the string of the message into words separated by spaces. If a word is an URL or and '@' character a link is created if not a 'label' is created . For the future this code needs to be improved with a better technique to identify urls.

The following code shows the definition of TwitterMainSubject:


class TwitterMainSubject user: userName password: pswd = Subject (
"Main subject."
|

user = userName.
password = pswd.
data
images = Dictionary new.
twitterClient = TwitterClient user: userName password:pswd.

|
)
createPresenter ^ = (
data:: twitterClient getFriendsTimeline.
^TwitterMainPresenter new subject: self
)

twits = (
data:: twitterClient getFriendsTimeline.
^data collect: [:t | TwitSubject withTwit: t images: imagesDictionary].
)

updateStatus: statusText = (
twitterClient updateStatus: statusText.
)
...


This class receives the used and password of the Twitter account. The TwitterClient class provides the access to the Twitter services.

Here's the definition of the TwitterMainPresenter class.


class TwitterMainPresenter = Presenter (
"Presenter for the main section of the GUI client."
|
editor
twitsHolder
charCountHolder

|
)
(

definition ^ <Fragment> = (
^column: {
row: { label: 'What are you doing?'.} .
row: { elastic:twitEditor.} .
row: { getCharCountHolder.
filler.
button: 'Update' action:[updateStatus: (editor editedText asString)].
blank: 5.
button: 'Refresh' action:[twitsHolder refresh].
blank: 5.
}.
row: {
blank: 5.
elastic:: getTwitsHolder
}}
))


The getTwitsHolder method create an instance of an HolderComposer object that allows the content to be recalculated using the refresh method. Here's the definition:


getTwitsHolder = (
twitsHolder:: holder: [ list:: subject twits collect: [ :i | i presenter ] ].
^twitsHolder.
)


Notice also that here we are requesting the list of twits to the subject, which calls the web service again getting new content.

Final words



The experience of using the Newspeak and the Hopscotch framework to create this program was very nice.

One thing that I need to find out is how to prevent the application from blocking when requesting the data from the services.

Code for this post can be found here.

Thursday, April 2, 2009

Parsing JSON with Newspeak

In this post I'm going to show a JSON parser written using the Newspeak parser combinator library.

Newspeak



Newspeak is a new programmming language. From its webpage http://newspeaklanguage.org/:


Newspeak is a new programming language in the tradition of Self and Smalltalk. Newspeak is highly dynamic and reflective - but designed to support modularity and security. It supports both object-oriented and functional programming.


In The Newspeak Programming Platform the authors give a nice introduction to the language and platform.

The first time I heard about Newspeak was by watching the Lang.NET 2008 symposium presentation video by Gilad Braha (available here). In this video, a nice parser combinator library is presented . This parser combinator library is described in the Executable Grammars in Newspeak paper by Gilad Bracha.

Code in this post was written using the Newspeak prototype released February 27 2009.

JSON



In order to learn about the language and platform I decided to create a little parser for JSON (Javascript Object Notation).

JSON is a simple data-interchange format defined in http://www.json.org/ .An example of it:


[{ "name": "Wiston Smith",
"description" :"Protagonist"},
{ "name": "Julia",
"description" :"Lover"},
{ "name": "O Brien",
"description" :"Goverment agent"}]


Parser structure



The parser is defined as a single class with a couple of nested classes. The following image shows the definition of JSON Parser in the Newspeak environment.

definition of the JSONParser class

As described in the "Modularity" section of the The Newspeak Programming Platform paper, top-level classes (in this case JSONParser) doesn't have access to its surrounding scope, it only has access to its own or inherited definitions. This is the reason why the JSONParser has the following definitions:


class JSONParser withParserLib: parserLibrary usingLib: platform = (
"Experiment for JSON parser based on the description from http://www.json.org/fatfree.html "
|
ExecutableGrammar = parserLibrary ExecutableGrammar.
CharParser = parserLibrary CharParser.
PredicateTokenParser = parserLibrary PredicateTokenParser.
Dictionary = platform Dictionary.
OrderedCollection = platform OrderedCollection.
Number = platform Number.
|
)
...


The "withParserLib: parserLibrary usingLib: platform" part defines parameters for the construction of JSONParser. These parameters are used to 'import' classes defined elsewhere.

The following code shows a way to create an instance of the JSONParser class:

|platform parser|
platform:: Platform new.
parser = (JSONParser withParserLib: (BlocklessCombinatorialParsing usingLib: platform) usingLib: platform).
...


The JSONParser nested classes are the following:

  1. CharExceptForParser which is a parser that accepts any character except for the one specified (this is for internal use)
  2. JSONGrammar The definition of the JSON grammar
  3. JSONGrammarWithAST which defines the way the AST is created
  4. JSONObject which is used in the representation of the JSON AST



Grammar



The following code shows the JSON grammar defined using the parsing combinators:


class JSONGrammar = ExecutableGrammar (
"Experiment for JSON grammar based on the description from http://www.json.org/fatfree.html "
|
doubleQuote = (char: $").
backslash = (char: $\).
str = doubleQuote,((backslash, ( char: $" )) |
(backslash, ( char: $/ )) |
(backslash, backslash) |
(backslash, ( char: $r )) |
(backslash, ( char: $n )) |
(backslash, ( char: $t )) |
(charExceptFor: $")) star, doubleQuote.
string = tokenFor: str.

negSign = (char: $-).
plusSign = (char: $+).
digit = (charBetween: $0 and: $9).
dot = (char: $. ) .
num = negSign opt, digit, digit star, dot opt,digit star, ((char: $e) | (char: $E)) opt, (plusSign | negSign) opt,digit star.
number = tokenFor: num.

leftbrace = tokenFromChar: ${.
rightbrace =tokenFromChar: $}.
colon = tokenFromChar: $:.
comma = tokenFromChar: $,.
definition = string,colon,value.
obj = leftbrace, (definition starSeparatedBy: comma),rightbrace.
object = tokenFor: obj.

leftbracket = tokenFromChar: $[.
rightbracket = tokenFromChar: $].
arr = leftbracket, (value starSeparatedBy: comma), rightbracket.
array = tokenFor: arr.

ttrue = tokenFromSymbol: #true.
tfalse = tokenFromSymbol: #false.
null = tokenFromSymbol: #null.

value = string | number | object | array | ttrue | tfalse | null.

|
)
...




For more information on the how this library works, check the Executable Grammars in Newspeak paper.


AST construction



We need to define a way to represent the tree structure(AST) parsed by JSONGrammar. As described in the "Executable Grammars is Newspeak" paper, one of the nice things about Newspeak is that we don't have the modify the grammar definition to add AST construction code. We can do that by inheriting from the original grammar:


class JSONGrammarWithAST = JSONGrammar(
"Parses a JSON File and generates and Ast"
|

|
)
('as yet unclassified'
array = (
^super array wrapper: [:a | (a token at: 2) ].
)


null = (
^ super null wrapper: [:o | nil].
)

number = (
^super number wrapper: [:o | Number readFrom: (flattenCharCollectionToString: (o token)) ].
)

object = (
^super object wrapper:
[:obj | JSONObject withContent:
(Dictionary newFrom: ((obj token at: 2) collect: [:e | (e at: 1) -> (e at: 3)]))].
)

parse: input = (
^super value parse: input.
)

string = (
^super string wrapper:
[:t | flattenCollectedString: (t token at: 2)].
)

tfalse = (
^super tfalse wrapper: [:o | false].
)

ttrue = (
^super ttrue wrapper: [:o | true].
)

...
)


As shown here (omitting some method definitions) the arrays are converted to Ordered collections, the numbers,strings,booleans to its equivalents and JSON objects to instances of JSONObject(described below).

JSONObject



In order to make it easy to use a JSON object in Newspeak the JSONParser class was defined:


class JSONObject withContent: dContent = (
"Instances of this class represent JSON objects."
|
content = dContent.
|
)
('as yet unclassified'
doesNotUnderstand: message = (
| fieldName |
fieldName:: message selector string.
(fieldName beginsWith: 'json_')
ifTrue: [fieldName:: fieldName allButFirst: 5].
^content at: fieldName ifAbsent: [nil].
)

)


This class receives a Dicionary as parameter. This dictionary contains all the name/value pairs of the JSON object definition. We create a definition of the doesNotUnderstand method which as in Smalltalk is called when a message sent to an object doesn't have a explicit way to respond it. We take the name of the message being called and check it against the dictionary.

If the message is prefixed by 'json_', the string after it is used as the key in the dictionary. This is defined this way because JSONObject has definitions inherited from Object (such as 'name').


...
| parsed |
parsed:: parserWithAST
parse: (streamFromString: '[{ "name": "Wiston Smith",
"description" :"Protagonist"},
{ "name": "Julia",
"description" :"Lover"},
{ "name": "O Brien",
"description" :"Goverment agent"}]'
).
assert:[((parsed at: 2) description) = 'Lover'].
assert:[((parsed at: 3) json_name) = 'O Brien'].




In the following post I'm going to use this parser to explore the GUI library provided with Newspeak.

Code for this post can be found here.

Sunday, March 15, 2009

Support for the LookupSwitch opcode in AbcExplorationLib

The LookupSwitch AVM2 opcode is used to represent the ActionScript switch statement.

This is an interesting branch instruction because it has multiple targets. It is almost a direct translation of the switch statement because it has two parameters, a default case and an array of possible targets.

Recently support for this opcode was added to AbcExplorationLib.

Given the following ActionScript code:


var x;
for(x = 1;x < 5;x++) {
switch(x) {
case 1:
print("one");
break;
case 2:
print("two");
break;
case 3:
print("three");
break;
case 4:
print("four");
break;
}
}


We compile this file to a .abc file using the following command:


$ java -jar /opt/flex3sdk/lib/asc.jar testswitch.as

testswitch.abc, 280 bytes written


By using a little IronPython example included with AbcExplorationLib we can see how this library interprets the LookupSwitch opcode:


$ mono /opt/IronPython-2.0/ipy.exe ../ipyexample/abccontents.py testswitch.abc
...
Instructions:
getlocal_0
pushscope
pushbyte 1
getglobalscope
swap
setslot 1
jump dest177
dest12:
label
jump dest74
dest17:
label
findpropertystrict M.print
pushstring "one"
callprop M.print
pop
jump dest165
dest30:
label
findpropertystrict M.print
pushstring "two"
callprop M.print
pop
jump dest165
dest43:
label
findpropertystrict M.print
pushstring "three"
callprop M.print
pop
jump dest165
dest56:
label
findpropertystrict M.print
pushstring "four"
callprop M.print
pop
jump dest165
dest69:
label
jump dest165
dest74:
getglobalscope
getslot 1
setlocal_1
pushbyte 1
getlocal_1
ifstrictneq dest91
pushshort 0
jump dest143
dest91:
pushbyte 2
getlocal_1
ifstrictneq dest104
pushshort 1
jump dest143
dest104:
pushbyte 3
getlocal_1
ifstrictneq dest117
pushshort 2
jump dest143
dest117:
pushbyte 4
getlocal_1
ifstrictneq dest130
pushshort 3
jump dest143
dest130:
pushfalse
iffalse SolvedReference dest141
pushshort 4
jump dest143
dest141:
pushshort 4
dest143:
kill
lookupswitch dest69 dest17,dest30,dest43,dest56,dest69
dest165:
getglobalscope
getslot 1
increment
setlocal_1
getlocal_1
getglobalscope
swap
setslot 1
kill
dest177:
getglobalscope
getslot 1
pushbyte 5
iflt dest12
returnvoid


As described in the documentation the parameters of the LookupSwitch instruction are specified as relative byte offsets that specify the target. In order to give a higher level representation of the code, these relative offsets are converted to symbolic references. This process is detailed in the "Using F# Active Patterns to encapsulate complex conditions" post.

This process starts in the following functions:


static member ReadAndProcessInstructions(aInput:BinaryReader,
count,
constantPool:ConstantPoolInfo) =
let instructionsAndOffsets =
(AvmMethodBody.ReadingInstructions([],
aInput,
count,
constantPool)) in
let destinations =
AvmMethodBody.CollectDestinations(instructionsAndOffsets,
Map.empty,
instructionsAndOffsets)
in
AvmMethodBody.UpdateCodeWithDestinations(
destinations,
instructionsAndOffsets,[]) |> List.to_array


The CollectDestinations method collects all the absolute offsets used by branch instructions and stores them in a dictionary with a generated label. The UpdateCodeWithDestinations method modifies the instruction list use the generated labels.

For example to add support in CollectDestinations the following code was added:


static member CheckLookupSwitchCase baseOffset
(totalInstructions:(int64*AbcFileInstruction)
list) =
fun (destinations:Map<int64,string>) target ->
match target with
| UnSolvedReference(relativeOffset) when
(AvmMethodBody.IsDestinationDefined(int(baseOffset+relativeOffset),
totalInstructions)) ->
destinations.Add(int64(baseOffset+relativeOffset),
sprintf "dest%d" (baseOffset+relativeOffset))
| _ -> destinations

static member CollectDestinations(instructions:(int64*AbcFileInstruction) list,
destinations:Map<int64,string>,
totalInstructions:(int64*AbcFileInstruction) list) =
match instructions with
...
| ((offset,(LookupSwitch(defaultBranch,cases)))::rest) ->
let baseOffset = int(offset) in
AvmMethodBody.CollectDestinations(rest,
Seq.append [defaultBranch] cases |>
Seq.fold (AvmMethodBody.CheckLookupSwitchCase baseOffset totalInstructions ) destinations,
totalInstructions)
...


Then the code is modified in the UpdateCodeWithDestinations method to use the generated labels.


static member UpdateCodeWithDestinations(destinations:Map<int64,string>,
instructions,
resultingInstructions) =
let processedInstructions =
match instructions with
...
| ((offset,LookupSwitch(defaultCase,cases))::rest) ->
(offset,
LookupSwitch(AvmMethodBody.SolveSwitchCase defaultCase offset destinations,
Array.map (fun c->AvmMethodBody.SolveSwitchCase c offset destinations) cases))::rest
| _ -> instructions
...

Tuesday, March 3, 2009

Manipulating AVM2 byte code with F#

In this post I'm going to show an example of using AbcExplorationLib to manipulate simple AVM2 byte code (ActionScript). This example show how load a .ABC file and write it back to disk.

AbcExplorationLib is a library that will allow the manipulation of AVM2 Byte Code(described here). Although it's still incomplete, some basic examples work as the ones presented in this post .

The following ActionScript code will be compiled to byte code .


var i = 0;
for( i = 0;i < 10;i++) {
print("inside loop");
}
print("Done");


To generate the ".abc" file we type:

c:\test\> java -jar c:\flexsdk\lib\asc.jar test.as



Loading the compiled file



We're going to use the F# REPL(fsi.exe) to manipulate the file. We start by referencing the library.


> #r "abcexplorationlib.dll";;

--> Referenced 'C:\test\abcexplorationlib.dll'

> open Langexplr.Abc;;

Now we load the file:


> let abcFile = using (new System.IO.FileStream("test.abc",System.IO.FileMode.Open)) (
fun s -> AvmAbcFile.Create(s));;


Now abcFile contains the code of the compiled program.


> abcFile;;
val it : AvmAbcFile
= Langexplr.Abc.AvmAbcFile {Classes = [];
Scripts = [Langexplr.Abc.AvmScript];}




Inspecting the instructions



We're interested in the instructions of the top-level script for this .abc file. By typing the following expression we can get to this section:


> abcFile.Scripts.[0].InitMethod.Body.Value.Instructions;;
val it : AbcFileInstruction array
= [|GetLocal0; PushScope; PushByte 0uy; GetGlobalScope; Swap; SetSlot 1;
PushByte 0uy; GetGlobalScope; Swap; SetSlot 1;
Jump (SolvedReference "dest39"); ArtificialCodeBranchLabel "dest18"; Label
FindPropertyStrict
(MQualifiedName
([|Ns ("",CONSTANT_Namespace); Ns ("test.as$0",CONSTANT_PrivateNs)|],
"print")); PushString "inside loop";
CallProperty
(MQualifiedName
([|Ns ("",CONSTANT_Namespace); Ns ("test.as$0",CONSTANT_PrivateNs)|],
"print"),1); Pop; GetGlobalScope; GetSlot 1; Increment; SetLocal_2;
GetLocal2; GetGlobalScope; Swap; SetSlot 1; Kill 2;
ArtificialCodeBranchLabel "dest39"; GetGlobalScope; GetSlot 1;
PushByte 10uy; IfLt (SolvedReference "dest18");
FindPropertyStrict
(MQualifiedName
([|Ns ("",CONSTANT_Namespace); Ns ("test.as$0",CONSTANT_PrivateNs)|],
"print")); PushString "Done";
CallProperty
(MQualifiedName
([|Ns ("",CONSTANT_Namespace); Ns ("test.as$0",CONSTANT_PrivateNs)|],
"print"),1); CoerceA; SetLocal_1; GetLocal1; ReturnValue; Kill 1|]



We're going to define the following function to assist in the presentation of instruction listings.


> open Langexplr.Abc.InstructionPatterns;;
> let pr (i:AbcFileInstruction) =
- match i with
- | ArtificialCodeBranchLabel t -> printf "%s:\n" <| t.ToString()
- | i & UnsolvedSingleBranchInstruction(d,_) -> printf " %s %d\n" i.Name d
- | i & SolvedSingleBranchInstruction(l,_) -> printf " %s %s\n" i.Name l
- | _ -> printf " %s\n" i.Name;;

val pr : AbcFileInstruction -> unit


Now we can type:


> abcFile.Scripts.[0].InitMethod.Body.Value.Instructions |> Array.iter pr;;
getlocal_0
pushscope
pushbyte
getglobalscope
swap
setslot
pushbyte
getglobalscope
swap
setslot
jump dest39
dest18:
label
findpropertystrict
pushstring
callprop
pop
getglobalscope
getslot
increment
setlocal_2
getlocal_2
getglobalscope
swap
setslot
kill
dest39:
getglobalscope
getslot
pushbyte
iflt dest18
findpropertystrict
pushstring
callprop
coerce_a
setlocal_1
getlocal_1
returnvalue
kill
val it : unit = ()




A note on branch instructions



In order to make it easy to manipulate and analyze the code AbcExplorationLib adds a non-existing instruction called ArtificialCodeBranchLabel to mark the position where a branch instruction will jump. When these labels are generated the branch instructions are modified to point to the label's name instead of a relative byte offset. Details on how this process is briefly described in "Using F# Active Patterns to encapsulate complex conditions"

Converting from label references to byte offsets is also necessary to write code back to an .abc file. This process is performed by a function called ConvertSymbolicLabelsToByteReferences, for example:


> let c = AbcFileCreator();;

val c : AbcFileCreator

> abcFile.Scripts.[0].InitMethod.Body.Value.Instructions |>
- InstructionManipulation.ConvertSymbolicLabelsToByteReferences c |>
- Array.iter pr;;
getlocal_0
pushscope
pushbyte
getglobalscope
swap
setslot
pushbyte
getglobalscope
swap
setslot
jump 21
dest18:
label
findpropertystrict
pushstring
callprop
pop
getglobalscope
getslot
increment
setlocal_2
getlocal_2
getglobalscope
swap
setslot
kill
dest39:
getglobalscope
getslot
pushbyte
iflt -30
findpropertystrict
pushstring
callprop
coerce_a
setlocal_1
getlocal_1
returnvalue
kill
val it : unit = ()
>



Modifying the code



Values for branch instruction targets are adjusted if new code added, for example, lets add some code to print "Hola!" inside the loop.


> let printName = CQualifiedName(Ns("",NamespaceKind.CONSTANT_Namespace),"print"
- ) ;;

val printName : QualifiedName

> let printCode = [| FindPropertyStrict printName ;
- PushString "Hola!" ;
- CallProperty(printName,1);
- Pop |] ;;

val printCode : AbcFileInstruction array

> Seq.append instr.[0..16] <| Seq.append printCode instr.[17..] |>
- Seq.to_array |>
- InstructionManipulation.ConvertSymbolicLabelsToByteReferences c |>
- Array.iter pr;;
getlocal_0
pushscope
pushbyte
getglobalscope
swap
setslot
pushbyte
getglobalscope
swap
setslot
jump 29
dest18:
label
findpropertystrict
pushstring
callprop
pop
findpropertystrict
pushstring
callprop
pop

getglobalscope
getslot
increment
setlocal_2
getlocal_2
getglobalscope
swap
setslot
kill
dest39:
getglobalscope
getslot
pushbyte
iflt -38
findpropertystrict
pushstring
callprop
coerce_a
setlocal_1
getlocal_1
returnvalue
kill
val it : unit = ()



Writing the new file



We can write this code back to a .abc file by doing this:


> let newCode = Seq.append instr.[0..16] <| Seq.append printCode instr.[17..] |
- > Seq.to_array;;

val newCode : AbcFileInstruction array

> let newBody = AvmMethodBody(oldbody.Method,
- oldbody.MaxStack,
- oldbody.LocalCount,
- oldbody.InitScopeDepth,
- oldbody.MaxScopeDepth,
- newCode,
- oldbody.Exceptions,
- oldbody.Traits);;

val newBody : AvmMethodBody

> let newFile = AvmAbcFile( [AvmScript( abcFile.Scripts.[0].InitMethod.CloneWithBody(newBody), abcFile.Scripts.[0].Members)], []);;

val newFile : AvmAbcFile
> open System.IO;;
> let c = AbcFileCreator();;

val c : AbcFileCreator

>
- using (new BinaryWriter(new FileStream("test_modified.abc",FileMode.Create)))
-
- (fun f -> let file = newFile.ToLowerIr(c) in file.WriteTo(f));;
val it : unit = ()


Running this program using Tamarin shows:


c:\test\>avmplus_sd.exe test_modified.abc
inside loop
Hola!
inside loop
Hola!
inside loop
Hola!
inside loop
Hola!
inside loop
Hola!
inside loop
Hola!
inside loop
Hola!
inside loop
Hola!
inside loop
Hola!
inside loop
Hola!
Done



The AbcExplorationLib library is still pretty incomplete. Also there's a lot to improve, for example name handling and instruction modification. Future posts will present new features/experiments.

Thursday, February 12, 2009

Writing Xml with IronPython, XmlWriter and the 'with' statement

This post shows a little example of wrapping calls to System.Xml.XmlWriter inside a Python's 'with' statement using IronPython.

Writing Xml



While reading some code examples from the XIST HTML/XML generation library I noticed the nice use of Python's 'with' statement to represent the target HTML or XML.

The System.Xml.XmlWriter class provided with .NET already gives you a way to write well formed Xml documents. In this post I'm going to show how to use an XmlWriter instance in conjunction with Python's 'with' statement.

We want to write the following code:


from __future__ import with_statement

...

w = XmlWriter.Create(System.Console.Out,XmlWriterSettings(Indent=True))
x = XWriter(w)

with x.element('tableofcontents'):
with x.element('section',{'page' : '10'}):
x.text('Introduction')
with x.element('section',{'page' : '12'}):
x.text('Main topic')
with x.element('section',{'page' : '14'}):
x.text('Extra topic')


To generate the following Xml file:


<tableofcontents>
<section page="10">Introduction</section>
<section page="12">Main topic</section>
<section page="14">Extra topic</section>
</tableofcontents>


The 'with' statement



The 'with' statement was introduced in Python 2.5 . This statement is used wrap the execution of a series of statements with some special code. For example it is used to implement the try...except...finally pattern.

As described in the documentation the following statement:


with context expression:
statements...


Will be executed as follows:


  1. Evaluate the context expression to obtain the context manager

  2. Invoke the context manager's __enter__() method

  3. Execute the statements

  4. When the execution of the statements finishes(even with an exception), the context manager's __exit()__ method is called.



Given these steps we're going to implement a context manager that assist in the creation of Xml documents using the System.Xml.XmlWriter .NET class.

The following code shows a class that wraps the XmlWriter instance and helps with the creation of context managers:


class XWriter(object):
def __init__(self,writer):
self.writer = writer

def element(self,name,atts = {}):
return ElementCtxt(name,atts,self)

def nselement(self,prefix,name,namespace,atts = {}):
return NamespaceElementCtxt(prefix,name,namespace,atts,self)


def text(self,text):
self.writer.WriteString(text)

def cdata(self,text):
self.writer.WriteCData(text)



Notice that the element method creates an instance of the ElementCtxt class using the element name and an optional dictionary with the attributes. As the following listing shows this class performs the calls to WriteStartElement and WriteEndElement in the __enter__ and __exit__ methods.


class ElementCtxt(object):
def __init__(self,elementName,atts,writer):
self.elementName = elementName
self.atts = atts
self.writer = writer

def processAttributes(self):
for att in self.atts:
self.writer.writer.WriteAttributeString(att,self.atts[att].__str__())

def processStartTag(self):
self.writer.writer.WriteStartElement(self.elementName)
self.processAttributes()

def __enter__(self):
self.processStartTag()
return self

def __exit__(self,t,v,tr):
self.writer.writer.WriteEndElement()
return t == None



The XWriter.nselement method is used to write elements with namespace and prefix. This call generates an instance of the following context manager:


class NamespaceElementCtxt(ElementCtxt):
def __init__(self,prefix,elementName,namespace,atts,writer):
ElementCtxt.__init__(self,elementName,atts,writer)
self.namespace = namespace
self.prefix = prefix
def processStartTag(self):
self.writer.writer.WriteStartElement(self.prefix,self.elementName,self.namespace)
self.processAttributes()


Final example



The following code shows how to create a little SVG file:


from __future__ import with_statement
from xmlwriterw import XWriter

import clr

clr.AddReference('System.Xml')

from System.Xml import *
import System


w = XmlWriter.Create(System.Console.Out,\
XmlWriterSettings(Indent=True))
x = XWriter(w)

svgNs = 'http://www.w3.org/2000/svg'

with x.nselement('s','svg',svgNs,{'version': '1.1',
'viewBox': '0 0 100 100',
'style':'width:100%; height:100%; position:absolute; top:0; left:0; z-index:-1;'}):
with x.nselement('s','linearGradient',svgNs, { 'id' : 'gradient' }):
with x.nselement('s','stop',svgNs, {'class' : 'begin',
'offset' : '0%',
'stop-color':'red'}):
pass
with x.nselement('s','stop',svgNs, {'class' : 'end',
'offset' : '100%'}):
pass
with x.nselement('s','rect',svgNs, { 'x':0,
'y':0,
'width':100,
'height':100,
'style':'fill:url(#gradient)'} ):
pass
for i in range(1,5):
with x.nselement('s','circle',svgNs, { 'cx': 50,
'cy': 50,
'r': 30 - i*3,
'style':'fill:url(#gradient)'} ):
pass

w.Close()


Running this program shows:


<s:svg viewBox="0 0 100 100" style="width:100%; height:100%; position:absolute; top:0; left:0; z-index:-1;" version="1.1" xmlns:s="http://www.w3.org/2000/svg">
<s:linearGradient id="gradient">
<s:stop offset="0%" class="begin" stop-color="red" />
<s:stop offset="100%" class="end" />
</s:linearGradient>
<s:rect x="0" height="100" width="100" style="fill:url(#gradient)" y="0" />
<s:circle cx="50" cy="50" style="fill:url(#gradient)" r="27" />
<s:circle cx="50" cy="50" style="fill:url(#gradient)" r="24" />
<s:circle cx="50" cy="50" style="fill:url(#gradient)" r="21" />
<s:circle cx="50" cy="50" style="fill:url(#gradient)" r="18" />
</s:svg>

Wednesday, January 21, 2009

Some notes on using F# code from IronPython

This post presents a couple of things I learned about the interaction between IronPython and F#.

Most information about this topic can be found by looking at the code generated by the F# compiler, using tools like ILDASM or Reflector. Also the F# from C# section of the F# 1.1.12 documentation is very helpful.

FSharp.Code library



In order to call some F# basic functionality we need to add a reference to FSharp.Core.

Python

import clr
clr.AddReference('FSharp.Core')



Type parameters



IronPython provides a way to specify type parameters to methods and classes by using square brackets followed by the names of the types to be used. This mechanism can be used to create F# constructs that require them.

The following example shows how to create an F# list:

Python

>>> import clr
>>> clr.AddReference('FSharp.Core')
>>> from Microsoft.FSharp.Collections import List
>>> l1 = List[str].Cons('World',List[str].Empty)
>>> l2 = List[str].Cons('Hello',l1)
>>> " ".join(l2)
'Hello World'



Type parameters, for methods that require them, also can be specified using the same syntax. For example the following code shows the use of Seq.length with an Python sequence.

Python

print Seq.length[int]([45,234,52,345,2,346,657])



Tuples



Tuples in F# are instances of the Microsoft.FSharp.Core.Tuple class (with several definitions with depending on the number of arguments). A tuple instance contains properties Item# where # is the position of the element.

The following example shows how to use a tuple created in F# from IronPython.

F#

module MyFunctions = begin
...
let numberEvenTest x = x,x % 2 = 0
...
end


Python

print MyFunctions.numberEvenTest(46)
print MyFunctions.numberEvenTest(46).Item1
print MyFunctions.numberEvenTest(46).Item2



This program will print:


(46,True)
46
True


To create a tuple inside IronPython the Tuple[type1,type2,...](value1,value2,...) constructor must be used.

For example to given the following definition:

F#

module MyFunctions = begin
...
let movePoints points xdistance ydistance =
Seq.map (fun (x,y) -> (x+xdistance,y+ydistance)) points
...
end


We can used from IronPython like this:
Python

from Microsoft.FSharp.Core import Tuple
...
points = [ Tuple[int,int](1,2), Tuple[int,int](5,3) ]

for p in MyFunctions.movePoints(points,20,40):
print p


It is tempting to use the Python syntax for assigning a list of variables to the values of the tuples. For example:


x,y,z = 2,3,5


However this is not possible since F# tuples are not compatible with IronPython sequences.

Operators



F# operators are created using the same conventions as the C# operators. Thus they can be used directly in IronPython.

For example the following F# definition:


type Complex(real:double,img:double) = class
member this.real with get() = real
member this.img with get() = img
static member (+) (c1:Complex,c2:Complex) =
Complex(c1.real+c2.real,c1.img+c2.img)
override this.ToString() = sprintf "%f + %fi" real img
end


Can be used in IronPython:

Python


c1 = Complex(20,234)
c2 = Complex(34.35,32.2)
c3 = (c1+c2)



Discriminated unions



Discriminated unions in F# provide a compact way to define data structures. The following example shows the definition of simple math expressions that include addition, subtraction and numeric literals.

F#

type MathExpr =
| Addition of MathExpr * MathExpr
| Subtraction of MathExpr * MathExpr
| Literal of double


The following code snippet shows how to create instance of these math expression elements from IronPython.

Python

expr = MathExpr.Addition(MathExpr.Literal(10),\
MathExpr.Subtraction(\
MathExpr.Literal(40), \
MathExpr.Literal(24)))


A instance of the discriminated union has methods IsXXXX to verify the kind. Also properties a generated with the name of the constructor followed by a number (for example Addition1) to access values given to the constructor.

The following example shows how to evaluate a math expression.

Python

def evalExpr(expr):
if expr.IsAddition():
return evalExpr(expr.Addition1) + evalExpr(expr.Addition2)
elif expr.IsSubtraction():
return evalExpr(expr.Subtraction1) - evalExpr(expr.Subtraction2)
else:
return expr.Literal1


Functions



Functions passed as parameters need to be converted to a special F# element that represent them. In order to do this the FuncConvert.ToFastFunc function is used.

For example the following code shows how to call Seq.iter to print all the elements of a Python list.


from Microsoft.FSharp.Core import FuncConvert
...
aList = [5,3,42,6]

def foo(x):
print(x)

Seq.iter[int](FuncConvert.ToFastFunc[int]( foo),aList)



Since IronPython and F# use IEnumerable<T> types to represent collections and sequences we can use F# functions to manipulate IronPython collections.

For example given the following definition of a Python generator for Fibonacci numbers:


def fibo():
last1 = 0
last2 = 1
while True:
yield last1
next = last1+last2
last2 = last1
last1 = next


We can call F# functions to process values generated by fibo. For example the following IronPython code prints the first 15 squared Fibonacci numbers.



for n in Seq.map[int,int]( FuncConvert.ToFastFunc[int,int](lambda x: x*x) ,\
Seq.take[int](15,fibo())):
print n




Code for this post was created using IronPython 2.0 and F# 1.9.6.2 .