Monday, August 3, 2009

Creating Dynamic JSON array finders using the DLR

One of the things that really impressed me while reading about Ruby on Rails was the use of method_missing to implement dynamic finders. The technique is described on the "How dynamic filters work". In this post I'm going to show a little experiment of creating a similar technique for querying JSON arrays using the .NET's Dynamic Language Runtime infrastructure.

Code for this post was created using Visual Studio 2010 Beta 1, IronRuby for .NET 4 beta 1 and IronPython 2.6 beta 4 for .NET 4.

JSON.NET



For this post I'm using the JSON.NET library for loading the JSON data. This library provides several ways of loading JSON data. Here I'll be using a set of predefined classes: JObject for JSON objects, JArray for arrays, JValue for literal values, etc. All these classes inherit from JToken.
Code in this post use the JSON data returned by the Twitter REST API. An example of this data:


[
{"in_reply_to_screen_name":null,
"text":"...",
"user": { "following":null,
"description":"...",
"screen_name":"...",
"utc_offset":0,
"followers_count":10,
"time_zone":"...",
"statuses_count":155,
"created_at":"...",
"friends_count":1,
"url":"...",
"name":"...",
"notifications":null,
"protected":false,
"verified":false,
"favourites_count":0,
"location":"...",
"id": ...,
...
},
"truncated":false,
"created_at":"...",
"in_reply_to_status_id":null,
"in_reply_to_user_id":null,
"favorited":false,
"id":...,
"source":"...."
},
...
]



Dynamic queries on JSON data



Finders will be implemented for JSON arrays. As with Rail's dynamic finders the names of the required fields will be encoded in the name of the invoked method.

The following C# 4.0 code shows an example of this wrapper class in conjunction with the dynamic keyword.


JsonTextReader reader = new JsonTextReader(rdr);
JsonSerializer serializer = new JsonSerializer();
JArray o = (JArray)serializer.Deserialize(reader);
dynamic dArray = new FSDynArrayWrapper(o);

string name = "ldfallas";
foreach (var aJObject in dArray.FindAllByFavoritedAlsoByUserWithScreen_Name("false",name))
{
dynamic tobj = new FSDynJObjectWrapper(aJObject);
Console.WriteLine("========");
Console.WriteLine(tobj.user.screen_name);
Console.Write("\t'{0}'",tobj.text);
}




A small definition of the syntax used for names is the following.


method-name = "FindAllBy" ,
property-name , ("With", property-name)? ,
("AlsoBy" property-name , ("With", property-name)? ) *
property-name = valid json property name


In order to be more flexible the following syntax will also be allowed:


method-name = "find_all_by_" ,
property-name , ("_with_", property-name)? ,
("_also_by" property-name , ("_with_", property-name)? ) *
property-name = valid json property name


A sample name for this query methods look like this:


array.FindAllByFavoritedAlsoByUserWithScreen_Name("true","ldfallas")


This method will accept two parameters and is going to :


Find all the object elements from the array that has a 'Favorited' property equal to 'true' and also has an object with a 'User' property associated with an object which has a 'Screen_Name' property which is equal to the 'ldfallas'


Interoperability



One of the nice things of using the DLR infrastructure to create this feature, is that it can be used by other DLR languages. The following example is an IronRuby snippet:

require 'FsDlrJsonExperiments.dll'
include Langexplr::Experiments

while true do
print "Another try\n"
str = System::Net::WebClient.new().download_string("http://twitter.com/statuses/public_timeline.json")
json = FSDynArrayWrapper.CreateFromReader(System::IO::StringReader.new(str))

for i in json.find_all_by_user_with_time_zone('Central America') do
print i.to_string()
end
sleep(5)
end



The following IronPython code shows a little example of this wrapper class.


fReader = StreamReader(GetTwitterPublicTimeline())
jReader = JsonTextReader(fReader)
serializer = JsonSerializer()

json = FSDynArrayWrapper( serializer.Deserialize(jReader) )

for i in json.FindAllByFavoritedAlsoByUserWithScreen_Name("false","ldfallas"):
print i


Implementation



In order to implement the functionality presented here, the IDynamicMetaObjectProvider interface and the DynamicMetaObject class were used. By using these we can generate the code for the call site as a expression tree. For more information on how to use this interface see
Getting Started with the DLR as a Library Author document (available here) .

The code generated to do the filtering is an expression which uses the Where method from System.Linq.Enumerable
. The generated expression written in source using a pseudo C# looks like this:


{
object tmp;
array.Where(c => (((c Is JObject) &&
CompareHelper(
GetJObjectPropertyCI(((JObject)c), "Favorited"),
"true"))
&&
((((tmp = GetJObjectPropertyCI(((JObject)c), "User")) As JObject) != null) &&
CompareHelper(
GetJObjectPropertyCI(((JObject)tmp), "Screen_Name"),
"ldfallas"))))

}


Where GetJObjectPropertyCI is a helper method that gets a property from a JObject by case-intensive name . And CompareHelper is a helper method to do the comparison.

The implementation for FSDynArrayWrapper was written in F#. Mainly because it's a nice language to implement this kind of features. However there's no easy way to consume this feature using F# since it doesn't use the DLR.

Here's the definition:


type FSDynArrayWrapper(a:JArray) =
member this.array with get() = a
static member CreateFromReader(stream : System.IO.TextReader) =
...
static member CreateFromFile(fileName:string) =
...
interface IDynamicMetaObjectProvider with
member this.GetMetaObject( parameter : Expression) : DynamicMetaObject =
FSDynArrayWrapperMetaObject(parameter,this) :> DynamicMetaObject


As you can see, the interesting part is in the implementation of FSDynArrayWrapperMetaObject. The CreateFromReader and CreateFromFile methods are only utility methods to load data from a document.

The implementation of FSDynArrayWrapperMetaObject looks like this:


type FSDynArrayWrapperMetaObject(expression : Expression, value: System.Object) =
inherit DynamicMetaObject(expression,BindingRestrictions.Empty,value)

...

override this.BindInvokeMember(binder : InvokeMemberBinder, args: DynamicMetaObject array) =
match QueryInfo.GetQueryElements(binder.Name) with
| Some( elements ) ->
(new DynamicMetaObject(
this.GenerateCodeForBinder(
elements,
Array.map
(fun (v:DynamicMetaObject) ->
Expression.Constant(v.Value.ToString()) :> Expression) args),
binder.FallbackInvokeMember(this,args).Restrictions))
| None -> base.BindInvokeMember(binder,args)


The BindInvokeMember creates the expression tree for the code that will be executed for a given invocation of a dynamic finder method. Here the QueryInfo.GetQueryElements method is called to extract the elements of the name as described above. The value returned by this method is QueryElement list option where:


type QueryElement =
| ElementQuery of string
| SubElementQuery of string * string


ElementQuery specifies the "Favorited" part in FindAllByFavoritedAlsoByUserWithScreen and the SubElementQuery belongs to the "ByUserWithScreen" part in FindAllByFavoritedAlsoByUserWithScreen .

If the name of the invoked method corresponds is a supported name for a finder, the GenerateCodeForBinder is called to generate the expression tree. The last argument of this method is a collection of the arguments provided for this invocation.


member this.GenerateCodeForBinder(elements, arguments : Expression array) =
let whereParameter = Expression.Parameter(typeof<JToken>, "c") in
let tmpVar = Expression.Parameter(typeof<JToken>, "tmp") in
let whereMethodInfo =
(typeof<System.Linq.Enumerable>).GetMethods()
|> Seq.filter (fun (m:MethodInfo) -> m.Name = "Where" && (m.GetParameters().Length = 2))
|> Seq.map (fun (m:MethodInfo) -> m.MakeGenericMethod(typeof<JToken>))
|> Seq.hd
let queryElementsConditions =
elements
|> Seq.zip arguments
|> Seq.map
(fun (argument,queryParameter) ->
this.GetPropertyExpressionForQueryArgument(queryParameter,argument,whereParameter,tmpVar)) in
let initialCondition = Expression.TypeIs(whereParameter,typeof<JObject>) in

let resultingExpression =
Expression.Block(
[tmpVar],
Expression.Call(
whereMethodInfo,
Expression.Property(
Expression.Convert(
this.Expression,this.LimitType),"array"),
Expression.Lambda(
Seq.fold
(fun s c -> Expression.And(s,c) :> Expression)
(initialCondition :> Expression)
queryElementsConditions,
whereParameter))) in
resultingExpression



The most important parts of this method is the definition of queryElementsConditions and resultingExpression. The resulting expression specifies the invocation to the Where method


The queryElementsConditions take each argument extracted from the name of the method and tries to generate the necessary conditions for the value provided as an argument. In order to do this the GetPropertyExpressionForQueryArgument method is used:


member this.GetPropertyExpressionForQueryArgument(parameter:QueryElement,argument,cParam,tmpVar) : Expression =
match parameter with
| ElementQuery(propertyName) ->
this.CompareExpression(
this.GetJObjectPropertyExpression(cParam,propertyName) ,
argument)
| SubElementQuery(propertyName,subPropertyName) ->
Expression.And(
Expression.NotEqual(
Expression.TypeAs(
Expression.Assign(
tmpVar,
this.GetJObjectPropertyExpression(cParam,propertyName)),
typeof<JObject>),
Expression.Constant(null)),
this.CompareExpression(
this.GetJObjectPropertyExpression(
tmpVar,
subPropertyName),argument)) :> Expression



This method generates a different expression depending on the kind of query element that is requested.

Considerations for IronPython



As a curious note, in IronPython the BindInvokeMember method is not called in the FSDynArrayWrapperMetaObject when a method is invoked. It seems that IronPython calls the BindGetMember method and then tries to apply the result of getting the method.

So to make this object work with IronPython a implementation of the BindGetMember method was created that returns a lambda expression tree with the generated Where invocation.


override this.BindGetMember(binder: GetMemberBinder) =
match QueryInfo.GetQueryElements(binder.Name) with
| Some( elements ) ->
let parameters =
List.mapi ( fun i _ ->
Expression.Parameter(
typeof<string>,
sprintf "p%d" i)) elements
(new DynamicMetaObject(
Expression.Lambda(
this.GenerateCodeForBinder(
elements,
parameters
|> List.map (fun p -> p :> Expression)
|> List.to_array ),
parameters),
binder.FallbackGetMember(this).Restrictions))
| None -> base.BindGetMember(binder)


Accessing using different names



The QueryInfo.GetQueryElements method is used to allow the "FindAllBy..." and "find_all_by..." method names .


module QueryInfo = begin

...

let GetQueryElements(methodName:string) =
match methodName with
| str when str.StartsWith("FindAllBy") ->
Some(ExtractQueryElements(str.Substring("FindAllBy".Length),"AlsoBy","With"))
| str when str.StartsWith("find_all_by_") ->
Some(ExtractQueryElements(str.Substring("find_all_by_".Length),"_also_by_","_with_") )
| _ -> None
end


Code


Code for this post can be found here.