One of the things that really impressed me while reading about Ruby on Rails was the use of
method_missing to implement
dynamic finders. The technique is described on the
"How dynamic filters work". In this post I'm going to show a little experiment of creating a similar technique for querying JSON arrays using the .
NET's Dynamic Language Runtime infrastructure.
Code for this post was created using Visual Studio 2010 Beta 1,
IronRuby for .NET 4 beta 1 and
IronPython 2.6 beta 4 for .NET 4.
JSON.NET
For this post I'm using the
JSON.NET library for loading the
JSON data. This library provides several ways of loading JSON data. Here I'll be using a set of predefined classes:
JObject for JSON objects,
JArray for arrays,
JValue for literal values, etc. All these classes inherit from
JToken.
Code in this post use the JSON data returned by the
Twitter REST API. An example of this data:
[
{"in_reply_to_screen_name":null,
"text":"...",
"user": { "following":null,
"description":"...",
"screen_name":"...",
"utc_offset":0,
"followers_count":10,
"time_zone":"...",
"statuses_count":155,
"created_at":"...",
"friends_count":1,
"url":"...",
"name":"...",
"notifications":null,
"protected":false,
"verified":false,
"favourites_count":0,
"location":"...",
"id": ...,
...
},
"truncated":false,
"created_at":"...",
"in_reply_to_status_id":null,
"in_reply_to_user_id":null,
"favorited":false,
"id":...,
"source":"...."
},
...
]
Dynamic queries on JSON data
Finders will be implemented for JSON arrays. As with Rail's dynamic finders the names of the required fields will be encoded in the name of the invoked method.
The following C# 4.0 code shows an example of this wrapper class in conjunction with the
dynamic keyword.
JsonTextReader reader = new JsonTextReader(rdr);
JsonSerializer serializer = new JsonSerializer();
JArray o = (JArray)serializer.Deserialize(reader);
dynamic dArray = new FSDynArrayWrapper(o);
string name = "ldfallas";
foreach (var aJObject in dArray.FindAllByFavoritedAlsoByUserWithScreen_Name("false",name))
{
dynamic tobj = new FSDynJObjectWrapper(aJObject);
Console.WriteLine("========");
Console.WriteLine(tobj.user.screen_name);
Console.Write("\t'{0}'",tobj.text);
}
A small definition of the syntax used for names is the following.
method-name = "FindAllBy" ,
property-name , ("With", property-name)? ,
("AlsoBy" property-name , ("With", property-name)? ) *
property-name = valid json property name
In order to be more flexible the following syntax will also be allowed:
method-name = "find_all_by_" ,
property-name , ("_with_", property-name)? ,
("_also_by" property-name , ("_with_", property-name)? ) *
property-name = valid json property name
A sample name for this query methods look like this:
array.FindAllByFavoritedAlsoByUserWithScreen_Name("true","ldfallas")
This method will accept two parameters and is going to :
Find all the object elements from the array that has a 'Favorited' property equal to 'true' and also has an object with a 'User' property associated with an object which has a 'Screen_Name' property which is equal to the 'ldfallas'
Interoperability
One of the nice things of using the DLR infrastructure to create this feature, is that it can be used by other DLR languages. The following example is an IronRuby snippet:
require 'FsDlrJsonExperiments.dll'
include Langexplr::Experiments
while true do
print "Another try\n"
str = System::Net::WebClient.new().download_string("http://twitter.com/statuses/public_timeline.json")
json = FSDynArrayWrapper.CreateFromReader(System::IO::StringReader.new(str))
for i in json.find_all_by_user_with_time_zone('Central America') do
print i.to_string()
end
sleep(5)
end
The following IronPython code shows a little example of this wrapper class.
fReader = StreamReader(GetTwitterPublicTimeline())
jReader = JsonTextReader(fReader)
serializer = JsonSerializer()
json = FSDynArrayWrapper( serializer.Deserialize(jReader) )
for i in json.FindAllByFavoritedAlsoByUserWithScreen_Name("false","ldfallas"):
print i
Implementation
In order to implement the functionality presented here, the
IDynamicMetaObjectProvider interface and the
DynamicMetaObject class were used. By using these we can generate the code for the call site as a
expression tree. For more information on how to use this interface see
Getting Started with the DLR as a Library Author document (available
here) .
The code generated to do the filtering is an expression which uses the
Where method from
System.Linq.Enumerable
. The generated expression written in source using a pseudo C# looks like this:
{
object tmp;
array.Where(c => (((c Is JObject) &&
CompareHelper(
GetJObjectPropertyCI(((JObject)c), "Favorited"),
"true"))
&&
((((tmp = GetJObjectPropertyCI(((JObject)c), "User")) As JObject) != null) &&
CompareHelper(
GetJObjectPropertyCI(((JObject)tmp), "Screen_Name"),
"ldfallas"))))
}
Where
GetJObjectPropertyCI is a helper method that gets a property from a
JObject by case-intensive name . And
CompareHelper is a helper method to do the comparison.
The implementation for
FSDynArrayWrapper was written in F#. Mainly because it's a nice language to implement this kind of features. However there's no easy way to consume this feature using F# since it doesn't use the DLR.
Here's the definition:
type FSDynArrayWrapper(a:JArray) =
member this.array with get() = a
static member CreateFromReader(stream : System.IO.TextReader) =
...
static member CreateFromFile(fileName:string) =
...
interface IDynamicMetaObjectProvider with
member this.GetMetaObject( parameter : Expression) : DynamicMetaObject =
FSDynArrayWrapperMetaObject(parameter,this) :> DynamicMetaObject
As you can see, the interesting part is in the implementation of
FSDynArrayWrapperMetaObject. The
CreateFromReader and
CreateFromFile methods are only utility methods to load data from a document.
The implementation of
FSDynArrayWrapperMetaObject looks like this:
type FSDynArrayWrapperMetaObject(expression : Expression, value: System.Object) =
inherit DynamicMetaObject(expression,BindingRestrictions.Empty,value)
...
override this.BindInvokeMember(binder : InvokeMemberBinder, args: DynamicMetaObject array) =
match QueryInfo.GetQueryElements(binder.Name) with
| Some( elements ) ->
(new DynamicMetaObject(
this.GenerateCodeForBinder(
elements,
Array.map
(fun (v:DynamicMetaObject) ->
Expression.Constant(v.Value.ToString()) :> Expression) args),
binder.FallbackInvokeMember(this,args).Restrictions))
| None -> base.BindInvokeMember(binder,args)
The
BindInvokeMember creates the expression tree for the code that will be executed for a given invocation of a dynamic finder method. Here the
QueryInfo.GetQueryElements method is called to extract the elements of the name as described above. The value returned by this method is
QueryElement list option where:
type QueryElement =
| ElementQuery of string
| SubElementQuery of string * string
ElementQuery specifies the "Favorited" part in
FindAllByFavoritedAlsoByUserWithScreen and the
SubElementQuery belongs to the "ByUserWithScreen" part in
FindAllByFavoritedAlsoByUserWithScreen .
If the name of the invoked method corresponds is a supported name for a finder, the
GenerateCodeForBinder is called to generate the expression tree. The last argument of this method is a collection of the arguments provided for this invocation.
member this.GenerateCodeForBinder(elements, arguments : Expression array) =
let whereParameter = Expression.Parameter(typeof<JToken>, "c") in
let tmpVar = Expression.Parameter(typeof<JToken>, "tmp") in
let whereMethodInfo =
(typeof<System.Linq.Enumerable>).GetMethods()
|> Seq.filter (fun (m:MethodInfo) -> m.Name = "Where" && (m.GetParameters().Length = 2))
|> Seq.map (fun (m:MethodInfo) -> m.MakeGenericMethod(typeof<JToken>))
|> Seq.hd
let queryElementsConditions =
elements
|> Seq.zip arguments
|> Seq.map
(fun (argument,queryParameter) ->
this.GetPropertyExpressionForQueryArgument(queryParameter,argument,whereParameter,tmpVar)) in
let initialCondition = Expression.TypeIs(whereParameter,typeof<JObject>) in
let resultingExpression =
Expression.Block(
[tmpVar],
Expression.Call(
whereMethodInfo,
Expression.Property(
Expression.Convert(
this.Expression,this.LimitType),"array"),
Expression.Lambda(
Seq.fold
(fun s c -> Expression.And(s,c) :> Expression)
(initialCondition :> Expression)
queryElementsConditions,
whereParameter))) in
resultingExpression
The most important parts of this method is the definition of
queryElementsConditions and
resultingExpression. The resulting expression specifies the invocation to the
Where method
The
queryElementsConditions take each argument extracted from the name of the method and tries to generate the necessary conditions for the value provided as an argument. In order to do this the
GetPropertyExpressionForQueryArgument method is used:
member this.GetPropertyExpressionForQueryArgument(parameter:QueryElement,argument,cParam,tmpVar) : Expression =
match parameter with
| ElementQuery(propertyName) ->
this.CompareExpression(
this.GetJObjectPropertyExpression(cParam,propertyName) ,
argument)
| SubElementQuery(propertyName,subPropertyName) ->
Expression.And(
Expression.NotEqual(
Expression.TypeAs(
Expression.Assign(
tmpVar,
this.GetJObjectPropertyExpression(cParam,propertyName)),
typeof<JObject>),
Expression.Constant(null)),
this.CompareExpression(
this.GetJObjectPropertyExpression(
tmpVar,
subPropertyName),argument)) :> Expression
This method generates a different expression depending on the kind of query element that is requested.
Considerations for IronPython
As a curious note, in IronPython the
BindInvokeMember method is not called in the
FSDynArrayWrapperMetaObject when a method is invoked. It seems that IronPython calls the
BindGetMember method and then tries to apply the result of getting the method.
So to make this object work with IronPython a implementation of the
BindGetMember method was created that returns a
lambda expression tree with the generated
Where invocation.
override this.BindGetMember(binder: GetMemberBinder) =
match QueryInfo.GetQueryElements(binder.Name) with
| Some( elements ) ->
let parameters =
List.mapi ( fun i _ ->
Expression.Parameter(
typeof<string>,
sprintf "p%d" i)) elements
(new DynamicMetaObject(
Expression.Lambda(
this.GenerateCodeForBinder(
elements,
parameters
|> List.map (fun p -> p :> Expression)
|> List.to_array ),
parameters),
binder.FallbackGetMember(this).Restrictions))
| None -> base.BindGetMember(binder)
Accessing using different names
The
QueryInfo.GetQueryElements method is used to allow the "FindAllBy..." and "find_all_by..." method names .
module QueryInfo = begin
...
let GetQueryElements(methodName:string) =
match methodName with
| str when str.StartsWith("FindAllBy") ->
Some(ExtractQueryElements(str.Substring("FindAllBy".Length),"AlsoBy","With"))
| str when str.StartsWith("find_all_by_") ->
Some(ExtractQueryElements(str.Substring("find_all_by_".Length),"_also_by_","_with_") )
| _ -> None
end
Code
Code for this post can be found
here.
