Saturday, November 24, 2007

Creating Netbeans Ruby Hints with Scala, Part 2

In this part I'm going to show a quick fix for the Netbeans Ruby hint presented in part 1.

The quick fix class

The definition of the quick fix for the hint presented in part 1 is the following:


class IfToStatementModifierFix(
info : CompilationInfo,
condition : Node,
statement : Node,
ifNode : Node,
kind : StatementModifierFix) extends Fix {

def isSafe = true
def isInteractive = false

...

}


As shown in this fragment the class that represents the fix requires a reference to each tree element involved in the problem. In this case the important parts are the complete if statement, the condition and the then statement.

Also a kind argument(specific for this quick fix) is used to determine if a if or a unless statement modifier needs to be created. The definition for StatementModifierFix, looks like this:

abstract class StatementModifierFix
case class IfStatementModifierFix extends StatementModifierFix
case class UnlessStatementModifierFix extends StatementModifierFix


The quick fix implementation

The IfToStatementModifierFix class needs to implement the implement method in order to do its work, here's the code:


def implement() = {
val doc = info.getDocument.asInstanceOf[BaseDocument]
val conditionRange = AstUtilities.getRange(condition)
val conditionText = getConditionText(doc,conditionRange)
val statementRange = AstUtilities.getRange(statement)
val statementText = doc.getText(statementRange.getStart,statementRange.getLength)
val ifRange = AstUtilities.getRange(ifNode)

val resultingStatement = statementText + getModifierText + conditionText
doc.atomicLock
doc.replace(ifRange.getStart,ifRange.getLength,resultingStatement,null)
doc.atomicUnlock
}


In order to generate the unless statement we need to change the operator from != to ==.
The getConditionText method does this job:


def getConditionText(doc : BaseDocument,range: OffsetRange) = {
val text = doc.getText(range.getStart,range.getLength)
kind match {
case IfStatementModifierFix() => text
case UnlessStatementModifierFix() => text.replace("!=","==")
}
}


The last thing we need to do is to connect the hint with this quick fix. In order to do this we need to modify the run method of the IfToStatementModifierHint class presented in part 1. We need to add a fix for each identified case.

The case when a method is used as a condition:


...
case ifStat@IfStatement(methodCall : CallNode,
NewlineNode(thenStat),
null) => {

val range = AstUtilities.getRange(node)

val fixes = new java.util.ArrayList
fixes.add(
new IfToStatementModifierFix(info,methodCall,thenStat,ifStat,IfStatementModifierFix))

val desc =
new Description(this,
recomendationText,
info.getFileObject,range,
fixes,
600)
result.add(desc)
}
...


The case when the condition is a not-equal comparison with nil:


case ifStat@IfStatement(condition@NotNode(eqExp@EqualComparison(x,_ : NilNode)),
NewlineNode(thenStat),
null) => {


val range = AstUtilities.getRange(node)

val fixes = new java.util.ArrayList
fixes.add(
new IfToStatementModifierFix(info,condition,thenStat,ifStat,IfStatementModifierFix))
fixes.add(
new IfToStatementModifierFix(info,eqExp,thenStat,ifStat,UnlessStatementModifierFix))

val desc =
new Description(this,
recomendationText,
info.getFileObject,range,
fixes,
600)
result.add(desc)
}


Note that for this case we also offer the quick fix for unless.

Example 1

Activating the hint on this code:



Now displays the following quick fix:



And changes the code as:



Example 2

Also for code like this:



The following options are presented:



By selecting option 2 the code is changed as:



Code for this experiment can be found here.

Monday, November 12, 2007

Creating Netbeans Ruby Hints with Scala, Part 1

In this post I'm going to show a little experiment of creating a Netbeans Ruby Hint using the Scala language. This is the first of a two-part series of posts on this topic.

Netbeans Ruby Hints and Quick Fixes is a very nice feature that will be part of Netbeans 6.0 Ruby integration. This feature can be used to identify potential problems or to promote best practices in Ruby programs. I first learned about it by reading Tor Norbye's blog.

I really like the idea of mixing languages to accomplish something. This is the main reason I chose Scala to do this experiment. Also I wanted to try to apply some ideas from previous posts. For a future experiment it will be very nice to write this code using JRuby.

For this post I'm using Netbeans 6.0 Beta 2. As described below the APIs for writing hints are still not officially public because they're still under development. Again, as with previous posts, this is only an experiment to see how different programming languages could be used solve special tasks.

Motivation

Several weeks ago I wrote a post on Ruby's yield statement, there I showed a function that calculates the Fibonacci sequence using a while statement in a pretty traditional way. Then someone posted a comment with a much compact version of it by using some nice Ruby features. That got me thinking how Netbeans Ruby Hints could also be used to help Ruby beginners(like me) to find out about alternative ways to do something.

The Hint

One of the things I like about Ruby is statement modifiers. That's one of the first things you notice while reading the Why's (Poignant) Guide to Ruby.

I think statement modifiers makes the code look nice if used carefully.

So the hint that will be implement it's going to identify opportunities to apply the if or unless statement modifiers. For example:


table = Hash.new
...
if table.empty? then
puts "No items"
end
...


Could also be written as:


table = Hash.new
...
puts "No items" if table.empty?
...


Creating the Netbeans module

The biggest challenge I had was to create a Netbeans module using only Scala. The main issue is not having all the nice GUI facilities available to create to module.

Luckily there's a series of articles in Geertjan's Weblog called NetBeans Modules for Dummies Part 1,2,3,4. These articles provided a lot of information on how to create an NBM file using Ant tasks .

I also had to create a some Netbeans modules using Java to dissect the resulting NBM file and to provide some of the missing arguments. The resulting Ant file is can be found here.

Another challenge that I had was that some of the classes that I needed to access were only available to friend packages, this mainly because I was trying to access APIs that are still changing. Luckly, Tor Norbye from the Netbeans Ruby development list, provided a solution for this issue which could be found here. Again this is a consequence of using API's that are still not public.

The last issue that I had was to add the Scala Runtime Jar file as part of the package. The How do module dependencies/classloading work? Netbeans Faq entry was very helpful to solve this.

The code

By reading the code of existing hints, I learned that, the way to define hints (which is still under development) is very nice and very simple. Also there's useful documentation in the Writing New Ruby Hints entry from the Netbeans wiki.

Here's the code:

class IfToStatementModifierHint extends AstRule {

def appliesTo(info : CompilationInfo) = true

def getKinds() : Set = {
val nodeKinds = new HashSet()
nodeKinds.add(NodeTypes.IFNODE)
nodeKinds
}

def run(info :CompilationInfo,
node : Node,
path : AstPath ,
caretOffset : int,
result : java.util.List) : unit = {

node match {
case IfStatement(callNode : CallNode,
NewlineNode(thenStat),
null) => {

val range = AstUtilities.getRange(node)

val desc =
new Description(this,
recomendationText,
info.getFileObject,range,
Collections.emptyList,
600)
result.add(desc)
}
case IfStatement(NotNode(EqualComparison(x,_ : NilNode)),
NewlineNode(thenStat),
null) => {


val range = AstUtilities.getRange(node)

val desc =
new Description(this,
recomendationText,
info.getFileObject,range,
Collections.emptyList,
600)
result.add(desc)
}
case _ =>
{
}
}
}

def getId() = "TestHint1"

def getDisplayName() = "If statement as statement modifier"

def getDescription() = "Identifies certain ..."

def getDefaultEnabled() = true

def getDefaultSeverity() = HintSeverity.WARNING

def getCustomizer(node: Preferences) : JComponent = null

def showInTasklist() = true

def recomendationText = "Consider changing..."

}


The most interesting methods of this class are getHints and run.

The getHints method defines the kinds of AST nodes this method applies to. The run method identifies which kinds of if statement apply to this hint. To do this a couple of extractor objects defined in previous posts are used. Two special cases are identified:

  • One if statement with a method call as condition, one statement in the then section and no statements in the else section.

  • One if statement with an equal comparison to nil as condition, one statement in the then section and no statements in the else section.


This Hint is shown as a WARNING because it's the how severity that seems to apply. Although it will be very nice to have a RECOMMENDATION severity which seems to be more appropriate for this hint.

I really like this way of defining hints because you have to write exactly the code related to your problem.

Here's and example in action:




Code for this experiment can be found here.

For the next post I'll try to implement a quick fix for this hint.

Saturday, November 10, 2007

Exploring JRuby Syntax Trees With Scala, Part 2

In this post I'm going to show a couple of examples of using Scala to extract elements of Ruby programs.

Because of its functional influence Scala provides nice features to manipulate tree structures. As I mentioned in previous posts, I think one of the most interesting features provided by Scala is pattern matching. Among other pattern matching constructs, Scala provides one that I particularly like : Extractors.

The Matching Object With Patterns paper by Burak Emir , Martin Odersky , and John Williams, describes several techniques for exploring complex object data structures. One of this techniques is the Visitor design pattern which is one of the most used techniques that is used to navigate a syntax tree structure. I think the pattern matching approach can be combined with the visitors to solve different tasks.

In this post I'm interested in showing pattern matching using Scala extractors on the JRuby AST. The JRuby AST seems to provide nice visitor mechanism which will be explored in future posts.

Scala extractor objects allow existing Scala or Java Platform classes to participate in pattern matching just as case classes. Extractor objects can be used to expose only interesting elements for a certain problem and also makes the definition of the class independent of the way it is used in a match construct. A nice description of this feature for Scala is provided in the Matching Object With Patterns paper. A similar feature is implemented in F# as active patterns.This concept was originated by work on Views by Philip Walder .

I'm going to show a couple of examples of using this feature with the JRuby AST.

Example 1:

Say that you want to identify the key value that is given to Hash assignment: (here highlighted in red)


x["a"] = 1


We need to define extractors for related objects. In order to do this I'm going to use the tool from the previous post to see which classes we need to represent.



By reading this code we noticed that we need at least extractors for

  • BlockNode

  • NewlineNode

  • AttrAssignNode

  • LocalVarNode

  • ArrayNode

  • StrNode

  • FixnumNode



Basic extractors for the child nodes of these elements could be defined as follows:


object NewlineNode {
def unapply(astNode : Node) =
if (astNode.isInstanceOf[NewlineNode]) {
Some(astNode.asInstanceOf[NewlineNode].getNextNode)
} else {
None
}
}

object LocalVarNode {
def unapply(astNode : Node) = {
if (astNode.isInstanceOf[LocalVarNode]) {
Some (astNode.asInstanceOf[LocalVarNode].getName)
} else {
None
}
}
}

object StrNode {
def unapply(astNode : Node) = {
if (astNode.isInstanceOf[StrNode]) {
Some (astNode.asInstanceOf[StrNode].getValue)
} else {
None
}
}
}

object BlockNode {
def unapply(astNode : Node) = {
if (astNode.isInstanceOf[BlockNode]) {
Some (List.fromIterator(
new JavaIteratorWrapper[Node](
astNode.asInstanceOf[BlockNode].childNodes.iterator())))
} else {
None
}
}
}

object FixnumNode {
def unapply(astNode : Node) = {
if (astNode.isInstanceOf[FixnumNode]) {
Some (astNode.asInstanceOf[FixnumNode].getValue)
} else {
None
}
}
}


object AttrAssignNode {
def unapply(astNode : Node) = {
if (astNode.isInstanceOf[AttrAssignNode]) {
val attAsgn = astNode.asInstanceOf[AttrAssignNode]
Some (attAsgn.getReceiverNode,attAsgn.getArgsNode)
} else {
None
}
}
}

object ArrayNode {
def unapply(astNode : Node) =
if (astNode.isInstanceOf[ArrayNode]) {
val theArrayNode = astNode.asInstanceOf[ArrayNode]
Some(List.fromIterator(new JavaIteratorWrapper[Node](theArrayNode.childNodes.iterator)))
} else {
None
}
}


By defining these extractors we can combine them with existing Scala pattern matching constructs to create the required pattern:


val rubyCode2 = "x = Hash.new\n"+"x[\"a\"] = 1"
val rootNode2 = r.parse(rubyCode2,"dummy.rb",r.getCurrentContext.getCurrentScope,0).asInstanceOf[RootNode]

rootNode2.getBodyNode match {
case BlockNode(
List(_,
NewlineNode(
AttrAssignNode(
LocalVarNode("x"),
ArrayNode(
List(key,
FixnumNode(1))))))) =>
System.out.println("Matches for: "+ key.toString())
case _ => System.out.println("No Match!")
}


Example 2:

For the next example say that we want to extract the identifier which is compared to some number in the condition of an if statement. For example:


if the_variable == 1 then
...
else
...
end


The tree representation of a code similar to this looks like:



The following extractors will be used to identify this pattern:


object IfStatement {
def unapply(astNode : Node) =
if (astNode.isInstanceOf[IfNode]) {
val theIfNode = astNode.asInstanceOf[IfNode]
Some (theIfNode.getCondition,theIfNode.getThenBody,theIfNode.getElseBody)
} else {
None
}
}


object EqualComparison {
def unapply(astNode : Node) =
if (astNode.isInstanceOf[CallNode]) {
val theCallNode = astNode.asInstanceOf[CallNode]
if (theCallNode.getName.equals("==") &&
theCallNode.getArgsNode.isInstanceOf[ArrayNode] &&
theCallNode.getArgsNode.asInstanceOf[ArrayNode].size == 1) {
Some (theCallNode.getReceiverNode,
theCallNode.getArgsNode.asInstanceOf[ArrayNode].get(0))
} else {
None
}
} else {
None
}
}


Note that in this case I'm using a shortcut, instead of defining an extractor for CallNode and using it with ArrayNode a single extractor is used to identify the binary equal comparison.

By using this extractors we can define the pattern as follows:


val rubyCode3 = "if the_variable == 30 then\n puts 2\nelse\n puts 3\nend"
val rootNode3 = r.parse(rubyCode3,"dummy.rb",r.getCurrentContext.getCurrentScope,0).asInstanceOf[RootNode]
val nodeToMatch = rootNode3.getBodyNode.asInstanceOf[NewlineNode].getNextNode
nodeToMatch match {
case IfStatement(EqualComparison(leftSide,_ : FixnumNode),
_,
_) =>
System.out.println("Matches for: "+ leftSide.toString())
case _ => System.out.println("No Match!")
}



For future post I'm going to be using these extractors to create more examples in other contexts.

Wednesday, November 7, 2007

Exploring JRuby Syntax Trees With Scala, Part 1

In this post I'm going to show how to use Scala to have access to JRuby syntax trees.

Abstract syntax trees(AST) provide a representation of the source code that can be manipulated by another program. These trees are used by compilers and IDEs to do their work.

The JRuby distribution includes a parser and a nice tree representation for Ruby source code. This AST is used by the JRuby implementation to do its work, and also recently I noticed that the Netbeans Ruby support uses this representation for some tasks.

Calling JRuby Parser is pretty straightforward, just do the following:


package org.langexplr.jrubyasttests;
import org.jruby._
import org.jruby.ast._


object ScalaTestApp {
def main(args : Array[String]) : Unit = {
val rubyCode = "puts \"Hello\""

val r = Ruby.getDefaultInstance
val rootNode = r.parse(rubyCode,"dummy.rb",r.getCurrentContext.getCurrentScope,0).asInstanceOf[RootNode]
...
}
}


The rootNode variable contains the root of the syntax tree for the code stored in rubyCode.

In order to explore the syntax tree I wrote a small program that shows a Swing JTreecomponent with the result of parsing a snippet. Creating this program was easy because Node, the base class of all the elements in JRuby AST, provides a childNodes method. The model for the AST is the following:


package org.langexpr.jrubyasttest;

import javax.swing.tree.TreeModel
import javax.swing.event.TreeModelListener
import javax.swing.tree.TreePath
import org.jruby._
import org.jruby.ast._

class JRubyTreeModel() extends TreeModel {
var rootNode : Node = null;
def this(rubyCode : String) = {
this()
val r = Ruby.getDefaultInstance
this.rootNode = r.parse(rubyCode,"dummy.rb",r.getCurrentContext.getCurrentScope,0)
}

def getRoot : Object = rootNode

def isLeaf(node: Object ) =
node.asInstanceOf[Node].childNodes.size == 0

def getChildCount( parent : Object) : int =
parent.asInstanceOf[Node].childNodes.size

def getChild(parent : Object , index : int) : Object = {
parent.asInstanceOf[Node].childNodes.get(index)
}
def getIndexOfChild( parent : Object, child : Object) : int = {
parent.asInstanceOf[Node].childNodes.indexOf(child)
}
def valueForPathChanged( path : TreePath, newValue : Object) : unit ={

}
def addTreeModelListener( l : TreeModelListener) : unit = {

}
def removeTreeModelListener( l : TreeModelListener) : unit = {

}
}


With this model now we can create the main program:


package org.langexpr.jrubyasttest;


import javax.swing._
import java.awt.event._
import java.awt.BorderLayout
import java.awt.Dimension

object JRubyAstTest {
val treeVisualizer = new JTree
val input = new JTextArea


def createFrame = {
val theFrame = new JFrame("JRuby AST test")
theFrame.setSize(new Dimension(640,450))
val content = theFrame.getContentPane
content.setLayout(new BorderLayout)
content.add(treeVisualizer,BorderLayout.CENTER)


val bottomPanel = new JPanel
bottomPanel.setLayout(new BorderLayout)
content.add(bottomPanel,BorderLayout.SOUTH)

bottomPanel.add(new JScrollPane(input),BorderLayout.CENTER)
input.setText("x=1\ny = 2\n")

treeVisualizer.setModel(new JRubyTreeModel(input.getText))

val button = new JButton("Parse!")
button.addActionListener( new ActionListener() {
def actionPerformed(ae : ActionEvent) =
{
treeVisualizer.setModel(new JRubyTreeModel(input.getText))
}
})

bottomPanel.add(button,BorderLayout.SOUTH)

theFrame
}
def main(args : Array[String]) : Unit = {
val f = createFrame
f.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
f.setVisible(true)
}
}


Now we can visually explore the AST using this little program.

The following code:

def foo(x)
return x*x
end


Is visualized like this:



Examples in this post could also be implemented easily using JRuby but in the second part I'm going to a couple of Scala features that I specially like to manipulate this data structure.