Archive for April, 2010

Scala compiler advanced options

Recently I found myself trying to find a definitive list of the options supported by scalac, the Scala compiler. It turns out that I could have just asked the compiler itself (the -X option lists all “advanced” options and the -Y option lists all “private” options). But this is very difficult to find via Google.

So I thought that I’d create this page and hopefully save anyone else looking for the same information some time.

$ scalac -version
Scala compiler version 2.8.0.RC1 -- Copyright 2002-2010, LAMP/EPFL

Advanced Options

$ scalac -X
Usage: scalac  
Possible advanced options include:
  -Xassem-extdirs          List of directories containing assemblies, defaults to `lib'
  -Xassem-name             Name of the output assembly (only relevant with -target:msil)
  -Xassem-path             List of assemblies referenced by the program (only relevant with -target:msil)
  -Xcheck-null                   Emit warning on selection of nullable reference
  -Xcheckinit                    Add runtime checks on field accessors. Uninitialized accesses result in an exception being thrown.
  -Xdisable-assertions           Generate no assertions and assumptions
  -Xelide-below                  Generate calls to @elidable-marked methods only if method priority is greater than argument.
  -Xexperimental                 Enable experimental extensions
  -Xfuture                       Turn on future language features
  -Xgenerate-phase-graph   Generate the phase graphs (outputs .dot files) to fileX.dot
  -Xlog-implicits                Show more info on why some implicits are not applicable
  -Xmigration                    Warn about constructs whose behavior may have changed between 2.7 and 2.8
  -Xno-forwarders                Do not generate static forwarders in mirror classes
  -Xno-uescape                   Disables handling of u unicode escapes
  -Xnojline                      Do not use JLine for editing
  -Xplugin-disable:      Disable a plugin
  -Xplugin-list                  Print a synopsis of loaded plugins
  -Xplugin-require:      Abort unless a plugin is available
  -Xplugin:                Load a plugin from a file
  -Xpluginsdir             Path to search compiler plugins
  -Xprint-icode                  Log internal icode to *.icode files
  -Xprint-pos                    Print tree positions (as offsets)
  -Xprint-types                  Print tree types (debugging option)
  -Xprint:                Print out program after  or "all"
  -Xprompt                       Display a prompt after each error (debugging option)
  -Xresident                     Compiler stays resident, files to compile are read from standard input
  -Xscript               Compile as a script, wrapping the code into object.main()
  -Xshow-class            Show class info
  -Xshow-object          Show object info
  -Xshow-phases                  Print a synopsis of compiler phases
  -Xsource-reader     Specify a custom method for reading source files
  -Xsourcedir         When -target:msil, the source folder structure is mirrored in output directory.
  -Xstrict-warnings              Emit warnings about lots of things.
  -Xwarninit                     Warn about possible changes in initialization semantics
  -Y                             Print a synopsis of private options

Private Options

$ scalac -Y
Usage: scalac  
Possible private options include:
  -Ybrowse:               Browse the abstract syntax tree after  or "all"
  -Ybuild-manager-debug          Generate debug information for the Refined Build Manager compiler.
  -Ybuilder-debug:       Compile using the specified build manager (none,refined,simple)
  -Ycheck:                Check the tree at the end of  or "all"
  -Yclosure-elim                 Perform closure elimination
  -Ycompact-trees                Use compact tree printer when displaying trees
  -Ydead-code                    Perform dead code elimination
  -Ydebug                        Output debugging messages
  -Ydetach                       Perform detaching of remote closures
  -Yfatal-warnings               Fail the compilation if there are any warnings.
  -Yide-debug                    Generate, validate and output trees using the interactive compiler.
  -Yinline                       Perform inlining when possible
  -Yjenkins-hashCodes            Use jenkins hash algorithm for case class generated hashCodes.
  -Ylinearizer:           Linearizer to use (normal,dfs,rpo,dump)
  -Ylog-classpath                Output information about what classpath is being applied.
  -Ylog:                  Log operations in  or "all"
  -Yno-completion                Disable tab-completion in the REPL
  -Yno-generic-signatures        Suppress generation of generic signatures for Java
  -Yno-imports                   Compile without any implicit imports
  -Yno-predefs                   Compile without any implicit predefined values
  -Ypmat-debug                   Trace all pattern matcher activity.
  -Ypmat-naive                   Desugar matches as naively as possible..
  -Yrangepos                     Use range positions for syntax trees.
  -Yrecursion                    Recursion depth used when locking symbols
  -Yrepl-debug                   Trace all repl activity.
  -Yself-in-annots               Include a "self" identifier inside of annotations
  -Yshow-trees                   Show detailed trees when used in connection with -print:phase
  -Yskip:                 Skip  or "all"
  -Ysqueeze:            if on, creates compact code in matching (on,off)
  -Ystatistics                   Print compiler statistics
  -Ystop:                 Stop after phase  or "all"
  -Ystruct-dispatch:     Selects dispatch method for structural refinement method calls (no-cache,mono-cache,poly-cache,invoke-dynamic)
  -Ysuppress-vt-typer-warnings   Suppress warnings from the typer when testing the virtual class encoding, NOT FOR FINAL!
  -Ytailrecommend                Alert methods which would be tail-recursive if private or final.
  -Ytyper-debug                  Trace all type assignements
  -Ywarn-catches                 Emit warnings about catch blocks which catch everything.
  -Ywarn-dead-code               Emit warnings for dead code
  -Ywarn-shadowing               Emit warnings about possible variable shadowing.

Scala 2.8.0.RC1, sbt and ScalaTest step-by-step

For an updated version of this post that covers Scala 2.9.1 and sbt 0.10, go here.

I spent quite a bit of time beating my head against the details of getting Scala 2.8.0.RC1, sbt and ScalaTest to play nicely with each other. So here’s a step-by-step guide which will hopefully save others time.

  1. Install sbt
  2. Create a new project as follows:
    $ mkdir aproject
    $ cd aproject/
    $ sbt
    Project does not exist, create new project? (y/N/s) y
    Name: aproject
    Organization: com.example
    Version [1.0]:
    Scala version [2.7.7]: 2.8.0.RC1
    sbt version [0.7.3]:
    
  3. Create a build configuration file in project/build/AProject.scala containing:
    import sbt._
    
    class AProject(info: ProjectInfo) extends DefaultProject(info) {
      val scalaToolsSnapshots = ScalaToolsSnapshots
      val scalatest = "org.scalatest" % "scalatest" %
        "1.0.1-for-scala-2.8.0.RC1-SNAPSHOT"
    }
    

    This tells sbt that you will be using the pre-release snapshot version of ScalaTest for Scala 2.8.0.RC1.

  4. Download the ScalaTest library with sbt update. After running this, you should see that you have a lib_managed directory containing the ScalaTest jar.
  5. Create src/main/scala/Widget.scala containing:
    package com.example
    
    class Widget {
      def colour = "Blue"
      def disposition = "Awesome"
    }
  6. Create src/test/scala/WidgetSpec.scala containing:
    package com.example.test
    
    import org.scalatest.Spec
    import com.example.Widget
    
    class WidgetSpec extends Spec {
    
      describe("A Widget") {
    
        it("should be blue") {
          expect("Blue") { new Widget().colour }
        }
    
        it("should be awesome") {
          expect("Awesome") { new Widget().disposition }
        }
      }
    }
    
  7. Run your tests with sbt test. You should see:
    [info] == com.example.test.WidgetSpec ==
    [info] A Widget
    [info] Test Starting: A Widget should be blue
    [info] Test Passed: A Widget should be blue
    [info] Test Starting: A Widget should be awesome
    [info] Test Passed: A Widget should be awesome
    [info] == com.example.test.WidgetSpec ==
    
  8. Create src/main/scala/Main.scala containing:
    package com.example
    
    object Main {
      def main(args: Array[String]) {
        val w = new Widget()
        println("My new widget is "+ w.colour)
      }
    }
    
  9. Run your program with sbt run. You should see:
    [info] Running com.example.Main
    My new widget is Blue
    

Update: Use the pre-defined ScalaToolsSnapshots repo (thanks to Erick Fleming).

Update: Fixed a typo in step 8 (thanks to Jason).

Fuzzy about Scala

I’ve recently started working on a new project (SwiftKey), and my immediate priority is to get our testing sorted out. To that end, I’m constructing a fuzz test suite. Partly because I think that it’s the right tool for the job, and partly because it’s a good opportunity for me to learn more about the language, I’m writing it in Scala.

A fuzz test interacts with the System Under Test in a random, yet structured manner. The object of the exercise is to construct sequences of API calls (or network traffic, or whatever) that flush out the kind of bugs that are less likely to be discovered by unit tests tests and the like. Fuzz tests are particularly good at finding things like security problems, buffer overruns, memory leaks etc.

Clearly randomness is a large part of a fuzz test, so I’m creating some utilities that make it easier to randomly choose between several different actions. I thought that I’d publish them just in case anyone else finds them useful, and also to solicit feedback to help me improve my use of Scala (bear in mind that I’m still finding my feet, so be gentle!).

So, here’s my first attempt:

val random = new Random

def oneAtRandom(actions: () => Unit*)() =
  () => actions(random.nextInt(actions.length))()

oneAtRandom takes a set of functions and returns another function which, each time it’s called, executes one of them at random. For example:

val randomAction = oneAtRandom(
  () => print('a'), () => print('b'), () => print('c'))

for(_ <- 1 to 10)
  randomAction()

Which gives the following output:

aabaacbbcc

So far, so good. But what if we want different functions to be executed with different frequencies? We might have some actions that we want to only happen occasionally? Here’s another version of oneAtRandom that takes a weighted set of functions and calls them in proportion to their relative weights:

def oneAtRandom(actions: (Int, () => Unit)*)() = {
  val totalWeight = actions.foldLeft(0) {_ + _._1}
  () => {
    val r = random.nextInt(totalWeight)
    var w = 0
    val (_, action) = actions.find {
        case(weight, _) => w += weight; w > r
      }.get
    action()
  }
}

Here’s how it’s used:

val randomAction = oneAtRandom(
    (1, () => println("rare")),
    (10, () => println("occasional")),
    (50, () => println("frequent"))
  )

for(_ <- 1 to 100)
  randomAction()

And to prove that it’s doing what it claims:

$ scala random2.scala | sort | uniq -c
  88 frequent
  11 occasional
   1 rare

$ scala random2.scala | sort | uniq -c
  82 frequent
  12 occasional
   6 rare

So I’ve got it working, which I’m happy about. But I’m sure that it could be improved. I’m sure, for example, that there must be a “nice” functional way to implement the loop inside the weighted version (instead of using var w as an accumulator), but each of my attempts to do so have ended up being considerably more complex than the current version.

I’d also love to use the -> notation to pass the arguments, for example:

val randomAction = oneAtRandom(
    1 -> () => println("rare"),
    10 -> () => println("occasional"),
    50 -> () => println("frequent")
  )

But doing so results in a parse error—the only way I’ve found to avoid it is to add additional parentheses:

val randomAction = oneAtRandom(
    1 -> (() => println("rare")),
    10 -> (() => println("occasional")),
    50 -> (() => println("frequent"))
  )

which is starting to feel a bit too much like Lisp for my taste 🙂

Finally, I’m really missing some of the handy little utilities provided by Ruby. In Ruby, for example, I could write the loops that call randomAction like this:

10.times { randomAction }

I know that it’s trivial to create this kind of utility, but one of the nice things about Ruby is that they all come as standard out of the box.

In any case, I’d be very grateful for any and all feedback on the above. Thanks in advance!