Scalide

Simplified case class Generator for Scalacheck

2010-06-15T18:51:00.000-07:00

I just wrote a fairly handy generator for making very simple vanilla case classes. It basically uses the apply method defined in the companion of them to make things pretty simple. I am not sure if something similar is in the library already.

basically you can just type in your specification.

implicit val arbPoint = ArbFunc(Point.apply _)

and then arbitrary points will be provided as needed

If I posted the code here i have feeling it would get severely mangled so I put it in a gist:

http://gist.github.com/440002

I think it is a simpler way to make the generators for case classes where you don't care that much about constraining the members of the case class.

posted to the mailing list, waiting for moderation.

Working around Dependent Method Types with Structural Types

2010-06-14T13:45:00.000-07:00

Until dependent method types move out of experimental status in the scala compiler (they can be enabled with -Xexperimental) we have been using Structural Typing to work around their absence.

Here's how were doing it:

First, Given:

trait AToB {
    type B
    def b : B
}

and an implementation:

trait AToInt extends AToB {
    final override type B = Int
}

object AToInt {
    def apply(x : Int) = new AToInt {
        final override val b = x
    }
}

We would like to write a function with given any AToB
returns the type of B specified by that A:

if we had dependent method types we could write this as

def bOf[A <: AToB](a : A) : a.B = a.b

however this gives the error:

error: illegal dependent method type
          def bOf[A <: AToB](a : A) : a.B = a.b

To work around this however and still preserve type safety we have been doing this:

def bOf[AA <: AToB {type B = BB}, BB](a : AA) : BB = a.b

Basically we are using a structural type to fix the abstract type B in AToB to a concrete
type BB. This will enforce the type constraint that we want. There is one annoyance that remains
however and I haven't figured out a solution for it yet, the compiler is inferring the wrong type
for BB.

val x : Int = bOf(AToInt(5))

gives the compiler error:

error: inferred type arguments [java.lang.Object with AToInt,Nothing] do not conform to method bOf's type parameter bounds [AA <: AToB{type B = BB},BB]

yuck, the work around is to say:

val x : Int = bOf[AToInt, AToInt#B](AToInt(5))

Which will compile correctly.

A few more examples which will not compile:

trait AToString extends AToB {
    final override type B = String
}

object AToString {
    def apply(x : String) = new AToString {
        final override val b = x
    }
}


//Will not compile
val x : String = bOf[AToInt, AToInt#B](AToInt(5))
//Will compile
val x : String = bOf[AToString, AToString#B](AToString("Good Times"))
//Will not compile
val x : Int = bOf[AToString, AToString#B](AToString("Good Times"))

Pot of coffee gone, Intellij + Scala + sbt + bash-support plugin = Continuous Unit Testing In The IDE

2010-02-03T00:58:00.000-08:00

Got it working, sweet, no more neck strain looking back and forth from the terminal on every error, just their just a quick press away, don't even have to move the mouse and Intellij uses next occurrence to traverse between stacktrace elements in the active console.

Here's the patch for sbt it anyone is interested.
http://gist.github.com/293488

bedtime...

(previous post: http://scalide.blogspot.com/2010/02/trying-to-get-sbt-to-run-in-intellij.html)
(bash-support plugin for intellij http://plugins.intellij.net/plugin/?id=4230)
(mailing list posting http://groups.google.com/group/simple-build-tool/browse_thread/thread/a0cdaccc76c2384a)

Trying to get sbt to run in intellij with navigable errors and continuous compiles.

2010-02-02T22:34:00.000-08:00

So close, time to brew another pot of coffee, roll up my sleeves, and get to hacking!

http://groups.google.com/group/simple-build-tool/browse_thread/thread/a0cdaccc76c2384a

The dream of a 95% of the way there realtime compilation / unit test runner for scala in intellij is soooo close!

A Better Scala Enumeration Type

2010-01-20T16:11:00.000-08:00

I am going to post a link to the Enum type that I wrote due to the limitations of the builtin Scala Version. I don't use the built in enumeration type anymore as I find it very limiting.

(source & test / example):
http://gist.github.com/282446

Originally I wrote this as answer to this question on StackOverflow, however felt that it would be useful here too.

Follow up to Product.hashCode collisions

2010-01-05T23:46:00.000-08:00

After some experimentation here is the result that I came up. Note that you might need to increase the Heap Space to get it to run in the interpreter because of sort / removeDuplicates.

I posted it as a gist [http://gist.github.com/270117]

You should be able to just copy and paste it into a scala 2.8 interpreter
This yields a result of:

(ideal = 1000000.0, actual = 999880, uniqueRatio = 0.99988)
meaning a collision rate < .1%

Possible Collision Issue with Product.hashCode

2010-01-05T22:02:00.000-08:00

First Off I am no expert at hashing algo's & perhaps this is some sort of Birthday Problem so I am not sure if what I am pointing out is actually a big issue. It would be interesting to see what would happen on a project that uses large hash maps with Products in it if the distribution at the bottom was improved.

Quick Summary:
Right now it seems like the hashCode for Products of Arity N is the same regardless of the actual class implementing it & that it doesn't distribute very well:


case class Point(x : Int, y : Int)
case class Point2(x : Int, y : Int)
(1,2).hashCode == Point(1,2).hashCode == Point2(1,2).hashCode == 3405


Welcome to Scala version 2.8.0.r20331-b20100101020206 (Java HotSpot(TM) 64-Bit Server VM, Java 1.6.0_15).
List((1,2), (2,1), Point(1,2), Point(2,1), Point2(1,2), Point2(2,1)).map(_.hashCode)
res0: List[Int] = List(3405, 3445, 3405, 3445, 3405, 3445)

In a little more depth: I think this behavior is much better than in 2.7.5 where the ordering of the product elements did not affect the hashCode. (Point(x,y) would clash with Point(y,x))

in 2.7.5:


res0: List[Int] = List(1838716253, 1838716293, -1163646902, -1163646862, -270917823, -270917783)

Digging a little further on what this means for a distribution =>

Keep in mind that a big problem is that java's Integer.hashCode == the number being hashed, which is going to cause problems.

Using this code to get rudimentary collision distribution:


val tot = 500
val ideal = tot * tot * 1.0
val hashes = (for(x <- -(tot/2) until (tot/2); y <- -(tot/2) until (tot/2)) yield (x,y).hashCode ).toList
val hset = Set() ++ hashes
val hsSize = hset.size 
val hlSize = hashes.removeDuplicates.size
println((ideal, hsSize, hsSize/ideal, hlSize, hlSize/ideal))

Here is the 2.7.5 result (takes a few minutes on a Core i7 920, removeDuplicates is terribly slow):


(250000.0, 20959,0.083836,20959,0.083836)

and Here is the 2.8 nightly result (surprisingly the same distribution, that takes only a few seconds):

(250000.0, 20959,0.083836,20959,0.083836)

Both cases are going to collide 90+% of the time (notice that changing tot to 100 reduces the collision % quite significantly so I suspect there is a quadratic relationship at play here)

Running with a tot =100 gives:
(10000.0,4159,0.4159,4159,0.4159)
Still a 58% collide rate, but a lot better than the larger case.

Finally for the 3 Tuple cases with tot = 100
(1000000.0,170578,0.170578,170578,0.170578)

Perhaps I should look into a better case than just integers to see what the collision rates actually are like.

The Magic Mixin, Simple Version

2009-12-02T02:14:00.000-08:00

For boilerplate equality of classes, where the performance of equals is not all that important. I use a simple trait that I implement a few methods from. It's pretty handy so I figured I would post it.

Check out the Gist

Updates on the Intellij Plugin, Error Console is Less Grabby

2009-07-10T14:22:00.000-07:00

It's still a work in progress but I have managed to make the errors window less grabby. Right now it prints some harmless messages to standard out, just ignore them.

As before:

If you are an Intellij user and want to check out the modified plugin, here is the source and here is a download of a jar that you can drop in your /plugins folder

Intellij Plugin

2009-07-09T22:24:00.000-07:00

I recently switched to using Intellij 8 for scala development from Eclipse. On the whole it is a lot more polished than the current incarnation of the Scala IDE for Eclipse, I recommend it highly. Out of the 3 big java IDEs Intellij has the best Scala support at the moment. I will try out the Scala IDE for Eclipse again, once Miles gets done with all the refactoring he is doing.

One of the features I missed most from Eclipse was how eclipse compiles automatically in the background when files are saved. Luckily there is a plugin for Intellij (mentioned in this faq) called EclipseMode to add that behaviour, though it has a few kinks. [1]

I wasn't able to fix all of the issues I have with it (yet), however I did make one simple change that adds a toggle for it (called Compile On Save) to the build menu.

If you are an Intellij user and want to check out the modified plugin, here is the source and here is a download of a jar that you can drop in your /plugins folder

[1] kinks
One caveat, as others have mentioned: Is there any way to keep the messages window from auto-opening and from stealing focus? I tried mucking around with the plugin source, and couldn't figure it out, seems that problems view is not part of the openAPI and very hard to get to.

Also it doesn't seem to work with intellij 9, (tho i was able to hack it to get it to work, seems like they changed some of the non open api stuff the plugin uses).

GetAnyClass Tip

2009-06-04T01:47:00.000-07:00

I've been having some trouble with calling getClass on generic/primitive types.
It seems that it's a known issue with scala that wontfix and hence this code won't compile

def convert[A,B](instance : A, toClass : Class[B]) : Option[B] = {
//Fails here with the error below
instance.getClass
...Rest of implementation...
}

(The error):

:6: error: type mismatch;
found : A
required: ?{val getClass: ?}
Note that implicit conversions are not applicable because they are ambiguous:
both method any2stringadd in object Predef of type (Any)scala.runtime.StringAdd
and method any2ArrowAssoc in object Predef of type [A](A)ArrowAssoc[A]
are possible conversion functions from A to ?{val getClass: ?}
instance.getClass
^

the problem has a quick fix though suggested by paulp on the #scala irc channel on Freenode:

you can encapsulate that though:
scala> def getAnyClass(x: Any) = x.asInstanceOf[AnyRef].getClass
getAnyClass: (x: Any)java.lang.Class[_ <: java.lang.Object
casting to AnyRef (for this purpose) is a no-op on non-primitives, so...
you could plant that as an implicit if you want so 1.getAnyClass will work.

what this translates to is basically defining a class like this somewhere in your project:

/** Predef for brewmaster, maybe package objects will obsolete this
*/
private[brewmaster] object Brewdef {
implicit def any2GetAnyClass(x : Any) = new GetAnyClass(x)
class GetAnyClass(val x : Any) {
def getAnyClass = x.asInstanceOf[AnyRef].getClass
}
}

then chaning the code above slightly to use the .getAnyClass method it with

Publish Post

import Brewdef._
def convert[A,B](instance : A, toClass : Class[B]) : Option[B] = {
instance.getAnyClass
...
}

voila!
this will also allow things like 5.getAnyClass to compile (.getClass fails)

Immutable Objects and Named Parameters in scala

2009-06-01T12:30:00.000-07:00

--Updated--
Jorge Ortiz pointed out that 2.8 is already going to include a copy method in every case class:

Every case class already comes with a "copy" method defined something like:

def copy(x: Int = this.x, y: Int = this.y, z: Int = this.z) = {
Point3d(x, y, z)
}

And used like:

p1.copy(z = 2)

-Jorge Ortiz

Here's a cool way the named parameters feature being added in Scala 2.8 might help with immutable objects. It should help clients avoid errors when copying an immutable class, also it should end up hiding them from certain changes in that class.

//An overly simple example class
case class Point3d(x : Int, val y : Int, val z : Int) {
//Add method like this to make copying immutable classes easier
def copy (x : Option[Int] = None, val y : Option[Int] = None, val z : Option[Int] = None) = {
Point3d(x.getOrElse(this.x), y.getOrElse(this.y), z.getOrElse(this.z))
}
}

//Original class
val p1 = Point3d(1,1,1)

//"Old Way"
//notice every field is copied, this
//gets nasty as the class acquires more than just 3 fields
//Also notice how directly tied to the factory method
//this style is
val pOld = Point3d(p1.x, p1.y, 2)

//"New way"
//here we only care about the part of the
//class that we want to change. The class we are
//changing was refactored and additional fields were
//added to it, we would have to change all the "Old Way"
//calls, however this call will still work fine.
val pNew = p1.copy(z=Some(2))

Hashcode & Equals in the Scala 2.8 Collections Library

2009-05-18T17:32:00.000-07:00

[Edit]
Martin commented that the disparity between equals and hashCode has been changed on the Scala 2.8 trunk so that they now follow the hashCode-equals contract, therefore most of this post is irrelevant.

I just finished reading the whitepaper on the collections library that is going to be put into scala 2.8

First let me say that I really enjoy the new collections library, it seems like it is going to be a vast improvement over the old, and give scala one of the nicest, most growable collections libraries of any language out there. I really think that the new library with how well abstracted everything is will be a really cool starting point for allowing users to grow the language and give Scala what will in a very short time could be the end-all be-all of collections libraries.

Having just watched Guy Steele's talk on Growing A Language (thanks to debasishg@twitter) for tweeting it. It seems like Martin Odersky and the other architects of the new library really channelled what he was getting at in his talk. This new library while being fairly complete in it's own right, is going to give users a both the flexibility to implement collections that are highly optimized and/or the ability to implement them very quickly by reusing existing code. It will let the vendor of the Collection class focus more time on his implementation and less time on having to provide a bazillion methods that clients reasonably expect from a collection class.

There was a one thing that jumped out at me as being a bad idea in the new library, it was on page 15 and relates to how the new collections library is going to handle hashCode for mutable collections, and equals for all collections.

Per the whitepaper:

It does not matter for the equality check whether a collection is mutable or immutable. For a mutable collection one simply considers its current elements at the time the equality test is performed. This means that a mutable collection might be equal to different collections at different times, depending
what elements are added or removed. This is a potential trap when using a mutable collection as a key in a hashmap.
Example:
val map = HashMap[Array[Int], Int]()
val xs = Array(1, 2, 3)
map += (xs -> 6)
xs(0) = 0
map(xs)
In this example, the access on the last line will most likely fail because the hashcode of the array xs has changed in the second-to-last line. Therefore, the hashcode-based lookup will look at a different place than the one where xs was stored.
To avoid this trap, mutable collection classes will not support taking their hashCode—they will throw an UnsupportedOperationException with the message “unsuitable as hash key” if hashCode is called

I have a very bad feeling about any class throwing an exception when you ask it for its hashCode, the same way I would have a bad feeling if a class throw an exception just for calling its toString method, or its equals methods. I am generally not a big fan of surprise exceptions and while I am well aware of the dangers of using mutable field in a hash map, I just don't think that the hashCode method should be throwing exceptions. It's going to be a real gotcha for any programmer.

Programmer add gotcha to brain: "I can generally call hashCode on any object, except for this sub-type of objects, which are going to throw an exception at me because it is dangerous to use them in hashmaps. If I am writing generic code that calls hashCode on object I need to catch exceptions, because, certain objects decide they really did not want to ever be used in a hashmap, even though my code doesn't mutate them afterwards, as I am just sending these objects right out of the system for serialization and I need a good key to use for them that obeys the contract of hashcode. In this case, I am going to be in trouble as there is now no longer a guaranteed way for me to get hashCode that obeys the contract of hashCodes/equals. What do I do?"

Further, If a mutable collection is excluded from being used as a key because it could be mutated after being added to a map, shouldn't any 'case class' with a var field also be Unhashable? (A Case Class in scala provides some syntatic sugar by auto-implementing the equals and hashCodes methods based on the fields of the class.) Isn't Unhashable really just a synonym for Mutable? And is the real issue here forcing all client's to never use a mutable value as a key in a hashmap? And shouldn't then any mutable class throw UnsupportedOperationException when the hashCode of that class is called? Is it really a good idea to enforce this constraint this hard?

Having said that, and given the subtlety of the errors that happen when programmers do accidentally use mutable values as hash keys, then change them later on, I think that making some sort of bell go off that says: "Warning, you just used a mutable value as a key in a hashmap! I hope you know what you're doing, because you probably just messed up" is a good idea.

Another thing that scares about the 'This is Unhashable because it is mutable ideology' is that it is going to force everyone to implement their own hash functions for objects that are mutable, or always first coerce them into an immutable version just to get a hashCode (with possible severe performance impacts), if they need to for some reason get a hashCode for that class (maybe for some sort of db serialization scenario, who knows). Since their is no integrated way to get a hashCode for a mutable collection, they now are going to have make their own hashing function, and that is generally a bad idea, as making good hash functions can be hard. It's better to do it right once in the library than to have everyone do it their own way. Can it be said with absolute certainty that just because something is mutable, it must never generate a hashCode for any reason?

It would be really nice if it could be enforced by the compiler as a constrait on the type parameters of HashMaps, namely the Key type of a hashmap should be able to somehow say something like:
HashMap[K <: !Unhashable, V]
where K <: !Unhashable means K is not a subtype of Unhashable, and the compiler enforces the constraint, rather than the runtime.

The other thing that stuck out was a potential gotcha around the way equality is handled.

The Whitepaper on Equality:

The collection libraries have a uniform approach to equality and hashing. The idea is, first, to divide collections into sets, maps, and sequences. Collections in different categories are always unequal. For instance, Set(1, 2, 3) is unequal to List(1, 2, 3) even though they contain the same elements. On the other hand, within the same category, collections are equal if and only if they have the same elements (for sequences: the same elements in the same oder). For example, List(1, 2, 3) == Array(1, 2, 3), and HashSet(1, 2) == Treeset(2, 1).

From above: seems that a List(1,2,3) will now equal an Array(1,2,3) while they have different hashCodes (List is some defined value, and Array is Nothing because it throws an exception) which violates the contract of hashCode. Maybe this doesn't matter that much in this case because the mutable collections can never be used in a hashmap, but what about two different implementations of an immutable collection within the same category? If this is the direction being taken, then clients probably should never should override equals or hashCode for their own implementations of collections. Conceptually equals and hashCode should be final at the level of Map, Set, and Sequence otherwise any client who writes their own version of a collection inheriting from one of these has to ensure that they generate the same hashCode for all other collections of the same category that contain the same values or else they risk breaking the contract put forth in the whitepaper.

Another approach might be to just make a method in Sequence, Map, & Set called sameElements (or deepEquals or equalElems or whatever) that returns true iff these two collections are of the same category and contain the same elements.

I am not really sure which is preferable, on the one hand I like the idea of treating all sets, maps, and sequences uniformly, in the sense that if they have the same values, then they are the same, on the other hand, It still would be nice to quickly tell if certain maps have the same ordering as well they are iterated across (I suppose that something like this could probably do that as well):
pseudocode
def sameOrder = (map1.iterator zip map2.iterator).find((e1, e2) => {e1 != e2}).isEmpty

Planning to do a scalide screencast sometime

2009-05-07T20:30:00.000-07:00

I am hoping to learn how to record a brief 5 minute screen cast in the next few weeks to walk through some of the features of the interpreter. I know that the user interface leaves a lot to be desired and I haven't had much time to work on it, however hopefully I can explain it better for people new to Scala as it can be a good way to help learn the language. (I use it as a calculator now). To people who have used mathematica before the interface might be only extremely horrible, for everyone else it's probably entirely unusable ;-)

I am also going to look over the recent changes that have been checked into the scala tools package by extempore to see if I can add some auto-complete support which would make the interpreter much more usuable, stay tuned :-)

New Scalide Release

2008-09-30T11:54:00.000-07:00

I released newer updated version of Scalide a few weeks ago with a newer version of the Scala interpreter so check it out and let me know what you think.

Another update

2008-05-19T23:01:00.000-07:00

Probably the last one of the night, just clarified a few things in help.
http://scalide.googlecode.com/files/scalide-2008-05-20_00-59-10.jar

Another update

2008-05-19T22:39:00.001-07:00

Fixed a few more bugs and added the ability to erase cells

http://scalide.googlecode.com/files/scalide-2008-05-20_00-36-12.jar

Updated Release Tonight

2008-05-19T18:59:00.000-07:00

I just released a new version that adds several minor enhancements and fixes some hang on input issues.

Check it out here:
http://scalide.googlecode.com/files/scalide-2008-05-19_20-55-32.jar

New Version With Saving/Loading

2008-05-15T22:38:00.000-07:00

I just finished a new version that can save and load interpreter sessions.

Check it out here:

http://scalide.googlecode.com/files/scalide-2008-05-16_02-05-36.jar

First post

2008-05-15T06:53:00.000-07:00

I am going to use this blog to discuss topics related to the Scala programming language and my pet project I am using to learn called Scalide which is a visual front-end to the Scala interpreter. Check it out at http://scalide.googlecode.com and discuss it at http://groups.google.com/group/scalide