Derek Wyatt's Blog

I’ve been working on a book since March, 2012 called Akka Concurrency and on Monday, Oct 15th 2012, it went live at Artima. I’m really happy to see it make this milestone – it’s been a long time coming!

The book builds on the existing, and excellent Akka Documentation by going beyond the what and the basic how of Akka and into the why, as well as covering some more of the esoteric bits of the how as well. It’s a unique pleasure to see the wonderful documentation that the Akka team has developed. If they hadn’t done such a great job, I believe that I wouldn’t have wanted to write this book in the first place.

I hope that the Akka community finds it helpful and it brings more developers into this fantastic toolkit. Please head over to Artima and have a look at the book. Take a look at the table of contents, as well as the sample chapter and, if you like it, you just might want to grab a copy!

I’d like to say that I write useful code all the time, but let’s face facts… useful code doesn’t come around all that often. In my new job I’m using Maven instead of SBT and I miss the cool tilde modifier in a big way. And now that I’m using the Artima book building strategy – which is based on Ant – I miss it even more.

So I finally wrote this little script that’s saving my ass.

I tend to use it in a number of different ways, but one of the simple ones is like this:

And that’s all she wrote.

A little while ago, I gave up on Zsh. There were a number of people that showed me how I could have fixed up Zsh so it would suck less, but I just didn’t have time. Well, I found myself with some time yesterday that couldn’t be spent doing much useful, so I made the switch.

It now feels like Bash but a little bit better, so I’m pretty happy. Thanks to @datanoise, and others for the tips.

Here’s what I did to make it useful:

• unsetopt nomatch: I have no idea why this defaults to “on”. It’s useful about %0.00001 of the time and a pain in the bloody ass the other %99.99999 of the time.
• Use bindkey -v instead of the vi-mode plugin: This is also odd. Most of what is needed (the prompt stuff) is pointless eye-candy and the zle reset-prompt is just an awful feature. You hit ESC-\ and it deletes the line above… which a lot of the time, carried data that you really wanted to read.
• Ported my directory stacking stuff to Zsh (just barely :D). I can’t leave this feature behind… without it, running around the filesystem is a major drag.

My config can be found on GitHub if you’re interested in having a look.

I recently enjoyed a couple of days with Heiko Seeberger and Josh Suereth as they dragged me through some Advanced Scala Training and it was quite awesome. As an exercise, to test out my new found skills I pimped a new method onto “numbers” in Scala and I thought it might be instructive to share.

The Goal

The goal is really quite simple. All we want to do is add a new method, squared onto “numbers”. I don’t mean Int directly, but anything that Scala considers a number. We should see things like this:

5.squared    // == 25
25.0.squared // == 625.0

The Code

The code is wonderfully simple for this.

implicit def numeric2Powerable[A : Numeric](i: A) = new {
  def squared: A = implicitly[Numeric[A]].times(i, i)
}

Done. What we’ve managed to do in such a small amount of code is quite surprising. Let’s break it down.

• implicit: The implicit puts the conversion definition into the local scope so that we can get a transparent conversion from our number to our pimped object.
• [A : Numeric]: This is a context bound. It states that the type parameter we’re going to constrain this implicit conversion to must have an instance of the Numeric type class in scope. It’s this constraint that limits the types we’re pimping down to being numbers.
• implicitly[...]: Because we’ve chosen the “clean” declaration for the context bound (as opposed to the slightly uglier curried parameter version) the instance of the type class has not been bound to an identifier that we can later use. In order to get access to the instance, we must “look it up”. Because scala.Predef is always imported by default, and because implicitly is defined there, we have access to it, and this is what we use to resolve the implicit instance.

The reset is self-explanatory.

That’s all there is to it. We’ve defined an implicit conversion from a number to a new pimp-class, used a type class to narrow the target types for the pimp, and pulled in the instance of the type class to give us a concrete tool to use in performing the computation. Not bad for a couple of lines of code.

Addendum: Missingfaktor showed me another little fun bit that you can add on to the code above to make the implementation feel more natural. We can import the implicits that are defined on Numeric to give us some infix notation instead.

implicit def numeric2Powerable[A : Numeric](i: A) = new {
  import Numeric.Implicits._
  def squared: A = i * i
}

It doesn’t change the API at all, but it does look a little niftier with respect to the implementation. It’s a little less simple, from an instructive point of view, because you start to wonder where the implicitly wandered away to. So, some more explanation is probably in order.

Specifically, what we’re importing is:

object Numeric {
  trait ExtraImplicits {
    implicit def infixNumericOps[T](x: T)(implicit num: Numeric[T]): Numeric[T]#Ops = new num.Ops(x)    
  }
  object Implicits extends ExtraImplicits { }
}

So, it’s really the infixNumericOps that we’re pulling in, and it’s delegating to Ops. So, what’s Ops?

Ops is defined as an inner class to the Numeric type trait and looks like:

class Ops(lhs: T) {
  def +(rhs: T) = plus(lhs, rhs)
  def -(rhs: T) = minus(lhs, rhs)
  def *(rhs: T) = times(lhs, rhs)
  def unary_-() = negate(lhs)
  def abs(): T = Numeric.this.abs(lhs)
  def signum(): Int = Numeric.this.signum(lhs)
  def toInt(): Int = Numeric.this.toInt(lhs)
  def toLong(): Long = Numeric.this.toLong(lhs)
  def toFloat(): Float = Numeric.this.toFloat(lhs)
  def toDouble(): Double = Numeric.this.toDouble(lhs)    
}

So, when we stitch this together, here’s what happens:

1. We bring infixNumericOps into scope.
2. We then invoke i * i, which is equivalent to i.*(i).
3. The * can’t be found on i since i is of type A.
4. However, now that we have infixNumericOps in scope, the compiler can use that conversion to create an instance of Numeric[A]#Ops.
5. Numeric[A]#Ops does have * defined for type A, thus, a new Ops is constructued, holding our value i within it.
6. The compiler then applies the * operator to to the second value of i and we’re done.

Cooler, but a hell of a lot more complicated to explain :)

I recently responded to someone’s question on StackOverflow and now I have to rant about something.

There is a large problem I find with programmers that I meet (generally in Real Life, as opposed to on the ‘net because on the ‘net I associate with passionate programmers) when it comes to learning new “stuff”. The first thing they do is try to draw it close to something they already understand. For example, they pick up Python and try to relate it to something like Java. This is not a problem.

The problem comes from the fact that it’s not just the first thing they do, it’s also the last thing they do. Why is this a problem? Oh for so many reasons…

It’s new

Relating this ‘new’ thing to something you are familiar with is vital in the first stages in order to get a foothold on a new experience. But once you’ve got that, you need to forget it.

Let’s take a new language as an example. If you’re a Java coder and you’ve never coded in Ruby before, then when you see Ruby you’re going to be looking for objects. This is all well and good, and it gets you into the language so you can start expressing your ideas. Awesome. But if Ruby was just a scripted Java then it wouldn’t have a lot of use, would it?

What you’re going to miss with Ruby (among about a million other things) is that Ruby is very much about functions and not so much about objects (Ok, I don’t want to get in a big debate with people here… I get it, it’s about objects too… and a lot of other stuff). And missing this means you’re going to miss out on a solid opportunity to enhance the way you think about designing and writing software.

Languages are not nearly as important as the paradigms they help you express. All we do for a living is think and the better you can think about your problems and their solutions, the better you are at what you do.

It’s not that other thing…

If the “other thing” you’re used to is worth anything at all, then it’s going to have some “good points” to it – it damn well better have. The “new thing” is going to have good points too, and they’re going to be different than those in your wheel house. If you’re used to feature X, chances are you’re not going to see feature X in this new thing. Why? It’s not that other thing. Seems obvious, right? If it’s so obvious then why do I see so many people not getting it?

Here’s an example. There’s a colleague of mine that loves Erlang – great language, and it’s awesome to love it. When talking about Akka, he asked me whether or not you could hibernate an Actor (closest equivalent to an Erlang ‘process’ as far as our discussion went). I said “no”. I was immediately met with statements like, “It must have that or it’s no good,” and stuff like that. What he didn’t grasp, until explained, was that Akka is modeled completely differently and has no need of the hibernate functionality. This is a fundamental difference between the two designs – and there are definite trade-offs being made for many different reasons. The point I’m making here is that, unless I explained it, he would have written off the technology based on a fabricated fantasy.

If you expect a new thing to be exactly the same as the old thing, then why would you look at the new thing? It’s not that other thing!

It misses the whole point

I’ve already said this earlier but it’s a very general idea. How many ways are there to solve problems in our industry? It’s pretty much infinite, right? I can select any language I want, for starters. I can then choose any tool within that language, or I can choose another language and bolt that on to help solve the problem. I can use existing programs or libraries from anywhere else to do it. And I haven’t even started coding yet. Once I start coding, I can code in nearly any paradigm I want to, or any hybrid set of paradigms that I want to (made easier by a multi-paradigm language of course).

So, let me ask you this: If all you know is one language, and one set of tools, and you view every other language and every other tool through the foggy lens of “what you already know”, then how many ways are there to solve problems when you’re the one solving it? Not many, right? You’ll probably come up with solutions, but if they suck, you’re stuck – you have no other tools in your head to help solve these things. You’re stuck with sucking. Not awesome.

The whole point of learning new things is to learn new things. Please, if you find yourself looking at something and scoffing and saying “Well, it’s not like this other thing that I’m comfortable with”, then say “GREAT!” If you find yourself outside your comfort zone then you’re doing it right.

If it were comfortable and familiar then you wouldn’t be learning, and if you’re not learning, then stop what you’re doing and go back to doing something productive with what you already know. Save that time for finding something to learn that you’re really not comfortable with.

Ok, so everyone and their grandmother appears to be using Zsh and I was starting to wonder what was wrong with me. Why was I still using Bash? So I grabbed Oh-My-Zsh (which was pretty cool, BTW) and dove in… but here’s the problem – I didn’t have time to dive in.

I’m sure that, had I days to dedicate to getting Zsh working, that I would probably find the “right” way to use it, and make myself happy. But, given that I just wanted to start using it and figure things out later… well, it absolutely f’ing sucked! I got no benefit from this approach and a lot of pain. Here’s why it sucked:

• In bash, if I type — grep thing.*other myfile.txt — and — thing.*other — doesn’t glob, do you know what I get? That’s right, I get — thing.*other. With Zsh, I get an error.
• Same thing for — sbt ~test. Blurgh!
• I like a two line status. It keeps all of that long path information, git stuff, and whatnot on a line by itself, and then gives me a whole fresh line for writing commands. I get this in Bash, and I get this in Zsh. Difference? Every time I hit — <ESC> k — in Zsh, it would erase the line above it – a rendering bug. You know what? I really needed to read that line, thanks very much.
• The directory stacking is pretty terrible, which is insane considering that “cd” is the command you probably run more than any other in the shell, by far.

There were some other annoying bits too, but not too big of a deal to mention.

The bottom line here, is that before I would be able to successfully use Zsh, I would have to figure out how to get it to stay the hell out of my way. Said another way: If there’s a feature of Zsh that I only use 0.5% of the time, then it should not be defaulted to be on 100% of the time such that 99.5% of the time I have to protect myself from it.

I know, I know… I didn’t give it a chance. Very true. What I was expecting was that I could use it and access features when I was willing to head down that path, but what I found was that I spent nearly all my time working around annoyances just to get things done.

I’ve recently been devouring Learn You a Haskell for Great Good. I’ve been meaning to do it for ages, but a recent post by Debasish Ghosh pushed me to go through it cover to cover… I’m almost there :)

I was wrestling over the complexities of applying the sequence function to a list of functions. More succinctly, I was confused on the mechanics of how this worked:

ghci> sequence [(> 4), (< 10), odd] 7
[True, True, True]

To figure it out, all you have to do is reduce it, and then reduce it, and then reduce it some more. Let’s do that.

First, sequence is defined as (I’m using the definition of sequenceA from LYAH, as opposed to the definition of sequence from the Haskell API):

sequence :: (Applicative f) => [f a] -> f [a]
sequence = foldr (liftA2 (:)) (pure [])

And the definition of the Applicative Functor instance for functions is:

instance :: Functor ((->) r) where
    pure x = (\_ -> x)
    f <*> g = (\x -> f x (g x))

Now, let’s expand (note, I’m not going into nauseating detail here, and assume you’ve gone part of the way thus far):

sequence [(> 4), (< 10), odd]
-- becomes
foldr (liftA2 (:)) (pure []) [(> 4), (< 10), odd]
-- which becomes, due to the definition of liftA2
((:) . (> 4)) <*> (((:) . (< 10)) <*> (((:) . odd) <*> (pure [])))

Ok, so we’ve expanded the foldr and now we can expand the <*> and the pure []:

(\x -> ((:) . (> 4)) x (
    \x -> ((:) . (< 10)) x (
        \x -> ((:) . odd) x ((\_ -> []) x)) x) x)

Ok, that’s a bit rough to read… the key to it is noting how the 'x' gets propagated. Let’s take another look:

When we pass in the 7 it gets propagated to all of the little x‘s, and all of those little x‘s get passed to their composed functions, and each of those results in a new element on the list:

(\x -> ((:) . odd) x ((\_ -> []) x)) 7
-- reduces to
((:) . odd) 7 (\_ -> []) 7)
-- which reduces to
(: True [])
-- which becomes
[True]

-- Now, we use that to reduce the next level up
((:) . (< 10) 7) [True]
-- which becomes
(: True [True])
-- which becomes
[True, True]

-- And now our last level
((:) . (> 4) 7) [True, True]
-- which becomes
(: True [True, True])
-- which becomes
[True, True, True]

Ah, that’s better… now that I ‘get it’, I can move on.

Cheers!

A couple of people pointed out that my posting about why Monads are so awesome left a bit too much to the imagination. I tossed in an update that should make it less mysterious about what the heck I’m talking about.

Update: In the comments below, Alex is trying to educate me on something… I’m still not entirely sure what he’s on about but it’s been keeping me thinking about this post. I’ve modified it a bit here and there to try and be more correct. Essentially, I think I’ve been attributing too much to Monads where only part of what I like is attributed to them where the other part is attributed to partial functions (or function composition).

Ok, I’m no Monad expert by any stretch of the imagination… seriously. But, I did something today that has me so jazzed about Monads, partial functions and Scala in general, that I just gotta share.

So my task was to take some JSON and convert it to a structure of my own classes, but still really general – i.e. we’re not talking about “real” marshalling here.

So I took a simple approach, and used the Scala Library’s JSON Module. I had to deal with type-casting, defaulting of values, and constraint checking (i.e. field X must be defined). The code to do all of this is so simple and so awesome that it’s just hard to believe. Now, it’s not perfect… I’ll shore it up over the next while as need requires… but it’s passing all of my tests pretty nicely right now.

I want to convert the following JSON:

{
  "vendor"  : "Whoever", // required
  "colour"  : "blue",    // default to "None"
  "widgets" :            // default to Empty list of strings
  [
    {
      "name"    : "One",   // required
      "flavour" : "Lime"   // default to Strawberry
    },
    {
      "name"    : "Two",   // required
      "flavour" : "Banana" // default to Strawberry
    }
  ]
}

to the following concrete Scala:

case class Widget(name: String, flavour: String)
case class Contract(vendor: String, colour: String, widgets: List[Widget])
Contract("Whoever", "blue", List(Widget("One", "Lime"),
                                 Widget("Two", "Banana")))

I used a Pattern Matching example for this sort of thing from an awesome StackOverflow post as a starter and then fleshed it out to meet my requirements. Here it is:

// Types we need
case class Widget(name: String, flavour: String)
case class Contract(vendor: String, colour: String, widgets: List[Widget])

object Contract {
  // The class caster that came from the StackOverflow page
  // Yeah, this can throw exceptions... I don't care about
  // those for this example
  class ClassCaster[T] {
    def unapply(a: Any): Option[T] = Some(a.asInstanceOf[T])
  }

  // Concrete instances of the casters for pattern matching
  object AsMap extends ClassCaster[Map[String, Any]]
  object AsList extends ClassCaster[List[Any]]
  object AsString extends ClassCaster[String]

  // It can happen
  case class ContractException(message: String) extends Exception(message)

  // Returns an Either that is a hell of a lot more deterministic than
  // throwing exceptions around
  def fromJSON(json: String): Either[ContractException, Contract] = {
    // Helps us get rid of boilerplate
    def error[T](prefix: String): Option[T] =
      throw ContractException("%s must be specified".format(prefix))

    // Catch the exceptions and turn them into Left() instances
    try {
      val parsed = JSON.parseFull(json)
      if (parsed.isEmpty) throw ContractException("Unable to parse JSON.")

      // Here comes the monad / partial function love...
      val contract = for {
        Some(AsMap(map))  < - List(parsed)
        AsString(vendor)  <- map get "vendor" orElse error("vendor")
        AsString(colour)  <- map get "colour" orElse Some("None")
        AsList(widgets)   <- map get "widgets"
      } yield Contract(vendor, colour, for {
          AsMap(widget)     <- widgets
          AsString(name)    <- widget get "name" orElse error("name")
          AsString(flavour) <- widget get "flavour" orElse Some("Strawberry")
        } yield Widget(name, flavour)
      )
      Right(contract.head)
    } catch {
      case e: ContractException =>
        Left(e)
      case e =>
        Left(ContractException(e.getMessage))
    }
  }
}

It’s very impressive what this code actually handles when you start digging into it. I’m sure you can easily think about what it would be like to code this in Java (or spaghetti monster forbid C++) and know that it’s going to be a lot more verbose and probably a lot less reliable.

UPDATE: One of the key reasons why the above is so cool is this:

<- map get "colour" orElse Some("None")
<- widget get "name" orElse error("name")

By using a generator syntax (<-) and the get function on Map we get to work at a higher abstraction that allows us to chain the decision making process rather than what the imperative approach would demand of us. It’s the monadic and compositional nature of Option that allows us to do this. Consider:

String colour = map.get("colour");
if (colour == null)
  colour = "None";

String name = widget.get("name");
if (name == null)
  error("name");

I’m going to editorialise (I mean, this is my blog after all) and point out the following problems:

• It’s unclear. Sure, if you’ve seen it a million times (and let’s face, you have… we all have) you know exactly what this “means” but that’s not because its meaning is clear.
• It’s not syntactically atomic. Some nasty merge, or some careless programmer can easily insert something between lines 1 and 2 that uses colour.
• It’s repetitive in a bad way (is there a good way?). That pattern is going to be copy-pasted over and over again. How many times do you see “colour” in there? Yeah… three. So if I copy-paste that and forget to change the third one, you get this:

String flavour = widget.get("flavour");
if (flavour == null)
  colour = "None";

GHACK!

We need to program at a higher level.

I saw a post on Supuser.com that asked about how to go “back” to a directory rather than “up”. The right answer was “cd -“, of course, but I’ve been using a home-grown tool to do this for years and it works really well.

It started out as a Korn Shell so there are some remnants of that in it, but it does work for Bash.

It does the usual thing of replacing the ‘cd‘ built-in command with a function by aliasing it away. Every time you change directories, it saves the last place on the stack (up to a max of 15). You can look at the directory history and switch to any of them based on their number and also a regular expression.

This sort of functionality, whether you use mine or someone else’s or build your own, is vital to efficient command line kung-fu. How the hell have you been surviving this long without it???

Here’s the Gist of the code that you can grab.

• cd – You know what this does. And, yes, both cd - and cd {pat} {subst} should also work just fine.
• ss – Short for “show stack”.
• csd – Short for “change to stacked directory”. This is the fun one that I’ll show in an example below.

Here’s how you use it (I’ll leave it to you to read what I’m doing and figure out how you can use it):

--(/home/dwyatt/scala/scala/src/library/scala/collection/generic)--------------------------------------
--> ss
1) /home/dwyatt/scala/scala/src/library/scala
2) /home/dwyatt/scala/scala/src/library/scala/concurrent
3) /home/dwyatt/scala/scala/src/library/scala/collection/mutable
4) /home/dwyatt/scala/scala/src/library/scala/collection/immutable
5) /home/dwyatt/scala/scala
6) /home/dwyatt/scala
7) /home/dwyatt
--(/home/dwyatt/scala/scala/src/library/scala/collection/generic)--------------------------------------
--> csd 4
--(/home/dwyatt/scala/scala/src/library/scala/collection/immutable)------------------------------------
--> ss
1) /home/dwyatt/scala/scala/src/library/scala/collection/generic
2) /home/dwyatt/scala/scala/src/library/scala
3) /home/dwyatt/scala/scala/src/library/scala/concurrent
4) /home/dwyatt/scala/scala/src/library/scala/collection/mutable
5) /home/dwyatt/scala/scala
6) /home/dwyatt/scala
7) /home/dwyatt
--(/home/dwyatt/scala/scala/src/library/scala/collection/immutable)------------------------------------
--> csd ent$
--(/home/dwyatt/scala/scala/src/library/scala/concurrent)----------------------------------------------
--> ss
1) /home/dwyatt/scala/scala/src/library/scala/collection/immutable
2) /home/dwyatt/scala/scala/src/library/scala/collection/generic
3) /home/dwyatt/scala/scala/src/library/scala
4) /home/dwyatt/scala/scala/src/library/scala/collection/mutable
5) /home/dwyatt/scala/scala
6) /home/dwyatt/scala
7) /home/dwyatt
--(/home/dwyatt/scala/scala/src/library/scala/concurrent)----------------------------------------------
--> csd 5
--(/home/dwyatt/scala/scala)---------------------------------------------------------------------------
--> ss
1) /home/dwyatt/scala/scala/src/library/scala/concurrent
2) /home/dwyatt/scala/scala/src/library/scala/collection/immutable
3) /home/dwyatt/scala/scala/src/library/scala/collection/generic
4) /home/dwyatt/scala/scala/src/library/scala
5) /home/dwyatt/scala/scala/src/library/scala/collection/mutable
6) /home/dwyatt/scala
7) /home/dwyatt
--(/home/dwyatt/scala/scala)---------------------------------------------------------------------------
--> cd ~
--(/home/dwyatt)--------------------------------------------------------------------------------------------
--> cd -
--(/home/dwyatt/scala/scala)---------------------------------------------------------------------------
-->