Recently, I invested some time into automating some of the work that goes into a Finatra release.

The work consists of updating:

• The version in the XML fragment of the main README.markdown

• The version in the pom.xml of the example app

• Any API changes in the example app

• The version in the template pom.xml of the app generator

• The generated unit test of the app generator that demonstrates testing any new API features

• Any API changes inside the app template of the app generator

Using sub, I was able to create a finatra command that automated all of the above based on a single template, which also happens to be the main unit test. This ensures that the README, the example app, and the app generator never fall out of sync with the frameworks API.

Last week we released 1.1.0, and the README was completely generated, as was the example app. Not to mention, all generated apps would also contain the latest templates and examples!

Let’s dive into how it all works:

The source of truth

I annotated our main unit test with special tokens, like so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

class ExampleSpec extends SpecHelper {

  /* ###BEGIN_APP### */

  class ExampleApp extends Controller {

    /**
     * Basic Example
     *
     * curl http://localhost:7070/hello => "hello world"
     */
    get("/") { request =>
      render.plain("hello world").toFuture
    }

  }

  val app = new ExampleApp

  /* ###END_APP### */


  /* ###BEGIN_SPEC### */

  "GET /hello" should "respond with hello world" in {
    get("/")
    response.body should equal ("hello world")
  }

  /* ###END_SPEC### */
}

Using the special /* ### */ comments, the main app and its test can be extracted from the code of our test.

The app generator

Now that we have our “template”, we can build our app generator to use it. I customized base and ended up with: script/finatra/libexec/finatra-new

You can then run:

1

$ ./finatra new com.example.myapp

and it will generate myapp/ based on the tested example code from the test suite above.

The example app

The example app is just a generated app using the latest app generator:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

#!/bin/bash
# Usage: finatra update-example
# Summary: generates the example app from the template

set -e

source $_FINATRA_ROOT/lib/base.sh

tmpdir=$(mktemp -d /tmp/finatra_example.XXX)

$_FINATRA_ROOT/bin/finatra new com.twitter.finatra_example $tmpdir

cp -Rv $tmpdir/finatra_example/ $EXAMPLE_REPO

rm -rf $tmpdir

cd $EXAMPLE_REPO && mvn test

This also tests the app generator and the generated app!

Updating the README

Lastly, there’s a command for updating the README with the new example and version number.

After months of work Finatra 1.0.0 is finally available! Finatra is a scala web framework inspired by Sinatra built on top of Finagle.

The API

The API looks like what you’d expect, here’s a simple endpoint that uses route parameters:

1
2
3
4

get("/user/:username") { request =>
  val username = request.routeParams.getOrElse("username", "default_user")
  render.plain("hello " + username).toFuture
}

The toFuture call means that the response is actually a Future, a powerful concurrency abstraction worth checking out.

Testing it is just as easy:

1
2
3
4

"GET /user/foo" should "responsd with hello foo" in {
  get("/user/foo")
  response.body should equal ("hello foo")
}

A super quick demo

1
2
3

$ git clone https://github.com/capotej/finatra.git
$ cd finatra
$ ./finatra new com.example.myapp /tmp

Now you have an /tmp/myapp you can use:

1
2

$ cd /tmp/myapp
$ mvn scala:run

A simple app should’ve started up locally on port 7070, verify with:

1
2

$ curl http://locahost:7070
hello world

You can see the rest of the endpoints at /tmp/myapp/src/main/scala/com/example/myapp/App.scala

Heroku integration

The generated apps work in heroku out of the box:

1
2
3
4
5

$ heroku create
$ git init
$ git add .
$ git commit -am 'stuff'
$ git push heroku master

Make sure to see the full details in the README and check out the example app.

Props to @twoism and @thisisfranklin for their code, feedback and moral support.

Finally got tired of copy pasting other projects and gutting them to make new ones, so I created base, a shell command that creates new scala projects.

Creating the project:

1
2
3
4
5
6
7
8
9
10

$ base new com.capotej.newproj
creating project: newproj
  creating App.scala
  creating AppSpec.scala
  creating pom.xml
  creating .gitignore
  creating .travis.yml
  creating LICENSE
  creating README.markdown
Done! run mvn scala:run to run your projec

Based on the package name, it infered that the project name is newproj and created the project under that folder. Let’s build and run it:

1
2
3
4

$ cd newproj
$ mvn compile scala:run
(... maven output ...)
hello world

This uses the new incremental compiler for maven, zinc, which dramatically speeds up compile times (except for the first time you run it). It also sets you up with the latest scalatest maven plugin, which gives you sweet looking test output, like so:

See the base README for installation instructions.

A few months ago, I gave a presentation on how Posterous uses Riak for it’s post cache; At #ricon2012 I ended up retelling this story to numerous people, so I thought I’d post the slides and video here.

UPDATE: this is now available as a sub command, here: kiev

Cooked this up last night when I needed a simple key/value store for use in a shell script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

#!/bin/sh

DBFILE=example.db

put(){
  echo "export kv_$1=$2" >> $DBFILE
}

del(){
  echo "unset kv_$1" >> $DBFILE
}

get(){
  source $DBFILE
  eval r=\$$(echo "kv_$1")
  echo $r
}

list(){
  source $DBFILE
  for i in $(env | grep "kv_" | cut -d= -f1 ); do
    eval r=\$$i; echo $(echo $i | sed -e 's/kv_//') $r;
  done
}

## cmd dispatch

if [ ${1:-0} == "set" ]; then
  put $2 $3
elif [ ${1:-0} == "get" ] ; then
  get $2
elif [ ${1:-0} == "list" ] ; then
  list
elif [ ${1:-0} == "del" ] ; then
  del $2
else
  echo "unknown cmd"
fi

Use it like so:

$ ./db.sh set foo bar

$ ./db.sh get foo

$ ./db.sh set foo baz

$ ./db.sh get foo

$ ./db.sh del foo

$ ./db.sh list

How it works

Every time you update/set/delete a value, it writes a shell expression to an append-only log, exporting a shell variable (key) with that value. By sourcing the file every time we read a value, we replay the log, bringing our environment to a consistent state. Then, reading the value is just looking up that dynamic variable (key) in our shell environment.

I’ve been fascinated by the concepts in finagle for some time, but being a scala noob, I never knew how to bootstrap a finagle project. Turns out twitter has a gem, scala-bootstrapper, that generates a simple thirft based key/value store for you. There’s even atutorial  on how to extend the example project into a distributed search service.

This is a guide on setting it all up locally, it assumes you have Git, Homebrew, and OS X.

Install scala 2.8.1

1
2
3
4
5

$ brew versions scala
$ cd/usr/local/(or wherever you have homebrew installed)
$ git checkout -b scala281 0e16b9d(make sure the SHA matches versions output)
$ brew install scala
$ git checkout master$git branch -D scala281

Install sbt 0.7.4 (assumes you have a ~/bin in your $PATH)

1
2

$ curl -O http://simple-build-tool.googlecode.com/files/sbt-launch-0.7.4.jar > ~/bin/sbt-launch.jar
$ echo 'java -Xmx1G -jar `dirname $0`/sbt-launch.jar "$@"'> ~/bin/sbt

Install scala-bootstrapper

1

$ gem install scala-bootstrapper

Generate finagle project**

1
2
3
4
5

$ mkdir newbird
$ cd newbird
$ scala-bootstrapper newbird
$ sbt update
$ sbt test

Add a Client class

create newbird/src/main/scala/com/twitter/newbird/Client.scala with

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

package com.twitter.newbird

import com.twitter.finagle.builder.ClientBuilder
import com.twitter.finagle.thrift.ThriftClientFramedCodec
import com.twitter.newbird.thrift._
import org.apache.thrift.protocol.TBinaryProtocol

import java.net.InetSocketAddress

class Client {  

  val service = ClientBuilder().hosts(Seq(newInetSocketAddress("localhost",9999)))
    .codec(ThriftClientFramedCodec())    
    .hostConnectionLimit(1)    
    .build()  
  val client = new NewbirdServiceClientAdapter(
    new thrift.NewbirdService.ServiceToClient(service,newTBinaryProtocol.Factory))  

  def get(key:String) = client.get(key)()  
  def put(key:String, value:String) = client.put(key,value)()

}

Running the server

1
2

$ cd newbird
$ sbt> run -f config/development.scala

Playing with the client**

1
2
3
4
5
6

$ cd newbird
$ sbt console
scala> import com.twitter.newbird.Client
scala> val client = new Client()
scala> client.put("foo","bar")
scala> client.get("foo")

Bonus

finagle exports a stats url you can curl:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

$ curl http://localhost:9900/stats.txt
counters:  Newbird/connects: 1  
Newbird/requests: 4  
Newbird/success: 4
gauges:  
  Newbird/connections: 0  
  Newbird/pending: 0  
  jvm_heap_committed: 588251136  
  jvm_heap_max: 2146828288  
  jvm_heap_used: 64354560  
  jvm_nonheap_committed: 83267584  
  jvm_nonheap_max: 318767104  
  jvm_nonheap_used: 68655360  
  jvm_num_cpus: 4  
  jvm_start_time: 1327511164928  
  jvm_thread_count: 14  
  jvm_thread_daemon_count: 9  
  jvm_thread_peak_count: 14  
  jvm_uptime: 2626505
labels:
  metrics:  
    Newbird/connection_duration: (average=2590412, count=1, maximum=2590412, minimum=2590412, p25=2590412, p50=2590412, p75=2590412, p90=2590412, p99=2590412, p999=2590412, p9999=2590412)  
    Newbird/connection_received_bytes: (average=192, count=1, maximum=192, minimum=192, p25=192, p50=192, p75=192, p90=192, p99=192, p999=192, p9999=192)  
    Newbird/connection_requests: (average=4, count=1, maximum=4, minimum=4, p25=4, p50=4, p75=4, p90=4, p99=4, p999=4, p9999=4)  
    Newbird/connection_sent_bytes: (average=120, count=1, maximum=120, minimum=120, p25=120, p50=120, p75=120, p90=120, p99=120, p999=120, p9999=120)  
    Newbird/request_latency_ms: (average=14, count=4, maximum=39, minimum=2, p25=2, p50=8, p75=10, p90=39, p99=39, p999=39, p9999=39)

While looking for a way to add wunderlist tasks via alfred, I came upon wunderlist-for-alfred

Looked cool, but I wanted to write my own that didn’t depend on php.

I used lsof to figure out the location of the db, then used file to see what kind of db it was. Luckily, it was sqlite3, so I was able to poke around and figure out the sql to create a task.

Here’s the alfred extenstion that ties it all together:

1
2
3

user=`whoami`
wunderdb="/Users/$user/Library/Wunderlist/wunderlist.db"
sqlite3 $wunderdb "insert into tasks (name, list_id) values ('{query}', 1)"

Download it here

Last night I delightfully discovered that Adium Message Styles are just html, css, and javascript rendered inside a webview. The next natural step was to write something in it, so I wrote a Message Style that tries to render any image link directly inline the conversation (campfire style).

The code was written at midnight after a long day, so its not best. Basically, it’s a setInterval that runs every 2.5 seconds that loops through all message elements, appending an img tag to the body of the message if an image link is detected. It also removes the processing class as to not reprocess the same messages.

Installation is simple, just download: 

http://dl.dropbox.com/u/42561/Stockholm.AdiumMessageStyle.zip

and extract into ~/Library/Adium 2.0/Message Styles (create if necessary). Then choose the TOP Stockholm theme (no idea why there are two entries), and close your chat window. It should be activated next time a chat window opens.

A rich man builds a pool containing only bottled water and invites a few friends over. He gathers everyone around and says, “Guys, I know it’s tempting and convenient to pee in the pool, but this pool is 100% Evian, let’s all try to use the bathroom next to the pool instead.” So everyone listens and enjoys a urine-free, delicious pool for the rest of the day. Until, of course, one of them breaks the rule, ruining it for everyone.

That pool is your commit history and the merge commits are the pee in the pool. So please, keep your pee out of the pool and rebase instead.

Disclaimer: I don’t really care if you do or do not pee in regular pools. This is all made up to illustrate a point.

Here’s how you disable lion’s dumb default gestures that break front/back on browsing and next/prev in emacs: