Recently, I’ve really gotten into tmux for managing all my terminal sessions/windows. Not so much for the panes, but more for keeping a highly contextual environment per project or task.

As the number of sessions grew, they became difficult to tell apart. For a few days now, I’ve had the idea of hashing the name of the session into a unique color, so that every session had its own status-bg color.

First, the tmuxHashColor function:

1
2
3
4
5

tmuxHashColor() {
  local hsh=$(echo $1 | cksum | cut -d ' ' -f 1)
  local num=$(expr $hsh % 255)
  echo "colour$num"
}

In our ns function (new session), we hash the supplied session name to a color, then use tmux send-keys to set its status-bg color to it:

1
2
3
4
5
6
7
8
9
10

ns() {
  if [ -z $1 ]; then
    1=$(basename $(pwd))
  fi
  tmux new-session -d -s $1
  local color=$(tmuxHashColor $1)
  tmux send-keys -t $1 "tmux set-option status-bg $color" C-m
  tmux send-keys -t $1 "clear" C-m
  tmux attach -t $1
}

Now every session has it’s own distinct status-bg color!

Most modern web stacks allow the “filtering” of requests via stackable/composable middleware, allowing you to cleanly separate cross-cutting concerns from your web application. This weekend I needed to hook into go’s http.FileServer and was pleasantly surprised how easy it was to do.

Let’s start with a basic file server for /tmp:

1
2
3

func main() {
    http.ListenAndServe(":8080", http.FileServer(http.Dir("/tmp")))
}

This starts up a local file server at :8080. How can we hook into this so we can run some code before file requests are served? Let’s look at the method signature for http.ListenAndServe:

1

func ListenAndServe(addr string, handler Handler) error

So it looks like http.FileServer returns a Handler that knows how to serve files given a root directory. Now let’s look at the Handler interface:

1
2
3

type Handler interface {
    ServeHTTP(ResponseWriter, *Request)
}

Because of go’s granular interfaces, any object can be a Handler so long as it implements ServeHTTP. It seems all we need to do is construct our own Handler that wraps http.FileServer’s handler. There’s a built in helper for turning ordinary functions into handlers called http.HandlerFunc:

1

type HandlerFunc func(ResponseWriter, *Request)

Then we just wrap http.FileServer like so:

1
2
3
4
5
6
7
8
9
10
11
12

func OurLoggingHandler(h http.Handler) http.Handler {
  return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    fmt.Println(*r.URL)
    h.ServeHTTP(w, r)
  })
}

func main() {
    fileHandler := http.FileServer(http.Dir("/tmp"))
    wrappedHandler := OurLoggingHandler(fileHandler)
    http.ListenAndServe(":8080", wrappedHandler)
}

Go has a bunch of other builtin handlers like TimeoutHandler and RedirectHandler that can be mixed and matched the same way.

This week, @bradfitz (of memcached fame) released groupcache at OSCON 2013. I’m already a big fan of memcached and camlistore, so I couldn’t wait to download it and kick the tires.

By the way, I strongly recommend you go through the slides and README before going further.

What groupcache isn’t

After downloading it (without reading the slides), I instinctively searched around for how to actually start the server(s), only to find nothing. Turns out, groupcache is more of a library with a server built in, rather than a traditional standalone server. Another important consideration is that theres no support for set/update/evict operations, all you get is GET. Really fast, consistent, distributed GET’s.

What it is

Once you realize that groupcache is more of a smart, distributed LRU cache, rather than an outright memcached replacement, it all makes much more sense. Especially considering what it was built for, caching immutable file blobs for dl.google.com.

How to use it

For groupcache to work, you have to give it a closure in which: given a key, fill up this dest buffer with the bytes for the value of that key, from however you store them. This could be hitting a database, a network filesystem, anything. Then you create a groupcache group object, which knows the addresses of all the other groupcache instances. This is pluggable, so you can imagine rigging that up to zookeeper or the like for automatic node discovery. Finally, you start groupcache up by using go’s built in net/http and a ServeHTP provided by the previously constructed group object.

Running the demo

In order to really try out groupcache, I realized I needed to create a mini test infrastructure, consisting of a slow database, frontends, and a client. Visit the Github Repo for more details. This is what the topology looks like:

Setup

1. git clone git@github.com:capotej/groupcache-db-experiment.git
2. cd groupcache-db-experiment
3. sh build.sh

Start database server

1. cd dbserver && ./dbserver

Start Multiple Frontends

1. cd frontend
2. ./frontend -port 8001
3. ./frontend -port 8002
4. ./frontend -port 8003

Use the CLI to play around

Let’s set a value into the database:

./cli -set -key foo -value bar

Now get it out again to make sure it’s there:

./cli -get -key foo

You should see bar as the response, after about a noticeable, 300ms lag.

Let’s ask for the same value, via cache this time:

./cli -cget -key foo

You should see on one of the frontend’s output, the key foo was requested, and in turn requested from the database. Let’s get it again:

./cli -cget -key foo

You should have gotten this value instantly, as it was served from groupcache.

Here’s where things get interesting; Request that same key from a different frontend:

./cli -port 9002 -cget -key foo

You should still see bar come back instantly, even though this particular groupcache node did not have this value. This is because groupcache knew that 9001 had this key, went to that node to fetch it, then cached it itself. This is groupcache’s killer feature, as it avoids the common thundering herd issue associated with losing cache nodes.

Node failure

Let’s simulate single node failure, find the “owner” of key foo (this is going to be the frontend that said “asking for foo from dbserver”), and kill it with Ctrl+C. Request the value again:

./cli -cget -key foo

It’ll most likely hit the dbserver again (unless that particular frontend happens to have it), and cache the result on one of the other remaining frontends. As more clients ask for this value, it’ll spread through the caches organically. When that server comes back up, it’ll start receiving other keys to share, and so on. The fan out is explained in more detail on this slide.

Conclusion / Use cases

Since there is no support (by design) for eviction or updates, groupcache is a really good fit with read heavy, immutable content. Some use cases:

• Someone like Github using it to cache blobrefs from their file servers
• Large websites using it as a CDN (provided their assets were unique logo-0492830483.png)
• Backend for Content-addressable storage

Definitely a clever tool to have in the distributed systems toolbox.

Shout out to professors @jmhodges and @mrb_bk for proof reading this project and post

Recently, I invested some time into automating some of the work that goes into a Finatra release.

The work consists of updating:

• The version in the XML fragment of the main README.markdown

• The version in the pom.xml of the example app

• Any API changes in the example app

• The version in the template pom.xml of the app generator

• The generated unit test of the app generator that demonstrates testing any new API features

• Any API changes inside the app template of the app generator

Using sub, I was able to create a finatra command that automated all of the above based on a single template, which also happens to be the main unit test. This ensures that the README, the example app, and the app generator never fall out of sync with the frameworks API.

Last week we released 1.1.0, and the README was completely generated, as was the example app. Not to mention, all generated apps would also contain the latest templates and examples!

Let’s dive into how it all works:

The source of truth

I annotated our main unit test with special tokens, like so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

class ExampleSpec extends SpecHelper {

  /* ###BEGIN_APP### */

  class ExampleApp extends Controller {

    /**
     * Basic Example
     *
     * curl http://localhost:7070/hello => "hello world"
     */
    get("/") { request =>
      render.plain("hello world").toFuture
    }

  }

  val app = new ExampleApp

  /* ###END_APP### */


  /* ###BEGIN_SPEC### */

  "GET /hello" should "respond with hello world" in {
    get("/")
    response.body should equal ("hello world")
  }

  /* ###END_SPEC### */
}

Using the special /* ### */ comments, the main app and its test can be extracted from the code of our test.

The app generator

Now that we have our “template”, we can build our app generator to use it. I customized base and ended up with: script/finatra/libexec/finatra-new

You can then run:

1

$ ./finatra new com.example.myapp

and it will generate myapp/ based on the tested example code from the test suite above.

The example app

The example app is just a generated app using the latest app generator:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

#!/bin/bash
# Usage: finatra update-example
# Summary: generates the example app from the template

set -e

source $_FINATRA_ROOT/lib/base.sh

tmpdir=$(mktemp -d /tmp/finatra_example.XXX)

$_FINATRA_ROOT/bin/finatra new com.twitter.finatra_example $tmpdir

cp -Rv $tmpdir/finatra_example/ $EXAMPLE_REPO

rm -rf $tmpdir

cd $EXAMPLE_REPO && mvn test

This also tests the app generator and the generated app!

Updating the README

Lastly, there’s a command for updating the README with the new example and version number.

After months of work Finatra 1.0.0 is finally available! Finatra is a scala web framework inspired by Sinatra built on top of Finagle.

The API

The API looks like what you’d expect, here’s a simple endpoint that uses route parameters:

1
2
3
4

get("/user/:username") { request =>
  val username = request.routeParams.getOrElse("username", "default_user")
  render.plain("hello " + username).toFuture
}

The toFuture call means that the response is actually a Future, a powerful concurrency abstraction worth checking out.

Testing it is just as easy:

1
2
3
4

"GET /user/foo" should "responsd with hello foo" in {
  get("/user/foo")
  response.body should equal ("hello foo")
}

A super quick demo

1
2
3

$ git clone https://github.com/capotej/finatra.git
$ cd finatra
$ ./finatra new com.example.myapp /tmp

Now you have an /tmp/myapp you can use:

1
2

$ cd /tmp/myapp
$ mvn scala:run

A simple app should’ve started up locally on port 7070, verify with:

1
2

$ curl http://locahost:7070
hello world

You can see the rest of the endpoints at /tmp/myapp/src/main/scala/com/example/myapp/App.scala

Heroku integration

The generated apps work in heroku out of the box:

1
2
3
4
5

$ heroku create
$ git init
$ git add .
$ git commit -am 'stuff'
$ git push heroku master

Make sure to see the full details in the README and check out the example app.

Props to @twoism and @thisisfranklin for their code, feedback and moral support.

Finally got tired of copy pasting other projects and gutting them to make new ones, so I created base, a shell command that creates new scala projects.

Creating the project:

1
2
3
4
5
6
7
8
9
10

$ base new com.capotej.newproj
creating project: newproj
  creating App.scala
  creating AppSpec.scala
  creating pom.xml
  creating .gitignore
  creating .travis.yml
  creating LICENSE
  creating README.markdown
Done! run mvn scala:run to run your projec

Based on the package name, it infered that the project name is newproj and created the project under that folder. Let’s build and run it:

1
2
3
4

$ cd newproj
$ mvn compile scala:run
(... maven output ...)
hello world

This uses the new incremental compiler for maven, zinc, which dramatically speeds up compile times (except for the first time you run it). It also sets you up with the latest scalatest maven plugin, which gives you sweet looking test output, like so:

See the base README for installation instructions.

A few months ago, I gave a presentation on how Posterous uses Riak for it’s post cache; At #ricon2012 I ended up retelling this story to numerous people, so I thought I’d post the slides and video here.

UPDATE: this is now available as a sub command, here: kiev

Cooked this up last night when I needed a simple key/value store for use in a shell script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

#!/bin/sh

DBFILE=example.db

put(){
  echo "export kv_$1=$2" >> $DBFILE
}

del(){
  echo "unset kv_$1" >> $DBFILE
}

get(){
  source $DBFILE
  eval r=\$$(echo "kv_$1")
  echo $r
}

list(){
  source $DBFILE
  for i in $(env | grep "kv_" | cut -d= -f1 ); do
    eval r=\$$i; echo $(echo $i | sed -e 's/kv_//') $r;
  done
}

## cmd dispatch

if [ ${1:-0} == "set" ]; then
  put $2 $3
elif [ ${1:-0} == "get" ] ; then
  get $2
elif [ ${1:-0} == "list" ] ; then
  list
elif [ ${1:-0} == "del" ] ; then
  del $2
else
  echo "unknown cmd"
fi

Use it like so:

$ ./db.sh set foo bar

$ ./db.sh get foo

$ ./db.sh set foo baz

$ ./db.sh get foo

$ ./db.sh del foo

$ ./db.sh list

How it works

Every time you update/set/delete a value, it writes a shell expression to an append-only log, exporting a shell variable (key) with that value. By sourcing the file every time we read a value, we replay the log, bringing our environment to a consistent state. Then, reading the value is just looking up that dynamic variable (key) in our shell environment.

I’ve been fascinated by the concepts in finagle for some time, but being a scala noob, I never knew how to bootstrap a finagle project. Turns out twitter has a gem, scala-bootstrapper, that generates a simple thirft based key/value store for you. There’s even atutorial  on how to extend the example project into a distributed search service.

This is a guide on setting it all up locally, it assumes you have Git, Homebrew, and OS X.

Install scala 2.8.1

1
2
3
4
5

$ brew versions scala
$ cd/usr/local/(or wherever you have homebrew installed)
$ git checkout -b scala281 0e16b9d(make sure the SHA matches versions output)
$ brew install scala
$ git checkout master$git branch -D scala281

Install sbt 0.7.4 (assumes you have a ~/bin in your $PATH)

1
2

$ curl -O http://simple-build-tool.googlecode.com/files/sbt-launch-0.7.4.jar > ~/bin/sbt-launch.jar
$ echo 'java -Xmx1G -jar `dirname $0`/sbt-launch.jar "$@"'> ~/bin/sbt

Install scala-bootstrapper

1

$ gem install scala-bootstrapper

Generate finagle project**

1
2
3
4
5

$ mkdir newbird
$ cd newbird
$ scala-bootstrapper newbird
$ sbt update
$ sbt test

Add a Client class

create newbird/src/main/scala/com/twitter/newbird/Client.scala with

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

package com.twitter.newbird

import com.twitter.finagle.builder.ClientBuilder
import com.twitter.finagle.thrift.ThriftClientFramedCodec
import com.twitter.newbird.thrift._
import org.apache.thrift.protocol.TBinaryProtocol

import java.net.InetSocketAddress

class Client {  

  val service = ClientBuilder().hosts(Seq(newInetSocketAddress("localhost",9999)))
    .codec(ThriftClientFramedCodec())    
    .hostConnectionLimit(1)    
    .build()  
  val client = new NewbirdServiceClientAdapter(
    new thrift.NewbirdService.ServiceToClient(service,newTBinaryProtocol.Factory))  

  def get(key:String) = client.get(key)()  
  def put(key:String, value:String) = client.put(key,value)()

}

Running the server

1
2

$ cd newbird
$ sbt> run -f config/development.scala

Playing with the client**

1
2
3
4
5
6

$ cd newbird
$ sbt console
scala> import com.twitter.newbird.Client
scala> val client = new Client()
scala> client.put("foo","bar")
scala> client.get("foo")

Bonus

finagle exports a stats url you can curl:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

$ curl http://localhost:9900/stats.txt
counters:  Newbird/connects: 1  
Newbird/requests: 4  
Newbird/success: 4
gauges:  
  Newbird/connections: 0  
  Newbird/pending: 0  
  jvm_heap_committed: 588251136  
  jvm_heap_max: 2146828288  
  jvm_heap_used: 64354560  
  jvm_nonheap_committed: 83267584  
  jvm_nonheap_max: 318767104  
  jvm_nonheap_used: 68655360  
  jvm_num_cpus: 4  
  jvm_start_time: 1327511164928  
  jvm_thread_count: 14  
  jvm_thread_daemon_count: 9  
  jvm_thread_peak_count: 14  
  jvm_uptime: 2626505
labels:
  metrics:  
    Newbird/connection_duration: (average=2590412, count=1, maximum=2590412, minimum=2590412, p25=2590412, p50=2590412, p75=2590412, p90=2590412, p99=2590412, p999=2590412, p9999=2590412)  
    Newbird/connection_received_bytes: (average=192, count=1, maximum=192, minimum=192, p25=192, p50=192, p75=192, p90=192, p99=192, p999=192, p9999=192)  
    Newbird/connection_requests: (average=4, count=1, maximum=4, minimum=4, p25=4, p50=4, p75=4, p90=4, p99=4, p999=4, p9999=4)  
    Newbird/connection_sent_bytes: (average=120, count=1, maximum=120, minimum=120, p25=120, p50=120, p75=120, p90=120, p99=120, p999=120, p9999=120)  
    Newbird/request_latency_ms: (average=14, count=4, maximum=39, minimum=2, p25=2, p50=8, p75=10, p90=39, p99=39, p999=39, p9999=39)

While looking for a way to add wunderlist tasks via alfred, I came upon wunderlist-for-alfred

Looked cool, but I wanted to write my own that didn’t depend on php.

I used lsof to figure out the location of the db, then used file to see what kind of db it was. Luckily, it was sqlite3, so I was able to poke around and figure out the sql to create a task.

Here’s the alfred extenstion that ties it all together:

1
2
3

user=`whoami`
wunderdb="/Users/$user/Library/Wunderlist/wunderlist.db"
sqlite3 $wunderdb "insert into tasks (name, list_id) values ('{query}', 1)"

Download it here