Introducing Scales - A highly scalable, performant RPC library for Python

TLDR: Check out scales on github and pypi as scales-rpc

Some back story...

This post has been long in the making.

Almost a year ago, at TellApart, a coworker was debugging a weird issue with some of our Python thrift services (the client side was python, server was Java, finagle specifically). Randomly, but fairly frequently, we'd see our python clients getting the wrong responses from their RPC calls. Specifically, after investigating more, we were seeing them getting the response for the PREVIOUS call!

This seemingly impossible scenario makes sense once you understand how the python thrift TCP client works. It maintains a single TCP connection between RPC calls, and sends framed request / response messages over that connection. A full RPC call is just basically:

  1. Serialize message onto the wire.
  2. Wait for a response (by blocking on a socket read).
  3. Read the response.

Now, if everything is going well, you end up with a pipeline looking like:
Req1 -> Resp1 -> Req2 -> Resp2

However, if the client is interrupted between step 2 and 3 (that is, it never reads the response), your pipeline is now in an undefined state, looking something like:
Req1 -> Resp1 -> Req2 -> |Interupted| -> Req3 -> Resp2 -> Req4 -> Resp3

As you can see, since Resp2 was never read, and the connection was never reset, all responses are now for the request BEFORE them.

"How can this happen?" you may ask. Well, the answer is actually a fairly common operation, client side timeouts. In our case, we would perform a client side timeout by aborting the greenlet that was running the request. Since there was no code to clean up the socket after the timeout, the result was the connection getting into the above state.

Now, the short-term solution is fairly simple, after performing a timeout, the connection can simply be reset via closing an reopening the socket. However, this is a fairly expensive operation, and our service timeouts are fairly low (in the order of 20-30ms). Because of this, the overhead of reconnecting can be prohibitive.

At this point, you may be seeing where this is going. We quickly realized that since Thrift itself has no solution to performing efficient client side timeouts, we needed to look for another protocol that does.

Very conveniently, as I said above, our services were running on Twitter's Finagle RPC framework. Finagle provides another thrift transport called ThriftMux, that was designed pretty much exactly for this use case. ThriftMux is a multiplexed protocol, supporting multiple pending requests over a single socket, and additionally supporting robust client initiated timeouts. We had already been using ThriftMux for services on the Java side, so there was nothing to do on the server side for this to work in python.

So, I looked around for an existing python library that could talk ThriftMux. Sadly, finding none, I decided to write my own. What started out as a simple ThriftMux implementation (also with the aim of improving the terrible python thrift experience), turned into a full-featured RPC library. Thus, Scales was born.

Introducing Scales

Scales was designed from the ground up to be a scalable, reliable, extensible, and asynchronous RPC library for python.

Lets look into each of those buzz words.

Scalable

Scales was designed (and named) to support the very high performance and latency requirements at TellApart. Our python services would need to complete RPC calls in milliseconds, with end-to-end request processing time requirements being in the sub-100ms range.
Scales is built on top of gevent, using many of its features and optimizations to support high degrees of concurrency across many clients.
Scales supports hundreds of downstream servers per client, and uses ZooKeeper as a first-class service discovery mechanism. However, any other service discovery strategy (etcd, consul, etc) could be plugged in.

Reliable

The bundled Scales load balancers and transports are built from the ground up to be highly reliable in the event of downstream failures. Both bundled load balancers maintain the state of downstream servers and attempt to avoid ones that have failed, as well as performing robust, exponential back-off reconnects to servers marked down.

Scales has been battle-tested at TellApart and performs over 200,000 requests/sec across our production python services.

Extensible

All components of Scales are extensible, and implementing new service types typically only requires plugging in serialization and transport logic.

A modular design via message sinks encourages separation of concerns for a service, typically partitioning logic into serialization, transport, routing, etc.

Asynchronous

As above, Scales is built with gevent at its core, however, importantly it does not rely on monkey patching at all. Scales will work perfectly fine (and just as importantly, will be just as performant) in a non-monkey patched environment.

One of the key design points of Scales is that all operations are asynchronous, and this is visible in all aspects of the stack. All operations on a Scales client have asynchronous as well as blocking versions.

Protocol support

Scales supports multiple protocols out of the box, exposing a consistent interface across all of them via a builder pattern.

  • ThriftMux
  • Thrift
  • HTTP
  • Kafka (producer only)
  • Redis (highly experimental)

Getting started

Getting started is simple, just pip install scales-rpc and start making RPC calls! All service clients are created via a NewClient method, which just wraps NewBuilder().Build()

A simple example is making a HTTP client to get pages from www.google.com

from scales.http import Http  
client = Http.NewClient('tcp://www.google.com:80')  
response = client.Get('/')  
print(response.text)  

More to come

In the next few weeks I'll be writing more posts explaining advanced uses of Scales, as well as going into how it was implemented.

For now, check it out on github or download it from pypi.

Imagine There’s No Server

I just wrote a blog post on TellApart's engineering blog about how we use Aurora, Mesos, and Docker to manage our infrastructure. I also talk about some major contributions we've made to the project. Check it out!

Setting up Ghost in Google Compute Engine

This post will detail how to set up a new VM in google compute engine to run Ghost. I was writing this while setting up this blog's server, if I'm missing anything let me know! These steps were based on a combination of Installing Ghost Deploying Ghost and Using a custom domain.

Updated: I've created a shell script to do all of this for you. See https://github.com/steveniemitz/gce-ghost. All you need to run now is curl https://raw.githubusercontent.com/steveniemitz/gce-ghost/master/install-ghost.sh | sh

The steps below are roughly what the script runs.

  1. Start by making a new VM in your developer console. I chose a f1-micro instance, debian-7 as the image, and give it a new static IP address. You'll also want to check "allow HTTP" to automatically create a firewall rule for HTTP in.

  2. SSH into your new instance. You can do this either by adding a new private key when setting up the instance, or the "SSH" link in the menu.

  3. You'll need unzip, which is not installed by default. First update your packages via sudo apt-get update, then install it via sudo apt-get install unzip.

  4. From your home directory (cd ~) download the latest ghost package.
    curl -L https://ghost.org/zip/ghost-latest.zip -o ghost.zip

  5. Make a place for ghost, I chose /var/www/ghost since it's the default in a later step.
    sudo mkdir -p /var/www/ghost

  6. Create a new user for ghost, we'll use this later.
    sudo useradd -r ghost -U

  7. Unzip ghost into this new directory.
    sudo unzip -uo ~/ghost.zip -d /var/www/ghost

  8. Change the owner of that new directory to ghost:ghost
    sudo chown -R ghost:ghost /var/www/ghost

  9. Get Node.js.
    sudo apt-get install nodejs

  10. Set up node instead of nodejs. Debian doesn't allow node by default, run this to set up a symlink.
    sudo update-alternatives --install /usr/bin/node nodejs /usr/bin/nodejs 100

  11. Get npm (which doesn't have a package in wheezy)
    curl https://www.npmjs.org/install.sh | sudo sh

  12. Install build-essential package. This is needed to build sqlite3
    sudo apt-get install build-essential

  13. Set it up
    npm install --production npm start

    Now kill the server with Ctrl+C twice once the first start is complete

  14. Set up an init script for ghost
    sudo curl https://raw.githubusercontent.com/TryGhost/Ghost-Config/master/init.d/ghost -o /etc/init.d/ghost sudo chmod 755 /etc/init.d/ghost

  15. Start ghost via the init script
    sudo service ghost start

  16. Set ghost to auto-start on boot
    sudo update-rc.d ghost defaults sudo update-rc.d ghost enable

  17. Install nginx and follow the instructions

At this point you should have a fully functional ghost install running on google compute engine!

Changing Blog Software

I decided I wanted to try out a new blog platform, driven mostly by the fact that my current (old) one looked kind of crappy and outdated. I like the more modern, simpler, cleaner look that a lot of newer platforms have, and I wanted to move away from a HTML based editor, since I do mostly coding.

I ended up chosing Ghost, a very simple blogging platform running in node.js. I also chose to host it in Google Compute Engine, a competitor to Amazon's EC2 and azure. I'm running on a f1-micro instance, which should be more than enough to run this blog. It's 1 "CPU", and .6 GB of RAM for less than $7 / month.

Migrating from BlogEngine.NET -> Ghost wasn't really a supported path, so I decided to write a quick little utility to take the BlogML XML file BlogEngine.NET can output and convert it to the Ghost import format, which has a pretty close 1-1 mapping to the BlogML schema. I've put the utility on GitHub here in case it's helpful to anyone else.

I've set up rewrite rules on my old blog (hosted on ASP.NET) to redirect to this new site, along with the same for the RSS feed, so the transition should be seamless.

So far I like ghost a lot, it's super simple to use and set up, and looks really nice. I haven't even played around with other themes yet either.

Implementing SOS with SPT - Part 3 of N - DumpMD & IP2MD

This is part 3 of N of my SOS series

It's a Saturday, so I'm going to do a nice easy one today, DumpMD, and IP2MD.  I group these together because ip2md is just a level of indirection from DumpMD.

Let's start with a quick refresher of !dumpmd:

0:036> !dumpmd 5b2351ec 
Method Name:  System.Web.HttpContext.Init(System.Web.HttpRequest, System.Web.HttpResponse)
Class:        5b220e1c
MethodTable:  5b458094
mdToken:      06002627
Module:       5b201000
IsJitted:     yes
CodeAddr:     5b3bb0f0
Transparency: Safe critical

Most things here are self explanatory.  The only thing many people may not be familiar with is the "Transparency" row.  Transparency is a core part of the .NET security model, I'm not going to get into it, but you can read about it on MSDN.

DumpMD is very simple because almost all of the displayed fields are obtained via IXCLRDataProcess3::GetMethodDescData, which takes a methodDesc address and returns a ClrMethodDescData structure.  Name we can get via IXCLRDataProcess3::GetMethodDescName.  The EEClass ("Class") can be retrieved via IXCLRDataProcess3::GetMethodTableData with the methodTable address on ClrMethodDescData.  All we're left with is Transparency, which we can get with IXCLRDataProcess3::GetMethodDescTransparencyData.

IP2MD adds one layer on top of DumpMD, it figures out the MD first, then runs the rest.  Given an arbitrary IP, IXCLRDataProcess3 has a convenient method to resolve it to a MethodDesc, unsurprisingly named GetMethodDescPtrFromIP.  Calling this will give us the MethodDesc address for that IP, which we can then pass to DumpMD.

As always, the source for this is on my github.