Imagine There’s No Server

I just wrote a blog post on TellApart's engineering blog about how we use Aurora, Mesos, and Docker to manage our infrastructure. I also talk about some major contributions we've made to the project. Check it out!

Setting up Ghost in Google Compute Engine

This post will detail how to set up a new VM in google compute engine to run Ghost. I was writing this while setting up this blog's server, if I'm missing anything let me know! These steps were based on a combination of Installing Ghost Deploying Ghost and Using a custom domain.

Updated: I've created a shell script to do all of this for you. See https://github.com/steveniemitz/gce-ghost. All you need to run now is curl https://raw.githubusercontent.com/steveniemitz/gce-ghost/master/install-ghost.sh | sh

The steps below are roughly what the script runs.

  1. Start by making a new VM in your developer console. I chose a f1-micro instance, debian-7 as the image, and give it a new static IP address. You'll also want to check "allow HTTP" to automatically create a firewall rule for HTTP in.

  2. SSH into your new instance. You can do this either by adding a new private key when setting up the instance, or the "SSH" link in the menu.

  3. You'll need unzip, which is not installed by default. First update your packages via sudo apt-get update, then install it via sudo apt-get install unzip.

  4. From your home directory (cd ~) download the latest ghost package.
    curl -L https://ghost.org/zip/ghost-latest.zip -o ghost.zip

  5. Make a place for ghost, I chose /var/www/ghost since it's the default in a later step.
    sudo mkdir -p /var/www/ghost

  6. Create a new user for ghost, we'll use this later.
    sudo useradd -r ghost -U

  7. Unzip ghost into this new directory.
    sudo unzip -uo ~/ghost.zip -d /var/www/ghost

  8. Change the owner of that new directory to ghost:ghost
    sudo chown -R ghost:ghost /var/www/ghost

  9. Get Node.js.
    sudo apt-get install nodejs

  10. Set up node instead of nodejs. Debian doesn't allow node by default, run this to set up a symlink.
    sudo update-alternatives --install /usr/bin/node nodejs /usr/bin/nodejs 100

  11. Get npm (which doesn't have a package in wheezy)
    curl https://www.npmjs.org/install.sh | sudo sh

  12. Install build-essential package. This is needed to build sqlite3
    sudo apt-get install build-essential

  13. Set it up
    npm install --production npm start

    Now kill the server with Ctrl+C twice once the first start is complete

  14. Set up an init script for ghost
    sudo curl https://raw.githubusercontent.com/TryGhost/Ghost-Config/master/init.d/ghost -o /etc/init.d/ghost sudo chmod 755 /etc/init.d/ghost

  15. Start ghost via the init script
    sudo service ghost start

  16. Set ghost to auto-start on boot
    sudo update-rc.d ghost defaults sudo update-rc.d ghost enable

  17. Install nginx and follow the instructions

At this point you should have a fully functional ghost install running on google compute engine!

Changing Blog Software

I decided I wanted to try out a new blog platform, driven mostly by the fact that my current (old) one looked kind of crappy and outdated. I like the more modern, simpler, cleaner look that a lot of newer platforms have, and I wanted to move away from a HTML based editor, since I do mostly coding.

I ended up chosing Ghost, a very simple blogging platform running in node.js. I also chose to host it in Google Compute Engine, a competitor to Amazon's EC2 and azure. I'm running on a f1-micro instance, which should be more than enough to run this blog. It's 1 "CPU", and .6 GB of RAM for less than $7 / month.

Migrating from BlogEngine.NET -> Ghost wasn't really a supported path, so I decided to write a quick little utility to take the BlogML XML file BlogEngine.NET can output and convert it to the Ghost import format, which has a pretty close 1-1 mapping to the BlogML schema. I've put the utility on GitHub here in case it's helpful to anyone else.

I've set up rewrite rules on my old blog (hosted on ASP.NET) to redirect to this new site, along with the same for the RSS feed, so the transition should be seamless.

So far I like ghost a lot, it's super simple to use and set up, and looks really nice. I haven't even played around with other themes yet either.

Implementing SOS with SPT - Part 3 of N - DumpMD & IP2MD

This is part 3 of N of my SOS series

It's a Saturday, so I'm going to do a nice easy one today, DumpMD, and IP2MD.  I group these together because ip2md is just a level of indirection from DumpMD.

Let's start with a quick refresher of !dumpmd:

0:036> !dumpmd 5b2351ec 
Method Name:  System.Web.HttpContext.Init(System.Web.HttpRequest, System.Web.HttpResponse)
Class:        5b220e1c
MethodTable:  5b458094
mdToken:      06002627
Module:       5b201000
IsJitted:     yes
CodeAddr:     5b3bb0f0
Transparency: Safe critical

Most things here are self explanatory.  The only thing many people may not be familiar with is the "Transparency" row.  Transparency is a core part of the .NET security model, I'm not going to get into it, but you can read about it on MSDN.

DumpMD is very simple because almost all of the displayed fields are obtained via IXCLRDataProcess3::GetMethodDescData, which takes a methodDesc address and returns a ClrMethodDescData structure.  Name we can get via IXCLRDataProcess3::GetMethodDescName.  The EEClass ("Class") can be retrieved via IXCLRDataProcess3::GetMethodTableData with the methodTable address on ClrMethodDescData.  All we're left with is Transparency, which we can get with IXCLRDataProcess3::GetMethodDescTransparencyData.

IP2MD adds one layer on top of DumpMD, it figures out the MD first, then runs the rest.  Given an arbitrary IP, IXCLRDataProcess3 has a convenient method to resolve it to a MethodDesc, unsurprisingly named GetMethodDescPtrFromIP.  Calling this will give us the MethodDesc address for that IP, which we can then pass to DumpMD.

As always, the source for this is on my github.

Implementing SOS with SPT - Part 2 of N - DumpStackObjects

This is part 2 of N of my SOS series.

For part 2, I’m going to talk about DumpStackObjects.  This one is fairly straight forward, and we already have most of the functionality required built in SPT, so all that is left is to hook it up.

The general algorithm for scraping the stack is this:

  1. Find the stack limits (stackBase, stackLimit)
    • stackBase can be pulled from the current thread’s TEB (thread environment block)
    • stackLimit is the value of [e/r]sp for that thread.
  2. Get the bounds of the GC heap segments.  This algorithm can be found in CSDbgExt::EnumHeapSegments.
  3. Start at stackLimit, moving 1 pointer at a time, until we hit stackBase
    1. Read a pointer at our current stack location
    2. Use IClrProcess::IsValidObject to check if the address is valid or not
    3. If valid, also make sure it’s within the GC heap segments found in (2)
    4. We can find the MT using IXCLRDataProcess3::GetObjectData.  For strings, we can read their value easily using IXCLRDataProcess3::GetObjectStringData.

Once we’ve done this, there’s one more thing to consider: some registers may also contain CLR objects.  We can evaluate all the registers in a similar way to #3 above.

You can find the source for this on github here.