Matt Writes Code

Quick Tip: Protect Important EC2 Instances

Tue Feb 26 00:00:00 -0800 2013

I did something really stupid today. I was doing a load test with Bees With Machine Guns! that I thought was only going to hit Apache, and not the database. Turns out there is a small database call on the page I was hitting, and caused the database to spike. I tried exiting out of Bees, but it wouldn't. The site started to crash, so I went to terminate the bee instances in the AWS console, but didn't realize I had the production database selected. After hitting terminate, all hell broke loose.

Luckily we had a slave database running, that we were able to switch to relativly quickly. The EBS disks were still around, so we were able to get our production database back up and running. But what if we didn't have the slave (if you don't, you really should), or the EBS disks were terminated with the instance? We would have been SOL.

Luckily there is a termination protection feature in EC2 that you can use. Just enable termination protection, and you can't terminate your instance. I did that with the slave database, the replacement database, and a few other "mission critical" instances. Now some dumbass like me can't accidentally delete all your data.

Turning this on, of course, does not mean you shouldn't be backing up your data. There's no reason you shouldn't be doing that.

Installing pcntl On Lion

Wed Nov 21 00:00:00 -0800 2012

I needed the pcntl module for a project I'm working on in PHP. By default this module is not installed when you install PHP, so you'll need to compile it. There is a brew package for it, but I couldn't get it to work, and I think it's just as simple to compile it.

First you'll need to download the PHP source. Run php -v to get your PHP version and browse the PHP releases page for your version. In my case it was 5.3.10.

curl -o php-5.3.10.tar.gz http://us1.php.net/distributions/php-5.3.10.tar.gz
tar xzvf php-5.3.10.tar.gz
cd php-5.3.10/ext/pcntl
phpize
./configure --enable-pcntl

Some people reported that they got an error while running ./configure. I did not run into this issue, but if you do, you'll need to specify your system architecture.

CFLAGS='-arch x86_64' CXXFLAGS='-arch x86_64' LDFLAGS='-arch x86_64' ./configure --enable-pcntl

All that's left to do is compile, install it, and enable it.

make
make test # optional
sudo make install
sudo echo "extension=pcntl.so" >> /etc/php.ini

If you are using Apache, you will need to restart it to take affect.

Why I Do Side Projects

Tue Nov 20 00:00:00 -0800 2012

I, like every other developer, have side projects. Why work on side projects? I think there are four primary reasons. 1) learn an existing language or tool better, 2) learn a new language or tool, 3) make some extra income, or 4) fill a need you have.

I have done (or am doing) side projects that cover all of these reasons, but I think the main reason I do a lot of my side projects is because of reason 2, learn a new language or tool.

Learn A New Language Or Tool

When I learn something new, I'll generally read up on it, maybe buy a book, write a few code samples, but to really solidify what I've learned, I need to write something. For someone new to a language or tool this can be scary. That's why I rely on existing open source projects. If I run into an issue, I can see a real world example of how that was solved.

When I started writing Tea-Fueled Does, I copied a lot of ideas from other open source frameworks. It helped me out when I was having issues and was able to see how others solved it (and even different ways to solve the same problem). After seeing how the code worked, I was able to refine my framework and it become it's own thing.

The same thing with Playr. I was learning Ruby and things just weren't clicking. So I decided I would try to write a clone of Play. When I ran into an issue, or didn't know what to do next, I would look at Play's source code and was able to figure it out.

In both these cases, I never copied and pasted the project. I would have learn nothing that way. I would attempt to get as far as I could without any help, and when I ran into an issue or wanted to see how someone else did it, I would look at the code. Even then did I rarely copied the code verbatim. I would try to do something similar until I found the best solution.

Open Source Rocks

That's why I appreciate open source projects. They have helped me learn so much, and because of this I post a lot of my code to GitHub. At this moment I have 16 repos, two that are forks, and two that are private (but I intend on making public some day).

That's the main reason I hack on side projects, to learn something new.

Introducing OmniBot

Fri Nov 09 00:00:00 -0800 2012

Today I released OmniBot 0.2.1. OmniBot is an IRC chat bot written in NodeJS and CoffeeScript.

Some History

The company I worked for was using Google Talk (and still do), for chats. We would have launch nights every other Thursday, and need a group chat. We used GTalk for this, but it was a bit lacking. After throwing around the idea of using IRC for this, I setup a server, and tested it out. To this day we still use IRC.

After I got the server setup, I started working on a chat bot. I had worked with NodeJS a little before, but not enough to really do anything with. So I decided this was a chance to learn Node a little better as well. A couple weeks later I had a chat bot that would say hello when you joined a room and give you the weather conditions. I then built it to be a little more modular and released it as an npm package.

The New OmniBot

This past week I rewrote OmniBot from the ground up using CoffeeScript. I've used CoffeeScript for other projects before and I think it makes writing JavaScript a lot more simpler.

I've dropped all planned support for other chat types and have made OmniBot strictly IRC. If you are using another chat service, I would recommend checking out Hubot, GitHub's chat bot.

Getting Started

If you don't have CoffeeScript installed on your machine, you'll first need to install that.

npm install -g coffee-script

Next you'll want to install OmniBot and OmniBot-Modules npm packages.

npm install omnibot omnibot-modules

Writing The Bot

OmniBot is the actual chat bot, while OmniBot-Module is a collection of default modules for OmniBot (we'll get into writing our own in another post).

Now we need to create our script, I've named mine bot.coffee. We need to require omnibot and create a new instance of it.

Robot = require 'omnibot'

irc =
  server: 'irc.example.com'
  channels: [ '#bot' ]

bot = new Robot 'RobotName', irc

The irc variable stores all of the IRC connection details. We'll be connecting to irc.example.com and the #bot channel. If you have an IRC server running on a different port, or have a password on your server, you would add those to the irc variable.

You can run the script by running coffee bot.coffee in the command line.

We now have a chat bot, but if we connect, we won't see our chat bot. That's because we need to start the bot before it will connect to the server.

...
bot = new Robot 'RobotName', irc

bot.start()

Now if we connect, we'll see that our bot is connected, but if we try to interact with it, it won't do anything. That's because we don't have any modules loaded.

Loading Modules

Modules are ways to hook into OmniBot without modifing any of it's source. It also provides a lot of helpers to make things a bit easier. We'll get more into modules in a future post, what they look like and how to write our own. For now we'll use omnibot-modules, a collection of modules I have put together to use with OmniBot. There is a method already included that makes loading them easy.

We can either load the modules before we call start, or pass a callback to start, and load the modules right after it connects. In this case, we want the modules loaded before we start the bot.

...
bot = new Robot 'RobotName', irc

modules = [ 'join', 'joke', 'weather' ]

bot.loadModules modules

bot.start()
...

As you can see, we pass any array of module names to the loadModules method. You can view a list of the modules on OmniBot's website along with their commands.

Now if you say RobotName joke, it will respond with a joke.

There are some modules that have config options. The weather module is one of these. Using the set method we can set the config option.

...
modules = [ 'join', 'joke', 'weather' ]

bot.set 'weather_zip', 55555

bot.loadModules modules
...

Now RobotName weather will give use the weather for the zip we set.

Host Your Own Chat Bot

I hope that gets you off the ground with OmniBot. I'll be writing another post about modules and more OmniBot features here soon that should allow you to do even more cool stuff. I'm also actively working on improving and adding new features to OmniBot.

Language Specific Indents In Sublime Text 2

Thu Aug 23 00:00:00 -0700 2012

Ask any developer spaces or tabs and you'll get different responses. I was hardcore in the tabs camp until recently. Now I'm trying to use spaces. The biggest issue with using spaces 100% was my text editor. I use Sublime, and tried to set it to use spaces by default, but couldn't seem to get it to work (later I would find out it was a mis-typed option). So every time I would create a new document I would set the tab size (2 for Ruby, 4 for PHP and Python) and covert it to spaces. Half the time I would forget to do this, so I would end up with projects half in spaces and half in tabs. Today I finally got sick of doing this every time, so I set out determined to figure this out once and for all, and I did.

One of the things I like about Sublime (and vim as well) is you can customize pretty much every thing to your liking. There are default settings in Sublime that you can modify, or you can create user settings that will override the default options (which is the preferred method). There are also syntax or language specific settings. We're going to focus on how to modify the syntax specific settings. There are two methods of doing this.

The Hard Way

This is how I modified my settings. There is a much easier way to do this, that I found out after I had done it this way.

Sublime stores all of it's settings and completions in directory structure on disk. On OS X it's located in ~/Library/Application Support/Sublime Text 2. All of the syntax specific settings are in the Packages directory. If you want to modify the settings for Ruby, you would edit ~/Libra/Application Support/Sublime Text 2/Packages/Ruby/Ruby.sublime-settings. Remember how Sublime has default and user settings? This is the default setting for Ruby, you want to modify the user settings. This is something I realized after the fact. You'll instead want to edit ~/Libra/Application Support/Sublime Text 2/Packages/User/Ruby.sublime-settings.

The Easy Way

After I had modified the settings manually, I was playing around with Sublime's settings and found a way to edit syntax specific settings right in Sublime (Sublime Text 2 -> Preferences -> Settings - More -> Syntax Specific - User). When you click this, it will open up a settings file for the language you are currently working in. So if you want to edit settings for PHP, you'll have to open a PHP file and click Syntax Specific.

The Settings

I've showed you how to modify syntax specific settings, but what's the actual code for using spaces instead of tabs? As you may or may not know, Sublime settings are in JSON format. Unless you've already created a settings file for the language, you'll have to create the structure first, which is just a set of curly braces ({}).

There are two settings that deal with tabs/spaces. The first is the tabSize (tab_size) and the second is translateTabsToSpaces (translate_tabs_to_spaces). Tab size is an integer and deals with, you guessed it, the tab size length. If you are using tabs, this will define how tabs are spaced. The Sublime default is 4. Translate tabs to spaces is a Boolean and defines the use of spaces vs. tabs. If you set it to true, it will use spaces (as defined by tabSize). If it is false (which is the Sublime default), it will use tabs.

{
    "tab_size": 2,
    "translate_tabs_to_spaces": true
}

My Ruby settings

{
    "tab_size": 4,
    "translate_tabs_to_spaces": true
}

My PHP and Python settings

You can also set other settings such as the color scheme, whether or not to use spell check or any other Sublime setting.

Securing Your Passwords

Tue Jun 26 00:00:00 -0700 2012

Today TutsPlus Premium got hacked. They used a third party plugin that stored passwords in plaintext. This is ironic, because NetTuts+, a sibling company posts about security once in a while. It's also very frustrating for users (both present and past) because they have to change their password not only on Tuts+, but other sites as well.

If you're storing passwords in plaintext, please stop for the sake of your users and your company/product. I'm going to show you that it's easy to implement securing your user's password.

An Introduction To bcrypt

I personally use bcrypt to hash all my passwords. It's a hash that incorporates a salt (to protect against rainbow attacks) and is adaptive, meaning it can be made slower over time (as computers speed up) to protect against brute-force attacks. This is by far the most recommended hashing function out there because of these features. I'll show you how to implement it in both PHP and Ruby.

PHP

If you are using PHP 5.2 or earlier, you may not have bcrypt available to you. In these versions it was implemented on the system side rather then in PHP itself (as in 5.3 and later). You can check if it is available on your system.

if (CRYPT_BLOWFISH == 1) {
    echo "Yes";
} else {
    echo "No";
}

To implement, we'll use PHP's crypt function. This supports different varieties of hash types, and what one will be used is based on the salt.

Bcrypt's salt starts with '$2a$' followed by a two digit cost, another '$' and 22 characters from './0-9A-Za-z'. The cost parameter relates to how much load it will take to crypt the password. You'll have to find a number (04 to 31) that isn't too slow or too fast on your system. Here are two different ways to generate a bcrypt salt.

function generate_salt($cost = 12) {
    return '$2a$' . str_pad($cost, 2, '0', STR_PAD_LEFT) . '$' . substr(sha1(mt_rand()),0,22);
}

function secure_generate_salt($cost = 12) {
    $salt = '$2a$' . str_pad($cost, 2, '0', STR_PAD_LEFT) . '$';
    $salt .= substr(str_replace('+', '.', base64_encode(openssl_random_pseudo_bytes(16))), 0, 22);
    return $salt
}

The secure_generate_salt is a little more secure as it used openssl to generate a random string, but you will need the openssl extension enabled.

Once you have a salt, hashing your password is simple.

$salt = secure_generate_salt();
$hash = crypt('password', $salt);

Now you're storing secure passwords, congratulations. But how do we verify if the user's input matches our stored hash?

function verify_hash($input, $hash) {
    return crypt($password, $hash) === $hash;
}

As simple as that. If you put that together in a class, you can have a really handy tool.

class Crypter {

    private static $cost = 12;

    public static function generate_salt($cost = null) {
        if (is_null($cost)) $cost = self::$cost;
        if ($cost < 4 || $cost > 31) throw new Exception('Cost must be between 4 and 31');
        $salt = '$2a$' . str_pad($cost, 2, '0', STR_PAD_LEFT) . '$';
        $salt .= substr(str_replace('+', '.', base64_encode(openssl_random_pseudo_bytes(16))), 0, 22);
        return $salt
    }

    public static function hash($input, $cost = null) {
        return crypt($input, self::generate_salt($cost));
    }

    public static function verify($input, $hash) {
        return crypt($input, $hash) === $hash;
    }

}

Ruby

Why yes there is a bcrypt gem, which makes it super simple to implement in your application. Install it with gem install bcrypt-ruby and require bcrypt.

Generate a hash with BCrypt::Password.create.

require 'bcrypt'

password = BCrypt::Password.create("password")
# cost defaults to 10, you can manually set the cost
password = BCrypt::Password.create("password", :cost => 12)

And verify a password with BCrypt::Password.new

require 'bcrypt'

password = BCrypt::Password.new(hash)
password == input # true if password

Forgot Passwords

Since you're not storing passwords in plaintext anymore, you won't be able to send passwords in emails if someone forgot theirs. Instead you'll have to build a password reset system. You could either have a temporary link or generate a random password and send it to them or another method.

Updating Costs

If you find that the costs you've been using are either too fast or too slow you can't just change the cost parameter in the hash. You can instead rehash the password on a successful login using your new cost.

So Why Aren't You Using bcrypt?

As you can see, bcrypt is both secure and easy to use. You have no good reason to continue storing passwords in plaintext.

Parallel Processes In Bash

Wed Jun 13 00:00:00 -0700 2012

I wrote a script to speed up MySQL imports. I'll write more on this later, but for now I want to focus on background and parallel processes in Bash. Part of the script loads tables at the same time, trying to speed up the import. You can have multiple processes in Bash by sending them to the background using the & character.

for table in TABLES; do
    import_table $table &
done

The above example will send all of your process to the background. If you have a lot of process (assuming the process takes some time and resources), this is going to slow down your system, and potentially crash it. Instead, I wanted to only run 4 commands at the same time. First I tried making a pid for each background process. Then when an table started, it would check for the number of pids. If it was less then the threads specified, it would start one, otherwise it would wait.

import_table() {
    table="$1"
    while [[ `ls *.pid | wc -l | tr -d ' '` -ge 4 ]]; then sleep 5; done
    echo $PPID > $table.pid
    # import
    rm $table.pid
}

for table in TABLES; do
    import_table $table &
done

While this solution did what I wanted, it left a lot of processes still running. I did a little digging and found a solution to fire off another process when one died. By using a named pipe, you can do better parallel processes in Bash, and it's really simple to do.

mkfifo pipe
exec 3<>pipe
rm -f pipe

{ sleep 1; echo >&3; } &
{ sleep 2; echo >&3; } &
{ sleep 3; echo >&3; } &
{ sleep 4; echo >&3; } &

while read; do
    { process_func; echo >&3; } &
done <&3

This script makes a named pipe, assigns it to the 3 descriptor for reads and writes and removes the pipe (it will keep intact until the script is done). I'm then firing up 4 processes, offsetting them so they don't all start at the same time. The while loop will then read every time the pipe is written to (when ever the process is done) and run the loop. This way we only have 4 sub-processes running at a time.

There are a lot more cool things you can do with named pipes, I would recommend reading up on them.

Why You Should Write Tests

Sun May 20 00:00:00 -0700 2012

Last week I started writing tests for my framework. I've got to be honest, I rarely write tests for my code. This was the first time I really wrote tests, but I recommend it.

It's Easy

I was always under the impression that writing tests for code was complex. After writing some tests for PHPUnit, I was wrong. The class extends PHPUnit_Framework_TestCase and you've got a bunch of assertions methods. For example take a look at this code I've pulled right from my tests.

<?php

    require 'test.bootstrap.php';

    use TFD\Auth;

    class AuthTest extends PHPUnit_Framework_TestCase {

        /**
         * Test Auth::login method.
         */

        public function testLogin() {
            $fingerprint = Auth::login('username', 'secret');
            $this->assertInternalType('string', $fingerprint);

            return $fingerprint;
        }

        /**
         * Test Auth::valid method.
         * 
         * @depends testLogin
         */

        public function testValid($fingerprint) {
            $this->assertTrue(Auth::valid($fingerprint, 'username', 'secret'));
        }

    }

That took me less then ten minutes to write.

There's No Guessing

When you write tests, you are making sure that something is working how it's suppose to. There's no guessing if it's going to work. You write a test, if it passes, the code works, if it fails, there's something wrong. When you change something you can verify that it still works by running the tests again.

It Automates Testing

Normally when I write code, I make sure it does what it's supposed to do and then forget about it. For example with Tea-Fueled Does, I would create a route or add something to a view to make sure something's working and then delete it. I would never test it again. With tests you can rerun those tests, and with PHPUnit, you can run all your tests in a single command.

It Creates Quality Code

How many times have you released something with bugs in it? I've done a couple commits where I find out later, I have a bug in it. Writing tests helps you prevent that.

Test-Driven Development is also a great development process. You write a test for a new function/method or an improvement and fix the code until it passes. This creates concise code.

Getting Started

PHPUnit's docs are a great place to start. NetTuts's also has an article on it SimpleTest, another PHP test suite. And if you need real life examples, check out my tests for Tea-Fueled Does.

Getting Around CORS in CloudFront

Thu May 10 00:00:00 -0700 2012

We ran into an issue at work the other day with CloudFront and S3. We were trying to load assets via Ajax from our CloudFront distribution, but kept getting an "Origin http://example.com is not allowed by Access-Control-Allow-Origin" error. There is a W3 spec called CORS (Cross-Origin Resource Sharing) that prevents retrieving data from another site. To get around this, you would normally set an "Access-Control-Allow-Origin" header, but S3 limits the headers you can set, and that's not one of them. I'm not the only one who finds this problem annoying. Amazon has said there are plans to implement this feature, but they also said that two years ago.

I couldn't wait around for this fix to come about (if it ever does), so I looked at other options; using another CDN, building my own CDN. Then this morning I came across an article on Hacker News called How we use Amazon CloudFront for dynamically generated content. By default, CloudFront's origin server is an S3 bucket, but there is an option for setting a custom origin. This means you can set up your own server to use as a CloudFront origin server. I spun up an EC2 instance with nginx and the "Access-Control-Allow-Origin" header. Then I created a new CloudFront distribution and set the instance as the origin. I reloaded the page, and it loaded the asset. While there are other issues with CloudFront (SSL with CNAMEs), this makes it more usable.

Playr - The Ruby Music Manager

Sat Feb 18 00:00:00 -0800 2012

A while back I started to learn Ruby. When I learn something I have to be actively working with it. That's why I started Playr. The goal was to create a webapp that could manage music. After a month I got a basic webapp that you could upload music to and queue up music, and after a little more work, I actually got it to play music.

I deployed it on the music computer my company at the time was using, and it was received pretty well. After a couple of weeks in production, I realized some bugs in it, and went to work on version 2.

At first, version 2 was just suppose to fix the process issues Playr dealt with in v1. After diving into the code again, I decided to throw most of it out and start again. To force myself to learn more tools, I used haml, SASS, and CoffeeScript in v2. After some slowing issues (from rendering SASS and CoffeeScript every request), I went to Unicorn and some Rack plugins to render SASS and CoffeeScript assets. But the biggest issue from v1 was still there, processes crashing. In v1 all processes were in a single file in different forks. This was really hard to manage, and often times when one crashed, the rest would crash. I looked into process management tools, and found god. God is a process monitoring framework. Once I got the webapp (Sinatra), music, and WebSocket processes over to god (and some configuring) the processes became stable and easy to stop and restart. Because adding another process was almost nothing, I added a simple task management process. There are a couple more additions/fixes from v1, but I won't go into them.

I am so happy with Playr, that I am open-sourcing it today. You can find the code on GitHub, and a wiki to get you going with Playr.