rapid habits ❧ mustafa paksoy

How Dropbox saved my command line

I live in the command line across many Unix-like platforms and networks. I have two Mac laptops, two Linux workstations, shell logins scattered across the multiverse, and even a couple Windows VMs to boot. I’ve tried versioning my shell configuration files, but that requires me to check in/check out my changes across all environments. This turned out to be less than ideal. More manual effort, more hassle.

Enter Dropbox that syncs the files across multiple machines seamlessly and even gives you a bit of revision history. I concocted a system on top of Dropbox that allows me to have global, per OS, and per machine shell configs. For example, here are all the Bash config files I keep in sync via Dropbox (machine names changed to protect the innocent):

$ ls ~/Dropbox/shell/bash
bashbootstrap  bashrc
bashrc-Darwin  bashrc-Darwin-laptopname  bashrc-Darwin-mininame
bashrc-Linux  bashrc-Linux-machineone  bashrc-Linux-machinetwo

Let’s ignore the bashbootstrap file for the moment. You will notice that we have a globally-applied config file, bashrc, two OS specific config files, bashrc-Linux, bashrc-Darwin, and several machine specific ones. (By the way, Darwin is the name of OS X’s BSD-like kernel.)

What ties it all together is the bashbootstrap file. It loads each applicable config file in order of increasing specificity, this allows per OS and per machine overrides to have higher precedence. Additionally, we silently skip missing config files; you need not create empty config files for each of your machines to keep the script happy.

On a new machine, after installing Dropbox on ~/Dropbox, I move away the default .bashrc and just symlink the bootstrap file in its place instead:

$ mv ~/.bashrc ~/.bashrc.bak
$ ln -s ~/Dropbox/shell/bash/bashbootstrap  ~/.bashrc

Oh, and here are the contents of the bashbootstrap file:

if [ -z "$PS1" ]; then
   return
fi

dropboxshelldir=~/Dropbox/shell
dropboxdir=$dropboxshelldir/bash
masterbashrc=$dropboxdir/bashrc
osbashrc=$masterbashrc-`uname`
localbashrc=$osbashrc-`hostname | cut -d. -f1`

echo -n "Applicable shell configs: "
for bashfile in "$masterbashrc" "$osbashrc" "$localbashrc"; do
  if [ -r $bashfile ]; then
    . $bashfile
    echo -n "`basename $bashfile` "
  fi
done
echo

# Set convenience aliases
myed=${VISUAL:-${EDITOR:-vim}}
alias editbashrc="$myed $masterbashrc"
alias editosbashrc="$myed $osbashrc"
alias editlocalbashrc="$myed $localbashrc"

One final note, this script also provides three convenience aliases for editing your Bash config files without having to remember where they are stored.

• editbashrc: Edit the global config file.
• editosbashrc: Edit the OS-specific config file.
• editlocalbashrc: Edit the machine-specific config file.

I only tested this on Bash, but it could work on other Bash like shells. But, as they say, your mileage may vary.

Did I mention that if you sign up to Dropbox with my referral we both get 250 megs of additional storage? Neat!

The poor man’s Bash Tab-completion

Bash has a built-in tab completion utility that sports an impressive array features and the ability to generate suggestions using complex logic. But what if you just want to add a static list of suggestions to a command? It took a little digging around the Bash manual but I finally found the magic words you need to add to you .bashrc file:

complete -o default -W "list of space separated words" [command]

Obviously, you need to replace list of space separated words and [command] with values of your choice. For the curious, let’s take this command apart and see what’s going on:

• complete: Built in Bash command for controlling built-in tab completion behavior
• -o default: Tell Bash to fall back on the default filename completion if no matches are found.
• -W “list of space separated words”: This is where the magic happens. -W allows us to just provide a static list of suggestions. There are other flags that will dynamically evaluate suggestions at runtime, but we’ll leave that to a future post.
• [command]: The command that tab-completion will apply to.

Say I want to just add the suggestions “all” and “clean” to the command make. The line I need to append to my .bashrc becomes:

complete -o default -W "all clean" make

Bonus: get a little dynamic

Using command substitution we can also generate the list of suggestions at Bash start time. This is very useful for completing names that can be found in other configuration files (like, say, SSH host-name completion).

complete -o default -W "`echo $(cat /path/to/file | grep 'lines i want')`" \
    [command]

(You might notice that we are using command substitution twice in the example above, once with back-ticks the other time using $(command). This is necessary because the -W argument does not accept new-line characters as delimiters. echoing will convert these new-lines in to spaces.)

Update: The friendly folks over at HN have pointed out that there non-stupid ways of putting together the aforementioned command substitution. Thanks for the correction folks! Here’s a better way to do it:

complete -o default -W "$(grep 'lines i want' /path/to/file | tr '\n' ' ')" \
    [command]

Log analysis with MySQL

Digging through log files has got to be the least-sexy aspect of web app development. I’ve found that using MySQL makes this process a little less painful and easier to standardize. At the very least, you don’t end up with 5 perl scripts every time you try to analyze a particular piece of data that you can never reuse. Here’s the workflow that I’ve found to work well for me:

1. Grab logs with your preferred method (scp, wget, curl you name it)
2. Select the columns you want out of the log file using cut. If your input is not Tab-delimited, you will have to specify your delimiter with -d and override your output delimiter with --output-delimiter="\t". For example, to select the first and third columns out of a CSV file: cut -d, -f1,3 --output-delimiter="\t".
3. Create a new MySQL table that has a column per log line column. For example: `mysql -e “create table pageviews (page varchar(32), cputime int)” logs
4. Import the data from the text file:mysqlimport --local pageviews.txt

Please note: the table that you are importing into needs to have the same name as the text file. So if the file is called httperrors.txt your MySQL table needs to be called httperrors.

That’s about it. Now you can use familiar SQL to run any analysis on your data that you want.

For example: SELECT page, COUNT(1) FROM log WHERE cputime > 500 GROUP BY page ORDER BY 2 DESC;

That will give me all the requests that took longer than 500 ms of CPU time broken down by page in my app. As simple as that.

How is that for data warehousing?