rapid habits ❧ mustafa paksoy

Find file, cd to parent directory

Tue, 09 Mar 2010 17:48:00 -0500

For jumping around in elaborate source trees.

Usage:

cdto [search root directory] [case insensitive filename]

If you just want to search current directory:

cdto [case insensitive filename]

Automatically appends * to search filename.

Works as intended

Mon, 22 Feb 2010 11:53:00 -0500

See[Mike]Code is a nifty tool for conducting remote programming interviews. One of the comments on the site bemoans the potential downsides of asking real programming questions during interviews:

The problem is often with the interview questions, for example:

“Write a loop that counts from 1 to 10” or “What’s the number after F in hexadecimal?”

As a web developer who has 5 years experience and build a number of large web applications for satisfied clients, I have never had to do anything with hex and simply would not know the answer in an interview.

However if I needed to do it in a job I would reference the API’s and be able to answer this in less than 2 minutes….

— ProgrammerJ

I don’t doubt that the commenter is a successful web developer, but no amount of API documentation would fix not knowing what hex notation is. Some basic computer science knowledge is required to capably consume technical documentation.

Highcharts: Beautiful Charts in Pure JavaScript

Tue, 01 Dec 2009 14:51:00 -0500

Highcharts: Beautiful Charts in Pure JavaScript:

Beautiful pure JavaScript HTML charts. Doesn’t even use the canvas element from what I can tell, which means better IE support. Also, plenty of DOM nodes to attach even handlers to.

Spending server cycles to generate raster images of charts is getting silly.

Photo

Sat, 14 Nov 2009 10:47:00 -0500

Honor among developers: self-assessing productivity

Sun, 25 Oct 2009 16:23:00 -0400

There is a discussion brewing on HN today about measuring programmer productivity. Reg Braithwaite chimes in to ask a very pertinent question:

Programmers talk about productivity all the time. They say they’re more productive with static typing or without it. With Ruby or with Java. … It seems every programmer I meet has strong opinions about when they are more productive and when they are not. So, here’s my question:

How are all these people measuring programmer productivity?

The observation is spot-on. Most developers know exactly what makes them more, or, less productive. They may not be able to assess if something they haven’t used yet will boost their productivity; but of the tools and habits they have tried in the past, they have strong feelings about their current choice.

In an intangible way, every good developer I’ve met knows when she has had a productive day. It’s mostly a feeling that comes without rational explanation. This intuition is what developers are using when they express their choice for Java or Ruby. They have used both, and they know which felt right and which did not.

But there is a catch: you can’t harness this intuition if there are incentives to assess your own productivity too positively. Say, like, an annual performance review that your job depends on. Yet, this isn’t because developers are crooks. Something so intangible like an intuition is susceptible to being skewed for very innocent psychological reasons. This is also why a good manager will allow you to freely raise the issue when you haven’t been productive. It’s the only way she can find out why and address the root cause whatever it may be.

Professional assessment attaches very real consequences to being productive. You can’t let a developer search her own soul about if she should get a raise this year. She would really like to believe she should. However, if you simply want to make your team as productive as possible, let them be honest with you about it (and, by extension with themselves).

Push notifications in the age firewalls

Sat, 17 Oct 2009 12:59:00 -0400

In the world of client-server networking, web browsers are the ultimate clients. They are always sending SYN packets, but never receiving any. They are always establishing outgoing connections, but never get to listen to any ports.

Due to the ubiquity of the web, we have built our networks around the need to browse. Most NATs will not let any incoming connection near the private subnet. Similarly, our corporate firewalls cull incoming connections with extreme prejudice. As a result, an outgoing SYN packet on port 80 is the X86 of network connectivity. It’s as portable a networking tool as it gets, but it’s getting kludgier by the minute.

Polling is a particularly unfortunate result of this outgoing connection based architecture. Take RSS readers: imagine a website that posts 10 new items a day, if you poll this site every 5 minutes, you are wasting your time 97% of the time. Granted, you are getting 304 Not Modified responses without any HTTP response body. But these requests still consume threads, CPU and network resources on the server. At the end of the day we are also only getting 2.5 minutes of average latency, which in computer terms, is not very fast. In short, polling sucks.

The PubSub Hubbub protocol aims to solve this problem by allowing the server to push content to the readers directly. Unfortunately, as explained above, the server cannot establish a connection to a lot of clients out there. You would be able to connect to web based feed readers like Google Reader, because Google understands how to configure a network to allow incoming connections. But the average user either is not permitted to modify their network configuration (office) or don’t know how to do it themselves (home). As cool as PubSub is, it just does not solve the problem at hand.

Which brings me to the actual solution to the HTTP push problem: long polling. With long polling, instead of promptly responding with a 304, the server lets the connection go idle as long as there is no content to send. The client on the other hand, reinitiates a new request as soon as it gets a response. The server also does not need to allocate threads to each incoming request, a single thread can wait on multiple incoming requests and respond to them as new data becomes available. The client gets an almost immediate notification, the server does not have to waste time handing out 304s. Everybody is happy.

With that we have implemented yet another version reverse-tunneling to deal with NAT. Honestly, for the near future, we are going to have to fit the square peg of push notification into the round hole of browser acting as a client.

Come on Neo, Ctrl-C sends SIGTERM, not CTRL-X. (From the...

Mon, 17 Aug 2009 23:41:00 -0400

Come on Neo, Ctrl-C sends SIGTERM, not CTRL-X.

(From the beginning of first movie of course.)

Only Twitter can save us from URL shorteners

Mon, 17 Aug 2009 10:01:00 -0400

Twitter should make URL shortening moot by allowing a link to be included with each Tweet at a fixed character cost of 5-10 characters.

Everybody hates URL shorteners. They slow down page loads, can frame your content, steal your PageRank and even drop dead without notice (and come back alive in a similarly suprising fashion). I mean, even Jeff Atwood thinks they are hack, and he’s a very practical fellow.

To remedy these issues, some well-intentioned people are putting together a spec for webmasters to declare their own short URLs. The thinking goes, other short URL services would first check the target page and use the webmaster specified short URL if one exists. The difficulty of getting URL shorteners to cooperate aside, there is a more fundamental problem here: we should not need short URLs in the first place.

Let me be more direct: Twitter is the only reason URL shorteners are rearing their collective head again. This, at the end of the day, is simply a design defect. While the 140 character limit is what makes Twitter special, the interaction between this limit and URLs at large has the harmful side-effect of requiring third parties to build applications for the sole-purpose of providing 301 redirects. See above for potential issues with such applications.

As new usage patterns emerge for their platform, Twitter do a great job improving their API to provide the smooth, seamless experience that their users have come to expect. Even better, their developer community does a great job including such improvements in their apps. This, after all, is exactly what is happening with the Re-tweet API right now.

The solution to the URL shortening problem is for Twitter to allow one URL to be included with each Tweet for a negligible character cost (say, 5-10 characters). They could then use their default short URL provider for SMS updates, which do have technical character count limit. More importantly, Twitter could sell the click-through stats of all URLs, shortened or not. Power users of all stripes might be willing to pay for this functionality.

Any experienced developer can tell you that there are a million ways of implementing any piece of functionality. Some are just better than others. They cause less exposure to bugs, implementation issues, or even attack surface. Sure, you could try to implement transport-layer encryption with JavaScript but it’s just not the right place to do it. Similarly, making long URLs play nice with the character limit is a task best handled within Twitter’s API by Twitter themselves.

How Dropbox saved my command line

Mon, 10 Aug 2009 13:32:28 -0400

I live in the command line across many Unix-like platforms and networks. I have two Mac laptops, two Linux workstations, shell logins scattered across the multiverse, and even a couple Windows VMs to boot. I’ve tried versioning my shell configuration files, but that requires me to check in/check out my changes across all environments. This turned out to be less than ideal. More manual effort, more hassle.

Enter Dropbox that syncs the files across multiple machines seamlessly and even gives you a bit of revision history. I concocted a system on top of Dropbox that allows me to have global, per OS, and per machine shell configs. For example, here are all the Bash config files I keep in sync via Dropbox (machine names changed to protect the innocent):

$ ls ~/Dropbox/shell/bash
bashbootstrap  bashrc
bashrc-Darwin  bashrc-Darwin-laptopname  bashrc-Darwin-mininame
bashrc-Linux  bashrc-Linux-machineone  bashrc-Linux-machinetwo

Let’s ignore the bashbootstrap file for the moment. You will notice that we have a globally-applied config file, bashrc, two OS specific config files, bashrc-Linux, bashrc-Darwin, and several machine specific ones. (By the way, Darwin is the name of OS X’s BSD-like kernel.)

What ties it all together is the bashbootstrap file. It loads each applicable config file in order of increasing specificity, this allows per OS and per machine overrides to have higher precedence. Additionally, we silently skip missing config files; you need not create empty config files for each of your machines to keep the script happy.

On a new machine, after installing Dropbox on ~/Dropbox, I move away the default .bashrc and just symlink the bootstrap file in its place instead:

$ mv ~/.bashrc ~/.bashrc.bak
$ ln -s ~/Dropbox/shell/bash/bashbootstrap  ~/.bashrc

Oh, and here are the contents of the bashbootstrap file:

if [ -z "$PS1" ]; then
   return
fi

dropboxshelldir=~/Dropbox/shell
dropboxdir=$dropboxshelldir/bash
masterbashrc=$dropboxdir/bashrc
osbashrc=$masterbashrc-`uname`
localbashrc=$osbashrc-`hostname | cut -d. -f1`

echo -n "Applicable shell configs: "
for bashfile in "$masterbashrc" "$osbashrc" "$localbashrc"; do
  if [ -r $bashfile ]; then
    . $bashfile
    echo -n "`basename $bashfile` "
  fi
done
echo

# Set convenience aliases
myed=${VISUAL:-${EDITOR:-vim}}
alias editbashrc="$myed $masterbashrc"
alias editosbashrc="$myed $osbashrc"
alias editlocalbashrc="$myed $localbashrc"

One final note, this script also provides three convenience aliases for editing your Bash config files without having to remember where they are stored.

editbashrc: Edit the global config file.
editosbashrc: Edit the OS-specific config file.
editlocalbashrc: Edit the machine-specific config file.

I only tested this on Bash, but it could work on other Bash like shells. But, as they say, your mileage may vary.

Did I mention that if you sign up to Dropbox with my referral we both get 250 megs of additional storage? Neat!

The poor man’s Bash Tab-completion

Sat, 08 Aug 2009 18:14:00 -0400

Bash has a built-in tab completion utility that sports an impressive array features and the ability to generate suggestions using complex logic. But what if you just want to add a static list of suggestions to a command? It took a little digging around the Bash manual but I finally found the magic words you need to add to you .bashrc file:

complete -o default -W "list of space separated words" [command]

Obviously, you need to replace list of space separated words and [command] with values of your choice. For the curious, let’s take this command apart and see what’s going on:

complete: Built in Bash command for controlling built-in tab completion behavior
-o default: Tell Bash to fall back on the default filename completion if no matches are found.
-W “list of space separated words”: This is where the magic happens. -W allows us to just provide a static list of suggestions. There are other flags that will dynamically evaluate suggestions at runtime, but we’ll leave that to a future post.
[command]: The command that tab-completion will apply to.

Say I want to just add the suggestions “all” and “clean” to the command make. The line I need to append to my .bashrc becomes:

complete -o default -W "all clean" make

Bonus: get a little dynamic

Using command substitution we can also generate the list of suggestions at Bash start time. This is very useful for completing names that can be found in other configuration files (like, say, SSH host-name completion).

complete -o default -W "`echo $(cat /path/to/file | grep 'lines i want')`" \
    [command]

(You might notice that we are using command substitution twice in the example above, once with back-ticks the other time using $(command). This is necessary because the -W argument does not accept new-line characters as delimiters. echoing will convert these new-lines in to spaces.)

Update: The friendly folks over at HN have pointed out that there non-stupid ways of putting together the aforementioned command substitution. Thanks for the correction folks! Here’s a better way to do it:

complete -o default -W "$(grep 'lines i want' /path/to/file | tr '\n' ' ')" \
    [command]

Add ellipses (…) to overflowing text content

Wed, 04 Mar 2009 19:32:00 -0500

Add ellipses (…) to overflowing text content:

This is one of those rare moments when IE has a neat bit of CSS functionality that other browsers don’t have. Sure you could try emulating this behavior in other browsers with the onoverflow event handler, but it would be a hackish and fragile implementation.

Minimalist standards documents aside, some pieces of functionality are best supported by the browser, like contentEditable.

"Participants seeking to dodge troubles [related to derivatives] face the same problem as someone..."

Sat, 28 Feb 2009 19:25:03 -0500

“Participants seeking to dodge troubles [related to derivatives] face the same problem as someone seeking to avoid venereal disease. It’s not just whom you sleep with, but also whom they are sleeping with.”

- Warren Buffet as quoted in the NY Times.

Sita Sings the Blues

Sat, 28 Feb 2009 02:07:00 -0500

Sita Sings the Blues:

Lovely animated film about Ramayana by Nina Paley, the complete thing is available online for free. via kottke.org.

Add your favorite editors to the Mac Finder

Mon, 02 Feb 2009 11:42:00 -0500

Add your favorite editors to the Mac Finder:

I confess, I still use Vim for most of my text editing.

Background meet foreground.

Tue, 16 Sep 2008 17:12:26 -0400

Background meet foreground.

Public interfaces and incremental improvement

Sun, 14 Sep 2008 10:28:00 -0400

Facebook just came out with their new layout. From what I’ve seen on the internets, most people hate it.

I’m not a very heavy Facebook user, so maybe I’m not hitting all the use-cases that my fellow internet citizens are hitting. The only problem I can see with the new layout is that “info” and “wall” now live in separate tabs. Even this is pretty acceptable for me. The wall and info sections of a profile usually work separately, so not having the context of one while using the other is not cumbersome.

If I had to guess (and I do), I’d say that they are trying to “solve” profiles with too many apps installed. There’s simply no way that 10 independently developed applications can live on a single page harmoniously. The page becomes too busy with all the ~~useles~~different apps competing for screen real estate. By creating a tab for each app, sane Facebook users can install more apps without worrying about their MySpace tendencies shining through. To put it another way, with a fixed amount of profile screen real-estate, the benefit of adding a new app diminishes with each installation. With the new layout, you get more screen space as you install apps, and they can be as useful as they can possibly be.

Which brings me to what I really wanted to say: once an interface (UI or API) is public, anything short of opt-in incremental changes are going to cause public uproar. This is regardless of the benefits, the change itself is most burdensome for the users.

Log analysis with MySQL

Mon, 11 Aug 2008 18:37:00 -0400

Digging through log files has got to be the least-sexy aspect of web app development. I’ve found that using MySQL makes this process a little less painful and easier to standardize. At the very least, you don’t end up with 5 perl scripts every time you try to analyze a particular piece of data that you can never reuse. Here’s the workflow that I’ve found to work well for me:

Grab logs with your preferred method (scp, wget, curl you name it)
Select the columns you want out of the log file using cut. If your input is not Tab-delimited, you will have to specify your delimiter with -d and override your output delimiter with --output-delimiter="\t". For example, to select the first and third columns out of a CSV file: cut -d, -f1,3 --output-delimiter="\t".
Create a new MySQL table that has a column per log line column. For example: `mysql -e “create table pageviews (page varchar(32), cputime int)” logs
Import the data from the text file:mysqlimport --local pageviews.txt

Please note: the table that you are importing into needs to have the same name as the text file. So if the file is called httperrors.txt your MySQL table needs to be called httperrors.

That’s about it. Now you can use familiar SQL to run any analysis on your data that you want.

For example: SELECT page, COUNT(1) FROM log WHERE cputime > 500 GROUP BY page ORDER BY 2 DESC;

That will give me all the requests that took longer than 500 ms of CPU time broken down by page in my app. As simple as that.

How is that for data warehousing?

Lego fun snacks

Sun, 22 Jun 2008 17:58:42 -0400

Lego fun snacks

Let me enumerate the times…

Sat, 24 May 2008 09:06:00 -0400

Sometimes you have to iterate over a collection and count the steps you’ve taken. For example, let’s write a really simple function that returns the index of the leftmost occurrence of a given character in the given string.

def indexof(str, char):
  i = 0
  for c in str:
    if c == char:
      return i
    i += 1
  return -1

Or, we could write it as:

def indexof(str, char):
  for i in range(len(str)):
    if str[i] == char:
      return i
  return -1

The second one looks a little bit better. Still, I wish we could write the loop so that the for statement (i.e. the first line of the loop) expresses all aspects of the iteration. In other words, if we are both stepping over the string and incrementing a counter, it is best if they happen simultaneously each time we step through the loop.

Turns out it’s not uncommon to need a counter variable while iterating over the elements of some collection. So Python has a builtin function specifically for this purpose: enumerate (new in Python 2.3). At each iteration it returns a 2-tuple of (counter, object). So our indexof becomes:

def indexof(str, char):
  for index, curchar in enumerate(str):
    if curchar == char:
      return index
  return -1

Now we don’t have to clumsily declare a counter variable and increment it manually. Our final implementation also works with generators and unindexable collections that we cannot subscript (i.e. do collection[index]).

Is it OK to remove code you don't understand?

Wed, 21 May 2008 16:44:23 -0400

No.

Should be obvious but we all do it. People are merciless (or, dare I say, reckless) when trimming Other People’s Code. I was reminded of this fact by a recently well publicized occurrence of let’s-just-remove-the-line-that-warns.

First of all let me just say, yes, you should always check your warnings. They are great for catching your own suboptimal programming practices. And in languages like C where there aren’t almost any foot-shooting prevention mechanisms built into the language, compiler warnings are a godsend (people invented lint for a reason). So it’s no surprise many programmers get indoctrinated at an early age not to ignore compiler/debugger warnings.

Unfortunately, sometimes one gets a little overzealous and removes a line, say, that makes a random number generator, um, random. Apparently such a line of code in the Debian SSL library was generating warnings and had to be silenced with prejudice.

So let this be my plea to the code-literate masses: “Do not remove code you don’t understand.”