How to make programming hard for yourself I posed a simple question: why do we have trouble admitting that programming is hard?

This generated a tempest in a teapot of comments. Most focused on the red herring of whether programming is an innate talent. That's actually irrelevant to the fundamental question.

Is programming actually hard?

The evidence is in that software development as an engineering practice is very hard. One person on reddit commented that he had worked for three successive game companies that shipped working software on time. Other than that comment, my own experience, the experience of everyone I've ever talked to, and the industry folklore is that software projects are almost always late, short of promised functionality, buggy, or all three.



Now I admit that lots of people think that this is a management problem. Their view is that the actual programming is quite tractable, it's just that getting people to work on the right things at the right time is the hard part of software development. This argument takes the same world view that the Egyptian engineers had about building pyramids: the labour is so easy that ignorant slaves can do it.

My view is that this is true for a certain class of programs. Most of these can be found in current business contexts. Quite honestly, most business applications are what I would call wide and shallow. There's a lot of work to be done, but little of it requires deep thinking. The challenges are around managing requirements and user expectations. These are hard problems, but they are not in the same class as, say, figuring out how to implement a fast join on two distributed hash tables.

I think that easy programming problems drive a lot of decisions about people and tools. If you perceive that a problem isn't very hard, why hire the 'cream' of the programming crop to solve it? And if you eschew the cream for a plain glass of 1% milk, won't you insist that the program be written in Blub instead of Python, Ruby, or Lisp?

This becomes a feedback loop. The 'B' players become hiring managers one day and they hire 'C' players. The 'C' players are asked to design applications, and they design things that they know how to build. This second effect is a huge lever pushing programming problems downwards in business.

Programs are written to solve problems posed by people, be they programmers, program managers, business analysts, or what-have-you. The difficulty of writing the programs is driven by the difficulty of the problems. The difficulty of the problems is limited by the imagination of the people posing the problems.

If your feedback loop is pushing difficulty downwards, the only problems you will have are the ones that can be solved by anyone who can pick up a Learn Blub in Twenty-One Days book.

Is this a problem?

Quite honestly, no. If what you want is a steady paycheque, and you can sleep at night knowing that your job might be outsourced one day, there's nothing wrong with chasing the easy work. It's no different than playing saxophone in a dance band at a wedding. Charlie Parker did it, you can do it.

But just because almost everybody does easy work doesn't mean there isn't hard work to be done. Hard problems are their own reward for some people, myself included (in my case I mean "hard for me," I know more than a few people that consider my hard problems to be recreational diversions). And some hard problems have lucrative consequences if you have a taste for starting companies.

Why it's hard to find hard problems

It's hard to find problems that are so challenging that you don't know how to solve them. It's especially hard if you also want a problem that can actually be solved. And it's very, very hard to find a hard problem that makes business sense, even in a startup. Most good startups solved the right problem rather than the hardest problem.

Very few people have the luxury of writing requirements for a program that they have no idea how to write, much less the imagination to see the results. Jump in your time machine and zoom back to 1980. Start asking people do sketch a word processor design on a napkin. How many people specify bit-mapped graphics and mouse-driven selection?

I compared programming to music and athletics. I suggested that programming is hard and the reason we don't see that is because we aren't competitive enough. That may be true, but now I sense a deeper challenge: in programming, we tend to pose questions for which we already know the answer.

In music, most people's reach far exceeds their grasp. It's quite easy to find pieces of music that are difficult or impossible for someone to play but nevertheless pleasing to that same person's ear.

If you are a wanna-be bassist and you listen to Jaco Pastorius' harmonics on "Portrait of Tracy," it is immediately obvious to you that you could be better and that being better is worthwhile. I leave it to your imagination to draw the same parallel to sports and athletics.

But in programming... we have a nasty habit of being very satisfied with the tools we already know. After all, they are good enough for all of the programs we have ever written. And, surprise surprise, they just might be good enough for all of the programs we will ever want to write.

How to make life hard for yourself

Most of us need outside forces to shake us up. Many bassists were shocked in 1976 when they heard Portrait of Tracy. Harmonics were used as accents here and there, but nobody dared to play an entire piece with harmonics as the main technique. This shook things up, and now playing harmonics is considered a basic skill on electric bass (although it's still fiendishly hard to play Portrait of Tracy!)

For all of my professed admiration of Ruby on rails, I personally don't think that easier and more productive CRUD application writing will shake things up. I personally care very much about writing applications in a tenth of the time, but using Rails is like listening to Jaco Pastorius. The real learning experience comes when you try to duplicate the feat.

If you want Rails to help you improve as a programmer, you need to look at the source code and figure out how to use Ruby to write Domain Specific Languages. Next, you'll have to write a few DSLs to become immersed in how programming changes when you can write DSLs. Now you're ready to look for a problem that is barely possible if you can solve it with a DSL.

Before you knew how to write your own languages, you probably couldn't design a program that couldn't be written without a DSL. Now you can.

This general template works surprisingly well: Learning things that force you to solve existing problems in new ways will help you find hard problems that can only be solved with the aid of your new techniques.

So when you're done learning how to write DSLs just like Rails, go ahead and write a web application using continuations as the basic metaphor. And don't forget to look under the hood and figure out how the framework makes continuations scale in a web context.

Come to think of it, figure out how to write continuation-based Rails applications that live on a Google-scaled grid and you might be headed towards a place where the inhabitants have no fear of hard.

I'll see you there.

Update: I found an early take of Jaco playing Portrait of Tracy (MP3). Nice.

Labels: passion

  ¶ 2:36 PM