Two years ago, according to my diary, I was working on a small but interesting programming project. It turned out quite well — the results are still in use — and perhaps the tale in retrospect is inspirational enough to be worth telling.

A smart and diligent colleague, AP, had performed a "tool inventory" of information resources and computer programs used in various offices. She organized the material into a structured database, with fields for attributes such as tool name, category, date of development, cost, size, operating system, hardware requirements, etc. The challenge: how to make the tool inventory accessible to everyone, so people could easily look up what was available and avoid reinventing wheels?

In November 1998 the tool inventory was a remote island. It lived on an isolated server, reachable by only a few privileged souls, hidden behind the fog of a hostile user interface. AP and I needed to figure out a way to put it on the shared network in a simple enough format that anybody could find what they needed quickly. The "obvious" solution, according to the professional software developers who looked at the task, was to write routines to perform database queries and translate the results into web pages. The folks involved happened to be learning Java at the time, and so they proposed to spend some months building the interfaces in that language.

AP and I were skeptical. The developers, we felt, didn't understand what real end users wanted to do with the tools inventory. Worse, the dynamic interface that they proposed to build sounded likely to be slow, ugly, and limited in its capabilities. And we lacked confidence that they could actually deliver something on time that would work, even in a minimal fashion. Their learning experience, we feared, would teach them how hard their approach was without resulting in a functional product.

But what did we know? We knew that the tools inventory database was relatively static, with many hundreds but not thousands of entries and with a slow rate of growth, maybe a few percent per month. We also knew that most users, most of the time, would want to look up a tool by name (if they already knew what it was called and just needed a few details) or by category (if they wanted to browse and identify candidate solutions to their problems).

This sounded tailor-made for a simple flat structure of static HTML pages, one per tool, plus simple index pages with links to each tool page sorted alphabetically by name or category. By avoiding dynamic server-side fancy database interfaces, we thought we could get quick response and high reliability. We figured that a standard web search engine would be able to spider our pages and build a free-text retrieval system to find any of the words in the tool descriptions — so we would get that feature for free. We knew we could build æsthetically pleasing pages that demanding users would enjoy the look and feel of. And we could see how to do all this by hand. The only remaining challenge was to automate the process, so that whenever the tools inventory database changed (every few weeks or so) we could painlessly rebuild the tool description pages and the cross-reference ("Xref") index pages.

In a sense, then, our solution was to take a snapshot of a slowly-changing database and then publish that photograph. We didn't get permission; we just did it, as a two-person mini-skunk-works, during odd chunks of time we found between other jobs. AP already knew how to export crude HTML from the database, so she took on the task of cleaning that process up and making nice web pages come out of it.

For my contribution, I wrote a tiny "compiler" — a program to take a directory full of individual tool listing HTML files, scan through them, pull out tool names and categories, and then build index pages full of links sorted by those attributes. I wrote the "compiler" in Javascript, an interpreted language with several critical advantages:

• it was free;
• it was available;
• it offered adequate performance; and
• it had excellent pattern-matching and string-manipulation features, which we needed to parse the tool pages and pull out tagged data.

So I crafted the "compiler" in JavaScript. Like many programming efforts, it took only a few hours to do the core routines, but an order of magnitude longer to debug, tweak, generalize, rewrite, document, and explain.

But when we were done, in just a couple of weeks of part-time coding, AP and I had a prototype solution that really worked! It was simple, easy to understand, straightforward to maintain, and extensible to solve new problems or add new features. It was adequately fast: it indexed half a dozen tool pages per second, so a complete rebuild of the cross-reference pages took only a few minutes.

And the fancy solution that the professional software developers were attempting? Somehow it never got finished. They moved on to other tasks, and real people used our quick-and-dirty program.

What were the secrets of our success? Perhaps:

• We understood the ultimate requirements — since we were close enough to our customers to know what they really needed to do.
• We understood the inputs that we had to work with — so we knew what were the big hurdles and what were the non-problems.
• We understood our development tools — so we could gauge the size and duration of the programming task accurately.
• We factored our work efficiently — so we minimized time spent on meetings, interpersonal communications, and redundant overlap of effort, but we maximized productive collaboration.
• We simplified, ruthlessly — so we had fewer challenges to overcome and more reserves for unanticipated (but predictable!) surprises.
• We avoided bureaucracy — by keeping the project small and using only what resources we could scrounge locally.
• We weren't proud — so we were happy when our results were taken over and used by others, and we weren't surprised when neither of us got much recognition for what we did.

Bottom line: both AP and I had fun, learned a lot, and helped our fellow creatures. Who could ask for more?

Sunday, November 26, 2000 at 06:39:30 (EST) = 2000-11-26

TopicProgramming - TopicPersonalHistory

(correlates: SixProjectStages, DavidCopperfieldInFashion, WickedWork, ...)