Bailing Frantically
For hints, tricks and ideas about better ways to build embedded systems, subscribe to The Embedded Muse, a free biweekly e-newsletter. No hype, just down to earth embedded talk. Click here to subscribe.
Bailing Frantically
Sadly, business is booming.
In my last real job I developed and sold in-circuit
emulators. Talk about a tough product! The designer needs deep insight into
every quirk of the target processor, which never works quite as advertised. The
firmware is huge and hideously complex, and insane timing requirements make the
hardware equally difficult. Just the sort of thing that appeals to geeks like
me. But over the years I found an even more fascinating aspect of the business:
the customers.
Anyone using a debugging tool is living on the edge of
product development. They're in that difficult phase of the project where
things just don't work. Typically the schedule is in a state of collapse, the
boss's temper flares, and everyone is tired from working too much overtime. In
my role as an ICE vendor I worked with thousands of these developers, looking
metaphorically or physically over their shoulders as they struggled to complete
their projects. I found that few - astonishingly few - follow any sort of
even vaguely-disciplined process. Yet the minority who did use reasonable
methods outperformed all other developers by astonishing margins. They delivered
good code fast. The difference in performance of the two sorts of teams was so
staggering I decided to sell the business and become a firmware evangelist.
So now I write and lecture about better ways to create
firmware. So many projects are in so much trouble that the demand for this is,
sadly, quite high.
It's incredibly satisfying to see how a little bit of
information and encouragement can change a development team. But a couple of
times a month I hear from a group that can neither attend one of my lectures nor
come to an Embedded Systems Conference because they're desperately behind on
the latest project. Further inquiry generally shows that the group is always
in panic mode. There's no time to learn how better development techniques can
bring projects to fruition on-time.
Like sailors on a sinking ship they are too busy bailing to
fix the leak. The water slowly rises so they bail ever more frantically. Sooner
or later they're going down, but working faster and harder staves of the
inevitable end for just a while longer.
An old cartoon shows a fierce battle, the soldiers wielding
swords and spears. The general turns away a machine-gun salesman. He complains:
"I don't have time to talk to you - can't we see we're fighting?"
The brochure for this year's ESC Boston just arrived. It
advertises over 75 classes on all aspects of building embedded systems. There is
no other venue that packs so much information into a single week. A press
release from conference management suggests that about 5000 attendees are
expected. That's a healthy increase over last year, but, in my opinion, an
appallingly low number.
We're part of a fraternity of about a quarter million
embedded developers. Why is it only 2% of us take advantage of this incredible
learning opportunity? Is everyone too busy bailing?
Elusive Perfection
Why are so many firmware projects so late and so
bug-ridden? A lot of theories abound about software's complexity and other
contributing factors, but I believe the proximate cause is that coding is not
something suited to Homo Sapiens. It requires a level of accuracy that is truly
super-human. And most of us ain't superman.
Cavemen did not have to bag every gazelle they hunted -
just enough to keep from starving. Farmers never expect the entire bag of seeds
to sprout; a certain wastage is implicit and accepted. Any merchant providing a
service expects to delight most, but not all, customers.
The kid who brings home straight As (not mine, darn it)
thrills his parents. Yet we get an A for being 90% correct. Perfection isn't
required. Most endeavors in life succeed if we score an A, if we miss our mark
by 10% or less.
Except in software. 90% correct is an utter disaster,
resulting in an unusable product. 99.9% correct means we're shipping junk.
100K lines of code with 99.9% accuracy suggests some 100 lurking errors.
That's not good enough. Software requires near-perfection, which defies the
nature of intrinsically error-prone people.
Software is also highly entropic. Anyone can write a
perfect 100 line-of-code system, but as the size soars perfection, or near
perfection, requires ever-increasing investments of energy. It's as if the
bits are wandering cattle trying to bust free from the corral; coding cowboys
work harder and harder to avoid strays as the size of the herd grows.
So what's the solution? Is there an answer?
In my opinion, software will always be a problem, and there
will never be a silver bullet that satisfies all of the stakeholders. But there
are some well-known, though rarely practiced, strategies that offer hope.
The first is to fight featurism. A nifty feature is just a
software change, right? We can slap in a bit of extra functionality in no time
at all. This attitude has changed cell phones from basically simple devices to
million line-of-code firmware monstrosities. The easiest feature to implement is
the one that's not there. Prune unneeded requirements to get the code done
better and faster. Unhappily, customers - or at least the marketing droids -
usually demand far more than any product really needs. We developers usually
have little influence over the requirements.
Relentlessly partition the system. Cheat complexity growth
by breaking the system up into many small pieces, each of which is as
independent as possible. Minimize coupling between functions and modules. As the
firmware evolves invariably coupling increases. We add a few more parameters in
function calls, sneak in a global or 10, and morph the simple keypad driver into
a display/keypad/LED handler. Though I struggle with a lot of what the XP folks
mandate, they laudably require that crummy code gets refactored rather than
beaten into submission.
Practice reuse. You won't; no one does. But reuse is, in
my opinion, our only hope of salvation. Unfortunately, reuse goes against the
nature of early 21st century capitalism, which stresses short term
profits uber alles. Reuse fails unless one makes a large up-front effort in
generalizing functions. Few bosses accept the argument "if we double
development time on this project, we can halve it on the next dozen."
Artistes
Most importantly, use a disciplined development process.
In the olden days all products were hand-made, the work of
artistes rather than manufacturers. Plows, shears and guns were all turned out
one at a time, each was subtly different than the one the blacksmith finished
just the day before. Over time industry learned about the value of making
standardized, interchangeable parts, and ultimately about mass production.
Though there's long been a retro movement that admires handwork, for most
products mass production has yielded far better products at much lower costs.
But not in software, because we view ourselves as artistes,
cranking out hand-made code highly optimized for each specific application.
Engineering will never resemble a factory, but the utter chaos that rules most
firmware shops is unacceptable.
I estimate under 2% of firmware developers have any sort of
process that they use religiously. It matters little what that approach
may be: XP, CMM, Inspections, Standards, and maybe even Drunken Orgies are all
valuable ways to improve the code! when used religiously. None offer
much benefit when casually or intermittently employed.
We software artistes have missed the "process" boat.
Most other industries use various sorts of defined processes to work
efficiently. One way to identify an amateur organization of any sort, be they
accountants, lawyers, craft shops or software developers, is by a lack of
process. By contrast, an efficient company like MacDonald's defines a rigorous
way to do just about everything. Even a teenager, using Ray Kroc's process,
can crank out Big Macs that taste exactly as bad as in any MacDonald's in the
world.
We firmware folks must look beyond the world of software
engineering for insight into better ways to build our products. I highly
recommend Michael Gerber's book "The E-Myth Revisited: Why Most Small
Businesses Don't Work and What to Do About It" (HarperBusiness, 1995 ISBN
0887307280). Skip the book's irrelevant last half. Gerber says that poor
businesspeople work "in" the business - they are technicians daily busy
with making the product or service. The business cannot succeed without that
individual, who may be a genius at providing some product or service, but spend
their days firefighting. He feels the brilliant company owners work "on" the
business. They build systems, processes, and techniques so the business runs
smoothly. These awesome managers don't just solve problems, they invent
solutions that eliminate the problem forever, or that automatically deal with
the issue when it comes up again.
They stop bailing and plug the leak.
We must adopt the same philosophy. For instance,
I've observed that most teams spend 50% of the project debugging the code. Try
something else - maybe pair programming a la eXtreme Programming, or (in my
opinion much better) code inspections.
Developers often blame management for lousy development
methods. They get neither support nor direction from the boss, who is
preoccupied with just shipping the product no matter what it takes. So the
developers employ heroics, working ever harder to produce bigger and more
complex products.
Nonsense. We cannot abdicate our responsibility to do
things right. Another great non-software book is "Quality Is Personal: A
Foundation for Total Quality Management" by Harry Roberts (Free Press, 1993,
ISBN 0029266254). The author contends that "working harder" is a lousy way
to solve problems. The couch potato who promises to watch less TV will probably
fail. Better: make a profound change. Throw the TV away, or put it on a timer.
Dieting? Empty the fridge, buy only low-cal foods, don't carry money so you
don't eat out, but stop making futile promises to do better.
Roberts' philosophy applies to software engineering. Are
variable names always totally meaningless? Skip that meeting where everyone
promises to do a better job with variables on the next project. It'll fail.
Instead, change something. Maybe adopt naming conventions and software
standards. Appoint a naming czar. Build a tool that identifies stupid names.
If the boss is indeed capricious and unsupportive, if he
thwarts every attempt to improve engineering, you can still change something.
Adopt a stealth process, one that requires no buy-in from management, and one
you can implement with neither his consent nor knowledge. One that's great is
Watts Humphrey's Personal Software Process. The PSP will improve your work
even if everyone else on the team is a cowboy.
Conclusion
Most of the developers I talk to do have at least an
inkling about how to get their efforts on track. Some understand the potential
benefits from using a reasonable process, and others have adopted some or all of
these techniques. The vast majority, though, sheepishly admit that they do not
employ them religiously.
If the pimple-faced MacDonald's worker doesn't change
the deep-frying oil at prescribed intervals, all of the food will be even worse
than normal. So the company has a system that insures oil changes happen
on-schedule, all of the time. If we don't adopt the same determination to
always use a decent process - whatever it is - our code will be a mess.
Guaranteed. Cranking C code is, to me, less interesting than finding better ways
to crank C.
Stop bailing and plug the leaks.
|
