How Slow Programs Are Like Christmas: a Sales Pitch

My company, Method R Corporation, makes systems faster. And we make people who make systems faster, faster. We train people to become performance optimization heroes.

Here is my story about why you should be interested in our training.

%%

Slow programs remind me of Christmas when I was a kid. In early December, my parents would put a present for me under our Christmas tree. It would be wrapped up so I couldn't see what was inside it. The unwrapping would not happen until Christmas morning, December 25. (That was the best-case scenario. If my Dad couldn't be home because of work, then Christmas morning would come a day or two late.)

So, every day, for nearly a month, I would see that present under the tree and wonder what was in it. 

I'd take any clue I could find. What shape is it? What does it weigh? What does it sound like when I shake it? (Sometimes, my Mom and Dad would prohibit me from shaking it.) No matter how desperate the curiosity, all I could do was guess.

When Christmas came, I'd finally get to tear the paper off, and I would now see, plain as day, what had been in that box the whole time. All the clues and possibilities would collapse into a single reality. Finally, there was no more mystery, no more guessing.

Slow programs are like that. The clues aren't enough. You guess a lot. But with slow programs, there's no specially designated morning when your programs reveal their mysteries to you. They just keep irritating you, with no end in sight. You need somebody who knows how to tear the wrapping paper off those programs so you can see what they're doing wrong. 

That's the somebody I like being. The role scares a lot of people, but it doesn't scare me. That's because I trust my preparation. I know that I have three particular assets that tilt the game in my favor. 

Those assets are knowledge, tools, and community. With the knowledge and tools I have, I don't get stumped very often. But when I do, I have a network of friends who'll help me out. My friends and I can solve just about anything. These three assets are huge for both my effectiveness and my confidence.

These three assets aren't just for me. They're for you, too. 

That's the aim of my online course called "Mastering Oracle Trace Data." In this course, I've bundled everything you need to claim those three assets for your own:

1. You'll learn the details about Oracle traces and the stories they're trying to tell you. This is your knowledge asset.
2. You'll have, for the duration of your choosing, full-feature access to Method R Workbench, the most comprehensive software in the world for mining, managing, and manipulating Oracle traces. This is your tools asset.
3. You'll have access to our Slack channel, a global community of Oracle trace enthusiasts that can help you whenever you get stumped. You won't be alone. You'll have people who are there for you. This is your community asset. 

You can also fortify all three of your new assets by purchasing office hours for the course. Office hours are Zoom sessions, where you can spend time with my friends and me, discussing your questions about the material.

If you're interested in becoming a more effective and confident optimizer, you can get started now. Just visit our course page for details.

%%

That's my story. I hope you'll contact me at method-r.com if you're interested.

And if you like stories like this, you'll find a lot more in my How To Make Things Faster book, available wherever books are sold.

A Design Decision

This week, my team at Method R devoted some time to an enhancement request that required an interesting design decision. This post is about the analysis behind that decision.

The enhancement request was for our flagship product called Method R Workbench. It's an application that people use to mine, manage, and manipulate Oracle trace files.

One of its features, called mrskew, is a tool that allows a Workbench user to filter, group, and sort the raw dbcall and syscall data that Oracle Database processes write to their trace files. You can use mrskew from within the Workbench application, or from a *nix command line.

Here's an example of a mrskew command. It's what you would use to find out how long your program spent reading Oracle blocks from secondary storage. It will show you which blocks it read, how long they took, and how many times each block was read:

mrskew --name='db.*read' \
  --group='sprintf("%8d %8d", $p1, $p2)' \
  x_ora_1492.trc

Here's the output:

sprintf("%8d %8d", $p1, $p2)  DURATION       %  CALLS      MEAN  ...
----------------------------  --------  ------  -----  --------  
                  2        2  0.072918    1.0%     26  0.002805  ...
                 33   698186  0.051940    0.7%      1  0.051940  ...
                 50   339841  0.049261    0.7%      1  0.049261  ...
...

The important thing in this report is the meaning of the $p1 and $p2 variables. The combination of these two variables happens to represent the data block address (the file number and block number) of an Oracle block that was read by some kind of an Oracle read call. It would be nice for the report to tell you that instead of just telling you that the first two columns of numbers are the output of an sprintf function call.

We have a command-line option for that. The ‑‑group-label option lets you assign your own title for the group column. So, with some careful character counting, you could use…

‑‑group-label='    FILE    BLOCK'

…to get exactly the heading you want:

    FILE    BLOCK  DURATION       %  CALLS      MEAN  ...
-----------------  --------  ------  -----  --------  
       2        2  0.072918    1.0%     26  0.002805  ...
      33   698186  0.051940    0.7%      1  0.051940  ...
      50   339841  0.049261    0.7%      1  0.049261  ...
...

That makes sense. Now it's easy to see that Oracle has read one block (file #2, block #2) 26 times, consuming a total of 0.072918 seconds reading it.

The group label fits the output, only because of the careful character counting. The enhancement request was to allow the ‑‑group-label option to take an expression, not just a string. Like this:

--group-label='sprintf("%8s %8s", "FILE", "BLOCK")'

That way, he could print out the header he wanted, perfectly aligned, by just syncing his ‑‑group‑label expression to his ‑‑group expression, without having to count space characters that are literally invisible.

It's a smart idea. The group label option should have been designed that way from the beginning. We eagerly approved the enhancement request and began thinking about the design.

When we thought it through, we ended up with two different ideas about how we could implement this new idea:

1. Redefine ‑‑group‑label to take an expression instead of a string. mrskew will calculate the value of the expression before printing the column label.
2. Create a new option, say, ‑‑new‑group‑label, that takes an expression as its argument. And leave ‑‑group‑label as it is.

The first idea is how the enhancement request was worded. The second idea entered our minds because the first idea creates a compatibility problem: if we change the spec of the existing ‑‑group‑label option, it will break some existing mrskew scripts. For example, these will work in Workbench 9.2.5:

--group-label=FILE
--group-label="FILE BLOCK"

But if we redefine ‑‑group‑label to take an expression instead of a string, then these won't work anymore. People will need to quote their string expressions like this:

--group-label='"FILE"'
--group-label='"FILE BLOCK"'

In the end, we decided to redefine the existing option and live with the compatibility breach.

The way we make decisions like this is that we create strenuous arguments for each idea. Here are some of the arguments we considered en route to our decision.

First, the customer experience (cognitive expenditure).

Everyone who participated in the debate had the customer experience foremost in mind. But how can we objectively measure "customer experience"? How do you structure a scientific debate about the superiority of one experience over another?

One way to do it is to measure cognitive expenditure—the amount of mental effort that a user has to invest to get the desired outcome from our software. We want to minimize cognitive expenditure, to maximize a customer's return on investment of effort.

We began by realizing that responding to this enhancement request with one of our two ideas would necessarily force the user into one of two new regimes:

1. The syntax of ‑‑group-label has changed.
2. There's a new ‑‑new-group-label option.

In regime 1, our users would have to learn the new syntax. That's a cognitive expenditure. But it's a one-time expenditure, which is good. The new syntax would be consistent with the existing ‑‑group syntax, which is actually a cognitive savings for our users over what we have now. However, if a customer had saved any scripts that used the old syntax, then the customer would have to convert those scripts. That's a cognitive expenditure in a loop (one for each script), which is bad.

In regime 2, our users would have to learn about ‑‑new-group‑label, which is a cognitive expenditure. They'd still have to remember (or relearn) about ‑‑group‑label, too, which is a similar cognitive expenditure as the one in regime 1. They wouldn't have to modify any old scripts, but they would have to make the choice of whether to use ‑‑group‑label or ‑‑new-group‑label, every time they wrote a script in the future. That's another cognitive expenditure in a loop (one for each script), which is bad.

Second, the developer experience (technical debt).

We also need to consider the developer's experience. We don't want to create code that increases technical debt that makes the product unnecessarily difficult to support.

If we redefine ‑‑group-label, there's no long-term effect to worry about. But if we add ‑‑new‑group‑label to the story, I would expect for people to wonder, why are there two such similar group label options, when one (the one that takes an expression) is clearly superior? And why does the inferior one have the better name?

At some point in the future, I envision wanting to clean up the cruft and have just the one group label feature. Naturally, the right name for it would be ‑‑group‑label. But of course, changing the spec that way would introduce a compatibility problem. To make things worse, this would occur in the future when—one would hope, if our business is growing—such a decision would impact even more customers than it would today. So then, why create the cruft in the first place? It'll be a worse problem later than it is now.

The question that really seals the deal, is who will the change really affect? It's really a probability question about customer experiences.

Most users who use the Workbench application will never experience our group label option directly. It's there for everybody to use, but our Workbench has so many predefined reports built into it, most users never need to touch the group label option for themselves. When they do need to modify it, they're usually tweaking a report that we've predefined for them, which is a low–cognitive-expenditure experience.

In the end, Method R Corporation bears almost the entire cost of the ‑‑group‑label redefinition. It required us to revise:

• The code for mrskew ‑‑group‑label
• The prepackaged actions in Method R Workbench
• The mrskew .rc files
• The mrskew manual page
• The Mastering Oracle Trace Data book
• The Mastering Oracle Trace Data course

Most users will experience the benefit of the ‑‑group‑label change, without ever knowing that, once upon a time, it changed. And that's the way we want it. We want the product to be as smart as possible so that our customers get the most out of their investment, both cognitive and financial.

Catchphrase

It finally happened.

Several years ago, I asked my friend Mauro Pagano to help me spread a new catchphrase. I had made up this little saying. It was weird enough that I knew, if I heard it in the wild somewhere, then it had to have come from me. It would be a fun little experiment.

The only problem with this new catchphrase was the "catch" part. It was a perfectly good "phrase," but we were having a hard time coming up with situations where we could use it. We agreed we'd keep on the lookout. 

Skip forward several years.

Today, my colleague Jeff Holt and I had lunch at Weinberger's Deli, as we do at least every week or two. We got there just a few minutes after they opened, but people were already queued out the door. That's ok, the line would work its way down shortly.

As we advanced nearer the counter, I found myself increasingly distracted by The Table Situation. No matter how full Weinberger's is, we're almost always able to find a table. But today, since everyone ahead of us had just sat down, it was easy to imagine that no tables would be opening up anytime soon.

This wasn't just idle worry. It was required prep for the dreaded question, "For here? Or to go?" It was already 96ºF outside, on its way to 106ºF, so we'd prefer not to sit out there. But also we would even more prefer not to take our lunch in bags back to the office. We took a risk, mitigated by the two open tables outside. We answered, "For here."

While I was paying for my order, a guy finished his lunch, leaving an open two-top. Victory. We would eat inside, comfortably, at a table. 

I shouldn't be surprised. I almost always get a table. The whole thing—worry, worry, worry, get a table—it happens nearly every time. But somehow, I'm almost always surprised.

But today, in a burst of excitement, a little program that's been running in the background of my mind for years popped its confetti cannon: THIS is the perfect situation for my catchphrase!

So I used it. Here it is:

It's like mouse balls. You think it can't possibly work, but it almost always does.  

A Side Trip to Wonderhell

I should be jubilant. But I'm not. 

I'm habitually non-jubilant. I don't know why.

This weekend, I returned from the kind of a trip that I don't take very often: an all-by-msyself vacation. This year, I attended the IPMS/USA National Convention 2023 in sunny San Marcos, Texas. IPMS is the International Plastic Modelers' Society. The convention is three things: a contest, a meetup, and a vendor showcase. This is the first time I've ever attended. 

The contest is a big deal. Some people spend years preparing for it. And I won two gold medals for a model (a "replica") that took me seven years to make. It was awarded 1st place in its category and then it was named the best of its class: one of only eight such awards among a field of over 3,000 entries.

I should be jubilant. But I'm not.

There are modelers out there who have worked hard over the past who-knows-how-long, suffering now that their work didn't result in the level of appreciation they had hoped for. They probably have a right to be offended that I'm not jubilant.

One of the reasons I'm not jubilant is because I promised myself before the show that I wouldn't get too emotionally wrapped up in how my entry was judged. There are just too many things that could conspire that could deny—rightfully or not—my little model from winning a prize. So, before the show, I preloaded my mind with the following sentence: 

When you put your ego in other people's hands, you're asking for trouble. 

My wife and I have similar conversations with our kids. As she says, sometimes we need to ride the monorail, not the roller coaster. For example, when you believe you're awesome because a local sportswriter says that you are, then it's also tempting to believe that you're trash when that same writer doesn't include you in his all-area team. If you get too high or too low because of how someone else defines you, you're almost guaranteed to crash eventually. 

That sentence of mine sustained me for the first three days of the show. The 3,000+ models on display in the contest room were absolutely overwhelming. There's no way I could walk in that room and feel like a legitimate competitor, no matter how hard I knew I had worked on my model. It would have been ludicrous to think I was going to win something. Even when I did, it felt like it couldn't be real.

Now that I'm home, I'm in this weird predicament of having summited an audacious, multi-year goal, certainly a good feeling, but simultaneously there's this looming feeling of "now what?" Sure, I won this huge award three days ago, but what have I done lately?! 

It makes me feel like there's something wrong with me. But I know I'm not the only one. What I'm feeling happens so often, it even has a name: Laura Gassner Otting calls it Wonderhell.

At the end of all this, the feeling I'm trying to savor and cultivate is gratitude. Thankful for the awards, of course, but also thankful for what I really was hoping for when I signed up for the show. What really drove me while I was building my model was the hope that people might look at my model, enjoy it, and want to talk about how I built it. 

That conversation happens every month in my club (my model couldn't have been what it was without my club). And it happened enough times in San Marcos to make it fully worth submitting myself to the harrowing competition part of the convention.

Maybe I'll do it again someday, I'm not sure. But for now, my monthly club meetings will be enough. Right now, I need to focus on improving my business so much that someday soon I can blog about being oddly not jubilant about that.

A Better Way to Think about %

A lot of people get confused by the "%" symbol. I can understand why. Even https://en.wikipedia.org/wiki/Percentage seems way more confusing than it should be.

Well, maybe I can help. 

Here's a simplifying little idea that I learned by reading ISO 80000-1: 

The percent symbol (%) is just a constant, just like π or e. Its value is 0.01 (or 1/100, if you prefer).

Let me show it to you in a table. Maybe that'll clear it up:

Symbol	Value
π	≈ 3.14159
e	≈ 2.71828
%	= 0.01

What this means is that anywhere you see the "%" symbol, you're free to substitute the value 0.01 if you want:

50% = 50(0.01) = 0.5

So, how does that help? Well, it gives you a simple rule you can apply instead of having to intuit how to convert something to or from a percentage. 

For example, I used to find myself wondering, "If I want to convert this percentage to a real number, do I multiply by 100? Or divide?" I hate memorizing crap like that.

But knowing that % = 0.01 makes it easy. For example, converting 42% to a number without the % sign, I simply substitute, like this:

42% = 42(0.01) = 0.42

When you know that % = 0.01, it's easy to see that 100% is just another way of expressing the number 1:

100% = 100(0.01) = 1

Converting a number to a percentage is easy, too. 

I can of course multiply anything I want by 100% and still have the same quantity I started with. Here's how to convert 0.0005 to a percentage:

0.0005 = 0.0005 × 1

= 0.0005 × 100%

= (0.0005 × 100)%

= 0.05% 

Yep, ISO 80000-1... I don't do everything it says, but this percentage thing was a nice revelation.

Version 1 Is Never the Answer

I create for a living. I make presentations and software and books... Even my hobbies are creation hobbies. I make tools and furnishings and, these days, “museum quality aviation replicas.”

Many of the things I’ve made have worked out really well, but one result I can count on: my version 1 of pretty much anything is rarely going to be a keeper. My good work is almost always a v2, or v3, or beyond. When I succeed, it’s usually more a persistence thing than a genius thing. 

For the past couple of weeks, I’ve been working on a 20-minute session I’ll present at P99Conf. Creating a presentation, for me, is a mostly private experience. I simulate in my mind what an audience might like to see. When either I think I’m finished, or I feel stuck, the next step in my process is to find an audience and go through the material. In other words, I use the material in front of other people.

Today, because I felt stuck, I showed my P99Conf material to the audience on my Tuesday Zoom call.

Here’s what I learned. The slides I had made were awful. Not nearly relevant enough to match my audience. I was already off the rails by the fourth slide. In the discussion though, the discussion helped to put me on the right track. The group was very helpful, identifying a couple of slides that were actually pretty good, and suggesting fixes for the worst parts. I went into the call, stuck. I came out inspired, ready to get back to work.

The thing that I suspect might surprise you, that did not surprise me, is how this process felt. I did not feel insulted or denigrated or “vulnerable.” The feelings I did have, because of the call, were these:

1. Gratitude – I was appreciative that the group would lend me their time
2. Motivation – I was inspired now, and interested; instead of stuck
3. Shame – I was a tiny bit embarrassed that my v1 hadn’t been better
4. Confidence – I knew my work was going to be better because of the review

I did not feel surprised, because I don’t expect any v1 to be a production release. Most of my v1’s I expect to throw away. A v1 is simply part of the process; you can’t have v2 without v1. More precisely, and this is the critical realization in the whole process: 

You can’t have v2 without feedback, and you can’t have feedback without v1.

My job, then, is not to try to create a v1 that is production worthy. It is to create a v1 that is feedback-worthy. And then go get the feedback. That feedback loop is essential. It’s not a step that I aspire to ever “optimize away.” It is a vital part of the process that I embrace.

If you’re interested in this story, then you might be interested in a paper I wrote in 2011, called “My Case for Agile.” 

Why the Oak Table Was So Great

This weekend, I watched a wonderful TEDx video by Barbara Sher, called “Isolation Is the Dream-Killer, Not Your Attitude.” Please watch this video. It’s 21 minutes, 18 seconds long.

It reminded me about what was so great about the Oak Table. That’s right: was. It’s not anymore.

Here’s what it was. People I admired, trusted, and liked would gather at a home in Denmark owned by a man named Mogens Nørgaard. Mogens is the kindest and most generous host I have ever encountered. He would give his whole home—every inch—to keep as many of us as he could, for a week, once or twice a year. Twenty, maybe thirty of us. We ate, drank, and slept, all for free, as much and for as long as we wanted. 

And the “us” in that sentence was no normal, regular, everyday “us.” It was Tom Kyte, Lex de Haan, Anjo Kolk, Jonathan Lewis, Graham Wood, Tanel Põder, Toon Koppelaars, Chris Antognini, Steve Adams, Stephan Haisley, James Morle, John Beresniewicz, Jože Senegačnik, Bryn Llewellyn, Tuomas Pystynen, Andy Zitelli, Johannes Djernæs, Michael Möller, Peter Gram, Dan Norris, Carel Jan-Engel, Pete Sharman, Tim Gorman, Kellyn Pot'Vin, Alex Gorbachev, Frits Hoogland, Karen Morton, Robyn Sands, Greg Rahn, and—my goodness—I’m leaving out even more people than I’m listing.

We spent a huge amount of our time sitting together at Mogens’s big oak table, which was big enough for about eight people. Or, in actuality, about twice that. We’d just work. And talk. If there wasn’t a meal on the table, then it would be filled with laptops and power cords covering every square inch. Oops, I mean millimeter. That table had millimeters.

And here’s what was so great about the Oak Table: you could say what you wanted—whatever it was!—and you could have it. You could just say your dream and your obstacle, and someone around the table would know how to make your dream come true.

It’s tricky even trying to remember good examples of people’s dreams, because I’m so far removed from it now. Some of them were nerdy things like, “I wonder how long an Oracle PARSE call would take if we did a 256-table join?” You’d hear, “Hmm, interesting. I think I have a test for that,” and then the next thing you know, Jonathan Lewis would be working on your problem. Or, “Hey, does anyone know how to do such-and-such in vim?” And Johannes Djernæs or Michael Möller would show you how easy it was.

I got into a career-saving conversation late one night with Robyn Sands. She had asked, “Is anybody else having trouble finding good PL/SQL developers? I can’t figure out where they are, if there even are any. Are there?” We talked for a while about why they were so scarce, and then I connected the dots that, hey, I have two superb PL/SQL developers at home on the bench, and I had been desperately trying to find them good work. The story that Robyn and I started some 3:00am over beers resulted in a superb consumer femtocell device for Robyn and a year’s worth of much-needed revenue for my tiny little team.

It was a world where you could have anything you want. Better yet, it was a world where you could dream properly. Today, in isolation, it’s hard to even dream right. After nearly two years of being locked away, I can barely conceive of a world that’s plentiful and joyous like those Oak Table years. I feel much smaller now. (Oh, and it wasn’t COVID-19 that killed that Oak Table experience. It died years before that—but, obviously, it’s a factor today.)

I want it back. I want my friends back. How are we going to do this?