Brown Noise in Written Language

Today’s email brought a loaf of spam with this in it:

[Name withheld] is a world-class developer and provider of leading-edge solutions that help customers optimize the physical infrastructure through simplification, agility, and operational efficiency.

This passage is the informational equivalent of this audio file. If you can read it without feeling sad, sarcastic, vaguely scummy, or bitter about humanity’s perverse unwillingness to combine thought and language in a useful way, then I beg you to read Revising Prose and Rework.

Please.

That is all.

My Actual OTN Interview

And now, the actual OTN interview (9:11) is online. Thank you, Justin; it was a lot of fun. And thank you to Oracle Corporation for another great show. It's an ever-growing world we work in, and I'm thrilled to be a part of it.

New Method R Blogs

Today we installed two new blogs over at our method-r.com web page. We created them to give us—all of us at Method R—a place to talk about our products and professional experiences. I hope you'll come have a look.

I'll still be posting here, too.

My OTN Interview at OOW2010 (which hasn’t happened yet)

Yesterday, Justin Kestelyn from Oracle Technology Network sent me a note specifying some logistics for our OTN interview together called “Coding for Performance, with Cary Millsap.” It will take place at Oracle OpenWorld on Monday, September 20 at 11:45am Pacific Time.

One of Justin’s requests in his note was, “Topics: Please suggest 5 topics for discussion?” So, I thought for a couple of minutes about questions I’d like him to ask me, and I wrote down the first one. And then I thought to myself that I might as well write down the answer I would hope to say to this; maybe it’ll help me remember everything I want to say. Then I wrote another question, and the answer just flowed, and then another, and another. Fifteen minutes later, I had the whole thing written out.

I told Justin all this, and we agreed that it would be fun to post the whole interview here on my blog, before it ever happened. And then during the actual interview, we’ll see what actually happens. It’ll all be in Justin’s hands by then.

So, here we go. Justin Kestelyn’s interview, “Coding for Performance, with Cary Millsap.” Which hasn’t happened yet.

◆ ◆ ◆

Justin: Hi, Cary. Welcome to the show, etc., etc.

Cary: Hi, Justin. It’s great to be here. Thank you for having me, etc., etc.


Justin: So tell me, ... What is the most important thing to know about performance?

Cary: Performance is all about code path. There are only three ways that a program can consume your time. There’s (1) the actual execution of your program’s instructions. There’s (2) queueing delay, which is what you get when you visit a resource that’s busy serving someone else (CPU, disk, network, etc.). And there’s (3) coherency delay, which is when you await some other process’s permission to execute your next step. The code you’re running controls all three of those ways you can spend time. So understanding performance is all about understanding code, whether it’s Java or PHP or C# that you wrote, or the C code that the Oracle Database kernel developers have written for you.


Justin: Is tuning SQL or PL/SQL any different from tuning Java or PHP or C#?

Cary: The tools are a little different, but the fundamentals are exactly the same. You find out which code path in your application is consuming your time, and then you go after it. The best thing to do is figure out a way to execute that code path less often (because the fastest way to do anything is to not do it at all). The next best thing to do is try to figure out a way to make any instructions that can’t be eliminated, faster. That’s the whole trick.


Justin: You make it sound easy.

Cary: It usually is easy once you can collect the data you need to guide you. ...Once you know how to get the system to tell you where it’s spending your time. People make it hard on themselves anytime they try to use performance data that includes information about anything other than the specific user experience they’re trying to fix. Like when they try to fix the performance of some click on a web form by looking at CPU utilization data on their application server or their database server.

Another thing that makes it really hard is the design of the application. More tiers means more complexity when it comes time to diagnose performance problems. And some User Interface designs are just guaranteed to create performance problems. My presentation here called “Messed-Up Apps” is a showcase of a few of those kinds of designs. The message there is that performance is something that has to be designed into an application from the start, like any other feature. Performance is not something you can paint on at the end.


Justin: What can developers do to maximize the performance of the applications they write?

Cary: The most important thing is to remember a couple of key ideas. First, Barry Boehm showed that the cost of repairing defects increases hyperbolically, the later you find them in your development and deployment life cycle. That’s true for performance defects just like it is for functional defects. Second, what Donald Knuth wrote 40 years ago is still true today: when developers try to guess where their code is slow, they do an awful job. Even great developers, when they profile the response time of their code, they’re often surprised at where that code is spending their (or their users’) time. So, profiling early in the software development life cycle is vital.

Next, it’s important to test. Not just functional requirements, but test performance requirements, too. Finally, it’s important to realize that there’s no way that your testing can catch every performance problem that can go wrong, so it’s important to make your application code easy to diagnose and repair in production. You do that with good instrumentation so your production system managers can profile in production when they need to, just like the developers do on the development and test systems.


Justin: How do you—a developer—profile your code?

Cary: Every development language has profiling tools that go with it. They’re tools that you can point at your application when it runs to show exactly where every smidgen of response time is being consumed within that code. The first profiler I was ever aware of is the -pg flag on C compilers. You gcc -pg to compile your code, and then after you run your code, you can use gprof to profile where your time went. Java has profilers, PHP has them, Perl, C#, C++, all of them.

Even the Oracle Database has a profiling capability, but they don’t call it “profiling” (that name means something else in the Oracle documentation). The extended SQL trace data that Oracle emits when you do the right DBMS_MONITOR.SESSION_TRACE_ENABLE call is a written record of where every bit of your response time went. That’s “profiling,” in the computer science sense of the word. Those files have been the basis of my career as a performance analyst and software tools author for the past 20 years or so.


Justin: Tell us a little bit about your company, Method R. You founded it a couple of years ago?

Cary: Yes, I started a new company called Method R Corporation in April 2008. We’ve had a great time writing tools for people and performing services (teaching and consulting) to help people solve their performance problems. Our core business is building tools that help people do for themselves what we know how to do with performance. The trace data that Oracle emits is very complicated, and we have software tools that make it easy to get what you need from those trace files.

We also have an extension for SQL Developer that makes it easy to get the trace data itself, while you’re developing a new SQL- or PL/SQL-based application. We’re also working on a number of very large development projects for customers in which we’re writing complex application code that has to scale to outrageous workloads. We’re always looking for ways we can help people.


Justin: Well, that’s all the time we have today. I really enjoyed talking with you, etc., etc.

Cary: Oh, it was my pleasure. Thank you for having me, and good luck with the rest of the Show.

◆ ◆ ◆

...Which hasn’t happened yet. ☺

I hope you enjoyed, and I’ll look forward to seeing you at Oracle OpenWorld 2010.

I’ll be presenting “Messed-Up Apps: a study of performance antipatterns” at the ODTUG User Group Forum on Sunday at 3:00pm, and “Thinking Clearly about Performance” at the main event on Tuesday at 12:30pm. See you there!

Mister Trace

For the past several weeks, my team at Method R have been working hard on a new software tool that we released today. It is an extension for Oracle SQL Developer called Method R Trace. We call it MR Trace for short.

MR Trace is for SQL and PL/SQL developers who care about performance. Every time you execute code from a SQL Developer worksheet, MR Trace automatically copies a carefully scoped trace file to your SQL Developer workstation. There, you can open it with any application you want, just by clicking. You can tag it for easy lookup later. There’s a 3-minute video if you’re interested in seeing what it looks like.

I’m particularly excited about MR Trace because it’s the smallest software tool we’ve ever designed. That may sound funny to a lot of people, but it won’t sound funny to you if you’ve read Rework by Jason Fried and David Heinemeier Hansson of 37signals. MR Trace does a seemingly very small thing—it gets your trace file for you—but if you’ve ever done that job yourself, you might get a kick out of seeing it happen so automatically, so simply, and so quickly.

The thing is, the normal process of getting trace files is raw misery for many of our friends. It’s a common story: “If I trace some SQL, then to get my trace files, I have to call up my SA or DBA. I apologize for the interruption and hope he’s in a good mood. I tell him what system I need him to look at. He has to figure out which trace files are the ones I need, and then he FTPs them over to where I can get to them. I try not to bother him, but there’s no other way.”

Most places don’t have any security reasons to prohibit developers from getting their trace files, but they just don’t have the time or the interest to create procedures that developers can use to fetch only the files they’re supposed to see. The resulting bother is so labor-intensive and so demotivating that developers stop fighting and just move on without trace files to guide them.

That’s a big problem: if you can’t see how the code you write consumes response time, then how can you optimize it? How can you even know if you should try to? If you have to guess where your code spends time, then you can’t possibly think clearly about performance.

We have tried to design MR Trace to be a beautiful little application that does exactly what you need by staying out of your way. If we did it right, then you won’t be thinking about MR Trace whenever you use it; you’ll just have the trace files you need, right where and when you need them. And you’ll have them with no extra typing, no extra clicks, and—for goodness’ sake—certainly no more phone calls or trips down the hall. ...Unless it’s to to show off a performance problem you found and fixed before anyone else ever noticed it.

Key information:

	Name:		Method R Trace
	Type:		Extension for Oracle SQL Developer
	Function:		Zero-click trace file collector
	Price:		$49.95 USD
	Risk:		30-day free trial
	URL:		http://method-r.com/software/mrtrace
	Designer:		Method R Corporation

Thinking Clearly is more important than the Right Answer

Have you ever met anyone who attracted your attention because he had the right idea, but the more you got to know how he arrived at that idea, the less attracted you felt?

All our lives, we learn how important it is to be correct, to have the right answer. You gotta have the right answer to make good grades in school, to nail that interview, to be accepted by your peers and your families and your supervisors, .... But too many people think that an education is merely a sequence of milestones at which you demonstrate that you know the right answer. That view of education is unfortunate.

Here’s a little trick that will help me demonstrate. I’m sure you already know how to “cancel” factors in fractions, like I showed in my Filter Early post, to make division simpler. Like this:But, did you know that you can do this, too?
You never knew you could do that, did you?

Well, that’s because you can’t. Canceling the nines produces the right answer in this case: 95/19 is in fact 5/1. But the trick works only in a few special cases. It doesn’t, for example, work here:
Canceling digits like this is not a reliable technique for reducing fractions. (Here’s a puzzle for you. For how many two-digit number pairs will this digit-canceling trick work? What are they? How did you figure it out?)

The trick’s problem is precisely its lack of reliability. A process is reliable only if it works every time you use it. Incomplete reliability is the most insidious of vices. If you have a tool that never works, you learn quickly never to depend upon it, so it doesn’t hurt you too badly. But if you have a tool that works sometimes, then you can grow to trust it—which increases the stakes—and then it really hurts you when it fails.

Of course, you can make a partially reliable tool useful with some extra work. You can determine under what limited circumstances the tool is reliable, and under what circumstances it isn’t. Engineers do it all the time. Aluminum is structurally unreliable in certain temperature ranges, so when a part needs to operate in those ranges, they don’t build it out of aluminum. In some cases, a tool is so unreliable—like our cancel-the-digits trick—that you’re better off abandoning it entirely.

So, if your student (your child) were to compute 95/19 = 5/1 by using the unreliable cancel-the-digits method, should you mark the problem correctly solved? It’s the right answer; but in this case, the correctness of the answer is actually an unfortunate coincidence.

I say unfortunate, because any feedback that implies, “you can reduce fractions by canceling digits,” helps to create a defect in the student’s mind. It creates a bug—in the software sense—that he’ll need to fix later if he wants to function properly. That’s why showing your work is so important for students. How can someone evaluate your thinking if all you show is your final answer?

Being a good teacher requires many of the same skills as being a good software tester. It’s not just about whether the student can puke out the right answers, it’s whether the process in the student’s mind is reliable. For example, if a student is prone to believing in an unreliable trick like cancel-the-digits, then a test where all the problems submit nicely to that trick is a really bad test.

Likewise, being a good student requires many of the same skills as being a good software developer. It’s not just fitting your mind to the problems in the book; it’s exploring how the things you’re learning (both code path and data) can help you solve other problems, too. Being a good student means finding out “Why?” a lot. Why does this work? Does it always work? When does it not work?

Clear thinking is more important than the right answer. Certainly you want the right answer, but knowing how to find the right answer is far more important. It’s the difference between having a fish and knowing how to catch more.

Filter Early

Yesterday, my 12 year-old son Alex was excited to tell me that he had learned a new trick that made it easier to multiply fractions. Here’s the trick:

The neat thing for me is that this week I’m working on my slides for ODTUG Kaleidoscope 2010 (well, actually, for the Performance Symposium that’ll occur on Sunday 27 June), and I need more examples to help encourage application developers to write code that will Filter Early. This “trick” (it’s actually an application of the Multiplicative Inverse Law) is a good example of the Filter Early principle.

Filter Early is all about throwing away data that you don’t need, as soon as you can know that you don’t need it. That’s what this trick of arithmetic is all about. Without the trick, you would do more work to multiply 4/7 × 3/4 = (4 × 3)/(7 × 4) = 12/28, and then you would do even more work again to figure out that 12 and 28 both share a factor of 4, which is what you need to know before you then divide 12/4 = 3 and then 28/4 = 7 to reduce 12/28 to 3/7. It’s smarter, faster, and more fun to use the trick. Multiplying fractions without the trick is a Filter Late operation, which is just dumb and slow.

Here are some other examples of the Filter Early pattern’s funnier (unless you’re the victim of it), sinister antipattern, Filter Late. You shouldn’t do these things:

• Drop a dozen brass needles into a haystack, shuffle the haystack, and then try to retrieve the needles. (Why did I specifically choose brass? Two reasons. Can you guess?)
  
• Pack everything you own into boxes, hire a moving company to move them to a new home, and then, after moving into your new home, determine that 80% of your belongings are junk that should be thrown away.
  
• Return thousands of rows to the browser, even though the user only wants one or two.
• To add further insult to returning thousands of rows to the browser, return the rows in some useless order. Make the user click on an icon that takes time to sort those rows into an order that will allow him to figure out which one or two he actually wanted.
• Execute a database join operation in a middle-tier application instead of the database. I’m talking about the Java application that fetches 100,000 rows from table A and 350,000 rows from table B, and then joins the two result sets in a for loop, in an operation that makes 100,000 comparisons to figure out that the result set of the join contains two rows, which the database could have told you much more efficiently.
• Slog row-by-row through a multimillion-row table looking for the four rows you need, instead of using an index scan to compute the addresses of those four rows and then access them directly.
  

Converting a Filter Late application into a Filter Early application can make performance unbelievably better.

One of my favorite features of the Oracle Exadata machine is that it applies the Filter Early principle where a lot of people would have never thought to try it. It filters query results in the storage server instead of the database server. Before Exadata, the Oracle Database passed disk blocks (which contain some rows you do need, but also some rows you don’t) from the storage server to the database. Exadata passes only the rows you need back to the database server (Chris Antognini explains).

How many Filter Early and Filter Late examples do you know?