Richard Feynman defined science as "the belief in the ignorance of experts." Science begins by questioning established ideas. ...Even those ideas promoted by so-called experts.
The value of science that's obvious to everybody is the chance you might discover some valuable truth that nobody else has discovered before. That's the glamorous idea that might motivate you to begin the hard work that science sometimes requires. Science is also valuable to you when you learn that an established idea, no matter how much you may not like it, really is true after all. That second value of science is not as glamorous, but it's just as important. My little prayer with respect to that possibility is, "If an idea I believe is wrong, please let me find out before anybody else does."
Everyone can do science. Not just "scientists"; all of us. But you need to do science "right," or it's not science. Do it right, and you accumulate a little bit of truth. Do it wrong, and and you've wasted your time, or worse, you've doomed yourself to waste more of your time in the future, too.
The difference between "right" and "wrong" in science is not some snooty, bureaucratic concept. You don't need a license or a blessing to do science right. You just need to ensure that the cause-effect relationships you choose to believe are actually correct. One of the rules for doing science right is that you measure instead of just asserting your opinion.
Different people have different thresholds of skepticism. Some people believe new ideas, whether they're true or false, with very little persuasion. The people who are persuaded easily to believe false things cannot contribute much useful new knowledge to their communities (irrespective of how much they might publish).
People at the opposite end of the spectrum have very strict standards for what they accept as truth. They're careful about what rows they insert and commit into their minds. There aren't as many of those people, but they're more interesting, with respect to science, because they're the people who can contribute new knowledge to the community.
A lot of what we do in the Oracle world comes down to a person demonstrating, say, in sqlplus, that some certain cause produces some certain effect in some certain version of the database on some certain operating system, and so on. Then the next step is when you have to look at that result and decide for yourself (or perhaps with someone's help), how relevant that result is for you.
Innumerable Oracle debates funnel into the argument that, "Yes, I can see plainly that this situation can happen, but that situation will probably never happen for me." This is what happens, for example, when the doubter believes the prover is using an example that is too contrived to be realistic, or when the doubter's context is different from the prover's. ...Like when the doubter is running a data warehouse, but the prover talks only about transaction processing.
That's where another level of value begins, and it's another place where science can help. It's where the issue becomes proving how relevant a given proposal really is for a given circumstance. One of the nice things about software (and Oracle Database software in particular) is that it's usually easy to write code that will tell you how often something happens, or how long it takes when it does. With software, I don't have to guess. I can measure and know. Software is unusual in the world in that it can be used to measure itself.
At this week's ODTUG conference in Monterey, California, a debate has formed that I'm interested in following. The established idea of the experts in this debate is a passionate belief that you should declare, enable, and index foreign key (FK) relationships. The counter-argument is that you should not.
I've heard probably most of the arguments on the pro side of the debate. (a) If you don't declare/enable/index your FKs, then you have to ensure the correct relationships in your application code, which is error-prone in both obvious and highly subtle ways. (b) Not implementing full FK integrity in the database is developmentally inefficient, because it violates the important principle that you should never duplicate code in an application. (c) Furthermore, the absence of declared/enabled/indexed FK integrity is operationally inefficient at run time, because it blocks the Oracle query optimizer from using code paths that can be hundreds of times faster and more scalable than when it can't rely upon database-enforced FK integrity.
I haven't heard a single compelling argument for the other side of the debate. But, you see, there could be a compelling argument that I haven't heard.
This is what makes this new debate interesting for me. One side or the other is on the brink of learning something important, if the debate is conducted properly.
The first thing the two sides will need to do is agree on whether both sides really are in disagreement and, if they are, what exactly the disagreement is. In lots of debates I've seen, we've figured out after defining the terms and deciding the context of the argument (for example, what kind of application we're talking about), that there's really no debate of principle at all. When that happens, it's debate closed. Each side agrees with the other, and maybe the two sides learn a little more about life on the other side of the fence.
If after the suitable definition-of-terms process, the two sides really do still have a debate left, then there will be some kind of attestation of facts as both sides see them. One side, for example, will show sqlplus session output to demonstrate the subtle and not-so-subtle ways in which not doing the declare/enable/index thing causes corrupted data and horrific performance penalties. The other side of the argument will then show some contrary evidence that counts in favor of not doing the declare/enable/index thing.
If the sides can't agree on the truth of the "facts" presented, then the debate will collapse, and at least one side (possibly both) will have learned nothing. If each side succeeds in impressing the other side that the "facts" thus presented are actually true, then the debate will move to the discussions of the relevance of the facts just demonstrated. This is where one guy might say something like, "I know that queries with FKs in the database are faster, but I can't afford the performance penalty at data load time." To which the other guy might say, "Yes, loads are a little slower with referential integrity checking, but that's time spent executing necessary code path to ensure that only correct data can get into your database. And besides, it's not a good trade-off to endure a thousand slow queries a day so that one load can go faster." Rebuttal, counter-rebuttal, and so on. You get the picture.
I am biased in my estimation of how this will turn out. But I sincerely respect when someone thoughtfully and sincerely challenges an important idea in my professional domain, no matter how well entrenched that idea may be. In fact, the more entrenched, the better. The debate will remain interesting as long as the counter-argument is thoughtful and sincere. As soon as the evidence for the challenging new idea reveals itself to be nonsense, or if the counterargument context seems irrelevant to me (and the people I'm trying to represent), then I'll lose interest.
The best debate ends in a handshake (real or virtual) in which the people representing both sides learn something they didn't know before, one side perhaps more than the other. The best debaters value learning more than being right. The best debaters respect each other more after the debate because each has helped the other (and the community around them) to advance.
The worst debaters confuse the principles of factual correctness and personal correctness. When the debate shifts from factual to personal, it may become interesting to some people because of the enhanced drama, but the actual usefulness of the debate evaporates. ([Grin] I guess if the debate were solid to begin with, then the usefulness actually sublimates.)
No matter which idea wins this FK debate (right, I said idea, not people), I expect to be happy for the debate to have occurred. That's because I expect the debate to end with a resolution, and either I'm going to learn something completely new, or I'm going to fortify an existing belief. Something new is obviously exciting, and fortification will make me a more effective teacher of my existing belief. Either result benefits me and, I believe, the community.
With science, you get suspense, drama, plot twists, surprises, fortifications, .... Science is fun. I wish more people knew.
Tuesday, June 23, 2009
Thursday, June 18, 2009
Profiling with my Boy
We have an article online called "Can you explain Method R so even my boss could understand it?" Today I'm going to raise the stakes, because yesterday I think I explained Method R so that an eleven year-old could understand it.
Yesterday I took my 11 year-old son Alex to lunch. I talked him into eating at one of my favorite restaurants, called Mercado Juarez, over in Irving, so it was a half hour in the car together, just getting over there. It was a big day for the two of us because we were very excited about the new June 17 iPhone OS 3.0 release. I told him about some of the things I've learned about it on the Internet over the past couple of weeks. One subject in particular that we were both interested in was performance. He likes not having to wait for click results just as much as I do.
According to Apple, the new iPhone OS 3.0 software has some important code paths in it that are 3× faster. Then, upgrading to the new iPhone 3G S hardware is supposed to yield yet another 3× performance improvement for some code paths. It's what Philip Schiller talks about at 1:42:00 in the WWDC 2009 keynote video. Very exciting.
Alex of course, like many of us, wants to interpret "3× faster" as "everything I do is going to be 3× faster." As in everything that took 10 seconds yesterday will take 3 seconds tomorrow. It's a nice dream. But it's not what seeing a benchmark run 3× faster means. So we talked about it.
I asked him to imagine that it takes me an hour to do my grocery shopping when I walk to the store. Should I buy a car? He said yes, probably, because a car is a lot faster than walking. So I asked him, what if the total time I spent walking to and from the grocery store was only one minute? Then, he said, having a car wouldn't make that much of a difference. He said you might want a car for other reasons, but he wouldn't recommend it just to make grocery shopping faster.
Good.
I said, what if grocery shopping were taking me five hours, and four of it was time spent walking? "Then you probably should get the car," he told me. "Or a bicycle."
Commit.
On the back of his menu (photo above: click to zoom), I drew him a sequence diagram (A) showing how he, running Safari on an iPhone 3G might look to a performance analyst. I showed him how to read the sequence diagram (time runs top-down, control passes from one tier to another), and I showed him two extreme ways that his sequence diagram might turn out for a given experience. Maybe the majority of the time would be spent on the 3G network tier (B), or maybe the majority of the time would be spent on the Safari software tier (C). We talked about how if B were what was happening, then a 3× faster Safari tier wouldn't really make any difference. Apple wouldn't be lying if they said their software was 3× faster, but he really wouldn't notice a performance improvement. But if C were what was happening, then a 3× faster Safari tier would be a smoking hot upgrade that we'd be thrilled with.
Sequence diagrams, check. Commit.
Now, to profiles. So I drew up a simple profile for him, with 101 seconds of response time consumed by 100 seconds of software and 1 second of 3G (D):
Oops. Rollback.
He made the same mistake that just about every forty year-old I've ever met makes. He figured if one component of response time were 2× faster, then the total response time must be 2× faster, too. Nope. In this case, the wrong answer was close to the right answer, but only because of the particular numbers I had chosen.
So, to illustrate, I drew up another profile (E):
Commit.
Next step. "What if I told you it takes me an hour to get into my email at home? Do I need to upgrade my network connection?" A couple of minutes of conversation later, he figured out that he couldn't answer that question until he got some more information from me. Specifically, he had to ask me how much of that hour is presently being spent by the network. So we did this over and over a few times. I'd say things like, "It takes me an hour to run my report. Should I spend $4,800 tuning my SQL?" Or, "Clicking this button takes 20 seconds. Should I upgrade my storage area network?"
And, with just a little bit of practice, he learned that he had to say, "How much of the however-long-you-said is being spent by the whatever-it-was-you-said?" I was happy with how he answered, because it illustrated that he had caught onto the pattern. He realized that the specific blah-blah-blah proposed remedies I was asking him about didn't really matter. He had to ask the same question regardless. (He was answering me with a sentence using bind variables.)
Commit.
Alex hears me talk about our Method R Profiler software tool a lot, and he knows conceptually that it helps people make their systems faster, but he's never known in any real detail very much about what it does. So I told him that the profile tables are what our Profiler makes for people. To demonstrate how it does that, I drew him up a list of calls (F), which I told him was a list of visits between a disk and a CPU. ...Just a list that says the same thing that a sequence diagram (annotated with timings) would say:
The finish-up conversation in the car ride back was about how to use everything we had talked about when you fix people's performance problems. I told him the most vital thing about helping someone solve a performance problem is to make sure that the operation (the business task) that you're analyzing right now is actually the most important business task to fix first. If you're looking at anything other than the most important task first, then you're asking for trouble.
I asked him to imagine that there are five important tasks that are too slow. Maybe every one of those things has its response time dominated by a different component than all the others. Maybe they're all the same. But if they're all different, then no single remedy you can perform is going to fix all five. A given remedy will have a different performance impact upon each of the five tasks, depending on how much of the fixed thing that task was using to begin with.
So the important thing is to know which of the five profiles it is that you ought to be paying attention to first. Maybe one remedy will fix all five tasks, maybe not. You just can't know until you look at the profiles. (Or until you try your proposed remedy. But trial-and-error is an awfully expensive way to find out.)
Commit.
It was a really good lunch. I'll look forward to taking my 9-year-old (Alex's little brother) out the week after next when I get back from ODTUG.
Yesterday I took my 11 year-old son Alex to lunch. I talked him into eating at one of my favorite restaurants, called Mercado Juarez, over in Irving, so it was a half hour in the car together, just getting over there. It was a big day for the two of us because we were very excited about the new June 17 iPhone OS 3.0 release. I told him about some of the things I've learned about it on the Internet over the past couple of weeks. One subject in particular that we were both interested in was performance. He likes not having to wait for click results just as much as I do.
According to Apple, the new iPhone OS 3.0 software has some important code paths in it that are 3× faster. Then, upgrading to the new iPhone 3G S hardware is supposed to yield yet another 3× performance improvement for some code paths. It's what Philip Schiller talks about at 1:42:00 in the WWDC 2009 keynote video. Very exciting.
Alex of course, like many of us, wants to interpret "3× faster" as "everything I do is going to be 3× faster." As in everything that took 10 seconds yesterday will take 3 seconds tomorrow. It's a nice dream. But it's not what seeing a benchmark run 3× faster means. So we talked about it.
I asked him to imagine that it takes me an hour to do my grocery shopping when I walk to the store. Should I buy a car? He said yes, probably, because a car is a lot faster than walking. So I asked him, what if the total time I spent walking to and from the grocery store was only one minute? Then, he said, having a car wouldn't make that much of a difference. He said you might want a car for other reasons, but he wouldn't recommend it just to make grocery shopping faster.
Good.
I said, what if grocery shopping were taking me five hours, and four of it was time spent walking? "Then you probably should get the car," he told me. "Or a bicycle."
Commit.
On the back of his menu (photo above: click to zoom), I drew him a sequence diagram (A) showing how he, running Safari on an iPhone 3G might look to a performance analyst. I showed him how to read the sequence diagram (time runs top-down, control passes from one tier to another), and I showed him two extreme ways that his sequence diagram might turn out for a given experience. Maybe the majority of the time would be spent on the 3G network tier (B), or maybe the majority of the time would be spent on the Safari software tier (C). We talked about how if B were what was happening, then a 3× faster Safari tier wouldn't really make any difference. Apple wouldn't be lying if they said their software was 3× faster, but he really wouldn't notice a performance improvement. But if C were what was happening, then a 3× faster Safari tier would be a smoking hot upgrade that we'd be thrilled with.Sequence diagrams, check. Commit.
Now, to profiles. So I drew up a simple profile for him, with 101 seconds of response time consumed by 100 seconds of software and 1 second of 3G (D):
Software 100 3G 1 ------------- Total 101I asked him, if we made the software 2× faster, what would happen to the total response time? He wrote down "50" in a new column to the right of the "100." Yep. Then I asked him what would happen to total response time. He said to wait a minute, he needed to use the calculator on his iPod Touch. Huh? A few keystrokes later, he came up with a response time of 50.5.
Oops. Rollback.
He made the same mistake that just about every forty year-old I've ever met makes. He figured if one component of response time were 2× faster, then the total response time must be 2× faster, too. Nope. In this case, the wrong answer was close to the right answer, but only because of the particular numbers I had chosen.
So, to illustrate, I drew up another profile (E):
Software 4 3G 10 ------------- Total 14Now, if we were to make the software 2× faster, what happens to the total? We worked through it together:
Software 4 2 3G 10 10 ------------------ Total 14 12Click. So then we spent the next several minutes doing little quizzes. If this is your profile, and we make this component X times faster, then what's the new response time going to be? Over and over, we did several more of these, some on paper (F), and others just talking.
Commit.
Next step. "What if I told you it takes me an hour to get into my email at home? Do I need to upgrade my network connection?" A couple of minutes of conversation later, he figured out that he couldn't answer that question until he got some more information from me. Specifically, he had to ask me how much of that hour is presently being spent by the network. So we did this over and over a few times. I'd say things like, "It takes me an hour to run my report. Should I spend $4,800 tuning my SQL?" Or, "Clicking this button takes 20 seconds. Should I upgrade my storage area network?"
And, with just a little bit of practice, he learned that he had to say, "How much of the however-long-you-said is being spent by the whatever-it-was-you-said?" I was happy with how he answered, because it illustrated that he had caught onto the pattern. He realized that the specific blah-blah-blah proposed remedies I was asking him about didn't really matter. He had to ask the same question regardless. (He was answering me with a sentence using bind variables.)
Commit.
Alex hears me talk about our Method R Profiler software tool a lot, and he knows conceptually that it helps people make their systems faster, but he's never known in any real detail very much about what it does. So I told him that the profile tables are what our Profiler makes for people. To demonstrate how it does that, I drew him up a list of calls (F), which I told him was a list of visits between a disk and a CPU. ...Just a list that says the same thing that a sequence diagram (annotated with timings) would say:
D 2 C 1 D 6 D 4 D 8 C 3I told him to make a profile for these calls, and he did (H):
Disk 20 CPU 4 --------- Total 24Excellent. So I explained that instead of adding up lists in our head all day, we wrote the Profiler to aggregate the call-by-call durations (from an Oracle extended SQL trace file) for you into a profile table that lets you answer the performance questions we had been practicing over lunch. ...Even if there are millions of lines to add up.
The finish-up conversation in the car ride back was about how to use everything we had talked about when you fix people's performance problems. I told him the most vital thing about helping someone solve a performance problem is to make sure that the operation (the business task) that you're analyzing right now is actually the most important business task to fix first. If you're looking at anything other than the most important task first, then you're asking for trouble.
I asked him to imagine that there are five important tasks that are too slow. Maybe every one of those things has its response time dominated by a different component than all the others. Maybe they're all the same. But if they're all different, then no single remedy you can perform is going to fix all five. A given remedy will have a different performance impact upon each of the five tasks, depending on how much of the fixed thing that task was using to begin with.
So the important thing is to know which of the five profiles it is that you ought to be paying attention to first. Maybe one remedy will fix all five tasks, maybe not. You just can't know until you look at the profiles. (Or until you try your proposed remedy. But trial-and-error is an awfully expensive way to find out.)
Commit.
It was a really good lunch. I'll look forward to taking my 9-year-old (Alex's little brother) out the week after next when I get back from ODTUG.
Subscribe to:
Posts (Atom)