Tuesday, February 26, 2008

How the OFA Began, Part 2

I’m looking this moment at a pair of nicely bound light beige books called 1991 International Oracle User Week Proceedings, Volumes 1 and 2. Toon Koppelaars just mentioned to me a couple of days ago that his international speaking debut was at the same event, and sure enough, there’s his paper #808, right in Volume 2 with mine: “User Experiences with Oracle Server for OS/2.”

It’s fun remembering the Good Old Days before laptop PCs. But I’ll tell you what, it was a lot harder work back then trying to communicate. When I thumb to my paper #513 (29+1), the format reminds me that I used to use typesetting software called TeX (pronounced “tech,” with no x sound). TeX was written by Donald Knuth, who took a 15-year hiatus from his “proper” research in computer science to develop—at long last—a decent phototypesetter for people like me to use.

TeX was a superb and wonderful thing, and it still is. It has fallen aside in some environments including, unfortunately, my own, in favor of the so-called WYSIWYG editors, like Microsoft Word. I still can’t figure out how an application that makes you type Ctrl-Fn-Alt-; to get an em-dash gets to be called “WYSIWYG,” but, well, I guess they expect that if you know and care what an em-dash is, then you’re probably going to be able to figure out how to type one. Learning TeX taught me a lot about typesetting, which to this day I regard as a Very Good Thing. I took a look at some of my old TeX source just a few days ago, and it still looks good to me. (Em dashes in TeX get entered as ‘---’ or ‘\emdash’, by the way.)

I made a lot of course material with TeX in the old days, too. I had different templates for making pages with bigger fonts that were suitable for printing on transparencies which could then be displayed on those old overhead projectors. I remember carrying those heavy things around on airplanes. The boxes of plastic foils, not the projectors.

But a slide show that I would present at IOUW, that’s another matter. That’s no place for black and white transparencies. For IOUW, I obtained permission from my boss at the time, Robert W. Rudzki, to produce 35mm slides in actual color. I owe a lot to Bob (or “Bwob,” as we who love him called him, because he seemed to enjoy letting us mock his Pittsburgh accent). I don’t know how many strings he had to pull to get Oracle to approve of this kid in his group to create color 35mm slides, but I presume it was a pretty big deal.

I remember the slides costing something like $400 to prepare. (See the Big Deal?) I think there were around 35 slides in total. I remember who prepared them, too: a couple named Guy and Karen Lucien. I don’t think I ever met Guy or Karen, but they did a very nice job on my slides. I remember telephone conversations with them over faxes (back when faxes were black and white), saying stuff like, ”I expected the disk on the right to be red.” Or, “There needs to be an arrow going from here to there.”

Anyway, I remember doing all the hard work to put together the material, make all these slides, and deal with the nervousness of having what I had hoped would be a 500-person audience see what I had to say. I forget when my talk was scheduled, but I had my flight booked, and I was all ready to go.

Then, about two weeks before the event, I got my summons in the mail. I was supposed to report to the Dallas County Court House for Jury Duty on exactly the morning I was supposed to present in Miami. At least the time was off by an hour. No, wait, no, if you allow for the one-hour time zone difference, I was to report exactly when I was supposed to begin my presentation in Miami. All the detailed information on the summons about being NOT EXCUSED FOR BUSINESS REASONS, ...that really helped me to work up my confidence.

I wish I still had a copy of the letter I wrote to the Judge. I would have written it in TeX. Whatever I said, it worked. I got a return letter in the mail, just in time, saying that my date was postponed, so it was a couple of weeks later that I got to go fulfill my civic duty. I sat for a couple of hours before being dismissed without even so much as a role in a voir dire. By that time, I had seen my 500 people in Miami.

Wednesday, February 13, 2008

How the OFA Began, Part 1

By all outward appearances, my Oracle career began in Miami when I presented paper 513 at 1991 International Oracle User Week. It was a paper called “Configuring a growing Oracle V6 database for optimal performance.” That paper was an early milestone en route to what would eventually be known as the OFA Standard:
The OFA Standard is a set of installation guidelines that will give you faster, more reliable, Oracle databases that require less work to maintain.
This is the story of how OFA—the Optimal Flexible Architecture—began.

I joined Oracle in October 1989. Three months prior to that, I had never heard of “Oracle.” I owe my friend Gary Goodman, whom I met in 1988 at SMU, for introducing me to the existence of this database company called Oracle. (A database company?! After the first hierarchical database course I had taken in 1984—on HP MPE running Image—I had religiously avoided just about everything having to do with databases that I could. My career path had led me to language design and compiler development. The only relational database experience I had was the result of reading a book that showed how to implement a relational database on Unix filesystems with awk and grep.)

Turned out that, because of my Unix development experience, I was reasonably well suited for a few of the more technical tasks that most of my Oracle Consulting Services colleagues weren’t particularly well suited for: installing, tuning, and upgrading Oracle databases. By the end of 1990, I had installed several dozen databases around the country, and I had been called in to fix problems in several existing installations. The problems were pretty comical in hindsight, including things like:
  • Unix system administrators kept deleting Oracle database files. Especially the temporary tablespace data files, which had been stored in /tmp.

  • Systems were sloooooooow, because in spite of being installed upon an 8-disk system, all the Oracle database files were stored in the $ORACLE_HOME/dbs directory on a single Unix filesystem, on a single disk.
One of my first steps when I’d get to a new site was to run some SQL scripts upon the Oracle database dictionary just to figure out where all the database files were. They were always in different places. Most people tried to put all their files in the single dbs directory, because that’s where the Oracle installer put them. But what happens when you need more space, and the filesystem under that dbs directory had filled to capacity? Right: you put some Oracle database files someplace else. So they would end up in the legaldocs filesystem or anywhere else they’d fit.

Most of the places I visited had files all over the place. In addition to vertical (space consumption) growth within a database, there was also horizontal growth (second and third databases) to contend with. One common type of problem was accidentally deleting a data file from database #1 when a DBA had been trying to reclaim space from a no-longer-needed database #2. Another common problem had to do with writing backup scripts. When people added database files, they’d tend to forget to update the backup script (which now needed to look for a new file in legaldocs), and so when it came time to recover their database, they’d be met with a nasty surprise.

So I created myself a standard. At least at the customer sites I would visit a second time, I was able to find my files without having to use SQL. The first formal documentation of my little standard was for a system in the Dallas office, where I was based. I had been asked to install Oracle on a demo system in the office—I think the machine was called DALHP. This was a sales demo machine, so I knew that there would be sales consultants installing different versions of Oracle on the box, creating new databases and so on. I was scheduled to leave for a 1-week vacation shortly after my installation, and I didn’t want anyone messing up my nice, clean structure while I was gone.

So I wrote down some instructions. I wrote down where to put new data files in case the database filled up with demonstration data. I wrote down what directories to create and where to put all the new files if the guys needed to create new databases. That document was the embryo of what would later become the OFA Standard.

The new document was of course very useful to me at clients sites as well. You see, my consulting engagement pattern had evolved to...
  1. Arrive at client site.
  2. Listen attentively to a description of all the problems.
  3. Explain how my little standard would solve those problems.
  4. Repeat step 3 as many times as necessary for the system administrator to actually change the way the whole filesystem is laid out.
I don’t know about you, but when I practice some task over and over again, I do tend to get better at it. For a while. Then I start to get bored, and when I get bored, I get sloppy. It’s why I became a programmer in the first place: so I can do something a few times, get pretty good at it, explain to a machine how to do it, and then perform the task perfectly, over and over again for the rest of my life. It’s a good formula.

With this OFA argument of mine, I had the fullest conviction that it was good and that everyone should do it. But I wasn’t always able to convince people of that. Especially the system administrators who were going to have to do a lot of work as a result of that convincing. I was losing at step 4 sometimes when I shouldn’t. And even when I won, it was taking me longer than I felt like it should have to get my points across.

The solution was easy, though. I shipped my “standards” document to my client in advance of my arrival. By the time I arrived on site, the people I’d be working with would have read the latest and most polished version of my argument. From there, I remember only two results: the “Yes, let’s go” result, and the “I have a few questions first” result. Either one of those worked just fine. Sometimes, the people I was visiting would have reconfigured all their filesystems over the weekend before I showed up. That helped us work faster and save the client money.

I don’t remember exactly where the name OFA came into the picture. I did have a difficult time coming up with a name, but I did have an awareness of what I sometimes call Marketing Rule #1:
Marketing Rule #1: If it doesn’t have a name, it doesn’t exist.
I had to name it something so that people could talk about it. I was never really sure whether the word “Architecture” was right. It wasn’t really an architecture, it was a configuration standard. Somehow, though, as I think of all the times I’ve heard OFA pronounced as the word “ofah,” I don’t think the name “OFCS” would have turned out as well as OFA did.

In my next post, I’ll tell you a little bit about my presentation in Miami, including why it almost never happened.

Wednesday, February 6, 2008

Hello, world

Hello, world. Welcome to my blog.

If you found this page, then you're probably already aware of our annual big event, the Hotsos Symposium, which we're hosting March 2-6 this year in Las Colinas (just outside of Dallas). Preparation for that event is where most of my time's going these days.

One friend suggested to me that people might find it interesting if I told the story of how the performance symposium idea started (thanks, Marco).

It actually all began back in January 1998, when I was still at Oracle Corporation. Through much internal sales-pitching, I was able to convince my SVP to let me host an event we called the "International Symposium on System Architecture Services." We had roughly 140 people show up, all Oracle employees who had gathered to discuss issues that DBAs and architects and their managers would find interesting. We held the event at the Hilton DFW Airport hotel in Grapevine. Our party featured watching Super Bowl XXXII on a huge screen in our auditorium. I watched John Elway lead the winning drive with my then one-year-old son parked on my lap.

In the welcome address the next morning, I told the general assembly that I had hoped this event would (1) "spark your imagination," (2) "grow your personal worth," (3) "grow your business," and (4) "build professional community." I introduced Sandy Sanderson, SVP of Oracle Consulting and my boss at the time, who would deliver the keynote for the event. And then I promptly boarded a plane for a client site I was responsible for taking care of. I missed my own event.

I hosted the second event in July 1999, this time at Caesar's Palace in Las Vegas. We had 36 presentations given in two days within four parallel tracks, featuring people like Michael Erwin, Mogens Nørgaard, Virag Saksena, Espen Braekken, Jim Littlefield, Graham Wood, Dominic Delmolino, Curt Bennett, and many others. Our keynote speaker was Dr. Daniel Menascé from George Mason University, who spoke to us about his work in computer systems capacity planning.

I did actually attend this one. I remember it being a beautiful show; absolutely inspirational. But I had a secret at the time: I was leaving Oracle. My departure date was going to be October 1999, but I couldn't tell anyone yet. My boss knew, but I had committed to a plan that prohibited my divulging my departure to anyone else until later. I can remember walking the halls of the event feeling pretty nervous about leaving a great job at Oracle--where I could host events like this--to start up a new company.

In October 1999, we created Hotsos. Thus ended the performance symposium business for a while.

In 2002, we decided our company was healthy enough to take the plunge. We'd host an event in Dallas on February 10-12, 2003. Our small business had to make a pretty breathtaking commitment to our hotel for guaranteed room nights in order to secure the conference facilities. But people came. This time, we had a 2.5-day, 2-track model featuring Anjo Kolk, Jonathan Lewis, Tom Kyte, Mogens again, Stephan Haisley, Gaja Vaidyanatha, Wolfgang Breitling, Kyle Hailey, Steven Feuerstein, and Julian Dyke. Oh, and me. I gave the keynote entitled "The Future of Performance Tuning ...Or Else."

Since our first Hotsos Symposium held February 2003, we've hosted an event each year in Dallas, where we've been able to attract over 400 professionals looking to dedicate their week to stimulating themselves with new ideas about Oracle performance. Now, we book the event each March, so we won't collide with the RMOUG Training Days show (Denver) held in February every year. We've kept our focus on Oracle performance, and with our small 2-track agenda, I think we've been able to keep our standards of quality high.

My hope is that you find us consistently able to put some of the best researchers and speakers in our field on stage for you. One accomplishment I enjoy considering is that we've been able to create a channel for introducing new talent into your span of perception. I'm very proud of our affiliation with people like Steve Adams, Jonathan Lewis, our late friend Lex de Haan, Anjo Kolk, Kyle Hailey, Tanel Põder, Jože Senegacnik, Andy Zitelli, Toon Koppelaars, Chris Antognini, Wolfgang Breitling, Doug Burns, Stephan Haisley, Jarod Jensen, Tapio Lahdehmäki, and many others, whom we've helped introduce to you. Maybe you've met (or will meet) some of these folks for the first time through us.

In Hotsos Symposium 2008, our sixth annual Hotsos show, it's my turn again to deliver the keynote address. I'll spend a half hour or so reprising some of the ideas I opened up to you in that first Symposium in 2003. Let me know if you have ideas for that keynote that you'd like me to consider.

My overall hope for the event is that again we'll help you accomplish the goals we set out to achieve in that very first event I hosted ten long years ago: to (1) spark your imagination, (2) grow your personal worth, (3) grow your business, and (4) build professional community. I'll look forward to seeing you there.