Wednesday, February 13, 2008

How the OFA Began, Part 1

By all outward appearances, my Oracle career began in Miami when I presented paper 513 at 1991 International Oracle User Week. It was a paper called “Configuring a growing Oracle V6 database for optimal performance.” That paper was an early milestone en route to what would eventually be known as the OFA Standard:
The OFA Standard is a set of installation guidelines that will give you faster, more reliable, Oracle databases that require less work to maintain.
This is the story of how OFA—the Optimal Flexible Architecture—began.

I joined Oracle in October 1989. Three months prior to that, I had never heard of “Oracle.” I owe my friend Gary Goodman, whom I met in 1988 at SMU, for introducing me to the existence of this database company called Oracle. (A database company?! After the first hierarchical database course I had taken in 1984—on HP MPE running Image—I had religiously avoided just about everything having to do with databases that I could. My career path had led me to language design and compiler development. The only relational database experience I had was the result of reading a book that showed how to implement a relational database on Unix filesystems with awk and grep.)

Turned out that, because of my Unix development experience, I was reasonably well suited for a few of the more technical tasks that most of my Oracle Consulting Services colleagues weren’t particularly well suited for: installing, tuning, and upgrading Oracle databases. By the end of 1990, I had installed several dozen databases around the country, and I had been called in to fix problems in several existing installations. The problems were pretty comical in hindsight, including things like:
  • Unix system administrators kept deleting Oracle database files. Especially the temporary tablespace data files, which had been stored in /tmp.

  • Systems were sloooooooow, because in spite of being installed upon an 8-disk system, all the Oracle database files were stored in the $ORACLE_HOME/dbs directory on a single Unix filesystem, on a single disk.
One of my first steps when I’d get to a new site was to run some SQL scripts upon the Oracle database dictionary just to figure out where all the database files were. They were always in different places. Most people tried to put all their files in the single dbs directory, because that’s where the Oracle installer put them. But what happens when you need more space, and the filesystem under that dbs directory had filled to capacity? Right: you put some Oracle database files someplace else. So they would end up in the legaldocs filesystem or anywhere else they’d fit.

Most of the places I visited had files all over the place. In addition to vertical (space consumption) growth within a database, there was also horizontal growth (second and third databases) to contend with. One common type of problem was accidentally deleting a data file from database #1 when a DBA had been trying to reclaim space from a no-longer-needed database #2. Another common problem had to do with writing backup scripts. When people added database files, they’d tend to forget to update the backup script (which now needed to look for a new file in legaldocs), and so when it came time to recover their database, they’d be met with a nasty surprise.

So I created myself a standard. At least at the customer sites I would visit a second time, I was able to find my files without having to use SQL. The first formal documentation of my little standard was for a system in the Dallas office, where I was based. I had been asked to install Oracle on a demo system in the office—I think the machine was called DALHP. This was a sales demo machine, so I knew that there would be sales consultants installing different versions of Oracle on the box, creating new databases and so on. I was scheduled to leave for a 1-week vacation shortly after my installation, and I didn’t want anyone messing up my nice, clean structure while I was gone.

So I wrote down some instructions. I wrote down where to put new data files in case the database filled up with demonstration data. I wrote down what directories to create and where to put all the new files if the guys needed to create new databases. That document was the embryo of what would later become the OFA Standard.

The new document was of course very useful to me at clients sites as well. You see, my consulting engagement pattern had evolved to...
  1. Arrive at client site.
  2. Listen attentively to a description of all the problems.
  3. Explain how my little standard would solve those problems.
  4. Repeat step 3 as many times as necessary for the system administrator to actually change the way the whole filesystem is laid out.
I don’t know about you, but when I practice some task over and over again, I do tend to get better at it. For a while. Then I start to get bored, and when I get bored, I get sloppy. It’s why I became a programmer in the first place: so I can do something a few times, get pretty good at it, explain to a machine how to do it, and then perform the task perfectly, over and over again for the rest of my life. It’s a good formula.

With this OFA argument of mine, I had the fullest conviction that it was good and that everyone should do it. But I wasn’t always able to convince people of that. Especially the system administrators who were going to have to do a lot of work as a result of that convincing. I was losing at step 4 sometimes when I shouldn’t. And even when I won, it was taking me longer than I felt like it should have to get my points across.

The solution was easy, though. I shipped my “standards” document to my client in advance of my arrival. By the time I arrived on site, the people I’d be working with would have read the latest and most polished version of my argument. From there, I remember only two results: the “Yes, let’s go” result, and the “I have a few questions first” result. Either one of those worked just fine. Sometimes, the people I was visiting would have reconfigured all their filesystems over the weekend before I showed up. That helped us work faster and save the client money.

I don’t remember exactly where the name OFA came into the picture. I did have a difficult time coming up with a name, but I did have an awareness of what I sometimes call Marketing Rule #1:
Marketing Rule #1: If it doesn’t have a name, it doesn’t exist.
I had to name it something so that people could talk about it. I was never really sure whether the word “Architecture” was right. It wasn’t really an architecture, it was a configuration standard. Somehow, though, as I think of all the times I’ve heard OFA pronounced as the word “ofah,” I don’t think the name “OFCS” would have turned out as well as OFA did.

In my next post, I’ll tell you a little bit about my presentation in Miami, including why it almost never happened.


Marco Gralike said...

Great topic! Doug did a great presentation on OFA during UKOUG. When I brought the topic OFA up in my company I was really amazed that it was treated as almost an un-interesting topic. Until Doug's presentation I wasn't aware of your involvement. I almost wanted to say, it saved my life several times (OK, with less enthusiasm...), it made my life easier. Aware that it is that lighthearted nowadays, I started "preaching" about it...

In all. Thanks.

Niall said...

Great entry Cary and thanks for the background. I also saw Doug's presentation which both delighted me (great presentation on a great topic) and depressed me (we'll be seeing presentations on why you should use timing information to resolve time critical issues for at least another ten years.

Dan Norris said...

I was surprised to learn that Oracle actually "created" a product name that started with "O", but didn't have the word Oracle in it. For the first year or two of my DBA career (oh so many years ago), OFA was defined for me as the "Oracle Filesystem Architecture" which seemed reasonable. When I found the original Oracle Whitepaper where it was defined, I thought the whitepaper was a hoax due to the name not being what I had learned.

Great info--I'll be watching for part 2!

Unknown said...

Gosh, I could say a lot in reply to this post!

I'll limit myself to saying I'm glad you wrote it and that I'll be sending it to all of my colleagues in the morning ;-)

Andy Rivenes said...

Hi Cary, I assume you have your original papers, but if not I have a revised edition from February 7, 1992. I assume you gave this to Neil when you visited way back then. It's titled "An Optimal Flexible Architecture for a Growing ORACLE V6 Database" and it lists you as part of the Oracle National Product Specialist Team. I can remember seeing you present this in Florida in 1992 or 1993 at an IOUW conference and thinking "this is what we do". Thanks for the history, it is interesting to think back about how some this started.