To Jeanne
About 35 years ago, you painted this quote for me on a watercolor:
“Come grow old with me, the best is yet to be.” There are no regrets. It is a wonderful journey with you by my side.
In the practice of performance tuning, change is inevitable. New techniques are created; old ones are updated. The volume of metrics, especially those contained in Oracle’s Automatic Workload Repository (AWR), is multiplying almost daily. Despite the autonomous database, monitoring tools, and other similar tools, the level of expertise required to analyze database performance problems has increased. To survive and thrive in this field, innovation is everything. Nothing remains the same for long. We either innovate and adapt to change or change happens to us.
The purpose of this book is not simply to help you adapt to change but to equip you with a method of analysis that will enable you to thrive and even lead in the application of creative big data science methods for Oracle performance analysis and tuning.
Before I begin to tell you about a process that I think you will find very potent, albeit very different from what you might be currently using, I want to tell you about my background in performance tuning. I’d like you to know that I’ve seen a lot of change and I’ve been where many of you are.
I began using Oracle as a software developer in 1985 (version 4.1.4). One important aspect of my work was to ensure high performance of enterprise class applications, so tuning was a necessary part of the job. At that time, there was little to no Oracle internal instrumentation exposed to the average user, so tuning involved a lot of guesswork. I relied on a careful study of the application architecture and used custom instrumentation for coarse-grained metrics.
When Oracle began to introduce internal metrics to the users, they did this through a series of views called “dynamic performance views” but commonly referred to as “v$ views” since they are prefixed with “v$.” The v$ views include a host of metrics, many of which will be familiar to tuning experts, including v$session, v$sqlarea, v$sysstat, and v$OSstat.
Like many of my tuning peers, I quickly embraced these metrics as a primary source of metrics for finding problems in databases that needed tuning. The v$ views provide values for metrics for near real time. They are useful for solving problems that are happening at the present time, but since the metrics are not persisted over time, they are not very helpful when solving problems that happened in the past. This, of course, constrained the kinds of problems that I could solve quickly or the reproducibility of the problems that I was trying to solve.
With Oracle’s introduction of the bStat/eStat utilities, I could see what happened during an interval of time, but I still had to be present when the problem was occurring (see sidebar note on bStat and eStat utilities).
Oracle made an improvement to the v$ views and the bStat/eStat utilities mechanisms by providing StatsPack, which captured many of those metrics over time and then persisted historic values for the metrics. With this added capability, I could look at past performance even when the problem was no longer apparent [occurring on the database]. I took a similar approach to querying StatsPack data as I used with the v$ view counterparts.
Note on the bStat/eStat utilities: This is a much-antiquated approach provided by Oracle prior to version 7 where you manually start and end a performance evaluation period—“bStat” is short for begin snapshot and is run prior to evaluating some SQL for performance; “eStat” is short for end snapshot and is run at the end of the performance evaluation period. Together these tools provide performance metrics for the manual snapshot period.
I began to use the enormous warehouse of AWR data somewhat differently than most developers/DBAs I knew. Instead of looking at AWR reports (or ASH or ADDM reports) as an initial analysis step in the tuning process, I began my analysis by querying specific individual DBA_HIST views for detailed information that I believed would lead me to discover performance problems more quickly. I chose the metrics I wanted to look at based on the type of metric and my past experience of how useful that information was for problem solving. I became quite adept at selecting where to look in the AWR for solving specific problems. I refer to this approach of selecting specific metrics as a “small model approach” because it uses only a subset of all the data available. I found this small model approach self-limiting and not always effective or efficient. It required a lot of knowledge of where to look for solving specific problems and a good amount of trial and error. I was able to gain proficiency over time, but it was a difficult and sometimes tedious learning process.
All of the tuning resources I have evaluated to date use some variation of this small model approach. Again, I use the term “small model” because they all rely on a subset of the available metrics—a predefined set of metrics is put forward as being highly predictive of finding performance problems. These methods and metrics are put forth by people who have a wealth of experience, and I grant that they are very useful in identifying problems in the areas for which the metrics were designed, but the downfall of the small model approach is that the decreased visibility into the tens of thousands of available metrics results in a less complete, less accurate understanding of the performance problem.
The method of analysis presented in this book represents an innovative, step change paradigm shift away from the traditional methods of analyzing Oracle database performance data. The method I present leverages big data and advanced analytical techniques (using plain SQL code) so that instead of relying on a few hand-picked/favorite metrics, the performance tuner is able to draw on “all” the available data without being overwhelmed by sheer volume and lack of a clear starting point. By formatting the data in a particular manner (discussed in Chapter 3 ), the tuning professional can apply big data methods to a tremendous number of metrics to draw impactful, focused performance improvement conclusions.
The Dynamic Oracle Performance Analytics (DOPA) process I created substantially increases the number of metrics that can be included in the analysis while simultaneously reducing the complexity of the analysis by identifying the metrics most relevant for solving a particular performance tuning problem. It is a predictive modeling technique. It enables the performance tuning to be efficient, targeted, and specific. The tuner will be able to quickly, accurately, and consistently identify a “root cause” with confidence.
When I first started this work about a year and a half ago, I had no intention of writing a book on the subject—I simply wanted a better way to analyze the thousands of Oracle performance metrics available. I’d like to thank my wife, Jeanne Cornejo, for encouraging me to write the book and for the many hours she spent working with me. Dynamic Oracle Performance Analytics is a big and somewhat complex topic; Jeanne, a physical therapist by profession, worked tirelessly throughout the writing to help me break the material down so that even a nontechnical person can understand it. If you’ve ever seen the Johnny English movie where he breaks out into what appears to be a mini-seizure whenever he heard the word “Mozambique,” it kind of got that way for her with “DOPA.”
I’d also like to thank my oldest son, David Cornejo (PhD Operations Research), who provided me with inspiration and ideas as to how to get started with a lightweight version for implementing some rather complex big data analytical techniques (which are easily implemented in the Oracle SQL language).
Jonathan Gennick from Apress deserves credit for recognizing the potential value of this material to a wider audience and for reaching out to me and helping me get started with the proposal. A special thank you as well to the technical reviewer, Bill Padfield, who stuck with the review even though some key points needed more fleshing out.
has been an Oracle enthusiast since 1985 (versions 4–12c). He has many and varied experiences on large enterprise class Oracle applications, not only in performance troubleshooting and tuning but also in systems architecture, information modeling, and software development/project management. For the past ten years, Roger’s main focus has been database performance analysis and tuning, with much of his time spent exploring the complexities and usefulness of AWR* tuning data. In his work, Roger is relied upon to produce Oracle database tuning results across 18c/12c/11g/10g (and occasionally 9i) databases. Roger is a thought-leader in his field and has been recognized for his expertise in tuning through such opportunities as presenting at the past eight East Coast Oracle Conferences, as well as at COLLABORATE14 and COLLABORATE18, RMOUG16, and Hotsos 2017–2018.
is an Oracle Certified Professional, working for a large telecommunications company in Denver, Colorado, as a senior database administrator. Bill helps administer and manage a large data warehouse environment consisting of more than 100 databases. Bill has been an Oracle database administrator for more than 16 years and has been in the IT industry since 1985. Bill also teaches graduate database courses at Regis University and currently resides in Aurora, Colorado, with his wife, Oyuna, and son, Evan.