Author’s Note: As with a number of my past articles for The RAM Review, this one also is an amalgam of several similar event investigations. Any details that would otherwise allow a specific company or person to be identified have been generalized, combined, or removed. Furthermore, while the story (and, by extension, its heroine) are focused on the power-gen industry, the approach to the problem, investigation, and conclusions is applicable to a variety of operations.

THE SITUATION BACKGROUND
Mary Doe’s current title is assistant plant systems engineer. She works for a multiple-unit, electrical-power facility located somewhere in midwestern North America. Although a relatively new employee at the facility, having been on the job for six months, this is certainly not her first rodeo. She previously worked several years in smaller power facilities within the same organization conducting root cause investigations. A licensed professional engineer, she was invited to work at her current position to improve operating reliability at the flagship facility.

Ms. Doe’s first significant assignment from her boss, a plant systems engineer, was to implement a leak monitoring program for one of the three 625-megawatt units at the site. The facility-operations manager noted there had been higher-than-normal numbers of water, oil, and hydraulic-fluid leaks in various places in that power train. The assignment to implement such a program was then passed from the facility-operations manager to the systems-maintenance manager of that particular power train, then to Ms. Doe’s boss, and finally to Ms. Doe.

AN ABOVE & BEYOND APPROACH
Ms. Doe began her assignment by reviewing a model leak-monitoring program, as described in an EPRI (Electric Power Research Institute) publication in the company library. She read through the model-program handbook and put together an implementation plan tailored to the power train in question with an eye to perhaps adapting the final version to all three 625-megawatt units at the facility. After consultation with Operations, her leak-monitoring plan was approved by management.

The first task was to assemble a leak-inventory database. Ms. Doe reviewed the maintenance and inspection reports for each reported leak of water, oil, and hydraulic fluid. The review included not only the most recent reports (since the last outage), but also any similar reports in the same area or same equipment prior to the last outage. In reviewing the paperwork, she noted the following.

♦   The absolute frequency and number of leak reports occurring two or more outages prior to the present were significantly less than the current frequency and number.

♦   The leaks mostly originated in the following items or areas:

• • • mechanical-end face seals
    • segmented bushing seals with constant barrier fluid purges
    • close-tolerance labyrinth seals
    • in locations near large roller bearings not rated for significant side loads
    • disc-pack couplings and similar
    • small-diameter support piping, less than a half inch, for the same items as above.

♦   In response to the increased leak incidence in some items, some compensating maintenance changes had been made.

• • • In some cases, scheduled part-replacement intervals were shortened to match the
      new, shorter intervals between excessive leakage.
    • In some cases, equivalent parts made by a different manufacturer were substituted,
      based upon the assumption that something had changed in the parts that had been reliably
      used for many years.
    • In some cases, the prior maximum leak rates were re-evaluated by Engineering and higher
      maximum leakage rates were permitted. Similarly, prior maximum vibration levels were
      re-evaluated by Engineering and, in some cases, higher maximum levels permitted.

Ms. Doe then made field visits to each leak location. As part of her documentation, she observed and measured the current leak rate under normal, steady-state operating conditions. She also photographed the general area around the leak area, noting the adjacent equipment upstream and downstream of the leak, and jotted down particulars related to the environment specific to the area around the leak.

After organizing, reviewing, and entering the collected information in a database, Ms. Doe again visited the location of each leak and measured the leakage rate under the same operating conditions as before. The data collected to this point allowed her to make some short-term extrapolations, which she documented with tables and graphs in a report. The report was used by the Operations and Maintenance Departments to assess possible equipment risks related to the leaks and to determine if any remedial work might be needed prior to the next outage. Some items were listed to be replaced on a contingency basis during any unscheduled outage that might occur.

Additionally, Ms. Doe measured vibration levels and the temperatures of equipment associated with the leaks. In that regard, she made the following observations.

♦   Almost every piece of equipment from which leaking was reported had measured vibration levels higher than the good velocity versus operating-rpm regions of a generalized equipment-vibration chart used by the Maintenance Department. Many levels were in the fair or slightly rough region of the chart. However, in most cases the vibration levels had not yet reached the rough portion of the chart. The dominant vibration frequencies were typically in the 1x, 2x and 3x operating frequencies.

♦   Nearly every piece of major equipment related to a leak had a surface temperature that was well above the plant’s ambient air temperature.

♦   As for small-diameter support piping that circulated local water, oil or hydraulic-fluids, inspection of leak origins indicated pipe sealant at joints had cracked over time due to shaking. Since this local piping wasn’t supported, constrained, or otherwise damped like large-diameter pipes, it appeared to have been sympathetically excited by vibrational energy transmitted from the equipment it supported.

♦   Close inspection of equipment related to the leaks found otherwise inconspicuous, light chisel marks in several places on the equipment and its base or support. Although they appeared to be match marks, Mrs. Doe noted those on the equipment items didn’t line up exactly with the corresponding marks on the supports. When questioned, the mechanics or techs accompanying Ms. Doe on these rounds indicated that they had never noticed the marks, but guessed they were made for alignment purposes during installation of the equipment.

ADDITIONAL CONSIDERATIONS
Having collected the above information, Ms. Doe also considered the following facts:

♦   General vibration charts are often based upon what is considered either acceptable or unacceptable to promote bearing life. A check of the origin of the vibration chart used by the Maintenance department by Ms. Doe found that this was also the case here. While recent vibration measurements were often fair or slightly rough, they were not deemed high enough to warrant shutting down the equipment in an unscheduled outage to remediate the condition.

♦   Mechanical-end face seals. labyrinth seals, and similar close-tolerance seals, along with some precision couplings often cannot tolerate for long the same level of vibration or misalignment that well-designed roller bearings can tolerate. Over time, those vibration levels shorten the service life of such seals and couplings. Typically the readily observable indication of an excessive vibration or misalignment problem in those items is leakage.

INVESTIGATIVE FINDINGS
With all this in mind, Ms. Doe reviewed various maintenance reports from the last outage and some from prior outages of equipment in locations corresponding to the current problem leak areas. She discovered the following.

♦   Prior to the last two outages, the Maintenance Department performed both cold and hot alignments of major equipment items that heated up during normal operation. Vibration levels measured after hot-alignment procedures were completed and the equipment’s operational temperatures stabilized were typically in the smooth or very-smooth When those vibration levels were achieved after a hot alignment was completed, match marks were left on the equipment to facilitate future hot-alignment work.

♦   Further, cold- and hot-alignment work prior to the last two outages was done primarily (countless times), by two experienced mechanics working with the Operations Department. But, after many years of fine service to the company, both gentlemen had retired prior to the last two outages. And before leaving, neither of them was asked to show other mechanics at the site how they had performed hot alignments.

♦   When cold- and hot-alignment work was completed during subsequent operation of equipment, either no leakage or minimal leakage was reported coming from the seals, couplings, or supporting small-diameter piping listed in Ms. Doe’s database.

♦   In the last two outages, though, only cold alignments were performed. Doe found records of the vibration levels measured after those alignments that were generally in the smooth range. Conspicuously missing, however, were measurements of or references to vibration levels after hot alignments. Further review of the outage records found that the cold-alignment work was performed by an outside contractor using the utility company’s alignment instrumentation.

During person-to-person interviews with various Operations and Maintenance personnel, Ms. Doe learned that in the two most recent outages, after cold alignment was completed by the contractor, no hot alignment was done. Two reasons were cited as the basis for this decision. The first was that the outage was running behind schedule and the equipment was on the critical path. The second was that the cold-alignment measurements were in the smooth range.

Consequently, the time required for hot alignments on critical-path equipment items was sacrificed to make up lost time on the outage’s critical-path schedule. Vibration measurements made right after the facility went back on line, which in some cases had increased to the good range, were deemed to be an acceptable trade for the decrease in critical-path time. The omission of outage hot-alignment work for significant plant equipment was then carried over into the next outage schedule and repeated.

Finally, when she checked with the Training Department, Ms. Doe found that no recent training on equipment alignment had been conducted. The reason: It had been decided that an outside contractor would be aligning major pieces of equipment just as they had done in the last two outages.

THE ROOT-CAUSE REPORT
Ms. Doe took the above information and prepared a root-cause report on the leakage problem. She explained in the report that when certain equipment heats up during operation, the temperature expansion effects coupled with dynamic loading effects often cause eccentricity in alignment to increase. The increase in eccentricity often leads to increased vibration levels, usually in the 1x, 2x, and 3x RPM ranges of the equipment, as compared to the cold alignment vibration measurements.

To be clear, the increase in vibration levels may not, in the short term, affect major bearings, which usually have ample margin built into their design for long life. However, the increase can impact close-tolerance mechanical seals and couplings, and possibly excite sympathetic vibrations in small diameter piping connected to these items that is not well constrained. Thus, Ms. Doe’s report concluded that the increase in leakage was the direct result of not performing hot alignments to minimize alignment eccentricities at operating conditions.

Her report further noted that the leakage problem was also a result of a loss of skill and knowledge. Following the retirement of two mechanics who were experienced in conducting hot alignment of important equipment, the company became dependent upon outside contractors. While the contractors performed acceptable cold-alignment work as assigned, subsequent hot alignment work was not done due to schedule pressure. It wasn’t recognized at the time that a trade had been made: The longer service life of components such as precision seals, close-tolerance couplings, and the like was traded for an immediate reduction in outage time.

REAL-WORLD BOTTOM LINE
What really was the root cause of the facility’s leakage problem? This story provides plenty of fodder for some animated discussions.

However, rather than split hairs in a operation about whether this or that item is the real root cause of a problem, it would be more productive to address deficient practices, such as those uncovered in Ms. Doe’s investigations.

It’s also worth noting how Ms. Doe went beyond simply implementing a successful leak-monitoring program as assigned. Her approach helped identify the underlying causes of the facility’s leakage problems, an approach that would eventually make a leak-monitoring program somewhat moot. It’s an approach that others would be wise to take.TRR

---

ABOUT THE AUTHOR
Randall Noon is a registered professional engineer and author of several books and articles about failure analysis. He has conducted root-cause investigations for four decades, in both nuclear and non-nuclear power facilities. Contact him at noonrandy@icloud.com.

---

Tags: reliability  availability, maintenance, RAM, root-cause analysis, fluid leaks, leak investigation, leak monitoring

---