October 20, 1997 MEMORANDUM FOR THE SECRETARY FROM: John C. Layton Inspector General SUBJECT: INFORMATION: Report on "Audit of the Contractor Incentive Program at the Nevada Operations Office" BACKGROUND: The Department of Energy (Department) is using performance-based contracts to address problems associated with its traditional management and operating contracts. Under this approach, contractor performance is to be evaluated against performance measures which are clearly stated, results-oriented, and established prior to performance. The performance measures, which reflect the Department's expectations of the contractor, are the basis for rewarding superior contractor performance through the use of incentive fees. The purpose of our audit was to determine whether performance-based contracting, as incorporated in the Bechtel Nevada Corporation (Bechtel) contract for the management of the Department's Nevada Test Site and associated activities, conformed to these principles. DISCUSSION: The audit disclosed that the performance measures associated with the Bechtel contract did not conform to requirements set forth in the Contract Reform Team report and the Bechtel contract. The Nevada Operations Office (Nevada) established measurement milestones after the work had actually been completed by Bechtel. Further, many of the performance measures were vague and non- specific and, as a result, Nevada rewarded performance that could not be objectively validated. These problems were attributable to the general difficulties in transitioning to the new contracting concept. As a result, the success of the effort to implement performance-based contracting at Nevada was at risk. We recommended that the Manager, Nevada Operations Office, establish performance measures that are clearly stated and results-oriented. Further, such measures should be established before the work is performed and the results should be validated before incentive fees are disbursed. We also recommended that Nevada review all performance measure incentive fees paid and seek recovery where other work was accomplished prior to setting the measure, where the performance measure was not met, or where the savings cannot be demonstrated. Management concurred with both recommendations. Attachment cc: Deputy Secretary Under Secretary U.S. DEPARTMENT OF ENERGY OFFICE OF INSPECTOR GENERAL AUDIT OF THE CONTRACTOR INCENTIVE PROGRAM AT THE NEVADA OPERATIONS OFFICE The Office of Inspector General wants to make the distribution of its reports as customer friendly and cost effective as possible. Therefore, this report will be available electronically through the Internet at the following alternative addresses: Department of Energy Headquarters Gopher gopher.hr.doe.gov Department of Energy Headquarters Anonymous FTP vm1.hqadmin.doe.gov Department of Energy Human Resources and Administration Home Page http://www.hr.doe.gov/ig Your comments would be appreciated and can be provided on the Customer Response Form attached to the report. This report can be obtained from the U.S. Department of Energy Office of Scientific and Technical Information P.O. Box 62 Oak Ridge, Tennessee 37831 Report Number: DOE/IG-0412 Western Regional Audit Office Date of Issue: October 20, 1997 Albuquerque, New Mexico 87185 AUDIT OF THE CONTRACTOR INCENTIVE PROGRAM AT THE NEVADA OPERATIONS OFFICE TABLE OF CONTENTS Page SUMMARY......................................... 1 PART I - APPROACH AND OVERVIEW........................... 2 Introduction.................................... 2 Scope and Methodology........................... 2 Background...................................... 3 Observations and Conclusions.................... 3 PART II - FINDING AND RECOMMENDATIONS...................... 4 Performance Measures.............................. 4 PART III- MANAGEMENT AND AUDITOR COMMENTS....................9 PART IV - APPENDIX Summary of Related Office of Inspector General Reports U.S. DEPARTMENT OF ENERGY OFFICE OF INSPECTOR GENERAL OFFICE OF AUDIT SERVICES AUDIT OF THE CONTRACTOR INCENTIVE PROGRAM AT THE NEVADA OPERATIONS OFFICE Audit Report Number: DOE/IG-0412 SUMMARY As a result of recommendations in the 1994 report, Making Contracting Work Better and Cost Less, the Department of Energy (Department) has adopted performance-based contracting for the management and operation of its major facilities. Under this approach, contractor performance is to be evaluated against performance measures which are clearly stated, results-oriented, and established prior to performance. The performance measures, which reflect the Department's expectations of the contractor, are the basis for rewarding superior contractor performance through the use of incentive fees. The purpose of our audit was to determine whether performance-based contracting, as incorporated in the Bechtel Nevada Corporation (Bechtel) contract for the management of the Department's Nevada Test Site and associated activities, conformed to these principles. The audit disclosed that the performance measures associated with the Bechtel contract did not conform to requirements set forth in the Contract Reform Team report and the Bechtel contract. The Nevada Operations Office (Nevada) established measurement milestones after the work had actually been completed by Bechtel. Further, many of the performance measures were vague and non-specific and, as a result, Nevada rewarded performance that could not be objectively validated. These problems were attributable to the general difficulties in transitioning to the new contracting concept. As a result, the success of the effort to implement performance-based contracting at Nevada was at risk. We recommended that the Manager, Nevada Operations Office, establish performance measures that are clearly stated and results-oriented. Further, such measures should be established before the work is performed and the results should be validated before incentive fees are disbursed. We also recommended that Nevada review all performance measure incentive fees paid and seek recovery where the work was accomplished prior to setting the measure, where the performance measure was not met, or where the savings cannot be demonstrated. Nevada concurred with both recommendations. ________(Signed)_______________ Office of Inspector General PART I APPROACH AND OVERVIEW INTRODUCTION As a result of recommendations in the 1994 report, Making Contracting Work Better and Cost Less, the Department has adopted performance-based contracting for the management and operation of its major facilities. Under this approach, contractor performance is to be evaluated against performance measures which are clearly stated, results-oriented, and established prior to performance. The performance measures, which reflect the Department's expectations of the contractor, are the basis for rewarding superior contractor performance through the use of incentive fees. The purpose of our audit was to determine whether performance-based contracting, as incorporated in the Bechtel contract, conformed to these principles. Specifically, we wanted to determine if Nevada paid incentive fees to Bechtel for performance that was measured against standards established in advance of the activity and to determine if the performance measures were sufficiently specific to justify the incentive fees. SCOPE AND METHODOLOGY The audit was conducted at Nevada and at Bechtel facilities in North Las Vegas, Nevada, from February through June 1997. To accomplish the audit objectives, we: o reviewed performance measures, including revisions; o compared available completion documentation for the measures with the measure requirements; o compared performance measure requirements with strategic planning documents; o compared performance measures with the Department's report on contract reform; and, o interviewed Department and contractor officials. We reviewed 6 of 27 Fiscal Year (FY) 1996 performance measures in detail and made a limited review of the 25 performance measures for FY 1997. The audit was conducted according to generally accepted Government Auditing Standards for performance audits, which included tests of internal controls and compliance with laws and regulations to the extent necessary to satisfy the audit objective. We assessed significant internal controls designed to ensure the proper payment of incentive fees associated with performance measures. Because our review was limited, it would not necessarily have disclosed all internal control deficiencies that may have existed at the time of our audit. The audit did not rely extensively on computer processed data. Therefore, we did not fully examine the reliability of the computerized data used. We discussed the results of our review during the course of the audit and at an exit conference with Nevada officials on August 27, 1997. BACKGROUND In January 1996, Bechtel became the prime management contractor for the Department's Nevada Test Site and associated activities. Consistent with the recommendations of the Contract Reform Team, this was a performance-based contract which incorporated the performance measure and incentive fee concepts. The FY 1996 performance measures covered the first nine months of the contract, January 1, 1996 through September 30, 1996. During this period, Bechtel's operating budget was $347 million. In total, performance incentive fees of $19 million were available for this period and $14.6 million was subsequently paid. Bechtel's operating budget for FY 1997 was $259 million and the maximum available incentive fee was $16.7 million. OBSERVATIONS AND CONCLUSIONS This audit report raises a number of concerns regarding the implementation and administration of the contractor performance program at Nevada. Further, this is one of several Office of Inspector General (OIG) reports that have identified similar issues at other Department locations. In responding to a recent report on contract reform at Rocky Flats, Audit of the Contractor Incentive Programs at the Rocky Flats Environmental Technology Site, the Department recognized that improvements could be made. As a result, the Deputy Assistant Secretary for Procurement and Assistance Management directed the following actions: o Operations Offices shall establish a senior management team to perform an integrated review of all performance incentives prior to their being finalized. o Performance incentives shall be subject to Headquarters' review and approval until lessons learned and remedial guidelines have been implemented. We support these actions and believe that their implementation along with the recommendations in this report will not only help the Rocky Flats Field Office but also allow Nevada to improve its implementation and administration of contract reform. PART II FINDING AND RECOMMENDATIONS Performance Measures FINDING The Department's commitment to performance-based contracting required the establishment of performance measures that formalized contractor performance expectations. The performance measures were to be clearly stated, results-oriented, and were to define the anticipated performance in advance to ensure that the Department's objectives were accomplished. The audit disclosed, however, that the performance measures associated with the Bechtel contract did not conform to these standards. Nevada established measurement milestones after the work had actually been completed by Bechtel. Further, many of the performance measures were vague and non-specific and, as a result, Nevada rewarded performance that could not be objectively validated. These problems were attributable to the general difficulties in transitioning to the new contracting concept. As a result, the success of the effort to implement performance-based contracting at Nevada was at risk. RECOMMENDATIONS We recommend that the Manager, Nevada Operations Office: 1) establish performance measures that are clearly stated and results-oriented. Further, such measures should be established before the work is performed and the results should be validated before incentive fees are disbursed; and, 2) review all performance measure incentive fees paid and seek recovery where the work was accomplished prior to setting the measure, where the performance measure was not met, or where the savings cannot be demonstrated. MANAGEMENT REACTION The Manager, Nevada Operations Office, concurred with both recommendations. Part III of this report includes detailed management and auditor comments. DETAILS OF FINDING The Bechtel contract required that the parties agree on a set of contractor performance measures that identified performance in advance, and that were clearly stated and results- oriented. Such measures were to specify the Department's expectations so that the contractor could focus on the best way to carry out its responsibilities. Further, clearly defined performance measures were to be the baseline against which Bechtel's performance was to be evaluated and rewarded through an incentive fee structure. FY 1996 Performance Measures The Bechtel contract included performance measures for FY 1996 that did not establish performance requirements in advance, were not clearly stated, and were not results-oriented. In fact, many of the performance milestones were established after the contractor had completed the required work effort. Further, some of the measures were so vague as to be ineffective as performance evaluation tools. Consequently, Nevada accepted performance claims from the contractor and paid significant incentive fees to Bechtel despite the fact that the claims could not be objectively validated. We took exception to three of the six measures examined. Establishing Milestones For two of the measures, the Department's requirements had not been established in advance of contractor performance. The Subcritical Experiments measure required Bechtel to support two experiments scheduled during the fiscal year. The Laboratory Readiness measure required Bechtel to support the requirement for above-ground experiments. The available fees for the measures were $1.9 million and $950,000, respectively. Of this total, $1.7 million or 60 percent was earmarked for work "to be determined." As the fiscal year progressed, milestones were established that identified the work to be performed for Bechtel to earn its incentive fee. However, many of the milestones were added after the work had already been accomplished by Bechtel. Ten such milestones were added to the Subcritical Experiments measure during the course of FY 1996. On June 25, 1996, for example, Nevada established milestones to (1) replace a hoist for the underground facility and (2) perform a "safety walkdown." However, the actual work associated with these milestones had been completed on February 5, and April 1, 1996, respectively. On September 10, 1996, seven additional milestones were added to the measure even though the required work had been completed by the contractor between June 20, and August 30, 1996. On October 15, 1996, the last milestone was added to the measure which had been met by the contractor on September 26, 1996. The Laboratory Readiness measure included two milestones added on September 5, 1996. Again, this was after the contractor had already completed the required work. Performance milestones established after the fact do not incentivize future contractor performance. This practice created a retroactive, artificial basis to support the payment of contractor fees and was incompatible with the basic principles of performance-based contracting. In these specific cases, Bechtel received an incentive fee of $775,600. We concluded that payment of these fees could not be supported. Although Nevada acknowledged that there were delays in getting metrics (milestones) formally approved, it believed that sufficient evidence existed to demonstrate that the metrics were developed and agreed to before accomplishing the work. However, Nevada was unable to provide any documentation to support its position. Measuring Performance The Indirect Cost Reductions performance measure was neither clearly stated nor results-oriented. Thus, Bechtel's success or lack of success in accomplishing this measure could not be validated objectively. The indirect cost reduction measure required Bechtel to reduce indirect and fringe benefit costs at the Nevada site. Specifically, such costs were to be reduced by up to $38 million with the "savings" made available for alternate Departmental uses. The process to use cost reduction savings for alternative uses within the Department was referred to as "reuse." To incentivize Bechtel's efforts, Nevada offered a performance incentive fee of up to $3.2 million. Ultimately, Bechtel was paid a fee of $1.8 million for claimed savings of $10.2 million. Since the ability to validate claimed cost savings is crucial in determining incentive fees, we attempted to validate the largest of Bechtel's claimed savings. We could not do so because budget baseline and change documents were not available. The baseline information is the criteria against which cost reduction claims are evaluated. One portion of the savings, according to Bechtel's claim, resulted from an initiative to reduce the total number of internal procedures. The audit disclosed that Nevada accepted Bechtel's claim that 1,000 unnecessary internal procedures had been eliminated. Bechtel claimed that it saved $1.5 million annually through enhanced personnel productivity by eliminating these unneeded procedures. However, Bechtel could not relate the staff time spent on each procedure to a direct dollar saving. Nevada, nonetheless, paid Bechtel a cost reduction incentive fee of $301,000. Neither Nevada nor Bechtel could demonstrate that the claimed savings had been made available for Departmental reuse either through a deobligation to the contract or a reprogramming of funds. The reuse provision was a key element in the Department's procedure to validate the cost savings claimed by the contractor. Because Bechtel could not show how hard dollar savings had been reused, we questioned the validity of the fee paid to Bechtel. In another instance, Bechtel claimed a savings of $2.8 million because employee fringe benefit costs were less than those of the previous site operating contractors. For this action, the Department paid Bechtel a cost reduction incentive fee of $343,500. Once again, the "reuse" test failed to confirm contractor claims of cost savings. Neither Nevada nor Bechtel could demonstrate that the claimed savings had been deobligated from its contract or reprogrammed for other use. Therefore, the incentive fee of $343,500 paid to Bechtel appeared questionable. We discussed our concerns regarding the difficulty in tracking this particular claim and contractor savings claims, in general, with a senior Bechtel official and with Nevada. The Bechtel official agreed that it may not be possible to demonstrate how all claimed savings were reused. But, he contended, the savings did occur. In response to our concern, Nevada acknowledged that its validation of claimed savings was subjective because of the difficulty in establishing baselines from which to measure and pointed out that this difficulty was compounded by the transition from three contractors to one. It believed that when the contractor could identify an action taken to reduce cost and when the savings claimed appeared reasonable and consistent with the objectives of the performance measure, the claim was valid. However, neither Nevada nor Bechtel provided documentation to demonstrate that claimed savings actually resulted in reduced contract cost or increased work scope. By continuing to subjectively determine performance award fees, Nevada has demonstrated its fundamental misunderstanding of performance award fee contracts. FY 1997 Performance Measures The FY 1997 performance measures were established in February 1997 or in the fifth month of the fiscal year. According to the contract, they should have been established before the fiscal year began. Further, many of the FY 1997 measures, as was the case in FY 1996, were neither clearly stated nor were they results-oriented. They did not specifically describe the Department's expectations for contractor performance. Consequently, determining the incentive fees for FY 1997 will likely require a great deal of interpretation by the Department and the contractor. Many of the FY 1997 measures were subjective and process- oriented. For example, one of the key measures described the expected performance as follows: Reduce the cost of doing business through business development, more efficient processes and practices, and the identification of innovative problem solving solutions; demonstrate a strong commitment to the business management and oversight program; demonstrate the cost effective and efficient implementation of necessary and sufficient standards; and demonstrate that open and candid communication and trust is a recognized Bechtel Nevada priority and a company-wide effort. Despite the fact that there was a potential incentive fee of $2.5 million associated with this measure, its vague and indefinite language makes it nearly impossible for the Department to objectively evaluate contractor performance. We found that other FY 1997 performance measures were defined in similar terms. This imprecise language, which leads to subjective judgments, is not, in our opinion, the type of criteria envisioned in a performance-based contract. Transitioning to a New Contract The problems identified in this audit were generally attributed to the challenges associated with the transition from the management and operating contract to the new performance- based contract. One division manager expressed frustration with the limited time given at the start of the performance-based contract to define measures and with the lack of procedures or policy related to the establishment of milestones throughout the year. He explained that there was a big change in mindset from the traditional award fee process where fee was decided at the end of a period to the performance-based contract where expectations for contractor performance and the associated incentive fee were to be established and fully understood at the outset of the period of performance. Another division manager explained that with a totally new contract vehicle, Nevada struggled with its role of setting performance targets. Further, he explained that he was initially unaware of the need to validate contractor claims of incentive fee earned. However, he stated that he now requires one of his staff members to validate claims and document the results. Finally, financial officials said that the use of the word "reuse" as a prerequisite to the successful accomplishment of contractor initiated cost reductions was unclear. They explained that when the contractor could demonstrate action was taken to reduce costs and the estimate of costs saved appeared reasonable, the performance was deemed acceptable. In summary, all of the officials we talked to contended that the cause of difficulties in implementing an effective performance-based contracting process was the inexperience in designing performance measures. Continued Improvement In moving to performance-based contracting, the Department sought to obtain superior contractor performance by establishing expectations, measuring results, and incentivizing contractor activities. However, we concluded that, despite its intentions, the Department paid Bechtel at least $1.42 million in incentive fees which could not be fully justified. As a result, we questioned these payments. We believe that Nevada needs to adopt performance measures which are clear and measurable so that contractor performance is enhanced through appropriate incentives. PART III MANAGEMENT AND AUDITOR COMMENTS The Manager, Nevada Operations Office, concurred with both recommendations. The comments made with regard to the recommendations and our responses follow. Recommendation 1 Recommendation. We recommend that the Manager, Nevada Operations Office, establish performance measures that are clearly stated and results-oriented. Further, such measures should be established before the work is performed and the results should be validated before incentive fees are disbursed. Management Comments. Nevada concurred that performance measures should be clearly stated and results-oriented. Nevada also concurred that measures should be established before the work is performed and results validated before incentive fees are disbursed. As recognized in the audit report, FY 1996 was the first year that Nevada administered a performance-based management contract. Nevada stated that it fully recognized that opportunities for improvement would be identified as this process evolved, and has been proactive in instituting changes which are designed to meet both the spirit and intent of the recommendation. To demonstrate its commitment to improving the administration of the incentive fee process consistent with the recommendation, Nevada has developed a procedural instruction institutionalizing the processes for scorecard development, change control, and performance evaluation. This instruction provides specific guidelines and requirements for documenting the process and rationale for selecting the proposed performance objectives, performance measures, and proposed weights. The process for defining the types of changes that can be made also has been formalized and is designed to ensure that there is adequate analysis and coordination by all parties prior to making any changes to the performance measures. For FY 1998, Nevada has revised the approach to fee administration for the performance-based management contract to provide for an award fee component to address those overall activities which do not readily lend themselves to the incentive fee concept. Under the revised process as outlined in the procedural instruction, the incentive fee component provides management emphasis on Nevada's critical few management and program performance objectives. This will facilitate the focus on performance measures that are results-oriented and the evaluation of performance based on objective criteria. Auditor Comments. Nevada's comments are responsive to the recommendation. Recommendation 2 Recommendation. We recommend that the Manager, Nevada Operations Office, review all performance measure incentive fees paid and seek recovery where the work was accomplished prior to setting the measure, where the performance measure was not met, or where the savings cannot be demonstrated. Management Comments. Nevada concurred and said that in June 1997, a joint Nevada and Headquarters select team commissioned under the auspices of the Secretary's Performance Based Incentive Review initiative conducted a thorough review of all FY 1996 performance based incentives in the Bechtel Nevada contract. This review and other independent oversight activities performed by Nevada identified two instances in which fee was determined not to be warranted based on strict application of the performance metrics. In each of these instances, the questioned fee was recovered by Nevada from the contractor. Nevada stated that this review and other oversight activities it performed, when coupled with audits by the Bechtel Nevada Internal Audit Group and the OIG, provide reasonable assurance that, with the exception of the two instances noted, the FY 1996 fee determinations were justified and supportable and that further reassessments of the FY 1996 performance measures would be unnecessary. In summarizing its position, Nevada acknowledged the difficulty and complexity of implementing this process in the first year. Nevada further stated that FY 1996 performance measures have been thoroughly reviewed and the audits have resulted in approximately the same procedural recommendations. As noted by the Nevada and Headquarters joint review team, Nevada had already identified a number of these issues through its ongoing efforts to improve contract administration and contractor performance. Nevada has worked aggressively to implement all recommendations from these review activities. Auditor Comments. Although Nevada concurred in the recommendation and said that action had been taken to recover fee that was not warranted, it did not specifically state or provide us with data demonstrating that the amounts questioned in this report were fully collected or resolved. Until this is clarified, this issue remains open. PART IV APPENDIX Summary of Related Office of Inspector General Reports 1. DOE/IG-0401, Inspection of the Performance Based Incentive Program at the Richland Operations Office, March 1997. This inspection reviewed the Richland Operations Office's (Richland) Performance Based Incentive Program. The report showed that Richland did not always make the best use of the incentive fees paid to the management and operating contractor. The inspection found examples of incentive fees paid that were (1) excessive when compared with the cost of labor and material to perform the work, (2) for work that was accomplished before Richland's program was established, (3) for work that was not completed, and (4) for work that was easily achieved by the contractor. There was also an instance where the contractor compromised quality and safety in order to earn an incentive fee. The report also contained a number of administrative weaknesses and observations. 2. DOE/IG-0410, Audit of Environmental Restoration at the Los Alamos National Laboratory, July 1997. This audit reviewed the Los Alamos National Laboratory's performance criteria for remediating contaminated sites. The report showed that the performance criteria used to evaluate cost effectiveness of remediating contaminated sites were not always reasonable, measurable, and complete. 3. DOE/IG-0411, Audit of the Contractor Incentive Programs at the Rocky Flats Environmental Technology Site, August 1997. This audit reviewed the Rocky Flats cost savings awards and performance fees paid. The report showed that (1) Rocky Flats approved cost savings awards that were not innovative and generally did not return savings to the Department as required by Departmental guidance, and (2) Rocky Flats performance measures did not always include clearly defined criteria, were not structured to encourage and reward superior performance, and were often process- rather than results-oriented as required by the report on contract reform. Report No. DOE/IG-0412 CUSTOMER RESPONSE FORM The Office of Inspector General has a continuing interest in improving the usefulness of its products. We wish to make our reports as responsive as possible to our customers' requirements, and therefore ask that you consider sharing your thoughts with us. On the back of this form, you may suggest improvements to enhance the effectiveness of future reports. Please include answers to the following questions if they are applicable to you: 1. What additional background information about the selection, scheduling, scope, or procedures of the audit or inspection would have been helpful to the reader in understanding this report? 2. What additional information related to findings and recommendations could have been included in this report to assist management in implementing corrective actions? 3. What format, stylistic, or organizational changes might have made this report's overall message more clear to the reader? 4. What additional actions could the Office of Inspector General have taken on the issues discussed in this report which would have been helpful? Please include your name and telephone number so that we may contact you should we have any questions about your comments. Name ___________________________ Date__________________________ Telephone ______________________ Organization__________________ When you have completed this form, you may telefax it to the Office of Inspector General at (202) 586-0948, or you may mail it to: Office of Inspector General (IG-1) U.S. Department of Energy Washington, D.C. 20585 ATTN: Customer Relations If you wish to discuss this report or your comments with a staff member of the Office of Inspector General, please contact Wilma Slaughter at (202) 586-1924.