*************************************************************** **This document is an ASCII formatted version of a printed ** **document. The page numbers in this eletronic version may ** **not be in the same order as the printed document. The ** **printed document may also contain charts and photographs ** **which are not reproduced in this electronic version. If ** **you require the printed version of this document, contact ** **the Office of Inspector General (IG-1), Department of ** **Energy, 1000 Independence Avenue, SW, Washington, D.C. ** **20585 or call Wilma Slaughter at (202) 586D1924. ** *************************************************************** August 2, 1995 IG-1 INFORMATION: "Audit of Program Administration by the Office of Energy Research" The Secretary BACKGROUND: Congressional and Departmental initiatives envision improved contract and program performance by requiring program managers to set measurable performance expectations. Congress recognized the need for performance expectations in passing Public Law 103D62 entitled "Government Performance and Results Act of 1993." The Act required agencies to set performance goals, measure program performance against those goals, and report publicly on program progress. Also, the Department's Contract Reform Team noted that the pursuit and use of clear expectations, accurate performance metrics, and evaluation against those metrics could resolve many of the Department's contract management problems. The audit was performed to determine whether the Office of Energy Research (Energy Research) had established performance expectations, including performance criteria and metrics, and used these expectations to monitor progress for basic and applied research performed at the Department's national laboratories. DISCUSSION: The audit disclosed that Energy Research generally did not clearly specify--at either an aggregated program level or an individual task level--performance expectations for research at the Department's national laboratories. Specifically, 237 of 264 tasks examined did not contain a clear statement of the work to be performed, resource limits, milestones, specific deliverables, or any other performance criteria and metrics that could be used to measure performance. The absence of documented performance criteria and metrics in the work authorizations we examined made it impossible for us to determine whether contractors performance of specific research tasks or programs met Departmental expectations. We also noted that Energy Research's current administrative process gives the appearance of decision making at an individual task level as it requires the proposing, funding, and accounting for research at the individual task level. However, as stated by officials in Energy Research, and confirmed by our audit, management decisions are made at an -2- aggregated program level rather than at the individual task level. The current administrative process does not provide Departmental elements responsible for performance-based contract management a method of determining whether schedules were met, resources were properly used, deliverables were as specified, and the research performed was within the proper mission. Performance criteria and metrics would also assist external reviewers in evaluating contractor management of research. We, therefore, recommended that the Director, Office of Energy Research review the administrative process and make appropriate changes. Specifically, consideration should be given to: (1) authorizing work based on requests received, and (2) evaluating research progress based on the metrics in these authorizations. The Director agreed in part to the finding and recommendations and initiated corrective action. (Signed) John C. Layton Inspector General Attachment cc: Deputy Secretary Under Secretary U.S. DEPARTMENT OF ENERGY OFFICE OF INSPECTOR GENERAL AUDIT OF PROGRAM ADMINISTRATION BY THE OFFICE OF ENERGY RESEARCH The Office of Inspector General wants to make the distribution of its reports as customer friendly and cost effective as possible. Therefore, this report will be available electronically through the Internet five to seven days after publication at the following alternative addresses: Department of Energy Headquarters Gopher gopher.hr.doe.gov Department of Energy Headquarters Anonymous FTP vm1.hqadmin.doe.gov U.S. Department of Energy Human Resources and Administration Home Page http://www.hr.doe.gov/refshelf.html Your comments would be appreciated and can be provided on the Customer Response Form attached to the report. Report Number: DOE/IG-0376 Eastern Regional Audit Office Date of Issue: August 2, 1995 Oak Ridge, Tennessee 37830 REPORT ON AUDIT OF PROGRAM ADMINISTRATION BY THE OFFICE OF ENERGY RESEARCH TABLE OF CONTENTS Page SUMMARY ................................. 1 PART I D APPROACH AND OVERVIEW ................... 3 Introduction ............................ 3 Scope and Methodology ................... 3 Background .............................. 4 Observations and Conclusions ............ 5 PART II D FINDING AND RECOMMENDATIONS ............. 7 Program Administration................... 7 PART III D MANAGEMENT AND AUDITOR COMMENTS ......... 14 Appendix A- Energy Research Funds Provided to the Management and Operating Contractors............................ 16 Appendix B- Fusion Energy Program Example of Expectations........................... 17 Appendix C- Office of Energy Research Response ...... 18 to the Official Draft Report #=1 U. S. DEPARTMENT OF ENERGY OFFICE OF INSPECTOR GENERAL OFFICE OF AUDIT SERVICES REPORT ON AUDIT OF PROGRAM ADMINISTRATION BY THE OFFICE OF ENERGY RESEARCH Audit Report Number: DOE/IG-0376 SUMMARY The Office of Energy Research (Energy Research) carries out a broad range of advisory, coordination, and program management activities for research in basic energy sciences, high energy and nuclear physics, magnetic fusion energy, and biomedical and environmental sciences. This basic scientific and applied research is conducted by scientists at the Department's national laboratories, which are operated by management and operating contractors (contractors). The objective of the audit was to determine whether Energy Research had established performance expectations, including performance criteria and metrics, and used these expectations to monitor progress for basic and applied research performed at the Department's national laboratories. Congressional and Departmental initiatives envision improved contract and program performance by requiring program managers to set measurable performance expectations. Even though research outcomes are inherently unpredictable, performance expectations can and should be established for scopes of work, milestones, resource limits and deliverables. However, Energy Research generally did not clearly specify--at either an aggregated program or individual task level--such expectations for research at the Department's national laboratories. While information was available in the contractor's research proposals, Energy Research essentially relied on the contractors to initiate and execute the research without agreement on expectations. This practice provided the Department with little basis to measure and evaluate contractor performance. Energy Research agreed in part with the finding and will take action on the recommendations in the report. However, Energy Research is concerned that unnecessary requirements on its research performers could reduce desirable flexibility, stifle creativity, and lead to inferior results. (Signed) Office of Inspector General PART I APPROACH AND OVERVIEW INTRODUCTION Energy Research provided funds of about $1.4 billion annually for both applied and basic research programs conducted at the Department's national laboratories operated by contractors including universities and consortia of universities. The objective of the audit was to determine whether Energy Research had established performance expectations, including performance criteria and metrics, and used them to monitor progress for research performed by these contractors. SCOPE AND METHODOLOGY The audit was performed from March 1994 through November 1994, at the Office of Energy Research, Chicago Operations Office (Chicago) and Argonne National Laboratory (Argonne), and covered Fiscal Years 1992 through 1994. Our audit included the following methodologies: o Reviewed research proposals submitted by the contractors; o Evaluated the budget validation process; o Determined the level of funding provided by the Department for work to be accomplished by the selected contractors; o Judgmentally selected 264 research tasks representing over $600 million of authorized research at the Departmental laboratories. o Evaluated the guidance provided by Energy Research in administering funds; o Interviewed Argonne, Chicago, and Energy Research personnel; and o Analyzed the methods used by Energy Research to evaluate performance of research. The audit was performed according to generally accepted Government auditing standards for performance audits and included tests of internal controls and compliance with laws and regulations to the extent necessary to satisfy the audit objective. We assessed significant internal controls with respect to providing and administering funds for basic and applied research to contractors. Our assessment consisted of a review of policies and procedures associated with the proposal of research, the authorization of that research, and the guidance provided by Energy Research over the contractors' management of research. Because our review was limited, it would not necessarily have disclosed all internal control deficiencies that may have existed at the time of our audit. We discussed our finding with officials in the Office of Energy Research during an exit conference on June 29, 1995. BACKGROUND Energy Research engages contractors, primarily individual universities or consortia of universities, to perform a broad range of basic and applied energy research activities at Departmental laboratories. Contractors, for example, perform research in basic energy sciences, high energy and nuclear physics, magnetic fusion, and environmental and health sciences. During the three fiscal years ending 1994, Energy Research funded about $1.4 billion annually for these research activities. The chart in Appendix A provides a summary by contractor of Energy Research funding to Departmental laboratories for Fiscal Years 1992 through 1994. The contractual language binding the contractors and the Department recognizes a close relationship between the parties concerning the scope and development of the research work. Energy Research's management practices have allowed program offices within Energy Research to fund Departmental laboratories using a system that places extensive reliance on the contractors that operate the laboratories. The contractors and Energy Research define the mission of laboratories through an annual institutional planning process. Energy Research uses the institutional planning process to direct the contractors to develop specific laboratory missions, 20-year strategic plans, scientific initiatives, research programs, technology transfer activities, and the expected use of resources including facilities, funding and personnel. Contractors use their scientific and technical management expertise to develop plans as to how future resources should be utilized to meet challenges facing the scientific community and make the most advantageous use of the Department's scientific research facilities. Contractors also use the institutional planning process to guide them in developing research initiatives and resource estimates for the annual budget request to Energy Research. Energy Research authorizes funds to contractors for the performance of research through documents called work authorizations. These work authorizations provide the formal channel for Energy Research to communicate performance expectations for specific work tasks to contractors. The inclusion of performance expectations in work authorizations would provide an opportunity to assess contractors' management of resources, adherence to schedules, and compliance with an agreed to deliverable. Energy Research relies on periodic reviews to understand and evaluate research at the national laboratories. Individual program offices evaluate contractor progress throughout the year using, where appropriate, external peer review at the task level. At the aggregated program level the Department relies on ad hoc peer review teams, which include specialists from outside the program offices, and Departmentally chartered advisory groups to monitor research. We recognize that the goals of the basic and applied research performed by these contractors are complex and provide challenges for Energy Research program managers to set expectations. For example, defining the specific outcomes of these complex research activities under study may be impossible to specify. However, the resources and tasks needed to accomplish the goals of the research can be established and used to evaluate how effectively and efficiently the contractors are managing the research. OBSERVATIONS AND CONCLUSIONS We found that Energy Research, except for its Office of Fusion Energy, generally did not specify performance criteria and metrics, even though useful information was available in contractor research proposals. Specifically, 237 of the 264 tasks we selected for review did not specify the scope of work, set resource limits, establish milestones, or specify deliverables even though such information was proposed by contractors. The audit disclosed that the Energy Research office responsible for fusion energy developed measurable performance expectations from information in the contractor proposals. These expectations were communicated to contractors as part of the written authorization to perform a specific research task. This occurred because Energy Research generally envisioned its role as one of providing funds to support basic and applied research at Departmental laboratories without constraining the scientific freedom of the contractors to manage research. In effect, the execution of this practice did not provide documented performance expectations so that Departmental elements responsible for performance-based contract management could determine whether contractors met schedules, properly used resources, and provided deliverables as specified. Thus, we recommend that the Director, Office of Energy Research, require program offices to include performance criteria and metrics when authorizing work at Departmental laboratories and use those metrics to determine program progress. PART II FINDING AND RECOMMENDATIONS Program Administration FINDING Performance criteria and metrics are essential for the Department to focus and coordinate research objectives and hold contractors accountable for work performed. However, Energy Research, excepting the office responsible for fusion energy, generally did not include performance criteria and metrics in work authorizations that provided for research at Departmental laboratories. Specifically, work authorizations for 237 of 264 tasks selected for review did not contain a clear statement of the work to be performed, resource limits, milestones, or specific deliverables. While information was available in the contractor's research proposals, Energy Research essentially relied on the contractors to initiate and execute the research without agreement on expectations. This practice precludes the establishment of documented performance criteria and metrics that Departmental elements responsible for performance-based contract management can use to determine whether contractors met the objectives of the Department for their research efforts. RECOMMENDATIONS We recommend that the Director of Energy Research review its administrative process and make appropriate changes. Specifically, consider: 1. authorizing work based on requests received, and 2. evaluating research progress based on the metrics in these authorizations. MANAGEMENT REACTION The Office of Energy Research agreed in part with the finding and will take action on the recommendations in the report. However, Energy Research is concerned that unnecessary requirements on its research performers could reduce desirable flexibility, stifle creativity, and lead to inferior results. A summary of management comments and the auditors reply are in Part III of this report. DETAILS OF FINDING ENERGY RESEARCH'S RESPONSIBILITY TO MANAGE PROGRAMS Performance expectations are essential to provide a basis for guiding and measuring performance and, thus, ensuring contractor accountability for work performed. Congress recognized the need for performance expectations in passing Public Law 103D62 entitled "Government Performance and Results Act of 1993." The Act required agencies to set performance goals, measure program performance against those goals, and report publicly on program progress. The Secretary established a Contract Reform Team to improve contract management. The team's report emphasized that aggressive changes were needed in the way the Department conducted business. The team noted that the pursuit and use of clear expectations, accurate performance metrics, and evaluation against those metrics could resolve many of the Department's contract management problems. The contract reform team's recommendations echo established procurement principles which recognize that performance expectations are essential to a quality procurement program. These principles also apply to the acquisition of scientific research services as well as the more conventional acquisition of goods and services. In either case, program managers should ensure the existence and use, as appropriate, of performance expectations that measure success in regard to technical achievement or progress, meeting cost and schedule goals, and validating the usefulness of methods or approaches proposed. PERFORMANCE INFORMATION AVAILABLE BUT NOT USED Energy Research generally did not use the information available in contractor work proposals to establish performance criteria and metrics to evaluate contractor management of resources to accomplish research. Contractors, for example, annually submitted work proposals which contained information concerning current technical progress, proposed work objectives, descriptions of work, expected milestones, expected future accomplishments, and proposed resource needs. Energy Research could have evaluated the reasonableness of contractors' information and used it to establish performance criteria and metrics that could have included: o deliverables for the tasks to be performed, including statements of the area of exploration, experiments to be performed, and objectives of the research; o resource limitations, including the name of the principal investigator, time to be charged to the effort and other constraints that may have effected the results of the effort; and, o specific reporting requirements, including papers to be submitted for publication, and progress reports for program managers detailing developments in the research. Expectations of this type could have been used to evaluate whether contractors performed the tasks specified, remained within resource limits, met milestone schedules, and reported accomplishments in a timely manner. However, except for the Office of Fusion Energy, Energy Research program offices generally did not utilize the contractors' proposals as a basis for developing specific performance criteria and metrics at the task or aggregated program level. The audit disclosed that the work authorizations for 237 of 264 research tasks selected for review did not contain clear statements of work, resource limitations, milestone schedules, or specific deliverables. The Office of Fusion Energy used the contractors' proposals to develop performance criteria and metrics that provided a clear statement of the work to be performed, resource restrictions, milestone schedules, and specific deliverables. In Fiscal Year 1993, for example, Lawrence Livermore National Laboratory (Livermore) submitted 18 proposals for research tasks in fusion energy, of which 9 were selected for funding. In authorizing each of the 9 tasks, the fusion energy office identified the funding level, set the staffing level, included a milestone schedule, and specified the deliverable. Appendix B, for example, illustrates the inclusion of performance expectations in a task to perform theoretical research at Livermore. In contrast to the Office of Fusion Energy, the other Energy Research program offices did not use contractor proposals to develop performance criteria and metrics for inclusion in work authorizations at the task or aggregated program level. The respective work authorizations provided to Argonne National Laboratory (Argonne) and Ames Laboratory (Ames), by the Office of Basic Energy Sciences (Basic Energy Sciences), did not include performance criteria and metrics. For example, they did not identify funds for specific work proposals, set resource limits, establish milestones, or specify deliverables. However, both Argonne and Ames proposals contained information that Energy Research could have used as performance criteria and metrics in its work authorizations or the program guidance. o Argonne submitted 80 research proposals, with a total cost of $145 million, all of which contained information such as the proposed work to be performed, use of financial and human resources, milestones, progress reporting, and expected deliverables that could have been used to establish performance criteria and metrics. Rather than use Argonne's proposals to identify tasks and establish criteria and metrics, Basic Energy Sciences issued work authorizations totaling $103 million using broad statements of work which did not clearly specify which of the 80 proposals were funded. Since specific funds were not linked to specific proposals to be funded, we could not evaluate whether the contractor performed within the proposed limits of funding and staffing, met milestone schedules, or provided the expected deliverables. o Ames submitted 23 proposals for material sciences research, with a total cost of $12.2 million, all of which contained information in sufficient detail to establish performance expectations. Subsequently, Basic Energy Sciences authorized $9.6 million for research using broad statements of work which did not clearly specify which of the 23 proposals were funded. Since the proposals were not directly identified in the statements of work, specific resource limitations could not be defined, milestones could not be established and deliverables could not be determined. As in the Argonne example, we could not evaluate the contractor's performance because the work authorizations did not include performance criteria and metrics. The lack of performance criteria and metrics in the work authorizations provided to Argonne and Ames made it impossible for us to evaluate their performance of specific tasks or programs within defined constraints. Of importance, however, was that Energy Research could not illustrate that these contractors were evaluated against performance criteria and metrics in the work authorizations. Energy Research's work authorizations simply did not contain performance criteria and metrics that could be used to hold these contractors accountable for performing specific tasks or research areas within defined parameters. In work authorizations issued to Oak Ridge National Laboratory (Oak Ridge), the Office of Health and Environmental Research also did not identify performance criteria and metrics. The work authorizations, for example, did not specify funding levels for specific proposals, define resource limitations, milestones, or deliverables even though such information was available in the contractor's proposals. Oak Ridge, for example, submitted 91 research proposals, estimated to cost $31 million, for biological and environmental research. Each of these proposals included proposed financial and human resources, a statement of the work to be performed, milestones, reporting schedules, and deliverables. Although information was available in the contractor's research proposals, the Office of Health and Environmental Research authorized $25.6 million for Fiscal Year 1992 without specifying the funding level for each proposal, resource limitations, milestones or deliverables. Even in cases where funds could be identified for specific proposals, program offices did not establish performance criteria and metrics such as financial and human resource limits, milestones or deliverables. The Office of High Energy and Nuclear Physics, for example, did not include performance criteria and metrics when it authorized $1.3 million for Brookhaven National Laboratory (Brookhaven). Brookhaven's proposal provided information concerning staffing, approaches to developing specific detectors and instruments, and a variety of other information that was not used to establish performance criteria and metrics. The Office of High Energy and Nuclear Physics authorized research with a statement of work that read "funds are provided for research and development related to detectors and instrumentation used in the high energy research program." The work authorization did not set financial or human resource restrictions, establish milestones, or specify deliverables. These examples illustrate the lack of performance criteria and metrics in documents used to authorize contractors to perform research at Departmental laboratories. The lack of performance criteria and metrics make it extremely difficult, if not impossible, to objectively evaluate the contractor's progress in furthering the Department's mission. REASON FOR NOT ESTABLISHING PERFORMANCE EXPECTATIONS Energy Research allows contractors broad latitude to propose initiatives for the Department's research agenda that is of current interest to the scientific community. Energy Research envisions its role as one of providing funds to Departmental laboratories so that contractors can pursue these initiatives for the advancement of science. Accordingly, under the Department's institutional planning process, which is the major oversight mechanism for these laboratories, Energy Research has relied on the contractors to (1) define their missions within broad research areas of which Energy Research has traditionally funded or shown an interest in funding, (2) develop budget estimates without Departmental technical evaluation to determine the reasonableness of the estimates, (3) expend funds within broad areas of research, (4) develop their own methods of marking and measuring performance, and (5) provide the results of peer reviews used to evaluate the quality of the research. Energy Research relies on the contractors to manage the technical aspects as well as the resources associated with the research activity. Program managers said that they informally communicate performance technical and resource guidance to the principal investigators and laboratory management. Therefore, they contended that the broad guidance used in their work authorizations is sufficient to hold contractors accountable for efficiently and effectively managing research activity. Also, Energy Research uses a system of internal and external technical peer reviews to determine the quality and progress of the contractors' research. EFFECT OF NOT ESTABLISHING PERFORMANCE EXPECTATIONS This informal system does not include documented performance criteria and metrics that Departmental elements responsible for performance-based contract management can use to determine whether schedules were met, resources were properly used, deliverables were as specified, and the research performed was within the proper mission. Performance criteria and metrics also assist external reviewers in evaluating contractor management of research activity as well as provide a basis for stakeholders to align objectives. The need for measurable performance criteria was made clear in our review of the conclusions reached by a scientific peer review panel. The scientific review panel gave high marks for the research performed. However, the panel recommended that the following be provided in writing as well as at the oral presentation. o A clearly stated overall goal as well as the specific aim of the research for the project period. o A summary statement of accomplishments in relation to the principal investigator's perception of the goal. o The funding level should be made clear and should include a summary budget specifying the number of individuals supported, the travel, the supplies, and equipment allocations. The personnel funded by the project should be identified and their academic level specified. The inclusion of this type of information in work authorizations would improve both internal and external stakeholders ability to evaluate the program as well as the business management aspect of contractor research activity. The inclusion of performance criteria and metrics could serve to align the research objectives between the Department and its contractors. Performance expectations are of greater importance considering the current environment of constrained Federal resources. Contractor performance judged in relation to established expectations defined by the Department should be used to make decisions concerning future budgetary allocations. PART III SUMMARY OF MANAGEMENT COMMENTS AND OFFICE OF INSPECTOR GENERAL REPLY The Office of Energy Research provided comments on the official draft of this report and agreed in part with the report's recommendations. In response to the report, Energy Research will convene a process improvement team to: (1) examine the format of field research proposals to see whether changes in requested information and aggregation levels would improve their use as a research management tool; (2) consider modifications to research authorizations to better indicate Energy Research's expectations and performance criteria with respect to modified field research proposals; and (3) develop a phased implementation plan to incorporate the recommendations of the process improvement team into the FY 1998 and FY 1999 budget cycles. This action is expected to be completed by July 1, 1996. However, in regard to the analysis used to support the finding, Energy Research: o Agreed in part with our finding (conclusion) that a failure to include specific resource limits, milestones, and deliverables in work authorizations precludes the establishment of documented criteria which can be used to determine whether contractors meet the objectives of the Department; o Is concerned that unnecessary requirements on its research performers could reduce desirable flexibility and stifle creativity and lead to inferior results; and o Uses a variety of techniques, some informal and not all of which were examined by the Inspector General, to ensure that it gives appropriate technical direction to laboratory researchers. The full text of Energy Research's response is included as Appendix C to this report. Auditor Reply We recognize the inherent unpredictability of scientific research and the impossibility of specifying outcomes, particularly basic research. Even in a research environment, however, performance expectations are essential to properly manage. Milestones, together with other basic planning information, such as scope of work, resource commitments, and deliverables, are examples of information essential to understand, evaluate, and correct contractor performance. The level and specificity of items used to establish performance expectations should be appropriate to the type of research. Our finding was that Energy Research had not established such expectations at any level--individual tasks or overall aggregated programs. We are not recommending a level at which Energy Research should manage its programs. Our recommendations for including performance criteria and metrics, and evaluating progress based on those metrics, are valid at whichever level Energy Research chooses to manage its research. In addition to the need for performance expectations, Energy Research's administrative support processes are inconsistent with its management decision making. The current administrative process gives the appearance of decision making at an individual task level as it requires the proposing, funding, and accounting for research at the individual task level. However, as stated by officials in Energy Research, and confirmed by our audit, management decisions are made at an aggregated program level rather than at the individual task level. Our analyses were based on the administrative processes currently employed by Energy Research. We agree that Energy Research's process improvement team should examine these processes and modify or eliminate them as appropriate. Appendix A Energy Research Funds Provided to the Management and Operating Contractors (In Millions) FY 1992 FY 1993 FY 1994 Ames Laboratory $ 20.3 $ 21.8 $ 20.9 Argonne National Laboratory 112.8 129.5 150.0 Brookhaven National Laboratory 186.9 183.4 179.2 Continuous Electron Beam Accl. Fac. 23.4 28.3 43.8 Fermi National Accl. Laboratory 164.6 166.7 167.6 Lawrence Berkeley Laboratory 131.8 129.0 129.8 Lawrence Livermore National Lab. 88.9 83.7 85.5 Los Alamos National Laboratory 111.0 65.2 90.8 Oak Ridge National Laboratory 174.4 176.6 165.3 Pacific Northwest Laboratory 49.8 50.0 69.3 Princeton Plasma Physics Lab. 109.2 98.1 101.5 Sandia National Laboratory 28.9 28.8 28.6 Stanford Linear Accl. Center 120.4 131.4 120.2 Superconducting Supercollider 91.6 105.5 * Other Laboratories 31.8 33.1 34.1 Total $1,445.8 $1,431.1 $1,386.6 MMMMMMMM MMMMMMMM MMMMMMMM * $610 million for closing the facility is not included in the $1,386.6 billion total. Appendix B is a separate document. To obtain a copy of this appendix, please call Wilma Slaughter at (202) 586-1924. Appendix C OFFICE OF ENERGY RESEARCH RESPONSE TO INSPECTOR GENERAL OFFICIAL DRAFT REPORT "AUDIT OF PROGRAM ADMINISTRATION BY THE OFFICE OF ENERGY RESEARCH" SUMMARY The Inspector General report recommends that the Director of Energy Research review its administrative process and make appropriate changes. Specifically, consider (1) authorizing work based on requests received, and (2) evaluating research progress based on the metrics in these authorizations. Energy Research's response to the report findings and recommendations is provided below. I. Findings Energy Research recognizes the requirement for appropriate performance measures to support the evaluation of contractor performance, and has an effort under the Contract Reform Team to establish such performance measures and integrate them into contracts as part of the contract reform process. The Office of Energy Research disagrees with the Inspector General's finding that a failure to include specific resource limits, milestones and deliverables in work authorizations precludes the establishment of documented criteria which can be used to determine whether contractors met the objectives of the Department. Energy Research is concerned that unnecessary requirements on its research performers could reduce desirable flexibility and stifle creativity and lead to inferior results. Energy Research uses a variety of techniques, some informal and not all of which were exemined by the Inspector General, to ensure that it gives appropriate technical direction to laboratory researchers. II. Recommendations 1. Authorize work based on requests received. Response: Energy Research agrees in part. a. Energy Research accepts the value of "closing the loop" by specifically relating authorizations to field research proposals. However, we believe that it is important not to limit flexibility of researchers to follow up on unexpected results. We agree with the Inspector General's statement on page 5 that "...defining the specific outcomes of these complex research activities under study may be impossible to specify." In response to this recommendation, Energy Research will convene a process improvement team to: (1) examine the format of field research proposals to see whether changes in requested information and aggregation levels would improve their use as a research management tool; (2) consider modifications to research authorizations to better indicate Energy Research's expectations and performance criteria with respect to modified field research proposals; and (3) develop a phased implementation plan to incorporate the recommendations of the process improvement team into the FY 1998 and FY 1999 budget cycles. This action is expected to be completed by July 1, 1996. b. Energy Research does not agree that performance criteria and metrics contained in individual research proposals should be utilized in work authorizations at the individual task level. In 1993, the Office of Energy Research began working with stakeholders to develop a set of performance criteria for measuring R&D performance. These criteria have been accepted by the Contract Reform Team and are the basis for the scientific measures in the model contract developed for Energy Research laboratories. The criteria and performance measures developed by Energy Research have been incorporated into the contract signed with the University of Chicago for the operation of Argonne National Laboratory; the contract for the operation of Brookhaven National Laboratory, which will be signed shortly, includes them as well. As laboratory contracts come up for renegotiation, each will include similar criteria for measuring R&D performance. 2. Evaluate research progress based on the metrics in these authorizations. Response: Energy Research agrees in part. a. As stated in our response to recommendation 1, Energy Research wishes to point out that research progress should not be measured only against performance criteria and metrics at the individual task level. The basic process by which Energy Research ensures that research programs are of high quality and meet Departmental objectives is by convening reviews by outside experts using established performance criteria. These peer reviews provide a written evaluation, which includes some scoring of tasks. The reviews are used by program managers to adjust, redirect or stop tasks, as appropriate. This process is recognized as the best practice in the field of R&D management and is also used by other government agencies, such as the National Science Foundation and the National Institutes of Health. b. Energy Research intends to incorporate the performance criteria discussed under 1.b. above, in its guidance to reviewers. This will be implemented as part of the process improvement team activities referred to in 1.a. above. IG Report No. DOE/IG-0376 CUSTOMER RESPONSE FORM The Office of Inspector General has a continuing interest in improving the usefulness of its products. We wish to make our reports as respon- sive as possible to our customers' requirements, and therefore ask that you consider sharing your thoughts with us. On the back of this form, you may suggest improvements to enhance the effectiveness of future reports. Please include answers to the following questions if they are applicable to you: 1. What additional background information about the selection, scheduling, scope, or procedures of the audit or inspection would have been helpful to the reader in understanding this report? 2. What additional information related to findings and recommenda- tions could have been included in this report to assist management in implementing corrective actions? 3. What format, stylistic, or organizational changes might have made this report's overall message more clear to the reader? 4. What additional actions could the Office of Inspector General have taken on the issues discussed in this report which would have been helpful? Please include your name and telephone number so that we may contact you should we have any questions about your comments. Name Date Telephone Organization When you have completed this form, you may telefax it to the Office of Inspector General at (202) 586D0948, or you may mail it to: Office of Inspector General (IG-1) Department of Energy Washington, D.C. 20585 ATTN: Customer Relations If you wish to discuss this report or your comments with a staff member of the Office of Inspector General, please contact Wilma Slaughter at (202) 586D1924.