中英文資料外文翻譯文獻(xiàn) 基于價(jià)值的軟件測(cè)試管理 魯?shù)婪?斯蒂芬,保羅 沈青松譯 摘要: 根據(jù)研究表明測(cè)試已經(jīng)成為軟件開(kāi)發(fā)過(guò)程中一個(gè)很重要的環(huán)節(jié)，它占據(jù)了整個(gè)軟件開(kāi)發(fā)成本的百分之三十到五十。測(cè)試通常不是用來(lái)組織商業(yè)價(jià)值的最大化，也不是肩負(fù)著項(xiàng)目的使命。路徑測(cè)試、分支測(cè)試、指導(dǎo)測(cè)試、變換測(cè)試、場(chǎng)景測(cè)試以及需求測(cè)試等對(duì)于軟件的所有方面都是同等重要的。然而在實(shí)踐中百分之八十的價(jià)值往往來(lái)自百分之二十的軟件。為了從軟件測(cè)試中得到最大的投資回報(bào)，測(cè)試管理需要最大化它的價(jià)值貢獻(xiàn)。在本章，我們將更加促進(jìn)對(duì)基于價(jià)值的測(cè)試的需要，描述支持基于價(jià)值的測(cè)試管理的實(shí)踐，勾畫(huà)出基于價(jià)值的測(cè)試管理的框架，并舉例說(shuō)明該框架。 關(guān)鍵詞: 基于價(jià)值的軟件測(cè)試,基于價(jià)值的測(cè)試,測(cè)試成本,測(cè)試?yán)?測(cè)試管理 11.1 前言 測(cè)試是軟件質(zhì)量保證過(guò)程中最重要和最廣泛使用的方法。校驗(yàn)和驗(yàn)證旨在通過(guò)綜合分析, 測(cè)試軟件確保其正確運(yùn)行功能，確保軟件的質(zhì)量和軟件的可靠性。在IEEE610.12(1990)中,測(cè)試被定義為在規(guī)定條件下對(duì)執(zhí)行的系統(tǒng)或者組件進(jìn)行觀察和記錄,并對(duì)系統(tǒng)或者組件進(jìn)行評(píng)價(jià)的活動(dòng)。 測(cè)試在實(shí)踐過(guò)程中被廣泛的使用，在保證質(zhì)量策略的諸多組織中扮演著重要的角色。軟件影響著成千上萬(wàn)人的日常生活，擔(dān)負(fù)著艱巨的任務(wù)。因此軟件在不久的將來(lái)將顯得尤其的重要。研究表明，測(cè)試通常消耗軟件開(kāi)發(fā)成本的30%至50%。對(duì)于安全危急系統(tǒng)，甚至更高的比例也不足為奇。因此軟件測(cè)試具有挑戰(zhàn)的就是尋找更多的有效途徑進(jìn)行有效的測(cè)試。 軟件測(cè)試管理的價(jià)值在于努力減少測(cè)試成本和滿足需求。有價(jià)值的測(cè)試管理對(duì)于項(xiàng)目目標(biāo)和商業(yè)價(jià)值也能有很好的向?qū)?。在第一章，Boehm 列舉了很多方面的潛在測(cè)試成本。該例子說(shuō)明了利用客戶結(jié)賬類型的7%的成本來(lái)完成50%的軟件測(cè)試?yán)妗１M管百分百測(cè)試是一個(gè)不太切實(shí)際的目標(biāo), 然而通過(guò)調(diào)整測(cè)試方法, 仍有很大的空間來(lái)改進(jìn)和節(jié)省達(dá)到預(yù)期的價(jià)值?；谲浖こ痰膬r(jià)值動(dòng)力在于目前軟件工程的實(shí)踐研究都是把需求, 測(cè)試案例, 測(cè)試對(duì)象和產(chǎn)品缺陷看的同等重要。這對(duì)測(cè)試顯然是正確的，它對(duì)產(chǎn)品價(jià)值間接的做出貢獻(xiàn)。開(kāi)發(fā)和測(cè)試的分離使得這問(wèn)題顯得更為突出。測(cè)試往往是一個(gè)純粹的技術(shù)問(wèn)題，將使得測(cè)試和商業(yè)決策之間曾解鏈的關(guān)系變的更加緊密。本章主要描述了提高基于價(jià)值的測(cè)試管理的需求, 解釋其基本要素, 討論現(xiàn)有的實(shí)例來(lái)支持基于價(jià)值的測(cè)試, 以及基于價(jià)值的測(cè)試管理的基本框架。本章接下來(lái)部分的基本結(jié)構(gòu)如下：11.2節(jié)討論測(cè)試管理的貢獻(xiàn)價(jià)值； 11.3節(jié)討論已測(cè)案例對(duì)于測(cè)試管理的支持； 11.4節(jié)用例子描述基于價(jià)值的測(cè)試管理的框架。本章最后將對(duì)具體的再作更進(jìn)一步的研究。 11.2 基于價(jià)值測(cè)試的描述 基于價(jià)值的校驗(yàn)和驗(yàn)證的目標(biāo)被定義為確保軟件的實(shí)施能夠滿足其預(yù)期的目標(biāo)價(jià)值利益。如果我們從價(jià)值的角度去考慮，那么測(cè)試的貢獻(xiàn)應(yīng)該在哪呢？從根本上來(lái)說(shuō), 可以從兩個(gè)方面來(lái)考慮: 內(nèi)部方面包括測(cè)試成本以及測(cè)試效益, 外部方面強(qiáng)調(diào)未來(lái)系統(tǒng)的威脅以及機(jī)遇。對(duì)于基于價(jià)值的測(cè)試，其關(guān)鍵就是要把這兩個(gè)方面結(jié)合起來(lái),也就是說(shuō)通過(guò)客戶和市場(chǎng)需求調(diào)整內(nèi)部測(cè)試的過(guò)程。 為了使內(nèi)部和外部?jī)煞矫嬉蛩叵嘟Y(jié)合，只專注于技術(shù)方面的測(cè)試顯然是不恰當(dāng)?shù)?。相反?測(cè)試管理需要有個(gè)全局的把握。 例41描述了測(cè)試管理內(nèi)外雙方面之間的依賴關(guān)系。內(nèi)部方面的就如同測(cè)試經(jīng)理對(duì)整個(gè)項(xiàng)目的控制。這方面的費(fèi)用主要來(lái)自于軟件測(cè)試實(shí)踐以及短期長(zhǎng)期的測(cè)試。而外部方面被認(rèn)為是測(cè)試主管所能控制以外的一些利益和參數(shù)數(shù)據(jù).基于價(jià)值的軟件測(cè)試管理使得測(cè)試能夠滿足以利益為重的價(jià)值主張，以及使整個(gè)小組聚焦在有價(jià)值的測(cè)試方向上。 對(duì)于軟件測(cè)試的外部觀點(diǎn)的首要問(wèn)題是“我們?nèi)绾未_保軟件系統(tǒng)的價(jià)值目標(biāo)？” 這目標(biāo)就是通過(guò)協(xié)調(diào)價(jià)值主張來(lái)測(cè)試軟件集中的有價(jià)值部分，最重要的品質(zhì)以及項(xiàng)目風(fēng)險(xiǎn)的及時(shí)調(diào)整等.回答這樣的問(wèn)題包括市場(chǎng)機(jī)遇，項(xiàng)目的價(jià)值主張以及成本效益。參考第一章關(guān)于機(jī)遇和風(fēng)險(xiǎn)的詳細(xì)介紹以及參考第七章的價(jià)值主張的引出與調(diào)和。 內(nèi)部觀點(diǎn)是建立在價(jià)值主張的利益之上，以及測(cè)試的預(yù)算代表著整個(gè)項(xiàng)目的一個(gè)大概水平。這主要的問(wèn)題就是如何把測(cè)試作為一項(xiàng)投資活動(dòng)。為了能夠高效迅速的測(cè)試及降低開(kāi)發(fā)預(yù)算。適當(dāng)?shù)膬?nèi)外部交流協(xié)調(diào)能夠滿足測(cè)試的利益價(jià)值。 測(cè)試的價(jià)值貢獻(xiàn) 測(cè)試與其他的開(kāi)發(fā)環(huán)節(jié)諸如代碼和用戶界面設(shè)計(jì)相比，它不能立即對(duì)產(chǎn)品創(chuàng)造價(jià)值。然而測(cè)試提供和支持軟件開(kāi)發(fā)過(guò)程中產(chǎn)生的有價(jià)值的任務(wù)活動(dòng)。理解測(cè)試貢獻(xiàn)價(jià)值的關(guān)鍵點(diǎn)在于測(cè)試的貢獻(xiàn)效應(yīng)。測(cè)試的貢獻(xiàn)建立了測(cè)試與最終產(chǎn)品價(jià)值利益之間的關(guān)系。 最直接的客戶是直接與測(cè)試小組有密切聯(lián)系的程序開(kāi)發(fā)者和項(xiàng)目主管。在基于價(jià)值的軟件工程測(cè)試過(guò)程中的集中力量是顧客和用戶（見(jiàn)第七章）。顧客和用戶通過(guò)設(shè)定語(yǔ)境和范圍來(lái)進(jìn)行測(cè)試達(dá)到測(cè)試的價(jià)值目標(biāo)。 測(cè)試的客戶 開(kāi)發(fā)人員,項(xiàng)目主管,質(zhì)量主管,顧客,分析者,最終用戶或者維修人員們都得益于軟件系統(tǒng)的分析,依靠反饋來(lái)檢測(cè)問(wèn)題,降低其不確定性,做出相關(guān)的決定來(lái)加快產(chǎn)品進(jìn)程。 下面的例子顯示了不同組對(duì)于測(cè)試需求的反饋信息： · 顧客和用戶關(guān)于多大程度上需求一致是否滿意以及在多大程度上滿足軟件的價(jià)值利益.測(cè)試還對(duì)項(xiàng)目的進(jìn)程提供可見(jiàn)性和洞察力.通過(guò)測(cè)試的結(jié)果可以了解已通過(guò)的測(cè)試案例.當(dāng)驗(yàn)收測(cè)試時(shí)出現(xiàn)不實(shí)用的或者失敗的顯示在實(shí)際環(huán)境中才能出現(xiàn)的問(wèn)題，α和β測(cè)試提供了一個(gè)更加堅(jiān)實(shí)的基礎(chǔ)來(lái)驗(yàn)證結(jié)果。 · 銷售和產(chǎn)品主管從測(cè)試計(jì)劃、定價(jià)、促銷和分配方面獲取相關(guān)的信息。產(chǎn)品實(shí)際的質(zhì)量與顧客和用戶所期望的質(zhì)量之間的差異很容易會(huì)導(dǎo)致誤解和錯(cuò)誤的設(shè)想以至于降低或者阻止了真正價(jià)值的實(shí)現(xiàn)。為了能夠成功地達(dá)到這些期望以及滿足個(gè)人或組織的目標(biāo)，通過(guò)客戶需求來(lái)調(diào)整產(chǎn)品設(shè)計(jì)滿足某些功能。 · 對(duì)于項(xiàng)目主管，測(cè)試支持了風(fēng)險(xiǎn)管理和項(xiàng)目進(jìn)程的估計(jì)。重點(diǎn)是識(shí)別和排除潛在的價(jià)值破壞和抑制價(jià)值功績(jī)的風(fēng)險(xiǎn)。早期的大幅降低項(xiàng)目績(jī)效的嚴(yán)重缺陷,是一個(gè)主要的目的。測(cè)試降低了不確定性和幫助項(xiàng)目主管對(duì)于清除缺陷、系統(tǒng)穩(wěn)定性以及產(chǎn)品更新發(fā)布能夠作出更好、更明智的決定。 · 質(zhì)量主管對(duì)于問(wèn)題的識(shí)別以及對(duì)特定問(wèn)題的動(dòng)態(tài)趨勢(shì)較為感興趣。測(cè)試結(jié)果對(duì)于項(xiàng)目評(píng)估、對(duì)于質(zhì)量策略的保證以及進(jìn)程的改進(jìn)提供了幫助。Rosenberg討論了測(cè)試如何對(duì)確保質(zhì)量作出貢獻(xiàn)并展示了測(cè)試問(wèn)題如何驗(yàn)證如何修正以此來(lái)提升項(xiàng)目進(jìn)程。開(kāi)發(fā)者和用戶了解相關(guān)問(wèn)題的當(dāng)前狀態(tài)，并且提供相關(guān)數(shù)據(jù)來(lái)衡量及預(yù)測(cè)軟件的質(zhì)量和可靠性。 · 開(kāi)發(fā)人員通常需要獲取反饋信息來(lái)驗(yàn)證測(cè)試實(shí)施是否完整，是否符合標(biāo)準(zhǔn)，是否滿足質(zhì)量要求。為了保證穩(wěn)定性，測(cè)試提供了相關(guān)缺陷的詳細(xì)信息，提示測(cè)試失敗的原因。除此之外，測(cè)試對(duì)于項(xiàng)目缺陷的改進(jìn)作出反饋。例如，通過(guò)相關(guān)的修改以后需要測(cè)試其是否隨著相關(guān)的改動(dòng)使原先的功能有所改動(dòng)或者出現(xiàn)衰退的情況，這些都是需要注意的。 · 對(duì)于需求工程師來(lái)說(shuō)，測(cè)試對(duì)于驗(yàn)證和確認(rèn)需求是很有價(jià)值的。Weinberg曾指出“最有效的方法之一就是通過(guò)測(cè)試案例就象測(cè)試一個(gè)完整的系統(tǒng)來(lái)形成測(cè)試需求”。黑盒測(cè)試通過(guò)其需求幫助能夠保證它們的完整性，準(zhǔn)確性，透明性及簡(jiǎn)明性等。因此測(cè)試能夠提升要求且向著測(cè)試驅(qū)動(dòng)的方向發(fā)展。 簡(jiǎn)言之，測(cè)試能夠通過(guò)降低計(jì)劃的不確定性和風(fēng)險(xiǎn)性來(lái)提升利益，作出相關(guān)的決定，努力控制把不必要的消耗減少到最低程度(內(nèi)部原因)。尤為重要的是，它有助于實(shí)現(xiàn)預(yù)期的價(jià)值利益。這些利益的得來(lái)也并不是免費(fèi)的，測(cè)試的代價(jià)通常也是有意義的(外部原因).測(cè)試可以被理解為購(gòu)買信息，也可被認(rèn)為降低成本風(fēng)險(xiǎn)，減少不確定性的投資活動(dòng)。在成本和利益的投資上需要對(duì)測(cè)試需求作出相關(guān)的決定。因此接下來(lái)的兩個(gè)問(wèn)題是：什么是測(cè)試成本？什么是價(jià)值活動(dòng)的測(cè)試?yán)妫? Value-Based Management of Software Testing Rudolf Ramler, Stefan Biffl and Paul Grünbacher Abstract: Testing is one of the most resource-intensive activities in software development and consumes between 30 and 50% of total development costs according to many studies. Testing is however often not organized to maximize business value and not aligned with a project’s mission. Path, branch, instruction, mutation, scenario, or requirement testing usually treat all aspects of software as equally important, while in practice 80% of the value often comes from 20% of the software. In order to maximize the return of investment gained from software testing, the management of testing needs to maximize its value contribution. In this chapter we motivate the need for value-based testing, describe practices supporting the management of value-based testing, outline a framework for value-based test management, and illustrate the framework with an example. Keywords: Value-based software engineering, value-based testing, cost of testing, benefits of testing, test management. 11.1 Introduction Testing is one of the most important and most widely used approaches for validation and verification (V&V). V&V aims at comprehensively analyzing and testing software to determine that it performs the intended functions correctly, to ensure that it performs no unintended functions, and to measure its quality and reliability (Wallace and Fujii, 1989). According to IEEE 610.12 (1990) testing is defined as “an activity in which a system or component is executed under specified conditions, the results are observed or recorded, and an evaluation is made of some aspect of the system or component.” Testing is widely used in practice and plays a central role in the quality assurance strategies of many organizations. As software pervades more and more critical tasks and affects everyday life, security, and well being of millions of people (Ferscha and Mattern, 2004), the importance of testing will increase in the future. Studies show that testing already consumes between 30 and 50% of software development costs (Beizer, 1990). Even higher percentages are not uncommon for safety-critical systems. Finding more efficient ways to perform effective testing is therefore a key challenge in testing (Harrold, 2000). Managing software testing based on value considerations promises to tackle increasing testing costs and required effort. Value-based test management could also provide guidance to better align testing investments with project objectives and business value. In Chapter 1, Boehm presents an impressive example of potential test cost savings (on project level as well as on global scale) by focusing testing on the most valuable aspects. The example illustrates that with an investment-oriented focus on testing 7% of the customer billing types (1 in 15) achieved 50% of the benefits of testing the software. Completely testing the system requires a constantly increasing effort and, due to decreasing marginal benefits, results in a negative return on investment. Although a “100% tested” status is not a practical goal, there is still room for a considerable amount of improvement and savings by better adjusting testing to its value contribution. The motivation for value-based software engineering comes from the fact that “much of current software engineering practice and research is done in a value neutral setting, in which every requirement, use case, object, and defect is treated as equally important” (Boehm, 2003). This is especially true for testing, where its indirect contribution to product value leads to a value-neutral perception of testing. The common separation of concerns between development and testing exacerbates the problem. Testing is often reduced to a purely technical issue leaving the close relationship between testing and business decisions unlinked and the potential value contribution of testing unexploited. The objectives of this chapter are to motivate the need for value-based management of testing, to explain its underlying elements, to discuss existing practices that support value-based testing, and to outline a general framework for value-based test management. The remainder of this chapter is thus structured as follows. In Section 11.2 we discuss test management under the light of its value contribution. In Section 11.3 we describe existing practices that support value-based testing. Section 11.4 depicts a value-based test management framework using an example for illustration. An outlook on further research directions closes the chapter. 11.2 Taking a Value-Based Perspective on Testing The objectives of value-based verification and validation are defined as “ensuring that a software solution satisfies its value objectives” and organizing V&V tasks to operate as an investment activity” (Boehm and Huang, 2003). What are the contributions of testing if we look at it from a value-based perspective? Fundamentally, we can consider two dimensions: The internal dimension of testing covers costs and benefits of testing. The external dimension emphasizes the opportunities and risks of the future system that have to be addressed. The key challenge in value-based testing is to integrate these two dimensions, i.e., align the internal test process with the value objectives coming from the customers and the market. It becomes clear that a pure focus on the technical aspects of testing (e.g., the testing methods and tools) is inappropriate to align the internal and external dimensions. Instead, test management activities need to adopt a value-based perspective. Figure 41 illustrates the external and internal dimensions of test management and their interdependencies. The internal dimension is similar to the scope of control of the test manager in the project. This dimension addresses costs from software testing practice as well as short-term and long-term benefits of testing. The external dimension considers stakeholders and parameters outside the scope of control of the test manager. Value-based test management organizes testing to satisfy value propositions of the stakeholders and to focus the team on the most worthwhile testing targets. The key question coming from the external view of software testing is: “How can we ensure the value objectives of the software system?” The goal is to reconcile stakeholder value propositions by focusing testing efforts on the most worthwhile parts of the software, the most important quality characteristics, and the most urgent symptoms of risks that threaten the value contribution of the project. Answering this question involves market opportunities and threats, project-specific customer value propositions, as well as costs and benefits. Please refer to Chapter 1 for details about opportunities and risks and to Chapter 7 for elicitation and reconciliation of stakeholder value propositions. The internal view builds on the stakeholder value propositions and the test budget that represents the possible level of testing effort in a project. The key question in this view is: “How can we organize testing as an investment activity?”The goal is to achieve effective and efficient testing considering changes in development and budget reductions. Internal project stakeholders consider how plans for software development and associated testing activities can contribute to stakeholder value propositions by supplying system functionality and performance, but also by limiting the impact of project-relevant risks. Appropriate communication is necessary to balance the external and internal dimensions of testing to assure the consistency of testing objectives with stakeholder value propositions. Value Contribution of Testing Compared to other development activities such as coding or user interface design, testing does not create immediate product value. Instead, testing informs and supports other value generating tasks in software development. A key to understanding the value contribution of testing is the contribution chain of testing (see the benefits realization approach described in Chapter 1). The contribution chain establishes the relation of testing to the final product that ultimately creates value for the stakeholders. Usually, the contribution chain of testing is complex and involves several different “clients,” who benefit from testing. Direct clients of testing are developers and project managers, who directly interact with the testing team (representing the internal dimension). However, in the spirit of value-based software engineering important parties for testing are customers and users (representing the external view). Customers and users are the source of value objectives (see Chapter 7), which set the context and scope of testing. Within this context testing informs developers and project managers to what extent value objectives are met and where improvement is required. Clients of Testing Developers, project managers, quality managers, customers, analysts, end users, or maintenance staff benefit from a thorough analysis of the software system and rely on feedback for detecting problems, reducing uncertainty, making decisions, or improving products and processes. The following examples show the kind of feedback from testing required by different groups: · Customers and users get information as to what extent mutually agreed requirements are satisfied and to what extent the software meets their value propositions. Testing also provides visibility and insights about project progress. Passed tests reduce the odds of misbehavior and acceptance decisions are thus frequently based on the results of tests. When acceptance tests are impractical or fail to reveal hidden problems that become visible only in real-world conditions, alpha and beta testing provide a more solid foundation for acceptance decisions. · Marketing and product managers require information from testing for planning releases, pricing, promotion, and distribution. A gap between the actual quality and the quality expected by customers and users most certainly leads to misleading expectations and wrong assumptions that diminish or prevent value realization (Boehm, 2000b). In order to successfully manage these expectations and to satisfy individual and organizational objectives, reconciling customer needs with product design has to consider quality in addition to functionality. · For project managers testing supports risk management and progress estimation. The focus is on identifying and eliminating risks that are potential value breakers and inhibit value achievements. Early detection of severe defects that significantly reduce project performance is a major objective. Ideally, testing reduces uncertainty and helps project managers to take better, more informed decisions, e.g., for defect removal, system stabilization, and release decisions. · Quality managers are interested in the identification of problems and in particular problem trends. Results from testing are the input for the assessment of development performance and provide the basis for quality assurance strategies and process improvement. Rosenberg (2003) discusses how testing contributes to quality assurance and shows that problems need to be documented, corrected, and can then be used for process improvement; after assessing problem reports for their validity corrective actions are implemented in accordance with customer-approved solutions; developers and users are informed about the problem status; and data for measuring and predicting software quality and reliability is provided. · Developers require feedback from testing to gain confidence that the implementation is complete and correct, conforming to standards, and satisfying quality requirements. For stabilization, testing provides details about defects and their estimated severity, information for reproducing defects, and support for revealing the cause of the failures. Besides, testing provides feedback for improvement and learning from defects. For example, throughout maintenance a detailed and reproducible description of problems contributes to the efficient implementation of changes and regression tests ensuring that these changes do not break existing functionality. · For requirements engineers, testing is valuable to validate and verify requirements. Gause and Weinberg (1989) point out that “… one of the most effective ways of testing requirements is with test cases very much like those for testing a complete system.” Deriving black-box tests from requirements helps to assure their completeness, accuracy, clarity, and conciseness early on. Tests thus enhance requirements and enable development in a test-driven manner. To summarize, testing helps to realize benefits by reducing planning uncertainty, mitigating risks, making more informed decisions, controlling efforts, and minimizing downstream costs (the internal dimension).More importantly, it helps to realize the expected stakeholder value propositions (the external dimension). These benefits, however, do not come for free and the costs of testing are often significant. Testing can be perceived as buying information and can be considered as an investment activity as it reduces the costs of risks, uncertainties, and the reward of taking risks. Making sound decisions about the investment in testing requires understanding their implications on both costs and benefits. The underlying questions therefore are: What are the costs of testing, and what are the benefits of testing for value generating activities? 11