TMMi history, benefit and future Session 3 Klaus Olsen, CEO of TMMi Foundation klaus.olsen@tmmi.org
1. TMMi Foundation History 2. Benefit of using TMMi Model 3. The future of TMMi Foundation
Klaus Olsen Founder and owner of the company Softwaretest.dk since 2000 Has used the past 23 years to focus on software testing, test process improvements, teaching and mentoring Author of Softwaretest how to get started in Danish Founding member of TMMi Foundation CEO of the TMMi Management Executives Member of ISTQB, representing Denmark Co-author of ISTQB Foundation and Advanced Syllabus Chair of ISTQB Foundation Working Group Certified ISTQB Foundation and Test Manager Advanced Certified Scrum Master
Public presentation includes EuroSTAR 98 in Münich, Germany Second World Congress on Software Quality 2000 in Yokohama, Japan EuroSTAR 2001 in Stockholm, Sweden Quality Week 2001 in San Francisco, USA EuroSTAR 2003 in Amsterdam, Holland ASTA 2007 in Seoul, Korea Test 2008 in New Delhi, India EuroSTAR 2008 in Haag, Holland ANZTB Test2009 conference Sydney, Australia JSTQB カンファレンス 2010, in Tokyo, Japan ASTA 2010 in Seoul, Korea Czechtest 2011 in Prague, Czech Republic TAPOST 2011 in Riga, Latvia TMMi World Conference 2011 in Seoul, Korea Czechtest 2012 in Prag, Czech Republic Nordic Testing Days 2012 in Tallinn, Estonia ANZTB Test2013 conference in Canberra, Australia FiSTB Testing Assembly 2013 in Helsinki, Finland Testing Portugal 2013 in Lisbon, Portugal DSTB 2014 in Copenhagen, Denmark Sixth World Congress on Software Quality 2014 in London, UK Teszt & Tea 2014 in Budapest, Hungary Czechtest 2015 in Prag, Czech Republic SEETEST 2015 Sofia, Bulgaria
Experience with TMM - TMMi First experience with TMM in 1997 and presentation about this at EuroSTAR 98 in Munich, Germany TMM is developed by Illinois Institute of Technology and published in two articles in U.S. Air Force magazine Crosstalk in 1996 Meeting with Ilene Burnstein, author of TMM, in USA in 2000 Ilene Burnstein has described her original TMM model in detail in her book Several assessment of companies in Denmark and Norway during the past 10 years Participating in creating the current TMMi model
Those who cannot remember the past are condemned to repeat it George Santayana 1863 in Spain 1952 in Italy. He was a philosopher, essayist, poet and novelist.
Maturity History Timeline Crosby 1979 CMM v1.1 1993 TPI 1997 CMMi 2000 TMMi full version 2010 CMM v1.0 1991 TMM 1996 TMM DK version 2000 TMMi 2005 TMMi Foundation established 2005 in London, UK
TMMi Foundation - Board of Directors Elected by the member of TMMi Foundation Chairman Vice-chair Treasure Les Murray Secretary Mac Miller CEO Fran O Hara Geoff Thompson Klaus Olsen 9
TMMi Foundation - Board of Advisor Appointed by the BoD Lev Lesokhin Bill Curtis Brian Ford Steve Connell Brian Wells 10
TMMi Foundation Management Executives Appointed by the BoD CEO Deputy CEO Accreditation Meile Posthuma Klaus Olsen Secretary Training Erik van Veenendaal TMMi Professional Geoff Thompson Marketing Marc Gebauer Andrew Goslin 11
TMMi Foundation is: 33 Certified companies, among them 5 from Korea TMMi Level 2: ITSTAR Co., LTD. (Development & Operation Division) Onycom Inc (QA, ICT Solution, IT Service Division) TMMi Level 3: LG CNS (DC Test and Performance Test Teams) Hanwha Thales Co., LTD (R&D Division) LIG Nex1 (Research & Development Division) 22 training providers, among them STA from Korea 15 supporting organisation 1.757 LinkedIn members 1.686 members of the TMMi Foundation 12
Test Maturity Model Integration, TMMi Level 5: Optimization Defect Prevention Test Process Optimization Quality Control Level 4: Measured Test Measurement Software Quality Evaluation Advanced Peer Reviews Level 3: Defined Test Organization Test Training Program Test Life Cycle and Integration Non-Functional Testing Peer Reviews Level 1: Initial Level 2: Managed Test Policy and Strategy Test Planning Test Monitoring and Control Test Design and Execution Test Environment 16 Process Areas TMMi maturity levels and process areas
TMMi Structure and components Maturity Levels Indicates Testing Process Capability contains Process Areas Achieves Goals contains Generic and Specific Goals contains Generic and Specific Practices Describes Implementation
TMMi Structure and components Maturity Level Process Area 1 Process Area 2 Process Area n Generic Goals Specific Goals Generic Practices Specific Practices
TMMi Structure and components Managed TMMi level 2 Test Policy and Strategy Test Planning Process Area.. Institutionalize a Managed Process Develop a test plan Specific Goal 4 Generic Goal 2 Generic Practice 2.2 Plan the process Establish the test plan Specific Practice 4.5
1. TMMi Foundation History 2. Benefit of using TMMi Model 3. The future of TMMi Foundation
TMMi Foundation benefit we provide TMMi model free available from our website in pdf TMMi Professional Certification TMMi Professional Syllabus from our website in pdf TMMi Professional trial exam questions from our website in pdf TAMAR, requirement to build an assessment method TAM, TMMi Foundation provide an assessment method on license 18
How you can utilise TMMi If you don t know where you are, a map won t help TMMi provides the framework for an assessment of your company s now situation Watts Humphrey If you don t know where you want to go, then any road will do TMMi provides a map of improvement possibilities from level 2 to level 5
How you can utilise TMMi 1. Start by using the TMMi model to locate your GPS position on the TMMi Model 2. Once you know where you are, use the TMMi model as a reference for where you want to go
How you can utilise TMMi Plan Act Do Study Study by measuring your companies test maturity level, through an assessment Act by identifying the weakest areas in your com pany when using the TMMi Model Plan by creating a test improvement plan based on the TMMi Model Do by implementing the identified task from Act and Plan above
Assessment how they are done a story Highlight areas in need of an improvement Go through proposal action plan Highlight good practices they must repeat Score result for each project for all process areas And more Test reports Test case examples Test Plan Test strategy Let the company decide what to do next Present result of assessment for the company Create proposal for an Action Plan based on result of assessment Create overall TMMi assessment report with result Evaluate all input from interviews and artefacts Get access to artefacts discussed in interviews Assessment Storytelling Engaging with a company who needs an assessment Selecting 3-5 projects in the company Select 3-4 people from each project for interview Often also 1 or 2 persons from a Qualtiy or Method department Run interview with all participants Should represent both large and small size projects Should represent the different development methods used Project Manger Test Manger Developer Tester X questions for each process area
96% defect detection rate reached 96% DDP reached in 2004 Better and more stable quality Moved from 78% DDP to 96% DDP during 3 years Effectiveness of test in company Test effectiveness = Numbe r of Defects found during t est / total number of defect s after 90 days Time for test % Effectiveness of test %
96% defect detection rate 96% DDP reached Better and more stable quality Moved from 78% to 96% DDP during 3 years Test effectiveness = Number of Defects found duri ng test / total number of defects after 90 days Year Test Effectiveness in Average Time for Test of total project 200x 78% DDP 23% 200x+1 94% DDP 28% 200x+2 96% DDP 25% 200x+3 95% DDP 28% 200x+4 92% DDP 19%
Metrics from a company using TMMi Improved higher level of quality in projects Now some with a 100% DDP others between 75% and 90% DDP, in average DDP were 92% Effectiveness in projects Time for test % Effectiveness of test %
1. TMMi Foundation History 2. Benefit of using TMMi Model 3. The future of TMMi Foundation
TMMi our future TMMi and Agile Alumni Club Free for all TMMi Professional Certified Able to vote on our AGM for the BoD Able to participate in our working groups 28
Currently TMMi Foundation are working with the following organizations to grow recognition of the TMMi Model on a global scale
TMMi Professional Training providers Belgium Bolivia China Denmark France Germany Hungary India India India India Ireland Ireland Korea Malaysia Netherlands Netherlands Scotland Spain UK UK UK USA 30
Make a Difference and use TMMi Clive Bates / clive.bates@experimentus.com Session 4
1. The challenges you face 2. Examples of client benefits 3. How to move forward with TMMi 4. TMMi trends 5. Don t fear failure embrace it
Experimentus services 3
Typical reasons for test improvement Many Organizations are still facing tough business objectives in the current climate e.g. Reduced time to market Increased functionality and complexity Increased reliability Increased speed of performance Increased number of users Reduced prices being paid (cheaper products/applications) Need to invest in and utilizing new technologies and compete on a global stage. Need to innovate or be left behind Re-design of the workplace including co-developed or outsourced development Meeting client requirements Therefore: It can be said that applications are now more challenging to build! 4
The result of all this is. Systems and applications are playing an ever increasing part in every day life (social and business) There is a greater emphasis to focus on quality aspects Customers and users are becoming more and more demanding Software failures are much more visible now and can have major impact on day to day activities economically, socially and internationally etc.. Testing is becoming a key activity for helping to improve quality, and the performance of the development process The fast pace of change in the industry consider what has happened over the last 5 or 10 years? What will we see next? 5
Whose having assessments & where? Service providers Mining company Pharmaceutical Banking Financial Services Retail Insurance Mobile Telephone Consulting, System Integrato rs, Outsourcing Utilities Software developers TV and Communications Etc UK India USA Malaysia Australia China Ukraine Spain France Mexico 6
Example benefits seen - summary Retain and/or gain contracts. Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business Releases now meeting scheduled dates Organisation have working processes in place for projects no longer a game of chess, consistent approach Management receiving relevant progress information through correct metrics Ferrari type processes not bicycle ones Conducted internal assessments over period of time and introduced relevant improvements achieved TMMi levels straight away 7
Example benefits seen - summary Able to involve product owners and identified risks were handled much better Able to motivate staff Found shortcomings in risk and processes despite having had an industry based audit that did not find problems Improved quality products, smoother releases We can help quantify savings through improvement 8
How to improve? Start simple do an online checkout first http://www.experimentus.com/customer-journey/assess/ Use the TMMi Framework as a guide Be realistic what target level you want to aim for Have an informal assessment to your target TMMi level use the framework against your organisation and type of business Identify the strengths and any gaps Hold a workshop to discuss the gaps Create a strategy defining what is to be improved and when and how they will be monitored 9
How to improve? Get yourself a champion/owner Communicate what you are doing and why Split improvements to quick wins and longer term goals Have access to a TMMi expert to help you Allocate the right amount of time to do this Gain commitment from individuals and ensure they are open to change Have an improvement plan and target metrics Use an improvement model like IDEAL ** COMMUNICATE PROGRESS ** 10
IDEAL model 11
Define/monitor improvements Priority Item # Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual Leader Estimated Effort Target (# Days) Completion Date Dependencies Training / Mentoring Requirements Status Need Additional Clarification? Comments / Questions 1 2 3 4 2.1 Test Policy & Strategy SG1 Establish a Test Policy SG2 Establish a Test Strategy SP 1.1 Define Test Goals Define and maintain test goals based upon business needs and objectives. SP 1.2 Define Test Policy A test policy, aligned with the business (quality) policy, is defined based on the test goals and agreed upon by the stakeholders. SP 1.3 Distribute Policy to Stakeholders The test policy and test goals are presented and explained to stakeholders inside and outside testing. SP 2.1 Perform a generic product risk assessment A generic product risk assessment is performed to identify the typical critical areas for testing. 12
TMMi Certification Trends 13
IT Service Provider Results IT Service Provider Sector Results 3.4 Non Functional Testing 3.5 Peer Reviews 2.1 Test Policy and Strategy 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 2.2 Test Planning 2.3 Test Monitoring and Control 2015 2014 3.3 Test Lifecyle and Integration 2.4 Test Design and Specification 3.2 Test Training program 2.5 Test Environments 3.1 Test Organisation 14
Level 2 Level 3 Results across all industries TMMi Level 2 & 3 Process Areas 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Test Organisation 54% 32% 14% Test Training Programme 62% 27% 12% Test Lifecycle and Integration 56% 34% 10% Non Functional Testing 40% 45% 15% Peer Reviews 45% 43% 12% Test Policy and Strategy 51% 36% 13% Test Planning 45% 43% 12% Test Monitoring and Control 49% 42% 9% Test Design and Execution 69% 25% 6% Test Environment 60% 33% 7% Fully Achieved Partially Achieved Not Achieved 15
Recognise and embrace the failures The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future See how failure sits alongside process improvement and how risk fits in all of this 16
Is this right?? 17
General views on Failure Comparing Aviation to Healthcare Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress Matthew Syed Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved When we fail we don t get a process improvement because the informat ion about errors, problems etc.. are misinterpreted or ignored. This we call a closed loop. An open loop does lead to progress because you act on feedback o If we fear failure we don t even try o If we don t try we don t take risks o If we don t take the risk then we don t fail and learn how to improve 18
Amy Edmondson - Harvard Business School Learning from failures is anything but straightforward. The attitudes and activities required to effectively detect and analyse failures are in short supply in most companies, and the need for context-specific learning strategies is underappreciated. Organisations need new and better ways to go beyond lessons that are superficial 19
Why do we fear failure? Failure has created a bad feeling from a young age from childhood we learn that admitting to failure can lead to blame, low marks, ridicule etc. so we've had 12 years or so of training to fear failure often as we grow up we don t shift to the rewards of learning from failure sometimes we fear failure even before we start - aim low you get low We can fear other people s view of us and this can lead to a lack of confidence We imagine others care as deeply about our own success and failure as we do personally We fear shame and our responses to this can manifest itself in many ways So overall failure can and has created negative connotations in many of us 20
How we should really see failure Organisations need to make it a safe place to admit and report failure with the right mind-set, the concern that accepting failures will lead to poor quality is not relevant Google X Astro Teller says You must reward people for failing. If not, they won't take risks and make breakthroughs. If you don't reward failure, people will hang on to doomed ideas for fear of the consequences. That wastes time and can damage an organisation's spirit. Harvard Business Review research developed a Spectrum of Reasons for Failure Recognise that no-one deliberately wants to fail! We learn from failure with the intention that we can then improve and learn 21
Spectrum of failure 22
How we should really see failure Opportunity to: Learn and grow from the experience Consider and have a healthy approach to risk Forget perfection fail well, take risks and learn from them. Helps you to find the gaps and take actions to fix Be resilient and keep trying e.g. practise at music to get better, ice skating, ski jumping etc.. Recognise that young people don t naturally fear failure so they can learn In a UK school they have a failure week to de-mystify the word. Failure happens! Then have a blow your trumpet week to recognise and celebrate success and achievements Be expert learners not just experts True success is not avoiding failure, but learning what to do with it 23
Wimbledon High School FAILURE WEEK A top girls' school is planning a "failure week" to teach pupils to embrace risk, b uild resilience and learn from their mistakes. The emphasis will be on the value of having a go, rather than playing it safe and perhaps achieving less. Pupils at Wimbledon High School will be asked how they feel when they fail. The headmistress, Heather Hanbury, said she wanted to show "it is completely acceptable and completely normal not to succeed at times in life. Ms Hanbury' s pupils achieve some of the highest exam scores - but from Monday they will b e invited to focus on failure. There will be workshops, assemblies, and activities for the girls, with parents and tutors joining in with tales of their own failures. There will be YouTube clips of famous and successful people who have failed al ong the way and moved on. The emphasis will be discussions on the merits of failure and on the negative si de of trying too hard not to fail. 'Courage in the classroom' Ms Hanbury told BBC News that she had placed a great emphasis on developin g resilience and robustness among the girls since she arrived at the school four years ago. "The girls need to learn how to fail well - and how to get over it and cope with it," she said. "Fear of failing can be really crippling and stop the girls doing things they really want to do."the pupils are hugely successful but can so metimes overreact to failure even though it can sometimes be enormously ben eficial to them. "We want them to be brave - to have courage in the classroom, " she added. Wimbledon High is an independent school, part of the Girls' Day S chool Trust. GDST chief executive, Helen Fraser, said: "Resilience is so important in working l ife these days. "Wimbledon High School is showing how making mistakes is not necessarily a b ad thing, that it is fine to try - and fail - and then pick yourself up and try again - or as Samuel Beckett said, 'fail better'." 24 "Blow your own trumpet" week, run by Wimbledon High School (motto : "Stepping in, Striding out"), aims to h elp its over-pressured, high-achieving pupils, who find it difficult to accept th at they have been successful. Head He ather Hanbury explains: "Some teenag ers will continue to push themselves, n ever recognising when they've done e nough." The school has also run a succ essful "failure" week.
Key quotes 25
What does success involve? 26
Project Retrospectives Can Help Definition of Retrospective: Relating to or thinking about the past Where were the failures any failures! Identify what worked well and what needs improvement Led by an independent facilitator Held in a safe place quiet, not disturbed Not a witch hunt looking at processes Plan the event issues questions at the start of the project Have the right people involved Record results in open area, allocate and monitor actions Use the results in future projects!! Be honest 27
Retrospectives Norm Kerth s Prime Directive: Regardless of what we discover, we must understand and truly believe that everyone did the best job he or she could, given what was known at the time, his or her skills, abilities, the resources available, and the situation at hand. 28
Emotional Histogram 29
Richard Branson His views I ve been failing for as long as I can remember. In fact, I ve been failing even longer than that I fell over many times as a baby before learning how to walk. The pattern has continued into adulthood and my life as an entrepreneur, and I have learned and loved every step of the way. We re celebrating failure all month at Virgin and teamed up for a Hangout on How to Turn Failure into Success. 30
Thomas Edison - Quote American Businessman and Inventor. Among other things he invented the Electric Light Bulb. 31
Lessons to learn 32
Everyone should aim to: TRANSFORM THE FEAR OF FAILURE TO THE ACTION OF SUCCESS 33
I Dare you to Fail It does not mean we should aim at failure, rather that creativity is a journey that involves taking wrong turns along the way 34
Summary Looked at: The challenges you face Examples of client benefits How to move forward with TMMi TMMi trends Don t fear failure embrace it 35
TMMi 모델을활용한 Quality Process 개선사례 H.H. Cho / hhcho@sta.co.kr Session 5
Ⅰ. The importance of (SW) Quality Ⅱ. Introduction to TMMi Ⅲ. SW Quality Improvement using TMMi
Question 1 Are you aware of TMMi using small i and CMMI using Large I? Yes No 3
Question 2 Do you know why TMMi using small i and CMMI using Large I? 4
Question 3 Are you new to SIX-SIGMA(DFSS), LEAN or Lean-Six Sigma? Yes No 5
Ⅰ. The importance of (SW) Quality Ⅱ. Introduction to TMMi Ⅲ. SW Quality Improvement using TMMi
품질이슈 Quality can make the difference between excellence and disaster. Accident Reason Root Cause Quality Focused Challenger Explosion(1986) O-ring Flaw Lack of O-ring Test in a very low temperature NASA disregarded warning from engineers Lack of factual based decision making Toyota Recall (2009) Sudden unintended acceleration Lack of quality control for the TPS, and so on Growth-oriented policies BP s Oil Spill(2010) Safety of drilling equipment (Blowout Presenver) will not operate hydraulic leaks and battery failure Poor test for cement borehole, and so on Lack of systematic approach, Lack of factual approach to decision making 7
SW 품질과비즈니스영향력 (1) < Forrester 의북미 (US, Canada), 유럽 (the UK, France, and Germany) IT 종사자 Survey > 다양한 SW 제품개발 [ Base : 336 product development and IT professionals involved with software development ] 다양한직무 [ Source : Software Integrity Risk Report, April 2011, Forrester Research ] 8
SW 품질과비즈니스영향력 (2) There is tight link between SW Quality and Business Impact SW 코드결함으로가장큰영향을받는것은무엇인가? Customer satisfaction Product release schedule/time-to-market Brand image Company revenues Product uptime Don t have an opinion None of the above [ Source : Software Integrity Risk Report, April 2011, Forrester Research ]
비즈니스목표와 SW Quality 노력 Software Quality(Risks) Have a Direct Impact on Business Success 품질개선활동을추진하는주요이유가무엇인가? 분명한비즈니스목표 5% 4% 3% 3% 3% 2% 2% 7% 7% 7% 11% 13% 15% 19% [ Base : 336 product development and IT professionals involved with software development ] * Software Integrity Risk Report, April 2011, Forrester Research 10
Ⅰ. The importance of (SW) Quality Ⅱ. Introduction to TMMi Ⅲ. SW Quality Improvement using TMMi
왜테스팅이개선되어야하나? Software industry has invested substantial effort to improve the quality of its products, but the software industry is still far from zero defects. Testing often takes 30-40% of project costs 60% Succeed Failed Challenged We can no longer imitate the three monkeys : Hear no failures See no failures Speak no failures 50% 40% 30% 20% 10% IT Projects are harder than climbing Everest 0% 2004 2006 2008 2010 2012 [ Source: The Standish Group, Chaos Manifesto, 2013 ]
SW 테스팅을개발앞단계에서부터 ~~ Early removal of defects contribute to mitigating risks and improving SW quality. 개발프로젝트의성공에가장큰영향을줄이슈는무엇인가? 27% 22% 14% 10% 10% 8% 4% 2% 1% 1% [ Source : Software Integrity Risk Report, April 2011, Forrester Research ]
프로세스개선프로그램 많은조직들이품질과생산성향상을위해 전사관점의프로세스개선프로그램을수립함 (company-wide process improvement program) 이러한프로그램은종종많은양의프로세스문서를만들고 결국엔부담이되어무시되어버리는결과를초래함 (become a burden or are ignored completely) In many cases, the organization is left with very little benefit to show for its efforts.
린식스시그마 Timeline Walter Shewhart SPC 개발 Deming SPC 확산, 14Points 제안 SPC TQM Motorola Introduces Six Sigma AlliedSignal GE Adapt LSS to Business Processes 1900 1930 1950 1980 1990 2000 Just-in-Time 포드 - Assembly line - Flow Concept 도요타 TPS - Flow concept from ford - Pull concept from grocery store
CMM(I), SPICE, TMM(i) Timeline CMM V1.0 CMM V1.1 ISO 15504(SPICE) first draft released TMMi V2.0 (Level 2 &3) released ISO 33063 released 1991 1993 1995 1996 2009 2015 TMM
개선을위한두가지 Approach 벤치마크정보활용 => 모델기반 Compare yourself to others who have similar strategies (but not necessarily similar products). Top-down strategy of borrowing from others to learn something new. Decomposes the improvement needs into processes. 문제해결을위한 Flexible way Improve the quality of your processes by eliminating waste and variation. Bottom up strategy where improvement begins at the work-team level. Used to solve specific problems.
TMMi Model Maturity Level High Maturity L5 : Optimization Defect Prevention Test Process Optimization Quality Control L4 : Measured Test Measurement Product Quality Evaluation Advanced Peer Reviews Infrastructure for Process Improvement L3 : Defined Test Organization Test Training Program Test Lifecycle and Integration Non-functional testing Peer Reviews L2 : Managed Test Policy and strategy Test Planning Test Monitoring and Control Test Design and Execution Test Environment Level 1 : Initial
TMMi Maturity Level(4/5) Definition Features Level 4 Measured Testing is a thoroughly defined, well-founded and measurable process. Testing is perceived as evaluation Measuring Test Performance and cost Measuring Product Quality Measuring Review Result Support factbased decision making Visibility Quality and process efficiency are being statistically measured through the whole development lifecycle. A test measurement program supports predictions relating to test performance and cost. IN OUT
TMMi Maturity Level(5/5) Level 5 Optimization Definition Optimized test process is focused on defect prevention Features Continuously improved Fine-tuning methods and technique Tool support as much as possible Process reuse Six Sigma(DFSS) & Lean is helpful for CMMI and TMMi Level 5 Visibility Test Process is optimized and continually improving its processes IN OUT
The Benefits of TMMi An IT organization that reached TMMi Level 3, as one of the first testing organizations worldwide, reported a higher Defect Detection Percentage during the system test. (DDP) 75 70 65 60 55 50 1 st year 2 nd tear 3 rd year 4 th year * DDP(Defect Detection Percentage) [ Source : Erik van Veenendaal and Jan Jaap Cannegieter, The Little TMMi, UTN Publishers, 2011 ]
The Benefits of TMMi An IT organization at TMMi Level 2, reported better predictability of the testing process. A Financial organization reached TMMi Level 2, reported the improvement of the DDP. (%) (DDP) 50 40 30 20 10 0 1 st year 2 nd tear 3 rd year Deviation of test time spent versus estimated test time (percentage) Defect Detection Percentage of the system test [ Source : Erik van Veenendaal and Jan Jaap Cannegieter, The Little TMMi, UTN Publishers, 2011 ]
Ⅰ. The importance of (SW) Quality Ⅱ. Introduction to TMMi Ⅲ. SW Quality Improvement using TMMi
Case Study of Defense Industry Company L CMMI L5 SIX SIGMA(DFSS), LEAN SIX SIGMA 경영진의품질및테스트인식수준이높음 Continuous improvement activities Perform a step-by-step risk analysis(from generic risk analysis..) Strong Points Company H CMMI L5 SIX SIGMA(DFSS), LEAN SIX SIGMA 경영진의품질및테스트인식수준이높음 Continuous improvement activities Well known about process (Recently enhanced dev. quality control gate) Some members do not well known about test processes Lack of risk tracking Lack of test log managing Weak Points Lack of dedicated test organization Lack of professional test expertise Lack of risk tracking
Case Study of Defense Industry Similarities and differences of both companies Company L Company H CMMI Level 5 Development CMMI Level 5 TMMi Level 3 Testing Level TMMi Level 3 Common risk analysis based on quality attributes Define new procedures to derive Test conditions Newly create new Master Test Plan(MTP) Risk Based Testing Test condition Test Plan Requirements based risk analysis Apply to the existing test Design documents Updated Original Master Test Plan(STP) V-Model-based testing activities Integration with Dev. lifecycle V-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation CMMi Level Process Area Process Area TMMi Level Level 5 (Optimizing) Level 4 (Quan. Managed) Org. Innovation and Deployment Causal analysis and Resolution Organizational Process Performance Quantitative Project Management Defect Prevention Test Process Optimization Quality Control Level 5 (Optimization) Requirements Development Technical Solution Product Integration Verification Test Measurement Product Quality Evaluation Advanced Peer Reviews Level 4 (Measured) Level 3 (Defined) Organizational Process Focus Organizational Process Definition Organizational Training Integrated Project Management Integrated Supplier Management Risk Management Decision Analysis and Resolution Test Organization Test Training Program Test Lifecycle and Integration Non-functional testing Peer Reviews Level 3 (Defined) Level 2 (Managed) Requirements Management Project Planning Project Control and Monitor Supplier Agreement Management Measurement and Analysis Process and Product Quality Assurance Configuration Management Test Policy and strategy Test Planning Test Monitoring and Control Test Design and Execution Test Environment Level 2 (Managed)
L 사사례 WHY L 사가 TMMi 기반테스트프로세스개선을추진한이유? Everyone else is doing it An executive heard about it or read about it somewhere A customer demands it If they don t do something, they won t survive In order to maintain a leader & have a good position in bidding situation ( 국방산업분야최초인증!)
L 사사례 프로세스개선활동 R&D 품질경영시스템 국방정책 R&D Process HW based Process SW based Process SW Process Update HW/SW Integration Process Measurement and Continuous Improvement Measurement and Continuous Improvement Defense embedded SW guideline SW related contents update MRA* update * MRA : Manufacturing Readiness Assessment * CMMI : Capability Maturity Model Integration * TMMi : Test Maturity Model integration * TMMi based Test Process Improvement, March 2015, LIG Nex1
L 사사례 - SW 테스트프로세스구현 Establish Test Policy and Strategy Strengthen Test Design Activities Organizational Test Policy Product Risk based test strategy Establish Test Performance Indicators Adopt formal test techniques Make test design specification templates Identify early test item and design for the dynamic test Perform integrated test planning activities Integrate Level test plan Establish product risk analysis activities and testing strategy Clarification of the testing exit criteria Establish Master Test Plan Try to separate test organization SW Test and Whole System Test Develop test Job description & Career Path Develop test training Program Develop Test organization and training program * TMMi based Test Process Improvement, March 2015, LIG Nex1
L 사사례 - Testing Activities SW Coding guideline Static Testing/ Quality Management SW Reliability Test TMMi based Process Implementation TMMi based Process Institutionalization SW Coding guideline SW Testing Pilot project Extend SW Testing Scope SW Reliability Test(Organizational) Build SW testing process TMMi Level 3 TMMi institutionalization Independent testing Static Testing tool - QAC++, CodeSonar Static Testing tool - QAC Adopt SW Incident management tool - QualityCenter introducing dynamic testing tools - VectorCAST, LDRA SW Resource Management tool - SharePoint SW Dynamic Test tool - DT10 SW Static test tool - CodeSonar(USB) SW Test management tool - Jama SW Build Automation tool - Jenkins SW Static test tool - QAC++(USB), QAV Setup Partner s SW Process Educate Partner s SW reliability test Perform Partner s SW reliability test Develop SW Quality Training Program and Train(3 classes) Develop SW Quality Training Program and Train( 6 classes) Quality Management Team Development Team Leads the Activities and Quality Team is supportive Suppliers are also applied * TMMi based Test Process Improvement, March 2015, LIG Nex1
TMMi 적용효과 - L 사 # of Defects/ KLOC SW 개발수명주기앞부분에서결함검출율향상 Applied TMMi based Processes, 7 Projects Not applied TMMi Processes, 33 Projects - The defect density of the coding and unit testing and SW integration testing phase increased significantly - The defect density of System integration testing is reduced * TMMi based Test Process Improvement, March 2015, LIG Nex1
TMMi 적용효과 - L 사 Organizational Pride The successes we've had in process improvement have been a source of great pride for our quality assurance team. 사내에품질경영을위한기본적인품질관리프로세스가 CMMi 등으로정착이되어있는상태에서테스트를위한프로세스를보완해품질관리의성과가더욱빛나도록함!
H 사사례 WHY Why did Company L apply Process Improvement using TMMi? Everyone else is doing it An executive heard about it or read about it somewhere A customer demands it If we don t do something, we won t survive A compelling reason for change At the VOC meeting, The CEO announced to initiate TMMi
H 사사례 - Process Improvement Defense Industry Policy (Quality Management System) R&D Process HW based Process SW based Process SW Process Update HW/SW Integration Process Measurement and Continuous Improvement Measurement and Continuous Improvement Defense embedded SW guideline SW related contents update TMMi based Process update Applied * DFSS : Design for Six Sigma
H 사사례 - SW 테스트프로세스구현 Establish Test Policy and Strategy Organizational Test Policy Product Risk based test strategy Reused Test Performance Indicators Strengthen Test Design Activities Adopt formal test techniques Make test design specification templates Identify early test item and design for the dynamic test Update original Master test plan(stp) Integrate Level test plan Establish product risk analysis activities and testing strategy Clarification of the testing exit criteria Establish Master Test Plan SE WPL and SW WPL have Responsible for Testing and Development work Develop test Job description & Career Path Develop test training Program Develop Test organization and training program
TMMi 를효과적으로적용하기! To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful, appropriate manner Identifying problems as objectively as possible and tie process improvement to your business goals Tailoring (interpreting) TMMi to fit your context and needs Not forcing inappropriate solutions simply because a practice is identified in the TMMi model Use TMMi as a Guide not as a Rule Book.
TMMi 기반프로세스개선의핵심요소 A compelling reason for change Leadership of the change effort by the top executive Informed commitment of the top management team Designation of a primary change agent and an adequate mandate for change Sound performance measures that drive change Without these, it s very likely that you will experience a false start of your process improvement program.
테스트프로세스개선의 Manifesto Detailed Process IP1. Flexibiltiy IP2. Best Practices IP3. Deployment oriented IP4. Peer reviews Templates Process oriented Quality assurance Model driven IP5. Business driven [ Source : Erik van Veenendaal and Jan Jaap Cannegieter, The Little TMMi, UTN Publishers, 2011 ]
Conclusions 1. TMMi can be a good tool for test process improvement. 2. TMMi has similar structure with CMMI, so a company using CMMi can easily apply TMMi practices. 3. Needs a compelling reason for change. Just trying to get to TMMi Leve3 without understanding the business context will be fail in the short or long term. 4. The improvement team must understand the business problem. 5. As TMMi is a relatively recent model, the number of publications about benefits in practice is limited. However, applying the TMMi maturity will improve test process and have a positive impact on product quality.
TMMi based improvement case study of software reliability testing process at LIG Nex1 LIG Nex1 김명호 / myonghokim91@gmail.com LIG Nex1 신영섭 / busigee@naver.com Session 6
1. 회사소개 2. LIG Nex1 테스트프로세스개선현황 3. 소프트웨어신뢰성시험개요 4. TMMi 기반의테스트 Process 구축 5. 소프트웨어신뢰성시험프로세스개선 6. 결론및향후계획
1. 회사소개 LIG 넥스원은정밀유도무기등방위산업 전분야에걸친최첨단무기체계를 연구개발 / 생산기업임. 3
1. 회사소개 4
2. 테스트프로세스개선현황 국방정책및 Global 모델 (CMMI, TMMi 등 ) 에따라지속적인프로세스개선활동을수행하고있음. 5
2. 테스트프로세스개선현황 07 년부터 SW 신뢰성을향상을위한프로세스, 인프라, 교육등의활동을수행함. 6
3. 소프트웨어신뢰성시험개요 무기체게소프트웨어는 무기체계소프트웨어개발및관리매뉴얼 을기준으로시험을수행함. 소프트웨어신뢰성확보활동 이란소프트웨어의잠재적결함을최소화시키기위한 제반활동 ( 신뢰성확보계획수립, 검증및확인, 신뢰성시험등 ) 을말한다. 소프트웨어신뢰성시험 이라함은소프트웨어코드가일으킬수있는결함을사전에 식별하여제거하기위한시험을말하며정적시험및동적시험으로구분한다. 1) 소프트웨어정적 (Static) 시험 이란소프트웨어를실행하지않은상태에서잠재적인결함을검출하는시험을말하며, 코딩규칙 (Coding Rule) 검증, 취약점점검및소스코드메트릭점검으로구성된다. 2) 소프트웨어동적 (Dynamic) 시험 이란소프트웨어의요구사항및구조를기반으로소프트웨어를실행하면서기능의정상동작여부를확인하는시험을말한다. 7
3. 소프트웨어신뢰성시험개요 소프트웨어신뢰성시험의시험기준은지속적으로높아지고있음. 정적시험분야 구분 11 년 14 년 16 년 코딩규칙 방위사업청코딩규칙적용 - 공통규칙 : 45 개 - C 언어전용 : 5 개 - C++ 언어전용 : 10 개 방위사업청코딩규칙적용 - 공통규칙 : 45 개 - C 언어전용 : 5 개 - C++ 언어전용 : 10 개 국제표준을사업별조정및합의 - C 언어 : MISRA C (143 개규칙 ) - C ++ 언어 : MISRA C++ (228 개규칙 ) - C # 언어 : Microsoft (62 개규칙 ) - JAVA : Oracle 방위사업청코딩규칙은유지 취약점 실행시간오류검출목록 : CWE-658/659 사업별로선별적용 실행시간오류검출목록 : CWE-658/659 사업별로선별적용 실행시간오류검출목록 : CWE-658/659/660 CWE 목록중도구가지원하지않는항목에대해서는대안과사유를제출하고협의하여결정. 주석률은코딩규칙으로이동 소스코드메트릭 주석률 30% 이상 주석률 30% 이상 Cyclomatic Complexity : 20 이하 Number of Call Levels : 6 이하 Number of Function Parameters : 8 이하 Number of Calling Functions : 8 이하 Number of Called Functions : 10 이하 Number of Executable Code Lines : 200 이하 8
3. 소프트웨어신뢰성시험개요 동적시험분야 11 년 14 년 16 년 임무중요도, 기능안정성및통제능력, 사용빈도에따라목표값설정후구조기반동적시험수준결정 N/A, Statement, Branch, MC/DC 국제기준및규격 (MIL-STD-882E, DO- 178B, IEC 61508, ISE 26262 등 ) 을참고하여시험대상결정후위험수준별구조기반동적시험수준결정 N/A, Statement, Branch, MC/DC 결함발생빈도, 영향성및제어가능성에따라수준별요구사항기반동적시험수행 Statement, Branch, MC/DC ( N/A 없음.) 단, 국제표준 (MIL-STD-882E, DO-178, IEC 61508, ISE 26262 등 ) 을적용하여개발한경우는해당표준을따를수있다. 9
3. 소프트웨어신뢰성시험 Open Source Software 11년 14 년 16 년 소프트웨어신뢰성시험대상 - - 상세설계단계에서사용필요성및신뢰성확보제시하여승인시제외가능. 오픈소스사용시는라이선스조항을반드시확인하여야하며, 소스코드공개의무가있는라이선스는사용이불가함. 10
4. TMMi 기반의테스트프로세스구축 LIG 넥스원은 Level 2 Process Area 5 개, Level 3 Process Area 5 개총 10 개영역을적용함. 조직차원 4 개영역 : 테스트정책및전략, 시험조직, 테스트훈련프로그램, 테스트수명주기 & 통합 프로젝트차원 6 개영역 : 테스트계획, 테스트모니터링 & 제어, 테스트설계 & 실행, 테스트환경, 비기능테스팅, 동료 검토 Level 5 Level 1 Initial Level 2 Managed 1. 테스트정책 & 전략 2. 테스트계획 3. 테스트모니터링 & 제어 4. 테스트설계 & 실행 5. 테스트환경 Level 3 Defined 1. 테스트조직 2. 테스트훈련프로그램 3. 테스트수명주기 & 통합 4. 비기능테스팅 5. 동료검토 Level 4 Measured 1. 테스트측정 2. 제품품질평가 3. 발전된동료검토 Optimization 1. 결함예방 2. 테스트프로세스최적화 3. 품질제어 TMMi 공식인증시작 (2010 년 ~) 11
4. TMMi 기반의테스트프로세스구축 LIG 넥스원에구축된 SW 테스트프로세스를 TMMi 영역별로도식화하면다음과같은연관관계가있음. 조직차원정의 PA3.3 Test Lifecycle and Integration 개발과시험수명주기통합모델 PA2.1 Test Policy and Strategy 시험정책및전략 PA2.3 Test Monitoring and Control 시험모니터링 PA3.5 Peer Reviews 동료검토 PA3.1 Test Organization 시험조직 PA2.2 Test Planning 시험계획 PA2.4 Test Design & Execution PA2.5 Test Environment 시험설계및수행 PA3.2 Test Training Program 시험훈련프로그램 프로젝트차원정의 시험환경 PA3.4 Non-Functional Testing 비기능시험 12
4. TMMi 기반의테스트프로세스구축 TMMi Level 2~3 의 10 개 Process Area 는사내표준프로세스에포함되어있음. 13
4. TMMi 기반의테스트프로세스구축 소프트웨어시험프로세스는방위사업청규정및 MIL-STD 기반으로 CMMI 레벨 5 및 TMMi 레벨 3 요건을충족시키는총 33 종의표준프로세스가정의되어있음. 14
4. TMMi 기반의테스트프로세스구축 기본방침은 V 모델기반으로개발초기부터신뢰성시험활동을수행토록함. 15
5. 소프트웨어신뢰성시험프로세스개선 배경 : 소프트웨어신뢰성시험기준변경 추진방법 Step 1. 신뢰성시험의주요변경사항분석및이슈도출 Step 2. 도출된이슈를 Process, People, Product 및 Environment 관점에서개선과제도출및우선순위를고려한개선계획수립 Step 3. 개선과제중신뢰성시험에대한 As-Is 프로세스를정의및문제점도출 Step 4. 개정된규정에의해추가되어야할활동을정리 Step 5. 3 단계와 4 단계를통해개선및추가되어야할활동을통합하여 To-Be 프로세스를구축 16
5. 소프트웨어신뢰성시험프로세스개선 Step 1. 신뢰성시험의주요변경사항분석및이슈도출 총 16개주요변경사항및이슈 - 공통분야 : 8개 - 정적시험분야 : 4개 - 동적시험분야 : 4개 17
5. 소프트웨어신뢰성시험프로세스개선 Step 2. 도출된이슈를 Process, People, Product 및 Environment 관점에서개선과제도출및우선순위를고려한개선계획수립 개별사업별해결 : 3 개 과제우선순위 - A : 즉시실행 - B : 17 년실행과제 18
5. 소프트웨어신뢰성시험프로세스개선 Step 3. 개선과제중신뢰성시험에대한 As-Is 프로세스를정의및문제점도출 Process Modeling : BPMN 2.0 개발단계별 Process 정의 19
5. 소프트웨어신뢰성시험프로세스개선 Step 4. 개정된규정에의해추가되어야할활동을정리 20
6. 결론및향후계획 To-Be Process 특징 첫째, 신뢰성시험을시험항목별로연관성을분석하여시험프로세스를표준화하고시험항목별목표치를설정하여재작업을최소화 둘째, 개발초기부터전문부서의검토단계를추가하여시험단계에발생할수있는문제를사전에최소화할수있도록하였다. 셋째, 신뢰성시험전문조직의역할을늘리고개발조직과의협업프로세스를구축하여시험생산성을향상하였다. 향후계획 프로세스개선효과측정 : 테스트생산성, 결함검출률등 L4 & L5 기반프로세스개선 : 동료검토생산성향상, 정적시험목표치설정등 21
대한민국 SW 가치증진을선도하고산업을리딩하는 SW 테스팅전문기업네트워크 대한민국 SW 품질을책임집니다! 회원사명단회장사 : 와이즈와이어즈임원사 :MDS 테크놀로지,STA 테스팅컨설팅, 어니컴, 와이즈스톤, 지티지, 파수닷컴회원사 : 가온테스팅, 두루이디에스, 링크제니시스, 솔루션링크, 슈어소프트테크, 씨엔큐소프트, 아이오에이솔루션, 아이지오, 아주큐엠에스, 에스이앤티소프트, 에이취큐텍, 엠스텍, 지텍에스, 큐머드, 한국전력공사전력연구원,NIPASW 공학센터, 컴퓨터월드 /ITDAILY ( 자문 ) 숭실대학교 SW 학부 ( 박진호교수 ) 회원가입문의 한국 SW 산업협회배범진책임 연락처 :bbj99@sw.or.kr 한국 SW 테스팅협의회에서는관련전문기업및 조직의참여를기다리고있습니다.