《A Continuous Quality Improvement Approach to Evaluation》由会员分享，可在线阅读，更多相关《A Continuous Quality Improvement Approach to Evaluation（93页珍藏版）》请在金锄头文库上搜索。
1、A Continuous Quality Improvement Approach to EvaluationSmaller World CommunicationsBarb van MarisHealth Communications UnitSpecial Topics WorkshopLearning Objectives1. Participants will understand the similarities between evaluation and quality improvement.2. Participates will be able to explain the
2、 benefits of approaching evaluation from a quality improvement perspective 3. Participants will understand how evaluation fits-in to quality improvement4. Participations will be able to describe the Model for Improvement5. Participations will identify different methods and tools used ina CQI approac
3、h to evaluationLearning ObjectivesTools we will cover todayNeeds assessmentSWOT analysisFlow chartsFishbone diagramAffinity DiagramBrainstormingPrioritization MatrixVarious measurement toolsPresentation OutlineOverview of evaluation and quality improvementDiscussion.how does this relate to your prac
4、tice?Challenges experienced with traditional approach to program evaluationBenefitsCreating a CQI approach to evaluationThe Model for ImprovementDrawbacksConclusionsSmaller World CommunicationsPerformance measurement and evaluation Primarily in the health care and health promotion fieldClients inclu
5、de (health units, hospitals, not for profit organizations, professional colleges, funding agencies)Some work in the private sectorThe evolution of an evaluation practiceClinical ResearchProgram EvaluationControlled experiments in the labClinical trials in hospitals4Research Design4Threats to validit
6、y/reliability4Highly controlled4Mutli-variate statisticsApplied research skills to evaluating health promotion programs4Evaluation Design4Indicators4Goals/objectives/outcomes4Internal evaluation4Lack of experimental control The evolution of an evaluation practiceEvaluation of PT Clinics/ QA Programs
7、 for Professional CollegesPerformance MeasurementAdapted to language of standardsand RHPA4Move from QA to Continuous Learning4Demonstrate effectiveness for accountability (Insurance Co.)4Demonstrate evaluating practice for College Requirements Applied market research skills and program evaluation in
8、 hospital setting4Indicators4Client and employee opinions4Clinical outcomes 4Balance Score CardsThe evolution of an evaluation practiceOrganizational DevelopmentNatural progression to increase utilization of performance measurement data4Facilitation skills4Team building4Business strategy4Leadership4
9、Continuous Quality ImprovementOverview: Evaluation and CQIThe two disciplines developed in parallel with the same goal “to improve our services and programs”.But, they stemmed from two different areas of studyProgram evaluation - social sciences researchCQI - organizational development/businessEvalu
10、ationApplication of social science methodsEmerged from the Education and Public Health fields prior to WWI By 1930s, applied social science and program evaluation grew at a rapid rateEvaluation of government programs first took-off in the U.S. with Canada not far behindEvaluation“Evaluation research
11、 is the systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of social intervention programs.”Rossi and Freeman, 1993The increase in popularity of program evaluation emerged from the need for government programs to be accountab
12、leAlthough most text books state there are many purposes or uses to evaluation the underlying tone is still to demonstrate effectiveness or proper use of resourcesEvaluationEvaluationTreasury Board Policy on the Evaluation of Programs.GuidelinesProgram evaluation in federal departments and agencies
13、should involve the systematic gathering of verifiable information on a program and demonstrate evidence on its results and cost-effectiveness. Its purpose should be to periodically produce credible, timely, useful and objective findings on programs appropriate for resource allocation, program improv
14、ement and accountability.(1981)Treasury Board of Canada (1981 ). Guide on the Program Evaluation FunctionEvaluation helps you to make decisions about:- the optimal use of time and resources- determining if the program/service is meeting the needs of participants- ways of improving a program/service-
15、 demonstrating the effectiveness of a program to funders and other stakeholder groupsEvaluationEvaluations focus is on measurement In order to collect valid and reliable information, evaluators utilize, as much as possible the scientific method for collecting information.It was quickly recognized wh
16、at we need to measure depends on the programs stage of developmentEvaluation Types of Program Evaluation- Formative- Process- Summative (outcome)Evaluation Formative - information you collect to help you plan or implement a program/service- needs assessments- pre-testing program materials- audience
17、analysisEvaluation Process - studies program implementation- tracking quantity and description of people who are reached- tracking quantity and types of services- description of how services are provided- quality of services provided- was everything implemented the way you thoughtEvaluation Outcome
18、- studies the outcomes of the program/service- changes in attitudes, knowledge or behaviour- changes in morbidity or mortality rates- changes in policy- program results in relation to program costs Evaluation Different approaches emergedExternal evaluation - highly rigorous and objectiveInternal eva
19、luationParticipatory evaluationEmpowerment evaluationUtilization-focused evaluation EvaluationProgram EvaluationInternal Evaluation ApproachDefinition: Carried out by “persons who are responsible for the evaluation process” in an organization (John Mayne, 1992)BenefitsAn organization better understa
20、nds their own programs and environment by doing the evaluationThere is greater acceptance for the changes requiredEvaluationParticipatory Evaluationengages stakeholders in all or key aspects of the evaluationcollective-learningsharing of powerStep 1: Clarify your programStep 2: Engage StakeholdersSt
21、ep 3: Assess Resources for the EvaluationStep 4: Design the EvaluationStep 5: Determine Appropriate Methods of Measurement and ProceduresSteps in Evaluating HP ProgramsSteps in Evaluating HP ProgramsSteps in Evaluating HP ProgramsIntroduction to Evaluating Health Promotion Programs - HCUStep 6: Deve
22、lop Work Plan, Budget and Timeline for EvaluationStep 7: Collect the Data Using Agreed-upon Methods and ProceduresStep 8: Process and Analyze the DataStep 9: Interpret and Disseminate the ResultsStep 10:Take ActionSteps in Evaluating HP ProgramsSteps in Evaluating HP ProgramsSteps in Evaluating HP P
23、rogramsIntroduction to Evaluating Health Promotion Programs - HCUContinuous Quality ImprovementStems from work in the organizational development field1950s - Dr. E. Deming introduced Total Quality Control to Japanese manufacturers1980s - Total Quality Management begins in the U.S.1990s - TQM fades a
24、s a fad, yet focus is still on continually improving products and services (CQI)Continuous Quality ImprovementPrinciples of CQI1Develop a strong customer focus2Continually improve all processes3Involve employees4Mobilize both data and team knowledge to improve decision makingBrassard M., and Ritter
25、D., 1994. A Pocket Guide of Tools for Continuous Improvement and Effective PlanningContinuous Quality Improvement3 Key QuestionsWhat are we trying to accomplish?How will we know that a change is an improvementWhat changes can we make that will result in an improvementPlan- Do - Study - ActPlanDoStud
26、yActIdentify purpose and goals, formulate theory. Define how to measure. Plan activities.Execute plan, applying our best knowledge to the pursuit of our desired purpose and goalsMonitor the outcomes. We study the results for signs of progress or success or unexpected outcomes. Integrate the lessons
27、learned and adjust the program. Do we need to reformulate the theory? Identify what more we need to learn.Scholtes, 1998. The Leaders Handbook (Based on the work of Dr. W. Edwards Deming)PDSA cycle creates continuous learningTheoriesApplicationPlanAct- PlanAct- PlanDo-StudyDo-StudyDo-StudyThe nature
28、 of true learning.Activity #1Select someone in your group to:facilitate the discussionrecord your ideas on the flip chartkeep track of timeActivity #1Identify where group members are already doing program evaluation.What are some of the challenges you are facing?Challenges to the Traditional Approac
29、h Program staff are resistant!Program staff focus on showing effectiveness rather than looking at what needs to be improvedThey also get hung-up on what design and statistical techniques are needed, which in many cases are beyond their skills or necessary for the evaluationPrograms are expected to b
30、e effective in an unrealistic time frame.it takes time for programs to evolveLevels of AccomplishmentLevels of accomplishmentIssue mappingCapacity buildingEnvironmental shiftBehaviour changeWithin each Level of Accomplishment, identify relevant performance indicators that might signal progress towar
31、d health promotion goals.Michael Hayes (1999) Ontario Tobacco Research UnitIMPACTPrograms EvolveNEEDActivitiesShort term OutcomesIntermediate term OutcomesLong termOutcomesFormative & ProcessSummative1. Relationships& Capacity2. Quality andEffectiveness3. Magnitude &SatisfactionSome summativeRealist
32、ic EvaluationExtended impact analysisKellogg Foundation - CES Conference 1999A CQI approach to evaluationFocus is not on showing what we did well, or whether the program passed or failed but what we can do better and the changes we can make to improve our work!You measure what you need to know to im
33、prove your program and to determine whether it works(process and outcome)All evaluation becomes formative in some wayStaff are encouraged to look for what is not working and why notA CQI approach to evaluationMethods of measurement are still the same, but there are additional tools we can adapt root
34、 cause analysis, flow diagrams, affinity diagrams, etc.Measurement would become a continuous aspect of any program where staff could utilize results and see the benefitsMeasurement - decision making cycle is fasterThe approach and in some cases the language used is differentA CQI approach to evaluat
35、ionA CQI approach doesnt mean an ineffective program should not be terminated.A CQI approach to evaluationNeed to create a learning culture“safe”increase understanding and benefits of ongoing measurementdebunk the myth that measurement is difficultbegin by measuring what you can in the best way poss
36、iblethen improve on it as you goutilize staff observations and hard dataempower staff to critically assess and observe their programsA CQI approach to evaluationFocus staff on the positive change they are trying to create and not on their defined program and activitiesKey short term evaluation quest
37、ions: What information will help us improve our program?Think about this month or the next 6 monthsActivity #2Review Case Study Each table is going to focus on one of the key elements we just discussed and brainstorm about strategies or ways the staff of this program could incorporate them 5 minutes
38、 to write ideas down independently on post-it notes10 minutes to put ideas on flip chartActivity #21. Making it safe to measure2. Increase understanding and benefits of measurement3. Debunk the myth that measurement is difficult4. Empower staff to critically assess and observe their program5. Focus
39、staff on the change they are trying to create and not on the defined program and activities6. What would some of your key short term evaluation questionsA CQI approach to evaluationSmall scale changes to make improvementsMeasure both processes and monitor outcomesBuilt in process for changing progra
40、m based on what is learnedWhat to MeasureAudience ReachedWho are you reaching/who are you not reachingNumbers reachedActivities implementedWhat activities did you do? What did you not do and.How well did you do them? What could you have done better?Challenges to implementationClient satisfactionOutc
41、omes achieved and not achievedEffect of program on those reached, were there any unintended effects?What is it changing? What is it not changing?.Costs (in-kind, staff time and $)External influences on program successWHY NOT?WHY NOT?WHY NOT?Program EvaluationTreasury Board Evaluation PolicyThe Gover
42、nment of Canada is committed to becoming a learning organization. Evaluation supports this aim by helping to find out what works and what does not; and by identifying cost-effective alternative ways of designing and improving policies, programs and initiatives.(February 1, 2001)Treasury Board of Can
43、ada Secretariate (2001 ). Evaluation PolicyKey DefinitionsEvaluationPerformance measurementBenchmarkingResults based managementQuality improvementContinuous Quality ImprovementOrganization/Program/Policy DevelopmentPerformance MeasurementEvaluationClinical Research or Academic ResearchInformsBenchma
44、rkingThe Improvement ModelImprovement ModelWhat are we trying to accomplish? AIMHow will we know that a change is an improvement? INDICATORSWhat changes will result in an improvement? ACTIVITIES or SOLUTIONSPlanDoStudyActImprovement or program development CycleLangley, Nolan et. Al. The Improvement
45、GuideSetting Aims 1a. Understand your program/service-client needs and expectations-goals and objectives-what are you currently doing?-strengths - what is working-challenges - what is not working.why not?What are we trying to accomplish?Tools-needs assessment-SWOT analysis -Flowcharting (illustrates
46、 a process)-Fishbone diagram - Cause/Effect (getting to the root causes of a problem)Setting Aims What are we trying to accomplish?How do you conduct a needs assessment?-find out from clients and/or stakeholders what they need and expect out of the program or service-define clients and stakeholders.
47、who do you need to talk to?-hold focus groups, informal interviews, conduct a survey-how many people do you ask?Setting Aims What are we trying to accomplish?Steps in doing a needs assessment1. Decide who your program/service is intended for. Be as specific as possible.2. Identify what you know abou
48、t them and what you dont know about them.Demographics (age, gender, education, culture, etc)Where are they currently accessing services?What are their service needs?What are some of their barriers to receiving service?What is the best way to communicate with them?Steps in doing a needs assessment3.
49、Who else is involved with their care? How are they involved? What are their needs?4. Decide on the best way to collect information from themCan you reach them through another program or service? Can you reach them through the mail? Internet? Phone? In-person?Decide on a method of measurement (way to
50、 collect information)Steps in doing a needs assessment5. Formulate your questions based on what you dont know about them and what would help you plan your program/service6. Put the questions into the data collection format you plan to use7. Recruit people to answer your questions or participate in a
51、 focus group8. Collect and analyze the dataTips for conducting focus groupsHold a group at a time that is convenient for your target audienceProvide refreshments, daycare, transportation etc.Recruit more people than you need (10-12)Call a day ahead to confirm participationFacilitator should not be e
52、xpected to recordAudio-taping is best but may not be necessaryRun at least 2 groups and ideally 3-5 groupsDevelop and use a moderators guide which outlines the questions you need answeredAnalyze the groups independently and then look for what is common across groupsSWOT Analysis looks at your organi
53、zations capabilities and the current environment-Strengths-Weaknesses-Opportunities-ThreatsSetting Aims What are we trying to accomplish?Flowcharting-Determine boundaries.identify where the process starts and ends-Identify major steps in the process-Sequence the steps-unless it is a new process docu
54、ment what “is” happening not what you would like to be happeningSetting Aims What are we trying to accomplish?Flowcharting - the basicsUsed to show starts or ends in the processUsed to show a task or activityUsed to show points in the process where decisions are madeUsed to show direction or flowSet
55、ting Aims What are we trying to accomplish?Fishbone diagram - find the causes Teenagers not attending the clinicSetting Aims What are we trying to accomplish?CostTimeinconvenientuncomfortableI dont need itI dont need itinconvenientuncomfortableCostWhy?Why?Fishbone diagram - find the causes Teenagers
56、 not attending the clinicPrograms are not for meNot able to get transportationLocation of clinicUncomfortableSetting Aims What are we trying to accomplish?CostTimeinconvenientuncomfortableI dont need itenvironmentStaffI dont need itCostTimeInconvenientNot able to get transportationLocation of clinic
57、Not sure what you doCultural beliefsCultural beliefsActivity #3Read the case studyClarify the program or issue by listing on a flip chart what the main concerns areReview the flow chartBrainstorm about possible opportunities for improvement1b. Decide on your Aim.give a general description of what yo
58、u are going to do-focus on the “results” you want, not the “activities” you will do to achieve them-explain in broad terms what is to be accomplished and whyyou may or may not decide to use numerical targetsSetting Aims What are we trying to accomplish?1c. Guidance statements.include information tha
59、t helps to answer the fundamental questions about the program or the improvement effort.-suggestions for measures-aspects of system you will focus on -boundaries in which changes are developed Setting Aims What are we trying to accomplish?Improvement ModelWhat are we trying to accomplish? AIMHow wil
60、l we know that a change is an improvement? INDICATORSWhat changes will result in an improvement? ACTIVITIES or SOLUTIONSPlanDoStudyActImprovement or program development CycleHow will you know you made an improvement?Select 6 or fewer measuresKeep the interests of the customer represented in the meas
61、urements chosenComparison data from before and after is helpful but not always possible or necessarySometimes the data needed to measure the impact of a change is not available for a long time. Select intermediate measures.Use multiple measures to balance competing interests and to assure the system
62、 as a whole is improved How will you know you made an improvement?Levels of measurementGlobal measures - relates directly to the aim of the study. Big system changes. Improvement signifies accomplishment of aim.Intermediate measures (milestones) - related to the global measures but is not sufficient
63、 to ensure the accomplishment of the aim.Process measures - assess whether actions were implemented as planned. Helps to explain why or why not the intermediate and global measures are reached.How will you know you made an improvement?Examples of measuresGlobal measures# low birth weight babies# tee
64、n pregnancies% population smoking# accidents due to drinking and drivingHow will you know you made an improvement?Examples of measuresIntermediate measuresincrease in awarenessincrease in knowledgeparticipation in physical activityuse of birth controlattendance at quit smoking clinicExamples of meas
65、uresProcess measures# participantsdemographics of participantsclient satisfactionemployee satisfactionpartnershipsHow will you know you made an improvement?Examples of process measuresEfficiency and costsTime to receive serviceTime for tests resultsCost per client visitHow will you know you made an
66、improvement?Examples of process measuresResource utilization# staff hoursamount of non productive hours# medications dispensed/dayHow will you know you made an improvement?Examples of process measuresStakeholders perceptions/satisfactionclient satisfactionemployee satisfaction# complaints# repeat cl
67、ients or requestsHow will you know you made an improvement?Activity #4 - Selecting indicatorsIdentify 3 global measures3 intermediate measures3 process measuresImprovement ModelWhat are we trying to accomplish? AIMHow will we know that a change is an improvement? INDICATORSWhat changes will result i
68、n an improvement? ACTIVITIES or SOLUTIONSPlanDoStudyActImprovement or program development CycleWhat changes will result in improvement?For simple systems and/or programs a list of changes or activities could be developed and tested quicklyFor more complex systems which require fundamental redesign o
69、r multiple program components consider:developing the design concept firstthen identify detailed activities/changes for each componentWhat changes will result in improvement?What do other organizations or programs do?Research other programsSearch the internet for ideasWhat does the literature sugges
70、t?What do the stakeholders suggest?What changes will result in improvement?Making changes to existing systemsCritically think about the current systemFlow charting and documenting where the challenges arise can quickly highlight opportunitiesConsider automating/eliminating redundanciesCreative think
71、ing - Tool (Brainstorming)Use Change ConceptsAffinity DiagramBrainstorm theissue, problem,opportunity.items identified inthebrainstorm sessionitems identified in thebrainstorm sessionitems identified inthebrainstorm sessionitems identified in thebrainstorm sessionitems identified in thebrainstorm se
72、ssionitems identified in thebrainstorm sessionitems identified inthebrainstorm sessionitems identified inthebrainstorm sessionitems identified inthebrainstorm sessionitems identified inthebrainstorm sessionitems identified inthebrainstorm sessionitems identified inthebrainstorm sessionitems identifi
73、ed in thebrainstorm sessionSilently sort theitems raised into groups or themesitems identified in thebrainstorm sessionitems identified in thebrainstorm sessionitems identified in thebrainstorm sessionitems identified in thebrainstorm sessionitems identified in thebrainstorm sessionitems identified
74、in thebrainstorm sessionitems identified in thebrainstorm sessionitems identified in thebrainstorm sessionitems identified in thebrainstorm sessionitems identified in thebrainstorm sessionitems identified in thebrainstorm sessionitems identified in thebrainstorm sessionitems identified in thebrainst
75、orm sessionWrite descriptive headers for each theme groupingitems identified in thebrainstorm sessionitems identified in thebrainstorm sessionitems identified in thebrainstorm sessionitems identified in thebrainstorm sessionitems identified in thebrainstorm sessionitems identified in thebrainstorm s
76、essionitems identified in thebrainstorm sessionitems identified in thebrainstorm sessionitems identified in thebrainstorm sessionitems identified in thebrainstorm sessionitems identified in thebrainstorm sessionitems identified in thebrainstorm sessionitems identified in thebrainstorm sessionheader
77、to name the themeheader to name the themeheader to name the themeheader to name the themeActivity #5 - Brainstorming solutions-Brainstorming exercise-Affinity diagramSetting PrioritiesVoting ToolsPrioritization matrixMulti-voting - openMulti-voting - closedPrioritization MatrixStepsGenerate or clari
78、fy the optionsDetermine and agree on criteria Compare the options to each other using the criteriaVote on the best optionPrioritization Matrix1. Create a matrix with the options on the vertical and the criteria across the top.2. Assign a score for each option for each criteria.3. Tally totals to det
79、ermine the score for each option.4. Each participant could do this independently and then take the average score for each optionPlan- Do - Study - ActPlanDoStudyActIdentify purpose and goals, formulate theory. Define how to measure. Plan activities.Execute plan, applying our best knowledge to the pu
80、rsuit of our desired purpose and goalsMonitor the outcomes. We study the results for signs of progress or success or unexpected outcomes. Integrate the lessons learned and adjust the program. Do we need to reformulate the theory? Identify what more we need to learn.Testing out changes or a new progr
81、amPDSA Cycles-allows you to study your new ideas on a small scale-learn from tests and then build on themLinking Cycles on a “Ramp”KnowledgeTimeTheoriesand ideasChanges thatresult inimprovementThe Plan - Do - Study - Act CyclePrimary means for turning the ideas into action and for connecting action
82、to learningCan use the cycle to build knowledge when you do not have sufficient information to answer one or more of the three questionsCan use the cycle to test improvements or programs on small scalebased on a trial and learning approach to improvement or developmentAllows you to gradually impleme
83、nt a program or changeIncrease promotionRun clinic at one highschoolPeer counsellingSelecting physicianMultiple tests at one timeWhat if you are developing a new program?Creating new programsIdentify aim or goalsIdentify indicatorsIdentify components needed to achieve program goalWhat activities are
84、 needed for each componentLook for best practice examplesWhat do other organizations do?What does the theory suggest is needed?Prepare a program logic model - diagram of your programBenefitsStaff are more open to collecting information on how to improve their program. Less threatening for staff Incr
85、eases likelihood results will be usedProgram planners can be more responsive to what is working and not working.Creates a learning environment for both program staff and FundersDrawbacksMay be criticized for not being “objective” enoughMore challenging More skills neededNeed to develop a culture of
86、critical assessment and quality improvement in order for the evaluation to be as objective as possibleRequires staff time and trainingConclusionsThe techniques used in program evaluation are not that different to continuous quality improvementThey evolved from completely separate disciplines but eve
87、n the language is not that far offThe main difference is: program evaluation stemmed from the need to demonstrate effectiveness - accountabilityCQI stemmed from the need to find ways to make programs/systems better ConclusionsMost programs take time to evolve before we can expect them to be effectiv
88、e. A continuous quality improvement approach to evaluation will facilitate that evolution and increase the likelihood that a program, given time, will be effective and have an impact.ConclusionsA CQI approach can Transform Evaluation Practice to Meet New Challenges Opportunity to learn from the organizational development fieldOpportunity to close the gap between management/business strategy and program planning and evaluation
