Consumer Satisfaction Focus Group
July 31, 1998
Jean Campbell, Ph.D., Principal Investigator
Kim Einspahr, Research Assistant
Supported by:
Department of Health and Human Services
Substance Abuse and Mental Health Services Administration (SAMHSA)
Center for Mental health Services (CMHS)
Missouri Institute of Mental Health
Program in Consumer Studies and Training
5400 Arsenal Street (314) 644-7829
St. Louis, MO 63109 F=(314) 644-7934
Introduction
The Missouri Institute of Mental Health has been contracted by the Center for Mental Health Services to write a monograph on consumer satisfaction. The project involves the production of a consumer satisfaction best practices guidebook, as well as a review of the literature, a glossary of research terms, and a review of best practices throughout the United States. To establish a general paradigm for the concept of satisfaction among mental health consumers, the principal investigator, Jean Campbell, Ph.D., designed a focus group, which would examine, via teleconference, the concept of satisfaction. Focus group participants were selected because they were mental health consumers/survivors who worked in either service delivery, monitoring and assistance, or peer support. Initially, those recruited received a letter detailing the nature of the project and the purpose of the focus group. In addition, those who agreed to participate were told they would be paid for actively participating in the focus group activities. The activities of the focus group included reviewing a list of questions to be discussed during the teleconference (see Appendix I), participating in the teleconference, and reviewing the transcript of the call to edit, add or amend their comments. Eleven participants, seven women and four men, agreed to participate in the focus group (see Appendix II). The teleconference was held July 31, 1998 at 2:00 p.m. central time and was moderated by Jean Campbell, Ph.D. The following is a summary of the 60-page transcription from the teleconference.
Defining Consumer Satisfaction
It was the intent of the principal investigator to explore the concept of satisfaction by first defining satisfaction and then deciding whether or not it is a valid concept in mental health research. While some participants agreed that satisfaction is defined as a measurement of the extent to which the services received met the goals they were intended to achieve, others contend that satisfaction indicates whether or not services met the "minimum standards of treatment." According to Darby Penney, "'satisfaction' seems like a bare-bones concept. It implies that a service meets some minimal criteria of not being harmful or offensive."
Overall, responses of the participants indicate general agreement that the concept of satisfaction is too vague to generate a clear definition. This vagueness transpires from the misuse of satisfaction assessment by providers in mental health services. According to consumer David Gettys, satisfaction is a "vehicle by which providers may falsely claim that they are rendering satisfactory service." This political use of satisfaction assessments being noted (see section on Political Concerns for greater detail), it was generally concluded that satisfaction is not a valid concept when assessing metal health services, because of the true nature and selection of the services themselves. People who receive mental health services have been disempowered, making few decisions, if any, about the services they receive, and having little involvement in their treatment and recovery plan. Therefore, as several participants pointed out, by not having the opportunity to choose their services, they have nothing with which to compare their current status. One participant has found that the concept of satisfaction and the usefulness of services do not seem to be correlated in people's minds. For example, Darby Penney stated that "if you ask people if they are satisfied, by and large people say 'yes.' But, if you ask them 'How has this program been helpful to you?' most will say that it hasn't been very helpful." As a result, participants agreed that satisfaction assessments, as they have been traditionally executed, do not reflect the true feelings of consumers regarding the usefulness, appropriateness, and satisfaction rates of mental health services.
Some participants supported the use of the satisfaction concept, but stressed that it must first be extensively defined. According to Katherine Roberts, an adequate definition of satisfaction can be determined by dividing the measurement tool into domains based on what clients like or dislike about their experiences with service providers. Jon Grobe and Darby Penney suggest tailoring satisfaction instruments to measure the empowerment of mental health consumers and to "reveal and access the critical elements which ultimately empower the individual with the skills of self-help and recovery" (Grobe).
In summary, participants deemed the concept of satisfaction invalid for the following reasons:
Survey Instrument: Development, Implementation and Application
Participants identified many obstacles related to survey/instrument development and execution when conducting satisfaction assessments for mental health consumers. These obstacles were specifically related to the structure of the assessment, the type of questions asked, and the validity of the responses. As Paul Weaver noted, there appear to be three types of consumer satisfaction assessments:
Generally, participants expressed support for the greater usefulness of qualitative data versus quantitative data. Quantitative information consistently revealed high satisfaction rates. Participants felt these findings were invalid based on their own experiences and the writing and testimony of others. On the other hand, qualitative information tended to be more negative about traditional mental health services and to identify different issues of concern. Along the same lines, several participants shared successful assessment stories, which used face-to-face interviews and open-ended questions.
There were additional concerns expressed about problems associated with confidentiality and trying to obtain truthful responses from people who are afraid that their responses will be traced back to them. Katherine Roberts pointed out that if assessments are conducted consumer-to-consumer, respondents may feel more comfortable. Although participants did not focus on confidentiality extensively, the importance of this aspect when conducting valid assessments was noted.
Request for Inclusion and Diversity
There was a request for a more inclusive approach to mental health research, which would involve consumer/survivors who are people of color, in all aspects of the research design. This involvement would include participation in focus groups, in implementation of the tool to consumers, and in quality improvement activities following distribution of assessment results. The extensive inclusion of these groups will ultimately add new dimensions to the assessment process, will increase honest feedback of consumers, will empower a group of people with hands-on experience and will improve the entire mental health system. Participants felt strongly that satisfaction assessments need consumer voices and consumer participation to be significant in the process of instrument development, data collection and interpretation.
Jacki McKinney expressed a need for people of color to "[know] that they're really represented and that their thinking is representative." She also stressed the importance and power of sharing people's stories when developing assessments or speaking on behalf of consumers. She stated that "the next time that you talk with anybody about this issue...tell a story of someone who you know that this has actually impacted on...At some point we have to tell those stories....not only should we talk about the process, but we talk about the people." Providers and others need to recognize the importance of an individual's experience in the mental health system. By emphasizing individuals, there is a hope that people conducting assessments will understand the implications of their results.
Similarly, Mary Auslander emphasized the need for additional "research [which is] done on people's experience of the mental health system...partly based on what they knew about it before, partly based on what it was they thought they were going to get when they went to it, and...partly based on alternatives."
It was strongly noted that assessments must be derived, executed, and presented with a cultural context that is representative of the mental health consumer's successes, struggles, and experiences. As expressed by John Grobe, "It must be recognized that the most important consideration in gathering good survey data is the social and organizational culture in which the consumer/survivor lives (which include society's attitude towards the mentally ill). Relationship of the powerful to the vulnerable is more important than the factual content of the satisfaction survey. The consumer must experience safety and personal empowerment as a direct result of the surveying process in order to provide good data."
In summary, consumer participants expressed support for the following aspects of instrument development:
The Politicization of Consumer Assessments: A Persistent Concern
Throughout the discussion of satisfaction, there was considerable concern expressed among all participants regarding the intended use of the information that is obtained in satisfaction assessments. These concerns included knowing whether assessments are used to achieve funding to sustain a particular program, to provide support for managed care programs, or if they are really intended to measure the situations and feelings of consumers. Darby Penney expressed her concern as, "whether or not you call it satisfaction, if you're asking people about whether the services are useful, valuable, helpful, what they want, et cetera, it's within a cultural framework that assumes that the goals of the mental health system are to help people. And if you can't make that assumption, then what you're doing is really invalid." According to participants, a system of evaluation that is useful to consumers and is there to help them is what is necessary.
Although some assessments may be done to determine the current feelings of consumers towards mental health services, other assessments are guided by political itineraries which serve to support programs and funding for providers, rather than to assess the needs and feelings of mental health consumers. Judi Chamberlin noted the distinct difference in "usefulness" between assessments endorsed and conducted by people with power (often the providers) versus those conducted by the disempowered (often the consumers). For those in power, assessments may be used to sustain services, while for the disempowered, assessments may be used to provide successful changes in services and treatment. Chamberlin commented that "the idea that the leadership of the mental health system or the political leadership is going to fix things is naïve because if you're looking at this in terms of a political process between people with power and disempowered people, nowhere in history have people in power ever voluntarily given it up and said oh yes we realize now this change is a good thing." Further, she supported the politicization of satisfaction assessments by consumers, asserting that "you have to look at something the mental health system never wants to look at, which is the essential coercive nature of the system. And that's always ignored. And therefore these things, while they pretend to be some sort of objective measures, are very highly political."
Some participants questioned the utility of information in current satisfaction assessments, and suggested that use of satisfaction surveys could conflict with the goals the consumer movement seeks to promote. On one end of the spectrum is the belief that researchers/consumers need evaluations to "occur whereby there is a commitment to change" (Jean Campbell) and need follow-up to ensure that those changes are made. At the heart of the issue is the desire for consumers' honest opinions to be heard, and that "outcomes or changes...are made as a result of what people...have issues about" (Gayle Bluebird). However, other consumers believe that the continued efforts to improve the quality of services undermine the consumer movement if the services that are measured are not voluntary. According to Jon Brock, "I think the consumer/survivor movement and its participants need to rethink this whole process...I don't believe that's the true purpose of the consumer movement in having effect on the services."
In summary, participants expressed the following political concerns regarding consumer satisfaction assessment:
Final Note
The noticeable power difference between providers and consumers, as expressed by John Grobe and several participants throughout the teleconference, is seemingly the underlying theme throughout this discussion of consumer satisfaction assessments. These participants expressed a desire and need to eliminate the "so-called" professional definition of satisfaction as created by providers, and to institute their own measures of empowerment, well-being, personhood, and of recovery. During the teleconference, a suggestion was made by Paul Weaver to develop action steps that would enable consumers to make necessary changes to satisfaction assessments. In the following statement he may have defined the first action step:
"We who are leaders need to take things to the top, to the decision makers and work with them and educate them. I think part of the problem we face is that they don't understand recovery. They don't understand empowerment. They don't understand quality of life. I think we have a great responsibility to do."
Appendix I
Teleconference Focus Questions:
Appendix II
Satisfaction Focus Group, Friday, July 31 @ 2pm CDST, 3pm EDST
Jean Campbell, Ph.D., Principal Investigator
Missouri Institute of Mental Health
5400 Arsenal Street
St. Louis, MO 63139
(314) 644-7829
F=(314) 644-7934
Mary Auslander
Box 230
East Dennis, MA 02641
(508) 385-8198
F=(508) 385-6850
Gayle Bluebird
110 SW 8th Ave.
Fort Lauderdale, FL 33312
(954) 467-1431
John Brock
816 Conroy Road
Birmingham, ALA 35222
(205) 595-5662
(205) 591-8520
F=(205) 591-8520
Judi Chamberlin
2 Dow Street
West Sommerville, MA 02144
(617) 628-8438
F=(617) 628-8438
William David Gettys
3204 Lumberjack Road
Nashville, TN 37214
(615) 874-1389
John Grobe
127 Chester Parkway
Duluth, MN 55805
(218) 728-6054
(218) 226-4205
F=(218) 728-6054 (call first)
Irene E. Lynch
39 Fairway East
Colts Neck, NJ 07722-1418
(732) 946-4489
F=(732) 946-3344
Jacki McKinney
5124 Newhall Street
Philadelphia, PA 19144
(215) 844-2540
Darby Penny
NYS Office of Mental Health
44 Holland Ave.
Albany, NY 12229
(518) 473-6579
F=(518) 473-7926
Katherine M. Roberts
3335 Overcreek Road
Columbia, SC 29206
(803) 782-6780
F=(803) 782-6780
Paul Weaver
KDMHMRS
100 Fair Oaks Lane, 4W-C
Frankfort, KY 40621
(502) 564-4448
F=(502) 564-9010