DHHS

Substance Abuse and Mental Health Services Administration (SAMHSA)

Center for Mental Health Services (CMHS)

(314) 644-8787

1. John Allen, MD
2. Frank Armstead, DE; Consumer Client Satisfaction Survey
3. Larry Belcher, WV; West Virginia Consumer Survey
4. Gayle Bluebird, FL; "CATS" Consumer Assessment Team Specialist
5. Jon Brock, AL; Consumer Satisfaction Survey
6. Terri Bronner, NY; Community Residence Report Card
7. Vicki Cousins, SC; Consumer to Consumer Evaluation Team Project
8. Larry Fricks, GA; GEST, Georgia Evaluation and Satisfaction Team
9. John Grobe, MN; Lake County Mental Health Service Local Advisory Council
10. David Hilton, NH; Mental Health Satisfaction Project
11. Sharon Mason, AL; Community Q.I. Survey
12. Dan Powers, NE; Mental Health Quality Review Teams and Consumer Survey
13. Ginny Riley, GA; GEST, Georgia Evaluation and Satisfaction Team
14. Andrea Schmook, IL; Consumer Survey of State Hospitals
15. Mary Smith, IL; Consumer Survey in Hospital and Community
16. Christine Squibb, MO; Consumer Satisfaction Survey
17. Joe Swinford, TN; Ten Care Partners Focus Group Study
18. Pamela Trammel, AL; FACTS (Family & Consumer Team Satisfaction Project)
19. Paul Weaver, KY; Kentucky Early Evaluation Project

Note: Although the GEST survey project was discussed twice, Larry Fricks reviewed GEST as it was piloted for the first time in Georgia, and Ginny Riley discussed GEST as an ongoing survey project that is tailored to individual contractors. Throughout this summary, the people who discussed projects will be referred to as "respondents", and sample populations, who were part of the satisfaction surveys, will be referred to as "participants, subjects, or interviewees".

Satisfaction Survey Locations:

Satisfaction surveys were conducted in a variety of locations, including mental health centers, clubhouses, day treatment centers, group homes, residential facilities, independent living apartments, hospitals, community programs, drop-in sites, substance abuse agencies, and satellite offices. In addition, one project conducted their survey in a children's agency, and two projects specifically targeted both rural and metro areas.

Of the surveys conducted, seven were pilot projects, four were snapshot surveys, and nine are ongoing surveys of satisfaction and quality improvement. Survey participants were primarily consumers of Mental Retardation and Developmental Disabilities, psychiatric services, and/or substance abuse services. More specifically, one project targeted primary, former, and prospective users of the mental health system, while another surveyed outpatient consumers and those who were receiving substance abuse treatment. One project focused its assessment on Medicaid clients receiving behavioral health care, while another project surveyed consumers who were discharged from a hospital. One project engaged people in a focus group to discuss services for mental health. Three projects conducted additional surveys for family members. All surveys rated the services used by consumers.

According to the respondents, the numbers of participants varied considerably according to the nature of the survey. Three survey projects had less than 50 participants, four projects obtained 50-150 participants, four projects had 250-800 participants, and three projects had over 1000 participants. Participant-rate data was unavailable for three of the projects included in this study. One respondent was beginning to implement a satisfaction survey and anticipated a response rate greater than 2000.

Respondents in our study were asked to describe the overall benefits and problems associated with doing consumers satisfaction surveys. Respondents reported experiencing methodological benefits, mental health system benefits, and benefits to consumer interviewers and consumer participants. Methodological benefits included using face-to-face specifically peer-to-peer interviews to obtain a greater response rate. One respondent noted that the use of consumers as interviewers enhanced the participants' interest because such interviewers were people who had utilized mental health services, knew about the mental health system, and could establish rapport with the participants and therefore get more valid responses. In addition, one respondent identified the use of a trend analysis as more beneficial than statistical reporting, because the results are more easily understandable and useful. Conducting interviews in places comfortable to consumers and the use of anonymity were also noted as beneficial to assessments. One respondent noted that the overall benefit was the help of consumers in the development of a reusable and valid tool.

Mental health system benefits included the importance of obtaining concrete information about consumer perceived need, as well as the belief that services are satisfactory, helpful, and individually beneficial. This information can then be used to identify problems, which will be followed up with continuous quality improvement. In addition, conducting assessments allowed mental health departments and related offices to integrate which promoted a greater understanding of consumer needs and created a corroborative effort to meet those needs.

Benefits to consumers who participated in the survey included increased self-esteem, self-worth, and life skills, as well as the belief that they are helping people who are experiencing the same things they have experienced. Finally, it was noted that surveys are beneficial because they make participants feel important and appreciated.

Ironically, some of the same practices that were noted as beneficial to assessments were also considered problematic for a variety of reasons. Specifically, face-to-face interviews and report writing were considered problematic by some because they are time consuming, labor intensive, and require extensive training. In addition, consumer confidentiality, earlier stated as a benefit when doing surveys, was also noted as problematic for several reasons. Specifically, it may be important to know, or be able to code, the types of services individuals are. One respondent expressed concern that surveying consumers in general could provoke a breach of confidentiality in the mental health system. Similarly, another respondent felt that regardless of the seemingly confidential process guaranteed by the interviewers, it is possible that consumer's names and responses will be linked and made available to providers.

Several respondents noted problems with obtaining responses from outpatients and people who had been discharged from services, and with conducting follow up interviews because often people move or relocate. In addition, one respondent noted problems obtaining equal response rates from all sites. These problems coincided with the belief that consumer satisfaction surveys are unable to reach a representative service population because they exclude the homeless, illiterate, and dissatisfied. One respondent expressed her/his belief that satisfaction surveys "fail" because they attempt to measure services of people who have a very limited frame of reference and have rarely been offered a choice of services. Similarly, another respondent stated that "one-time" evaluations reveal only small portions of the consumer experience.

Other problems noted by respondents included difficulties in obtaining funding for an entire project at the onset, adequately planning enough time for consumers to complete the survey, and administering the survey in a timely fashion and in a form that was easy to understand. One respondent expressed concern that relying on focus does not provide "hard core" data or represent all services recipients. Finally, several respondents noted problems obtaining interviewers in rural areas and finding ways to decrease the fear among some consumers that they would lose their social security check if they were paid to be interviewers.

We asked respondents to describe what they felt was the best overall practices in doing consumer satisfaction surveys. The following best practices were noted:

• Conducting face-to-face interviews
• Using consumers as interviewers
• Conducting peer-to-peer interviews (Family members to family members, consumers to consumers, and providers to providers)
• Organizing focus groups to include a diverse group of people
• Interviewing at locations where consumers are comfortable
• Using consumer-developed instruments
• Developing open-ended, neutral questions
• Ensuring privacy and confidentiality of interviewees
• Using the Wave Approach: conducting surveys at different periods
• Providing training and on-site support for consumer interviewers
• Maximizing all available agency information to make the sample representative of the population
• Keeping length of interview flexible
• Conducting follow-up and comparing findings with other surveys

Of the survey projects included, three were considered primarily consumer projects, three were considered mental health systems/professionally run projects, nine were considered professional/consumer partnership projects, and four noted that they were a combination of two or more types of projects. Specifically, two projects were a combination of a family/consumer/mental health systems project, one project was primarily a consumer/family run project, and another was a combination of consumer/professional/mental health systems.

The stakeholders in the projects were typically members of consumer advisory boards, consumer consultants, and administrative service providers. Specifically, seven projects used consumer advisory boards, eight projects reported using consumer consultants, and four of the projects were guided by the decisions of the administrative service provider or contractor. In addition, three projects relied on additional advisory boards, two projects reported using non-consumer consultants, one relied on a group of consumer affairs coordinators, one using a consumer researcher, one used a family-member consultant, one used an advisory task force made up of 50% family members and 50% consumers, and one was guided by an outside agency. Typically, consumer advisory boards consisted of family members, consumers, executive directors, and some even included a therapist. Additionally, three projects reported that the majority of the workload rested with consumers and family members. For other projects, the day-to-day proceedings were typically overseen by a staff, which may have been made up of consumers, non-consumers, consultants, and researchers. One project relied on an oversight committee made up of mental hygiene staff, the Office of Consumer Affairs, and the administrative service organization to oversee the process. Another project was a partnership involving the Department of Mental Health, the Consumer Affairs Office, and a consumer advocacy board. Interestingly, one respondent noted the necessity of political connections when trying to get the project funded and approved.

Of the projects we surveyed, eight paid their interviewers, five paid staff and family members, four paid professionals, two paid interviewees, two reimbursed for costs including mileage, and four paid no one. Only one project could not submit this information.

Respondents frequently mentioned that consumer involvement increased the flow of information into the survey development, implementation, and interpretation. The experiences that consumers brought to the project allowed them to provide greater insight into issues facing the consumer population, which could then be used to develop an adequate tool. In addition, consumer interviewers promoted a greater response rate because participants felt comfortable talking to a peer who shared their experiences. According to one respondent, allowing consumers to have input encourages other consumers also to respond. It was noted that many consumers were excited about the opportunity to give constructive feedback about the system and that did not use it simply as an opportunity to complain. Other respondents reported that having consumers involved with professionals and staff led to more respectful relationships, promoted a greater understanding of the consumer experience, and resulted in successful action steps necessary for system redesign.

Consumer involvement provided many benefits to the consumers as well, including the opportunity to carry out paid work, and to become involved in skilled positions. These experiences were beneficial in helping to increase self-esteem and feelings of empowerment because they felt as though they were helping others. One consumer respondent in our study, who carried out the data collection independently, discovered that it was easier for him/her to approach people and get feedback because of his/her prior experiences. S/he had the ability to relate to participants' feelings.

There were several problems noted regarding consumer involvement. Specifically, several respondents noted that a select group of staff and professionals were initially resistant and defensive. This group did not cooperate when presented with information from consumers about how to improve the system. These problems were often solved through education, which emphasized that the changes would benefit everyone. It was also noted that problems arose in situations in which consumers were involved in the beginning stages but were denied decision-making opportunities in the later aspects of the project. In contrast, one respondent expressed concern that the data received are unlikely to be relevant because of years of consumer "mental health conditioning" by the system.

Several respondents reported experiencing problems related to expenses. Specifically, one respondent noted problems when trying to cover the cost of implementing changes suggested by consumers. Additionally, some respondents acknowledged that from the business perspective it was difficult to hire people who might later drop out due to illness or better employment. One respondent reported problems finding highly functioning consumers to be interviewers in all parts of the state, while another noted problems securing transportation for consumers to. It was also noted that, at times, consumers were hesitant to participate because they feared having their benefits or social security checks taken away.

Finally, one respondent experienced problems maintaining consistent interviewing techniques with male and female interviewees. Specifically, the respondent found that women tended to talk at greater length than men. Although this provided more in-depth information, it also presented time constraints.

The following best practices were noted for consumer involvement in satisfaction surveys:

• Using consumers in every aspect of the project, including focus groups, tool development, interviewing, research, and data analysis
• Flexibility and fairness with all consumers; equal opportunity to be involved
• Consistent training program
• Using face-to-face or small group interviews with consumers as interviewers
• Consumers in charge of the project and/or evaluation teams
• Paying consumers for their participation
• Partnership of consumers and professionals
• Open-ended question format, with consumers as facilitators, during focus groups
• Providing consumer support to other consumers (on-site or by telephone)
• Recruiting a staff that is trusted by consumers

Of the projects included in our study, three used a quantitative survey, three used a qualitative process, and twelve used an approach that combined both methods. This information was not available for one project.

Reportedly, nine assessments were conducted face-to-face, three surveys projects were carried out through the mail, three were a combination of face-to-face interviews and telephone interviews, and two were a combination of all three. One assessment was given out face-to-face and could then be returned anonymously, and another assessment was given out by the provider at the point of service and could then be returned by mail.

Nine projects reportedly used team models to implement their survey; however, the make-up of these teams varied. Teams typically consisted of two to five consumers and family members, and some teams included professionals. These teams would then go on-site to administer the surveys. One project team went to drop-in centers, collected common responses from consumers, and then reported the frequencies of those responses. Another team consisted of a consumer from each division who memorized a set of questions, conducted the interview without any written materials, memorized the responses, and then later wrote the report.

Two projects reportedly used only a mail format, six projects had individuals go on-site to administer the survey, and two projects arrived at the site, explained the purpose of the project, and then handed the survey to individuals to complete.

Twelve projects developed assessment tools especially for their projects, two projects used a standard survey tool, and three projects combined the standard tool with additional questions to make it more specific to their needs. One project reportedly changes their assessment for each contractor, has used both combinations and standard tools, and prefers to develop a tool from scratch specific to the needs of each population. This information was not available for one of the projects in our study.

Eighteen of the projects in our survey involved consumers at some level in the development of the tool. Only one survey project reported that consumers were involved minimally, indicating that a select group of consumers were asked to look at the tool before it was implemented. However, one respondent noted that although consumers were involved, the extent of their involvement was unknown. Reportedly, consumers were involved individually, as members of committees or advocacy groups, and in focus groups. Twelve respondents reported using focus groups in their project, which served to establish the purpose of the assessment, and to bring people and ideas together. However, one project that reportedly did not initially use focus groups, appointed a task force to assess the issues that needed to be addressed in a hospital setting. One respondent indicated that consumers and staff received training before beginning to develop a tool. Other respondents reported that consumers were involved in the project from the beginning, designing the instrument from scratch. One respondent indicated that consumers were involved in the development of the tool through focus groups, and were then given a copy of the tool to review before implementation. Despite the numerous ways in which consumers were involved in most projects, one respondent expressed irritation that at times consumers were not involved as much as they could and should have been.

Nine survey assessments reportedly field-tested their tool before implementation, five projects reported that they did not conduct a field-test, and two respondents reported that they are in the process of field-testing their tool. This information was not available for one project included in our study.

Interview length varied considerably among assessments. Several respondents reported differences in reading and verbal ability that accounted for time variations. Overall, ten respondents indicated that their interviews/surveys took less than 20 minutes, five interviews/surveys were approximately 20-45 minutes, and two interview/surveys lasted approximately 30-60 minutes. One project conducted an introductory interview that lasted five to seven minutes, and then followed up with another interview that lasted 15-20 minutes. In addition, this project selected 10-15 people interview periodically over then next year. A different project that relied on information from focus groups reportedly held a group interview of 20 people that lasted two to two and half-hours. This information was not available for two projects in our study.

There were many benefits of the methods used to conduct satisfaction surveys; however, the most frequently mentioned benefit was the involvement of consumers in the development of the tool and the implementation process. Specifically, respondents indicated that consumers involved in focus groups and the development of the tool taught others about issues of satisfaction that were important to consumers, ultimately leading to a useful tool tailored to the contractor/provider. Several respondents reported feeling that consumers had contributed points of view that might otherwise have been missed, and that without consumer participation the tool would have been entirely different. In addition, consumer involvement in the implementation of the survey was also noted as beneficial because participants were more likely to respond to consumer interviewers and new types of questions. One respondent noted the importance of consumers conducting face-to-face interviews to reach people who may be unable to complete a written survey. Overall, respondents expressed their belief that the project benefited by maximizing consumer input in all aspects of the project.

Focus groups were also highly regarded as an effective method to encourage consumers and others to express their ideas and opinions. One respondent explained that their success in using both consumers and direct-care staff throughout the process got everyone involved and provided a greater understanding of both points of view. The team approach was also beneficial because it provided on-site support for difficult interviews or questions. Other benefits included being able to pay interviewers and interviewees for their time, and a standardized instrument with the wave approach.

Problems pertaining to methods were related largely to the formation of questions. Specifically, respondents reported experiencing difficulty when trying to make questions neutral and understandable and to include a comprehensive range of questions that stakeholders wanted know. Additionally, several respondents noted that surveys were time consuming and that when developing questions, it was difficult to incorporate every point of view into a survey. One respondent indicated experiencing problems during the development of the survey because it was difficult to ask questions and then trust in the confidentiality of the responses. Additional problems with the methodology included trying to classify information, ensuring random sampling, getting the provider to endorse the use of the tool, and realizing the limitations of mail surveys when trying to reach participants of varying reading and writing abilities. One respondent commented that it is important to realize that focus groups are not designed to produce reliable measures, but rather to develop ideas.

• Using dialogue/focus groups to involve more people and allow a fuller expression of ideas
• Involving consumers in every step of the project
• Compensating interviewers and subjects
• Field-testing of instruments
• Ensuring consumer authority over survey development
• Using a written survey instead of a team interview approach, because it is more cost-effective and straightforward
• Face-to-face interviewing techniques, when possible, or a combination of mail and face-to-face
• Taking time during interviews, assuring confidentiality, and using a conversational style
• Using information from a variety of sources, including providers and consumers, and keeping all stakeholders involved
• Involving consumers who are good at negotiating

Of the respondents included in our survey, sixteen used consumer interviewers, and one project used non-consumers as interviewers. This information was not applicable for two projects. The numbers of consumers used ranged from three to 30. Specifically, of the sixteen projects that used consumer interviewers, eight projects used fewer than 10 interviewers, five used 10 or more consumer interviewers. This information was not available for three of the projects in our survey.

Training typically consisted of role-playing, a manual, and a discussion of the survey procedures. Of the projects, eight used role-playing, five had a manual, three participated in an on-site visit, and three conducted continuous training. In addition, two projects reportedly paid their interviewers for the training. Length of training varied considerably. Some projects conducted training for two to five hours, and one reportedly had training for 30 days. Most of the projects trained interviewees for one to two days. Some projects spent significant time discussing survey methods and techniques used to obtain non-biased responses, while others included history and project concept. Many respondents covered report writing and confidentiality in the initial training and then held additional training as needed. One respondent reported giving their interviewers telephone cards to call if they needed support.

Generally, respondents reported positive experiences with training and indicated that it gave consumers an opportunity to express their feelings and ideas and to ask questions. In addition, it helped consumers to develop skills for future jobs, while working with professional staff. Some felt that role-playing and on-site visits helped consumers to become more comfortable before going out into the field by themselves, and that practicing report writing increased confidence. One respondent noted that it was beneficial to provide additional support to consumers on-site, to be flexible, and to support consumer interviewers. Another respondent reported that it was beneficial to have an Office of Consumer Relations, which could connect all of the areas in which the consumers were involved during training. Peer-to-peer interviews were again noted as beneficial to both interviewers and participants. One respondent who did not use consumer interviewers noted that for his/her area, provider-based surveys are the best way to do assessments at this time. However, the preference would be to do more consumer-interactive surveys in the future. Finally, one respondent, who helped to administer the survey, noted that it was beneficial to see the possibilities of satisfaction surveys, although surveys are not in themselves the delivery of improved services.

There were few problems noted with consumer interviewer training. However, several respondents reported problems finding enough time for training, making sure that interviewers followed through with planned interviews and were on time, and ensuring that consumers received enough training to produce accurate and readable reports. One respondent noted that several consumers required tutoring throughout the assessment process. Another respondent reported that some of the interviewers were not functioning well at the time of the survey. Respondents also noted difficulty finding enough consumers statewide to serve as interviewers.

Similarly, a respondent pointed out that many consumers who could be trained as interviewers might already be employed. One individual encountered problems with professionals failing to approach the idea of consumer interviewers with an open mind. He/she suggested that all people involved in the survey process need to think of consumers as employees.

• Using role-playing for training
• Conducting an on-site visit, which includes report writing and debriefing
• Providing a continuous training program, or repeat training when necessary
• Using a manual
• Consumers functioning as the point of delivery for the survey
• Ensuring that interviewers have patience, empathy, sensitivity, and understanding of the cultural and educational levels targeted by the survey
• Involving consumers in every aspect of the process, including the opportunity to lead the training
• Paying consumers for their time or providing other compensation (e.g. lunch)
• Using a team effort when writing the report, ensuring that reports are reviewed by many people before being finalized
• Educating consumers about the history of the project
• Providing consumers with a written script for dealing with problems

Thirteen respondents reportedly did not use an IRB, while two assessments did. However, three assessments were examined in a similar process by a governing board. This information was not available for one project included in our study.

Ten projects included in our study used a method of confidentiality in which only evaluators knew the identity of the respondent. Of these, five projects used a unique identification number and four linked their responses through an information system. Additionally, eight projects reportedly kept the identity of all participants anonymous by not asking for any type of identifying information; however, several projects coded their surveys to determine which type of service the individual was receiving. Interestingly, one respondent reported that there were no regulatory safeguards to assure confidentiality. However, participants in the dialogue/focus groups were assured that their comments would remain anonymous.

Ten projects used an informed consent process, eight projects did, and one project reported using it only when the provider required it.

All projects that responded to this question (18) reported that participants could refuse to participate and still receive services.

The most widely recognized benefit was the use of consumers to obtain consent. Specifically, the respondents suggested that consumers helped to increase the response rate because they made participants feel as though they could establish trust and rapport with someone who had also used the system. Participants were therefore not afraid to express their feelings. It was also reported that consent was better obtained face-to-face or when the interviews were conducted at a location convenient to the consumers. Respondents also noted that ensuring confidentiality granted researchers the opportunity to communicate ethical standards and establish trust with the participants. This also encouraged participants to speak their minds for the first time about the services they received. Finally, one respondent felt that anonymity was beneficial because it reduced the chance that participants' responses would be traced back to them.

There were several problems that respondents encountered regarding the issue of confidentiality. It was difficult to make sure that the respondent knew and trusted that it was a confidential process. At times, the process of confidentiality and consent intimidated people. One respondent, who used audio recorders in satisfaction interviews, acknowledged the difficulty of making sure that the participants knew they were being taped but that the interviewers would hold the information confidential. Another respondent noted that it was not always possible to guarantee confidentiality because there were times when personal problems would come up that needed to be identified, including issues of abuse. Still another respondent reported that it was difficult to assure confidentiality when the sample size was small and people's responses could be easily identified. It was also noted that the process of choosing people to survey often compromised the issue of confidentiality. However, another respondent noted that anonymity prohibited analysis of the correlation between satisfaction, dissatisfaction, and the wellness of the individual.

In one project providers viewed the consumer interviewers as poor "holders" of confidentiality, who had poor judgement, lacked credibility, and were trying to resolve issues of power, control, and authority. Finally, two respondents reported a low response rate representing only highly satisfied people. These respondents believed that those people who were not satisfied feared being identified and consequently chose not to participate.

• Using a face-to-face process to assure confidentiality, gain consent, and promote trust between interviewer and respondent
• Ensuring total anonymity; do not use any names or records
• Using only consumer-to-consumer interviews
• Conducting interviews in privacy
• Ensuring anonymity or confidentiality at the beginning of the study
• Selecting interviewers with capacity for confidentiality and then provide additional training. Be up-front, tell people what you are doing, maintain anonymity and the choice of refusing to participate
• Secure consent of participants
• Clearly state how the results will be used

Eleven of the projects involved consumers in the analysis of the data or interviews, while four projects reported that consumers were not involved in this stage. This information was not available for five projects. Responses from our survey indicated that the roles of consumers who participated in data analysis and interpretation varied considerably. Some consumers were involved in the survey team, putting together the questions and tool development, and then analyzing the data, while others were involved in the debriefing that followed the study. Several respondents indicated that consumers participated in collaborative to share insight about the findings, and how the findings were related to their own experiences, and then to form conclusions about how the data could reach consumers and be used to make changes. In some projects, consumer staff wrote reports, completed the data, and translated it into percentages. Of course, several respondents noted that some of the consumers were not capable or did not have the necessary knowledge to perform the analyses. In those situations, consultants were asked to analyze the data, but the consumers may have entered or assisted in directing questions to be asked. Several of the projects had not yet begun their data analysis or interpretation and could therefore not supply information about consumer involvement. However, several respondents noted the intention to involve consumers in every way possible and to train them in skills of analysis and trend identification.

Fourteen of the respondents expressed beliefs that the information collected accurately reflected the opinions of the interview or survey participants accurately. One respondent explained that they ensured accuracy in the focus groups by using a chart to help participants to visualize and verify the information that was being reported.

Of the remaining five respondents, two said that they felt the data were somewhat accurate, while three respondents expressed doubt. Specifically, one respondent noted that the survey did not get at certain issues known through discussion to be important, and results did not reveal any significant differences among the groups. Several respondents noted that there were problems with the questions being worded in a biased way, making it difficult to determine if the participants really felt as positive about items as they indicated. However, it was also suggested that the surveys could have been accurate, but tapped into only those participants who were satisfied, while ignoring those who were not satisfied. Still another respondent noted that the consumers surveyed needed support and education to identify and then express their concerns. It was also noted that because the survey responses were very subjective, it was difficult to tell whether people expressed their true feelings. Finally, one respondent noted that the survey, although it appeared to elicit accurate responses, failed to ask if and how services were linked to the real life issues of the clients.

The benefits of data analysis and interpretation were varied. Some found that the use of qualitative data enabled them to receive more quality of life information. The use of consumers at the analysis level not only proved to be an empowering employment experience for them, but it also contributed an aspect of expertise to the team. Others noted that it was beneficial to work with a consultant on the analyses, and to see how the data related to. One respondent found it beneficial to employ a non-affiliated Ph.D. to help with the data and to keep the analyses neutral. Finally, one respondent noted that using data tables helped in understanding the interpretation.

Some respondents who did not use consumers in the analysis noted problems interpreting the results. They felt that researchers needed to be sensitive to the consumer needs conveyed in the data and that this could only be accomplished with the input of consumers. However, others noted that consumers in their staff required informational, emotional, and economic resources to contribute meaningfully to the survey process. For others, analyses were very technical, difficult, and staff intensive, making the accuracy of the reporting process a challenge. As a result, some respondents reported having difficulties grouping the responses into meaningful categories that could be used to promote change in the mental health system. Finally, one respondent noted that unequal response rates made analysis problematic.

• Using trend analysis to examine what people do/don't like
• Including consumers, statisticians, and researchers to analyze and interpret data
• Using a combination of qualitative and quantitative questions
• Don't be afraid to use a long instrument; people enjoy the interaction and attention
• Employing a neutral professional researcher to handle the analyses
• Providing ongoing self-help and self-advocacy education as part of a lifelong learning process
• Teaching consumers the skills necessary for data analysis or involve them in the interpretation of data
• Replicating analyses to ensure the positive and negative aspects reliable
• Using an established survey and doing analyses that are appropriate for the survey.
• Making sure the sample is representative of the population you are trying to measure
• Completing analyses quickly, and presenting the results to groups who can effectively use the information

The projects involved in our survey were at different stages of the data presentation and distribution process. Some had not yet begun this phase, whereas others had distributed some of the information but planned to do more. Seven of the projects used data tables, graphs, or pie charts in their summaries. One respondent reported presenting items in rank order; another included only percentages in their summary. Most respondents reported that their summaries were easy to use, or specifically designed them to be easy to understand. One respondent reported presenting their information in notebook style in which each program was summarized in a section and direct statements were presented parallel to questions. Once this was compiled, the researchers met with the director to review each aspect of the report. Although no respondents indicated that they had printed the information in the newspaper, several respondents noted that some of the information would be included in a publication that went to consumers, advocacy groups, the state council, and providers. None of the respondents had put their information on-line, but three respondents indicated that they plan to do in the future. Finally, one respondent noted that although summaries of the information were effective and easy to understand, because they included both the positive and negative information; some of the internal people were upset, and education about the results were needed to promote understanding and acceptance. Ultimately, these problems resulted less publicity than hope for.

In thirteen of the projects, consumers were involved in deciding how to present the data. Three respondents reported that consumers were not involved in this aspect of the project. This information was not yet applicable to three projects.

Nine of the respondents included indicated that consumers had access to all of the data, and three indicated that consumers would have access to all of the data when compilation was completed. Also, two respondents noted that consumers would be granted access to all of the data upon request. Three respondents reported that because of the way the data was presented and summarized, consumers outside of the staff had access to only some of the data. One respondent noted that this was due in part to an unequal response rate, which made the data less applicable and representative of some sites. Another respondent, who recognized the importance of consumer access to all of the data, expressed concern that this access was not always provided. He/she indicated that although information was sent to providers, asking them to post it, it was difficult to know if providers always followed through with this request. Finally, one respondent noted that although consumers had access to all of the summarized information, these reports themselves might be twisted and manipulated in ways, which flatter the provider.

Generally, this information was distributed in summary reports or newsletters to the internal staff and other stakeholders, including consumer advocacy groups, providers, the state planning council, mental health centers, Offices of Consumer Affairs, family groups, and consumers. In some cases, the information was made directly available to the public. One respondent also reported that they plan to hold a conference to present the information in detail, provide interpretation, and follow up.

Several respondents noted that it was beneficial to provide an easily understood report that could be read and used by many different groups. Presenting and explaining the results of the project to executive directors, boards of directors, providers, federal organizations, and other stakeholders who make decisions about the programs was also viewed. This process allowed these people to hear the voices of consumers and family members who had not previously been heard. Similarly, when results were made public, people who participated could feel the impact of their involvement. In one case, interviewees who received the information showed interest in the results and expressed their gratification that people had returned to follow up and share the results. Finally, the reports were also noted as beneficial because they exemplified strong indications of necessary consumer quality improvement and then provided the groundwork for change by presenting this information to people who were responsible.

There were several problems noted with the distribution of data, including problems in ensuring that providers wanted to hear and accept the results. Consequently, for some respondents, data presentation was slower than anticipated, and may have been poorly received by people who did not read the reports. To counter this problem, one respondent explained a process that to ensure that people are receiving and understanding the information. Accountability meetings are conducted to discuss the findings and make sure that all stakeholders are informed. One respondent noted that it is important to consider the political environment when reporting the information and stressed the need to write accurately but to also be sensitive to how people will respond. Some respondents felt they needed to include more data, graphs, and tables in the future, finding ways to include all of the information that stakeholders find important, and were concerned that the data was not presented as much as it could and should have been. Finally, one respondent indicated that the results of their surveys indicated little dissatisfaction, or so that little change seemed needed.

• Presenting data in a style which is readable and friendly, with little jargon
• Issuing reports in a timely manner
• Involving consumers in the report writing
• Including consumers and family members in the distribution process
• Conducting forums or meetings to ask questions about the results
• Issuing a follow-up report to ensure that changes are made and follow up with a report on those changes
• Distributing copies of the reports to consumers
• Using a professional to write the summaries
• Comparing the results with similar assessments being conducted in the country
• Holding accountability meetings to inform all stakeholders of the results
• Issuing a dissatisfaction survey to capture dissatisfaction as well as satisfaction
• Publishing results in journals

7. Data Use and Feedback:

Nine of the respondents included in our study indicated that the information from their survey has been used to improve services. Specifically, the data was used to improve the services for consumers by making changes suggested by individuals. In one case, services were improved through increasing consumers' dignity and respect shown by staff. This was accomplished by retraining the staff to respect consumers' rights and to treat them as human beings. Several respondents noted that they submitted reports to providers, agencies, and other stakeholders, and then followed up to make sure that improvements were made. At this point, ten respondents indicated that they had not yet approached the aspect of improvement but anticipate many changes. One respondent indicated that they plan to implement a performance contract for consumers with managed care providers. This process would allow consumers to walk away from a program if it was not satisfactory, have long-term information up-front, and to choose services from year to year. One respondent noted that although a few services were re-targeted, no fundamental needs were addressed.

Nine respondents indicated that consumers participated in the improvement process. Generally, consumers were involved as members of the board of directors, in the reorganization process, or in generating ideas based on the responses of consumers. Only one respondent indicated that consumers did not participate in the improvement process. Seven respondents had not reached this stage and planned for consumers to be involved but did not know the extent that of involvement. Two respondents indicated that consumer involvement was typically left to the discretion of the provider. In one case consumers were required to be involved by quality improvement committees.

According to the respondents in our study, there were many improvements made as a result of the surveys, including medication education for nurses, a change in the focus of skills that consumers were being taught, the implementation of a new job program to help consumers seek jobs, the establishment of "comfort" rooms, installation of bulletin boards for information, establishment of a library, new activities, and the introduction of a consumer advocate. In addition, there were more basic improvements, including changes in temperature and room maintenance, consistent medication checks, changes in the gowning practices at a children's facility, a policy for grievance procedures established, and increased telephone. In addition, it was felt that consumers were given more voice in establishing programs as a result of the surveys. One respondent expressed hope that an increase in consumer participation will help incorporate a new understanding of recovery and empowerment, make services more cost-effective, and provide for a healthier and happier client.

Respondents indicated that changes were most beneficial when they were made promptly, indicating to the consumer that their voices are important. It was also beneficial to know exactly what consumers want from their services and how to improve them. Several respondents noted an improved sense of empowerment for consumers who witnessed changes and knew that their voice had made a difference in their lives and the lives of others. One respondent reported that it was beneficial to identify and reveal things that providers were not aware of, while another respondent indicated that it was beneficial to know what services consumers were satisfied with.

Although the benefits of improvements were extensive, there were several problems noted as well. Specifically, at times it was difficult to implement change quickly because of the time or funding involved. In addition, one respondent noted that it was risky to identify changes that needed to be made, because the program would risk losing funding. Interestingly, one respondent reported facing a tough political environment, which limited the amount of funding and change made within their system. Several respondents noted similar forms of resistance from people in management and providers, and one respondent expressed an irritation because of his/her belief that the survey was distributed as a mandated formality without any intentions to make fundamental improvements. Additional problems included convincing skeptical people of the value of projects, determining how much information to share with people (specifically those who did not participate) and teaching consumers in stakeholder positions how to use the information effectively to bring about lasting improvements.

• Ensuring follow-up with prompt and effective changes
• Guaranteeing that advocacy groups have all of the information, so that they can advocate for useful changes
• Handling changes in a partnership fashion with both consumers and system stakeholders
• Having an ongoing system of improvement
• Redesigning the system to focus on outcome measurement
• Equipping consumers with the necessary organizational and political resources
• Encouraging professionals to view consumers as an integral part of the assessment process, beginning with their participation at the local level.
• Involving people who are responsible for services in feedback and change
• Sending reports to people who were surveyed
• Determining whether follow-up changes have been made
• Ensuring that consumers are aware of the direct effects of their participation
• Analyzing and improving services for individual centers and for the entire system
• Ensuring that every point of the assessment and improvement process is evaluated by consumers

As demonstrated by these respondents, the involvement of consumers in every aspect of the assessment process is considered the fundamental best practice when conducting consumer satisfaction surveys. Consumers have previous experience with the mental health system and an inherent appreciation and understanding of the emotions of fellow consumers which allows them to contribute to the survey process in ways that non-consumers cannot. Consumers tend to be greatly invested in the projects they become involved in because they understand the importance of improving the situations and experiences of fellow consumers. In addition, they may possess a more diverse understanding of the mental health system, because they are exposed to different aspects of the services than are providers or other non-consumers. Therefore, consumers may have an advantage when developing survey questions, administering the survey, interpreting the data, and proposing changes for the system.

To effectively use consumers in the survey project, adequate training needs to be conducted. Although there was considerable variety in the types of training programs people used, the most frequently noted activities included role-playing, a manual, and information on report writing and analysis. Throughout the training process, it is important to have patience with and understanding for each consumer participant and to provide support and continuous training if needed. Although some projects found a team approach useful when conducting training and the surveys in the field, other projects were successful with using individual interviewers. Therefore, choice of the team or individual model should be determined by the preference of the consumer interviewers and what they believe would be most effective to implement their tool and obtain responses.

Overall, consumers were not only beneficial to the projects, but they also experienced many benefits, including increased skills, self-esteem, knowledge, and, in some cases, a financial opportunity. Although it was difficult for many respondents to provide prolonged employment opportunities, other respondents were quick to realize that this was a great opportunity for consumers to help people while learning new skills, regardless of the duration of employment.

Another frequently mentioned best practice was the use of consumes to conduct interviews face-to-face. This technique serves to personalize the interviews/surveys by conveying to the respondent that people are genuinely interested in their opinions and, ultimately, this increases response rates. However, it is important to note that not all participants may be receptive to face-to-face interviews, because such interviews are time consuming and difficult to schedule. In addition, in some areas confidentiality is a continuous concern for interviewees, making it difficult for interviewers to receive honest responses. Therefore, it is important to consider telephone interviews and mail surveys to reach people who may need to voice their opinions anonymously, or who may experience other barriers to face-to-face interviews. Stakeholders make adequately assess the needs and restrictions of their population before beginning to develop the survey so that they can determine the necessary methods to reach their target group.

Survey development continues to face problems with wording. Many of the respondents in our survey are constructing their own tools. It is possible that they will improve previous studies by asking more relevant questions important to consumers. Survey length was not a problem for survey participants. They appreciated the opportunity to be heard. On the other hand, long surveys were also considered problematic for several respondents who were working under deadlines. One possible solution is to institute a system of ongoing assessment, which focuses on gathering quality responses rather than on the quantity. In addition, ongoing assessments appear to provide a more accurate view of consumers' needs and opinions.

Once the survey process has been completed, it is clearly important to follow-up with sessions that explain the results of the survey to the stakeholders, including consumers and providers. This data sharing process demonstrates to consumers, the importance of their voices. It also involves all people in the improvement process, an involvement that ultimately, will promote greater understanding and appreciation of consumers' experiences.

Finally, as demonstrated in this review, there are many successful satisfaction survey projects working to improve the lives of consumers. Our intent is to provide a guide to facilitate best practices and to help improve surveys by sharing information.