The broad long-term objectives of this project was to develop measures to evaluate, research and improve the delivery of mental health, substance abuse, and domestic violence services in the state of Oklahoma. The project is funded under a State Mental Health Reform Grant to integrate, synthesize, and distribute data. It began with the development of a comprehensive list of treatment and support measures from the perspective of the consumers of the Oklahoma Department of Mental Health and Substance Abuse Services. The second stage involved the construction of satisfaction and outcome measures based on consumer identified items to measure outcomes of services and satisfaction. Various modalities of data collection based on the consumer-oriented measurement tools of outcomes and satisfaction of service recipients is now underway. Results of the data will be distributed as a report card for consumers of the various agencies.

In "Caring for People with Severe Mental Disorders: A National Plan of Research to Improve Services," the NIMH identified the study of the impact of services provided-whether specific treatments, broad social interventions, or both-as one of the most critical elements of clinical services research. However, the national research plan acknowledged that such studies of outcome and satisfaction are not easily accomplished due to the variety of complex ways serious mental illness manifests itself, including not only deteriorations in intellectual and emotional functioning, but in "marred social and vocational adaptation, the destruction of well-being and personal potential, and massive impediments to normal existence in the community." Difficulties in assessment of such factors, development of adequate instrumentation, and research designs that would permit valid and reliable conclusion challenge mental health service research (NIMH, 1991).

Further, in a review of research literature that compared perceived needs and goals of mental health consumers with mental health professionals and caregivers, it was found that people who have experience mental illness and are/have been the recipients of various treatment modalities, perceive traditional psychiatric interventions and supports in significant domains of everyday life and mental health services very differently (Ridgway, 1988). One study of patients and professionals on a psychiatric ward found that the staff viewed patient "insight" as the primary goal, while the patients themselves placed "insight" at the bottom of their list of goals. Patients wanted specific means to adapt to their situation, symptom relief, or other goals that could not be addressed through psychotherapeutic interventions. The researchers concluded that if the patients were treated in a milieu that was consistent with their goals, client satisfaction would be increased and length of hospitalization reduced (Dimsdale et al., 1979).

In general, mental health professionals and caregivers are trained to focus on pathology and incapacity, and to practice mental health care as a specialized intervention. For example, many rehabilitation approaches consist of services aimed at remediation of a series of deficits (Cook, 1992). Also, individuals diagnosed with mental illness are generally viewed as needing different things than "more ordinary" people (Wilson & Blanch, 1987).

In survey research on what factors promote or deter the well-being and quality of life of people receiving mental health services, consumers reported that their needs for respect, dignity, and choice were more significant than those treatment needs identified by mental health professionals as primary. Consumers appear to have a more global world view of mental health and recovery that encompasses the needs to recognize and nourish personal strengths and support a quality life (Campbell & Schraiber, 1989).

Clearly, mental health consumer perceptions about treatment efficacy and satisfaction with the type of treatment received, as well as the way it is delivered, determine which treatments are sought or complied with and which outcomes are valued (Mirin & Namerow, 1991). Considering the rising cost of mental health care and shrinking social resources, the results of treatment as evaluated by consumers and their subsequent help-seeking and coping patterns as related to service utilization and recovery are critical to any evaluation of mental health services.

However, the perceptions of consumers as a generic category have in general long been ignored in public-sector services, and in particular, in the research and practice of medicine. In addition to tensions in group or individual perceptions and role definitions, the philosophy and interventions based on Total Quality Management have clearly defined outcome measures and product satisfaction, rather than primarily designing research, programs and evaluations based on system needs, goals, and outcomes.

This strategy is particularly salient when assessing the effects of mental health services and institutionalization and the impact of those effects on the lives of persons diagnosed with mental illness. There are many assumptions underlying every outcome measure that determines the expected results of services, how service effects are to be measured, when effects should be measured, who should be measured, and how data is analyzed (Cook, 1992).

According to consumers, the assumptions of traditional mental health systems have been grounded in the pathologizing of problems in living, low expectations of consumer achievement, paternalism, limited range of options, and the definition of anger as symptomatic, among others (Focus Group Meeting on Client Outcomes, June 2, 1992). Often researchers lack insight to ask the questions that would capture detrimental effects of treatment and care. In a review of quality of life instruments, for example, only studies developed by consumers measured personal freedom and decision-making power were found even though these factors represented deep-seated cultural values in American society (Campbell, 1992). Such studies reveal that consumers often approach traditional mental health services with ambivalence and fear. The treat of involuntary treatments, subtle forms of coercion and lack of respect towards consumers by mental health professionals and providers, and debilitating side-effects from medications are some of the key concerns identified by mental health consumers (Campbell & Schraiber, 1989).

On the other hand, in the past few years interest and support of research on service priorities and desired outcomes o mental health consumer, the relationship between consumer and provider/researcher outcome incongruence, and the rejection or resistance of mental health consumers to traditional treatment and services has grown. In a series of focus group sessions, the Consumer/Survivor Mental Health Research and Policy Work Groups began a systematic articulation and exploration of consumer values and outcomes. Recovery, personhood, well-being, and liberty were identified as relevant outcomes that are seldom measured or operationalized in traditional mental health research or program evaluation (Consumer/Survivor Mental Health and Policy Work Group Task Force, June 2, 1992; July 13, 1992; September 30, 1992).

Given concern about the diversity of persons who come to be recipients of mental health services, plans were developed by the Consumer/Survivor Outcomes Task Force to conduct fieldwork in various settings. Concept-mapping was selected as a method for inquiry because it is a structured, replicable process that is both participatory and democratic. Each individual has the opportunity to provide crucial input at the various steps of the process. Concept-mapping has been utilized in mental health planning and evaluation context (Trochim, 1989). Dumont (1989, 1993) found it a useful and valid tool for theory development and measurement with persons who are consumers of the psychiatric system.

One concept-mapping pilot was supported by the National Association of State Mental Health Program Directors (NASMHPD) Research Institute to provide input into the State Mental Health Agency Profiling System. Seventeen consumers brainstormed 98 statements that described "...specific consumer/survivor defined individual and/or system outcome indicators or measures that should be part of a mental health system of measurement." The major classes of outcomes involved those related to mental health system issues, control and voluntariness, personhood (including the damaging effects of a system on individuals), the inner process of healing, and life in the community (emphasizing quality of life). Participants assessed the methodology as useable with other consumer groups for further instrument development with the caveat that adaptations might be needed, particularly if used with persons currently on hospital wards.

Progress was also made elsewhere to develop outcome measures from the perspective of recipients of services. For example, the Accreditation Council (1992) developed value-based performance measures after initial input through focus groups from people with disabilities in one state program. Another effort in California developed quantifiable client outcome measures that were value-based (personal communication, Kathy Styc, 1993). However, there are at least three important ways-besides the method-in which the development of these measures differed from the Oklahoma State Mental Health Reform project: In the Oklahoma effort, consumers of services were the primary researchers and participants; sites were selected for different demographic groups of recipients of different types of services; and the development of measures included a synthesis of results across sites.

The Oklahoma project combined multivariate statistical methodologies to achieve pictorial representations of peoples' thinking to find out: (1)what outcomes, positive and negative, consumers of services think ought to be measured; (2) what are the relationships among the identified issues; and (3) how to operationalize the results of the mapping-sessions? The methodology used was concept-mapping (Trochim & Linton, 1986) in a mental health, substance abuse, and domestic violence context with a discrimination between male and female participants.

The mapping involves three steps: generation, structuring, and representation. The notation for the design in this study is described in Trochim & Linton (1986) and is considered highly structured because each of the three process steps is performed separately. Notation for the design is:

7. (S) (R) p(S) P

where

7. is the generation of conceptual domain by group brainstorming

19. is the structuring of the data by individuals in the generation sample, i.e. unstructured sorting and ratings by individuals

18. is the representing of the structured data by algorithmic techniques producing the representation, i.e. statistical programs for multidimensional scaling and cluster analysis

p is the pictorial representation combining multidimensional scaling and cluster analysis

19. is the group structuring of the representation by the generation sample, i.e. naming of the clusters and regions by the group

P is the final group concept map or pictorial representation.

These steps were followed from each of the three sites rendering three pictorial representations or group concept maps. One site (family members of people with mental illness) did not participate due to insufficient participants.

SAMPLES:

The study populations consisted of three purposive samples chosen from two service types: inpatient substance abuse and domestic violence shelter. The study populations were chosen to capture special populations, culturally or racially diverse groups, women, and people with comorbid problems to reflect diversity and to seek shared perspectives within this diversity. Participants were selected based on the following criteria: (1) they were current consumers in a defined service/program; and (2) they shared the goal of wanting to participate in the development of measures.

The intent was to include approximately 15 persons per groups to generate and structure the data and interpret the maps.

A specified group conceptualization process described in Trochim & Linton (1986) was used to develop satisfaction and outcome items from the perspective of service recipients. Recipients responded to the question, "What things do you like or dislike about the services and the providers of the services?" Five groups were selected to participate in the concept-mapping groups. They included: male and female substance abuse service recipients at the Norman Alcohol and Drug Treatment Center: survivors of domestic violence at the Young Women's Christian Association facility in Oklahoma city: a group of mental health consumers with serious mental illness at the Thunderbird Clubhouse in Norman; and parents of people with mental illness at the Oklahoma Alliance for the Mentally Ill. The parent group was unsuccessful due to lack of attendance, and the mental health consumer group failed because the mapping sessions as organized was too complex for the participants to complete.

Each mapping process resulted in two products. Brainstormed items were ranked and then presented in descending order for each group. Maps which were pictures of the thinking of the various groups about the brainstormed items provided a conceptual framework. The maps enabled the Oklahoma Department of Mental Health and Substance Abuse Services to see the aggregated similarities and differences in relation to each other. The picture revealed not only what participants know they think but also thinking they may not be aware, particularly the thinking that shifts when trying to incorporate diversity (Linton, 1992). The maps captured how the groups thought as well as indicated on what underlying dimensions participants organized their thinking. Therefore, the maps and the rank ordering could be used as a tool for developing measurement in this context (Dumont, 1993; 1988).

The method as adapted involved four steps: generation or expansion, structuring or contraction, analysis and interpretation. A computer software program, Concepts System has been developed (Trochim, 1989) to process both text and numerical data generated by the first two steps. The focus group input was accomplished in three and a half hour sessions.

• Step 1: Generation or Expansion

In this step, participants created the conceptual domain. They contributed the ideas that constituted the meaning of the concepts satisfaction and outcomes of services. Brainstorming was used in order to encourage participants to let go of strictly organized approaches and get the broadest possible view of what is involved in the concepts. Ideas in the form of phrases or short sentences were brainstormed first individually on paper, and then by the group by voice, and then recorded on a flip chart and in the computer program. A time limit (approximately one hour) was set for the actual brainstorming. Before moving to the next step, the groups were asked if they thought minimal editing of the ideas was needed. If so, it was done at this time.

• Step 2: Structuring or Contraction

In this step the participants organized the ideas. This was done in two ways: by sorting, and by rating. The procedures for sorting and rating were modified to accommodate the needs and abilities of participants.

The ideas generated by the brainstorming in Step 1 were printed on paper in mailing label format. Each participant was given a listing of the statements laid out in mailing label format and asked to cut the listing into slips with one statement on each slip. Each participant then sorted the statements into piles in any way that made sense to him/her. The only exception is that one could not put all the slips in one pile or make each pile a single slip of paper. After sorting the statements, each participant was asked to place each pile in an envelope and write the name of each pile on the outside.

Participants were given a list of the brainstormed ideas and asked to rate each one as to level of importance on a scale from 1-5. Asking participants to rate for importance , even relative importance, was somewhat problematical and not highly discriminating. The cluster average ratings ranged in value from 3.70-4.21.

• Step 3: Analysis

Multidimensional scaling and cluster analysis was used on the sorting data. This analysis relies almost entirely on mathematical algorithms programmed in the Concept System, although there are some analytical decision points with respect to the number of clusters that best represents the thinking of the group. The consultant entered the data and examined the results to select the number of clusters. Multidimensional scaling locates the ideas on a two dimensional map by searching for the joint occurrence of all possible pairs of ideas among all the members piles. The final placement of ideas on the map (x, y coordinate values) reflects the level of agreement among sorters as to which ideas make sense to put together and which to separate. The closer the ideas appear together on the map, the higher the level of agreement reflected.

A hierarchical cluster analysis procedure successively divides the multidimensional scaling output. Somewhere between seeing the map as one gigantic cluster to seeing every point as a cluster unto itself is the object of this analysis. The Concept System program allows the analyst to purview the clustering procedure in sequence or select at random or by rule of thumb a cluster resolution.

• Step 4: Interpretation

The consultant and the Department explored the meaning of the map in this final step. An interpretation sessions included a review of the process to date, an explanation of the information to be used in the sessions, a process to name the clusters, a process to identify regions of the map, a discussion of relationships of individual ideas, cluster of ideas, and regions, a view of the total map, a discussion of the fit of the map to the group's understanding of its view of the concept, and a discussion of how the group wanted to use the conceptual framework.

Following the four steps, the consultant wrote up the interpretation and participants comments about the method. In each of the mapping sessions the following products resulted: (1) a list of the brainstormed ideas; (2) cluster names for the ideas; (3) region names; (4) a synthesis of the discussion of the meaning of the inter-relationships among the ideas and larger concepts; and (5) means, medians and standard deviations for the ratings of the individual ideas and clusters.

After the group conceptualization process occurred at the three sites, the consultant compared results across the sites and the Department aggregated the findings based on a process of categorization and the ratings.

The instrument development phase involved two activities. A list of 60 items from the MHSIP Report Card (CMHS, 1996), the Oklahoma Mental Health State Planning Council, the Florida BHRS (Dow & Ward, 1996), the Texas consumer survey, and the WICHE Report Card were generated in order to develop and implement a behavioral health report card. In addition to the outcome measures, the Department plans a brief discharge survey of all clients, and an in-depth, face-to-face survey from a statewide, random sample. The results by treatment agency of the outcome study and satisfaction surveys are to be disseminated in the report card. Further, from the concept-mapping sessions, the top-rated focus group issues to enhance well-being and recovery were classified by Venita Johnson, data analyst, at the request of Rock Richardson of the Planning Division, and a report was prepared for the DMHSAS Regional Advisory Boards (August, 1997). The report was developed to "...assist Regional Advisory board members in assessing the strengths and weaknesses of the regional service delivery system and promote local involvement of stakeholders in problem-solving." The ranked items were sorted by Department staff into the categories of "Resources", "Education" and "Policy" as the types of solutions needed to resolve the issues. The top ten items in each category for each of the populations were identified. "Resources" related to the accentuation or expansion of services, "Education" to increasing community awareness, and "Policy" to existing staff, rules, regulations, policies and practices.