This document (101 KB) is viewed best with Netscape Navigator/Communicator. Those using the Microsoft Internet Explorer web browser may experience difficulties viewing and reading the entire document.

TABLE OF CONTENTS

Background

The Oversight Board for the multi-site study of Mental Health Service Use, Needs, Outcomes, and Costs in Child and Adolescent Populations (UNOCCAP) had a dual charge (A roster for the Board is presented in Appendix A). The first was to provide scientific guidance and administrative advice to the UNOCCAP investigators as they finished pilot and feasibility work (See Appendix B). This task is now completed, and the necessary work is underway and important products are anticipated. As the UNOCCAP pilot work draws to a close, the National Institute of Mental Health (NIMH) needs to reassess its action plan for charting the mental health status and service needs of children.

Drafting an action plan was the Board's second charge and the impetus for this report to NIMH and the National Advisory Mental Health Council (NAMHC). Specifically, Dr. Hyman, Director of NIMH and Chairperson of the NAMHC, asked the Board to assess the research needs regarding the prevalence and etiology of children's mental health problems and the use, need, quality, and costs of service associated with these problems.

It is the Board's hope that this report will spark a spirited discussion within the research and advocacy communities. These discussions should identify complementary, practical, and productive approaches to addressing important and urgent research questions concerning the responsiveness of this Nation's mental health service system to the needs of children.

Consensus must be developed quickly since the science is ready to address many of these critical research questions. A sustained program of research, such as that proposed in this report, carried out in conjunction with the Institute's research portfolio in the treatment and prevention of childhood mental disorders, remains a timely investment that promises great benefit for children, for their families, and for the future of the Nation.

Identifying and Addressing Critical Questions

Efficiently advancing this research agenda will require a fresh start based on identifying important questions that the field can address rigorously. The Board chose to focus on significant questions facing policymakers as the extensive changes across this Nation's mental health, education, and justice systems unfold. Interestingly, these policy-driven questions are completely consistent with scientifically important questions:

1. What are the frequencies of symptoms and impairment in U.S. children and adolescents?

• How do these frequencies vary over time and with changing policies?
• How do these frequencies vary across demographic factors such as age, socioeconomic status, location, gender, and ethnicity?
• Which symptoms cluster to form patterns or diagnostic groups?

1. What is the service use associated with various symptoms and impairment?

• How does service use change over time?
• How does service use differ across demographic groups?

1. A. What are the costs of mental health services to youth in treatment?

• How do costs vary by demographic groups?
• How do costs vary by mental health problem?

• What are the effects of managed care on service use and costs?
• What benefit packages are associated with variation in service use and cost?
• How do parity laws affect the cost of treating children?

1. What are the paths into and out of disorder and into and out of service use?

• How do these paths vary across different demographic groups?
• Are the paths different for different disorders?

1. Which services are effective for which children and adolescents?

• How does effectiveness vary over demographic groups?
• Who are the children who need, but do not receive, effective services?
• What are the costs of effective services?
• How does quality vary across settings and provider characteristics?

Developing an Action Plan to Address These Questions

The path for answering these questions will be difficult. First, no one study can address each of these questions well simultaneously. There is no way to combine these questions into one method or design without sacrificing the quality of data. Second, the many disciplines that are allied in children's mental health research are not yet ready to answer each of these questions. Some areas require additional conceptualization, better instrumentation, or more empirical work to generate or test hypotheses. This critical basic research must be conducted to lay the groundwork for answering all of these questions.

The Board's recommendations, therefore, were shaped by both the urgency with which these questions must be addressed and by the realities of today's scientific abilities. The guideline was a simple maxim: if it's ready, use it; if it's not ready, develop it. This guideline resulted in a program of research, each component of which is ready for initiation today. The program, with new, complementary initiatives spanning large-scale surveys and fine-grain developmental assessments of psychopathology, establishes:

• A mechanism for answering selected priority questions with the best available tools
• The vehicle for constructing the needed tools and methods for answering all of the remaining priority questions
• A rich data source to speed the process of scientific discovery and policy-making in child and adolescent mental health

The Board recommendations call for a program of research having multiple components. Some of the components are designed to stimulate new concepts and measures crucial to elaborating our understanding of children's mental health. Some of the components are designed to provide a monitoring of prevalence of mental health impairments and related service use, as a national indicator of the seriousness of the phenomenon. Some of the components are designed to inform scientific inquiries concerning pathways into mental health impairments. Some of the components are designed to measure the costs of services to those using them. Thus, the recommendations attempt to enhance the science side of children's mental health, supporting research addressing causal hypothesis, at the same time as they serve basic informational needs of the society about the magnitude of the problems faced within the child mental health domain.

NIMH Role

NIMH must provide qualified staff, access to expert and timely peer and administrative review, adequate budgets, and competition through appropriate funding mechanisms. The staff must ensure that these activities are supported in economical, yet feasible ways. In particular, the Board encourages NIMH to begin looking for a cosponsor for these activities, either Federal or private. There could be significant cost savings on large-scale efforts by imbedding the NIMH questions within on-going surveys. Further, a suitable partner could provide NIMH with much needed expertise in survey methodology.

Recommendations to Establish a Program of Research

• Recommendation 1: Initiate an ongoing national survey to monitor children's symptoms, impairment, and service use.

The Board concluded that questions on prevalence and service use are best answered, and are ready to be answered, at a national level through a large-scale survey. A rigorous study is required to describe symptoms, symptom patterns, impairments, and service use in a sample large enough to reveal important sources of demographic variation in the United States. It is of critical importance to launch this study soon, since valuable time has already been lost for informing the policies and strategies that are changing service provision and financing. The Board also places great emphasis on NIMH creating the necessary infrastructure to initiate this activity immediately.

The Board recommends a national survey, employing cross-sectional samples to monitor the frequencies of symptoms and impairments, as well as service use. Each assessment wave would have a new sample of children administered a minimum core set of measures (i.e., symptom, impairment, service use, and demographic variables) and a developmental module of test questions, estimated to take 20-30 minutes of the respondent's time. The first year, and perhaps every five years or so, a benchmark assessment would be conducted that would contain the core battery and an augmented set of assessment measures (estimated to be 90 minutes for the full symptom and impairment checklists, service use measure, and demographic variables). The benchmark assessments would provide more detailed information and allow for the recalibration of the core measures with better questions from the developmental modules. In addition, topical modules could be included, to assess critical scientific or policy issues.

This ongoing monitoring design would provide a cost-efficient platform for: 1) monitoring symptoms, impairment, and service use nationally and revealing variation over time and demographic groups; 2) generating unanticipated inductive hypotheses about symptom patterns and diagnostic categories; 3) providing a time series of indicators to help determine the effects of policy changes; and 4) testing and refining new measures as they mature (e.g., symptom patterns or diagnostic categories).

• Recommendation 2: Initiate a national survey of the type of services used by children in treatment, the costs of these services, and how benefit coverage affects service use and costs.

The proposed monitoring survey in Recommendation 1 will not capture a significant number of children who use mental health services and therefore will not be able to examine the factors associated with more costly services. Therefore, the Board recommends conducting a national survey of children who use mental health services to determine the overall use of these services, the costs of these services, and the factors associated with the use and costs of services. This separate survey should be initiated immediately.

• Recommendation 3: Develop infrastructure activities to support the two large-scale surveys in Recommendations 1 and 2.
• Once collected, the national survey data should be released to the field, after ensuring the participants' confidentiality.

• A program announcement should be issued for secondary analysis of the survey data.

• Recommendation 4: Establish an initiative to enrich the conceptualization and measurement of childhood mental problems and impairment from infancy to 18 years of age.

Conceptualizing and measuring children's mental health problems require a sustained effort by NIMH. Although the Institute has funded work in this area, it is now time to foster multiple approaches to this issue through consensus activities, theoretical work, development and testing of measures, and comparisons between measures and theoretical models.

• Recommendation 5: Establish an initiative to further the understanding of emotional functioning and mental health problems in young children.

As difficult as the area of child and adolescent psychopathology is, of particular concern within this area is the study of the young child, where efforts are just beginning. Measures of normative functioning are being developed and tested, but what they mean for children's mental health and later problems is unknown. NIMH is encouraged to develop an RFA for stimulating research in this key area, which is ready to move ahead now.

• Recommendation 6: Establish a program of local or regional longitudinal studies in three separate areas: 1) paths into and out of disorder; 2) paths into and out of service use; and 3) effectiveness and quality of mental health services.

Longitudinal research has been consistently supported through the Institute, but the portfolio should be reviewed for what is known and moved toward synthesizing and testing these findings regarding risk factors for disorders and for service use. The Institute must consider and direct its efforts toward understanding these developmental processes.

Understanding how the provision of service improves outcome is an area of research that also requires longitudinal study. Children in services must be tracked to see if they do get better or if further impairment is prevented. This work must begin immediately because it is a prerequisite for understanding and indexing the service need and service quality (i.e., how effective services are provided in a particular service sector or in a sector site).

Because of the need for deeply textured studies and for tracking participants over time, these studies are proposed for local or regional work rather than large-scale survey approaches.

• Recommendation 7: Establish a program to foster innovative methods or analyses in child mental health.

Questions of continuity and change remain intriguing for all of the life sciences and for policy development. Innovative methods for conceptualizing, measuring, and analyzing these processes should be fostered at NIMH. Current efforts should be augmented with a new and immediate emphasis on supporting the infrastructure for methodological, statistical, and software development.

Overview

Lessons Learned

The Board had a rich context for identifying the lessons learned from UNOCCAP. The reader should note that the lessons are being reported with the benefit of hindsight. Even though the Board is serving as an "arm-chair quarterback" in this capacity, the lessons should be shared. They were learned through the pursuit of an important goal: NIMH was working to conduct large-scale science. Projects of this scope are relatively rare for NIMH. NIMH must build its capacity to develop and conduct such large-scale projects to generate the type of data that can inform clinical practice and service systems.

The overarching lessons for NIMH are:

• NIMH leadership must recognize its relative inexperience in large-scale ventures and seek consultation on future efforts from other NIH Institutes and Federal agencies that routinely conduct such science.
• NIMH staff must have the necessary experience with and expertise in large-scale survey methodology and program management. This knowledge must be available among the staff participating in scientific or in administrative roles. Among this group of experts, there should be one NIMH leader.
• The team of grantees or contractors must also have one leader to sustain the scientific vision for the project and make administrative decisions.
• Multiple vehicles to ensure accountability must be established for both the NIMH and the extramural leader, such as the NAMHC, a project advisory board, and peer review.
• One study cannot address all of the questions in children's diagnostic and service issues. Even national surveys are bounded by feasibility issues and by limits on statistical inference.
• Each research question must be answered through appropriate methods that reflect the best that science has to offer.
• Some areas do not have the necessary tools to address rigorously the questions posed, and resources must be devoted to developing and refining these tools.
• Mounting a national, one-time survey, where data sharing is postponed for years, places undue pressure on the participating investigators, the field, and the Institute.

The last point requires clarification. If NIMH plans one national survey as the only such large-scale initiative that the field will see for decades, then this puts a huge burden on the eventual design to be all things to all people. There is a great temptation to address too many research questions.

More important, science is not ready for such an approach. Quite simply, the necessary conceptualization, tools, and designs are not uniformly available in all substantive areas. If resources were diverted into prematurely conducting such an immense and elaborate effort, the Board is concerned that it would generate data that policymakers should not use and that a substantial portion of the researchers would not accept as credible.

Process for Identifying and Prioritizing Research Questions

Navigating through the substantive areas to specify recommendations was far more difficult than recognizing the lessons of UNOCCAP. Each domain in child and adolescent mental health needs large, well-documented data sets. But as said before, each area is not equally ready to embark on such a costly mission. Some areas are better served by devoting resources to conceptual development, creating or refining measurements, or collecting richly textured data.

The guideposts of utility, readiness, and building the science shaped all of the Board's considerations. The remainder of this chapter is devoted to exploring these issues in each of the target areas specified by Dr. Hyman in his charge to the Board. The following portion provides a description of the steps the Board followed in generating and prioritizing the policy questions that can be answered in a scientifically meaningful way.

In each area, the reader will see how the Board followed certain steps, similar to a task analysis, to generate their conclusions. The Board offers these steps as a helpful template to assess the feasibility and readiness of the future large-scale studies at NIMH. The Board has also outlined the products that should be expected to serve as briefing documents for the Council, NIMH leadership, and as the historical record.

What Does Ready Mean?

The steps outlined above ask for repeated determinations about readiness, yet defining this term is not simple. Ready does not mean perfect or even finished. Ready means serviceable. For a large-scale survey, the value rests with the application of standardized measurement to vast numbers of persons. Inferential power to the U.S. population and important subpopulations is the chief asset of surveys. With this in mind, and acknowledging the large costs of a national survey, most fields choose to use instruments that have passed the test of time. This does not mean that the measure is flawless, but that its psychometric properties are known.

Cutting-edge measurement is not often tried in large-scale efforts because this approach risks large sums of funding and may potentially misguide a field for years. Typically, smaller-scale studies are used to test cutting-edge methods, leading to the understanding of the measure's psychometric properties. Then, researchers can decide the value of the instrument for inclusion in a later large-scale or national survey. But how does one balance the excitement of the newest measures or technologies with the need to commit to a level of technology for initiation of a project?

The space program provides a clarifying example. When Neil Armstrong walked on the moon in l969, it was not the science of 1969 that put him there. A decision was made years earlier that the science was good enough to send an astronaut to the moon and allow a safe return. With this decision made, the technology was "frozen" so that the ship and its many systems could be built. Even though science marched on and new technologies were incorporated into subsequent moon walks, a commitment to a satisfactory level of technology for a particular project was made so that planning, testing, refining, and retesting could occur.

In summary, this technological "freeze" is necessary for initiating a large-scale project. Work for better methods continues along its own course and will be utilized in a timely manner, but the large-scale project is not delayed for the purpose of including these ever improving measures.

The remainder of the chapter contains a section on each of the substantive areas Dr. Hyman asked the Board to consider and a brief review of survey methodologies.

National Prevalence Rates of Childhood Mental Health Problems: What to Measure?

The Board repeatedly debated the need for national rates of children's mental health problems. The outcome of the debate hinged upon two interrelated questions: what measures of mental health status were ready to be used, and then what would the value of these data be at a national level?

• Measuring Mental Health Status

Some definitions would be helpful first. Mental health problems can be measured in many ways. As previously discussed, when a child's mental heath or behavioral problem can be classified as a disorder or diagnosis remains unsettled.

Here are the key concepts in measuring mental health problems:

Symptom. A symptom is the physical or mental expression of an illness. In mental disorders, symptoms can be abnormalities of emotions, thinking, behavior, or relationships. In a few instances, a single symptom is sufficient to define an illness or a diagnosis (e.g., elective mutism), but most illnesses are recognized by the expression of a reproducible cluster of symptoms.

Symptom patterns. Symptom patterns refer to clusters of symptoms that are highly correlated with each other. Symptom patterns should be reproducible over time within the same person and between persons believed to have the same illness in terms of frequency, intensity, and duration.

Impairment. An impairment is the reduction or deterioration in the capacity to perform a valued role. Impairments can be judged using three criteria: educational, social or vocational restriction; interference with development; and/or negative effect on others. Impairments occur in work or social functioning associated with being a family member, an employee, a student, or a companion. In short, impairment is a measure of the extent to which symptoms interfere with full functioning.

Diagnosis. A specific diagnosis should consist of a symptom pattern and level of impairment that warrants therapeutic concern due to the frequency, intensity, or duration of the symptoms or impairments. Attempts have been made in community surveys to classify children into different categories of disorder, and some diagnostic categories appear to be of utility for both the researcher and the clinician.

The challenges to measuring the prevalence of childhood problems within an acceptable range of precision and accuracy are considerable. Instruments conceived from a categorical perspective may be most appropriate for problems that lend themselves to such nosological distinctions, while instruments such as symptom checklists may be most appropriate when the task is to discriminate deviancy from a population norm. In considering this diagnosis-versus-symptoms quandary, several other realities such as the use of multiple informants and the need for documented reliability and validity must be recognized.

The Board determined that, as discussed above, too many of the childhood diagnoses remain under-conceptualized or not easily measured. The Board therefore turned to the use of symptoms and impairment as ready and valid measures for indexing children's mental health problems. The Board found that using current symptom and impairment measures can provide an urgently needed starting point. Once collected, the "building blocks" of symptoms and impairment can be examined to look for symptom patterns and to establish or test possible diagnostic categorization.

Counting and Monitoring

The Board also determined that generating national frequencies of symptoms and impairments, or prevalences, was of value. Nonetheless, what is even more important is the ability to provide policymakers and researchers with indicators of change in the population. Tracking these changes in the population may reveal how new policies affect children and their care. In addition, differences in problems among demographic groups are of concern. Such variation in the frequencies of mental health problems across age, geographic, cultural, ethnic, socioeconomic, and gender subgroupings will interest policymakers and scientists.

The Board identified the critical questions on prevalence as:

What are the frequencies of symptoms and impairment in U.S. children and adolescents?

How do these frequencies vary over time?
How do these frequencies vary across demographic factors such as age, socioeconomic status, location, gender, and ethnicity?
Do these symptoms cluster to form patterns or diagnostic groups?

The Board referred to this need to follow children's mental health status through measurable indicators as the monitoring function. As with all social indicators, such tracking must be done through repeated surveys, but it need not be done on the same children over time.

Service Use

The past decade has witnessed a remarkable transition in the financing, organization, and delivery of mental health services in the United States, as public and private payers have sought to decrease costs and improve accountability in mental health care. These include the development of comprehensive provider networks, discounted fees or other risk sharing payment systems, and intensive utilization management which together make up the key components of the heterogeneous plans known as managed care organizations. Although most policymakers believe costs are no longer increasing at double-digit rates, it is unclear if the quality of care has improved or deteriorated. Similar privatization efforts are starting in other sectors serving children and adolescents. Utilization management and capitation arrangements are being organized for child welfare, juvenile justice, and educational programs for children and adolescents with mental disorders.

These forces will change the way services are delivered for years to come. The implications of such continuing change for any assessments of psychopathology and services use in a national study are clear. Study designs that do not measure changes in the population over time with sufficiently detailed community and regional data, are unlikely to lead to meaningful findings on policy issues for services research. Moreover, designs that do not allow repeated assessments over longer periods of time on populations may not capture larger trends in access, utilization, and cost that emerge as new systems become entrenched.

The Board identified the following priority questions for service use:

What is the service use associated with specific symptoms and impairments?

How does service use change over time?
How does service use differ across demographic groups?

Although the field of service research in children is relatively new, the measurement issues for service use seem to be straightforward for certain service vectors. In particular, the UNOCCAP investigators should take great pride in their work on the Services Assessment for Children and Adolescents (SACA), since it stands out as the measure of choice for capturing the architecture of children's services. Further, the imbedded questions involving the respondent's insurance will provide a limited, but important, index of access to services.

Service Use and Costs for Children in Treatment

Generating an overall cost estimate for the Nation of children's mental health services is difficult since services may be received from multiple providers (e.g., schools, welfare, health care, or justice). Such an estimate may not be particularly accurate since tracking a child through these multiple providers is problematic. Further, the utility of such total cost estimates, with this level of accuracy, for policymakers, payers, or researchers is not clear. However, the study of some children and adolescents receiving services may provide critical information on important policy questions related to system design, insurance issues, or organizational factors.

The Board identified the following priority questions in costs:

A. What are the costs of providing mental health services to children and adolescents?

How do costs vary by demographic groups?
How do costs vary by mental health problem?

B. How does the structure of insurance benefits influence the use of services and the costs of care?

What benefit packages are associated with variation in service use and cost?
What are the effects of managed care on service use and costs?
How do parity laws affect the cost of treating children?

From a methodological standpoint, the problem that must be addressed is that a small number of individuals--those using residential or inpatient care or partial hospitalization--account for a majority of the aggregate costs of mental health services. Most cross-sectional samples of the U.S. population, even with a national sampling frame, will identify too few service users to provide stable cost estimates. For this reason, studies on the cost of mental health services for children in treatment will require directly sampling service providers and the children receiving care.

• Into and Out of Disorder

One of the research issues in the National Plan is charting the pathways into and out of disorder. This captures the notion of development, and how children with various risk and protective factors change. Many childhood conditions can be traced to complex interactions among psychological, social, and biological phenomena that unfold over time. To address questions about the pathways into and out of disorders, longitudinal studies should reflect this complexity through multiple developmental assessments over time. In this way, the progressive unfolding of the child's abilities in relation to changing environmental demands can be captured. Without this developmental perspective, assessment of the course of children's development may miss important markers of their evolving biological, psychological, and social capacities.

The factors that predict which individuals with moderate symptoms go on to develop a problem or avoid a problem need to be verified. Additionally, since many individuals have multiple disorders (comorbidity) or have many symptoms (co-occurrence), it is important to chart the timing and patterning of multiple conditions and symptoms. Related issues include an examination of the persistence versus transience of various disorders and the strength of the association between moderate symptoms at one point of time and the likelihood of developing a disorder later.

The Board identified these questions:

What are the paths into and out of disorder?

Are these paths similar for different demographic groups?
Are these paths similar for different disorders?

Into and Out of Services

What are the factors associated with the use of services? In addition to sociodemographic and health status factors, the following factors are of particular policy and scientific interest: the effect of the presence or absence of insurance, the type of insurance, the structure of the cost-sharing provisions, the extent of managed care, etc. Also of interest is the question of how the use of services is influenced by the availability of service providers and differences in the cost of the service.

Examining prevalence and service use questions at one point in time does not account for the interplay of disorders and symptoms, or the changes in risk and protective factors over time. Children's lives and circumstances change over time and such changes are associated with changes in patterns of mental health and service use. If rich, developmentally sensitive, longitudinal studies are not initiated, these changes cannot be modeled. Concomitantly, nothing can be learned about the course of mental problems in children.

The Board identified the following questions:

What are the paths into and out of service use?

Are these paths similar for different demographic groups?

Are these paths similar for different disorders?

Service Need, Effectiveness, and Quality

Eventually, planning for mental health service delivery systems and their evaluation will require careful assessment of the needs of individuals and populations for mental health services. Subsequently, providers, patients, and payers will require assessments of the quality of service delivery in the mental health sector. Unfortunately, the Board judged that considerable work is necessary in mental health research before such careful analyses can take place. Because so little is known about which treatments work in which settings and for which individuals, it is not possible to speculate on the level of need for any particular service. In the past, need has been estimated for populations through the use of such gross indicators as diagnoses or more recently, impairment. However, current treatment models relying on such indicators have not yet proven accurate for children and adolescents.

The Board identified the following priority questions related to service need and quality:

Which services are effective for which children and adolescents?

How does effectiveness vary over demographic groups?
Who are the children who need, but do not receive, effective services?
What are the costs of these services?
How does quality vary across settings and provider characteristics?

Need for Longitudinal, Local Designs

Only richly textured, longitudinal studies provide answers to these questions. Given the depth and breadth of the data required, studies cannot be reasonably done at this time through a large-scale national survey technique. Nonetheless, large samples, albeit geographically confined, are needed to test the interplay between multiple factors.

Longitudinal surveys permit the comparison of a person's attributes over time, allowing one to study the precursors and sequelae of mental health impairments. Longitudinal surveys offer abilities to study pathways that no other design can rival.

However, longitudinal surveys on fixed samples of individuals become less representative of the full population over time, as attrition in the sample and cumulative effects of emigration and immigration occur.

Rotating panel surveys are designs that measure the same sample of persons for multiple times and then remove them from the measurements, replacing them with a new sample of the current population. They assemble shorter time series of data on each sample person but keep the sample representative of the current, full population.

All longitudinal surveys face problems of tracking "movers" over time and of contaminating effects of repeated measurements on the same topics. Movers are survey participants who enter the survey, but leave the geographic area before the next assessment. The mover problem leads to attrition and to higher costs from trying to trace these individuals. Further, young families tend to move at higher rates than others do. Also, the repeated measurement of individuals using the same instruments can lead to behaviors that reduce the participant's interview burden (e.g., not reporting symptoms because such reports are known to lead to further burdensome questions). Finally, longitudinal surveys quickly lose representation of the youngest ages and need more frequent refreshing to remain informative about young children. In short, longitudinal surveys also have limitations.

An advantage of local longitudinal surveys over national longitudinal surveys is the relative ease of appending richly textured community and service system attributes to the survey data. Doing this in a national survey is cumbersome and expensive.

In sum, longitudinal surveys are a key tool for the study of pathways, but must be viewed as complementary to other designs (e.g., national, repeated, cross-sections) that more efficiently measure prevalence characteristics.

State of the Art in Large, Population Based, National Surveys

Because large-scale surveys are multivariate in nature, they allow statistical analyses to measure the relative frequency of a selected phenomenon on small subclasses of the national population. This allows policymakers to have faith that, for example, no unobserved regional factors are influencing the data. Further, such surveys can examine whether measurable attributes combine in unexpected ways to produce unusual susceptibility to problems. For example, gender differences in exposure to risk factors, types of symptoms, and their severity may vary across subgroups resulting in particular patterns of vulnerability (e.g., stronger or weaker links to impairment or perhaps shorter or longer duration of suffering).

It is essential to review the current status of survey research technology, as background to the Board's recommendations on how this technology can be applied to children's mental health research. Also, this methodology is vital to incorporate if the national studies are to be scientifically rigorous, economical, and generative over time.

The state of the art in the development of survey measurements involves:

• Qualitative investigations that study the linguistic and conceptual variation within the population regarding the phenomena;
• Semi-structured group and one-on-one interviews that explore the ease of comprehension of terminology to be used in survey questions;
• "Cognitive interviewing" that asks laboratory-based respondents to report their cognitive processes in constructing an answer to a survey question;
• Household-based pre-testing of fully structured instruments, sometimes with reinterviews or validity-check components;
• Development of training protocols for professional interviewers to use the instrumentation;
• "Dress rehearsals" to study the performance of the instrumentation in the context of professional survey interviewers;
• Implantation of design features within the survey to provide statistical estimates of accuracy and precision (reliability and validity) of survey measurements, as they were actually implemented in the survey.

This last step deserves elaboration because it is critical to the Board's vision, and yet contrary to the culture of traditional psychometric practice of mounting reliability and validity studies separate from the study in which they are used. The literature on surveys has uncovered repeated evidence that surveys are sensitive to biases from many characteristics of the interviewing environment that one cannot anticipate in the laboratory. Survey designers attempt to eliminate these sources of systematic error in measurement (e.g., training interviewers to deliver questions and probe answers in a non-directive manner). Many of these influences, however, cannot be completely removed from the survey measurements.

Instead of relying on estimates of reliability and validity from prior small scale-studies (often using different interviewing staffs, different sites of data collection, different protocols), survey researchers increasingly attempt to imbed side-studies that provide reliability and validity estimates on the survey data as collected in vivo. These studies can use probability subsamples of the full sample, with reinterviews involving clinical examinations or more in-depth questioning or replication of the original measurements. The value of imbedding such studies in the actual survey is that multivariate models can be used to estimate the data quality for improving the inferential statements derived from the statistics.

Summary

The Board is recommending immediately moving forward on monitoring the levels of symptoms and impairment in children and adolescents and their service use because this research would be important for children and because the field is ready to do so. Also, a large-scale national survey on the patterns of service use, the costs of services, and the factors that influence both use and costs could be conducted now.

Priority questions regarding paths into and out of both disorders and service use, and understanding the need and effectiveness of services should be supported through local and regional longitudinal studies of sufficient size and with sufficient breadth of variables. NIMH has a strong history of supporting these efforts which should provide this new initiative with the necessary groundwork.

Determining readiness and setting priorities are always difficult processes. The Board expects that each of its assumptions and determinations will be subjected to criticism, and this criticism should be viewed as part of the necessary dialectic process for initiating a major research effort.

Overview

Addressing the questions identified by the Board will require a sustained program of research on children's mental health status, services, and the factors that influence each. This plan for a sustained program of research contains a subset of questions that require national data and are ready to be addressed. In addition, the plan calls for initiatives to foster better conceptualization, new efforts in measurement development, and hypothesis-generating studies to build the groundwork for answering the remaining questions.

As depicted in Figure 1 below, the components of the program of research are actually freestanding. However, the components are also seen as synergistic. For instance, the national survey work will provide a testing ground for the new conceptual work and serve as a reference-base for the local and regional studies. Further, when the data from surveys are "democratized" (i.e., given promptly and freely to the field), researchers can examine whether the data are consistent with competing hypotheses from various perspectives. The remainder of this section will outline the following seven components of the proposed program.

National Monitoring Survey in Prevalence and Service Use

There are many possible designs for conducting a national assessment of children's mental health problems and their service use. These designs vary in the precision, timeliness, and quality of estimates, as well as costs. In considering the merits of alternative designs, the Board used five criteria:

a) Ability to estimate change in prevalence of symptoms, impairments, and use of services over time;

b) That the specific question is best addressed (and ready to be addressed) through a national design;

c) Ability to foster the development and evaluation of new measurement tools, taking advantage of advances in the field;

d) Extent of openness to the entire field, permitting participation of large numbers of researchers;

e) Possibility of partnerships with ongoing data collection systems that will reduce costs to NIMH.

As an aside, the Board wants to recognize a noteworthy asset of the monitoring study, even though this comment is beyond its purview. Mental health service researchers are faced with many problems when looking to see the effects of a policy shift on the public's mental health or service use. One of the most frustrating must be the delay between an announced policy change and obtaining funds to evaluate the policy's impact. Having an ongoing monitoring study, also called an indicators study by service researchers, with public data files provides the immediacy researchers and policymakers need. This asset is not limited to children's mental health services and should be considered for adults as well.

Figure 2 depicts the monitoring program, beginning with a benchmark assessment (about 90 minutes of interviewing time) in the initial year. The Board recommends that the first benchmark assessment consists of a selected number of demographic variables, a symptom checklist, an impairment checklist, and the SACA. Future benchmark surveys, conducted on a planned schedule every 4 to 6 years, might offer enhanced diagnostic measures, multiple reporters of the child's attributes, insurance and provider follow-backs, or psychiatric diagnostic interviews on subsamples.

The Board believes that the intensive benchmark measures will not be needed in each assessment wave. Rather, most assessment waves would consist of a very brief set of core indicators and a short developmental module (about 15-20 minutes total interview time). The core indicators would be a small subset of the benchmark measures. The indicators would be brief, sensitive measures of mental health status and service use that could be used to estimate change between core survey years or between core survey years and the benchmark survey year.

The developmental module of each core survey would be the "laboratory" for innovation in children's mental health measurement, fulfilling criterion c) above. NIMH would announce a competition for these modules. Proposals could involve new measurements on subsets of children (e.g., the young child), measures on specific disorders that have previously eluded accurate measurement, or urgent policy questions. The proposals might involve small-scale preliminary studies prior to the introduction into the next available developmental module. Each edition of the developmental module would be analyzed to assess its success in providing unprecedented insights into the mental health of children. Individual measures or questions would be candidates to replace the benchmark measures or core assessments.

The repeated benchmark assessments would offer the field a tool to recalibrate its core indicators based on changes in the population or service characteristics or based on inventions of improved measurements. Each core assessment and benchmark assessment is envisioned as a new cross-sectional sample of U.S. children.

In summary, the proposed monitoring survey can be seen as similar to the National Aeronautics and Space Administration's (NASA) shuttle program. NASA invests in running the shuttle program and a stream of ongoing research for developing shuttle technology. Additionally, NASA reserves funds, airtime, and staff for conducting add-on projects to the ongoing procedures. That is, NASA will solicit research on a given topic, and the competitive winners are provided time during the space flight for their studies. NIMH can adopt this model to promote timely and optimal application of scientific advances, as well as coordination in its large-scale initiatives.

National Survey on Use and Costs among Children in Treatment

The monitoring system of prevalence, insurance coverage, and service utilization is a powerful method for identifying overall trends on use of services across the population. The study will not, however, provide much information about the nature of treatment received by children. The main reason for this is that the national design will not identify large numbers of children who are receiving treatment in specialty mental health care services. (It should have the power to identify children who are less seriously ill and using outpatient services and frequently prescribed medications.) Consequently, it will not provide precise information on many of the questions that NIMH is asked about the treatment of seriously ill children and how it is affected by the financing and organization of care.

Some of these questions can be addressed by regional and single-site studies, but regional and single-site studies will not be able to answer questions on the impact of parity laws, differences in the way that managed care carve-outs or health plans are organized, and geographic and ethnic differences in the use of intensive services. The Board therefore recommends that a national study of children in treatment be conducted to examine processes of service access, use and financing.

The Board is not making specific design recommendations about how to execute such a study, but offers general advice. The study can be initiated promptly since the instruments necessary for the conduct of such an effort are already available through the efforts of NIMH grantees and the UNOCCAP investigators. In addition, the qualities outlined above in the survey section must be reflected in this study as well.

Considerable thought should be given to the particular sampling of mental health service settings, geographic areas, and populations to take advantage of the natural variation in the spread of managed care, provider types, and public laws related to the delivery of mental health care. Natural variation across states in policy changes or the provision of services should influence the sampling frame. For example, states or regions undergoing early welfare reform should be contrasted with those not undergoing reform at the same pace.

The sample and assessments should probably include youth in various types of restrictive services to include partial hospitalization, intensive outpatient therapy, and in-home crisis services.

The proposed study of service use for children in treatment is not designed to provide a national total cost estimate. The field is not yet ready to tap into all of the streams of costs for children receiving services in non-specialty settings (e.g., health, welfare, justice and education). Making these links across service settings will be a vital area for developing the field and improving the current cost estimates. The Board believes, however, that the proposed study on the architecture of service use and costs would be highly informative for both services researchers, payers, and for policymakers.

The Board concluded that questions on prevalence and service use are best answered, and are ready to be answered, at a national level through a large-scale survey. A rigorous study is required to describe symptoms, symptom patterns, impairments, and service use in a sample large enough to reveal important sources of demographic variation in the United States. It is of critical importance to launch this study soon, since valuable time has already been lost for informing the policies and strategies that are changing service provision and financing. The Board also places great emphasis on NIMH creating the necessary infrastructure to initiate this activity immediately.

Generativity and Availability of the Data

The survey data would be publicly available as soon as possible after data collection. This rapid availability is standard procedure for surveys conducted through other governmental agencies, such as the National Health Interview Survey and the National Ambulatory Medical Care Survey.

Because the data sets developed by the proposed project would be public use data as with many other federal data sets, resources to support their analyses and use are essential to maximize return. A variety of options are possible including targeted contracts for specific analyses or a grant program for analyses. Of course, the data set will require documentation and seminars on access and use of the data to maximize its utility. These activities may be undertaken by NIMH staff or contracted.

The Board believes that if data were publicly available and analysis supported by NIMH, would be generative of new sets of hypotheses to stimulate the field. These might arise from unexpected differences among subgroups of children in prevalence or service use patterns. They might arise from evidence of changes over time in prevalence that suggest causal mechanisms undetected in clinic samples or homogeneous groups studied previously. They might also arise from newer scientists or scientists not bound by the dominant paradigm, testing novel hypotheses with the observational data and finding support. In short, the national, local, and regional studies should complement and reinforce one another.

Answering Questions on Diagnosis, Pathways, and Outcomes

As described in this chapter, NIMH should provide initiatives for building the areas that are seen as critical to achieving the goals in the National Plan for Research on Child and Adolescent Mental Disorders. Issues of measurement and diagnosis are critical to advance research in children's mental health problems. Better understanding of disorders and their course will enhance treatment research. Service programs cannot be targeted to needy children until research documents what works for whom. The hand-in-glove nature of research and policy, as well as basic and applied research, must be recognized and fostered. Certain preliminary, basic work must be accomplished before the questions on diagnosis, pathways, and outcomes can be posed and feasibly answered. In addition to continued support of investigator-initiated projects, Program Announcements, contracts, or RFAs should be issued as quickly as possible to stimulate research in the areas proposed in the remainder of this section.

Conceptualization and Measurement

The Board concluded that researchers trying to understand child diagnostic issues are wrangling with a complex and enduring conceptual problem. The debate over the classification of disorders and problems, as well as their measurement, is healthy and necessary. Inductive as well as deductive approaches must be advanced and tested. For example, large-scale data collection on the natural distribution of symptoms and impairments in children may provide a new structure of how these problems coalesce into disorders. The Institute is encouraged to support many alternative models to assessment in children. Further, comparisons between and among methods for validity and utility are strongly supported. However, methods for certain subgroups of children are not as ready as others for assessment. The Board recommends particular attention to developing these areas.

Young Child

The assessment of mental health in older children and adolescents is far more developed than procedures for determining mental health in younger children. For example, it is not clear that the Diagnostic Interview Schedule for Children (DISC), as currently designed, taps the symptoms of various disorders very well for children under the age of 7 to 8 years of age. In addition, not only are there unanswered questions vis-à-vis the validity of parental reports of young children's symptoms as mapping onto disorders, but reliability questions remain. The Board is heartened by the efforts of the researchers working on the DISC and the UNOCCAP Young Child project in addressing these issues.

Alternative techniques for assessing the mental health of young children are being developed. Several of these laboratory-based measures appear to be very promising, but more research is needed to see if these measures can be modified for use in the field. Questions also remain on the usefulness of some of the techniques for children with low verbal comprehension and expression.

The Board encourages the further development of such assessment techniques in field settings and with various groups of children (i.e., different ethnic groups, bilingual children, immigrant children). It also encourages the assessment of young children using multimethod approaches (i.e., parental report using the DISC, observation of parents and children, emotional regulation tasks). Finally, the Board encourages short-term longitudinal follow-up studies in order to see if these techniques have predictive value.

This necessary work should not hold up the national monitoring survey. Depending on the psychometric properties of the selected symptom and impairment checklists for young children, they may either be included or not included in the initial sampling waves. If the selected checklist is reliable and valid for this age group, then younger children should be included. If the selected measures are not yet validated for young children, this testing and refinement should be undertaken. But such testing should not delay the initiation of the monitoring survey. In this latter case, full incorporation of the younger age group into the national monitoring survey can be a future goal, and the young child initiative will be an important catalyst for the field in reaching this goal.

Longitudinal Research Initiatives

Pathways. NIMH has a long history of supporting risk factor research. Providing synthesis for this research and refining important testable questions is an important activity for the Institute to sustain. Small-scale investigator-initiated projects may not be sufficient for this activity. The Institute is encouraged to work with other funding agencies and ongoing surveys to collect data on samples that are large enough and sufficiently representative to examine the factors and circumstances that predispose some children to develop mental disorders. Of particular importance is the need to identify the factors that predispose some children to short bouts of problems while other have more persistent problems. The monitoring study will not allow for an examination of how some children, who live in difficult circumstances, avoid problem behavior or mental disorders over time.

Applying the best of services research, epidemiology, developmental psychopathology, and social sciences will be necessary to understand the paths into and out of services. The complexities are profound since children receive services in so many sectors and neither they nor their parents are good reporters of the services received. The Board's best advice is to start small and focus on mental health services first, working toward building linkages with the educational, health, welfare, and justice systems. Brief measures of mental health status and service use will be invaluable in making these ties.

Outcomes and Quality. The conceptualization and assessment of treatment outcomes for children have progressed markedly in the past decade in conjunction with a growing recognition of the importance of functional status/impairment and family perspectives. However, much additional work needs to be done. In particular, better characterization of child mental health status remains elusive. Similarly, investigators have yet to elucidate the key components of treatment necessary to link outcomes with services in the examination of effectiveness. The Board heartily supports NIMH's efforts to develop methods for assessing child and family outcomes and for better describing components of mental health services.

Further complicating the inclusion of treatment outcomes and quality of care assessments in a national study is the lack of data on effective treatments for many childhood disorders. It is essential that local or regional studies examining the translation of efficacious treatments into community settings be conducted as soon as possible, especially in the face of numerous studies documenting limited effectiveness for routine mental health services. The Board encourages all reasonable efforts to develop effective treatments for a wider range of children and adolescents. The assessment of the quality of care for a national study will progress rapidly once such treatments are available.

Innovations in Methods and Analyses

Many fields (e.g., astronomy) have made advances in theoretical and applied knowledge by directly funding the development of new measurement tools. These investments have multiplier effects because inventions in measurement can be used to improve the insights from hundreds of scientists. The Board believes that a revitalized program is needed in this area to ensure that methodological and analytic opportunities rapidly advance the quality of research in mental health. Some agencies devote a certain percentage of its data collection budget to such an infrastructure building initiative.

Research in this program might involve addressing issues of new standardized measurement of very young children. It might involve studies of the impact of linguistic constructions on question comprehension in self-reports concerning mental impairments. It might involve development of statistical methods to reduce the harmful effects of participant attrition. It may involve survey methods to combine rapidly and accurately administrative data from health systems with self-report data from survey participants. It might involve the development of computer-assisted measurement protocols that decrease the effects of social desirability on self-report of mental health problems and risk behaviors. It may involve the development of statistical analytic techniques to provide insights into the role of the family or schools in the course of mental disorders.

In general, mental health researchers cannot be expected to define the cutting edge of developments in survey design, data collection, and analysis as well as be on the forefront of mental health. The field of survey methodology is a fast-moving one, involving technological advancements in both data collection and estimation. To sponsor an innovative grant program in methods, NIMH would want to be open to applications from design statisticians, analytic statisticians, computer scientists, survey methodologists, cognitive scientists and others, who are building the key theories and applications in research methods. NIMH should encourage collaborations with mental health researchers, but these methodologists should be supported as principal investigators as well.

The review of these applications should involve a significant representation of specialists in measurement. That is, a majority of the reviewers should be basic scientists actively contributing to the field of measurement, as well as mental health researchers who are working on measurement issues.

Integrative and Generative Nature of the Plan

These seven recommendations are seen as an integrated package, even though each initiative can be freestanding. The synergistic value of the components will be evident over time as new findings are integrated into or tested through the monitoring survey. As an example of envisioned interplay, the national services use and costs survey for children receiving services can begin now. From this short-term survey, important hypotheses or measures will be generated that can be included in the national monitoring survey either as a developmental or topical module in four to five years.

The effectiveness and path issues need more time for basic research and conceptualization, but initiating these activities now will speed the field in answering the important policy questions related to developmental processes. By supporting longitudinal studies and methods development today, the Board foresees future possibilities for topical modules in the monitoring study. These modular add-ons should be competitive and may also include solicitation of research to address pressing policy or scientific questions as they arise.

The initiation and management of large-scale efforts require both accountability and scientific vision. NIMH staff must balance administrative necessities with achieving the scientific and public health goals of any large-scale effort. The Board offers a series of suggestions on the planning, staffing, and oversight of such efforts. The Board also encourages the staff to talk to colleagues at other agencies and institutes to identify other possibilities for enabling large-scale projects. Some of the ideas mentioned here were initially raised at the NAMHC's May l997 policy meeting.

Planning

The Board has noted the urgency of mounting the proposed program of research as quickly as possible. Nonetheless, the planning phase cannot be short-changed. The Board reviews below some of the critical components of planning a large-scale effort, and encourages a period of open planning that engages the field and other Federal agencies and institutes.

• Selecting a Feasible Funding Mechanism

Deciding among contracts, cooperative agreements, and grants is an administrative decision, based on the type of activity. However, the execution of each of these mechanisms can be done in many ways as demonstrated throughout NIH. The NIMH should consider the following in initiating these programs of research.

Cooperative agreements are solicited through RFAs. Individuals or teams of individuals compete by submitting their research plans and protocols to a review committee. The review committee carefully considers these applications against the review criteria. NIMH then awards individual projects. Typically, NIMH then convenes the successful applicants, who then design a shared protocol. There is the potential for many problems in this approach, and NIMH should consider improving this process so that:

• Teams of researchers respond to the RFA. Awards can then be made to a self-composed team that already has or can easily build a workable alliance.
• In the extramural team, there must be one designated leader. The leader's role and how this role is assigned must be specified in the solicitation.
• The review process could be segmented. There could be a minimum application based on a capacity review. The successful applicants would then submit a more involved application including the proposed design.

When the best way to achieve the programming goal requires scientific expertise from investigators, a cooperative agreement is initiated. However, when the NIMH staff knows what needs to be done and how it should be accomplished, a contract should be considered. One issue regarding the use of contracts is that a contract requires OMB clearance for a survey. This can be difficult to achieve in a timely manner, but should not dissuade NIMH staff from using the contract mechanism.

In drafting the RFA or RFP, staff should focus on a few, feasible scientific and/or public health issues that must be addressed. Having such focus will enrich the process from drafting the RFA to making design decisions, to the final analysis of the data.

• Establishing a Time Line

Planning involves drafting a feasible time line. The Board stresses feasibility, even though the needs of children are so urgent. As an example, the Board projected a time line for initiating the proposed monitoring survey. If the Institute wants to be in the field with the national monitoring survey in September of 2000, the time line for developing, competing, reviewing, and awarding the initiative is already behind schedule:

Budgets must be fully anticipated for flagship studies such as national surveys. Staff must reconnoiter the estimated costs with the questions to be addressed. If the budget does not permit the answering of these questions, changes must be made.

In addition, NIMH should plan on complex projects having problems during implementation. Some of these problems can be fixed with additional financing, and the budget should have a reserve fund for this purpose.

Efforts to equalize payments to large-scale projects over funding years should never jeopardize the science or financial accountability. That is, although budgeting may run more smoothly if yearly costs are equivalent, large-scale studies rarely have identical yearly costs. NIMH should adopt a pay-as-you-go policy, with sufficient monitoring.

Once the applications/proposals have been received and reviewed, the budget estimates should be reassessed. Staff should examine whether initial estimates will permit the projected work to be completed as required.

• Identifying Co-sponsors for Large-scale Projects

In a world of constrained research funding, a legitimate question for NIMH to discuss is how much of its resource base should be devoted to large-scale observational studies and surveys versus small-scale experimentation. This is a question that faces all funders of scientific activities. There appears to be at least four alternative approaches to this issue: a) fund the survey research using agency funds, b) recruit a set of partners who will pay for their own compatible modules on the NIMH survey, c) seek out another survey of compatible design as a vehicle for the NIMH questions, and d) use existing survey data to provide proxy indicators relevant to the agency's needs.

For the monitoring survey, the Board urges NIMH to begin contacting other Federal agencies, charged with tracking the health and welfare of the Nation?s children, to determine whether partnerships of the sort in b) and c) above might be possible. Much of the survey data collected by the Federal government on the household population comes from the U.S. Census Bureau. The collection of mental health-related measurements on children would be a departure from normal practice of that agency, and the Board is doubtful that such a partnership would be feasible. Candidates for partners include the follow-ups to the National Health Interview Survey of the National Center for Health Statistics, the Behavioral Risk Factor Surveys of the Centers for Disease Control and Prevention, the Current Population Survey of the Bureau of Labor Statistics, the National Household Education Survey of the National Center for Education Statistics, the National Household Survey on Drug Abuse of the Substance Abuse and Mental Health Services Administration, the National Immunization Survey of the Centers for Disease Control, among others.

For instance, in the monitoring survey, the core survey years could involve the addition of the NIMH measures to an existing ongoing survey. The size of the question set would be determined by the amount that the partner survey could incorporate. For discussion purposes, it would be less than 20 minutes of questions focused on one selected child in the sample household.

NIMH would enter into an agreement with the sponsor of the survey to "purchase" some amount of time with sample respondents to ask a set of questions and transfer funds to pay for the increase in survey costs for that core. By taking advantage of the infrastructure of the base survey, NIMH could reduce costs relative to mounting its own survey (addressing point d) above).

An NIMH-led survey could also be initiated along with a partner. NIMH might wish to consider a co-funder with interests in collecting similar data. Alternatively, NIMH may want to "rent" a service, such as the sampling of households from an agency using desirable sampling procedures, but seeking different respondents (e.g., National Institute on Aging (NIA) and NIMH sharing respondents over 65 or under 19 in identified residences). Yet another possibility for a partnership would be the ability of another agency to provide survey expertise or other needed staff. If no partnership can be established that would be mutually attractive, the work should continue nonetheless.

Governance and Staffing Needs

The ongoing nature of the proposed program combined with the need for accountability suggest that an ad hoc governance structure for each project will not be adequate and that a permanent, or at least long-term, administrative component should be established. The budgetary and oversight responsibility for the project should rest within NIMH. Survey contract and collaborative functions could reside within NIMH or within a co-sponsoring agency like the National Center for Health Statistics that has similar relationships with other agencies.

To ensure that the problems are identified quickly and handled appropriately and fairly, NIMH must have a separate unit of scientifically appropriate staff to serve both as administrators for these projects. These include survey statisticians and methodological experts as well as experts in mental health measurement. Such staff members are a requirement for the duration of the project. Administrative staff must be able to assess when scientific progress is being made and its adequacy. They must have credibility with the field, the NIMH leadership, and the participating researchers.

Among these staff members, there must be one individual with the scientific credentials to serve as the NIMH leader for this effort. The organizational location of this individual and reporting requirements must be formalized and maintained. The Board suggests looking across NIH for possible models, considering Nationa Instiute on Drug Abuse's Drug Abuse Warning Network program or certain National CancerInstitute projects.

Accountability

• Peer Review

The role of peer review should not be reserved for concept clearance and review of applications. For instance, typically RFAs and RFPs have been reviewed for their concept only. That is, an external group of scientists is convened to review the concept, but not the specifics of the anticipated request. Frequently, the concept is of undeniable appeal, but the feasibility cannot be addressed because too few details are provided. The advantage of this approach is that the concept reviewers, because they do not know the specifics of the request, may still compete for the award. The down side is that when the applications go to the peer review committee, reviewers are asked to assess the applications relative to the criteria in the RFA or RFP, even if the request is flawed.

As previously mentioned, the current NIMH approach to cooperative agreements is to select participants based on their single-site applications. Then these peer-reviewed designs are dropped from consideration to form the collaborative design. Not only is this a problem for establishing productive collaborations, but it may result in a final design that has not been peer-reviewed prior to fielding. This can happen in contracts as well, when designing the study is an initial step in the scope of work. Regardless, the final protocol should be subjected to peer review (in a timely manner) to ensure the design's ability to test the proposed hypotheses and its staffing, budget and time frame. A peer review would also be helpful in the event of significant staff or budget changes.

Separately, but equally as important, some of the specific initiatives will require special review groups or standing review panels to evaluate applications for added modules, instrument development, or the young child area. These review activities would likely focus on two primary aspects, scientific content and policy implication. For the former, expertise in survey design, statistics and sampling, psychometrics, and content areas would be required. Policy analysis expertise would also be required to assess the relative contributions of various studies to the national debates around child mental health.

• Advisory Boards

Ongoing, timely oversight, may best be provided by establishing an advisory board at the initiation of these large-scale efforts. The quality of the advisory board may even be part of the RFA's or RFP's review criteria, asking each team to nominate their own advisors and letting the peer-review group assess the adequacy of the board. This is consistent with NIH policy and will prevent time lags in constructing multiple ad hoc review groups and the lack of continuity in approach as well.

In addition to the project's advisory board, large-scale projects would require significant oversight by the NAMHC. A program of research, such as that proposed in this report, will need oversight that only the Council or a subcommittee, with its perspective of NIMH's entire portfolio, can provide.

The Oversight Board welcomed the opportunity to integrate this information and to advise NIMH and its Council; however, the recommendations in this report will require broad debate and scrutiny to best shape the future initiatives. Certainly NIMH should reach out to the research community and to professional groups, but the Board encourages even broader consultation. Key players in the debate include the service providers and payers who can enhance the utility of the data to be collected in any new initiative. Parents of children with mental disorders should also be made aware of this effort so that they can provide NIMH with the realities of referring their children for diagnosis and treatment, identifying services, and financing these services.

The closing words must be dedicated to the children of this Nation. On behalf of those children living with mental problems, the Board reminds NIMH of the critical need to conduct research so that these children can be identified and effective services provided. Achieving this goal will not be easy, but the necessary initial steps can be taken immediately. Do not wait for the perfect moment or the perfect measure. The Board advises NIMH to adopt its maxim for moving forward--if it's ready, use it now; if it is not ready, develop it.