ASU SWP Pharmaceutical Protocol Case Study

Refer to the Clinical Research Management (2022) textbook and review the SWP Pharmaceutical Protocol (Lateral Epicondylitis – Drug XYZ). Carefully read the study question (study objectives),the study endpoints and the measurement tools.

PART 1: (5 points)
List the tools that are being used for measurement.

PART 2: (15 points)
Do you think that considering the study question and the study endpoints, the measurement tools are accurate and precise – Yes or no and justify your answer (5 points)
Are they all necessary – Yes or no and justify your answer (5 points)
Identify a different measurement scale for pain scores for this study (you might want to use an appropriate orthopedic scale) and briefly justify your choice of pain scale. (e.g: Wong-Baker Faces Pain Scale). Do not use this example for your answer (5 points)

Designing Clinical Research FOURTH EDITION Designing Clinical Research FOURTH EDITION Stephen B. Hulley, MD, MPH Professor and Chair, Emeritus Department of Epidemiology & Biostatistics School of Medicine, University of California, San Francisco Steven R. Cummings, MD Founding Director, San Francisco Coordinating Center Senior Scientist, California Pacific Medical Center Research Institute Professor Emeritus, Department of Medicine, and of Epidemiology & Biostatistics School of Medicine, University of California, San Francisco Warren S. Browner, MD, MPH Chief Executive Officer, California Pacific Medical Center Adjunct Professor, Department of Epidemiology & Biostatistics School of Medicine, University of California, San Francisco Deborah G. Grady, MD, MPH Professor of Medicine Associate Dean for Clinical and Translational Research School of Medicine, University of California, San Francisco Thomas B. Newman, MD, MPH Professor of Epidemiology & Biostatistics, and of Pediatrics Chief, Division of Clinical Epidemiology Attending Physician, Department of Pediatrics School of Medicine, University of California, San Francisco Acquisitions Editor: Rebecca Gaertner Product Manager: Tom Gibbons Production Project Manager: David Orzechowski Senior Manufacturing Coordinator: Beth Welsh Marketing Manager: Kimberly Schonberger Design Coordinator: Teresa Mallon Production Service: S4Carlisle © 2013 by LIPPINCOTT WILLIAMS & WILKINS, a WOLTERS KLUWER business Two Commerce Square 2001 Market Street Philadelphia, PA 19103 USA LWW.com © 2007 by Lippincott Williams & Wilkins, a Wolters Kluwer business. All rights reserved. This book is protected by copyright. No part of this book may be reproduced in any form by any means, including ­photocopying, or utilized by any information storage and retrieval system without written permission from the copyright owner, except for brief quotations embodied in critical articles and reviews. Materials appearing in this book prepared by individuals as part of their official duties as U.S. government employees are not covered by the above-mentioned copyright. Printed in China Library of Congress Cataloging-in-Publication Data Designing clinical research / Stephen B Hulley . . . [et al.]. — 4th ed.     p. ; cm. Includes bibliographical references and index. ISBN 978-1-60831-804-9 (pbk.) I. Hulley, Stephen B. [DNLM: 1. Epidemiologic Methods. 2. Research Design. WA 950] R853.C55 610.72—dc23 2013009915 DISCLAIMER Care has been taken to confirm the accuracy of the information presented and to describe generally accepted practices. However, the authors, editors, and publisher are not responsible for errors or omissions or for any consequences from application of the information in this book and make no warranty, expressed or implied, with respect to the currency, completeness, or accuracy of the contents of the publication. Application of the information in a particular situation remains the professional responsibility of the practitioner. The authors, editors, and publisher have exerted every effort to ensure that drug selection and dosage set forth in this text are in accordance with current recommendations and practice at the time of publication. However, in view of ongoing research, changes in government regulations, and the constant flow of information relating to drug therapy and drug reactions, the reader is urged to check the package insert for each drug for any change in indications and dosage and for added warnings and precautions. This is particularly important when the recommended agent is a new or infrequently employed drug. Some drugs and medical devices presented in the publication have Food and Drug Administration (FDA) clearance for limited use in restricted research settings. It is the responsibility of the health care provider to ascertain the FDA status of each drug or device planned for use in their clinical practice. To purchase additional copies of this book, call our customer service department at (800) 638-3030 or fax orders to (301) 223-2320. International customers should call (301) 223-2300. Visit Lippincott Williams & Wilkins on the Internet at LWW.com. Lippincott Williams & Wilkins customer service representatives are available from 8:30 AM to 6 PM, EST. 10 9 8 7 6 5 4 3 2 1 To our families and our students Contents Contributing Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii SECTION I. Basic Ingredients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1 2 3 4 5 6 Getting Started: The Anatomy and Physiology of Clinical Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Stephen B. Hulley, Thomas B. Newman, and Steven R. Cummings Conceiving the Research Question and Developing the Study Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Steven R. Cummings, Warren S. Browner, and Stephen B. Hulley Choosing the Study Subjects: Specification, Sampling, and Recruitment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Stephen B. Hulley, Thomas B. Newman, and Steven R. Cummings Planning the Measurements: Precision, Accuracy, and Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Stephen B. Hulley, Thomas B. Newman, and Steven R. Cummings Getting Ready to Estimate Sample Size: Hypotheses and Underlying Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Warren S. Browner, Thomas B. Newman, and Stephen B. Hulley Estimating Sample Size and Power: Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 Warren S. Browner, Thomas B. Newman, and Stephen B. Hulley SECTION II. Study Designs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 7 Designing Cross-Sectional and Cohort Studies . . . . . . . . . . . . . . . . . . . . 85 8 Designing Case–Control Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Stephen B. Hulley, Steven R. Cummings, and Thomas B. Newman Thomas B. Newman, Warren S. Browner, Steven R. Cummings, and Stephen B. Hulley vii viii Contents 9 Enhancing Causal Inference in Observational Studies . . . . . . . . . . . . 117 10 Designing a Randomized Blinded Trial . . . . . . . . . . . . . . . . . . . . . . . . . 137 11 Alternative Clinical Trial Designs and Implementation Issues . . . . . . . 151 12 Designing Studies of Medical Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 13 Research Using Existing Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 Thomas B. Newman, Warren S. Browner, and Stephen B. Hulley Steven R. Cummings, Deborah G. Grady, and Stephen B. Hulley Deborah G. Grady, Steven R. Cummings, and Stephen B. Hulley Thomas B. Newman, Warren S. Browner, Steven R. Cummings, and Stephen B. Hulley Deborah G. Grady, Steven R. Cummings, and Stephen B. Hulley SECTION III. Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208 14 Addressing Ethical Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 15 Designing Questionnaires, Interviews, and Online Surveys . . . . . . . . 223 16 Data Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 17 Implementing the Study and Quality Control . . . . . . . . . . . . . . . . . . . 250 18 Community and International Studies . . . . . . . . . . . . . . . . . . . . . . . . . 268 19 Writing a Proposal for Funding Research . . . . . . . . . . . . . . . . . . . . . . 277 Bernard Lo and Deborah G. Grady Steven R. Cummings, Michael A. Kohn, and Stephen B. Hulley Michael A. Kohn, Thomas B. Newman, and Stephen B. Hulley Deborah G. Grady and Stephen B. Hulley Norman Hearst and Thomas Novotny Steven R. Cummings, Deborah G. Grady, and Stephen B. Hulley Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292 Answers to Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351 Contributing Authors Norman Hearst, MD, MPH Professor of Family and Community Medicine School of Medicine, University of California, San Francisco Attending Physician, University of California Medical Center San Francisco, California Michael A. Kohn, MD, MPP Associate Professor of Epidemiology and Biostatistics School of Medicine, University of California, San Francisco Attending Physician, Emergency Department Mills-Peninsula Medical Center, Burlingame, California Bernard Lo, MD President, The Greenwall Foundation Professor of Medicine, Emeritus Director of Program in Medical Ethics, Emeritus University of California, San Francisco Thomas Edward Novotny, MD, MPH Professor and Associate Director for Border and Global Health Graduate School of Public Health San Diego State University, San Diego, California ix Introduction T his fourth edition of Designing Clinical Research (DCR) marks the 25th anniversary of the publication of our first edition. It has become the most widely used textbook of its kind, with more than 130,000 copies sold and foreign language editions produced in Spanish, Portuguese, Arabic, Chinese, Korean, and Japanese. We designed it as a manual for clinical research in all its flavors: clinical trials, observational epidemiology, translational science, patient-oriented research, behavioral science, and health services research. We used epidemiologic terms and principles, presented advanced conceptual material in a practical and reader-friendly way, and suggested common sense approaches to the many judgments involved in designing a study. Many of our readers are physicians, nurses, pharmacists, and other health scientists who, as trainees and junior faculty, are developing careers in clinical research and use this book as a guide in designing and carrying out their studies. Many others are clinicians in residency programs and pre-doctoral students in professional schools—medicine, nursing, pharmacy, and public health among others—who use DCR to help them become discerning readers with a grasp of the strengths and limitations of the research studies that inform evidence-based clinical practice. A third audience consists of undergraduate students preparing to apply to these schools who are interested in looking ahead at the world of clinical research. What’s new in the fourth edition? The most visible innovation is color, which, in addition to improving the esthetics, will speed comprehension of the color-coded components. A larger innovation that accompanies each purchase of the paperback text is an interactive digital experience powered by Inkling®, viewable through a browser or as a download to tablet or smartphone. Its features include rapid index-based search options that link to a newly created glossary; bookmarking, highlighting, and annotating capability; cross-linking of relevant content; the ability to cut-and-paste figures or text into PowerPoint presentations; and live Internet links to jump instantly from citations to articles on PubMed, and to Google topics. The substantive revisions to the fourth edition include updated and tightened text, figures, and tables in every chapter; many new examples and references; and new sections covering recent advances in the field. For example: • The chapters on observational studies have been reorganized with an entire chapter now • • • • devoted to various case–control designs, including the incidence-density approach for addressing changes in risk factor levels and differences in follow-up time. The chapters on clinical trials have an expanded section on the non-inferiority trials that have become popular in comparative effectiveness research, and they address subgroup analysis and effect modification more fully. The chapter on studying medical tests has a new section on the growing practice of developing clinical prediction rules. The chapter on utilizing existing data sets emphasizes attractive options for beginning investigators to publish rapidly and inexpensively. The chapter on research ethics is updated to reflect current policy on whole genome sequencing and other topics, with new cases that illustrate the resolution of ethical dilemmas in clinical research. xi xii Introduction • The chapter on data management has been extensively updated with the latest Web-based approaches. • The chapter on getting funded has strategies for addressing the new NIH grant-writing re- quirements, as well as updates on funding by foundation and corporate sponsors. The fourth edition is accompanied by an upgraded DCR website at www.epibiostat.ucsf. edu/dcr/ that contains materials for teaching DCR, including links to a detailed syllabus for the­ 4- and 7-week DCR workshops that we present to 300 trainees each year at UCSF. There are also instructor’s notes for the workshops that faculty who teach this material will find useful, and links to our Training In Clinical Research (TICR) master’s degree program at UCSF, with more than 30 other courses and their materials. In addition, there are useful tools for investigators, including an excellent interactive sample size calculator. Many things have not changed in the fourth edition. It is still a simple book that leaves out unnecessary technicalities and invites the investigator to focus on the important things: how to find a good research question and how to plan an efficient, effective, ethical design. The chapters on estimating sample size continue to demystify the process and enable readers with minimal training in statistics to make these calculations themselves, thoughtfully, and without needing to wrestle with formulas. The book still works best when combined with the essential ingredient of one or more long-term mentors. It still does not address the important areas of how to analyze, present, and publish the findings of clinical research—topics that our readers can pursue with other books (e.g., 1–4). And we still do use the feminine pronoun in the first half of the book, masculine in the second, the goal (besides avoiding the passive tense) being to symbolically empower clinical investigators of both genders. The process of becoming an independent clinical scientist can be challenging, especially getting over the hump of acquiring a substantial grant for the first time. But it is gratifying that many of our former trainees who used this book have achieved this goal, discovered that they like doing research, and settled into a great career. For those with inquiring minds, the pursuit of truth can become a lifelong fascination. For perfectionists and craftsmen, there are endless challenges in creating elegant studies that conclusively answer questions, large and small, at an affordable cost in time and money. Investigators who enjoy teamwork will develop rewarding relationships with colleagues, staff, and students, as well as friendships with collaborators working in the same field in distant places. And for those with the ambition to make a lasting contribution to society, there is the prospect that with skill and tenacity they will participate in the incremental advances in clinical and public health practice that is the natural order of our science. REFERENCES 1. Vittinghoff E, Glidden DV, Shiboski SC, et al. Regression methods in biostatistics: linear, logistic, survival, and repeated measures models, 2nd ed. New York: Springer-Verlag, 2011. 2. Katz MH. Multivariable analysis: a practical guide for clinicians and public health researchers, 3rd ed. New York: Cambridge University Press, 2011. 3. Newman TB, Kohn MA. Evidence-based diagnosis. Cambridge, MA: Cambridge University Press, 2009. 4. Browner WS. Publishing and presenting clinical research, 3rd ed. Philadelphia, PA: Lippincott Williams & Wilkins, 2012. Acknowledgments W e are grateful to the Andrew W. Mellon Foundation for bringing the five of us together 30 years ago to begin the five-year journey of developing the teaching materials that became the first edition; to our publisher for steadily inviting a fourth edition until resistance became futile, and for providing exceptionally talented and supportive professionals to help us put it together; to our families for their patient support as we labored over this opus; to many colleagues at UCSF and beyond, whose ideas have influenced ours; to our students over the years, whose accomplishments have been fun to watch and stimulating to our thinking; and to our readers who have put this book to use. xiii SECTION I Basic Ingredients 1 CHAPTER 1 Getting Started: The Anatomy and Physiology of Clinical Research Stephen B. Hulley, Thomas B. Newman, and Steven R. Cummings T his chapter introduces clinical research from two viewpoints, setting up themes that run together throughout the book. One is the anatomy of research—what it’s made of. This includes the tangible elements of the study plan: research question, design, subjects, measurements, sample size calculation, and so forth. An investigator’s goal is to design these components in a fashion that will make the project feasible and efficient. The other theme is the physiology of research—how it works. Studies are useful to the extent that they yield valid inferences, first about what happened in the study sample and then about how these findings generalize to people outside the study. The goal is to minimize the errors, random and systematic, that threaten conclusions based on these inferences. Separating the two themes is artificial in the same way that the anatomy of the human body doesn’t make much sense without some understanding of its physiology. But the separation has the same advantage: It clarifies our thinking about a complex topic. ■ ANATOMY OF RESEARCH: WHAT IT’S MADE OF The structure of a research project is set out in its protocol, the written plan of the study. Protocols are well known as devices for seeking grant funds and Institutional Review Board (IRB) approval, but they also have a vital scientific function: helping the investigator organize her research in a logical, focused, and efficient way. Table 1.1 outlines the components of a protocol. We introduce the whole set here, expand on each component in the ensuing chapters of the book, and return to put the completed pieces together in Chapter 19. Research Question The research question is the objective of the study, the uncertainty the investigator wants to resolve. Research questions often begin with a general concern that must be narrowed down to a concrete, researchable issue. Consider, for example, the general question: • Should people eat more fish? This is a good place to start, but the question must be focused before planning efforts can begin. Often this involves breaking the question into more specific components, and singling out one or two of these to build the protocol around: • • • • • 2 How often do Americans eat fish? Does eating more fish lower the risk of cardiovascular disease? Is there a risk of mercury toxicity from increasing fish intake in older adults? Do fish oil supplements have the same effects on cardiovascular disease as dietary fish? Which fish oil supplements don’t make your breath smell like fish? Chapter 1 • Getting Started: The Anatomy and Physiology of Clinical Research 3 TABLE 1.1 ANATOMY OF RESEARCH: THE STUDY PLAN DESIGN COMPONENTS PURPOSE Research questions What questions will the study address? Background and significance Why are these questions important? Design How is the study structured? Time frame Epidemiologic design Subjects Who are the subjects and how will they be selected? Selection criteria Sampling design Variables What measurements will be made? Predictor variables Confounding variables Outcome variables Statistical issues How large is the study and how will it be analyzed? Hypotheses Sample size Analytic approach A good research question should pass the “So what?” test. Getting the answer should contribute usefully to our state of knowledge. The acronym FINER denotes five essential characteristics of a good research question: It should be feasible, interesting, novel, ethical, and relevant (Chapter 2). Background and Significance A brief background and significance section in a protocol sets the proposed study in context and gives its rationale: What is known about the topic at hand? Why is the research question important? What kind of answers will the study provide? This section cites relevant previous research (including the investigator’s own work) and indicates the problems with the prior research and what uncertainties remain. It specifies how the findings of the proposed study will help resolve these uncertainties, lead to new scientific knowledge, or influence practice guidelines or public health policy. Often, the literature review and synthesis done for the significance section will lead the investigator to modify the research question. Design The design of a study is a complex issue. A fundamental decision is whether to take a passive role in making measurements on the study subjects in an observational study or to apply an intervention and examine its effects in a clinical trial (Table 1.2). Among observational studies, two common designs are cohort studies, in which observations are made in a group of subjects that is followed over time, and cross-sectional studies, in which observations are made on a single occasion. Cohort studies can be further divided into prospective studies that begin in the present and follow subjects into the future, and retrospective studies that examine information collected over a period of time in the past. A third common option is the case–control design, in which the investigator compares a group of people who have a disease or other outcome with another group who do not. Among clinical trial options, the randomized blinded trial is 4 Section I • Basic Ingredients TABLE 1.2 EXAMPLES OF CLINICAL RESEARCH DESIGNS TO FIND OUT WHETHER FISH INTAKE REDUCES CORONARY HEART DISEASE RISK EPIDEMIOLOGIC DESIGN KEY FEATURE EXAMPLE Cohort study A group of subjects identified at the beginning and followed over time The investigator measures fish intake in a group of subjects at baseline and periodically examines them at followup visits to see if those who eat more fish have fewer coronary heart disease (CHD) events. Cross-sectional study A group examined at one point in time She interviews a group of subjects about current and past history of fish intake and correlates results with history of CHD and current coronary calcium score. Case–control study Two groups selected based on the presence or absence of an outcome She examines a group of patients with CHD (the “cases”) and compares them with a group who do not have CHD (the “controls”), asking about past fish intake. Observational Designs Clinical Trial Design Randomized blinded trial Two groups created by a ­random process, and a blinded intervention She randomly assigns subjects to receive fish oil supplements or a placebo that is identical in appearance, then follows both treatment groups for several years to observe the incidence of CHD. usually the best design but nonrandomized or unblinded designs may be all that are feasible for some research questions. No one approach is always better than the others, and each research question requires a judgment about which design is the most efficient way to get a satisfactory answer. The randomized blinded trial is often held up as the best design for establishing causality and the effectiveness of interventions, but there are many situations for which an observational study is a better choice or the only feasible option. The relatively low cost of case–control studies and their suitability for rare outcomes makes them attractive for some questions. Special considerations apply to choosing designs for studying diagnostic tests. These issues are discussed in Chapters 7 through 12, each dealing with a particular set of designs. A typical sequence for studying a topic begins with observational studies of a type that is often called descriptive. These studies explore the lay of the land—for example, describing distributions of health-related characteristics and diseases in the population: • What is the average number of servings of fish per week in the diet of Americans with a his- tory of coronary heart disease (CHD)? Descriptive studies are usually followed or accompanied by analytic studies that evaluate associations to permit inferences about cause-and-effect relationships: • Do people with a CHD who eat a lot of fish have a lower risk of recurrent myocardial infarc- tion than people with a history of CHD who rarely eat fish? The final step is often a clinical trial to establish the effects of an intervention: • Does treatment with fish oil capsules reduce total mortality in people with CHD? Chapter 1 • Getting Started: The Anatomy and Physiology of Clinical Research 5 Clinical trials usually occur relatively late in a series of research studies about a given question, because they tend to be more difficult and expensive, and to answer more definitively the narrowly focused questions that arise from the findings of observational studies. It is useful to characterize a study in a single sentence that summarizes the design and research question. If the study has two major phases, the design for each should be mentioned. • This is a cross-sectional study of dietary habits in 50- to 69-year-old people with a history of CHD, followed by a prospective cohort study of whether fish intake is associated with lower risk of subsequent coronary events. This sentence is the research analog to the opening sentence of a medical resident’s report on a new hospital admission: “This 62-year-old white policewoman was well until 2 hours before admission, when she developed crushing chest pain radiating to the left shoulder.” Some designs do not easily fit into the categories listed above, and classifying them with a single sentence can be surprisingly difficult. It is worth the effort—a concise description of the design and research question clarifies the investigator’s thoughts and is useful for orienting colleagues and consultants. Study Subjects Two major decisions must be made in choosing the study subjects (Chapter 3). The first is to specify inclusion and exclusion criteria that define the target population: the kinds of people best suited to the research question. The second decision concerns how to recruit an appropriate number of people from an accessible subset of this population to be the subjects of the study. For example, the study of fish intake in people with CHD might identify subjects seen in the clinic with diagnostic codes for myocardial infarction, angioplasty, or coronary artery bypass grafting in their electronic medical record. Decisions about which patients to study often represent trade-offs; studying a random sample of people with CHD from the entire country (or at least several different states and medical care settings) would enhance generalizability but be much more difficult and costly. Variables Another major set of decisions in designing any study concerns the choice of which variables to measure (Chapter 4). A study of fish intake in the diet, for example, might ask about different types of fish that contain different levels of omega-3 fatty acids, and include questions about portion size, whether the fish was fried or baked, and use of fish oil supplements. In an analytic study the investigator studies the associations among variables to predict outcomes and to draw inferences about cause and effect. In considering the association between two variables, the one that occurs first or is more likely on biologic grounds to be causal is called the predictor variable; the other is called the outcome variable.1 Most observational studies have many predictor variables (age, race, sex, smoking history, fish and fish oil supplement intake) and several outcome variables (heart attacks, strokes, quality of life, unpleasant odor). Clinical trials examine the effects of an intervention—a special kind of predictor variable that the investigator manipulates, such as treatment with fish oil capsules. This design allows her to observe the effects on the outcome variable using randomization to minimize the influence of confounding variables—other predictors of the outcome such as smoking or income level that could be associated with dietary fish and confuse the interpretation of the findings. 1 Predictors are sometimes termed independent variables and outcomes dependent variables, but the meaning of these terms is less self-evident and we prefer to avoid their use. 6 Section I • Basic Ingredients Statistical Issues The investigator must develop plans for estimating sample size and for managing and analyzing the study data. This generally involves specifying a hypothesis (Chapter 5). Hypothesis: 50- to 69-year-old women with CHD who take fish oil supplements will have a lower risk of recurrent myocardial infarction than those who do not. This is a version of the research question that provides the basis for testing the statistical significance of the findings. The hypothesis also allows the investigator to calculate the sample size—the number of subjects needed to observe the expected difference in outcome between study groups with reasonable probability (an attribute known as power) (Chapter 6). Purely descriptive studies (what proportion of people with CHD use fish oil supplements?) do not involve tests of statistical significance, and thus do not require a hypothesis; instead, the number of subjects needed to produce acceptably narrow confidence intervals for means, proportions, or other descriptive statistics can be calculated. ■ PHYSIOLOGY OF RESEARCH: HOW IT WORKS The goal of clinical research is to draw inferences from findings in the study about the nature of the universe around it. Two major sets of inferences are involved in interpreting a study ­(illustrated from right to left in Figure 1.1). Inference #1 concerns internal validity, the degree to which the investigator draws the correct conclusions about what actually happened in the study. Inference #2 concerns external validity (also called generalizability), the degree to which these conclusions can be appropriately applied to people and events outside the study. When an investigator plans a study, she reverses the process, working from left to right in the lower half of Figure 1.1 with the goal of maximizing the validity of these inferences at the end of the study. She designs a study plan in which the choice of research question, subjects, and measurements enhances the external validity of the study and is conducive to implementation with a high degree of internal validity. In the next sections we address design and then implementation before turning to the errors that threaten the validity of these inferences. Designing the Study Consider this simple descriptive question: What is the prevalence of daily ingestion of fish oil supplements among people with CHD? This question cannot be answered with perfect accuracy because it would be impossible to study all patients with CHD and our approaches to discovering whether a person has CHD Drawing Conclusions TRUTH IN THE UNIVERSE Designing and Implementing Research question Infer Design EXTERNAL VALIDITY TRUTH IN THE STUDY Study plan Infer Implement FINDINGS IN THE STUDY Actual study INTERNAL VALIDITY ■ FIGURE 1.1 The process of designing and implementing a research project sets the stage for drawing conclusions based on inferences from the findings. Chapter 1 • Getting Started: The Anatomy and Physiology of Clinical Research 7 and is taking fish oil are imperfect. So the investigator settles for a related question that can be answered by the study: Among a sample of patients seen in the investigator’s clinic who have a previous CHD ­diagnosis and respond to a mailed questionnaire, what proportion report taking daily fish oil supplements? The transformation from research question to study plan is illustrated in Figure 1.2. One major component of this transformation is the choice of a sample of subjects that will represent the population. The group of subjects specified in the protocol can only be a sample of the population of interest because there are practical barriers to studying the entire population. The decision to study patients in the investigator’s clinic identified through the electronic medical record system is a compromise. This is a sample that is feasible to study but has the disadvantage that it may produce a different prevalence of fish oil use than that found in all people with CHD. The other major component of the transformation is the choice of variables that will represent the phenomena of interest. The variables specified in the study plan are usually proxies for these phenomena. The decision to use a self-report questionnaire to assess fish oil use is a fast and inexpensive way to collect information, but unlikely to be perfectly accurate because people usually do not accurately remember or record how much they take in a typical week. In short, each of the differences in Figure 1.2 between the research question and the study plan has the purpose of making the study more practical. The cost of this increase in practicality, however, is the risk that design choices may cause the study to produce a wrong or misleading conclusion because it is designed to answer a somewhat different question from the research question of interest. Implementing the Study Returning to Figure 1.1, the right-hand side is concerned with implementation and the degree to which the actual study matches the study plan. At issue here is the problem of a wrong answer TRUTH IN THE UNIVERSE Infer TRUTH IN THE STUDY Errors Research question Study plan Design Target population People with CHD Intended sample All patients with a history of CHD seen in clinic last year Phenomena of interest The proportion who take fish oil supplements Intended variables Self-reported use of fish oil supplements EXTERNAL VALIDITY ■ FIGURE 1.2 Design errors and external validity: If the intended sample and variables do not sufficiently represent the target population and phenomena of interest, these errors may distort inferences about what actually happens in the population. 8 Section I • Basic Ingredients to the research question because the way the sample was actually drawn, or the measurements made, differed in important ways from the way they were designed (Figure 1.3). The actual sample of study subjects is almost always different from the intended sample. The plans to study all eligible clinic patients with CHD, for example, could be disrupted by incomplete diagnoses in the electronic medical record, wrong addresses for the mailed questionnaire, and refusal to participate. Those subjects who are reached and agree to participate may have a different prevalence of fish oil use than those not reached or not interested. In addition to these problems with the subjects, the actual measurements can differ from the intended measurements. If the format of the questionnaire is unclear subjects may get confused and check the wrong box, or they may simply omit the question by mistake. These differences between the study plan and the actual study can alter the answer to the research question. Figure 1.3 illustrates that errors in implementing the study join errors of design in leading to a misleading or wrong answer to the research question. Causal Inference A special kind of validity problem arises in studies that examine the association between a predictor and an outcome variable in order to draw causal inference. If a cohort study finds an association between fish intake and CHD events, does this represent cause and effect, or is fish intake just an innocent bystander in a web of causation that involves other variables? Reducing the likelihood of confounding and other rival explanations is one of the major challenges in designing an observational study (Chapter 9). The Errors of Research Recognizing that no study is entirely free of errors, the goal is to maximize the validity of inferences from what was observed in the study sample to what is happening in the population. TRUTH IN THE STUDY Infer FINDINGS IN THE STUDY Errors Study plan Intended sample Actual study implement Actual subjects All 215 patients with a history of CHD seen in clinic last year The 104 patients with a CHD diagnosis in chart last year who filled out questionnaire Intended variables Actual measurements Self-reported use of fish oil supplements Responses to questionnaire items on fish oil supplements INTERNAL VALIDITY ■ FIGURE 1.3 Implementation errors and internal validity: If the actual subjects and measurements do not sufficiently represent the intended sample and variables, these errors may distort inferences about what happened in the study. Chapter 1 • Getting Started: The Anatomy and Physiology of Clinical Research 9 Erroneous inferences can be addressed in the analysis phase of research, but a better strategy is to focus on design and implementation (Figure 1.4), preventing errors from occurring in the first place to the extent that this is practical. The two main kinds of errors that interfere with research inferences are random error and systematic error. The distinction is important because the strategies for minimizing them are quite different. Random error is a wrong result due to chance—sources of variation that are equally likely to distort measurements from the study in either direction. If the true prevalence of daily fish oil supplement use in the several hundred 50- to 69-year-old patients with CHD in the investigator’s clinic is 20%, a well-designed sample of 100 patients from that population might contain exactly 20 patients who use these supplements. More likely, however, the sample would contain a nearby number such as 18, 19, 21, or 22. Occasionally, chance would produce a substantially different number, such as 12 or 28. Among several techniques for reducing the influence of random error (Chapter 4), the simplest is to increase the sample size. The use of a larger sample diminishes the likelihood of a substantially wrong result by increasing the precision of the estimate—the degree to which the observed prevalence approximates 20% each time a sample is drawn. Systematic error is a wrong result due to bias—sources of variation that distort the study findings in one direction. An illustration is the decision in Figure 1.2 to study patients in the investigator’s clinic, where the local treatment patterns have responded to her interest in the topic and her fellow doctors are more likely than the average doctor to recommend fish oil. Increasing the sample size has no effect on systematic error. The best way to improve the ­accuracy of the estimate (the degree to which it approximates the true value) is to design the study in a way that reduces the size of the various biases. Alternatively, the investigator can seek additional information to assess the importance of possible biases. An example would be to compare results with those from a second sample of patients with CHD drawn from another setting, for example, examining whether the findings of such patients seen in a cardiology clinic are different from those seen in a primary care clinic. The examples of random and systematic error in the preceding two paragraphs are components of sampling error, which threatens inferences from the study subjects to the population. Infer Error Random error Solution Improve design (Ch. 7–13) Enlarge sample size 5 strategies to increase precision (Ch. 4) Infer Error Random error Solution Quality control (Ch.17) Systematic Improve design (Ch. 7–13) 7 strategies to increase error accuracy (Ch. 4) Systematic Quality control (Ch.17) error Design Implement EXTERNAL VALIDITY INTERNAL VALIDITY ■ FIGURE 1.4 Research errors. This blown-up detail of the error boxes in Figures 1.2 and 1.3 reveals strategies for controlling random and systematic error in the design and implementation phases of the study. 10 Section I • Basic Ingredients Infer TRUTH IN THE UNIVERSE Research question Infer TRUTH IN THE STUDY Random and systematic error Target population Study plan FINDINGS IN THE STUDY Random and systematic error Actual subjects Intended sample Design Phenomena of interest Actual study Implement Actual measurements Intended variables EXTERNAL VALIDITY INTERNAL VALIDITY ■ FIGURE 1.5 Physiology of research—how it works. Both random and systematic errors can also contribute to measurement error, threatening the inferences from the study measurements to the phenomena of interest. An illustration of random measurement error is the variation in the response when the diet questionnaire is administered to the patient on several occasions. An example of systematic measurement error is underestimation of the prevalence of fish oil use due to lack of clarity in how the question is phrased. Additional strategies for controlling all these sources of error are presented in Chapters 3 and 4. The concepts presented in the last several pages are summarized in Figure 1.5 Getting the right answer to the research question is a matter of designing and implementing the study in a fashion that minimizes the magnitude of inferential errors. ■ DESIGNING THE STUDY Study Plan The process of developing the study plan begins with the one-sentence research question that specifies the main predictor and outcome variables and the population. Three versions of the study plan are then produced in sequence, each larger and more detailed than the preceding one. • Study outline (Table 1.1 and Appendix 1). This one-page summary of the design serves as a standardized checklist to remind the investigator to address all the components. As important, the sequence has an orderly logic that helps clarify the investigator’s thinking on the topic. • Study protocol. This expansion on the study outline usually ranges from 5 to 15 pages, and is used to plan the study and to apply for IRB approval and grant support. The protocol parts are discussed throughout this book and summarized in Chapter 19. • Operations manual. This collection of specific procedural instructions, questionnaires, and other materials is designed to ensure a uniform and standardized approach to carrying out the study with good quality control (Chapters 4 and 17). The research question and study outline should be written out at an early stage. Putting thoughts down on paper leads the way from vague ideas to specific plans and provides a concrete basis for getting advice from colleagues and consultants. It is a challenge to do it (ideas are easier to talk about than to write down), but the rewards are a faster start and a better project. Appendix 1 is an example of a study outline. This one-page outline deals more with the anatomy of research (Table 1.1) than with its physiology (Figure 1.5), so the investigator must remind herself to worry about the errors that may result when it is time to draw inferences Chapter 1 • Getting Started: The Anatomy and Physiology of Clinical Research 11 from measurements in the study sample to phenomena of interest in the population. A study’s virtues and problems can be revealed by explicitly considering how the question the study is likely to answer differs from the research question, given the plans for acquiring subjects and making measurements, and given the likely problems of implementation. With the study outline in hand and the intended inferences in mind, the investigator can proceed with the details of her protocol. This includes getting advice from colleagues, drafting specific recruitment and measurement methods, considering scientific and ethical appropriateness, modifying the study question and outline as needed, pretesting specific recruitment and measurement methods, making more changes, getting more advice, and so forth. This iterative process is the nature of research design and the topic of the rest of this book. Trade-offs Regretably, errors are an inherent part of all studies. The main issue is whether the errors will be large enough to change the conclusions in important ways. When designing a study, the investigator is in much the same position as a labor union official bargaining for a new contract. The union official begins with a wish list—shorter hours, more money, health care benefits, and so forth. She must then make concessions, holding on to the things that are most important and relinquishing those that are not essential or realistic. At the end of the negotiations is a vital step: She looks at the best contract she could negotiate and decides if it has become so bad that it is no longer worth having. The same sort of concessions must be made by an investigator when she transforms the research question to the study plan and considers potential problems in implementation. On one side are the issues of internal and external validity; on the other, feasibility. The vital last step of the union negotiator is sometimes omitted. Once the study plan has been formulated, the investigator must decide whether it adequately addresses the research question and whether it can be implemented with acceptable levels of error. Often the answer is no, and there is a need to begin the process anew. But take heart! Good scientists distinguish themselves not so much by their uniformly good research ideas as by their alacrity in turning over those that won’t work and moving on to better ones. ■ SUMMARY 1. The anatomy of research is the set of tangible elements that make up the study plan: the research question and its significance, and the design, study subjects, and measurement approaches. The challenge is to design elements that are relatively inexpensive and easy to implement. 2. The physiology of research is how the study works. The study findings are used to draw inferences about what happened in the study sample (internal validity), and about events in the world outside (external validity). The challenge here is to design and implement a study plan with adequate control over two major threats to these inferences: random error (chance) and systematic error (bias). 3. In designing a study the investigator may find it helpful to consider Figure 1.5, the relationships between the research question (what she wants to answer), the study plan (what the study is designed to answer), and the actual study (what the study will actually answer, given the errors of implementation that can be anticipated). 4. A good way to develop the study plan is to begin with a one-sentence version of the ­research question that specifies the main variables and population, and expand this into a one-page outline that sets out the study elements in a standardized sequence. Later on the study plan will be expanded into the protocol and the operations manual. 5. Good judgment by the investigator and advice from colleagues are needed for the many trade-offs involved, and for determining the overall viability of the project. APPENDIX 1 Outline of a Study This is the one-page study plan of a project carried out by Valerie Flaherman, MD, MPH, begun while she was a general pediatrics fellow at UCSF. Most beginning investigators find observational studies easier to pull off, but in this case a randomized clinical trial of modest size and scope was feasible, the only design that could adequately address the research question, and ultimately successful—see publication by Flaherman et al (1) for the findings, which, if confirmed, could alter policy on how best to initiate breast feeding. ■ Title: Effect of Early Limited Formula Use on Breastfeeding Research question: Among term newborns who have lost ≥ 5% of their birth weight before 36 hours of age, does feeding 10 cc of formula by syringe after each breastfeeding before the onset of mature milk production increase the likelihood of subsequent successful breastfeeding? Significance: 1. Breast milk volume is low until mature milk production begins 2–5 days after birth. 2. Some mothers become worried if the onset of mature milk production is late and their baby loses a lot of weight, leading them to abandon breastfeeding within the first week. A strategy that increased the proportion of mothers who succeed in breastfeeding would have many health and psycho-social benefits to mother and child. 3. Observational studies have found that formula feeding in the first few days after birth is associated with decreased breastfeeding duration. Although this could be due to confounding by indication (see Chapter 9), the finding has led to WHO and CDC guidelines aimed at reducing the use of formula during the birth hospitalization. 4. However, a small amount of formula combined with breastfeeding and counseling might make the early breastfeeding experience more positive and increase the likelihood of success. A clinical trial is needed to assess possible benefits and harms of this strategy. Study design: Unblinded randomized control trial with blinded outcome ascertainment Subjects: • Entry criteria: Healthy term newborns 24–48 hours old who have lost ≥ 5% of their birth weight in the first 36 hours after birth • Sampling design: Consecutive sample of consenting patients in two Northern California academic medical centers Predictor variable, randomly assigned but not blinded: • Control: Parents are taught infant soothing techniques. • Intervention: Parents are taught to syringe-feed 10 cc of formula after each breastfeeding until the onset of mature milk production. 12 Chapter 1 • Getting Started: The Anatomy and Physiology of Clinical Research 13 Outcome variables, blindly ascertained: 1. Any formula feeding at 1 week and 1, 2, and 3 months 2. Any breastfeeding at 1 week and 1, 2, and 3 months 3. Weight nadir Primary null hypothesis: Early limited formula does not affect the proportion of women who are breastfeeding their baby at 3 months. Reference 1. Flaherman VJ, Aby J, Burgos AE, et al. Effect of early limited formula on duration and exclusivity of breastfeeding in at-risk infants: an RCT. Pediatrics, in press. CHAPTER 2 Conceiving the Research Question and Developing the Study Plan Steven R. Cummings, Warren S. Browner, and Stephen B. Hulley T he research question is the uncertainty that the investigator wants to resolve by ­performing her study. There is no shortage of good research questions, and even as we succeed in answering some questions, we remain surrounded by others. Clinical trials, for example, established that treatments that block the synthesis of estradiol (aromatase inhibitors) reduce the risk of breast cancer in women who have had early stage cancer (1). But this led to new questions: How long should treatment be continued; does this treatment prevent breast cancer in patients with BRCA 1 and BRCA 2 mutations; and what is the best way to prevent the osteoporosis that is an adverse effect of these drugs? Beyond that are primary prevention questions: Are these treatments effective and safe for preventing breast cancer in healthy women? The challenge in finding a research question is defining an important one that can be transformed into a feasible and valid study plan. This chapter presents strategies for accomplishing this (Figure 2.1). ■ ORIGINS OF A RESEARCH QUESTION For an established investigator the best research questions usually emerge from the findings and problems she has observed in her own prior studies and in those of other workers in the field. A new investigator has not yet developed this base of experience. Although a fresh perspective is sometimes useful by allowing a creative person to conceive new approaches to old problems, lack of experience is largely an impediment. Infer Infer TRUTH IN THE UNIVERSE TRUTH IN THE STUDY Error FINDINGS IN THE STUDY Error Actual study Study plan Research question Design Implement Target population Intended sample Actual subjects Phenomena of interest Intended variables Actual measurements EXTERNAL VALIDITY INTERNAL VALIDITY ■ FIGURE 2.1 This chapter focuses on the area within the dashed green line, the challenge of choosing a research question that is of interest and can be tackled with a feasible study plan. 14 Chapter 2 • Conceiving the Research Question and Developing the Study Plan 15 A good way to begin is to clarify the difference between a research question and a research interest. Consider this research question: • Does participation in group counseling sessions reduce the likelihood of domestic violence among women who have recently immigrated from Central America? This might be asked by someone whose research interest involves the efficacy of group counseling, or the prevention of domestic violence, or improving health in recent immigrants. The distinction between research questions and research interests matters because it may turn out that the specific research question cannot be transformed into a viable study plan, but the investigator can still address her research interest by asking a different question. Of course, it’s impossible to formulate a research question if you are not even sure about your research interest (beyond knowing that you’re supposed to have one). If you find yourself in this boat, you’re not alone: Many new investigators have not yet discovered a topic that interests them and is susceptible to a study plan they can design. You can begin by considering what sorts of research studies have piqued your interest when you’ve seen them in a journal. Or perhaps you were bothered by a specific patient whose treatment seemed inadequate or inappropriate: What could have been done differently that might have improved her outcome? Or one of your attending physicians told you that hypokalemia always caused profound thirst, and another said the opposite, just as dogmatically. Mastering the Literature It is important to master the published literature in an area of study: Scholarship is a necessary precursor to good research. A new investigator should conduct a thorough search of published literature in the areas pertinent to the research question and critically read important original papers. Carrying out a systematic review is a great next step for developing and establishing expertise in a research area, and the underlying literature review can serve as background for grant proposals and research reports. Recent advances may be known to active investigators in a particular field long before they are published. Thus, mastery of a subject entails participating in meetings and building relationships with ­experts in the field. Being Alert to New Ideas and Techniques In addition to the medical literature as a source of ideas for research questions, it is helpful to attend conferences in which new work is presented. At least as important as the formal presentations are the opportunities for informal conversations with other scientists at posters and during the breaks. A new investigator who overcomes her shyness and engages a speaker at the coffee break may find the experience richly rewarding, and occasionally she will have a new senior colleague. Even better, for a speaker known in advance to be especially relevant, it may be worthwhile to look up her recent publications and contact her in advance to arrange a meeting during the conference. A skeptical attitude about prevailing beliefs can stimulate good research questions. For example, it was widely believed that lacerations which extend through the dermis required sutures to assure rapid healing and a satisfactory cosmetic outcome. However, Quinn et al. noted personal experience and case series evidence that wounds of moderate size repair themselves regardless of whether wound edges are approximated (2). They carried out a randomized trial in which all patients with hand lacerations less than 2 cm in length received tap water irrigation and a 48-hour antibiotic dressing. One group was randomly assigned to have their wounds sutured, and the other group did not receive sutures. The suture group had a more painful and time-consuming treatment in the emergency room, but blinded assessment revealed similar time to healing and similar cosmetic results. This has now become a standard approach used in clinical practice. 16 Section I • Basic Ingredients The application of new technologies often generates new insights and questions about familiar clinical problems, which in turn can generate new paradigms (3). Advances in imaging and in molecular and genetic technologies, for example, have spawned translational research studies that have led to new treatments and tests that have changed clinical medicine. Similarly, taking a new concept, technology, or finding from one field and applying it to a problem in a different field can lead to good research questions. Low bone density, for example, is a risk factor for fractures. Investigators applied this technology to other outcomes and found that women with low bone density have higher rates of cognitive decline (4), stimulating research for factors, such as low endogenous levels of estrogen, that could lead to loss of both bone and memory. Keeping the Imagination Roaming Careful observation of patients has led to many descriptive studies and is a fruitful source of research questions. Teaching is also an excellent source of inspiration; ideas for studies often occur while preparing presentations or during discussions with inquisitive students. Because there is usually not enough time to develop these ideas on the spot, it is useful to keep them in a computer file or notebook for future reference. There is a major role for creativity in the process of conceiving research questions, imagining new methods to address old questions, and playing with ideas. Some creative ideas come to mind during informal conversations with colleagues over lunch; others arise from discussing recent research or your own ideas in small groups. Many inspirations are solo affairs that strike while preparing a lecture, showering, perusing the Internet, or just sitting and thinking. Fear of criticism or seeming unusual can prematurely quash new ideas. The trick is to put an unresolved problem clearly in view and allow the mind to run freely around it. There is also a need for tenacity, returning to a troublesome problem repeatedly until a resolution is reached. Choosing and Working with a Mentor Nothing substitutes for experience in guiding the many judgments involved in conceiving a research question and fleshing out a study plan. Therefore an essential strategy for a new investigator is to apprentice herself to an experienced mentor who has the time and interest to work with her regularly. A good mentor will be available for regular meetings and informal discussions, encourage creative ideas, provide wisdom that comes from experience, help ensure protected time for research, open doors to networking and funding opportunities, encourage the development of independent work, and put the new investigator’s name first on grants and publications whenever appropriate. Sometimes it is desirable to have more than one mentor, representing different disciplines. Good relationships of this sort can also lead to tangible resources that are needed—office space, access to clinical populations, data sets and specimen banks, specialized laboratories, financial resources, and a research team. A bad mentor, on the other hand, can be a barrier. A mentor can harm the career of the new investigator, for example, by taking credit for findings that arise from the new investigator’s work, or assuming the lead role on publishing or presenting it. More commonly, many mentors are simply too busy or distracted to pay attention to the new investigator’s needs. In either case, once discussions with the mentor have proved fruitless, we recommend finding a way to move on to a more appropriate advisor, perhaps by involving a neutral senior colleague to help in the negotiations. Changing mentors can be hazardous, emphasizing the importance of choosing a good mentor in the first place; it is perhaps the single most important decision a new investigator makes. Your mentor may give you a database and ask you to come up with a research question. In that situation, it’s important to identify (1) the overlap between what’s in the database and your own research interests, and (2) the quality of the database. If there isn’t enough overlap or the data are irrevocably flawed, find a way to move on to another project. Chapter 2 • Conceiving the Research Question and Developing the Study Plan 17 ■ CHARACTERISTICS OF A GOOD RESEARCH QUESTION The characteristics of a research question that lead to a good study plan are that it be Feasible, Interesting, Novel, Ethical, and Relevant (which form the mnemonic FINER; Table 2.1). Feasible It is best to know the practical limits and problems of studying a question early on, before wasting much time and effort along unworkable lines. • Number of subjects. Many studies do not achieve their intended purposes because they can- not enroll enough subjects. A preliminary calculation of the sample size requirements of the study early on can be quite helpful (Chapter 6), together with an estimate of the number of subjects likely to be available for the study, the number who would be excluded or refuse to participate, and the number who would be lost to follow-up. Even careful planning often produces estimates that are overly optimistic, and the investigator should assure that there are enough eligible and willing subjects. It is sometimes necessary to carry out a pilot survey or chart review to be sure. If the number of subjects appears insufficient, the investigator can consider several strategies: expanding the inclusion criteria, eliminating unnecessary exclusion criteria, lengthening the time frame for enrolling subjects, acquiring additional sources of subjects, developing more precise measurement approaches, inviting colleagues to join in a multicenter study, and using a different study design. • Technical expertise. The investigators must have the skills, equipment, and experience needed for designing the study, recruiting the subjects, measuring the variables, and managing and analyzing the data. Consultants can help to shore up technical aspects that are unfamiliar to the investigators, but for major areas of the study it is better to have an experienced colleague steadily involved as a coinvestigator; for example, it is wise to include a statistician as a member of the research team from the beginning of the planning process. It is best to use familiar and established approaches, because the process of developing new TABLE 2.1 FINER CRITERIA FOR A GOOD RESEARCH QUESTION AND STUDY PLAN Feasible Adequate number of subjects Adequate technical expertise Affordable in time and money Manageable in scope Fundable Interesting Getting the answer intrigues the investigator and her colleagues Novel Provides new findings Confirms, refutes, or extends previous findings May lead to innovations in concepts of health and disease, medical practice, or methodologies for research Ethical A study that the institutional review board will approve Relevant Likely to have significant impacts on scientific knowledge, clinical practice, or health policy May influence directions of future research 18 Section I • Basic Ingredients methods and skills is time-consuming and uncertain. When a new approach is needed, such as measurement of a new biomarker, expertise in how to accomplish the innovation should be sought. • Cost in time and money. It is important to estimate the costs of each component of the project, bearing in mind that the time and money needed will generally exceed the amounts projected at the outset. If the projected costs exceed the available funds, the only options are to consider a less expensive design or to develop additional sources of funding. Early recognition of a study that is too expensive or time-consuming can lead to modification or abandonment of the plan before expending a great deal of effort. • Scope. Problems often arise when an investigator attempts to accomplish too much, making many measurements at repeated contacts with a large group of subjects in an effort to answer too many research questions. The solution is to narrow the scope of the study and focus only on the most important goals. Many scientists find it difficult to give up the opportunity to answer interesting side questions, but the reward may be a better answer to the main ­question at hand. • Fundability. Few investigators have the personal or institutional resources to fund their own research projects, particularly if subjects need to be enrolled and followed, or expensive measurements must be made. The most elegantly designed research proposal will not be feasible if no one will pay for it. Finding sources of funding is discussed in Chapter 19. Interesting An investigator may have many motivations for pursuing a particular research question: because it will provide financial support, because it is a logical or important next step in building a career, or because getting at the truth of the matter is interesting. We like this last reason; it is one that grows as it is exercised and that provides the intensity of effort needed for overcoming the many hurdles and frustrations of the research process. However, it is wise to confirm that you are not the only one who finds a question interesting. Speak with mentors, outside experts, and representatives of potential funders such as NIH project officers before devoting substantial energy to develop a research plan or grant proposal that peers and funding agencies may consider dull. Novel Good clinical research contributes new information. A study that merely reiterates what is already established is not worth the effort and cost and is unlikely to receive funding. The novelty of a proposed study can be determined by thoroughly reviewing the literature, consulting with experts who are familiar with unpublished ongoing research, and searching for abstracts of projects in your area of interest that have been funded using the NIH Research Portfolio Online Reporting Tools (RePORT) website (http://report.nih.gov/categorical_spending.aspx.) Reviews of studies submitted to NIH give considerable weight to whether a proposed study is innovative (5) such that a successful result could shift paradigms of research or clinical practice through the use of new concepts, methods, or interventions (Chapter 19). Although novelty is an important criterion, a research question need not be totally original—it can be worthwhile to ask whether a previous observation can be replicated, whether the findings in one population also apply to others, or whether a new measurement method can clarify the relationship between known risk factors and a disease. A confirmatory study is particularly useful if it avoids the weaknesses of previous studies or if the result to be confirmed was unexpected. Ethical A good research question must be ethical. If the study poses unacceptable physical risks or invasion of privacy (Chapter 14), the investigator must seek other ways to answer the question. Chapter 2 • Conceiving the Research Question and Developing the Study Plan 19 If there is uncertainty about whether the study is ethical, it is helpful to discuss it at an early stage with a representative of the institutional review board (IRB). Relevant A good way to decide about relevance is to imagine the various outcomes that are likely to occur and consider how each possibility might advance scientific knowledge, influence practice guidelines and health policy, or guide further research. NIH reviewers emphasize the significance of a proposed study: the importance of the problem, how the project will improve scientific knowledge, and how the result will change concepts, methods, or clinical services. ■ DEVELOPING THE RESEARCH QUESTION AND STUDY PLAN It helps a great deal to write down the research question and a brief (one-page) outline of the study plan at an early stage (Appendix 1). This requires some self-discipline, but it forces the investigator to clarify her ideas about the plan and to discover specific problems that need attention. The outline also provides a basis for specific suggestions from colleagues. Problems and Approaches Two complementary approaches to the problems involved in developing a research question deserve special emphasis. The first is the importance of getting good advice. We recommend a research team that includes representatives of each of the major disciplines involved in the study, and that includes at least one senior scientist. In addition, it is a good idea to consult with specialists who can guide the discovery of previous research on the topic and the choice and design of measurement techniques. Sometimes a local expert will do, but it is often useful to contact individuals in other institutions who have published pertinent work on the subject. A new investigator may be intimidated by the prospect of writing or calling someone she knows only as an author in the Journal of the American Medical Association, but most scientists respond favorably to such requests for advice. The second approach is to allow the study plan to gradually emerge from an iterative process of making incremental changes in the study’s design, estimating the sample size, reviewing with colleagues, pretesting key features, and revising. Once the one-page study outline is specified, formal review by colleagues will usually result in important improvements. As the protocol takes shape pilot studies of the availability and willingness of sufficient numbers of subjects may lead to changes in the recruitment plan. The preferred imaging test may turn out to be prohibitively costly and a less expensive alternative sought. Primary and Secondary Questions Many studies have more than one research question. Experiments often address the effect of the intervention on more than one outcome; for example, the Women’s Health Initiative was designed to determine whether reducing dietary fat intake would reduce the risk of breast ­cancer, but an important secondary hypothesis was to examine the effect on coronary events (5). Almost all cohort and case–control studies look at several risk factors for each outcome. The advantage of designing a study with several research questions is the efficiency that can result, with several answers emerging from a single study. The disadvantages are the increased complexity of designing and implementing the study and of drawing statistical inferences when there are multiple hypotheses (Chapter 5). A sensible strategy is to establish a single primary research question around which to focus the study plan and sample size estimate, adding ­secondary research questions about other predictors or outcomes that may also produce v­ aluable conclusions. 20 Section I • Basic Ingredients ■ TRANSLATIONAL RESEARCH Translational research refers to studies of how to translate findings from the ivory tower into the “real world,” how to assure that scientific creativity has a favorable impact on public health. Translational research (6) comes in two main flavors (Figure 2.2): • Applying basic science findings from laboratory research in clinical studies of patients (sometimes abbreviated as T1 research), and • Applying the findings of these clinical studies to alter health practices in the community (sometimes abbreviated as T2 research). Both forms of translational research require identifying a “translation” opportunity. Just as a literary translator first needs to find a novel or poem that merits translating, a translational research investigator must first target a scientific finding or new technology that could have an important impact on clinical research, practice, or public health. Among the strategies for making this challenging choice, it may be helpful to pay attention to colleagues when they talk about their latest findings, to presentations at national meetings about novel methods, and to speculation about mechanisms in published reports. Translating from Laboratory to Clinical Research (T1) A host of tools have become available for clinical investigations, including DNA sequencing, gene expression arrays, molecular imaging, and proteomics. From the viewpoint of a clinical investigator, there is nothing epidemiologically different about these novel measurements, technologies, or test results. The chapter on measurements will be useful in planning studies involving these types of measurements (Chapter 4), as will the advice about study design (Chapters 7–12), population samples (Chapter 3), and sample size (Chapter 6). Especially relevant to genomics and other “omics” will be the concern with multiple hypothesis testing (Chapter 5). Compared with ordinary clinical research, being a successful T1 translational investigator often requires having an additional skill set or working with a collaborator with those skills. Bench-tobedside research necessitates a thorough understanding of the underlying basic science. Although many clinical researchers believe that they can master this knowledge—just like many laboratory-based researchers believe doing clinical research requires no special training—in reality, the skills hardly overlap. For example, suppose a basic scientist has identified a gene that Laboratory Research Clinical Research T1 Population Research T2 ■ FIGURE 2.2 Translational research is the component of clinical research that interacts with basic science research (hatched area T1) or with population research (hatched area T2). Chapter 2 • Conceiving the Research Question and Developing the Study Plan 21 affects circadian rhythm in mice. A clinical investigator whose expertise is in sleep has access to a cohort study with data on sleep cycles and a bank of stored DNA, and wants to study whether there is an association between variants in the human homolog of that gene and sleep. In order to propose a T1 study of that association she needs collaborators who are familiar with that gene, as well as the advantages and limitations of the various methods of genotyping. Similarly, imagine that a laboratory-based investigator has discovered a unique pattern of gene expression in tissue biopsy samples from patients with breast cancer. She should not propose a study of its use as a test for predicting the risk of recurrence of breast cancer without collaborating with someone who understands the importance of clinical research issues, such as test-retest reliability, sampling and blinding, and the effects of prior probability of disease on the applicability of her discovery. Good translational research requires expertise in more than one area. Thus a research team interested in testing a new drug may need scientists familiar with molecular biology, pharmacokinetics, pharmacodynamics, phase I and II clinical trials, and practice patterns in the relevant field of medicine. Translating from Clinical to Population Research (T2) Studies that attempt to apply findings from clinical trials to larger and more diverse populations often require expertise in identifying high-risk or underserved groups, understanding the difference between screening and diagnosis, and knowing how to implement changes in health care delivery systems. On a practical level, this kind of research usually needs access to large groups of patients (or clinicians), such as those enrolled in health plans or large clinics. Support and advice from the department chair, the chief of the medical staff at an affiliated hospital, the leader of a managed care organization, or a representative from a community organization may be helpful when planning these studies. Some investigators take a short cut when doing this type of translational research, expanding a study in their own clinic by studying patients in their colleagues’ practices (e.g., a house staff-run clinic in an academic medical center) rather than involving practitioners in the community. This is a bit like translating Aristophanes into modern Greek—it will still not be very useful for English-speaking readers. Chapter 18 emphasizes the importance of getting as far into the community as possible. Testing research findings in larger populations often requires adapting methods to fit organizations. For example, in a study of whether a new office-based diet and exercise program will be effective in the community, it may not be possible to randomly assign individual patients. One solution would be to randomly assign physician practices instead. This may require collaborating with an expert on cluster sampling and clustered analyses. Many T2 research projects aimed to improve medical care use proxy “process” variables as their outcomes. For example, if clinical trials have established that a new treatment reduces mortality from sepsis, a translational research study comparing two programs for implementing and promoting use of the new treatment might not need to have mortality as the outcome. Rather, it might just compare the percentages of patients with sepsis who received the new treatment. Moving research from settings designed for research into organizations designed for medical care or other purposes requires flexibility and creativity in applying principles that assure as much rigor and validity of the study results as possible. ■ SUMMARY 1. All studies should start with a research question that addresses what the investigator would like to know. The goal is to find one that can be developed into a good study plan. 2. Scholarship is essential to developing research questions that are worth pursuing. A systematic review of research pertinent to an area of research interest is a good place to start. Attending conferences and staying alert to new results extends the investigator’s expertise beyond what is already published. 22 Section I • Basic Ingredients 3. The single most important decision a new investigator makes is her choice of one or two senior scientists to serve as her mentor(s): experienced investigators who will take time to meet, provide resources and connections, encourage creativity, and promote the independence and visibility of their junior scientists. 4. Good research questions arise from finding new collaborators at conferences, from critical thinking about clinical practices and problems, from applying new methods to old issues, and from considering ideas that emerge from teaching, daydreaming, and tenacious ­pursuit of solutions to vexing problems. 5. Before committing much time and effort to writing a proposal or carrying out a study, the investigator should consider whether the research question and study plan are “FINER”: feasible, interesting, novel, ethical, and relevant. Those who fund research give priority to proposals that may have innovative and significant impacts on science and health. 6. Early on, the research question should be developed into a one-page written study outline that specifically describes how many subjects will be needed, how the subjects will be selected, and what measurements will be made. 7. Developing the research question and study plan is an iterative process that includes consultations with advisors and friends, a growing familiarity with the literature, and pilot studies of the recruitment and measurement approaches. 8. Most studies have more than one question, and it is useful to focus on a single primary question in designing and implementing the study. 9. Translational research is a type of clinical research that studies the application of basic ­science findings in clinical studies of patients (T1) and how to apply these findings to improve health practices in the community (T2); it requires collaborations between ­laboratory and population-based investigators, using the clinical research methods ­presented in this book. REFERENCES 1. The ATAC Trialists Group. Anastrazole alone or in combination with tamoxifen versus tamoxifen alone for adjuvant treatment of postmenopausal women with early breast cancer: first results of the ATAC randomized trials. Lancet 2002;359:2131–2139. 2. Quinn J, Cummings S, Callaham M, et al. Suturing versus conservative management of lacerations of the hand: randomized controlled trial. BMJ 2002;325:299–301. 3. Kuhn TS. The structure of scientific revolutions. Chicago, IL: University of Chicago Press, 1962. 4. Yaffe K, Browner W, Cauley J, et al. Association between bone mineral density and cognitive decline in older women. J Am Geriatr Soc 1999;47:1176–1182. 5. Prentice RL, Caan B, Chlebowski RT, et al. Low-fat dietary pattern and risk of invasive breast cancer. JAMA 2006;295:629–642. 6. Zerhouni EA. US biomedical research: basic, translational and clinical sciences. JAMA 2005;294:1352–1358. CHAPTER 3 Choosing the Study Subjects: Specification, Sampling, and Recruitment Stephen B. Hulley, Thomas B. Newman, and Steven R. Cummings A good choice of study subjects serves the vital purpose of ensuring that the findings in the study accurately represent what is going on in the population of interest. The protocol must specify a sample of subjects that can be studied at an acceptable cost in time and money (i.e., modest in size and convenient to access), yet large enough to control random error and representative enough to allow generalizing study findings to populations of interest. An important precept here is that generalizability is rarely a simple yes-or-no matter; it is a complex qualitative judgment that depends on the investigator’s choice of population and of sampling design. We will come to the issue of choosing the appropriate number of study subjects in ­Chapter 6. In this chapter we address the process of specifying and sampling the kinds of subjects who will be representative and feasible (Figure 3.1). We also discuss strategies for recruiting these people to participate in the study. ■ BASIC TERMS AND CONCEPTS Populations and Samples A population is a complete set of people with specified characteristics, and a sample is a subset of the population. In lay usage, the characteristics that define a population tend to be Infer Infer TRUTH IN THE UNIVERSE TRUTH IN THE STUDY Error FINDINGS IN THE STUDY Error Study plan Research question Design Actual study Implement Target population Intended sample Actual subjects Phenomena of interest Intended variables Actual measurements EXTERNAL VALIDITY INTERNAL VALIDITY ■ FIGURE 3.1 This chapter focuses on choosing a sample of study subjects that represent the population of interest for the research question. 23 24 Section I • Basic Ingredients geographic—for example, the population of Canada. In research, the defining characteristics are also clinical, demographic, and temporal: • Clinical and demographic characteristics define the target population, the large set of people throughout the world to which the results may be generalized—teenagers with asthma, for example. • The accessible population is a geographically and temporally defined subset of the target population that is available for study—teenagers with asthma living in the investigator’s town this year. • The intended study sample is the subset of the accessible population that the investigator seeks to include in the study. • The actual study sample is the group of subjects that does participate in the study. Generalizing the Study Findings The classic Framingham Study was an early approach to scientifically designing a study to allow inferences from findings observed in a sample to be applied to a population (Figure 3.2). The sampling design called for identifying all the families in Framingham with at least one person aged 30–59, listing the families in order by address, and then asking age-­eligible persons in the first two of every set of three families to participate. This “systematic” ­sampling design is not as tamperproof as choosing each subject by a random process (as discussed later in this chapter), but two more serious concerns were the facts that one-third of the Framingham residents selected for the study refused to participate, and that in their place the investigators accepted age-eligible residents who were not in the sample and volunteered (1). Because respondents are often healthier than nonrespondents, especially if they are volunteers, the characteristics of the actual sample undoubtedly differed from those of the intended sample. Every sample has some errors, however, and the issue is how much damage has been TRUTH IN THE UNIVERSE Target population (GENERALIZATION FAIRLY SECURE) Same association exists in all suburban U.S. adults Accessible population EXTERNAL VALIDITY INFERENCE #2 Same association exists in all Framingham adults EXTERNAL VALIDITY INFERENCE #1 TRUTH IN THE STUDY FINDINGS IN THE STUDY Intended sample Same association exists in the designed sample of Framingham adults Actual subjects Association between hypertension and INTERNAL CHD observed in VALIDITY INFERENCE the actual sample of Framingham adults (GENERALIZATION LESS SECURE) Same association exists in: (a) Other U.S. adults (e.g., inner city Blacks) (b) People living in other countries (c) People living in 2030 (d) etc. ■ FIGURE 3.2 Inferences in generalizing from the study subjects to the target populations proceed from right to left. Chapter 3 • Choosing the Study Subjects: Specification, Sampling, and Recruitment TRUTH IN THE UNIVERSE 25 TRUTH IN THE STUDY STEP # 1: STEP # 2: STEP # 3: Target populations Accessible population Intended sample Specify clinical and demographic characteristics Specify temporal and geographical characteristics Design an approach to selecting the sample CRITERIA Well suited to the research question CRITERIA Representative of target populations and available CRITERIA Representative of accessible population and easy to study Specification Sampling ■ FIGURE 3.3 Steps in designing the protocol for choosing the study subjects. done. The Framingham Study sampling errors do not seem large enough to invalidate the conclusion that risk relationships observed in the study—for example, that hypertension is a risk factor for coronary heart disease (CHD)—can be generalized to all the residents of Framingham. The next concern is the validity of generalizing the finding that hypertension is a risk factor for CHD from the accessible population of Framingham residents to target populations elsewhere. This inference is more subjective. The town of Framingham was selected not with a scientific sampling design, but because it seemed fairly typical of middle-class white communities in the United States and was convenient to the investigators. The validity of generalizing the Framingham risk relationships to populations in other parts of the country involves the precept that, in general, analytic studies and clinical trials that address biologic relationships produce more widely generalizable results across diverse populations than descriptive studies that address distributions of characteristics. Thus, the strength of hypertension as a risk factor for CHD is similar in Caucasian Framingham residents to that observed in inner city African Americans, but the prevalence of hypertension is much higher in the latter population. Steps in Designing the Protocol for Acquiring Study Subjects The inferences in Figure 3.2 are presented from right to left, the sequence used for interpreting the findings of a completed study. An investigator who is planning a study reverses this sequence, beginning on the left (Figure 3.3). She begins by specifying the clinical and demographic characteristics of the target population that will serve the research question well. She then uses geographic and temporal criteria to specify a study sample that is representative and practical. ■ SELECTION CRITERIA If an investigator wants to study the efficacy of low dose testosterone supplements versus placebo for enhancing libido in postmenopausal women, she can begin by creating selection criteria that define the population to be studied. 26 Section I • Basic Ingredients Establishing Selection Criteria Inclusion criteria define the main characteristics of the target population that pertain to the research question (Table 3.1). Age is often a crucial factor, and in this study the investigator might decide to focus on women in their fifties, speculating that in this group the benefit-toharm ratio of the drug might be optimal; another study might make a different decision and focus on older decades. The investigator also might incorporate African American, Hispanic, and Asian women in the study in an effort to expand generalizability. This is generally a good idea, but it’s important to realize that the increase in generalizability is illusory if there is other evidence to suggest that the effects differ by race. In that case the investigator would need enough women of each race to statistically test for the presence of effect modification (an effect in one race that is different from that in other races, also known as “an interaction”; Chapter 9); the number needed is generally large, and most studies are not powered to detect effect modification. Inclusion criteria that address the geographic and temporal characteristics of the accessible population often involve trade-offs between scientific and practical goals. The investigator may find that patients at her own hospital are an available and inexpensive source of subjects. But she must consider whether peculiarities of the local referral patterns might interfere with generalizing the results to other populations. On these and other decisions about inclusion criteria, there is no single course of action that is clearly right or wrong; the important thing is to make decisions that are sensible, that can be used consistently throughout the study, and that can be clearly described to others who will be deciding to whom the published ­conclusions apply. TABLE 3.1 DESIGNING SELECTION CRITERIA FOR A CLINICAL TRIAL OF LOW DOSE TESTOSTERONE VERSUS PLACEBO TO ENHANCE LIBIDO IN MENOPAUSE DESIGN FEATURE Inclusion criteria (be specific) Exclusion criteria (be parsimonious) EXAMPLE Specifying populations relevant to the research question and efficient for study: Demographic characteristics Women 50 to 59 years old Clinical characteristics Good general health Has a sexual partner Is concerned about decreased libido Geographic (administrative) characteristics Patients attending clinic at the investigator’s hospital Temporal characteristics Between January 1 and December 31 of specified year Specifying subsets of the population that will not be studied because of: A high likelihood of being lost to follow-up Alcoholic Plans to move out of state An inability to provide good data Disoriented Has a language barrier* Being at high risk of possible adverse effects History of myocardial infarction or stroke *Alternatives to excluding those with a language barrier (when these subgroups are sizeable and important to the research question) would be collecting nonverbal data or using bilingual staff and questionnaires. Chapter 3 • Choosing the Study Subjects: Specification, Sampling, and Recruitment 27 Specifying clinical characteristics for selecting subjects often involves difficult judgments, not only about which factors are important to the research question, but about how to define them. How, for example, would an investigator put into practice the criterion that the subjects be in “good health”? She might decide not to include patients with any self-reported illness, but this would likely exclude large numbers of subjects who are perfectly suitable for the research question at hand. More reasonably, she might exclude only those with diseases that could interfere with follow-up, such as metastatic cancer. This would be an example of “exclusion criteria,” which indicate individuals who meet the inclusion criteria and would be suitable for the study were it not for characteristics that might interfere with the success of follow-up efforts, the quality of the data, or the acceptability of randomized treatment (Table 3.1). Difficulty with the English language, psychological problems, alcoholism, and serious illness are examples of exclusion criteria. Clinical trials differ from observational studies in being more likely to have exclusions mandated by concern for the safety of an intervention in certain patients; for example, the use of drugs in pregnant women (Chapter 10). A good general rule that keeps things simple and preserves the number of potential study subjects is to have as few exclusion criteria as possible. Clinical Versus Community Populations If the research question involves patients with a disease, hospitalized or clinic-based patients are easier to find, but selection factors that determine who comes to the hospital or clinic may have an important effect. For example, a specialty clinic at a tertiary care medical center attracts patients from afar with serious forms of the disease, giving a distorted impression of the features and prognosis that are seen in ordinary practice. Sampling from primary care practices can be a better choice. Another common option in choosing the sample is to select subjects in the community who represent a healthy population. These samples are often recruited using mail, e-mail, or advertising via Internet, broadcast, or print media; they are not fully representative of a general population because some kinds of people are more likely than others to volunteer or be active users of Internet or e-mail. True “population-based” samples are difficult and expensive to recruit, but useful for guiding public health and clinical practice in the community. One of the largest and best examples is the National Health and Nutrition Examination Survey (NHANES), a representative sample of U.S. residents. The size and diversity of a sample can be increased by collaborating with colleagues in other cities, or by using preexisting data sets such as NHANES and Medicare data. Electronically ­accessible data sets from public health agencies, healthcare providing organizations, and medical insurance companies have come into widespread use in clinical research and may be more representative of national populations and less time-consuming than other possibilities (Chapter 13). ■ SAMPLING Often the number of people who meet the selection criteria is too large, and there is a need to select a sample (subset) of the population for study. Nonprobability Samples In clinical research the study sample is often made up of people who meet the entry criteria and are easily accessible to the investigator. This is termed a convenience sample. It has obvious advantages in cost and logistics, and is a good choice for some research questions. A consecutive sample can minimize volunteerism and other selection biases by consecutively selecting subjects who meet the entry criteria. This approach is especially desirable, for example, when it amounts to taking the entire accessible population over a long enough period to include seasonal variations or other temporal changes that are important to the research question. 28 Section I • Basic Ingredients The validity of drawing inferences from any sample is the premise that, for the purpose of answering the research question at hand, it sufficiently represents the accessible population. With convenience samples this requires a subjective judgment. Probability Samples Sometimes, particularly with descriptive research questions, there is a need for a scientific basis for generalizing the findings in the study sample to the population. Probability sampling, the gold standard for ensuring generalizability, uses a random process to guarantee that each unit of the population has a specified chance of being included in the sample. It is a scientific approach that provides a rigorous basis for estimating the fidelity with which phenomena observed in the sample represent those in the population, and for computing statistical significance and confidence intervals. There are several versions of this approach. • A simple random sample is drawn by enumerating (listing) all the people in the population from which the sample will be drawn, and selecting a subset at random. The most common use of this approach in clinical research is when the investigator wishes to select a representative subset from a population that is larger than she needs. To take a random sample of the cataract surgery patients at her hospital, for example, the investigator could list all such patients on the operating room schedules for the period of study, then use a table of random numbers to select individuals for study (Appendix 3). • A systematic sample resembles a simple random sample in the first step, enumerating the population, but differs in that the sample is selected by a preordained periodic process (e.g., the Framingham approach of taki…

A Sample Answer For the Assignment: ASU SWP Pharmaceutical Protocol Case Study

Title: ASU SWP Pharmaceutical Protocol Case Study

Click here to ORDER an A++ paper from our Verified MASTERS and DOCTORATE WRITERS:ASU SWP Pharmaceutical Protocol Case Study