Observational Research: Choosing the right research approach for the right question

Fig. 1. Quality, time and cost elements of major research approaches.

Observational research—evaluating outcomes in patients in a naturalistic rather than an experimental setting–has the potential to answer many research questions. But in order to do so effectively, researchers must select “the right research approach for the right question.” Doing so requires thorough evaluation of feasibility of delivery, specialized operational approaches with an understanding of pitfalls to avoid, and an upfront plan to communicate the results to stakeholders for whom the results are relevant.

The starting point for selecting “right research/right question” is to refine the question into something specific that can be transformed into a viable study. The research question represents the uncertainty that the sponsor wants to resolve by performing the study. Sometimes, the research question is confused with the research interest, which is a broader concept. To refine the research question requires understanding the strategic objectives of any research, and reviewing the current body of literature and any ongoing research to ensure that the question is relevant, specific and novel.

The right research approach is determined by practical considerations such as time and budget, ethical considerations and the strength of evidence required to meet healthcare stakeholder needs. Various study design options exist, with varying strengths and limitations that can be evaluated against these considerations. For example, a randomized controlled trial (RCT), while providing the highest strength of evidence, might cost too much and take too long. There may be ethical questions concerning randomization. A cross-sectional study conducted in a database might be quick and cheap, but might not provide sufficient certainty of evidence for decision-making. All of these factors should be weighed against each other to determine the right research approach.

Feasibility considerations

The feasibility of a study is a key ingredient in its ability to generate the right quality evidence in the right time at the right cost. There are several research approaches to choose from (Fig. 1). The quality of data generated via each research approach varies, as does the time and cost to generate that evidence. In general, the time needed to gather data prospectively is longer than approaches using real-world data sources, and the cost of research can vary from less than $100,000 to tens of millions of dollars. RCTs and prospective observational cohort studies provide the most certainty of evidence for decision-makers, but take longest and cost most. In contrast, research using routine data sources—while relatively quick and cost-effective–may not be considered sufficient by some decision-makers.

The various types of research approaches—such as prospective observational cohort studies and patient registries, chart reviews, database studies, and hybrid designs–have different features, advantages and disadvantages.

For example, features of prospective observational cohort studies include the fact that they are non-randomized and non-interventional, and they monitor cohorts over time. Patient registries may focus on disease, product or exposure. Advantages include the fact that prospective observational cohort studies and patient registries can:

• Examine longer-term outcomes in populations typically excluded from trials
• Examine risks for uncommon harms & factors that modify risk
• Obtain more representative data on a range of outcomes
• Be more cost-effective than randomized trial designs
• Assess actual use (including off-label) to identify potential new indications.

Disadvantages of prospective observational cohort studies and patient registries include the fact that they are prone to bias and confounding. Optimal design, conduct and analysis are critical for producing strong evidence.

Chart reviews and database studies use existing clinical or administrative data present in patient paper charts or electronic databases. Advantages of using existing data sources include the ability to assess benefits and harms across an extremely large population, and the fact that this is a cost- and time-efficient approach compared to prospective, longitudinal research. Disadvantages are:

• Underlying information is not collected in a systematic way
• It is difficult to interpret missing data which may introduce bias
• Non-randomization of patients to treatment introduces the possibility of confounding
• Data abstraction from paper records is resource-intensive
• Complete medical and clinical histories may not be available
• Administrative databases and electronic health records are not uniformly available across all countries
• Privacy issues may create the need to aggregate data.

Fig. 2. Advantages and disadvantages of hybrid designs.

Hybrid designs can capitalize on the speed, cost-efficiency and strengths of different methods, using a combination of primary and secondary data to shorten time and lower costs (Fig. 2). With this approach, sub-studies—focusing on health economic data or patient reported outcomes–are conducted alongside larger prospective studies, providing added value.

For all types of study design, strategies to ensure a feasible study include:

• Early planning: Much of the data cited as difficult to provide to decision-makers around the time of launch can be captured during development, or at least planned for during Phase 3.
• Multifunctional collaboration within companies: This allows the right scientific, commercial and operational input into study designs.
• A knowledge of data gaps: This guides research questions, which in turn determine data needs. To gather required data demands an understanding of where these may exist in routine data sources or whether they must be gathered prospectively.

Specialized operational delivery

Importantly, observational research requires a different operational approach from that used in experimental clinical research. It is nonetheless common for companies to use clinical development teams to conduct observational research. Observational research requires specialized people, processes and technology in four major areas:

• Research question and study design should be informed by stakeholder needs in terms of endpoint definition, design and execution.
• Data sources, data tools and data elements should be selected with the needs of observational approaches in mind. Data need to be available and accessible. For a truly observational study, data should be captured as part of routine care and should not be requested purely for the purpose of the study. If a database is being used, it should be accessible and any necessary permissions obtained for data use. Most technologies for data capture in clinical research were designed with clinical trials in mind. Companies should consider technologies compatible with usual care. Selection of data elements is another area requiring a different approach from that for clinical trials. Potential for confounding and bias—which can invalidate the results of observational studies–must be understood. Failure to capture data elements critical to a full analysis of a study may lead to false conclusions; epidemiologists should be employed to advise teams on the selection of the right data elements.
• Site and patient recruitment and retention are key areas requiring specialized approaches. For many observational studies, it is important that sites are representative of the target population of sites. The starting point to achieve a representative sample of sites is to understand the target population profile. For example, to understand primary versus secondary care treatment, rural versus urban distribution, and for product registries, prescribing trends. Randomizing a site target list may appear an attractive choice but can lead to an insufficient site list once site interest has been determined. More appropriate is to determine site interest from the universe of sites and to back this list into the ideal site sample. Ensuring a representative sample of patients can also prove challenging when site enrollment may be uneven. Strategies such as patient caps might be employed but should be carefully balanced against time and cost considerations. Recruitment and retention strategies may be employed on observational studies but options are more limited than for clinical trials due to the need to avoid the introduction of bias.
• Statistical analysis, interpretation and reporting are also important. Specialized methodological approaches are critical to ensure appropriate use of analytical techniques and interpretation of that analysis.

In practice, observational research requires the full integration of strategy, science and operations throughout the lifecycle of every study. Regulatory and commercial teams are needed to provide global and local insights into evidentiary needs and the tools for evidence dissemination. Epidemiologists and outcomes researchers are involved in writing the study protocol; reviewing data management and biostatistical deliverables; leading site and patient sampling; reviewing data for missing elements, bias and confounding; and interpreting, analyzing and reporting data throughout the study. Epidemiological input is needed to all aspects of operational delivery to minimize bias and confounding. Dedicated observational research operational teams work on specialized observational study delivery processes, and leveraging fit-for-purpose data capture technology ensure high-quality delivery.

Risks to avoid

Several pitfalls may be encountered during the design and implementation of observational studies. First, researchers may mistakenly assume that such studies are “simple” to design and conduct working off a light-touch clinical trial delivery template. This is far from true. Indeed, applying clinical trial operational approaches may lead to a lower quality study than appropriate observational study approaches.

Operational standards for the design, conduct and reporting of observational studies have not been harmonized. This means that each nation governs observational studies to its own standards and requirements. Moreover, specific standards may not exist in some countries, even today, and interpretation of the requirements may vary within countries. A need for representative sites brings the possibility of an extended site selection process, and the likelihood of working with many research-naïve sites. Such sites require training and monitoring approaches tailored to observational research.

The lack of randomization increases the risk of bias and confounding of results, requiring specialized approaches to design and analysis. The ‘real world’ provides a research environment that is less controlled than the gold standard randomized clinical trial, bringing with it specific design and operational challenges.

Second, there may be the temptation to omit robust pre-study feasibility to inform best practice design and conduct. However, working in the real world requires an understanding of patient pathways, product launch dates and other factors relevant to usual care. Understanding these factors is critical to ensuring robust study plans.

Third, researchers may fail to invest in extracting the value from studies through analyses, reports and publications. The value of the study resides in the results that are reported and published. For these to demonstrate value to decision-makers, the results must live far beyond the life of the study. A desire to meet study timelines can blind teams to strategic goals, leading them to invest less time in study outputs (analyses, reports and publications) and more time in study inputs (such as first patient in, first patient out or last site closed).

Communication of results to stakeholders

The final element of observational research design involves creating a comprehensive communication plan with specific strategies designed for each stakeholder involved. There are four critical success factors to maximize the impact of the study. First, identify appropriate target journals for publication, enabling the research to be tailored to meet their needs. Many journals provide guidance on what they look for. For example, the British Medical Journal lists factors that make study publication less likely, including: unethical research; an insufficiently important or clear research question; an insufficiently original, relevant or important overall message; an inappropriate study design for the research question; suboptimal study design; low internal validity/robustness of the study or external validity/generalizability of the study; or the study not being publicly registered. This highlights the importance of the second success factor, which is to register the observational study.

The third success factor is having a study-focused publication plan, including clarity around criteria for authorship during site selection, and a plan for early analyses to maximize the impact of the study from the start. Many long observational studies include interim analyses to disseminate results, and these should be coordinated with key journal and conference abstract submission dates. The process should be transparent with respect to the types of publications expected, and cross-company collaboration should be encouraged to optimize internal communication of results.

Engaging the study physicians is the fourth success factor. Payment for participation in observational studies can be low, and for most physicians, research does not play a major role in their day-to-day work. Additional approaches can boost physician motivation to participate in such studies, including:

• The creation of advisory panels involving or establishing national leaders and advisors, and holding face-to-face or virtual regional advisor panel meetings that link in to the sponsor company and the established Global Steering Committee. Facilitating information sharing through an online portal can enable a ‘bottom-up’ and ‘top-down’ flow of ideas and information, and provide a forum for discussion and proposal of publications.
• Expanding the use of the data by engaging multiple investigators with preferred content and communication channels, as established through regional advisory panels. This can use a combination of online portal resources and face-to-face meetings. Additional online tools can communicate the data to a broader community, including monitored and closed social networking sites (such as bulletin board meetings, with a facilitated Q&A over several days), and roll-out to the entire investigator community for debate.
• Broaden communications with other interested stakeholders, such as specialist nurses (engaging them in the study and sharing clinical insights with investigators) and patients (offering education on the importance of the study within the disease area).

Real-world evidence generation

Successfully selecting ‘the right research approach for the right question’ is critical for best practice real-world evidence generation. Robust feasibility, specialized operational delivery approaches, understanding and mitigating risks are critical. Engaging stakeholders through creation of a comprehensive communications and publication plan and establishing an advisory board will maximize the value of the study back to society. Delivering the right quality of evidence in the right time at the right cost will aid healthcare decision-making and ultimately improve patient care.

ABOUT THE AUTHOR

Louise Parmenter, PhD, is Global Head of Operations, Epidemiology & Outcomes Research, Real-World & Late Phase Research, Quintiles. She is a specialist in real-world and late-phase research with 23 years of related global operational and strategic experience. She is currently responsible for a team of epidemiologists and outcomes researchers primarily based in the United States, with growing teams in Europe and Asia. Dr. Parmenter holds a BSc in Physiology & Biochemistry and a PhD in Neurophysiology from the University of Southampton, UK. She is completing an MSc in Epidemiology from the London School of Hygiene and Tropical Medicine, UK.