Research Question, Study Design and Continuous Research Education and Training Exercises (CREATE) Program

Volume 1, Jan 2012

Tal Burt, Pooja Sharma, Sanjay Mittal; Durham, NC, USA; Gurgaon, India

Page: 35-43

Introduction

Need of doctors to receive continuing clinical research education

In this era of evidence-based medicine, clinicians’ ability to interpret and conduct clinical research is critical. Clinicians today have increasing demands on their time and this is cutting into the opportunity to stay abreast of clinical research literature and interpret it critically. Studies indicate that knowledge of current medical care is inversely related to time since graduation from medical school (1, 2).  Hence, by the time clinicians are experienced enough to contribute to the body of literature, they have dissociated themselves from recent developments and have little time to become current again.  In addition, due to inadequate preparation many medical professionals are also unable to properly design, execute, and disseminate their own research. The result is a failure to create and translate scientific discoveries into tangible human benefit (3, 4).Mission statements of many healthcare institutions today combine emphasis on healthcare delivery with excellence in education and research.  There are indications that institutions that engage in clinical research provide better care (5), though others have found a failure of research programs to improve patient care (6).  Studies have suggested that learning critical appraisal in medical school is one of the most important skills learnt in medicine (7, 8). It is known that exposure in research methodology and good clinical practice (GCP) has positive effect on attitudes of physicians toward science and research, in addition to the appreciation of such courses in improving critical appraisal skills and the ability to conduct research (9–11).  

Goldhamer and colleagues emphasize the need to train the “endangered species” of the physician investigator to conduct clinical research through a clinical effectiveness program (12).There are ample Continuing Medical Education (CME) programs and in many countries it is mandatory to have fixed CME Credits for continuing with medical registration.  Bernd Löwe in 2007 assessed the effectiveness of a 1-year resident training program in clinical research through a reproducible program and found higher rates of writing and grantsmanship in the trained residents (13). Lipiraand colleagues measured the change in clinical research self-efficacy after participating in pre-doctoral and post-doctoral clinical research training programs at Washington University School of Medicine in 2010, and found that clinical research self-efficacy did increase 1 year after clinical research training (14). Their concern was whether this short-term outcome correlates with long-term clinical research productivity. Supino and Borer, who designed a comprehensive course on research methodology for physicians, and have successfully implemented it over the last 15 years, remark that “programmatic goal now is to develop an objective strategy to measure the impact of this program on key methodological competencies and, ultimately, research productivity” (8).

A cohesive and continuous plan toward continuous research education (CRE) can result not only in the translation of research into practice but also in the creation of a skilled set of investigators qualified for further scientific research. The final assessment of any CRE program will be the productivity of relevant research.The clinical research process is a lengthy one and involves considerable utilization of resources even for simple and small studies (Figure 1).

We propose CREATE (Continuous Research Education and Training Exercises)—a continuous education program dedicated to clinical research as a peer-based, research-team-focused, interactive, analytical, customized, and accrediting program with didactic, training, mentoring, administrative, and professional support to enhance clinical research knowledge and skills of healthcare professionals and promote the generation of original investigator- and peer-initiated research projects.

The CREATE program thus emphasizes the “continuous” nature of the experience. Not only is the plan for the program to continuously provide clinical research education at a given healthcare environment but for a given project support is continuous from inception to completion.  We first describe the principles governing the formulation of optimal research questions and choice of associated study design as the first steps in the research project and then illustrate their role as case studies in the implementation of the CREATE program.  


Research Question

Meaningful unknown

Formulating the research question is arguably one of the most important aspects of a research project.  Many key components of the research project depend directly on it; including the ability to ensure a positive benefit-to-risk ratio, the ability to make efficient use of resources, to obtain a convincing and clear answer to the research question, and make a meaningful impact with that answer.The answer to the research question should fill a “meaningful unknown,” a “knowledge gap” with relevance to healthcare practice and patients’ outcomes (15). Thorough familiarity with relevant knowledge is required.  A scholarly review of the literature and consultation with experts in the field is advised.  Novice researchers or those venturing into new research fields may benefit from a relationship with a mentor who is a seasoned researcher in the field. A “thought experiment,” a “what-if-the-question-were-answered” exercise is advised to explore the impact of potential answers on relevant stakeholders and the practice of medicine. Sometimes the question may have already been answered in some form but there is justification to study it again.  A common justification is to replicate the study to increase the validity of the findings. Other justifications include conducting the study in different settings to increase the generalizability of the data.  Different settings may include different study populations, with different inclusion/exclusion criteria, over different periods, different primary outcomes, in combination with other interventions (16).  

The FINER criteria (Feasible, Interesting, Novel, Ethical, and Relevant) have been proposed to describe the characteristics of the ultimate research question (15, 16).  To these we would like to add  “scientifically sound” and “impactful.” The research question should also be one that is meaningful to one’s personal development and career.

The process of identifying and refining the research question culminates with the formulation of the research hypothesis—a clear and specific “true or false” statement that the study will be designed to prove or disprove.  To get to the optimal research question and hypothesis the following process is recommended:

1. Overall review of the literature on the topic of interest—the features of the research question that are the focus of this phase are novelty, impact, relevance

2. Local and personal relevance—features of focus here are interest (including personal interest), feasibility

3. Research question—when honing-in on the research question of choice the features of focus are ethics and the science

4. Research hypothesis—the main focus at this stage is to make sure the research hypothesis and hence the study are scientifically sound

Study Design

The “study design” is the set of methodological instructions aimed at answering the research question with minimal impact from the two main methodological threats: variability and bias, or random errors and systematic errors, or the lack of precision and lack of accuracy, respectively.

Adequate study design will ensure:

    o Study results reflect the “reality” of the study—“internal validity”
    o Study results can be generalized and reflect the “reality of the world”—in differe
locations, at different times, with different individuals.  If so, the study has “external validity”

Once completed, the “study design” will be followed by the detailed set of operational instructions to constitute the “study protocol.

The principles governing the choice of “study design” parameters have been variously defined in the literature.  One such scheme, the PICOT format, identifies Patient population, Intervention, Comparison, Outcome of Interest, and Time Frame (PICOT) as the key parameters (17).  In addition, a carefully designed statistical analysis plan needs to be identified prior to analysis of the data and preferably prior to initiation of the study (18)

We propose the following options and considerations to be part of the choice of study design parameters:

   o General design (type of study) 
   „X Prospective/retrospective
   „X Intervention/observatio
   „X Cross-sectional/longitudinal 
   „X Allocation, e.g., randomized, consecutive
   „X Control group—parallel group, cross-over, single group, placebo/sham, acti
control, treatment-as-usual, waiting list, historical control
   „X Blindness,  e.g., open-label, single-blind, double-blind

   o Specific parameters
   „X Sample size
   „X Participant characteristics (inclusion/exclusion criteria)  
   „X Type of intervention (including comparator where applicable)
   „X Primary outcome (as well as secondary outcomes
   „X Time parameters—duration and frequency of observations and interventions

These design parameters are consistent with reporting requirements of the CONSORT statement (19).

It is critical to identify the right design to answer the research question.  The foremost goal of study design is to ensure safety and ethical handling of human subjects.  From a methodological point of view, the purpose of study design is to provide an accurate and precise way of answering the research question—an answer that has minimal vulnerability to systematic and random errors, minimal bias, and variability. Appropriate designs depend not only on ethical and methodological considerations but also on available resources (funds, staff, skills, patient/participant populations, infrastructure, equipment, and analytics) and the constraints of the research question. Conversely, choice of the research question needs to take into account available study designs options. 

“Research question” and “study design” principles are utilized here to illustrate the function and utility of the CREATE program in engaging healthcare professional peers in a decision-making process aimed at distilling and selecting consensus parameters, most appropriate for the group being trained or the team who will execute the research project.  

The assimilation of the aforementioned research question and study design principles constitutes the first steps in the CREATE program and the first steps in undertaking any specific research project.   The “clinical research workshops” first teach the aforementioned principles and then utilize interactive group exercises to train the learned material (Appendices A and B).

Continuous Research Education and Training Exercises (CREATE) Program

The Continuous Research Education and Training Exercises (CREATE) Program is a peer- and group-based, interactive, analytical, customized, and accrediting program with didactic, training, mentoring, administrative, and professional support to enhance clinical research knowledge and skills among healthcare professionals, promote the generation of original research projects, increase the chances of their successful completion, and optimize the potential for their meaningful impact.  

The term peer-based is meant to indicate an institution-based, indigenous program that can be tailored to the needs of the institution’s staff and be executed with available resources following the general CREATE guidelines herein provided.  The key features of the program are successive intra- and inter-group discussions and confrontational thematic challenges among participating peers aimed at capitalizing on the groups’ knowledge, experience, and skills, and combined intellectual processing capabilities to optimize choice of research project elements and stakeholders’ decision-making. The program was applied in collaborations between Duke—National University of Singapore (Duke-NUS) and healthcare professionals at Singapore General Hospital (SGH) in Singapore and between Duke Medicine and Medanta – The Medicity in Gurgaon, India.  

The program was highly endorsed by participants that included mostly physicians and nursing staff.

The program consists of
1. Investigator survey 
2. Educational curriculum
   a. Presentations on clinical research methodology and ethics
   b. Workshops and group discussions on topics matching the presentation
3. Production of original research questions, study designs, study protocols
4. Provision of personal tutoring and mentoring
5. Provision of auxiliary professional clinical research support (e.g., biostatistics,               regulatory, budgetary support)
6. Pre- and post-assessments 
7. Accreditation system

Target Audience

While primarily targeting existing and aspiring clinical research staff (e.g., investigators, coordinators, research nurses), CREATE’s interactive format may accommodate and be attractive to participants from other disciplines related to the healthcare and medical research sectors (e.g., biostatisticians, regulatory personnel, industry sponsors, monitors, patientadvocacy groups, research administrators, study participants) as well as the public at large (e.g., media representatives, policy makers, scholars).  This is similar to the model of Independent Review Boards and Ethics Committees where the critique and vetting of human research projects is the product of input from the community where the research takes place.  The purpose is to engage all stakeholders in the research project in meaningful exchange and obtain all relevant input at the start of the project.

The CREATE program can be used to

1. Educate and train healthcare providers in the principles of clinical research

2. Generate research projects

3. Cross-fertilize ideas and initiate collaboration among individuals from different subject matter expertise areas (e.g., oncologists and immunologists), backgrounds, skills, and operational capabilities (e.g., physicians, nurses, lab technicians, statisticians, administrators)

4. Bridge the divide between basic scientists and clinician scientists to facilitate exchange of information that is essential to the development of common research projects

5. Bridge the divide between academia and industry and identify projects that make best use of combined skills, capabilities, resources, and interests 

6. Bridge the divide between those who do research (industry, academia, clinicians) and those who receive it (patients and the public at large)

Components of the CREATE Program

1.  Investigator survey: “first, know your audience”

A structured survey administered to potential investigators.  The purpose of this survey is to provide a framework for the assessment of potential investigators for participation in educational activities and clinical research projects. The survey will be used to create a database of investigators’ subject matter experience, expertise, interests, and availability as they pertain to clinical research and help characterize clinical research capabilities and culture and be used to guide policy modifications resource allocation and facilitate interactions with potential sponsors.

2.  Educational and training curriculum: Clinical research workshops

The core feature of the CREATE experience is the clinical research workshops—a peer- and group-based, interactive, analytical, and customized education and training tool aimed at enhancing clinical research knowledge and skills with the goal of promoting the generation and execution of original clinical research projects.

The methodology behind clinical research workshops was developed through a repeating process of group interactions and personal mentorship and the systematic gathering of feedback from those interactions.  Its main feature is the utilization of multiple and concurrent cognitive modalities to process the learned information, enhance retention, establish the desired skills, and enable initiation and execution of independent research projects.  

The basic unit of the clinical research workshop is a 2-hour session comprising, in the following order:

A. Lecture, providing the information necessary to execute the group exercise that follows, describing the exercise, and answering any questions (about 30 minutes)

B. Small group exercise, with 4–8 individuals in each group.  Pairs of groups are presented with the same task, one distinct task per pair of groups so that later, during the large group discussion, at least one other group has been able to give careful thought to the task and be in a position to contest the presenting groups’ conclusions (about 40 minutes)

C. Presentation and whole group discussion, optimally with a total of 30–70 individuals divided into no more than eight groups (in four pairs).  Each group presents its conclusions to the whole group, contests its paired group, and addresses any challenges from the whole group.  After the paired group has made its own presentation, the whole group arrives at an overall consensus (5 minutes per group presentation for a total of about 40 minutes)

D. Summary (about 10 minutes).  By the end of the workshop each task will have been processed in four distinct times: at the individual, small group, group pair, and whole group levels

Appendices A and B are provided to illustrate the group training exercises for research question and study design workshops, respectively.  The study design assignment follows and builds on the conclusions arrived at during the research question workshop.

The clinical research workshops can cover most topics relevant to the conception, practice of skills, and execution of clinical research projects.  Specific topics to be covered by the clinical research workshops are:

 A. Research question
 B. Study design
 C. Protocol writing
 D. Protocol execution 
 E. Report writing
 F. Publications
 G. Grant submission
H. Biostatistics
  I. Data management (EDC)
 J. Ethics
 K. Regulatory
 L. Drug development process
 M. Reading and writing research articles (+ CONSORT Guidelines)
 N. Journal clubs

3.  Conceptualization and production of research projects

The same format of the clinical research workshop can be used by research teams and peer groups to generate research projects and improve the quality of study execution.  The opportunity to intellectually challenge and build consensus over topics fundamental to formulation of the research question, choice of study design, and conduct can be instrumental to team bonding and efficiency of function.  The research project would be investigator-initiated and supported from conceptualization to completion by the CREATE program.  The components of the research project are:

  A. Choosing the research question
  B. Study design and protocol writing
  C. IRB approval
  D. Research execution
  E. Report writing
  F. Publication

4.  Provision of personal tutoring and mentoring

Each clinician scientist will have one or more tutor/mentor/s assigned to his/her research project/s.  The tutor/mentor will provide ad-hoc training as needed, feedback and guidance in advancing the research project from conceptualization through completion and publication.  The tutor/mentor will also coordinate interaction with auxiliary professional support personnel (see below).

5.  Provision of auxiliary professional clinical research support

Besides the tutor/mentor other specialized support personnel would be available as needed.  These could be subject matter, regulatory, and biometrics experts, or clinical research accountants to assist with budgetary considerations and grant submissions.

6.  Pre- and post-assessments

The program will aim to demonstrate the impact it makes on knowledge retention, skills, and productivity relevant to execution of clinical research projects.  It will also aim to document specific changes achieved and areas still in need of change as well as feedback on experience and preferences from the recipients of the program.  

Parameters assessed can include those relevant to the validity, quality, and integrity of research.  Choices made by participants during a survey, test, or personal mentoring may be used to complete the assessments.

Short of having the impractical control group of randomly selected participants who are excluded from the program CREATE will use the following, alone or in combination, to document outcome:

   A. “Historical control”—compare a period in the past to a period in the future regarding the amount and quality of research done.  This has some methodological disadvantages but can nevertheless be indicative of change.  Parameters may include number of projects per clinician and collective “impact factor” for a department or institution

   B. “Baseline assessment of knowledge and skills”—administer a proficiency test and/or collect baseline information on amount of research done by each individual

   C. “Compare with non-participants”—the productivity of participants could be compared with those that did not participate.  While there is a potential for confound, those that do not participate may not be interested in research to begin with

7.  Accreditation system

A credit system (e.g., point-based) may be arranged in conjunction with participants’ employers and supervisors to provide an additional incentive to the workshop and integrate it with a larger research development process (e.g., completion of own research projects, publications) and a general career development plan.  Participants in the CREATE program will be eligible for accreditation points for their participation and milestone achievements.  Such credit could be similar to CME credits and may be used toward clinician-scientists career development, personal remuneration, or both.  A proposal for a credit scheme is the following:

A. Training/education: 
  I. 1 hour = 1 point
  II. 1 workshop session = 2 hours = 2 points
  III. Total of 8 sessions and 16 possible points per year—the limit is proposed so
that there is incentive to regain points through an original research project

B. Research project
  I. Research question:  2 points
  II. Research proposal (including protocol writing):  4 points
  III. Grant award:  4 points
  IV. IRB approval:  2 points
  V. Study execution:  4 points (additional points given for complexity factors, e.g.
study duration, number of procedures, recruitment difficulties)
  VI. Report writing:  2 points
  VII. Publication:  2 points (+ additional 2 points for each 2 points of Impact Factor
beyond 2.00)

Conclusions

We proposed CREATE (Continuous Research Education and Training Exercises) program to assist busy clinicians in learning clinical research principles, practicing associated skills, and collaborating with peers to generate ethical, impactful, feasible, scientifically sound, novel, relevant, and interesting research projects.  We have provided specific details on the first two steps in the course of a clinical research project:  the formulation of the “research question” and choice of “study design” parameters.

References

1. Perneger TV, Ricou B, Boulvain M, Bovier PA, Herrmann FR, Perrier A, Burnand B. Medical researchers evaluate their methodological skills.  J ClinEpidemiol. 2004; 57(12):1323–9.

2. Ramsey PG, Carline JD, Inui TS, Larson EB, LoGerfo JP, Norcini JJ, Wenrich MD. Changes over time in the knowledge base of practicing internists.  JAMA. 1991; 266(8):1103–7.

3. Sung NS, Crowley WF Jr, Genel M, Salber P, Sandy L, Sherwood LM, Johnson SB, Catanese V, Tilson H, Getz K, Larson EL, Scheinberg D, Reece EA, Slavkin H, Dobs A, Grebb J, Martinez RA, Korn A, Rimoin D. Central challenges facing the national clinical research enterprise. JAMA. 2003; 289(10):1278–87.

4. Olatunbosun OA, Edouard L, Pierson RA. Physicians’ attitudes toward evidence based obstetric practice: a questionnaire survey.  BMJ. 1998; 316(7128):365–6.

5. Majumdar SR, Roe MT, Peterson ED, Chen AY, Gibler WB, Armstrong PW.  Better outcomes for patients treated at hospitals that participate in clinical trials.  Arch Intern Med. 2008; 168(6):657–62.

6. Davis DA, Thomson MA, Oxman AD, Haynes RB. Changing physician performance. A systematic review of the effect of continuing medical education strategies.  JAMA. 1995; 274(9):700–5.

7. Grimes DA, Bachicha JA, Learman LA. Teaching critical appraisal to medical students in obstetrics and gynecology.  Obstet Gynecol. 1998; 92(5):877–82.

8. Inam SB. Experience of teaching critical appraisal of scientific literature to undergraduate and postgraduate students at the Ziauddin medical university, Karachi, Pakistan.  Int J Health Sci (Qassim). 2007; 1(1):119–24.

9. Supino PG, Borer JS. Teaching clinical research methodology to the academic medical community: a fifteen-year retrospective of a comprehensive curriculum.  Med Teach. 2007; 29(4):346–52.

10. Vujaklija A, Hren D, Sambunjak D, Vodopivec I, Ivanis A, Marusi A, Marusi M. Can teaching research methodology influence students’ attitude toward science? Cohort study and nonrandomized trial in a single medical school.  J Investig Med. 2010; 58(2):282–6.

11. Kuusisto H, Virkki M, Wuolijoki E, Kernen T. Hospital training program increases awareness of Good Clinical Practice (GCP).  Contemp Clin Trials. 2011; 32(3):339–41.

12. Goldhamer ME, Cohen AP, Bates DW, Cook EF, Davis RB, Singer DE, Simon SR.  Protecting an endangered species: training physicians to conduct clinical research. Acad Med. 2009; 84(4):439–45.

13. Löwe B, Hartmann M, Wild B, Nikendei C, Kroenke K, Niehoff D, Henningsen P, Zipfel S, Herzog W. Effectiveness of a 1-year resident training program in clinical research: a controlled before-and-after study. J Gen Intern Med. 2008; 23(2):122–8.

14. Lipira L, Jeffe DB, Krauss M, Garbutt J, Piccirillo J, Evanoff B, Fraser V. Evaluation of clinical research training programs using the clinical research appraisal inventory. Clin Transl Sci. 2010; 3(5):243–8.

15. Hulley SB, Cummungs SR, Browner WS, Grady D, Newman TB. Designing Clinical Research, ed 3. Philadelphia, PA: Lippincott Williams & Wilkins, 2007.

16. Thabane L, Thomas T, Ye C, Paul J.  Posing the research question: not so simple.  Can J Anaesth. 2009; 56(1):71–9.

17. Richardson WS, Wilson MC, Nishikawa J, Hayward RS. The well-built clinical question: a key to evidence-based decisions.  ACP J Club. 1995; 123:A12–3.

18. Jones JB.  Research fundamentals: statistical considerations in research design: a simple person’s approach.  Acad Emerg Med. 2000; 7:194–199.

19. Schulz KF, Altman DG, Moher D; CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. PLoS Med. 2010; 7(3):e1000251.

Appendix A

“Research Question” Exercise

Question parameters:

Ethical
Impactful
Feasible
Scientifically sound
Novel
Relevant
Interesting

PolyHeme and Hemorrhagic Shock

Background:  

Trauma patients not receiving blood have 17–54% mortality before they reach the hospital. Most ambulances do not carry blood.  

PolyHeme is a synthetic form of hemoglobin compatible with all blood types and capable of providing oxygen-carrying capacity in conditions of life-threatening anemia where blood is not available.

Research questions/hypotheses (multiple-choice statements):

1. What is the best method of fluid replacement for trauma patients?

2. Is PolyHeme providing better oxygen-carrying capacity than autologous blood per unit volume?

3. Does administration of PolyHeme improve the mortality rate in trauma patients when administered during the ride in an ambulance to the hospital?

4. Is PolyHeme superior to IV fluids in maintaining BP in trauma patients?

5. Is outcome of treatment with PolyHeme superior to IV fluids during delivery of trauma patients to hospital? 

6. Is tertiary-care hospital care better than PolyHeme administration in trauma patients?

7. Is PolyHeme effective in managing intraoperative blood loss during elective surgery?

8. Is PolyHeme superior to no infusion at all in acute trauma patients?

9. Suggest a question/hypothesis:  _____________________________________

Appendix B

“Study Design” Exercise

Study design parameters:

• General design

    • Prospective/retrospective
    • Intervention/observation
    • Cross-sectional/longitudinal
    • Allocation, e.g., randomized, consecutive
        Control group—placebo/sham, active control, treatment-as-usual, waiting list, historical control
    • Blindness,  e.g., open-label, single-blind, double-blind

• Sample size
    • Participant characteristics (inclusion/exclusion criteria)
    • Type of intervention (including comparator where applicable)
    • Primary outcome (and secondary outcomes)
    • Time parameters: duration and frequency of observations and interventions

PolyHeme and Hemorrhagic Shock

Background:  

Trauma patients not receiving blood have 17–54% mortality before they reach the hospital.  Most ambulances do not carry blood.  

PolyHeme is a synthetic form of hemoglobin compatible with all blood types and capable of providing oxygen-carrying capacity in conditions of life-threatening anemia where blood is not available.

Research Question: Is outcome of treatment with PolyHeme superior to IV fluids during delivery of trauma patients to hospital? 

Refute the “Null-Hypothesis”:  Outcome of treatment with PolyHeme is the same as outcome of treatment with IV fluids.

Study designs (multiple-choice statements):

   1. Give PolyHeme to trauma patients during the ambulance ride to the hospital and compare their survival to reported survival in the literature.

   2. Give PolyHeme to some patients but reserve the usual treatment to the more severe patients during the ambulance ride to the hospital.  Compare survival between the two groups.

   3. Give PolyHeme or usual treatment to patients during the ambulance ride to the hospital and during the first 24 hours at the hospital.

   4. Allow the treating physician to decide which patients are more likely to benefit from the PolyHeme and give it only to such patients.

   5. Randomly assign patients to receive PolyHeme or usual treatment during the ambulance ride.  Compare survival between the two groups.

   6. Randomly assign patients to receive PolyHeme or no treatment during the ambulance ride. Compare survival between the two groups.

   7. Randomly assign patients to receive PolyHeme or usual treatment during the ambulance ride. Compare patient satisfaction between the two groups.

   8. Randomly assign trauma patients to receive PolyHeme or emergency room care. 

   9. Suggest a study design:  _____________________________________