Module Two:

Assessing Research Quality

Learning Objective

Learn some characteristics of quality research.

What You Need to Know

Research designs take different forms depending on the research’s purpose. The first step in choosing a design is deciding whether to take a qualitative or quantitative approach. 

A qualitative approach relies on data collected by the researcher from observations, interviews, questionnaires, focus groups, and artifacts. A quantitative approach utilizes observational/descriptive, correlational, experimental, and quasi-experimental designs. 

Quantitative Designs

Below are descriptions of quantitative designs, including randomized control trials, which some deem the “gold standard” for research methodologies in education.

  • Observational / Descriptive Design

    Non-experimental design that focuses on the characteristics of the population or question being studied (the ‘what’ vs. the ‘why). Data collection tools can include surveys, interviews, focus groups, ethnographic and case study methods.

  • Correlational Design

    Focuses on the relationship between two variables, but does not include manipulation or control of either variable by the investigator; therefore, causation cannot be determined when using a correlational design.

  • Experimental Design

    Has one variable that is controlled and one that is manipulated by the investigator to determine the existence of a cause and effect relationship.

  • Quasi-Experimental Design

    Does not rely on random subject assignment but is used when experimental design cannot be used for ethical or practical reasons and researchers still seek to find a cause and effect type relationship. Data collection includes large-scale surveys or panel data of a quantitative nature.

Steps

  • 1. Start with a Reliable Source

    Make sure the research or evidence has been peer-reviewed and/or vetted by a trusted source like an academic journal, clearing house reviewers, or a professional association with a well-developed and strong reputation for providing accurate and reliable information.

  • 2. Consult with a Subject Matter Expert or Researcher

    If possible, connect with a researcher or expert, who can help assess the quality of a study or evidence.

  • 3. Assess Critical Components of the Research or Evidence

    Such as methodology, the sample, outcomes, and replicability.

Guiding Questions

  1. What was the research methodology that was used? Was there a control group?

  2. Is the sample size adequate, inclusive, and representative? 

  3. Was the research submitted for an independent expert review?

Case Study

After looking for research that provides evidence of impact and on effective design principles, the superintendent chooses to look more closely at a recent study of 9th and 10th graders who received high-dosage math tutoring during the 2013-14 and 2014-15 school years. 

To justify the high cost of a high-dosage tutoring program, she wants to ensure the study is high quality. She examines the following components:

  • Methodology:  Researchers conducted two randomized control trials (RCT) of the tutoring model. RCT is considered the ‘gold standard.’

  • Sample:  In the first RCT, there were 2,633 9th and 10th graders from two representative Chicago public schools in the treatment group, and in the second RCT, there were 2,710 9th and 10th graders.

  • Outcomes:  Participation in the tutoring program increased math test scores by 0.16 standard deviations (SDs) and increased grades in math and non-math courses, with results persisting into future years.

  • Replicability:  Researchers replicated the results of the first RCT in a separate RCT with 2,710 students and found even larger math test score impacts and similar grade impacts. 

She concludes that the research design and outcomes are strong. She also finds an article that she can share with her board to help communicate the intervention’s effectiveness.

Resources

  • Assessing the Strength of Evidence in the Education Sector:

    This guide was produced by the Building Evidence in Education (BE2 ) working group composed of the Department for International Development (DFID), United Nations International Children's Emergency Fund (UNICEF), United States Agency for International Development (USAID) and The World Bank Group. It provides comprehensive descriptions of different types of research and includes rubrics to assess quality and determine what evidence should and should not be used.

  • How to Evaluate Educational Research:

    This article from T.H.E. Journal offers a “refresher primer” on evaluating education research and includes rules of thumb and questions compiled by Dr. Doris Redfield, Vice President for Research and Director, Regional Education Laboratory at AEL (Appalachia Education Laboratory).

  • Identifying and Implementing Educational Practices Supported by Rigorous Evidence:

    A user friendly guide from the Institute of Education Sciences (IES) National Center for Education Evaluation and Regional Assistance (NCEE) defines and describes the use of Randomized Control Trials and its ability to yield strong evidence.