researchmethods

**Social Science Research Methods**
The following terms and resources may assist you in better understanding important aspects of the research process. You may wish to refer to this as you read/critique studies throughout the semester as well as when you begin to think about the design of your own action research project.

This fully hyperlinked online text covers key aspects of the research process including: formulating research questions; sampling (probability and nonprobability); measurement (surveys, scaling, qualitative, unobtrusive); research design (experimental and quasi-experimental); data analysis; and, writing the research paper. It also addresses the major theoretical and philosophical underpinnings of research including: the idea of validity in research; reliability of measures; and ethics.
 * [|Research Methods General Knowledge Base]** (Trochim, 2006)

[|Key Understandings about the Design and Interpretation of Experimental Studies in Education] (Robert Coe, Durham University) An excellent summary of issues such as the nature of causality in educational research, practice and policy, threats to the validity of causal inference, the ethics of doing experiments, the analysis of data from experiments and the interpretation of systematic reviews. Section 3 on threats to causal inference is one of the best I have seen - in very simple language.


 * Key terms to be familiar with...** (please feel free to add terms to the bottom of this list as you encounter them in your readings - we can define and discuss them in class the next week)


 * METHODOLOGIES**
 * **Quantitative approaches of inquiry** typically involve experiments (experimental and quasi-experimental) and surveys. Quantitative approaches seek to reduce ambiguity in numerical ways.
 * **Experimental research design** – a “true experiment” - is a study that involves random assignment of participants to conditions, some receiving an intervention of interest to the researcher and others receiving a control intervention. The random assignment, with sufficient numbers, results in two groups of people that are so similar that we can be reasonably confident that one can stand in for the other. Therefore, any differences in achievement at the conclusion of the experiment must have resulted from the intervention.
 * **Quasi-experimental research design** often involves nonrandom assignment to groups that are matched and/or controlled for characteristics (e.g., gender, intelligence, ability level) the researchers believe could have an impact on the way individuals respond to the intervention.
 * **Qualitative approaches of inquiry** seek to explore and document the whys and hows of interventions or phenomena in non-numerical ways. Examples of qualitative approaches include ethnographies [studies of intact cultural group over time]; grounded theory [derive a theory grounded in views/actions of participants]; case studies [explore an activity/process in depth with one or more cases (individuals, schools, teachers); or narrative studies (studies that retell participants' life stories combined with the researcher's stories).
 * **Mixed-methodologies** incorporate both quantitative and qualitative approaches to study an issue. Sometimes the approaches are employed sequentially (one of the other) and other times they are employed concurrently where both types of data converge at the same time to address different aspects of related research questions.
 * **Replication study** seeks to verify data from a previous study, generally with different situations and different subjects, to determine if the basic findings of the original study can be generalized to other participants and circumstances.
 * **Meta-analysis** combines the results of across several experimental and quasi-experimental studies that address a similar set of hypotheses. This analysis averages effect sizes over similar studies and gives more generalized information about when an intervention works and when it does not.
 * **Effect size** is a measure of the strength/size of the relationship between two variables. In inferential statistics, an effect size helps to determine whether a "//statistically significant difference//" has any "//practical significance//" of a difference that really matters in the real world. Pearson's r correlation, one of the most widely used effect sizes, varies between -1 and 1 (perfectly negative or positive linear relationship) and Cohen (1992) indicates an effect size of r =0.1 is small; r =0.3 is medium; and r =0.5 is large.
 * **Statistically significant difference** indicates there is a difference that is unlikely to have occurred by chance.
 * **Practical significance** looks at whether the difference is large enough to be of value in a practical sense.
 * **Grounded theory analysis** involves watching participants actions/responses, developing categories, noting the relationships between the categories, and eventually generating a theory (list of principles) that can be translated into hypothesis.
 * **Cross-case analysis** involves examining patterns across cases to identify similarities and differences in the topics/variables of interest


 * STATISTICS AND RESEARCH DESIGN**
 * **Statistics** allow us to detect and evaluate group differences that are small compared to individual differences.
 * **Descriptive statistics** are used to describe and visualize the data (distributions, summary measures, mean, median, mode, etc).
 * **Inferential statistics** are used to draw inferences about populations on the basis of samples from the populations using statistical tests. These provide an objective numerical way to quantify the strength of the evidence for a hypothesis.
 * **Independent T-test**: tests the mean difference between two independent groups
 * **Analysis of Variance (ANOVA**): tests the mean differences in two or more groups
 * **Analysis of Covariance (ANCOVA)**: tests the mean differences in two or more groups after controlling for/holding constant one or more variables
 * **Correlational analysis** tests the strength and direction of a relationship between two variables. Remember, a positive linear correlation does NOT indicate causation!
 * **Regression analysis** involves using a correlation (relationship between two variables) to predict one variable by knowing the score on the other variable. This usually involves a linear regression that tries to find the "best fitting" straight line for the data. Multiple regression involves two or more independent variables being combined to predict the dependent variable. For more information, read [|StatSoft].
 * **Factor analysis** is used to study the relationship patterns among many dependent variables. It usually seeks to explore (//exploratory factor analysis//) or confirm (//confirmatory factor analysis//) how many different factors are needed to explain the relationship among several variables, to characterize the nature of those factors, and to investigate how well these "factors" explain the observed data both individually and collectively. For more specific information, read [|Darlington's chapter].
 * **Random selection** occurs when actual participants are randomly selected from the larger population with similar demographics.
 * **Random assignment** occurs when large numbers of participants are randomly assigned to either treatment or control classrooms.
 * **Convenience sample** was derived by the researcher selecting participants by convenience (they are available, close, easy to access, or volunteers). But there is no evidence this sample is representative of the population the researcher wants to generalize the findings to.
 * **Purposive sample** was selected by the researcher with a particular purpose in mind - often a member of a specific, predefined group.
 * **Matching participants** involves first, matching up each set of similar participants and then, putting one partner in the treatment group and the other partner in the control group. After "random assignment", this is the next-best sampling method for eliminating sources of experimental variation other than the one being studied. Matching is usually done at the outset of a study when the sample is selected. Studies that use matched samples are quasi-experimental rather than experimental.
 * **Independent variable** is the independent part that the researcher changes to do the experiment. Changes in these variables are directly caused by the researcher.
 * **Dependent variable** is what changes as a result of changing the independent variable. The researcher can not "control" this variable - it depends on the outcome of the independent variable.
 * **Fidelity of treatment** is how faithfully and accurately the intervention was delivered - was it delivered as intended across all treatments? Treatment fidelity increases the likelihood that the differences in outcome performance between conditions were due to the intended differences in the treatment conditions (and not some alternative explanation/difference between the groups).
 * **Construct Validity** determines whether or not a study validly measures what it was intended to measure. It considers how well the researcher's ideas were translated into actual programs and/or measures. It can help to think of it as a "truth in labeling" issue.
 * **Internal validity** provides evidence that what was done in one study (e.g., treatment or intervention) actually caused the outcome to happen, rather than some alternative explanation. This is only relevant to studies that try to establish a causal relationship.
 * **External validity** involves being able to confidently generalize from one study to other people, places, or times.
 * **Generalizability of the findings** involves the extent to which the findings can be generalized to the larger population of "like participants" (and not just the particular sample involved in the study)
 * **Reliability of measures (or test-retest reliability**) is the consistency of a measure from one time to another - the more stable the results, the more reliable the measure over time.
 * **Inter-rater reliability**: the degree to which different raters/observers give consistent estimates/scores of the same observations
 * **Circumscribed interpretation** is an interpretation of findings that is defined or marked off carefully to include or exclude certain types of participants and/or situations. This type of interpretation tells us nothing about how an intervention might impact different types of readers in different types of settings. Findings from one study can only apply to/generalize to the same type of participant or situation.
 * **Alternative explanation** is any explanation other than the planned intervention that might be used to explain the outcomes of a study.
 * **Confounding variables** are two "explanatory" variables that are confounded because their effects can not be separated from each other. In this case, it is not possible to determine which of the related variables actually caused the outcome.
 * **Hawthorne effect** is a phenomenon that participants change simply because they are getting attention from the researchers, rather than because of the effects of the intervention itself. It suggests an individual's behavior will change to meet the expectations of the observer if they are aware that their behavior is being observed.
 * **Matthew effect** is the phenomenon where "the rich get richer and the poor get poorer". Keith Stanovich used the term to describe a phenomenon that has been observed in research on how new readers acquire the skills to read: early success in acquiring reading skills usually leads to later successes in reading as the learner grows, while failing to learn to read before the third or fourth year of schooling may be indicative of life-long problems in learning new skills
 * **Immediate effects** considers the outcomes observed immediately or very soon after the treatment was given versus **long-term effects** are outcomes that are observed after some time (can vary) has elapsed between the treatment and measured performance


 * Additional terms added through the semester:**
 * **Chi-square analysis** is used with "discrete data" (e.g., categories) that are not continuous. It tests the difference between the frequency distribution of nonparametric data and the theoretical distribution, otherwise known as a "goodness of fit" test. {Does your mental model fit with reality?} It can also be used as a "test of independence" to test whether paired observations on two variables are independent of each other.