Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions DESCRIPTION
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,10 @@ Description: Imports variables from 'ReaderBench' (Dascalu et al.,
2018)<doi:10.1007/978-3-319-66610-5_48>, 'Coh-Metrix' (McNamara et
al., 2014)<doi:10.1017/CBO9780511894664>, and/or 'GAMET' (Crossley et
al., 2019) <doi:10.17239/jowr-2019.11.02.01> output files; downloads
predictive scoring models described in Mercer & Cannon
(2022)<doi:10.31244/jero.2022.01.03> and Mercer et
al.(2021)<doi:10.1177/0829573520987753>; and generates predicted
predictive scoring models documented in Keller-Margulis et al.
(2026)<doi:10.35542/osf.io/36dn5_v1>, Mercer & Cannon
(2022)<doi:10.31244/jero.2022.01.03>, and Mercer et
al. (2021)<doi:10.1177/0829573520987753>; and generates predicted
writing quality and curriculum-based measurement (McMaster & Espin,
2007)<doi:10.1177/00224669070410020301> scores.
License: MIT + file LICENSE
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,8 @@ Also see the list of code [contributors](https://github.com/shmercer/writeAlizer

### Journal Articles

Keller-Margulis, M. A., Mercer, S. H., Matta, M., & Duran, B. (2026). *Identifying optimal indicators of performance using automated written-expression curriculum-based measurement: An examination of genre effects* [Preprint]. EdArXiv. https://osf.io/preprints/edarxiv/36dn5_v1/

Matta, M., Keller-Margulis, M. A., & Mercer, S. H. (2025). Improving written-expression curriculum-based measurement feasibility with automated writing evaluation programs. *School Psychology, 40*(6), 707–717. https://doi.org/10.1037/spq0000691

Matta, M., Mercer, S. H., & Keller-Margulis, M. A. (2023). Implications of bias in automated writing quality scores for fair and equitable assessment decisions. *School Psychology, 38*, 173–181. https://doi.org/10.1037/spq0000517
Expand Down
2 changes: 1 addition & 1 deletion codemeta.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"@context": "https://doi.org/10.5063/schema/codemeta-2.0",
"@type": "SoftwareSourceCode",
"identifier": "writeAlizer",
"description": "Imports variables from 'ReaderBench' (Dascalu et al., 2018)<doi:10.1007/978-3-319-66610-5_48>, 'Coh-Metrix' (McNamara et al., 2014)<doi:10.1017/CBO9780511894664>, and/or 'GAMET' (Crossley et al., 2019) <doi:10.17239/jowr-2019.11.02.01> output files; downloads predictive scoring models described in Mercer & Cannon (2022)<doi:10.31244/jero.2022.01.03> and Mercer et al.(2021)<doi:10.1177/0829573520987753>; and generates predicted writing quality and curriculum-based measurement (McMaster & Espin, 2007)<doi:10.1177/00224669070410020301> scores.",
"description": "Imports variables from 'ReaderBench' (Dascalu et al., 2018)<doi:10.1007/978-3-319-66610-5_48>, 'Coh-Metrix' (McNamara et al., 2014)<doi:10.1017/CBO9780511894664>, and/or 'GAMET' (Crossley et al., 2019) <doi:10.17239/jowr-2019.11.02.01> output files; downloads predictive scoring models documented in Keller-Margulis et al. (2026)<doi:10.35542/osf.io/36dn5_v1>, Mercer & Cannon (2022)<doi:10.31244/jero.2022.01.03>, and Mercer et al. (2021)<doi:10.1177/0829573520987753>; and generates predicted writing quality and curriculum-based measurement (McMaster & Espin, 2007)<doi:10.1177/00224669070410020301> scores.",
"name": "writeAlizer: Generate Predicted Writing Quality Scores",
"relatedLink": ["https://shmercer.github.io/writeAlizer/", "https://CRAN.R-project.org/package=writeAlizer"],
"codeRepository": "https://github.com/shmercer/writeAlizer/",
Expand Down
9 changes: 9 additions & 0 deletions vignettes/references.bib
Original file line number Diff line number Diff line change
@@ -1,3 +1,12 @@
@misc{Keller-Margulis2026,
author = {Keller-Margulis, Milena A. and Mercer, Sterett H. and Matta, Michael and Duran, Brenda},
title = {Identifying optimal indicators of performance using automated written-expression curriculum-based measurement: An examination of genre effects},
year = {2026},
type = {Preprint},
publisher = {EdArXiv},
url = {https://osf.io/preprints/edarxiv/36dn5_v1/}
}

@article{Keller-Margulis2021,
author = {Keller-Margulis, Milena A. and Mercer, Sterett H. and Matta, Michael},
title = {Validity of automated text evaluation tools for written-expression curriculum-based measurement: A comparison study},
Expand Down
8 changes: 4 additions & 4 deletions vignettes/scoring-model-development.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -2053,11 +2053,11 @@ Each column sums to approximately 100, allowing for rounding (values describe re

ReaderBench Model 3, recommended for current use, is an ensemble (formed by averaging predicted quality scores) of three genre-specific models, detailed below.

The models were trained on ReaderBench scores from 15 min narrative, expository, and persuasive writing samples from students in Grades 2-5 to predict holistic writing quality on the samples (theta scores calculated from paired comparisons).
The models were trained on ReaderBench scores from 15 min narrative, expository, and persuasive writing samples from students in Grades 3–5 to predict holistic writing quality on the samples (theta scores calculated from paired comparisons) [@Keller-Margulis2026].

Highly correlated ReaderBench metrics (_r_ > |.90|) were excluded during pre-processing (see section on [Scoring Model Development](#scoring-model-development) for more details).

More details on the sample will be provided once peer review is complete on the main study using this model.
See [@Keller-Margulis2026] for additional details on the study sample, writing tasks, human scoring, and model-development procedure.

## ReaderBench Model 3narr

Expand Down Expand Up @@ -3825,11 +3825,11 @@ Each column sums to approximately 100, allowing for rounding (values describe re

Coh-Metrix Model 3, recommended for current use, is an ensemble (formed by averaging predicted quality scores) of three genre-specific models, detailed below.

The models were trained on Coh-Metrix scores from 15 min narrative, expository, and persuasive writing samples from students in Grades 2-5 to predict holistic writing quality on the samples (theta scores calculated from paired comparisons).
The models were trained on Coh-Metrix scores from 15 min narrative, expository, and persuasive writing samples from students in Grades 3–5 to predict holistic writing quality on the samples (theta scores calculated from paired comparisons) [@Keller-Margulis2026].

Highly correlated CohMetrix metrics (_r_ > |.90|) were excluded during pre-processing (see section on [Scoring Model Development](#scoring-model-development) for more details).

More details on the sample will be provided once peer review is complete on the main study using this model.
The training sample, writing tasks, and paired-comparison writing-quality criterion are described in [@Keller-Margulis2026].

## CohMetrix Model 3narr

Expand Down
2 changes: 2 additions & 0 deletions vignettes/writealizer-getting-started.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,8 @@ ReaderBench Model 3 keys also accept an explicit `_v2` suffix; the output column
| `coh_mod2` | `pred_coh_mod2a` through `pred_coh_mod2c` | `pred_coh_mod2_mean` |
| `coh_mod1` | `pred_coh_mod1a` through `pred_coh_mod1f` | `pred_coh_mod1_mean` |

The ReaderBench and Coh-Metrix Model 3 families were trained on the same Grades 3–5 sample of 15-minute narrative, expository, and persuasive writing described in [@Keller-Margulis2026].

Model 2 is a simplified version of Model 1. Published Model 1 applications include [@Keller-Margulis2021; @Matta2022].

### GAMET and the offline demonstration
Expand Down
Loading