When does a better model actually perform better?
A replication of a published readmission model, comparing its performance with a GRU baseline before and after hyperparameter tuning.
30
Inside the study trials per configuration
using synthetic patient data
Experiment 01 / Model comparison
Model performance
Replication and tuned configurations.
Replication: PR-AUC, mean and standard deviationCONTENT: 0.6391 plus or minus 0.0025. GRU baseline: 0.6401 plus or minus 0.0017. Full results are available in the table below.
CONTENT0.6391
GRU baseline0.6401
The added architecture did not show a statistically significant PR-AUC advantage in our initial configuration.
Both configurations use 2 epochs; the original study used 6. Bars show standard deviations, not confidence intervals. The axis is expanded for comparison.
View all results & source
| Configuration | Model | PR-AUC ± SD | ROC-AUC ± SD |
|---|---|---|---|
| Replication | CONTENT | 0.6391 ± 0.0025 | 0.7970 ± 0.0014 |
| Replication | GRU baseline | 0.6401 ± 0.0017 | 0.7979 ± 0.0013 |
| After tuning | CONTENT | 0.6464 ± 0.0022 | 0.8026 ± 0.0014 |
| After tuning | GRU baseline | 0.6391 ± 0.0025 | 0.7970 ± 0.0014 |
Read the report, pp. 3–4 ↗. The report’s statistical claim concerns PR-AUC; this figure makes no separate significance claim for ROC-AUC.
