Course Description
It is easier than ever to make and use prediction models. With frameworks such as tidymodels, scikit-learn, H2O, and AutoGluon, it takes no more than a few lines of code to build state-of-the-art machine-learning models. This led to an explosion in the use of prediction models in both research and application, which was accompanied by an explosion of over-promising and under-delivering prediction models. Model building might be easier than ever, but evaluating the model’s predictive accuracy – its ability to predict outcomes in new samples – requires as much care and clarity as ever before.
In this workshop, we will start by covering basic principles of machine learning; that is, the main goals and challenges of prediction-making. This should provide participants with a common set of intuitions and a shared language to build upon. Following that, we’ll delve into the core of the workshop: evaluating predictive performance. We will focus on understanding and addressing the fundamental pitfalls of overfitting, data leakage, and distribution shifts. Finally, we’ll zoom out and briefly touch on other challenges, including model explainability, quantification of predictive uncertainty, and identification of causally-relevant predictors. With this, participants should come away able to address key obstacles to evaluating predictive accuracy, and to think critically about predictive research.
The workshop will include practical examples in R that illustrate common pitfalls in evaluating predictive accuracy, giving participants hands-on experience implementing methods to overcome them. Furthermore, the workshop will offer an opportunity to reflect on the role of prediction-making in participants’ own research and in science more generally, and to consider how predictive modeling can be combined with other scientific approaches to answer a broader range of research questions.
Prerequisites
- A basic understanding of R is strongly recommended for this workshop.
- Participants should have the following R libraries installed & loaded: tidyverse, tidymodels
Reading Materials
-
Watch ‘The Odyssey’ (2026) movie.
-
Read Yarkoni, T., & Westfall, J. (2017). Choosing prediction over explanation in psychology: Lessons from machine learning. Perspectives on Psychological Science, 12(6), 1100-1122. https://pmc.ncbi.nlm.nih.gov/articles/PMC6603289/pdf/nihms-845851.pdf
Optional
-
Collins, G. S., Dhiman, P., Ma, J., Schlussel, M. M., Archer, L., Van Calster, B., … & Riley, R. D. (2024). Evaluation of clinical prediction models (part 1): from development to external validation. bmj, 384. https://www.bmj.com/content/bmj/384/bmj-2023-074819.full.pdf
-
Riley, R. D., Archer, L., Snell, K. I., Ensor, J., Dhiman, P., Martin, G. P., … & Collins, G. S. (2024). Evaluation of clinical prediction models (part 2): how to undertake an external validation study. bmj, 384. https://www.bmj.com/content/bmj/384/bmj-2023-074820.full.pdf
-
Moons, K. G., Damen, J. A., Kaul, T., Hooft, L., Navarro, C. A., Dhiman, P., … & Van Smeden, M. (2025). PROBAST+ AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods. bmj, 388. https://www.bmj.com/content/bmj/388/bmj-2024-082505.full.pdf
-
Collins, G. S., Moons, K. G., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., … & Logullo, P. (2024). TRIPOD+ AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. bmj, 385. https://www.bmj.com/content/bmj/385/bmj-2023-078378.full.pdf
Capacity
This course has a maximum capacity of 35 participants.
Time and Location
This workshop will be held on-site only at Eindhoven University of Technologyon November 27, 2026. Details will be provided to all attendees over email after registration for the workshop.
Workshops start from 9:30 to 16:30 with a lunch break from 12:30 to 13:30. Lunch will not be provided but can be purchased at the university canteen or the on-campus supermarket.
Registration
This workshop is not yet open for registration.
Instructors
Dr. Benny Markovitch
Benny Markovitch is a postdoctoral researcher at Eindhoven University of Technology whose research focuses on game-based cognitive assessment and its use for predictive modeling. After obtaining master’s degrees in research psychology and statistics (with a specialization in data science), he worked for two years as a data scientist and biostatistician.