Tutorial at useR! 2019
Today, the pre-conference tutorials kick off this year’s useR! 2019 conference. The organizers kindly invited R/exams for one of these tutorials. This blog post provides some accompanying resources which may be of interest to those who do not attend the conference/tutorial. The plan is to roughly follow the outline below.
|Introduction (overview, installation, written exam)
Dynamic exercises (numeric, single choice, multiple choice, cloze)
One-for-all (plain PDF, plain HTML, Moodle XML)
|15:30||— Break —|
|E-Learning (Moodle, Canvas, ARSnova, …)
Written exams (NOPS)
Below, the examples, code, and links from the slides and R script are also provided which might be more convenient for some readers.
To set up your computer for this tutorial, follow Steps 1-4 in the tutorial http://www.R-exams.org/tutorials/installation/.
Subsequently, we start out with setps described in more detail in http://www.R-exams.org/tutorials/first_steps/. The package is loaded and first we create an “exams skeleton” with:
exercises/folder with all
templates/folder with various customizable templates
nops/folder (empty) for
To check whether everything works we render a single-choice question (swisscapital, knowledge quiz about the Swiss capital) into a plain HTML and PDF format, respectively.
And then we do the same for a numeric question with mathematical notation: (deriv, product rule for derivatives). By default
exams2html() uses MathML for the mathematical notation and expects the browser to render it which works ok in Firefox and Safari. But for Chrome it is better to use MathJax, see this post about mathematical notation in web exams for more details.
exams2html("deriv.Rmd") exams2html("deriv.Rmd", converter = "pandoc-mathjax") exams2pdf("deriv.Rmd")
Finally, a useful trick for checking whether the meta-information associated with an exercise is processed correctly:
This section gives a tour of the dynamic exercises shipped along with the package. See http://www.R-exams.org/intro/dynamic/ for the supported exercise types and http://www.R-exams.org/templates/ for a list of all example exercises.
And the following exercises are used to illustrate how to go from a static to a dynamic numeric or single-choice question, see http://www.R-exams.org/tutorials/static_num_schoice/ for more details. While the tutorial uses a basic economic example about computing elasticities, the exercises below are basic arithmetic exercises about derivatives.
||Fixed parameters and numeric solution.|
||As in #1 but with single-choice solution (five answer alternatives).|
||Randomly drawn parameters with dynamic computation of correct solution, based on #1.|
||Randomly drawn parameters (as in #3) with dynamically-generated single-choice solution (as in #2), computed by
||As in #4 but with the last alternative: None of the above.|
All of these extra exercise files can be downloaded from within R and stored in the skeleton set up above:
extra <- c("conferences", paste0("expderiv", 1:5)) extra <- c(paste0(extra, ".Rnw"), paste0(extra, ".Rmd")) for(i in extra) download.file(paste0("http://www.R-exams.org/assets/posts/2019-07-09-user2019/", i), i)
To take a tour of the available exercise files, we switch to the
exams2html(c( "deriv.Rmd", ## product rule for derivatives "tstat.Rmd", ## 1-sample t-test statistic "dist.Rmd", ## distances and the Pythagorean theorem "regression.Rmd", ## simple linear regression (by hand) "fruit.Rmd", ## image-based systems of linear equations (numeric) "lagrange.Rmd", ## method of Lagrange multipliers "currency8.Rmd" ## convert currencies (UTF-8 encoding) ))
exams2html(c( "swisscapital.Rmd", ## knowledge quiz question about the Swiss capital "Rlogo.Rmd", ## knowledge quiz question about the R logo "deriv2.Rmd", ## product rule for derivatives (single-choice, via num_to_schoice) "tstat2.Rmd", ## 1-sample t-test statistic (single-choice, by hand) "dist3.Rmd", ## distances and the Pythagorean theorem (single-choice, via num_to_schoice) "fruit2.Rmd", ## image-based systems of linear equations (single-choice, via num_to_schoice) "hessian.Rmd", ## 2x2 Hessian matrix (single-choice) "logic.Rmd" ## interpretation of logic gates (using TikZ) ))
exams2html(c( "switzerland.Rmd", ## knowledge quiz question about Switzerland "gaussmarkov.Rmd", ## knowledge quiz question about Gauss-Markov assumptions "anova.Rmd", ## 1-way analysis of variance "boxplots.Rmd", ## interpretation of 2-sample boxplots "scatterplot.Rmd", ## interpretation of a scatterplot "ttest.Rmd", ## interpretation of 2-sample t test "relfreq.Rmd", ## interpretation of relative frequency tables "cholesky.Rmd", ## Cholesky decomposition "automaton.Rmd" ## interpretation of automaton diagrams (using TikZ) ))
exams2html(c( "function.Rmd", ## string question about R functions "countrycodes.Rmd" ## string question about ISO country codes ))
These combine several num/schoice/mchoice/string parts:
exams2html(c( "lm.Rmd", ## simple linear regression (with CSV data, schoice/num) "boxhist.Rmd", ## univariate exploration of a CSV file (schoice/num) "confint2.Rmd", ## 2-sided confidence interval (num) "dist2.Rmd", ## three distances in a Cartesian coordinate system (num) "fourfold.Rmd" ## fourfold table (num) ))
Stress testing dynamic exercises
stresstest_exercise() can help to do some basic quality control of dynamic exercise templates. It runs the exercises
n times (default: 100) with seeds
n and stores run time, solution, and accompanying variables (of length 1). This can help to spot problems or patterns in the exercises.
A simple example for a numeric exercise is:
s3 <- stresstest_exercise("expderiv3.Rmd") plot(s3) plot(s3, type = "solution")
And for a single-choice exercise:
s4 <- stresstest_exercise("expderiv4.Rmd") plot(s4) plot(s4, type = "solution")
To focus on typical errors the plots against the variables
err2 can either be created by the
plot() method or “by hand” with some customization:
plot(s4, type = "solution", variables = c("err1", "err2")) plot(res ~ err1, data = s4$objects) abline(0, 1) plot(res ~ err2, data = s4$objects) abline(0, 1)
Finally, we switch back to the original folder:
A basic design idea for R/exams is that the same exercise can, in principle, be rendered into a wide range of different output formats. For standalone files the most important formats are PDF, HTML, and possibly also Docx:
demo_exam <- c("swisscapital.Rmd", "deriv.Rmd") exams2html(demo_exam, edir = "exercises") exams2pdf(demo_exam, edir = "exercises") exams2pandoc(demo_exam, edir = "exercises")
The layout can be customized using suitable templates, e.g.,
exams2pdf(demo_exam, edir = "exercises", template = "templates/exam.tex") exams2pdf(demo_exam, edir = "exercises", template = "templates/solution.tex")
exams2nops() internally sets up a template on the fly and then calls
exams2pdf(). And e-learning exams (Moodle XML, QTI 1.2/2.1, …) are illustrated in the next section.
E-Learning (Moodle, Canvas, ARSnova, …)
Output for various e-learning formats can be generated in pretty much the same way as the standalone files above. The main difference is that not one file per exam is generated but a combined output file comprising the desired number of exam replications. For example for Moodle:
elearn_exam <- c("conferences.Rmd", "deriv.Rmd", "ttest.Rmd", "boxplots.Rmd", "function.Rnw", "lm.Rmd", "fourfold2.Rmd", "confint3.Rmd") set.seed(2019-07-09) exams2moodle(elearn_exam, n = 30, name = "useR-2019", dir = "output", edir = "exercises")
For the registered participants of the course the resulting online quiz is made available at https://eeecon.uibk.ac.at/~moodle/. You can login with your e-mail address and the numeric code sent to you.
Analogously, the numeric and single-/multiple-choice questions (but not yet cloze) can be prepared for Canvas by:
set.seed(2019-07-09) exams2canvas(elearn_exam[1:5], n = 3, name = "useR-2019", dir = "output", edir = "exercises", duration = 15)
For OpenOLAT all exercises except one Moodle-specific exercise can be used as well:
set.seed(2019-07-09) exams2openolat(elearn_exam[1:7], n = 3, name = "useR-2019-olat", dir = "output", edir = "exercises")
For ARSnova only single- and multiple-choice questions have good support:
choice_exam <- list( "conferences.Rmd", "deriv2.Rmd", c("ttest.Rmd", "boxplots.Rmd") ) set.seed(2019-07-09) exams2arsnova(unlist(choice_exam), n = 1, name = "useR-2019", dir = "output", edir = "exercises")
Written exams (NOPS)
The written demo exam from the tutorial can be replicated by:
set.seed(2019-07-09) exams2nops(choice_exam, n = 35, name = "useR-2019", dir = "nops", edir = "exercises", reglength = 8, blank = 1, date = "2019-07-09", title = "Demo Exam", institution = "useR! 2019")
More details about creating and evaluating NOPS exams is provided in the tutorial at http://www.R-exams.org/tutorials/exams2nops/.