In addition to completely customizable PDF output, R/exams provides a standardized format called “NOPS” for written exams with multiplechoice (and/or singlechoice) exercises that can be automatically generated, scanned, and evaluated. In order to assure that the automatic scanning works the title page has a fixed layout that cannot be modified by R/exams users. However, to internationalize the format there is a possibility for customizing the natural language support. A number of languages is already available but it is not difficult to add further languages or to tweak existing languages if desired.
To illustrate how the language support works, once it has been fully incorporated into the
exams
package, we set up a short exam with three exercises:
deriv2, tstat2,
swisscapital. All of these are readily available
in the package (and are actually in English).
library("exams")
myexam < c("deriv2.Rnw", "tstat2.Rnw", "swisscapital.Rnw")
Then we set up PDF output in English (en), German (de), and Spanish (es).
By setting language
most text on the title page is modified, only the name of the
institution
and the title
of the exam have to be set separately. For the English
example we produce n = 1
PDF output file in the output directory nops_pdf
(created
automatically).
set.seed(403)
exams2nops(myexam, n = 1, language = "en",
institution = "R University", title = "Exam",
dir = "nops_pdf", name = "en", date = "20180108")
Then we do the same for the other two languages.
set.seed(403)
exams2nops(myexam, n = 1, language = "de",
institution = "R Universit\\\"at", title = "Klausur",
dir = "nops_pdf", name = "de", date = "20180108")
set.seed(403)
exams2nops(myexam, n = 1, language = "es",
institution = "R Universidad", title = "Examen",
dir = "nops_pdf", name = "es", date = "20180108")
The title pages of the resulting PDF files then have the desired languages.
To add a new language, essentially just a single text file (say lang.dcf
) is needed containing
suitable translations of all the phrases on the title page as well as a few additional phrases,
e.g., occuring in the HTML evaluation reports etc.
As an example, the first few phrases in English (en.dcf
) are:
PersonalData: Personal Data Name: Name FamilyName: Family Name GivenName: Given Name Signature: Signature RegistrationNumber: Registration Number Checked: checked NoChanges: In this section \textbf{no} changes or modifications must be made! ...
And the corresponding translations to German (de.dcf
) are:
PersonalData: Pers{\"o}nliche Daten Name: Name FamilyName: Nachname GivenName: Vorname Signature: Unterschrift RegistrationNumber: Matrikelnummer Checked: gepr{\"u}ft NoChanges: In diesem Feld d{\"u}rfen \textbf{keine} Ver{\"a}nderungen der Daten vorgenommen werden! ...
Note that here LaTeX markup is used for the German umlaute and for bold highlighting. Alternatively,
special characters can be added in a suitable encoding (typically UTF8) but then the encoding has
to be declared when calling exams2nops()
(e.g., encoding = "UTF8"
).
Most of the phrases required in the .dcf
are very straightforward and only some are a bit technical.
There are also a couple of coordinates (MarkExample*
) necessary for aligning some text lines.
If you have set up your own lang.dcf
you can easily pass it to exams2nops()
by setting
language = "/path/to/lang.dcf"
. The same has to be done for nops_eval()
when evaluating the exam.
Due to the kind support from friends and various dedicated R/exams users, there is already support
for many important Western languages as well as a few other languages/countries. All of these
are directly available in the R package (note that "tr"
requires the current
development version from RForge, though). But for convenience
and manual inspection the .dcf
files are also linked here.
File  Language  Contributor 

de.dcf  German  Achim Zeileis 
en.dcf  English  Achim Zeileis 
es.dcf  Spanish  Maria Kogelnik 
fi.dcf  Finnish  Klaus Nordhausen 
fr.dcf  French  Arthur Allignol 
hu.dcf  Hungarian  Gergely Daróczi & Dénes Tóth 
it.dcf  Italian  Domenico Zambella 
nl.dcf  Dutch  Niels Smits 
pt.dcf  Portugese  Mauricio Calvão & Fabian Petutschnig 
ro.dcf  Romanian  Cristian Gatu 
tr.dcf  Turkish  Emrah Er 
If you want to contribute a new language, simply set up a .dcf
file starting out from one of the examples
above and send the file or a link to
<info at Rexams.org>.
Do not worry if not everything is 100% perfect, yet, we can still sort this out together!
For Western languages (e.g., dk
, se
, no
are still missing) it is probably the most robust solution to
code special characters in LaTeX. For languages requiring other alphabets (e.g., ru
would be nice or Asian languages…)
it is probably easiest to use UTF8 encoding. Get in touch through email, the
support forum
or on Twitter (@AchimZeileis) if you want to know more or need further details.
Many statisticians regularly teach large lecture courses on statistics, probability, or mathematics for students from other fields such as business and economics, social sciences and psychology, etc. A common challenge in such courses is that many variations of similar exercises are needed for written exams, online tests conducted in learning management systems (LMS, such as Moodle, Blackboard, etc.), or live quizzes with voting via smartphones or tablets.
Here, it is shown how the Department of Statistics at Universität Innsbruck manages the firstyear mathematics 101 course for economics and business students, attended by about 500 to 1,000 students per semester. The course is accessible through the university’s LMS (an OpenOLAT system), including anonymous “Guest access” for most materials. So you can either look at the screen shots presented below in this blog post or try out the course yourself: https://lms.uibk.ac.at/url/RepositoryEntry/3051978752?guest=true. (Notes: The course materials are in German but also nonGermanspeakers should get a good impression of how things are tied together. The online tests contain mathematical equations in MathML which is displayed by browsers with MathML support like Firefox or Safari  but not Chrome.)
As highschool knowledge about mathematics is rather heterogeneous among the students, the course has been designed so that they can learn either synchronously (in a classical lecture that is also streamed live) or asynchronously (with a textbook and/or screencasts). This way students can decide which parts of the content just need reviewing and which need deepening. Similarly, they can get feedback (not directly relevant for their grade) during lecture and tutorial sessions synchronously or asynchronously “in their own time”. Finally, the assessment is also done in two parts: online tests that they can do over the course of the semester (at any time within a 3.5day window each) and a classical written exam at the end of the semester.
Learning  Feedback  Assessment  

Synchronous  Lecture Live stream 
Live quiz (+ Tutorial) 
Written exam 
Asynchronous  Textbook Screencast 
Self test (+ Forum) 
Online test 
Subsequently, it will be discussed how the different feedback and assessment formats are generated with R/exams from the same pool of exercise templates.
The core of the course are about 400 dynamic exercise templates where students have to compute some number based on a problem description or a short story. Some examples from this pool have also been translated into English and included in R/exams, e.g., deriv, lagrange, hessian. Almost all of these exercises have been established as num exercises first (with numeric result, employed in online tests) and then converted to singlechoice format (schoice, see e.g. deriv2) as this format can be easily scanned automatically in written exams. Additionally, there are about 70 dynamic multiplechoice questions (mchoice) that are used either in online tests or live quizzes.
As illustrated in the following, these exercises are the thread running through the enitre course, starting with the lecture slides where typically one static snapshot from an exercise is used (potentially with some additional explanations, incremental slides, graphics, etc.) Below we show screenshots from such a curve sketching exercise in the screencasts (based on the slides). It depicts a simple cubic polynomial of type a x^{3} + b x^{2} + c x + d for which the two local optima and the inflection point have to be found.
The corresponding dynamic exercise templates can then be used in an online test. In fact, here there are three dynamic templates that draw different coefficients a, b, c, and d and either ask for the function value f(x) in one of the local optima (A or C), or the second derivative f’‘(x) in one of the local optima (A or C), or the slope f’(x) in the inflection point B.
In the screenshots below the second derivative f’‘(x) in C is demanded, but the argument x is C incorrectly entered/saved by the student, prompting OpenOLAT to display the full correct approach to the solution.
To import an online test into OpenOLAT, an XMLbased exchange format by the IMS Global Learning Consortium is used:
IMS Question & Test Interoperability, Version 1.2 (QTI 1.2).
The R/exams function exams2qti12()
generates a zipped XML file
that can be imported into OpenOLAT as a complete online test. To do so
we first set up a list of exercises à la:
otest < list(
c("derivpoly.Rnw", "derivsqrt.Rnw", "derivexp.Rnw", ...),
c("graphderivA.Rnw", "graphderivB.Rnw", "graphderivC.Rnw"),
...
)
And then exams2qti12()
can proceed to draw many replications from this combining two forms of
randomization: (a) Sampling one item from a given vector of exercise templates (e.g., here
the deriv*.Rnw
templates for the first item). (b) In each .Rnw
template random
numbers are drawn and inserted into the exercise (e.g., the coefficients a, b, c, d
in the cubic polynomial). For example:
exams2qti12(otest, n = 500, name = "MathematikOnlinetest01",
solutionswitch = FALSE, maxattempts = 2, cutvalue = 1000)
draws 500 random versions for each of the items in the test. We don’t
show the correct solution approach at the end (only the correct numeric result, solutionswitch = FALSE
)
but allow our students to attempt each item within the test twice (i.e., one wrong attempt is
tolerated). Because individual tests cannot be passed or failed (they just collect points for
the overall course), we set cutvalue = 1000
which can never be attained. Due to the large
exercise pools and the large number of random variations, the students cannot simply copy
a single number or a certain solution formula. Of course, during a 3.5day test it is possible
to help each other and in fact we encourage the students to form small learning groups.
Further control arguments of exams2qti12()
include the number of points
and the eval
strategy (e.g.,
for negative points upon incorrect results etc.). See ?exams2qti
for more technical
details and also for an example you can run yourself based on exercises from the
template gallery.
In addition to the online tests (= assessment) we employ self tests (= feedback), either in the
form of selected demo exercises (“Musteraufgaben”) for which the full solution approach is shown
(solutionswitch = TRUE
), short tests that sample exercises from almost the entire exercise
pool (“Selbsttests”), or “mock exams” based on past written exams (“Probeklausuren”).
Course element 
Item type 
Templates per item 
Full solution 
Show points during test? 
Time (min.) 
Attempts per item 
Attempts per test 
Purpose 

Online test  num or mchoice 
320  x  2  1  Assessment during semester 

Demo exercises (Musteraufgaben) 
num or mchoice 
1  x  x  1  ∞  Feedback after lecture 

Self test (Selbsttest) 
schoice  320  x  1  ∞  Feedback during semester 

Mock exam (Probeklausur) 
schoice  1  90  ∞  ∞  Feedback before exam 
As the written exams use singlechoice versions of the exercises, the latter two test types also practice using this item type (see the screenshot below).
The course ends in a “strict” written exam (“Gesamtprüfung”) in the sense that now each student really
has to solve his/her own exam without further help in a time window of 90 minutes.
The function employed for generating the PDF files is exams2nops()
and its usage
is illustrated in detail in the tutorial on written exams.
First, a vector (as opposed to a list) of exercises is set up à la:
gp < c(
"derivsqrt.Rnw",
"graphderivB.Rnw",
...
)
Thus, for written exams we assure that every student gets the same exercise template, only with some random variations of the numbers/graphics/etc. Thus, we try to make the exam as fair as possible while assuring that cheating is very hard. If there are, say, 400 participants in the exam, there are 400 different exams with singlechoice exercises. These can be generated (typically after setting a random seed for reproducibility) via:
set.seed(20170405)
exams2nops(gp, n = 400, name = "GP", title = "GP Mathematik",
institution = "Universit\\\"at Innsbruck", logo = "uibklogobw.png",
language = "de", course = "403100200", date = "20170405", reglength = 7,
pages = c("/path/to/formulary.pdf", "/path/to/normaltable.pdf"))
Thus, in addition to some information about course/exam, we specify the language (German), length of the registration ID (7 digits), and add PDF pages with the formulary and normal tables. The first few pages of one of the resulting PDFs are displayed below. More details on the evaluation process can be found in the exams2nops tutorial.
A very nice benefit of using R/exams here is that we can share an interactive online
“mock exam” (“Probeklausur”) in addition to the PDF version in the exam archive. This
can again be generated by exams2qti12()
with time limit and evaluation settings
corresponding to the actual written exam:
exams2qti12(gp, n = 10, name = "MathematikGP20170405",
maxattempts = Inf, cutvalue = 18, solutionswitch = FALSE,
points = 2, eval = list(partial = FALSE, negative = 0.25),
duration = 90)
The screen shots below show first the interactive version of the mock exam and then one of the PDFs in the resources archive.
Finally, the same exercise templates can also be imported into live quizzes when the
audience response system ARSnova is used (see https://arsnova.eu/). As we do not use
the system for assessment but only for feedback it is not as important to have a lot
of random variation in the exercises, but it certainly helps to avoid duplication
across courses taught in parallel or in subsequent semesters.
Moreover, the R/exams function exams2arsnova()
can directly import the drawn exercises
into ARSnova using its REST interface (via Rcurl and suitable JSON data). Alternatively,
JSON files can be generated on the hard disk and subsequently imported manually.
As an illustration we use again a vector of exercise templates and export random draws from these directly into a previouslygenerated ARSnova session (using one session for the enitre semester). Authentication can be done through the JSESSIONID.
quiz < c("derivsqrt.Rnw", "graphderivB.Rnw", ...)
exams2arsnova(quiz, name = "Quiz", url = "https://arsnova.uibk.ac.at",
sessionkey = "79732490", jsessionid = "34A6B8C1D83CB7D0EC66F14DA7044DB1",
fix_choice = FALSE, abstention = FALSE)
The students can then view the exercises on the mobile phones or tablets, practicing for the exam, followed by a phase of peer instruction to resolve remaining problems.
Exercises like shown in the two screenshots above are used for exam practice during lecture or tutorial sessions. Additionally, knowledge quiz questions are frequently used for checking whether the students can follow the lecture or not. These are typically in the form of multiplechoice questions where each subitem can either be true or false. See gaussmarkov for a worked example.
This blog post has shown how the same pool of exercises can be used flexibly for different elements of a largelecture course. Over the last years we experimented quite a bit with these and kept those elements we (or the students) found useful but discontinued using others. Of course, the hard part is establishing the large exercise pool but then R/exams enables us to try out different formats fairly easily.
If you have further questions about these tools or want to let us know how you use R/exams in your course, please get in touch! Either via our forum, StackOverflow, Twitter, or simply by email.
]]>Our colleagues over at the economics department became interested in using R/exams for
generating largescale exams in their introductory economics courses. However, they face the challenge that
so far they had been writing static exercises and modified them by hand if they wanted to reuse them in a
different exam in another semester. To let R/exams do this job it is illustrated how a static arithmetic
exercise can be turned into a dynamic exercise template either in num
format with a numeric solution or into schoice
format with a singlechoice solution.
The idea for the exercise is a very basic price elasticity of demand task:
Consider the following inverse demand function: $p(x)=ab\xb7x$ for the price $p$ given the demanded quantity $x$. What is the price elastiticy of demand at a price of $p$? 
The natural candidates for “parameterizing” this exercise are the price $p$ and the parameters $a$ and $b$ of the inverse demand function. Based on these the solution is simply:
First, we obtain the demand function by inverting the inverse demand function: $x=D(p)=(ap)/b$. Then, at $p$ the price elasticity of demand is $\frac{D\text{'}(p)}{D(p)}p=\frac{1/b}{x}p.$ 
Below various incarnations of this exercise are provided in both R/Markdown Rmd
and R/LaTeX Rnw
format.
The following table gives a brief overview of all available versions along with a short description of the idea behind it.
More detailed explanations are provided in the subsequent sections.
#  Exercise templates  Dynamic?  Type  Description 

1  elasticity1.Rmd elasticity1.Rnw 
No  num 
Fixed parameters and numeric solution. 
2  elasticity2.Rmd elasticity2.Rnw 
No  schoice 
As in #1 but with singlechoice solution (five answer alternatives). 
3  elasticity3.Rmd elasticity3.Rnw 
Yes  num 
Randomly drawn parameters with dynamic computation of correct solution, based on #1. 
4  elasticity4.Rmd elasticity4.Rnw 
Yes  schoice 
Randomly drawn parameters (as in #3) with dynamicallygenerated singlechoice solution (as in #2), computed by num_to_schoice() . 
5  elasticity5.Rmd elasticity5.Rnw 
Yes  schoice 
As in #4 but with the last alternative: None of the above. 
The starting point is a completely static exercise as it had been used in a previous introductory economics exam. The parameters had been set to $p=5$, $a=50$, and $b=0.5$. This implies that $x=90$, leading to an elasticity of $0.111111$.
The corresponding R/exams templates simply hardcode these numbers into the question/solution and wrap everything into R/Markdown (elasticity1.Rmd) or R/LaTeX (elasticity1.Rnw). Note that LaTeX is used in either case for the mathematical notation. In case you are unfamiliar with the R/exams format, please check out the First Steps tutorial.
The metainformation simply sets extype
to num
, supplies the exsolution
with a precision of three digits,
and allows an extol
tolerance of 0.01. To see what the result looks like download the files linked above
and run exams2html()
and/or exams2pdf()
. (The examples below always use the R/Markdown version but the
R/LaTeX version can be used in exactly the same way.)
library("exams") exams2html("elasticity1.Rmd") exams2pdf("elasticity1.Rmd")
Singlechoice versions of exercises are often desired for use in written exams because they can be conveniently scanned and automatically evaluated. Thus, we need to come up with a number of incorrect alternative solutions (or “distractors”). If desired, these could include typical wrong solutions or a None of the others alternative.
The R/exams templates elasticity2.Rmd and elasticity2.Rnw are essentially copies of the static numeric exercise above but:
extype
has been changed to schoice
.exsolution
now contains a binary coding of the correct solution.exshuffle
to TRUE
. (Subsampling more than five alternatives would also be possible and would add some further randomization.)As above exams2html()
and/or exams2pdf()
can be used to display the exercise interactively in R/exams.
Next, the static exercise from above is made dynamic by drawing the parameters from a suitable datagenerating process. In this case, the following works well:
## p = a  b * x p < sample(5:15, 1) fctr < sample(c(2, 4, 5, 10), 1) x < sample(5:15, 1) * fctr b < sample(1:5, 1) / fctr a < p + b * x ## elasticity sol < 1/b * p/x
Note that in order to obtain “nice” numbers a common scaling factor fctr
is used for both x
and b
.
Also, while the examinees are presented with parameters a
and b
and have to compute x
,
the datagenerating process actually draws x
and b
and computes a
from that. Again, this makes it
easier to obtain “nice” numbers.
The R/exams templates
elasticity3.Rmd and
elasticity3.Rnw include
the datagenerating process above as a code chunk either in R/Markdown or R/LaTeX format.
The parameters are then inserted into question/solution/metainformation using `r a`
(R/Markdown)
or \Sexpr{a}
(R/LaTeX). Sometimes the fmt()
function is used for formatting the parameters
with a desired number of digits. (See ?fmt
for more details.)
As before exams2html()
and/or exams2pdf()
can be used to display the random draws from
the exercise templates. For checking that the metainformation is included correctly, it is often
helpful to run
exams_metainfo(exams2html("elasticity3.Rmd"))
Furthermore, some tweaking is usually required when calibrating the parameter ranges in the
datagenerating process. The stresstest_exercise()
function draws a large number of random replications
and thus can help to spot errors that occur or to find solutions that are “extreme” (e.g., much larger or smaller than usual). To clean up your
global environment and run the function use something like this:
rm(list = ls()) s < stresstest_exercise("elasticity3.Rmd", n = 200) plot(s) plot(s, type = "solution")
The latter command plots the correct solution against the (scalar) parameters that were
generated in the exercise. This might show patterns for which parameters the solution becomes
too large or too small etc. See ?stresstest_exercise
for further features/details.
To go from the dynamic numeric exercise to a dynamic singlechoice exercise we need to
extend the datagenerating process to also produce a number of wrong alternatives.
The function num_to_schoice()
helps with this by providing different sampling
mechanisms. It allows to set a range in which the alternatives have to be, a minimum
distance between all alternatives, possibly include typical wrong solutions, etc.
It also shuffles the resulting alternatives and tries to make sure that the correct
solution is not a certain order statistic (e.g., almost always the largest or
smallest alternative).
The R/exams templates
elasticity4.Rmd and
elasticity4.Rnw first
wrap the datagenerating process into a while loop with while(sol > 0.11)
. This makes sure
that there is enough “space” for four wrong alternatives between 0.11 and 0, using a minimum
distance of 0.017. Subsequently, the four wrong alternatives are generated by:
## singlechoice incl. typical errors err < c(1/sol, sol/p, p/sol) err < err[(err > 5) & (err < 0.2) & abs(err  sol) > 0.01] rng < c(min(1.5 * sol, 1), 0.01) sc < num_to_schoice(sol, wrong = err, range = rng, delta = 0.017, method = "delta", digits = 3)
This suggests a number of typical wrong solutions err
(provided that they are not too small
or too large) and makes sure that the range rng
is large enough. With these arguments
num_to_schoice()
is run, see ?num_to_schoice
for the details. The resulting sc
list
contains suitable $questions
and $solutions
that can be easily embedded in an answerlist()
and in the metainformation. Sometimes it is useful to wrap num_to_schoice()
into another
while()
loop to make sure a valid result is found. See the deriv2
template for illustraion.
As before exams2html()
and/or exams2pdf()
can be used to display the random draws from
this exercise template. And some stresstesting could be carried out by:
rm(list = ls()) s < stresstest_exercise("elasticity4.Rmd", n = 200) plot(s) plot(s, type = "ordering")
As a final variation we could include None of the above as the last alternative. This is very easy because we can simply replace the fifth element of the question list with the corresponding string:
sc$questions[5] < "None of the above."
No matter whether this was actually the correct alternative or one of the incorrect alternatives, the corresponding solution will stay the same. The R/exams templates elasticity5.Rmd and elasticity5.Rnw incorporate this additional line of code in the datagenerating process.
]]>Welcome everybody, we are proud to introduce the brand new web page and blog http://www.Rexams.org/. This provides a central access point for the opensource software “exams” implemented in the R system for statistical computing. R/exams is a oneforall exams generator using (potentially) dynamic exercises written in R/Markdown or R/LaTeX and it can export a variety of formats from learning management systems to classical written exams. This post gives a brief overview of what has happened so far and what you can expect to find here in the next months.
The package was originally started more than a decade ago to facilitate classical written exams with randomized exercises for largelecture courses. Like many other teachers of introductory statistics and mathematics courses, we were in need of infrastructure for conducting written exams with about 1,0001,500 students. A team effort of colleagues at WU Wirtschaftsuniversität Wien lead to a large collection of dynamic exercises and the software was eventually released at https://CRAN.Rproject.org/package=exams.
Over the years learning management systems (like Moodle, Blackboard, OLAT, etc.) became easily available at virtually every university, creating a desire to employ the same dynamic exercises also for online tests and quizzes. Hence, the R/exams infrastructure was completely reimplemented allowing to export the same exercises not only to written exams (with automatic scanning and evaluation) but also to learning management systems, the live voting system ARSnova, as well as customizable standalone files in PDF, HTML, Docx, and beyond.
Despite (or rather because of) the flexibility of the software, novice R/exams users often struggled with adopting it because the documentation provided in the package is either somewhat technical and/or targeted towards more experienced R users.
Hence, this web page and blog make it easy for new users to explore the possibilities of R/exams before reading about a lot of technical details. It also provides accessible guides to common tasks and examples for dynamic exercises with different complexity. For a first tour you can check out the oneforall approach of the package based on (potentially) dynamic exercise templates for generating large numbers of personalized exams/quizzes/tests, either for elearning or classical written exams (with automatic evaluation).
Some tutorials already explain the installation of R/exams (with dependencies like LaTeX or pandoc) and the first steps in writing dynamic exercises using either Markdown or LaTeX markup along with randomization in R. There is also a gallery of exercise templates, ranging from basic multiplechoice questions with simple shuffling to more advanced exercises with random data, graphics, etc.
For the next months we plan to write more tutorial blog posts that help to accomplish common tasks, e.g., handson guidance for exporting exercises from R to Moodle or tips how to write good randomized exercises. If you want to give us feedback or ask us to cover certain topics please feel free to contact us  be it via email, discussion forum, or twitter. Also if you want to link R/examsbased materials or share share experiences of using R/exams in a guest post, please let us know.
Big shout to all contributors that helped R/exams to grow so much over the last years. A special thank you goes to Patrik Keller, Hanna Krimm, and Reto Stauffer who established the infrastructure for this web page (using R/Markdown and Jekyll) and designed graphics/icons/photos/etc. Thanks also to everyone sharing this post, especially on http://www.Rbloggers.com/ and https://RWeekly.org/.
]]>The first step in conducting a written exam with multiplechoice
(and/or singlechoice) exercises in R/exams' NOPS format is to create the exam in PDF format.
First, we load the R exams
package and then simply create a list of exercise file names.
library("exams")
myexam < list(
"tstat2.Rnw",
"ttest.Rnw",
"relfreq.Rnw",
"anova.Rnw",
c("boxplots.Rnw", "scatterplot.Rnw"),
"cholesky.Rnw"
)
Here, we use a number of schoice
and mchoice
questions that are directly shipped
within the package. In practice, you would use files that you have authored and stored
somewhere locally. Above, exercises in .Rnw format are used but all of the examples
are also available in .Rmd format, leading to virtually identical output.
Then, we create a small exam with only n = 2
randomlydrawn versions, storing the
resulting PDF files (plus metainformation) on the disk in a new output directory nops_pdf
.
To customize the exam we assign a different number of points to the different exercises
and also show the respective number of points at the beginning of each question.
set.seed(403)
ex1 < exams2nops(myexam, n = 2,
dir = "nops_pdf", name = "demo", date = "20150729",
points = c(1, 1, 1, 2, 2, 3), showpoints = TRUE)
A random seed is set to make the generated exams exactly reproducible for you (or ourselves at some point in the future). The output directory now contains three files that were generated.
dir("nops_pdf")
## [1] "demo.rds" "demo1.pdf" "demo2.pdf"
The two PDF files are the two exams we requested above.
Furthermore, the metainfromation about the exam (exam IDs, questions, correct and wrong
answer alternatives) is stored in a demo.rds
file (serialized R data). This crucial
for being able to evaluate the exam later on.
exams2nops()
have a blank second page for
duplex printing (without content on the back of the exam sheet). For nonduplex printing
simply set duplex = FALSE
when creating the PDFs
with exams2nops()
.The exam is conducted as usual. But if you used the possibilities of dynamic exercises in R/exams, the risk of cheating is greatly reduced. At the end of the exam you just need to collect the completed exam sheet (first page). Of course, you can also collect the rest of the exam papers (e.g., to keep future students from seeing the exercises). However, an advantage of letting the students keep their exercises reduces the need of having postexam reviews etc.
Each completed exam sheet has information on the exam ID, student ID, and the
checked answers. This needs to be scanned into images (PDF or PNG) and can then
be processed by nops_scan()
.
Typically, it's easy to use the photocopiers provided by your university to scan
all sheets into PDF or PNG files. For example, our university provides us with
Canon ImageRunners and the sheet feeder can easily take about 4050 sheets and
render them into a single PDF file.
Practical recommendations:
rotate = TRUE
in nops_scan()
below.For demonstration, we use two completed demo sheets that are provided along with
the exams
package and copy them to a dedicated directory nops_scan
.
img < dir(system.file("nops", package = "exams"), pattern = "nops_scan",
full.names = TRUE)
dir.create("nops_scan")
file.copy(img, to = "nops_scan")
(Note that the first scanned image corresponds to one of the PDFs above while the other one was generated with custom title/logo/language/etc.)
Using the function nops_scan()
we can now read all scanned images (i.e.,
all locally available PNG and/or PDF files) and collect everything in a ZIP file.
(Note that if there were PDF files that need to be scanned, then the
PDF toolkit pdftk
and the function convert
from ImageMagick need to be
available outside of R on the command line.)
nops_scan(dir = "nops_scan")
dir("nops_scan")
## [1] "nops_scan_20170810004404.zip" "nops_scan1.png"
## [3] "nops_scan2.png"
The resulting file nops_scan_20170810004404.zip contains copies of the
PNG files along with a file called Daten.txt
(for historical reasons) that
contains the scanned information in machinereadable from.
See ?nops_scan
for more details, e.g., multicore support or options for
rotating PDF files prior to scanning.
In the previous scanning step the exam sheets have just been read but not
yet evaluated, i.e., it has not yet been assessed which questions were answered
(partially) correctly and which were wrong, and no points have been assigned.
Therefore, we use nops_eval()
to carry out these computations and to make
the results available  both in a format easy to read for machines (a CSV file)
and a format for humans (one HTML page for each student).
To do so, three files are required:
exams2nops()
above.nops_scan()
above.write.table(data.frame(
registration = c("1501090", "9901071"),
name = c("Jane Doe", "Ambi Dexter"),
id = c("jane_doe", "ambi_dexter")
), file = "Exam20150729.csv", sep = ";", quote = FALSE, row.names = FALSE)
The resulting file is Exam20150729.csv.
Now the exam can be evaluated creating an output data frame (also stored as a CSV file) and individual HTML reports (stored in a ZIP file). Here, we employ an evaluation scheme without partial points in the multiplechoice questions and differing points across questions.
ev1 < nops_eval(
register = "Exam20150729.csv",
solutions = "nops_pdf/demo.rds",
scans = Sys.glob("nops_scan/nops_scan_*.zip"),
eval = exams_eval(partial = FALSE, negative = FALSE),
interactive = FALSE
)
dir()
## [1] "Exam20150729.csv" "nops_eval.csv" "nops_eval.zip"
## [4] "nops_pdf" "nops_scan"
The evaluated data can be inspected by opening nops_eval.csv in some spreadsheet software or we can directly look at the data in R. Based on this information, the marks could be entered into the university’s information system.
ev1
## registration name id exam scrambling
## 1501090 1501090 Jane Doe jane_doe 15072900001 00
## 9901071 9901071 Ambi Dexter ambi_dexter 15072900002 00
## scan points mark answer.1 solution.1 check.1 points.1
## 1501090 nops_scan1.png 4 5 00100 00100 1 1
## 9901071 nops_scan2.png 0 5 10100 10000 0 0
## answer.2 solution.2 check.2 points.2 answer.3 solution.3 check.3
## 1501090 11101 11100 0 0 00000 00000 1
## 9901071 10111 11001 0 0 01000 01010 0
## points.3 answer.4 solution.4 check.4 points.4 answer.5 solution.5
## 1501090 1 00100 00110 0 0 00010 00010
## 9901071 0 00000 01011 0 0 00000 11010
## check.5 points.5 answer.6 solution.6 check.6 points.6
## 1501090 1 2 01101 01111 0 0
## 9901071 0 0 11100 00011 0 0
And nops_eval.zip
contains subdirectories with HTML reports for each of the two participants.
These HTML reports could then be uploaded into a learning management system, put on some other web server, or even sent out via email.
]]>The basic idea of R/exams is to have a oneforall exams generator where each exercise is a standalone file, so that the same exercise can be rendered easily into a variety output formats such as elearning systems or written exams. To reduce the risk of cheating, R/exams offers different mechanisms for drawing random variations of exercises:
Each exercise (or rather “exercise template”) is a standalone file with up to four elements:
Only the question (2) and the metainformation (4) are mandatory while data generation (1) and solution (3) are optional. Solutions can be used to either inform teachers about the correct solution for some randomly drawn exercise  or for giving students feedback about how to solve a given exercise. Programming a good datagenerating process is often the hardest part of authoring a new dynamic exercise but is, of course, crucial for drawing a very large number of random variations.
For formatting the text within question/solution (e.g., italics, bold face, lists, etc.)
two options are available: Markdown or
LaTeX. The former is very easy to learn
and recommended for authors who do not know LaTeX yet. If mathematic notation and
formulas are needed, then LaTeX can be used  either within a LaTeX or also within
a Markdown question. The file suffix for R/Markdown
is .Rmd
while .Rnw
is conventionally used for R/LaTeX, known as
Sweave.
In the following, a first session should help new R/exams users to explore the materials available and get a feeling for the workflow. Furthremore, two exercise templates are shown in more detail:
To explore what is available in R/exams two things are necessary: a couple of exercise
templates to draw inspiration from, and some demo R scripts that illustrate how the
different exams2xyz()
functions can produce output in a variety of formats. (Here,
xyz
is a placeholder for the output format like html
, pdf
, moodle
, etc.)
The function exams_skeleton()
is a helper function that generates exactly these
materials in a specified folder. For example, when exams
has already been
installed, the following can be carried out
in R:
library("exams")
exams_skeleton(markup = "markdown", encoding = "UTF8",
writer = c("exams2html", "exams2pdf", "exams2moodle"))
This copies all R/Markdown files to the current working directory along with demo scripts,
illustrating the usage of exams2html()
and exams2pdf()
for customizable HTML and
PDF output, respectively, along with Moodle output via exams2moodle()
. Specifically,
the working directory then contains:
dir()
## [1] "demoall.R" "demohtml.R" "demomoodle.R" "demopdf.R"
## [5] "exercises" "templates"
Simply open the demo script
demoall.R for the first steps and then continue with
more details in demohtml.R, demopdf.R, or
demomoodle.R, respectively.
More information about all the exercises
can be found in the
exercise template gallery online.
Knowledge quiz question (about the Swiss capital) with 1 correct and 6 false alternative.
4 out of the 6 false alternatives are sampled randomly and shuffled. Otherwise the
exercise is static and contains no R code. Below the structure of the exercise is
highlighted sidebyside in both .Rmd
and .Rnw
, illustrating that the additional
exercise markup is similar in spirit to the respective typesetting system.
To see what the output looks like, run the following code within R or take a look at swisscapital in the exercise gallery.
library("exams") exams2html("swisscapital.Rmd") exams2pdf("swisscapital.Rmd")
library("exams") exams2html("swisscapital.Rnw") exams2pdf("swisscapital.Rnw")
The question is straightforward and lists 7 answer alternatives. The correct solution (Bern) is declared in the metainformation below.
Question ======== What is the seat of the federal authorities in Switzerland (i.e., the de facto capital)? Answerlist  * Basel * Bern * Geneva * Lausanne * Zurich * St. Gallen * Vaduz
\begin{question} What is the seat of the federal authorities in Switzerland (i.e., the de facto capital)? \begin{answerlist} \item Basel \item Bern \item Geneva \item Lausanne \item Zurich \item St.~Gallen \item Vaduz \end{answerlist} \end{question}
The optional solution can provide some general feedback (that explains why Bern is correct in this question) as well as more details for each answer alternative (only listing true/false here). It is also possible to have no solution, or just a general text, or just the answer list.
Solution ======== There is no de jure capital but the de facto capital and seat of the federal authorities is Bern. Answerlist  * False * True * False * False * False * False * False
\begin{solution} There is no de jure capital but the de facto capital and seat of the federal authorities is Bern. \begin{answerlist} \item False. \item True. \item False. \item False. \item False. \item False. \item False. \end{answerlist} \end{solution}
In the metainformation, each exercise must specify at least an extype
,
and an exsolution
, and should typically also have a short descriptive
exname
. The type for singlechoice questions is schoice
and the
exsolution
is simply a binary coding of the 7 answer alternatives.
For randomization, additionally exshuffle
is set to 5 so that 1 correct
and 4 random wrong alternatives are subsampled and shuffled.
Metainformation ================ exname: Swiss Capital extype: schoice exsolution: 0100000 exshuffle: 5
\exname{Swiss Capital} \extype{schoice} \exsolution{0100000} \exshuffle{5}
Arithmetic question for computing the first derivative of a product function with two factors, using the product rule. The parameters of the function are drawn randomly in R and subsequently inserted into the question and the solution (which is not shown below for brevity). LaTeX is used for typesetting the mathematical formulas  both in the Markdown and the LaTeX version.
To see what the output looks like, run the following code within R or take a look at deriv in the exercise gallery.
library("exams") exams2html("deriv.Rmd") exams2pdf("deriv.Rmd")
library("exams") exams2html("deriv.Rnw") exams2pdf("deriv.Rnw")
The R code simply draws the two parameters a
and b
as well as the argument c
from a discrete uniform distribution. (See the question below for the actual function used.)
Subsequently, the random numbers are inserted into the correct solution and storing
it as res
.
```{r, echo=FALSE, results="hide"} ## parameters a < sample(2:9, 1) b < sample(seq(2, 4, 0.1), 1) c < sample(seq(0.5, 0.8, 0.01), 1) ## solution res < exp(b * c) * (a * c^(a1) + b * c^a) ```
<<echo=FALSE, results=hide>>= ## parameters a < sample(2:9, 1) b < sample(seq(2, 4, 0.1), 1) c < sample(seq(0.5, 0.8, 0.01), 1) ## solution res < exp(b * c) * (a * c^(a1) + b * c^a) @
The question asks for the derivative of a particular function and simply inserts the parameters drawn above into the question template. (The solution is set up analogously in the exercise template but not shown here for brevity.)
Question ======== What is the derivative of $f(x) = x^{`r a`} e^{`r b` x}$, evaluated at $x = `r c`$?
\begin{question} What is the derivative of $f(x) = x^{\Sexpr{a}} e^{\Sexpr{b}x}$, evaluated at $x = \Sexpr{c}$? \end{question}
Again, the metainformation lists extype
(which is num
for numeric exercises),
exsolution
(which is inserted dynamically from the variable res
computed above),
and a short exname
. Additionally, the tolerance extol
which is used for automatically
evaluating answers, e.g., in an online test/exam.
Metainformation ================ extype: num exsolution: `r fmt(res)` exname: derivative exp extol: 0.01
\extype{num} \exsolution{\Sexpr{fmt(res)}} \exname{derivative exp} \extol{0.01}
The R/exams system is an extension for the R system for statistical computing and hence the first installation step is the base R system.
Linux: While it is possible to download from CRAN by hand, it is easier for most distributions to install the packaged binary. For example, on Debian/Ubuntu:
aptget install rbasecore rbasedev
There is a wide variety of interfaces for using R including simply the shell, Emacs, or dedicated graphical user interfaces for Windows and OS X, respectively. Moreover, RStudio is an opensource crossplatform integrated development environment that facilitates many common tasks for R beginners.
The core of R/exams is the opensource R package “exams”, also available from CRAN. It can be easily installed interactively from within R with a single command. If necessary, the development version of the package is also available, which may provide some new features or small improvements.
Stable version:
install.packages("exams", dependencies = TRUE)
Development version:
install.packages("exams", repos = "http://RForge.Rproject.org")
Details: Several additional R packages, automatically installed by the command above,
are needed for certain tasks: base64enc
(HTMLbased output: Base64 encoding of
supplements), knitr
(R/Markdownbased exercises), png
(NOPS exams: reading
scanned PNG images), RCurl
(ARSnova: posting exercises), RJSONIO
(ARSnova:
JSON format), rmarkdown
(pandocbased conversion), tth
(HTML output from
R/LaTeX exercises).
For producing PDF output, the typesetting system LaTeX is used internally by R/exams.
For certain conversions performed internally in R/exams, specifically when Markdown is involved, the universal document converter pandoc is employed. If you have installed RStudio, then pandoc is provided along with it and nothing else needs to be done.
Otherwise pandoc can be obtained from its web page (linked above) or standard repositories, e.g., for Debian/Ubuntu:
aptget install pandoc
Note: Unless you want to process written NOPS exams from scanned PDF files, this section can be skipped.
If the scanned images of written NOPS exams (from your photocopier) are in PDF format,
they need to be converted to PNG first using the PDF Toolkit pdftk
and
ImageMagick’s convert
.
Linux: Install PDFTk and ImageMagick from your distribution, e.g., for Debian/Ubuntu:
aptget install pdftk imagemagick
To check that the software from Steps 14 works, try to run some examples from the exercise template gallery, e.g., dist or ttest.
]]>ttest
t.test()
in R need to be interpreted regarding significance, type of alternative, and comparison of the underlying empirical means.(Note that the HTML output contains mathematical equations in MathML. It is displayed by browsers with MathML support like Firefox or Safari  but not Chrome.)
Demo code:
library("exams")
set.seed(1090)
exams2html("ttest.Rnw")
set.seed(1090)
exams2pdf("ttest.Rnw")
set.seed(1090)
exams2html("ttest.Rmd")
set.seed(1090)
exams2pdf("ttest.Rmd")
]]>tstat2
(Note that the HTML output contains mathematical equations in MathML. It is displayed by browsers with MathML support like Firefox or Safari  but not Chrome.)
Demo code:
library("exams")
set.seed(1090)
exams2html("tstat2.Rnw")
set.seed(1090)
exams2pdf("tstat2.Rnw")
set.seed(1090)
exams2html("tstat2.Rmd")
set.seed(1090)
exams2pdf("tstat2.Rmd")
]]>tstat
(Note that the HTML output contains mathematical equations in MathML. It is displayed by browsers with MathML support like Firefox or Safari  but not Chrome.)
Demo code:
library("exams")
set.seed(1090)
exams2html("tstat.Rnw")
set.seed(1090)
exams2pdf("tstat.Rnw")
set.seed(1090)
exams2html("tstat.Rmd")
set.seed(1090)
exams2pdf("tstat.Rmd")
]]>