Guest post by Nathaniel P. Graham (Texas A&M International University, Division of International Banking and Finance Studies).

While R/exams was originally written for large statistics classes, there is nothing specific to either large courses or statistics. In this article, I will describe how I use R/exams for my “Introduction to Finance (FIN 3310)” course. While occasionally I might have a class of 60 students, I generally have smaller sections of 20–25. These courses are taught and exams are given in-person (some material and homework assignments are delivered via the online learning management system, but I do not currently use R/exams to generate that).

My written exams are generated by `exams2nops()`

and have two components: a multiple-choice portion and a short-answer portion. The former are generated using R/exams’s dynamic exercise format while the latter are included from static PDF documents. Example pages from a typical exam are linked below:

Because I have a moderate number of students, I grade all my exams manually. This blog post outlines how the exercises are set up and which R scripts I use to automate the generation of the PDF exams.

The actual questions in my exams are not different from many other R/exams examples, except that they are about finance instead of statistics. An example is included below with LaTeX markup and both the R/LaTeX (.Rnw) and R/Markdown (.Rmd) versions can be downloaded: goalfinance.Rnw, goalfinance.Rmd. I have 7 possible answers, but only 5 will be randomly chosen for a given exam (always including the 1 correct answer). To do so, many of my non-numerical questions take advantage of the `exshuffle`

option here. (If you are one of my students, I *told* you this would be on the exam!)

Periodic updates to individual questions (stored as separate files) keep the exams fresh.

```
\begin{question}
The primary goal of financial management is to maximize the \dots
\begin{answerlist}
\item Present value of future operating cash flows
\item Net income, according to GAAP/IFRS
\item Market share
\item Value of the firm
\item Number of shares outstanding
\item Book value of shareholder equity
\item Firm revenue
\end{answerlist}
\end{question}
\exname{Goal of Financial Management}
\extype{schoice}
\exsolution{0001000}
\exshuffle{5}
```

Since this is for a finance class, there are numerical exercises as well. Every semester, I promise my students that this problem *will* be on the second midterm. The problem is simple: given some cash flows and a discount rate, calculate the net present value (NPV), see npvcalc.Rnw, npvcalc.Rmd. Since the cash flows, the discount rate, and the answers are generated from random numbers, I would not gain much from defining more than five possible, the way I did in the qualitative question above. As you might expect, some of the incorrect answers are common mistakes students make when approaching NPV, so I have not lost anything relative to the carefully crafted questions and answers I used in the past to find out who really learned the material.

```
<<echo=FALSE, results=hide>>=
discountrate <- round(runif(1, min = 6.0, max = 15.0), 2)
r <- discountrate / 100.0
cf0 <- sample(10:20, 1) * -100
ocf <- sample(seq(200, 500, 25), 5)
discounts <- sapply(1:5, function(i) (1 + r) ** i)
npv <- round(sum(ocf / discounts) + cf0, 2)
notvm <- round(sum(ocf) + cf0, 2)
wrongtvm <- round(sum(ocf / (1.0 + r)) + cf0, 2)
revtvm <- round(sum(ocf * (1.0 + r)) + cf0, 2)
offnpv <- round(npv + sample(c(-200.0, 200.0), 1), 2)
@
\begin{question}
Assuming the discount rate is \Sexpr{discountrate}\%, find the
net present value of a project with the following cash flows, starting
at time 0: \$\Sexpr{cf0}, \Sexpr{ocf[1]}, \Sexpr{ocf[2]}, \Sexpr{ocf[3]},
\Sexpr{ocf[4]}, \Sexpr{ocf[5]}.
\begin{answerlist}
\item \$\Sexpr{wrongtvm}
\item \$\Sexpr{notvm}
\item \$\Sexpr{npv}
\item \$\Sexpr{revtvm}
\item \$\Sexpr{offnpv}
\end{answerlist}
\end{question}
\exname{Calculating NPV}
\extype{schoice}
\exsolution{00100}
\exshuffle{5}
```

While there are other methods of generating randomized exams out there, few are as flexible as R/exams, in large part because we have full access to R. The next example also appears on my second midterm; it asks students to compute the internal rate of return for a set of cash flows, see irrcalc.Rnw, irrcalc.Rmd. Numerically, this is a simple root-finding problem, but systems that do not support more advanced mathematical operations (such as Blackboard’s “Calculated Formula” questions) can make this difficult or impossible to implement directly.

```
<<echo=FALSE, results=hide>>=
cf0 <- sample(10:16, 1) * -100
ocf <- sample(seq(225, 550, 25), 5)
npvfunc <- function(r) {
discounts <- sapply(1:5, function(i) (1 + r) ** i)
npv <- (sum(ocf / discounts) + cf0) ** 2
return(npv)
}
res <- optimize(npvfunc, interval = c(-1,1))
irr <- round(res$minimum, 4) * 100.0
wrong1 <- irr + sample(c(1.0, -1.0), 1)
wrong2 <- irr + sample(c(0.25, -0.25), 1)
wrong3 <- irr + sample(c(0.5, -0.5), 1)
wrong4 <- irr + sample(c(0.75, -0.75), 1)
@
\begin{question}
Find the internal rate of return of a project with the following cash flows,
starting at time 0: \$\Sexpr{cf0}, \Sexpr{ocf[1]}, \Sexpr{ocf[2]},
\Sexpr{ocf[3]}, \Sexpr{ocf[4]}, \Sexpr{ocf[5]}.
\begin{answerlist}
\item \Sexpr{wrong1}\%
\item \Sexpr{wrong2}\%
\item \Sexpr{irr}\%
\item \Sexpr{wrong3}\%
\item \Sexpr{wrong4}\%
\end{answerlist}
\end{question}
\exname{Calculating IRR}
\extype{schoice}
\exsolution{00100}
\exshuffle{5}
```

While `exams2nops()`

has some support for open-ended questions that are scored manually, it is very limited. I create and format those questions in the traditional manner (Word or LaTeX) and save the resulting file as a PDF. Alternatively, a custom `exams2pdf()`

could be used for this. Finally, I add this PDF file on to the end of the exam using the `pages`

option of `exams2nops()`

(as illustrated in more detail below). As an example, the short-answer questions from the final exam above are given in the following PDF file:

To facilitate automatic inclusion of these short-answer questions, a naming convention is adopted in the scripts below. The script looks for a file named `examNsa.pdf`

, where `N`

is the exam number (1 for the first midterm exam, 2 for the second, and 3 for the final exam) and sets `pages`

to it. So for the final exam, it looks for the file `exam3sa.pdf`

. If I want to update my short-answer questions, I just replace the old PDF with a new one.

I keep the questions for each exam in their own directory, which makes the script below that I use to generate exams 1 and 2 straightforward. For the moment, the script uses all the questions in an exam’s directory, but I if want to guarantee that I have exactly (for example) 20 multiple choice questions, I could set `nsamp = 20`

instead of however many Rnw (or Rmd) files it finds.

Note that `exlist`

is a list with one element (a vector of file names), so that R/exams will randomize the order of the questions as well. If I wanted to specify the order, I could make `exlist`

a list of N elements, where each element was exactly one file/question. The `nsamp`

option could then be left unset (the default is `NULL`

).

When I generate a new set of exams, I only need to update the first four lines, and the script does the rest. Note that I use a modified version of the English `en.dcf`

language configuration file in order to adapt some terms to TAMIU’s terminology, e.g., “ID Number” instead of the `exams2nops()`

default “Registration Number”. See en_tamiu.dcf for the details. Since the TAMIU student ID numbers have 8 digits, I use the `reglength = 8`

argument, which sets the number of digits in the ID Number to 8.

```
library("exams")
## configuration: change these to make a new batch of exams
exid <- 2
exnum <- 19
exdate <- "2019-04-04"
exsemester <- "SP19"
excourse <- "FIN 3310"
## options derived from configuration above
exnam <- paste0("fin3310exam", exid)
extitle <- paste(excourse, exsemester, "Exam", exid, sep = " ")
saquestion <- paste0("SA_questions/exam", exid, "sa.pdf")
## exercise directory (edir) and output directory (odir)
exedir <- paste0("fin3310_exam", exid, "_exercises")
exodir <- "nops"
if(!dir.exists(exodir)) dir.create(exodir) ## in case it was previously deleted
exodir <- paste0(exodir, "/exam", exid, "_", format(Sys.Date(), "%Y-%m-%d"))
## exercise list: one element with a vector of all available file names
exlist <- list(dir(path = exedir, pattern = "\.Rnw$"))
## generate exam
set.seed(54321) # not the actual random seed
exams2nops(file = exlist,
n = exnum, nsamp = length(exlist[[1]]), dir = exodir, edir = exedir,
name = exnam, date = exdate, course = excourse, title = extitle,
institution = "Texas A\\&M International University",
language = "en_tamiu.dcf", encoding = "UTF-8",
pages = saquestion, blank = 1, logo = NULL, reglength = 8,
samepage = TRUE)
```

My final exam is comprehensive, so I would like to include questions from the previous exams. I do not want to keep separate copies of those questions just for the final, in case I update one version and forget to update the other, so I need a script that gathers up questions from the first two midterms and adds in some questions specific to the final.

The script below uses a feature that has long been available in R/exams but was undocumented up to version 2.3-2: You can set `edir`

to a directory and all its subdirectories will be included in the search path. I have specified some required questions that I want to appear on every student’s exam; each student will also get a random draw of other questions from the first two midterms in addition to some questions that only appear on the final. How many of each is controlled by `exnsamp`

, which is passed to the `nsamp`

argument. They add up to 40 currently, so `exams2nops()`

’s 45 question limit does not affect me.

```
library("exams")
## configuration: change these to make a new batch of exams
exnum <- 19
exdate <- "2019-04-30"
exsemester <- "SP19"
excourse <- "FIN 3310"
## options derived from configuration above
extitle <- paste(excourse, exsemester, "Exam", exid, sep = " ")
## exercise directory (edir) and output directory (odir)
exedir <- getwd()
exodir <- "nops"
if(!dir.exists(exodir)) dir.create(exodir)
exodir <- paste0(exodir, "/exam3", "_", format(Sys.Date(), "%Y-%m-%d"))
## exercises: required and from previous midterms
exrequired <- c("goalfinance.Rnw", "pvcalc.Rnw", "cashcoverageortie.Rnw",
"ocfcalc.Rnw", "discountratecalc.Rnw", "npvcalc.Rnw", "npvcalc2.Rnw",
"stockpriceisabouttopaycalc.Rnw", "annuitycalc.Rnw", "irrcalc.Rnw")
exlist1 <- dir(path = "fin3310_exam1_exercises", pattern = "\.Rnw$")
exlist2 <- dir(path = "fin3310_exam2_exercises", pattern = "\.Rnw$")
exlist3 <- dir(path = "fin3310_exam3_exercises", pattern = "\.Rnw$")
## final list and corresponding number to be sampled
exlist <- list(
exrequired,
setdiff(exlist1, exrequired),
setdiff(exlist2, exrequired),
exlist3
)
exnsamp <- c(10, 5, 10, 15)
## generate exam
set.seed(54321) # not the actual random seed
exams2nops(file = exlist,
n = exnum, nsamp = exnsamp, dir = exodir, edir = exedir,
name = "fin3310exam3", date = exdate, course = excourse, title = extitle,
institution = "Texas A\\&M International University",
language = "en_tamiu.dcf", encoding = "UTF-8",
pages = "SA_questions/exam3sa.pdf", blank = 1, logo = NULL,
reglength = 8, samepage = TRUE)
```

R/exams can read scans of the answer sheet, automating grading for the multiple-choice questions. For a variety of reasons, I do not use any of those features. If I had to teach larger classes I would doubtless find a way to make it convenient, but for the foreseeable future I will continue to use the function below to produce the answer key for each exam, which I grade by hand. This is not especially onorous, since I have to grade the short-answer questions by hand anyway.

Given the serialized R data file (.rds) produced by `exam2nops()`

the corresponding `exams_metainfo()`

can be extracted. This comes with a `print()`

method that displays all correct answers for a given exam. Starting from R/exams version 2.3-3 (current R-Forge devel version) it is also possible to split the output into blocks of five for easy reading (matching the blocks on the answer sheets). As an example the first few correct answers of the third exam are extracted:

```
fin3310exam3 <- readRDS("fin3310exam3.rds")
print(exams_metainfo(fin3310exam1), which = 3, block = 5)
```

```
## 19043000003
## 1. Discount Rate: 1
## 2. Calculating IRR: 4
## 3. Present Value: 3
## 4. Calculating stock prices: 5
## 5. Calculating NPV: 3
##
## 6. Annuity PV: 3
## 7. Goal of Financial Management: 1
## 8. Finding OCF: 5
## ...
```

It can be convenient to display the correct answers in a customized format, e.g., with letters instead of numbers and omitting the exercise title text. To do so, the code below sets up a function `exam_solutions()`

, applying it to the same exam as above.

```
exam_solutions <- function(examdata, examnum) {
solutions <- LETTERS[sapply(examdata[[examnum]],
function(x) which(x$metainfo$solution))]
data.frame(Solution = solutions)
}
split(exam_solutions(fin3310exam3, examnum = 3), rep(1:8, each = 5))
```

```
## $`1`
## Solution
## 1 A
## 2 D
## 3 C
## 4 E
## 5 C
##
## $`2`
## Solution
## 6 C
## 7 A
## 8 E
## ...
```

The only downside of the manual grading approach is that I do not have students’ responses in an electronic format for easy statistical analysis, but otherwise grading is very fast.

Using the system above, R/exams works well even for small courses with in-person, paper exams. I do not need to worry about copies of my exams getting out and students memorizing it, or any of the many ways students have found to cheat over the years. By doing all the formatting work for me, R/exams helps me avoid a lot of the finicky aspects of adding, adjusting, or removing questions from an existing exam, and generally keeps exam construction focused on the important part: writing questions that gauge students’ progress.

]]>A popular use case of the R/exams package is the generation of dynamic exercises for online learning management systems in large-scale courses in mathematics, statistics, or physics. Often, these contain some mathematical notation using LaTeX markup. While LaTeX can be easily rendered into PDF for printing written exams, the options for rendering it into HTML for online exams are somewhat more diverse and the best strategies depend on a number of factors. As these involve a number of technical details this tutorial aims to give a brief overview and makes a few practical recommendations. Factors include the following:

*Original markup in exercise source files:*

The R/exams exercises may employ either Markdown or LaTeX for formatting and structuring the text (see the First steps tutorial for more details). In either case LaTeX is used for the mathematical notation. Thus, also in Markdown the mathematical content is embedded as LaTeX code.*Math markup in HTML:*

For display of the formatted text online in a browser, the original markup (Markdown/LaTeX) needs to be converted (see below) to HTML with some suitable formatting for the mathematical content. Either the LaTeX can be preserved or turned into MathML, a dedicated XML format for describing mathematical content.*Rendering of math markup:*

The most widely-used way of rendering mathematical content in web pages is MathJax (see also the corresponding Wikipedia page), an open-source JavaScript library that works in all browsers without any setup by the user (except enabling JavaScript). It can render LaTeX directly but also MathML. Moreover, some browsers (mainly Firefox and Safari but*not*Chrome) have also native support for displaying MathML, i.e, without the need for an additional display engine such as MathJax. The advantage of employing the browser for MathML rendering is that it is faster and typically blends better into the regular text. The disadvantage is obviously that it is not supported by all browsers and does not support rendering of LaTeX directly.*Conversion to HTML:*

The original markup from the exercises templates needs to be converted to HTML and the most powerful document converter for this is Pandoc (see also the corresponding Wikipedia page). It can convert both Markdown and LaTeX source files to HTML, either with LaTeX embedded for MathJax or with MathML (among various other options). Moreover, LaTeX (but not Markdown) exercises can be converted to HTML using the TeX-to-MathML converter TtM. (R/exams also provides a few further options which are typically of less interest due to lower quality, such as TtH or rendered images.)*Defaults in R/exams:*

All`exams2xyz()`

functions that produce HTML-based output offer a`converter =`

argument. By default, this is configured to produce HTML with embedded MathML because this can be rendered both by MathJax in all browsers as well as by some browsers directly. The default is is`converter = "pandoc"`

(equivalent to`"pandoc-mathml"`

) for R/Markdown exercises and`converter = "ttm"`

for R/LaTeX exercises, respectively. Whether MathJax is enabled varies across output formats or might depend on the learning management system.

As a simple illustration of the strengths and weaknesses of the different approaches, the deriv exercise template (computation of a derivative using the product rule) is converted to HTML using `exams2html()`

. Here, the R/LaTeX version of the exercise is used (`"deriv"`

or equivelently `"deriv.Rnw"`

) but using the R/Markdown version (`"deriv.Rmd"`

) yields almost the same output.

The following examples and resulting screenshots contrast the output between Firefox and Chrome. By clicking on the screenshots you can also see what the HTML pages look like in your own browser.

By default, `exams2html()`

generates HTML with MathML embedded and uses a template that does *not* enable MathJax.

```
library("exams")
set.seed(1090)
exams2html("deriv")
```

The screenshots that the native MathML support in Firefox leads to output that renders fast and smoothly. However, the lack of native MathML support in Chrome means that the exercise cannot be displayed correctly.

To easily explore the effect of MathJax rendering `exams2html()`

supports the argument `mathjax = TRUE`

which inserts the MathJax `<script>`

into the template (loaded from RStudio’s content delivery network).

```
set.seed(1090)
exams2html("deriv", mathjax = TRUE)
```

Now the math output looks good in both browsers. However, for Firefox users the version without MathJax might still be preferable as it renders faster with somewhat smoother output.

To preserve the LaTeX equations, the argument `converter = "pandoc-mathjax"`

can be used. Then, Pandoc converts the LaTeX text to HTML but preserves the LaTeX equations (in minimal HTML markup).

```
set.seed(1090)
exams2html("deriv", converter = "pandoc-mathjax", mathjax = TRUE)
```

The output is very similar to the MathML rendered MathJax above. However, note that the alignment in the equations is changed (from left to right). This is caused by Pandoc replacing the LaTeX `{eqnarray*}`

environment employed in the `"deriv"`

exercise by `{aligned}`

environments.

*Original markup in exercise source files:*

Whether to use R/Markdown (`.Rmd`

) or R/LaTeX (`.Rnw`

) markup in the exercise source files is mostly a matter of taste. The former is probably somewhat easier to learn for beginners but generally differences are small if there is only moderate text formatting. It is also good practice to keep the formatting simple to be robust across the different output formats and more advanced math constructs should be checked carefully.*MathML as default math markup:*

As of today (January 2019) it might seem more natural to use LaTeX rendered by MathJax as`rmarkdown`

does. However, when HTML conversion was added to R/exams in early 2012, MathML rendered by Firefox was the more robust choice. MathML has been preserved as the default since then because it can be rendered both by MathJax and some browsers.*Pandoc vs. TtM:*

Pandoc is the default converter for all`.Rmd`

exercises while TtM is still used by default for`.Rnw`

exercises. The latter is mostly for backward compatibility but might change to Pandoc in the future. However, differences are not very large for most exercises anyway but some mathematical LaTeX constructs are just supported by one and not the other converter.*Enabling MathJax rendering:*

Many modern learning management systems have MathJax enabled by default, e.g., in Moodle, Canvas, or OpenOLAT. There are a couple of caveats, though: First, the default MathJax configuration in Moodle and OpenOLAT switches off rendering of MathML. Second, some systems do not host their own copy of the MathJax library but employ a content delivery network (CDN). Thus, there is a small risk that the learning management system might be up and running but there are problems loading MathJax from the CDN.*Moodle:*

As noted above the default configuration for Moodle has MathJax support (via a CDN) but switches off rendering of MathML. As`exams2moodle(...)`

currently uses MathML markup by default, this necessitates Firefox or Safari for viewing the quizzes in Moodle. In contrast,`exams2moodle(..., converter = "pandoc-mathjax")`

would use LaTeX math markup and render it by MathJax (unless the Moodle configuration switched off MathJax support).

**Request:**Feedback from Moodle users would be appreciated on whether they prefer the current default or`converter = "pandoc-mathjax"`

. It would be especially useful to find out whether the latter works in their Moodle installations.*Canvas:*

We are currently working on a dedicated`exams2qti12()`

-based function`exams2canvas()`

for generating quizzes for Canvas. This will keep the MathML-based default for the math notation as this is rendered smoothly by Canvas’ own MathJax support.*OpenOLAT:*

A new dedicated interface`exams2openolat()`

for generating quizzes for OpenOLAT has been added recently to R/exams (version 2.3-1). By default, this is a wrapper to`exams2qti21(..., converter = "pandoc-mathjax")`

because OpenOLAT provides MathJax rendering of LaTeX math (but not MathML). Some additional tweaks are necessary, though, because OpenOLAT expects the LaTeX to be embedded slightly differently from standard Pandoc output.

The mathematical equation in the random draw of the deriv exercise in LaTeX is: `f(x) = x^{8} e^{3.4x}`

. Here, we highlight that all converters yield almost equivalent output when rendered by MathJax:

`"pandoc-mathjax"` |
`"ttm"` |
`"pandoc-mathml"` |
---|---|---|

\(f(x) = x^{8} e^{3.4x}\) | $f(x)={x}^{8}{e}^{3.4x}$ | $f(x) = x^{8} e^{3.4x}$ |

(**Note:** If you are viewing this on R-bloggers or another aggregator some or all of the equations will not display correctly. Refer to the R/exams site for a version with MathJax properly enabled.)

The underlying LaTeX code generated by `converter = "pandoc-mathjax"`

is simply the original LaTeX code with some minimal HTML markup:

```
<span class="math inline">\(f(x) = x^{8} e^{3.4x}\)</span>
```

The MathML code generated by `converter = "ttm"`

differs slightly from the of `converter = "pandoc"`

(or equivalently `"pandoc-mathml"`

). The former yields:

```
<math xmlns="http://www.w3.org/1998/Math/MathML">
<mrow><mi>f</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo>=</mo>
<msup><mrow><mi>x</mi></mrow><mrow><mn>8</mn></mrow>
</msup>
<msup><mrow><mi>e</mi></mrow><mrow><mn>3</mn><mo>.</mo><mn>4</mn><mi>x</mi></mrow>
</msup>
</mrow></math>
```

The Pandoc version is very similar but contains some more markup and annotation:

```
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics>
<mrow><mi>f</mi>
<mo stretchy="false" form="prefix">(</mo><mi>x</mi><mo stretchy="false" form="postfix">)</mo>
<mo>=</mo>
<msup><mi>x</mi><mn>8</mn></msup><msup><mi>e</mi><mrow><mn>3.4</mn><mi>x</mi></mrow></msup>
</mrow><annotation encoding="application/x-tex">f(x) = x^{8} e^{3.4x}</annotation>
</semantics></math>
```

]]>Version 2.3-2 of the one-for-all exams generator R/exams has been published on the Comprehensive R Archive Network at https://CRAN.R-project.org/package=exams. In the next days this will propagate to other CRAN mirrors along with Windows and OS X binary packages. The development version of the package is now version 2.3-3 on R-Forge at http://R-Forge.R-project.org/forum/?group_id=1337.

- Support for further languages is available in
`exams2nops()`

and`nops_eval()`

: Russian (`ru`

, contributed by Boris Demeshev) and Serbian (`sr`

, contributed by Tatjana Kecojevic). Furthermore, Croatian (`hr`

) was streamlined along with Serbian by Tatjana Kecojevic. - In
`exams2nops()`

the`header`

argument can also be specified simply as`header = "\\mycommand{value}"`

rather than`header = list(mycommand = "value")`

. The former is more flexible, e.g., for passing additional options or more complex commands. Internally, the former is turned into an unnamed list which is then processed correspondingly by`exams2pdf()`

. - Scanning of written NOPS exams is enhanced and registration IDs are processed more reliably: First,
`nops_scan()`

gained a new argument`trim = 0.3`

that controls how much of the check boxes is trimmed in order to shave the borders prior to determining the average gray level. In versions up to 2.3-1 this implicitly had a value of`0.25`

hard-coded. Now the default increased to`0.3`

in order to shave box borders more reliably, e.g., in more pixelated scans. Second,`nops_scan()`

tries to process registration numbers more reliably. In case one of the registration columns contains more than one potential mark, the heuristics of determining the intended mark have been improved. `nops_eval()`

gained a new argument`labels = NULL`

that can be used to give labels for the marks that differ from the default`(length(mark) + 1):1`

.

- The new exercise confint3 (Rmd/Rnw) illustrates how to use the
`verbatim`

type for advanced processing in Moodle when asking for the computation of a 2-sided confidence interval. Rather than using a single correct value for each numeric result (along with a corresponding tolerance interval) as in`num`

exercises (see confint2), the exercise also provides a second partially correct value for each result. More precisely, the exercise yields 100% of the points for the correct solution based on t quantiles but still 50% for the solution based on normal quantiles. This is possible by setting the type to`verbatim`

and specifying the solution via the necessary Moodle XML markup directly, e.g.,`:NUMERICAL:=<value1>:<tolerance1>~%50%<value2>:<tolerance2>#<comment>`

. The exercise is a contribution by Ulrike Groemping.

- Rather than fully importing the basic dependencies
`stats`

,`graphics`

,`grDevices`

,`tools`

, and`utils`

, only the required functions from those packages are imported selectively. The main motivation for this was that otherwise the code evaluated in R/exams exercises might also use the same NAMESPACE semantics as the`exams`

package - instead of employing the setup in the user’s environment. The issue was raised in this StackOverflow thread by Florian Oswald. - In
`include_supplement()`

an argument`target = NULL`

is added to optionally include the supplement with a different file name than the original file name. - In all LaTeX templates that use Helvetica (phv) as the font for the main text, this is also used now in math mode by
`\usepackage[helvet]{sfmath}`

(rather than`\usepackage{sfmath}`

, as employed previously). In particular, this affects`exams2nops()`

and`tex2image()`

output.

- In previous versions the
`pandoc`

-based HTML`converter`

erroneously produced unbalanced`<p>`

tags in certain situations. - The QTI 2.1 XML output from
`exams2qti21()`

is somewhat cleaner, fixing some computations for correct/incorrect answers in`schoice`

/`mchoice`

exercises.

`confint3`

Computing the 2-sided confidence interval at 95% level for the mean based on a random sample. The exercise is a cloze with two numeric answers for the lower and upper bound of the confidence interval, respectively. Using the 'verbatim' clozetype for Moodle, the exercises yields 100% of the points for the correct solution based on t quantiles but still 50% for a partially correct solution based on normal quantiles.

Yes

Random numbers

Yes

No

No

No

*(Note that the HTML output contains mathematical equations in MathML. It is displayed by browsers with MathML support like Firefox or Safari - but not Chrome.)*

**Demo code:**

```
library("exams")
set.seed(1090)
exams2html("confint3.Rnw")
set.seed(1090)
exams2pdf("confint3.Rnw")
set.seed(1090)
exams2html("confint3.Rmd")
set.seed(1090)
exams2pdf("confint3.Rmd")
```

]]>R/exams was presented in a keynote lecture by Achim Zeileis at eRum 2018, the European R Users Meeting, this time organized by a team around Gergely Daróczi in Budapest. It was a great event with many exciting presentations, reflecting the vibrant R community in Europe (and beyond).

This blog post provides various resources accompanying the presentation which may be of interest to those who did not attend the meeting as well as those who did and who want to explore the materials in more detail.

Most importantly the presentation slides are available in PDF format (under CC-BY):

The eRum organizers did a great job in making the meeting accessible to those useRs who could not make it to Budapest. All presentations were available in a livestream on YouTube where also videos of all lectures were made available after the meeting (Standard YouTube License):

To illustrate the e-learning capabilities supported by R/exams, the presentation started with a live quiz using the audience response system ARSnova. The original version of the quiz was hosted on the ARSnova installation at Universität Innsbruck. To encourage readers to try out ARSnova for their own purposes, a copy of the quiz was also posted on the official ARSnova server at Technische Hochschule Mittelhessen (where ARSnova is developed under the General Public License, GPL):

The presentation briefly also showed an online test generated by R/exams and imported into OpenOLAT, an open-source learning management system (available under the Apache License). The online test is made available again here for anonymous guest access. *(Note however, that the system only has one guest user so that when you start the test there may already be some test results from a previous guest session. In that case you can finish the test and also start it again.)*

The presentation slides show how to set up an exam using the R package and then rendering it into different output formats. In order to allow the same exam to be rendered into a wide range of different output formats, only single-choice and multiple-choice exercises were employed (see the `choice`

list below). However, in the e-learning test shown in OpenOLAT all exercises types are supported (see the `elearn`

list below). All these exercises are readily provided in the package and also introduced online: deriv/deriv2, fruit/fruit2, ttest, boxplots, cholesky, lm, function. The code below uses the R/LaTeX (.Rnw) version but the R/Markdown version (.Rmd) could also be used instead.

```
## package
library("exams")
## single-choice and multiple-choice only
choice <- list("deriv2.Rnw", "fruit2.Rnw", c("ttest.Rnw", "boxplots.Rnw"))
## e-learning test (all exercise types)
elearn <- c("deriv.Rnw", "fruit.Rnw", "ttest.Rnw", "boxplots.Rnw",
"cholesky.Rnw", "lm.Rnw", "function.Rnw")
```

First, the exam with the choice-based questions can be easily turned into a PDF exam in NOPS format using `exams2nops`

, here using Hungarian language for illustration. Exams in this format can be easily scanned and evaluated within R.

```
set.seed(2018-05-16)
exams2nops(choice, institution = "eRum 2018", language = "hu")
```

Second, the choice-based exam version can be exported into the JSON format for ARSnova: Rexams-1.json. This contains an entire ARSnova session that can be directly imported into the ARSnova system as shown above. It employs a custom exercise set up just for eRum (conferences.Rmd) as well as a slightly tweaked exercise (fruit3.Rmd) that displays better in ARSnova.

```
set.seed(2018-05-16)
exams2arsnova(list("conferences.Rmd", choice[[1]], "fruit3.Rmd", choice[[3]]),
name = "R/exams", abstention = FALSE, fix_choice = TRUE)
```

Third, the e-learning exam can be generated in QTI 1.2 format for OpenOLAT, as shown above: eRum-2018.zip. The `exams2openolat`

command below is provided starting from the current R/exams version 2.3-1. It essentially just calls `exams2qti12`

but slightly tweaks the MathJax output from pandoc so that it is displayed properly by OpenOLAT.

```
set.seed(2018-05-16)
exams2openolat(elearn, name = "eRum-2018", n = 10, qti = "1.2")
```

In the last part of the presentation a couple of new and ongoing efforts within the R/exams project are highlighted. First, the natural language support in NOPS exams is mentioned which was recently described in more detail in this blog. Second, the relatively new “stress tester” was illustrated with the following example. (A more detailed blog post will follow soon.)

```
s <- stresstest_exercise("deriv2.Rnw")
plot(s)
```

Finally, a psychometric analysis illustrated how to examine exams regarding: Exercise difficulty, student performance, unidimensionality, fairness. The replication code for the results from the slides is included below (omitting some graphical details for simplicity, e.g., labeling or color).

```
## load data and exclude extreme scorers
library("psychotools")
data("MathExam14W", package = "psychotools")
mex <- subset(MathExam14W, nsolved > 0 & nsolved < 13)
## raw data
plot(mex$solved)
## Rasch model parameters
mr <- raschmodel(mex$solved)
plot(mr, type = "profile")
## points per student
MathExam14W <- transform(MathExam14W,
points = 2 * nsolved - 0.5 * rowSums(credits == 1)
)
hist(MathExam14W$points, breaks = -4:13 * 2 + 0.5, col = "lightgray")
abline(v = 12.5, lwd = 2, col = 2)
## person-item map
plot(mr, type = "piplot")
## principal component analysis
pr <- prcomp(mex$solved, scale = TRUE)
plot(pr)
biplot(pr, col = c("transparent", "black"),
xlim = c(-0.065, 0.005), ylim = c(-0.04, 0.065))
## differential item functioning
mr1 <- raschmodel(subset(mex, group == 1)$solved)
mr2 <- raschmodel(subset(mex, group == 2)$solved)
ma <- anchortest(mr1, mr2, adjust = "single-step")
## anchored item difficulties
plot(mr1, parg = list(ref = ma$anchor_items), ref = FALSE, ylim = c(-2, 3), pch = 19)
plot(mr2, parg = list(ref = ma$anchor_items), ref = FALSE, add = TRUE, pch = 19, border = 4)
legend("topleft", paste("Group", 1:2), pch = 19, col = c(1, 4), bty = "n")
## simultaneous Wald test for pairwise differences
plot(ma$final_tests)
```

]]>`fruit2`

A system of three linear equations has to be solved and the solution has to be entered into a fourth equation. However, the system is not defined through a verbal description or mathermatical notation but through images (clip art of tropical fruits). The problem can be interpreted as prices of three fruits (banana, orange, pineapple) and corresponding fruit baskets with different combinations of fruits. Images are stored in Base64 encoding within the exercise files and embedded dynamically into the output. A set of five answer alternatives is generated based on two potential mistakes and two random solutions from a suitable range. PDFs are best generated from the Rnw version, HTML is best generated with pandoc from either the Rmd version (where pandoc is used by default) or the Rnw version (where ttm is used by default, but pandoc can be easily used as well.)

Yes

Random numbers, shuffled graphics

Yes

No

Yes

No

*(Note that the HTML output contains mathematical equations in MathML. It is displayed by browsers with MathML support like Firefox or Safari - but not Chrome.)*

**Demo code:**

```
library("exams")
set.seed(1090)
exams2html("fruit2.Rnw")
set.seed(1090)
exams2pdf("fruit2.Rnw")
set.seed(1090)
exams2html("fruit2.Rmd")
set.seed(1090)
exams2pdf("fruit2.Rmd")
```

]]>`fruit`

A system of three linear equations has to be solved and the solution has to be entered into a fourth equation. However, the system is not defined through a verbal description or mathermatical notation but through images (clip art of tropical fruits). The problem can be interpreted as prices of three fruits (banana, orange, pineapple) and corresponding fruit baskets with different combinations of fruits. Images are stored in Base64 encoding within the exercise files and embedded dynamically into the output. PDFs are best generated from the Rnw version, HTML is best generated with pandoc from either the Rmd version (where pandoc is used by default) or the Rnw version (where ttm is used by default, but pandoc can be easily used as well.)

Yes

Random numbers, shuffled graphics

Yes

No

Yes

No

*(Note that the HTML output contains mathematical equations in MathML. It is displayed by browsers with MathML support like Firefox or Safari - but not Chrome.)*

**Demo code:**

```
library("exams")
set.seed(1090)
exams2html("fruit.Rnw")
set.seed(1090)
exams2pdf("fruit.Rnw")
set.seed(1090)
exams2html("fruit.Rmd")
set.seed(1090)
exams2pdf("fruit.Rmd")
```

]]>Version 2.3-1 of the one-for-all exams generator R/exams has been published on the Comprehensive R Archive Network at https://CRAN.R-project.org/package=exams. In the next days this will propagate to other CRAN mirrors along with Windows binary packages. The development version of the package is now version 2.3-2 on http://R-Forge.R-project.org/forum/?group_id=1337.

- Added new interface
`exams2openolat()`

for the open-source OpenOLAT learning management system. This is only a convenience wrapper to`exams2qti12()`

or`exams2qti21()`

with some dedicated tweaks for optimizing MathJax output for OpenOLAT. - New function
`include_tikz()`

that facilitates compiling standalone TikZ figures into a range of output formats, especially PNG and SVG (for HTML-based output). This is useful when including TikZ in R/Markdown exercises or when converting R/LaTeX exercises to HTML. Two examples have been added to the package that illustrate the capabilities of`include_tikz()`

: automaton, logic. A dedicated blog post is also planned.

- Following the blog post on Written R/exams around the World several users have been kind enough to add language support for: Croatian (hr.dcf, contributed by Krunoslav Juraić), Danish (da.dcf, contributed by Tue Vissing Jensen and Jakob Messner),Slovak (sk.dcf, contributed by Peter Fabsic), Swiss German (gsw.dcf, contributed by Reto Stauffer), Turkish (tr.dcf, contributed by Emrah Er). Furthermore, Portuguese has been distinguished into pt-PT.dcf (Portuguese Portuguese) vs. pt-BR.dcf (Brazilian Portuguese) with pt.dcf defaulting to the former (contributed by Thomas Dellinger).
- After setting a random seed
`exams2nops()`

and`exams2pdf()`

now yield the same random versions of the exercises. Previously, this was not the case because`exams2nops()`

internally generates a single random trial exam first for a couple of sanity checks. Now, the`.GlobalEnv$.Random.seed`

is restored after generating the trial exam. - Fixed the support for
`nsamp`

argument in`exams2nops()`

. Furthermore, current limitations of`exams2nops()`

are pointed out more clearly in error messages and edge cases caught. - Allow varying points within a certain exercise in
`nops_eval()`

.

- In
`exams2html()`

and other interfaces based on`make_exercise_transform_html()`

the option`base64 = TRUE`

now uses Base64 encoding for all file extensions (known to the package) whereas`base64 = NULL`

only encodes image files (previous default behavior). - Bug fixes and improvements in HTML transformers:
- Only
`="file.ext"`

(with`="`

) for supplementary files embedded into HTML is replaced now by the corresponding Base64-encoded version. `href="file.ext"`

is replaced by`href="file.ext" download="file.ext"`

prior to Base 64 replacement to assure that the file name is preserved for the browser/downloader.`alt="file.ext"`

and`download="file.ext"`

are preserved without the Base64-encoded version of`file.ext`

.

- Only
- Include further file URIs for Base64 supplements, in particular .sav for SPSS data files.
- In
`exams2blackboard(..., base64 = FALSE, ...)`

the`base64 = FALSE`

was erroneously ignored. No matter how base64 was specified essentially`base64 = TRUE`

was used, it is honored again now.

`\exshuffle{<num>}`

can now also be used for schoice exercises with more than one`TRUE`

answer. In a first step only one of the`TRUE`

answers is selected and then`<num>-1`

items from the`FALSE`

answers.- Function
`include_supplement(..., dir = "foo")`

- without full path to`"foo"`

- now also works if`"foo"`

is not a local sub-directory but a sub-directory to the exercise directory`edir`

(if specified). - Enable passing of
`envir`

argument from`exams2html()`

to`xweave()`

in case of R/Markdown (.Rmd) exercises. - When using
`exams2html(..., mathjax = TRUE)`

for testing purposes, mathjax.rstudio.com is used now rather than cdn.mathjax.org which is currently redirecting and will eventually be shut down completely. - Added support for
`\tightlist`

(as produced by pandoc) in all current LaTeX templates as well as`exams2nops()`

.

- Fixed a bug in
`stresstest_exercise()`

where the “rank” (previously called “order”) of the correct solution was computed incorrectly. Additional enhancements in plots and labels. - Fixed a bug for
`tex2image(..., tikz = TRUE)`

where erroneously`\usetikzlibrary{TRUE}`

was included. Also`tex2image(..., Sweave = TRUE)`

(the default) did not run properly on Windows, fixed now. - Better warnings if
`\exshuffle{<num>}`

could not be honored due to a lack of sufficiently many (suitable) answer alternatives. - Bug fix in CSV export of
`exams2arsnova()`

. Recent ARSnova versions use “mc” (rather than “MC”) and “abcd” (rather than “SC”) to code multiple-choice and single-choice questions, respectively.

`logic`

Gate diagrams for three logical operators (sampled from: and, or, xor, nand, nor) are drawn with TikZ and have to be matched to a truth table for another randomly drawn logical operator. Depending on the exams2xyz() interface the TikZ graphic can be rendered in PNG, SVG, or directly by LaTeX.

Yes

Shuffling, text blocks, and graphics

No

No

Yes

No

**Demo code:**

```
library("exams")
set.seed(1090)
exams2html("logic.Rnw")
set.seed(1090)
exams2pdf("logic.Rnw")
set.seed(1090)
exams2html("logic.Rmd")
set.seed(1090)
exams2pdf("logic.Rmd")
```

]]>`automaton`

An automaton diagram with four states A-D is drawn with TikZ and is to be interpreted, where A is always the initial state and one state is randomly picked as the accepting state. Five binary 0/1 input sequences acceptance have to be assessed with approximately a quarter of all sequences being accepted. Depending on the exams2xyz() interface the TikZ graphic can be rendered in PNG, SVG, or directly by LaTeX.

Yes

Random numbers, text blocks, and graphics

No

No

Yes

No

**Demo code:**

```
library("exams")
set.seed(1090)
exams2html("automaton.Rnw")
set.seed(1090)
exams2pdf("automaton.Rnw")
set.seed(1090)
exams2html("automaton.Rmd")
set.seed(1090)
exams2pdf("automaton.Rmd")
```

]]>