A popular use case of the R/exams package is the generation of dynamic exercises for online learning management systems in large-scale courses in mathematics, statistics, or physics. Often, these contain some mathematical notation using LaTeX markup. While LaTeX can be easily rendered into PDF for printing written exams, the options for rendering it into HTML for online exams are somewhat more diverse and the best strategies depend on a number of factors. As these involve a number of technical details this tutorial aims to give a brief overview and makes a few practical recommendations. Factors include the following:

*Original markup in exercise source files:*

The R/exams exercises may employ either Markdown or LaTeX for formatting and structuring the text (see the First steps tutorial for more details). In either case LaTeX is used for the mathematical notation. Thus, also in Markdown the mathematical content is embedded as LaTeX code.*Math markup in HTML:*

For display of the formatted text online in a browser, the original markup (Markdown/LaTeX) needs to be converted (see below) to HTML with some suitable formatting for the mathematical content. Either the LaTeX can be preserved or turned into MathML, a dedicated XML format for describing mathematical content.*Rendering of math markup:*

The most widely-used way of rendering mathematical content in web pages is MathJax (see also the corresponding Wikipedia page), an open-source JavaScript library that works in all browsers without any setup by the user (except enabling JavaScript). It can render LaTeX directly but also MathML. Moreover, some browsers (mainly Firefox and Safari but*not*Chrome) have also native support for displaying MathML, i.e, without the need for an additional display engine such as MathJax. The advantage of employing the browser for MathML rendering is that it is faster and typically blends better into the regular text. The disadvantage is obviously that it is not supported by all browsers and does not support rendering of LaTeX directly.*Convertesion to HTML:*

The original markup from the exercises templates needs to be converted to HTML and the most powerful document converter for this is Pandoc (see also the corresponding Wikipedia page). It can convert both Markdown and LaTeX source files to HTML, either with LaTeX embedded for MathJax or with MathML (among various other options). Moreover, LaTeX (but not Markdown) exercises can be converted to HTML using the TeX-to-MathML converter TtM. (R/exams also provides a few further options which are typically of less interest due to lower quality, such as TtH or rendered images.)*Defaults in R/exams:*

All`exams2xyz()`

functions that produce HTML-based output offer a`converter =`

argument. By default, this is configured to produce HTML with embedded MathML because this can be rendered both by MathJax in all browsers as well as by some browsers directly. The default is is`converter = "pandoc"`

(equivalent to`"pandoc-mathml"`

) for R/Markdown exercises and`converter = "ttm"`

for R/LaTeX exercises, respectively. Whether MathJax is enabled varies across output formats or might depend on the learning management system.

As a simple illustration of the strengths and weaknesses of the different approaches, the deriv exercise template (computation of a derivative using the product rule) is converted to HTML using `exams2html()`

. Here, the R/LaTeX version of the exercise is used (`"deriv"`

or equivelently `"deriv.Rnw"`

) but using the R/Markdown version (`"deriv.Rmd"`

) yields almost the same output.

The following examples and resulting screenshots contrast the output between Firefox and Chrome. By clicking on the screenshots you can also see what the HTML pages look like in your own browser.

By default, `exams2html()`

generates HTML with MathML embedded and uses a template that does *not* enable MathJax.

```
library("exams")
set.seed(1090)
exams2html("deriv")
```

The screenshots that the native MathML support in Firefox leads to output that renders fast and smoothly. However, the lack of native MathML support in Chrome means that the exercise cannot be displayed correctly.

To easily explore the effect of MathJax rendering `exams2html()`

supports the argument `mathjax = TRUE`

which inserts the MathJax `<script>`

into the template (loaded from RStudio’s content delivery network).

```
set.seed(1090)
exams2html("deriv", mathjax = TRUE)
```

Now the math output looks good in both browsers. However, for Firefox users the version without MathJax might still be preferable as it renders faster with somewhat smoother output.

To preserve the LaTeX equations, the argument `converter = "pandoc-mathjax"`

can be used. Then, Pandoc converts the LaTeX text to HTML but preserves the LaTeX equations (in minimal HTML markup).

```
set.seed(1090)
exams2html("deriv", converter = "pandoc-mathjax", mathjax = TRUE)
```

The output is very similar to the MathML rendered MathJax above. However, note that the alignment in the equations is changed (from left to right). This is caused by Pandoc replacing the LaTeX `{eqnarray*}`

environment employed in the `"deriv"`

exercise by `{aligned}`

environments.

*Original markup in exercise source files:*

Whether to use R/Markdown (`.Rmd`

) or R/LaTeX (`.Rnw`

) markup in the exercise source files is mostly a matter of taste. The former is probably somewhat easier to learn for beginners but generally differences are small if there is only moderate text formatting. It is also good practice to keep the formatting simple to be robust across the different output formats and more advanced math constructs should be checked carefully.*MathML as default math markup:*

As of today (January 2019) it might seem more natural to use LaTeX rendered by MathJax as`rmarkdown`

does. However, when HTML conversion was added to R/exams in early 2012, MathML rendered by Firefox was the more robust choice. MathML has been preserved as the default since then because it can be rendered both by MathJax and some browsers.*Pandoc vs. TtM:*

Pandoc is the default converter for all`.Rmd`

exercises while TtM is still used by default for`.Rnw`

exercises. The latter is mostly for backward compatibility but might change to Pandoc in the future. However, differences are not very large for most exercises anyway but some mathematical LaTeX constructs are just supported by one and not the other converter.*Enabling MathJax rendering:*

Many modern learning management systems have MathJax enabled by default, e.g., in Moodle, Canvas, or OpenOLAT. There are a couple of caveats, though: First, the default MathJax configuration in Moodle and OpenOLAT switches off rendering of MathML. Second, some systems do not host their own copy of the MathJax library but employ a content delivery network (CDN). Thus, there is a small risk that the learning management system might be up and running but there are problems loading MathJax from the CDN.*Moodle:*

As noted above the default configuration for Moodle has MathJax support (via a CDN) but switches off rendering of MathML. As`exams2moodle(...)`

currently uses MathML markup by default, this necessitates Firefox or Safari for viewing the quizzes in Moodle. In contrast,`exams2moodle(..., converter = "pandoc-mathjax")`

would use LaTeX math markup and render it by MathJax (unless the Moodle configuration switched off MathJax support).

**Request:**Feedback from Moodle users would be appreciated on whether they prefer the current default or`converter = "pandoc-mathjax"`

. It would be especially useful to find out whether the latter works in their Moodle installations.*Canvas:*

We are currently working on a dedicated`exams2qti12()`

-based function`exams2canvas()`

for generating quizzes for Canvas. This will keep the MathML-based default for the math notation as this is rendered smoothly by Canvas’ own MathJax support.*OpenOLAT:*

A new dedicated interface`exams2openolat()`

for generating quizzes for OpenOLAT has been added recently to R/exams (version 2.3-1). By default, this is a wrapper to`exams2qti21(..., converter = "pandoc-mathjax")`

because OpenOLAT provides MathJax rendering of LaTeX math (but not MathML). Some additional tweaks are necessary, though, because OpenOLAT expects the LaTeX to be embedded slightly differently from standard Pandoc output.

The mathematical equation in the random draw of the deriv exercise in LaTeX is: `f(x) = x^{8} e^{3.4x}`

. Here, we highlight that all converters yield almost equivalent output when rendered by MathJax:

`"pandoc-mathjax"` |
`"ttm"` |
`"pandoc-mathml"` |
---|---|---|

\(f(x) = x^{8} e^{3.4x}\) | $f(x)={x}^{8}{e}^{3.4x}$ | $f(x) = x^{8} e^{3.4x}$ |

(**Note:** If you are viewing this on R-bloggers or another aggregator some or all of the equations will not display correctly. Refer to the R/exams site for a version with MathJax properly enabled.)

The underlying LaTeX code generated by `converter = "pandoc-mathjax"`

is simply the original LaTeX code with some minimal HTML markup:

```
<span class="math inline">\(f(x) = x^{8} e^{3.4x}\)</span>
```

The MathML code generated by `converter = "ttm"`

differs slightly from the of `converter = "pandoc"`

(or equivalently `"pandoc-mathml"`

). The former yields:

```
<math xmlns="http://www.w3.org/1998/Math/MathML">
<mrow><mi>f</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo>=</mo>
<msup><mrow><mi>x</mi></mrow><mrow><mn>8</mn></mrow>
</msup>
<msup><mrow><mi>e</mi></mrow><mrow><mn>3</mn><mo>.</mo><mn>4</mn><mi>x</mi></mrow>
</msup>
</mrow></math>
```

The Pandoc version is very similar but contains some more markup and annotation:

```
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics>
<mrow><mi>f</mi>
<mo stretchy="false" form="prefix">(</mo><mi>x</mi><mo stretchy="false" form="postfix">)</mo>
<mo>=</mo>
<msup><mi>x</mi><mn>8</mn></msup><msup><mi>e</mi><mrow><mn>3.4</mn><mi>x</mi></mrow></msup>
</mrow><annotation encoding="application/x-tex">f(x) = x^{8} e^{3.4x}</annotation>
</semantics></math>
```

]]>Version 2.3-2 of the one-for-all exams generator R/exams has been published on the Comprehensive R Archive Network at https://CRAN.R-project.org/package=exams. In the next days this will propagate to other CRAN mirrors along with Windows and OS X binary packages. The development version of the package is now version 2.3-3 on R-Forge at http://R-Forge.R-project.org/forum/?group_id=1337.

- Support for further languages is available in
`exams2nops()`

and`nops_eval()`

: Russian (`ru`

, contributed by Boris Demeshev) and Serbian (`sr`

, contributed by Tatjana Kecojevic). Furthermore, Croatian (`hr`

) was streamlined along with Serbian by Tatjana Kecojevic. - In
`exams2nops()`

the`header`

argument can also be specified simply as`header = "\\mycommand{value}"`

rather than`header = list(mycommand = "value")`

. The former is more flexible, e.g., for passing additional options or more complex commands. Internally, the former is turned into an unnamed list which is then processed correspondingly by`exams2pdf()`

. - Scanning of written NOPS exams is enhanced and registration IDs are processed more reliably: First,
`nops_scan()`

gained a new argument`trim = 0.3`

that controls how much of the check boxes is trimmed in order to shave the borders prior to determining the average gray level. In versions up to 2.3-1 this implicitly had a value of`0.25`

hard-coded. Now the default increased to`0.3`

in order to shave box borders more reliably, e.g., in more pixelated scans. Second,`nops_scan()`

tries to process registration numbers more reliably. In case one of the registration columns contains more than one potential mark, the heuristics of determining the intended mark have been improved. `nops_eval()`

gained a new argument`labels = NULL`

that can be used to give labels for the marks that differ from the default`(length(mark) + 1):1`

.

- The new exercise confint3 (Rmd/Rnw) illustrates how to use the
`verbatim`

type for advanced processing in Moodle when asking for the computation of a 2-sided confidence interval. Rather than using a single correct value for each numeric result (along with a corresponding tolerance interval) as in`num`

exercises (see confint2), the exercise also provides a second partially correct value for each result. More precisely, the exercise yields 100% of the points for the correct solution based on t quantiles but still 50% for the solution based on normal quantiles. This is possible by setting the type to`verbatim`

and specifying the solution via the necessary Moodle XML markup directly, e.g.,`:NUMERICAL:=<value1>:<tolerance1>~%50%<value2>:<tolerance2>#<comment>`

. The exercise is a contribution by Ulrike Groemping.

- Rather than fully importing the basic dependencies
`stats`

,`graphics`

,`grDevices`

,`tools`

, and`utils`

, only the required functions from those packages are imported selectively. The main motivation for this was that otherwise the code evaluated in R/exams exercises might also use the same NAMESPACE semantics as the`exams`

package - instead of employing the setup in the user’s environment. The issue was raised in this StackOverflow thread by Florian Oswald. - In
`include_supplement()`

an argument`target = NULL`

is added to optionally include the supplement with a different file name than the original file name. - In all LaTeX templates that use Helvetica (phv) as the font for the main text, this is also used now in math mode by
`\usepackage[helvet]{sfmath}`

(rather than`\usepackage{sfmath}`

, as employed previously). In particular, this affects`exams2nops()`

and`tex2image()`

output.

- In previous versions the
`pandoc`

-based HTML`converter`

erroneously produced unbalanced`<p>`

tags in certain situations. - The QTI 2.1 XML output from
`exams2qti21()`

is somewhat cleaner, fixing some computations for correct/incorrect answers in`schoice`

/`mchoice`

exercises.

`confint3`

Computing the 2-sided confidence interval at 95% level for the mean based on a random sample. The exercise is a cloze with two numeric answers for the lower and upper bound of the confidence interval, respectively. Using the 'verbatim' clozetype for Moodle, the exercises yields 100% of the points for the correct solution based on t quantiles but still 50% for a partially correct solution based on normal quantiles.

Yes

Random numbers

Yes

No

No

No

*(Note that the HTML output contains mathematical equations in MathML. It is displayed by browsers with MathML support like Firefox or Safari - but not Chrome.)*

**Demo code:**

```
library("exams")
set.seed(1090)
exams2html("confint3.Rnw")
set.seed(1090)
exams2pdf("confint3.Rnw")
set.seed(1090)
exams2html("confint3.Rmd")
set.seed(1090)
exams2pdf("confint3.Rmd")
```

]]>R/exams was presented in a keynote lecture by Achim Zeileis at eRum 2018, the European R Users Meeting, this time organized by a team around Gergely Daróczi in Budapest. It was a great event with many exciting presentations, reflecting the vibrant R community in Europe (and beyond).

This blog post provides various resources accompanying the presentation which may be of interest to those who did not attend the meeting as well as those who did and who want to explore the materials in more detail.

Most importantly the presentation slides are available in PDF format (under CC-BY):

The eRum organizers did a great job in making the meeting accessible to those useRs who could not make it to Budapest. All presentations were available in a livestream on YouTube where also videos of all lectures were made available after the meeting (Standard YouTube License):

To illustrate the e-learning capabilities supported by R/exams, the presentation started with a live quiz using the audience response system ARSnova. The original version of the quiz was hosted on the ARSnova installation at Universität Innsbruck. To encourage readers to try out ARSnova for their own purposes, a copy of the quiz was also posted on the official ARSnova server at Technische Hochschule Mittelhessen (where ARSnova is developed under the General Public License, GPL):

The presentation briefly also showed an online test generated by R/exams and imported into OpenOLAT, an open-source learning management system (available under the Apache License). The online test is made available again here for anonymous guest access. *(Note however, that the system only has one guest user so that when you start the test there may already be some test results from a previous guest session. In that case you can finish the test and also start it again.)*

The presentation slides show how to set up an exam using the R package and then rendering it into different output formats. In order to allow the same exam to be rendered into a wide range of different output formats, only single-choice and multiple-choice exercises were employed (see the `choice`

list below). However, in the e-learning test shown in OpenOLAT all exercises types are supported (see the `elearn`

list below). All these exercises are readily provided in the package and also introduced online: deriv/deriv2, fruit/fruit2, ttest, boxplots, cholesky, lm, function. The code below uses the R/LaTeX (.Rnw) version but the R/Markdown version (.Rmd) could also be used instead.

```
## package
library("exams")
## single-choice and multiple-choice only
choice <- list("deriv2.Rnw", "fruit2.Rnw", c("ttest.Rnw", "boxplots.Rnw"))
## e-learning test (all exercise types)
elearn <- c("deriv.Rnw", "fruit.Rnw", "ttest.Rnw", "boxplots.Rnw",
"cholesky.Rnw", "lm.Rnw", "function.Rnw")
```

First, the exam with the choice-based questions can be easily turned into a PDF exam in NOPS format using `exams2nops`

, here using Hungarian language for illustration. Exams in this format can be easily scanned and evaluated within R.

```
set.seed(2018-05-16)
exams2nops(choice, institution = "eRum 2018", language = "hu")
```

Second, the choice-based exam version can be exported into the JSON format for ARSnova: Rexams-1.json. This contains an entire ARSnova session that can be directly imported into the ARSnova system as shown above. It employs a custom exercise set up just for eRum (conferences.Rmd) as well as a slightly tweaked exercise (fruit3.Rmd) that displays better in ARSnova.

```
set.seed(2018-05-16)
exams2arsnova(list("conferences.Rmd", choice[[1]], "fruit3.Rmd", choice[[3]]),
name = "R/exams", abstention = FALSE, fix_choice = TRUE)
```

Third, the e-learning exam can be generated in QTI 1.2 format for OpenOLAT, as shown above: eRum-2018.zip. The `exams2openolat`

command below is provided starting from the current R/exams version 2.3-1. It essentially just calls `exams2qti12`

but slightly tweaks the MathJax output from pandoc so that it is displayed properly by OpenOLAT.

```
set.seed(2018-05-16)
exams2openolat(elearn, name = "eRum-2018", n = 10, qti = "1.2")
```

In the last part of the presentation a couple of new and ongoing efforts within the R/exams project are highlighted. First, the natural language support in NOPS exams is mentioned which was recently described in more detail in this blog. Second, the relatively new “stress tester” was illustrated with the following example. (A more detailed blog post will follow soon.)

```
s <- stresstest_exercise("deriv2.Rnw")
plot(s)
```

Finally, a psychometric analysis illustrated how to examine exams regarding: Exercise difficulty, student performance, unidimensionality, fairness. The replication code for the results from the slides is included below (omitting some graphical details for simplicity, e.g., labeling or color).

```
## load data and exclude extreme scorers
library("psychotools")
data("MathExam14W", package = "psychotools")
mex <- subset(MathExam14W, nsolved > 0 & nsolved < 13)
## raw data
plot(mex$solved)
## Rasch model parameters
mr <- raschmodel(mex$solved)
plot(mr, type = "profile")
## points per student
MathExam14W <- transform(MathExam14W,
points = 2 * nsolved - 0.5 * rowSums(credits == 1)
)
hist(MathExam14W$points, breaks = -4:13 * 2 + 0.5, col = "lightgray")
abline(v = 12.5, lwd = 2, col = 2)
## person-item map
plot(mr, type = "piplot")
## principal component analysis
pr <- prcomp(mex$solved, scale = TRUE)
plot(pr)
biplot(pr, col = c("transparent", "black"),
xlim = c(-0.065, 0.005), ylim = c(-0.04, 0.065))
## differential item functioning
mr1 <- raschmodel(subset(mex, group == 1)$solved)
mr2 <- raschmodel(subset(mex, group == 2)$solved)
ma <- anchortest(mr1, mr2, adjust = "single-step")
## anchored item difficulties
plot(mr1, parg = list(ref = ma$anchor_items), ref = FALSE, ylim = c(-2, 3), pch = 19)
plot(mr2, parg = list(ref = ma$anchor_items), ref = FALSE, add = TRUE, pch = 19, border = 4)
legend("topleft", paste("Group", 1:2), pch = 19, col = c(1, 4), bty = "n")
## simultaneous Wald test for pairwise differences
plot(ma$final_tests)
```

]]>`fruit2`

A system of three linear equations has to be solved and the solution has to be entered into a fourth equation. However, the system is not defined through a verbal description or mathermatical notation but through images (clip art of tropical fruits). The problem can be interpreted as prices of three fruits (banana, orange, pineapple) and corresponding fruit baskets with different combinations of fruits. Images are stored in Base64 encoding within the exercise files and embedded dynamically into the output. A set of five answer alternatives is generated based on two potential mistakes and two random solutions from a suitable range. PDFs are best generated from the Rnw version, HTML is best generated with pandoc from either the Rmd version (where pandoc is used by default) or the Rnw version (where ttm is used by default, but pandoc can be easily used as well.)

Yes

Random numbers, shuffled graphics

Yes

No

Yes

No

*(Note that the HTML output contains mathematical equations in MathML. It is displayed by browsers with MathML support like Firefox or Safari - but not Chrome.)*

**Demo code:**

```
library("exams")
set.seed(1090)
exams2html("fruit2.Rnw")
set.seed(1090)
exams2pdf("fruit2.Rnw")
set.seed(1090)
exams2html("fruit2.Rmd")
set.seed(1090)
exams2pdf("fruit2.Rmd")
```

]]>`fruit`

A system of three linear equations has to be solved and the solution has to be entered into a fourth equation. However, the system is not defined through a verbal description or mathermatical notation but through images (clip art of tropical fruits). The problem can be interpreted as prices of three fruits (banana, orange, pineapple) and corresponding fruit baskets with different combinations of fruits. Images are stored in Base64 encoding within the exercise files and embedded dynamically into the output. PDFs are best generated from the Rnw version, HTML is best generated with pandoc from either the Rmd version (where pandoc is used by default) or the Rnw version (where ttm is used by default, but pandoc can be easily used as well.)

Yes

Random numbers, shuffled graphics

Yes

No

Yes

No

*(Note that the HTML output contains mathematical equations in MathML. It is displayed by browsers with MathML support like Firefox or Safari - but not Chrome.)*

**Demo code:**

```
library("exams")
set.seed(1090)
exams2html("fruit.Rnw")
set.seed(1090)
exams2pdf("fruit.Rnw")
set.seed(1090)
exams2html("fruit.Rmd")
set.seed(1090)
exams2pdf("fruit.Rmd")
```

]]>Version 2.3-1 of the one-for-all exams generator R/exams has been published on the Comprehensive R Archive Network at https://CRAN.R-project.org/package=exams. In the next days this will propagate to other CRAN mirrors along with Windows binary packages. The development version of the package is now version 2.3-2 on http://R-Forge.R-project.org/forum/?group_id=1337.

- Added new interface
`exams2openolat()`

for the open-source OpenOLAT learning management system. This is only a convenience wrapper to`exams2qti12()`

or`exams2qti21()`

with some dedicated tweaks for optimizing MathJax output for OpenOLAT. - New function
`include_tikz()`

that facilitates compiling standalone TikZ figures into a range of output formats, especially PNG and SVG (for HTML-based output). This is useful when including TikZ in R/Markdown exercises or when converting R/LaTeX exercises to HTML. Two examples have been added to the package that illustrate the capabilities of`include_tikz()`

: automaton, logic. A dedicated blog post is also planned.

- Following the blog post on Written R/exams around the World several users have been kind enough to add language support for: Croatian (hr.dcf, contributed by Krunoslav Juraić), Danish (da.dcf, contributed by Tue Vissing Jensen and Jakob Messner),Slovak (sk.dcf, contributed by Peter Fabsic), Swiss German (gsw.dcf, contributed by Reto Stauffer), Turkish (tr.dcf, contributed by Emrah Er). Furthermore, Portuguese has been distinguished into pt-PT.dcf (Portuguese Portuguese) vs. pt-BR.dcf (Brazilian Portuguese) with pt.dcf defaulting to the former (contributed by Thomas Dellinger).
- After setting a random seed
`exams2nops()`

and`exams2pdf()`

now yield the same random versions of the exercises. Previously, this was not the case because`exams2nops()`

internally generates a single random trial exam first for a couple of sanity checks. Now, the`.GlobalEnv$.Random.seed`

is restored after generating the trial exam. - Fixed the support for
`nsamp`

argument in`exams2nops()`

. Furthermore, current limitations of`exams2nops()`

are pointed out more clearly in error messages and edge cases caught. - Allow varying points within a certain exercise in
`nops_eval()`

.

- In
`exams2html()`

and other interfaces based on`make_exercise_transform_html()`

the option`base64 = TRUE`

now uses Base64 encoding for all file extensions (known to the package) whereas`base64 = NULL`

only encodes image files (previous default behavior). - Bug fixes and improvements in HTML transformers:
- Only
`="file.ext"`

(with`="`

) for supplementary files embedded into HTML is replaced now by the corresponding Base64-encoded version. `href="file.ext"`

is replaced by`href="file.ext" download="file.ext"`

prior to Base 64 replacement to assure that the file name is preserved for the browser/downloader.`alt="file.ext"`

and`download="file.ext"`

are preserved without the Base64-encoded version of`file.ext`

.

- Only
- Include further file URIs for Base64 supplements, in particular .sav for SPSS data files.
- In
`exams2blackboard(..., base64 = FALSE, ...)`

the`base64 = FALSE`

was erroneously ignored. No matter how base64 was specified essentially`base64 = TRUE`

was used, it is honored again now.

`\exshuffle{<num>}`

can now also be used for schoice exercises with more than one`TRUE`

answer. In a first step only one of the`TRUE`

answers is selected and then`<num>-1`

items from the`FALSE`

answers.- Function
`include_supplement(..., dir = "foo")`

- without full path to`"foo"`

- now also works if`"foo"`

is not a local sub-directory but a sub-directory to the exercise directory`edir`

(if specified). - Enable passing of
`envir`

argument from`exams2html()`

to`xweave()`

in case of R/Markdown (.Rmd) exercises. - When using
`exams2html(..., mathjax = TRUE)`

for testing purposes, mathjax.rstudio.com is used now rather than cdn.mathjax.org which is currently redirecting and will eventually be shut down completely. - Added support for
`\tightlist`

(as produced by pandoc) in all current LaTeX templates as well as`exams2nops()`

.

- Fixed a bug in
`stresstest_exercise()`

where the “rank” (previously called “order”) of the correct solution was computed incorrectly. Additional enhancements in plots and labels. - Fixed a bug for
`tex2image(..., tikz = TRUE)`

where erroneously`\usetikzlibrary{TRUE}`

was included. Also`tex2image(..., Sweave = TRUE)`

(the default) did not run properly on Windows, fixed now. - Better warnings if
`\exshuffle{<num>}`

could not be honored due to a lack of sufficiently many (suitable) answer alternatives. - Bug fix in CSV export of
`exams2arsnova()`

. Recent ARSnova versions use “mc” (rather than “MC”) and “abcd” (rather than “SC”) to code multiple-choice and single-choice questions, respectively.

`logic`

Gate diagrams for three logical operators (sampled from: and, or, xor, nand, nor) are drawn with TikZ and have to be matched to a truth table for another randomly drawn logical operator. Depending on the exams2xyz() interface the TikZ graphic can be rendered in PNG, SVG, or directly by LaTeX.

Yes

Shuffling, text blocks, and graphics

No

No

Yes

No

**Demo code:**

```
library("exams")
set.seed(1090)
exams2html("logic.Rnw")
set.seed(1090)
exams2pdf("logic.Rnw")
set.seed(1090)
exams2html("logic.Rmd")
set.seed(1090)
exams2pdf("logic.Rmd")
```

]]>`automaton`

An automaton diagram with four states A-D is drawn with TikZ and is to be interpreted, where A is always the initial state and one state is randomly picked as the accepting state. Five binary 0/1 input sequences acceptance have to be assessed with approximately a quarter of all sequences being accepted. Depending on the exams2xyz() interface the TikZ graphic can be rendered in PNG, SVG, or directly by LaTeX.

Yes

Random numbers, text blocks, and graphics

No

No

Yes

No

**Demo code:**

```
library("exams")
set.seed(1090)
exams2html("automaton.Rnw")
set.seed(1090)
exams2pdf("automaton.Rnw")
set.seed(1090)
exams2html("automaton.Rmd")
set.seed(1090)
exams2pdf("automaton.Rmd")
```

]]>Online tests and quizzes conducted in learning management systems are a highly useful tool for students to practice course and materials and for teachers to monitor the progress students make while working on their exercises. However, to allow for enough variation and avoid cheating among students, it is often necessary to generate a large number of variations of exercises for such online tests.

In the following it is demonstrated how R/exams and its `exams2blackboard()`

function can be used to create “dynamic” online tests and quizzes as well as large item pools for the learning management system Blackboard. The discussion is based on experiences at the University of Amsterdam, in the Bachelor programme Child Development and Education, where Blackboard testing facilities are used extensively in courses in methods and statistics: Students make formative tests in their preparation of practicals, and teachers use student results to optimize the content of those practicals.

In the Blackboard system, the *Tests, Surveys and Pools* section can be accessed from the *Control Panel* under *Course Tools*.

If an exam created with R/exams is imported into Blackboard, the material will appear both in the ‘Tests’ module and ‘Pools’ module. Tests are sets of questions that make up an exam. Once an exam is created here, it may be deployed within a content folder for administering it to students. Test results appear in the Grade Centre, which is a spreadsheet-like database for storing student results. Pools are sets of questions that can be added to any Test, and are useful for storing questions in the system. So, R/exams may be used to either construct a full exam that can be readily deployed in Blackboard, or to create item pools that can be used as input for exams created within the Blackboard system itself. For documentation by Blackboard on its testing facilities, see the Blackboard Help pages.

In its testing facilities, Blackboard employs the international QTI 1.2 (question & test interoperability) standard. However, as Blackboard created its own special flavor of QTI 1.2, it was not possible to simply use `exams2qti12()`

from R/exams but it was necessary to create a separate function `exams2blackboard()`

. This supports four types of R/exams exercises to be included in the Blackboard testing system:

R/exams name | Blackboard type |
---|---|

`num` |
Calculated numeric |

`schoice` |
Multiple choice |

`mchoice` |
Multiple answer |

`string` |
Fill in the blank |

The R/exams `cloze`

type is part of the QTI 1.2 standard but not officially supported in Blackboard. Hence this type of item is not supported in the current version of `exams2blackboard()`

either as no workaround for implementation in Blackbard has been found (yet).

For an overview of question types in Blackboard’s testing module, see Question types.

The first step in creating an exam is to construct its constituting questions; for an introduction into designing exercises using R/exams, see the First Steps tutorial. Next, the exercises are gathered into an exam. To that end, in R, after loading the R/exams package, a list of the exercise file names is created:

```
library("exams")
myexam <- list(
"tstat.Rnw",
"tstat2.Rnw",
"relfreq.Rnw",
"anova.Rnw",
"boxplots.Rnw"
)
```

Here, we use a collection of one `num`

(tstat), one `schoice`

(tstat2), and three `mchoice`

(relfreq, anova, boxplots) questions, respectively, that are a part of the package. Above, the items have a .Rnw format, but all of the examples are also available in .Rmd format, leading to virtually identical output.

Next, as a first check of the exam and its questions, it is easier to create one HTML version on-the-fly in the browser before importing one or more versions into Blackboard. (Alternatively or additionally, a PDF version can be easily used.)

```
exams2html(myexam)
exams2pdf(myexam)
```

After a thorough inspection, the exam may be converted to Blackboard format. Let’s start with the simple case of using a single copy per exercise:

```
set.seed(1234)
exams2blackboard(myexam)
```

As the five exercises have a dynamic nature (i.e., numbers are drawn randomly), a random seed is set to make the generated exams exactly reproducible for you (or ourselves at some point in the future). In the working directory a zip file named blackboard.zip is created which contains an exam consisting of five item pools, each containing a single copy of an exercise.

Next, the exam `blackboard.zip`

is imported in Blackboard using *Import Test* in the *Tests* menu. Below, it can be seen that importing was successful: It now appears as *blackboard-test* in the menu along with three other exams.

The exam may be inspected, and exercises may be editted and saved:

The preview that is available in this menu *does not* provide an accurate impression of what exercises look like during the actual administration of the test. Therefore, it is advised to activate the exam (for instructors only) and inspect it in an actual administration. Below your see an example of a test administration:

Several arguments may be specified to adapt the exam to meet one’s preferences. For example, the name of the zip file may be set, and the number of copies drawn for each exercise may be changed:

```
set.seed(1234)
exams2blackboard(myexam, n = 3, name = "myexam")
```

This code results in a zip file named myexam.zip in the working directory with three copies for each exercise.

When a non-integer numerical answer is required for `num`

exercises, it is important that the correct decimal mark is used when typing the number in the answer box in Blackboard’s testing environment. The decimal mark depends on the language settings of the *individual* student in Blackboard. For example, if the language options are either English or Dutch, English requires a point (“.”) as decimal mark and Dutch requires a comma (“,”). For student level language settings, see Personal settings. It is advised that students be extensively informed about which decimal mark to use in their tests, because using an inappropriate decimal mark will lead to an incorrect answer. Alternatively, one may Enforce the Language Pack, forcing students to use a given language (and matching decimal sign). It is stressed that this issue is specific to Blackboard and has nothing to do with how exercises are created with R/exams; it is simply an issue we experienced at the University of Amsterdam while deploying online tests.

By default `exams2blackboard()`

converts mathematical symbols to MathML. Hence, it is advised that Blackboard tests are administered in a web browser that supports MathML such as Mozilla Firefox and Safari. Of these two the former shows the best performance and the latter does a fairly decent job. Therefore students should be instructed to open their tests in either one of them. Note that Chrome does not support MathML.

Alternatively, `exams2blackboard(..., converter = "pandoc-mathjax")`

would allow to embed mathematical formulas for the MathJax plugin that would need to be available in the Blackboard system. As we did not have access to a MathJax-enabled Blackboard system, we were not able to test this option, though.

Evaluation of `num`

, `schoice`

, and `string`

item types generated by R/exams is relatively straightforward: Answers can either be correct or wrong. However, for `mchoice`

exercises there is more flexibility: Either all parts of an answer have to be exactly correct or partial credits can be assigned. The `exams2blackboard()`

function allows for specifying evaluation policies and passing them to Blackboard. For more details, see `?exams_eval`

.

If the test constructor develops exercises for the purpose of creating Blackboard *item pools* (as opposed to *exams*), it is advised not to mix exercises into one pool, as is done with `regression`

and `anova`

in:

```
library("exams")
myexam <- list(
"boxplots",
"tstat",
c("regression", "anova"),
"scatterplot",
"relfreq"
)
```

When combining two or more different exercise types, the name and type of the exercise that is first drawn (in the random sampling) will appear as the respective name and type of the item pool in Blackboard, which may lead to administrative issues in the system.

For creating *exams*, however, mixing is no issue.

Various other learning management systems allow for importing tests and item pools created in Blackboard. Thus, in principle, exams created using `exams2blackboard()`

maybe uploaded in those systems as well We did some limited trials with *Desire2Learn (D2L)* and *Canvas* suggesting that this route for import works. However, more trials and possibly further finetuning may be needed for things to work properly.