A controlled vocabulary is a predefined set of terms intended to be used to be used in a specific context. They can solve a number of problems of data standardisation and cleaning in data science. controller provides functions for working with controlled vocabularies in R.
This vignette explains the main features of the package. It
introduces the control() verb, which recodes values in a
vector using a lookup table of preferred and variant terms (a
thesaurus).
The control() verb
A common data-tidying problem is standardising variant terms for the same concept. Imagine we have a dataset that uses a number of different names for shades of the same colour. As data analysts, we naturally want to recode the data to eliminate this messy creativity, for example using dplyr::recode():
library(dplyr, warn.conflicts = FALSE)
shades <- c("daffodil", "purple", "magenta", "azure", "navy", "violet")
recode(shades,
daffodil = "yellow",
purple = "purple",
magenta = "pink",
azure = "blue",
navy = "blue",
violet = "purple")
#> [1] "yellow" "purple" "pink" "blue" "blue" "purple"But recoding this way can be tedious, especially if there are a large
number of terms. With control(), we can instead use a data
frame containing a thesaurus to replace the values:
library(controller)
data("colour_thesaurus")
control(shades, colour_thesaurus)
#> Replaced values:
#> ℹ daffodil → yellow
#> ℹ azure → blue
#> ℹ navy → blue
#> ℹ violet → purple
#> Warning: Some values of `x` were not matched in `thesaurus`:
#> ✖ magenta
#> [1] "yellow" "purple" "magenta" "blue" "blue" "purple"By default, control() issues a message listing the
values that were replaced and a warning for any values that could not be
matched in the thesaurus. These can be suppressed with
quiet = TRUE and warn_unmatched = FALSE
respectively.
Use control_names() to control the names of an object
rather than its values. This is useful for standardising column names in
data frames:
df <- data.frame(temp = 20, humid = 65, `wind speed` = 10, date = "2024-01-01")
df
#> temp humid wind.speed date
#> 1 20 65 10 2024-01-01
control_names(df, thesaurus = data.frame(
preferred = c("temperature", "humidity", "wind_speed"),
variant = c("temp", "humid", "wind speed")
))
#> Replaced values:
#> ℹ temp → temperature
#> ℹ humid → humidity
#> Warning: Some values of `x` were not matched in `thesaurus`:
#> ✖ wind.speed
#> ✖ date
#> temperature humidity wind.speed date
#> 1 20 65 10 2024-01-01Fuzzy matching
control() also supports fuzzy matching, removing the
need to exhaustively list variants for common causes of differing
terminology. For example, to perform a case insensitive match to the
thesaurus:
shades_ci <- toupper(shades)
control_ci(shades_ci, colour_thesaurus)
#> Replaced values:
#> ℹ DAFFODIL → yellow
#> ℹ PURPLE → purple
#> ℹ AZURE → blue
#> ℹ NAVY → blue
#> ℹ VIOLET → purple
#> Warning: Some values of `x` were not matched in `thesaurus`:
#> ✖ MAGENTA
#> [1] "yellow" "purple" "MAGENTA" "blue" "blue" "purple"control_fuzzy() goes further by also ignoring
differences in word boundaries and character encoding. Fuzzy boundary
matching treats word boundaries like spaces, hyphens, and underscores as
equivalent:
df <- data.frame(preferred = "foo bar", variant = "foo-bar")
control(c("foo bar", "foo_bar", "foobar"), df,
fuzzy_boundary = TRUE, quiet = FALSE)
#> Replaced values:
#> ℹ foo_bar → foo bar
#> ℹ foobar → foo bar
#> [1] "foo bar" "foo bar" "foo bar"Fuzzy encoding matching ignores non-ASCII characters, so plain ASCII input can match variants with diacritics or wrongly encoded characters (‘mojibake’):
df <- data.frame(preferred = "bar", variant = "fo\u00f6")
control("foo", df, fuzzy_encoding = TRUE, quiet = FALSE)
#> Replaced values:
#> ℹ foo → bar
#> [1] "bar"To inspect which type of match was used for each value, use
control_matches(). It returns a data frame with a column
for each match type, rather than a single vector:
control_matches(shades_ci, colour_thesaurus, case_insensitive = TRUE)
#> term exact_match case_insensitive_match
#> 1 DAFFODIL <NA> yellow
#> 2 PURPLE <NA> purple
#> 3 MAGENTA <NA> <NA>
#> 4 AZURE <NA> blue
#> 5 NAVY <NA> blue
#> 6 VIOLET <NA> purpleThesaurus format
A thesaurus is a data frame with two columns containing preferred terms and the variants to be controlled. Each row maps one variant to one preferred term. The variants must be unique values – otherwise they cannot be unambiguously matched to a preferred term.
colour_thesaurus,1 used in the examples above, is an example
of a thesaurus bundled with the package:
head(colour_thesaurus)
#> colour shade
#> 1 white white
#> 2 white pearl
#> 3 white alabaster
#> 4 white snow
#> 5 white ivory
#> 6 white creamBecause a thesaurus is simply a data frame, it can be created or read into R with standard tools:
my_thesaurus <- data.frame(
preferred = c("feline", "canine"),
variant = c("cat", "dog")
)
animals <- c("cat", "dog", "parrot")
control(animals, my_thesaurus)
#> Replaced values:
#> ℹ cat → feline
#> ℹ dog → canine
#> Warning: Some values of `x` were not matched in `thesaurus`:
#> ✖ parrot
#> [1] "feline" "canine" "parrot"Use the thesaurus_cols argument to specify which columns
of the data frame contain the preferred and variant terms
respectively:
my_wider_thesaurus <- data.frame(
code = c("FEL", "CAN"),
label = c("feline", "canine"),
variant = c("cat", "dog")
)
control(animals, my_wider_thesaurus, thesaurus_cols = c("label", "variant"))
#> Replaced values:
#> ℹ cat → feline
#> ℹ dog → canine
#> Warning: Some values of `x` were not matched in `thesaurus`:
#> ✖ parrot
#> [1] "feline" "canine" "parrot"The package can also read controlled vocabularies in Historic
England’s FISH
(Forum on Information Standards in Heritage) format using
read_fish(). The path argument can be a local
.zip file, an uncompressed directory, or a URL:
nationality_zip <- system.file("extdata", "fish-nationality.zip",
package = "controller")
fish_nationality <- read_fish(nationality_zip)
head(fish_nationality)
#> preferred term
#> 1 FAROESE FaeroeseThe result is a data frame with columns preferred and
term, ready to use with control().