---
title: "Regular Expressions in R"
author: "Martin Schweinberger"
date: ""
output:
  bookdown::html_document2
bibliography: bibliography.bib
link-citations: yes
---

```{r uq1, echo=F, eval = T, fig.cap="", message=FALSE, warning=FALSE, out.width='100%'}
knitr::include_graphics("https://slcladal.github.io/images/uq1.jpg")
```

# Introduction

This tutorial introduces regular expressions and how they can be used when working with language data. Regular expressions are powerful tools used to search and manipulate text patterns. They provide a way to find specific sequences of characters within larger bodies of text. Think of them as search patterns on steroids. Regular expressions are useful for tasks like extracting specific words, finding patterns, or replacing text in bulk. They offer a concise and flexible way to describe complex text patterns using symbols and special characters. Regular expressions have applications in linguistics and humanities research, aiding in tasks such as text analysis, corpus linguistics, and language processing. Understanding regular expressions can unlock new possibilities for exploring and analyzing textual data.


```{r diff, echo=FALSE, out.width= "15%", out.extra='style="float:right; padding:10px"'}
knitr::include_graphics("https://slcladal.github.io/images/gy_chili.jpg")
```

This tutorial is aimed at beginners and intermediate users of R with the aim of showcasing how to use regular expression (or wild cards) in R. The aim is not to provide a fully-fledged analysis but rather to show and exemplify selected useful functions and methods associated with regular expressions. 



<div class="warning" style='padding:0.1em; background-color:#f2f2f2; color:#51247a'>
<span>
<p style='margin-top:1em; text-align:center'>
To be able to follow this tutorial, we suggest you check out and familiarize yourself with the content of the following **R Basics** tutorials:<br>
</p>
<p style='margin-top:1em; text-align:left'>
<ul>
  <li>[Getting started with R](https://ladal.edu.au/intror.html) </li>
  <li>[Loading, saving, and generating data in R](https://ladal.edu.au/load.html) </li>
  <li>[String Processing in R](https://ladal.edu.au/string.html) </li>
</ul>
</p>
<p style='margin-top:1em; text-align:center'>
Click [**here**](https://ladal.edu.au/content/regex.Rmd)^[If you want to render the R Notebook on your machine, i.e. knitting the document to html or a pdf, you need to make sure that you have R and RStudio installed and you also need to download the [**bibliography file**](https://slcladal.github.io/content/bibliography.bib) and store it in the same folder where you store the Rmd file.] to download the **entire R Notebook** for this tutorial.<br><br>
[![Binder](https://mybinder.org/badge_logo.svg)](https://binderhub.atap-binder.cloud.edu.au/v2/gh/SLCLADAL/interactive-notebooks-environment/main?urlpath=git-pull%3Frepo%3Dhttps%253A%252F%252Fgithub.com%252FSLCLADAL%252Finteractive-notebooks%26urlpath%3Dlab%252Ftree%252Finteractive-notebooks%252Fnotebooks%252Fregex_cb.ipynb%26branch%3Dmain)<br>
Click [**here**](https://binderhub.atap-binder.cloud.edu.au/v2/gh/SLCLADAL/interactive-notebooks-environment/main?urlpath=git-pull%3Frepo%3Dhttps%253A%252F%252Fgithub.com%252FSLCLADAL%252Finteractive-notebooks%26urlpath%3Dlab%252Ftree%252Finteractive-notebooks%252Fnotebooks%252Fregex_cb.ipynb%26branch%3Dmain) to open an interactive Jupyter notebook that allows you to execute, change, and edit the code as well as to upload your own data. <br>
</p>
<p style='margin-left:1em;'>
</p></span>
</div>

<br>

How can you search texts for complex patterns or combinations of patterns? This question will answered in this tutorial and at the end you will be able to perform very complex searches yourself. The key concept of this tutorial is that of a regular expression. A regular expression (in short also called *regex* or *regexp*) is a special sequence of characters (or string) for describing a search pattern. You can think of regular expressions as very powerful combinations of wildcards or as wildcards on steroids. 

If you would like to get deeper into regular expressions, I can recommend @friedl2006mastering and, in particular, chapter 17 of @peng2016r for further study (although the latter uses base R rather than tidyverse functions, but this does not affect the utility of the discussion of regular expressions in any major or meaningful manner). Also, [here](https://rstudio.com/wp-content/uploads/2016/09/RegExCheatsheet.pdf) is a so-called *cheatsheet* about regular expressions written by  Ian Kopacka and provided by RStudio. Nick Thieberger has also recorded a very nice [*Introduction to Regular Expressions*](https://www.youtube.com/watch?v=8ILToE0CNpM) for humanities scholars to YouTube.

**Preparation and session set up**

This tutorial is based on R. If you have not installed R or are new to it, you will find an introduction to and more information how to use R [here](https://slcladal.github.io/intror.html). For this tutorials, we need to install certain *packages* from an R *library* so that the scripts shown below are executed without errors. Before turning to the code below, please install the packages by running the code below this paragraph. If you have already installed the packages mentioned below, then you can skip ahead and ignore this section. To install the necessary packages, simply run the following code - it may take some time (between 1 and 5 minutes to install all of the libraries so you do not need to worry if it takes some time).

```{r prep1, echo=T, eval = F, message=FALSE, warning=FALSE}
# set options
options(stringsAsFactors = F)         # no automatic data transformation
options("scipen" = 100, "digits" = 4) # suppress math annotation
# install packages
install.packages("tidyverse")
install.packages("flextable")
install.packages("htmlwidgets")
# install klippy for copy-to-clipboard button in code chunks
install.packages("remotes")
remotes::install_github("rlesur/klippy")
```

In a next step, we load the packages.

```{r prep2, message=FALSE, warning=FALSE}
library(tidyverse)
library(flextable)
# activate klippy for copy-to-clipboard button
klippy::klippy()
```

Once you have installed RStudio and have initiated the session by executing the code shown above, you are good to go.

# Getting started with Regular Expressions

To put regular expressions into practice, we need some text that we will perform out searches on. In this tutorial, we will use texts from wikipedia about grammar.

```{r regex01, echo=T, eval = T, message=FALSE, warning=FALSE}
# read in first text
text1 <- readLines("https://slcladal.github.io/data/testcorpus/linguistics02.txt")
et <-  paste(text1, sep = " ", collapse = " ")
# inspect example text
et
```

In addition, we will split the example text into words to have another resource we can use to understand regular expressions

```{r regex02, echo=T, eval = T, message=FALSE, warning=FALSE}
# split example text
set <- str_split(et, " ") %>%
  unlist()
# inspect
head(set)
```

Before we delve into using regular expressions, we will have a look at the regular expressions that can be used in R and also check what they stand for.

There are three basic types of regular expressions:

* regular expressions that stand for individual symbols and determine frequencies

* regular expressions that stand for classes of symbols

* regular expressions that stand for structural properties

The regular expressions below show the first type of regular expressions, i.e. regular expressions that stand for individual symbols and determine frequencies.

```{r regex05, echo=F, eval = T}
symbols1 <- c(fixed("?"), fixed("*"), fixed("+"), "{n}", "{n,}", "{n,m}")
explanation1 <- c("The preceding item is optional and will be matched at most once", "The preceding item will be matched zero or more times", "The preceding item will be matched one or more times", "The preceding item is matched exactly n times", "The preceding item is matched n or more times", "The preceding item is matched at least n times, but not more than m times")
example1 <- c("walk[a-z]? = walk, walks", 
             "walk[a-z]* = walk, walks, walked, walking", 
             "walk[a-z]+ = walks, walked, walking", 
             "walk[a-z]{2} = walked", 
             "walk[a-z]{2,} = walked, walking", 
             "walk[a-z]{2,3} = walked, walking")
df_regex <- data.frame(symbols1, explanation1, example1)
colnames(df_regex) <- c("RegEx Symbol/Sequence", "Explanation", "Example")
df_regex %>%
  as.data.frame() %>%
#   tibble::rownames_to_column("Hedge") %>%
  head(15) %>%
  flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "Regular expressions that stand for individual symbols and determine frequencies.")  %>%
  flextable::border_outer()
```

The regular expressions below show the second type of regular expressions, i.e. regular expressions that stand for classes of symbols.

```{r regex03, echo=F, eval = T}
symbols2 <- c("[ab]", "[a-z]", "[AB]", "[A-Z]", "[12]", "[0-9]", "[:digit:]", "[:lower:]", "[:upper:]", "[:alpha:]", "[:alnum:]", "[:punct:]", "[:graph:]", "[:blank:]", "[:space:]", "[:print:]")
explanations2 <- c("lower case a and b",
                   "all lower case characters from a to z",
                   "upper case a and b",
                   "all upper case characters from A to Z",
                   "digits 1 and 2", 
                   "digits: 0 1 2 3 4 5 6 7 8 9",
                   "digits: 0 1 2 3 4 5 6 7 8 9", 
                   "lower case characters: a–z", 
                   "upper case characters: A–Z", 
                   "alphabetic characters: a–z and A–Z", 
                   "digits and alphabetic characters", 
                   "punctuation characters: . , ; etc.", 
                   "graphical characters: [:alnum:] and [:punct:]", 
                   "blank characters: Space and tab", 
                   "space characters: Space, tab, newline, and other space characters", 
                   "printable characters: [:alnum:], [:punct:] and [:space:]")
df_regex <- data.frame(symbols2, explanations2)
colnames(df_regex) <- c("RegEx Symbol/Sequence", "Explanation")
df_regex %>%
  as.data.frame() %>%
#   tibble::rownames_to_column("Hedge") %>%
  head(15) %>%
  flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "Regular expressions that stand for classes of symbols.")  %>%
  flextable::border_outer()
```

The regular expressions that denote classes of symbols are enclosed in `[]` and `:`. The last type of regular expressions, i.e. regular expressions that stand for structural properties are shown below.

```{r regex09, echo=F, eval = T}
symbols3 <- c(fixed("\\\\w"), fixed("\\\\W"), fixed("\\\\s"), fixed("\\\\S"), 
              fixed("\\\\d"), fixed("\\\\D"), fixed("\\\\b"), fixed("\\\\B"), 
              fixed("<"), fixed(">"), fixed("^"), fixed("$"))
explanations3 <- c("Word characters: [[:alnum:]_]",
                   "No word characters: [^[:alnum:]_]",
                   "Space characters: [[:blank:]]",
                   "No space characters: [^[:blank:]]",
                   "Digits: [[:digit:]]",
                   "No digits: [^[:digit:]]",
                   "Word edge",
                   "No word edge",
                   "Word beginning",
                   "Word end",
                   "Beginning of a string",
                   "End of a string")
df_regex <- data.frame(symbols3, explanations3)
colnames(df_regex) <- c("RegEx Symbol/Sequence", "Explanation")
df_regex %>%
  as.data.frame() %>%
#   tibble::rownames_to_column("Hedge") %>%
  head(15) %>%
  flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "Regular expressions that stand for structural properties.")  %>%
  flextable::border_outer()
```

# Practice

In this section, we will explore how to use regular expressions. At the end, we will go through some exercises to help you understand how you can best utilize regular expressions.

Show all words in the split example text that contain `a` or `n`.

```{r regex10, echo=T, eval = T, message=FALSE, warning=FALSE}
set[str_detect(set, "[an]")]
```

Show all words in the split example text that begin with a lower case `a`.

```{r regex11, echo=T, eval = T, message=FALSE, warning=FALSE}
set[str_detect(set, "^a")]
```

Show all words in the split example text that end in a lower case `s`.

```{r regex12, echo=T, eval = T, message=FALSE, warning=FALSE}
set[str_detect(set, "s$")]
```

Show all words in the split example text in which there is an `e`, then any other character, and than another `n`.

```{r regex13, echo=T, eval = T, message=FALSE, warning=FALSE}
set[str_detect(set, "e.n")]
```

Show all words in the split example text in which there is an `e`, then two other characters, and than another `n`.

```{r regex14, echo=T, eval = T, message=FALSE, warning=FALSE}
set[str_detect(set, "e.{2,2}n")]
```

Show all words that consist of exactly three alphabetical characters in the split example text.

```{r regex15, echo=T, eval = T, message=FALSE, warning=FALSE}
set[str_detect(set, "^[:alpha:]{3,3}$")]
```

Show all words that consist of six or more alphabetical characters in the split example text.

```{r regex16, echo=T, eval = T, message=FALSE, warning=FALSE}
set[str_detect(set, "^[:alpha:]{6,}$")]
```

Replace all lower case `a`s with upper case `E`s in the example text.

```{r regex17, echo=T, eval = T, message=FALSE, warning=FALSE}
str_replace_all(et, "a", "E")
```

Remove all non-alphabetical characters in the split example text.

```{r regex18, echo=T, eval = T, message=FALSE, warning=FALSE}
str_remove_all(set, "\\W")
```

Remove all white spaces in the example text.

```{r regex19, echo=T, eval = T, message=FALSE, warning=FALSE}
str_remove_all(et, " ")
```

**Highlighting patterns**

We use the `str_view` and `str_view_all` functions to show the occurrences of regular expressions in the example text.

To begin with, we match an exactly defined pattern (`ang`).

```{r regex20, echo=T, eval = T, message=FALSE, warning=FALSE}
str_view_all(et, "ang")
```

Now, we include . which stands for any symbol (except a new line symbol).

```{r regex23, echo=T, eval = T, message=FALSE, warning=FALSE}
str_view_all(et, ".n.")
```


***

<div class="warning" style='padding:0.1em; background-color:#51247a; color:#f2f2f2'>
<span>
<p style='margin-top:1em; text-align:center'>
<b>EXERCISE TIME!</b></p>
<p style='margin-left:1em;'>
</p></span>
</div>

<div class="question">` 

1. What regular expression can you use to extract all forms of *walk* from a text?

<details>
  <summary>Answer</summary>
  `[Ww][Aa][Ll][Kk].*`
</details>

More exercises will follow - bear with us ;)

</div>`

***


# Citation & Session Info {-}

Schweinberger, Martin. 2022. *Regular Expressions in R*. Brisbane: The University of Queensland. url: https://ladal.edu.au/regex.html (Version 2022.11.17).

```
@manual{schweinberger2022regex,
  author = {Schweinberger, Martin},
  title = {Regular Expressions in R},
  note = {https://ladal.edu.au/regex.html},
  year = {2022},
  organization = {The University of Queensland, School of Languages and Cultures},
  address = {Brisbane},
  edition = {2022.11.17}
}
```


```{r fin}
sessionInfo()
```


***

[Back to top](#introduction)

[Back to LADAL home](https://ladal.edu.au)

***

# References {-}
