---
title: "Concordancing with R"
author: "Martin Schweinberger"
date: ""
output:
  bookdown::html_document2: default
bibliography: bibliography.bib
link-citations: yes
---

```{r uq1, echo=F, fig.cap="", message=FALSE, warning=FALSE, out.width='100%'}
knitr::include_graphics("https://slcladal.github.io/images/uq1.jpg")
knitr::opts_chunk$set(warning = FALSE, message = FALSE) 
```

# Introduction{-}

This tutorial introduces how to extract concordances and keyword-in-context (KWIC) displays with R using the `quanteda` package [@quanteda2018package]. 

<div class="warning" style='padding:0.5em; background-color:rgba(215,209,204,.3); color:#51247a'>
<span>
<p style='margin-top:1em; text-align:center'>
Please cite as: <br>Schweinberger, Martin. 2023. *Concordancing with R*. Brisbane: The Language Technology and Data Analysis Laboratory (LADAL). url: https://ladal.edu.au/kwics.html (Version 2023.09.23).<br>
</p>
</span>
</div>


```{r diff, echo=FALSE, out.width= "15%", out.extra='style="float:right; padding:10px"'}
knitr::include_graphics("https://slcladal.github.io/images/gy_chili.jpg")
```

This tutorial is aimed at beginners and intermediate users of R with the aim of showcasing how to extract keywords and key phrases from textual data and how to process the resulting concordances using R. The aim is not to provide a fully-fledged analysis but rather to show and exemplify selected useful methods associated with concordancing. 



<div class="warning" style='padding:0.1em; background-color:rgba(215,209,204,.3); color:#51247a'>
<span>
<p style='margin-top:1em; text-align:center'>
To be able to follow this tutorial, we suggest you check out and familiarize yourself with the content of the following **R Basics** tutorials:<br>
</p>
<p style='margin-top:1em; text-align:left'>
<ul>
  <li>[Getting started with R](https://ladal.edu.au/intror.html) </li>
  <li>[Loading, saving, and generating data in R](https://ladal.edu.au/load.html) </li>
  <li>[String Processing in R](https://ladal.edu.au/string.html) </li>
  <li>[Regular Expressions in R](https://ladal.edu.au/regex.html) </li>
</ul>
</p>
<p style='margin-top:1em; text-align:center'>
Click [**here**](https://ladal.edu.au/content/kwics.Rmd)^[If you want to render the R Notebook on your machine, i.e. knitting the document to html or a pdf, you need to make sure that you have R and RStudio installed and you also need to download the [**bibliography file**](https://slcladal.github.io/content/bibliography.bib) and store it in the same folder where you store the Rmd file.] to download the **entire R Notebook** for this tutorial.<br><br>
[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/SLCLADAL/interactive-notebooks-environment/main?urlpath=git-pull%3Frepo%3Dhttps%253A%252F%252Fgithub.com%252FSLCLADAL%252Finteractive-notebooks%26urlpath%3Dlab%252Ftree%252Finteractive-notebooks%252Fnotebooks%252Fkwics_cb.ipynb%26branch%3Dmain)<br>
Click [**here**](https://mybinder.org/v2/gh/SLCLADAL/interactive-notebooks-environment/main?urlpath=git-pull%3Frepo%3Dhttps%253A%252F%252Fgithub.com%252FSLCLADAL%252Finteractive-notebooks%26urlpath%3Dlab%252Ftree%252Finteractive-notebooks%252Fnotebooks%252Fkwics_cb.ipynb%26branch%3Dmain)  to open a Jupyter notebook that allows you to follow this tutorial interactively. This means that you can execute, change, and edit the code used in this tutorial to help you better understand how the code shown here works (make sure you run all code chunks in the order in which they appear - otherwise you will get an error).  </p> <br>
</p>
<p style='margin-left:1em;'>
</p></span>
</div>

<br>


<div class="warning" style='padding:0.1em; background-color:#51247a; color:#f2f2f2'>
<span>
<p style='margin-top:1em; text-align:center'>
**CONCORDANCING TOOL** 
</p>
<p style='margin-top:1em; text-align:center'>
Click on this [![Binder](https://mybinder.org/badge_logo.svg)](https://binderhub.atap-binder.cloud.edu.au/v2/gh/SLCLADAL/interactive-notebooks-environment/main?urlpath=git-pull%3Frepo%3Dhttps%253A%252F%252Fgithub.com%252FSLCLADAL%252Finteractive-notebooks%26urlpath%3Dlab%252Ftree%252Finteractive-notebooks%252Fnotebooks%252Fkwictool.ipynb%26branch%3Dmain) badge to open an notebook-based tool <br>that allows you to upload your own text(s), perform concordancing on them, and download the resulting kwic(s). </p><br>
<p style='margin-left:1em;'>
</p></span>
</div>

<br>

In the language sciences, concordancing refers to the extraction of words from a given text or texts [@lindquist2009corpus 5]. Commonly, concordances are displayed in the form of keyword-in-context displays (KWICs) where the search term is shown in context, i.e. with preceding and following words. Concordancing are central to analyses of text and they often represents the first step in more sophisticated analyses of language data [@stafanowitsch2020corpus]. The play such a key role in the language sciences because concordances are extremely valuable for understanding how a word or phrase is used, how often it is used, and in which contexts is used. As concordances allow us to analyze the context in which a word or phrase occurs and provide frequency information about word use, they also enable us to analyze collocations or the collocational profiles of words and phrases [@stafanowitsch2020corpus 50-51]. Finally, concordances can also be used to extract examples and it is a very common procedure. 

```{r antc1, echo=FALSE, out.width= "100%", out.extra='style="float:left; padding:2px"', fig.cap="\\label{fig:Fig1} Concordances in AntConc."}
knitr::include_graphics("https://slcladal.github.io/images/AntConcConcordance.png")
```

<br>

There are various very good software packages that can be used to create concordances - both for offline use. There are, for example

+ [*AntConc*](https://www.laurenceanthony.net/software/antconc/) [@anthony2004antconc]
+ [*SketchEngine*](https://www.sketchengine.eu/)[@kilgarriff2004itri]
+ [*MONOCONC*](https://www.monoconc.com/)[@barlow1999monoconc]
+ [*ParaConc*](https://paraconc.com/))[@barlow2002paraconc] 
+ [*Web Concordancers*](https://lextutor.ca/conc/)
+ [*CorpusMate*](https://corpusmate.com/) 


In addition, many corpora that are available such as the [BYU corpora](https://corpus.byu.edu/overview.asp) can be accessed via a web interface that have in-built concordancing functions.  
  
```{r coca1, echo=FALSE, out.width= "100%", out.extra='style="float:right; padding:2px"', fig.cap="\\label{fig:Fig2} Online concordances extracted from the COCA corpus that is part of the BYU corpora."}
knitr::include_graphics("https://slcladal.github.io/images/KwicCocaLanguage.png")
```

  
While these packages are very user-friendly, offer various additional functionalities, and almost everyone who is engaged in analyzing language has used concordance software, they all suffer from shortcomings that render R a viable alternative. Such issues include that these applications  
  
* are black boxes that researchers do not have full control over or do not know what is going on within the software

* they are not open source

* they hinder replication because the replications is more time consuming compared to analyses based on Notebooks.

* they are commonly not free-of charge or have other restrictions on use (a notable exception is *AntConc*)


R represents an alternative to ready-made concordancing applications because it:

* is extremely flexible and enables researchers to perform their entire analysis in a single environment

* allows full transparency and documentation as analyses can be based on Notebooks

* offer version control measures (this means that the specific versions of the involved software are traceable)

* makes research more replicable as entire analyses can be reproduced by simply running the Notebooks that the research is based on 

Especially the aspect that R enables full transparency and replicability is relevant given the ongoing *Replication Crisis* [@atlantic2018psychology; @fivethirtyeight2018replication; @noba2019replication; @physorg2018replication; @sciencealert2018replication]. The Replication Crisis is a ongoing methodological crisis primarily affecting parts of the social and life sciences beginning in the early 2010s [see also @fanelli2009fraud]. Replication is important so that other researchers, or the public for that matter, can see or, indeed, reproduce, exactly what you have done. Fortunately, R allows you to document your entire workflow as you can store everything you do in what is called a script or a notebook (in fact, this document was originally a R notebook). If someone is then interested in how you conducted your analysis, you can simply share this notebook or the script you have written with that person.

**Preparation and session set up**

This tutorial is based on R. If you have not installed R or are new to it, you will find an introduction to and more information how to use R [here](https://slcladal.github.io/intror.html). For this tutorials, we need to install certain *packages* from an R *library* so that the scripts shown below are executed without errors. Before turning to the code below, please install the packages by running the code below this paragraph. If you have already installed the packages mentioned below, then you can skip ahead and ignore this section. To install the necessary packages, simply run the following code - it may take some time (between 1 and 5 minutes to install all of the libraries so you do not need to worry if it takes some time).

```{r prep1, echo=T, eval = F}
# install packages
install.packages("quanteda")
install.packages("dplyr")
install.packages("stringr")
install.packages("writexl")
install.packages("here")
install.packages("flextable")
# install klippy for copy-to-clipboard button in code chunks
install.packages("remotes")
remotes::install_github("rlesur/klippy")
```


Now that we have installed the necessary packages, we activate them as shown below.

```{r prep2, message=FALSE, warning=FALSE, class.source='klippy'}
# activate packages
library(quanteda)
library(dplyr)
library(stringr)
library(writexl)
library(here)
library(flextable)
# activate klippy for copy-to-clipboard button
klippy::klippy()
```

Once you have installed R and RStudio and also initiated the session by executing the code shown above, you are good to go.

## Loading and processing textual data{-}

For this tutorial, we will use Lewis Caroll's  *Alice's Adventures in Wonderland*. You can use the code below to load this text into R (but you have to have access to the internet to do so).

```{r skwic2, warning=F, message=F}
text <- base::readRDS(url("https://slcladal.github.io/data/alice.rda", "rb"))
```

```{r skwic2b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
text %>%
  as.data.frame() %>%
  head(10) %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 text elements of the example text.")  %>%
  flextable::border_outer()
```

The table above shows that the example text requires formatting so that we can use it. Therefore, we collapse it into a single object (or text) and remove superfluous white spaces.


```{r skwic3, message=F, warning=F}
text <- text %>%
  # collapse lines into a single  text
  paste0(collapse = " ") %>%
  # remove superfluous white spaces
  str_squish()
```

```{r skwic3b, echo = F, message=FALSE, warning=FALSE}
# inspect data
text %>%
  substr(start=1, stop=1000) %>%
  as.data.frame() %>%
  flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 1000 characters of of the example text.")  %>%
  flextable::border_outer()
```

The result confirms that the entire text is now combined into a single character object. 

## Creating simple concordances{-}

Now that we have loaded the data, we can easily extract concordances using the `kwic` function from the `quanteda` package. The `kwic` function takes the text (`x`) and the search pattern (`pattern`) as it main arguments but it also allows the specification of the context window, i.e. how many words/elements are show to the left and right of the key word (we will go over this later on).

```{r basekwic2, message=F, warning=F}
mykwic <- quanteda::kwic(
  # define text
  quanteda::tokens(text), 
  # define search pattern
  pattern = "Alice") %>%
  # make it a data frame
  as.data.frame()
```


```{r basekwic2b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
mykwic %>%
  head(10) %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 concordances for the keyword Alice in our example text.")  %>%
  flextable::border_outer()
```


You will see that you get a warning stating that you should use `token` before extracting concordances. This can be done as shown below. Also, we can specify the package from which we want to use a function by adding the package name plus :: before the function (see below)

```{r basekwic3, message=F, warning=F}
mykwic <- quanteda::kwic(
  # define and tokenize text
  quanteda::tokens(text), 
  # define search pattern
  pattern = "alice") %>%
  # make it a data frame
  as.data.frame()
```

```{r basekwic3b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
mykwic %>%
  head(10) %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 concordances for the keyword *alice* in example text.")  %>%
  flextable::border_outer()
```

We can easily extract the frequency of the search term (*alice*) using the `nrow` or the `length` functions which provide the number of rows of a tables (`nrow`) or the length of a vector (`length`).

```{r basekwic4}
nrow(mykwic)
```


```{r basekwic5}
length(mykwic$keyword)
```

The results show that there are `r length(mykwic$keyword)` instances of the search term (*alice*) but we can also find out how often different variants (lower case versus upper case) of the search term were found using the `table` function. This is especially useful when searches involve many different search terms (while it is, admittedly, less useful in the present example). 

```{r basekwic6}
table(mykwic$keyword)
```

To get a better understanding of the use of a word, it is often useful to extract more context. This is easily done by increasing size of the context window. To do this, we specify the `window` argument of the `kwic` function. In the example below, we set the context window size to 10 words/elements rather than using the default (which is 5 word/elements).


```{r basekwic8, message=F, warning=F}
mykwic_longer <- quanteda::kwic(
  # define text
  quanteda::tokens(text), 
  # define search pattern
  pattern = "alice", 
  # define context window size
  window = 10) %>%
  # make it a data frame
  as.data.frame()
```


```{r skwic8b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
mykwic_longer %>%
  head(10) %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 concordances for the keyword *alice* in the example text with extended context (10 elements).")  %>%
  flextable::border_outer()
```

***

<div class="warning" style='padding:0.1em; background-color:#51247a; color:#f2f2f2'>
<span>
<p style='margin-top:1em; text-align:center'>
<b>EXERCISE TIME!</b></p>
<p style='margin-left:1em;'>
</p></span>
</div>

<div class="question">` 



1. Extract the first 10 concordances for the word *confused*.<br>

<details>
  <summary>Answer</summary>
  ```{r ex1_1, class.source = NULL, eval = T}
  kwic_confused <- quanteda::kwic(x = quanteda::tokens(text), pattern = "confused")
  # inspect
  kwic_confused %>%
  as.data.frame() %>%
  head(10)
  ```
</details>


2. How many instances are there of the word *wondering*?<br>

<details>
  <summary>Answer</summary>
  ```{r ex1_2, class.source = NULL, eval = T}
  quanteda::kwic(x = quanteda::tokens(text), pattern = "wondering") %>%
  as.data.frame() %>%
  nrow()
  ```
</details>


3. Extract concordances for the word *strange* and show the first 5 concordance lines.<br>

<details>
  <summary>Answer</summary>
  ```{r  ex1_3, class.source = NULL, eval = T}
  kwic_strange <- quanteda::kwic(x = quanteda::tokens(text), pattern = "strange")
  # inspect
  kwic_strange %>%
  as.data.frame() %>%
  head(5)
  ```
</details>

</div>`

***


## Exporting concordances {-}

To export or save a concordance table as an MS Excel spreadsheet, you can use the `write_xlsx` function from the `writexl` package as shown below. Be aware that we use the `here` function from the `here` package to define where we want to save the file (in this case this will be in the current working directory. If you work with Rproj files in RStudio - as you should - then the current working directory is the directory or folder where your Rproj file is).

```{r eval = F, warning=F, message=F}
write_xlsx(mykwic, here::here("mykwic.xlsx"))
```

## Extracting more than single words{-}

While extracting single words is very common, you may want to extract more than just one word. To extract phrases, all you need to so is to specify that the pattern you are looking for is a phrase, as shown below.

```{r multikwic2, message=FALSE, warning=FALSE}
kwic_pooralice <- quanteda::kwic(quanteda::tokens(text), 
                                 pattern = phrase("poor alice")) %>%
  as.data.frame()
```

```{r multikwic2b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
kwic_pooralice %>%
  head(10) %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 concordances for the keyphrase *poor alice* in the example text.")  %>%
  flextable::border_outer()
```

You may also want to extract more or less fixed patterns rather than exact words or phrases. To search for patterns that allow variation rather than specific, exactly-defined words, you need to include regular expressions in your search pattern. 

***

<div class="warning" style='padding:0.1em; background-color:#51247a; color:#f2f2f2'>
<span>
<p style='margin-top:1em; text-align:center'>
<b>EXERCISE TIME!</b></p>
<p style='margin-left:1em;'>
</p></span>
</div>

<div class="question">` 



1. Extract the first 10 concordances for the phrase *the hatter*.<br>

<details>
  <summary>Answer</summary>
  ```{r ex2_1, class.source = NULL, eval = T}
  kwic_thehatter <- quanteda::kwic(x = quanteda::tokens(text), pattern = phrase("the hatter"))
  # inspect
  kwic_thehatter %>%
  as.data.frame() %>%
  head(10)
  ```
</details>


2. How many instances are there of the phrase *the hatter*?<br>

<details>
  <summary>Answer</summary>
  ```{r ex2_2, class.source = NULL, eval = T}
  kwic_thehatter %>%
  as.data.frame() %>%
  nrow()
  ```
</details>


3. Extract concordances for the phrase *the cat* and show the first 5 concordance lines.<br>

<details>
  <summary>Answer</summary>
  ```{r  ex2_3, class.source = NULL, eval = T}
  kwic_thecat <- quanteda::kwic(x = quanteda::tokens(text), pattern = phrase("the cat"))
  # inspect
  kwic_thecat %>%
  as.data.frame() %>%
  head(5)
  ```
</details>

</div>`

***

## Searches using regular expressions{-}

Regular expressions allow you to search for abstract patterns rather than concrete words or phrases which provides you with an extreme flexibility in what you can retrieve. A regular expression (in short also called *regex* or *regexp*) is a special sequence of characters that stand for are that describe a pattern. You can think of regular expressions as very powerful combinations of wildcards or as wildcards on steroids. For example, the sequence `[a-z]{1,3}` is a regular expression that stands for one up to three lower case characters and if you searched for this regular expression, you would get, for instance, *is*, *a*, *an*, *of*, *the*, *my*, *our*, *etc*, and many other short words as results.

There are three basic types of regular expressions:

* regular expressions that stand for individual symbols and determine frequencies

* regular expressions that stand for classes of symbols

* regular expressions that stand for structural properties

The regular expressions below show the first type of regular expressions, i.e. regular expressions that stand for individual symbols and determine frequencies.

```{r regex05, echo=F, eval = T, message=FALSE, warning=FALSE}
symbols1 <- c(fixed("?"), fixed("*"), fixed("+"), "{n}", "{n,}", "{n,m}")
explanation1 <- c("The preceding item is optional and will be matched at most once", "The preceding item will be matched zero or more times", "The preceding item will be matched one or more times", "The preceding item is matched exactly n times", "The preceding item is matched n or more times", "The preceding item is matched at least n times, but not more than m times")
example1 <- c("walk[a-z]? = walk, walks", 
             "walk[a-z]* = walk, walks, walked, walking", 
             "walk[a-z]+ = walks, walked, walking", 
             "walk[a-z]{2} = walked", 
             "walk[a-z]{2,} = walked, walking", 
             "walk[a-z]{2,3} = walked, walking")
df_regex <- data.frame(symbols1, explanation1, example1)
colnames(df_regex) <- c("RegEx Symbol/Sequence", "Explanation", "Example")
```

```{r regex05b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
df_regex %>%
  as.data.frame() %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .75, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "Regular expressions that stand for individual symbols and determine frequencies.")  %>%
  flextable::border_outer()
```


The regular expressions below show the second type of regular expressions, i.e. regular expressions that stand for classes of symbols.

```{r regex03, echo=F, eval = T, message=FALSE, warning=FALSE}
symbols2 <- c("[ab]", "[AB]", "[12]", "[:digit:]", "[:lower:]", "[:upper:]", "[:alpha:]", "[:alnum:]", "[:punct:]", "[:graph:]", "[:blank:]", "[:space:]", "[:print:]")
explanations2 <- c("lower case a and b", 
                   "upper case a and b", 
                   "digits 1 and 2", 
                   "digits: 0 1 2 3 4 5 6 7 8 9", 
                   "lower case characters: a–z", 
                   "upper case characters: A–Z", 
                   "alphabetic characters: a–z and A–Z", 
                   "digits and alphabetic characters", 
                   "punctuation characters: . , ; etc.", 
                   "graphical characters: [:alnum:] and [:punct:]", 
                   "blank characters: Space and tab", 
                   "space characters: Space, tab, newline, and other space characters", 
                   "printable characters: [:alnum:], [:punct:] and [:space:]")
df_regex <- data.frame(symbols2, explanations2)
colnames(df_regex) <- c("RegEx Symbol/Sequence", "Explanation")
```


```{r regex07b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
df_regex %>%
  as.data.frame() %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .75, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "Regular expressions that stand for classes of symbols.")  %>%
  flextable::border_outer()
```

The regular expressions that denote classes of symbols are enclosed in `[]` and `:`. The last type of regular expressions, i.e. regular expressions that stand for structural properties are shown below.

```{r regex09, echo=F, eval = T, message=FALSE, warning=FALSE}
symbols3 <- c(fixed("\\\\w"), fixed("\\\\W"), fixed("\\\\s"), fixed("\\\\S"), 
              fixed("\\\\d"), fixed("\\\\D"), fixed("\\\\b"), fixed("\\\\B"), 
              fixed("<"), fixed(">"), fixed("^"), fixed("$"))
explanations3 <- c("Word characters: [[:alnum:]_]",
                   "No word characters: [^[:alnum:]_]",
                   "Space characters: [[:blank:]]",
                   "No space characters: [^[:blank:]]",
                   "Digits: [[:digit:]]",
                   "No digits: [^[:digit:]]",
                   "Word edge",
                   "No word edge",
                   "Word beginning",
                   "Word end",
                   "Beginning of a string",
                   "End of a string")
df_regex <- data.frame(symbols3, explanations3)
colnames(df_regex) <- c("RegEx Symbol/Sequence", "Explanation")
```

```{r regex09b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
df_regex %>%
  as.data.frame() %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .5, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "Regular expressions that stand for structural properties.")  %>%
  flextable::border_outer()
```

To include regular expressions in your KWIC searches, you include them in your search pattern and set the argument `valuetype` to `"regex"`. The search pattern `"\\balic.*|\\bhatt.*"` retrieves elements that contain `alic` and `hatt` followed by any characters and where the `a` in `alic` and the `h` in `hatt` are at a word boundary, i.e. where they are the first letters of a word. Hence, our search would not retrieve words like *malice* or *shatter*. The `|` is an operator (like `+`, `-`, or `*`) that stands for *or*.

```{r rkwic6, message=FALSE, warning=FALSE}
# define search patterns
patterns <- c("\\balic.*|\\bhatt.*")
kwic_regex <- quanteda::kwic(
  # define text
  quanteda::tokens(text), 
  # define search pattern
  patterns, 
  # define valuetype
  valuetype = "regex") %>%
  # make it a data frame
  as.data.frame()
```

```{r rkwic6b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
kwic_regex %>%
  head(10) %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 concordances for the regular expression \\balic.* and \\bhatt.*.")  %>%
  flextable::border_outer()
```


***

<div class="warning" style='padding:0.1em; background-color:#51247a; color:#f2f2f2'>
<span>
<p style='margin-top:1em; text-align:center'>
<b>EXERCISE TIME!</b></p>
<p style='margin-left:1em;'>
</p></span>
</div>

<div class="question">` 



1. Extract the first 10 concordances for words containing *exu*.<br>

<details>
  <summary>Answer</summary>
  ```{r ex3_1, class.source = NULL, eval = T}
  kwic_exu <- quanteda::kwic(x = quanteda::tokens(text), pattern = ".*exu.*", valuetype = "regex")
  # inspect
  kwic_exu %>%
  as.data.frame() %>%
  head(10)
  ```
</details>


2. How many instances are there of words beginning with *pit*?<br>

<details>
  <summary>Answer</summary>
  ```{r ex3_2, class.source = NULL, eval = T}
  quanteda::kwic(x = quanteda::tokens(text), pattern = "\\bpit.*", valuetype = "regex") %>%
  as.data.frame() %>%
  nrow()
  ```
</details>


3. Extract concordances for words ending with *ption* and show the first 5 concordance lines.<br>

<details>
  <summary>Answer</summary>
  ```{r  ex3_3, class.source = NULL, eval = T}
  quanteda::kwic(x = quanteda::tokens(text), pattern = "ption\\b", valuetype = "regex")  %>%
  as.data.frame() %>%
  head(5)
  ```
</details>

</div>`

***


## Piping concordances{-}

Quite often, we only want to retrieve patterns if they occur in a certain context. For instance, we might be interested in instances of *selection* but only if the preceding word is *natural*. Such conditional concordances could be extracted using regular expressions but they are easier to retrieve by piping. Piping is done using the `%>%` function from the `dplyr` package and the piping sequence can be translated as *and then*. We can then filter those concordances that contain *natural* using the `filter` function from the `dplyr` package. Note the the `$` stands for the end of a string so that *natural$* means that *natural* is the last element in the string that is preceding the keyword.

```{r pipekwic7, echo=T, eval = T, message=FALSE, warning=FALSE}
kwic_pipe <- quanteda::kwic(x = quanteda::tokens(text), pattern = "alice") %>%
  as.data.frame() %>%
  dplyr::filter(stringr::str_detect(pre, "poor$|little$"))
```

```{r pipekwic7b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
kwic_pipe %>%
  head(10) %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 concordances for instances of *alice* that are preceeded by *poor* or *little*.")  %>%
  flextable::border_outer()
```

Piping is a very useful helper function and it is very frequently used in R - not only in the context of text processing but in all data science related domains.

## Arranging concordances and adding frequency information{-}

When inspecting concordances, it is useful to re-order the concordances so that they do not appear in the order that they appeared in the text or texts but by the context. To reorder concordances, we can use the `arrange` function from the `dplyr` package which takes the column according to which we want to re-arrange the data as it main argument. 

In the example below, we extract all instances of *alice* and then arrange the instances according to the content of the `post` column in alphabetical.

```{r orderkwic2, echo=T, eval = T, message=FALSE, warning=FALSE}
kwic_ordered <- quanteda::kwic(x = quanteda::tokens(text), pattern = "alice") %>%
  as.data.frame() %>%
  dplyr::arrange(post)
```

```{r orderkwic2b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
kwic_ordered %>%
  head(10) %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 re-ordered concordances for instances of alice.")  %>%
  flextable::border_outer()
```


Arranging concordances according to alphabetical properties may, however, not be the most useful option. A more useful option may be to arrange concordances according to the frequency of co-occurring terms or collocates. In order to do this, we need to extract the co-occurring words and calculate their frequency. We can do this by combining the  `mutate`, `group_by`, `n()` functions from the `dplyr` package with the `str_remove_all` function from the `stringr` package. Then, we arrange the concordances by the frequency of the collocates in descending order (that is why we put a `-` in the arrange function). In order to do this, we need to 

1. create a new variable or column which represents the word that co-occurs with, or, as in the example below, immediately follows the search term. In the example below, we use the `mutate` function to create a new column called `post_word`. We then use the `str_remove_all` function to remove everything except for the word that immediately follows the search term (we simply remove everything and including a white space).

2. group the data by the word that immediately follows the search term.

3. create a new column called `post_word_freq` which represents the frequencies of all the words that immediately follow the search term.

4. arrange the concordances by the frequency of the collocates in descending order.

```{r orderkwic4, echo=T, eval = T, message=FALSE, warning=FALSE}
kwic_ordered_coll <- quanteda::kwic(
  # define text
  x = quanteda::tokens(text), 
  # define search pattern
  pattern = "alice") %>%
  # make it a data frame
  as.data.frame() %>%
  # extract word following the keyword
  dplyr::mutate(post_word = str_remove_all(post, " .*")) %>%
  # group following words
  dplyr::group_by(post_word) %>%
  # extract frequencies of the following words
  dplyr::mutate(post_word_freq = n()) %>%
  # arrange/order by the frequency of the following word
  dplyr::arrange(-post_word_freq)
```

```{r orderkwic4b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
kwic_ordered_coll %>%
  head(10) %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 re-ordered concordances for instances of natural.")  %>%
  flextable::border_outer()
```

We add more columns according to which we could arrange the concordance following the same schema. For example, we could add another column that represented the frequency of words that immediately preceded the search term and then arrange according to this column.

## Ordering by subsequent elements{-}

In this section, we will extract the three words following the keyword (*alice*) and organize the concordances by the frequencies of the following words. We begin by inspecting the first 6 lines of the concordance of *selection*.

```{r, message=FALSE, warning=FALSE}
head(mykwic)
```

Next, we take the concordances and create a clean post column that is all in lower case and that does not contain any punctuation.


```{r, message=FALSE, warning=FALSE}
mykwic %>%
  # create new CleanPost
  dplyr::mutate(CleanPost = stringr::str_remove_all(post, "[:punct:]"),
                CleanPost = stringr::str_squish(CleanPost),
                CleanPost = tolower(CleanPost))-> mykwic_following
# inspect
head(mykwic_following)
```

In a next step, we extract the 1^st^, 2^nd^, and 3^rd^ words following the keyword.


```{r, message=FALSE, warning=FALSE}
mykwic_following %>%
  # extract first element after keyword
  dplyr::mutate(FirstWord = stringr::str_remove_all(CleanPost, " .*")) %>%
  # extract second element after keyword
  dplyr::mutate(SecWord = stringr::str_remove(CleanPost, ".*? "),
                SecWord = stringr::str_remove_all(SecWord, " .*")) %>%
  # extract third element after keyword
  dplyr::mutate(ThirdWord = stringr::str_remove(CleanPost, ".*? "),
                ThirdWord = stringr::str_remove(ThirdWord, ".*? "),
                ThirdWord = stringr::str_remove_all(ThirdWord, " .*")) -> mykwic_following
# inspect
head(mykwic_following)
```

Next, we calculate the frequencies of the subsequent words and order in descending order from the  1^st^ to the 3^rd^ word following the keyword.

```{r, message=FALSE, warning=FALSE}
mykwic_following %>%
  # calculate frequency of following words
  # 1st word
  dplyr::group_by(FirstWord) %>%
  dplyr::mutate(FreqW1 = n()) %>%
  # 2nd word
  dplyr::group_by(SecWord) %>%
  dplyr::mutate(FreqW2 = n()) %>%
  # 3rd word
  dplyr::group_by(ThirdWord) %>%
  dplyr::mutate(FreqW3 = n()) %>%
  # ungroup
  dplyr::ungroup() %>%
  # arrange by following words
  dplyr::arrange(-FreqW1, -FreqW2, -FreqW3) -> mykwic_following
# inspect results
head(mykwic_following, 10)
```

The results now show the concordance arranged by the frequency of the words following the keyword.

## Concordances from transcriptions{-}

As many analyses use transcripts as their primary data and because transcripts have features that require additional processing, we will now perform concordancing based on on transcripts. As a first step, we load five example transcripts that represent the first five files from the Irish component of the [International Corpus of English](https://www.ice-corpora.uzh.ch/en.html).

```{r trans2, echo=T, eval = T, message=FALSE, warning=FALSE}
# define corpus files
files <- paste("https://slcladal.github.io/data/ICEIrelandSample/S1A-00", 1:5, ".txt", sep = "")
# load corpus files
transcripts <- sapply(files, function(x){
  x <- readLines(x)
  })
```


```{r trans2b, echo = F, message=FALSE, warning=FALSE}
# inspect data
transcripts[[1]][1:10] %>%
  as.data.frame() %>%
  flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 utterances in the sample transcripts.")  %>%
  flextable::border_outer()
```

The first ten lines shown above let us know that, after the header (`<S1A-001 Riding>`) and the symbol which indicates the start of the transcript (`<I>`), each utterance is preceded by a sequence which indicates the section, file, and speaker (e.g. `<S1A-001$A>`). The first utterance is thus uttered by speaker `A` in file `001` of section `S1A`. In addition, there are several sequences that provide meta-linguistic information which indicate the beginning of a speech unit (`<#>`), pauses (`<,>`), and laughter (`<&> laughter </&>`).

To perform the concordancing, we need to change the format of the transcripts because the `kwic` function only works on character, corpus, tokens object- in their present form, the transcripts represent a list which contains vectors of strings. To change the format, we collapse the individual utterances into a single character vector for each transcript.

```{r trans4, echo=T, eval = T, message=FALSE, warning=FALSE}
transcripts_collapsed <- sapply(files, function(x){
  # read-in text
  x <- readLines(x)
  # paste all lines together
  x <- paste0(x, collapse = " ")
  # remove superfluous white spaces
  x <- str_squish(x)
})
```

```{r trans4b, echo = F, message=FALSE, warning=FALSE}
# inspect data
transcripts_collapsed %>%
    substr(start=1, stop=500) %>%
  as.data.frame() %>%
  flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 500 characters of the collapsed sample transcripts.")  %>%
  flextable::border_outer()
```


We can now extract the concordances.

```{r trans6, echo=T, eval = T, message=FALSE, warning=FALSE}
kwic_trans <- quanteda::kwic(
  # tokenize transcripts
  quanteda::tokens(transcripts_collapsed), 
  # define search pattern
  pattern = phrase("you know")) %>%
  # make it a data frame
  as.data.frame()

```

```{r trans6b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
kwic_trans %>%
  head(10) %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 concordances for you know in three example transcripts.")  %>%
  flextable::border_outer()
```

The results show that each non-alphanumeric character is counted as a single word which reduces the context of the keyword substantially. Also, the *docname* column contains the full path to the data which make it hard to parse the content of the table. To address the first issue, we specify the tokenizer that we will use to not disrupt the annotation too much. In addition, we clean the *docname* column and extract only the file name. Lastly, we will expand the context window to 10 so that we have a better understanding of the context in which the phrase was used.

```{r trans8, echo=T, eval = T, message=FALSE, warning=FALSE}
kwic_trans <- quanteda::kwic(
  # tokenize transcripts
  quanteda::tokens(transcripts_collapsed, what = "fasterword"), 
  # define search
  pattern = phrase("you know"),
  # extend context
  window = 10) %>%
  # make it a data frame
  as.data.frame() %>%
  # clean docnames
  dplyr::mutate(docname = str_replace_all(docname, ".*/([A-Z][0-9][A-Z]-[0-9]{1,3}).txt", "\\1"))
```

```{r trans8b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
kwic_trans %>%
  head(10) %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 concordances for you know in three example transcripts.")  %>%
  flextable::border_outer()
```


Extending the context can also be used to identify the speaker that has uttered the search pattern that we are interested in. We will do just that as this is a common task in linguistics analyses.

To extract speakers, we need to follow these steps:

1. Create normal concordances of the pattern that we are interested in.

2. Generate concordances of the pattern that we are interested in with a substantially enlarged context window size.

3. Extract the speakers from the enlarged context window size.

4. Add the speakers to the normal concordances using the `left-join` function from the `dplyr` package.

```{r trans10,  message=FALSE, warning=FALSE}
kwic_normal <- quanteda::kwic(
  # tokenize transcripts
  quanteda::tokens(transcripts_collapsed, what = "fasterword"), 
  # define search
  pattern = phrase("you know")) %>%
  as.data.frame()
kwic_speaker <- quanteda::kwic(
    # tokenize transcripts
  quanteda::tokens(transcripts_collapsed, what = "fasterword"), 
  # define search
  pattern = phrase("you know"), 
  # extend search window
  window = 500) %>%
  # convert to data frame
  as.data.frame() %>%
  # extract speaker (comes after $ and before >)
  dplyr::mutate(speaker = stringr::str_replace_all(pre, ".*\\$(.*?)>.*", "\\1")) %>%
  # extract speaker
  dplyr::pull(speaker)
# add speaker to normal kwic
kwic_combined <- kwic_normal %>%
  # add speaker
  dplyr::mutate(speaker = kwic_speaker) %>%
  # simplify docname
  dplyr::mutate(docname = stringr::str_replace_all(docname, ".*/([A-Z][0-9][A-Z]-[0-9]{1,3}).txt", "\\1")) %>%
  # remove superfluous columns
  dplyr::select(-to, -from, -pattern)
```

```{r trans10b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
kwic_combined %>%
  as.data.frame() %>%
  head(10) %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 concordances for you know in three example transcripts with speakers that uttered them..")  %>%
  flextable::border_outer()
```

The resulting table shows that we have successfully extracted the speakers (identified by the letters in the `speaker` column) and cleaned the file names (in the `docnames` column).

## Customizing concordances{-}

As R represents a fully-fledged programming environment, we can, of course, also write our own, customized concordance function. The code below shows how you could go about doing so. Note, however, that this function only works if you enter more than a single file. 

```{r customkwic2, message=F, warning=F}
mykwic <- function(txts, pattern, context) {
  # activate packages
  require(stringr)
  # list files
  txts <- txts[stringr::str_detect(txts, pattern)]
  conc <- sapply(txts, function(x) {
    # determine length of text
        lngth <- as.vector(unlist(nchar(x)))
    # determine position of hits
    idx <- str_locate_all(x, pattern)
    idx <- idx[[1]]
    ifelse(nrow(idx) >= 1, idx <- idx, return(NA))
    # define start position of hit
    token.start <- idx[,1]
    # define end position of hit
    token.end <- idx[,2]
    # define start position of preceding context
    pre.start <- ifelse(token.start-context < 1, 1, token.start-context)
    # define end position of preceding context
    pre.end <- token.start-1
    # define start position of subsequent context
    post.start <- token.end+1
    # define end position of subsequent context
    post.end <- ifelse(token.end+context > lngth, lngth, token.end+context)
    # extract the texts defined by the positions
    PreceedingContext <- substring(x, pre.start, pre.end)
    Token <- substring(x, token.start, token.end)
    SubsequentContext <- substring(x, post.start, post.end)
    Id <- 1:length(Token)
    conc <- cbind(Id, PreceedingContext, Token, SubsequentContext)
    # return concordance
    return(conc)
    })
  concdf <- do.call(rbind, conc) %>%
    as.data.frame()
  return(concdf)
}
```

We can now try if this function works by searching for the sequence *you know* in the transcripts that we have loaded earlier. One difference between the `kwic` function provided by the `quanteda` package and the customized concordance function used here is that the `kwic` function uses the number of words to define the context window, while the `mykwic` function uses the number of characters or symbols instead (which is why we use a notably higher number to define the context window).

```{r customkwic4, message=F, warning=F}
kwic_youknow <- mykwic(transcripts_collapsed, "you know", 50)
```


```{r customkwic4b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
kwic_youknow %>%
  as.data.frame() %>%
  head() %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 6 concordances for *you know* extracted using the mykwic function.")  %>%
  flextable::border_outer()
```

As this concordance function only works for more than one text, we split the text into chapters and assign each section a name.


```{r customkwic6, message=FALSE, warning=FALSE}
# read in text
text_split <- text %>%
  stringr::str_squish() %>%
  stringr::str_split("[CHAPTER]{7,7} [XVI]{1,7}\\. ") %>%
  unlist()
text_split <- text_split[which(nchar(text_split) > 2000)]
# add names
names(text_split) <- paste0("text", 1:length(text_split))
# inspect data
nchar(text_split)
```

Now that we have named elements, we can search for the pattern *poor alice*. We also need to clean the concordance as some sections do not contain any instances of the search pattern. To clean the data, we select only the columns `File`, `PreceedingContext`, `Token`, and `SubsequentContext` and then remove all rows where information is missing. 

```{r customkwic8, message=FALSE, warning=FALSE}
mykwic_pooralice <- mykwic(text_split, "poor Alice", 50)
```

```{r customkwic8b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
mykwic_pooralice %>%
  as.data.frame() %>%
  head() %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 6 concordances of *poor alice* extracted using the mykwic function.")  %>%
  flextable::border_outer()
```

You can go ahead and modify the customized concordance function to suit your needs. 


# Citation & Session Info {-}

Schweinberger, Martin. 2023. *Concordancing with R*. Brisbane: The Language Technology and Data Analysis Laboratory (LADAL). url: https://ladal.edu.au/kwics.html (Version 2023.09.23).


```
@manual{schweinberger2023kwics,
  author = {Schweinberger, Martin},
  title = {Concordancing with R},
  note = {https://ladal.edu.au/kwics.html},
  year = {2023},
  organization = {The Language Technology and Data Analysis Laboratory (LADAL)},
  address = {Brisbane},
  edition = {2023.09.23}
}
```


```{r fin}
sessionInfo()
```




***

[Back to top](#introduction)

[Back to HOME](https://ladal.edu.au)

***

# References {-}

