---
title: "Concordancing with R"
author: "Martin Schweinberger"
date: ""
output:
  bookdown::html_document2: default
bibliography: bibliography.bib
link-citations: yes
---

```{r uq1, echo=F, fig.cap="", message=FALSE, warning=FALSE, out.width='100%'}
knitr::include_graphics("https://slcladal.github.io/images/uq1.jpg")
knitr::opts_chunk$set(warning = FALSE, message = FALSE) 
```

# Introduction{-}

This tutorial introduces how to extract concordances and keyword-in-context (KWIC) displays with R using the `quanteda` package [@quanteda2018package]. 

<div class="warning" style='padding:0.5em; background-color:rgba(215,209,204,.3); color:#51247a'>
<span>
<p style='margin-top:1em; text-align:center'>
Please cite as: <br>Schweinberger, Martin. 2024. *Concordancing with R*. Brisbane: The Language Technology and Data Analysis Laboratory (LADAL). url: https://ladal.edu.au/kwics.html (Version 2024.05.07).<br>
</p>
</span>
</div>


```{r diff, echo=FALSE, out.width= "15%", out.extra='style="float:right; padding:10px"'}
knitr::include_graphics("https://slcladal.github.io/images/gy_chili.jpg")
```

This tutorial is aimed at beginners and intermediate users of R with the aim of showcasing how to extract words and phrases from textual data and how to process the resulting concordances using R. The aim is not to provide a fully-fledged analysis but rather to show and exemplify selected useful methods associated with concordancing. 



<div class="warning" style='padding:0.1em; background-color:rgba(215,209,204,.3); color:#51247a'>
<span>
<p style='margin-top:1em; text-align:center'>
To be able to follow this tutorial, we suggest you check out and familiarize yourself with the content of the following **R Basics** tutorials:<br>
</p>
<p style='margin-top:1em; text-align:left'>
<ul>
  <li>[Getting started with R](https://ladal.edu.au/intror.html) </li>
  <li>[Loading, saving, and generating data in R](https://ladal.edu.au/load.html) </li>
  <li>[String Processing in R](https://ladal.edu.au/string.html) </li>
  <li>[Regular Expressions in R](https://ladal.edu.au/regex.html) </li>
</ul>
</p>
<p style='margin-top:1em; text-align:center'>
Click [**here**](https://ladal.edu.au/content/kwics.Rmd)^[If you want to render the R Notebook on your machine, i.e. knitting the document to html or a pdf, you need to make sure that you have R and RStudio installed and you also need to download the [**bibliography file**](https://slcladal.github.io/content/bibliography.bib) and store it in the same folder where you store the Rmd file.] to download the **entire R Notebook** for this tutorial.<br><br>
[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/SLCLADAL/interactive-notebooks-environment/main?urlpath=git-pull%3Frepo%3Dhttps%253A%252F%252Fgithub.com%252FSLCLADAL%252Finteractive-notebooks%26urlpath%3Dlab%252Ftree%252Finteractive-notebooks%252Fnotebooks%252Fkwics_cb.ipynb%26branch%3Dmain)<br>
Click [**here**](https://mybinder.org/v2/gh/SLCLADAL/interactive-notebooks-environment/main?urlpath=git-pull%3Frepo%3Dhttps%253A%252F%252Fgithub.com%252FSLCLADAL%252Finteractive-notebooks%26urlpath%3Dlab%252Ftree%252Finteractive-notebooks%252Fnotebooks%252Fkwics_cb.ipynb%26branch%3Dmain)  to open a Jupyter notebook that allows you to follow this tutorial interactively. This means that you can execute, change, and edit the code used in this tutorial to help you better understand how the code shown here works (make sure you run all code chunks in the order in which they appear to avoid running into errors).  </p> <br>
</p>
<p style='margin-left:1em;'>
</p></span>
</div>

<br>


<div class="warning" style='padding:0.1em; background-color:#51247a; color:#f2f2f2'>
<span>
<p style='margin-top:1em; text-align:center'>
**CONCORDANCING TOOL** 
</p>
<p style='margin-top:1em; text-align:center'>
Click on this [![Binder](https://mybinder.org/badge_logo.svg)](https://binderhub.atap-binder.cloud.edu.au/v2/gh/SLCLADAL/interactive-notebooks-environment/main?urlpath=git-pull%3Frepo%3Dhttps%253A%252F%252Fgithub.com%252FSLCLADAL%252Finteractive-notebooks%26urlpath%3Dlab%252Ftree%252Finteractive-notebooks%252Fnotebooks%252Fkwictool.ipynb%26branch%3Dmain) badge to open an notebook-based tool <br>that allows you to upload your own text(s), perform concordancing on them, and download the resulting kwic(s). </p><br>
<p style='margin-left:1em;'>
</p></span>
</div>

<br>

Concordancing is a fundamental tool in language sciences, involving the extraction of words from a given text or texts [@lindquist2009corpus 5]. Typically, these extractions are displayed through keyword-in-context displays (KWICs), where the search term, also referred to as the *node word*, is showcased within its surrounding context, comprising both preceding and following words. Concordancing serves as a cornerstone for textual analyses, often serving as the initial step towards more intricate examinations of language data [@stafanowitsch2020corpus]. Their significance lies in their capacity to provide insights into how words or phrases are utilized, their frequency of occurrence, and the contexts in which they appear. By facilitating the examination of a word or phrase's contextual usage and offering frequency data, concordances empower researchers to delve into collocations or the collocational profiles of words and phrases [@stafanowitsch2020corpus 50-51]. 

Examples of Concordancing Applications:

1. **Linguistic Research**: Linguists often use concordances to explore the usage patterns of specific words or phrases in different contexts. For instance, researchers might use concordancing to analyze the various meanings and connotations of a polysemous word across different genres of literature.

2. **Language Teaching**: Concordancing can be a valuable tool in language teaching. Educators can create concordances to illustrate how vocabulary words are used in authentic contexts, helping students grasp their usage nuances and collocational patterns.

3. **Stylistic Analysis**: Literary scholars use concordances to conduct stylistic analyses of texts. By examining how certain words or phrases are employed within a literary work, researchers can gain insights into the author's writing style, thematic concerns, and narrative techniques.

4. **Translation Studies**: Concordancing is also applied in translation studies to analyze the usage of specific terms or expressions in source and target languages. Translators use concordances to identify appropriate equivalents and ensure accurate translation of idiomatic expressions and collocations.

5. **Lexicography**: Lexicographers employ concordances to compile and refine dictionaries. By examining the contexts in which words appear, lexicographers can identify common collocations and usage patterns, informing the creation of more comprehensive and accurate lexical entries.

These examples underscore the usefulness of concordancing across various disciplines within the language sciences, showcasing its role in facilitating nuanced analyses and insights into language usage.

```{r antc1, echo=FALSE, out.width= "100%", out.extra='style="float:left; padding:2px"', fig.cap="\\label{fig:Fig1} Concordances in AntConc."}
knitr::include_graphics("https://slcladal.github.io/images/AntConcConcordance.png")
```

<br>

There are various very good software packages that can be used to create concordances - both for offline use. There are, for example

+ [*AntConc*](https://www.laurenceanthony.net/software/antconc/) [@anthony2004antconc]
+ [*SketchEngine*](https://www.sketchengine.eu/) [@kilgarriff2004itri]
+ [*MONOCONC*](https://www.monoconc.com/) [@barlow1999monoconc]
+ [*ParaConc*](https://paraconc.com/)) [@barlow2002paraconc] 
+ [*Web Concordancers*](https://lextutor.ca/conc/)
+ [*CorpusMate*](https://corpusmate.com/) [@corpusmate]


In addition, many corpora that are available such as the [BYU corpora](https://corpus.byu.edu/overview.asp) can be accessed via a web interface that have in-built concordancing functions.  
  
```{r coca1, echo=FALSE, out.width= "100%", out.extra='style="float:right; padding:2px"', fig.cap="\\label{fig:Fig2} Online concordances extracted from the COCA corpus that is part of the BYU corpora."}
knitr::include_graphics("https://slcladal.github.io/images/KwicCocaLanguage.png")
```

 While many concordance software packages boast user-friendly interfaces and additional functionalities, their inherent limitations make R a compelling alternative. These shortcomings include:

* **Lack of Transparency**: Traditional concordance software often operates as black boxes, leaving researchers without full control or insight into the inner workings of the software. This opacity can compromise the reproducibility and transparency of analyses, as researchers are unable to fully understand or document the analytical process.

* **Closed Source**: The proprietary nature of many concordance software packages restricts access to their source code, hindering collaborative development and innovation within the research community. In contrast, R's open-source framework encourages transparency, collaboration, and customization, empowering researchers to tailor analyses to their specific needs.

* **Replication Challenges**: Replicating analyses conducted using traditional concordance software can be time-consuming and labor-intensive, particularly when compared to analyses documented in Notebooks. R's integrated scripting environment facilitates reproducibility by allowing researchers to document their entire workflow in a clear, executable format, streamlining the replication process and enhancing research transparency.

* **Cost and Restrictions**: Many concordance software packages are not free-of-charge or impose usage restrictions, limiting accessibility for researchers with budget constraints or specific licensing requirements. While exceptions like *AntConc* exist, the prevalence of cost barriers or usage restrictions underscores the importance of open-source alternatives like R, which offer greater accessibility and flexibility for researchers across diverse settings.

By addressing these key limitations, R emerges as a robust and versatile platform for concordancing, offering researchers greater transparency, flexibility, and accessibility. As such, R offers a compelling alternative to off-the-shelf concordancing applications for several reasons:

* **Flexibility**: R provides unparalleled flexibility, allowing researchers to conduct their entire analysis within a single environment. This versatility enables users to tailor their analysis precisely to their research needs, incorporating custom functions and scripts as required.

* **Transparency and Documentation**: One of R's strengths lies in its capacity for full transparency and documentation. Analyses can be documented and presented in R Notebooks, providing a comprehensive record of the analytical process. This transparency ensures that every step of the analysis is clearly documented, enhancing the reproducibility of research findings.

* **Version Control**: R offers robust version control measures, enabling researchers to track and trace specific versions of software and packages used in their analysis. This ensures that analyses are conducted using consistent software environments, minimizing the risk of discrepancies or errors due to software updates or changes.

* **Replicability**: Replicability is paramount in scientific research, particularly in light of the ongoing Replication Crisis [@atlantic2018psychology; @fivethirtyeight2018replication; @noba2019replication; @physorg2018replication; @sciencealert2018replication]. By conducting analyses within R and documenting the entire workflow in scripts or Notebooks, researchers facilitate easy replication of their analyses by others. This not only enhances the credibility and robustness of research findings but also fosters greater collaboration and transparency within the scientific community.

The Replication Crisis, which has affected various fields since the early 2010s [@fanelli2009fraud], underscores the importance of transparent and replicable research practices. R's capability to document workflows comprehensively in scripts or Notebooks addresses this challenge directly. Researchers can readily share their scripts or Notebooks with others, enabling full transparency and facilitating replication of analyses. This emphasis on transparency and replicability aligns closely with best practices in scientific research, ensuring that findings can be scrutinized, validated, and built upon by the broader research community.

**Preparation and session set up**

This tutorial is conducted within the R environment. If you're new to R or haven't installed it yet, you can find an introduction to R and further instructions on how to use it [here](https://ladal.edu.au/intror.html). To ensure smooth execution of the scripts provided in this tutorial, it's necessary to install specific packages from the R library. Before proceeding to the code examples, please ensure you've installed these packages by running the code provided below. If you've already installed the required packages, feel free to skip ahead and disregard this section. To install the necessary packages, simply execute the following code. Please note that installation may take some time (usually between 1 and 5 minutes), so there's no need to be concerned if it takes a while.

```{r prep1, echo=T, eval = F}
# install packages
install.packages("quanteda")
install.packages("dplyr")
install.packages("stringr")
install.packages("writexl")
install.packages("here")
install.packages("flextable")
# install klippy for copy-to-clipboard button in code chunks
install.packages("remotes")
remotes::install_github("rlesur/klippy")
```

Now that we've installed the required packages, let's activate them using the following code snippet:

```{r prep2, message=FALSE, warning=FALSE, class.source='klippy'}
# activate packages
library(quanteda)
library(dplyr)
library(stringr)
library(writexl)
library(here)
library(flextable)
# activate klippy for copy-to-clipboard button
klippy::klippy()
```

Once you've initiated the session by executing the provided code, you're ready to proceed.

## Loading and Processing Text{-}

For this tutorial, we will use Lewis Carroll's classic novel *Alice's Adventures in Wonderland* as our primary text dataset. This whimsical tale follows the adventures of Alice as she navigates a fantastical world filled with peculiar characters and surreal landscapes. 

**Loading Text (from Project Gutenberg)**

To access the text within R, you can use the following code snippet, ensuring you have an active internet connection:

```{r skwic2, warning=F, message=F}
# Load Alice's Adventures in Wonderland text into R
rawtext <- readLines("https://www.gutenberg.org/files/11/11-0.txt")
```

```{r skwic2b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
rawtext %>%
  as.data.frame() %>%
  dplyr::filter(.  != "") %>%
  head(25) %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 25 non-empty lines of the example text.")  %>%
  flextable::border_outer()
```

After retrieving the text from Project Gutenberg, it becomes available for analysis within R. However, upon loading the text into our environment, we notice that it requires some processing. This includes removing extraneous elements such as the table of contents to isolate the main body of the text. Therefore, in the next step, we process the text. This will include consolidating the text into a single object and eliminating any non-essential content. Additionally, we clean up the text by removing superfluous white spaces to ensure a tidy dataset for our analysis.


**Data Preparation**

Data processing and preparation play a crucial role in text analysis, as they directly impact the quality and accuracy of the results obtained. When working with text data, it's essential to ensure that the data is clean, structured, and formatted appropriately for analysis. This involves tasks such as removing irrelevant information, standardizing text formats, and handling missing or erroneous data.

The importance of data processing and preparation lies in its ability to transform raw text into a usable format that can be effectively analyzed. By cleaning and pre-processing the data, researchers can mitigate the impact of noise and inconsistencies, enabling more accurate and meaningful insights to be extracted from the text.

However, it's worth noting that data preparation can often be a time-consuming process, sometimes requiring more time and effort than the actual analysis task itself. The extent of data preparation required can vary significantly depending on the complexity of the data and the specific research objectives. While some datasets may require minimal processing, others may necessitate extensive cleaning and transformation.

Ultimately, the time and effort invested in data preparation are essential for ensuring the reliability and validity of the analysis results. By dedicating sufficient attention to data processing and preparation, researchers can enhance the quality of their analyses and derive more robust insights from the text data at hand.



```{r skwic3, message=F, warning=F}
text <- rawtext %>%
  # collapse lines into a single  text
  paste0(collapse = " ") %>%
  # remove superfluous white spaces
  stringr::str_squish() %>%
  # remove everything before "CHAPTER I."
  stringr::str_remove(".*CHAPTER I\\.")
```

```{r skwic3b, echo = F, message=FALSE, warning=FALSE}
# inspect data
text %>%
  substr(start=1, stop=1000) %>%
  as.data.frame() %>%
  flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 1000 characters of of the example text.")  %>%
  flextable::border_outer()
```


<div class="warning" style='padding:0.1em; background-color:#51247a; color:#f2f2f2'>
<span>
<p style='margin-top:1em; text-align:center'>
<b>ADDITIONAL INFO</b>: the regular expression ".*CHAPTER I\\." can be interpreted as follows:<br> Match any sequence of characters (.*) followed by the exact text "CHAPTER I" and ending with a period (.). This pattern is commonly used to locate occurrences of a chapter heading labeled "CHAPTER I" within a larger body of text.
</p></span>
</div>

<br>

The entire content of Lewis Carroll's *Alice's Adventures in Wonderland* is now combined into a single character object and we can begin with generating concordances (KWICs). 

## Generating Basic KWICs{-}

Now, extracting concordances becomes straightforward with the `kwic` function from the `quanteda` package. This function is designed to enable the extraction of keyword-in-context (KWIC) displays, a common format for displaying concordance lines. 

To prepare the text for concordance extraction, we first need to tokenize it, which involves splitting it into individual words or tokens. Additionally, we specify the `phrase` argument in the `kwic` function, allowing us to extract phrases consisting of more than one token, such as "poor alice". 

The `kwic` function primarily requires two arguments: the tokenized text (`x`) and the search pattern (`pattern`). Additionally, it offers flexibility by allowing users to specify the context window, determining the number of words or elements displayed to the left and right of the nodeword. We'll delve deeper into customizing this context window later on.

```{r basekwic2, message=F, warning=F}
mykwic <- quanteda::kwic(
  # define and tokenise text
  quanteda::tokens(text), 
  # define search pattern and add the phrase function
  pattern = phrase("Alice")) %>%
  # convert it into a data frame
  as.data.frame()
```


```{r basekwic2b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
mykwic %>%
  head() %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 concordances for the nodeword Alice in our example text.")  %>%
  flextable::border_outer()
```

The resulting table showcases how "Alice" is used within our example text. However, since we use the `head` function, the table only displays the first six instances.

After extracting a concordance table, we can easily determine the frequency of the search term ("alice") using either the `nrow` or `length` functions. These functions provide the number of rows in a table (`nrow`) or the length of a vector (`length`).

```{r basekwic4}
nrow(mykwic)
```


```{r basekwic5}
length(mykwic$keyword)
```

The results indicate that there are `r length(mykwic$keyword)` instances of the search term ("alice"). Moreover, we can also explore how often different variants of the search term were found using the table function. While this may be particularly useful for searches involving various search terms (although less so in the present example). 

```{r basekwic6}
table(mykwic$keyword)
```

To gain a deeper understanding of how a word is used, it can be beneficial to extract more context. This can be achieved by adjusting the size of the context window. To do so, we simply specify the `window` argument of the `kwic` function. In the following example, we set the context window size to 10 words/elements, deviating from the default size of 5 words/elements.

```{r basekwic8, message=F, warning=F}
mykwic_long <- quanteda::kwic(
  # define text
  quanteda::tokens(text), 
  # define search pattern
  pattern = phrase("alice"), 
  # define context window size
  window = 10) %>%
  # make it a data frame
  as.data.frame()
```


```{r skwic8b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
mykwic_long %>%
  head() %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 concordances for the nodeword *alice* in the example text with extended context (10 elements).")  %>%
  flextable::border_outer()
```

***

<div class="warning" style='padding:0.1em; background-color:#51247a; color:#f2f2f2'>
<span>
<p style='margin-top:1em; text-align:center'>
<b>EXERCISE TIME!</b></p>
<p style='margin-left:1em;'>
</p></span>
</div>

<div class="question">` 



1. Extract the first 10 concordances for the word *confused*.<br>

<details>
  <summary>Answer</summary>
  ```{r ex1_1, class.source = NULL, eval = T}
  kwic_confused <- quanteda::kwic(x = quanteda::tokens(text), pattern = phrase("confused"))
  # inspect
  kwic_confused %>%
  as.data.frame() %>%
  head(10)
  ```
</details>


2. How many instances are there of the word *wondering*?<br>

<details>
  <summary>Answer</summary>
  ```{r ex1_2, class.source = NULL, eval = T}
  quanteda::kwic(x = quanteda::tokens(text), pattern = phrase("wondering")) %>%
  as.data.frame() %>%
  nrow()
  ```
</details>


3. Extract concordances for the word *strange* and show the first 5 concordance lines.<br>

<details>
  <summary>Answer</summary>
  ```{r  ex1_3, class.source = NULL, eval = T}
  kwic_strange <- quanteda::kwic(x = quanteda::tokens(text), pattern = phrase("strange"))
  # inspect
  kwic_strange %>%
  as.data.frame() %>%
  head(5)
  ```
</details>

</div>`

***


## Exporting KWICs {-}

To export or save a concordance table as an MS Excel spreadsheet, you can utilize the `write_xlsx` function from the `writexl` package, as demonstrated below. It's important to note that we employ the `here` function from the `here` package to specify the location where we want to save the file. In this instance, we save the file in the current working directory. If you're working with Rproj files in RStudio, which is recommended, then the current working directory corresponds to the directory or folder where your Rproj file is located.

```{r eval = F, warning=F, message=F}
write_xlsx(mykwic, here::here("mykwic.xlsx"))
```

## Extracting Multi-Word Expressions{-}

While extracting single words is a common practice, there are situations where you may need to extract more than just one word at a time. This can be particularly useful when you're interested in extracting phrases or multi-word expressions from your text data. To accomplish this, you simply need to specify that the pattern you are searching for is a phrase. This allows you to extract contiguous sequences of words that form meaningful units of text.

For example, if you're analyzing a text and want to extract phrases like "poor alice", "mad hatter", or "cheshire cat", you can easily do so by specifying these phrases as your search patterns.

```{r multikwic2, message=FALSE, warning=FALSE}
# extract concordances for the phrase "poor alice" using the kwic function from the quanteda package
kwic_pooralice <- quanteda::kwic(
  # tokenizing the input text
  quanteda::tokens(text),  
  # specifying the search pattern as the phrase "poor alice"
  pattern = phrase("poor alice") ) %>%
  # converting the result to a data frame for easier manipulation
  as.data.frame()                                  

```

```{r multikwic2b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
kwic_pooralice %>%
  head(10) %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 concordances for the nodephrase *poor alice* in the example text.")  %>%
  flextable::border_outer()
```

In addition to exact words or phrases, there are situations where you may need to extract more or less fixed patterns from your text data. These patterns might allow for variations in spelling, punctuation, or formatting. To search for such flexible patterns, you need to incorporate regular expressions into your search pattern.

Regular expressions (regex) are powerful tools for pattern matching and text manipulation. They allow you to define flexible search patterns that can match a wide range of text variations. For example, you can use regex to find all instances of a word regardless of whether it's in lowercase or uppercase, or to identify patterns like dates, email addresses, or URLs.

To incorporate regular expressions into your search pattern, you can use functions like `grepl()` or `grep()` in base R, or `str_detect()` and `str_extract()` in the `stringr` package. These functions allow you to specify regex patterns to search for within your text data.

***

<div class="warning" style='padding:0.1em; background-color:#51247a; color:#f2f2f2'>
<span>
<p style='margin-top:1em; text-align:center'>
<b>EXERCISE TIME!</b></p>
<p style='margin-left:1em;'>
</p></span>
</div>

<div class="question">` 



1. Extract the first 10 concordances for the phrase *the hatter*.<br>

<details>
  <summary>Answer</summary>
  ```{r ex2_1, class.source = NULL, eval = T}
  kwic_thehatter <- quanteda::kwic(x = quanteda::tokens(text), pattern = phrase("the hatter"))
  # inspect
  kwic_thehatter %>%
  as.data.frame() %>%
  head(10)
  ```
</details>


2. How many instances are there of the phrase *the hatter*?<br>

<details>
  <summary>Answer</summary>
  ```{r ex2_2, class.source = NULL, eval = T}
  kwic_thehatter %>%
  as.data.frame() %>%
  nrow()
  ```
</details>


3. Extract concordances for the phrase *the cat* and show the first 5 concordance lines.<br>

<details>
  <summary>Answer</summary>
  ```{r  ex2_3, class.source = NULL, eval = T}
  kwic_thecat <- quanteda::kwic(x = quanteda::tokens(text), pattern = phrase("the cat"))
  # inspect
  kwic_thecat %>%
  as.data.frame() %>%
  head(5)
  ```
</details>

</div>`

***

## Concordancing Using Regular Expressions{-}

Regular expressions provide a powerful means of searching for abstract patterns within text data, offering unparalleled flexibility beyond concrete words or phrases. Often abbreviated as *regex* or *regexp*, a regular expression is a special sequence of characters that describe a pattern to be matched in a text.

You can conceptualize regular expressions as highly potent combinations of wildcards, offering an extensive range of pattern-matching capabilities. For instance, the sequence `[a-z]{1,3}` is a regular expression that signifies one to three lowercase characters. Searching for this regular expression would yield results such as "is", "a", "an", "of", "the", "my", "our", and other short words.

There are three fundamental types of regular expressions:

1. **Regular expressions for individual symbols and frequencies**: These regular expressions represent single characters and determine their frequencies within the text. For example, `[a-z]` matches any lowercase letter, `[0-9]` matches any digit, and `{1,3}` specifies a range of occurrences (one to three in this case).

2. **Regular expressions for classes of symbols**: These regular expressions represent classes or groups of symbols with shared characteristics. For instance, `\d` matches any digit, `\w` matches any word character (alphanumeric characters and underscores), and `\s` matches any whitespace character.

3. **Regular expressions for structural properties**: These regular expressions represent structural properties or patterns within the text. For example, `^` matches the start of a line, `$` matches the end of a line, and `\b` matches a word boundary.

The regular expressions below show the first type of regular expressions, i.e. regular expressions that stand for individual symbols and determine frequencies.

```{r regex05, echo=F, eval = T, message=FALSE, warning=FALSE}
symbols1 <- c(fixed("?"), fixed("*"), fixed("+"), "{n}", "{n,}", "{n,m}")
explanation1 <- c("The preceding item is optional and will be matched at most once", "The preceding item will be matched zero or more times", "The preceding item will be matched one or more times", "The preceding item is matched exactly n times", "The preceding item is matched n or more times", "The preceding item is matched at least n times, but not more than m times")
example1 <- c("walk[a-z]? = walk, walks", 
             "walk[a-z]* = walk, walks, walked, walking", 
             "walk[a-z]+ = walks, walked, walking", 
             "walk[a-z]{2} = walked", 
             "walk[a-z]{2,} = walked, walking", 
             "walk[a-z]{2,3} = walked, walking")
df_regex <- data.frame(symbols1, explanation1, example1)
colnames(df_regex) <- c("RegEx Symbol/Sequence", "Explanation", "Example")
```

```{r regex05b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
df_regex %>%
  as.data.frame() %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .75, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "Regular expressions that stand for individual symbols and determine frequencies.")  %>%
  flextable::border_outer()
```


The regular expressions below show the second type of regular expressions, i.e. regular expressions that stand for classes of symbols.

```{r regex03, echo=F, eval = T, message=FALSE, warning=FALSE}
symbols2 <- c("[ab]", "[AB]", "[12]", "[:digit:]", "[:lower:]", "[:upper:]", "[:alpha:]", "[:alnum:]", "[:punct:]", "[:graph:]", "[:blank:]", "[:space:]", "[:print:]")
explanations2 <- c("lower case a and b", 
                   "upper case a and b", 
                   "digits 1 and 2", 
                   "digits: 0 1 2 3 4 5 6 7 8 9", 
                   "lower case characters: a–z", 
                   "upper case characters: A–Z", 
                   "alphabetic characters: a–z and A–Z", 
                   "digits and alphabetic characters", 
                   "punctuation characters: . , ; etc.", 
                   "graphical characters: [:alnum:] and [:punct:]", 
                   "blank characters: Space and tab", 
                   "space characters: Space, tab, newline, and other space characters", 
                   "printable characters: [:alnum:], [:punct:] and [:space:]")
df_regex <- data.frame(symbols2, explanations2)
colnames(df_regex) <- c("RegEx Symbol/Sequence", "Explanation")
```


```{r regex07b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
df_regex %>%
  as.data.frame() %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .75, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "Regular expressions that stand for classes of symbols.")  %>%
  flextable::border_outer()
```

The regular expressions that denote classes of symbols are enclosed in `[]` and `:`. The last type of regular expressions, i.e. regular expressions that stand for structural properties are shown below.

```{r regex09, echo=F, eval = T, message=FALSE, warning=FALSE}
symbols3 <- c(fixed("\\\\w"), fixed("\\\\W"), fixed("\\\\s"), fixed("\\\\S"), 
              fixed("\\\\d"), fixed("\\\\D"), fixed("\\\\b"), fixed("\\\\B"), 
              fixed("<"), fixed(">"), fixed("^"), fixed("$"))
explanations3 <- c("Word characters: [[:alnum:]_]",
                   "No word characters: [^[:alnum:]_]",
                   "Space characters: [[:blank:]]",
                   "No space characters: [^[:blank:]]",
                   "Digits: [[:digit:]]",
                   "No digits: [^[:digit:]]",
                   "Word edge",
                   "No word edge",
                   "Word beginning",
                   "Word end",
                   "Beginning of a string",
                   "End of a string")
df_regex <- data.frame(symbols3, explanations3)
colnames(df_regex) <- c("RegEx Symbol/Sequence", "Explanation")
```

```{r regex09b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
df_regex %>%
  as.data.frame() %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .5, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "Regular expressions that stand for structural properties.")  %>%
  flextable::border_outer()
```

To incorporate regular expressions into your KWIC searches, you include them in your search pattern and set the `valuetype` argument to `"regex"`. This allows you to specify complex search patterns that go beyond exact word matches.

For example, consider the search pattern `"\\balic.*|\\bhatt.*"`. In this pattern:

- `\\b` indicates a word boundary, ensuring that the subsequent characters are at the beginning of a word.
- `alic.*` matches any sequence of characters (`.*`) that begins with `alic`.
- `hatt.*` matches any sequence of characters that begins with `hatt`.
- The `|` operator functions as an OR operator, allowing the pattern to match either `alic.*` or `hatt.*`.

As a result, this search pattern retrieves elements that contain `alic` or `hatt` followed by any characters, but only where `alic` and `hatt` are the first letters of a word. Consequently, words like "malice" or "shatter" would not be retrieved.

By uaing regular expressions in your KWIC searches, you can conduct more nuanced and precise searches, capturing specific patterns or variations within your text data.

```{r rkwic6, message=FALSE, warning=FALSE}
# define search patterns
patterns <- c("\\balic.*|\\bhatt.*")
kwic_regex <- quanteda::kwic(
  # define text
  quanteda::tokens(text), 
  # define search pattern
  patterns, 
  # define valuetype
  valuetype = "regex") %>%
  # make it a data frame
  as.data.frame()
```

```{r rkwic6b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
kwic_regex %>%
  head(10) %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 concordances for the regular expression \\balic.* and \\bhatt.*.")  %>%
  flextable::border_outer()
```


***

<div class="warning" style='padding:0.1em; background-color:#51247a; color:#f2f2f2'>
<span>
<p style='margin-top:1em; text-align:center'>
<b>EXERCISE TIME!</b></p>
<p style='margin-left:1em;'>
</p></span>
</div>

<div class="question">` 



1. Extract the first 10 concordances for words containing *exu*.<br>

<details>
  <summary>Answer</summary>
  ```{r ex3_1, class.source = NULL, eval = T}
  kwic_exu <- quanteda::kwic(x = quanteda::tokens(text), pattern = ".*exu.*", valuetype = "regex")
  # inspect
  kwic_exu %>%
  as.data.frame() %>%
  head(10)
  ```
</details>


2. How many instances are there of words beginning with *pit*?<br>

<details>
  <summary>Answer</summary>
  ```{r ex3_2, class.source = NULL, eval = T}
  quanteda::kwic(x = quanteda::tokens(text), pattern = "\\bpit.*", valuetype = "regex") %>%
  as.data.frame() %>%
  nrow()
  ```
</details>


3. Extract concordances for words ending with *ption* and show the first 5 concordance lines.<br>

<details>
  <summary>Answer</summary>
  ```{r  ex3_3, class.source = NULL, eval = T}
  quanteda::kwic(x = quanteda::tokens(text), pattern = "ption\\b", valuetype = "regex")  %>%
  as.data.frame() %>%
  head(5)
  ```
</details>

</div>`

***


## Concordancing and Piping{-}

Quite often, we want to retrieve patterns only if they occur in a specific context. For instance, we might be interested in instances of "alice", but only if the preceding word is "poor" or "little". While such conditional concordances could be extracted using regular expressions, they are more easily retrieved by piping.

Piping is achieved using the `%>%` function from the `dplyr` package, and the piping sequence can be interpreted as "and then". We can then filter those concordances that contain "alice" using the `filter` function from the `dplyr` package. Note that the `$` symbol stands for the end of a string, so "poor$" signifies that "poor" is the last element in the string that precedes the nodeword.

```{r pipekwic7, echo=T, eval = T, message=FALSE, warning=FALSE}
# extract KWIC concordances
quanteda::kwic(        
  # input  tokenized text
  x = quanteda::tokens(text),                     
  # define search pattern ("alice")
  pattern = "alice"  
  # pipe (and then)
) %>%
  # convert result to data frame
  as.data.frame() %>%                              
  # filter concordances with "poor" or "little" preceding "alice"
  # save result in object called "kwic_pipe"
  dplyr::filter(stringr::str_detect(pre, "poor$|little$")) -> kwic_pipe  
``` 

```{r pipekwic7b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
kwic_pipe %>%
  head(10) %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 concordances for instances of *alice* that are preceeded by *poor* or *little*.")  %>%
  flextable::border_outer()
```

In this code:

- `quanteda::kwic`: This function extracts KWIC concordances from the input text.
- `quanteda::tokens(text)`: The input text is tokenized using the `tokens` function from the `quanteda` package.
- `pattern = "alice"`: Specifies the search pattern as "alice".
- `%>%`: The pipe operator (`%>%`) chains together multiple operations, passing the result of one operation as the input to the next.
- `as.data.frame()`: Converts the resulting concordance object into a data frame.
- `dplyr::filter(...)`: Filters the concordances based on the specified condition, which is whether "poor" or "little" precedes "alice".

Piping is an indispensable tool in R, commonly used across various data science domains, including text processing. This powerful function, denoted by `%>%`, allows for a more streamlined and readable workflow by chaining together multiple operations in a sequential manner.

Instead of nesting functions or creating intermediate variables, piping allows to take an easy-to-understand and more intuitive approach to data manipulation and analysis. With piping, each operation is performed "and then" the next, leading to code that is easier to understand and maintain.

While piping is frequently used in the context of text processing, its versatility extends far beyond. In data wrangling, modeling, visualization, and beyond, piping offers a concise and elegant solution for composing complex workflows.

By leveraging piping, R users can enhance their productivity and efficiency, making their code more expressive and succinct while maintaining clarity and readability. It's a fundamental tool in the toolkit of every R programmer, empowering them to tackle data science challenges with confidence and ease.

## Ordering and Arranging KWICs{-}

When examining concordances, it's often beneficial to reorder them based on their context rather than the order in which they appeared in the text or texts. This allows for a more organized and structured analysis of the data. To reorder concordances, we can utilize the `arrange` function from the `dplyr` package, which takes the column according to which we want to rearrange the data as its main argument.

### Ordering Alphabetically {-}

In the example below, we extract all instances of "alice" and then arrange the instances according to the content of the post column in alphabetical order.

```{r orderkwic2, echo=T, eval = T, message=FALSE, warning=FALSE}
# extract KWIC concordances
quanteda::kwic(                  
  # input  tokenized text
  x = quanteda::tokens(text),
  # define search pattern ("alice")
  pattern = "alice" 
  # end function and pipe (and then)
) %>%
  # convert result to data frame
  as.data.frame() %>%   
  # arrange concordances based on the content of the "post" column
  # save result in object called "kwic_ordered"
  dplyr::arrange(post) -> kwic_ordered                            
```

```{r orderkwic2b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
kwic_ordered %>%
  head(10) %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 re-ordered concordances for instances of alice.")  %>%
  flextable::border_outer()
```


### Ordering by Co-Occurrence Frequency {-}

Arranging concordances based on alphabetical properties may not always be the most informative approach. A more insightful option is to arrange concordances according to the frequency of co-occurring terms or collocates. This allows us to identify the most common words that appear alongside our search term, providing valuable insights into its usage patterns.

To accomplish this, we need to follow these steps:

1. Create a new variable or column that represents the word that co-occurs with the search term. In the example below, we use the `mutate` function from the `dplyr` package to create a new column called `post_word`. We then use the `str_remove_all` function from the `stringr` package to extract the word that immediately follows the search term. This is achieved by removing everything except for the word following the search term (including any white space).

2. Group the data by the word that immediately follows the search term.

3. Create a new column called `post_word_freq` that represents the frequencies of all the words that immediately follow the search term.

4. Arrange the concordances by the frequency of the collocates in descending order. This is accomplished by using the `arrange` function and specifying the column `post_word_freq` in descending order (indicated by the `-` symbol).

```{r arrange_kwic, echo=T, eval = T, message=FALSE, warning=FALSE}
quanteda::kwic(
  # define text
  x = quanteda::tokens(text), 
  # define search pattern
  pattern = "alice") %>%
  # make it a data frame
  as.data.frame() %>%
  # extract word following the nodeword
  dplyr::mutate(post1 = str_remove_all(post, " .*")) %>%
  # group following words
  dplyr::group_by(post1) %>%
  # extract frequencies of the following words
  dplyr::mutate(post1_freq = n()) %>%
  # arrange/order by the frequency of the following word
  dplyr::arrange(-post1_freq) -> kwic_ordered_coll
``` 



```{r orderkwic4j, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
kwic_ordered_coll %>%
  head(10) %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 re-ordered concordances.")  %>%
  flextable::border_outer()
```

In this code:

- `mutate`: This function from the `dplyr` package creates a new column in the data frame.
- `str_remove_all`: This function from the `stringr` package removes all occurrences of a specified pattern from a character string.
- `group_by`: This function from the `dplyr` package groups the data by a specified variable.
- `n()`: This function from the `dplyr` package calculates the number of observations in each group.
- `arrange`: This function from the `dplyr` package arranges the rows of a data frame based on the values of one or more columns.


We add more columns according to which we could arrange the concordance following the same schema. For example, we could add another column that represented the frequency of words that immediately preceded the search term and then arrange according to this column.

## Ordering by Multiple Co-Occurrence Frequencies{-}

In this section, we extract the three words preceding and following the nodeword "alice" from the concordance data and organize the results by the frequencies of the following words (you can also order by the preceding words which is why we also extract them). 

We begin by iterating through each row of the concordance data using `rowwise()`. Then, we extract the three words following the nodeword ("alice") and the three words preceding it from the `post` and `pre` columns, respectively. These words are split using the `strsplit` function and stored in separate columns (`post1`, `post2`, `post3`, `pre1`, `pre2`, `pre3`). 

Next, we group the data by each of the following words (`post1`, `post2`, `post3`, `pre1`, `pre2`, `pre3`) and calculate the frequency of each word using the `n()` function within each group. This allows us to determine how often each word occurs in relation to the nodeword "alice".

Finally, we arrange the concordances based on the frequencies of the following words (`post1`, `post2`, `post3`) in descending order using the `arrange()` function, storing the result in the `mykwic_following` data frame.

```{r}
mykwic %>%
  dplyr::rowwise() %>%  # Row-wise operation for each entry
  # Extract words preceding and following the node word
  # Extracting the first word following the node word
  dplyr::mutate(post1 = unlist(strsplit(post, " "))[1],  
                # Extracting the second word following the node word
                post2 = unlist(strsplit(post, " "))[2], 
                # Extracting the third word following the node word
                post3 = unlist(strsplit(post, " "))[3],  
                # Extracting the last word preceding the node word
                pre1 = unlist(strsplit(pre, " "))[length(unlist(strsplit(pre, " ")))],  
                # Extracting the second-to-last word preceding the node word
                pre2 = unlist(strsplit(pre, " "))[length(unlist(strsplit(pre, " ")))-1], 
                # Extracting the third-to-last word preceding the node word
                pre3 = unlist(strsplit(pre, " "))[length(unlist(strsplit(pre, " ")))-2]) %>%
  # Extract frequencies of the words around the node word
  # Grouping by the first word following the node word and counting its frequency
  dplyr::group_by(post1) %>% dplyr::mutate(npost1 = n()) %>%  
  # Grouping by the second word following the node word and counting its frequency
  dplyr::group_by(post2) %>% dplyr::mutate(npost2 = n()) %>%  
  # Grouping by the third word following the node word and counting its frequency
  dplyr::group_by(post3) %>% dplyr::mutate(npost3 = n()) %>% 
  # Grouping by the last word preceding the node word and counting its frequency
  dplyr::group_by(pre1) %>% dplyr::mutate(npre1 = n()) %>%  
  # Grouping by the second-to-last word preceding the node word and counting its frequency
  dplyr::group_by(pre2) %>% dplyr::mutate(npre2 = n()) %>% 
  # Grouping by the third-to-last word preceding the node word and counting its frequency
  dplyr::group_by(pre3) %>% dplyr::mutate(npre3 = n()) %>%  
  # Arranging the results
  # Arranging in descending order of frequencies of words following the node word
  dplyr::arrange(-npost1, -npost2, -npost3) -> mykwic_following  
```

```{r orderkwic4h, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
mykwic_following %>%
  head(10) %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 lines of re-ordered concordances.")  %>%
  flextable::border_outer()
```


The updated concordance now presents the arrangement based on the frequency of words following the node word. This means that the words occurring most frequently immediately after the keyword "alice" are listed first, followed by less frequent ones.

It's essential to note that the arrangement can be customized by modifying the arguments within the `dplyr::arrange` function. By altering the order and content of these arguments, you can adjust the sorting criteria. For instance, if you want to prioritize the frequency of words preceding the node word instead, you can simply rearrange the arguments accordingly. This flexibility empowers users to tailor the arrangement to suit their specific analysis goals and preferences.

## Concordances from Transcriptions{-}

Since many analyses rely on transcripts as their main data source, and transcripts often require additional processing due to their specific features, we will now demonstrate concordancing using transcripts. To begin, we will load five example transcripts representing the first five files from the Irish component of the [International Corpus of English](https://www.ice-corpora.uzh.ch/en.html)[^1]. These transcripts will serve as our dataset for conducting concordance analysis.

We first load these files so that we can process them and extract KWICs. To load the files, the code below  dynamically generates URLs for a series of text files, then reads the content of each file into R, storing the text data in the transcripts object. This is a common procedure when working with multiple text files or when the filenames follow a consistent pattern.

```{r trans2, echo=T, eval = T, message=FALSE, warning=FALSE}
# define corpus files
files <- paste("https://slcladal.github.io/data/ICEIrelandSample/S1A-00", 1:5, ".txt", sep = "")
# load corpus files
transcripts <- sapply(files, function(x){
  x <- readLines(x)
  })
```


```{r trans2b, echo = F, message=FALSE, warning=FALSE}
# inspect data
transcripts[[1]][1:10] %>%
  as.data.frame() %>%
  flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 utterances in the sample transcripts.")  %>%
  flextable::border_outer()
```

The first ten lines shown above let us know that, after the header (`<S1A-001 Riding>`) and the symbol which indicates the start of the transcript (`<I>`), each utterance is preceded by a sequence which indicates the section, file, and speaker (e.g. `<S1A-001$A>`). The first utterance is thus uttered by speaker `A` in file `001` of section `S1A`. In addition, there are several sequences that provide meta-linguistic information which indicate the beginning of a speech unit (`<#>`), pauses (`<,>`), and laughter (`<&> laughter </&>`).

To perform the concordancing, we need to change the format of the transcripts because the `kwic` function only works on character, corpus, tokens object- in their present form, the transcripts represent a list which contains vectors of strings. To change the format, we collapse the individual utterances into a single character vector for each transcript.

```{r trans4, echo=T, eval = T, message=FALSE, warning=FALSE}
transcripts_collapsed <- sapply(files, function(x){
  # read-in text
  x <- readLines(x)
  # paste all lines together
  x <- paste0(x, collapse = " ")
  # remove superfluous white spaces
  x <- str_squish(x)
})
```

```{r trans4b, echo = F, message=FALSE, warning=FALSE}
# inspect data
transcripts_collapsed %>%
    substr(start=1, stop=500) %>%
  as.data.frame() %>%
  flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 500 characters of the collapsed sample transcripts.")  %>%
  flextable::border_outer()
```


We now move on to extracting concordances. We begin by splitting the text simply by white space. This ensures that tags and markup remain intact, preventing accidental splitting. Additionally, we extend the context surrounding our target word or phrase. While the default is five tokens before and after the keyword, we opt to widen this context to 10 tokens. Furthermore, for improved organization and readability, we refine the file names. Instead of using the full path, we extract only the name of the text. This simplifies the presentation of results and enhances clarity when navigating through the corpus.

```{r trans6, echo=T, eval = T, message=FALSE, warning=FALSE}
kwic_trans <- quanteda::kwic(
  # tokenize transcripts
  quanteda::tokens(transcripts_collapsed, what = "fasterword"), 
  # define search pattern
  pattern = phrase("you know"),
  # extend context
  window = 10) %>%
  # make it a data frame
  as.data.frame() %>%
  # clean docnames / filenames / text names
  dplyr::mutate(docname = str_replace_all(docname, ".*/(.*?).txt", "\\1"))

```

```{r trans6b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
kwic_trans %>%
  head(10) %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 10 concordances for you know in three example transcripts.")  %>%
  flextable::border_outer()
```


## Custom Concordances{-}

As R represents a fully-fledged programming environment, we can, of course, also write our own, customized concordance function. The code below shows how you could go about doing so. Note, however, that this function only works if you enter more than a single file. 

```{r customkwic2, message=F, warning=F}
mykwic <- function(txts, pattern, context) {
  # activate packages
  require(stringr)
  # list files
  txts <- txts[stringr::str_detect(txts, pattern)]
  conc <- sapply(txts, function(x) {
    # determine length of text
        lngth <- as.vector(unlist(nchar(x)))
    # determine position of hits
    idx <- str_locate_all(x, pattern)
    idx <- idx[[1]]
    ifelse(nrow(idx) >= 1, idx <- idx, return(NA))
    # define start position of hit
    token.start <- idx[,1]
    # define end position of hit
    token.end <- idx[,2]
    # define start position of preceding context
    pre.start <- ifelse(token.start-context < 1, 1, token.start-context)
    # define end position of preceding context
    pre.end <- token.start-1
    # define start position of subsequent context
    post.start <- token.end+1
    # define end position of subsequent context
    post.end <- ifelse(token.end+context > lngth, lngth, token.end+context)
    # extract the texts defined by the positions
    PreceedingContext <- substring(x, pre.start, pre.end)
    Token <- substring(x, token.start, token.end)
    SubsequentContext <- substring(x, post.start, post.end)
    Id <- 1:length(Token)
    conc <- cbind(Id, PreceedingContext, Token, SubsequentContext)
    # return concordance
    return(conc)
    })
  concdf <- do.call(rbind, conc) %>%
    as.data.frame()
  return(concdf)
}
```

We can now try if this function works by searching for the sequence *you know* in the transcripts that we have loaded earlier. One difference between the `kwic` function provided by the `quanteda` package and the customized concordance function used here is that the `kwic` function uses the number of words to define the context window, while the `mykwic` function uses the number of characters or symbols instead (which is why we use a notably higher number to define the context window).

```{r customkwic4, message=F, warning=F}
kwic_youknow <- mykwic(transcripts_collapsed, "you know", 50)
```


```{r customkwic4b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
kwic_youknow %>%
  as.data.frame() %>%
  head() %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 6 concordances for *you know* extracted using the mykwic function.")  %>%
  flextable::border_outer()
```

As this concordance function only works for more than one text, we split the text into chapters and assign each section a name.


```{r customkwic6, message=FALSE, warning=FALSE}
# read in text
text_split <- text %>%
  stringr::str_squish() %>%
  stringr::str_split("[CHAPTER]{7,7} [XVI]{1,7}\\. ") %>%
  unlist()
text_split <- text_split[which(nchar(text_split) > 2000)]
# add names
names(text_split) <- paste0("text", 1:length(text_split))
# inspect data
nchar(text_split)
```

Now that we have named elements, we can search for the pattern *poor alice*. We also need to clean the concordance as some sections do not contain any instances of the search pattern. To clean the data, we select only the columns `File`, `PreceedingContext`, `Token`, and `SubsequentContext` and then remove all rows where information is missing. 

```{r customkwic8, message=FALSE, warning=FALSE}
mykwic_pooralice <- mykwic(text_split, "poor Alice", 50)
```

```{r customkwic8b, echo=F, message=FALSE, warning=FALSE, class.source='klippy'}
# inspect data
mykwic_pooralice %>%
  as.data.frame() %>%
  head() %>%
  flextable::flextable() %>%
  flextable::set_table_properties(width = .95, layout = "autofit") %>%
  flextable::theme_zebra() %>%
  flextable::fontsize(size = 12) %>%
  flextable::fontsize(size = 12, part = "header") %>%
  flextable::align_text_col(align = "center") %>%
  flextable::set_caption(caption = "First 6 concordances of *poor alice* extracted using the mykwic function.")  %>%
  flextable::border_outer()
```

You can go ahead and modify the customized concordance function to suit your needs. 


# Citation & Session Info {-}

Schweinberger, Martin. 2024. *Concordancing with R*. Brisbane: The Language Technology and Data Analysis Laboratory (LADAL). url: https://ladal.edu.au/kwics.html (Version 2024.05.07).


```
@manual{schweinberger2024kwics,
  author = {Schweinberger, Martin},
  title = {Concordancing with R},
  note = {https://ladal.edu.au/kwics.html},
  year = {2024},
  organization = {The Language Technology and Data Analysis Laboratory (LADAL)},
  address = {Brisbane},
  edition = {2024.05.07}
}
```


```{r fin}
sessionInfo()
```




***

[Back to top](#introduction)

[Back to HOME](https://ladal.edu.au)

***

# References {-}


[^1]: This data is freely available after registration. To get access to the data represented in the Irish Component of the International Corpus of Eng;lish (or any other component), you or your institution will need a valid licence. You need to send your request from an academic edu e-mail to proof your educational status. To get an academic licence with download access please fill in [the licence form (PDF, 82 KB)](https://www.ice-corpora.uzh.ch/dam/jcr:7ae594b2-ee97-4935-8022-7d2d91b60be4/ICElicence_UZH.pdf) and send it to `ice@es.uzh.ch`. You should get the credentials for downloading [here](https://www.ice-corpora.uzh.ch/en/access/corpus-download.html) and unpacking the corpora within  about 10 working days.