Illustration of a spam message

Objective

We illustrate how to use Bayes theorem to design a simple spam email detector. \[ \Pr(spam\mid money )=\frac{\Pr(money\mid spam) \Pr(spam)}{\Pr(money)}\]

Toy example

Firstly, we introduce the problem with this simple toy example.

Example: SMS spam data

Consider the file sms.csv which contains a study of SMS records classified as spam or ham.

rm(list=ls())
sms <- read.csv("sms.csv",sep=",")
names(sms)

## [1] "type" "text"

head(sms)

##   type
## 1  ham
## 2  ham
## 3  ham
## 4 spam
## 5 spam
## 6  ham
##                                                                                                                                                                text
## 1                                                                                                                 Hope you are having a good week. Just checking in
## 2                                                                                                                                           K..give back my thanks.
## 3                                                                                                                       Am also doing in cbe only. But have to pay.
## 4            complimentary 4 STAR Ibiza Holiday or Â£10,000 cash needs your URGENT collection. 09066364349 NOW from Landline not to lose out! Box434SK38WP150PPM18+
## 5 okmail: Dear Dave this is your final notice to collect your 4* Tenerife Holiday or #5000 CASH award! Call 09061743806 from landline. TCs SAE Box326 CW25WX 150ppm
## 6                                                                                                                Aiya we discuss later lar... Pick u up at 4 is it?

We want to use a naive Bayes classifier to build a spam filter based on the words in the message.

Prepare the Corpus

A corpus is a collection of documents

library(tm)

## Warning: package 'tm' was built under R version 4.1.2

## Loading required package: NLP

## Warning: package 'NLP' was built under R version 4.1.1

corpus <- Corpus(VectorSource(sms$text))
inspect(corpus[1:3])

## <<SimpleCorpus>>
## Metadata:  corpus specific: 1, document level (indexed): 0
## Content:  documents: 3
## 
## [1] Hope you are having a good week. Just checking in
## [2] K..give back my thanks.                          
## [3] Am also doing in cbe only. But have to pay.

Here, VectorSource tells the Corpus function that each document is an entry in the vector.

Clean the Corpus

Different texts may contain Hello!, Hello, hello, etc. We would like to consider all of these the same. We clean up the corpus with the tm_map function. Translate all letters to lower case:

clean_corpus <- tm_map(corpus, tolower)

## Warning in tm_map.SimpleCorpus(corpus, tolower): transformation drops documents

inspect(clean_corpus[1:3])

## <<SimpleCorpus>>
## Metadata:  corpus specific: 1, document level (indexed): 0
## Content:  documents: 3
## 
## [1] hope you are having a good week. just checking in
## [2] k..give back my thanks.                          
## [3] am also doing in cbe only. but have to pay.

Remove numbers:

clean_corpus <- tm_map(clean_corpus, removeNumbers)

## Warning in tm_map.SimpleCorpus(clean_corpus, removeNumbers): transformation
## drops documents

Remove punctuation:

clean_corpus <- tm_map(clean_corpus, removePunctuation)

## Warning in tm_map.SimpleCorpus(clean_corpus, removePunctuation): transformation
## drops documents

Remove common non-content words, like to, and, the,.. These are called stop words. The function stopwords reports a list of about 175 such words.

stopwords("en")[1:10]

##  [1] "i"         "me"        "my"        "myself"    "we"        "our"      
##  [7] "ours"      "ourselves" "you"       "your"

clean_corpus <- tm_map(clean_corpus, removeWords,
stopwords("en"))

## Warning in tm_map.SimpleCorpus(clean_corpus, removeWords, stopwords("en")):
## transformation drops documents

Remove the excess white space:

clean_corpus <- tm_map(clean_corpus, stripWhitespace)

## Warning in tm_map.SimpleCorpus(clean_corpus, stripWhitespace): transformation
## drops documents

Word clouds

We create word clouds to visualize the differences between the two message types, ham or spam. First, obtain the indices of spam and ham messages:

spam_indices <- which(sms$type == "spam")
spam_indices[1:3]

## [1] 4 5 9

ham_indices <- which(sms$type == "ham")
ham_indices[1:3]

## [1] 1 2 3

library(wordcloud)

## Warning: package 'wordcloud' was built under R version 4.1.2

## Loading required package: RColorBrewer

## Warning: package 'RColorBrewer' was built under R version 4.1.1

wordcloud(clean_corpus[ham_indices], min.freq=40, scale=c(3,.5))

wordcloud(clean_corpus[spam_indices], min.freq=40)

Building a spam filter

Divide into training and test data. Use 75% training and 25% test.

nobs=dim(sms)[1]
train = 1:round(nobs*0.75)
test=(round(nobs*0.75)+1):nobs
sms_train <- sms[train,]
sms_test <- sms[test,]

And the clean corpus:

corpus_train <- clean_corpus[train]
corpus_test <- clean_corpus[test]

Compute the frequency of terms

Using DocumentTermMatrix, we create a sparse matrix data structure in which the rows of the matrix refer to document and the columns refer to words.

sms_dtm <- DocumentTermMatrix(clean_corpus)
inspect(sms_dtm[1:4, 3:10])

## <<DocumentTermMatrix (documents: 4, terms: 8)>>
## Non-/sparse entries: 8/24
## Sparsity           : 75%
## Maximal term length: 6
## Weighting          : term frequency (tf)
## Sample             :
##     Terms
## Docs also back cbe hope just kgive thanks week
##    1    0    0   0    1    1     0      0    1
##    2    0    1   0    0    0     1      1    0
##    3    1    0   1    0    0     0      0    0
##    4    0    0   0    0    0     0      0    0

Divide the matrix into training and test rows.

sms_dtm_train <- sms_dtm[train,]
sms_dtm_test <- sms_dtm[test,]

Identify frequently used words

Don’t muddy the classifier with words that may only occur a few times. To identify words appearing at least 5 times:

five_times_words <- findFreqTerms(sms_dtm_train, 5)
length(five_times_words)

## [1] 1228

five_times_words[1:5]

## [1] "checking" "good"     "hope"     "just"     "week"

Create document-term matrices using frequent words:

sms_dtm_train <- DocumentTermMatrix(corpus_train, control=list(dictionary = five_times_words))
sms_dtm_test <- DocumentTermMatrix(corpus_test, control=list(dictionary = five_times_words))

Convert count information to Yes or No

Naive Bayes classification needs present or absent info on each word in a message. We have counts of occurrences. To convert the document-term matrices:

convert_count <- function(x){
y <- ifelse(x > 0, 1,0)
y <- factor(y, levels=c(0,1), labels=c("No", "Yes"))
y
}

Convert document-term matrices

sms_dtm_train <- apply(sms_dtm_train, 2, convert_count)
sms_dtm_train[1:4, 30:35]

##     Terms
## Docs lar  later pick much ask  father
##    1 "No" "No"  "No" "No" "No" "No"  
##    2 "No" "No"  "No" "No" "No" "No"  
##    3 "No" "No"  "No" "No" "No" "No"  
##    4 "No" "No"  "No" "No" "No" "No"

sms_dtm_test <- apply(sms_dtm_test, 2, convert_count)
sms_dtm_test[1:4, 3:10]

##     Terms
## Docs home  can   come  plan  point room  weekend cool 
##    1 "Yes" "No"  "No"  "No"  "No"  "No"  "No"    "No" 
##    2 "No"  "Yes" "Yes" "Yes" "Yes" "Yes" "Yes"   "No" 
##    3 "No"  "No"  "No"  "No"  "No"  "No"  "No"    "Yes"
##    4 "No"  "No"  "No"  "No"  "No"  "No"  "No"    "No"

Create a Naive Bayes classifier

We will use a Naive Bayes classifier provided in the package e1071. We create the classifier using the training data.

library(e1071)

## Warning: package 'e1071' was built under R version 4.1.1

classifier <- naiveBayes(sms_dtm_train, sms_train$type)
class(classifier)

## [1] "naiveBayes"

Evaluate the performance on the test data

Given the classifier object, we can use the predict function to test the model on new data.

predictions <- predict(classifier, newdata=sms_dtm_test)

Classifications of messages in the test set are based on the probabilities generated with the training set. # Check the predictions against reality We have predictions and we have a factor of real spam-ham classifications. Generate a table.

table(predictions, sms_test$type)

##            
## predictions  ham spam
##        ham  1202   31
##        spam    5  152

Spam filter performance:

It correctly classifies 99% of the ham;
It correctly classifies 83% of the spam;

This is good balance.

Bayesian Naive Bayes classifier

There may be problems with the standard Naive Bayes classifier when there are one or various words that do not appear in a certain class. A useful solution is considering a Bayesian approach for the Naive Bayes classifier.

Firsly, we will illustrate the Bayesian approach using the toy example .

The Bayesian Naive Bayes with uniform priors is equivalent to the frequently called’ ‘’Laplacian smoothing’’. In the sms example, it can be incorporated using:

B.clas <- naiveBayes(sms_dtm_train, sms_train$type,laplace = 1)
class(B.clas)

## [1] "naiveBayes"

B.preds <- predict(B.clas, newdata=sms_dtm_test)
table(B.preds, sms_test$type)

##        
## B.preds  ham spam
##    ham  1189   18
##    spam   18  165

The Bayesian Spam filter performance is slightly better than the classical approach.

Summary

We have shown how to use the Bayes theorem to construct a spam email detector.
First, we have constructed a clean Corpus based on a collection of texts.
Then, we have obtain a word could for each group of texts.
Using a (training) data subset, we have constructed a document-term matrix using the most frequent words.
We have constructed the naive Bayes classiffier based on empirical frequencies.
Finally, we have shown how to extend the naive Bayes classifier using Bayesian estimates which avoid zero probabilities in the classifier.

Exercise

Consider the file twits.csv which contain a collection of twits (in Spanish) that are favourable or against death penalty. Construct a clean corpus based on these sample of text and obtain word could for each group. Later, construct a a document-term matrix using the most frequent words and use a (frequentist) naive Bayes classifier. Finally, obtain a Bayesian naive Bayes classifierand compare results.

Case Study I: Naive Bayesian spam filtering