Solution, compute TF, IDF vectorization of given corpus

What does tf-idf mean?

Tf-idf stands for term frequency-inverse document frequency, and the tf-idf weight is a weight often used in information retrieval and text mining. This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus. The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus. Variations of the tf-IDF weighting scheme are often used by search engines as a central tool in scoring and ranking a document's relevance given a user query.

One of the simplest ranking functions is computed by summing the tf-IDF for each query term; many more sophisticated ranking functions are variants of this simple model.

Tf-idf can be successfully used for stop-words filtering in various subject fields including text summarization and classification.

How to Compute:

Typically, the tf-idf weight is composed by two terms: the first computes the normalized Term Frequency (TF), aka. the number of times a word appears in a document, divided by the total number of words in that document; the second term is the Inverse Document Frequency (IDF), computed as the logarithm of the number of the documents in the corpus divided by the number of documents where the specific term appears.

TF: Term Frequency, which measures how frequently a term occurs in a document. Since every document is different in length, it is possible that a term would appear much more time in long documents than shorter ones. Thus, the term frequency is often divided by the document length (aka. the total number of terms in the document) as a way of normalization:

𝑇𝐹(𝑡)=Number of times term t appears in a document/Total number of terms in the document

IDF: Inverse Document Frequency, which measures how important a term is. While computing TF, all terms are considered equally important. However, it is known that certain terms, such as "is", "of", and "that", may appear a lot of times but have little importance. Thus we need to weigh down the frequent terms while scaling up the rare ones, by computing the following:

𝐼𝐷𝐹(𝑡)=log𝑒Total number of documents/Number of documents with term t in it

for numerical stability we will be changing this formula little bit

𝐼𝐷𝐹(𝑡)=log𝑒Total number of documents/Number of documents with term t in it+1.

Example

Consider a document containing 100 words wherein the word cat appears 3 times. The term frequency (i.e., tf) for cat is then (3 / 100) = 0.03. Now, assume we have 10 million documents and the word cat appears in one thousand of these. Then, the inverse document frequency (i.e., idf) is calculated as log(10,000,000 / 1,000) = 4. Thus, the Tf-idf weight is the product of these quantities: 0.03 * 4 = 0.12.

1. Build a TFIDF Vectorizer & compare its results with Sklearn

As a part of this task, you will be implementing the TFIDF vectorizer on a collection of text documents.

You should compare the results of your own implementation of the TFIDF vectorizer with that of sklearns implementation TFIDF vectorizer.

Sklearn does few more tweaks in the implementation of its version of TFIDF vectorizer, so to replicate the exact results you would need to add following things to your custom implementation of TF-IDF vectorizer:
1. Sklearn has its vocabulary generated from IDF sorted in alphabetical order
2. Sklearn applies L2-normalization on its output matrix.
3. The final output of sklearn TF-IDF vectorizer is a sparse matrix.

Steps to approach this task:

You would have to write both fit and transform methods for your custom implementation of TF-IDF vectorizer.
Print out the alphabetically sorted vocab after you fit your data and check if its the same as that of the feature names from sklearn TF-IDF vectorizer.
Print out the IDF values from your implementation and check if its the same as that of sklearns TF-IDF vectorizer idf values.
Once you get your vocab and IDF values to be the same as that of sklearns implementation of the TF-IDF vectorizer, proceed to the below steps.
Make sure the output of your implementation is a sparse matrix. Before generating the final output, you need to normalize your sparse matrix using L2 normalization. You can refer to this link

https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.normalize.html

After completing the above steps, print the output of your custom implementation and compare it with sklearns implementation of the TF-IDF vectorizer.

To check the output of a single document in your collection of documents, you can convert the sparse matrix related only to that document into a dense matrix and print it.

Read corpus:

## SkLearn# Collection of string documents

corpus = [
     'this is the first document',
     'this document is the second document',
     'and this is the third one',
     'is this the first document',
]

SkLearn Implementation

from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer()
vectorizer.fit(corpus) 
skl_output = vectorizer.transform(corpus)

# sklearn feature names, they are sorted in alphabetic order by default.
print(vectorizer.get_feature_names())

Output:

['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this']

# Here we will print the sklearn tfidf vectorizer idf values after applying the fit method
# After using the fit function on the corpus the vocab has 9 words in it, and each has its idf value.
print(vectorizer.idf_)

Output:

[1.91629073 1.22314355 1.51082562 1. 1.91629073 1.91629073 1. 1.91629073 1. ]

# shape of sklearn tfidf vectorizer output after applying transform method.
skl_output.shape

(4, 9)

# sklearn tfidf values for first line of the above corpus.
# Here the output is a sparse matrix 
print(skl_output[0])

Output:

(0, 8) 0.38408524091481483 (0, 6) 0.38408524091481483 (0, 3) 0.38408524091481483 (0, 2) 0.5802858236844359 (0, 1) 0.46979138557992045

# sklearn tfidf values for first line of the above corpus.
# To understand the output better, here we are converting the sparse output matrix to dense matrix and printing it.
# Notice that this output is normalized using L2 normalization. sklearn does this by default.
print(skl_output[0].toarray())

Output:

[[0. 0.46979139 0.58028582 0.38408524 0. 0. 0.38408524 0. 0.38408524]]

Contact us for this machine learning assignment Solutions by Codersarts Specialist who can help you mentor and guide for such machine learning assignments.

If you have project or assignment files, You can send at contact@codersarts.com directly