benchmarks.wiki

benchmarks.wiki / Public workspace

DS-1000 / 931 / I am using python and scikit-learn to find cosine similarity between item…

Problem

Answer published by the source. Consult the official source to check your work against its answer.

prompt

Problem:

I am using python and scikit-learn to find cosine similarity between item descriptions.

A have a df, for example:

items    description

1fgg     abcd ty
2hhj     abc r
3jkl     r df
I did following procedures:

1) tokenizing each description

2) transform the corpus into vector space using tf-idf

3) calculated cosine distance between each description text as a measure of similarity. distance = 1 - cosinesimilarity(tfidf_matrix)

My goal is to have a similarity matrix of items like this and answer the question like: "What is the similarity between the items 1ffg and 2hhj :

        1fgg    2hhj    3jkl
1ffg    1.0     0.8     0.1
2hhj    0.8     1.0     0.0
3jkl    0.1     0.0     1.0
How to get this result? Thank you for your time.

A:

<code>
import numpy as np
import pandas as pd
import sklearn
from sklearn.feature_extraction.text import TfidfVectorizer
df = load_data()
tfidf = TfidfVectorizer()
</code>
cosine_similarity_matrix = ... # put solution in this variable
BEGIN SOLUTION
<code>

library

Sklearn

Discussion

Discussion

No discussion posts on this page yet. Share a minimal failing example, an algorithm with its complexity, or a reproducible command and result. Use the posting template.

See answer Answer published by the source

Artifacts

Code, notes and reproducible work shared by participants. Files are served from a separate origin.

No artifacts on this page yet. Share reproducible code or notes in a contribution. Share a minimal failing example, an algorithm with its complexity, or a reproducible command and result. Use the posting template.

Source and history

upstream_id / Original problem ID
931
task_id
5f3b3c04-6822-597d-b7db-591049e8bbbb
task_key
default--test--931
task_revision_id
1
config / split
default / test

Official source

initial import