Neighborhood-based Collaborative Filtering: Phương pháp gợi ý dựa trên láng giềng gần nhất (P2)

Thứ Hai, 30/12/2019

Tram Ho

Ở phần 2 này thì mình sẽ viết demo cho Neighborhood-based Collaborative Filtering (NBCF).

Lý thuyết mình đã trình bày ở phần 1, nếu muốn các bạn có thể xem lại tại đây

1. Về hàm khoảng cách

Trong bài này, mình sẽ demo với 2 hàm khoảng cách cơ bản là cosine và pearson

Với cosine, mình sử dụng hàm cosine_similarity có sẵn của sklearn.

Còn pearson, mình sử dụng pearsonr của scipy. Nhưng do hàm này không nhận ma trận (mảng 2 chiều) nên mình phải convert lại như vậy:

<span class="token keyword">from</span> scipy<span class="token punctuation">.</span>stats<span class="token punctuation">.</span>stats <span class="token keyword">import</span> pearsonr

<span class="token keyword">def</span> <span class="token function">pearson</span><span class="token punctuation">(</span>X<span class="token punctuation">,</span> Y <span class="token operator">=</span> <span class="token boolean">None</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
    x <span class="token operator">=</span> X<span class="token punctuation">.</span>shape<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span>
    y <span class="token operator">=</span> X<span class="token punctuation">.</span>shape<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span>
    a <span class="token operator">=</span> np<span class="token punctuation">.</span>zeros<span class="token punctuation">(</span><span class="token punctuation">(</span>x<span class="token punctuation">,</span> x<span class="token punctuation">)</span><span class="token punctuation">)</span>
    u <span class="token operator">=</span> np<span class="token punctuation">.</span>zeros<span class="token punctuation">(</span><span class="token punctuation">(</span>x<span class="token punctuation">,</span> y<span class="token punctuation">)</span><span class="token punctuation">)</span>
    temp <span class="token operator">=</span> <span class="token number">0</span>
    
    <span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>x<span class="token punctuation">)</span><span class="token punctuation">:</span>
        <span class="token keyword">for</span> j <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>y<span class="token punctuation">)</span><span class="token punctuation">:</span>
            u<span class="token punctuation">[</span>i<span class="token punctuation">]</span><span class="token punctuation">[</span>j<span class="token punctuation">]</span> <span class="token operator">=</span> X<span class="token punctuation">[</span>i<span class="token punctuation">,</span> j<span class="token punctuation">]</span>
    <span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>x<span class="token punctuation">)</span><span class="token punctuation">:</span>
        <span class="token keyword">for</span> j <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>x<span class="token punctuation">)</span><span class="token punctuation">:</span>
            temp <span class="token operator">=</span> pearsonr<span class="token punctuation">(</span>u<span class="token punctuation">[</span>i<span class="token punctuation">]</span><span class="token punctuation">,</span> u<span class="token punctuation">[</span>j<span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span>
            a<span class="token punctuation">[</span>i<span class="token punctuation">]</span><span class="token punctuation">[</span>j<span class="token punctuation">]</span> <span class="token operator">=</span>  temp <span class="token keyword">if</span> <span class="token operator">not</span> np<span class="token punctuation">.</span>isnan<span class="token punctuation">(</span>temp<span class="token punctuation">)</span> <span class="token keyword">else</span> <span class="token number">0</span>
    
    <span class="token keyword">return</span> a

from scipy.stats.stats import pearsonr

def pearson(X, Y = None):

x = X.shape[0]

y = X.shape[1]

a = np.zeros((x, x))

u = np.zeros((x, y))

temp = 0

for i in range(x):

for j in range(y):

u[i][j] = X[i, j]

for j in range(x):

temp = pearsonr(u[i], u[j])[0]

a[i][j] = temp if not np.isnan(temp) else 0

return a

2. Xây dựng class NBCF

Hàm khởi tạo:

Tham số đầu vào:

Y: ma trận Utility, gồm 3 cột, mỗi cột gồm 3 số liệu: user_id, item_id, rating.
k: số lượng láng giềng lựa chọn để dự đoán rating.
uuCF: Nếu sử dụng uuCF thì uuCF = 1 , ngược lại uuCF = 0. Tham số nhận giá trị mặc định là 1.
dist_f: Hàm khoảng cách, như đã nói ở mục 1, ở đây mình sử dụng 2 hàm cosine và pearson. Tham số nhận giá trị mặc định là hàm cosine_similarity của klearn.
limit: Số lượng items gợi ý cho mỗi user. Mặc định bằng 10.

<span class="token keyword">def</span> <span class="token function">__init__</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> Y<span class="token punctuation">,</span> k<span class="token punctuation">,</span> uuCF <span class="token operator">=</span> <span class="token number">1</span><span class="token punctuation">,</span> dist_f <span class="token operator">=</span> cosine_similarity<span class="token punctuation">,</span> limit <span class="token operator">=</span> <span class="token number">10</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
        self<span class="token punctuation">.</span>uuCF <span class="token operator">=</span> uuCF
        self<span class="token punctuation">.</span>f <span class="token operator">=</span> <span class="token builtin">open</span><span class="token punctuation">(</span><span class="token string">'danhgiaNBCF.dat'</span><span class="token punctuation">,</span> <span class="token string">'a+'</span><span class="token punctuation">)</span>
        self<span class="token punctuation">.</span>Y <span class="token operator">=</span> Y <span class="token keyword">if</span> uuCF <span class="token keyword">else</span> Y<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">]</span><span class="token punctuation">]</span>
        self<span class="token punctuation">.</span>Ybar <span class="token operator">=</span> <span class="token boolean">None</span>
        self<span class="token punctuation">.</span>k <span class="token operator">=</span> k
        self<span class="token punctuation">.</span>limit <span class="token operator">=</span> limit
        self<span class="token punctuation">.</span>dist_func <span class="token operator">=</span> dist_f
        self<span class="token punctuation">.</span>users_count <span class="token operator">=</span> <span class="token builtin">int</span><span class="token punctuation">(</span>np<span class="token punctuation">.</span><span class="token builtin">max</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>Y<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token operator">+</span> <span class="token number">1</span>
        self<span class="token punctuation">.</span>items_count <span class="token operator">=</span> <span class="token builtin">int</span><span class="token punctuation">(</span>np<span class="token punctuation">.</span><span class="token builtin">max</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>Y<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token operator">+</span> <span class="token number">1</span>
        self<span class="token punctuation">.</span>Pu <span class="token operator">=</span> <span class="token boolean">None</span>
        self<span class="token punctuation">.</span>Ru <span class="token operator">=</span> <span class="token boolean">None</span>

def __init__(self, Y, k, uuCF = 1, dist_f = cosine_similarity, limit = 10):

self.uuCF = uuCF

self.f = open('danhgiaNBCF.dat', 'a+')

self.Y = Y if uuCF else Y[:, [1, 0, 2]]

self.Ybar = None

self.k = k

self.limit = limit

self.dist_func = dist_f

self.users_count = int(np.max(self.Y[:, 0])) + 1

self.items_count = int(np.max(self.Y[:, 1])) + 1

self.Pu = None

self.Ru = None

Lưu ý, class NBCF đồng bộ cho cả hai phương pháp iiCF và uuCF nên khi tính toán bằng uuCF, ma trận Utility truyền vào là Y, ngược lại thì cột 0 và 1 của Y được đổi chỗ cho nhau (hoán đổi vị trí của user và item)

Hàm chuẩn hóa

Như lý thuyết trong Phần 1, mỗi rating của mỗi user được chuẩn hóa bằng cách trừ đi trung bình cộng các rating mà user đó đã đánh giá:

<span class="token keyword">def</span> <span class="token function">normalizeY</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
        users <span class="token operator">=</span> self<span class="token punctuation">.</span>Y<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">]</span>
        self<span class="token punctuation">.</span>Ybar <span class="token operator">=</span> self<span class="token punctuation">.</span>Y<span class="token punctuation">.</span>copy<span class="token punctuation">(</span><span class="token punctuation">)</span>
        self<span class="token punctuation">.</span>mu <span class="token operator">=</span> np<span class="token punctuation">.</span>zeros<span class="token punctuation">(</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>users_count<span class="token punctuation">,</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
        <span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>users_count<span class="token punctuation">)</span><span class="token punctuation">:</span>
            ids <span class="token operator">=</span> np<span class="token punctuation">.</span>where<span class="token punctuation">(</span>users <span class="token operator">==</span> i<span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">.</span>astype<span class="token punctuation">(</span><span class="token builtin">int</span><span class="token punctuation">)</span>
            ratings <span class="token operator">=</span> self<span class="token punctuation">.</span>Y<span class="token punctuation">[</span>ids<span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">]</span>
            m <span class="token operator">=</span> np<span class="token punctuation">.</span>mean<span class="token punctuation">(</span>ratings<span class="token punctuation">)</span>
            <span class="token keyword">if</span> np<span class="token punctuation">.</span>isnan<span class="token punctuation">(</span>m<span class="token punctuation">)</span><span class="token punctuation">:</span>
                m <span class="token operator">=</span> <span class="token number">0</span>
            self<span class="token punctuation">.</span>mu<span class="token punctuation">[</span>i<span class="token punctuation">]</span> <span class="token operator">=</span> m
            self<span class="token punctuation">.</span>Ybar<span class="token punctuation">[</span>ids<span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">]</span> <span class="token operator">=</span> ratings <span class="token operator">-</span> self<span class="token punctuation">.</span>mu<span class="token punctuation">[</span>i<span class="token punctuation">]</span>
        self<span class="token punctuation">.</span>Ybar <span class="token operator">=</span> sparse<span class="token punctuation">.</span>coo_matrix<span class="token punctuation">(</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>Ybar<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
            <span class="token punctuation">(</span>self<span class="token punctuation">.</span>Ybar<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">,</span> self<span class="token punctuation">.</span>Ybar<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">(</span>self<span class="token punctuation">.</span>items_count<span class="token punctuation">,</span> self<span class="token punctuation">.</span>users_count<span class="token punctuation">)</span><span class="token punctuation">)</span>
        self<span class="token punctuation">.</span>Ybar <span class="token operator">=</span> self<span class="token punctuation">.</span>Ybar<span class="token punctuation">.</span>tocsr<span class="token punctuation">(</span><span class="token punctuation">)</span>

def normalizeY(self):

users = self.Y[:, 0]

self.Ybar = self.Y.copy()

self.mu = np.zeros((self.users_count,))

for i in range(self.users_count):

ids = np.where(users == i)[0].astype(int)

ratings = self.Y[ids, 2]

m = np.mean(ratings)

if np.isnan(m):

m = 0

self.mu[i] = m

self.Ybar[ids, 2] = ratings - self.mu[i]

self.Ybar = sparse.coo_matrix((self.Ybar[:, 2],

(self.Ybar[:, 1], self.Ybar[:, 0])), (self.items_count, self.users_count))

self.Ybar = self.Ybar.tocsr()

Hàm tính khoảng cách tương đồng

<span class="token keyword">def</span> <span class="token function">similarity</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
        self<span class="token punctuation">.</span>S <span class="token operator">=</span> self<span class="token punctuation">.</span>dist_func<span class="token punctuation">(</span>self<span class="token punctuation">.</span>Ybar<span class="token punctuation">.</span>T<span class="token punctuation">,</span> self<span class="token punctuation">.</span>Ybar<span class="token punctuation">.</span>T<span class="token punctuation">)</span>

def similarity(self):

self.S = self.dist_func(self.Ybar.T, self.Ybar.T)

Hàm dự đoán rating và đưa ra danh sách items

   <span class="token keyword">def</span> <span class="token function">pred</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> u<span class="token punctuation">,</span> i<span class="token punctuation">,</span> normalized <span class="token operator">=</span> <span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
        ids <span class="token operator">=</span> np<span class="token punctuation">.</span>where<span class="token punctuation">(</span>self<span class="token punctuation">.</span>Y<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">]</span> <span class="token operator">==</span> i<span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">.</span>astype<span class="token punctuation">(</span><span class="token builtin">int</span><span class="token punctuation">)</span>
        <span class="token keyword">if</span> ids <span class="token operator">==</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">:</span>
            <span class="token keyword">return</span> <span class="token number">0</span>
        users <span class="token operator">=</span> <span class="token punctuation">(</span>self<span class="token punctuation">.</span>Y<span class="token punctuation">[</span>ids<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">.</span>astype<span class="token punctuation">(</span><span class="token builtin">int</span><span class="token punctuation">)</span>
        sim <span class="token operator">=</span> self<span class="token punctuation">.</span>S<span class="token punctuation">[</span>u<span class="token punctuation">,</span> users<span class="token punctuation">]</span>
        a <span class="token operator">=</span> np<span class="token punctuation">.</span>argsort<span class="token punctuation">(</span>sim<span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token operator">-</span>self<span class="token punctuation">.</span>k<span class="token punctuation">:</span><span class="token punctuation">]</span>
        nearest <span class="token operator">=</span> sim<span class="token punctuation">[</span>a<span class="token punctuation">]</span>
        r <span class="token operator">=</span> self<span class="token punctuation">.</span>Ybar<span class="token punctuation">[</span>i<span class="token punctuation">,</span> users<span class="token punctuation">[</span>a<span class="token punctuation">]</span><span class="token punctuation">]</span>
        
        <span class="token keyword">if</span> normalized<span class="token punctuation">:</span>
            <span class="token keyword">return</span> <span class="token punctuation">(</span>r<span class="token operator">*</span>nearest<span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token operator">/</span><span class="token punctuation">(</span>np<span class="token punctuation">.</span><span class="token builtin">abs</span><span class="token punctuation">(</span>nearest<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token builtin">sum</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">+</span> <span class="token number">1e</span><span class="token operator">-</span><span class="token number">8</span><span class="token punctuation">)</span>

        <span class="token keyword">return</span> <span class="token punctuation">(</span>r<span class="token operator">*</span>nearest<span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token operator">/</span><span class="token punctuation">(</span>np<span class="token punctuation">.</span><span class="token builtin">abs</span><span class="token punctuation">(</span>nearest<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token builtin">sum</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">+</span> <span class="token number">1e</span><span class="token operator">-</span><span class="token number">8</span><span class="token punctuation">)</span> <span class="token operator">+</span> self<span class="token punctuation">.</span>mu<span class="token punctuation">[</span>u<span class="token punctuation">]</span>
        
        
    <span class="token keyword">def</span> <span class="token function">_pred</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> u<span class="token punctuation">,</span> i<span class="token punctuation">,</span> normalized <span class="token operator">=</span> <span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
        <span class="token keyword">if</span> self<span class="token punctuation">.</span>uuCF<span class="token punctuation">:</span> <span class="token keyword">return</span> self<span class="token punctuation">.</span>pred<span class="token punctuation">(</span>u<span class="token punctuation">,</span> i<span class="token punctuation">,</span> normalized<span class="token punctuation">)</span>
        <span class="token keyword">return</span> self<span class="token punctuation">.</span>pred<span class="token punctuation">(</span>i<span class="token punctuation">,</span> u<span class="token punctuation">,</span> normalized<span class="token punctuation">)</span>

def pred(self, u, i, normalized = 1):

ids = np.where(self.Y[:, 1] == i)[0].astype(int)

if ids == []:

return 0

users = (self.Y[ids, 0]).astype(int)

sim = self.S[u, users]

a = np.argsort(sim)[-self.k:]

nearest = sim[a]

r = self.Ybar[i, users[a]]

if normalized:

return (r*nearest)[0]/(np.abs(nearest).sum() + 1e-8)

def _pred(self, u, i, normalized = 1):

if self.uuCF: return self.pred(u, i, normalized)

return self.pred(i, u, normalized)

Lưu ý, với uuCF = 0, mình sẽ thực hiện hàm đổi chỗ 2 tham số u và i khi thực hiện hàm pred

<span class="token keyword">def</span> <span class="token function">recommend</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> u<span class="token punctuation">)</span><span class="token punctuation">:</span>
        <span class="token keyword">if</span> self<span class="token punctuation">.</span>uuCF<span class="token punctuation">:</span>
            ids <span class="token operator">=</span> np<span class="token punctuation">.</span>where<span class="token punctuation">(</span>self<span class="token punctuation">.</span>Y<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">]</span> <span class="token operator">==</span> u<span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">.</span>astype<span class="token punctuation">(</span><span class="token builtin">int</span><span class="token punctuation">)</span>
            items_rated_by_user <span class="token operator">=</span> self<span class="token punctuation">.</span>Y<span class="token punctuation">[</span>ids<span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">.</span>tolist<span class="token punctuation">(</span><span class="token punctuation">)</span>
            n <span class="token operator">=</span> self<span class="token punctuation">.</span>items_count
        <span class="token keyword">else</span><span class="token punctuation">:</span>
            ids <span class="token operator">=</span> np<span class="token punctuation">.</span>where<span class="token punctuation">(</span>self<span class="token punctuation">.</span>Y<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">]</span> <span class="token operator">==</span> u<span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">.</span>astype<span class="token punctuation">(</span><span class="token builtin">int</span><span class="token punctuation">)</span>
            items_rated_by_user <span class="token operator">=</span> self<span class="token punctuation">.</span>Y<span class="token punctuation">[</span>ids<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">.</span>tolist<span class="token punctuation">(</span><span class="token punctuation">)</span>
            n <span class="token operator">=</span> self<span class="token punctuation">.</span>users_count
        a <span class="token operator">=</span> np<span class="token punctuation">.</span>zeros<span class="token punctuation">(</span><span class="token punctuation">(</span>n<span class="token punctuation">,</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
        recommended_items <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
        <span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>n<span class="token punctuation">)</span><span class="token punctuation">:</span>
            <span class="token keyword">if</span> i <span class="token operator">not</span> <span class="token keyword">in</span> items_rated_by_user<span class="token punctuation">:</span>
                a<span class="token punctuation">[</span>i<span class="token punctuation">]</span> <span class="token operator">=</span> self<span class="token punctuation">.</span>_pred<span class="token punctuation">(</span>u<span class="token punctuation">,</span> i<span class="token punctuation">)</span>
        <span class="token keyword">if</span> <span class="token builtin">len</span><span class="token punctuation">(</span>a<span class="token punctuation">)</span> <span class="token operator">&lt;</span> self<span class="token punctuation">.</span>limit<span class="token punctuation">:</span>
            recommended_items <span class="token operator">=</span> np<span class="token punctuation">.</span>argsort<span class="token punctuation">(</span>a<span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token operator">-</span><span class="token builtin">len</span><span class="token punctuation">(</span>a<span class="token punctuation">)</span><span class="token punctuation">:</span><span class="token punctuation">]</span>
        <span class="token keyword">else</span><span class="token punctuation">:</span>
            recommended_items <span class="token operator">=</span> np<span class="token punctuation">.</span>argsort<span class="token punctuation">(</span>a<span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token operator">-</span>self<span class="token punctuation">.</span>limit<span class="token punctuation">:</span><span class="token punctuation">]</span>
        <span class="token keyword">return</span> recommended_items

def recommend(self, u):

if self.uuCF:

ids = np.where(self.Y[:, 0] == u)[0].astype(int)

items_rated_by_user = self.Y[ids, 1].tolist()

n = self.items_count

else:

ids = np.where(self.Y[:, 1] == u)[0].astype(int)

items_rated_by_user = self.Y[ids, 0].tolist()

n = self.users_count

a = np.zeros((n,))

recommended_items = []

for i in range(n):

if i not in items_rated_by_user:

a[i] = self._pred(u, i)

if len(a) < self.limit:

recommended_items = np.argsort(a)[-len(a):]

else:

recommended_items = np.argsort(a)[-self.limit:]

return recommended_items

3. Đánh giá thuật toán

Tương tự với Content-base, ở đây mình cũng đánh giá thuật toán bằng RMSE và precision recall . Các bạn có thể tham khảo một chút:

<span class="token keyword">def</span> <span class="token function">RMSE</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> data_size<span class="token punctuation">,</span> Data_test<span class="token punctuation">,</span> test_size <span class="token operator">=</span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
        SE <span class="token operator">=</span> <span class="token number">0</span>
        n_tests <span class="token operator">=</span> Data_test<span class="token punctuation">.</span>shape<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span>
        <span class="token keyword">for</span> n <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>n_tests<span class="token punctuation">)</span><span class="token punctuation">:</span>
            <span class="token keyword">if</span> Data_test<span class="token punctuation">[</span>n<span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">]</span> <span class="token operator">==</span> <span class="token number">1681</span><span class="token punctuation">:</span>
                pred <span class="token operator">=</span> <span class="token number">0</span>
            <span class="token keyword">else</span><span class="token punctuation">:</span>
                pred <span class="token operator">=</span> self<span class="token punctuation">.</span>_pred<span class="token punctuation">(</span>Data_test<span class="token punctuation">[</span>n<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">,</span> Data_test<span class="token punctuation">[</span>n<span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">,</span> normalized <span class="token operator">=</span> <span class="token number">0</span><span class="token punctuation">)</span>
            SE <span class="token operator">+=</span> <span class="token punctuation">(</span>pred <span class="token operator">-</span> Data_test<span class="token punctuation">[</span>n<span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token operator">**</span><span class="token number">2</span> 

        RMSE <span class="token operator">=</span> np<span class="token punctuation">.</span>sqrt<span class="token punctuation">(</span>SE<span class="token operator">/</span>n_tests<span class="token punctuation">)</span>

        <span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'%s::%d::%d::cosine_similarity::%r::%rrn'</span> <span class="token operator">%</span> <span class="token punctuation">(</span><span class="token builtin">str</span><span class="token punctuation">(</span>data_size<span class="token punctuation">)</span><span class="token punctuation">,</span> self<span class="token punctuation">.</span>uuCF<span class="token punctuation">,</span> self<span class="token punctuation">.</span>k<span class="token punctuation">,</span> test_size<span class="token punctuation">,</span> RMSE<span class="token punctuation">)</span><span class="token punctuation">)</span>
        self<span class="token punctuation">.</span>f<span class="token punctuation">.</span>write<span class="token punctuation">(</span><span class="token string">'%s::%d::%d::cosine_similarity::%r::%rrn'</span> <span class="token operator">%</span> <span class="token punctuation">(</span><span class="token builtin">str</span><span class="token punctuation">(</span>data_size<span class="token punctuation">)</span><span class="token punctuation">,</span> self<span class="token punctuation">.</span>uuCF<span class="token punctuation">,</span> self<span class="token punctuation">.</span>k<span class="token punctuation">,</span> test_size<span class="token punctuation">,</span> RMSE<span class="token punctuation">)</span><span class="token punctuation">)</span>
        
    <span class="token keyword">def</span> <span class="token function">evaluate</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> data_size<span class="token punctuation">,</span> Data_test<span class="token punctuation">,</span> test_size <span class="token operator">=</span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
        sum_p <span class="token operator">=</span> <span class="token number">0</span>
        n <span class="token operator">=</span> self<span class="token punctuation">.</span>users_count <span class="token keyword">if</span> self<span class="token punctuation">.</span>uuCF <span class="token keyword">else</span> self<span class="token punctuation">.</span>items_count
        self<span class="token punctuation">.</span>Pu <span class="token operator">=</span> np<span class="token punctuation">.</span>zeros<span class="token punctuation">(</span><span class="token punctuation">(</span>n<span class="token punctuation">,</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
        <span class="token keyword">for</span> u <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>n<span class="token punctuation">)</span><span class="token punctuation">:</span>
            recommended_items <span class="token operator">=</span> self<span class="token punctuation">.</span>recommend<span class="token punctuation">(</span>u<span class="token punctuation">)</span>
            ids <span class="token operator">=</span> np<span class="token punctuation">.</span>where<span class="token punctuation">(</span>Data_test<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">]</span> <span class="token operator">==</span> u<span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span>
            rated_items <span class="token operator">=</span> Data_test<span class="token punctuation">[</span>ids<span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">]</span>
            <span class="token keyword">for</span> i <span class="token keyword">in</span> recommended_items<span class="token punctuation">:</span>
                <span class="token keyword">if</span> i <span class="token keyword">in</span> rated_items<span class="token punctuation">:</span>
                    self<span class="token punctuation">.</span>Pu<span class="token punctuation">[</span>u<span class="token punctuation">]</span> <span class="token operator">+=</span> <span class="token number">1</span>
            sum_p <span class="token operator">+=</span> self<span class="token punctuation">.</span>Pu<span class="token punctuation">[</span>u<span class="token punctuation">]</span>
        p <span class="token operator">=</span> sum_p<span class="token operator">/</span><span class="token punctuation">(</span>n <span class="token operator">*</span> self<span class="token punctuation">.</span>limit<span class="token punctuation">)</span>
        r <span class="token operator">=</span> sum_p<span class="token operator">/</span><span class="token punctuation">(</span>Data_test<span class="token punctuation">.</span>shape<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span> <span class="token operator">+</span> <span class="token number">1</span><span class="token punctuation">)</span>
        <span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'%s::%d::%d::cosine_similarity::%r::%rrn'</span> <span class="token operator">%</span> <span class="token punctuation">(</span><span class="token builtin">str</span><span class="token punctuation">(</span>data_size<span class="token punctuation">)</span><span class="token punctuation">,</span> self<span class="token punctuation">.</span>uuCF<span class="token punctuation">,</span> self<span class="token punctuation">.</span>limit<span class="token punctuation">,</span> p<span class="token punctuation">,</span> r<span class="token punctuation">)</span><span class="token punctuation">)</span>
        self<span class="token punctuation">.</span>f<span class="token punctuation">.</span>write<span class="token punctuation">(</span><span class="token string">'%s::%d::%d::cosine_similarity::%r::%rrn'</span> <span class="token operator">%</span> <span class="token punctuation">(</span><span class="token builtin">str</span><span class="token punctuation">(</span>data_size<span class="token punctuation">)</span><span class="token punctuation">,</span> self<span class="token punctuation">.</span>uuCF<span class="token punctuation">,</span> self<span class="token punctuation">.</span>limit<span class="token punctuation">,</span> p<span class="token punctuation">,</span> r<span class="token punctuation">)</span><span class="token punctuation">)</span>

def RMSE(self, data_size, Data_test, test_size = 0):

SE = 0

n_tests = Data_test.shape[0]

for n in range(n_tests):

if Data_test[n, 1] == 1681:

pred = 0

else:

pred = self._pred(Data_test[n, 0], Data_test[n, 1], normalized = 0)

SE += (pred - Data_test[n, 2])**2

RMSE = np.sqrt(SE/n_tests)

print('%s::%d::%d::cosine_similarity::%r::%rrn' % (str(data_size), self.uuCF, self.k, test_size, RMSE))

self.f.write('%s::%d::%d::cosine_similarity::%r::%rrn' % (str(data_size), self.uuCF, self.k, test_size, RMSE))

def evaluate(self, data_size, Data_test, test_size = 0):

sum_p = 0

n = self.users_count if self.uuCF else self.items_count

self.Pu = np.zeros((n,))

for u in range(n):

recommended_items = self.recommend(u)

ids = np.where(Data_test[:, 0] == u)[0]

rated_items = Data_test[ids, 1]

for i in recommended_items:

if i in rated_items:

self.Pu[u] += 1

sum_p += self.Pu[u]

p = sum_p/(n * self.limit)

r = sum_p/(Data_test.shape[0] + 1)

print('%s::%d::%d::cosine_similarity::%r::%rrn' % (str(data_size), self.uuCF, self.limit, p, r))

self.f.write('%s::%d::%d::cosine_similarity::%r::%rrn' % (str(data_size), self.uuCF, self.limit, p, r))

Vậy là kết thúc 2 phần của Neighborhood-based Collaborative Filtering. Còn Matrix Factorization nữa thôi là mình sẽ kết thúc chủ đề này. Hy vọng là mình sẽ hoàn thành được

Dưới đây là link source code và tài liệu tham khảo. Hẹn gặp bạn ở bài viết tiếp theo nhé

Source code

Link tài liệu tham khảo

Chia sẻ bài viết ngay

Nguồn bài viết : Viblo

Neighborhood-based Collaborative Filtering: Phương pháp gợi ý dựa trên láng giềng gần nhất (P2)

1. Về hàm khoảng cách

2. Xây dựng class NBCF

Hàm khởi tạo:

Hàm chuẩn hóa

Hàm tính khoảng cách tương đồng

Hàm dự đoán rating và đưa ra danh sách items

3. Đánh giá thuật toán

TikTok trở thành nền tảng mảng xã hội lớn thứ hai ở Nam Phi

Mất giá nhanh nhất sau 9 tháng ra mắt, iPhone 14 Pro Max tiếp tục phá đáy tại Việt Nam

Hướng dẫn sử dụng ngôn ngữ R cho người mới bắt đầu

10 plugin cần thiết của SublimeText dành cho các lập trình viên JavaScript