Backpropni qo'lda hisoblash - Neyron tarmoqlar - noldan

Formulani ko'rish va uni tushunish - ikki xil narsa. Bu bo'limda kichik tarmoqni raqamlar bilan, har qadamni alohida chop etib hisoblaymiz.

Tarmoq va ma'lumot #

Eng kichik ma'noli tarmoq: 2 kirish → 2 yashirin → 1 chiqish. Bitta namuna.

Python

import numpy as np


def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))


X = np.array([[1.0, 2.0]])            # bitta namuna, ikkita belgi
y = np.array([[1.0]])                 # to'g'ri javob

W1 = np.array([[0.10, 0.40],
               [0.20, 0.30]])         # (2, 2)
b1 = np.array([0.05, -0.05])
W2 = np.array([[0.60],
               [-0.70]])              # (2, 1)
b2 = np.array([0.10])

Python

print("X :", X, X.shape)
print("y :", y, y.shape)
print("W1:", W1.tolist(), W1.shape)
print("W2:", W2.tolist(), W2.shape)

Natija

X : [[1. 2.]] (1, 2)
y : [[1.]] (1, 1)
W1: [[0.1, 0.4], [0.2, 0.3]] (2, 2)
W2: [[0.6], [-0.7]] (2, 1)

1-qadam: `Z1` #

Natija

Z1 = X @ W1 + b1

Python

Z1 = X @ W1 + b1

qolda_1 = 1.0 * 0.10 + 2.0 * 0.20 + 0.05
qolda_2 = 1.0 * 0.40 + 2.0 * 0.30 + (-0.05)

print("Z1 (kod):  ", np.round(Z1, 6))
print("Z1 (qo'lda):", [round(qolda_1, 6), round(qolda_2, 6)])
print("mos keladi:", bool(np.allclose(Z1, [[qolda_1, qolda_2]])))

Natija

Z1 (kod):   [[0.55 0.95]]
Z1 (qo'lda): [0.55, 0.95]
mos keladi: True

Birinchi yashirin neyron: 1·0.10 + 2·0.20 + 0.05 = 0.55. Ikkinchisi: 1·0.40 + 2·0.30 - 0.05 = 0.95.

2-qadam: `A1` (ReLU) #

Python

Z1 = X @ W1 + b1
A1 = np.maximum(0.0, Z1)
print("Z1:", np.round(Z1, 4))
print("A1:", np.round(A1, 4))
print("ikkalasi ham musbat - ReLU hech nimani kesmadi:", bool((Z1 > 0).all()))

Natija

Z1: [[0.55 0.95]]
A1: [[0.55 0.95]]
ikkalasi ham musbat - ReLU hech nimani kesmadi: True

3-qadam: `Z2` va `A2` #

Python

Z1 = X @ W1 + b1
A1 = np.maximum(0.0, Z1)
Z2 = A1 @ W2 + b2
A2 = sigmoid(Z2)

qolda_z2 = 0.55 * 0.60 + 0.95 * (-0.70) + 0.10

print("Z2 (kod):   ", np.round(Z2, 6))
print("Z2 (qo'lda):", round(qolda_z2, 6))
print("A2 = sigmoid(Z2):", np.round(A2, 6))

Natija

Z2 (kod):    [[-0.235]]
Z2 (qo'lda): -0.235
A2 = sigmoid(Z2): [[0.441519]]

Tarmoq 0.4415 dedi, to'g'ri javob esa 1.0. Model ikkilanmoqda va biroz noto'g'ri tomonga qaragan.

Har son ma'lumot beradi: ishorasi yo'nalishni, kattaligi ta'sirni

4-qadam: yo'qotish #

Python

def krossentropiya(y, a, eps=1e-12):
    a = np.clip(a, eps, 1 - eps)
    return float(-np.mean(y * np.log(a) + (1 - y) * np.log(1 - a)))


Z1 = X @ W1 + b1
A1 = np.maximum(0.0, Z1)
A2 = sigmoid(A1 @ W2 + b2)
L = krossentropiya(y, A2)

print("A2 =", np.round(A2, 6))
print("L  =", round(L, 6))
print("qo'lda: -log(0.441519) =", round(float(-np.log(0.441519)), 6))
print("tasodifiy modeldan yomonroqmi:", L > float(np.log(2)))

Natija

A2 = [[0.441519]]
L  = 0.817534
qo'lda: -log(0.441519) = 0.817534
tasodifiy modeldan yomonroqmi: True

y = 1 bo'lgani uchun formulaning ikkinchi qismi ((1-y)·log(1-a)) nolga ko'paytiriladi va faqat -log(a) qoladi.

5-qadam: `dZ2` #

Python

Z1 = X @ W1 + b1
A1 = np.maximum(0.0, Z1)
A2 = sigmoid(A1 @ W2 + b2)
n = len(y)

dZ2 = (A2 - y) / n
print("A2 - y =", np.round(A2 - y, 6))
print("dZ2    =", np.round(dZ2, 6), " (n =", n, ")")
print("manfiy - demak A2 ni oshirish kerak:", bool((dZ2 < 0).all()))

Natija

A2 - y = [[-0.558481]]
dZ2    = [[-0.558481]]  (n = 1 )
manfiy - demak A2 ni oshirish kerak: True

Gradient ishorasini o'qish

dZ2 manfiy. Yangilanish qoidasi Z2 = Z2 - tezlik·dZ2, ya'ni manfiy gradient Z2 ni oshiradi.

Bu to'g'ri: A2 = 0.44 bo'lib, kerakli javob 1.0. Z2 oshsa, sigmoid ham oshadi.

Gradient ishorasini o'qishni odat qiling:

Gradient	Ma'nosi
Manfiy	Bu qiymatni oshirish yo'qotishni kamaytiradi
Musbat	Bu qiymatni kamaytirish kerak
Nolga yaqin	Bu qiymat deyarli ta'sir qilmayapti

Uchinchi holat ayniqsa muhim: agar biror parametr gradienti doim nolga yaqin bo'lsa, u o'qimayapti - va sababini qidirish kerak (o'lik ReLU, yo'qolayotgan gradient, yoki shunchaki keraksiz parametr).

6-qadam: `dW2` va `db2` #

Python

Z1 = X @ W1 + b1
A1 = np.maximum(0.0, Z1)
A2 = sigmoid(A1 @ W2 + b2)
dZ2 = (A2 - y) / len(y)

dW2 = A1.T @ dZ2
db2 = dZ2.sum(axis=0)

qolda_dW2 = [0.55 * float(dZ2[0, 0]), 0.95 * float(dZ2[0, 0])]

print("dW2 (kod):   ", np.round(dW2.ravel(), 6))
print("dW2 (qo'lda):", [round(v, 6) for v in qolda_dW2])
print("db2:", np.round(db2, 6))
print("shakl W2 bilan bir xilmi:", dW2.shape == W2.shape)

Natija

dW2 (kod):    [-0.307165 -0.530557]
dW2 (qo'lda): [-0.307165, -0.530557]
db2: [-0.558481]
shakl W2 bilan bir xilmi: True

Ikkinchi yashirin neyron (A1 = 0.95) birinchisidan (0.55) kuchliroq signal bergani uchun uning og'irligi ham kattaroq gradient oldi.

7-qadam: `dA1` va `dZ1` #

Python

Z1 = X @ W1 + b1
A1 = np.maximum(0.0, Z1)
A2 = sigmoid(A1 @ W2 + b2)
dZ2 = (A2 - y) / len(y)

dA1 = dZ2 @ W2.T
dZ1 = dA1 * (Z1 > 0)

qolda_dA1 = [float(dZ2[0, 0]) * 0.60, float(dZ2[0, 0]) * (-0.70)]

print("dA1 (kod):   ", np.round(dA1, 6))
print("dA1 (qo'lda):", [round(v, 6) for v in qolda_dA1])
print("ReLU maskasi:", (Z1 > 0).astype(int))
print("dZ1:", np.round(dZ1, 6))

Natija

dA1 (kod):    [[-0.335089  0.390937]]
dA1 (qo'lda): [-0.335089, 0.390937]
ReLU maskasi: [[1 1]]
dZ1: [[-0.335089  0.390937]]

Ishoralar W2 dan meros bo'ladi

dA1 ning birinchi komponenti manfiy, ikkinchisi musbat. Sabab W2 ning ishoralarida: [0.60, -0.70].

Mantiq: birinchi yashirin neyron chiqishga musbat hissa qo'shadi (W2 = 0.6), demak A2 ni oshirish uchun uni ham oshirish kerak - gradient manfiy.

Ikkinchisi esa manfiy hissa qo'shadi (W2 = -0.7), demak A2 ni oshirish uchun uni kamaytirish kerak - gradient musbat.

Backprop aynan shu mantiqiy zanjirni avtomatik bajaradi. Uni bir marta qo'lda kuzatgan odam keyinchalik formulaga ishonch bilan qaraydi.

8-qadam: `dW1` va `db1` #

Python

Z1 = X @ W1 + b1
A1 = np.maximum(0.0, Z1)
A2 = sigmoid(A1 @ W2 + b2)
dZ2 = (A2 - y) / len(y)
dZ1 = (dZ2 @ W2.T) * (Z1 > 0)

dW1 = X.T @ dZ1
db1 = dZ1.sum(axis=0)

print("dW1:")
print(np.round(dW1, 6))
print("db1:", np.round(db1, 6))
print("dW1 shakli W1 bilan bir xilmi:", dW1.shape == W1.shape)
print("ikkinchi satr birinchisidan 2 barobar kattami:",
      bool(np.allclose(dW1[1], 2 * dW1[0])))

Natija

dW1:
[[-0.335089  0.390937]
 [-0.670177  0.781874]]
db1: [-0.335089  0.390937]
dW1 shakli W1 bilan bir xilmi: True
ikkinchi satr birinchisidan 2 barobar kattami: True

Ikkinchi satr birinchisining aynan ikki barobari - chunki X = [1.0, 2.0], ya'ni ikkinchi belgi ikki barobar katta.

Hammasini sonli gradient bilan tekshirish #

Python

def krossentropiya(y, a, eps=1e-12):
    a = np.clip(a, eps, 1 - eps)
    return float(-np.mean(y * np.log(a) + (1 - y) * np.log(1 - a)))


def yoqotish_hisobla(W1, b1, W2, b2):
    A1 = np.maximum(0.0, X @ W1 + b1)
    A2 = sigmoid(A1 @ W2 + b2)
    return krossentropiya(y, A2)


def sonli(param, ozgartir, h=1e-6):
    natija = np.zeros_like(param)
    tekis = param.ravel()
    chiqish = natija.ravel()
    for i in range(tekis.size):
        eski = tekis[i]
        tekis[i] = eski + h; yuqori = ozgartir()
        tekis[i] = eski - h; past = ozgartir()
        tekis[i] = eski
        chiqish[i] = (yuqori - past) / (2 * h)
    return natija


# analitik
A1 = np.maximum(0.0, X @ W1 + b1)
A2 = sigmoid(A1 @ W2 + b2)
dZ2 = (A2 - y) / len(y)
dW2 = A1.T @ dZ2
dZ1 = (dZ2 @ W2.T) * ((X @ W1 + b1) > 0)
dW1 = X.T @ dZ1

W1_nusxa = W1.copy(); W2_nusxa = W2.copy()
sonli_W1 = sonli(W1_nusxa, lambda: yoqotish_hisobla(W1_nusxa, b1, W2, b2))
sonli_W2 = sonli(W2_nusxa, lambda: yoqotish_hisobla(W1, b1, W2_nusxa, b2))

print("W1 gradienti mos keladi:", bool(np.allclose(dW1, sonli_W1, atol=1e-6)))
print("W2 gradienti mos keladi:", bool(np.allclose(dW2, sonli_W2, atol=1e-6)))
print("eng katta ayirma W1:", f"{np.abs(dW1 - sonli_W1).max():.2e}")
print("eng katta ayirma W2:", f"{np.abs(dW2 - sonli_W2).max():.2e}")

Natija

W1 gradienti mos keladi: True
W2 gradienti mos keladi: True
eng katta ayirma W1: 1.26e-10
eng katta ayirma W2: 9.51e-11

Qo'lda hisoblangan formulalar sonli gradientga o'ninchi kasrgacha mos keldi - ayirma 1e-10 tartibida.

Bitta yangilanish #

Python

def krossentropiya(y, a, eps=1e-12):
    a = np.clip(a, eps, 1 - eps)
    return float(-np.mean(y * np.log(a) + (1 - y) * np.log(1 - a)))


W1y, b1y = W1.copy(), b1.copy()
W2y, b2y = W2.copy(), b2.copy()
tezlik = 1.0

A1 = np.maximum(0.0, X @ W1y + b1y)
A2 = sigmoid(A1 @ W2y + b2y)
print("oldin:  A2 =", np.round(A2, 6), " L =", round(krossentropiya(y, A2), 6))

dZ2 = (A2 - y) / len(y)
dW2 = A1.T @ dZ2; db2 = dZ2.sum(axis=0)
dZ1 = (dZ2 @ W2y.T) * ((X @ W1y + b1y) > 0)
dW1 = X.T @ dZ1; db1 = dZ1.sum(axis=0)

W2y -= tezlik * dW2; b2y -= tezlik * db2
W1y -= tezlik * dW1; b1y -= tezlik * db1

A1 = np.maximum(0.0, X @ W1y + b1y)
A2 = sigmoid(A1 @ W2y + b2y)
print("keyin:  A2 =", np.round(A2, 6), " L =", round(krossentropiya(y, A2), 6))

Natija

oldin:  A2 = [[0.441519]]  L = 0.817534
keyin:  A2 = [[0.951722]]  L = 0.049482

Bitta qadamda bashorat 0.44 dan 0.95 ga ko'tarildi, yo'qotish esa 0.82 dan 0.05 ga tushdi.

tezlik = 1.0 - bu misolda ishladi, amalda emas

Bitta namuna va kichik tarmoqda 1.0 tezlik xavfsiz bo'ldi. Haqiqiy tarmoqda bu deyarli har doim portlashga olib keladi.

Sabab: bu yerda gradientlar kichik (0.3 atrofida) va parametrlar kam. Yuz minglab parametrli tarmoqda gradientlar to'planadi va bitta katta qadam hamma narsani buzadi.

Amaliy boshlang'ich qiymatlar:

Optimizator	Odatiy tezlik
SGD	`0.01` - `0.1`
SGD + momentum	`0.01`
Adam	`0.001`

15-bo'limda bularni batafsil ko'ramiz. Hozircha qoida: kichikdan boshlang.

Amaliy topshiriq

Z1 ni qo'lda hisoblab, kod natijasi bilan solishtiring.
ReLU dan keyin nima o'zgarganini tekshiring.
Z2 va A2 ni hisoblang.
Yo'qotishni -log(a) orqali tasdiqlang.
dZ2 ishorasini o'qing va nima anglatishini ayting.
dW2 ni qo'lda hisoblang.
dA1 ishoralari W2 dan qanday kelib chiqqanini tushuntiring.
dW1 ning ikkinchi satri nega ikki barobar ekanini ayting.
Barcha gradientlarni sonli gradient bilan tekshiring.
Bitta yangilanish bajarib, yo'qotishning tushishini ko'ring.

Xulosa #

Kichik tarmoqni qo'lda hisoblash formulalarga ishonch beradi.
Z1 = X @ W1 + b1 - har neyron o'z kirishlarini yig'adi.
L = -log(a) - y = 1 bo'lganda formula soddalashadi.
dZ2 = A2 - y - gradient manfiy bo'lsa, qiymatni oshirish kerak.
dW2 = A1.T @ dZ2 - kuchliroq neyron kattaroq gradient oladi.
dA1 ishoralari W2 dan meros bo'ladi.
ReLU maskasi (Z1 > 0) - manfiy neyronlarga gradient o'tmaydi.
dW1 satrlari kirish belgilariga proporsional.
Sonli gradient bilan tekshiruv - formulaning yagona isboti.
tezlik = 1.0 faqat o'yinchoq misolda xavfsiz.

Keyingi bo'limda bu formulalarni ixtiyoriy chuqurlikdagi tarmoq uchun umumlashtiramiz.

Tarmoq va ma'lumot #

1-qadam: Z1 #

2-qadam: A1 (ReLU) #

3-qadam: Z2 va A2 #

4-qadam: yo'qotish #

5-qadam: dZ2 #

6-qadam: dW2 va db2 #

7-qadam: dA1 va dZ1 #

8-qadam: dW1 va db1 #

Hammasini sonli gradient bilan tekshirish #

Bitta yangilanish #

Xulosa #

Xatolik topdingizmi?

1-qadam: `Z1` #

2-qadam: `A1` (ReLU) #

3-qadam: `Z2` va `A2` #

5-qadam: `dZ2` #

6-qadam: `dW2` va `db2` #

7-qadam: `dA1` va `dZ1` #

8-qadam: `dW1` va `db1` #