Orqaga tarqalish - g'oyasi - Neyron tarmoqlar - noldan

Orqaga tarqalish (backpropagation) - zanjir qoidasini tarmoqqa tartibli qo'llash usuli. U barcha gradientlarni bitta o'tishda hisoblaydi.

Muammo: `W1` xatoga qanday ta'sir qiladi #

Ikki qatlamli tarmoqda W1 yo'qotishga bevosita ta'sir qilmaydi. Uning ta'siri zanjir bo'ylab o'tadi:

Natija

W1 → Z1 → A1 → Z2 → A2 → L

Demak dL/dW1 ni topish uchun beshta hosilani ko'paytirish kerak.

Oldinga - qiymat, orqaga - mas'uliyat

Eng oddiy holat: bitta neyron #

Python

import numpy as np


def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))


def krossentropiya(y, a, eps=1e-12):
    a = np.clip(a, eps, 1 - eps)
    return float(-np.mean(y * np.log(a) + (1 - y) * np.log(1 - a)))

Python

x = np.array([2.0, -1.0])
w = np.array([0.5, 0.3])
b = 0.1
y = 1.0

z = float(x @ w + b)
a = float(sigmoid(z))
L = krossentropiya(np.array([y]), np.array([a]))

print(f"z = {z:.4f}")
print(f"a = {a:.6f}")
print(f"L = {L:.6f}")

Natija

z = 0.8000
a = 0.689974
L = 0.371101

Endi gradientlarni orqaga hisoblaymiz:

Python

x = np.array([2.0, -1.0])
w = np.array([0.5, 0.3])
b = 0.1
y = 1.0

z = float(x @ w + b)
a = float(sigmoid(z))

dL_dz = a - y              # 6-bo'limdagi natija
dL_dw = dL_dz * x          # z = x @ w + b  =>  dz/dw = x
dL_db = dL_dz              # dz/db = 1

print(f"dL/dz = {dL_dz:.6f}")
print(f"dL/dw = {np.round(dL_dw, 6)}")
print(f"dL/db = {dL_db:.6f}")

Natija

dL/dz = -0.310026
dL/dw = [-0.620051  0.310026]
dL/db = -0.310026

Sonli gradient bilan tekshiramiz:

Python

x = np.array([2.0, -1.0])
w = np.array([0.5, 0.3])
b = 0.1
y = np.array([1.0])


def yoqotish(w, b):
    a = sigmoid(np.array([float(x @ w + b)]))
    return krossentropiya(y, a)


h = 1e-6
sonli_w = np.zeros(2)
for i in range(2):
    w_yuqori = w.copy(); w_yuqori[i] += h
    w_past = w.copy(); w_past[i] -= h
    sonli_w[i] = (yoqotish(w_yuqori, b) - yoqotish(w_past, b)) / (2 * h)
sonli_b = (yoqotish(w, b + h) - yoqotish(w, b - h)) / (2 * h)

a = float(sigmoid(float(x @ w + b)))
formula_w = (a - 1.0) * x
formula_b = a - 1.0

print("sonli   dL/dw:", np.round(sonli_w, 6))
print("formula dL/dw:", np.round(formula_w, 6))
print("mos keladi:", bool(np.allclose(sonli_w, formula_w, atol=1e-5)))
print("sonli dL/db:", round(sonli_b, 6), " formula:", round(formula_b, 6))

Natija

sonli   dL/dw: [-0.620051  0.310026]
formula dL/dw: [-0.620051  0.310026]
mos keladi: True
sonli dL/db: -0.310026  formula: -0.310026

dL/dw = dL/dz · x - eng ko'p ishlatiladigan naqsh

z = x·w + b bo'lgani uchun dz/dw = x. Ya'ni og'irlikning gradienti - uning kirishiga ko'paytirilgan yuqoridan kelgan gradient.

Buning intuitiv ma'nosi bor:

Kirish `x`	Og'irlik gradienti
Katta	Katta - bu og'irlik ko'p ta'sir qildi
Nolga yaqin	Nolga yaqin - bu og'irlik deyarli qatnashmadi
Manfiy	Ishora almashadi

Ikkinchi qator muhim: agar kirish belgisi doim nol bo'lsa, unga bog'langan og'irlik hech qachon o'zgarmaydi.

Bu 17-bo'limdagi normalizatsiya sababi ham: turli masshtabdagi kirishlar turli tezlikda o'qiydigan og'irliklarni yaratadi.

Ikki qatlam: qadam-baqadam #

Python

gen = np.random.default_rng(1)
X = np.array([[1.0, 0.5], [-0.5, 2.0], [1.5, -1.0]])
y = np.array([[1.0], [0.0], [1.0]])

W1 = gen.normal(0, 0.5, (2, 3))
b1 = np.zeros(3)
W2 = gen.normal(0, 0.5, (3, 1))
b2 = np.zeros(1)

# --- oldinga
Z1 = X @ W1 + b1
A1 = np.maximum(0.0, Z1)
Z2 = A1 @ W2 + b2
A2 = sigmoid(Z2)

print("Z1:", Z1.shape, " A1:", A1.shape)
print("Z2:", Z2.shape, " A2:", A2.shape)
print("yo'qotish:", round(krossentropiya(y, A2), 6))

Natija

Z1: (3, 3)  A1: (3, 3)
Z2: (3, 1)  A2: (3, 1)
yo'qotish: 0.737786

Python

gen = np.random.default_rng(1)
X = np.array([[1.0, 0.5], [-0.5, 2.0], [1.5, -1.0]])
y = np.array([[1.0], [0.0], [1.0]])
W1 = gen.normal(0, 0.5, (2, 3)); b1 = np.zeros(3)
W2 = gen.normal(0, 0.5, (3, 1)); b2 = np.zeros(1)

Z1 = X @ W1 + b1
A1 = np.maximum(0.0, Z1)
Z2 = A1 @ W2 + b2
A2 = sigmoid(Z2)
n = len(y)

# --- orqaga
dZ2 = (A2 - y) / n
dW2 = A1.T @ dZ2
db2 = dZ2.sum(axis=0)
dA1 = dZ2 @ W2.T
dZ1 = dA1 * (Z1 > 0)
dW1 = X.T @ dZ1
db1 = dZ1.sum(axis=0)

for nom, g in [("dZ2", dZ2), ("dW2", dW2), ("db2", db2),
               ("dA1", dA1), ("dZ1", dZ1), ("dW1", dW1), ("db1", db1)]:
    print(f"{nom}: shakl {str(g.shape):<8}")

Natija

dZ2: shakl (3, 1)  
dW2: shakl (3, 1)  
db2: shakl (1,)    
dA1: shakl (3, 3)  
dZ1: shakl (3, 3)  
dW1: shakl (2, 3)  
db1: shakl (3,)

Gradient shakli parametr shakli bilan bir xil bo'lishi shart

Bu eng foydali tekshiruv:

Parametr	Shakli	Gradient	Shakli
`W1`	`(2, 3)`	`dW1`	`(2, 3)`
`b1`	`(3,)`	`db1`	`(3,)`
`W2`	`(3, 1)`	`dW2`	`(3, 1)`
`b2`	`(1,)`	`db2`	`(1,)`

Sabab oddiy: W = W - tezlik * dW amalida shakllar mos kelmasa, translyatsiya ishga tushadi va jimgina noto'g'ri natija beradi (2-bo'lim).

Har backprop funksiyasi oxiriga shu tekshiruvni qo'ying:

Python

assert dW1.shape == W1.shape
assert db1.shape == b1.shape

Bir daqiqa vaqt oladi, soatlab qidiruvdan saqlaydi.

Python

gen = np.random.default_rng(1)
W1 = gen.normal(0, 0.5, (2, 3)); b1 = np.zeros(3)
W2 = gen.normal(0, 0.5, (3, 1)); b2 = np.zeros(1)
X = np.array([[1.0, 0.5], [-0.5, 2.0], [1.5, -1.0]])
y = np.array([[1.0], [0.0], [1.0]])

Z1 = X @ W1 + b1; A1 = np.maximum(0.0, Z1)
Z2 = A1 @ W2 + b2; A2 = sigmoid(Z2)
n = len(y)
dZ2 = (A2 - y) / n
dW2 = A1.T @ dZ2; db2 = dZ2.sum(axis=0)
dZ1 = (dZ2 @ W2.T) * (Z1 > 0)
dW1 = X.T @ dZ1; db1 = dZ1.sum(axis=0)

print("dW1 == W1 shakli:", dW1.shape == W1.shape)
print("db1 == b1 shakli:", db1.shape == b1.shape)
print("dW2 == W2 shakli:", dW2.shape == W2.shape)
print("db2 == b2 shakli:", db2.shape == b2.shape)

Natija

dW1 == W1 shakli: True
db1 == b1 shakli: True
dW2 == W2 shakli: True
db2 == b2 shakli: True

Nima uchun transponirlash #

Python

A1 = np.array([[1.0, 2.0, 3.0],
               [4.0, 5.0, 6.0]])      # (2, 3) - 2 namuna, 3 neyron
dZ2 = np.array([[0.1], [0.2]])        # (2, 1) - 2 namuna, 1 chiqish

print("A1:", A1.shape, " dZ2:", dZ2.shape)
print("A1.T @ dZ2:", (A1.T @ dZ2).shape, "  <- W2 shakli (3, 1) bilan bir xil")

try:
    A1 @ dZ2
except ValueError:
    print("A1 @ dZ2 - shakllar mos kelmaydi")

Natija

A1: (2, 3)  dZ2: (2, 1)
A1.T @ dZ2: (3, 1)   <- W2 shakli (3, 1) bilan bir xil
A1 @ dZ2 - shakllar mos kelmaydi

Transponirlashni eslab qolish qoidasi

Formulani yodlash o'rniga shakllardan chiqaring:

Natija W shakliga teng bo'lishi kerak;
Qatnashuvchi ikkita massivning shakllarini yozing;
Qaysi transponirlash kerakli shaklni beradi - o'sha to'g'ri.

Misol: W2 shakli (3, 1). Bizda A1 (2, 3) va dZ2 (2, 1). Faqat (3, 2) @ (2, 1) = (3, 1) ishlaydi - demak A1.T @ dZ2.

Xuddi shu usul dA1 uchun ham: kerakli shakl (2, 3), bizda dZ2 (2, 1) va W2 (3, 1). Faqat (2, 1) @ (1, 3) = (2, 3) ishlaydi - demak dZ2 @ W2.T.

Bu usul har qanday qatlam uchun ishlaydi va formulani yodlashdan ancha ishonchli.

`*` va `@` - bu yerda farq hal qiluvchi #

Python

dA1 = np.array([[0.5, -0.2, 0.3],
                [0.1, 0.4, -0.6]])
Z1 = np.array([[1.0, -2.0, 3.0],
               [-1.0, 2.0, -3.0]])

relu_hosila = (Z1 > 0).astype(float)
print("relu hosilasi:")
print(relu_hosila)
print("dZ1 = dA1 * hosila:")
print(dA1 * relu_hosila)

Natija

relu hosilasi:
[[1. 0. 1.]
 [0. 1. 0.]]
dZ1 = dA1 * hosila:
[[ 0.5 -0.   0.3]
 [ 0.   0.4 -0. ]]

Faollashtirish har neyronga alohida qo'llanadi, shuning uchun uning hosilasi ham elementma-element ko'paytiriladi.

Amal	Qachon
`@`	Qatlamlar orasida - og'irliklar bilan
`*`	Faollashtirish hosilasi bilan

@ va * ni almashtirish - jimgina xato

dZ1 = dA1 @ relu_hosila yozsangiz:

Shakllar (2,3) @ (2,3) - xato beradi, yaxshi;
Lekin kvadrat matritsalarda ((3,3) @ (3,3)) ishlaydi va butunlay noto'g'ri natija qaytaradi.

Yashirin qatlam o'lchami namunalar soniga teng bo'lib qolsa, bu xato hech qanday xabar bermaydi - faqat tarmoq o'qimaydi.

Bu 2-bo'limdagi ogohlantirishning takrori: shakl tekshiruvi yetarli emas. Gradient tekshiruvi (11-bo'lim) esa buni darhol tutadi.

Bir qadam o'qitish #

Python

gen = np.random.default_rng(1)
X = np.array([[1.0, 0.5], [-0.5, 2.0], [1.5, -1.0]])
y = np.array([[1.0], [0.0], [1.0]])
W1 = gen.normal(0, 0.5, (2, 3)); b1 = np.zeros(3)
W2 = gen.normal(0, 0.5, (3, 1)); b2 = np.zeros(1)
tezlik = 0.5

for qadam in range(4):
    Z1 = X @ W1 + b1; A1 = np.maximum(0.0, Z1)
    Z2 = A1 @ W2 + b2; A2 = sigmoid(Z2)
    L = krossentropiya(y, A2)
    print(f"qadam {qadam}: yo'qotish = {L:.6f}")

    n = len(y)
    dZ2 = (A2 - y) / n
    dW2 = A1.T @ dZ2; db2 = dZ2.sum(axis=0)
    dZ1 = (dZ2 @ W2.T) * (Z1 > 0)
    dW1 = X.T @ dZ1; db1 = dZ1.sum(axis=0)

    W2 -= tezlik * dW2; b2 -= tezlik * db2
    W1 -= tezlik * dW1; b1 -= tezlik * db1

Natija

qadam 0: yo'qotish = 0.737786
qadam 1: yo'qotish = 0.683122
qadam 2: yo'qotish = 0.639546
qadam 3: yo'qotish = 0.593561

Yo'qotish har qadamda kamayyapti - orqaga tarqalish ishlayapti.

Amaliy topshiriq

Bitta neyron uchun z, a va L ni hisoblang.
dL/dz, dL/dw, dL/db ni yozing.
Ularni sonli gradient bilan tekshiring.
dL/dw = dL/dz · x ning intuitiv ma'nosini tushuntiring.
Ikki qatlamli tarmoqda oldinga tarqalishni bajaring.
Yetti gradientni ketma-ket hisoblang va shakllarini chop eting.
Har gradient shakli parametr shakliga tengligini tasdiqlang.
Nima uchun A1.T @ dZ2 ekanini shakllardan chiqaring.
* va @ farqini ReLU hosilasi misolida ko'rsating.
To'rt qadam o'qitib, yo'qotishning kamayishini kuzating.

Xulosa #

Backprop - zanjir qoidasini tartibli qo'llash.
W1 yo'qotishga bevosita emas, zanjir orqali ta'sir qiladi.
Chiqish qatlamida dL/dZ2 = (A2 - y) / n - hammasi qisqaradi.
Og'irlik gradienti - uning kirishiga ko'paytirilgan yuqori gradient.
Gradient shakli doim parametr shakliga teng.
Transponirlashni yodlamang - shakllardan chiqaring.
Qatlamlar orasida @, faollashtirish hosilasi bilan *.
@ va * ni almashtirish kvadrat matritsalarda jimgina o'tadi.
assert dW.shape == W.shape - bir daqiqalik ish, soatlab tejaydi.
Yo'qotish kamaysa - backprop to'g'ri ishlayapti.

Keyingi bo'limda gradientlarni qo'lda, raqamlar bilan hisoblab, formulalarni oxirigacha tushunamiz.

Muammo: W1 xatoga qanday ta'sir qiladi #

Eng oddiy holat: bitta neyron #

Ikki qatlam: qadam-baqadam #

Nima uchun transponirlash #

* va @ - bu yerda farq hal qiluvchi #

Bir qadam o'qitish #

Xulosa #

Xatolik topdingizmi?

Muammo: `W1` xatoga qanday ta'sir qiladi #

`*` va `@` - bu yerda farq hal qiluvchi #