Optimizatorlar - momentum va Adam - Neyron tarmoqlar - noldan

w = w - tezlik · gradient - eng oddiy qoida. U ishlaydi, lekin sekin. Uch xil yaxshilanish uni sezilarli tezlashtiradi.

Muammo: tor vodiy #

Python

import numpy as np


def vodiy(v):
    """Bir yo'nalishda tik, boshqasida tekis funksiya."""
    x, y = v
    return 50.0 * x ** 2 + 0.05 * y ** 2


def vodiy_gradient(v):
    x, y = v
    return np.array([100.0 * x, 0.1 * y])


def yol(yangilash_fn, boshlanish, qadamlar=120):
    v = np.array(boshlanish, dtype=float)
    holat = {}
    tarix = [v.copy()]
    for _ in range(qadamlar):
        v = yangilash_fn(v, vodiy_gradient(v), holat)
        tarix.append(v.copy())
    return np.array(tarix)


def sgd(tezlik):
    def yangilash(v, g, holat):
        return v - tezlik * g
    return yangilash


def momentum(tezlik, beta=0.9):
    def yangilash(v, g, holat):
        holat["m"] = beta * holat.get("m", np.zeros_like(v)) + g
        return v - tezlik * holat["m"]
    return yangilash


def rmsprop(tezlik, beta=0.9, eps=1e-8):
    def yangilash(v, g, holat):
        holat["v"] = beta * holat.get("v", np.zeros_like(v)) + (1 - beta) * g ** 2
        return v - tezlik * g / (np.sqrt(holat["v"]) + eps)
    return yangilash


def adam(tezlik, b1=0.9, b2=0.999, eps=1e-8):
    def yangilash(v, g, holat):
        holat["t"] = holat.get("t", 0) + 1
        holat["m"] = b1 * holat.get("m", np.zeros_like(v)) + (1 - b1) * g
        holat["v"] = b2 * holat.get("v", np.zeros_like(v)) + (1 - b2) * g ** 2
        m_tuzatilgan = holat["m"] / (1 - b1 ** holat["t"])
        v_tuzatilgan = holat["v"] / (1 - b2 ** holat["t"])
        return v - tezlik * m_tuzatilgan / (np.sqrt(v_tuzatilgan) + eps)
    return yangilash

Python

print("f(1, 1)   =", vodiy(np.array([1.0, 1.0])))
print("gradient  =", vodiy_gradient(np.array([1.0, 1.0])))
print("x yo'nalishi 1000 barobar tikroq")

Natija

f(1, 1)   = 50.05
gradient  = [100.    0.1]
x yo'nalishi 1000 barobar tikroq

Bu shartli sonli yomon (ill-conditioned) funksiya: bir o'q bo'ylab juda tik, boshqasi bo'ylab deyarli tekis.

Zigzag - bu isrof; momentum uni to'g'rilaydi

Oddiy gradient tushish #

Python

def sgd(tezlik):
    def yangilash(v, g, holat):
        return v - tezlik * g
    return yangilash


tarix = yol(sgd(0.009), [1.0, 1.0])
print("boshlanish:", np.round(tarix[0], 4))
print("20 qadam:  ", np.round(tarix[20], 4))
print("120 qadam: ", np.round(tarix[120], 6))
print("f(oxirgi) =", round(vodiy(tarix[120]), 8))

Natija

boshlanish: [1. 1.]
20 qadam:   [0.     0.9822]
120 qadam:  [0.       0.897584]
f(oxirgi) = 0.04028285

x deyarli darhol nolga tushdi, y esa 120 qadamdan keyin ham 0.9 atrofida qoldi - deyarli qimirlamadi.

Python

for tezlik in [0.005, 0.009, 0.02, 0.021]:
    tarix = yol(sgd(tezlik), [1.0, 1.0])
    oxirgi = tarix[-1]
    holat = "portladi" if not np.isfinite(vodiy(oxirgi)) or abs(oxirgi[0]) > 10 else "barqaror"
    print(f"tezlik {tezlik:<6} oxirgi: [{oxirgi[0]:>12.4f}, {oxirgi[1]:>8.4f}]  {holat}")

Natija

tezlik 0.005  oxirgi: [      0.0000,   0.9418]  barqaror
tezlik 0.009  oxirgi: [      0.0000,   0.8976]  barqaror
tezlik 0.02   oxirgi: [      1.0000,   0.7864]  barqaror
tezlik 0.021  oxirgi: [  92709.0688,   0.7770]  portladi

Tezlikni oshirib bo'lmaydi - x portlaydi

y yo'nalishida tezlikni oshirish kerak, lekin x yo'nalishida chegara 2/100 = 0.02 da yotibdi.

Ya'ni bitta tezlik ikkala yo'nalishga ham to'g'ri kelmaydi:

Tezlik	`x` yo'nalishi	`y` yo'nalishi
Kichik	Barqaror	Juda sekin
Katta	Portlaydi	Yaxshi

Bu neyron tarmoqlarda doimiy holat: turli parametrlar juda turli masshtabdagi gradientlarga ega.

Uch yechim bor va ularning barchasi shu bo'limda:

G'oya	Nima qiladi
Momentum	Yo'nalishni "eslab qoladi", tebranishni so'ndiradi
RMSProp	Har parametrga o'z tezligini beradi
Adam	Ikkalasini birlashtiradi

Momentum #

Natija

tezlanish = beta · tezlanish + gradient
w = w - tezlik · tezlanish

Python

def momentum(tezlik, beta=0.9):
    def yangilash(v, g, holat):
        holat["m"] = beta * holat.get("m", np.zeros_like(v)) + g
        return v - tezlik * holat["m"]
    return yangilash


sgd_tarix = yol(sgd(0.009), [1.0, 1.0])
mom_tarix = yol(momentum(0.002, 0.9), [1.0, 1.0])

print("120 qadamdan keyin:")
print("  SGD:      ", np.round(sgd_tarix[-1], 6), " f =", f"{vodiy(sgd_tarix[-1]):.2e}")
print("  momentum: ", np.round(mom_tarix[-1], 6), " f =", f"{vodiy(mom_tarix[-1]):.2e}")

Natija

120 qadamdan keyin:
  SGD:       [0.       0.897584]  f = 4.03e-02
  momentum:  [6.38000e-04 7.97872e-01]  f = 3.19e-02

Momentum tebranishni qanday so'ndiradi

Momentum oldingi gradientlarni jamlaydi:

Yo'nalish	Gradientlar	Yig'indi
`x` (tik, zigzag)	`+`, `-`, `+`, `-`	Bir-birini yo'q qiladi
`y` (tekis, bir xil)	`+`, `+`, `+`, `+`	To'planadi va kuchayadi

Ya'ni momentum foydali yo'nalishni kuchaytiradi, zararli tebranishni esa so'ndiradi.

beta = 0.9 - amaliy standart. U taxminan oxirgi 10 ta gradientning o'rtachasini beradi (1/(1-0.9) = 10).

Fizik o'xshatish: gradient - kuch, momentum esa sharikning tezligi. Sharik vodiy devorlaridan sakramaydi, u tubi bo'ylab dumalab boradi.

Python

gradientlar_x = np.array([1.0, -1.0, 1.0, -1.0, 1.0, -1.0])
gradientlar_y = np.array([1.0, 1.0, 1.0, 1.0, 1.0, 1.0])

for nom, g in [("tebranuvchi (x)", gradientlar_x), ("bir xil (y)", gradientlar_y)]:
    m = 0.0
    for qiymat in g:
        m = 0.9 * m + qiymat
    print(f"{nom:<18} 6 qadamdan keyin momentum = {m:>7.4f}")

Natija

tebranuvchi (x)    6 qadamdan keyin momentum = -0.2466
bir xil (y)        6 qadamdan keyin momentum =  4.6856

Bir xil yo'nalishdagi gradient momentumni 15 barobar kattaroq qildi.

RMSProp - har parametrga o'z tezligi #

Natija

v = beta · v + (1-beta) · gradient²
w = w - tezlik · gradient / (sqrt(v) + eps)

Python

def rmsprop(tezlik, beta=0.9, eps=1e-8):
    def yangilash(v, g, holat):
        holat["v"] = beta * holat.get("v", np.zeros_like(v)) + (1 - beta) * g ** 2
        return v - tezlik * g / (np.sqrt(holat["v"]) + eps)
    return yangilash


rms_tarix = yol(rmsprop(0.02), [1.0, 1.0])
print("RMSProp 120 qadamdan keyin:", np.round(rms_tarix[-1], 6))
print("f =", f"{vodiy(rms_tarix[-1]):.2e}")

Natija

RMSProp 120 qadamdan keyin: [0. 0.]
f = 6.11e-37

RMSProp ikkala yo'nalishni ham mashina aniqligigacha nolga olib keldi - bu masalada u eng yaxshi natija berdi.

gradient / sqrt(gradient²) - masshtabni yo'q qiladi

RMSProp ning g'oyasi: gradientni uning o'z kattaligiga bo'lish.

Parametr	Gradient	`sqrt(v)`	Qadam
Tik yo'nalish	`20`	`≈ 20`	`≈ 1`
Tekis yo'nalish	`1`	`≈ 1`	`≈ 1`

Ya'ni har parametr bir xil kattalikdagi qadam tashlaydi, gradient qanchalik katta bo'lishidan qat'i nazar.

Bu turli masshtabdagi belgilar muammosini ham qisman hal qiladi (17-bo'lim) - lekin normalizatsiya o'rnini bosmaydi.

eps = 1e-8 - nolga bo'linishdan himoya. Uni tushirib qoldirish - nan olishning ishonchli yo'li.

Adam - ikkalasi birga #

Natija

m = b1·m + (1-b1)·g          (momentum)
v = b2·v + (1-b2)·g²         (RMSProp)
m̂ = m / (1 - b1^t)           (siljishni tuzatish)
v̂ = v / (1 - b2^t)
w = w - tezlik · m̂ / (sqrt(v̂) + eps)

Python

def adam(tezlik, b1=0.9, b2=0.999, eps=1e-8):
    def yangilash(v, g, holat):
        holat["t"] = holat.get("t", 0) + 1
        holat["m"] = b1 * holat.get("m", np.zeros_like(v)) + (1 - b1) * g
        holat["v"] = b2 * holat.get("v", np.zeros_like(v)) + (1 - b2) * g ** 2
        m_tuzatilgan = holat["m"] / (1 - b1 ** holat["t"])
        v_tuzatilgan = holat["v"] / (1 - b2 ** holat["t"])
        return v - tezlik * m_tuzatilgan / (np.sqrt(v_tuzatilgan) + eps)
    return yangilash


for nom, fn in [("SGD", sgd(0.009)), ("momentum", momentum(0.002, 0.9)),
                ("RMSProp", rmsprop(0.02)), ("Adam", adam(0.1))]:
    tarix = yol(fn, [1.0, 1.0])
    print(f"{nom:<10} 120 qadam -> f = {vodiy(tarix[-1]):.3e}   "
          f"nuqta = [{tarix[-1][0]:>9.6f}, {tarix[-1][1]:>9.6f}]")

Natija

SGD        120 qadam -> f = 4.028e-02   nuqta = [ 0.000000,  0.897584]
momentum   120 qadam -> f = 3.185e-02   nuqta = [ 0.000638,  0.797872]
RMSProp    120 qadam -> f = 6.114e-37   nuqta = [ 0.000000,  0.000000]
Adam       120 qadam -> f = 1.238e-04   nuqta = [-0.001573, -0.001573]

Siljishni tuzatish nima uchun kerak

Adam da m va v noldan boshlanadi. Birinchi qadamda:

Natija

m = 0.9 · 0 + 0.1 · g = 0.1 · g

Ya'ni haqiqiy gradientning atigi o'ndan bir qismi. Tuzatishsiz Adam birinchi o'nlab qadamlarda juda kichik qadamlar tashlagan bo'lardi.

m / (1 - 0.9^t) bo'linishi buni to'g'rilaydi:

`t`	`1 - 0.9^t`	Tuzatish
1	`0.1`	`× 10`
10	`0.65`	`× 1.5`
100	`≈ 1.0`	Deyarli yo'q

v uchun b2 = 0.999 bo'lgani sababli tuzatish ancha uzoq davom etadi - 1 - 0.999^t yuzlab qadamdan keyingina birga yaqinlashadi.

Python

for t in [1, 10, 100, 1000]:
    m_tuz = 1 - 0.9 ** t
    v_tuz = 1 - 0.999 ** t
    print(f"t = {t:>4}:  m koeffitsienti = {1/m_tuz:>8.3f}   "
          f"v koeffitsienti = {1/v_tuz:>10.3f}")

Natija

t =    1:  m koeffitsienti =   10.000   v koeffitsienti =   1000.000
t =   10:  m koeffitsienti =    1.535   v koeffitsienti =    100.451
t =  100:  m koeffitsienti =    1.000   v koeffitsienti =     10.503
t = 1000:  m koeffitsienti =    1.000   v koeffitsienti =      1.582

Haqiqiy tarmoqda #

Python

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))


class Tarmoq:
    def __init__(self, olchamlar, urug=42):
        gen = np.random.default_rng(urug)
        self.q = len(olchamlar) - 1
        self.W = [gen.normal(0, np.sqrt(2.0 / olchamlar[i]),
                             (olchamlar[i], olchamlar[i + 1])) for i in range(self.q)]
        self.b = [np.zeros(olchamlar[i + 1]) for i in range(self.q)]
        self.holat = [{} for _ in range(2 * self.q)]

    def oldinga(self, X):
        A = X
        kesh = {"A0": X}
        for i in range(self.q):
            Z = A @ self.W[i] + self.b[i]
            A = sigmoid(Z) if i == self.q - 1 else np.maximum(0.0, Z)
            kesh[f"Z{i+1}"] = Z
            kesh[f"A{i+1}"] = A
        return A, kesh

    def qadam(self, X, y, yangilash_fn):
        A, kesh = self.oldinga(X)
        dZ = (A - y) / len(y)
        for i in range(self.q - 1, -1, -1):
            dW = kesh[f"A{i}"].T @ dZ
            db = dZ.sum(axis=0)
            if i > 0:
                dZ = (dZ @ self.W[i].T) * (kesh[f"Z{i}"] > 0)
            self.W[i] = yangilash_fn(self.W[i], dW, self.holat[2 * i])
            self.b[i] = yangilash_fn(self.b[i], db, self.holat[2 * i + 1])

    def yoqotish(self, X, y, eps=1e-12):
        A, _ = self.oldinga(X)
        A = np.clip(A, eps, 1 - eps)
        return float(-np.mean(y * np.log(A) + (1 - y) * np.log(1 - A)))


gen = np.random.default_rng(0)
X = gen.normal(0, 1, (600, 10))
y = ((X[:, 0] * X[:, 1] + X[:, 2]) > 0).astype(float).reshape(-1, 1)

for nom, fn in [("SGD 0.05", sgd(0.05)), ("SGD 0.2", sgd(0.2)),
                ("momentum", momentum(0.05, 0.9)), ("Adam", adam(0.01))]:
    t = Tarmoq([10, 24, 24, 1], urug=3)
    for _ in range(400):
        t.qadam(X, y, fn)
    print(f"{nom:<12} 400 davrdan keyin yo'qotish = {t.yoqotish(X, y):.4f}")

Natija

SGD 0.05     400 davrdan keyin yo'qotish = 0.1849
SGD 0.2      400 davrdan keyin yo'qotish = 0.0258
momentum     400 davrdan keyin yo'qotish = 0.0055
Adam         400 davrdan keyin yo'qotish = 0.0004

Tartib aniq: Adam (0.0004) → momentum (0.0055) → SGD 0.2 (0.026) → SGD 0.05 (0.18).

Diqqat qiling: SGD ni 0.05 dan 0.2 ga oshirish uni yetti barobar yaxshiladi - ya'ni oddiy SGD da tezlikni tanlash juda muhim. Adam esa 0.01 bilan darhol eng yaxshi natijani berdi.

Amalda qaysi birini tanlash

Optimizator	Boshlang'ich tezlik	Qachon
Adam	`0.001`	Standart tanlov - deyarli har doim ishlaydi
SGD + momentum	`0.01` - `0.1`	Rasm tasniflashda ba'zan yaxshiroq umumlashadi
AdamW	`0.001`	Adam + to'g'ri og'irlik pasayishi (16-bo'lim)
RMSProp	`0.001`	Rekurrent tarmoqlarda

Amaliy tavsiya: Adam bilan boshlang. U giperparametrlarga kam sezgir va deyarli har doim ma'qul natija beradi.

Model ishlaganidan keyin, agar oxirgi foizlar muhim bo'lsa, SGD + momentum + tezlik jadvali bilan sinab ko'ring - ba'zan u biroz yaxshiroq umumlashtiradi.

Diqqat: yuqoridagi misolda Adam 0.01 tezlik bilan ishlatildi, chunki tarmoq kichik. Katta tarmoqlarda standart 0.001 odatda to'g'riroq.

Gradientni kesish #

Python

def kesilgan_adam(tezlik, chegara=1.0, b1=0.9, b2=0.999, eps=1e-8):
    ichki = adam(tezlik, b1, b2, eps)

    def yangilash(v, g, holat):
        norma = float(np.linalg.norm(g))
        if norma > chegara:
            g = g * (chegara / norma)
        return ichki(v, g, holat)
    return yangilash


katta_gradient = np.array([100.0, 200.0])
print("asl norma:", round(float(np.linalg.norm(katta_gradient)), 4))
kesilgan = katta_gradient * (1.0 / float(np.linalg.norm(katta_gradient)))
print("kesilgandan keyin norma:", round(float(np.linalg.norm(kesilgan)), 4))
print("yo'nalish saqlandimi:",
      bool(np.allclose(kesilgan / np.linalg.norm(kesilgan),
                       katta_gradient / np.linalg.norm(katta_gradient))))

Natija

asl norma: 223.6068
kesilgandan keyin norma: 1.0
yo'nalish saqlandimi: True

Gradient portlashi - kesish bilan hal qilinadi

Ba'zan bitta g'alati namuna juda katta gradient beradi va bitta qadam og'irliklarni buzadi.

Gradientni kesish (clipping) uning yo'nalishini saqlab, faqat uzunligini cheklaydi.

Usul	Formula
Norma bo'yicha	`g · min(1, chegara/‖g‖)` - afzal
Qiymat bo'yicha	`np.clip(g, -c, c)` - yo'nalishni buzadi

Birinchisi afzal, chunki u gradientning yo'nalishini o'zgartirmaydi.

Bu ayniqsa rekurrent tarmoqlarda va til modellarida majburiy. Oddiy tarmoqlarda esa kerak bo'lmaydi - lekin yo'qotish to'satdan nan bo'lsa, birinchi sinab ko'radigan chora shu.

Amaliy topshiriq

Tor vodiy funksiyasini yozing va gradientini hisoblang.
SGD bilan 60 qadam bajarib, y ning sekinligini ko'ring.
Tezlikni oshirib, x yo'nalishidagi chegarani toping.
Momentum ni yozing va SGD bilan solishtiring.
Tebranuvchi va bir xil gradientlarda momentum farqini hisoblang.
RMSProp ni yozing va natijani solishtiring.
Adam ni siljish tuzatishi bilan yozing.
Tuzatish koeffitsientini t = 1, 10, 100 uchun hisoblang.
To'rt optimizatorni haqiqiy tarmoqda solishtiring.
Gradient kesishni yozing va yo'nalish saqlanishini tekshiring.

Xulosa #

Tor vodiyda bitta tezlik barcha yo'nalishlarga to'g'ri kelmaydi.
Momentum oldingi gradientlarni jamlaydi: zigzag so'nadi, foydali yo'nalish kuchayadi.
beta = 0.9 ≈ oxirgi 10 ta gradientning o'rtachasi.
RMSProp gradientni o'z kattaligiga bo'ladi - har parametrga o'z tezligi.
Adam = momentum + RMSProp + siljish tuzatishi.
Siljish tuzatishi birinchi qadamlardagi kichik qadamlarni to'g'rilaydi.
Amalda Adam standart tanlov, 0.001 tezlik bilan.
SGD + momentum ba'zan yaxshiroq umumlashtiradi.
eps ni tushirib qoldirmang - nan olasiz.
Gradient portlasa - norma bo'yicha kesing, yo'nalish saqlanadi.

Keyingi bo'limda haddan ortiq moslashuvni va unga qarshi choralarni ko'ramiz.

Muammo: tor vodiy #

Oddiy gradient tushish #

Momentum #

RMSProp - har parametrga o'z tezligi #

Adam - ikkalasi birga #

Haqiqiy tarmoqda #

Gradientni kesish #

Xulosa #

Xatolik topdingizmi?