2.(basic) Neural Net & Back Propagation 구현(2)

Neural Net & Back Propagation 구현(2)

이전 포스트에서는 numpy만을 사용하여 back propagation과 neural network를 구현하였다. 이번에는 tensorflow2를 사용하여 이전에 한 것과 똑같은 것을 보다 쉽게 구현하였다.

1. Import libraries

import tensorflow as tf
from tensorflow.keras import layers
import numpy as np

2. Set hyperparameters

EPOCHS = 1000

3. Network Architecture

• Input layer : 2
• Hidden layer : 128 (Sigmoid)
• Output : 1 (Softmax)

class MyModel(tf.keras.Model):
    def __init__(self):
        super(MyModel, self).__init__()
        self.d1 = layers.Dense(128, input_dim=2, activation='sigmoid')
        self.d2 = layers.Dense(10, activation='softmax')
            
    def call(self, x, training=None, mask=None):
        x = self.d1(x)
        x = self.d2(x)
        return x

4. Train function

• 함수의 인자값 : model, input & label, loss_object, optimizer, train loss & train metric

@tf.function
def train_step(model, inputs, labels, loss_object, optimizer, train_loss, train_metric):
    with tf.GradientTape() as tape:
        pred = model(inputs) # 예측값
        loss = loss_object(labels, pred) # 예측 손실
    grads = tape.gradient(loss, model.trainable_variables) # dy/dx
    
    optimizer.apply_gradients(zip(grads, model.trainable_variables))
    train_loss(loss)
    train_metric(labels, predictions)

5. Import Data

np.random.seed(0)

pts = list()
labels = list()
center_pts = np.random.uniform(-8.0, 8.0, (10, 2))
for label, center_pt in enumerate(center_pts):
    for _ in range(100):
        pts.append(center_pt + np.random.randn(*center_pt.shape))
        labels.append(label)

pts = np.stack(pts, axis=0).astype(np.float32)
labels = np.stack(labels, axis=0)

train_ds = tf.data.Dataset.from_tensor_slices((pts, labels)).shuffle(1000).batch(32)

6. Modeling

(1) Model

model = MyModel()

(2) Loss Object & Optimizer

• Loss Object : spare categorical cross entropy, binary cross entropy..
• Optimizer : Adam, RMSprop, …

loss_object = tf.keras.losses.SparseCategoricalCrossentropy()
optimizer = tf.keras.optimizers.Adam()

(3) Metric ( loss & accuracy )

train_loss = tf.keras.metrics.Mean(name='train_loss')
train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='train_accuracy')

7. Train

for epoch in range(EPOCHS):
    for x, label in train_ds:
        train_step(model, x, label, loss_object, optimizer, train_loss, train_accuracy)
        
    template = 'Epoch {}, Loss: {}, Accuracy: {}'
    print(template.format(epoch + 1,
                          train_loss.result(),
                          train_accuracy.result() * 100))
    train_loss.reset_states()
    train_accuracy.reset_states()

Epoch 1, Loss: 0.5004048347473145, Accuracy: 85.82125091552734
Epoch 2, Loss: 0.4983041286468506, Accuracy: 85.854736328125
Epoch 3, Loss: 0.49625158309936523, Accuracy: 85.8885498046875
Epoch 4, Loss: 0.49433252215385437, Accuracy: 85.91976165771484
Epoch 5, Loss: 0.49232837557792664, Accuracy: 85.9474868774414
Epoch 6, Loss: 0.49037522077560425, Accuracy: 85.9784927368164
Epoch 7, Loss: 0.4885392487049103, Accuracy: 86.0060806274414
Epoch 8, Loss: 0.48661574721336365, Accuracy: 86.03408813476562
Epoch 9, Loss: 0.4847148656845093, Accuracy: 86.0625
...
Epoch 994, Loss: 0.27788347005844116, Accuracy: 89.2856216430664
Epoch 995, Loss: 0.2778545022010803, Accuracy: 89.28681945800781
Epoch 996, Loss: 0.2778218984603882, Accuracy: 89.287109375
Epoch 997, Loss: 0.277792751789093, Accuracy: 89.28839874267578
Epoch 998, Loss: 0.277757853269577, Accuracy: 89.28913879394531
Epoch 999, Loss: 0.27772510051727295, Accuracy: 89.29024505615234
Epoch 1000, Loss: 0.2776941955089569, Accuracy: 89.29043579101562

8. Save / Load parameters

• 학습한 모델의 weight를 따로 저장했다가 나중에 (다시 학습할 필요 없게) weight만 따로 불러들일 수 있다.

np.savez_compressed('ch2_dataset.npz', inputs=pts, labels=labels)

W_h, b_h = model.d1.get_weights()
W_o, b_o = model.d2.get_weights()
W_h = np.transpose(W_h)
W_o = np.transpose(W_o)
np.savez_compressed('ch2_parameters.npz',
                    W_h=W_h,
                    b_h=b_h,
                    W_o=W_o,
                    b_o=b_o)