3.(basic) RNN Implementation

RNN 구현

import tensorflow as tf

1. Hyperparameters

• EPOCH (에폭 수) : 10
• NUM_WORDS (추후에 불러올 imdb 데이터셋에서 사용할 vocabulary 수) : 10000

EPOCHS = 10
NUM_WORDS = 10000

2. Define Models

• Embedding : 우선 10000종류의 어휘 (총 10000차원)을, 16개의 차원으로 Embedding 시켜준다.

  ( 함수 : tf.keras.layers.Embedding(10000,16) )

• LSTM : LSTM의 neuron 개수를 지정해준다

  ( 함수 : tf.keras.layers.LSTM(32) )

• Dense : Fuclly Connected Layer. 마지막 dense layer에 1개의 Softmax 함수를 사용할 것이다

  ( 함수 : tf.keras.layers.Dense(1, activation='softmax') )

NUM_WORDS=10000

class MyModel(tf.keras.Model):
    def __init__(self):
        super(MyModel, self).__init__()
        self.embed = tf.keras.layers.Embedding(NUM_WORDS, 16)
        self.rnn = tf.keras.layers.LSTM(32)
        self.dense = tf.keras.layers.Dense(1, activation='softmax')
        
    def call(self, x, training=None, mask=None):
        x = self.embed(x)
        x = self.rnn(x)
        X = self.dense(x)
        return x

Training Loop & Testing Loop

tf.function을 함수에 붙여줄 경우, 여전히 다른 일반 함수들처럼 사용할 수 있습니다. 하지만 그래프 내에서 컴파일 되었을 때는 더 빠르게 실행하고, GPU나 TPU를 사용해서 작동하고, 세이브드모델(SavedModel)로 내보내는 것이 가능해집니다. ( 출처 : tensorflow.org )

# Implement training loop
@tf.function
def train_step(model, inputs, labels, loss_fn, opt, train_loss, train_acc):
    with tf.GradientTape() as tape:
        pred = model(inputs, training=True)
        loss = loss_fn(labels, pred)
    gradients = tape.gradient(loss, model.trainable_variables)
    opt.apply_gradients(zip(gradients, model.trainable_variables))
    train_loss(loss)
    train_acc(labels, pred)

# Implement testing loop
@tf.function
def test_step(model, images, labels, loss_fn, test_loss, test_acc):
    pred = model(images, training=False)
    loss = loss_fn(labels, pred)
    test_loss(loss)
    test_acc(labels, pred)

3. Import Dataset

데이터로는 imdb 데이터셋을 활용할 것이다.

( 영화, TV 등의 프로에 관련된 평가를 담은 text 형태의 데이터셋이다. )

• padding : 리뷰(평가)마다 문장 길이가 다르다.

  이 길이를 맞춰주기 위해 zero padding을 해준다. ( 최대 길이는 32까지로 )

  ( 함수 : tf.keras.preprocessing.sequence.pad_sequences())

• shuffle을 해주고, batch size는 32로 해준다.

imdb = tf.keras.datasets.imdb
(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=NUM_WORDS)

x_train = tf.keras.preprocessing.sequence.pad_sequences(x_train,value=0,
                                                       padding='pre',maxlen=32)
x_test = tf.keras.preprocessing.sequence.pad_sequences(x_test,value=0,
                                                      padding='pre',maxlen=32)

train_ds = tf.data.Dataset.from_tensor_slices((x_train, y_train)).shuffle(10000).batch(32)
test_ds = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(32)

4. Create Model

• step 1) 모델을 생성한다

• step 2) loss function & optimizer 정의

  • loss function : Sparse Categorical Crossentropy

    ( tf.keras.losses.SparseCategoricalCrossentropy() )

  • optimizer : Adam

    (tf.keras.optimizers.Adam())

• 3) 평가 metric 지정 ( accuracy )

  ( tf.keras.metrics.SparseCategoricalAccuracy() )

# step 1) Create model
model = MyModel()

# step2) Define loss and optimizer
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy()
opt = tf.keras.optimizers.Adam()

# 3) Define performance metrics
train_loss = tf.keras.metrics.Mean(name='train_loss')
train_acc = tf.keras.metrics.SparseCategoricalAccuracy(name='train_acc')

test_loss = tf.keras.metrics.Mean(name='test_loss')
test_acc = tf.keras.metrics.SparseCategoricalAccuracy(name='test_acc')

5. Train

• 전처리한 데이터셋과, 만든 모델을 통해 학습을 시킨다

for epoch in range(EPOCHS):
    for seqs, labels in train_ds:
        train_step(model, seqs, labels, loss_fn, opt, train_loss, train_acc)

    for test_seqs, test_labels in test_ds:
        test_step(model, test_seqs, test_labels, loss_fn, test_loss, test_acc)

    template = 'Epoch {}, Loss: {}, acc: {}, Test Loss: {}, Test acc: {}'
    print(template.format(epoch + 1,
                          train_loss.result(),
                          train_acc.result() * 100,
                          test_loss.result(),
                          test_acc.result() * 100))
    train_loss.reset_states()
    train_acc.reset_states()
    test_loss.reset_states()
    test_acc.reset_states()

Epoch 1, Loss: 0.500654935836792, acc: 74.24800109863281, Test Loss: 0.4538622498512268, Test acc: 78.59200286865234
Epoch 2, Loss: 0.36486417055130005, acc: 83.76799774169922, Test Loss: 0.4595658779144287, Test acc: 78.03199768066406
Epoch 3, Loss: 0.2961457669734955, acc: 87.59600067138672, Test Loss: 0.48797470331192017, Test acc: 77.38400268554688
Epoch 4, Loss: 0.23388780653476715, acc: 90.89199829101562, Test Loss: 0.5365427136421204, Test acc: 76.56000518798828
Epoch 5, Loss: 0.18335703015327454, acc: 93.19200134277344, Test Loss: 0.6383700370788574, Test acc: 75.85199737548828
Epoch 6, Loss: 0.1514805555343628, acc: 94.48799896240234, Test Loss: 0.7883667349815369, Test acc: 75.40399932861328
Epoch 7, Loss: 0.12500645220279694, acc: 95.42000579833984, Test Loss: 0.9673071503639221, Test acc: 74.78799438476562
Epoch 8, Loss: 0.10475290566682816, acc: 96.16000366210938, Test Loss: 1.0729326009750366, Test acc: 74.80799865722656
Epoch 9, Loss: 0.08867021650075912, acc: 96.89599609375, Test Loss: 1.0573757886886597, Test acc: 73.19599914550781
Epoch 10, Loss: 0.06874625384807587, acc: 97.6240005493164, Test Loss: 1.3542929887771606, Test acc: 71.81999969482422