27.TensorFlow Serving REST API - 鐵人賽示範

• 本篇為鐵人賽示範，參考官方範例修改而成。
• 本篇適用在 Colab 環境。

下載資料及訓練模型

• 由於重點在如何起一個TF Severing 服務，資料採用keras.datasets.cifar10進行示範，CIFAR10為小型的影像分類資料集，具有50,000筆訓練資料集與10,000筆測試資料集，皆為32X32像素圖片。更多資訊參閱官方介紹。

Label	Description
0	airplane
1	automobile
2	bird
3	cat
4	deer
5	dog
6	frog
7	horse
8	ship
9	truck

# TensorFlow and tf.keras
# !pip install -Uq grpcio==1.26.0

import tensorflow as tf
from tensorflow import keras
import numpy as np
import matplotlib.pyplot as plt
import os
import subprocess

• 建立模型

fashion_mnist = keras.datasets.cifar10
(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()

# scale the values to 0.0 to 1.0
train_images = train_images / 255.0
test_images = test_images / 255.0

# reshape for feeding into the model
train_images = train_images.reshape(train_images.shape[0], 32, 32, 3)
test_images = test_images.reshape(test_images.shape[0], 32, 32, 3)

class_names = ['airplane','automobile','bird','cat','deer','dog','frog','horse','ship','truck']


print(f'train_images.shape: {train_images.shape}, of {train_images.dtype}')
print(f'test_images.shape: {test_images.shape}, of {test_images.dtype}')

• 訓練與評估模型

model = keras.Sequential([
  keras.layers.Conv2D(
      input_shape=(32,32,3), 
      filters=8, 
      kernel_size=3, 
      strides=2, 
      activation='relu', 
      name='Conv1'),
  keras.layers.Flatten(),
  keras.layers.Dense(10, name='Dense')
])
model.summary()

testing = False
epochs = 5

model.compile(
    optimizer='adam', 
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=[keras.metrics.SparseCategoricalAccuracy()]
    )
model.fit(train_images, train_labels, epochs=epochs)

test_loss, test_acc = model.evaluate(test_images, test_labels)
print(f'Test accuracy: {test_acc}')

儲存模型

• 將模型保存為SavedModel格式，以便將模型加載到 TensorFlow Serving 中。
• TensorFlow Serving允許我們在發出推理請求時選擇要使用的模型版本或“可服務”版本。每個版本將導出到給定路徑下的不同子目錄。為此，需在目錄創建 protobuf 文件，並將包含一個版本號。
• 以下會在/tmp/建立版次版次version = 1之相關檔案。

import tempfile

MODEL_DIR = tempfile.gettempdir()
version = 1
export_path = os.path.join(MODEL_DIR, str(version))
print(f'export_path = {export_path}')

tf.keras.models.save_model(
    model,
    export_path,
    overwrite=True,
    include_optimizer=True,
    save_format=None,
    signatures=None,
    options=None
)

print('\nSaved model:')
!ls -l {export_path}

檢查我們的Saved model

• saved_model_cli 可以檢查前述SavedModel中相關資訊，這對理解模型相當有用，包含:
  • MetaGraphDefs（模型）
  • SignatureDefs（您可以調用的方法）
• SavedModel CLI的詳細說明可參閱TensorFlow 指南。

!saved_model_cli show --dir {export_path} --all

建立 TensorFlow Serving 服務

• 依官方範例此為Colab環境所需設定內容，如使用本機端的 Notebook ，請注意相關提醒。

Add TensorFlow Serving distribution URI as a package source

• 我們準備使用Aptitude安裝 TensorFlow Serving，因為此 Colab 在 Debian 環境中運行。我們將把這個tensorflow-model-server包添加到 Aptitude 知道的包列表中。請注意，我們以 root 身份運行。
• 最簡單的方式是以 Docker 佈署，您可以參考此範例。

import sys
# We need sudo prefix if not on a Google Colab.
if 'google.colab' not in sys.modules:
  SUDO_IF_NEEDED = 'sudo'
else:
  SUDO_IF_NEEDED = ''

# This is the same as you would do from your command line, but without the [arch=amd64], and no sudo
# You would instead do:
# echo "deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | sudo tee /etc/apt/sources.list.d/tensorflow-serving.list && \
# curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | sudo apt-key add -

!echo "deb http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | {SUDO_IF_NEEDED} tee /etc/apt/sources.list.d/tensorflow-serving.list && \
curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | {SUDO_IF_NEEDED} apt-key add -
!{SUDO_IF_NEEDED} apt update

安裝 TensorFlow Serving

!{SUDO_IF_NEEDED} apt-get install tensorflow-model-server

啟動 TensorFlow Serving

加載後，我們可以開始使用 REST 發出推理請求，相關參數:

• rest_api_port： REST 請求的 Port。
• model_name：您將在 REST 請求的 URL 中使用它。
• model_base_path：保存模型的目錄的路徑。

os.environ["MODEL_DIR"] = MODEL_DIR

nohup tensorflow_model_server \
  --rest_api_port=8501 \
  --model_name=fashion_model \
  --model_base_path="${MODEL_DIR}" >server.log 2>&1

!tail server.log

以 Request 向 TensorFlow Serving 提出請求提出請求

• 先以亂數查看 test data。

def show(idx, title):
  plt.figure()
  plt.imshow(test_images[idx].reshape(32,32,3))
  plt.axis('off')
  plt.title(f'{title}', fontdict={'size': 16})

import random
rando = random.randint(0,len(test_images)-1)
test_label_name = class_names[int(test_labels[rando])]
show(rando, f'An Example Image: {test_label_name}')

• 測試請求一批JSON。

import json
from pprint import pprint

data = json.dumps(
    {"signature_name": "serving_default", 
     "instances": test_images[0:3].tolist()}
     )
pprint(f'Data: {data[:50]} ... {data[len(data)-52:]}')

發出 REST 請求

最新版本的 servable

• 以 POST 至 server，預設請求伺服器提供最新版次的內容。

# docs_infra: no_execute
!pip install -q requests

import requests
headers = {"content-type": "application/json"}
json_response = requests.post(
    'http://localhost:8501/v1/models/fashion_model:predict', 
    data=data, 
    headers=headers
    )

predictions = json.loads(json_response.text)['predictions']

show(0, 'The model thought this was a {} (class {}), and it was actually a {} (class {})'.format(
  class_names[np.argmax(predictions[0])], np.argmax(predictions[0]), class_names[int(test_labels[0])], test_labels[0]))

指定特定版本服務

• 以REST API 向伺服器請求指定版本version = 1。

# docs_infra: no_execute
version = 1

headers = {"content-type": "application/json"}
json_response = requests.post(
    f'http://localhost:8501/v1/models/fashion_model/versions/{version}:predict', 
    data=data, 
    headers=headers
    )

predictions = json.loads(json_response.text)['predictions']

for i in range(0,3):
  show(i, 'The model thought this was a {} (class {}), and it was actually a {} (class {})'.format(
    class_names[np.argmax(predictions[i])], np.argmax(predictions[i]), class_names[int(test_labels[i])], test_labels[i]))