Machine Learning

Machine Learning in Tizen

There are two categories related to Machine Learning inferences in Tizen: (a) Preloaded intelligence services and (b) Machine learning framework support. With (a), Tizen applications can call high-level APIs to invoke preloaded neural network models of Tizen. For example, "mv_face_detect()" in Media Vision APIs. With (b), Tizen applications or frameworks may execute their own neural network models (e.g., TensorFlow-Lite's .tflite files) with MachineLearning.Inference.Single APIs or create its own neural network pipelines with MachineLearning.Inference.Pipeline APIs.

In this document, for inferences, we describe the latter part (b). Note that the former part (a) is recommended to be implemented with the latter part (b).

Other than inferences, Tizen Machine Learning supports (c) MachineLearning.Training and (d) MachineLearning.Service APIs. (c) allows to train or update neural networks with nntrainer and (d) allows to create, deploy, or launch machine learning services (pipelines) for remote/local accesses of other applications, and allows to connect to such services. This allows among-device AI services via SmartThings and Matter as well.

Machine Learning APIs in Tizen

• MachineLearning.Inference
• MachineLearning.Inference.Single
• MachineLearning.Inference.Pipeline
• MachineLearning.Training
• MachineLearning.Service

Machine Learning Frameworks in Tizen

• NN Framework, a.k.a., "ONE".
  • NNFW-Runtime
  • NNFW-Compiler
• NN Framework is the default inference engine for Tizen applications and services, which provides a higher degree of runtime optimizations for embedded devices. However, in case of exceptional cases for application writers or system/device integrators, the following frameworks are ported for Tizen and can be loaded to a device.
  • Tensorflow-lite
  • Tensorflow
  • PyTorch/Caffe2
  • ARMNN

On-Device (Re-)Training in Tizen

Although Tizen device integrator may install and use Tensorflow or PyTorch to train neural network models in Tizen devices, they are not recommended due to their excessive resource requirements. For model re-training or personalizations in embedded devices, application or platform developers may use "nntrainer" with its experimental Tizen public APIs (6.0M2).

NNTrainer is ...

Neural Network Pipelines

In 2018-2019, there were various proposals on creating neural-network systems based on stream pipelines. Major companies including Samsung (NNStreamer), Google (MediaPipe), and NVidia (DeepStream) have proposed very similar approaches, suggesting that most neural-network systems may be expressed as stream pipelines (or the pipe and filter style).

Tizen supports such neural network pipelines natively with its public APIs: MachineLearning.Inference.Pipeline, which provides simple interfaces to create NNStreamer pipelines.

NNStreamer pipelines are GStreamer pipelines. Although Tizen API has restrictions on the usage of GStreamer elements (we manage a whitelist of elements that can be used with NNStreamer APIs), a NNStreamer pipeline is a general GStreamer piepline. With NNStreamer, users may process tensor streams as if they are media streams and process them with general neural network frameworks just as MediaPipe does with media and tensor data streams. Note that while MediaPipe and NNStreamer handles tensors as the first class citizen of data streams, DeepStream does not; DeepStream considers tensors as metadata of conventional media data streams, which makes it a bit challenging if you are writing a serious machine-learning application or service although it makes it easier if you are writing a demonstration or visualization application.

Tizen's Machine Learning APIs

Bold for new features.

Tizen 5.5 (M2 / Released)

• C API: SIngle, Pipeline
• .NET API: SIngle
• Web API: N/A

Tizen 6.0 (M1 / Released)

• C API: Single, Pipeline
• .NET API: Single, Pipeline
• Web API: N/A

Tizen 6.0 (M2 / Released)

• C API: SIngle, Pipeline, Training (experimental)
• .NET API: Single, Pipeline
• Web API: Single, Pipeline

Tizen 7.0 (M2 / Released)

• C API: Single, Pipeline, Training, Service (P1/P2)
• .NET API: Single, Pipeline, Training
• Web API: Single, Pipeline, Training

P1: register/unregister/fetch models and pipelines P2: launch a pipeline as a service, connect to a pipeline via remote/local connections P3: update models with versioning mechanisms. P4: launch a training session as a service. P5: Tizen ML Service fully supports the device side of MLOPS.

How to write pipeline descriptions (WIP)

• Refer to GStreamer for the pipeline syntax.
• The elements that can be used in a Tizen ML pipeline are limited. The whitelist is available at: TBD (Whitelist draft)

• Tizen ML is based on nnstreamer as its frontend, executing the whole on-device AI system as a GStreamer pipeline.
  • https://github.com/nnsuite/nnstreamer
  • NNStreamer is available for other operating systems as well.
• Tizen ML is going to be based on nnfw/nncc as its backend runtime, exeucting a single neural network
  • WIP: https://github.com/nnsuite/nnstreamer/blob/master/ext/nnstreamer/tensor_filter/tensor_filter_nnfw.c

Machine Learning Infrastructure in Tizen

Neural Network Framework Support @ 5.5 M2

• Tensorflow-Lite 1.13: Supported via MachineLearning.Inference.*

Neural Network Framework Support @ Tizen:Unified (daily build)

• Tensorflow-Lite 1.13: Supported via MachineLearning.Inference.* / We may upgrade it to 1.15.x for 6.0 M1.
• Tensorflow 1.13: Available at devel:AIC:Tizen:5.0:nnsuite:test for x86_64 only.
• Caffe: informally tested (/platform/upstream/caffe) Not Available at Tizen:Unified
• CaffeOnACL: Available at Tizen:Unified
• Caffe2/PyTorch: informally tested (/platform/upstream/pytorch) Not Available at Tizen:Unified
• Google-Coral EdgeTPU Runtime: Available at Tizen:Unified. Accessible via MachineLearning.Inference APIs
• OpenVINO w/ NCS: Available at Tizen:Unified. Accessible via MachineLearning.Inference APIs
• Intel NCSDK2 w/ NCS: Available at Tizen:Unified. Accessible via MachineLearning.Inference APIs

Robotics Support

• ROS1/minimal is available at Tizen devel:AIC:Tizen:5.0:nnsuite Many more ROS1 packages had been tested with Tizen, but not released at tizen.org.