Machine Learning

Machine Learning in Tizen

There are two categories related to Machine Learning in Tizen: (a) Preloaded intelligence services and (b) Machine learning framework support. With (a), Tizen applications can call high-level APIs to invoke preloaded neural network models of Tizen. For example, "mv_face_detect()" in Media Vision APIs. With (b), Tizen applications may execute its own neural network models (e.g., TensorFlow-Lite's .tflite files) with MachineLearning.Inference.Single APIs or create its own neural network pipelines with MachineLearning.Inference.Pipeline APIs.

In this document, we describe the latter part (b).

Machine Learning APIs in Tizen

• MachineLearning.Inference
• MachineLearning.Inference.Single
• MachineLearning.Inference.Pipeline
• MachineLearning.Training

Machine Learning Frameworks in Tizen

• NN Framework, a.k.a., "ONE".
  • NNFW-Runtime
  • NNFW-Compiler
• NN Framework is the default inference engine for Tizen applications and services, which provides a higher degree of runtime optimizations for embedded devices. However, in case of exceptional cases for application writers or system/device integrators, the following frameworks are ported for Tizen and can be loaded to a device.
  • Tensorflow-lite
  • Tensorflow
  • PyTorch/Caffe2
  • ARMNN

On-Device (Re-)Training in Tizen

Although Tizen device integrator may install and use Tensorflow or PyTorch to train neural network models in Tizen devices, they are not recommended due to their excessive resource requirements. For model re-training or personalizations in embedded devices, application or platform developers may use "nntrainer" with its experimental Tizen public APIs (6.0M2).

NNTrainer is ...

Neural Network Pipelines

In 2018-2019, there were various proposals on creating neural-network systems based on stream pipelines. Major companies including Samsung (NNStreamer), Google (MediaPipe), and NVidia (DeepStream) have proposed very similar approaches, suggesting that most neural-network systems may be expressed as stream pipelines (or the pipe and filter style).

Tizen supports such neural network pipelines natively with its public APIs: MachineLearning.Inference.Pipeline, which provides simple interfaces to create NNStreamer pipelines.

NNStreamer pipelines are GStreamer pipelines. Although Tizen API has restrictions on the usage of GStreamer elements (we manage a whitelist of elements that can be used with NNStreamer APIs), a NNStreamer pipeline is a general GStreamer piepline. With NNStreamer, users may process tensor streams as if they are media streams and process them with general neural network frameworks just as MediaPipe does with media and tensor data streams. Note that while MediaPipe and NNStreamer handles tensors as the first class citizen of data streams, DeepStream does not; DeepStream considers tensors as metadata of conventional media data streams, which makes it a bit challenging if you are writing a serious machine-learning application or service although it makes it easier if you are writing a demonstration or visualization application.

Tizen's Machine Learning APIs

Bold for new features.

Tizen 5.5 (M2 / Released)

• C API: SIngle, Pipeline
• .NET API: SIngle
• Web API: N/A

Tizen 6.0 (M1 / Released)

• C API: SIngle, Pipeline
• .NET API: SIngle, Pipeline
• Web API: N/A (dropped)

Tizen 6.0 (M2 / Planned)

• C API: SIngle, Pipeline, Training (experimental)
• .NET API: SIngle, Pipeline
• Web API: N/A

How to write pipeline descriptions (WIP)

• Refer to GStreamer for the pipeline syntax.
• The elements that can be used in a Tizen ML pipeline are limited. The whitelist is available at: TBD (Whitelist draft)

• Tizen ML is based on nnstreamer as its frontend, executing the whole on-device AI system as a GStreamer pipeline.
  • https://github.com/nnsuite/nnstreamer
  • NNStreamer is available for other operating systems as well.
• Tizen ML is going to be based on nnfw/nncc as its backend runtime, exeucting a single neural network
  • WIP: https://github.com/nnsuite/nnstreamer/blob/master/ext/nnstreamer/tensor_filter/tensor_filter_nnfw.c

Machine Learning Infrastructure in Tizen

Neural Network Framework Support @ 5.5 M2

• Tensorflow-Lite 1.13: Supported via MachineLearning.Inference.*

Neural Network Framework Support @ Tizen:Unified (daily build)

• Tensorflow-Lite 1.13: Supported via MachineLearning.Inference.* / We may upgrade it to 1.15.x for 6.0 M1.
• Tensorflow 1.13: Available at devel:AIC:Tizen:5.0:nnsuite:test for x86_64 only.
• Caffe: informally tested (/platform/upstream/caffe) Not Available at Tizen:Unified
• CaffeOnACL: Available at Tizen:Unified
• Caffe2/PyTorch: informally tested (/platform/upstream/pytorch) Not Available at Tizen:Unified
• Google-Coral EdgeTPU Runtime: Available at Tizen:Unified. Accessible via MachineLearning.Inference APIs
• OpenVINO w/ NCS: Available at Tizen:Unified. Accessible via MachineLearning.Inference APIs
• Intel NCSDK2 w/ NCS: Available at Tizen:Unified. Accessible via MachineLearning.Inference APIs

Robotics Support

• ROS1/minimal is available at Tizen devel:AIC:Tizen:5.0:nnsuite Many more ROS1 packages had been tested with Tizen, but not released at tizen.org.