IVI/Multiuser Audio

From Tizen Wiki
Jump to: navigation, search

Multiuser Audio


Summary and Scope

This is a feasibility study to understand how to implement true multiuser audio in Tizen (IVI).


  • Use case 1:
    • Driver is listening to music in loudspeakers and passenger is listening different music from his/her own headphones
  • Use case 2:
    • Passenger can share his music with other passenger or public speakers. The receiving user can choose if he wants to listen the available shared stream.
  • Use case 3:
    • System must be able to play notifications to all users, possibly wearing headphones. Like "passenger 4 please fasten your seat belt"


  • usb info via udev (i.e., which passenger seat the headset belongs). policy not making usb slot decision



udev and seats: configure audio devices

udev, and multiseat overall is best covered in systemd multi-seat document: http://www.freedesktop.org/wiki/Software/systemd/multiseat/

One notable item is there is always a seat with name "seat0", which is the default seat. In our case the driver seat should be seat0.

Normally there would be one GPU per seat. However, our reference hardware has only one GPU with two monitor ports. It's not currently possible to assign different monitor ports of the same GPU to different seats. To make this possible, the GPU drivers should create per-port device nodes. TODO: We should not assign the GPU to any seat, since it will be shared by multiple seats.

If we want to configure e.g. the on-board sound card of VTC-7 to be used for the speakers, we should not assign it to any seat, since the speakers are shared by all seats. This requires the sound card to not have the "seat" tag in its udev configuration. However, currently udev automatically tags all sound cards with "seat" in 71-seat.rules. We can work around this by overriding 71-seat.rules in /etc. Here's how to do it: first, copy /usr/lib/udev/rules.d/71-seat.rules to /etc/udev/rules/71-seat.rules. Then, apply these changes to the file under /etc:

--- /usr/lib/udev/rules.d/71-seat.rules	2014-06-23 13:30:30.000000000 -0700
+++ /etc/udev/rules.d/71-seat.rules	2014-08-26 01:52:52.144853486 -0700
@@ -7,11 +7,15 @@
 ACTION=="remove", GOTO="seat_end"
-TAG=="uaccess", SUBSYSTEM!="sound", TAG+="seat"
-SUBSYSTEM=="sound", KERNEL=="card*", TAG+="seat"
-SUBSYSTEM=="input", KERNEL=="input*", TAG+="seat"
-SUBSYSTEM=="graphics", KERNEL=="fb[0-9]*", TAG+="seat", TAG+="master-of-seat"
-SUBSYSTEM=="usb", ATTR{bDeviceClass}=="09", TAG+="seat"
+# If the device or any of its parents has the "seatless" tag set, then we
+# should not assign the device to any seat.
+TAGS=="seatless", ENV{SEATLESS}="1"
+ENV{SEATLESS}!="1", TAG=="uaccess", SUBSYSTEM!="sound", TAG+="seat"
+ENV{SEATLESS}!="1", SUBSYSTEM=="sound", KERNEL=="card*", TAG+="seat"
+ENV{SEATLESS}!="1", SUBSYSTEM=="input", KERNEL=="input*", TAG+="seat"
+ENV{SEATLESS}!="1", SUBSYSTEM=="graphics", KERNEL=="fb[0-9]*", TAG+="seat", TAG+="master-of-seat"
+ENV{SEATLESS}!="1", SUBSYSTEM=="usb", ATTR{bDeviceClass}=="09", TAG+="seat"
 # 'Plugable' USB hub, sound, network, graphics adapter
 SUBSYSTEM=="usb", ATTR{idVendor}=="2230", ATTR{idProduct}=="000[13]", ENV{ID_AUTOSEAT}="1"

Now udev will not assign any devices that have the "seatless" tag to any seats. To exclude the VTC-7 on-board sound card from seat assignment, we can create /etc/udev/rules.d/00-seatless.rules with the following contents:

ACTION=="remove", GOTO="end"

DEVPATH=="/devices/pci0000:00/0000:00:1b.0/sound/card0", TAG+="seatless"


If you want to adapt this for different hardware than VTC-7, you can use "loginctl seat-status seat0" to find the device path of the sound card that is currently assigned to seat0, then use the device path in 00-seatless.rules.

If we want to package the changes to 71-seat.rules, the approach of overriding the upstream version with a custom version under /etc isn't really the optimal solution. There is a thread on systemd-devel about supporting seatless sound cards in udev upstream configuration: http://thread.gmane.org/gmane.comp.sysutils.systemd.devel/22412


Cascaded Pulseaudio processes (user-session PA and system PA)

Multi Node Multi Pulse


system instance 
a pulseaudio process running as a system service
user instance 
a pulseaudio process running as a user (session) service

The system instance may run on the same or different machine as the user instances. Here we'll focus on the same machine case, the configuration would be a bit different in the different machine case.

How to start the system instance at boot

Basic steps:

  • Install pulseaudio.service
  • Hook the pulseaudio service to the boot sequence

Here's an example of a simple but working service file:

Description = PulseAudio system instance

ExecStart = @PA_BINARY@ --system --daemonize=no --disallow-exit

WantedBy = sound.target

"@PA_BINARY@" is meant to be automatically replaced with the executable path when compiling pulseaudio. If you just want to manually test this out, replace it yourself with "/usr/bin/pulseaudio".

The service file should be installed to /usr/lib/systemd/system/pulseaudio.service. After installing the service file, remember to enable it with "systemctl enable pulseaudio.service". Now pulseaudio should start automatically after a reboot.

Autospawning when the system instance is running

Autospawning, which Tizen heavily relies on, doesn't work if the system instance is running, because libpulse tries to connect to the system instance before resorting to autospawning, so if the system instance is available, it will be used instead of autospawning the user instance. libpulse should be changed so that it doesn't try to connect to the system instance unless explicitly requested.

Access control in the system instance

By default the system instance only accepts connections from users that are in the "pulse-access" group. It's also possible to not require this by giving the "auth-anonymous=yes" argument to module-native-protocol-unix, in which case there are no limits on who can connect to the system instance.

Users that get access to the system instance have full access to all pulseaudio functionality (except shutting down the daemon can be prevented with the "--disallow-exit" command line argument). This can be a security issue in the sense that users are not isolated from each other. It would be easy to change the code so that module loading and unloading would be forbidden for clients, which would be a small improvement to the situation.

Device access configuration

The system instance runs under the "pulse" user. The "pulse" user is in the "audio" group, so it has access to all alsa sound cards (/dev/snd/*). We don't want the system instance to access per-seat sound cards. To achieve that, we could modify system.pa so that it doesn't load module-udev-detect, but loads only a hard-coded set of module-alsa-card instances, or add a "seatless_only" argument for module-udev-detect that we'd use in system.pa. TODO: Discuss with udev developers whether and how udev should support marking devices as "seatless". Discussion ongoing: http://thread.gmane.org/gmane.comp.sysutils.systemd.devel/22412

The user instances should access only sound cards that are assigned to the seat of the user's currently active session (or possibly multiple sessions, but we probably don't need to support that). This is usually, on non-Tizen systems, achieved via the automatic device ACL management that logind does. PulseAudio monitors changes in the device permissions, and when the user loses permission to a device, PulseAudio stops using that device. On Tizen this doesn't currently work, at least not in the default Tizen Common setup without tlm nor with tlm, because the ACL management relies on active sessions being marked as active, and currently that doesn't happen. Also, users are currently in the "audio" group, which overrides any automatic ACL management anyway, and allows all users to use all sound cards all the time. So, if we want to use device permissions for managing which devices a user PulseAudio instance uses, there are two things to do: fix the session management so that active sessions are marked as active, and remove all users (except the "pulse" user) from the "audio" group.

Using the seat information

If device permissions are configured so that users only have access to per-seat alsa sound cards and the system instance only has access to shared alsa sound cards, then module-udev-detect doesn't necessarily need changes. However, if we don't want to rely only on the device permissions, it wouldn't be very hard to modify module-udev-detect to follow these rules:

  • If the pulseaudio process belongs to a seat, then only try to use those sound cards that are marked to belong to the process's seat.
  • If the pulseaudio process doesn't belong to a seat, then only try to use those sound cards that are not marked to belong to any seat.

What if per-card granularity isn't enough?

What if one alsa card has some devices that are shared and some devices that are per-seat, or some devices that belong to one seat and some devices that belong to another? Currently module-udev-detect works only at the card level, individual devices aren't considered. Udev can probably attach attributes to individual alsa device nodes (/dev/snd/*), but supporting this in pulseaudio in a general way is difficult, because there's no direct mapping from the alsa device names that pulseaudio uses to the device nodes that udev is concerned with. Luckily, hotpluggable devices are always represented by separate card objects, so this problem only affects the built-in audio devices of the car. Vendors should be able to write the pulseaudio configuration for the system and user pulseaudio instances so that they use only those portions of the built-in devices that they are supposed to.

But what if even per-device-node (/dev/snd/*) granularity isn't enough? It's possible to imagine a sound card that relies on alsa mixer only to control whether to play to the shared speakers or to per-seat headphones. This would mean that the shared speakers and the headphones would be mutually exclusive, which doesn't make much sense, so maybe we can ignore this case?

Connecting user instances to the system instance

Applications use the user instance, and if applications need to use devices that are managed by the system instance, then the user instance needs to somehow act as a proxy between the application and the system instance. Currently the basic tool for achieving that is to use tunnel sinks and sources. A tunnel sink is a virtual output device in the user instance that takes audio from applications and sends it to a hardware device in the system instance. A tunnel source is the same, but for input devices.

We could write a simple module that creates tunnels for every sink and source in the system instance. If the system instance has multiple devices to choose from, then the device choice would have to be done in the user context, which may not be a desirable thing. On the other hand, if the system instance has devices whose routing involves profile and port selection choices, those choices couldn't be done in the user context, because the tunnels, as currently implemented, don't expose the profiles and ports. Therefore, part of the routing policy would have to be in the user context and part in the system context, which is certainly not nice.

With additional work, we could push the entire routing policy to either to the user context or to the system context. (To clarify, we're talking about devices managed by the system instance only; routing policy that concerns devices that belong to the user instance should always be done in the user context.) To have all routing policy in the user context, we could implement "card tunnels" or "card proxies" so that the user instance could control the card profiles and ports also for devices managed by the system instance. To have all routing policy in the system context we could extend module-murphy-ivi so that it would create only a single routing node for the system instance. The node would be implemented so that when streams are routed to that node, module-murphy-ivi would load tunnel sinks dynamically for each stream separately. Then the system instance would be able to decide on per-stream basis which hardware device to use.

The best solution would probably be to push the routing policy where the devices are, i.e. the user instance should have only one routing node for the system instance and module-murphy-ivi should load per-stream tunnel sinks. If this doesn't seem like the best solution to everybody, we could of course support multiple approaches.

There are two parallel implementations of tunnel sinks and sources. The older implementation is eventually going to be replaced by the newer implementation, so from this point of view it would be much better to use the new one. On the other hand, the new one doesn't work very well yet. The new version should be fixable with reasonable effort, however.

Known issues in the new tunnel implementation:

  • The maximum latency is 10 seconds. 2 seconds would make more sense (easily changed).
  • While the new tunnel module supports dynamic latency (meaning that applications requiring low latency will get that), in practice this doesn't work well enough due to missing rewinding support. Rewinding support should be possible to implement, but it's probably not an easy task. Here are some examples of what the lack of rewinding means:
    • If the current latency is 2 seconds, it will take up to 2 seconds for new streams to become audible.
    • If the current latency is 2 seconds, it will take up to 2 seconds for pausing/unpausing to have effect.
    • If the current latency is 2 seconds, it will take up to 2 seconds for volume changes to become audible.
  • Since the dynamic latency feature doesn't work well enough, we should make the maximum latency configurable (easy task) and configure it to e.g. 100 ms. (For comparison, the old tunnel module doesn't support dynamic latency at all, and it has a fixed latency of 150 ms. Making that configurable would be equally easy compared to making the maximum latency configurable in the new tunnel module.)
  • If a tunnel sink is idle, i.e. there are no streams connected to it, it sends a constant stream of silence to the system instance. To conserve system resources, the stream should be paused when nothing is playing to the tunnel sink. (Should be pretty easy to fix, and actually there are already unreviewed patches on the pulseaudio mailing list for fixing this issue.)
  • The tunnel sink uses an approach for audio rendering that results in less efficient memory allocation and repeated debug log messages if the latency is big enough. This should be pretty easy to fix.

Both the old and new tunnel modules have the problem that each sink and source creates a new client connection to the system instance. This may or may not be a problem in practice, but it would be nice to refactor the code so that we could have just one client connection from each user instance to the system instance.

Avoiding tunnels

The tunnel-based two-hop streaming has some inherent downsides, such as negative impact on the lowest achievable latency and debuggability (it's harder to debug issues in streaming if there are two hops compared to just one hop). It's not feasible to change the design of the streaming protocol in the short term, but there are a couple of possible solutions to the two-hop problem that could be implemented at some point in the future.

The first idea (I'll call this the "stream forwarding idea") is that when a client creates a stream, and the user instance decides that the stream should played on a shared device, the user instance could say to the client that "please connect to the system instance and play the stream there". The client would then create another (parallel) connection to the system instance. The system instance connection would only be used for streaming purposes, for other communication the client should still use the user instance connection. The stream should have a representation also in the user instance to allow rerouting within the user instance.

The second idea (I'll call this the "fd passing idea") is that the user instance would get a file descriptor from the system instance for streaming, and then the user instance could pass on that descriptor to the client. This would be entirely transparent to the client. It doesn't need to know what's behind the streaming socket, is it the user instance or is it the system instance. The problem is that currently there are no separate file descriptors for audio streaming and other communication; everything goes through the same socket. Separating the two types of data is something that upstream would very much like to have, as it would solve other problems as well, it's just that implementing this is quite a lot of work. The "stream forwarding idea" would probably be somewhat easier to implement, but on the other hand, the "fd passing idea" would probably make more sense from architecture point of view.

Bluetooth considerations

Bluetooth devices are often seat specific in the sense that only one passenger uses them at a time (e.g. a headset), but the system has no idea on which seat the device is currently being used, unless there are per-seat bluetooth adapters. Therefore, bluetooth audio devices need to be managed in the system instance, and the user instance configuration should be modified to not load module-bluetooth-detect.

Managing the bluetooth devices in the system instance makes automatic routing to bluetooth devices difficult: how to figure out which user's audio should be routed to the bluetooth device when a new device connects to the system? Some user interaction is likely needed to establish the device-user connection. TODO: Do we need to come up with some details about what exactly is needed?

Seat switching within a session

We should support moving a session from one seat to another. If we don't rely solely on the /dev/snd/* permissions to reconfigure the set of managed devices within the user instance, PulseAudio needs to follow changes in what seat the session belongs to. This functionality doesn't exist at the moment.

Code changes

  • Required
    • Modify system.pa so that it doesn't load module-udev-detect but loads a hard-coded set of module-alsa-card instances, or add a "seatless_only" argument to module-udev-detect and use it in system.pa.
    • Fix session management so that active sessions are marked as active (necessary if we want to rely on device permissions for deciding which sound cards a user instance should use).
    • Remove users from the "audio" group (necessary if we want to rely on device permissions for deciding which sound cards a user instance should use).
    • Modify module-udev-detect so that it takes seat information into consideration before trying to use a sound card (necessary if we don't want to rely on device permissions only).
    • Modify module-udev-detect so that it monitors changes in what seat the session belongs to and adjusts the set of managed sound cards accordingly (necessary if we don't want to rely on device permissions only).
    • Write a simple module for loading tunnel sinks and sources in the user instance for each sink and source in the system instance (If we don't offer a better solution).
    • If we want to support pushing the routing policy concerning shared devices to the user instance: Write a "card proxy" module that would create a card object in the user instance to mirror a card object in the system instance. There would also need to be a module that would automatically load these card proxies for each card in the system instance.
    • Extend module-murphy-ivi so that it creates per-stream tunnel sinks and sources for each stream that is routed to the system instance, If we want to support pushing the routing policy concerning shared devices to the system instance.
    • Make the maximum latency of the new tunnel module configurable, if rewinding support isn't implemented.
    • Avoid wakeups when nothing is playing: Stop streaming from the user instance to the system instance when there are no streams playing to a tunnel sink.
    • Autospawning: Modify libpulse so that it doesn't try to connect to the system instance unless explicitly requested.
  • Nice to have
    • Additional security: Modify the --allow-module-loading=no behaviour so that it prevents clients from loading and unloading modules, but doesn't prevent hotplugging from working (i.e. server-initiated module loading should be allowed).
    • For convenience: Add --system and --user switches to pactl and other utilities for easy selection between the user and the system instance.
    • Lower the maximum latency of the new tunnel module from 10 seconds to 2 seconds. (this might actually not matter, if we configure the maximum latency to a lower value anyway)
    • Allow higher maximum latency: Implement rewind support in the new tunnel module.
    • Reduce the amount of client connections from one user instance to the system instance: Make it possible for multiple tunnel sinks to share one client connection.
    • Increase memory allocation efficiency (may not have noticeable effect, though) and to reduce debug log size: Fix the audio rendering approach of the new tunnel module (shouldn't use pa_sink_render_full()).

Single System Pulseaudio

Single Node Multi Pulse

Cascade user Pulseaudio with another sound server