Synavis is an open-source data production orchestration library focused on producing synthetic data using Unreal Engine and coupled simulations. It has been developed primarily for plant sciences, and many of its individual features are specifically tailored to this domain. A key limitation so far has been the inability to dynamically allocate data cameras, as the WebRTC initialization almost always had to be done prior to the actual scene startup, even if the streaming initialization event does not take place until a "player" is connected (see Streamer class specification).
The Plan
I am assessing whether we move to PixelStreaming2 or whether we will build our implementation directly upon a WebRTC implementation. Using PixelStreaming provides more expedient inclusion into the general workflow of the UE-WebRTC implementation and makes it more readily compatible with other methods. However, PixelStreaming2 is still implemented for active and human users, which Synavis is not necessarily.
The plan is to first implement a prototype using PixelStreaming2, and then evaluate whether this meets the needs of Synavis. If not, I will explore a more direct WebRTC implementation that allows for greater flexibility in camera management and data streaming. Ultimately, this is probably where we will go, since we do not gain the primary benefits of the native implementation anyways: the inclusion of the hardware encoders directly in the rendering pipeline. I could of course implement this myself as well, but the primary issue is that the GPUs are limited regarding the number of encoders for H264 and H265 streams, and we would be using VP9 for greater flexibility regarding the number of cameras. Second plus is that according to the documentation of PixelStreaming, VP9 does not suffer quality issues when the performance or throughput is low.
WebRTC orchestration
The user should be able to do most things dynamically. The ideal setup process will be adapted from the default process above. The SynavisDrone will be replaced by a "Data Camera" while the SynavisUE plugin mostly will be reduced to processing commands using the InputProcessing class already present within the plugin.
- The WebRTC connection will be initialized when the first data camera is created.
- Each data camera will create its own PeerConnection and DataChannel.
- The signaling server will manage multiple PeerConnections from the same UE instance.
- Cameras can be added and removed dynamically during runtime.
- The user can start and stop streaming for each camera independently.
- Users can choose the viewpoint, camera settings, and label data produced by each camera.
This approach will open up more possibilities, particularly in combination with how the signalling server is setup currently, as its implementation allows for matching. This matching only really makes sense if the data production is more fully dynamic within a certain domain-specific scene.
Development Plan and Use-Cases
The use-case development is not always an expedient choice, but in this case there are specific application cases that would benefit from the re-implementation, particularly scene evaluation of algorithms. A bidirectional use-case for the Synavis framework along with an implementation of dynamic data cameras is the primary target of re-implementing Synavis with a more flexible WebRTC integration.
Can I help?
Thank you so much for asking! I am prototyping but would welcome having a chat about the technical aspects. If you are interested in helping out, please reach out via email.