I am trying to use ThreeJS positional audio with WebRTC to build a sort of 3D Room audio chat feature. I am able to get the audio streams sent across the clients. However, the positional audio does not seem to work. Irrespective of where the user (camera) moves, the intensity of the audio remains the same. Some relevant code is being posted below:
The getUserMedia has the following in the promise response
// create the listener
listener = new THREE.AudioListener();
// add it to the camera object
camera.object3D.add( listener );
//store the local stream to be sent to other peers
localStream = stream;
Then on each WebRTC PEER connection, I set the stream that is received to a mesh using Positional Audio
// create the sound object
sound = new THREE.PositionalAudio( listener ); // using the listener created earlier
const soundSource = this.sound.context.createMediaStreamSource(stream);
// set the media source to as the sound source
sound.setNodeSource(soundSource);
// assume that I have handle to the obj where I need to set the sound
obj.object3D.add( sound )
This is done for each of the clients and these local streams are being sent via WebRTC to one another, however there is no sound from the speakers? Thanks
Related
I am using pyav to process a live RTSP stream:
import av
import time
URL = "RTSP_url"
container = av.open(
url, 'r',
options={
'rtsp_transport': 'tcp',
'stimeout': '5000000',
'max_delay': '5000000',
}
)
for packet in self.container.demux(video=0):
for frame in packet.decode():
# do something
time.sleep(10)
What happens if I do something too slow? Are frames / packets dropped or are they buffered?
I guess the same question would apply to libav or ffmpeg.
tcp is a guaranteed delivery protocol with built-in flow control. If you do not process the incoming data as fast as it is received, the tcp stack will buffer the data until its buffers are full at which time the tcp protocol will let the sender know that it cannot receive any more data. If this continues, the sender's output buffers will eventually fill up and then it is up to the sender to decide what to do.
An IP camera at that point may throw frames away or it may even drop the connection. Most IP cameras also use a keep-alive mechanism typically via RTCP packets sent over the RTSP stream. The camera may send Sender Reports and the receiver should send back Receiver Reports. If the camera does not get a Receiver Report within a timeout, it will drop the connection. I would have to assume that either the av library or ffmpeg is doing that.
You probably do not want to do time.sleep(10).
If you really feel that you need to discard packets, then you could examine your packets before calling decode to see if you are falling behind. If you are getting too far behind, you can discard packets that are not key frames until you catch up. The effect will be that the video will have jumps in it.
In my experience gstreamer could store in a buffer old frames and return them even minutes later. Not sure if PyAv would do the same.
I went through the publisher docs which has the methods publishVideo(value) and publishAudio(value).
Corresponding to the video part, the subscriber receives the event videoDisabled or videoEnabled with the reason publishVideo which allows me to determine whether the subscribed participant has intentionally turned off their video or not, but I can't find something similar for audio such as audioDisabled or audioEnabled. audioBlocked event supposedly only covers blocks by browser's autoplay policy: Dispatched when the subscriber's audio is blocked because of the browser's autoplay policy.
The audioLevelUpdated event provides the current audio level, but that could just be silence, not an intentional mute, so that doesn't look ideal for this purpose.
I want to show a audio muted icon on the subscribed participants element when they have intentionally turned off their audio by calling publishAudio() method. How can that be achieved?
Referenced docs:
Subscriber events: https://tokbox.com/developer/sdks/js/reference/Subscriber.html#events
Publisher methods: https://tokbox.com/developer/sdks/js/reference/Publisher.html#methods
Each stream has an hasAudio attribute that returns false if the user's audio is muted or their microphone is muted. Similarly, streams also have a hasVideo attribute. You can reference the stream docs at https://tokbox.com/developer/sdks/js/reference/Stream.html.
I personally use it like so:
session.streams.forEach((stream) => {
const name = stream.name;
const video = stream.hasVideo;
const audio = stream.hasAudio;
});
You can listen for these changes with the session.on('streamPropertyChanged') event: https://tokbox.com/developer/sdks/js/reference/StreamPropertyChangedEvent.html
Hav u tried audioLevelUpdated and checking the level of audio
If level is 0 then its muted.
https://tokbox.com/developer/sdks/js/reference/Subscriber.html#getAudioVolume
So steps are listen audioLevelUpdated and check AudioVolume, audio volume should get u the subscriber level point.
Set OTSubscriberKitNetworkStatsDelegate to subscriber as
subscriber.networkStatsDelegate = self
Then you can call the below function to get simultaneous changes on subscribers' microphone status
func subscriber(_ subscriber: OTSubscriberKit, audioNetworkStatsUpdated stats: OTSubscriberKitAudioNetworkStats) {
// call stream.hasAudio
}
You can also check from opentok audioNetworkStatsUpdated
I'm trying to stream some URLs to my Chromecast through a sender app. They're HLS/m3u8 URLs.
Here's one such example URL: https://qa-apache-php7.dev.kaltura.com/p/1091/sp/109100/playManifest/entryId/0_wifqaipd/protocol/https/format/applehttp/flavorIds/0_h65mfj7f,0_3flmvnwc,0_m131krws,0_5407xm9j/a.m3u8
However they never seem to load on the Chromecast, despite other HLS/m3u8 URLs working (example of an HLS stream that does work).
It's not related to CORS as they indeed have the proper CORS headers.
I notice they have separate audio groups in the root HLS manifest file.
When I hook it up to a custom receiver app, I get the following logs:
The relevant bits being (I think): Neither ID3 nor ADTS header was found at 0 and cast.player.api.ErrorCode.NETWORK/315 (which I believe is a result of the first)
These are perfectly valid/working HLS URLs. They play back in Safari on iOS and desktop perfectly, as well as VLC.
Is there something I need to be doing (either in my sender app or my receiver app) to enable something like the audio tracks? The docs seem to indicate something about that.
I also found this Google issue where a person had a similar issue, but solved it somehow that I can't understand. https://issuetracker.google.com/u/1/issues/112277373
How would I playback this URL on Chromecast properly? Am I to do something in code?
This already has a solution here but I will add this answer in case someone looks up the exact error message / code.
The problem lies in the hlsSegmentFormat which is initialized to TS for multiplexed segments but currently defaults to packed audio for HLS with alternate audio tracks.
The solution is to intercept the CAF LOAD request and set the correct segment format:
const context = cast.framework.CastReceiverContext.getInstance();
const playerManager = context.getPlayerManager();
// intercept the LOAD request
playerManager.setMessageInterceptor(cast.framework.messages.MessageType.LOAD, loadRequestData => {
loadRequestData.media.hlsSegmentFormat = cast.framework.messages.HlsSegmentFormat.TS;
return loadRequestData;
});
context.start();
Source: Google Cast issue tracker
For those who manage multiple video sources in various formats and who don't want to arbitrarily force the HLS fragment format to TS, I suggest to track the error and set a flag that force the format at the next retry (by default, the receiver tries 3 times before giving up).
First, have a global flag to enable the HLS segments format override:
setHlsSegmentFormat = false;
Then detect the error:
playerManager.addEventListener(cast.framework.events.EventType.ERROR,
event => {
if (event.detailedErrorCode == cast.framework.events.DetailedErrorCode.HLS_NETWORK_INVALID_SEGMENT) {
// Failed parsing HLS fragments. Will retry with HLS segments format set to 'TS'
setHlsSegmentFormat = true;
}
}
);
Finally, handle the flag when intercepting the playback request:
playerManager.setMediaPlaybackInfoHandler(
(loadRequest, playbackConfig) => {
if (setHlsSegmentFormat) {
loadRequest.media.hlsSegmentFormat = cast.framework.messages.HlsSegmentFormat.TS;
// clear the flag to not force the format for subsequent playback requests
setHlsSegmentFormat = false;
}
}
);
The playback will quickly fail the first time and will succeed at the next attempt. The loading time is a bit longer but the HLS segment format is only set when required.
we occurred an issue handling manually audio track and subtitles on sender web and mobile (v3 for both).
Basically, we are able to add some track info before load media, we found the added tracks on the receiver but there are present also the tracks that come from the manifest in two formats (AF and standard object).
There is a way to handle them once and to remove the original that comes from manifest on the receiver side?
Additionally, in this way, the senders will be notified of the change (eg. visible only audio track manually added)?
Many thanks for your support.
You can use message interception:
https://developers.google.com/cast/docs/caf_receiver_features#message-interception
Your interceptor should return the modified request or a Promise that resolves with the modified request value.
You can add your own tracks:
request.media.contentId = mediaUrl;
request.media.contentType = 'application/dash+xml';
request.media.tracks = [{
trackId: 1,
trackContentId: captionUrl,
trackContentType: 'text/vtt',
type: cast.framework.messages.TrackType.TEXT
}];
I have an application using Tokbox to create 1:1 video calls with the users. However is it possible to mask/morph the voice of a user as they speak while in a tokbox session? Pitch Modifier
It's possible but not using the officially supported API. You will need to intercept the getUserMedia call, make modifications to the audio track of the intercepted stream, and pass through the modified stream to opentok.js.
See https://tokbox.com/blog/camera-filters-in-opentok-for-web/ for an example on how to intercept the getUserMedia call to make modifications to the video track of the stream.
Here is a basic example of using the mockGetUserMedia function from the blog post to replace the audio track with a simple sine wave:
mockGetUserMedia((originalStream) => {
const audioContext = new window.AudioContext();
const destination = audioContext.createMediaStreamDestination();
const customStream = destination.stream;
originalStream.getVideoTracks().map(videoTrack => customStream.addTrack(videoTrack));
const oscillator = audioContext.createOscillator();
oscillator.start(audioContext.currentTime);
oscillator.connect(destination);
return customStream;
});
Remember: This is not an officially supported API, use at your own risk.