I have an application using Tokbox to create 1:1 video calls with the users. However is it possible to mask/morph the voice of a user as they speak while in a tokbox session? Pitch Modifier
It's possible but not using the officially supported API. You will need to intercept the getUserMedia call, make modifications to the audio track of the intercepted stream, and pass through the modified stream to opentok.js.
See https://tokbox.com/blog/camera-filters-in-opentok-for-web/ for an example on how to intercept the getUserMedia call to make modifications to the video track of the stream.
Here is a basic example of using the mockGetUserMedia function from the blog post to replace the audio track with a simple sine wave:
mockGetUserMedia((originalStream) => {
const audioContext = new window.AudioContext();
const destination = audioContext.createMediaStreamDestination();
const customStream = destination.stream;
originalStream.getVideoTracks().map(videoTrack => customStream.addTrack(videoTrack));
const oscillator = audioContext.createOscillator();
oscillator.start(audioContext.currentTime);
oscillator.connect(destination);
return customStream;
});
Remember: This is not an officially supported API, use at your own risk.
Related
I went through the publisher docs which has the methods publishVideo(value) and publishAudio(value).
Corresponding to the video part, the subscriber receives the event videoDisabled or videoEnabled with the reason publishVideo which allows me to determine whether the subscribed participant has intentionally turned off their video or not, but I can't find something similar for audio such as audioDisabled or audioEnabled. audioBlocked event supposedly only covers blocks by browser's autoplay policy: Dispatched when the subscriber's audio is blocked because of the browser's autoplay policy.
The audioLevelUpdated event provides the current audio level, but that could just be silence, not an intentional mute, so that doesn't look ideal for this purpose.
I want to show a audio muted icon on the subscribed participants element when they have intentionally turned off their audio by calling publishAudio() method. How can that be achieved?
Referenced docs:
Subscriber events: https://tokbox.com/developer/sdks/js/reference/Subscriber.html#events
Publisher methods: https://tokbox.com/developer/sdks/js/reference/Publisher.html#methods
Each stream has an hasAudio attribute that returns false if the user's audio is muted or their microphone is muted. Similarly, streams also have a hasVideo attribute. You can reference the stream docs at https://tokbox.com/developer/sdks/js/reference/Stream.html.
I personally use it like so:
session.streams.forEach((stream) => {
const name = stream.name;
const video = stream.hasVideo;
const audio = stream.hasAudio;
});
You can listen for these changes with the session.on('streamPropertyChanged') event: https://tokbox.com/developer/sdks/js/reference/StreamPropertyChangedEvent.html
Hav u tried audioLevelUpdated and checking the level of audio
If level is 0 then its muted.
https://tokbox.com/developer/sdks/js/reference/Subscriber.html#getAudioVolume
So steps are listen audioLevelUpdated and check AudioVolume, audio volume should get u the subscriber level point.
Set OTSubscriberKitNetworkStatsDelegate to subscriber as
subscriber.networkStatsDelegate = self
Then you can call the below function to get simultaneous changes on subscribers' microphone status
func subscriber(_ subscriber: OTSubscriberKit, audioNetworkStatsUpdated stats: OTSubscriberKitAudioNetworkStats) {
// call stream.hasAudio
}
You can also check from opentok audioNetworkStatsUpdated
I'm trying to stream some URLs to my Chromecast through a sender app. They're HLS/m3u8 URLs.
Here's one such example URL: https://qa-apache-php7.dev.kaltura.com/p/1091/sp/109100/playManifest/entryId/0_wifqaipd/protocol/https/format/applehttp/flavorIds/0_h65mfj7f,0_3flmvnwc,0_m131krws,0_5407xm9j/a.m3u8
However they never seem to load on the Chromecast, despite other HLS/m3u8 URLs working (example of an HLS stream that does work).
It's not related to CORS as they indeed have the proper CORS headers.
I notice they have separate audio groups in the root HLS manifest file.
When I hook it up to a custom receiver app, I get the following logs:
The relevant bits being (I think): Neither ID3 nor ADTS header was found at 0 and cast.player.api.ErrorCode.NETWORK/315 (which I believe is a result of the first)
These are perfectly valid/working HLS URLs. They play back in Safari on iOS and desktop perfectly, as well as VLC.
Is there something I need to be doing (either in my sender app or my receiver app) to enable something like the audio tracks? The docs seem to indicate something about that.
I also found this Google issue where a person had a similar issue, but solved it somehow that I can't understand. https://issuetracker.google.com/u/1/issues/112277373
How would I playback this URL on Chromecast properly? Am I to do something in code?
This already has a solution here but I will add this answer in case someone looks up the exact error message / code.
The problem lies in the hlsSegmentFormat which is initialized to TS for multiplexed segments but currently defaults to packed audio for HLS with alternate audio tracks.
The solution is to intercept the CAF LOAD request and set the correct segment format:
const context = cast.framework.CastReceiverContext.getInstance();
const playerManager = context.getPlayerManager();
// intercept the LOAD request
playerManager.setMessageInterceptor(cast.framework.messages.MessageType.LOAD, loadRequestData => {
loadRequestData.media.hlsSegmentFormat = cast.framework.messages.HlsSegmentFormat.TS;
return loadRequestData;
});
context.start();
Source: Google Cast issue tracker
For those who manage multiple video sources in various formats and who don't want to arbitrarily force the HLS fragment format to TS, I suggest to track the error and set a flag that force the format at the next retry (by default, the receiver tries 3 times before giving up).
First, have a global flag to enable the HLS segments format override:
setHlsSegmentFormat = false;
Then detect the error:
playerManager.addEventListener(cast.framework.events.EventType.ERROR,
event => {
if (event.detailedErrorCode == cast.framework.events.DetailedErrorCode.HLS_NETWORK_INVALID_SEGMENT) {
// Failed parsing HLS fragments. Will retry with HLS segments format set to 'TS'
setHlsSegmentFormat = true;
}
}
);
Finally, handle the flag when intercepting the playback request:
playerManager.setMediaPlaybackInfoHandler(
(loadRequest, playbackConfig) => {
if (setHlsSegmentFormat) {
loadRequest.media.hlsSegmentFormat = cast.framework.messages.HlsSegmentFormat.TS;
// clear the flag to not force the format for subsequent playback requests
setHlsSegmentFormat = false;
}
}
);
The playback will quickly fail the first time and will succeed at the next attempt. The loading time is a bit longer but the HLS segment format is only set when required.
I am trying to use ThreeJS positional audio with WebRTC to build a sort of 3D Room audio chat feature. I am able to get the audio streams sent across the clients. However, the positional audio does not seem to work. Irrespective of where the user (camera) moves, the intensity of the audio remains the same. Some relevant code is being posted below:
The getUserMedia has the following in the promise response
// create the listener
listener = new THREE.AudioListener();
// add it to the camera object
camera.object3D.add( listener );
//store the local stream to be sent to other peers
localStream = stream;
Then on each WebRTC PEER connection, I set the stream that is received to a mesh using Positional Audio
// create the sound object
sound = new THREE.PositionalAudio( listener ); // using the listener created earlier
const soundSource = this.sound.context.createMediaStreamSource(stream);
// set the media source to as the sound source
sound.setNodeSource(soundSource);
// assume that I have handle to the obj where I need to set the sound
obj.object3D.add( sound )
This is done for each of the clients and these local streams are being sent via WebRTC to one another, however there is no sound from the speakers? Thanks
we occurred an issue handling manually audio track and subtitles on sender web and mobile (v3 for both).
Basically, we are able to add some track info before load media, we found the added tracks on the receiver but there are present also the tracks that come from the manifest in two formats (AF and standard object).
There is a way to handle them once and to remove the original that comes from manifest on the receiver side?
Additionally, in this way, the senders will be notified of the change (eg. visible only audio track manually added)?
Many thanks for your support.
You can use message interception:
https://developers.google.com/cast/docs/caf_receiver_features#message-interception
Your interceptor should return the modified request or a Promise that resolves with the modified request value.
You can add your own tracks:
request.media.contentId = mediaUrl;
request.media.contentType = 'application/dash+xml';
request.media.tracks = [{
trackId: 1,
trackContentId: captionUrl,
trackContentType: 'text/vtt',
type: cast.framework.messages.TrackType.TEXT
}];
I need to draw in UI a proper volume level on client (sender) side when working with Chromecast Audio. I see there are two ways of receiving (and might setting as well) volume from Chromecast - from Receiver and Media namespaces. In my understanding Receiver namespace stores general device's volume while Media namespace stores volume of currently played track.
It seems that I can't receive media volume by using GET_STATUS request for Media namespace before I load any tracks with LOAD request. Then how do I correctly display volume that will be used before loading media? Changing in UI RECEIVER volume to MEDIA volume after media is loaded doesn't look like a good solution and will be a surprise for users.
I fail to control volume using SET_VOLUME request for Receiver namespace - I've got no reply from Chromecast
Json::Value msg, response;
msg["type"] = "SET_VOLUME";
msg["requestId"] = ++request_id;
msg["volume"]["level"] = value; // float
response = send("urn:x-cast:com.google.cast.receiver", msg);
If the following lines are used instead of the last one, media volume is controlled OK:
msg["mediaSessionId"] = m_media_session_id;
response = send("urn:x-cast:com.google.cast.media", msg);
What am I doing wrong here?
In order to set the volume on the receiver, you should be using the SDK's APIs instead of sending a hand-crafted message. For example, you should use setReceiverVolumeLevel(). Also, use the receiver volume and not the stream volume.