Framerate independent event in Unity3D - events

I use the libpd4unity package to communicate with Pure Data. I receive a bang from Pure Data with LibPD.Bang. On a bang event I play sound by FMOD.
Problem is, that I receive bangs frequently, for example once every 500 ms but event doesn't trigger in specific length of frame. Usually length change 1 frame less or more.
Is there a solution for this problem? For example a framerate independent event? I want to know if event (delegate) in Unity3D is framerate independent or not.
Because there is tempo for playing each sound and just 1 frame ruins rhythm.
I need to sync sounds for playing by each separate bang.

Regarding your question on whether delegates are dependent or independent from Unity's framerate, there's no straight answer. It depends on how your delegates are called. Are they called from a thread? Are they executed in a thread?
Coroutines are not framerate independent, they are executed in Unity's loop.
The following script should shine a light on the difference between handling delegates in coroutines and in threads.
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using System.Threading;
public class DelegatesAndFramerate : MonoBehaviour {
delegate void MyDelegate();
MyDelegate myDelegate1; // done with coroutines
MyDelegate myDelegate2; // done with threads
Thread thread;
bool threadDone = false;
private int frameCount = 0;
private int delegate1CallCount = 0;
private int delegate2CallCount = 0;
private int callerLoopsCount_coroutine = 0;
private int callerLoopsCount_thread = 0;
void Start () {
myDelegate1 += Elab1;
myDelegate2 += Elab2;
StartCoroutine(CallerCoroutine());
thread = new Thread(new ThreadStart(CallerThread));
thread.Start();
}
void Update()
{
frameCount++;
}
void Elab1()
{
delegate1CallCount++;
}
void Elab2()
{
delegate2CallCount++;
}
IEnumerator CallerCoroutine()
{
while(true)
{
callerLoopsCount_coroutine++;
myDelegate1();
yield return null;
}
}
void CallerThread()
{
while(!threadDone)
{
callerLoopsCount_thread++;
myDelegate2();
}
}
void OnDestroy()
{
Debug.Log("Frame Count: " + frameCount);
Debug.Log("Delegate Call Count (Coroutine): " + delegate1CallCount);
Debug.Log("Delegate Call Count (Thread): " + delegate2CallCount);
Debug.Log("Caller Loops Count (Coroutine): " + callerLoopsCount_coroutine);
Debug.Log("Caller Loops Count (Thread): " + callerLoopsCount_thread);
threadDone = true;
thread.Join();
}
}
If you attach it to a GameObject and let Unity play for some seconds you'll see that the times the delegate was called from a coroutine is equal to the number of executed frames whilst the times the delegate was called from the thread will be way bigger.
I have experience in interfacing softwares similar to Pure Data and I think what you need is a (rather typical) thread with all your delegates there, create a queue of commands for Unity and digest it in Unity's Update.
Not knowing libPD in the specific this might not be the best practice for the case but it is a widely used approach. Basically the producer-consumer pattern.
Basing on the example GUITextScript.cs, libPD only requires you to subscribe to the right delegates. You don't have control on when these are executed, the library has; so if you keep having this issue it's worth submitting a bug report to the developers I guess.

Related

Running a non-blocking, high-performance activity in nativescript/javascript

This question is about running a non-blocking, high-performance activity in nativescript that is needed for the simple task of reading and saving raw audio from the microphone by directly accessing the hardware through the native Android API. I believe I have brought the nativescript framework to the edge of its capabilities, and I need experts' help.
I'm building a WAV audio recorder in Nativescript Android. Native implementation is described here (relevant code below).
In short, this can be done by reading audio steam from an android.media.AudioRecord buffer, and then writing the buffer to a file in a separate thread, as described:
Native Android implementation
startRecording() is triggered by a button press, and starts a new Thread that runs writeAudioDataToFile():
private void startRecording() {
// ... init Recorder
recorder.startRecording();
isRecording = true;
recordingThread = new Thread(new Runnable() {
#Override
public void run() {
writeAudioDataToFile();
}
}, "AudioRecorder Thread");
recordingThread.start();
}
Recording is stopped by setting isRecording to false (stopRecording() is triggered by a button press):
private void stopRecording() {
isRecording = false;
recorder.stop();
recorder.release();
recordingThread = null;
}
Reading and saving buffer is stopped if isRecording = false:
private void writeAudioDataToFile() {
// ... init file and buffer
ByteArrayOutputStream recData = new ByteArrayOutputStream();
DataOutputStream dos = new DataOutputStream(recData);
int read = 0;
while(isRecording) {
read = recorder.read(data, 0, bufferSize);
for(int i = 0; i < bufferReadResult; i++) {
dos.writeShort(buffer[i]);
}
}
}
My Nativescript javascript implementation:
I wrote a nativescript typescript code that does the same as the native Android code above. The problem #1 I faced was that I can't run while(isRecording) because the javascript thread would be busy running inside the while loop, and would never be able to catch button clicks to run stopRecording().
I tried to solve problem #1 by using setInterval for asynchronous execution, like this:
startRecording() is triggered by a button press, and sets a time interval of 10ms that executes writeAudioDataToFile():
startRecording() {
this.audioRecord.startRecording();
this.audioBufferSavingTimer = setInterval(() => this.writeAudioDataToFile(), 10);
}
writeAudioDataToFile() callbacks are queued up every 10ms:
writeAudioDataToFile() {
let bufferReadResult = this.audioRecord.read(
this.buffer,
0,
this.minBufferSize / 4
);
for (let i = 0; i < bufferReadResult; i++) {
dos.writeShort(buffer[i]);
}
}
Recording is stopped by clearing the time interval (stopRecording() is triggered by button press):
stopRecording() {
clearInterval(this.audioBufferSavingTimer);
this.audioRecord.stop();
this.audioRecord.release();
}
Problem #2: While this works well, in many cases it makes the UI freeze for 1-10 seconds (for example after clicking a button to stop recording).
I tried to change the time interval that executes writeAudioDataToFile() from 10ms to 0ms and up to 1000ms (while having a very big buffer), but then the UI freezes were longer and, and I experienced loss in the saved data (buffered data that was not saved to the file).
I tried to offload this operation to a separate Thread by using a nativescript worker thread as described here, where startRecording() and stopRecording() are called by messages sent to the thread like this:
global.onmessage = function(msg) {
if (msg.data === 'startRecording') {
startRecording();
} else if (msg.data === 'stopRecording') {
stopRecording();
}
}
This solved the UI problem, but created problem #3: The recorder stop was not executed on time (i.e. recording stops 10 to 50 seconds after the 'stopRecording' msg.data is received by the worker thread). I tried to use different time intervals in the setInterval inside the worker thread (0ms to 1000ms) but that didn't solve the problem and even made stopRecording() be executed with greater delays.
Does anyone have an idea of how to perform such a non-blocking high-performance recording activity in nativescript/javascript?
Is there a better approach to solve problem #1 (javascript asynchronous execution) that I described above?
Thanks
I would keep the complete Java implementation in actual Java, you can do this by creating a java file in your plugin folder:
platforms/android/java, so maybe something like:
platforms/android/java/org/nativescript/AudioRecord.java
In there you can do everything threaded, so you won't be troubled by the UI being blocked. You can call the Java methods directly from NativeScript for starting and stopping the recording. When you build your project, the Java file will automatically be compiled and included.
You can generate typings from your Java class by grabbing classes.jar from the generated .aar file of your plugin ({plugin_name}.aar) and generate type declarations for it: https://docs.nativescript.org/core-concepts/android-runtime/metadata/generating-typescript-declarations
This way you have all the method/class/type information available in your editor.

How to get smooth animation in SkiaSharp

I am trying to achieve smooth animation using Xamarin SkiaSharp. The core issue is the the time between calling canvasView.InvalidateSurface(); and hitting the mathod OnCanvasViewPaintSurface to do the redraw can vary from 3 to 30 ms which gives a somewhat jerky appearance when you are moving an object across the screen. I have tried to mitigate this by adding a dead loop in the draw code, which helps some but is not a great solution. I do not understand why the time varies so much, and I do not see any way around this. You cannot put a sleep in the draw code. How do games achieve smooth animation? My code follows
async Task DoAnimationLoop()
{
while (DoAnimation)
{
AccumulatedTime = StopWatch1.ElapsedMilliseconds;
await Task.Delay(TimeSpan.FromMilliseconds(5));
if (AccumulatedTime > 50)
{
StopWatch1.Restart();
MoveItems();
SKCanvasView canvasView = Content as SKCanvasView;
TotalBounds = new Size(canvasView.Width,
canvasView.Height);
canvasView.InvalidateSurface();
}
}
}
private void OnCanvasViewPaintSurface(object sender,
SKPaintSurfaceEventArgs e)
{
AccumulatedTime = StopWatch1.ElapsedMilliseconds;
while (AccumulatedTime < 30)
{
AccumulatedTime = StopWatch1.ElapsedMilliseconds;
}
e.Surface.Canvas.Clear();
e.Surface.Canvas.DrawBitmap(Background, 0, 0);
foreach(Item item in AllItems)
{
e.Surface.Canvas.DrawBitmap(item.CurrentBitmap,
item.CurrentPositionX, item.CurrentPositionY);
}
}
For future readers:
From experience, I get the smoothest animations with SkiaSharp by creating SkiaSharp SKCanvasViews that have Bindable properties that can be incremented with Xamarin.Forms.Animate. Animate handles all the timing and sleeping code based on variables you configure it with. As you want a loop, you can set the repeat delegate to return true when calling Animate.Commit( ... repeat: () => true ...)
Here is a method example that animates a progress bar "filling" to 100 percent (without looping), by incrementing the ProgressBar's PercentageFilled Property. Note the timing settings you can configure: Refresh rate = 11ms (equates to "90 fps": 1000ms/90 = 11.11), timeToAnimate is the length of time the animation should take to complete in ms, and you can choose from several easing functions.
private void AnimateProgressBar()
{
double startPercentage = 0; //start at 0 percent
double endPercentage = 1; //fill to 100 percent (Forms.Animate will evenly increment
//between 0 and 1 , and in this case the ProgressBar's OnPaintSurface method knows how to draw based
//on the given decimal i.e. if PercentageFilled is .5 it will draw the bar to
//50 percent of its possible max length)
uint timeToAnimate = 1000;
Xamarin.Forms.Animation animation = new Animation(v => _ProgressBar.PercentageFilled = (float)v, startPercentage, endPercentage, easing: Easing.CubicOut);
animation.Commit(_ProgressBar, "FillPercentage", length: timeToAnimate, finished: (l, c) => animation = null, rate: 11);
}
When the PercentageFilled Property is changed it triggers InvalidateSurface by placing a call to InvalidateSurface() within the OnPropertyChanged method. To do this override OnPropertyChanged like so in your SKCanvasView derived class:
class ProgressBar: SKCanvasView
{
//...
public BindableProperty PercentageFilledProperty =
BindableProperty.Create(nameof(PercentageFilled), typeof(float), typeof(ProgressBar), 0f);
public float PercentageFilled
{
get { return (float)GetValue(PercentageFilledProperty ); }
set { SetValue(PercentageFilledProperty , value); }
}
protected override void OnPropertyChanged(string propertyName = null)
{
base.OnPropertyChanged(propertyName);
InvalidateSurface();
}
protected override void OnPaintSurface(SKPaintSurfaceEventArgs args)
{
//Draws progress bar based on the PercentageFilled filled property
}
//....
}
In the question's code it appears that a lot of items are being moved (MoveItems();) in sequence and this technique of drawing items one by one moving might be the cause of jitter? But MoveItems appears to be the place where you might want to use Forms.Animate that is you could Create a Forms.Animation with MoveItems() as the Callback. Look into the microsoft documentation for "Custom Animations in Xamarin.Forms" for more info on how to animate with callbacks.
Also check out "The basics to create custom Xamarin.Forms controls using SkiaSharp" by Konrad Müller on Medium which contains this helpful paragraph which might be useful to consider if you are making a game:
The basic SkiaSharp.Views.Forms provides two views you can use as a
base for your controls: SKCanvasView and SKGLView. The CanvasView uses
the CPU accelerated backend while the GLView uses OpenGL and therefore
the GPU. You might intuitively think that using the GPU for graphics
operations is always the better choice but in fact, OpenGL has a high
overhead when creating the GL context. The CanvasView simply allocates
the memory it needs and as long as enough CPU power is available, it
can render without problems. In theory, the CanvasView should be
better suited for less demanding renderings and GlView better for
complex renderings but the power of modern smartphones makes these
differences mostly unnoticable. I would recommend to simply stick to
the CanvasView and switch to the GlView if the rendering gets to
complex and you notice performance problems.

How to Poll For Controller Input In UWP App

I'm unsure about the best practice for obtaining and updating input received from a controller monitored using the GamePad class in UWP.
I've seen a couple of examples of people using Dispatch Timers and async loops inside the GamePadAdded event. In Win32 applications, I would have handled input in the WinMain update/message loop, but in UWP apps I don't know of anything similar.
Is there a loop in UWP apps that input should be collected/handled like in Win32 apps? What is the recommended protocol for polling for input from a input device (nominally a Xbox One controller)?
I'm happy to read more about UWP app development but I'm unsure of any guides that reference something like this.
Edit: It would be productive if, instead of downvoting and moving on, you shared thoughts on why this question deserved a downvote.
I've seen a couple of examples of people using Dispatch Timers and async loops inside the GamePadAdded event
This is the right way in UWP app to read Gamepad data. A little suggestion is, move the loop reading part on UI thread if you need to update UI frequently. See the solution in this blog
Is there a loop in UWP apps that input should be collected/handled like in Win32 apps
You may make a wrapper with custom event, see the open source implementation: XBoxGamepad
public class XBoxGamepad
{
private List<Gamepad> _controllers = new List<Gamepad>();
private bool _running = true;
Task backgroundWorkTask;
public event EventHandler<GamepadButtons> OnXBoxGamepadButtonPressA;
//omitted......
public XBoxGamepad()
{
Gamepad.GamepadAdded += Gamepad_GamepadAdded;
Gamepad.GamepadRemoved += Gamepad_GamepadRemoved;
backgroundWorkTask = Task.Run(() => PollGamepad());
}
//omitted......
private void Start()
{
_running = true;
}
public void Stop()
{
_running = false;
}
public async Task PollGamepad()
{
while (true)
{
if (_running)
{
foreach (Gamepad controller in _controllers)
{
if (controller.GetCurrentReading().Buttons == GamepadButtons.A)
{
OnXBoxGamepadButtonPressA(controller, controller.GetCurrentReading().Buttons);
}
//omitted......
}
}
await Task.Delay(50);
}
}
private void Gamepad_GamepadRemoved(object sender, Gamepad e)
{
_controllers.Remove(e);
}
private void Gamepad_GamepadAdded(object sender, Gamepad e)
{
_controllers.Add(e);
}
}

Not receiving messages after sometime

I am using JNA to access User32 functions (I dont think it has got to do with Java here, more of concept problem). In my application, I have a Java process which communicates with the Canon SDK. To dispatch any messages I am using the below function:
private void peekMessage(WinUser.MSG msg) throws InterruptedException {
int hasMessage = lib.GetMessage(msg, null, 0, 0);
if (hasMessage != 0) {
lib.TranslateMessage(msg);
lib.DispatchMessage(msg);
}
Thread.sleep(1);
}
peekMessage is called in a loop and it all works well. Whenever an Image is taken from camera, I get the event and do the rest.
But I have observed, say after about 15 seconds (sometimes never or sometimes just at start) of no activity with camera, taking picture does not give me any download event. Later the whole application becomes unusable as it doesn't get any events from camera.
What can be the reason for this? Please let me know of any other info needed, I can paste the respective code along.
Edit:
Initialization:
Map<String, Integer> options = new HashMap<String, Integer>();
lib = User32.INSTANCE;
hMod = Kernel32.INSTANCE.GetModuleHandle("");
options.put(Library.OPTION_CALLING_CONVENTION, StdCallLibrary.STDCALL_CONVENTION);
this.EDSDK = (EdSdkLibrary) Native.loadLibrary("EDSDK/dll/EDSDK.dll", EdSdkLibrary.class, options);
private void runNow() throws InterruptedException {
while (!Thread.currentThread().isInterrupted()) {
Task task = queue.poll();
if (task != null) {
int taskResult = task.call();
switch (taskResult) {
case (Task.INITIALIZE_STATE):
break;
case (Task.PROCESS_STATE):
break;
case (Task.TERMINATE_STATE): {
//queue.add(new InitializeTask());
Thread.currentThread().interrupt();
break;
}
default:
;
}
}
getOSEvents();
}
}
WinUser.MSG msg = new WinUser.MSG();
private void getOSEvents() throws InterruptedException {
if (isMac) {
receiveEvents();
} else {
peekMessage(msg);
}
}
Above, whenever I get my camera event, it add's it to the queue and in each loop I check the queue to process any Task. One more important information: This is a process running on cmd and has no window. I just need the events from my camera and nothing else.
The code where I register callback functions:
/**
* Adds handlers.
*/
private void addHandlers() {
EdSdkLibrary.EdsVoid context = new EdSdkLibrary.EdsVoid(new Pointer(0));
int result = EDSDK.EdsSetObjectEventHandler(edsCamera, new NativeLong(EdSdkLibrary.kEdsObjectEvent_All), new ObjectEventHandler(), context).intValue();
//above ObjectEventHandler contains a function "apply" which is set as callback function
context = new EdSdkLibrary.EdsVoid(new Pointer(0));
result = EDSDK.EdsSetCameraStateEventHandler(edsCamera, new NativeLong(EdSdkLibrary.kEdsStateEvent_All), new StateEventHandler(), context).intValue();
//above StateEventHandler contains a function "apply" which is set as callback function
context = new EdSdkLibrary.EdsVoid(new Pointer(0));
result = EDSDK.EdsSetPropertyEventHandler(edsCamera, new NativeLong(EdSdkLibrary.kEdsStateEvent_All), new PropertyEventHandler(), context).intValue();
//above PropertyEventHandler contains a function "apply" which is set as callback function
}
You are getting ALL messages from ALL windows that belong to this thread, that includes all mouse moves, paints etc. if you aren't rapidly calling this function your message queue will overflow and cause the behavior you describe.
The sleep you definitely don't want as GetMessage yields if no message is waiting.
So if there exists a normal message pump(s) (i.e GetMessage/DispatchMessage) loop somewhere else for this threads window(s) then you should let that pump do most of the work, perhaps use wMsgFilterMin, wMsgFilterMax to just get the event message you require; or even better in this case use peekmessage with PM_NOREMOVE (then you will need your sleep
call as peekmessage returns immediately).
Alternatively provide the hWnd of the window that generates the event to reduce the workload.
Use spy++ to look into which windows this thread owns and what messages are being produced.
To take this answer further please provide answers to: what else is this thread doing and what windows does it own; also is this message pump the only one or do you call into the SDK API where it may be pumping messages too?
There is an OpenSource project wrapping EDSDK with JNA and it has a version of your code that is probably working better:
https://github.com/kritzikratzi/edsdk4j/blob/master/src/edsdk/api/CanonCamera.java#L436
Unfortunately this is not platform independent and specifically the way things work on windows. I am currently in the process of trying to get a MacOS version of things working at:
https://github.com/WolfgangFahl/edsdk4j

Implementing an asynchronous delay in C++/CX

I am trying to write a function which, given a number of seconds and a callback, runs the callback after the given number of seconds. The callback does not have to be on the same thread. The target language is C++/CX.
I tried using Windows::System::Threading::ThreadPoolTimer, but the result is a memory access exception. The issue appears to be that the callback implementation (in native C++) can't be accessed from the managed thread that the timer is running its callback on.
ref class TimerDoneCallback {
private:
function<void(void)> m_callback;
public:
void EventCallback(ThreadPoolTimer^ timer) {
m_callback(); // <-- memory exception here
}
TimerDoneCallback(function<void(void)> callback) : m_callback(callback) {}
};
void RealTimeDelayCall(const TimeSpan& duration, function<void(void)> callback) {
auto t = ref new TimerDoneCallback(callback);
auto e = ref new TimerElapsedHandler(t, &TimerDoneCallback::EventCallback);
ThreadPoolTimer::CreateTimer(e, duration);
}
void Test() {
RealTimeDelayCall(duration, [](){}); //after a delay, run 'do nothing'
}
I don't want to create a thread and sleep on it, because there may be many concurrent delays.
The TimerDoneCallback instance is not kept alive - delegates in C++/CX take weak references to the target object (to avoid circular references). You can override this behavior by using the extended overload of the delegate constructor:
auto e = ref new TimerElapsedHandler(t, &TimerDoneCallback::EventCallback, CallbackContext::Any, true);
The final bool parameter should be true for strong references, and false for weak references. (False is the default.)
You could also consider using the timer class in PPL agents to make a delayed callback: http://msdn.microsoft.com/en-us/library/hh873170(v=vs.110).aspx to avoid needing to use ThreadPoolTimer.

Resources