I try to perform LipSync on a ReadyPlayer.me avatar in ThreeJS. The puppet already have Viseme Face Rigged Morph Target. I'm usings Microsoft Speech SDK to retrieve the audio and the Viseme values (timecode / ID) and Map it correctly.
Demo (the 1st part) : https://www.youtube.com/watch?v=vLbQ2arXzRk
Here is my algorithm / workflow:
Start playing audio
Reset N-1 MorphTargetInfluence to 0
Update MorphTargetInfluence head.morphTargetInfluences[ viseme.id ] = 0.7
Wait until next offset setTimeout(...)
On the AnimationLoop I call renderer.render( scene, camera );
Questions & Issues
The morphing is not smooth, how can I set smooth animation from 0 to 0.7 ?
Should I reset previous morphTarget to 0 ? (I assume yes)
In this Forum they use TWEEN but I don't think it is efficient to create an object every 50ms ? And how long should last the animation may be under 100 or 200ms it's not relevant ?
Thanks, I'm new to Three.js
The morphing is not smooth, how can I set smooth animation from 0 to 0.7 ?
It depends on your use case. It is possible to use an animation library like GSAP or Tween.js to animate morph target influences from one value to another. However, it's potentially more performant to setup an animation clip from sequence of morph targets and then use the standard animation system of three.js to playback the clip. The clip itself can be generated via the static factory method AnimationClip.CreateFromMorphTargetSequence().
Should I reset previous morphTarget to 0 ?
A single set of morph targets represents the geometry in a specific state. You normally want to set the morphTargetInfluences entry for all morph targets to 0 which should not affect the 3D object. That also means you want to animate the "previous" morph target in your list from its current influence value back to 0.
In this Forum they use TWEEN but I don't think it is efficient to create an object every 50ms ? And how long should last the animation may be under 100 or 200ms it's not relevant ?
See my first answer. It could be a performance issue if create too many objects per frame. You potentially need no third-party animation library for your use case if the built-in animation system works for you.
Related
I use this -
if (timeSinceLastClick > 60)
{
Time.timeScale = 0;
}
But that doesn't freeze either the ar camera/ar tracking or audio. Besides the animation, everything else seems to keep running.
I'd like to pause the app completely so the users can save their batteries and avoid heating
If you want to save battery/avoid heating I'd recommend either cull as much as you can or lower the target framerate.
Settings TimeScale to 0 will affect Rigidbodies and all scripts that multiply their movements by Time.deltaTime (which is highly recommended to achieve framerate-independency). It still computes every frame, as often as possible. So see above.
If you have occlusion culling baked, you could move the camera to an area that doesn't point at any renderable objects. A dynamic cube in front of the cam will probably not work as it's not baked into Occlusion Culling. But you could disable the main Camera and render the pause-menu on a second camera that is UI only for example.
I have a single gltf file, exported from Blender with 6 anims and 20 morph targets. When that's the only skinned gltf object in the scene, everything plays nicely - I can switch between bone anims (run, walk, idle, etc), and get all morph anims (for facial expressions) cycling on a timer, or triggered by events. Yay.
The problem is when I introduce a second skinned object, such as an NPC. At that point lots of weirdness starts to happen.
For example, when morph targets cycle expressions on/off the player object, the NPC model standing nearby scales down and disappears on the off cycle, then scales back up during the on cycle. Another example, at init time the NPC object might randomly turn into an instance of another loaded object (a tree or a building), or occasionally a mini version of some random object, at 10% normal scale, and then start rapidly bouncing around in unpredictable and inconsistent ways. I have no idea what's going on.
I thought it might have something to do with loading multiple mixers, but then that's what the docs state should be done - "When multiple objects in the scene are animated independently, one AnimationMixer may be used for each object." Unless I'm reading that wrong?
I'm using:
npcMixer = new THREE.AnimationMixer(npc);
virtually the same as what I do for the player:
playerMixer = new THREE.AnimationMixer(player);
Is this a bad/mistaken approach?
Perhaps worth noting: I had FBX versions of the player and NPC working just fine together when exported and accessed as individual files. Then I spent a lot of time converting to GLTF since it's faster and lets me wrap all the actions up in a single file, which the FBX exporter does not seem to support (If I'm wrong about FBX being able to export multiple actions in a single file for playback in the three.js context please let me know!).
Three.js r98
Blender 2.79
Thanks for any advice.
I currently have a big impact on the performances of my ThreeJS app when I render the very first frame. It causes the Edge and IE 11 browsers to freeze for 5 seconds with a pop-up indicating "This window does not respond", which may scare my users.
Using the Performance profiler of Chrome, it seems the problem come from several ThreeJS functions you can clearly identify in the screenshot below.
WebGLUniforms.upload : 425ms (50.7% frame rendering time)
WebGLProgram.constructor : 327ms (38.9% frame rendering time)
How can I minimize the duration of the functions call ?
Can I create the program over multiple frames? Or upload the uniforms?
Does the number of materials on my 3D models impact these functions ?
I've tried to hide all the models in the scene and show them one at a time, it seems to prevent the freeze, but each model takes 500ms to show, which is not perfect for user experience. Maybe, it's the only way to go.
Thanks for your time
EDIT : The number of materials or their nature (WebGLStandardMaterial?) seems to affect the performances
Put add a few objects to the scene per frame over time. Three.js inits WebGL resources on first use so if there's 100 objects on your scene all 100 objects get initialized the first time you call renderer.render.
So, just put N objects to the scene, call renderer.render, then next frame add N more to the scene, etc until all the objects have been added.
It's probably most important to do it by material and geometry. In other words if you have 10 different materials and 10 different geometries and you're rendering 100 different models (a model takes one material and one geometry), then you want to make sure the first N models you add don't use all the materials and all the models because those are the things that need to get initialized.
Post processing passes also need initialization so if you're using any of those maybe the first frame just init those, then start adding objects.
I'm working on a phisics based game and I have a question
Is it possible to make almost real phisics inside Unity engine??
Because when I put a rolling sphere at top of a ramp and let it roll... it moves very slow... and when I do it in real life... obviously the ball rolls with certain speed depending on the angle of the ramp... less angle = less speed... more angle = more speed
I tried:
Removing drag
Removing angular drag
Changing the values in interpolate and collision detection
Changing the mass value
Any help will be apreciated
Thanks in advance
Be sure to check the scale of your objects, it is very easy to set up a scene at the wrong scale because there's no easy frame of reference!
Unity's units map to 1 meter, so if your objects are extremely large, they will appear to move more slowly because the physics engine is set up to respect this scale by default.
A marble should have a diameter of roughly 0.025 units, and a person should be around 1.7 units tall!
I think your error is the scale objects. I found this link that may can help you: http://gamedevelopment.tutsplus.com/articles/how-to-fix-common-physics-problems-in-your-game--cms-21418
I was wondering how I should repeat a texture in Cocos2d 3. I have a background and I want to “tile” it across the screen. I have found this which is using ccTexParams with GL_REPEAT but those have been made private in version 3 of cocos.
I have found another solution which can be found here it creates a loop and positions a new child node based on the size of the texture and the size you want. But is that performant? Because when you have a 1px wide background texture and want to repeat that on a iPad retina, you have more than 2000 child nodes.
What is the best way to repeat a texture.
Well since there was no method for repeating without having a POT texture I made something of my own which takes care of it.
Might be useful for someone who has this same question. The code can be found here on Github.
CCTexture2D class has setTexParameters: method to set repeat mode. Also pay attention that the texture must have power-of-two width and height, otherwise repeating mode will be disabled.