Understanding animation loading with Assimp - animation

I have made a container class that basicly contains all the information I need for rendering an animation. I'm using the Assimp library to load the animation. Then assigning the data from the scene->mVertices etc. to my array buffers, what I'm having trouble figuring out is how I'm supposed to get that data for other frames into my buffers!
I know that there is a function called HasAnimations(), and also a aiAnimation **mAnimations. But what I can't find any data in there that is relevant to getting the next set of vertex data.
I have managed to load a series of obj-files with it and draw them in order to comfirm that my class works correctly. But obviously I would prefer or actaully need to use something else when I want to expand to the real deal. As loading 250 frames individually takes a couple of minutes. (Loading a simple animation should be done in about 5 seconds tops, right?)
I'm open to using any kind of file format. But I need to know how to set it up in Blender so that the animations will get exported. As I also seem to fail horribly at for now, as I have little experience with Blender.
I've been searching for tutorials on this library and blender exporting for 2 days now, and found almost nothing useful. I did check out the documentation for Assimp as well, which took me so far, but doesn't explain a thing about how aiAnimation affects the vertices. Or how I can get the other frames of data I need.

Well, I did manage to make it work after endless hours! Sort of... I made a model that is transformed in a loop x+5, y+5, x-5, y-5...
What I ended up doing, well it was the only thing I could think of anyways. Is reading the data from the scene->mAnimations[] and this basicly consists of an array of the keyframes only. So I had to interpolate all the vertices myself, (which is always a funny task to approach!).
Effectivly:
You get the time the keyframe should have interpolated fully.
Then subtract where the object currently is to figure out how much you need to move.
Now you need to figure out how much to move each step, so I took the easiest solution, divided it by how many frames the movement should be splitted over.
Now it was just a matter of updating all my vertices before sending them to the VBO. (This step is probably a little varying depending on your data-setup)
After those steps, I got something that looks like this:
Header:
class AssimpMesh {
private:
struct ShaderProgram {
//Shader data
GLuint program;
string programName;
vector <GLuint> shaders, uniforms;
};
struct MeshData {
//Mesh data
GLuint meshArray;
vector <GLuint> buffers;
vector <string> bufferNames;
//Shader data
ShaderProgram *shader;
vector <aiVector3D> transedVertices;
int totalIndices;
};
struct Frame {
vector <MeshData*> meshes;
};
struct Animation {
string name;
vector <Frame*> frames;
};
//Global shader data
ShaderProgram *globalShader;
//Model data
Assimp::Importer importer;
const aiScene *scene;
//Mesh data
bool initialized, perMeshShading;
vector <Animation*> animations;
int currentFrame, currentAnimation;
Uint32 lastFrameTicks;
Transform *transform;
glm::mat4 projectionView;
aiVector3D lightPosition;
void loadScene(string filePath);
void loadAnimation(Animation *animation, int numFrames);
void initMesh(aiMesh *mesh, MeshData *data);
...
public:
AssimpMesh(string filePath);
~AssimpMesh();
void draw();
...
};
Source:
void AssimpMesh::loadScene(string filePath) {
//Load animation file
scene = importer.ReadFile(filePath.c_str(), aiProcessPreset_TargetRealtime_MaxQuality);
if (scene) {
if (scene->HasAnimations()) {
for (int i = 0; i < scene->mNumAnimations; i++) {
aiAnimation *anime = scene->mAnimations[i];
int framesInAnimation = ceil(anime->mDuration * ANIMATION_IMPORT_FPS);
Animation *animation = new Animation();
animation->name = anime->mName.C_Str();
loadAnimation(animation, framesInAnimation);
animations.push_back(animation);
}
}
else {
Animation *animation = new Animation();
animation->name = "Default";
loadAnimation(animation, 1);
animations.push_back(animation);
}
printf("Done loading '%s'\n", filePath.c_str());
}
else {
//Report error
printf("Assimp error: %s\n", importer.GetErrorString());
}
}
void AssimpMesh::loadAnimation(Animation *animation, int numFrames) {
int nextKeyframe = -1;
int nextKeyframeId = -1;
int transedFrames = 0;
aiVector3D transPos = aiVector3D();
aiVector3D transVec = aiVector3D();
for (int f = 0; f < numFrames; f++) {
Frame *frame = new Frame();
if (f > nextKeyframe && nextKeyframe < numFrames) {
//Get the new keyframe
aiAnimation *anime = scene->mAnimations[animations.size()];
aiNodeAnim *aniNode = anime->mChannels[0];
aiVectorKey key = aniNode->mPositionKeys[++nextKeyframeId];
nextKeyframe = ceil(key.mTime * ANIMATION_IMPORT_FPS);
if (!nextKeyframeId) {
transVec = key.mValue;
transPos = key.mValue;
}
else {
int transFrames = nextKeyframe - (f - 1);
aiVector3D transDir = key.mValue - transPos;
transPos = key.mValue;
transVec = transDir;
transVec /= transFrames;
transedFrames = 0;
}
}
if (scene->HasLights()) {
aiLight *light = scene->mLights[0];
//lightPosition = light->mPosition;
}
//Put data into vertex arrays
transedFrames++;
aiMesh *mesh;
MeshData *data;
for (int i = 0; i < scene->mNumMeshes; i++) {
mesh = scene->mMeshes[i];
data = new MeshData();
if (!i) {
for (int j = 0; j < mesh->mNumVertices; j++) {
if (!f) {
data->transedVertices.push_back(mesh->mVertices[j] + transVec);
}
else {
data->transedVertices.push_back(animation->frames[f-1]->meshes[i]->transedVertices[j] + transVec);
}
}
}
//Assign VBO
initMesh(mesh, data);
//Assign shader
if (perMeshShading) {
initShader(mesh, data);
setUniforms(mesh, data);
}
frame->meshes.push_back(data);
}
animation->frames.push_back(frame);
}
}
void AssimpMesh::initMesh(aiMesh *mesh, MeshData *data) {
//Buffer for temporary storage of new ids
GLuint id;
//Make vertex array
if (!initialized) {
glGenVertexArrays(1, &id);
}
data->meshArray = id;
//Tell OpenGL to use this array
glBindVertexArray(id);
//Assign vertices
if (mesh->HasPositions()) {
//Make buffer
if (!initialized) {
glGenBuffers(1, &id);
}
data->buffers.push_back(id);
data->bufferNames.push_back("Positions");
//Set buffer data
glBindBuffer(GL_ARRAY_BUFFER, id);
if (data->transedVertices.size()) {
glBufferData(GL_ARRAY_BUFFER, sizeof(aiVector3D) * data->transedVertices.size(), &data->transedVertices[0], GL_STATIC_DRAW);
}
else {
glBufferData(GL_ARRAY_BUFFER, sizeof(aiVector3D) * mesh->mNumVertices, &mesh->mVertices[0], GL_STATIC_DRAW);
}
//Set shader attribute data
glEnableVertexAttribArray(VBO_VERTEX);
glVertexAttribPointer(VBO_VERTEX, 3, GL_FLOAT, GL_FALSE, NULL, NULL);
}
unsigned int matId = mesh->mMaterialIndex;
aiMaterial *material = scene->mMaterials[matId];
vector <aiColor3D> colors;
aiColor3D diffuse(0, 0, 0);
material->Get(AI_MATKEY_COLOR_DIFFUSE, diffuse);
for (int i = 0; i < mesh->mNumVertices; i++) {
colors.push_back(diffuse);
}
//Make buffer
if (!initialized) {
glGenBuffers(1, &id);
}
data->buffers.push_back(id);
data->bufferNames.push_back("Colors");
//Set buffer data
glBindBuffer(GL_ARRAY_BUFFER, id);
glBufferData(GL_ARRAY_BUFFER, sizeof(aiColor3D) * mesh->mNumVertices, &colors.front(), GL_STATIC_DRAW);
//Set shader attribute data
glEnableVertexAttribArray(VBO_COLOR);
glVertexAttribPointer(VBO_COLOR, 3, GL_FLOAT, GL_FALSE, NULL, NULL);
//Assign texture coords
if (mesh->HasTextureCoords(0)) {
//Make buffer
if (!initialized) {
glGenBuffers(1, &id);
}
data->buffers.push_back(id);
data->bufferNames.push_back("TextureCoords");
//Set buffer data
glBindBuffer(GL_ARRAY_BUFFER, id);
glBufferData(GL_ARRAY_BUFFER, sizeof(aiVector3D) * mesh->mNumVertices, &mesh->mTextureCoords[0], GL_STATIC_DRAW);
//Set shader attribute data
glEnableVertexAttribArray(VBO_TEXCORD);
glVertexAttribPointer(VBO_TEXCORD, 3, GL_FLOAT, GL_FALSE, NULL, NULL);
}
//Assign colors
if (mesh->HasNormals()) {
//Make buffer
if (!initialized) {
glGenBuffers(1, &id);
}
data->buffers.push_back(id);
data->bufferNames.push_back("Normals");
//Set buffer data
glBindBuffer(GL_ARRAY_BUFFER, id);
glBufferData(GL_ARRAY_BUFFER, sizeof(aiVector3D) * mesh->mNumVertices, &mesh->mNormals[0], GL_STATIC_DRAW);
//Set shader attribute data
glEnableVertexAttribArray(VBO_NORMAL);
glVertexAttribPointer(VBO_NORMAL, 3, GL_FLOAT, GL_FALSE, NULL, NULL);
}
if (mesh->HasFaces()) {
vector <unsigned int> indices;
aiFace face;
for (int i = 0; i < mesh->mNumFaces; i++) {
face = mesh->mFaces[i];
for (int j = 0; j < face.mNumIndices; j++) {
indices.push_back(face.mIndices[j]);
}
}
data->totalIndices = indices.size();
//Make buffer
if (!initialized) {
glGenBuffers(1, &id);
}
data->buffers.push_back(id);
data->bufferNames.push_back("Faces");
//Set buffer data
glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, id);
glBufferData(GL_ELEMENT_ARRAY_BUFFER, sizeof(unsigned int) * indices.size(), &indices.front(), GL_STATIC_DRAW);
}
}
Of course, it doesn't work for everything yet. Actually only translation and the entire model. Apparently it doesn't read name values correctly so I can't which meshes the animation is meant for. But it got me going, maybe someone will find this helpful. =)

Related

Controlling sprite with 2 dimensional array SDL2

I'm currently learning to create my first game ever, and I chose SDL2 to be the game engine. I can make animation for the main character sprite separately. However I figured that it would be ridiculously tedious to control the sprite that way.
So I loaded the sprite sheet into a 2 dimensional array hoping that I would be able to control the sprite with all kind of actions with just that one sprite sheet.
#define HOR_FRAMES 16
#define VER_FRAMES 16
// These are for the images and sprites
Sprite catGif;
SDL_Rect catRect[VER_FRAMES][HOR_FRAMES];
bool load()
{
bool success = true;
// Load cat sprite sheets
if (!catGif.loadFromFile("./assets/pets/fighter_cat_sprite/cat_sprite_base_64.png",
sceneRenderer, 0X00, 0x00, 0x00))
{
printf("%s", SDL_GetError());
success = false;
}
else
{
int initX = 0;
int initY = 0;
for (int ver = 0; ver < VER_FRAMES; ver++)
{
for (int hor = 0; hor < HOR_FRAMES; hor++)
{
catRect[ver][hor].x = initX;
catRect[ver][hor].y = initY;
catRect[ver][hor].w = 64;
catRect[ver][hor].h = 64;
initX += 64;
}
initX = 0;
initY += 64;
}
initX = 0;
initY = 0;
}
return success;
}
Then I created a simple enum to define the actions, which is the rows in the sprite sheet
enum Actions{
IDLE,
WALK,
TOTAL
};
After that, in main(), I add the control logic and set the variables in away that I think appropriately for rendering.
#include "init.h"
#include "Sprite.h"
LTexture background(SCREEN_WIDTH, SCREEN_HEIGHT);
SDL_Rect bgRect;
LTexture foreground(SCREEN_WIDTH, SCREEN_HEIGHT);
SDL_Rect fgRect;
int frame = 0;
int maxFrame = 0;
SDL_RendererFlip flipType;
int main(int argc, char* argv[])
{
init();
load();
int action;
bool quitFlag = false;
SDL_Event event;
while (!quitFlag)
{
action = IDLE;
while (SDL_PollEvent(&event) != 0) // Handle events in queue
{
if (event.type == SDL_QUIT ||
event.key.keysym.sym == SDLK_ESCAPE)
quitFlag = true;
}
const Uint8* states = SDL_GetKeyboardState(NULL);
if (states[SDL_SCANCODE_A])
{
maxFrame = 8;
action = WALK;
flipType = SDL_FLIP_HORIZONTAL;
catGif.moveLeft();
}
else if (states[SDL_SCANCODE_D])
{
maxFrame = 8;
action = WALK;
flipType = SDL_FLIP_NONE;
catGif.moveRight();
}
else
{
maxFrame = 4;
action = IDLE;
}
// Drawing
background.clipRender(0, 0, sceneRenderer, &bgRect);
foreground.clipRender(0, SCREEN_HEIGHT / 2, sceneRenderer, &fgRect);
SDL_Rect* currFrame = &catRect[action][frame / maxFrame];
int x = catGif.xGetter();
int y = catGif.yGetter();
catGif.render(x, y, sceneRenderer, currFrame, NULL, 0, flipType);
// Update
SDL_RenderPresent(sceneRenderer);
frame++;
if (frame / maxFrame == maxFrame)
{
frame = 0;
}
}
clean();
return 0;
}
The sprite played smoothly as I pressed the A/D buttons. However, problems appeared after I released the buttons, the sprite sheet continued to play itself through to the end and then the sprite disappeared, even though the "action" variable has been set and there is no way that it can increment itself, I'm pretty sure about that.
Please help me to understand how that happened, hopefully I can try to fix it, or take another approach. Thank you.
I use code::blocks 16.01 on Fedora linux.
And this is the sprite sheet: cat_sprite_sheet
You must reset frame to 0 when changing the animation state.
{
maxFrame = ...;
frame = 0;
...
}
The bug is there because of == check in your frame increment. If current frame is 40 and you release A/D, maxFrame is set back to 4. frame / maxFrame = 40 / 4 = 10 > 4, so it will never be equal to maxFrame, and frame keeps increasing forever, thus make your sprite to disappear when [frame / maxFrame] goes out of bound. Reset frame to 0 will solve this.

How to make waveform rendering more interesting?

I wrote a waveform renderer that takes an audio file and creates something like this:
The logic is pretty simple. I calculate the number of audio samples required for each pixel, read those samples, average them and draw a column of pixels according to the resulting value.
Typically, I will render a whole song on around 600-800 pixels, so the wave is pretty compressed. Unfortunately this usually results in unappealing visuals as almost the entire song is just rendered at almost the same heights. There is no variation.
Interestingly, if you look at the waveforms on SoundCloud almost none of them are as boring as my results. They all have some variation. What could be the trick here? I don't think they just add random noise.
I don't think SoundCloud is doing anything particularly special. There are plenty of songs I see on their front page that are very flat. It has more to do with the way detail is perceived and what the overall dynamics of the song are like. The main difference is that SoundCloud is drawing absolute value. (The negative side of the image is just a mirror.)
For demonstration, here is a basic white noise plot with straight lines:
Now, typically a fill is used to make the overall outline easier to see. This already does a lot for the appearance:
Larger waveforms ("zoomed out" in particular) typically use a mirror effect because the dynamics become more pronounced:
Bars are another way to visualize and can give an illusion of detail:
A pseudo routine for a typical waveform graphic (average of abs and mirror) might look like this:
for (each pixel in width of image) {
var sum = 0
for (each sample in subset contained within pixel) {
sum = sum + abs(sample)
}
var avg = sum / length of subset
draw line(avg to -avg)
}
This is effectively like compressing the time axis as RMS of the window. (RMS could also be used but they are almost the same.) Now the waveform shows overall dynamics.
That is not too different from what you are already doing, just abs, mirror and fill. For boxes like SoundCloud uses, you would be drawing rectangles.
Just as a bonus, here is an MCVE written in Java to generate a waveform with boxes as described. (Sorry if Java is not your language.) The actual drawing code is near the top. This program also normalizes, i.e., the waveform is "stretched" to the height of the image.
This simple output is the same as the above pseudo routine:
This output with boxes is very similar to SoundCloud:
import javax.swing.*;
import java.awt.*;
import java.awt.event.*;
import java.awt.image.*;
import java.io.*;
import javax.sound.sampled.*;
public class BoxWaveform {
static int boxWidth = 4;
static Dimension size = new Dimension(boxWidth == 1 ? 512 : 513, 97);
static BufferedImage img;
static JPanel view;
// draw the image
static void drawImage(float[] samples) {
Graphics2D g2d = img.createGraphics();
int numSubsets = size.width / boxWidth;
int subsetLength = samples.length / numSubsets;
float[] subsets = new float[numSubsets];
// find average(abs) of each box subset
int s = 0;
for(int i = 0; i < subsets.length; i++) {
double sum = 0;
for(int k = 0; k < subsetLength; k++) {
sum += Math.abs(samples[s++]);
}
subsets[i] = (float)(sum / subsetLength);
}
// find the peak so the waveform can be normalized
// to the height of the image
float normal = 0;
for(float sample : subsets) {
if(sample > normal)
normal = sample;
}
// normalize and scale
normal = 32768.0f / normal;
for(int i = 0; i < subsets.length; i++) {
subsets[i] *= normal;
subsets[i] = (subsets[i] / 32768.0f) * (size.height / 2);
}
g2d.setColor(Color.GRAY);
// convert to image coords and do actual drawing
for(int i = 0; i < subsets.length; i++) {
int sample = (int)subsets[i];
int posY = (size.height / 2) - sample;
int negY = (size.height / 2) + sample;
int x = i * boxWidth;
if(boxWidth == 1) {
g2d.drawLine(x, posY, x, negY);
} else {
g2d.setColor(Color.GRAY);
g2d.fillRect(x + 1, posY + 1, boxWidth - 1, negY - posY - 1);
g2d.setColor(Color.DARK_GRAY);
g2d.drawRect(x, posY, boxWidth, negY - posY);
}
}
g2d.dispose();
view.repaint();
view.requestFocus();
}
// handle most WAV and AIFF files
static void loadImage() {
JFileChooser chooser = new JFileChooser();
int val = chooser.showOpenDialog(null);
if(val != JFileChooser.APPROVE_OPTION) {
return;
}
File file = chooser.getSelectedFile();
float[] samples;
try {
AudioInputStream in = AudioSystem.getAudioInputStream(file);
AudioFormat fmt = in.getFormat();
if(fmt.getEncoding() != AudioFormat.Encoding.PCM_SIGNED) {
throw new UnsupportedAudioFileException("unsigned");
}
boolean big = fmt.isBigEndian();
int chans = fmt.getChannels();
int bits = fmt.getSampleSizeInBits();
int bytes = bits + 7 >> 3;
int frameLength = (int)in.getFrameLength();
int bufferLength = chans * bytes * 1024;
samples = new float[frameLength];
byte[] buf = new byte[bufferLength];
int i = 0;
int bRead;
while((bRead = in.read(buf)) > -1) {
for(int b = 0; b < bRead;) {
double sum = 0;
// (sums to mono if multiple channels)
for(int c = 0; c < chans; c++) {
if(bytes == 1) {
sum += buf[b++] << 8;
} else {
int sample = 0;
// (quantizes to 16-bit)
if(big) {
sample |= (buf[b++] & 0xFF) << 8;
sample |= (buf[b++] & 0xFF);
b += bytes - 2;
} else {
b += bytes - 2;
sample |= (buf[b++] & 0xFF);
sample |= (buf[b++] & 0xFF) << 8;
}
final int sign = 1 << 15;
final int mask = -1 << 16;
if((sample & sign) == sign) {
sample |= mask;
}
sum += sample;
}
}
samples[i++] = (float)(sum / chans);
}
}
} catch(Exception e) {
problem(e);
return;
}
if(img == null) {
img = new BufferedImage(size.width, size.height, BufferedImage.TYPE_INT_ARGB);
}
drawImage(samples);
}
static void problem(Object msg) {
JOptionPane.showMessageDialog(null, String.valueOf(msg));
}
public static void main(String[] args) {
SwingUtilities.invokeLater(new Runnable() {
#Override
public void run() {
JFrame frame = new JFrame("Box Waveform");
JPanel content = new JPanel(new BorderLayout());
frame.setContentPane(content);
JButton load = new JButton("Load");
load.addActionListener(new ActionListener() {
#Override
public void actionPerformed(ActionEvent ae) {
loadImage();
}
});
view = new JPanel() {
#Override
protected void paintComponent(Graphics g) {
super.paintComponent(g);
if(img != null) {
g.drawImage(img, 1, 1, img.getWidth(), img.getHeight(), null);
}
}
};
view.setBackground(Color.WHITE);
view.setPreferredSize(new Dimension(size.width + 2, size.height + 2));
content.add(view, BorderLayout.CENTER);
content.add(load, BorderLayout.SOUTH);
frame.pack();
frame.setResizable(false);
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
frame.setLocationRelativeTo(null);
frame.setVisible(true);
}
});
}
}
Note: for the sake of simplicity, this program loads the entire audio file in to memory. Some JVMs may throw OutOfMemoryError. To correct this, run with increased heap size as described here.

OpenNi Blob Tracking over binary image generated from depthMap

I am trying to generate a binary image from the depthMap()-function in OpenNI, which provides an array of type int. With that image I want to do blob-Tracking.
Problem is that I am not able to generate a clear binary image from the depthMap. In my understanding the depth image generates a bright pixel for everything that is closer to the sensor and the farer away from the sensor the darker they get. So I ask every Pixel in the (one-dimensional) Array if it is over my min and under my max-Threshold to make up a range from that I want the get the data.
Here is my code:
// import library
import SimpleOpenNI.*;
import processing.opengl.*; // opengl
import blobDetection.*; // blobs
// declare SimpleOpenNI object
SimpleOpenNI context;
BlobDetection theBlobDetection;
BlobBall blobBalls;
PrintWriter output;
// threshold for binaryImage
int minThreshold, maxThreshold;
// Size of the kinect Image
int kinectWidth = 640;
int kinectHeight = 480;
//
float globalX, globalY;
// Colors
color bgColor = color(0, 0, 123);
color white = color(255,255,255);
color black = color(0,0,0);
// PImage to hold incoming imagery
int[] distanceArray;
PImage cam, forBlobDetect;
void setup() {
output = createWriter("positions.txt");
// init threshold
minThreshold = 960;
maxThreshold = 2500;
// same as Kinect dimensions
size(kinectWidth, kinectHeight);
background(bgColor);
// initialize SimpleOpenNI object
context = new SimpleOpenNI(this);
if (context.isInit() == false) {
println("Can't init SimpleOpenNI, maybe the camera is not connected!");
exit();
}
else {
// mirror the image to be more intuitive
context.setMirror(true);
context.enableDepth();
// context.enableScene();
distanceArray = context.depthMap();
forBlobDetect = new PImage(width, height);
theBlobDetection = new BlobDetection(forBlobDetect.width, forBlobDetect.height);
theBlobDetection.setThreshold(0.2);
}
}
void draw() {
noStroke();
// update the SimpleOpenNI object
context.update();
// put the image into a PImage
cam = context.depthImage();
// copy the image into the smaller blob image
// forBlobDetect.copy(cam, 0, 0, cam.width, cam.height, 0, 0, forBlobDetect.width, forBlobDetect.height);
// blur the blob image
forBlobDetect.filter(BLUR, 2);
//
int pos = 0;
int currentDepthValue = 0;
distanceArray = context.depthMap();
for(int x = 0; x < cam.width; x++) {
for(int y = 0; y < cam.height; y++) {
pos = y*cam.width+x;
currentDepthValue = distanceArray[pos];
// println(currentDepthValue);
if((currentDepthValue > minThreshold) && (currentDepthValue < maxThreshold)) {
forBlobDetect.pixels[pos] = black;
} else {
forBlobDetect.pixels[pos] = white;
}
}
}
// for(int i=0; i < distanceArray.length; i++) {
// currentDepthValue = distanceArray[i];
// // println(currentDepthValue);
// if(currentDepthValue > minThreshold) /*&& (currentDepthValue < maxThreshold)*/) {
// forBlobDetect.pixels[pos] = white;
// } else {
// forBlobDetect.pixels[pos] = black;
// }
// }
// detect the blobs
theBlobDetection.computeBlobs(forBlobDetect.pixels);
// display the image
image(cam, 0, 0);
image(forBlobDetect, 0, 0, width/2, height/2);
// image(context.sceneImage(), context.depthWidth(), 0);
}
Really stupid mistake by myself because I missunderstood the 11-bit Array.
Thanks to the "Making things see" examples I solved it.
https://github.com/atduskgreg/Making-Things-See-Examples/tree/master/ax02_depth_range_limit

Video Delay/Buffer in Processing 2.0

I'm having a ton of trouble making a simple video delay in processing. I looked around on the internet and I keep finding the same bit of code and I can't get it to work at all. When I first tried it, it did nothing (at all). Here's my modified version (which at least seems to load frames into the buffer), I really have no idea why it doesn't work and I'm getting really tired of pulling out my hair. Please... please, for the love of god, please somebody point out the stupid mistake I'm making here.
And now, without further delay (hah, get it?), the code:
import processing.video.*;
VideoBuffer vb;
Movie myMovie;
Capture cam;
float seconds = 1;
void setup() {
size(320,240, P3D);
frameRate(30);
String[] cameras = Capture.list();
if (cameras.length == 0) {
println("There are no cameras available for capture.");
exit();
} else {
println("Available cameras:");
for (int i = 0; i < cameras.length; i++) {
println(cameras[i]);
}
cam = new Capture(this, cameras[3]);
cam.start();
}
vb = new VideoBuffer(90, width, height);
}
void draw() {
if (cam.available() == true) {
cam.read();
vb.addFrame(cam);
}
image(cam, 0, 0);
image( vb.getFrame(), 150, 0 );
}
class VideoBuffer
{
PImage[] buffer;
int inputFrame = 0;
int outputFrame = 0;
int frameWidth = 0;
int frameHeight = 0;
VideoBuffer( int frames, int vWidth, int vHeight )
{
buffer = new PImage[frames];
for(int i = 0; i < frames; i++)
{
this.buffer[i] = new PImage(vWidth, vHeight);
}
this.inputFrame = 0;
this.outputFrame = 1;
this.frameWidth = vWidth;
this.frameHeight = vHeight;
}
// return the current "playback" frame.
PImage getFrame()
{
return this.buffer[this.outputFrame];
}
// Add a new frame to the buffer.
void addFrame( PImage frame )
{
// copy the new frame into the buffer.
this.buffer[this.inputFrame] = frame;
// advance the input and output indexes
this.inputFrame++;
this.outputFrame++;
println(this.inputFrame + " " + this.outputFrame);
// wrap the values..
if(this.inputFrame >= this.buffer.length)
{
this.inputFrame = 0;
}
if(this.outputFrame >= this.buffer.length)
{
this.outputFrame = 0;
}
}
}
This works in Processing 2.0.1.
import processing.video.*;
Capture cam;
PImage[] buffer;
int w = 640;
int h = 360;
int nFrames = 60;
int iWrite = 0, iRead = 1;
void setup(){
size(w, h);
cam = new Capture(this, w, h);
cam.start();
buffer = new PImage[nFrames];
}
void draw() {
if(cam.available()) {
cam.read();
buffer[iWrite] = cam.get();
if(buffer[iRead] != null){
image(buffer[iRead], 0, 0);
}
iWrite++;
iRead++;
if(iRead >= nFrames-1){
iRead = 0;
}
if(iWrite >= nFrames-1){
iWrite = 0;
}
}
}
There is a problem inside your addFrame-Method. You just store a reference to the PImage object, whose pixels get overwritten all the time. You have to use buffer[inputFrame] = frame.get() instead of buffer[inputFrame] = frame. The get() method returns a copy of the image.

How to efficiently store past depth pixel data from depthMapRealWorld() method?

I can't get around a peculiar problem with SimpleOpenNI for Processing ao I'm asking for your help.
I'd like to store snapshots of pixel depth data (returned by .depthMapRealWorld() method as PVector arrays) on discrete time intervals, then process them further for a presentation. I tried adding them in an ArrayList, but it seems that the depthMapRealWorld() method is returning only a reference to a current depth data, not a real array. I tried in this sequence:
Just getting the data and adding it in an arraylist. On every call of the update() method the whole arraylist contained the same PVector array, even if the array at the zero position was added many iterations away!
Then I made the PVector array, along with its creation time, part of a class. Rewrote the sketch a little, but it didn't help. All of the arrays in the arraylist werw still the same.
Finally, in the constructor of the class, I "manually" copied the xyz coordinates of every vector from the PVector array into a int array. That seemed to solve the problem - the int arrays in the arraylist are now different from each other. But this solution introduced serious performance problems.
The question is: is there a more efficient way of storing these PVector arrays and retaining their value?
code:
import processing.opengl.*;
import SimpleOpenNI.*;
SimpleOpenNI kinect;
float rotation = 0;
int time = 0;
ArrayList dissolver;
ArrayList<Integer> timer;
int pSize = 10;
Past past;
void setup() {
dissolver = new ArrayList();
timer = new ArrayList();
size(1024, 768, OPENGL);
kinect = new SimpleOpenNI(this);
kinect.enableDepth();
translate(width/2, height/2, -100);
rotateX(radians(180));
stroke(255);
}
void draw() {
background(0);
translate(width/2, height/2, 500);
rotateX(radians(180));
kinect.update();
stroke (255, 255, 255);
past = new Past (kinect.depthMapRealWorld(), time);
if (dissolver.size() == pSize) { //remove the oldest arraylist element if when list gets full
dissolver.remove(0); //
}
if (time % 20 == 0) {
dissolver.add (past);
Past p1 = (Past) dissolver.get (0);
float [][] o2 = p1.getVector();
println ("x coord of a random point at arraylist position 0: " + o2[50000][0]); //for testing
}
if (dissolver.size() == pSize-1) {
//dissolve ();
}
time ++;
}
void dissolve () { //from the previous nonworking version; ignore
for (int offset = 0; offset < pSize-1; offset ++) {
PVector[] offPoints = (PVector[]) dissolver.get (offset);
int offTime = timer.get(offset);
for (int i = 0; i < offPoints.length; i+=10) {
int col = (time-offTime)*2; //why??
stroke (255, 0, col);
PVector currentPoint = offPoints[i];
if (currentPoint.z <1500) {
point(currentPoint.x, currentPoint.y, currentPoint.z); // - 2*(time-offTime) + random(0, 100)
}
}
}
}
class Past {
private PVector [] depth; //should contain this, not int
private float [][] depth1;
private int time;
Past (PVector [] now, int t) {
//should be like this: depth = now;
//clumsy and performancewise catastrophic solution below
depth1 = new float [now.length][3];
for (int i = 0; i< now.length; i+=10) {
PVector temp = now[i];
depth1 [i][0] = temp.x;
depth1 [i][1] = temp.y;
depth1 [i][2] = temp.z;
}
//arrayCopy(now, depth); this didn't work either
time = t;
}
float [][] getVector () {
return depth1;
}
int getTime () {
return time;
}
}
If I understood correctly, you want to store the 3D positions(ArrayList of PVectors) for each frame, right ?
If so, you should be able to simply store PVectors and reference them later.
Here's a basic sketch to illustrate this:
import processing.opengl.*;
import SimpleOpenNI.*;
SimpleOpenNI kinect;
ArrayList<ArrayList<PVector>> frames = new ArrayList<ArrayList<PVector>>();
ArrayList<PVector> frame;
boolean isRecording = true;
boolean isRecFrame;
void setup() {
size(1024, 768, OPENGL);
kinect = new SimpleOpenNI(this);
kinect.enableDepth();
stroke(255);
}
void draw() {
background(0);
translate(width/2, height/2, 500);
rotateX(PI);
translate(0,0,-1000);
kinect.update();
if(isRecording){
isRecFrame = (frameCount % 20 == 0);//record every 20 frames
int[] depthMap = kinect.depthMap();
int steps = 5; // to speed up the drawing, draw every N point
int index;
PVector realWorldPoint;
if(isRecFrame) frame = new ArrayList<PVector>();
for(int y=0;y < kinect.depthHeight();y+=steps)
{
for(int x=0;x < kinect.depthWidth();x+=steps)
{
index = x + y * kinect.depthWidth();
if(depthMap[index] > 0)
{
realWorldPoint = kinect.depthMapRealWorld()[index];
point(realWorldPoint.x,realWorldPoint.y,realWorldPoint.z);
if(isRecFrame) frame.add(realWorldPoint.get());
}
}
}
if(isRecFrame) frames.add(frame);
}else{//playback
ArrayList<PVector> currentFrame = frames.get(frameCount%frames.size());//playback is faster than recording now for testing purposes - add a decent frame counter here at some point
for(PVector p : currentFrame) point(p.x,p.y,p.z);
}
}
void keyPressed(){
if(key == ' ') isRecording = !isRecording;
}
Use the SPACE key to toggle between recording and playback.
The main thing to note is I'm storing a copy of the real world position for each depth pixel (frame.add(realWorldPoint.get());). Another thing to keep in mind is that currently you're storing these coordinates in memory which at some point will fill. If you only store a limited number of frames that should be fine, if not you might want to save to the points to disk. This way you can reuse recordings with other sketches. A basic way would be to sore them in a csv file:
void saveCSV(ArrayList<PVector> pts){
String csv = "x,y,z\n";
for(PVector p : pts) csv += p.x + "," + p.y + "," + p.z + "\n";
saveStrings("frame_"+frameCount+".csv",csv.split("\n"));
}
Another would be to use a more suitable format for point clouds, like PLY.
Saving an ASCII PLY is fairly straight forward:
void savePLY(ArrayList<PVector> pts){
String ply = "ply\n";
ply += "format ascii 1.0\n";
ply += "element vertex " + pts.size() + "\n";
ply += "property float x\n";
ply += "property float y\n";
ply += "property float z\n";
ply += "end_header\n";
for(PVector p : pts)ply += p.x + " " + p.y + " " + p.z + "\n";
saveStrings("frame_"+frameCount+".ply",ply.split("\n"));
}
You can later open/explore/process these files with tools like MeshLab.

Resources