Tensorflow.js: Resize image to specific byte size - image

For the prediction I need an image of the shape [null,7,7,256].
const image = tf.reshape(tf.fromPixels(loadedImage).resizeBilinear([?,?]), [null, 7, 7, 256]);
But I don't know how to resize the image to be exactly 7*7*256 big.
Error: Size(37632) must match the product of shape ,7,7,256
Edit: The code for the prediction is:
tf.loadModel(tf.io.browserFiles([uploadJSONInput.files[0], uploadWeightsInput.files[0]])).then(model => {
console.log("model loaded");
return model;
}).then(pretrainedModel => {
return loadImage2('http://localhost/myimg.jpeg', (src) => {
const loadedImage = document.createElement("img");
loadedImage.src = src;
loadedImage.width = "275"
loadedImage.height = "183"
console.log("image loaded");
const image = tf.fromPixels(loadedImage)
resized = tf.image.resizeBilinear(image, [7, 7])
const padded = resized.pad([[0, 0], [0, 0], [126, 127]])
const pretrainedModelPrediction = pretrainedModel.predict(padded);
const modelPrediction = model.predict(pretrainedModelPrediction);
const prediction = modelPrediction.as1D().argMax().dataSync()[0];
console.log(prediction);
});
})
Error:
Error: Error when checking : expected flatten_Flatten1_input to have 4 dimension(s), but got array with shape [7,7,256]

ResizeBilinear will resize the height and the width of the image, meaning that it does not affect the number of channel which is the last dimension of the shape of an image.
If your image has 256 as it last channel, then the following will work
tf.fromPixels(loadedImage).resizeBilinear([7,7])
Reshaping a tensor will only work if both sizes matches.
const image = tf.ones([183, 275, 3 ])
resized = tf.image.resizeBilinear(image, [7, 7])
console.log(resized.pad([[0, 0], [0, 0], [126, 127]]).shape);
An image is generally of shape [h, w, 3].
resize = tf.fromPixels(loadedImage).resizeBilinear([7,7]) // [7, 7, 3]
And then use tf.pad for the last dimension
const image = tf.ones([183, 275, 3 ])
resized = tf.image.resizeBilinear(image, [7, 7])
console.log(resized.pad([[0, 0], [0, 0], [126, 127]]).shape);// [7,7,256]
// reshape the tensor to be a 4d
resized.reshape([1,7,7,256])

Here's how to do it with a Uint8Array
const canvas: any = document.getElementById('canvas')
const context = canvas.getContext('2d')
const imageData: ImageData = context.getImageData(0, 0, canvas.width, canvas.height)
const uint8array = new Uint8Array(imageData.data.buffer)
const rgbaTens3d = tf.tensor3d(uint8array, [canvas.height, canvas.width, 4])
const rgbTens3d= tf.slice3d(rgbaTens3d, [0, 0, 0], [-1, -1, 3]) // strip alpha channel
const smallImg = tf.image.resizeBilinear(rgbTens3d, [192, 192]); // 192,192 is dictated by my model

Related

Find the Z coordinate of point within plane

From the 3 black points I found the plane
const { Vector3, Plane } = require('three')
const points = [new Vector3(0, 0, 0), new Vector3(1, 0, 1), new Vector3(1, 2, 0)]
const plane = new Plane().setFromCoplanarPoints(...points)
But how do I get the Z coordinate of the fourth red point (example: (0.75, 0.75, z)) that lies in the plane?
This doesn't seem to work:
const targetPoint = new Vector3()
plane.projectPoint(new Vector3(0.75, 0.75, 0), targetPoint)
/*
Vector3 {
x: 0.5833333333333334,
y: 0.8333333333333334,
z: 0.16666666666666666
}
*/
An answer with TurfJS would be also perfectly OK
Just for others to know, I solved with TurfJS using its method planepoint.
The method polygon has as 1st parameter an array of Linear Rings, and a Linear Ring must have the same first and last points, thus a triangle has 4 points. a, b, c represent the ordered heights.
const turf = require('#turf/turf')
const point = turf.point([0.75, 0.75])
const triangle = turf.polygon([[
[0, 0], [1, 0], [1, 2], [0, 0]
]], {
a: 0,
b: 1,
c: 0
})
const zValue = turf.planepoint(point, triangle) // 0.375

threejs applyMatrix4 appears to do nothing

The method applyMatrix4 seems like it does nothing...
Why can I not apply this transformation matrix to my vector?
const vec = new THREE.Vector3(1,1,1)
const geometry = new THREE.BoxGeometry(1,1,1)
const material = new THREE.MeshBasicMaterial({ color: 0xff0000 })
const mesh = new THREE.Mesh(geometry, material)
mesh.rotateX(Math.PI)
const rotatedVec = vec.applyMatrix4(mesh.matrix)
console.log(rotatedVec)
Expectation (taking the cross product):
{x: 1, y: -1, z: -1}
Reality (the vector is unchanged)
{x: 1, y: 1, z: 1}
My mesh's matrix has changed - it is not the identity matrix.
[
[1, 0, 0, 0],
[0, -1, 0, 0],
[0, 0, -1, 0],
[0, 0, 0, 1],
]
Object3D.rotateX() only affects the object's quaternion property. It does not update its local matrix. If you say your matrix has changed, I assume you have checked it at a later point when other engine logic triggers a recalculation.
You can solve this issue by adding mesh.updateMatrix(); after you have called Object3D.rotateX().
Or even better use Vector3.applyQuaternion(). In this way, you don't have to recompute the matrix because you don't need it anyway.
const rotatedVec = vec.applyQuaternion(mesh.quaternion)

expected first layer to have x dimensions but got an array with shape y

(I am just starting tensorflow.js on node)
I have been searching the web up and down for an answer.
The confusion
I have image data from image1 = tf.fromPixels(img) and I tried inputting it along with other image data to xs = tf.tensor([image1, image2]). The confusion is no matter how I input a bunch of images into xs for model.fit, the program outputs errors described below.
What I already tried
When I run the program I get the error Error: Error when checking input: expected conv2d_Conv2D1_input to have 4 dimension(s). but got array with shape 4,1
I know for a fact that I am not inputting the xs correctly. I read some articles online relating to how you need to input the array in a fashion like tf.tensor([[0.2, 0.1], [0.2, 0.4]]); and some batching of images of some sort. I looked at articles showing that for images, you need another set of layers:
model.add(tf.layers.conv2d({
inputShape: [scaleHeight, scaleWidth, 3],
kernelSize: 5,
filters: 8,
strides: 1,
activation: 'relu',
kernelInitializer: 'VarianceScaling'
}));
model.add(tf.layers.maxPooling2d({
poolSize: [2, 2],
strides: [2, 2]
}));
model.add(tf.layers.conv2d({
kernelSize: 5,
filters: 16,
strides: 1,
activation: 'relu',
kernelInitializer: 'VarianceScaling'
}));
model.add(tf.layers.maxPooling2d({
poolSize: [2, 2],
strides: [2, 2]
}));
model.add(tf.layers.dense({ // Output
units: 2,
kernelInitializer: 'VarianceScaling',
activation: 'softmax'
}));
model.compile({loss: 'categoricalCrossentropy', optimizer: tf.train.sgd(0.1), metrics: ['accuracy']});
Well I tried inputting that in, tried converting them into typedarray format, tried a lot of things. I am pretty lost on coming up with a proper xs variable for multiple images turned to tensors by tf.fromPixels(canvas) for model.fit(xs, ys, {epochs: 100, options....});
Code:
var tf = require('#tensorflow/tfjs');
var cv = require('canvas');
var {Image, createCanvas, ImageData} = cv;
tf.disableDeprecationWarnings();
var scaleWidth = 16;
var scaleHeight = 16;
function getImage(path){
var img = new Image();
return new Promise(function(resolve, reject){
img.onload = function(){
var element = createCanvas(scaleWidth, scaleHeight);
var ctx = element.getContext('2d');
ctx.drawImage(img, 0, 0);
ctx.scale(scaleWidth/img.width, scaleHeight/img.height);
//resolve(Array.from(tf.fromPixels(element).flatten().dataSync()));
resolve(tf.fromPixels(element));
};
img.src = path;
});
}
var log = function(input){console.log(input)};
const model = tf.sequential();
model.add(tf.layers.conv2d({
inputShape: [scaleHeight, scaleWidth, 3],
kernelSize: 5,
filters: 8,
strides: 1,
activation: 'relu',
kernelInitializer: 'VarianceScaling'
}));
model.add(tf.layers.maxPooling2d({
poolSize: [2, 2],
strides: [2, 2]
}));
model.add(tf.layers.conv2d({
kernelSize: 5,
filters: 16,
strides: 1,
activation: 'relu',
kernelInitializer: 'VarianceScaling'
}));
model.add(tf.layers.maxPooling2d({
poolSize: [2, 2],
strides: [2, 2]
}));
model.add(tf.layers.dense({ // Output
units: 2,
kernelInitializer: 'VarianceScaling',
activation: 'softmax'
}));
model.compile({loss: 'categoricalCrossentropy', optimizer: tf.train.sgd(0.1), metrics: ['accuracy']});
(async function(){
var cats = [], bland = [];
cats[0] = await getImage('cats/0.jpeg');
cats[1] = await getImage('cats/1.jpeg');
bland[0] = await getImage('bland/0.png');
bland[1] = await getImage('bland/1.png');
var testCats = await getImage('c.jpeg');
var testBland = await getImage('b.jpeg');
var xs = tf.tensor([cats[0], cats[1], bland[0], bland[1]]); // confusion occurs here
for(var c = 0; c < 10; c++){
var result = await model.fit(xs, tf.tensor([[0, 1], [0, 1], [1, 0], [1, 0]]), {epochs: 100});
console.log(result.history.loss[0]);
}
})();
And after I ran it, I expected to at least log the loss of the model but it thrown this error: Error: Error when checking input: expected conv2d_Conv2D1_input to have 4 dimension(s). but got array with shape 4,1
Looking at your code the data passed in to your model doesn't have the same shape as the model first layer inputShape.
How to go about solving the issue ?
check the data.shape.
console.log(xs.shape) // it will return (4,1)
compare with the inputShape
The data shape should one dimension higher than the inputShape (one more dimension for batchsize)
// Does `xs.inputShape.slice(1) ===[Scaleheight, scaleWidth,3]` ?
shape1 = xs.inputShape.slice(1)
shape2 = [Scaleheight, scaleWidth,3]
const same = (shape1.length == shape2.length) && shape1.every(function(e, i) {
return e === shape2[i];
});
If they are not equal, there are two ways to get the problem resolved
Reshaping the data if possible,using tf.reshape, tf.slice, tf.expandDims(), ...
Or simply changing the inputShape to be equal to our data shape
In your case here there is a clear mismatch between the inputShape and the data shape.
First thing first, the way you create your xs is wrong. Actually, xs has the shape (4, 1) with NaN values. It is as if you created a tf.tensor with an array of tensors. You can create the xs this way:
xs = tf.concat([...cats, ...blands], 0)
However it is not sure if this will solve completely the issue. You need to iterate over the step outlined above,ie, check the shape of xs, compare with the inputShape and so on ...

Custom coordinates and axis range on leaflet.js

I have a raster image with dimensions (in pixels) 16384-by-12288 which is successfully rendered in leaflet. I am using a my own CRS and I am placing point (0,0) at the bottomleft corner of the image point (16384, 12288) at its topright using the option: transformation: new L.Transformation(1 / 64, 0, -1 / 64, 256).
The axes of my image, however, have range x:[6150, 1370] and y:[12987, 18457]
How can I tell leaflet to use my range as a system of coordinates please? Hence a marker at location (6150, 12987) will correspond and show up at the bottomleft corner: (0,0). I have done this manually using the function below:
var grid = {x0: 6150, // range of plot in Matlab
x1: 13751,
y0: 12987,
y1: 18457};
var img = [16384,
12288];
function project(p, img, grid) {
var x = p[0],
y = p[1];
xx = img[0] / (grid.x1 - grid.x0) * (x - grid.x0);
yy = img[1] / (grid.y1 - grid.y0) * (y - grid.y0);
return [xx, yy]
}
I was wondering however that there must a more streamlined and better way to do this. My code is:
var yx = L.latLng;
var xy = function(x, y) {
if (L.Util.isArray(x)) { // When doing xy([x, y]);
return yx(x[1], x[0]);
}
return yx(y, x); // When doing xy(x, y);
};
var img = [
16384, // original width of image
12288 // original height of image
];
L.CRS.MySimple = L.extend({}, L.CRS.Simple, {
transformation: new L.Transformation(1 / 64, 0, -1 / 64, 256),
});
var bounds = L.latLngBounds([
xy(0, 0),
xy(img)
]);
var map = L.map('map', {
crs: L.CRS.MySimple,
maxBounds: bounds.pad(.5),
}).setView([img[1] / 2, img[0] / 2], 0);
L.tileLayer('myImage/{z}/{x}/{y}.png', {
bounds: bounds,
minZoom: 1,
maxZoom: 6
}).addTo(map);
L.marker([0, 0]).addTo(map).bindPopup("Zero");
L.marker([img[1] / 2, img[0] / 2]).addTo(map).bindPopup("[img[1] / 2, img[0] / 2]");
L.marker([img[1], img[0]]).addTo(map).bindPopup("img");
I think I did some progress. In case someone faces something similar in the future, here is my code: (comments more than welcome)
var yx = L.latLng;
var xy = function(x, y) {
if (L.Util.isArray(x)) { // When doing xy([x, y]);
return yx(x[1], x[0]);
}
return yx(y, x); // When doing xy(x, y);
};
var img = [
16384, // original width of image
12288 // original height of image
];
var mapSW = [0, 16384],
mapNE = [12288, 0];
var roi = { //range of interest
x0: 6150,
x1: 13751,
y0: 12987,
y1: 18457
};
a = img[0] / (roi.x1 - roi.x0)
b = -img[0] / (roi.x1 - roi.x0) * roi.x0
c = img[1] / (roi.y1 - roi.y0)
d = -img[1] / (roi.y1 - roi.y0) * roi.y0
// This transformation maps a point in pixel dimensions to our user defined roi
var t = new L.Transformation(a, b, c, d);
// The transformation in this CRS maps the the bottom right corner to (0,0) and the topleft to (256, 256)
L.CRS.MySimple = L.extend({}, L.CRS.Simple, {
transformation: new L.Transformation(1 / 64, 0, -1 / 64, 256),
});
var bounds = L.latLngBounds([
xy(0, 0),
xy(img)
]);
var map = L.map('map', {
crs: L.CRS.MySimple,
maxBounds: bounds.pad(.5),
}).setView([img[1] / 2, img[0] / 2], 0);
L.tileLayer('map/{z}/{x}/{y}.png', {
bounds: bounds,
minZoom: 1,
maxZoom: 6,
}).addTo(map);
// map.setMaxBounds(new L.LatLngBounds(
// map.unproject(mapSW, map.getMaxZoom()),
// map.unproject(mapNE, map.getMaxZoom()),
// ));
L.marker([0, 0]).addTo(map).bindPopup("Zero");
L.marker([img[1] / 2, img[0] / 2]).addTo(map).bindPopup("[img[1] / 2, img[0] / 2]");
L.marker([img[1], img[0]]).addTo(map).bindPopup("img");
var marker = L.marker(xy([10000, 0]), {
draggable: true
}).addTo(map);
marker.bindPopup("");
marker.on("dragend", function(e) {
m = marker.getLatLng();
proj = map.project(m, map.getMaxZoom());
marker.getPopup().setContent('Clicked ' +m.toString() + '<br />' +
'Pixels ' + proj.toString())
.openOn(map);
})
L.control.scale({
imperial: false
}).addTo(map);
var popup = L.popup();
function onMapClick(e) {
popup
.setLatLng(e.latlng)
.setContent("You clicked the map at " + e.latlng.toString())
.openOn(map);
}
map.on('click', onMapClick);
var p = t.transform(L.point(roi.x1, roi.y1));
L.circleMarker(xy([p.x, p.y])).addTo(map);
p = t.transform(L.point(10000, 12987));
L.circleMarker(xy([p.x, p.y])).addTo(map);
p = t.transform(L.point(13000, 12987));
L.circleMarker(xy([p.x, p.y])).addTo(map);
p = t.transform(L.point(6150, 18000));
L.circleMarker(xy([p.x, p.y])).addTo(map);

Correct transformation order for scene graph

I am working on a quick WebGL Engine with a scene graph to quickly prototype my game idea on reddit (https://www.reddit.com/r/gameideas/comments/3dsy8m/revolt/). Now, after I have got some basic rendering done, I can't figure out the correct order, well the one that looks right to most people, that I am supposed to use in order to transform the nodes in the scene graph.
It's hard to explain what is happening but I hope you get a understanding that it just isn't rotating the way that most would expect it to happen in any other engine.
Here is a simplified version of what I am currently doing.
Mat4 = glMatrix 0.9.5
Utils = Custom Utilitys
Node(Render):
#param {parentMatrix}
// Create Local Matrix
self.lMatrix = mat4.create();
mat4.identity(self.lMatrix);
mat4.translate(self.lMatrix, self.position);
mat4.rotate(self.lMatrix, self.rotation[0], [1, 0, 0]);
mat4.rotate(self.lMatrix, self.rotation[1], [0, 1, 0]);
mat4.rotate(self.lMatrix, self.rotation[2], [0, 0, 1]);
var wMatrix = mat4.create();
mat4.identity(wMatrix);
if(parentMatrix){
mat4.multiply(self.lMatrix, parentMatrix, wMatrix);
}
else{
mat4.set(self.lMatrix, wMatrix);
}
// Render
var children = this.children,
numChildren = children.length,
child;
for(var i = 0; i < numChildren; i++){
child = children[i];
child.render(wMatrix);
}
Entity(Render):
#param {parentMatrix}
// Set Transformation matrix
var tMatrix = mat4.create();
mat4.identity(tMatrix);
mat4.translate(tMatrix, self.position);
mat4.rotate(tMatrix, self.rotation[0], [1, 0, 0]);
mat4.rotate(tMatrix, self.rotation[1], [0, 1, 0]);
mat4.rotate(tMatrix, self.rotation[2], [0, 0, 1]);
mat4.scale(tMatrix, self.scale || [1, 1, 1]);
var wMatrix = mat4.create();
mat4.identity(wMatrix);
mat4.multiply(tMatrix, parentMatrix, wMatrix);
Utils.loadTMatrix(wMatrix);
this.texture.bind();
this.mesh.render();
The usual order is SRT, or scale, rotate then translate.
Also I am not sure if you can just do
mat4.rotate(tMatrix, self.rotation[0], [1, 0, 0]);
mat4.rotate(tMatrix, self.rotation[1], [0, 1, 0]);
mat4.rotate(tMatrix, self.rotation[2], [0, 0, 1]);
with euler angles and get the correct result orientation. I dont use euler angles so I dont fully grasp the details. Somebody please correct me if Im wrong. See this page for conversions between euler angle and rotation matrix: http://www.euclideanspace.com/maths/geometry/rotations/conversions/eulerToMatrix/.
I didn't find the original way that I was hoping for because I was previously caching matrices, and was hoping to continue doing it, but now I have found a much easier way after recreating my old engine from scratch.
Engine.prototype.NODE.prototype.render = function(parentMatrix){
var children = this.children,
numChildren = children.length,
child, pos, rot, scale;
// If has set matrix to a copy of it
if(parentMatrix){
this.matrix = mat4.clone(parentMatrix);
}
else{
// Else set it to a identity matrix
mat4.identity(this.matrix);
}
// If matrix needs updating reconstruct it
pos = [this.position.x,
this.position.y,
this.position.z];
rot = [this.rotation.x,
this.rotation.y,
this.rotation.z];
scale = [this.scale.x,
this.scale.y,
this.scale.z];
// Recreate Transformation matrix
mat4.translate(this.matrix, this.matrix, pos);
mat4.rotate(this.matrix, this.matrix, rot[0], [1, 0, 0]);
mat4.rotate(this.matrix, this.matrix, rot[1], [0, 1, 0]);
mat4.rotate(this.matrix, this.matrix, rot[2], [0, 0, 1]);
mat4.scale(this.matrix, this.matrix, scale);
// Render Children with this matrix
for(var i = 0; i < numChildren; i++){
child = children[i];
child.render(this.matrix);
}
}
what I am basically doing is that, if the matrix has a parent (it isn't the root node) then I am starting the matrix off as a clone of its parent, else I am setting the matrix to it's identity matrix. Then applying the regular transformations to it. If I find a way in order to continue caching matrices I will uploaded it as soon as possible.

Resources