I am new to google-project-tango, and I want to record the images from Tango's color camera image buffer rather than fisheye camera. I saw henderso's sharing -- reading of the GL buffer to save the image, and modified his code to record the images-- keep saving every image when we get any onFrameAvailable() callback. I also try to lock the buffer or use semaphore, when i got the onframeavailable callback. But the results look like this one--is it the synchronization problem of GL buffer?.
The MainActivity implements TangoCameraScreengrabCallback:
public void onFrameAvailable(int arg0) {
if (mIsRecording){
tangoCameraPreview.takeSnapShot();
try {
mutex_on_mIsRecording.acquire();
} catch (InterruptedException e) {
e.printStackTrace();
}
tangoCameraPreview.onFrameAvailable();
mutex_on_mIsRecording.release();
}
}
And tangoCameraPreview.onFrameAvailable() will do as follows:
public void onFrameAvailable() {
super.onFrameAvailable();
if(recording) {
synchronized (intBuffer){
screenGrabRenderer.grabNextScreen(SCREEN_GRAB_X, SCREEN_GRAB_Y, SCREEN_GRAB_W, SCREEN_GRAB_H, intBuffer);
}
}
}
Related
I'm using Xamarin/Android (not Forms), trying to integrate the camera2basic api sample into my project.
https://developer.xamarin.com/samples/monodroid/android5.0/Camera2Basic/
I changed nothing in the sample, and I am only interested in using the main camera and only taking a snapshot.
My project has a MainActivity and the camera2 is one of its Fragments that I'm calling like this:
string fragmentTag = this.Resources.GetString(Resource.String.camera_form);
// Begin the transaction
FragmentTransaction trans = this.FragmentManager.BeginTransaction();
// Replace the old fragment with the new one.
trans.Add(Resource.Id.fragment_container, camera2BasicFragment, fragmentTag);
// Add the transaction to the back stack.
// The tag is added so we can use PopBackStack to skip a screen on the back key
trans.AddToBackStack(fragmentTag);
// Don't forget to commit
trans.Commit();
Everything works the first time. It takes a photo and saves it to a folder.
The second time I run it, it shows a preview, then when I take a photo it crashes here, where the throw is:
public void CaptureStillPicture()
{
try
{
var activity = Activity;
if (null == activity || null == mCameraDevice)
{
return;
}
// This is the CaptureRequest.Builder that we use to take a picture.
if (stillCaptureBuilder == null)
stillCaptureBuilder = mCameraDevice.CreateCaptureRequest(CameraTemplate.StillCapture);
stillCaptureBuilder.AddTarget(mImageReader.Surface);
// Use the same AE and AF modes as the preview.
stillCaptureBuilder.Set(CaptureRequest.ControlAfMode, (int)ControlAFMode.ContinuousPicture);
SetAutoFlash(stillCaptureBuilder);
// Orientation
int rotation = (int)activity.WindowManager.DefaultDisplay.Rotation;
stillCaptureBuilder.Set(CaptureRequest.JpegOrientation, GetOrientation(rotation));
mCaptureSession.StopRepeating();
try
{
mCaptureSession.Capture(stillCaptureBuilder.Build(), new CameraCaptureStillPictureSessionCallback(this), null);
}
catch (System.Exception e)
{
throw;
}
}
catch (CameraAccessException e)
{
e.PrintStackTrace();
}
}
With this error:
{Java.Lang.IllegalArgumentException: CaptureRequest contains unconfigured Input/Output Surface!
at Java.Interop.JniEnvironment+InstanceMethods.CallIntMethod (Java.Interop.JniObjectReference instance, Java.Interop.JniMethodInfo method, Java.Interop.JniArgumentValue* args) [0x00069] in <286213b9e14c442ba8d8d94cc9dbec8e>:0
at Java.Interop.JniPeerMembers+JniInstanceMethods.InvokeAbstractInt32Method (System.String encodedMember, Java.Interop.IJavaPeerable self, Java.Interop.JniArgumentValue* parameters) [0x00014] in <286213b9e14c442ba8d8d94cc9dbec8e>:0
at Android.Hardware.Camera2.CameraCaptureSessionInvoker.Capture (Android.Hardware.Camera2.CaptureRequest request, Android.Hardware.Camera2.CameraCaptureSession+CaptureCallback listener, Android.OS.Handler handler) [0x00078] in <b781ed64f1d743e7881ac038e0fbdf85>:0
at RvsMobileApp.Activities.Camera2BasicFragment.CaptureStillPicture () [0x000b7] in C:\Source\RVS\rvs-mobile-app\src\Rvs.Mobile.App\Activities\Camera2BasicFragment.cs:807
--- End of managed Java.Lang.IllegalArgumentException stack trace ---
java.lang.IllegalArgumentException: CaptureRequest contains unconfigured Input/Output Surface!
at android.hardware.camera2.CaptureRequest.convertSurfaceToStreamId(CaptureRequest.java:674)
at android.hardware.camera2.impl.CameraDeviceImpl.submitCaptureRequest(CameraDeviceImpl.java:1066)
at android.hardware.camera2.impl.CameraDeviceImpl.capture(CameraDeviceImpl.java:936)
at android.hardware.camera2.impl.CameraCaptureSessionImpl.capture(CameraCaptureSessionImpl.java:173)
at md5bbb797339b35f7667da89d6634e22c37.CameraCaptureListener.n_onCaptureCompleted(Native Method)
at md5bbb797339b35f7667da89d6634e22c37.CameraCaptureListener.onCaptureCompleted(CameraCaptureListener.java:37)
at android.hardware.camera2.impl.CameraCaptureSessionImpl$1.lambda$onCaptureCompleted$3(CameraCaptureSessionImpl.java:640)
at android.hardware.camera2.impl.-$$Lambda$CameraCaptureSessionImpl$1$OA1Yz_YgzMO8qcV8esRjyt7ykp4.run(Unknown Source:8)
at android.os.Handler.handleCallback(Handler.java:873)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loop(Looper.java:214)
at android.os.HandlerThread.run(HandlerThread.java:65)
}
base: {Java.Lang.RuntimeException}
JniPeerMembers: {Android.Runtime.XAPeerMembers}
At first I thought it was a memory leak, so I made sure my fragment was killing itself off. Here is how I end the fragment when the finished button is pressed:
case Resource.Id.camera_finished:
// EventHandler<DialogClickEventArgs> nullHandler = null;
Activity activity = Activity;
if (activity != null)
{
// Send all of the data to the service
// SendPhotosAndDataToService();
// Call the paren activitity's back to END this Fragment
activity.FragmentManager.BeginTransaction().Remove(this).CommitNow();
//activity.OnBackPressed();
}
break;
Here are my steps to reproduce the error:
Start the Camera (load the fragment)
See a preview
Take a photo
Return to Main Activity (close fragment)
Start the Camera (load the fragment)
See a preview
Take a photo CRASH!!!
As long as I don't take any photos, I can load and unload the fragment as many times as I need to.
I Googled "CaptureRequest contains unconfigured Input/Output Surface!", and didn't get enough information to really understand the problem.
I think something is not cleaning itself up after the first run.
I've been working on this issue for days now.
As Alex Cohn pointed out, and I found when I read this article:
https://hofmadresu.com/2018/09/11/android-camera2-trials-and-tribulations.html
Which is an excellent resource BTW, the sample code was not releasing the stillCaptureBuilder so it can be used the second time.
public void CaptureStillPicture()
{
try
{
var activity = Activity;
if (null == activity || null == mCameraDevice)
{
return;
}
// THIS WAS NOT RELEASING THE RESOURCES AND SHOULD BE REMOVED FROM THE SAMPLE!
//// This is the CaptureRequest.Builder that we use to take a picture.
////if (stillCaptureBuilder == null)
//// stillCaptureBuilder = mCameraDevice.CreateCaptureRequest(CameraTemplate.StillCapture);
// This is the proper code
var stillCaptureBuilder = mCameraDevice.CreateCaptureRequest(CameraTemplate.StillCapture);
stillCaptureBuilder.AddTarget(mImageReader.Surface);
// Use the same AE and AF modes as the preview.
stillCaptureBuilder.Set(CaptureRequest.ControlAfMode, (int)ControlAFMode.ContinuousPicture);
SetAutoFlash(stillCaptureBuilder);
// Orientation
int rotation = (int)activity.WindowManager.DefaultDisplay.Rotation;
stillCaptureBuilder.Set(CaptureRequest.JpegOrientation, GetOrientation(rotation));
mCaptureSession.StopRepeating();
mCaptureSession.AbortCaptures();
try
{
mCaptureSession.Capture(stillCaptureBuilder.Build(), new CameraCaptureStillPictureSessionCallback(this), null);
}
catch (System.Exception e)
{
throw;
}
}
catch (CameraAccessException e)
{
e.PrintStackTrace();
}
}
I am documenting this so anyone else struggling with camera2 can learn.
Thank you as always
Have stillCaptureBuilder re-initialized when you restore the camera fragment. Even better, make sure you clean the stillCaptureBuilder up when the fragment is destroyed.
UPDATE: The initial question has been answered as to why the crashes happen but the lingering problem remains of why is the 'OnImageAvailable' callback called so may times? When it is called, I want to do stuff with the image, but whatever method I run at that time is called many times. Is this the wrong place to be using the resulting image?
I am using the sample code found here for a Xamarin Android implementation of the Android Camera2 API. My issue is that when the capture button is pressed a single time, the OnCameraAvalibleListener's OnImageAvailable callback gets called multiple times.
This is causing a problem because the image from AcquireNextImage needs to be closed before another can be used, but close is not called until the Run method of the ImageSaver class as seen below.
This causes these 2 errors:
Unable to acquire a buffer item, very likely client tried to acquire
more than maxImages buffers
AND
maxImages (2) has already been acquired, call #close before acquiring
more.
The max image is set to 2 by default, but setting it to 1 does not help. How do I prevent the callback from being called twice?
public void OnImageAvailable(ImageReader reader)
{
var image = reader.AcquireNextImage();
owner.mBackgroundHandler.Post(new ImageSaver(image, file));
}
// Saves a JPEG {#link Image} into the specified {#link File}.
private class ImageSaver : Java.Lang.Object, IRunnable
{
// The JPEG image
private Image mImage;
// The file we save the image into.
private File mFile;
public ImageSaver(Image image, File file)
{
if (image == null)
throw new System.ArgumentNullException("image");
if (file == null)
throw new System.ArgumentNullException("file");
mImage = image;
mFile = file;
}
public void Run()
{
ByteBuffer buffer = mImage.GetPlanes()[0].Buffer;
byte[] bytes = new byte[buffer.Remaining()];
buffer.Get(bytes);
using (var output = new FileOutputStream(mFile))
{
try
{
output.Write(bytes);
}
catch (IOException e)
{
e.PrintStackTrace();
}
finally
{
mImage.Close();
}
}
}
}
The method OnImageAvailable can be called again as soon as you leave it if there is another picture in the pipeline.
I would recommend calling Close in the same method you are calling AcquireNextImage. So, if you choose to get the image directly from that callback, then you have to call Close in there as well.
One solution involved grabbing the image in that method and close it right away.
public void OnImageAvailable(ImageReader reader)
{
var image = reader.AcquireNextImage();
try
{
ByteBuffer buffer = mImage.GetPlanes()[0].Buffer;
byte[] bytes = new byte[buffer.Remaining()];
buffer.Get(bytes);
// I am not sure where you get the file instance but it is not important.
owner.mBackgroundHandler.Post(new ImageSaver(bytes, file));
}
finally
{
image.Close();
}
}
The ImageSaver would be modified to accept the byte array as first parameter in the constructor:
public ImageSaver(byte[] bytes, File file)
{
if (bytes == null)
throw new System.ArgumentNullException("bytes");
if (file == null)
throw new System.ArgumentNullException("file");
mBytes = bytes;
mFile = file;
}
The major downside of this solution is the risk of putting a lot of pressure on the memory as you basically save the images in memory until they are processed, one after another.
Another solution consists in acquiring the image on the background thread instead.
public void OnImageAvailable(ImageReader reader)
{
// Again, I am not sure where you get the file instance but it is not important.
owner.mBackgroundHandler.Post(new ImageSaver(reader, file));
}
This solution is less intensive on the memory; but you might have to increase the maximum number of images from 2 to something higher depending on your needs. Again, the ImageSaver's constructor needs to be modified to accept an ImageReader as a parameter:
public ImageSaver(ImageReader imageReader, File file)
{
if (imageReader == null)
throw new System.ArgumentNullException("imageReader");
if (file == null)
throw new System.ArgumentNullException("file");
mImageReader = imageReader;
mFile = file;
}
Now the Run method would have the responsibility of acquiring and releasing the Image:
public void Run()
{
Image image = mImageReader.AcquireNextImage();
try
{
ByteBuffer buffer = image.GetPlanes()[0].Buffer;
byte[] bytes = new byte[buffer.Remaining()];
buffer.Get(bytes);
using (var output = new FileOutputStream(mFile))
{
try
{
output.Write(bytes);
}
catch (IOException e)
{
e.PrintStackTrace();
}
}
}
finally
{
image?.Close();
}
}
I too facing this issue for longer time and tried implementing #kzrytof's solution but didn't helped well as expected but found the way to get the onImageAvailable to execute once.,
Scenario: When the image is available then the onImageAvailable method is called right?
so, What I did is after closing the image using image.close(); I called the imagereader.setonImageAvailableListener() and made the listener = null. this way I stopped the execution for second time.,
I know, that your question is for xamarin and my below code is in native android java but the method and functionalities are same, so try once:
#Override
public void onImageAvailable(ImageReader reader) {
final Image image=imageReader.acquireLatestImage();
try {
if (image != null) {
Image.Plane[] planes = image.getPlanes();
ByteBuffer buffer = planes[0].getBuffer();
int pixelStride = planes[0].getPixelStride();
int rowStride = planes[0].getRowStride();
int rowPadding = rowStride - pixelStride * width;
int bitmapWidth = width + rowPadding / pixelStride;
if (latestBitmap == null ||
latestBitmap.getWidth() != bitmapWidth ||
latestBitmap.getHeight() != height) {
if (latestBitmap != null) {
latestBitmap.recycle();
}
}
latestBitmap.copyPixelsFromBuffer(buffer);
}
}
catch(Exception e){
}
finally{
image.close();
imageReader.setOnImageAvailableListener(null, svc.getHandler());
}
// next steps to save the image
}
once again I need some help:
yesterday I asked this question that was about the way to use a large jpg image as a Bitmap (http://stackoverflow.com/questions/13511657/problems-with-big-drawable-jpg-image) and I resolved myself (Is my own response on that question) but whenever I resume my activity, as it uses that bitmap as the GLRenderer texture it crashes. I've tried many things, the last try was to make that bitmap static in order to keep it as a member variable into the activity but it crashes because, I supose, it looses it's mBuffer.
More details on the Activity code:
I declared it as SingletonInstance into the manifest:
android:launchMode="singleInstance"
in order to keep the tiles for the renderer.
and here some code:
#Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
mGLSurfaceView = new GLSurfaceView(this);
mGLSurfaceView.setEGLConfigChooser(true);
mSimpleRenderer = new GLRenderer(this);
getTextures();
if (!mIsTileMapInitialized){
tileMap = new LandSquareGrid(1, 1, mHeightmap, mLightmap, false, true, true, 128, true);
tileMap.setupSkybox(mSkyboxBitmap, true);
mIsTileMapInitialized = true;
}
initializeRenderer();
mGLSurfaceView.setRenderer(mSimpleRenderer);
setContentView( R.layout.game_layout );
setOnTouchListener();
initializeGestureDetector();
myCompassView = (MyCompassView)findViewById(R.id.mycompassview);
// Once set the content view we can set the TextViews:
coordinatesText = (TextView) findViewById(R.id.coordDynamicText);
altitudeText = (TextView) findViewById(R.id.altDynamicText);
directionText = (TextView) findViewById(R.id.dirDynamicText);
//if (!mIsGLInitialized){
mOpenGLLayout = (LinearLayout)findViewById(R.id.openGLLayout);
mOpenGLLayout.addView(mGLSurfaceView);
mVirtual3DMap = new Virtual3DMap(mSimpleRenderer, tileMap);
if (mGameThread == null){
mGameThread = new Thread(mVirtual3DMap);
mGameThread.start();
}
}
On getTextures method I get few small textures and the largest one as in my last question self response:
if (mTerrainBitmap==null){
InputStream is = getResources().openRawResource(R.drawable.terrain);
try {
// Set terrain bitmap options to 16-bit, 565 format.
terrainBitmapOptions.inPreferredConfig = Bitmap.Config.RGB_565;
Bitmap auxBitmap = BitmapFactory.decodeStream(is, null, terrainBitmapOptions);
mTerrainBitmap = Bitmap.createBitmap(auxBitmap);
}
catch (Exception e){
}
finally {
try {
is.close();
}
catch (IOException e) {
// Ignore.
}
}
}
So, again, first time it works great but when I go back I do:
protected void onPause() {
super.onPause();
mGLSurfaceView.onPause();
}
#Override
protected void onStop() {
// TODO Auto-generated method stub
super.onStop();
if (mVirtual3DMap != null) {
try {
mVirtual3DMap.cancel();
mGameThread=null;
mVirtual3DMap = null;
mGLSurfaceView.destroyDrawingCache();
mSimpleRenderer=null;
System.gc();
} catch (Throwable e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
And whan I resume the activity:
#Override
protected void onResume() {
super.onResume();
mGLSurfaceView.onResume();
if (mVirtual3DMap != null) {
try {
mVirtual3DMap.resume();
} catch (Throwable e) {
e.printStackTrace();
}
}
}
And it crashes.
Why?? Ok, here is the exception cause on the GLThread:
java.lang.IllegalArgumentException: bitmap is recycled...
I tried this messy stuff because launching more than two times the original activity the application crashes bacuse of this or because of the amount of memory used and now I don't know if revert all these changes or what todo with this.
Is there a good way to keep in memory and usable, by this or another application activity, this bitmap?
Please, I need your advices.
Thanks in advance.
Do not handle resources manually or your app's surface will broke up. You can't handle your resources manually.
If you worry about reloading resources and you use API level 11+, you can use setPreserveEGLContextOnPause(). It will perserve your textures and FBOs.
If you can't use API 11+, you can port GLSurfaceView() to your app. You can check my own GLSurfaceView that is ported from ICS.
PS: Sorry about my poor english.
No. Let Android handle all the resources. You must handle the appropriate events and reload the bitmap when the activity is resumed. You cannot expect, that any OpenGL handles are still valid after the activity has been resumed.
Think of it as in the example of a laptop coming out from hibernation. Although all memory has been restored, you cannot expect that any open socket has still a real active connection going.
I am an Android noobie, so please correct me if I am wrong.
I am making an Flashlight app, in which I need to use the camera's LED constantly on pressing ON button and OFF it on pressing the same button. I followed this article Turning on the LED with the video camera using Reflection. The ON/OFF operation works fine only once. The Code is as:
private VideoCamera _videoCamera;
private VideoCameraVisualizer _videoCameraVisualizer;
bool _isFlashOff = true;
private void FlashButton_Click(object sender, RoutedEventArgs e)
{
try
{
if (_isFlashOff)
{
_isFlashOff = false;
// Check to see if the camera is available on the device.
if (PhotoCamera.IsCameraTypeSupported(CameraType.Primary))
{
// Use standard camera on back of device.
_videoCamera = new VideoCamera();
// Event is fired when the video camera object has been initialized.
_videoCamera.Initialized += VideoCamera_Initialized;
// Add the photo camera to the video source
_videoCameraVisualizer = new VideoCameraVisualizer();
_videoCameraVisualizer.SetSource(_videoCamera);
}
}
else
{
_isFlashOff = true;
_videoCamera.StopRecording();
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
private void VideoCamera_Initialized(object sender, EventArgs e)
{
_videoCamera.LampEnabled = true;
_videoCamera.StartRecording();
}
Since there was no implementation of StopRecording Method in VideoCamera class as specified in the article: Turning on the LED with the video camera using Reflection . I made the function as:
public void StopRecording()
{
// Invoke the stop recording method on the video camera object.
_videoCameraStopRecordingMethod.Invoke(_videoCamera, null);
}
The problem is when I again press ON button "Exception' is thrown as "TargetInvocationException". I am unable to figure out the problem that causes exception. Is StopRecording() function right..??
That's because you should initialize the camera only once. Do it during the OnNavigatedTo event, then re-use the same instances:
protected override void OnNavigatedTo(System.Windows.Navigation.NavigationEventArgs e)
{
// Use standard camera on back of device.
_videoCamera = new VideoCamera();
// Event is fired when the video camera object has been initialized.
_videoCamera.Initialized += VideoCamera_Initialized;
// Add the photo camera to the video source
_videoCameraVisualizer = new VideoCameraVisualizer();
_videoCameraVisualizer.SetSource(_videoCamera);
}
private void VideoCamera_Initialized(object sender, EventArgs e)
{
isInitialized = true;
}
bool isInitialized;
bool isFlashEnabled;
private void Button_Click(object sender, System.Windows.RoutedEventArgs e)
{
if (!isInitialized)
{
MessageBox.Show("Please wait during camera initialization");
return;
}
if (!isFlashEnabled)
{
isFlashEnabled = true;
// Check to see if the camera is available on the device.
if (PhotoCamera.IsCameraTypeSupported(CameraType.Primary))
{
_videoCamera.LampEnabled = true;
_videoCamera.StartRecording();
}
}
else
{
isFlashEnabled = false;
_videoCamera.StopRecording();
}
}
Try this:
http://msdn.microsoft.com/en-us/library/hh202949.aspx
I'm using netbeans IDE to practice some Java basic code. But I'm unsuccessful to draw a PNG image. Below is my code,
package JavaApplication1;
import javax.swing.*;
import java.awt.*;
class MyCanvas extends JComponent
{
Image img;
public MyCanvas(){
img = Toolkit.getDefaultToolkit().createImage("pengbrew.png");
}
public void paintComponent(Graphics g){
super.paintComponent(g);
g.drawImage(img, 0, 0, null);
}
}
public class JavaGame
{
public static void main(String args[]){
JFrame window = new JFrame("Image demo");
window.getContentPane().add( new MyCanvas() );
window.setSize(400,400);
window.setVisible(true);
}
}
I added the images in the netbeans project.
Please advise.
Many thanks.
easiest ways is look for JLabel that's best of JComponents for Icon/ImageIcon examples here
To load an image use ImageIO.read(File file). It's a newer API, easier to use and better supported. A tutorial on loading images is here http://docs.oracle.com/javase/tutorial/2d/images/loadimage.html if you need some more pointers.
Your code would look like this instead,
Image img;
public MyCanvas(){
try {
img = ImageIO.read(new File("pengbrew.png"));
} catch (IOException e) {
// Handle exception if there is one.
}
}
For the long answer about why the method you were using wasn't working is probably because of the following...
Your image may be in the proper location and instead Toolkit.createImage() hasn't finished loading the image before the frame comes up to paint. Toolkit.createImage() returns control back to your application before the underlying image has actually finished being loaded. You can usually verify if this is the issue by resizing your application to force it to repaint. If after a few seconds of trying to resize the application the image shows up it was due to the image not being loaded during the first paint calls.
To ensure that the image is loaded before you continue you need to use a MediaTracker. Here is some example code to ensure loading of Image is complete before using it.
Component component = new Component() {};
MediaTracker TRACKER = new MediaTracker(component);
...
Image image = Toolkit.getDefaultToolkit().createImage("imageToLoad.png");
synchronized(TRACKER) {
int id = ++mediaTrackerID;
TRACKER.addImage(image, id);
try {
TRACKER.waitForID(id, 0);
} catch (InterruptedException ex) {
image = null;
}
TRACKER.removeImage(image, id);
}
// Now you can use your image.
You'll see code just like this in the ImageIcon class in the method loadImage(Image image).