Correct way to passed the stream image to tensorflow-lite model using Xamarin - xamarin

im currently developing Image Classification using xamarin form, the model works well and the prediction result is good, but my problem is, it is only works in PickPhotoAsync passing to MediaFile then the MediaFile > get stream > get the byte > feed to model. i just used this for testing only
What i really need to do is to get the image from SignaturePad > get stream > get the Byte > feed to model
i got wrong prediction when i change it to signaturePad. i hope someone will help me, thank you in advance
here's my code.
protected async void PickPhotoAsync(object sender, EventArgs e)
{
var file = await CrossMedia.Current.PickPhotoAsync();
HandlePhoto(file);
}
private void HandlePhoto(MediaFile file)
{
var stream = file.GetStreamWithImageRotatedForExternalStorage();
var memoryStream = new MemoryStream();
stream.CopyTo(memoryStream);
var bytes = memoryStream.ToArray();
var len = stream.Length;
var pos = stream.Position;
var read = stream.CanRead;
var lens = bytes.Length;
List<ImageClassificationModel> classifyImage = DependencyService.Get<IClassify>().Classify(bytes);
var sortedList = classifyImage.OrderByDescending(x => x.Probability);
var top = sortedList.First();//Highest Prediction Result
var max = sortedList.Max(x => x.Probability);
}
Here is my code when i change it to SignaturePad (Source of image) which i got bad result
private async void SaveImagePad(object sender, EventArgs e)
{
Stream image = await PadView.GetImageStreamAsync(SignatureImageFormat.Png);
//get the stream from SignaturePad
if (image == null)
return;
BinaryReader br = new BinaryReader(image);
Byte[] All = br.ReadBytes((int)image.Length);
byte[] acImage = (byte[])All;
//Convert To Byte before passing to classifier
List<ImageClassificationModel> classifyImage = DependencyService.Get<IClassify>().Classify(acImage);
//File.Delete(path);
var sortedList = classifyImage.OrderByDescending(x => x.Probability);
var top = sortedList.First();
var max = sortedList.Max(x => x.Probability); // i got bad and random result
}

Related

Cognitive face detection is not working when I try to upload the image with a face mask

Any suggestions on how to get info of the image with a mask in cognitive face recognition?
When I upload image with headwear or eyeglasses then cognitive service returns the image information but when picking an image with mask, Cognitive service doesn't return any information. That means my implementation of cognitive service is not able to recognize the image with the mask. If anybody has faced this issue and resolved it please suggest me a solution.
public string subscriptionKey = "88c**************************f7";
public string uriBase = "https://westcentralus.api.cognitive.microsoft.com/face/v1.0/detect";
//Method to pick an image from the gallery
async void btnPick_Clicked(object sender, System.EventArgs e)
{
try
{
if (!CrossMedia.Current.IsPickPhotoSupported)
{
return;
}
var file = await CrossMedia.Current.PickPhotoAsync(new Plugin.Media.Abstractions.PickMediaOptions
{
PhotoSize = Plugin.Media.Abstractions.PhotoSize.Medium
});
if (file == null) return;
imgSelected.Source = ImageSource.FromStream(() => {
var stream = file.GetStream();
return stream;
});
MakeAnalysisRequest(file.Path);
}
catch (Exception ex)
{
string test = ex.Message;
}
}
> //convert Convert image to byte array
public byte[] GetImageAsByteArray(string imageFilePath)
{
using (FileStream fileStream =
new FileStream(imageFilePath, FileMode.Open, FileAccess.Read))
{
BinaryReader binaryReader = new BinaryReader(fileStream);
return binaryReader.ReadBytes((int)fileStream.Length);
}
}
> //Method to get image information from the detection Url
public async void MakeAnalysisRequest(string imageFilePath)
{
HttpClient client = new HttpClient();
client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", subscriptionKey);
string requestParameters = "returnFaceId=true&returnFaceLandmarks=false" +
"&returnFaceAttributes=age,gender,headPose,smile,facialHair,glasses," +
"emotion,hair,makeup,occlusion,accessories,blur,exposure,noise";
string uri = uriBase + "?" + requestParameters;
HttpResponseMessage response;
byte[] byteData = GetImageAsByteArray(imageFilePath);
using (ByteArrayContent content = new ByteArrayContent(byteData))
{
content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
response = await client.PostAsync(uri, content);
string contentString = await response.Content.ReadAsStringAsync();
//***************************************************
//Here it return null in case of mask else its working fine
//***************************************************
List<ResponseModel> faceDetails = JsonConvert.DeserializeObject<List<ResponseModel>>(contentString);
if (faceDetails.Count != 0)
{
lblTotalFace.Text = "Total Faces : " + faceDetails.Count;
lblGender.Text = "Gender : " + faceDetails[0].faceAttributes.gender;
lblAge.Text = "Total Age : " + faceDetails[0].faceAttributes.age;
Console.WriteLine(faceDetails[0].faceAttributes.accessories.FirstOrDefault(x => x.type == "mask").confidence);
}
}
}
There a 2 different things that you must have in mind:
Some faces might not be seen by the services, see doc:
Some faces might not be detected because of technical challenges.
Extreme face angles (head pose) or face occlusion (objects such as
sunglasses or hands that block part of the face) can affect detection.
Frontal and near-frontal faces give the best results.
There are currently 2 detection models in Face API: "detection_01" and "detection_02". This latest model (existing since May 2019 if I remember well) has better performances (especially for rotated or partially hidden faces) but is not providing all the information in output that model 1 is giving.
Difference in detection models
I made a quick demo using a the "Cognitive Workbench" demo portal (available here) with the following image:
With detection_01: no face found
With detection_02: the face is found, as you can see in this capture:
But if you need to use specific attributes from the face, this might not solve your problem. See API documentation extract:

Rgb 565 Pdf to Image

I am trying to convert a PDF page to an image, to create thumbnails. This is the code that I am using:
PdfRenderer pdfRenderer = new PdfRenderer(GetSeekableFileDescriptor(filePath));
var appDirectory = System.Environment.GetFolderPath(System.Environment.SpecialFolder.MyDocuments);
string fileName = System.IO.Path.GetFileNameWithoutExtension(filePath);
string directoryPath = System.IO.Path.Combine(appDirectory, "thumbnailsTemp", System.IO.Path.GetFileNameWithoutExtension(fileName));
if (!Directory.Exists(directoryPath))
{
Directory.CreateDirectory(directoryPath);
int pageCount = pdfRenderer.PageCount;
for (int i = 0; i < pageCount; i++)
{
Page page = pdfRenderer.OpenPage(i);
Android.Graphics.Bitmap bmp = Android.Graphics.Bitmap.CreateBitmap(page.Width, page.Height, Android.Graphics.Bitmap.Config.Rgb565 or Argb8888);
page.Render(bmp, null, null, PdfRenderMode.ForDisplay);
try
{
using (FileStream output = new FileStream(System.IO.Path.Combine(directoryPath, fileName + "Thumbnails" + i + ".png"), FileMode.Create))
{
bmp.Compress(Android.Graphics.Bitmap.CompressFormat.Png, 100, output);
}
page.Close();
}
catch (Exception ex)
{
//TODO -- GERER CETTE EXPEXPTION
throw new Exception();
}
}
return directoryPath;
}
I tried with ARGB 8888 and that was a success. But the rendering time was too slow for big PDF files. This is why I tried to improve it by changing the format to RGB 565. But my app is crashing with this Execption:
Unsuported pixel format
Any idea to fix this, or how to render a PDF to a bitmap faster? I was looking on google but didn't find a solution related to my code.
UPDATE
I did this but know, my app is crashing at this line of code :
await Task.Run(() =>
{
bytes = page.AsPNG(72);
});
My class :
public async Task<string> GetBitmaps(string filePath)
{
//TODO -- WORK ON THIS
PdfRenderer pdfRenderer = new PdfRenderer(GetSeekableFileDescriptor(filePath));
var appDirectory = System.Environment.GetFolderPath(System.Environment.SpecialFolder.MyDocuments);
string fileName = System.IO.Path.GetFileNameWithoutExtension(filePath);
string directoryPath = System.IO.Path.Combine(appDirectory, "thumbnailsTemp", System.IO.Path.GetFileNameWithoutExtension(fileName));
var stream = new MemoryStream();
using (Stream resourceStream = new FileStream(filePath, FileMode.Open))
{
resourceStream.CopyTo(stream);
}
for (int i = 0; i < pdfRenderer.PageCount; i++)
{
TallComponents.PDF.Rasterizer.Page page = new TallComponents.PDF.Rasterizer.Page(stream, i);
byte[] bytes = null;
await Task.Run(() =>
{
bytes = page.AsPNG(72);
});
using (FileStream output = new FileStream(System.IO.Path.Combine(directoryPath, fileName + "Thumbnails" + i + ".png"), FileMode.Create, FileAccess.Write))
{
output.Write(bytes, 0, bytes.Length);
}
}
return directoryPath;
}
you could draw a PDF page in app by converting a PDF page to a bitmap,here the PDF document itself is embedded as a resource.
var assembly = Assembly.GetExecutingAssembly();
var stream = new MemoryStream();
using (Stream resourceStream = assembly.GetManifestResourceStream("DrawPdf.Android.tiger.pdf"))
{
resourceStream.CopyTo(stream);
}
Page page = new Page(stream, 0);
// render PDF Page object to a Bitmap
byte[] bytes = null;
await Task.Run(() =>
{
bytes = page.AsPNG(72);
});
Bitmap bmp = global::Android.Graphics.BitmapFactory.DecodeByteArray(bytes, 0, bytes.Length);

Xamarin Forms UWP Capture Screenshot Include Signature Pad

I have a Xamarin Forms page using Signature Pad (https://github.com/xamarin/SignaturePad). I'm attempting to capture a screenshot of the entire view. It should include the signature as well.
However, using the following code I'm noticing the signature does not show up.
What is the best way to capture the full Page including the signature? (not just the signature)
public class ScreenshotService : IScreenshotService
{
public async Task<byte[]> CaptureAsync()
{
var rtb = new RenderTargetBitmap();
await rtb.RenderAsync(Window.Current.Content);
var pixelBuffer = await rtb.GetPixelsAsync();
var pixels = pixelBuffer.ToArray();
// Useful for rendering in the correct DPI
var displayInformation = DisplayInformation.GetForCurrentView();
var stream = new InMemoryRandomAccessStream();
var encoder = await BitmapEncoder.CreateAsync(BitmapEncoder.JpegEncoderId, stream);
encoder.SetPixelData(BitmapPixelFormat.Bgra8,
BitmapAlphaMode.Premultiplied,
(uint)rtb.PixelWidth,
(uint)rtb.PixelHeight,
displayInformation.RawDpiX,
displayInformation.RawDpiY,
pixels);
await encoder.FlushAsync();
stream.Seek(0);
var readStram = stream.AsStreamForRead();
var bytes = new byte[readStram.Length];
readStram.Read(bytes, 0, bytes.Length);
return bytes;
}
}
According to the "XAML visuals and RenderTargetBitmap capture capabilities" of RenderTargetBitmap class:
Content that can't be captured will appear as blank in the captured image, but other content in the same visual tree can still be captured and will render (the presence of content that can't be captured won't invalidate the entire capture of that XAML composition).
So it could be that the content of InkCanvas is not captureable. However, you can use Win2D. For more you could refer the following code.
public async Task<Stream> CaptureAsync(Stream Tem)
{
var rtb = new RenderTargetBitmap();
await rtb.RenderAsync(Window.Current.Content);
var pixelBuffer = await rtb.GetPixelsAsync();
var pixels = pixelBuffer.ToArray();
var displayInformation = DisplayInformation.GetForCurrentView();
var stream = new InMemoryRandomAccessStream();
var encoder = await BitmapEncoder.CreateAsync(BitmapEncoder.JpegEncoderId, stream);
encoder.SetPixelData(BitmapPixelFormat.Bgra8,
BitmapAlphaMode.Premultiplied,
(uint)rtb.PixelWidth,
(uint)rtb.PixelHeight,
displayInformation.RawDpiX,
displayInformation.RawDpiY,
pixels);
await encoder.FlushAsync();
stream.Seek(0);
var readStram = stream.AsStreamForRead();
var pagebitmap = await GetSoftwareBitmap(readStram);
var softwareBitmap = await GetSoftwareBitmap(Tem);
CanvasDevice device = CanvasDevice.GetSharedDevice();
CanvasRenderTarget renderTarget = new CanvasRenderTarget(device, rtb.PixelWidth, rtb.PixelHeight, 96);
using (var ds = renderTarget.CreateDrawingSession())
{
ds.Clear(Colors.White);
var page = CanvasBitmap.CreateFromSoftwareBitmap(device, pagebitmap);
var image = CanvasBitmap.CreateFromSoftwareBitmap(device, softwareBitmap);
ds.DrawImage(page);
ds.DrawImage(image);
}
InMemoryRandomAccessStream randomAccessStream = new InMemoryRandomAccessStream();
await renderTarget.SaveAsync(randomAccessStream, CanvasBitmapFileFormat.Jpeg, 1f);
return randomAccessStream.AsStream();
}
private async Task<SoftwareBitmap> GetSoftwareBitmap(Stream data)
{
BitmapDecoder pagedecoder = await BitmapDecoder.CreateAsync(data.AsRandomAccessStream());
return await pagedecoder.GetSoftwareBitmapAsync(BitmapPixelFormat.Bgra8, BitmapAlphaMode.Premultiplied);
}
IScreenshotServicecs interface
public interface IScreenshotServicecs
{
Task<Stream> CaptureAsync(Stream stream);
}
Usage
var stream = await SignatureView.GetImageStreamAsync(SignaturePad.Forms.SignatureImageFormat.Png);
var data = await DependencyService.Get<IScreenshotServicecs>().CaptureAsync(stream);
MyImage.Source = ImageSource.FromStream(() => data);
Here is my final implementation including converting to byte array.
public async Task<byte[]> CaptureAsync(Stream signatureStream)
{
var rtb = new RenderTargetBitmap();
await rtb.RenderAsync(Window.Current.Content);
var pixelBuffer = await rtb.GetPixelsAsync();
var pixels = pixelBuffer.ToArray();
var displayInformation = DisplayInformation.GetForCurrentView();
var stream = new InMemoryRandomAccessStream();
var encoder = await BitmapEncoder.CreateAsync(BitmapEncoder.JpegEncoderId, stream);
encoder.SetPixelData(BitmapPixelFormat.Bgra8,
BitmapAlphaMode.Premultiplied,
(uint)rtb.PixelWidth,
(uint)rtb.PixelHeight,
displayInformation.RawDpiX,
displayInformation.RawDpiY,
pixels);
await encoder.FlushAsync();
stream.Seek(0);
var readStram = stream.AsStreamForRead();
var pagebitmap = await GetSoftwareBitmap(readStram);
var softwareBitmap = await GetSoftwareBitmap(signatureStream);
CanvasDevice device = CanvasDevice.GetSharedDevice();
CanvasRenderTarget renderTarget = new CanvasRenderTarget(device, rtb.PixelWidth, rtb.PixelHeight, 96);
using (var ds = renderTarget.CreateDrawingSession())
{
ds.Clear(Colors.White);
var page = CanvasBitmap.CreateFromSoftwareBitmap(device, pagebitmap);
var image = CanvasBitmap.CreateFromSoftwareBitmap(device, softwareBitmap);
ds.DrawImage(page);
ds.DrawImage(image, 50, 55);
}
InMemoryRandomAccessStream randomAccessStream = new InMemoryRandomAccessStream();
await renderTarget.SaveAsync(randomAccessStream, CanvasBitmapFileFormat.Jpeg, 1f);
var fileBytes = new byte[randomAccessStream.Size];
using (var reader = new DataReader(randomAccessStream))
{
await reader.LoadAsync((uint)randomAccessStream.Size);
reader.ReadBytes(fileBytes);
}
return fileBytes;
}

Windows 10 Store App - save URI as file

I am writing a Windows 10 Store app. In the app the User can input a Text, and then press "Read Text" and Cortana reads the text loud. That works fine.
Now I want to add the feature, to press a button called "Save" or something like that and then save Cortanas output as mp3 file. This should work via a normal save-file dialog.
This is what I got so far.
private static MediaElement mediaplayer = new MediaElement();
/// ... mediaplayer element gets content ...
Uri file = mediaplayer.Source;
Instead of an Uri element I could also get an SpeechSynthesisStream with this information.
How can I save this Uri / Stream to a file?
EDIT:
this is the final code:
var stream2 = stream.CloneStream();
//... use stream2 as mediaelement ...
if(stream != null)
{
using (var reader = new DataReader(stream))
{
FileSavePicker savePicker = new FileSavePicker();
savePicker.SuggestedStartLocation = PickerLocationId.PicturesLibrary;
savePicker.FileTypeChoices.Add("WAV", new List<string>() { ".wav" });
savePicker.SuggestedFileName = "sound.wav";
StorageFile file = await savePicker.PickSaveFileAsync();
if (file != null)
{
using (var outputStream = await file.OpenAsync(FileAccessMode.ReadWrite))
{
using (var writer = new DataWriter(outputStream.GetOutputStreamAt(0)))
{
long writtenBytes = 0;
const int bufferSize = 8192;
uint loadedBytes = 0;
while ((loadedBytes = (await reader.LoadAsync(bufferSize))) > 0) //!!!
{
IBuffer buffer = reader.ReadBuffer(loadedBytes);
writer.WriteBuffer(buffer);
uint tmpWritten = await writer.StoreAsync(); //!!!
writtenBytes += tmpWritten;
}
}
}
}
}
}
If you're trying to write the output to a file instead (or as well as) rendering it audibly to a MediaElement, you probably want something like this in here as well.
SpeechSynthesisStream synthesisStream = await synthesizer.SynthesizeTextToStreamAsync(text);
var stream2 = synthesisStream.CloneStream();
FileSavePicker savePicker = new FileSavePicker();
savePicker.SuggestedStartLocation = PickerLocationId.MusicLibrary;
savePicker.FileTypeChoices.Add("WAV", new List<string>() { ".wav" });
savePicker.SuggestedFileName = "sound.wav";
StorageFile file = await savePicker.PickSaveFileAsync();
using (var reader = new DataReader(synthesisStream))
{
using (var outputStream = await file.OpenAsync(FileAccessMode.ReadWrite))
{
using (var writer = new DataWriter(outputStream.GetOutputStreamAt(0)))
{
long writtenBytes = 0;
const int bufferSize = 8192;
uint loadedBytes = 0;
while ((loadedBytes = (await reader.LoadAsync(bufferSize))) > 0) //!!!
{
IBuffer buffer = reader.ReadBuffer(loadedBytes);
writer.WriteBuffer(buffer);
uint tmpWritten = await writer.StoreAsync(); //!!!
writtenBytes += tmpWritten;
}
}
}
}
// Set the source and start playing the synthesized audio stream.
media.AutoPlay = true;
media.SetSource(stream2, synthesisStream.ContentType);
media.Play();
The one problem is that the synthesisStream isn't rewindable (so far as I can tell), so you might have to synthesize it twice, or make a second (in memory) copy of the stream if you want to make it audible at the same time.

How do you get from a byte[] or from a MemoryStream to a System.Windows.Controls.Image in WP7?

I need to make a GET request that requires oauth, for and image to display in a page. Is there a way to do this without building a custom webrequest or httpwebrequest?
If it's an image make sure that you are reading the stream as binary data.
Something like:
var httpRequest = (HttpWebRequest)ar.AsyncState;
var httpResponse = (HttpWebResponse)httpRequest.EndGetResponse(ar);
int lengthInBytes = Convert.ToInt32(httpResponse.ContentLength);
BinaryReader br = new BinaryReader(httpResponse.GetResponseStream());
byte[] imageInBytes = new byte[lengthInBytes];
using (br)
{
for (int i = 0; i < lengthInBytes; i++)
{
imageInBytes[i] = br.ReadByte();
}
}
DispatcherHelper.CheckBeginInvokeOnUI(() =>
{
MemoryStream rawBytesStream = new MemoryStream(imageInBytes);
BitmapImage img = new BitmapImage();
img.SetSource(rawBytesStream);
imageInUI.Source = img;
});

Resources