Client (javascript) unable to receive image or binary from server using boost beast and opencv - image

I used one of one of the example of boost::beast web server asynchronous to communicate with client javascript using websocket. I am attempting to do the simple receiving image and write it in server side.
The result I receive in the server side is a broken jpg image.
Thank you in advance.
client-side (javascript)
var canvas = document.createElement("canvas");
var ctx = canvas.getContext("2d");
var img = document.getElementById("image");
var ws = null;
connect.onclick = function() {
ws = new WebSocket(uri.value);
ws.onopen = function(ev) {
messages.innerText += "[connection opened]\n";
};
ws.onclose = function(ev) {
messages.innerText += "[connection closed]\n";
};
ws.onmessage = function(ev) {
messages.innerText += ev.data + "\n";
};
ws.onerror = function(ev) {
messages.innerText += "[error]\n";
console.log(ev);
};
};
disconnect.onclick = function() {
ws.close();
};
send.onclick = function() {
var canvas1 = ctx.getImageData(0,0,img.width,img.height);
var binary = new Uint8Array(img.data.length);
for (var i=0; i<img.data.length; ++i) {
binary[i] = img.data[i];
}
ws.send(binary.buffer);
}
server-side
void on_write(boost::system::error_code ec,
std::size_t bytes_transferred)
{
boost::ignore_unused(bytes_transferred);
if(ec)
return fail(ec, "write");
buffer_.consume(buffer_.size());
std::cout << "on write before do read\n";
ws_.async_read(
buffer_,
boost::asio::bind_executor(
strand_,
std::bind(
&session::on_read,
shared_from_this(),
std::placeholders::_1,
std::placeholders::_2)));
uint8_t *buf = new uint8_t [bytes_transferred];
boost::asio::buffer_copy(boost::asio::buffer(buf,bytes_transferred),buffer_.data(),bytes_transferred);
::cv::Mat mImg(400,320,CV_8UC3,buf);
::cv::imwrite("file.jpg",mImg);
delete [] buf;
}

I first recommend refactoring your code into small testable functions.
Write a function like this:
template <class AsyncStream, class CompletionToken>
auto
async_write_opencv_image(AsyncStream&, CompletionToken&& token)
{
// write composed op here
}
Using this, you can feed in a boost::beast::test::stream to the function which will allow you to manually inspect the buffers to ensure the image's correctness.
This has the advantage that now your image writing code is stream-agnostic so you can swap it out for any other type.
If you need any assistance in this regard, there's ample documentation on boost.org as well as live-help on the cpplang slack team.
Edit:
Here's a wandbox example how to potentially also setup a unit test for websockets as well:
https://wandbox.org/permlink/EixDmotCphJwiDZ1

Related

Exception: Service invoked too many times for one day: urlfetch

I created a script in Google Sheets, which is working well but after a while I'm getting the following error:
Exception: Service invoked too many times for one day: urlfetch
I think I called the function like 200-300 times in the day, for what I checked it should be below the limit.
I read we can use cache to avoid this issue but not sure how to use it in my code.
function scrapercache(url) {
var result = [];
var description;
var options = {
'muteHttpExceptions': true,
'followRedirects': false,
};
var cache = CacheService.getScriptCache();
var properties = PropertiesService.getScriptProperties();
try {
let res = cache.get(url);
if (!res) {
// trim url to prevent (rare) errors
url.toString().trim();
var r = UrlFetchApp.fetch(url, options);
var c = r.getResponseCode();
// check for meta refresh if 200 ok
if (c == 200) {
var html = r.getContentText();
cache.put(url, "cached", 21600);
properties.setProperty(url, html);
var $ = Cheerio.load(html); // make sure this lib is added to your project!
// meta description
if ($('meta[name=description]').attr("content")) {
description = $('meta[name=description]').attr("content").trim();
}
}
result.push([description]);
}
}
catch (error) {
result.push(error.toString());
}
finally {
return result;
}
}
how can I use cache like this to enhance my script please?
var cache = CacheService.getScriptCache();
var result = cache.get(url);
if(!result) {
var response = UrlFetchApp.fetch(url);
result = response.getContentText();
cache.put(url, result, 21600);
Answer:
You can implement CacheService and PropertiesService together and only retrieve the URL again after a specified amount of time.
Code Change:
Be aware that additional calls to retrieving the cache and properties will slow your function down, especially if you are doing this a few hundred times.
As the values of the cache can be a maximum of 100 KB, we will use CacheService to keep track of which URLs are to be retrieved, but PropertiesService to store the data.
You can edit your try block as so:
var cache = CacheService.getScriptCache();
var properties = PropertiesService.getScriptProperties();
try {
let res = cache.get(url);
if (!res) {
// trim url to prevent (rare) errors
url.toString().trim();
var r = UrlFetchApp.fetch(url, options);
var c = r.getResponseCode();
// check for meta refresh if 200 ok
if (c == 200) {
var html = r.getContentText();
cache.put(url, "cached", 21600);
properties.setProperty(url, html);
var $ = Cheerio.load(html); // make sure this lib is added to your project!
// meta description
if ($('meta[name=description]').attr("content")) {
description = $('meta[name=description]').attr("content").trim();
}
}
result.push([description]);
}
}
catch (error) {
result.push(error.toString());
}
finally {
return result;
}
References:
Class CacheService | Apps Script | Google Developers
Class Cache | Apps Script | Google Developers
Class PropertiesService | Apps Script | Google Developers
Related Questions:
Service invoked too many times for one day: urlfetch

Custom google cast receiver stuck in "Load is in progress"

My custom v3 CAF receiver app is successfully playing the first few live & vod assets. After that, it gets into a state were media commands are being queued because "Load is in progress". It is still (successfully) fetching manifests, but MEDIA_STATUS remains "buffering". The log then shows:
[ 4.537s] [cast.receiver.MediaManager] Load is in progress, media command is being queued.
[ 5.893s] [cast.receiver.MediaManager] Buffering state changed, isPlayerBuffering: true old time: 0 current time: 0
[ 5.897s] [cast.receiver.MediaManager] Sending broadcast status message
CastContext Core event: {"type":"MEDIA_STATUS","mediaStatus":{"mediaSessionId":1,"playbackRate":1,"playerState":"BUFFERING","currentTime":0,"supportedMediaCommands":12303,"volume":{"level":1,"muted":false},"currentItemId":1,"repeatMode":"REPEAT_OFF","liveSeekableRange":{"start":0,"end":20.000999927520752,"isMovingWindow":true,"isLiveDone":false}}}
CastContext MEDIA_STATUS event: {"type":"MEDIA_STATUS","mediaStatus":{"mediaSessionId":1,"playbackRate":1,"playerState":"BUFFERING","currentTime":0,"supportedMediaCommands":12303,"volume":{"level":1,"muted":false},"currentItemId":1,"repeatMode":"REPEAT_OFF","liveSeekableRange":{"start":0,"end":20.000999927520752,"isMovingWindow":true,"isLiveDone":false}}}
Fetch finished loading: GET "(manifest url)".
No errors are shown.
Even after closing and restarting the cast session, the issue remains. The cast device itself has to be rebooted to resolve it. It looks like data is kept between sessions.
It could be important to note that the cast receiver app is not published yet. It is hosted on a local network.
My questions are:
What could be the cause of this stuck behavior?
Is there any session data kept between session?
How to fully reset the cast receiver app, without the necessity to restart the cast device.
The receiver app itself is very basic. Other than license wrapping it resembles the vanilla example app:
const { cast } = window;
const TAG = "CastContext";
class CastStore {
static instance = null;
error = observable.box();
framerate = observable.box();
static getInstance() {
if (!CastStore.instance) {
CastStore.instance = new CastStore();
}
return CastStore.instance;
}
get debugLog() {
return this.framerate.get();
}
get errorLog() {
return this.error.get();
}
init() {
const context = cast.framework.CastReceiverContext.getInstance();
const playerManager = context.getPlayerManager();
playerManager.addEventListener(
cast.framework.events.category.CORE,
event => {
console.log(TAG, "Core event: " + JSON.stringify(event));
}
);
playerManager.addEventListener(
cast.framework.events.EventType.MEDIA_STATUS,
event => {
console.log(TAG, "MEDIA_STATUS event: " + JSON.stringify(event));
}
);
playerManager.addEventListener(
cast.framework.events.EventType.BITRATE_CHANGED,
event => {
console.log(TAG, "BITRATE_CHANGED event: " + JSON.stringify(event));
runInAction(() => {
this.framerate.set(`bitrate: ${event.totalBitrate}`);
});
}
);
playerManager.addEventListener(
cast.framework.events.EventType.ERROR,
event => {
console.log(TAG, "ERROR event: " + JSON.stringify(event));
runInAction(() => {
this.error.set(`Error detailedErrorCode: ${event.detailedErrorCode}`);
});
}
);
// intercept the LOAD request to be able to read in a contentId and get data.
this.loadHandler = new LoadHandler();
playerManager.setMessageInterceptor(
cast.framework.messages.MessageType.LOAD,
loadRequestData => {
this.framerate.set(null);
this.error.set(null);
console.log(TAG, "LOAD message: " + JSON.stringify(loadRequestData));
if (!loadRequestData.media) {
const error = new cast.framework.messages.ErrorData(
cast.framework.messages.ErrorType.LOAD_CANCELLED
);
error.reason = cast.framework.messages.ErrorReason.INVALID_PARAM;
return error;
}
if (!loadRequestData.media.entity) {
// Copy the value from contentId for legacy reasons if needed
loadRequestData.media.entity = loadRequestData.media.contentId;
}
// notify loadMedia
this.loadHandler.onLoadMedia(loadRequestData, playerManager);
return loadRequestData;
}
);
const playbackConfig = new cast.framework.PlaybackConfig();
// intercept license requests & responses
playbackConfig.licenseRequestHandler = requestInfo => {
const challenge = requestInfo.content;
const { castToken } = this.loadHandler;
const wrappedRequest = DrmLicenseHelper.wrapLicenseRequest(
challenge,
castToken
);
requestInfo.content = wrappedRequest;
return requestInfo;
};
playbackConfig.licenseHandler = license => {
const unwrappedLicense = DrmLicenseHelper.unwrapLicenseResponse(license);
return unwrappedLicense;
};
// Duration of buffered media in seconds to start/resume playback after auto-paused due to buffering; default is 10.
playbackConfig.autoResumeDuration = 4;
// Minimum number of buffered segments to start/resume playback.
playbackConfig.initialBandwidth = 1200000;
context.start({
touchScreenOptimizedApp: true,
playbackConfig: playbackConfig,
supportedCommands: cast.framework.messages.Command.ALL_BASIC_MEDIA
});
}
}
The LoadHandler optionally adds a proxy (I'm using a cors-anywhere proxy to remove the origin header), and stores the castToken for licenseRequests:
class LoadHandler {
CORS_USE_PROXY = true;
CORS_PROXY = "http://192.168.0.127:8003";
castToken = null;
onLoadMedia(loadRequestData, playerManager) {
if (!loadRequestData) {
return;
}
const { media } = loadRequestData;
// disable cors for local testing
if (this.CORS_USE_PROXY) {
media.contentId = `${this.CORS_PROXY}/${media.contentId}`;
}
const { customData } = media;
if (customData) {
const { licenseUrl, castToken } = customData;
// install cast token
this.castToken = castToken;
// handle license URL
if (licenseUrl) {
const playbackConfig = playerManager.getPlaybackConfig();
playbackConfig.licenseUrl = licenseUrl;
const { contentType } = loadRequestData.media;
// Dash: "application/dash+xml"
playbackConfig.protectionSystem = cast.framework.ContentProtection.WIDEVINE;
// disable cors for local testing
if (this.CORS_USE_PROXY) {
playbackConfig.licenseUrl = `${this.CORS_PROXY}/${licenseUrl}`;
}
}
}
}
}
The DrmHelper wraps the license request to add the castToken and base64-encodes the whole. The license response is base64-decoded and unwrapped lateron:
export default class DrmLicenseHelper {
static wrapLicenseRequest(challenge, castToken) {
const wrapped = {};
wrapped.AuthToken = castToken;
wrapped.Payload = fromByteArray(new Uint8Array(challenge));
const wrappedJson = JSON.stringify(wrapped);
const wrappedLicenseRequest = fromByteArray(
new TextEncoder().encode(wrappedJson)
);
return wrappedLicenseRequest;
}
static unwrapLicenseResponse(license) {
try {
const responseString = String.fromCharCode.apply(String, license);
const responseJson = JSON.parse(responseString);
const rawLicenseBase64 = responseJson.license;
const decodedLicense = toByteArray(rawLicenseBase64);
return decodedLicense;
} catch (e) {
return license;
}
}
}
The handler for cast.framework.messages.MessageType.LOAD should always return:
the (possibly modified) loadRequestData, or
a promise for the (possibly modified) loadRequestData
null to discard the load request (I'm not 100% sure this works for load requests)
If you do not do this, the load request stays in the queue and any new request is queued after the initial one.
In your handler, you return an error if !loadRequestData.media, which will get you into that state. Another possibility is an exception in the load request handler, which will also get you in that state.
I guess we have a different approach and send everything possible through sendMessage, when we loading stuff we create a new cast.framework.messages.LoadRequestData() which we load with playerManager.load(loadRequest).
But I guess that you might be testing this on an integrated Chromecast, we see this problems as well!?
I suggest that you do one or more
Enable gzip compression on all responses!!!
Stop playback playerManager.stop() (maybe in the interseptor?)
Change how the licenseUrl is set
How we set licenseUrl
playerManager.setMediaPlaybackInfoHandler((loadRequestData, playbackConfig) => {
playbackConfig.licenseUrl = loadRequestData.customData.licenseUrl;
return playbackConfig;
}
);

Dart component progress bar only updating at the end of loop

I'm developing an app in angular dart and trying to animate a progress bar which shows progress of a file upload. I'm simply parsing the JSON from the file, and sending them off to a service with a _service.create(....) method.
I've put all of this code in an async method which is called when my submit button is clicked:
void uploadFile() async {
submitButton.attributes.addAll({ 'disabled': '' });
progress.value = 0;
FileList files = input.files;
if (files.isEmpty) {
_handleError("No file selected");
//handle error, probably a banner
return;
}
File file = files.item(0);
FileReader reader = new FileReader();
//reader.result
reader.onLoadEnd.listen((e) async {
Map map = json.decode(reader.result);
var combinations = map['combinations'];
progress.max = combinations.length;
int loopCount = 0;
combinations.forEach((e) async {
await _service.create(VJCombination.fromJSON(e)).then((_) {
combinationCount++;
progress.value++;
loopCount++;
if (loopCount == combinations.length) {
submitButton.attributes.remove('disabled');
}
});
});
isLoadSuccessful = true;
});
reader.onError.listen((evt) => print(evt));
reader.readAsText(file);
progress.value = 10;
}
I'm getting the progress and the submitButton elements with the #ViewChild annotations:
#ViewChild('progress')
ProgressElement progress;
#ViewChild('submit')
ButtonElement submitButton;
The code works. The progress bar starts off empty, and after the file is read and the service gets the data, the progress bar is full.
My issue is that the UI is only updated after all of the combinations have been sent to the _service. So it seemingly goes from empty to full in one frame.
This code
combinations.forEach((e) async {
does not make sense, because forEach does not care about the returned Future
Rather use
for(var e in combinations) {
Not sure if this fixes your problem, but such code definitely needs to be changed.

.Audio Timeout Error: NET Core Google Speech to Text Code Causing Timeout

Problem Description
I am a .NET Core developer and I have recently been asked to transcribe mp3 audio files that are approximately 20 minutes long into text. Thus, the file is about 30.5mb. The issue is that speech is sparse in this file, varying anywhere between 2 minutes between a spoken sentence or 4 minutes of length.
I've written a small service based on the google speech documentation that sends 32kb of streaming data to be processed from the file at a time. All was progressing well until I hit this error that I share below as follows:
I have searched via google-fu, google forums, and other sources and I have not encountered documentation on this error. Suffice it to say, I think this is due to the sparsity of spoken words in my file? I am wondering if there is a programmatical centric workaround?
Code
I have used some code that is a slight modification of the google .net sample for 32kb streaming. You can find it here.
public async void Run()
{
var speech = SpeechClient.Create();
var streamingCall = speech.StreamingRecognize();
// Write the initial request with the config.
await streamingCall.WriteAsync(
new StreamingRecognizeRequest()
{
StreamingConfig = new StreamingRecognitionConfig()
{
Config = new RecognitionConfig()
{
Encoding =
RecognitionConfig.Types.AudioEncoding.Flac,
SampleRateHertz = 22050,
LanguageCode = "en",
},
InterimResults = true,
}
});
// Helper Function: Print responses as they arrive.
Task printResponses = Task.Run(async () =>
{
while (await streamingCall.ResponseStream.MoveNext(
default(CancellationToken)))
{
foreach (var result in streamingCall.ResponseStream.Current.Results)
{
//foreach (var alternative in result.Alternatives)
//{
// Console.WriteLine(alternative.Transcript);
//}
if(result.IsFinal)
{
Console.WriteLine(result.Alternatives.ToString());
}
}
}
});
string filePath = "mono_1.flac";
using (FileStream fileStream = new FileStream(filePath, FileMode.Open))
{
//var buffer = new byte[32 * 1024];
var buffer = new byte[64 * 1024]; //Trying 64kb buffer
int bytesRead;
while ((bytesRead = await fileStream.ReadAsync(
buffer, 0, buffer.Length)) > 0)
{
await streamingCall.WriteAsync(
new StreamingRecognizeRequest()
{
AudioContent = Google.Protobuf.ByteString
.CopyFrom(buffer, 0, bytesRead),
});
await Task.Delay(500);
};
}
await streamingCall.WriteCompleteAsync();
await printResponses;
}//End of Run
Attempts
I've increased the stream to 64kb of streaming data to be processed and then I received the following error as can be seen below:
Which, I believe, means the actual api timed out. Which is decidely a step in the wrong direction. Has anybody encountered a problem such as mine with the Google Speech Api when dealing with a audio file with sparse speech? Is there a method in which I can filter the audio down to only spoken words progamatically and then process that? I'm open to suggestions, but my research and attempts have only lead me to further breaking my code.
There is to way for recognize audio in Google Speech API:
normal recognize
long running recognize
Your sample is uses the normal recognize, which has a limit for 15 minutes.
Try to use the long recognize method:
{
var speech = SpeechClient.Create();
var longOperation = speech.LongRunningRecognize( new RecognitionConfig()
{
Encoding = RecognitionConfig.Types.AudioEncoding.Linear16,
SampleRateHertz = 16000,
LanguageCode = "hu",
}, RecognitionAudio.FromFile( filePath ) );
longOperation = longOperation.PollUntilCompleted();
var response = longOperation.Result;
foreach ( var result in response.Results )
{
foreach ( var alternative in result.Alternatives )
{
Console.WriteLine( alternative.Transcript );
}
}
return 0;
}
I hope it helps for you.

Clicking or hovering over multiple images in canvas?

I am working on a project for my programming class. I'd like to essentially have a canvas with a background element (say a room.jpg) and then maybe three interactive objects in the room (lamp.jpg, couch.jpg, desk.jpg). I'd like for it to be that if you hover over the lamp a small box or text pops out, giving you some information. Or maybe have it so if you click an image, the same concept happens. You know, something interactive with the objects in the canvas. Again, I'm new to canvas but we have to use it in our assignment. My current code is:
function loadImages(sources, callback) {
var images = {};
var loadedImages = 0;
var numImages = 0;
// get num of sources
for(var src in sources) {
numImages++;
}
for(var src in sources) {
images[src] = new Image();
images[src].onload = function() {
if(++loadedImages >= numImages) {
callback(images);
}
};
images[src].src = sources[src];
}
}
var sources = {
room: 'room.jpg',
title: 'title.jpg'
};
loadImages(sources, function(images) {
context.drawImage(images.room, 0,0);
context.drawImage(images.title, 0,0);
});
}
But from what I understand, it makes the two jpegs a "permanent" canvas element (unable to be messed with). I had been trying to get it so that when I clicked I'd go from the title.jpg to the room.jpg but I've since given up. Essentially, all I want now is just to have the room.jpg appear when the page is first loaded, and have three other png objects on top (as objects in the room). Are these able to be interacted with, or do I have to put the images into the canvas in another way? Thanks for all your help and your patience!
// --- Image Loader ----
var images = {};
var loadedImages = 0;
var pictures = {
room: 'room.jpg',
title: 'title.jpg'
lamp1: 'lampoff.jpg'
lamp2: 'lampon.jpg'
};
function loadImages(sources, callback) {
var numImages = 0;
for(var src in sources)numImages++;
for(var src in sources) {
images[src] = new Image();
images[src].onload = function() {
if(++loadedImages >= numImages) {
callback(images);
}
};
images[src].src = sources[src];
}
}
// --- Mouse Down Functionality ----
$('#canvas').addEventListener('mouseDown', function(e){
if(e.clientX){
var rect = this.getBoundingClientRect();
if(rect) clickCanvas(e.clientX - rect.left, e.clientY - rect.top)
else clickCanvas(e.clientX - this.offsetLeft, e.clientY - this.offsetTop);
}else if(e.offsetX) clickCanvas(e.offsetX, e.offsetY);
else if(e.layerX) clickCanvas(e.layerX, e.layerY);
else console.warn("Couldn't Determine Mouse Coordinates");
})
var lampOn;
function drawCanvas(showLamp){
lampOn = showLamp;
canvas.width = canvas.width //clears canvas
context.drawImage(images.room, 0,0);
context.drawImage(images.title, 0,0);
if(lampOn){
context.drawImage(images.lamp2, 100,100);
}else{
context.drawImage(images.lamp1, 100,100);
}
}
function clickCanvas(x,y){
console.log('clicked canvas at:',x,y)
if(clickedLamp){
drawCanvas(!lampOn)
}
}
loadImages(pictures, function(images) {
drawCanvas(false)
});
Make sure to replace "clickedLamp" and "#canvas"! The idea here is that you redraw the same canvas, using the same function. Everytime you modify any of it, you rerender ALL of it. See if you can get this example working, it will help clarify alot. If you don't understand something comment

Resources