I am trying to implement Blob storage via IndexedDB for long Media recordings.
My code works fine in Chrome and Edge (not tested in Safari yet) - but won't do anything in Firefox. There are no errors, it just doesn't try to fulfill my requests past the initial DB Connection (which is successful). Intuitively, it seems that the processing is blocked by something. But I don't have anything in my code which would be blocking.
Simplified version of the code (without heavy logging and excessive error checks which I have added trying to debug):
const dbName = 'recording'
const storeValue = 'blobs'
let connection = null
const handler = window.indexedDB || window.mozIndexedDB || window.webkitIndexedDB
function connect() {
return new Promise((resolve, reject) => {
const request = handler.open(dbName)
request.onupgradeneeded = (event) => {
const db = event.target.result
if (db.objectStoreNames.contains(storeValue)) {
db.deleteObjectStore(storeValue)
}
db.createObjectStore(storeValue, {
keyPath: 'id',
autoIncrement: true,
})
}
request.onerror = () => {
reject()
}
request.onsuccess = () => {
connection = request.result
connection.onerror = () => {
connection = null
}
connection.onclose = () => {
connection = null
}
resolve()
}
})
}
async function saveChunk(chunk) {
if (!connection) await connect()
return new Promise((resolve, reject) => {
const store = connection.transaction(
storeValue,
'readwrite'
).objectStore(storeValue)
const req = store.add(chunk)
req.onsuccess = () => {
console.warn('DONE!') // Fires in Chrome and Edge - not in Firefox
resolve(req.result)
}
req.onerror = () => {
reject()
}
req.transaction.oncomplete = () => {
console.warn('DONE!') // Fires in Chrome and Edge - not in Firefox
}
})
}
// ... on blob available
await saveChunk(blob)
What I tried so far:
close any other other browser windows, anything that could count as on "open connection" that might be blocking execution
refresh Firefox profile
let my colleague test the code on his own machine => same result
Additional information that might useful:
Running in Nuxt 2.15.8 dev environment (localhost:3000). Code is used in the component as a Mixin. The project is rather large and uses a bunch of different browser APIs. There might be some kind of collision ?! This is the only place where we use IndexedDB, though, so to get to the bottom of this without any errors being thrown seems almost impossible.
Edit:
When I create a brand new Database, there is a brief window in which Transactions complete fine, but after some time has passed/something triggered, it goes back to being queued indefinitely.
I found out this morning when I had this structure:
...
clearDatabase() {
// get the store
const req = store.clear()
req.transaction.oncomplete = () => console.log('all good!')
}
await this.connect()
await this.clearDatabase()
'All good' fired. But any subsequent requests were broken same as before.
On page reload, even the clearDatabase request was broken again.
Something breaks with ongoing usage.
Edit2:
It's clearly connected to saving a Blob instance without an id with the autoIncrement option. Not only does it fail silently, it basically completely corrupts the DB. If I manually assign an incrementing ID to a Blob object, it works! If I leave out the id field for a regular simple object, it also works! Anyone knows about this? I feel like saving blobs is a common use-case so this should have been found already?!
I've concluded, unless proven otherwise, that it's a Firefox bug and opened a ticket on Bugzilla.
This happens with Blobs but might also be true for other instances. If you find yourself in the same situation there is a workaround. Don't rely on autoIncrement and assign IDs manually before trying to save them to the DB.
Related
I am trying to create a Firefox extension to block search terms on school computers. I'd like to prohibit a list of keywords, but the blocking doesn't seem to be working.
I found an example through a plugin gallery here:
https://github.com/mdn/webextensions-examples/blob/master/proxy-blocker/background/proxy-handler.js
This plugin listens to blocked hosts, and then basically returns localhost. I'd like to do the same, but when search terms are added in. I used the code in the example above as a starting point.
Here is the code I have so far:
// Initialize the list of blocked hosts
let blockedHosts = ["www.duckduckgo.com", "www.google.com"];
let blockedTerms = ["games", "minecraft", "legos"];
// Set the default list on installation.
browser.runtime.onInstalled.addListener(details => {
browser.storage.local.set({
blockedHosts: blockedHosts
});
});
// Get the stored list
browser.storage.local.get(data => {
if (data.blockedHosts) {
blockedHosts = data.blockedHosts;
}
});
// Listen for changes in the blocked list
browser.storage.onChanged.addListener(changeData => {
blockedHosts = changeData.blockedHosts.newValue;
});
// Managed the proxy
// Listen for a request to open a webpage
browser.proxy.onRequest.addListener(handleProxyRequest, {urls: ["<all_urls>"]});
function handleProxyRequest(requestInfo) {
let urlToCheck = new URL(requestInfo.documentUrl)
let searchString = urlToCheck.search;
const url = new URL(requestInfo.url);
let found;
blockedTerms.map((term) =>{
if(searchString.search(term) != -1){
found = true
}
})
if ( blockedHosts.indexOf(url.hostname) != -1 & found) {
return {type: "https", host: "127.0.0.1", port: 65535};
}
// Return instructions to open the requested webpage
return {type: "direct"};
}
// Log any errors from the proxy script
browser.proxy.onError.addListener(error => {
console.error(`Proxy error: ${error.message}`);
});
The URL that the browser creates is https://duckduckgo.com/?t=ffab&q=games&ia=web for example. I can determine that the term "games" was found, and that it was found in a duck duck go search, but the proxy wont work and the browser wont stop the user from going to the page.
Any help would be appreciated!
To start with, in a school environment, I suppose they have to use school net connection. It would be a lot easier to block at the main internet connection instead of creating and installing an addon on each computer (that might be altered or bypassed with another browser).
However, to answer your question, the following would be one (simpler) way of doing that using webRequest.onBeforeRequest:
// add a listener for web requests
browser.webRequest.onBeforeRequest.addListener(process, {
urls: ['*://*/*']
},
['blocking']
);
function process(e) {
// e.url is the target url
// no need for storage as the filter-list is hard-coded
const blockedHosts = ['www.duckduckgo.com', 'www.google.com'];
const blockedTerms = ['games', 'minecraft', 'legos'];
const hostRegExp = new RegExp(`^https?://(${blockedHosts.join('|')})/`, 'i');
const termRegExp = new RegExp(`(${blockedTerms.join('|')})`, 'i');
// if matches above criteria, redirect to 127.0.0.1
if (hostRegExp.test(e.url) && termRegExp.test(e.url)) {
return {redirectUrl: 'https://127.0.0.1:65535/'};
}
}
I'm trying to add some additional attributes for new user through cloud code:
Parse.Cloud.beforeSave(Parse.User, (request) => {
if (!request.original) {
// New user
Parse.Config.get()
.then((config) => {
const ProfileIcon = Parse.Object.extend("ProfileIcon");
const iconId = config.get("defaultProfileIcon");
const user = request.object;
// ...many user.set
user.set("profileIcon", ProfileIcon.createWithoutData(iconId), {
useMasterKey: true,
}); // Pointer
// This will save as expected, but cause recursion
// user.save({ useMasterKey: true });
})
.catch((err) => {
console.error(err);
});
}
});
The code above triggered and executed without any error, but when I check the database, none of my custom attributes show up. Passing the master key also does nothing. How can I fix this?
Or is it because the request from the client (Android, have no access to master key), if so then how can I set master key for the request, since Parse.Cloud.useMasterKey() is deprecated?
Here, modifying object on save does not mention anything about return, but before save file does. So I thought I would give it a try, turns out it worked. Still not sure if this is the right way though.
I have a service worker that is supposed to cache an offline.html page that is displayed if the client has no network connection. However, it sometimes believes the navigator is offline even when it is not. That is, navigator.onLine === false. This means the user may get offline.html instead of the actual content even when online, which is obviously something I'd like to avoid.
This is how I register the service worker in my main.js:
// Install service worker for offline use and caching
if ('serviceWorker' in navigator) {
navigator.serviceWorker.register('/service-worker.js', {scope: '/'});
}
My current service-worker.js:
const OFFLINE_URL = '/mysite/offline';
const CACHE_NAME = 'mysite-static-v1';
self.addEventListener('install', (event) => {
event.waitUntil(
// Cache the offline page when installing the service worker
fetch(OFFLINE_URL, { credentials: 'include' }).then(response =>
caches.open(CACHE_NAME).then(cache => cache.put(OFFLINE_URL, response)),
),
);
});
self.addEventListener('fetch', (event) => {
const requestURL = new URL(event.request.url);
if (requestURL.origin === location.origin) {
// Load static assets from cache if network is down
if (/\.(css|js|woff|woff2|ttf|eot|svg)$/.test(requestURL.pathname)) {
event.respondWith(
caches.open(CACHE_NAME).then(cache =>
caches.match(event.request).then((result) => {
if (navigator.onLine === false) {
// We are offline so return the cached version immediately, null or not.
return result;
}
// We are online so let's run the request to make sure our content
// is up-to-date.
return fetch(event.request).then((response) => {
// Save the result to cache for later use.
cache.put(event.request, response.clone());
return response;
});
}),
),
);
return;
}
}
if (event.request.mode === 'navigate' && navigator.onLine === false) {
// Uh-oh, we navigated to a page while offline. Let's show our default page.
event.respondWith(caches.match(OFFLINE_URL));
return;
}
// Passthrough for everything else
event.respondWith(fetch(event.request));
});
What am I doing wrong?
navigator.onLine and the related events can be useful when you want to update your UI to indicate that you're offline and, for instance, only show content that exists in a cache.
But I'd avoid writing service worker logic that relies on checking navigator.onLine. Instead, attempt to make a fetch() unconditionally, and if it fails, provide a backup response. This will ensure that your web app behaves as expected regardless of whether the fetch() fails due to being offline, due to lie-fi, or due to your web server experiencing issues.
// Other fetch handler code...
if (event.request.mode === 'navigate') {
return event.respondWith(
fetch(event.request).catch(() => caches.match(OFFLINE_URL))
);
}
// Other fetch handler code...
I am using Puppeteer to take screenshots of a web page for my company. I need to test multiple people's accounts so that means visiting the page multiple times (150 times in this case). This results in our firewall kicking me out for making too many requests.
My solution is to just fetch the contents of the page and save them locally. Then I use puppeteer on that local file, overriding the function used to get data from our servers to instead just use data already loaded into Node from a CSV.
All of this works, but it looks like it's still making requests to our servers.
I tried giving it a userDataDir so it could cache any resources. In theory, if it's loading it from file://, it's caching the resources and there's no Ajax requests, it shouldn't be making any further requests, right?
I also tried installing a debugging proxy but since it's https I can't see what it's trying to request.
This is how I start it:
puppeteer.launch({
userDataDir: "temp/"
})
.then(browser => {
next(browser, links);
)
.catch(error => {
cb(error, null);
});
next will iterate through any links it needs to visit.
This part saves the page locally:
if (this._linkCache[baseLink] === undefined) {
fetch(baseLink)
.then(resp => resp.text())
.then(contents => {
fs.writeFile(fullFileName, contents, 'utf8', err => {
if (err) {
cb(err, null);
} else {
this._linkCache[baseLink] = fileUrl;
gotoPage(fileUrl);
}
});
})
.catch(error => {
cb(error, null);
});
}
// Go to the cached version
else {
gotoPage(this._linkCache[baseLink] + queryParams);
}
And this gets the screenshots:
const gotoPage = async(url) => {
try {
const page = await browser.newPage();
// Override 'fetchAccountData' function
await page.evaluateOnNewDocument(testData => {
window["fetchAccountData"] = (cb: (err: any, data: any)=>void) => {
cb(null, testData);
};
}, data);
// Go to page and get screenshot
await page.goto(url);
const screenie = `${outputPath}${uuid()}.png`;
await page.screenshot({ fullPage: true, path: screenie, type: "png" });
pageHtml.push(`<img src="file://${screenie}" />`);
next(browser, rest);
} catch (e) {
cb(e, null);
}
};
I was hoping this would be able to only make a few requests at the beginning while it saves the html locally and caches all the resources, but it seems to make a request for every link.
How can I stop it?
Good Day! I would like to implement a convenient method for uploading a multiple files to an sftp-server with methods of calling back each ofuploaded files to server.
I have already tried to implement some code that works, but I saw that there is a memory leak that does not allow to successfully close the connection to the sftp server server after all download.
it is absolutely not critical to constantly open the connection and close it for me.
I tweaked the code a little bit from here: how do I send (put) multiple files using nodejs ssh2-sftp-client?
code:
function sftpPutFiles(config, files, pathToDir, callbackStep, callbackFinish, callbackError) {
let Client = require('ssh2-sftp-client');
let PromisePool = require('es6-promise-pool');
const sendFile = (config, pathFrom, pathTo) => {
return new Promise(function (resolve, reject) {
let sftp = new Client();
console.log(pathFrom, pathTo);
sftp.on('keyboard-interactive', (name, instructions, instructionsLang, prompts, finish) => { finish([config.password]); });
sftp.connect(config).then(() => {
return sftp.put(pathFrom, pathTo);
}).then(() => {
console.log('finish '+pathTo);
callbackStep(pathTo);
sftp.end();
resolve(pathTo);
}).catch((err) => {
console.log(err, 'catch error');
callbackError(err);
});
});
};
// Create a pool.
let indexFile = 0;
let pool = new PromisePool(() => {
while (indexFile < files.length) {
let file = files[indexFile];
indexFile++;
return sendFile(config, file.path, `${pathToDir}/${file.name}`);
}
return null;
}, 10);
pool.start().then(function () {
console.log({"message":"OK"}); // res.send('{"message":"OK"}');
callbackFinish();
});
}
using
input.addEventListener('change', function (e) {
e.preventDefault();
sftpPutFiles(
{host: '192.168.2.201', username: 'crestron', password: 'ehAdmin'},
this.files,
`./Program01/test/`,
pathTo => {
let tr = document.createElement('tr');
let bodyTable = document.querySelector('.body');
tr.innerHTML = `<td>${bodyTable.children.length+1}</td><td>${pathTo}</td><td>OK</td>`;
bodyTable.appendChild(tr);
}, () => {
alert('Всё файлы загружены');
},
err => {
alert('Ошибка: '+err);
}
);
});
If there is an error uploading the file to the sftp server, the connection does not close and I cannot reconnect when I open the custom console. I would like to translate the code to Rxjs to better support and I think I can solve the problem of closing the connection and responsiveness of the application.
make sure your using the latest version of ssh2-sftp-client - there has been a fair amount of updates recently, including fixes to handle errors more consistently and ensure connections are closed correctly. (v4.1.0).
You are using sftp.on('keyboard-interaction', ...). There is nothing which emits events of this type in the module, so this listener will not fire.
If you just want to upload files, use the fastPut() method. It is much faster. Make sure the destination path includes the remote file name and not just the remote directory.
Have a look at Promise.all(). You could use this instead of the promise-pool and I think it would be a lot cleaner. Something like (untested)
let localPath = '/path/to/src-dir';
let remotePath = '/path/to/dst-dir';
let files = ['file1.txt', file2.txt','file3.txt'];
let client = new Client();
client.connect(config)
.then(() => {
let promises = [];
files.forEach(f => {
let from = path.join(localPath, f);
let to = path.join(remotePath, f);
promise.push(client.fastPut(from, to));
});
return Promise.all(promises);
}).then(res => { // res is array of resoved promise results
client.end();
}).catch(err => {
// deal with error
});