I'm using Puppeteer to make the requests (it uses Chromium in the background):
;(async () => {
const browser = await puppeteer.launch()
const page = await browser.newPage()
await page.goto(`https://www.google.com/search?tbm=bks&q=%22this+is%22`)
const result = await page.evaluate(() => {
const stats = document.querySelector('#resultStats')
return stats.textContent
})
await browser.close()
res.send(result)
})()
Since I live in Taiwan, the results are being returned in Chinese.
How can I modify the URL or puppeteer's config so I get results in English?
Just add &hl=en in url:
await page.goto(`https://www.google.com/search?tbm=bks&q=%22this+is%22&hl=en`)
Related
I am using Puppeteer in a Node.js module. I retrieve a WebSocket object Prototype with queryObjects and I need to extract the url property.
// Get a handle to the websocket object prototype
const prototypeHandle = await page.evaluateHandle(() => WebSocket.prototype);
// Query all websocket instances into a jsHandle object
const jsHandle = await page.queryObjects(prototypeHandle);
// Count amount of map objects in heap
// const count = await page.evaluate(maps => maps.length, jsHandle); // returns the expected amount (x2)
// How to iterate through jsHandle to collect the url of each websockets
await jsHandle.dispose();
await prototypeHandle.dispose();
You do not get any response because WebSocket is not a simple JSON object which can be stringified and given back to you when you evaluate using page.evaluate.
To get the URL of the connected websocket in the page, you can map through the collected WebSocket instances/objects and extract the url out of them.
const browser = await puppeteer.launch();
const page = (await browser.pages())[0];
// create a dummy websocket connection for testing purpose
await page.evaluate(() => new WebSocket('wss://echo.websocket.org/'));
const wsPrototypeHandle = await page.evaluateHandle(
() => WebSocket.prototype
);
const wsInstances = await page.queryObjects(wsPrototypeHandle);
const wsUrls = await page.evaluate(
(e) => e.map((e) => e['url']), // <-- simply access the object here
wsInstances
);
console.log(wsUrls);
Which will result in following,
[ 'wss://echo.websocket.org/' ]
Hi I'm build a wildfire app tracker with react using the nasa API it works in development by using the url directly witch is https://eonet.sci.gsfc.nasa.gov/api/v2.1/events
But when I deploy it. It does not get the data. I obviously need a api key witch I have, but how do I implement it in the url above ?
here is my code..
useEffect(() => {
const fetchEvents = async () => {
setLoading(true)
const res = await fetch('https://eonet.sci.gsfc.nasa.gov/api/v2.1/events')
const {events} = await res.json()
setEventData(events)
setLoading(false)
}
fetchEvents()
// eslint-disable-next-line
}, [])
You could try to create a .env file in which you can set URLS as
REACT_APP_PUBLIC_URL=https://eonet.sci.gsfc.nasa.gov/api/v2.1/events
and then in your app component import as
fetch(process.env.REACT_APP_PUBLIC_URL)
I'm writing a test that uses puppeteer to test a component that copies something to the clipboard when it is interacted with. I want to test that the contents of the clipboard are correct after interacting. Other resources like this github issue mention using a tool like clipboardy to accomplish this. I tried using that and it works locally, but when run in my headless CI server, it complains about not having access to the X environment. Is there a way to access the clipboard without starting an X server?
I'm writing a test like this:
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://mypage.com');
await page.click('#my-component');
// This fails in a headless environment
expect(clipboardy.readSync()).toEqual("Some text");
By adding the 'clipboard-read' permission to puppeteer and using the
Clipboard API, you can run navigator.clipboard.readText() to read from the
clipboard in a test. This will work even in a headless environment:
const browser = await puppeteer.launch();
const context = browser.defaultBrowserContext();
context.overridePermissions(/* browser origin */, ['clipboard-read'])
const page = await browser.newPage();
await page.goto('https://mypage.com');
await page.click('#my-component');
expect(await page.evaluate(() => navigator.clipboard.readText()))
.toEqual("Some text");
Documentation of context.overridePermissions()
In my case I couldn't override permissions, as Eduard suggested, because it requires to give origin as a parm. I'm injecting html content in my test to the page via setContent so the page address is about:blank. Setting origin to "*" doesn't work either, nor undefined.
I have ended up mocking the clipboard api:
await page.evaluate((dataInternal) => {
// mock clipboard
let clipboardText = null;
window["navigator"]["clipboard"] = {
writeText: text => new Promise(resolve => clipboardText = text),
readText: () => new Promise(resolve => resolve(clipboardText)),
}
}
then you can just do the following assert:
expect(await page.evaluate(() => navigator.clipboard.readText())).toBe("your text in clipboard");
In my chrome, navigator.clipboard = ... did not work, because navigator had getter and setter for clipboard.
Mocking clipboard:
await page.evaluate(() => {
const clipboard = {
async writeText(text: string) {
this.text = text;
},
};
Object.defineProperty(navigator, 'clipboard', { value: clipboard });
});
Reading clipboard content:
page.evaluate(() => navigator.clipboard.text);
I use puppeteer-sharp to dump data received and send by page via websockets. The code to dump data in C#:
async Task Dump()
{
var client = await _browser.Page.Target.CreateCDPSessionAsync();
await client.SendAsync("Network.enable");
client.MessageReceived += OnChromeDevProtocolMessage;
}
void OnChromeDevProtocolMessage(object sender, MessageEventArgs eventArgs)
{
if (eventArgs.MessageID == "Network.webSocketCreated")
{
Logger.Trace($"Network.webSocketCreated: {eventArgs.MessageData}");
}
else if (eventArgs.MessageID == "Network.webSocketFrameSent")
{
Logger.Trace($"Network.webSocketFrameSent: {eventArgs.MessageData}");
}
else if (eventArgs.MessageID == "Network.webSocketFrameReceived")
{
var cdpMessage = JsonConvert.DeserializeObject<CdpMessage>(eventArgs.MessageData.ToString());
ProcessMessage(cdpMessage);
}
}
Is there any way to send data to websockets using puppeteer or using directly Chrome Dev Protocol messages?
EDIT:
Or is it possible to get somehow WebSocket instanse (or handle) to use it in JavaScript code to send data using EvaluateFunctionAsync?
QueryObjects can be used in the command line API and also in Puppeteer in order to find the instance. After that you just use EvaluateFunction, to execute send method on object. In PuppeteerSharp it looks something like this:
//CODE SNIPPET
var prototype = await page.EvaluateExpressionHandleAsync("WebSocket.prototype");
var socketInstances = await page.QueryObjectsAsync(prototype);
await page.EvaluateFunctionAsync("(instances) => {let instance = instances[0]; instance.send('Hello')}, new[] {socketInstances}");
More information can be found in the documentation.
in case someone comes to this question looking for an answer in javascript:
const prototype = await page.evaluateHandle("WebSocket.prototype");
const socketInstances = await page.queryObjects(prototype);
await page.evaluate((instances) => {
let instance = instances[0];
instance.send('Hello');
}, socketInstances);
and for ViniCoder in the comments, pyppeteer in python:
prototype = await page.evaluateHandle("WebSocket.prototype")
socketInstances = await page.queryObjects(prototype)
await page.evaluate('''(instances) => {
let instance = instances[0];
instance.send('Hello');
}''', socketInstances)
Why do we do this
router.get('/data', async (ctx, next) => {
ctx.body = dummyjson.parse(data);
await next();
});
router.get('/data/:x', async (ctx, next) => {
const newData = dataRepeat.replace('%(x)', ctx.params.x);
ctx.body = dummyjson.parse(newData);
await next();
});
What is the use of await next()
It would work just fine without that. Similar thing was expected with koa 1. yield next was added at the end of the router.
I'll try to explain it using a very simple example:
const Koa = require('koa');
const app = new Koa();
// middleware
app.use(async function (ctx, next) {
console.log(1)
await next();
console.log(3)
});
// response
app.use(ctx => {
console.log(2)
});
app.listen(3000);
If you call localhost:3000 in your browser, the following will happen in your app:
The first app.use that you fired here was the middleware. So the request flow goes into that one first, logs 1to the console.
Then, when you see this await next(), it downstreams to the next use.
Here we just log 2 to the console. When this is finished (and no further await next is seen in the second use) the flow goes back to the first one which actually waited till the second one was finished.
Here we then continue with logging 3 to the console.
Hope this makes it a little more clear.
No, It is not necessary. It is depend on your requirement.
you use next() function when you call next middleware.
Check your router module and its version. I have use koa-router module and its version is 7.2.0 for routing. It self handle await next.
'use strict';
const Koa = require('koa'),
router = require('koa-router'),
app = new Koa();
let pubRouter = new router();
let securedRouter = new router();
let mapper = require('./mapper'),
// assign router to corresponding function
mapper(pubRouter, securedRouter);
app .use(logger(config.app.log))
.use(bodyParser())
.use(pubRouter.routes()).use(pubRouter.allowedMethods())
.use(jwt({
secret: publicKey,
algorithms: ['RS256']
}))
.use(async(ctx, next) => {
console.log('\n\n\n\n\n', ctx.state.user);
await next();
})
.use(securedRouter.routes()).use(securedRouter.allowedMethods())
.use(async(ctx, next) => {
ctx.body = 'Invalid URL!!!';
});
app.listen(port, () => console.log(`Server listening on port: ${port}`));