Puppeteer - how to iterate through queryObjects to collect the url of a WebSocket object? - websocket

I am using Puppeteer in a Node.js module. I retrieve a WebSocket object Prototype with queryObjects and I need to extract the url property.
// Get a handle to the websocket object prototype
const prototypeHandle = await page.evaluateHandle(() => WebSocket.prototype);
// Query all websocket instances into a jsHandle object
const jsHandle = await page.queryObjects(prototypeHandle);
// Count amount of map objects in heap
// const count = await page.evaluate(maps => maps.length, jsHandle); // returns the expected amount (x2)
// How to iterate through jsHandle to collect the url of each websockets
await jsHandle.dispose();
await prototypeHandle.dispose();

You do not get any response because WebSocket is not a simple JSON object which can be stringified and given back to you when you evaluate using page.evaluate.
To get the URL of the connected websocket in the page, you can map through the collected WebSocket instances/objects and extract the url out of them.
const browser = await puppeteer.launch();
const page = (await browser.pages())[0];
// create a dummy websocket connection for testing purpose
await page.evaluate(() => new WebSocket('wss://echo.websocket.org/'));
const wsPrototypeHandle = await page.evaluateHandle(
() => WebSocket.prototype
);
const wsInstances = await page.queryObjects(wsPrototypeHandle);
const wsUrls = await page.evaluate(
(e) => e.map((e) => e['url']), // <-- simply access the object here
wsInstances
);
console.log(wsUrls);
Which will result in following,
[ 'wss://echo.websocket.org/' ]

Related

How to control a websocket with Chrome DevTools?

I'm currently working on capturing crypto-currency price infos from OKEX. I found that, when a page navigate to https://www.okex.com/trade-spot/eth-usdt, it will initiate a websocket named public and send subscription orders via this websocket. Then, data corresponding sent subscriptions will flow through the same websocket.
My question is, in addition to passively inspecting dataflows in this websocket, is there any way to control it, namely to send subscriptions? If so, can this approch be automated(by puppeteer or something equivalent)?
After some research and learning, I solved this problem using evaluateHandle() and queryObjects(). However communication via selected websockets must be done in DOM context.
const puppeteer = require('puppeteer');
async function unsubscrible(page, wsHandle){
page.evaluate(arr=>{
arr[0].send(JSON.stringify({"op": "unsubscribe","args":[{"channel":"instruments","instType":"SPOT"},{"channel":"instruments","instType":"FUTURES"},{"channel":"instruments","instType":"SWAP"},{"channel":"instruments","instType":"OPTION"},{"channel":"tickers","instId":"ETH-USDT"},{"channel":"cup-tickers-3s","ccy":"USDT"},{"channel":"mark-price","instId":"ETH-USDT"},{"channel":"index-tickers","instId":"ETH-USDT"},{"channel":"itn-status"},{"channel":"optimized-books","instId":"ETH-USDT"},{"channel":"trades","instId":"ETH-USDT"}]}));
}, wsHandle);
};
async function ping(page, wsHandle){
page.evaluate(arr=>{
arr[0].send('ping');
}, wsHandle);
}
async function main() {
const browser = await puppeteer.launch({
headless : false,
args : [
'--auto-open-devtools-for-tabs',
]
});
const page = (await browser.pages())[0];
page.setDefaultNavigationTimeout(0);
await page.goto('https://www.okex.com/trade-spot/eth-usdt');
const wsHandle = await page.evaluateHandle(()=>WebSocket.prototype);
const ws = await page.queryObjects(wsHandle);
await unsubscrible(page, ws);
setTimeout(()=>{
for (i=0; i<10; i++) ping(page, ws);
}, 20000)
}
main();

Why is my koa application timing out when testing a route with Jest/Supertest

Summary of Problem
Receiving : Timeout - Async callback was not invoked within the 5000ms timeout specified by jest.setTimeout.Timeout when trying to run a test with Jest and supertest.
Specs
Koa2 project, Jest/Supertest testing, Babel 7.9.0 recommended configuration
What I've tried
I have a simple test from the same file running which I omitted from the code below for brevity. I've also tried sending an HTTP request from the browser - this file is imported & 'listen'ed in a server file. The request is failing because it is blocked by a CORS policy - I think this is a problem for another day and isn't affecting my test timing out.
I also tried removed .callback() from the supertest(..) call:
const response = await supertest(app).post('/save-material');
at which point I get TypeError: app.dress is not a function.
Here is the content of my test file:
process.env.NODE_ENV = 'test';
const app = require('../../src/server/app.js')
const supertest = require('supertest')
test('save-material returns response', async() => {
const response = await supertest(app.callback()).post('/save-material');
expect(response.status).toBe(200);
expect(response.body.status).toBe('success');
expect(response.body.msg).toBe('Material saved')
});
Here is the content of the imported file (app.js) from above:
require('#babel/register'); // not entry point - but is entry point for some tests
const Koa = require('koa');
var Router = require('koa-router')
const app = new Koa();
const router = new Router();
router
.post('/save-material', async(ctx) => {
ctx.response = {
status: 'success',
msg: 'Material saved'
}
return ctx;
})
app.use(router.routes());
app.use(router.allowedMethods());
module.exports = app;

How can I pass values between koa-router routes

I wanted to move the authentication procedure from all routes into one route (koa-router provides the all() middleware for all methods on a router for this). However, in the process, I decode a token whose decoding I need for further execution. How can I access this decoded token from another route?
const Router = require('koa-router');
const router = new Router({ prefix: '/test' });
router.all('/', async (ctx, next) => {
//decode
await next();
})
router.get('/', async ctx=> {
// Here I need to access decoded, too
});
the Koa Context object encapsulates the request, response and a state object, along with much more. This state object is the recommended namespace where you can pass data between middleware.
Modifiying the provided example gets:
const http = require('http')
const Koa = require('koa')
const Router = require('koa-router')
const app = new Koa()
const router = new Router({ prefix: '/test' })
router.all('/', async (ctx, next) => {
// decode token
const x = 'foo'
// assign decoded token to ctx.state
ctx.state.token = x
await next()
})
router.get('/', async ctx=> {
// access ctx.state
console.log(ctx.state.token)
})
app.use(router.routes())
http.createServer(app.callback()).listen(3000)
Navigate to http://localhost:3000/test and see the decoded token logged to the console.

How to send data to WebSocket using Puppeteer

I use puppeteer-sharp to dump data received and send by page via websockets. The code to dump data in C#:
async Task Dump()
{
var client = await _browser.Page.Target.CreateCDPSessionAsync();
await client.SendAsync("Network.enable");
client.MessageReceived += OnChromeDevProtocolMessage;
}
void OnChromeDevProtocolMessage(object sender, MessageEventArgs eventArgs)
{
if (eventArgs.MessageID == "Network.webSocketCreated")
{
Logger.Trace($"Network.webSocketCreated: {eventArgs.MessageData}");
}
else if (eventArgs.MessageID == "Network.webSocketFrameSent")
{
Logger.Trace($"Network.webSocketFrameSent: {eventArgs.MessageData}");
}
else if (eventArgs.MessageID == "Network.webSocketFrameReceived")
{
var cdpMessage = JsonConvert.DeserializeObject<CdpMessage>(eventArgs.MessageData.ToString());
ProcessMessage(cdpMessage);
}
}
Is there any way to send data to websockets using puppeteer or using directly Chrome Dev Protocol messages?
EDIT:
Or is it possible to get somehow WebSocket instanse (or handle) to use it in JavaScript code to send data using EvaluateFunctionAsync?
QueryObjects can be used in the command line API and also in Puppeteer in order to find the instance. After that you just use EvaluateFunction, to execute send method on object. In PuppeteerSharp it looks something like this:
//CODE SNIPPET
var prototype = await page.EvaluateExpressionHandleAsync("WebSocket.prototype");
var socketInstances = await page.QueryObjectsAsync(prototype);
await page.EvaluateFunctionAsync("(instances) => {let instance = instances[0]; instance.send('Hello')}, new[] {socketInstances}");
More information can be found in the documentation.
in case someone comes to this question looking for an answer in javascript:
const prototype = await page.evaluateHandle("WebSocket.prototype");
const socketInstances = await page.queryObjects(prototype);
await page.evaluate((instances) => {
let instance = instances[0];
instance.send('Hello');
}, socketInstances);
and for ViniCoder in the comments, pyppeteer in python:
prototype = await page.evaluateHandle("WebSocket.prototype")
socketInstances = await page.queryObjects(prototype)
await page.evaluate('''(instances) => {
let instance = instances[0];
instance.send('Hello');
}''', socketInstances)

How to make make a google request return the results in a certain language?

I'm using Puppeteer to make the requests (it uses Chromium in the background):
;(async () => {
const browser = await puppeteer.launch()
const page = await browser.newPage()
await page.goto(`https://www.google.com/search?tbm=bks&q=%22this+is%22`)
const result = await page.evaluate(() => {
const stats = document.querySelector('#resultStats')
return stats.textContent
})
await browser.close()
res.send(result)
})()
Since I live in Taiwan, the results are being returned in Chinese.
How can I modify the URL or puppeteer's config so I get results in English?
Just add &hl=en in url:
await page.goto(`https://www.google.com/search?tbm=bks&q=%22this+is%22&hl=en`)

Resources