I am trying to traverse a state database of court cases for homicides and other violent crimes research, using Casper.JS
The database works by the client first agreeing to Terms of Service via submitting an HTML form. After that a second form is presented in which you can enter in search terms to find up to 500 records in an HTML table.
This state database/site is very unreliable and often breaks down. The goal is to traverse it with Casper and host the data independently for a client.
My problem is that I can't get basic Casper tasks to work, besides loading the first form. It appears to be navigating, hits an about:blank and then stops. It doesn't log at critical junctures even though it is coded to do so.
The script I am running:
var casper = require('casper').create({
clientScripts: [
'jquery.min.js', // These two scripts will be injected in remote
'moment.min.js' // DOM on every request
],
pageSettings: {
loadImages: false, // The WebPage instance used by Casper will
loadPlugins: false // use these settings
},
logLevel: "debug",
verbose: true // log messages will be printed out to the console
});
casper.start('http://casesearch.courts.state.md.us/casesearch/processDisclaimer.jis', function(status) {
var y = (this)
$("input[name='disclaimer']").prop('checked', true);
$("input[name='action']").click();
casper.log(y, 'info');
setTimeout(function() {
y.download('http://casesearch.courts.state.md.us/casesearch/inquiry-results.jsp?middleName=&partyType=&lastName=T&filingEnd=&filingDate=&site=00&filingStart=&d-16544-p=20&countyName=&action=Search&courtSystem=B&firstName=N&company=N', 'phantom.csv')
}, 3000);
});
casper.run();
The output I am getting:
[info] [phantom] Starting...
[info] [phantom] Running suite: 2 steps
[debug] [phantom] opening url: http://casesearch.courts.state.md.us/casesearch/processDisclaimer.jis, HTTP GET
[debug] [phantom] Navigation requested: url=http://casesearch.courts.state.md.us/casesearch/processDisclaimer.jis, type=Other, willNavigate=true, isMainFrame=true
[debug] [phantom] url changed to "http://casesearch.courts.state.md.us/casesearch/processDisclaimer.jis"
[debug] [phantom] Automatically injected jquery.min.js client side
[debug] [phantom] Automatically injected moment.min.js client side
[debug] [phantom] Successfully injected Casper client-side utilities
[info] [phantom] Step anonymous 2/2 http://casesearch.courts.state.md.us/casesearch/processDisclaimer.jis (HTTP 200)
[debug] [phantom] Navigation requested: url=about:blank, type=Other, willNavigate=true, isMainFrame=true
[debug] [phantom] url changed to "about:blank"
Can someone instruct me as to why I'm not getting my console.log(y) and why I have no download or error message? It just terminates once that output is given.
I should also mention that I believe the download will only work if the database server knows that I accepted the Terms, as in, I need to send my cookies to it in my download request. How does Casper handle that?
Thank you.
Related
I have a web application to perform the update operation with the help of a Update button in UI mode.
During that time list of APIs are loaded into the Network tab with XHR type as below. I have to verify one of the API put call is triggered and the status is passed.
url.toString() -- https://abcdef.execute-api.ue-east-2.amazonaws.com/stg2
It contains the RequestedURL value. I manually verified that in Network tab for that particular API put call. Then kept that in cypress.json and reading the url to the current class
Base URL for the UI operation: https://abc-stg2.awsdev.fgh.com/
Note: The both url are modified and dummy one, since it is confidential to share. The understanding is API calls are in AWS and the UI urls are in other environment
Try #1
// After the UI operation
cy.intercept('PUT',url.toString()).as('urupdate')
cy.wait('#urupdate',{requestTimeout:20000})
.its('status')
.should('be.eq',200)
Output:
Try #2
cy.intercept('PUT', url.toString(), (req) => {
if (req.body.status == 200) {
cy.log('pass')
}
})
Output:
The log is not getting printed and till the if statement everything is getting passed
How can we verify the particular API is triggered and the status is 200?
I have gone through the intercept as above and other stuffs in Cypress. But that does not get me the solution. Share your suggestions
I am new to the service workers and trying to develop one to take care of background image uploading. I am using Workbox and firefox for testing. The service worker is loaded and registered correctly and whenever I try to upload an image offline these logs appear in the console:
workbox Request for '/photoUpload' has been added to background sync queue 'PhotoQueue'
workbox Using NetworkOnly to respond to '/photoUpload'
after some seconds before I get online, the following are printed in the log, and the photo is not uploaded to the server:
workbox Background sync replaying without background sync event
workbox Request for '/photoUpload' has been replayed in queue 'PhotoQueue'
workbox All requests in queue 'PhotoQueue' have successfully replayed; the queue is now empty!
here is my serviceWorker.js:
const showNotification = () => {
self.registration.showNotification('Post Sent', {
body: 'You are back online and your post was successfully sent!',
});
};
const bgSyncPlugin = new workbox.backgroundSync.Plugin('PhotoQueue', {
maxRetentionTime: 24 * 60, // Retry for max of 24 Hours
callbacks: {
queueDidReplay: showNotification
}
});
workbox.routing.registerRoute(
new RegExp('/photoUpload'),
new workbox.strategies.NetworkOnly({
plugins: [
bgSyncPlugin
]
}),
'POST'
);
is there a way that I can trigger the background sync event? why the workbox removing the POST request from the Queue before the image is uploaded to the server.
Firefox does not support the Background Sync API natively. workbox-background-sync will attempt to "polyfill" this missing API by automatically retrying the queue whenever the service worker starts up.
Chrome allows you to trigger the background sync event via its DevTools, but as mentioned, Firefox does not. There is no programmatic way to force a service worker to stop and then start again using DevTools in Firefox (as far as I know).
Are you sure that the photo isn't being uploaded to the server? Do you see anything in the Network panel of Firefox's DevTools corresponding to the upload attempt?
Using Cypress to test a web app, I have tests failing because of an unexpected XHR response from our test backend.
The screenshots and screencasts doesn't help understanding why we get this error message from the webapp. It would be very useful to get the actual XHR requests logs as an artifact.
It seems to be possible to capture some routes using cy.route, but it seems to be more suitable to stub requests.
What is the correct way to capture and write the XHR logs alongside the screenshots and videos ? It would be even better if it would delete the file if everything passes.
To record the network traffic in the same manner like the network tab in devtools, you can utilize the HAR file format (http://www.softwareishard.com/blog/har-12-spec/). To generate a HAR file during the execution of your Cypress tests you can use the cypress-har-generator plugin.
describe('my tests', () => {
before(() => {
// start recording
cy.recordHar();
});
after(() => {
// save the HAR file
cy.saveHar({ waitForIdle: true });
});
it('does something', () => {
cy.visit('https://example.com');
// ...
});
});
It will save a HAR that can be used to inspect the network activity, identify the performance issues or troubleshooting bugs. The file can be skimmed using any kind of viewer (e.g. https://developer.chrome.com/blog/new-in-devtools-76/#HAR).
I am using OneSignal in my Laravel/Vue app. I have included it within <head> as stated in documentation:
<script src="https://cdn.onesignal.com/sdks/OneSignalSDK.js" async=""></script>
<script>
var OneSignal = window.OneSignal || [];
OneSignal.push(function() {
OneSignal.init({
appId: "{{ env('ONESIGNAL_APP_ID') }}"
});
OneSignal.showNativePrompt();
});
</script>
<script>
if ('serviceWorker' in navigator) {
navigator.serviceWorker.register('/OneSignalSDKWorker.js')
.then(function () {
console.log('Service worker registered');
})
.catch(function (error) {
console.log('Service worker registration failed:', error);
});
} else {
console.log('Service workers are not supported.');
}
</script>
I also have a service worker of my own, so I've followed the documentation here as well.
What is happening after a hard reset is that service worker gets installed and it is all fine, however once I refresh the page I am getting:
OneSignalPageSDKES6.js?v=151102:1 Uncaught (in promise) InvalidStateError: The current environment does not support this operation.
at Function.getServiceWorkerHref (https://cdn.onesignal.com/sdks/OneSignalPageSDKES6.js?v=151102:1:41510)
at xe. (https://cdn.onesignal.com/sdks/OneSignalPageSDKES6.js?v=151102:1:144028)
at Generator.next ()
at r (https://cdn.onesignal.com/sdks/OneSignalPageSDKES6.js?v=151102:1:716)
And I have no idea what does that mean? What is "current environment"? Where to start debugging? I've tried putting console logs around it, however it led me nowhere...
You would start debugging by looking at the source code of the library.
In your case your library is the OneSignal SDK for browsers.
Let's do this!!!
We can see that this error is thrown by getServiceWorkerHref function (which is defined here) and the error message is driven by the InvalidStateReason enumeration:
case InvalidStateReason.UnsupportedEnvironment:
super(`The current environment does not support this operation.`);
break;
If you look at the first linked file, you will see the note on getServiceWorkerHref OneSignal developers left for the those who dare venture into their source code:
else if (workerState === ServiceWorkerActiveState.Bypassed) {
/*
if the page is hard refreshed bypassing the cache, no service worker
will control the page.
It doesn't matter if we try to reinstall an existing worker; still no
service worker will control the page after installation.
*/
throw new InvalidStateError(InvalidStateReason.UnsupportedEnvironment);
}
As you can see, the error is raised when the service worker has the "Bypassed" state. What is that, you may ask? Let's look at ServiceWorkerActiveState enumeration below, in the same file:
/**
* A service worker is active but not controlling the page. This can occur if
* the page is hard-refreshed bypassing the cache, which also bypasses service
* workers.
*/
Bypassed = 'Bypassed',
It seems, when the browser "hard-refreshes" the page, it bypasses the service worker and OneSignal can't properly initialize when that happens. Hard-refresh can happen for a number of reasons — here are some of them (to the best of my knowledge):
if you click the refresh button a bunch of times (usually seconds consecutive refresh within a short period of time may trigger this)
if you have caching disabled in your DevTools
if the server sets a no-cache header
What is happening after a hard reset
I don't know exactly what you mean by "hard reset", but that sounds like it would trigger this issue. I would suggest you close your browser and then visit the page you are working on without using "reset" functions — theoretically, the service worker should be used for caching on consecutive visits and that would ensure OneSignal can function.
I am trying to get my trigger.io app to receive push notifications from parse. I've set up an account with Parse.com, installed and configured the parse trigger.io module and added the relevant setup code to my app.
Here is the code from my client app:
forge.parse.installationInfo(success, error);
function success(info){
forge.logging.log("installation: "+JSON.stringify(info));
}
function error(info){
forge.logging.log("Parse error! "+JSON.stringify(info));
}
forge.parse.push.subscribe("beta-testers",function () {
forge.logging.info("subscribed to beta-tester push notifications!");
},function (err) {
forge.logging.error("error: "+ JSON.stringify(err));
});
forge.event.messagePushed.addListener(function (msg) {
forge.logging.log(JSON.stringify(msg));
});
The app appears to successfully connect to Parse.com.
It logs all the relevant confirmation messages:
(I have purposely blanked out my device ID)
[INFO] Pausing webview while application not focussed.
[INFO] Checking for reload update.
[INFO] Android remote debugging disabled.
[INFO] Android hardware acceleration enabled.
[INFO] Loading live page in webview: http://10.56.2.49:31337/src/index.html
[INFO] Application in focus, resuming webview.
[INFO] No reload update available.
[INFO] [FORGE] 'installation: {"id":"XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"}'
[INFO] [FORGE] 'subscribed to beta-tester push notifications!'
[INFO] Checking for reload update.
[INFO] No reload update available.
The devices I am testing (AVD emulator + .apk on phone) both seem to register on parse.com; when I choose recipients to send a notification to the blue menu bar reads "This will be sent to 2 devices".
So as far as I can tell everything is set up correctly.
However if I send the notification neither of my devices receives it.
The forge.io log remains quiet and the "Pushes Sent" column on the parse.com dashboard reads "0" for all my sends.
I've reached the stage where I'm not sure how to debug this any further.
Your help would be greatly appreciated!
Antoine van Gelder at Trigger.io support managed to help me solve this problem.
I was using Forge serve to test the app instead of running an actual compiled instance of the app.
Thank you Antoine!