Anyone used to work with Web Scraping in laravel - laravel

Route::get('/api', function() {
$crawler = Goutte::request('GET', 'https://www.alibaba.com/product-detail/wholesale-magnetic-charging-cable-flowing-light_62127638796.html?spm=a2700.details.deiletai6.5.472ada1akNmbnG&bypass=true');
$crawler->filter('.ma-title')->each(function ($node) {
dump($node->text());
});
});
I don't always receive data and sometimes I need to refresh 10 times to get data.
Can anyone advise why this might be?

Related

Kendo Autocomplete not posting to Server

I am a bit stumped as to why my Kendo Autocomplete is not posting to the server.
Can anyone see a problem with this?
#(Html.Kendo().AutoComplete()
.Name("LinkSearch")
.Filter("contains")
.MinLength(3)
.DataTextField("value")
.DataSource(source => {
source.Read(read =>
{
read.Action("_LinkSearch", "Record", new { area="record" })
.Data("onAdditionalData");
})
.ServerFiltering(true);
})
)
<script>
function onAdditionalData() {
return {
searchTerm: $("#LinkSearch").val()
};
}
</script>
As far as I am concerned this should work. This is based on the examples on the Kendo page. The rest of the Kendo controls on the page work without any issues.
The box renders perfectly fine, but just doesn't post to the server when the user types in it and therefore never returns any data.
I have a breakpoint on the Action and am monitoring the network traffic, but it never even tries to hit the server.
Cheers
Gareth
It turns out that the problem was due to the Routing somewhere.
Record is a base controller and by changing the controller in the read.Action to the controller that was inheriting from the base controller it worked fine. As far as I am aware it should have worked either way, but for some reason it doesn't.
It took a couple of us a good couple of hours to get to the bottom of it.

Whats different between AngularJS “Responsive” calls vs good old AJAX calls?

I was watching a free interactive course published at angularjs.org to learn Angular, Shaping up with Angular js.
On this course at the very first chapter they say, one of the main reasons to use AngularJS is, it can build “Responsive” web apps. (I know about "Responsive Design" but that's totally a different thing), and explaining it saying that, with Angular you don’t need to refresh your web page to update it with the data getting from the web server (They tell you this, like this is brand new tech!).
I think isn’t that the same thing we did for last 10 years using Ajax? Or is this something totally different?
Please help me to understand this coz I'm new to AngularJS.
From my view “Responsive” web apps. means type of application that updates View regards to model change (MVC).
Angular application UI is full of watchers. For each variable wrapped by {{}} in HTML, Angular creates new watcher and when we update during code running this value, Angular, by using digest cycle updates view respectively. Or ng-repeat directive that creates separate scope per list item and adds watcher as well.
On other hand in pure Javascript I need find my element by id and update it manually.
Consider following example in Fiddle
HTML
<ul>
<li ng-click="loadGeo()">click 1</li>
</ul>
<ul> <pre>
data: {{data|json}}
</pre>
</ul>
JS
var app = angular.module('myModule', ['ngResource']);
app.controller('fessCntrl', function ($scope, Data) {
$scope.data = false;
$scope.loadGeo = function () {
Data.query()
.then(function (result) {
$scope.data = result.data.results[0];
}, function (result) {
alert("Error: No data returned");
});
}
});
app.factory('Data', ['$http', '$q', function ($http, $q) {
var address = 'Singapore, SG, Singapore, 153 Bukit Batok Street 1';
var URL = 'http://maps.googleapis.com/maps/api/geocode/json?address=' + address + '&sensor=true';
var factory = {
query: function () {
var data = $http({
method: 'GET',
url: URL
});
var deferred = $q.defer();
deferred.resolve(data);
return deferred.promise;
}
}
return factory;
}]);
On start we have empty data: $scope.data = false;
We click on button, we get Geo data from factory and populate data with Geo output. Our GUI updates without any additional code.
This approach I would call “Responsive” web app
I suggest you to read this great post written by Josh David Miller:
how-do-i-think-in-angularjs-if-i-have-a-jquery-background

CasperJS form fill sometimes stays on the current page

I have a simple casperjs test to submit a search form on my homepage. Then I assert that the title on the landing page is correct.
Works fine on my computer (OSX 10.9.2) but on my colleague's laptops (a Win 7 and Win 8), the test fails randomly because casper "thinks" it is still on the search page.
casper.test.begin('Search', function(test) {
casper.start("http://localhost:8080/site", function() {
this.fill(searchForm, { query: goodQuery }, true);
});
casper.then(function() {
// sometimes fails, says it's "My Project" main title
test.assertTitle('Search Result', 'Search result title is ok');
});
}
Introducing a casper.waitFor(3000) before checking the page title does not change the outcome. I've also tried to replace the then step by a waitForUrl, but it fails after 5 secs, saying it is still on the current page.
Plenty of other tests work fine on all computers but it's the only one with form submition.
Any hints on how to solve or properly work around this? I'd rather not simulate a click on the submit button (more coupling to the form internals) if possible (but it would be okay I guess).
Thanks
$ casperjs --version
1.1.0-beta3
$ phantomjs --version
1.9.7
EDIT: submitting the form and waitForUrldid not help. I found out that the test actually runs fine on its own, even on the Windows 7 machine. But when I run two tests:
01 search.js (the one described above)
02 menu.js (a simple one, merely containing assertExists)
'search.js' fails most of the time... and sometimes 'menu.js' fails instead! I suspect some mishandled concurrent access, although it consistently works on OSX. I must be doing something wrong. Both tests have the same structure:
casper.test.begin('Some test', function(test) {
casper.start(someUrl, function() {
// some test
});
casper.run(function() {
test.done();
});
});
Any clue?
Try :
casper.test.begin('Search', function(test) {
casper.start("http://localhost:8080/site", function() {
this.fill(searchForm, {
query: goodQuery
},false);
this.click("your selector for submit button");
});
casper.then(function() {//you should use waitForUrl/Selector/Text() instead
// sometimes fails, says it's "My Project" main title
test.assertTitle('Search Result', 'Search result title is ok');
});
casper.run(function() {
this.test.comment('------ Tests over ----\n');
test.done();
});
});
It's better to submit the form by clicking. Sometimes (often) it doesn't pass putting the fill arg at true. Just put the correct selector for the submit button.
You should wait for an item to appear on the following page. I would change your code to the following:
casper.test.begin('Search', function(test) {
casper.start("http://localhost:8080/site", function() {
this.fill(searchForm, { query: goodQuery }, true);
});
casper.waitForSelector('#someSelectorOnNextPage', function() {
test.assertTitle('Search Result', 'Search result title is ok');
});
}
I also experience same issue. Suprisingly adding empty then() handler fixes that in v1.1.0-beta3. I don't think this is expected behavior though:
casper.test.begin('Search', function(test) {
casper.start("http://localhost:8080/site", function() {
this.fill(searchForm, { query: goodQuery }, true);
});
// Do nothing here, just call it as a placeholder
// Here http://localhost:8080/site sends us to the next endpoint
casper.then(function() {});
// Now this is the final page we actually want to assert
casper.then(function() {
test.assertTitle('Search Result', 'Search result title is ok');
});
}
EDIT:
Although question author says casper.waitForUrl() didn't work for them, it did work for me as an alternative solution.
What does look strange is that in verbose mode whatever returns a 301 status code along with Location Header is recognized as HTTP 200 response by Casper.
EDIT 2:
Well obviously it doesn't happen every time, but what I noticed is that Casper sometimes doubles the previous response (that's why I thought it recognizes some specific HTTP codes as 200 mistakenly and that's why author's code functioned as if it stayed on same page after form submission) and sometimes not.
waitForUrl() fixes that obviously but there is still some underneath issue in Casper which scares me a bit and I hope I will find some time to report it with all the dumps to Casper issue tracker.

Issues with angular $watch and retrieving data from database

I'm a novice programming trying to put together a web application with Angular, node.js, and the graph database neo4j.
I would like to load content from my database dynamically based on the user selecting (or rejecting) terms (clicking buttons). Every time a button is clicked the relevant term is added to an array (either exclude or include). The idea is a new call to the database would be made each time a new term is selected.
I'm stuck right now on how to go about making calls to the database to retrieve the content. I'm trying to watch the arrays for changes using $watch. Something is going wrong and I'm having issues troubleshooting the problem.
Here is the controller code:
angular.module('myApp.controllers', []).
controller('content',function($scope,$http, queryTerms, $watch){
//watch arrays of terms for changes and fetch results based on what is selected
$watch(function() { return angular.toJson( [ queryTerms.includedTerms, queryTerms.excludedTerms ] ) },
function() {
$http({
method:'get',
url:'/query/getContent',
params: {includeTerms:queryTerms.includedTerms , excludeTerms:queryTerms.excludedTerms}
}).
success(function(data){
//feed content data to display for viewing
}).
error(function(data){
$scope.test = "Error :("
});
});
});
I'm getting the following error when I use $watch:
Error: Unknown provider: $watchProvider <- $watch
Is this a terrible stupid way to go about this in general? Any advice would be greatly appreciated- I'm learning as I'm going and so far the advice I've gotten on here has be amazing. Thanks!
Use $scope.$watch instead.
controller('content', function ($scope, $http, queryTerms) {
$scope.$watch(function () {
return angular.toJson([queryTerms.includedTerms, queryTerms.excludedTerms])
},...

how to save parameters in session in nodejs. nowjs example

I use nowjs and have an issue with sessions.
here is the code:
now.on('connect', function(){
if(this.user.session){
this.user.session.views++;
console.log(this.user.session.views);
}
});
app.get('/', function(req, res){
var body = '';
body = body+' <script src="/nowjs/now.js"></script>';
if (req.session.views) {
++req.session.views;
} else {
req.session.views = 1;
body += '<p>First time visiting? view this page in several browsers :)</p>';
}
res.send(body + '<p>viewed <strong>' + req.session.views + '</strong> times.</p>');
console.log("body");
});
no the problem is that I can write anything into "this.user.session" but on next call everything is erased. in my example it counts how many times you have entered the site. and "views" is increasing by one each time, but it should be increasing by 2 because I of "this.user.session.views++;". I'm missing something or it can't be done?
if you have some examples of nowjs+sessions post is please
thank you
I was using nowjs with sessions and Passport for authentication... I wrote a simple gist for making them work together ...
https://gist.github.com/2266544

Resources