I can't understand why scrapy can't see some //div/text() on page http://www.alize.gen.tr/index.php?is=urun_detay&id=37 . for examle:
scrapy view http://www.alize.gen.tr/index.php?is=urun_detay&id=37
and
scrapy shell http://www.alize.gen.tr/index.php?is=urun_detay&id=37
>>> hxs.select("//td[#class='urun_adi']/div/text()").extract()
returns [u'\r\n'] but must be [u'\r\nANGORA GOLD']
where I'm wrong?
Works for me:
stav#maia:~$ scrapy shell "http://www.alize.gen.tr/index.php?is=urun_detay&id=37"
2013-03-28 20:36:39-0600 [scrapy] INFO: Scrapy 0.17.0 started (bot: scrapybot)
...
>>> hxs.select("//td[#class='urun_adi']/div/text()").extract()
[u'\r\nANGORA GOLD']
What version Scrapy are you using?
stav#maia:~$ scrapy version -v
Scrapy : 0.17.0
lxml : 2.3.2.0
libxml2 : 2.7.8
Twisted : 11.1.0
Python : 2.7.3 (default, Aug 1 2012, 05:14:39) - [GCC 4.6.3]
Platform: Linux-3.2.0-39-generic-x86_64-with-Ubuntu-12.04-precise
Related
I tried to create a new installation of my codebase and am running into this issue. As best as possible, I have tried to verify that both installations have the same exact versions of all dependencies.
The issue only seems to strike on endpoints that access the database, although maybe that is just a coincidence and the issue is not at all DB related. (The DB is definitely functioning because the old install can still access it.)
Let me know if there is any other helpful info I can gather to help debug.
Traceback:
File "venv/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
132. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "venv/lib/python2.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
58. return view_func(*args, **kwargs)
File "venv/lib/python2.7/site-packages/rest_framework/viewsets.py" in view
87. return self.dispatch(request, *args, **kwargs)
File "venv/lib/python2.7/site-packages/rest_framework/views.py" in dispatch
466. response = self.handle_exception(exc)
File "venv/lib/python2.7/site-packages/rest_framework/views.py" in dispatch
463. response = handler(request, *args, **kwargs)
File "venv/lib/python2.7/site-packages/rest_framework/mixins.py" in retrieve
56. instance = self.get_object()
File "venv/lib/python2.7/site-packages/rest_framework/generics.py" in get_object
84. queryset = self.filter_queryset(self.get_queryset())
File "venv/lib/python2.7/site-packages/rest_framework/generics.py" in filter_queryset
151. queryset = backend().filter_queryset(self.request, queryset, self)
File "venv/lib/python2.7/site-packages/rest_framework/filters.py" in filter_queryset
31. raise NotImplementedError(".filter_queryset() must be overridden.")
Exception Type: NotImplementedError at /my/endpoint
Exception Value: .filter_queryset() must be overridden.
Here is my package list from pip list:
Package Version
---------------------------------- ----------
argcomplete 0.8.1
asn1crypto 0.24.0
backports.shutil-get-terminal-size 1.0.0
beautifulsoup4 4.3.2
cffi 1.11.5
cryptography 2.3.1
Cython 0.28.5
decorator 4.3.0
Django 1.8.4
django-cors-headers 1.1.0
django-filter 0.11.0
djangorestframework 3.2.3
djangorestframework-jwt 1.7.1
dxpy 0.161.0
enum34 1.0.4
fusepy 2.0.2
futures 3.0.3
gunicorn 19.3.0
idna 2.0
impyla 0.10.0
ipaddress 1.0.14
ipdb 0.11
ipython 5.8.0
ipython-genutils 0.2.0
kazoo 2.2.1
Markdown 2.6.2
MySQL-python 1.2.5
ndg-httpsclient 0.4.0
pathlib2 2.3.2
pexpect 4.6.0
pickleshare 0.7.4
pip 18.0
prompt-toolkit 1.0.15
psutil 2.1.3
ptyprocess 0.6.0
pyasn1 0.1.8
pycparser 2.14
Pygments 2.2.0
PyJWT 1.4.0
pymssql 2.2.0.dev0
pyodbc 4.0.24
pyOpenSSL 17.5.0
python-dateutil 2.3
python-magic 0.4.6
requests 2.7.0
scandir 1.9.0
setuptools 40.0.0
simplegeneric 0.8.1
six 1.9.0
SQLAlchemy 1.2.11
thrift 0.9.2
traitlets 4.3.2
wcwidth 0.1.7
WebHDFS 0.2.0
wheel 0.24.0
ws4py 0.3.2
xattr 0.6.4
Here are my rest framework settings. I'm starting to think that the use of the BaseFilterBackend is the start of the issue. Seems like that class contains the filter_queryset() method that is not being overwritten:
REST_FRAMEWORK = {
'DEFAULT_PERMISSION_CLASSES': (
'rest_framework.permissions.IsAuthenticated',
),
'DEFAULT_AUTHENTICATION_CLASSES': (
'rest_framework.authentication.SessionAuthentication',
'rest_framework.authentication.BasicAuthentication',
'rest_framework_jwt.authentication.JSONWebTokenAuthentication',
),
'DEFAULT_FILTER_BACKENDS': ('rest_framework.filters.BaseFilterBackend',),
'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.LimitOffsetPagination',
}
EDIT: Here is the view causing the error:
from rest_framework import permissions
from rest_framework import renderers
from rest_framework import viewsets, mixins
from models import MyModel
from rest_framework.response import Response
from rest_framework.renderers import JSONRenderer
from rest_framework.parsers import JSONParser
from rest_framework import generics
from rest_framework import filters
import six
class MyViewSet(viewsets.ModelViewSet):
queryset = MyModel.objects.all()
serializer_class = MyModelSerializer
#Allows to filter using URL params and sorting
filter_backends = (filters.BaseFilterBackend, filters.OrderingFilter)
ordering = ('CreatedAt',)
filter_fields = ('Modelfield1', 'Modelfield2',)
In your settings, you've set DEFAULT_FILTER_BACKENDS' to ('rest_framework.filters.BaseFilterBackend',). Thus, BaseFilterBackend is being applied to most your views* and is throwing the NotImplementedError because BaseFilterBackend is a baseclass that is not actually intended to be applied directly -- only to be overridden by you or inherited by other filters. You should remove it from your defaults and anywhere else you're using it.
*Except the ones that manually specify their filters via filter_backends like you did in your sample view.
For example, the error can be resolved by replacing BaseFilterBackend with DjangoFilterBackend in the settings and in the view.
Installed ruby 2.4.0 using RVM, but after typing ruby, the command just freezes indefinitely. Can be ctrl-C'ed out, but ruby never loads.
Ruby info:
ruby-2.4.0:
system:
uname: "Linux waffleboy 4.8.0-58-generic #63~16.04.1-Ubuntu SMP Mon Jun 26 18:08:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux"
name: "Ubuntu"
version: "16.04"
architecture: "x86_64"
bash: "/bin/bash => GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)"
zsh: "/usr/bin/zsh => zsh 5.1.1 (x86_64-ubuntu-linux-gnu)"
remote path: "ubuntu/16.04/x86_64"
rvm:
version: "rvm 1.29.2 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io/]"
updated: "23 minutes 20 seconds ago"
path: "/home/waffleboy/.rvm"
autolibs: "[4] Allow RVM to use package manager if found, install missing dependencies, install package manager (only OS X)."
ruby:
interpreter: "ruby"
version: "2.4.0p0"
date: "2016-12-24"
platform: "x86_64-linux"
patchlevel: "2016-12-24 revision 57164"
full_version: "ruby 2.4.0p0 (2016-12-24 revision 57164) [x86_64-linux]"
homes:
gem: "/home/waffleboy/.rvm/gems/ruby-2.4.0"
ruby: "/home/waffleboy/.rvm/rubies/ruby-2.4.0"
binaries:
ruby: "/home/waffleboy/.rvm/rubies/ruby-2.4.0/bin/ruby"
irb: "/home/waffleboy/.rvm/rubies/ruby-2.4.0/bin/irb"
gem: "/home/waffleboy/.rvm/rubies/ruby-2.4.0/bin/gem"
rake: "/home/waffleboy/.rvm/rubies/ruby-2.4.0/bin/rake"
environment:
PATH: "/home/waffleboy/.rvm/gems/ruby-2.4.0/bin:/home/waffleboy/.rvm/gems/ruby-2.4.0#global/bin:/home/waffleboy/.rvm/rubies/ruby-2.4.0/bin:/storage/anaconda3/bin:/home/waffleboy/bin:/home/waffleboy/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin:/usr/bin:/storage/programfiles/:/home/waffleboy/bin:/storage/programfiles/spark-1.6.2/bin:/home/waffleboy/.rvm/bin"
GEM_HOME: "/home/waffleboy/.rvm/gems/ruby-2.4.0"
GEM_PATH: "/home/waffleboy/.rvm/gems/ruby-2.4.0:/home/waffleboy/.rvm/gems/ruby-2.4.0#global"
MY_RUBY_HOME: "/home/waffleboy/.rvm/rubies/ruby-2.4.0"
IRBRC: "/home/waffleboy/.rvm/rubies/ruby-2.4.0/.irbrc"
RUBYOPT: ""
gemset: ""
Rvm list:
rvm rubies
=* ruby-2.4.0 [ x86_64 ]
# => - current
# =* - current && default
# * - default
I've tried reinstalling and installing ruby multiple times but it just doesn't seem to work.
Can anyone point me in the right direction? :) Thank you!
This is normal; it is a feature, not a bug.
Running ruby, without any parameters, causes the program to run and wait to receive input from STDIN (ending with an EOF character). It will not execute anything until it receives this information.
If you want to run ruby in interactive mode, (like when you run python), then use ruby's built-in irb ("interactive ruby") command.
Alternatively, if you just want to display some basic information about the ruby version installed then you can try commands like:
ruby -v # Display version information
ruby -h # Display help about usage, switches and features
If you're looking for a REPL to type Ruby code into, have it executed, and see the results, then you're looking for the irb command, not ruby.
ruby is for running files (eg. ruby my_file.rb), or without arguments will read a script typed into standard input, which will get executed when you quit ruby with cmd+D.
Ruby 2.2.5
Upgraded Gems to the following:
-Cucumber 2.4.0
-Selenium Webdriver 3.0.0
-rake 11.3.0
-json 2.0.2
And just upgraded to Firefox 47. Installed Gecko and now I am getting the following error when trying to run in Firefox
unknown option: {:profile=>#<Selenium::WebDriver::Firefox::Profile:0x40bba78 #model=nil, #native_events=true, #secure_ssl=false, #untrusted_issuer=true, #load_no_focus_lib=false, #additional_prefs={"plugin.default.state"=>2}, #extensions={}>} (ArgumentError)
Does anyone have any ideas why?!
This question already has answers here:
What command opens Ruby's REPL?
(2 answers)
Closed 6 years ago.
When attempting to run ruby on Linux (ubuntu 14.04) I am not getting a prompt like I do in python (">>>".) Below is an example. I can run the ruby -v to obtain the version information and execute ruby code when the code file is an argument. When I run ruby without any arguments I get nothing and have to break out of it. Any ideas what I am missing?
james#james-Latitude-D830:~$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
james#james-Latitude-D830:~$ ruby -v
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-linux]
james#james-Latitude-D830:~$ ruby
help()
quit()
^C-: Interrupt
python and ruby are different languages you can open prompt for python using python but incase of ruby you have to use irb to open ruby 'prompt'
james#james-Latitude-D830:~$ irb
2.3.1p112 :001 > #here you can write your ruby code
Ruby's repl is opened with the 'irb' command. Just using ruby would be for calling programs. See this answer for more details - What command opens Ruby's REPL?
or just google ruby repl (read, evaluate, print, loop)
I'm trying to use Jekyll on my windows 7 (64bit) PC.
I have installed Ruby, Ruby Devkit, Python 2.7, Jekyll and Pygments but getting this error
E:\jekyll\my-site>jekyll serve
Configuration file: E:/jekyll/my-site/_config.yml
Source: E:/jekyll/my-site
Destination: E:/jekyll/my-site/_site
Generating... Liquid Exception: Failed to get header. in 2013-07-19-welcome-to-jekyll.markdown
error: Failed to get header.. Use --trace to view backtrace
I also got this error, and a couple of other similar errors too, and finally got to a solution: you need to have a couple of compatible versions of the required Ruby, Python and Pygments.
Install Ruby 1.9.3-p448 version (download here);
Install Python 2.7.5 version (download here);
Install Pygments 0.5.0
gem uninstall pygments.rb
gem install pygments.rb --version "=0.5.0"
I hope it helps.
Edited: Don't forget to add the correct PATHs to your system variables. In my case, it was:
C:\Python27;C:\Ruby193
In order to improve this content, there is a simple tutorial (written in Portuguese, but I guess you can understand the steps, always considering the versions of software/plugins listed above) - view tutorial.
Another possible cause for this error message on Windows is that the path to the Pygments scripts contains spaces. Took me a while to figure this one out...
For my situation, install pygments 0.5.0 or 0.5.4 both have problem. I fixed by re-install python 2.7.6. Please refer to following answer:
https://github.com/jekyll/jekyll/issues/1181#issuecomment-35963876
I had two versions of python installed on my system.(Windows 8.1)
Python22 and Python27.
I removed Python22, added Python27 to System variables and issue got fixed.