ajax and ruby script only returning 1 record instead of many - ruby

I am doing an Ajax call, using Ruby and Sinatra. The query should return multiple rows, it only returns one though.
The ajax script is:
$(document).ready(function() {
$(".showmembers").click(function(e) {
e.preventDefault();
alert('script');
var short_id = $('#shortmembers').val();
console.log(short_id);
$.getJSON(
"/show",
{ 'id' : short_id },
function(res, status) {
console.log(res);
$('#result').html('');
$('#result').append('<input type=checkbox value=' + res["email"] + '>');
$('#result').append( res["first"] );
$('#result').append( res["last"] );
$('#result').append( res["email"] );
});
});
});
and the Ruby script is:
get '/show' do
id = params['id']
DB["select shortname, first, last, email from shortlists sh JOIN shortmembers sm ON sm.short_id = sh.list_id JOIN candidates ca ON ca.id = sm.candidate_id where sh.list_id = ?", id].each do |row|
#shortname = row[:shortname]
#first = row[:first]
#last = row[:last]
#email = row[:email]
puts #shortname
puts #first
puts #last
puts #email
halt 200, { shortname: #shortname, first: #first, last: #last, email: #email }.to_json
end
end
If I run the query directly in the terminal on postgres I get 9 rows returned but, as above on my website, it just returns the first row only.
What's the problem? No error in the console, just one record.

You have halt 200 inside your loop. This will cause Sinatra to terminate the request processing and return the result back up the stack.
To return a full set of results, you will need to do something like the following:
get '/show' do
id = params['id']
results = DB["select shortname, first, last, email from shortlists sh
JOIN shortmembers sm ON sm.short_id = sh.list_id
JOIN candidates ca ON ca.id = sm.candidate_id
where sh.list_id = ?", id].map do |row|
{
:short_name => row[:shortname],
:first=>row[:first],
:last=>row[:last],
:email=>row[:email]
}
end
halt 200, results.to_json
end
This will return the selected fields from each row as an array of hashes.
In fact, as I look at the above, the solution might even be as simple as:
get '/show' do
id = params['id']
results = DB["select shortname, first, last, email from shortlists sh
JOIN shortmembers sm ON sm.short_id = sh.list_id
JOIN candidates ca ON ca.id = sm.candidate_id
where sh.list_id = ?", id]
halt 200, results.to_json
end
since you don't seem to be selecting anything but the columns you desire in the first place.

Related

problems with the leaderboard discord.py

The leaderboard shows the same username even if they are different users in case they have the same value.
I don't know how to solve it but when in the code I ask to resist a variable it gives me only 3 elements and not 4 even if 4 come out.
code:
#client.command(aliases = ["lb"])
async def leaderboard(ctx,x = 10):
leader_board = {}
total = []
for user in economy_system:
name = int(user)
total_amount = economy_system[user]["wallet"] + economy_system[user]["bank"]
leader_board[total_amount] = name
total.append(total_amount)
print(leader_board)
total = sorted(total,reverse=True)
embed = discord.Embed(
title = f"Top {x} Richest People",
description = "This is decided on the basis of raw money in the bank and wallet",
color = 0x003399
)
index = 1
for amt in total:
id_ = leader_board[amt]
member = client.get_user(id_)
name = member.name
print(name)
embed.add_field(
name = f"{index}. {name}",
value = f"{amt}",
inline = False
)
if index == x:
break
else:
index += 1
await ctx.send(embed=embed)
print resists this:
{100: 523967502665908227, 350: 554617490806800387, 1100: 350886488235311126}
Padre Mapper
Flore (Orsolinismo)
Aetna
Aetna
In theory there should also be 100: 488826524791734275 (i.e. my user id) but it doesn't find it.
Your problem comes from this line:
leader_board[total_amount] = name
If total_amount is already a key (eg. two users have the same amount of money), it will replace the previous value (which was a user ID) and replace it with another user ID. In this situation, if multiple users have the same amount of money, only one will be saved in leader_board.
Then, you have this line:
total.append(total_amount)
In this case, if two users have the same amount of money, you would just have two identical values, which is normal but, considering the problem above, this will create a shift.
Let's say you have ten users with two of them who have the same amount of money. leader_board will only contain 9 items whereas total will contain 10 values. That's the reason why you have two of the same name in your message.
To solve the problem:
#client.command(aliases = ["lb"])
async def leaderboard(ctx, x=10):
d = {user_id: info["wallet"] + info["bank"] for user_id, info in economy_system.items()}
leaderboard = {user_id: amount for user_id, amount in sorted(d.items(), key=lambda item: item[1], reverse=True)}
embed = discord.Embed(
title = f"Top {x} Richest People",
description = "This is decided on the basis of raw money in the bank and wallet",
color = 0x003399
)
for index, infos in enumerate(leaderboard.items()):
user_id, amount = infos
member = client.get_user(user_id)
embed.add_field(
name = f"{index}. {member.display_name}",
value = f"{amount}",
inline = False
)
await ctx.send(embed=embed)
If I guessed right and your dictionnary is organized like this, it should work:
economy_system = {
user_id: {"bank": x, "wallet": y}
}

How to get commit history of users in gitlab in rails

I am trying to create an app which will fetch the gitlab/github commit history of the user which i can just show in a side panel and it can be checked on or off depending on my criteria. I want to know if there is a way to fetch the current logged user's gitlab/github commit history. I tried to use the gem
https://github.com/libgit2/rugged
But couldn't find a way to implement my need. If anyone knows how to implement this it would be so much helpful. Thanks.
Update
I have now managed to get the user by using :
Gitlab.endpoint = 'https://gitlab.com/api/v4'
Gitlab.private_token = 'token'
g = Gitlab.client(
endpoint: 'https://gitlab.com/api/v4',
private_token: 'token',
httparty: {
headers: { 'Cookie' => 'gitlab_canary=true' }
}
)
By using the command g.user i am able to get the user but i need to get the commits of the user that he has done in gitlab.
Use this gitlab API GET /projects/:id/repository/commits to fetch all the commits on a repository gitlab api. Check the attached code for more details.
Basically this git log --author="user_name" command can give you git commit history for a specific user, you can even use email address just the first name or last name in the above command.
Once you have authenticated with gitlab you can run the following command from ruby.
cmd = 'git log --author="user_name"'
system(cmd)
Sample code form ruby to connect to gitlab using private token which is not ideal but just an example
require 'json'
require 'curb'
require 'net/http'
begin
def parseCoverageReport(report_text)
coverage_perc = report_text.match /All files\s+\|\s+(\d+\.?\d+).*\n/
if not coverage_perc then
coverage_perc = report_text.match /^TOTAL\s+\d+\s+\d+\s+(\d+)%$/
end
if coverage_perc then
#if we found coverage value in job trace
#puts "coverage_perc[1]: #{coverage_perc[1]}"
coverage_perc[1].to_i
end
end
gen_config = YAML.load_file("config/general.yml")
gitlab_config = YAML.load_file("config/gitlab.yml")
SCHEDULER.every gen_config[:job_trigger_interval], :first_in => 0 do |job|
table = {
title: "Projects",
hrows: Array.new.push({cols: [
{value: "Project name"},
{value: "Open Merge Requests"},
{value: "Code coverage"}
]}),
rows: Array.new
}
instances = gitlab_config['instances']
instances.each do |instance|
gitlab_url = gitlab_config['instances'][instance.first]['url']
# gitlab_token = gitlab_config['instances'][instance.first]['api_key']
gitlab_token = ENV[gitlab_config['instances'][instance.first]['api_key']]
red_threshold = gitlab_config['instances'][instance.first]['red_threshold']
orange_threshold = gitlab_config['instances'][instance.first]['orange_threshold']
cov_red_threshold = gitlab_config['instances'][instance.first]['cov_red_threshold']
cov_orange_threshold = gitlab_config['instances'][instance.first]['cov_orange_threshold']
projects = gitlab_config['instances'][instance.first]['projects']
projects.each do |name, project|
merge_reqs = JSON.parse(Curl.get("#{gitlab_url}/api/v4/projects/#{project['id']}/merge_requests?state=opened&private_token=#{gitlab_token}&per_page=200").body_str)
git_project = JSON.parse(Curl.get("#{gitlab_url}/api/v4/projects/#{project['id']}?private_token=#{gitlab_token}").body_str)
opened_mrs = merge_reqs.select { |merge_reqs| %w[opened].include? merge_reqs['state'] }
repo_name = git_project['name']
repo_url = git_project['web_url']
status = case
when opened_mrs.size >= red_threshold then 'danger'
when opened_mrs.size >= orange_threshold then 'warning'
else
'ok'
end
mrs_count = "#{opened_mrs.size}"
send_event("#{name}_mr", { current: mrs_count, status: status })
color = case
when opened_mrs.size >= red_threshold then 'red'
when opened_mrs.size >= orange_threshold then 'orange'
else
'green'
end
font_color = color == 'orange' ? 'black' : 'white'
cov_color = color
font_cov_color = 'white'
code_coverage = "---"
code_coverage_tag = "---"
cov_job_url = ''
jobs = JSON.parse(Curl.get("#{gitlab_url}/api/v4/projects/#{project['id']}/jobs?scope=success&private_token=#{gitlab_token}&per_page=30").body_str)
code_cov_job = jobs.find { |gitlab_job| !gitlab_job['coverage'].nil? }
if not code_cov_job then
#if no job has 'coverage' feature set up in Gitlab try to parse
#'coverage' from jobs trace manually
jobs.each do |job|
trace_report = Curl.get("#{gitlab_url}/api/v4/projects/#{project['id']}/jobs/#{job['id']}/trace?private_token=#{gitlab_token}").body_str
code_cov_percentage = parseCoverageReport(trace_report)
if code_cov_percentage then
code_cov_job = job
code_cov_job['coverage'] = code_cov_percentage
break
end
end
end
if code_cov_job then
#found code coverage data => process them
code_coverage = code_cov_job['coverage'].to_i
cov_job_url = code_cov_job['web_url'].to_s
#update code covergate SprintProgress widgets at the same job
widget_title = "code_coverage_progress_#{project['id']}"
send_event(widget_title, {
title: "Code Coverage - #{git_project['name']}",
sprintboard_url: cov_job_url,
min: 0,
max: 100,
value: code_coverage,
moreinfo: ''
})
cov_color = case
when code_coverage <= cov_red_threshold then 'red'
when code_coverage <= cov_orange_threshold then 'orange'
else
'green'
end
code_coverage = "#{code_coverage}%"
code_coverage_tag = "<a href='#{cov_job_url}' target='_blank'>#{code_coverage.to_s}</a>"
end
repo_name_a_tag = "<a href='#{repo_url}' target='_blank'>#{repo_name}</a>"
open_mrs_size = "<a href='#{repo_url}/merge_requests' target='_blank'>#{opened_mrs.size}</a>"
table[:rows].push({
cols: [
{ value: repo_name_a_tag, style: "color: #{font_color}; background-color: #{color}" },
{ value: open_mrs_size, style: "color: #{font_color}; background-color: #{color}" },
{ value: code_coverage_tag, style: "color: #{cov_color == 'orange' ? 'black' : 'white'}; background-color: #{cov_color}" }
]
})
end
end
send_event('open_merge_requests_table', table)
end
rescue Errno::ENOENT
puts "No config file found for gitlab - not starting the Gitlab job"
end
In the above ruby example please have a look at the following code snippet
merge_reqs = JSON.parse(Curl.get("#{gitlab_url}/api/v4/projects/#{project['id']}/merge_requests?state=opened&private_token=#{gitlab_token}&per_page=200").body_str)
git_project = JSON.parse(Curl.get("#{gitlab_url}/api/v4/projects/#{project['id']}?private_token=#{gitlab_token}").body_str)
opened_mrs = merge_reqs.select { |merge_reqs| %w[opened].include? merge_reqs['state'] }
repo_name = git_project['name']
repo_url = git_project['web_url']
In here what i am trying to do is connect to our gitlab instance using a private_token and then for a specific project id (which you can get it form the UI of gitlab) check for the open merge request. I also get the git_project from which i get the name and web_url (which was my use case).
For your use case you will have to get the project_id(for gitlab UI) and then use some appropriate method to get the commits.gitlab docs

Problem matching amounts with PaypalExpressGateway.setup_authorization

I'm using ruby client with activemerchant to create call for Paypal sandbox API. Script is being called from command line, so variables are filled with command line parameters. Here's the code sample:
login = ARGV[0]
password = ARGV[1]
signature = ARGV[2]
ip = ARGV[3]
subtotal = ARGV[4]
shipping = ARGV[5]
handling = ARGV[6]
tax = ARGV[7]
currency = ARGV[8]
return_url = ARGV[9]
cancel_return_url = ARGV[10]
allow_guest_checkout = ARGV[11]
items = JSON.parse(ARGV[12])
ActiveMerchant::Billing::Base.mode = :test
paypal_options = {
login: login,
password: password,
signature: signature
}
gateway = ActiveMerchant::Billing::PaypalExpressGateway.new(paypal_options)
setup_hash = {
ip: ip,
items: items,
subtotal: Integer(subtotal),
shipping: Integer(shipping),
handling: Integer(handling),
tax: Integer(tax),
currency: currency,
return_url: return_url,
cancel_return_url: cancel_return_url,
allow_guest_checkout: allow_guest_checkout
}
amount = subtotal.to_i + shipping.to_i + handling.to_i + tax.to_i
puts "amount: " + amount.to_s
puts "items: " + items.to_s
response = gateway.setup_authorization(amount, setup_hash)
if !(response.success?)
puts response.message.to_s
end
And here's what I get in console:
amount: 10000
items: [{"name"=>"sample", "description"=>"desc", "quantity"=>1, "amount"=>10000}]
The totals of the cart item amounts do not match order amounts.
So, how come 10000 in amount doesn't match 10000 in items?
After long and boring debugging I found out the following:
internal hash items should be [:{name=>"sample", :description=>"desc", :quantity=>1, :amount=>10000}] rather than [{"name"=>"sample", "description"=>"desc", "quantity"=>1, "amount"=>10000}] in example above.
So, I've changed JSON parser to nice_hash and with items = ARGV[12].json it works like a charm.

Google Analytics API ruby client - multiple metrics

I'm using the google API ruby client and I want to implement some more complex analytics queries such as suggested in this document
https://developers.google.com/analytics/devguides/reporting/core/v3/common-queries
This document suggests that metrics can be supplied as a comma delimited string of multiple metrics but the API client only accepts an expression.
How can I query on multiple metrics in a single query? The ruby client appears only to accept an expression which generally consists of a single metric such as sessions or pageviews like this:
metric = Google::Apis::AnalyticsreportingV4::Metric.new(expression: 'ga:sessions')
If I remove "expression" and enter a list of metrics I just get an error.
Invalid value 'ga:sessions;ga:pageviews' for metric parameter.
Here is my solution, together with a generic method for reporting Google Analytics data:
This answer should be read in conjunction with https://developers.google.com/drive/v3/web/quickstart/ruby
analytics = Google::Apis::AnalyticsreportingV4::AnalyticsReportingService.new
analytics.client_options.application_name = APPLICATION_NAME
analytics.authorization = authorize
def get_analytics_data( analytics,
view_id,
start_date: (Date.today + 1 - Date.today.wday) - 6,
end_date: (Date.today + 1 - Date.today.wday),
metrics: ['ga:sessions'],
dimensions: [],
order_bys: [],
segments: nil, # array of hashes
filter: nil,
page_size: nil )
get_reports_request_object = Google::Apis::AnalyticsreportingV4::GetReportsRequest.new
report_request_object = Google::Apis::AnalyticsreportingV4::ReportRequest.new
report_request_object.view_id = view_id
analytics_date_range_object = Google::Apis::AnalyticsreportingV4::DateRange.new
analytics_date_range_object.start_date = start_date
analytics_date_range_object.end_date = end_date
report_request_object.date_ranges = [analytics_date_range_object]
# report_request_metrics = []
report_request_object.metrics = []
metrics.each { |metric|
analytics_metric_object = Google::Apis::AnalyticsreportingV4::Metric.new
analytics_metric_object.expression = metric
report_request_object.metrics.push(analytics_metric_object) }
# report_request_object.metrics = report_request_metrics
unless dimensions.empty?
report_request_object.dimensions = []
dimensions.each { |dimension|
analytics_dimension_object = Google::Apis::AnalyticsreportingV4::Dimension.new
analytics_dimension_object.name = dimension
report_request_object.dimensions.push(analytics_dimension_object) }
end
unless segments.nil?
report_request_object.segments = []
analytics_segment_object = Google::Apis::AnalyticsreportingV4::Segment.new
analytics_dynamic_segment_object = Google::Apis::AnalyticsreportingV4::DynamicSegment.new
analytics_segment_definition_object = Google::Apis::AnalyticsreportingV4::SegmentDefinition.new
analytics_segment_filter_object = Google::Apis::AnalyticsreportingV4::SegmentFilter.new
analytics_simple_segment_object = Google::Apis::AnalyticsreportingV4::SimpleSegment.new
analytics_or_filters_for_segment_object = Google::Apis::AnalyticsreportingV4::OrFiltersForSegment.new
analytics_segment_filter_clause_object = Google::Apis::AnalyticsreportingV4::SegmentFilterClause.new
analytics_segment_metric_filter_object = Google::Apis::AnalyticsreportingV4::SegmentMetricFilter.new
analytics_dimension_object = Google::Apis::AnalyticsreportingV4::Dimension.new
analytics_dimension_object.name = 'ga:segment'
report_request_object.dimensions.push(analytics_dimension_object)
analytics_or_filters_for_segment_object.segment_filter_clauses = []
analytics_simple_segment_object.or_filters_for_segment = []
analytics_segment_definition_object.segment_filters = []
segments.each { |segment|
analytics_segment_metric_filter_object.metric_name = segment[:metric_name]
analytics_segment_metric_filter_object.comparison_value = segment[:comparison_value]
analytics_segment_metric_filter_object.operator = segment[:operator]
analytics_segment_filter_clause_object.metric_filter = analytics_segment_metric_filter_object
analytics_or_filters_for_segment_object.segment_filter_clauses.push(analytics_segment_filter_clause_object)
analytics_simple_segment_object.or_filters_for_segment.push(analytics_or_filters_for_segment_object)
analytics_segment_filter_object.simple_segment = analytics_simple_segment_object
analytics_segment_definition_object.segment_filters.push(analytics_segment_filter_object)
analytics_dynamic_segment_object.name = segment[:name]
analytics_dynamic_segment_object.session_segment = analytics_segment_definition_object
analytics_segment_object.dynamic_segment = analytics_dynamic_segment_object
report_request_object.segments.push(analytics_segment_object) }
end
unless order_bys.empty?
report_request_object.order_bys = []
order_bys.each { |orderby|
analytics_orderby_object = Google::Apis::AnalyticsreportingV4::OrderBy.new
analytics_orderby_object.field_name = orderby
analytics_orderby_object.sort_order = 'DESCENDING'
report_request_object.order_bys.push(analytics_orderby_object)}
end
unless filter.nil?
report_request_object.filters_expression = filter
end
unless page_size.nil?
report_request_object.page_size = page_size
end
get_reports_request_object.report_requests = [report_request_object]
response = analytics.batch_get_reports(get_reports_request_object)
end
If using dimensions, you can report data like this:
response = get_analytics_data(analytics, VIEW_ID, metrics: ['ga:pageviews'], dimensions: ['ga:pagePath'], order_bys: ['ga:pageviews'], page_size: 25)
response.reports.first.data.rows.each do |row|
puts row.dimensions
puts row.metrics.first.values.first.to_i
puts
end

For loop while using scrapy

I am trying to crawl a website. I want to do it for different dates. So i am storing date in a list. But while trying to access items of list, crawler works only for 1st value in list. Please help. following is my code:
class SpidyQuotesViewStateSpider(scrapy.Spider):
name = 'retail_price'
def start_requests(self):
print "start request"
urls = "http://fcainfoweb.nic.in/PMSver2/Reports/Report_Menu_web.aspx"
yield scrapy.Request(url=urls, callback=self.parse)
def parse(self, response):
dated = ["05/03/2017","04/03/2017"]
urls = "http://fcainfoweb.nic.in/PMSver2/Reports/Report_Menu_web.aspx"
#frmdata =
cookies1 ={}
val = response.headers.getlist('Set-Cookie')
print "login session values" ,response.headers.getlist('Set-Cookie')
if(len(val) != 0):
cookies1['ASP.NET_SessionId'] = str(response.headers.getlist('Set-Cookie')[0].split(";")[0].split("=")[1])
cookies1['path'] = str(response.headers.getlist('Set-Cookie')[0].split(";")[1].split("=")[1])
print cookies1;
for i in range(len(dated)):
yield scrapy.FormRequest(url=urls, callback=self.parse1, formdata={'ctl00$MainContent$btn_getdata1':"Get Data",
'ctl00$MainContent$Txt_FrmDate':dated[i],
'ctl00$MainContent$Ddl_Rpt_Option0':"Daily Prices",
'ctl00$MainContent$Rbl_Rpt_type':"Price report",
'ctl00$MainContent$ddl_Language':"English",
'ctl00$MainContent$Ddl_Rpt_type':"Retail",
'__EVENTVALIDATION':"IwZyKgfTXVzxiHxiPXGk/W8XQZBDb0EOPxJh6s8hofq0ffqOpiHSH77CafcxySF3PbkYgSMNFCJhLM2cGnL6SxT0PJuGDCJtV0V8Y4a94UErUCiSANiin+4uKckk9v9Ux8JqTVeaipppmlH+wyks2U9SgPfkNUsqw4eHCkDyB5akNNZImRIixOHHVY3JSXGkwXn7ueK9w+AgnqJzpXaWdMr9J1++M4VAFImSNF8brFSfPHe5kb/qzkGIwUr/KRouaRYK8WLWZh/Mbl9xwREwhDSxWJSOdihSE0WWoaqSMtpaR99rDDCsD3mdJqfu0aPIlREupTZRzlrmztXU0eS3949YW+ywdTRvykaMNgOW2Q4saYP5j/niKbRW6GiDnaLV2A38X/HW80+trrsjwJr9tjTKVFyikf6s/3gzyiTp11ivSkwIY2b3hutjYn7OfTDo",
#'__EVENTVALIDATION':"HqVo2xHk04clYwnBposXbZGhbIr181A7RbyeZv74Cia7rXSKmpOpbeSnn3XXnoDJKRxMK0W9nxKZFfkNje+P/K7gE5HVjHJr9Gr0Gs46TntzKDsvzyii8jZ7e0fdZgQCJKoXxQNgR2vNkWqChKcEldBuMHCOgJRqCNCF/JPFKpdKZoIWr7GU8rhzwLijf/Gkm+FuTULs/fl2HHK6Z1QQEozzEHFsDwzl0G4IiN//eNYfHuUBXKZ3wdZzPqG0s53WHEuSBzhqBC9AtCJOs4ZZhdtwFh8iyTJ4PlsLP9DLHYHRCOAd72UO0UH8gT7gAkKVo1I4L540DilowOR9SttH7MM/oOs9qhKlnG61FgqkYGW8zGzF/yNEXO+beVAK1RVvuO+FDnuq/g36TRnUieei5GpAZ+96CSoCIxykdvHx8R+smTNF/5erlowV4ci+tcI7",
'__VIEWSTATEENCRYPTED':"",
'__VIEWSTATEGENERATOR':"85862B00",
'__VIEWSTATE':"+a+3jrBEKxDdkPOzx2wXwKaTMWvCB60WPaHRfJUAZQrdFIpxSqFr5VseTclpGzeHXdxaFnxJe/PkxDKYa7sj3Wiv/os1bNeX0IEB3s45eFsHYWGiU8cvsXCGa5z7rrGRDL5hotg7k/MuUWj8w27xXZO423MN5OsHS+wh+tC/5/Xix+w3zxuQhi8jR5DnreimHbhGZn1sYaKYIGCc8mDIDRNl+w1OZ058F+3LAx96QUu5BYiMYOmrlyxrb9b2yPTmmIrI4NtC4ClBQlxuST5wMDP3vUqqWMhn4auk8ev5gHyPestCRrsAXWs07wDNnikemMwo/4wPiTEbnZQV6SLcDUw0gZpXjXwLI7mhsVjEyVNaQnJp6+Wi6FLsAEEMlFYmQut3JecpVIUkjF9uYSN2GLIbXHPs37AiEXPeQ8E/GyBMx3z1X5l8sw/xSNmFgYQC3riajn8V0+SdkuV2PbNbYKtc+uoSCNLppLYCqiOv5eWanGvAQro2Q67FBA4w2xY+V/K8mzHaGMLoDBxJxLslWyJpL5cX0C6qoXVUu8B028auAQM4eVzH1YPF5qrJiCDo",
#'__VIEWSTATE':"W+m8kNAS6QHiRPo+zFj00EDs/Dbq+y/XvtCmSNwOIkGKlikAlphT8HBAWQDskSm1vdNterBuo0Hy7m4xPbXMOnyEm6IlseXO3jPw+ofnI2WHAKknLil+GeS0IfMWGeoD5aNyiz3zh1jZkKU7R7hQsxwARoHRyjhf8UCooFbkVvL6ddHVYZbH5LcocmCF1BTOCqYN5y5yzfDfYbp3KNW9kH53pdmwCsjiEirdxxUGDoG1Ke3JBEXfSl+4XubirHSR8z+VlFmPPXZGU8mMogwq9Eg822RYjvbwvZG74djcf7kdfB9KXCPO9u6cWIjLiW+cfXHSXD+1XYFVf9ATU2/NV4YbUzsI4PJRwoGD4BryUNIm2JFeT4c8F4REYTA16shxz5mDTFQ6rbmg6SmqP8G9gAc2Hr9ABD8+2BUNabGhNZ8wDIZArfYS4pl5DNrlPlpqeCjhmvv0znKAJSOac3pCUej8G90ZGwQKOPORWbNVzQShoH7QvrXV8pCklcia6psuAGO+Oj72oDWPxedE4DjdjX5TbLoW4bzsk/YNfUv4JpjGR8DWpG8IFYJG9CCjMEYb",
'__LASTFOCUS':"",
'__EVENTARGUMENT':"",
'__EVENTTARGET':"",
'ctl00_MainContent_ToolkitScriptManager1_HiddenField':";;AjaxControlToolkit,+Version=4.1.51116.0,+Culture=neutral,+PublicKeyToken=28f01b0e84b6d53e:en-US:fd384f95-1b49-47cf-9b47-2fa2a921a36a:475a4ef5:addc6819:5546a2b:d2e10b12:effe2a26:37e2e5c9:5a682656:c7029a2:e9e598a9"},method='POST',cookies = cookies1)
def parse1(self, response):
path1 = "id('Panel1')"
value1 = response.xpath(path1).extract_first()
print value1
First of all, you are sending the spider more time on the same site, though with different form parameters. You have therefore to use dont_filter=True in the request, otherwise Scrapy blocks duplicate calls.
Then it seems to me that the site you are scraping don't allow you to make more than one request during the same session. Try for example to go to http://fcainfoweb.nic.in/PMSver2/Reports/Report_Menu_web.aspx with your browser, compile the form, get the data and than to go back to the initial page: It's impossible. So you have to modify your spider. Here's a very rough code just to give an idea. It works for me, but please don't use it in production!
class SpidyQuotesViewStateSpider(scrapy.Spider):
name = 'retail_price'
urls = "http://fcainfoweb.nic.in/PMSver2/Reports/Report_Menu_web.aspx"
def start_requests(self):
dated = ["01/03/2017","05/03/2017","04/03/2017"]
for i in dated:
request = scrapy.Request(url=self.urls, dont_filter=True, callback=self.parse)
request.meta['question'] = i
yield request
def parse(self, response):
thedate = response.meta['question']
cookies1 ={}
val = response.headers.getlist('Set-Cookie')
print("login session values" ,response.headers.getlist('Set-Cookie'))
if(len(val) != 0):
cookies1['ASP.NET_SessionId'] = str(str(response.headers.getlist('Set-Cookie')[0]).split(";")[0].split("=")[1])
cookies1['path'] = str(str(response.headers.getlist('Set-Cookie')[0]).split(";")[1].split("=")[1])
yield scrapy.FormRequest(url=self.urls, dont_filter=True, callback=self.parse1, formdata={'ctl00$MainContent$btn_getdata1':"Get Data",
'ctl00$MainContent$Txt_FrmDate': thedate,
'ctl00$MainContent$Ddl_Rpt_Option0':"Daily Prices",
'ctl00$MainContent$Rbl_Rpt_type':"Price report",
'ctl00$MainContent$ddl_Language':"English",
'ctl00$MainContent$Ddl_Rpt_type':"Retail",
'__EVENTVALIDATION':"IwZyKgfTXVzxiHxiPXGk/W8XQZBDb0EOPxJh6s8hofq0ffqOpiHSH77CafcxySF3PbkYgSMNFCJhLM2cGnL6SxT0PJuGDCJtV0V8Y4a94UErUCiSANiin+4uKckk9v9Ux8JqTVeaipppmlH+wyks2U9SgPfkNUsqw4eHCkDyB5akNNZImRIixOHHVY3JSXGkwXn7ueK9w+AgnqJzpXaWdMr9J1++M4VAFImSNF8brFSfPHe5kb/qzkGIwUr/KRouaRYK8WLWZh/Mbl9xwREwhDSxWJSOdihSE0WWoaqSMtpaR99rDDCsD3mdJqfu0aPIlREupTZRzlrmztXU0eS3949YW+ywdTRvykaMNgOW2Q4saYP5j/niKbRW6GiDnaLV2A38X/HW80+trrsjwJr9tjTKVFyikf6s/3gzyiTp11ivSkwIY2b3hutjYn7OfTDo",
#'__EVENTVALIDATION':"HqVo2xHk04clYwnBposXbZGhbIr181A7RbyeZv74Cia7rXSKmpOpbeSnn3XXnoDJKRxMK0W9nxKZFfkNje+P/K7gE5HVjHJr9Gr0Gs46TntzKDsvzyii8jZ7e0fdZgQCJKoXxQNgR2vNkWqChKcEldBuMHCOgJRqCNCF/JPFKpdKZoIWr7GU8rhzwLijf/Gkm+FuTULs/fl2HHK6Z1QQEozzEHFsDwzl0G4IiN//eNYfHuUBXKZ3wdZzPqG0s53WHEuSBzhqBC9AtCJOs4ZZhdtwFh8iyTJ4PlsLP9DLHYHRCOAd72UO0UH8gT7gAkKVo1I4L540DilowOR9SttH7MM/oOs9qhKlnG61FgqkYGW8zGzF/yNEXO+beVAK1RVvuO+FDnuq/g36TRnUieei5GpAZ+96CSoCIxykdvHx8R+smTNF/5erlowV4ci+tcI7",
'__VIEWSTATEENCRYPTED':"",
'__VIEWSTATEGENERATOR':"85862B00",
'__VIEWSTATE':"+a+3jrBEKxDdkPOzx2wXwKaTMWvCB60WPaHRfJUAZQrdFIpxSqFr5VseTclpGzeHXdxaFnxJe/PkxDKYa7sj3Wiv/os1bNeX0IEB3s45eFsHYWGiU8cvsXCGa5z7rrGRDL5hotg7k/MuUWj8w27xXZO423MN5OsHS+wh+tC/5/Xix+w3zxuQhi8jR5DnreimHbhGZn1sYaKYIGCc8mDIDRNl+w1OZ058F+3LAx96QUu5BYiMYOmrlyxrb9b2yPTmmIrI4NtC4ClBQlxuST5wMDP3vUqqWMhn4auk8ev5gHyPestCRrsAXWs07wDNnikemMwo/4wPiTEbnZQV6SLcDUw0gZpXjXwLI7mhsVjEyVNaQnJp6+Wi6FLsAEEMlFYmQut3JecpVIUkjF9uYSN2GLIbXHPs37AiEXPeQ8E/GyBMx3z1X5l8sw/xSNmFgYQC3riajn8V0+SdkuV2PbNbYKtc+uoSCNLppLYCqiOv5eWanGvAQro2Q67FBA4w2xY+V/K8mzHaGMLoDBxJxLslWyJpL5cX0C6qoXVUu8B028auAQM4eVzH1YPF5qrJiCDo",
#'__VIEWSTATE':"W+m8kNAS6QHiRPo+zFj00EDs/Dbq+y/XvtCmSNwOIkGKlikAlphT8HBAWQDskSm1vdNterBuo0Hy7m4xPbXMOnyEm6IlseXO3jPw+ofnI2WHAKknLil+GeS0IfMWGeoD5aNyiz3zh1jZkKU7R7hQsxwARoHRyjhf8UCooFbkVvL6ddHVYZbH5LcocmCF1BTOCqYN5y5yzfDfYbp3KNW9kH53pdmwCsjiEirdxxUGDoG1Ke3JBEXfSl+4XubirHSR8z+VlFmPPXZGU8mMogwq9Eg822RYjvbwvZG74djcf7kdfB9KXCPO9u6cWIjLiW+cfXHSXD+1XYFVf9ATU2/NV4YbUzsI4PJRwoGD4BryUNIm2JFeT4c8F4REYTA16shxz5mDTFQ6rbmg6SmqP8G9gAc2Hr9ABD8+2BUNabGhNZ8wDIZArfYS4pl5DNrlPlpqeCjhmvv0znKAJSOac3pCUej8G90ZGwQKOPORWbNVzQShoH7QvrXV8pCklcia6psuAGO+Oj72oDWPxedE4DjdjX5TbLoW4bzsk/YNfUv4JpjGR8DWpG8IFYJG9CCjMEYb",
'__LASTFOCUS':"",
'__EVENTARGUMENT':"",
'__EVENTTARGET':"",
'ctl00_MainContent_ToolkitScriptManager1_HiddenField':";;AjaxControlToolkit,+Version=4.1.51116.0,+Culture=neutral,+PublicKeyToken=28f01b0e84b6d53e:en-US:fd384f95-1b49-47cf-9b47-2fa2a921a36a:475a4ef5:addc6819:5546a2b:d2e10b12:effe2a26:37e2e5c9:5a682656:c7029a2:e9e598a9"},method='POST',cookies = cookies1)
def parse1(self, response):
path1 = "id('Panel1')"
value1 = response.xpath(path1).extract_first()[:574]
print(value1)

Resources