I'm trying to save image from web_page with mechanize. I use this code:
#current_agent.get( image_url ).save ( save_path )
Error (I think I have a problems with timeout settings):
I, [2013-03-25T14:42:13.924694 #31865] INFO -- : Net::HTTP::Get: /i?path=b0312211141_img_id282431557272802821.jpg
D, [2013-03-25T14:42:13.924757 #31865] DEBUG -- : request-header: accept => */*
D, [2013-03-25T14:42:13.924828 #31865] DEBUG -- : request-header: user-agent => Mozilla/5.0 (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3
D, [2013-03-25T14:42:13.924858 #31865] DEBUG -- : request-header: accept-encoding => gzip,deflate,identity
D, [2013-03-25T14:42:13.924884 #31865] DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
D, [2013-03-25T14:42:13.924915 #31865] DEBUG -- : request-header: accept-language => en-us,en;q=0.5
D, [2013-03-25T14:42:13.924942 #31865] DEBUG -- : request-header: host => mdata.yandex.net
I, [2013-03-25T14:42:14.151810 #31865] INFO -- : status: Net::HTTPOK 1.0 200 OK
D, [2013-03-25T14:42:14.151890 #31865] DEBUG -- : response-header: server => nginx/1.2.1
D, [2013-03-25T14:42:14.151919 #31865] DEBUG -- : response-header: date => Mon, 25 Mar 2013 13:43:54 GMT
D, [2013-03-25T14:42:14.151943 #31865] DEBUG -- : response-header: content-type => image/jpeg
D, [2013-03-25T14:42:14.151967 #31865] DEBUG -- : response-header: content-length => 212187
D, [2013-03-25T14:42:14.151991 #31865] DEBUG -- : response-header: last-modified => Tue, 12 Mar 2013 18:11:41 GMT
D, [2013-03-25T14:42:14.152015 #31865] DEBUG -- : response-header: expires => Wed, 24 Apr 2013 13:43:54 GMT
D, [2013-03-25T14:42:14.152039 #31865] DEBUG -- : response-header: cache-control => max-age=2592000
D, [2013-03-25T14:42:14.152062 #31865] DEBUG -- : response-header: x-original-host => mdata.somesite.ru
D, [2013-03-25T14:42:14.152086 #31865] DEBUG -- : response-header: accept-ranges => bytes
D, [2013-03-25T14:42:14.152109 #31865] DEBUG -- : response-header: x-cache => MISS from parser.myapp.com.ua
D, [2013-03-25T14:42:14.152133 #31865] DEBUG -- : response-header: x-cache-lookup => MISS from parser.notus.com.ua:1221
D, [2013-03-25T14:42:14.152157 #31865] DEBUG -- : response-header: via => 1.0 parser.myapp.com.ua (squid/3.1.10)
D, [2013-03-25T14:42:14.152180 #31865] DEBUG -- : response-header: connection => keep-alive
D, [2013-03-25T14:42:14.152464 #31865] DEBUG -- : Read 2521 bytes (2521 total)
D, [2013-03-25T14:42:14.152509 #31865] DEBUG -- : Read 598 bytes (3119 total)
D, [2013-03-25T14:42:14.199787 #31865] DEBUG -- : Read 1448 bytes (6613 total)
D, [2013-03-25T14:42:14.199887 #31865] DEBUG -- : Read 2648 bytes (9261 total)
D, [2013-03-25T14:42:14.200125 #31865] DEBUG -- : Read 2896 bytes (12157 total)
D, [2013-03-25T14:42:14.200286 #31865] DEBUG -- : Read 1200 bytes (13357 total)
D, [2013-03-25T14:42:14.248204 #31865] DEBUG -- : Read 2896 bytes (16253 total)
D, [2013-03-25T14:42:14.248436 #31865] DEBUG -- : Read 1200 bytes (17453 total)
D, [2013-03-25T14:42:14.248510 #31865] DEBUG -- : Read 1448 bytes (18901 total)
D, [2013-03-25T14:42:14.248609 #31865] DEBUG -- : Read 2648 bytes (21549 total)
D, [2013-03-25T14:42:14.248864 #31865] DEBUG -- : Read 2896 bytes (24445 total)
D, [2013-03-25T14:42:14.248985 #31865] DEBUG -- : Read 1200 bytes (25645 total)
D, [2013-03-25T14:42:14.249174 #31865] DEBUG -- : Read 1448 bytes (27093 total)
D, [2013-03-25T14:42:14.249354 #31865] DEBUG -- : Read 2648 bytes (29741 total)
D, [2013-03-25T14:42:14.296443 #31865] DEBUG -- : Read 1448 bytes (31189 total)
D, [2013-03-25T14:42:14.296583 #31865] DEBUG -- : Read 2648 bytes (33837 total)
D, [2013-03-25T14:42:14.296756 #31865] DEBUG -- : Read 1448 bytes (35285 total)
I see that it start scraping image, "Read 2896 bytes (12157 total)" and freeze after!!!
parsing image aren't compleates, image is't saved :(
How can I solve this problem?
You can set the agent's timeout (both read and connection) with:
#current_agent.open_timeout = 10 # in seconds
#current_agent.read_timeout = 10 # in seconds
Related
I use Ruby mechanize(Ruby version: 2.3.1p112, Mehanize version: 2.7.5) to scrape web page.
Some page crushes with "input conversion failed due to input error"
The page I want to scrape is http://www.jra.go.jp/JRADB/accessD.html
and My code and execute logs below.
agent = Mechanize.new
agent.log = Logger.new(STDOUT)
agent.post('http://www.jra.go.jp/JRADB/accessD.html', { "cname" => "pw01bmd0006201705080220171224/7A" })
D, [2017-12-24T18:34:32.039095 #37600] DEBUG -- : query: "cname=pw01bmd0006201705080220171224%2F7A"
I, [2017-12-24T18:34:32.039562 #37600] INFO -- : Net::HTTP::Post: /JRADB/accessD.html
D, [2017-12-24T18:34:32.039657 #37600] DEBUG -- : request-header: accept-encoding => gzip,deflate,identity
D, [2017-12-24T18:34:32.039742 #37600] DEBUG -- : request-header: accept => */*
D, [2017-12-24T18:34:32.039873 #37600] DEBUG -- : request-header: user-agent => Mechanize/2.7.5 Ruby/2.3.1p112 (http://github.com/sparklemotion/mechanize/)
D, [2017-12-24T18:34:32.039996 #37600] DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
D, [2017-12-24T18:34:32.040105 #37600] DEBUG -- : request-header: accept-language => en-us,en;q=0.5
D, [2017-12-24T18:34:32.040209 #37600] DEBUG -- : request-header: host => www.jra.go.jp
D, [2017-12-24T18:34:32.040323 #37600] DEBUG -- : request-header: referer => http://www.jra.go.jp/JRADB/accessD.html
D, [2017-12-24T18:34:32.040428 #37600] DEBUG -- : request-header: content-type => application/x-www-form-urlencoded
D, [2017-12-24T18:34:32.040547 #37600] DEBUG -- : request-header: content-length => 40
I, [2017-12-24T18:34:32.093773 #37600] INFO -- : status: Net::HTTPOK 1.1 200 OK
D, [2017-12-24T18:34:32.094155 #37600] DEBUG -- : response-header: server => Apache
D, [2017-12-24T18:34:32.094296 #37600] DEBUG -- : response-header: x-frame-options => SAMEORIGIN
D, [2017-12-24T18:34:32.094736 #37600] DEBUG -- : response-header: content-encoding => gzip
D, [2017-12-24T18:34:32.094953 #37600] DEBUG -- : response-header: content-length => 13989
D, [2017-12-24T18:34:32.095272 #37600] DEBUG -- : response-header: content-type => text/html
D, [2017-12-24T18:34:32.095590 #37600] DEBUG -- : response-header: date => Sun, 24 Dec 2017 09:34:32 GMT
D, [2017-12-24T18:34:32.095980 #37600] DEBUG -- : response-header: connection => keep-alive
D, [2017-12-24T18:34:32.096387 #37600] DEBUG -- : response-header: vary => Accept-Encoding
D, [2017-12-24T18:34:32.096894 #37600] DEBUG -- : Read 6790 bytes (6790 total)
D, [2017-12-24T18:34:32.105888 #37600] DEBUG -- : Read 7199 bytes (13989 total)
D, [2017-12-24T18:34:32.106710 #37600] DEBUG -- : gzip response
encoding error : input conversion failed due to input error, bytes 0xFA 0xBA 0x3C 0x2F
I understand this is cause of character-encoding problem and I want to ignore if parsing error is occured. How can I do this?
I am trying to push my fabric clock to an output LVDS pair on a ZedBoard. I am using the solution provided here by scary_jeff.
FCLK_CLK1 is using a DDR PLL set to 150 MHz. I created a testbench for the IP which shows the expeceted output (the lvds pairs following the input clock with the expected inversion between the pairs).
The vhdl used to generate the ip :
library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;
library UNISIM;
use UNISIM.Vcomponents.all;
entity clk_to_lvds_v1_0 is
port (
clk_in : in std_logic;
clk_out_p : out std_logic;
clk_out_n : out std_logic
);
end clk_to_lvds_v1_0;
architecture arch_imp of clk_to_lvds_v1_0 is
signal clk_buf : std_logic;
signal clk_inv : std_logic;
begin
-- V1
-- oddr_inst : oddr
-- generic map(
-- DDR_CLK_EDGE => "SAME_EDGE",
-- INIT => '0',
-- SRTYPE => "SYNC"
-- )
-- port map (
-- Q => clk_buf,
-- C => clk_in,
-- CE => '1',
-- D1 => '1',
-- D2 => '0',
-- R => '0',
-- S => '0'
-- );
-- V2
clk_inv <= not clk_in;
ODDR_pxlclk_p : ODDR2
generic map(
DDR_ALIGNMENT => "NONE",
INIT => '0',
SRTYPE => "SYNC")
port map(
Q => clk_buf,
C0 => clk_in,
C1 => clk_inv,
CE => '1',
D0 => '1',
D1 => '0',
R => '0',
S => '0'
);
obufds_inst : obufds
generic map ( IOSTANDARD=>"LVDS_25" )
port map(
O => clk_out_p,
OB => clk_out_n,
I => clk_buf
);
end arch_imp;
The output is being pushed to JC1_P/N pmod ports using LVDS_25 standard.
set_property PACKAGE_PIN AB7 [get_ports {clk_out_p[0]}]
set_property IOSTANDARD LVDS_25 [get_ports {clk_out_p[0]}]
I am measuring the output with a scope (6 GHz, 20 GS/s) and all I'm seeing is noise. I turned on some LEDs in the design to ensure that the bitstream is being loaded properly.
Is this still the preferred method by Xilinx to output a clock on I/O?
The problem was I wasn't initializing the ps7_init on the ARM cores which kicks the clocks off. After doing that I saw the output clock on a scope.
I am on Mac OSX (Yosemite)
Installed using homebrew.
➜ ~ brew list gearmand
/usr/local/Cellar/gearman/1.1.12/bin/gearadmin
/usr/local/Cellar/gearman/1.1.12/bin/gearman
/usr/local/Cellar/gearman/1.1.12/homebrew.mxcl.gearman.plist
/usr/local/Cellar/gearman/1.1.12/include/libgearman/gearman.h
/usr/local/Cellar/gearman/1.1.12/include/libgearman-1.0/ (39 files)
/usr/local/Cellar/gearman/1.1.12/lib/libgearman.8.dylib
/usr/local/Cellar/gearman/1.1.12/lib/pkgconfig/gearmand.pc
/usr/local/Cellar/gearman/1.1.12/lib/ (2 other files)
/usr/local/Cellar/gearman/1.1.12/sbin/gearmand
/usr/local/Cellar/gearman/1.1.12/share/man/ (156 files)
Added /usr/local/sbin to my path (.bash-path) so that gearmand is on path.
➜ ~ echo $PATH
/Users/davidvezzani/.rvm/gems/ruby-1.9.3-p551/bin:/Users/davidvezzani/.rvm/gems/ruby-1.9.3-p551#global/bin:/Users/davidvezzani/.rvm/rubies/ruby-1.9.3-p551/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/git/bin:/Users/davidvezzani/.rvm/bin:/Users/davidvezzani/bin
Checking if on path. And it is.
➜ ~ which gearmand
/usr/local/sbin/gearmand
Trying to start up gearmand. I was expecting to see some log statements indicating the server has started and that it's waiting for jobs to process. There's nothing.
➜ ~ gearmand --verbose DEBUG
Is it running?!
501 17847 17693 0 11:24AM ttys017 0:00.01 gearmand --verbose DEBUG
Yes. But I have no idea what it is doing. Let's run one of the examples from gearman-ruby.
rvm use 2.2.0
gem install gearman-ruby
mvim do_reverse.rb
Provide the following content
# reverse_do.rb
require 'rubygems'
require 'gearman'
# Add our servers
servers = ['127.0.0.1:4730']
# Initialize a new client
client = Gearman::Client.new(servers)
puts "Sending job...";
task = Gearman::Task.new('reverse', 'Hello World!', {})
result = client.do_task task
puts "Result: " + result
Run the example client.
➜ work3 ruby reverse_do.rb
D, [2015-07-21T14:00:40.717160 #21521] DEBUG -- : Performing health check for 127.0.0.1:4730 (connected: false)
D, [2015-07-21T14:00:40.717248 #21521] DEBUG -- : Attempt #0 to connect to 127.0.0.1:4730
D, [2015-07-21T14:00:40.718060 #21521] DEBUG -- : Health check response for 127.0.0.1:4730 (connected: true) is [:echo_res, "ping"]
Sending job...
D, [2015-07-21T14:00:40.719181 #21521] DEBUG -- : Available job servers: [#<Gearman::Connection:0x007f960e049b78 #hostname="127.0.0.1", #port=4730, #real_socket=#<TCPSocket:fd 7>>]
D, [2015-07-21T14:00:40.719215 #21521] DEBUG -- : Using 127.0.0.1:4730 (connected: true) to submit job
D, [2015-07-21T14:00:40.719473 #21521] DEBUG -- : Got job_created from 127.0.0.1:4730 (connected: true)
The client seems to run, but I should see "Result" at some point, but I am not seeing that. What am I missing?
From the Gearman documentation:
This document assumes a high-level understanding of the role of the job server, clients, and workers.
Since there are three players involved here, when running an example, at least one Gearman job server (this example shows running 2),
(Execute from terminal:)
gearmand --verbose DEBUG -p 4730
gearmand --verbose DEBUG -p 4731
... start up your workers,
# worker.rb
require 'rubygems'
require 'logger'
require 'gearman'
servers = ['localhost:4730', 'localhost:4731']
w = Gearman::Worker.new(servers)
logger = Logger.new(STDOUT)
# Add a handler for a "sleep" function that takes a single argument, the
# number of seconds to sleep before reporting success.
w.add_ability("sleep") do |data,job|
seconds = 10
logger.info "Sleeping for #{seconds} seconds"
(1..seconds.to_i).each do |i|
sleep 1
# Report our progress to the job server every second.
job.report_status(i, seconds)
end
# Report success.
true
end
w.add_ability("blah") do |data,job|
logger.info "Blah, blah, blah..."
# Report our progress to the job server every second.
job.report_status(1, 1)
# Report success.
true
end
loop { w.work }
(Execute from terminal:)
ruby worker.rb
... and have a client push a job on the stack.
# reverse_do.rb
require 'gearman'
servers = ['localhost:4730', 'localhost:4731']
client = Gearman::Client.new(servers)
taskset = Gearman::TaskSet.new(client)
task = Gearman::Task.new('sleep', 20)
task.on_complete {|d| puts "Finished with client: #{d}" }
task2 = Gearman::Task.new('blah')
task2.on_complete {|d| puts "Finished with client for blah: #{d}" }
taskset.add_task(task)
taskset.add_task(task2)
taskset.wait(100)
(Execute from terminal:)
ruby reverse_do.rb
Server results.
Even through I have turned on verbosity to DEBUG, I'm seeing no logging in the terminal.
Worker results.
➜ work3 ruby worker.rb
D, [2015-07-21T15:04:34.868178 #22866] DEBUG -- : Performing health check for localhost:4730 (connected: false)
D, [2015-07-21T15:04:34.868258 #22866] DEBUG -- : Attempt #0 to connect to localhost:4730
D, [2015-07-21T15:04:34.869435 #22866] DEBUG -- : Health check response for localhost:4730 (connected: true) is [:echo_res, "ping"]
D, [2015-07-21T15:04:34.869471 #22866] DEBUG -- : Performing health check for localhost:4731 (connected: false)
D, [2015-07-21T15:04:34.869493 #22866] DEBUG -- : Attempt #0 to connect to localhost:4731
D, [2015-07-21T15:04:34.869962 #22866] DEBUG -- : Health check response for localhost:4731 (connected: true) is [:echo_res, "ping"]
D, [2015-07-21T15:04:34.870030 #22866] DEBUG -- : Announced ability sleep
D, [2015-07-21T15:04:34.870107 #22866] DEBUG -- : Announced ability sleep
D, [2015-07-21T15:04:34.870160 #22866] DEBUG -- : Announced ability blah
D, [2015-07-21T15:04:34.870193 #22866] DEBUG -- : Announced ability blah
D, [2015-07-21T15:04:34.870238 #22866] DEBUG -- : Sending GRAB_JOB to localhost:4730 (connected: true)
I, [2015-07-21T15:04:34.870417 #22866] INFO -- : Got NO_JOB from localhost:4730 (connected: true)
D, [2015-07-21T15:04:34.870444 #22866] DEBUG -- : Sending GRAB_JOB to localhost:4731 (connected: true)
I, [2015-07-21T15:04:34.870620 #22866] INFO -- : Got NO_JOB from localhost:4731 (connected: true)
I, [2015-07-21T15:04:34.870657 #22866] INFO -- : Sending PRE_SLEEP and going to sleep for 30 second(s)
D, [2015-07-21T15:04:34.870764 #22866] DEBUG -- : Polling on 2 available server(s) with a 30 second timeout
D, [2015-07-21T15:04:53.263702 #22866] DEBUG -- : Received NOOP while sleeping... waking up!
D, [2015-07-21T15:04:53.263801 #22866] DEBUG -- : Sending GRAB_JOB to localhost:4730 (connected: true)
I, [2015-07-21T15:04:53.264074 #22866] INFO -- : Got NO_JOB from localhost:4730 (connected: true)
D, [2015-07-21T15:04:53.264101 #22866] DEBUG -- : Sending GRAB_JOB to localhost:4731 (connected: true)
I, [2015-07-21T15:04:53.264367 #22866] INFO -- : Got JOB_ASSIGN with handle H:Davids-iMac.local:6 and 0 byte(s) from localhost:4731 (connected: true)
I, [2015-07-21T15:04:53.264399 #22866] INFO -- : Blah, blah, blah...
D, [2015-07-21T15:04:53.264443 #22866] DEBUG -- : Sending WORK_COMPLETE for H:Davids-iMac.local:6 with 4 byte(s) to localhost:4731 (connected: true)
D, [2015-07-21T15:04:53.264491 #22866] DEBUG -- : Sending GRAB_JOB to localhost:4731 (connected: true)
I, [2015-07-21T15:04:53.264710 #22866] INFO -- : Got NO_JOB from localhost:4731 (connected: true)
I, [2015-07-21T15:04:53.264735 #22866] INFO -- : Sending PRE_SLEEP and going to sleep for 30 second(s)
D, [2015-07-21T15:04:53.264795 #22866] DEBUG -- : Polling on 2 available server(s) with a 30 second timeout
D, [2015-07-21T15:04:53.265126 #22866] DEBUG -- : Received NOOP while sleeping... waking up!
D, [2015-07-21T15:05:03.269974 #22866] DEBUG -- : Sending GRAB_JOB to localhost:4730 (connected: true)
I, [2015-07-21T15:05:03.270372 #22866] INFO -- : Got JOB_ASSIGN with handle H:Davids-iMac.local:4 and 2 byte(s) from localhost:4730 (connected: true)
I, [2015-07-21T15:05:03.270405 #22866] INFO -- : Sleeping for 10 seconds
D, [2015-07-21T15:05:13.301296 #22866] DEBUG -- : Sending WORK_COMPLETE for H:Davids-iMac.local:4 with 4 byte(s) to localhost:4730 (connected: true)
D, [2015-07-21T15:05:13.301415 #22866] DEBUG -- : Sending GRAB_JOB to localhost:4730 (connected: true)
I, [2015-07-21T15:05:13.301655 #22866] INFO -- : Got NO_JOB from localhost:4730 (connected: true)
D, [2015-07-21T15:05:13.301696 #22866] DEBUG -- : Sending GRAB_JOB to localhost:4731 (connected: true)
I, [2015-07-21T15:05:13.301920 #22866] INFO -- : Got NO_JOB from localhost:4731 (connected: true)
I, [2015-07-21T15:05:13.301951 #22866] INFO -- : Sending PRE_SLEEP and going to sleep for 30 second(s)
D, [2015-07-21T15:05:13.302023 #22866] DEBUG -- : Polling on 2 available server(s) with a 30 second timeout
Client results.
➜ work3 ruby reverse_do.rb
D, [2015-07-21T14:55:09.896887 #22800] DEBUG -- : Performing health check for localhost:4730 (connected: false)
D, [2015-07-21T14:55:09.896973 #22800] DEBUG -- : Attempt #0 to connect to localhost:4730
D, [2015-07-21T14:55:09.898392 #22800] DEBUG -- : Health check response for localhost:4730 (connected: true) is [:echo_res, "ping"]
D, [2015-07-21T14:55:09.898428 #22800] DEBUG -- : Performing health check for localhost:4731 (connected: false)
D, [2015-07-21T14:55:09.898451 #22800] DEBUG -- : Attempt #0 to connect to localhost:4731
D, [2015-07-21T14:55:09.898963 #22800] DEBUG -- : Health check response for localhost:4731 (connected: true) is [:echo_res, "ping"]
D, [2015-07-21T14:55:09.899153 #22800] DEBUG -- : Available job servers: [#<Gearman::Connection:0x007fb6a309fd88 #hostname="localhost", #port=4730, #real_socket=#<TCPSocket:fd 9>>, #<Gearman::Connection:0x007fb6a309e258 #hostname="localhost", #port=4731, #real_socket=#<TCPSocket:fd 10>>]
D, [2015-07-21T14:55:09.899182 #22800] DEBUG -- : Using localhost:4731 (connected: true) to submit job
D, [2015-07-21T14:55:09.899464 #22800] DEBUG -- : Got job_created from localhost:4731 (connected: true)
D, [2015-07-21T14:55:19.903692 #22800] DEBUG -- : Received WORK_STATUS for H:Davids-iMac.local:5: 1 / 1
D, [2015-07-21T14:55:19.903789 #22800] DEBUG -- : Received WORK_COMPLETE for H:Davids-iMac.local:5
Finished with client for blah: true
D, [2015-07-21T14:55:19.903933 #22800] DEBUG -- : Available job servers: [#<Gearman::Connection:0x007fb6a309fd88 #hostname="localhost", #port=4730, #real_socket=#<TCPSocket:fd 9>>, #<Gearman::Connection:0x007fb6a309e258 #hostname="localhost", #port=4731, #real_socket=#<TCPSocket:fd 10>>]
D, [2015-07-21T14:55:19.903993 #22800] DEBUG -- : Using localhost:4730 (connected: true) to submit job
D, [2015-07-21T14:55:19.904225 #22800] DEBUG -- : Got job_created from localhost:4730 (connected: true)
D, [2015-07-21T14:55:30.908445 #22800] DEBUG -- : Received WORK_STATUS for H:Davids-iMac.local:3: 1 / 10
D, [2015-07-21T14:55:31.911152 #22800] DEBUG -- : Received WORK_STATUS for H:Davids-iMac.local:3: 2 / 10
D, [2015-07-21T14:55:32.914036 #22800] DEBUG -- : Received WORK_STATUS for H:Davids-iMac.local:3: 3 / 10
D, [2015-07-21T14:55:33.916782 #22800] DEBUG -- : Received WORK_STATUS for H:Davids-iMac.local:3: 4 / 10
D, [2015-07-21T14:55:34.919513 #22800] DEBUG -- : Received WORK_STATUS for H:Davids-iMac.local:3: 5 / 10
D, [2015-07-21T14:55:35.922062 #22800] DEBUG -- : Received WORK_STATUS for H:Davids-iMac.local:3: 6 / 10
D, [2015-07-21T14:55:36.926507 #22800] DEBUG -- : Received WORK_STATUS for H:Davids-iMac.local:3: 7 / 10
D, [2015-07-21T14:55:37.930245 #22800] DEBUG -- : Received WORK_STATUS for H:Davids-iMac.local:3: 8 / 10
D, [2015-07-21T14:55:38.930387 #22800] DEBUG -- : Received WORK_STATUS for H:Davids-iMac.local:3: 9 / 10
D, [2015-07-21T14:55:39.933860 #22800] DEBUG -- : Received WORK_STATUS for H:Davids-iMac.local:3: 10 / 10
D, [2015-07-21T14:55:39.933983 #22800] DEBUG -- : Received WORK_COMPLETE for H:Davids-iMac.local:3
Finished with client: true
Understanding the high-level concepts will be important so it is understood what order the scripts should be executed after the gearmand servers are started so that a working example can be successfully executed.
I am using ruby mechanize gem to automate form submission and scrape the results. I have the following code:
require 'mechanize'
require 'logger'
url = "http://www.cebupacificair.com/Pages/default.aspx"
agent = Mechanize.new do |agent|
agent.log = Logger.new(STDOUT)
agent.follow_meta_refresh = true
end
page = agent.get(url)
search_results = page.form_with(:name => 'aspnetForm') do |form|
form['__EVENTARGUMENT'] = ''
form['__EVENTTARGET'] = 'ControlGroupSearchView$AvailabilitySearchInputSearchView$LinkButtonNewSearch'
form.radiobutton_with(:value => "RoundTrip").check
form.field_with(:name => "ddOrigin").options_with(:value => "MNL").select
form.field_with(:name => "ddDestination").options_with(:value => "SGN").select
form.field_with(:name => "_depmonthyear").options_with(:value => "2013-02").select
form.field_with(:name => "_depday").options_with(:value => "9").select
form.field_with(:name => "_retmonthyear").options_with(:value => "2013-02").select
form.field_with(:name => "_retday").options_with(:value => "11").select
form.field_with(:name => "_adults").options_with(:value => "1").select
form.field_with(:name => "_children").options_with(:value => "0").select
form.field_with(:name => "_infants").options_with(:value => "0").select
end.click_button
puts search_results.body
when I run the code above, it submits the form but it doesn't redirect to a page where I was redirected when I do the submit manually. Here's the log:
Starting run ...
I, [2012-08-27T08:37:08.010661 #4] INFO -- : Net::HTTP::Get: /Pages/default.aspx
D, [2012-08-27T08:37:08.010794 #4] DEBUG -- : request-header: accept => */*
D, [2012-08-27T08:37:08.010825 #4] DEBUG -- : request-header: user-agent => Mechanize/2.3 Ruby/1.9.2...more
D, [2012-08-27T08:37:08.010849 #4] DEBUG -- : request-header: accept-encoding => gzip,deflate,identity
D, [2012-08-27T08:37:08.010873 #4] DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
D, [2012-08-27T08:37:08.010898 #4] DEBUG -- : request-header: accept-language => en-us,en;q=0.5
D, [2012-08-27T08:37:08.010922 #4] DEBUG -- : request-header: host => www.cebupacificair.com
I, [2012-08-27T08:37:11.354481 #4] INFO -- : status: Net::HTTPOK 1.1 200 OK
D, [2012-08-27T08:37:11.354778 #4] DEBUG -- : response-header: cache-control => private, max-age=0
D, [2012-08-27T08:37:11.354869 #4] DEBUG -- : response-header: content-type => text/html; charset=utf-8
D, [2012-08-27T08:37:11.354945 #4] DEBUG -- : response-header: content-encoding => gzip
D, [2012-08-27T08:37:11.355017 #4] DEBUG -- : response-header: expires => Sun, 12 Aug 2012 08:37:09 GMT
D, [2012-08-27T08:37:11.355089 #4] DEBUG -- : response-header: last-modified => Mon, 27 Aug 2012 08:37:09 GMT
D, [2012-08-27T08:37:11.355161 #4] DEBUG -- : response-header: vary => Accept-Encoding
D, [2012-08-27T08:37:11.355232 #4] DEBUG -- : response-header: server => Microsoft-IIS/7.0
D, [2012-08-27T08:37:11.355313 #4] DEBUG -- : response-header: x-aspnet-version => 2.0.50727
D, [2012-08-27T08:37:11.355386 #4] DEBUG -- : response-header: x-powered-by => ASP.NET
D, [2012-08-27T08:37:11.355457 #4] DEBUG -- : response-header: microsoftsharepointteamservices => 12.0.0.6420
D, [2012-08-27T08:37:11.355528 #4] DEBUG -- : response-header: date => Mon, 27 Aug 2012 08:37:09 GMT
D, [2012-08-27T08:37:11.355599 #4] DEBUG -- : response-header: content-length => 19031
D, [2012-08-27T08:37:11.355668 #4] DEBUG -- : response-header: connection => close
D, [2012-08-27T08:37:11.355739 #4] DEBUG -- : response-header: set-cookie => MyCookie=8npRAWS5i10kju...more
D, [2012-08-27T08:37:11.356068 #4] DEBUG -- : Read 6698 bytes (6698 total)
D, [2012-08-27T08:37:11.356328 #4] DEBUG -- : Read 12333 bytes (19031 total)
D, [2012-08-27T08:37:11.356890 #4] DEBUG -- : gzip response
D, [2012-08-27T08:37:11.375285 #4] DEBUG -- : saved cookie: MyCookie=8npRAWS5i10kjuDl8xX/01gRq0obDLa...more
I, [2012-08-27T08:37:11.385797 #4] INFO -- : form encoding: utf-8
D, [2012-08-27T08:37:11.388509 #4] DEBUG -- : query: "MSOWebPartPage_PostbackSource=&MSOTlPn_Selecte...more
I, [2012-08-27T08:37:11.390797 #4] INFO -- : Net::HTTP::Post: /Pages/default.aspx
D, [2012-08-27T08:37:11.390897 #4] DEBUG -- : request-header: accept => */*
D, [2012-08-27T08:37:11.390927 #4] DEBUG -- : request-header: user-agent => Mechanize/2.3 Ruby/1.9.2...more
D, [2012-08-27T08:37:11.390966 #4] DEBUG -- : request-header: accept-encoding => gzip,deflate,identity
D, [2012-08-27T08:37:11.390991 #4] DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
D, [2012-08-27T08:37:11.391015 #4] DEBUG -- : request-header: accept-language => en-us,en;q=0.5
D, [2012-08-27T08:37:11.391039 #4] DEBUG -- : request-header: cookie => MyCookie=8npRAWS5i10kjuDl8xX...more
D, [2012-08-27T08:37:11.391063 #4] DEBUG -- : request-header: host => www.cebupacificair.com
D, [2012-08-27T08:37:11.391095 #4] DEBUG -- : request-header: referer => http://www.cebupacificair.c...more
D, [2012-08-27T08:37:11.391123 #4] DEBUG -- : request-header: content-type => application/x-www-form...more
D, [2012-08-27T08:37:11.391146 #4] DEBUG -- : request-header: content-length => 13733
D, [2012-08-27T08:37:11.391170 #4] DEBUG -- : request-header: if-modified-since => Mon, 27 Aug 2012 ...more
I, [2012-08-27T08:37:19.039011 #4] INFO -- : status: Net::HTTPOK 1.1 200 OK
D, [2012-08-27T08:37:19.039252 #4] DEBUG -- : response-header: cache-control => private
D, [2012-08-27T08:37:19.039348 #4] DEBUG -- : response-header: content-type => text/html; charset=utf-8
D, [2012-08-27T08:37:19.039428 #4] DEBUG -- : response-header: content-encoding => gzip
D, [2012-08-27T08:37:19.039501 #4] DEBUG -- : response-header: vary => Accept-Encoding
D, [2012-08-27T08:37:19.039580 #4] DEBUG -- : response-header: server => Microsoft-IIS/7.0
D, [2012-08-27T08:37:19.039652 #4] DEBUG -- : response-header: x-aspnet-version => 2.0.50727
D, [2012-08-27T08:37:19.039725 #4] DEBUG -- : response-header: x-powered-by => ASP.NET
D, [2012-08-27T08:37:19.039798 #4] DEBUG -- : response-header: microsoftsharepointteamservices => 12.0.0.6420
D, [2012-08-27T08:37:19.039895 #4] DEBUG -- : response-header: date => Mon, 27 Aug 2012 08:37:17 GMT
D, [2012-08-27T08:37:19.040003 #4] DEBUG -- : response-header: content-length => 20962
D, [2012-08-27T08:37:19.040074 #4] DEBUG -- : response-header: connection => close
D, [2012-08-27T08:37:19.040338 #4] DEBUG -- : Read 6906 bytes (6906 total)
D, [2012-08-27T08:37:19.040605 #4] DEBUG -- : Read 14056 bytes (20962 total)
D, [2012-08-27T08:37:19.041137 #4] DEBUG -- : gzip response
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html __...more
Finished: 12.393 seconds elapsed
runfinished
Anyone knows what am I missing here? Do I need a different settings for mechanize to work on asp forms?
Thoughts?
Thanks.
Try a different approach.
If it's a post request that it makes. Try making that request manually instead of using Mechanizes form handler:
The params it submits are:
__EVENTTARGET=ControlGroupSearchView%24AvailabilitySearchInputSearchView%24LinkButtonNewSearch
__EVENTARGUMENT=
__VIEWSTATE=
ControlGroupSearchView%24AvailabilitySearchInputSearchView%24RadioButtonMarketStructure=RoundTrip
ControlGroupSearchView%24AvailabilitySearchInputSearchView%24TextBoxMarketOrigin1=BWN
ControlGroupSearchView%24AvailabilitySearchInputSearchView%24TextBoxMarketDestination1=PEK
ControlGroupSearchView%24AvailabilitySearchInputSearchView%24DropDownListMarketDay1=11
ControlGroupSearchView%24AvailabilitySearchInputSearchView%24DropDownListMarketMonth1=2013-06
ControlGroupSearchView%24AvailabilitySearchInputSearchView%24DropDownListMarketDay2=11
ControlGroupSearchView%24AvailabilitySearchInputSearchView%24DropDownListMarketMonth2=2013-06
ControlGroupSearchView%24AvailabilitySearchInputSearchView%24DropDownListPassengerType_ADT=1
ControlGroupSearchView%24AvailabilitySearchInputSearchView%24DropDownListPassengerType_CHD=0
ControlGroupSearchView%24AvailabilitySearchInputSearchView%24DropDownListPassengerType_INFANT=0
ControlGroupSearchView%24AvailabilitySearchInputSearchView%24promoCodeID=
So what you could do is to:
# 1. Get the frontpage to get the cookie set:
agent = Mechanize.new
page = agent.get(url)
# 2. The use .search helper to extract the inputs you might need
# 3. Submit the form by sending the postparams manually
result_page = agent.post("http://book.cebupacificair.com/Search.aspx?culture=en-us", {'__EVENTTARGET' => 'ControlGroupSearchView%24AvailabilitySearchInputSearchView%24LinkButtonNewSearch'}) # And of cause add the remaining params...
I'm sure this approach will work for you...
I try to connect a server though SSH using Ruby, but I have an Net::SSH::AuthenticationFailed error. The problem is that it works in PHP.
Here's my ruby code:
require 'rubygems'
require 'net/ssh'
include Net
domain = 'ks2.so.d-sites.com' # insert IP address or domain name here
begin
Net::SSH.start( domain, 'webistrano', :verbose => :debug, :keys => ['/home/webistrano/.ssh/id_rsa', '/home/webistrano/.ssh/id_rsa.pub'], :port => 22 ) do |ssh|
print "connected"
end
rescue Net::SSH::HostKeyMismatch => e
puts "remembering new key: #{e.fingerprint}"
e.remember_host!
retry
end
And I get this errors:
D, [2012-03-06T20:40:02.974862 #24165] DEBUG -- net.ssh.transport.session[3f8c5d3a06a0]: establishing connection to ks2.so.d-sites.com:22
D, [2012-03-06T20:40:02.975912 #24165] DEBUG -- net.ssh.transport.session[3f8c5d3a06a0]: connection established
I, [2012-03-06T20:40:02.976039 #24165] INFO -- net.ssh.transport.server_version[3f8c5d39edc8]: negotiating protocol version
D, [2012-03-06T20:40:02.981160 #24165] DEBUG -- net.ssh.transport.server_version[3f8c5d39edc8]: remote is `SSH-2.0-OpenSSH_5.8p1-hpn13v10'
D, [2012-03-06T20:40:02.981231 #24165] DEBUG -- net.ssh.transport.server_version[3f8c5d39edc8]: local is `SSH-2.0-Ruby/Net::SSH_2.3.0 x86_64-linux'
D, [2012-03-06T20:40:02.982272 #24165] DEBUG -- tcpsocket[3f8c5d39f250]: read 856 bytes
D, [2012-03-06T20:40:02.982418 #24165] DEBUG -- tcpsocket[3f8c5d39f250]: received packet nr 0 type 20 len 852
I, [2012-03-06T20:40:02.982517 #24165] INFO -- net.ssh.transport.algorithms[3f8c5d39e418]: got KEXINIT from server
I, [2012-03-06T20:40:02.982690 #24165] INFO -- net.ssh.transport.algorithms[3f8c5d39e418]: sending KEXINIT
D, [2012-03-06T20:40:02.982883 #24165] DEBUG -- tcpsocket[3f8c5d39f250]: queueing packet nr 0 type 20 len 716
D, [2012-03-06T20:40:02.982960 #24165] DEBUG -- tcpsocket[3f8c5d39f250]: sent 720 bytes
I, [2012-03-06T20:40:02.983003 #24165] INFO -- net.ssh.transport.algorithms[3f8c5d39e418]: negotiating algorithms
D, [2012-03-06T20:40:02.983161 #24165] DEBUG -- net.ssh.transport.algorithms[3f8c5d39e418]: negotiated:
* kex: diffie-hellman-group-exchange-sha1
* host_key: ssh-rsa
* encryption_server: aes128-cbc
* encryption_client: aes128-cbc
* hmac_client: hmac-sha1
* hmac_server: hmac-sha1
* compression_client: none
* compression_server: none
* language_client:
* language_server:
D, [2012-03-06T20:40:02.983207 #24165] DEBUG -- net.ssh.transport.algorithms[3f8c5d39e418]: exchanging keys
D, [2012-03-06T20:40:02.983416 #24165] DEBUG -- tcpsocket[3f8c5d39f250]: queueing packet nr 1 type 34 len 20
D, [2012-03-06T20:40:02.983469 #24165] DEBUG -- tcpsocket[3f8c5d39f250]: sent 24 bytes
D, [2012-03-06T20:40:03.024146 #24165] DEBUG -- tcpsocket[3f8c5d39f250]: read 152 bytes
D, [2012-03-06T20:40:03.024270 #24165] DEBUG -- tcpsocket[3f8c5d39f250]: received packet nr 1 type 31 len 148
D, [2012-03-06T20:40:03.027331 #24165] DEBUG -- tcpsocket[3f8c5d39f250]: queueing packet nr 2 type 32 len 140
D, [2012-03-06T20:40:03.027409 #24165] DEBUG -- tcpsocket[3f8c5d39f250]: sent 144 bytes
D, [2012-03-06T20:40:03.032446 #24165] DEBUG -- tcpsocket[3f8c5d39f250]: read 720 bytes
D, [2012-03-06T20:40:03.032563 #24165] DEBUG -- tcpsocket[3f8c5d39f250]: received packet nr 2 type 33 len 700
D, [2012-03-06T20:40:03.035433 #24165] DEBUG -- tcpsocket[3f8c5d39f250]: queueing packet nr 3 type 21 len 20
D, [2012-03-06T20:40:03.035505 #24165] DEBUG -- tcpsocket[3f8c5d39f250]: sent 24 bytes
D, [2012-03-06T20:40:03.035617 #24165] DEBUG -- tcpsocket[3f8c5d39f250]: received packet nr 3 type 21 len 12
D, [2012-03-06T20:40:03.035954 #24165] DEBUG -- net.ssh.authentication.session[3f8c5e189fdc]: beginning authentication of `webistrano'
D, [2012-03-06T20:40:03.036081 #24165] DEBUG -- tcpsocket[3f8c5d39f250]: queueing packet nr 4 type 5 len 28
D, [2012-03-06T20:40:03.036130 #24165] DEBUG -- tcpsocket[3f8c5d39f250]: sent 52 bytes
D, [2012-03-06T20:40:03.074777 #24165] DEBUG -- tcpsocket[3f8c5d39f250]: read 52 bytes
D, [2012-03-06T20:40:03.074912 #24165] DEBUG -- tcpsocket[3f8c5d39f250]: received packet nr 4 type 6 len 28
D, [2012-03-06T20:40:03.075018 #24165] DEBUG -- net.ssh.authentication.session[3f8c5e189fdc]: trying password
E, [2012-03-06T20:40:03.075077 #24165] ERROR -- net.ssh.authentication.session[3f8c5e189fdc]: all authorization methods failed (tried password)
/usr/lib64/ruby/gems/1.8/gems/net-ssh-2.3.0/lib/net/ssh.rb:200:in `start': webistrano (Net::SSH::AuthenticationFailed)
from ssh-test.rb:6
I wrote a simple PHP code, and it works:
<?php
$connection = ssh2_connect('ks2.so.d-sites.com', 22);
if (ssh2_auth_pubkey_file($connection, 'webistrano',
'/home/webistrano/.ssh/id_rsa.pub',
'/home/webistrano/.ssh/id_rsa')) {
echo "Identification réussie en utilisant une clé publique\n";
} else {
die('Echec de l\'identification en utilisant une clé publique');
}
Is there someone that can help me?
Thanks a lot in advance!
I've found how to solve that problem: adding a :auth_methods => ['publickey','password'] parameter in the Net::SSH.start function.
I have seen that error before,try uninstalling the gem 'net-ssh',try this:gem uninstall net-ssh -v 2.8.0, it works for me,
refer to: https://stackoverflow.com/a/21566548/3159604
I didn't use the Net::SSH.start solution but opted to set :ssh_options, {:forward_agent => true, :auth_methods => 'publickey'} which seems to work on my end.
Noted that Initially, I just used set :ssh_options, {:forward_agent => true} on 3 machines running on FreeBSD, CentOS and MacOSX and it worked fine - even used on production releases, however after testing on a VBoxed Vagrant-based CentOS (6.4) machine it stopped working. So, adding auth_method did the trick.
Just running ssh-add (in the terminal) and trying again (to deploy) worked for me.