I am sending a get request to any host using sockets tcp, but I keep on getting "301 Moved Permanently" from pages with https.
I have tried to do it by changing the port from 80 to 443.
I have tried with the ssl library as well.
But keep getting 301 code
This is the code
import socket
import click
#click.command()
#click.option("-h", "--host", prompt=True)
#click.option("-p", "--port", type=int, prompt=True, default=80)
def cli(host, port):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((host, port))
message = f"GET / HTTP/1.1\r\nHost:{host}\r\nConnection: close\r\n\r\n"
request = message.encode('utf-8')
sent = 0
while sent < len(request):
sent = sent + sock.send(request[sent:])
response = b""
while True:
chunk = sock.recv(4096)
if len(chunk) == 0: # If no more data received, quitting
break
response = response + chunk
response_decode = response.decode('latin-1')
sock.close()
print(response_decode)
This is the response when I try to connect to www.eltiempo.com by port 80
HTTP/1.1 301 Moved Permanently
Server: AkamaiGHost
Content-Length: 0
Location: https://www.eltiempo.com/
Cache-Control: max-age=120
Expires: Sat, 12 Feb 2022 18:24:28 GMT
Date: Sat, 12 Feb 2022 18:22:28 GMT
Connection: close
Server-Timing: cdn-cache; desc=HIT
Server-Timing: edge; dur=1
version: desktop
I get this error with port 443
chunk = sock.recv(4096)
ConnectionResetError: [Errno 104] Connection reset by peer
Please tell me how to improve my code to avoid this 301 code.
Related
I need check if response is an image.
For requirements of the work I need to generate the url of the photos that can exist or no and record the url that contains an image.
When the url generated doesn't show a photo the response of the website is an html when the body is:
<body>No File Found</body>
also the response.status =200
The response header doesn't have a valuable info for both results with image and No File Found
For instance
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store, must-revalidate
Pragma: no-cache
Transfer-Encoding: chunked
Expires: 0
Server: Microsoft-IIS/8.5
X-Powered-By: ASP.NET
X-Frame-Options: AllowAll
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: *
Date: Tue, 13 Aug 2019 01:44:40 GMT
The way that I found to check if the response is an image for this case was:
try :
no_file_found = response.xpath("/html/body[contains(., 'No File Found')]")
except:
photo_url = response.url
photo = PhotoItem()
photo['id'] = id
photo['url'] = photo_url
yield photo
Because When the response is an image the line
no_file_found = response.xpath("/html/body[contains(., 'No File Found')]")
throw this exception:
raise NotSupported("Response content isn't text")
I know that this isn't an elegant solution , but for this context it works
Question
My question is If there is another way more elegant to solve this problem, that not use try to solve that.
Notice that I don't need to download the image just need to record the valid url
Any suggestion is welcome.
Thanks in advance!!!
The simplest way would probably be to just check the type of the response:
from scrapy.http.response.text import TextResponse
if not isinstance(response, TextResponse):
# it's probably an image; do image stuff
I'm having a bit of a hard time receiving an image using sockets. I think the problem is related to the fact that sockets send both a header and the actual image, and that the two need different decoding.
This is the code:
import socket
mysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysock.connect(('www.py4inf.com', 80))
mysock.send(
'GET http://www.py4inf.com/cover.jpg HTTP/1.0\n\n'.encode('utf-8'))
count = 0
fhand = open("stuff.jpg", "wb")
while True:
data = mysock.recv(512)
if len(data) < 1:
break
fhand.write(data)
mysock.close()
fhand.close()
Yes, there is a header. The end of it is after the first \r\n\r\n sequence. Once you see that sequence send the rest to a file. Here's a crude fix:
import socket
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as mysock:
mysock.connect(('www.py4inf.com', 80))
mysock.send(b'GET http://www.py4inf.com/cover.jpg HTTP/1.0\n\n')
header = b''
while True:
data = mysock.recv(512)
if not data:
raise RuntimeError('no header?')
header += data
# end-of-header in buffer yet?
eoh = header.find(b'\r\n\r\n')
if eoh != -1:
break
# split the header off and keep data read after it.
header,data = header[:eoh+4],header[eoh+4:]
print(header.decode())
with open("stuff.jpg", "wb") as fhand:
fhand.write(data)
while True:
data = mysock.recv(512)
if not data:
break
fhand.write(data)
Here's the header. Note that the content length is in the header, so if you were to send an HTTP request with a keepalive, you would have to read exactly that many bytes after the header. since Connection: close is specified, you only have to read until no more data is received.
HTTP/1.1 200 OK
Date: Sun, 22 May 2016 23:22:20 GMT
Server: Apache
Last-Modified: Fri, 04 Dec 2015 19:05:04 GMT
ETag: "b294001f-111a9-526172f5b7cc9"
Accept-Ranges: bytes
Content-Length: 70057
Connection: close
Content-Type: image/jpeg
I'm trying to test out the failure recovery behavior of ZeroMQ ( via pyzmq ) when using DEALER and ROUTER sockets. Here's my code:
import sys, zmq
import threading
import time, gc
import socket
def tprint(msg):
"""like print, but won't get newlines confused with multiple threads"""
sys.stdout.write(msg + '\n')
sys.stdout.flush()
class ClientWorker(threading.Thread):
def __init__(self, id, ports):
self.id = id
self.ports = ports
super(ClientWorker, self).__init__()
def run(self):
context = zmq.Context()
socket = context.socket(zmq.DEALER)
for port in self.ports:
socket.connect("tcp://localhost:%d" % port)
tprint("client %d started" % (self.id))
for ia in xrange(self.id*100,self.id*100+100):
socket.send_string('request %d' % (ia))
time.sleep(1)
socket.close()
context.term()
class ServerWorker(threading.Thread):
def __init__(self, port, maxReq=None):
self.port = port
self.maxReq = maxReq
super(ServerWorker, self).__init__()
def run(self):
context = zmq.Context()
socket = context.socket(zmq.ROUTER)
socket.bind("tcp://127.0.0.1:%d" % (self.port))
tprint("server started on port %d" % (self.port))
numReq = 0
while True:
ident, msg = socket.recv_multipart()
print self.port, ident, msg
numReq += 1
if self.maxReq and numReq >= self.maxReq:
tprint("server on port %d exiting" % (self.port))
break
socket.unbind("tcp://127.0.0.1:%d" % (self.port))
socket.close()
context.term()
def main():
ports = [5555,5556,5557]
servers = [ServerWorker(port,10 if port==5555 else None) for port in ports]
for s in servers: s.start()
for ia in xrange(1,6):
w = ClientWorker(ia, ports)
w.start()
servers[0].join()
servers[0] = None
gc.collect()
time.sleep(30)
tprint("restarting server")
s = ServerWorker(port)
s.start()
if __name__ == "__main__":
main()
The behavior I observe is as follows:
the server at 5555 will print out 10 items that it receives at which point it exits
the client workers will NOT detect this failure, and will continue sending items to that server
when I attempt to re-.bind() a new server thread to port 5555, I get the "Address in use" error
this despite my closing the socket, calling context.term(), attempting to gc the server object, etc.
Three questions:
Am I correct to expect that the DEALER sockets should be able to detect the failure of one of the servers and redistribute the work to the remaining servers? I suspect that perhaps the reason why it can't detect the failure is the same reason that the socket on port 5555 remains open?
Any ideas about the "Address in use" error?
Am I correct to expect that when I reconnect the server to port 5555, the clients will be able to detect the reconnection and resume sending messages to the server in a round-robin way taking into account the new server?
Am I correct to expect that the DEALER sockets should be able to detect the failure of one of the servers and redistribute the work to the remaining servers?
No, this isn't how DEALERS work. DEALERs that connect load-balance across their peers, whether or not they are there. That means that messages are still queued to worker 5555, even while it's down. Those messages will be delivered immediately when worker 5555 returns.
Any ideas about the "Address in use" error?
This is caused by the fact that port when you start the resumed worker is ports[-1], not ports[0], so it's binding to a port that's still in use by one of your workers, not the one that stopped.
Am I correct to expect that when I reconnect the server to port 5555, the clients will be able to detect the reconnection and resume sending messages to the server in a round-robin way taking into account the new server?
Yes, messages will resume being delivered to 5555 when it comes back, but I think you aren't quite right about which messages will be delivered there.
With some minor adjustments to your script, I get the output:
server started on port 5555
server started on port 5556
server started on port 5557
client 1 started
client 2 started
client 3 started
client 4 started
client 5 started
5555 00800041a7 request 100
5555 00800041a8 request 200
5555 00800041a9 request 300
5555 00800041aa request 400
5555 00800041ab request 500
5556 0060b7acd9 request 101
5556 0060b7acdb request 301
5556 0060b7acdc request 401
5556 0060b7acdd request 501
5556 0060b7acda request 201
5557 004431b782 request 102
5557 004431b784 request 302
5557 004431b783 request 202
5557 004431b785 request 402
5557 004431b786 request 502
5555 00800041a7 request 103
5555 00800041a9 request 303
5555 00800041ab request 503
5555 00800041a8 request 203
5555 00800041aa request 403
server on port 5555 exiting
5556 0060b7acd9 request 104
5556 0060b7acda request 204
5556 0060b7acdd request 504
5556 0060b7acdb request 304
5556 0060b7acdc request 404
5557 004431b782 request 105
5557 004431b786 request 505
5557 004431b783 request 205
5557 004431b784 request 305
5557 004431b785 request 405
5556 0060b7acd9 request 107 <- note jump from 405 to 107
5556 0060b7acdc request 407
5556 0060b7acdd request 507
5556 0060b7acda request 207
5556 0060b7acdb request 307
restarting server on 5555
server started on port 5555
5557 004431b786 request 508
5557 004431b782 request 108
5557 004431b785 request 408
5557 004431b783 request 208
5557 004431b784 request 308
5555 0041c8aac3 request 506 <- here are the X06 messages on the new 5555 worker
5555 0041c8aac4 request 306
5555 0041c8aac5 request 406
5555 0041c8aac6 request 106
5555 0041c8aac7 request 206
5555 0041c8aac7 request 209
5555 0041c8aac4 request 309
5555 0041c8aac3 request 509
5555 0041c8aac5 request 409
5555 0041c8aac6 request 109
5556 0060b7acdd request 510
5556 0060b7acdb request 310
5556 0060b7acda request 210
5556 0060b7acdc request 410
5556 0060b7acd9 request 110
5557 004431b784 request 311
5557 004431b786 request 511
...
Messages 106-506 were sent to 5555 and redelivered later. They were not re-routed to another worker when 5555 wasn't there to receive them.
You can use client_socket.hwm = N to limit how many messages may be pending on a worker before the client should start excluding it from round-robin, but you can't make it zero.
The version of your script that I used:
from binascii import hexlify
import threading
import socket
import sys
import time
import zmq
def tprint(msg):
"""like print, but won't get newlines confused with multiple threads"""
sys.stdout.write(msg + '\n')
sys.stdout.flush()
class ClientWorker(threading.Thread):
def __init__(self, id, ports):
self.id = id
self.ports = ports
super(ClientWorker, self).__init__()
def run(self):
context = zmq.Context.instance()
socket = context.socket(zmq.DEALER)
socket.hwm = 1 # limit messages sent to dead workers
for port in self.ports:
socket.connect("tcp://localhost:%d" % port)
tprint("client %d started" % (self.id))
for ia in xrange(self.id*100,self.id*100+100):
socket.send_string('request %d' % (ia))
time.sleep(1)
socket.close()
context.term()
class ServerWorker(threading.Thread):
def __init__(self, port, maxReq=None):
self.port = port
self.maxReq = maxReq
super(ServerWorker, self).__init__()
def run(self):
context = zmq.Context.instance()
socket = context.socket(zmq.ROUTER)
tprint("server started on port %d" % (self.port))
socket.bind("tcp://127.0.0.1:%d" % (self.port))
numReq = 0
while True:
ident, msg = socket.recv_multipart()
print self.port, hexlify(ident), msg
numReq += 1
if self.maxReq and numReq >= self.maxReq:
tprint("server on port %d exiting" % (self.port))
break
socket.close()
context.term()
def main():
ports = [5555,5556,5557]
servers = [ServerWorker(port,10 if port==5555 else None) for port in ports]
for s in servers: s.start()
for ia in xrange(1,6):
w = ClientWorker(ia, ports)
w.start()
servers[0].join()
time.sleep(10)
port = ports[0]
tprint("restarting server on %i" % port)
s = ServerWorker(port)
s.start()
if __name__ == "__main__":
ctx = zmq.Context.instance()
try:
main()
finally:
ctx.term()
1.From client:
root#amsys-LIFEBOOK-AH502:/home/amsys# radtest -t chap usr password 127.0.0.1 0 testing123
This is how,the way i sended a packet access-request packet from the client (here,loop back only).
2.From server.
the server responds to client as shown as below:
Ready to process requests.
Ignoring request to auth address * port 1812 as server default from unknown client 127.0.0.1 port 34962 proto udp
3.server to client
Sending Access-Request of id 67 from 0.0.0.0 port 47852 to 127.0.0.1 port 1812
User-Name = 'usr'
User-Password = 'password'
NAS-IP-Address = 127.0.1.1
NAS-Port = 0
Message-Authenticator = 0x00
radclient: no response from server for ID 67 socket 3
if anybody would aware about this thing,please give your prompt response and pleased me.thanking you.!
I'm using Jmeter and would like to identify the endtime of each request for each user.
Please take a look my testplan:
Thread group: 2 users
loop:1
2 HTTP request (request_1, request_2)
Start testing Web performance, the View Result tree shows: 4 results (2 for request_1, 2 for request_2)
request_2: 1 passed and 1 failed. Look in request table of result tree, I see:
Thread Name: jp#gc - Stepping Thread Group 1-1
Sample Start: 2014-04-18 09:28:06 ICT
Load time: 1100554
Latency: 550450
Size in bytes: 408190
Headers size in bytes: 4774
Body size in bytes: 403416
Sample Count: 1
Error Count: 0
Response code: 200
Response message: OK
Response headers:
HTTP/1.1 200 OK
Date: Fri, 18 Apr 2014 02:28:15 GMT
Server: Apache
X-Powered-By: PHP/5.3.3
Set-Cookie: ls23166422738597439695-runtime-publicportal=h4knpfldt76e3kvmunrn5i4u16; path=/limesurvey/
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
P3P: CP="IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT"
Last-Modified: Fri, 18 Apr 2014 02:36:09 GMT
Cache-Control: post-check=0, pre-check=0
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=utf-8
HTTPSampleResult fields:
ContentType: text/html; charset=utf-8
DataEncoding: utf-8
The questions are:
How to identify the time which cause request_2 is fail ? and how to display the endtime of each request for each user ?
How to displays information in the log panel of Jmeter (enable DEbug log mode on GUI), like "This is error....due to..."
Besides, as in the log panel (active log debug in GUI), some time the log entries stop at Thread 1-n (n=1,2...), after that 30s, the log is continue showing. So, I wonder about this time, web server has error, and in this time, Jmeter still send request or waiting Web server response ?
Thanks.
It can be done via Beanshell Pre Processor which you can add as a child of any "interesting" request.Example code would look like:
import java.util.Date;
long end_time_ms = prev.getEndTime(); // obtain sampler end time (in milliseconds from 1st Jan 1970)
Date end_time_date = new Date(end_time_ms); //convert it to human-readable date if you prefer
String response_message = prev.getResponseMessage(); // get initial response message
StringBuilder response = new StringBuilder(); // initialize StringBuilder to construct new response
response.append(response_message); // add initial response message
response.append(System.getProperty("line.separator")); // add new line
response.append("Thread finished at: ").append(end_time_date); // add thread finish date
prev.setResponseMessage(response.toString()); // set new response message
log.info("Thread finished at:" + end_time_date"); // to print it to the log
See above for Beanshell code and image below for UI impact
Never use GUI for anything apart from developing or debugging tests. If you want to add something to the log use log.info("something"); as above or JMeter __log() function