I have the following that does not work.
import asyncio
from pyppeteer import launch
import logging
logging.basicConfig(level=logging.DEBUG)
async def main():
browser = await launch()
page = await browser.newPage()
await page.goto('https://example.com')
await page.screenshot({'path': 'example.png'})
await browser.close()
asyncio.get_event_loop().run_until_complete(main())
It shows something like this.
...
[D:pyppeteer.connection.Connection] RECV: {"id":18,"result":{}}
DEBUG:websockets.client:< TEXT '{"method":"Target.targetDestroyed","params":{"t...B9D141622A1A8C504495"}}' [92 bytes]
DEBUG:websockets.client:= connection is CLOSING
DEBUG:websockets.client:> CLOSE 1000 (OK) [2 bytes]
DEBUG:websockets.client:< TEXT '{"method":"Target.targetDestroyed","params":{"t...1792091509A69FA2AACC"}}' [92 bytes]
DEBUG:websockets.client:! failing connection with code 1006
DEBUG:websockets.client:= connection is CLOSED
If I delete the following lines, it works.
import logging
logging.basicConfig(level=logging.DEBUG)
What is the correct way to show the requests sent in this example code yet still make sure the code works?
Also, the ws requests are not printed completely (you see "..."). How to show the ws requests completely.
Related
I have a strange problem using the SFTP-API from django-storages(https://github.com/jschneier/django-storages). I am trying to use it in order to fetch media-files, which are stored on a different server and thus needed to create a Proxy for SFTP Downloads, since plain Django just sends GET-requests to the MEDIA_ROOT. I figured that Middleware provides a good hook:
import mimetypes
from storages.backends.sftpstorage import SFTPStorage
from django.http import HttpResponse
from storages.backends.sftpstorage import SFTPStorage
class SFTPMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
# Code to be executed for each request before
# the view (and later middleware) are called.
response = self.get_response(request)
try:
path = request.get_full_path()
SFTP = SFTPStorage() # <- this is where the magic happens
if SFTP.exists(path):
file = SFTP._read(path)
type, encoding = mimetypes.guess_type(path)
response = HttpResponse(file, content_type=type)
response['Content-Disposition'] = u'attachment; filename="{filename}"'.format(filename=path)
except PermissionError:
pass
return response
which works fine, but obviously it opens a new connection every time a website call is issued which I don't want (it also crashes after 3 reloads or something, I think it has to many parallel connections by then). So I tried just opening one connection to the Server via SFTP by moving the SFTP = SFTPStorage()-initialization into the __init__()-method which is just called once:
import mimetypes
from storages.backends.sftpstorage import SFTPStorage
from django.http import HttpResponse
from storages.backends.sftpstorage import SFTPStorage
class SFTPMiddleware:
def __init__(self, get_response):
self.get_response = get_response
self.SFTP = SFTPStorage() # <- this is where the magic happens
def __call__(self, request):
# Code to be executed for each request before
# the view (and later middleware) are called.
response = self.get_response(request)
try:
path = request.get_full_path()
if self.SFTP.exists(path):
file = self.SFTP._read(path)
type, encoding = mimetypes.guess_type(path)
response = HttpResponse(file, content_type=type)
response['Content-Disposition'] = u'attachment; filename="{filename}"'.format(filename=path)
except PermissionError:
pass
return response
But this implementation doesn't seem to work, the program is stuck either before the SFTP.exists() or after the SFTP._read() methods.
Can anybody tell me how to fix this problem? Or does anybody even have a better idea as to how to tackle this problem?
Thanks in advance,
Kingrimursel
I noticed that one url makes pyppeteer hang forever. That's weird because I've set timeout. Do you know where is the problem? Or is it asyncio problem?
async def test():
url = 'https://ig.com.br/'
browser = await launch(headless=True)
page = await browser.newPage()
page.setDefaultNavigationTimeout(1000)
await page.setUserAgent(fake_useragent.UserAgent().random)
await page.goto(url, {'waitUntil': 'networkidle0','timeout':1000})
await browser.close()
asyncio.run(test())
Other websites work well - it timeouts because the timeout value is too small. But this website makes it hang forever.
Do you know why? I want to make sure that any website will either be fetched under 20 seconds or raise error.
Having read the following question:
How to save message from telegram channel as variable
I need to do the same but from a NewMessage event, storing the content of the message in a variable.
However, neither event.text nor event.raw_test seem to be storable in a variable
The following code:
import asyncio
from telethon import TelegramClient, events
import logging
logging.basicConfig(format='[%(levelname) 5s/%(asctime)s] %(name)s: %(message)s',
level=logging.WARNING)
client = TelegramClient('session', 'api_id', 'api_hash')
client.start()
channel = 'xxx'
async def main():
#client.on(events.NewMessage(chats=channel))
async def handler(event):
await print (event.text)
await client.run_until_disconnected()
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
works printing the new channel message, but it gives me two errors along the printed message:
await callback(event)
TypeError: object NoneType can't be used in 'await' expression
But when I change
await print (event.text)
for
msg = await event.text
print(msg)
I get the same two errors but this time nothing is printed...and I need to save the text from the message as a variable in order to continue the script.
It also doesnt work declaring the variable msg before the function and making it global inside it.
I dont know what else to try. Thanks in advance.
The Telethon docs covers this quite well (adapted for your use case):
from telethon import TelegramClient, events
client = TelegramClient('session', api_id, api_hash)
channel = "xxx"
#client.on(events.NewMessage(chats=channel))
async def my_event_handler(event):
print(event.text) # this doesn't need an "await"
client.start()
client.run_until_disconnected()
Also notice that I haven't put the event handler in another function, and the run_until_disconnected() call doesn't call any function, nor is it in one. You don't even need to import asyncio
from telethon import TelegramClient, events
import logging
logging.basicConfig(format='[%(levelname) 5s/%(asctime)s] %(name)s: %(message)s',
level=logging.WARNING)
client = TelegramClient('session', 'api_id', 'api_hash')
client.start()
channel = 'xxx'
#client.on(events.NewMessage(chats=channel))
async def handler(event):
print(event.message.message)
client.run_until_disconnected()
You don't need to wrap listener with another async function. And also, you don't need to await print, just use plain print
I try to use aiohttp 3.6.2 both server and client:
For webhook perform work:
1) Get JSON-request from service
2) Fast send HTTP 200 OK back to service
3) Made additional work after: make http-request to slow web-service(answer 2-5 sec)
I dont understand how to perform work after view(or handler) returned web.Response(text="OK")?
Current view:
(it's slow cause slow http_request perform before response)
view.py:
async def make_http_request(url):
async with aiohttp.ClientSession() as session:
async with session.get(url) as resp:
print(await resp.text())
async def work_on_request(request):
url = (await request.json())['url']
await make_http_request(url)
return aiohttp.web.Response(text='all ok')
routes.py:
from views import work_on_request
def setup_routes(app):
app.router.add_get('/', work_on_request)
server.py:
from aiohttp import web
from routes import setup_routes
import asyncio
app = web.Application()
setup_routes(app)
web.run_app(app)
So, workaround for me is to start one more thread with different event_loop, or may be you know how to add some work to current event loop?
Already not actual, cause i found desicion to add one more task to main event_loop:
//additionaly i created one global queue to interoperate coroutine between each other.
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
queue = asyncio.Queue(maxsize=100000)
loop.create_task(worker('Worker1', queue))
app = web.Application()
app['global_queue'] = queue
I use aiohttp to send a request like so:
async with ClientSession() as session:
res = await session.get("http://0.0.0.0:8000/./")
When I start up a http server using python like so:
python3 -m http.server
I see that the path is normalized, i.e., the server gets the following request:
GET / HTTP/1.1" 200
How do I disable this normalization, to enforce a behavior like in urrlib, for example, where urllib.request.urlopen("http://0.0.0.0:8000/./") results in the following request:
GET /./ HTTP/1.1.
aiohttp uses yarl for URL processing.
session.get('http://example.com') works as well as session.get(yarl.URL('http://example.com'))
You can disable URL encoding for yarl.URL with encoded=True, but you have to take care of URL correctness.
e.g.
import asyncio
import yarl
import aiohttp
async def test():
url = yarl.URL('https://stackoverflow.com/./', encoded=True)
async with aiohttp.ClientSession() as session:
async with session.get(url, allow_redirects=False) as resp:
print(resp.url)
asyncio.run(test())