I have some periodic tasks which I execute with Celery (parse pages).
Also I established a websocket with tornado.
I want to pass data from periodic tasks to tornado, then write this data to websocket and use this data on my html page.
How can I do this?
I tried to import module with tornado websocket from my module with celery tasks, but ofcourse, that didn't work.
I know only how to return some data, if I get a message from my client-side. Here is how I cope with it:
import tornado.httpserver
import tornado.websocket
import tornado.ioloop
import tornado.web
import socket
'''
This is a simple Websocket Echo server that uses the Tornado websocket handler.
Please run `pip install tornado` with python of version 2.7.9 or greater to install tornado.
This program will echo back the reverse of whatever it recieves.
Messages are output to the terminal for debuggin purposes.
'''
class handler():
wss = []
class WSHandler(tornado.websocket.WebSocketHandler):
def open(self):
print ('new connection')
if self not in handler.wss:
handler.wss.append(self)
def on_message(self, message):
print ('message received: ' + message)
wssend('Ihaaaa')
def on_close(self):
print ('connection closed')
if self in handler.wss:
handler.wss.remove(self)
def check_origin(self, origin):
return True
def wssend(message):
print(handler.wss)
for ws in handler.wss:
if not ws.ws_connection.stream.socket:
print ("Web socket does not exist anymore!!!")
handler.wss.remove(ws)
else:
print('I am trying!')
ws.write_message(message)
print('tried')
application = tornado.web.Application([
(r'/ws', WSHandler),
])
if __name__ == "__main__":
http_server = tornado.httpserver.HTTPServer(application)
http_server.listen(8888)
myIP = socket.gethostbyname(socket.gethostname())
print ('*** Websocket Server Started at %s***' % myIP)
main_loop = tornado.ioloop.IOLoop.instance()
main_loop.start()
The option is to make a handle in tornado and then post results of celery task to this handle.
After that, there will be an opportunity to pass this data to websocket.
Related
I'm trying to send data to server from client but I get bad namespace error.
I searched for the error but didn't find any solution...
Can anyone point out what mistake I've done.
server side (app.py)
from flask import Flask, render_template, request, jsonify
from flask_socketio import SocketIO
app = Flask(__name__)
socket_app = SocketIO(app)
#socket_app.on('userdata', namespace='/test')
def username(data):
print(data)
print(request.sid)
if __name__ == '__main__':
socket_app.run(app, debug=True, host='127.0.0.1', port=5000)
client side(client.py)
import socketio
sio = socketio.Client(engineio_logger=True)
data = {}
data['name'] = 'Mark'
sio.connect('http://127.0.0.1:5000')
sio.emit('userdata', data , namespace='/test')
sio.wait()
error
Attempting polling connection to http://127.0.0.1:5000/socket.io/?transport=polling&EIO=4
Polling connection accepted with {'sid': '1ipsM7udDVMj0WdKAAAA', 'upgrades': ['websocket'], 'pingTimeout': 5000, 'pingInterval': 25000}
Sending packet MESSAGE data 0
Attempting WebSocket upgrade to ws://127.0.0.1:5000/socket.io/?transport=websocket&EIO=4
WebSocket upgrade was successful
Traceback (most recent call last):
File "client.py", line 41, in <module>
sio.emit('userdata', data , namespace='/test')
File "/home/amit/anaconda3/envs/ANPR/lib/python3.6/site-packages/socketio/client.py", line 329, in emit
namespace + ' is not a connected namespace.')
socketio.exceptions.BadNamespaceError: /test is not a connected namespace.
Received packet NOOP data
flask & socketio versions
python-socketio==5.0.4 python-engineio==4.0.0 Flask==1.0.2
Flask-SocketIO==5.0.1
It looks like the OP figured out their own issue but when connecting to a non-default namespace you can specify it in the client connection:
sio.connect('http://127.0.0.1:5000', namespaces=['/test'])
I got this working by having the connect handler invoked in server.
#socket_app.on('connect', namespace='/test')
def test_connect():
print("connected")
emit('connect_custom', {'data': 'Connected'})
which then emits to client's connect_custom handler.
#sio.on('connect_custom', namespace='/test')
def on_connect(data):
print('client contacted by server')
print(data)
data = {}
data['name'] = 'Mark'
sio.emit('userdata', data, namespace='/test')
sio.connect('http://127.0.0.1:5000')
Good evening everyone. I'm not quite new to this place but finally decided to register and ask for a help. I develop a web application using Quart framework (asynchronous Flask). And now as application became bigger and more complex I decided to separate different procedures to different server instances, this is mostly because I want to keep web server clean, more abstract and free of computational load.
So I plan to use one web server with a few (if needed) identical procedure servers. All servers are based on quart framework, for now just for simplicity of development. I decided to use Crossbar.io router and autobahn to connect all servers together.
And here the problem occurred.
I followed this posts:
Running several ApplicationSessions non-blockingly using autbahn.asyncio.wamp
How can I implement an interactive websocket client with autobahn asyncio?
How I can integrate crossbar client (python3,asyncio) with tkinter
How to send Autobahn/Twisted WAMP message from outside of protocol?
Seems like I tried all possible approaches to implement autobahn websocket client in my quart application. I don't know how to make it possible so both things are working, whether Quart app works but autobahn WS client does not, or vice versa.
Simplified my quart app looks like this:
from quart import Quart, request, current_app
from config import Config
# Autobahn
import asyncio
from autobahn import wamp
from autobahn.asyncio.wamp import ApplicationSession, ApplicationRunner
import concurrent.futures
class Component(ApplicationSession):
"""
An application component registering RPC endpoints using decorators.
"""
async def onJoin(self, details):
# register all methods on this object decorated with "#wamp.register"
# as a RPC endpoint
##
results = await self.register(self)
for res in results:
if isinstance(res, wamp.protocol.Registration):
# res is an Registration instance
print("Ok, registered procedure with registration ID {}".format(res.id))
else:
# res is an Failure instance
print("Failed to register procedure: {}".format(res))
#wamp.register(u'com.mathservice.add2')
def add2(self, x, y):
return x + y
def create_app(config_class=Config):
app = Quart(__name__)
app.config.from_object(config_class)
# Blueprint registration
from app.main import bp as main_bp
app.register_blueprint(main_bp)
print ("before autobahn start")
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
runner = ApplicationRunner('ws://127.0.0.1:8080 /ws', 'realm1')
future = executor.submit(runner.run(Component))
print ("after autobahn started")
return app
from app import models
In this case application stuck in runner loop and whole application does not work (can not serve requests), it becomes possible only if I interrupt the runners(autobahn) loop by Ctrl-C.
CMD after start:
(quart-app) user#car:~/quart-app$ hypercorn --debug --error-log - --access-log - -b 0.0.0.0:8001 tengine:app
Running on 0.0.0.0:8001 over http (CTRL + C to quit)
before autobahn start
Ok, registered procedure with registration ID 4605315769796303
after pressing ctrl-C:
...
^Cafter autobahn started
2019-03-29T01:06:52 <Server sockets=[<socket.socket fd=11, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 8001)>]> is serving
How to make it possible to work quart application with autobahn client together in non-blocking fashion? So autobahn opens and keeps websocket connection to Crossbar router and silently listen on background.
Well, after many sleepless nights I finally found a good approach to solve this conundrum.
Thanks to this post C-Python asyncio: running discord.py in a thread
So, I rewrote my code like this and was able to run my Quart app with autobahn client inside, and both are actively working in nonblocking fashion.
The whole __init__.py looks like:
from quart import Quart, request, current_app
from config import Config
def create_app(config_class=Config):
app = Quart(__name__)
app.config.from_object(config_class)
# Blueprint registration
from app.main import bp as main_bp
app.register_blueprint(main_bp)
return app
# Autobahn
import asyncio
from autobahn import wamp
from autobahn.asyncio.wamp import ApplicationSession, ApplicationRunner
import threading
class Component(ApplicationSession):
"""
An application component registering RPC endpoints using decorators.
"""
async def onJoin(self, details):
# register all methods on this object decorated with "#wamp.register"
# as a RPC endpoint
##
results = await self.register(self)
for res in results:
if isinstance(res, wamp.protocol.Registration):
# res is an Registration instance
print("Ok, registered procedure with registration ID {}".format(res.id))
else:
# res is an Failure instance
print("Failed to register procedure: {}".format(res))
def onDisconnect(self):
print('Autobahn disconnected')
#wamp.register(u'com.mathservice.add2')
def add2(self, x, y):
return x + y
async def start():
runner = ApplicationRunner('ws://127.0.0.1:8080/ws', 'realm1')
await runner.run(Component) # use client.start instead of client.run
def run_it_forever(loop):
loop.run_forever()
asyncio.get_child_watcher() # I still don't know if I need this method. It works without it.
loop = asyncio.get_event_loop()
loop.create_task(start())
print('Starting thread for Autobahn...')
thread = threading.Thread(target=run_it_forever, args=(loop,))
thread.start()
print ("Thread for Autobahn has been started...")
from app import models
With this scenario we create task with autobahn's runner.run and attach it to the current loop and then run this loop forever in new thread.
I was quite satisfied with current solution.... but then then was found out that this solution has some drawbacks, that was crucial for me, for example: reconnect if connection dropped (i.e crossbar router becomes unavailable). With this approach if connection was failed to initialize or dropped after a while it will not try to reconnect. Additionally for me it wasn't obvious how to ApplicationSession API, i.e. to register/call RPC from the code in my quart app.
Luckily I spotted another new component API that autobahn used in their documentation:
https://autobahn.readthedocs.io/en/latest/wamp/programming.html#registering-procedures
https://github.com/crossbario/autobahn-python/blob/master/examples/asyncio/wamp/component/backend.py
It has auto reconnect feature and it's easy to register functions for RPC using decorators #component.register('com.something.do'), you just need to import component before.
So here is the final view of __init__.py solution:
from quart import Quart, request, current_app
from config import Config
def create_app(config_class=Config):
...
return app
from autobahn.asyncio.component import Component, run
from autobahn.wamp.types import RegisterOptions
import asyncio
import ssl
import threading
component = Component(
transports=[
{
"type": "websocket",
"url": u"ws://localhost:8080/ws",
"endpoint": {
"type": "tcp",
"host": "localhost",
"port": 8080,
},
"options": {
"open_handshake_timeout": 100,
}
},
],
realm=u"realm1",
)
#component.on_join
def join(session, details):
print("joined {}".format(details))
async def start():
await component.start() #used component.start() instead of run([component]) as it's async function
def run_it_forever(loop):
loop.run_forever()
loop = asyncio.get_event_loop()
#asyncio.get_child_watcher() # I still don't know if I need this method. It works without it.
asyncio.get_child_watcher().attach_loop(loop)
loop.create_task(start())
print('Starting thread for Autobahn...')
thread = threading.Thread(target=run_it_forever, args=(loop,))
thread.start()
print ("Thread for Autobahn has been started...")
from app import models
I hope it will help somebody. Cheers!
I am trying to handle multiple concurrent requests using flask_socketio and eventlet. However, it does not work as expected: When function test1() is running, it blocks the execution of function test2() as seen in the output log.
How can I achieve that the server handles both requests simultaneously?
Server (Python):
import eventlet
eventlet.monkey_patch()
from flask import Flask, render_template
from flask_socketio import SocketIO, send, emit
app = Flask(__name__)
socketio = SocketIO(app, async_mode='eventlet')
#socketio.on('test1')
def test1():
print('test1 started')
do_complicated_calculation() # takes some time
print('test1 done')
#socketio.on('test2')
def test2():
print('test2')
if __name__ == '__main__':
socketio.run(app)
Client (JavaScript):
import io from 'socket.io-client';
socket = io('http://localhost:5000');
socket.emit('test1');
socket.emit('test2');
Expected Output:
test1 started
test2
test1 done
Actual Output:
test1 started
test1 done
test2
As discussed on GitHub, you need to insert socketio.sleep(0) calls as often as you can inside your long computation, ideally inside a loop so that it happens at regular intervals. That will allow the eventlet scheduler to give the CPU to your second task while the first task is running.
I am trying to create a WebSocket server using Tornado. What I would like to do is execute a specific command, that will dispatch a message for every cycle of the IOLoop.
To make it more clear; let's say I have the following WebSocket handler
class MyHandler(websocket.WebSocketHandler):
def auto_loop(self, *args, **kwargs):
self.write_message('automatic message')
Is there any way to run auto_loop on every IOLoop cycle, without blocking the main thread?
I suppose that I can use greenlets for that, but I am searching for a more Tornado-native solution.
Thank you
You shouldn't write a message on every IOLoop cycle: you'll overwhelm your system. You want to send it every few milliseconds or seconds. A coroutine will do nicely:
import datetime
from tornado.ioloop import IOLoop
from tornado import gen
handlers = set()
#gen.coroutine
def auto_loop():
while True:
for handler in handlers:
handler.write_message('automatic message')
yield gen.Task(
IOLoop.current().add_timeout,
datetime.timedelta(milliseconds=500))
if __name__ == '__main__':
# ... application setup ...
# Start looping.
auto_loop()
IOLoop.current().start()
In MyHandler.open(), do handlers.add(self), and in MyHandler.on_close() do handlers.discard(self).
You might also be interested in PeriodicCallback
http://tornado.readthedocs.org/en/latest/ioloop.html#tornado.ioloop.PeriodicCallback
def callback(self):
self.write_message("Hi there!")
def open(self):
self.write_message("Connected.")
self.pCallback = PeriodicCallback(self.callback,
callback_time=250)
self.pCallback.start()
Inspired by ipython-notebook-proxy, and based on ipydra, and extending the latter to support more complex user authentication as well as a proxy, because in my use case, only port 80 can be exposed.
I am using flask-sockets for the gunicorn worker, but I am having troubles to proxy WebSockets. IPython uses three different WebSockets connections, /shell, /stdin, and /iopub, but I am only able to get the 101 Switching Protocols for the first two. And /stdin receives a Connection Close Frame as soon as is created.
This is the excerpt code in question:
# Flask imports...
from werkzeug import LocalProxy
from ws4py.client.geventclient import WebSocketClient
# I use my own LocalProxy because flask-sockets does not support Werkzeug Rules
websocket = LocalProxy(lambda: request.environ.get('wsgi.websocket', None))
websockets = {}
PROXY_DOMAIN = "127.0.0.1:8888" # IPython host and port
methods = ["GET", "POST", "PUT", "DELETE", "HEAD", "OPTIONS", "PATCH",
"CONNECT"]
#app.route('/', defaults={'url': ''}, methods=methods)
#app.route('/<path:url>', methods=methods)
def proxy(url):
with app.test_request_context():
if websocket:
while True:
data = websocket.receive()
websocket_url = 'ws://{}/{}'.format(PROXY_DOMAIN, url)
if websocket_url not in websockets:
client = WebSocketClient(websocket_url,
protocols=['http-only', 'chat'])
websockets[websocket_url] = client
else:
client = websockets[websocket_url]
client.connect()
if data:
client.send(data)
client_data = client.receive()
if client_data:
websocket.send(client_data)
return Response()
I also tried to create my own WebSocket proxy class, but it doesn't work either.
class WebSocketProxy(WebSocketClient):
def __init__(self, to, *args, **kwargs):
self.to = to
print(("Proxy to", self.to))
super(WebSocketProxy, self).__init__(*args, **kwargs)
def opened(self):
m = self.to.receive()
print("<= %d %s" % (len(m), str(m)))
self.send(m)
def closed(self, code, reason):
print(("Closed down", code, reason))
def received_message(self, m):
print("=> %d %s" % (len(m), str(m)))
self.to.send(m)
Regular request-response cycle works like a charm, so I removed that code. If interested, the complete code is hosted in hidra.
I run the server with
$ gunicorn -k flask_sockets.worker hidra:app
Here is my solution(ish). It is crude, but should serve as a starting point for building websocket proxy. The full code is available in unreleased project, pyramid_notebook.
This uses ws4py and uWSGI instead of gunicorn
We use uWSGI's internal mechanism to receive downstream websocket message loop. There is nothing like WSGI for websockets in Python world (yet?), but looks like every web server implements its own mechanism.
A custom ws4py ProxyConnection is created which can combine ws4py event loop with uWSGI event loop
The thing is started and messages start fly around
This uses Pyramid request (based on WebOb), but this really shouldn't matter and code should be fine for any Python WSGI app with little modifications
As you can see, this does not really take advantage of asynchronicity, but just sleep() if there is nothing coming in from the socket
Code goes here:
"""UWSGI websocket proxy."""
from urllib.parse import urlparse, urlunparse
import logging
import time
import uwsgi
from ws4py import WS_VERSION
from ws4py.client import WebSocketBaseClient
#: HTTP headers we need to proxy to upstream websocket server when the Connect: upgrade is performed
CAPTURE_CONNECT_HEADERS = ["sec-websocket-extensions", "sec-websocket-key", "origin"]
logger = logging.getLogger(__name__)
class ProxyClient(WebSocketBaseClient):
"""Proxy between upstream WebSocket server and downstream UWSGI."""
#property
def handshake_headers(self):
"""
List of headers appropriate for the upgrade
handshake.
"""
headers = [
('Host', self.host),
('Connection', 'Upgrade'),
('Upgrade', 'websocket'),
('Sec-WebSocket-Key', self.key.decode('utf-8')),
# Origin is proxyed from the downstream server, don't set it twice
# ('Origin', self.url),
('Sec-WebSocket-Version', str(max(WS_VERSION)))
]
if self.protocols:
headers.append(('Sec-WebSocket-Protocol', ','.join(self.protocols)))
if self.extra_headers:
headers.extend(self.extra_headers)
logger.info("Handshake headers: %s", headers)
return headers
def received_message(self, m):
"""Push upstream messages to downstream."""
# TODO: No support for binary messages
m = str(m)
logger.debug("Incoming upstream WS: %s", m)
uwsgi.websocket_send(m)
logger.debug("Send ok")
def handshake_ok(self):
"""
Called when the upgrade handshake has completed
successfully.
Starts the client's thread.
"""
self.run()
def terminate(self):
raise RuntimeError("NO!")
super(ProxyClient, self).terminate()
def run(self):
"""Combine async uwsgi message loop with ws4py message loop.
TODO: This could do some serious optimizations and behave asynchronously correct instead of just sleep().
"""
self.sock.setblocking(False)
try:
while not self.terminated:
logger.debug("Doing nothing")
time.sleep(0.050)
logger.debug("Asking for downstream msg")
msg = uwsgi.websocket_recv_nb()
if msg:
logger.debug("Incoming downstream WS: %s", msg)
self.send(msg)
s = self.stream
self.opened()
logger.debug("Asking for upstream msg")
try:
bytes = self.sock.recv(self.reading_buffer_size)
if bytes:
self.process(bytes)
except BlockingIOError:
pass
except Exception as e:
logger.exception(e)
finally:
logger.info("Terminating WS proxy loop")
self.terminate()
def serve_websocket(request, port):
"""Start UWSGI websocket loop and proxy."""
env = request.environ
# Send HTTP response 101 Switch Protocol downstream
uwsgi.websocket_handshake(env['HTTP_SEC_WEBSOCKET_KEY'], env.get('HTTP_ORIGIN', ''))
# Map the websocket URL to the upstream localhost:4000x Notebook instance
parts = urlparse(request.url)
parts = parts._replace(scheme="ws", netloc="localhost:{}".format(port))
url = urlunparse(parts)
# Proxy initial connection headers
headers = [(header, value) for header, value in request.headers.items() if header.lower() in CAPTURE_CONNECT_HEADERS]
logger.info("Connecting to upstream websockets: %s, headers: %s", url, headers)
ws = ProxyClient(url, headers=headers)
ws.connect()
# Happens only if exceptions fly around
return ""