NodeMCU info
> Lua 5.1.4
> SDK 2.2.1
> Memory Usage :
> Total : 3260490 bytes
> Used : 9287 bytes
> Remain: 3251203 bytes
Error I get when I try to send HTTP response with big json string response (json_response)
PANIC: unprotected error in call to Lua API (file.lua:5: out of memory)
Code:
-- a simple HTTP server
srv = net.createServer(net.TCP)
srv:listen(80, function(conn)
conn:on("receive", function(sck, payload)
sck:send("HTTP/1.0 200 OK\r\nContent-Type: text/html\r\n\r\n"..json_response)
end)
conn:on("sent", function(sck) sck:close() end)
end)
Yes, that won't work if you're trying to send a lot of data. You need to send this piece-by-piece. Our API documentation shows two approaches (you would find further references here on SO) the first being this:
srv = net.createServer(net.TCP)
function receiver(sck, data)
local response = {}
-- if you're sending back HTML over HTTP you'll want something like this instead
-- local response = {"HTTP/1.0 200 OK\r\nServer: NodeMCU on ESP8266\r\nContent-Type: text/html\r\n\r\n"}
response[#response + 1] = "lots of data"
response[#response + 1] = "even more data"
response[#response + 1] = "e.g. content read from a file"
-- sends and removes the first element from the 'response' table
local function send(localSocket)
if #response > 0 then
localSocket:send(table.remove(response, 1))
else
localSocket:close()
response = nil
end
end
-- triggers the send() function again once the first chunk of data was sent
sck:on("sent", send)
send(sck)
end
srv:listen(80, function(conn)
conn:on("receive", receiver)
end)
Related
I want to use a NodeMCU device (Lua based top level) to act as a websocket server to 1 or more browser clients.
Luckily, there is code to do this here: NodeMCU Websocket Server
(courtesy of #creationix and/or #moononournation)
This works as described and I am able to send a message from the client to the NodeMCU server, which then responds based on the received message. Great.
My questions are:
How can I send messages to the client without it having to be sent as a response to a client request (standalone sending of data)? When I try to call socket.send() socket is not found as a variable, which I understand, but cannot work out how to do it! :(
Why does the decode() function output the extra variable? What is this for? I'm assuming it will be for packet overflow, but I can never seem to make it return anything, regardless of my message length.
In the listen method, why has the author added a queuing system? is this essential or for applications that perhaps may receive multiple simultaneous messages? Ideally, I'd like to remove it.
I have simplified the code as below:
(excluding the decode() and encode() functions - please see the link above for the full script)
net.createServer(net.TCP):listen(80, function(conn)
local buffer = false
local socket = {}
local queue = {}
local waiting = false
local function onSend()
if queue[1] then
local data = table.remove(queue, 1)
return conn:send(data, onSend)
end
waiting = false
end
function socket.send(...)
local data = encode(...)
if not waiting then
waiting = true
conn:send(data, onSend)
else
queue[#queue + 1] = data
end
end
conn:on("receive", function(_, chunk)
if buffer then
buffer = buffer .. chunk
while true do
local extra, payload, opcode = decode(buffer)
if opcode==8 then
print("Websocket client disconnected")
end
--print(type(extra), payload, opcode)
if not extra then return end
buffer = extra
socket.onmessage(payload, opcode)
end
end
local _, e, method = string.find(chunk, "([A-Z]+) /[^\r]* HTTP/%d%.%d\r\n")
local key, name, value
for name, value in string.gmatch(chunk, "([^ ]+): *([^\r]+)\r\n") do
if string.lower(name) == "sec-websocket-key" then
key = value
break
end
end
if method == "GET" and key then
acceptkey=crypto.toBase64(crypto.hash("sha1", key.."258EAFA5-E914-47DA-95CA-C5AB0DC85B11"))
conn:send(
"HTTP/1.1 101 Switching Protocols\r\n"..
"Upgrade: websocket\r\nConnection: Upgrade\r\n"..
"Sec-WebSocket-Accept: "..acceptkey.."\r\n\r\n",
function ()
print("New websocket client connected")
function socket.onmessage(payload,opcode)
socket.send("GOT YOUR DATA", 1)
print("PAYLOAD = "..payload)
--print("OPCODE = "..opcode)
end
end)
buffer = ""
else
conn:send(
"HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: 12\r\n\r\nHello World!",
conn.close)
end
end)
end)
I can only answer 1 question, the others may be better suited for the library author. Besides, SO is a format where you ask 1 question normally.
How can I send messages to the client without it having to be sent as a response to a client request (standalone sending of data)?
You can't. Without the client contacting the server first and establishing a socket connection the server wouldn't know where to send the messages to. Even with SSE (server-sent events) it's the client that first initiates a connection to the server.
I've inherited a classic asp project and as part of the upgrade process we're moving a lot of the business logic to a REST API (WebApi 2.2)
The authorization endpoint for the api is written, and the asp site can call it, but it's very slow compared with calling directly via Postman.
(I'm a C# coder not a VBScript one so the below code may be offensive)
Asp Code:
' Send a prebuilt HTTP request and handle the response
' Returns true if the request returns a 200 response, False otherwise
' Response body is placed in Response
' ErrorMessage is set to return status text if an error code is returned
Function HandleRequest(ByRef objRequest, strBody)
set profiler = Server.CreateObject("Softwing.Profiler")
HandleRequest = False
' Add auth token if we have it
If Not m_accessToken&"" = "" Then
objRequest.SetRequestHeader "Authorization", "Bearer " & m_accessToken
End If
' Originating IP for proxy forwarding
If Not m_clientIp&"" = "" Then
objRequest.SetRequestHeader "X-Forwarded-For", m_clientIp
End If
On Error Resume Next
If (strBody&"" = "") Then
objRequest.Send()
Else
profiler.ProfileStart()
objRequest.Send(strBody)
flSendRequest = profiler.ProfileStop()
End If
If Err.Number = 0 Then
Dim jsonResponse
If (objRequest.ResponseText&"" <> "") Then
profiler.ProfileStart()
set jsonResponse = JSON.parse(objRequest.ResponseText)
flJson = profiler.ProfileStop()
set m_Response = jsonResponse
End If
If objRequest.Status = 200 Then
HandleRequest = True
m_errorMessage = ""
Else
m_errorMessage = objRequest.statusText
End If
Else
m_errorMessage = "Unable to connect to Api server"
End If
On Error GoTo 0
End Function
You can see there's some profiling code in there.
The following post request takes 392ms
POST localhost:5000/oauth/token
Content-Type application/x-www-form-urlencoded
client_id:ABCDEF0-ABCD-ABCD-ABCD-ABCDEF-ABCDEF01234
client_secret:aBcDeF0123456789aBcDeF0123456789=
username:demo
password:demo
grant_type:password
If I issue the same request direct to the Api via Postman it takes 30ms.
That's more than 13x slower.
What gives?
Edit
Raw result from Softwing Profiler:
flJson 10.9583865754112
flSendRequest 392.282022557137
So after a lengthy-ish discussion with the #J-Tolley it looks as though the issue is with the Softwing.Profiler documentation which states;
all results are given in milliseconds
even though earlier in the page it states;
has a ten milliseconds resolution
Have not used the Softwing.Profiler component alone before and would recommend anyone using in a Classic ASP environment to implement it using the SlTiming class library provided by 4GuysFromRolla.
In that article it even warns anyone using the Softwing.Profiler ProfileStop() method to;
Be aware that Softwing.Profiler's ProfileStop method returns a value in ticks (tenths of milliseconds).
My protocol is based on HTTP, I need a dissector to analyze the HTTP payload. How to get the http payload in the dissector function?
The chained dissector looks like:
local original_http_dissector = DissectorTable.get("tcp.port"):get_dissector(80)
local function my_dissector(buf, pkt, root)
-- 'buf' here contains all tcp data,
-- including the http header
-- How to get the http payload only(skip http header)?
local b = buf
end
function p_MM.dissector(buf, pkt, root)
if original_http_dissector:call(buf, pkt, root) then
my_dissector(buf, pkt, root)
end
end
I've struggled a bit with trying to do similar. The below (based on http_extra at https://wiki.wireshark.org/Lua/Dissector) puts the http content in a new data tab and then does some very basic processing (xor with 0xA5, which turns out to be a bit of a faff) and displays that in a second tab.
do
local http_proto = Proto("http_extra", "Further process HTTP traffic");
local f_http_data = Field.new("http.file_data")
local original_http_dissector
-- simple function to XOR data against 0xA5 to show some processing
-- it turns out it's actually quite hard to reconstruct a tvb for display
-- as you need it in hex string format
function xorf(data)
data = data:raw()
local d = {}
for i = 1, data:len() do
local x = bit32.bxor(data:byte(i), 0xA5)
local c = string.format("%02x", x)
table.insert(d, c)
end
return table.concat(d, "")
end
function http_proto.dissector(tvbuffer, pinfo, treeitem)
-- we've replaced the original http dissector in the
-- dissector table, but we still want the original to run,
-- especially because we need to read its data
original_http_dissector:call(tvbuffer, pinfo, treeitem)
-- validate packet length is adequate, otherwise quit
if tvbuffer:len() == 0 then return end
local a=f_http_data()
if a then
-- get the (whole) subset as a tvbrange
local tvbrange = a.range()
-- get a ByteArray composed of the bytes in the TvbRange
local data = tvbrange:bytes()
-- create a new tab
local tvc = ByteArray.tvb(data, "http.file_data")
-- process the http.file_data to change it
local tvy = ByteArray.tvb(ByteArray.new(xorf(data)), "xor'ed")
end
end
local tcp_dissector_table = DissectorTable.get("tcp.port")
-- save the original dissector so we can still get to it
original_http_dissector = tcp_dissector_table:get_dissector(443)
-- and take its place in the dissector table
tcp_dissector_table:add(443, http_proto)
end
I'm trying to send something via socket so I made a simple protocol which is :
[Message length (uint32)][Packet id(uint32)][Message]
So, how can I send a message with this protocol ? I tried this :
message = 'hi'
parent_socket.send([message.length, 2].pack('LL') + message, 0)
and it doesn't seems to work. What I get is just the length and the packet id.
How can I figure this out?
(Updated)
The code I'm using to receive the message is :
if child_socket.ready?
header = child_socket.read(8).unpack('LL')
length = header[0]
packet = header[1]
case packet
when 1
stdin.write(child_socket.read(length))
when 2
puts child_socket.read(length)
#send console
else
Console.show "Unknown packet : #{packet}"
end
The output is 10. Seems normal (4 + 4 + 2)
in that case, length is 2 and packet is 2. So it switch to 'when 2' but it output nothing.
The problem is you are only reading the first 8 byes.
header = child_socket.read(8).unpack('LL')
According to the docs for IO#read, if you don't pass a length, it will read to EOF, which should get everything. So just remove that length parameter:
header = child_socket.read.unpack('LL')
Maybe I've gotten my sockets programming way mixed up, but shouldn't something like this work?
srv = TCPServer.open(3333)
client = srv.accept
data = ""
while (tmp = client.recv(10))
data += tmp
end
I've tried pretty much every other method of "getting" data from the client TCPSocket, but all of them hang and never break out of the loop (getc, gets, read, etc). I feel like I'm forgetting something. What am I missing?
In order for the server to be well written you will need to either:
Know in advance the amount of data that will be communicated: In this case you can use the method read(size) instead of recv(size). read blocks until the total amount of bytes is received.
Have a termination sequence: In this case you keep a loop on recv until you receive a sequence of bytes indicating the communication is over.
Have the client closing the socket after finishing the communication: In this case read will return with partial data or with 0 and recv will return with 0 size data data.empty?==true.
Defining a communication timeout: You can use the function select in order to get a timeout when no communication was done after a certain period of time. In which case you will close the socket and assume every data was communicated.
Hope this helps.
Hmm, I keep doing this on stack overflow [answering my own questions]. Maybe it will help somebody else. I found an easier way to go about doing what I was trying to do:
srv = TCPServer.open(3333)
client = srv.accept
data = ""
recv_length = 56
while (tmp = client.recv(recv_length))
data += tmp
break if tmp.length < recv_length
end
There is nothing that can be written to the socket so that client.recv(10) returns nil or false.
Try:
srv = TCPServer.open(3333)
client = srv.accept
data = ""
while (tmp = client.recv(10) and tmp != 'QUIT')
data += tmp
end