get http payload when writing wireshark lua chained HTTP dissector - wireshark-dissector

My protocol is based on HTTP, I need a dissector to analyze the HTTP payload. How to get the http payload in the dissector function?
The chained dissector looks like:
local original_http_dissector = DissectorTable.get("tcp.port"):get_dissector(80)
local function my_dissector(buf, pkt, root)
-- 'buf' here contains all tcp data,
-- including the http header
-- How to get the http payload only(skip http header)?
local b = buf
end
function p_MM.dissector(buf, pkt, root)
if original_http_dissector:call(buf, pkt, root) then
my_dissector(buf, pkt, root)
end
end

I've struggled a bit with trying to do similar. The below (based on http_extra at https://wiki.wireshark.org/Lua/Dissector) puts the http content in a new data tab and then does some very basic processing (xor with 0xA5, which turns out to be a bit of a faff) and displays that in a second tab.
do
local http_proto = Proto("http_extra", "Further process HTTP traffic");
local f_http_data = Field.new("http.file_data")
local original_http_dissector
-- simple function to XOR data against 0xA5 to show some processing
-- it turns out it's actually quite hard to reconstruct a tvb for display
-- as you need it in hex string format
function xorf(data)
data = data:raw()
local d = {}
for i = 1, data:len() do
local x = bit32.bxor(data:byte(i), 0xA5)
local c = string.format("%02x", x)
table.insert(d, c)
end
return table.concat(d, "")
end
function http_proto.dissector(tvbuffer, pinfo, treeitem)
-- we've replaced the original http dissector in the
-- dissector table, but we still want the original to run,
-- especially because we need to read its data
original_http_dissector:call(tvbuffer, pinfo, treeitem)
-- validate packet length is adequate, otherwise quit
if tvbuffer:len() == 0 then return end
local a=f_http_data()
if a then
-- get the (whole) subset as a tvbrange
local tvbrange = a.range()
-- get a ByteArray composed of the bytes in the TvbRange
local data = tvbrange:bytes()
-- create a new tab
local tvc = ByteArray.tvb(data, "http.file_data")
-- process the http.file_data to change it
local tvy = ByteArray.tvb(ByteArray.new(xorf(data)), "xor'ed")
end
end
local tcp_dissector_table = DissectorTable.get("tcp.port")
-- save the original dissector so we can still get to it
original_http_dissector = tcp_dissector_table:get_dissector(443)
-- and take its place in the dissector table
tcp_dissector_table:add(443, http_proto)
end

Related

Pushing data to websocket browser client in Lua

I want to use a NodeMCU device (Lua based top level) to act as a websocket server to 1 or more browser clients.
Luckily, there is code to do this here: NodeMCU Websocket Server
(courtesy of #creationix and/or #moononournation)
This works as described and I am able to send a message from the client to the NodeMCU server, which then responds based on the received message. Great.
My questions are:
How can I send messages to the client without it having to be sent as a response to a client request (standalone sending of data)? When I try to call socket.send() socket is not found as a variable, which I understand, but cannot work out how to do it! :(
Why does the decode() function output the extra variable? What is this for? I'm assuming it will be for packet overflow, but I can never seem to make it return anything, regardless of my message length.
In the listen method, why has the author added a queuing system? is this essential or for applications that perhaps may receive multiple simultaneous messages? Ideally, I'd like to remove it.
I have simplified the code as below:
(excluding the decode() and encode() functions - please see the link above for the full script)
net.createServer(net.TCP):listen(80, function(conn)
local buffer = false
local socket = {}
local queue = {}
local waiting = false
local function onSend()
if queue[1] then
local data = table.remove(queue, 1)
return conn:send(data, onSend)
end
waiting = false
end
function socket.send(...)
local data = encode(...)
if not waiting then
waiting = true
conn:send(data, onSend)
else
queue[#queue + 1] = data
end
end
conn:on("receive", function(_, chunk)
if buffer then
buffer = buffer .. chunk
while true do
local extra, payload, opcode = decode(buffer)
if opcode==8 then
print("Websocket client disconnected")
end
--print(type(extra), payload, opcode)
if not extra then return end
buffer = extra
socket.onmessage(payload, opcode)
end
end
local _, e, method = string.find(chunk, "([A-Z]+) /[^\r]* HTTP/%d%.%d\r\n")
local key, name, value
for name, value in string.gmatch(chunk, "([^ ]+): *([^\r]+)\r\n") do
if string.lower(name) == "sec-websocket-key" then
key = value
break
end
end
if method == "GET" and key then
acceptkey=crypto.toBase64(crypto.hash("sha1", key.."258EAFA5-E914-47DA-95CA-C5AB0DC85B11"))
conn:send(
"HTTP/1.1 101 Switching Protocols\r\n"..
"Upgrade: websocket\r\nConnection: Upgrade\r\n"..
"Sec-WebSocket-Accept: "..acceptkey.."\r\n\r\n",
function ()
print("New websocket client connected")
function socket.onmessage(payload,opcode)
socket.send("GOT YOUR DATA", 1)
print("PAYLOAD = "..payload)
--print("OPCODE = "..opcode)
end
end)
buffer = ""
else
conn:send(
"HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: 12\r\n\r\nHello World!",
conn.close)
end
end)
end)
I can only answer 1 question, the others may be better suited for the library author. Besides, SO is a format where you ask 1 question normally.
How can I send messages to the client without it having to be sent as a response to a client request (standalone sending of data)?
You can't. Without the client contacting the server first and establishing a socket connection the server wouldn't know where to send the messages to. Even with SSE (server-sent events) it's the client that first initiates a connection to the server.

ESP8266 NodeMCU out of memory

NodeMCU info
> Lua 5.1.4
> SDK 2.2.1
> Memory Usage :
> Total : 3260490 bytes
> Used : 9287 bytes
> Remain: 3251203 bytes
Error I get when I try to send HTTP response with big json string response (json_response)
PANIC: unprotected error in call to Lua API (file.lua:5: out of memory)
Code:
-- a simple HTTP server
srv = net.createServer(net.TCP)
srv:listen(80, function(conn)
conn:on("receive", function(sck, payload)
sck:send("HTTP/1.0 200 OK\r\nContent-Type: text/html\r\n\r\n"..json_response)
end)
conn:on("sent", function(sck) sck:close() end)
end)
Yes, that won't work if you're trying to send a lot of data. You need to send this piece-by-piece. Our API documentation shows two approaches (you would find further references here on SO) the first being this:
srv = net.createServer(net.TCP)
function receiver(sck, data)
local response = {}
-- if you're sending back HTML over HTTP you'll want something like this instead
-- local response = {"HTTP/1.0 200 OK\r\nServer: NodeMCU on ESP8266\r\nContent-Type: text/html\r\n\r\n"}
response[#response + 1] = "lots of data"
response[#response + 1] = "even more data"
response[#response + 1] = "e.g. content read from a file"
-- sends and removes the first element from the 'response' table
local function send(localSocket)
if #response > 0 then
localSocket:send(table.remove(response, 1))
else
localSocket:close()
response = nil
end
end
-- triggers the send() function again once the first chunk of data was sent
sck:on("sent", send)
send(sck)
end
srv:listen(80, function(conn)
conn:on("receive", receiver)
end)

pyzmq recv_json can't decode message sent by send_json

Here is my code with the extraneous stuff stripped out:
coordinator.py
context = zmq.Context()
socket = context.socket(zmq.ROUTER)
port = socket.bind_to_random_port(ZMQ_ADDRESS)
poller = zmq.Poller()
poller.register(socket, zmq.POLLIN)
while True:
event = poller.poll(1)
if not event:
continue
process_id, val = socket.recv_json()
worker.py
context = zmq.Context()
socket = context.socket(zmq.DEALER)
socket.connect('%s:%s' % (ZMQ_ADDRESS, kwargs['zmq_port']))
socket.send_json(
(os.getpid(), True)
)
what happens when I run it:
process_id, val = socket.recv_json()
File "/Users/anentropic/.virtualenvs/myproj/lib/python2.7/site-packages/zmq/sugar/socket.py", line 380, in recv_json
return jsonapi.loads(msg)
File "/Users/anentropic/.virtualenvs/myproj/lib/python2.7/site-packages/zmq/utils/jsonapi.py", line 71, in loads
return jsonmod.loads(s, **kwargs)
File "/Users/anentropic/.virtualenvs/myproj/lib/python2.7/site-packages/simplejson/__init__.py", line 451, in loads
return _default_decoder.decode(s)
File "/Users/anentropic/.virtualenvs/myproj/lib/python2.7/site-packages/simplejson/decoder.py", line 406, in decode
obj, end = self.raw_decode(s)
File "/Users/anentropic/.virtualenvs/myproj/lib/python2.7/site-packages/simplejson/decoder.py", line 426, in raw_decode
raise JSONDecodeError("No JSON object could be decoded", s, idx)
JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)
and if I dig in with ipdb:
> /Users/anentropic/.virtualenvs/myproj/lib/python2.7/site-packages/zmq/sugar/socket.py(380)recv_json()
379 msg = self.recv(flags)
--> 380 return jsonapi.loads(msg)
381
ipdb> p msg
'\x00\x9f\xd9\x06\xa2'
hmm, that doesn't look like JSON... is this a bug in pyzmq? am I using it wrong?
Hmm, ok, found the answer.
There is an annoying asymmetry in the ØMQ interface, so you have to be aware of the type of socket you are using.
In this case my use of ROUTER/DEALER architecture means that the JSON message sent from the DEALER socket, when I do send_json, gets wrapped in multipart message envelope. The first part is a client id (I guess this is the '\x00\x9f\xd9\x06\xa2' that I got above) and the second part is the JSON string we are interested in.
So in the last line of my coordinator.py I need to do this instead:
id_, msg = socket.recv_multipart()
process_id, val = json.loads(msg)
IMHO this is bad design on the part of ØMQ/pyzmq, the library should abstract this away and have just send and recv methods, that just work.
I got the clue from this question How can I use send_json with pyzmq PUB SUB so it looks like PUB/SUB architecture has the same issue, and no doubt others too.
This is described in the docs but it's not very clear
http://zguide.zeromq.org/page:all#The-Asynchronous-Client-Server-Pattern
Update
In fact, I found in my case I could simplify the code further, by making use of the 'client id' part of the message envelope directly. So the worker just does:
context = zmq.Context()
socket = context.socket(zmq.DEALER)
socket.identity = str(os.getpid()) # or I could omit this and use ØMQ client id
socket.connect('%s:%s' % (ZMQ_ADDRESS, kwargs['zmq_port']))
socket.send_json(True)
It's also worth noting that when you want to send a message the other direction, from the ROUTER, you have to send it as multipart, specifying which client it is destined for, eg:
coordinator.py
context = zmq.Context()
socket = context.socket(zmq.ROUTER)
port = socket.bind_to_random_port(ZMQ_ADDRESS)
poller = zmq.Poller()
poller.register(socket, zmq.POLLIN)
pids = set()
while True:
event = poller.poll(1)
if not event:
continue
process_id, val = socket.recv_json()
pids.add(process_id)
# need some code in here to decide when to stop listening
# and break the loop
for pid in pids:
socket.send_multipart([pid, 'a string message'])
# ^ do your own json encoding if required
I guess there is probably some ØMQ way of doing a broadcast message rather than sending to each client in a loop as I do above. I wish the docs just had a clear description of each available socket type and how to use them.

Is it possible to get the headers of a request using Ruby's HTTPClient gem before the request completes?

I have a requirement to proxy a request in a Rails app. I was hoping I could proxy it with chunking (so, 1 chunk received, one chunk is sent). The app is working fine without chunking (load the request into memory, and transmit).
Here is my code to proxy the chunks through to the end-client:
self.response.headers['Last-Modified'] = Time.now.ctime.to_s
self.response_body = Enumerator.new do |y|
client = HTTPClient.new
http_response = client.get(proxy_url, nil, headers) do |chunk|
y << chunk
end
end
The problem is, I can't inspect "http_response" until all the chunks have been received, thus I can't set the headers based on the headers of the client.
What I'm trying to do is transmit the headers returned from the client before the first chunk is sent. Is this possible?
If not, is this pattern possible in any other Ruby HTTP client gem?
Update
I have a solution for you.
If you call get_async instead, it will retun immediately with an HTTPClient::Connection object that is updated with the header information as soon as it is received. This code sample demonstrates.
The patch to HTTPClient::Connection is almost certainly not necessary for you, but it lets you write things like conn.queue.size? and conn.queue.empty?.
conn.pop blocks until the response (or exception) has been pushed to the queue by the async thread and then returns the normal HTTP::Message object. (Note that, if you are using the monkey patch, you can use conn.queue.empty? to see if pop is going to block.)
resp.content returns an IO object which is a pipe read endpoint, and can be called as soon as pop hs returned. The other end is written by the async thread as the data arrives, and you can read the entire content in one go or in whatever size chunks you like using read.
require 'httpclient'
class HTTPClient::Connection
attr_reader :queue
end
client = HTTPClient.new
conn = client.get_async 'http://en.wikipedia.org/wiki/Ruby_(programming_language)'
resp = conn.pop
resp.header.all.each { |name, val| puts "#{name}=#{val}" }
puts
pipe = resp.content
while chunk = pipe.read(8192)
print chunk
end
You could parse the first chunk you receive to extract the headers, but I suggest you call head first to get the header information. Then do the get as well.
(Updated - the first chunk holds the beginning of the content so this won't work.)

Ruby TCPServer sockets

Maybe I've gotten my sockets programming way mixed up, but shouldn't something like this work?
srv = TCPServer.open(3333)
client = srv.accept
data = ""
while (tmp = client.recv(10))
data += tmp
end
I've tried pretty much every other method of "getting" data from the client TCPSocket, but all of them hang and never break out of the loop (getc, gets, read, etc). I feel like I'm forgetting something. What am I missing?
In order for the server to be well written you will need to either:
Know in advance the amount of data that will be communicated: In this case you can use the method read(size) instead of recv(size). read blocks until the total amount of bytes is received.
Have a termination sequence: In this case you keep a loop on recv until you receive a sequence of bytes indicating the communication is over.
Have the client closing the socket after finishing the communication: In this case read will return with partial data or with 0 and recv will return with 0 size data data.empty?==true.
Defining a communication timeout: You can use the function select in order to get a timeout when no communication was done after a certain period of time. In which case you will close the socket and assume every data was communicated.
Hope this helps.
Hmm, I keep doing this on stack overflow [answering my own questions]. Maybe it will help somebody else. I found an easier way to go about doing what I was trying to do:
srv = TCPServer.open(3333)
client = srv.accept
data = ""
recv_length = 56
while (tmp = client.recv(recv_length))
data += tmp
break if tmp.length < recv_length
end
There is nothing that can be written to the socket so that client.recv(10) returns nil or false.
Try:
srv = TCPServer.open(3333)
client = srv.accept
data = ""
while (tmp = client.recv(10) and tmp != 'QUIT')
data += tmp
end

Resources