I'm used to crafting ProducerRecord instances and schlepping them off to my cluster using
// build up properties
// build KakfaProducer with properties
ProducerRecord<String,String> reccord = new ProducerRecord<String,String>("myTopic","woohoo");
producer.send( reccord);
this all depends on having the org.apache.kafka.* packages available.
A different 3rd-party application that we can write custom code for in Java8 and Groovy does NOT have org.apache.kafka.* packages available so must craft messages by-hand.
Is there a way(s) to capture what gets sent over the wire for analysis? Perhaps a ByteArrayOutputStream then loading what gets written to the buffer into a bytearray for viewing. Perhaps javax.interceptors.* could be useful.
TIA,
Still-learning Steve
<!-- update 2022-06-26T00:10:10.000Z -->
This is all running on a Windoze 10 box to which I have Admin access but cannot add any software to i.e. wireshark. So I can run pktmon and set a filter for port 9002 which the kafka cluster is running on. I also have a PowerShell panel open running kafka-console-producer where I can manually send messages, and a corresponding PowerShell panel open running kafka-console-consumer where I can see the messages appearing. Between all this plus the Java producer and Java consumer there simply must be a way to capture the serialized data on the wire.
<!-- update 2022-06-27T05:20:10.000Z -->
Two of the proffered solutions are spot-on -- except both rely on wireshark, which I mentioned I cannot load. Current approach is to create a ServerSocket aimed at port 9092 and an InputStream and try to bind a listener on the port and stream the bytes to an array for inspection. Fingers crossed.
<!-- update 2022-06-30T00:40:10.000Z -->
A variation of this kinda worked - I shut down my instance of Kafka and started a simple server on port 9092 that simply printed the stream to console. Sending a message from kafka-console-producer caused this to be printed
0x0 0x0 0x0 0x1a 0x0 0x12 0x0 0x2 0x0 0x0 0x0 0x0 0x0 0x10 0x50 0x61 0x73 0x73 0x70 0x6f 0x72 0x74 0x50 0x72 0x6f 0x64 0x75 0x63 0x65 0x72
which works out to
64-bit integer == 26 == total number of bytes following
32-bit integer ???
32-bit integer ???
64-bit integer == 16 == number of bytes in following string
PassportProducer
but no topic, key, or value turns up. Which leads me to think there's some back-and-forth between source and sink. I've tried attaching my server on the same port as the kafka service but get a "port already in use" errmsg. Can't two apps share a port if one only listens?
Refer How do I view the full TCP packet that Apache Kafka produces?
You call also try using any of the following.
https://aristanetworks.force.com/AristaCommunity/s/article/how-to-capture-timestamped-packets-to-a-kafka-broker
https://wiki.wireshark.org/Kafka
Related
I have one energy meter and i am trying to retrieve voltage, freq values from meter on raspberry pi uart over RS485
My connections for raspberry pi and rs485 are as follows
Rs485 DI - Tx of raspberry pi
Rs485 R0 - Rx of raspberry pi
Rs485 DE/RE -Pin 7 of raspberry pi
my code is as follows:
import serial
import RPi.GPIO as GPIO
from pymodbus.client.sync import ModbusSerialClient as ModbusClient
from pymodbus.register_read_message import ReadInputRegistersResponse
from pymodbus.register_read_message import ReadInputRegistersRequest
import logging
logging.basicConfig() log = logging.getLogger()
log.setLevel(logging.DEBUG)
GPIO.setmode(GPIO.BOARD) GPIO.setup(7,GPIO.OUT,initial=GPIO.LOW)
client= ModbusClient(method = 'rtu', port='/dev/ttyS0',stopbits = 1,timeout =0.3, bytesize = 8, parity = 'N', baudrate = '9600')
connection = client.connect()
print "Connection" print connection
while 1:
volt=0
freq=0
if connection:
try:
voltage1= client.read_input_registers(0x000,4,unit=0x03)
print voltage1
except:
print "Error: No message Received"
client.close()
And i am receiving the output as follows
DEBUG:pymodbus.transaction:Current transaction state - TRANSACTION_COMPLETE
DEBUG:pymodbus.transaction:Running transaction 4
DEBUG:pymodbus.transaction:SEND: 0x3 0x4 0x0 0x0 0x0 0x4 0xf0 0x2b
DEBUG:pymodbus.framer.rtu_framer:Changing state to IDLE - Last Frame End - None, Current Time stamp - 1557304284.88
DEBUG:pymodbus.client.sync:New Transaction state 'SENDING'
DEBUG:pymodbus.transaction:Changing transaction state from 'SENDING' to 'WAITING FOR REPLY'
DEBUG:pymodbus.transaction:Changing transaction state from 'WAITING FOR REPLY' to 'PROCESSING REPLY'
DEBUG:pymodbus.transaction:RECV: 0x7b 0x20 0x31 0x20 0x31 0x20 0x32 0x36 0x2e 0x33 0x35 0x20 0x31
DEBUG:pymodbus.transaction:Changing transaction state from 'PROCESSING REPLY' to 'TRANSACTION_COMPLETE'
Modbus Error: [Input/Output] No Response received from the remote unit/Unable to decode response
DEBUG:pymodbus.transaction:Current transaction state - TRANSACTION_COMPLETE
DEBUG:pymodbus.transaction:Running transaction 5
DEBUG:pymodbus.transaction:Clearing current Frame : - 0x7b 0x20 0x31 0x20 0x31 0x20 0x32 0x36 0x2e 0x33 0x35 0x20 0x31
DEBUG:pymodbus.framer.rtu_framer:Resetting frame - Current Frame in buffer - 0x7b 0x20 0x31 0x20 0x31 0x20 0x32 0x36 0x2e 0x33 0x35 0x20 0x31
DEBUG:pymodbus.transaction:SEND: 0x3 0x4 0x0 0x0 0x0 0x4 0xf0 0x2b
DEBUG:pymodbus.framer.rtu_framer:Changing state to IDLE - Last Frame End - None, Current Time stamp - 1557304284.98
DEBUG:pymodbus.client.sync:New Transaction state 'SENDING'
WARNING:pymodbus.client.sync:Cleanup recv buffer before send: 0x37 0x2e 0x35 0x35 0x20 0x33
DEBUG:pymodbus.transaction:Changing transaction state from 'SENDING' to 'WAITING FOR REPLY'
DEBUG:pymodbus.transaction:Incomplete message received, Expected 13 bytes Recieved 7 bytes !!!!
DEBUG:pymodbus.transaction:Changing transaction state from 'WAITING FOR REPLY' to 'PROCESSING REPLY'
DEBUG:pymodbus.transaction:RECV: 0x2e 0x30 0x36 0x20 0x7d 0xd 0xa
DEBUG:pymodbus.transaction:Changing transaction state from 'PROCESSING REPLY' to 'TRANSACTION_COMPLETE'
Modbus Error: [Input/Output] No Response received from the remote unit/Unable to decode response
If I'm not getting it wrong, you're defining your GPIO pin correctly but you're never toggling it high and low. To be able to drive the DE/~RE signal on your RS485 chip you should take the GPIO high before you write on the bus and low right after to be able to read the answer from your meter.
Unfortunately, I'm afraid what you're trying to do is not possible with pyModbus out of the box. You can take a look at this link:
https://github.com/riptideio/pymodbus/issues/33
You might be able to tweak pyModbus and use the RTS alternative functions on your Pi (see here: https://github.com/mholling/rpirtscts), but I don't think this path will get you very far reliability wise.
As I wrote here: RS485: Inappropriate ioctl for device, you might be better off going for a hardware solution. If you can't get new hardware you could always try the 555 timer solution, at least as a temporary fix.
Good luck, and be sure to post your progress or any further ideas.
EDIT: Solution using libmodbus instead
The suggestion to use libmodbus was very successful. Follow these steps if you want to try it (tested with Raspberry Pi 3B):
1) Clone libmodbus fork with GPIO support to your Pi:
git clone https://github.com/dhruvvyas90/libmodbus
2) Configure, compile and install libmodbus library (same commands as for the main repo):
./autogen.sh && ./configure --prefix=/usr && make && sudo make install
3) Go to rpi-test folder and compile the example:
gcc -o test -I/usr/include/modbus test.c -lmodbus
4) Run test, you will need to change permissions or sudo it: sudo ./test
What you get is actually much better than I expected and probably good enough for most Modbus hardware:
In blue, you see the TX from the Pi's UART (pin number 8 on the connector) and in yellow you get your DE/~RE (pin number 11, GPIO17) that you should connect to your RS485 chip. As you can see there is a delay of 0.6 ms from the end of the Modbus data frame until the bus is free for the slave to answer. At the speed I was using (9600 bps) the minimum delay you need to comply with the Modbus specification is around 3 ms (3.5 chars), so it should be fine for most situations.
The only thing pending would be to add all these GPIO functions to the pylibmodbus wrapper but that should be quite easy. I'm planning to use this library from Python real soon in the field with my Pocket Chip computer to work as a Modbus handheld tester so if you or anybody else manage to find the time I'd be more than glad to test it.
As soon as I have more time I'll try libmodbus together with my FTDI serial port and take a couple of scope captures to compare hardware vs. software signaling.
I forgot to mention that the only changes I made to the test.c were:
Line 13: #define UART_PORT "/dev/serial0"
Line 14: #define BAUD_RATE 9600
The first one just to the name of the embedded serial port on my Pi and the second one is the speed I always use for testing purposes.
EDIT: Software vs. Hardware signalling
As promised I have tested the solution proposed with libmodbus but instead of the embedded UART on the Raspberry Pi I made it work together with my FTDI USB adaptor to compare the time it takes to release the bus.
You can see how the TXEN (magenta trace) manages to go low about 250 microseconds after the stop bit while the GPIO on the Pi (in blue) takes more or less the same time as in the capture above (500-600 microseconds).
So pending more extensive testing, my conclusion is libmodbus does a great job for UARTs where you don't have a TX enable signal available. I think it should be possible to have reliable Modbus communication for most scenarios.
Solution with pylibmodbus
I wrote the missing functions for the pylibmodbus library.
See here: https://github.com/marcosgatcomputer/pylibmodbus
Once you have everything installed (the libmodbus branch with support for GPIO and pylibmodbus from the link above) you can try the test file:
from pylibmodbus import ModbusRtu
#Define Modbus RTU client (Python 2.x)
client=ModbusRtu(device="/dev/serial0", baud=19200, parity="N", data_bit=8, stop_bit=1)
# For Python 3.x you have to explicitly indicate ASCII enconding
#client=ModbusRtu(device="/dev/serial0".encode("ascii"), baud=19200, parity="N".encode("ascii"), data_bit=8, stop_bit=1)
#Read and set timeout
timeout_sec = client.get_response_timeout()
client.set_response_timeout(timeout_sec+1)
#Connect
client.connect()
SERVER_ID=0
BCM_PIN_DE=17
BCM_PIN_RE=9
#Set Slave ID number
client.set_slave(SERVER_ID)
#Enable RPi GPIO Functions
client.enable_rpi(1)
#Define pin numbers to be used as Read Enable (RE) and Drive Enable (DE)
client.configure_rpi_bcm_pins(BCM_PIN_DE,BCM_PIN_RE)
#Export pin direction (set as outputs)
client.rpi_pin_export_direction()
#Write Modbus registers, 10 starting from 0
client.write_registers(0, [0]*10)
#Read 10 input registers starting from number 0
result=(client.read_registers(0, 10))
#Show register values
print result
#Release pins and close connection
client.rpi_pin_unexport_direction()
client.close()
This code works with Rpi 3B. For Pocket Chip I had to modify libmodbus to account for the GPIO pin numbers (the original code was not able to write on the /sys/class/gpio/export file to create the gpio1015 device). This issue might happen for hardware with 4 digit numbers (if you see folders like gpiochipxxxx on your /sys/class/gpio/)
I trying to read a DTC from a CAPL script. I am using "(0x19) ReadDtcInformation - Report DTC snapshot record by DTC number" protocol service. My DTC number is 0x062003. I am able to set DTC number correctly. But, I am not able to set DTC status bits.
1. diagRequest FR_Diagnostic.FaultMemory_ReadEnvironmentData PWM_Status; // 0x19 0x94
2. diagSetParameter(PWM_Status,"DTC",0x062003); // 0x06 0x20 0x03
3. diagSetParameter(PWM_Status,"DtcSnapshotRecordNumber",0xFF);
4. DiagSendRequest(PWM_Status);
Line #3 is not quite setting DtcSnapshotRecordNumber to 0xFF. I see this field as 0x00 in trace.
Question:
1. Am I using correct API in line #3? How do I set DtcSnapshotRecordNumber to read for all the status bits? (there are 8 status bits to monitor, thus it should be set to 0xFF).
Normaly you call service 0x19 (Diagnostic) then call Subfunction and then you provide a bitmask which contains information for what you are searching.
So everything you need is defined in ISO 14229 Road vehicles.
Most services regarding DTCs and OBD stuff are the same across all OEMs. Each OEM normaly has a Diagnostic Specification Supplement which describes OEM specific Diagnostic services and so on.
In CAPL there is normaly an easy way to read out DTCs. You don´t need to do it manually as it is provided by the CDD-File. You can see it in the Object Browser of CAPL Brower on the right side.
So for your case for egg. retrieve number of DTCs which match your Mask:
Subfunction Report DTCByStatusMask 0x02
0x1902
The mask which is your search filter will be build like this:
bit # hex state description
0 0x01 testFailed DTC failed at the time of the request
1 0x02 testFailedThisOperationCycle DTC failed on the current operation cycle
2 0x04 pendingDTC DTC failed on the current or previous operation cycle
3 0x08 confirmedDTC DTC is confirmed at the time of the request
4 0x10 testNotCompletedSinceLastClear DTC test not completed since the
last code clear
5 0x20 testFailedSinceLastClear DTC test failed at least once since last code clear
6 0x40 testNotCompletedThisOperationCycle DTC test not completed this operation cycle
7 0x80 warningIndicatorRequested Server is requesting warningIndicator to be active
So if you just want to read out DTC which are confirmed (stored) than you send:
0x190208
If you get a positive response than the DTCs will be retrieved in Hex Format. You´ll need a HEX->SAE converter which will convert retrieved DTCs to regular know format. Anyway you can Test it through Diagnostic Console.
Answering myself:
Yes, I was using the correct API.
It wasn't getting set properly because the datatype used in the database (.cdd) files was wrong.
I could have converted .cdd file to template and edit the template and work with updated .cdd file, but I chose to work it out using a fresh template copy.
I have configured our custom BSP to have 4 COM ports.
Out of these 3 COM ports work fine.
I have issue with the below COM port:
[HKEY_LOCAL_MACHINE\Drivers\BuiltIn\Serial3]
"SysIntr"=dword:16
"IoBase"=dword:03E8
"IoLen"=dword:8
"DeviceArrayIndex"=dword:2
"Prefix"="COM"
"IClass"="{CC5195AC-BA49-48a0-BE17-DF6D1B0173DD}"
"Dll"="Com16550.Dll"
"Order"=dword:0
"Flags"=dword:10 ; User MOde: DEVFLAGS_LOAD_AS_USERPROC
Any transmission of more than 16 bytes is truncated. On debugging we found that after transmitting first 16 bytes, it waits for IIR(Interrupt Identification Register) "Transmitter Holding Register Empty Interrupt" event to occur.
But this does not occurs.
Any thoughts on how to proceed with this.
I seem to remember that 'modern' UARTS have a 16 byte transmit (and receive) buffer to allow fewer interrupts to the main CPU. This would explain why you can transmit 16 bytes before it stops.
As for the reason your custom board doesn't interrupt, I would assume you haven't wired up the interrupt line from the UART to your main processor.
It was a BIOS issue and it got resolved after updating the BIOS
I have no real experience with hardware programming. I would like to know how to find out which registers, i.e their addresses, are used for an ethernet connection to send and receive information in a processer. In particular, for ATMEL's at91sam9g20 processor. I have searched the documentation and I'm not sure of the following of what I've found:
-Transmit data: signal name ETX0-ETX3. Receive data: signal name ERX0-ERX3.
Also, Offset: 0x18 Receive Buffer Queue Pointer Register and offset: 0x1C Transmit Buffer Queue Pointer Register.
I would appreciate any help as I'm very stuck on this issue.
Thank you
In the procesor documentation you find the following:
chapter 8 - general memory mapping - this gives the offset of the EMAC memory block as 0xFFFC:4000
chapter 36 - description of the Ethernet MAC subsystem
item 36.3.2 - memory interface - tells how the memory buffers for RX and TX is set up
item 36.5 - user interface - table 36-6 gives the names and offsets to all registers used by the subsystem
The signals (=pins) and register offsets you describe are correct.
What is the MozWebSocket firefox binary message client -> server packet structure?
Sending ArrayBuffer (0x01 0x00 0x01). Really sent 0x88 0x82 0xE7 0xEB 0x20 0x30 0xE4 0x03. First 2 bytes doesnt change on different sessions (id size?).
Here is for example chrome binary message packet structure -
0x82 - id
0xYY 1bit - is masked? 7bit - data length
0xZZ oxZZ oxZZ oxZZ - 4 byte mask
data (masked)
Note: Firefox currently only supports sending strings.
Seems like there is no way to send binary data using firefox?
Only Chrome currently supports sending binary object types. Firefox does not yet support sending binary types. However, Mozilla is working on the issue so there should be support in a Firefox release soon.
I believe the packet structure is identical to Chrome's (or any other browser's). Looking at the data framing section of the latest spec suggests that this is a connection close message:
0x88: The leading 8 shows that this is the final fragment of the message. The trailing 8 is the opcode for a connection close frame.
0x82: The msb shows that the message contains a mask. The remaining 7 bits give the length of data sent - 2 bytes in this case.
0xE7 0xEB 0x20 0x30: The mask used to decode the message.
0xE4 0x03: The message sent - (0xE4^0xE7) (0x03 ^ 0xEB) [ == 0xE7 0x0C I think]
Assuming you aren't closing the connection client side, you could check whether your version of Firefox supports sending binary messages.