I keep getting this error EXCEPTION: SocketException: Operation not permitted (select/poll failed) when I push frames to kinesis, this is followed by
f-stream is reported. Terminating...
DEBUG / KinesisVideo: Exception while sending data.
ERROR / KinesisVideo: 2018-06-09T16:26Z T31: EXCEPTION: RuntimeException: Exception thrown on sending thread: Exception while sending encoded chunk in MKV stream !
DEBUG / KinesisVideo: PutFrame index: 10, pts: 15285616115400000, dts: 15285616115400000, duration: 200000, keyFrame: false, flags: 0
com.amazonaws.kinesisvideo.producer.ProducerException: Failed to put a frame into the stream.
at com.amazonaws.kinesisvideo.producer.jni.NativeKinesisVideoProducerJni.putKinesisVideoFrame(Native Method)
at com.amazonaws.kinesisvideo.producer.jni.NativeKinesisVideoProducerJni.putFrame(NativeKinesisVideoProducerJni.java:440)
at com.amazonaws.kinesisvideo.producer.jni.NativeKinesisVideoProducerStream.putFrame(NativeKinesisVideoProducerStream.java:259)
at com.amazonaws.kinesisvideo.mediasource.ProducerStreamSink.onFrame(ProducerStreamSink.java:35)
at com.amazonaws.kinesis.custom.S3FileMediaSource.putFrame(S3FileMediaSource.java:114)
at com.amazonaws.kinesis.custom.S3FileMediaSource.access$3(S3FileMediaSource.java:112)
at com.amazonaws.kinesis.custom.S3FileMediaSource$1.onFrameDataAvailable(S3FileMediaSource.java:103)
at com.amazonaws.kinesis.custom.S3FrameSource.generateFrameAndNotifyListener(S3FrameSource.java:84)
at com.amazonaws.kinesis.custom.S3FrameSource.access$0(S3FrameSource.java:71)
at com.amazonaws.kinesis.custom.S3FrameSource$1.run(S3FrameSource.java:66)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
DEBUG / KinesisVideo: Received all data, close
DEBUG / KinesisVideo: Closing data stream
DEBUG / KinesisVideo: Stream unblocked notification.
DEBUG / KinesisVideo: Data availability notification. Upload handle: 0, Size: 0, Duration 0
DEBUG / KinesisVideo: Being notified to close stream streamName with uploadHandle 0
INFO / KinesisVideo: End-of-stream is reported. Terminating...
can't figure out why this is happening, any ideas ?
Finally, we used the PutMedia API instead, to insert MKV, but we could find that the above error was due to ending connection with Kinesis for the below 2 reasons
sending wrong fragments (order/time)
ending thread before finishing the Kinesis connection establishment
I would recommend to try your logic on EC2 instance or generally standalone application (JAR) and check the logs
Related
Current setup - Our Springboot application consumes messages from Kafka topic,We are processing one message at a time (we are not using streams).Below are the config properties and version being used.
ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG- 30000
ConsumerConfig.AUTO_OFFSET_RESET_CONFIG-earliest
ContainerProperties.AckMode-RECORD
Spring boot version-2.5.7
Spring-kafka version- 2.7.8
Kafks-clients version-2.8.1
number of partitions- 6
consumer group- 1
consumers- 2
Issue - When springboot application stays idle for longer time(idle time varying from 4 hrs to 3 days).We are seeing org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Exception error message - org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching kafka-2.broker.emh-dev.service.dev found.
2022-04-07 06:58:42.437 ERROR 24180 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer : Authentication/Authorization Exception, retrying in 10000 ms
After service recover we are seeing message duplication with same partition and offsets which is inconsistent.
Below are the exception:
Consumer clientId=XXXXXX, groupId=XXXXXX] Offset commit failed on partition XXXXXX at offset 354: The coordinator is not aware of this member
Seek to current after exception; nested exception is org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records
We are getting the Caused by: java.io.InterruptedIOException: timeout exception in the logs from the server. However, the server is not giving the response code to us.
I am looking for the standard practice for the timeout monitoring to be followed in Splunk or Appdynamics to plot the graph for number of timeouts being received per second.
Shall we add the error code like 408 in the exception at client side or we should plot the graph for on basis of text "timeout" count with over the time.
Exception Logs
java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:150)
Caused by: java.io.InterruptedIOException: timeout
at okhttp3.internal.connection.Transmitter.timeoutExit(Transmitter.kt:104)
at okhttp3.internal.connection.Transmitter.maybeReleaseConnection(Transmitter.kt:293)
at okhttp3.internal.connection.Transmitter.noMoreExchanges(Transmitter.kt:257)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.kt:192)
at okhttp3.RealCall.execute(RealCall.kt:66)
For AppDynamics the ideal solution would be for a "bad error code" to be returned from the server - this would cause an error to be detected (and mark any associated Business Transaction as being in error) - see https://docs.appdynamics.com/22.2/en/application-monitoring/troubleshooting-applications/errors-and-exceptions#ErrorsandExceptions-BusinessTransactionError
Else you can use Custom Error Configuration to set a logger which signals errors - see https://docs.appdynamics.com/22.2/en/application-monitoring/configure-instrumentation/error-detection#ErrorDetection-ErrorDetectionConfiguration
Else you can capture values using a Data Collector and then use these in Analytics to break out errors - see https://docs.appdynamics.com/22.2/en/application-monitoring/configure-instrumentation/data-collectors + https://docs.appdynamics.com/22.2/en/analytics/configure-analytics/collect-transaction-analytics-data
I have a requirement to delay events by 45 minutes and then invoke a method.
The events keep on coming and piling up. How to approach this using spring integration delayer or spring scheduler?
I went ahead and used JdbcMessageStore with Oracle DB. The messages are stored in INT_MESSAGE table. Somehow during retrieval we observe this error
--- [ 26] o.s.j.c.JdbcTemplate : Executing prepared SQL statement [SELECT MESSAGE_ID, CREATED_DATE, MESSAGE_BYTES from INT_MESSAGE where MESSAGE_ID=? and REGION=?]
DEBUG whdq7355
--- [ 26] o.s.j.d.DataSourceUtils : Fetching JDBC Connection from DataSource
DEBUG whdq7355
--- [ 26] o.s.j.s.l.DefaultLobHandler : Returning BLOB as bytes
DEBUG whdq7355
--- [ 26] o.s.j.d.DataSourceUtils : Returning JDBC Connection to DataSource
DEBUG whdq7355
--- [ 26] o.s.i.c.PublishSubscribeChannel : preSend on channel 'errorChannel', message: ErrorMessage [payload=org.springframework.core.serializer.support.SerializationFailedException: Failed to deserialize payload. Is the byte array a result of corresponding serialization for DefaultDeserializer?; nested exception is java.io.StreamCorruptedException: invalid stream header: 00540001, headers={id=a39571b4-747b-87e7-f10f-0fa360904a15, timestamp=1566325212736}]
There is a delayer component in Spring Integration. It can be configured for that amount of time when you convert it into milliseconds. See Docs for more info: https://docs.spring.io/spring-integration/docs/5.1.7.RELEASE/reference/html/#delayer. Also for such a long delay you definitely need to consider to use external message store on delayer to avoid messages loss and memory leaks.
I am running 3 instances of a service that I wrote using:
Scala 2.11.12
kafkaStreams 1.1.0
kafkaStreamsScala 0.2.1 (by lightbend)
The service uses Kafka streams with the following topology (high level):
InputTopic
Parse to known Type
Clear messages that the parsing failed on
split every single message to 6 new messages
on each message run: map.groupByKey.reduce(with local store).toStream.to
Everything works as expected but i can't get rid of a WARN message that keeps showing:
15:46:00.065 [kafka-producer-network-thread | my_service_name-1ca232ff-5a9c-407c-a3a0-9f198c6d1fa4-StreamThread-1-0_0-producer] [WARN ] [o.a.k.c.p.i.Sender] - [Producer clientId=my_service_name-1ca232ff-5a9c-407c-a3a0-9f198c6d1fa4-StreamThread-1-0_0-producer, transactionalId=my_service_name-0_0] Got error produce response with correlation id 28 on topic-partition my_service_name-state_store_1-repartition-1, retrying (2 attempts left). Error: UNKNOWN_PRODUCER_ID
As you can see, I get those errors from the INTERNAL topics that Kafka stream manage. Seems like some kind of retention period on the producer metadata in the internal topics / some kind of a producer id reset.
Couldn't find anything regarding this issue, only a description of the error itself from here:
ERROR CODE RETRIABLE DESCRIPTION
UNKNOWN_PRODUCER_ID 59 False This exception is raised by the broker if it could not locate the producer metadata associated with the producerId in question. This could happen if, for instance, the producer's records were deleted because their retention time had elapsed. Once the last records of the producer id are removed, the producer's metadata is removed from the broker, and future appends by the producer will return this exception.
Hope you can help,
Thanks
Edit:
It seems that the WARN message does not pop up on version 1.0.1 of kafka streams.
Spring boot properties for kafka producer:
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.client-id=bam
#spring.kafka.producer.acks= # Number of acknowledgments the producer requires the leader to have received before considering a request complete.
spring.kafka.producer.batch-size=0
spring.kafka.producer.bootstrap-servers=localhost:9092
#spring.kafka.producer.buffer-memory= # Total bytes of memory the producer can use to buffer records waiting to be sent to the server.
spring.kafka.producer.client-id=bam-producer
spring.kafka.consumer.auto-offset-reset=earliest
#spring.kafka.producer.compression-type= # Compression type for all data generated by the producer.
spring.kafka.producer.key-serializer= org.apache.kafka.common.serialization.StringSerializer
#spring.kafka.producer.retries= # When greater than zero, enables retrying of failed sends.
spring.kafka.producer.value-serializer= org.apache.kafka.common.serialization.StringSerializer
#spring.kafka.properties.*= # Additional properties used to configure the client.
I am getting below exception when i am trying to send message to a kafka topic :
Caused by: org.springframework.kafka.core.KafkaProducerException: Failed to send; nested exception is org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for bam-0 due to 30004 ms has passed since last append
at org.springframework.kafka.core.KafkaTemplate$1.onCompletion(KafkaTemplate.java:255)
at org.apache.kafka.clients.producer.internals.RecordBatch.done(RecordBatch.java:109)
at org.apache.kafka.clients.producer.internals.RecordBatch.maybeExpire(RecordBatch.java:160)
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortExpiredBatches(RecordAccumulator.java:245)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:212)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:135)
... 1 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for bam-0 due to 30004 ms has passed since last append
I am not able to figure our why i am getting this exception. Can some one please help ?
The producer is timing out trying to send messages. I notice you are using localhost in your bootstrap. Make sure a broker is available locally and listening on port 9092.
Issue resolved by setting advertised.listeners in server.properties to PLAINTEXT://<ExternalIP>:9092.
Note: Kafka is deployed over aws.