Problem when filling slots with custom form (Failed to extract slot) - rasa-nlu

I'm implementing an action form to collect all the required data for some processing but I got this error:
2020-04-09 09:19:13 ERROR rasa_sdk.endpoint - Failed to extract slot type with action places_form
2020-04-09 09:19:13 ERROR rasa.core.actions.action - Failed to extract slot type with action places_form
This custom form must collect two required slots: city and type. The first one is collected from an entity of an intent. For example: Give me all places for [Rennes](city). This works fine.
But the previous intent doesn't contain the type hint and the action form and I want an additional question to be triggered. For example: Which type of places?.
Here is the code of this action form:
class CustomFormAction(FormAction):
def name(self):
return ""
def request_next_slot(
self,
dispatcher: CollectingDispatcher,
tracker: Tracker,
domain: Dict[Text, Any],
) -> Optional[List[EventType]]:
"""Request the next slot and utter template if needed,
else return None"""
for slot in self.required_slots(tracker):
if self._should_request_slot(tracker, slot):
# logger.debug(f"Request next slot '{slot}'")
dispatcher.utter_message(
template=f"utter_ask_{self.name()}_{slot}", **tracker.slots
)
return [SlotSet(REQUESTED_SLOT, slot)]
return None
class PlacesForm(CustomFormAction):
def name(self) -> Text:
return "places_form"
#staticmethod
def required_slots(tracker: Tracker) -> List[Text]:
return ["city", "type"]
def submit(
self,
dispatcher: CollectingDispatcher,
tracker: Tracker,
domain: Dict[Text, Any],
) -> List[Dict]:
city = tracker.get_slot("city")
print("city : {}".format(city))
print("domain : {}".format(json.dumps(domain)))
elements = requests.get("http://...").json()
dispatcher.utter_message(template="utter_places")
dispatcher.utter_message(json.dumps(elements))
return []
Here is the way I defined the associated story:
## places
* places
- places_form
- form{"name": "places_form"}
And the places intent:
## intent:places
- Give me places around [Rennes](city)
- What are the places around [Rennes](city)
- I'm looking for places around [Rennes](city)
- Give me all places for [Rennes](city)
I have the FormPolicy in my config.yml file:
policies:
- name: FallbackPolicy
nlu_threshold: 0.75
- name: AugmentedMemoizationPolicy
- name: FormPolicy
- name: MappingPolicy
- name: TEDPolicy
epochs: 20
Here are the logs regarding the conversation:
Your input -> Give me all places for Rennes
2020-04-09 09:19:07 DEBUG rasa.core.tracker_store - Creating a new tracker for id '8fcaa471f975433ca5e4c1af83b8ca38'.
2020-04-09 09:19:07 DEBUG rasa.core.processor - Starting a new session for conversation ID '8fcaa471f975433ca5e4c1af83b8ca38'.
2020-04-09 09:19:07 DEBUG rasa.core.processor - Action 'action_session_start' ended with events '[<rasa.core.events.SessionStarted object at 0x7fb3d9ab0e48>, <rasa.core.events.ActionExecuted object at 0x7fb3d9ab0e10>]'.
2020-04-09 09:19:07 DEBUG rasa.core.processor - Current slot values:
city: None
requested_slot: None
type: None
2020-04-09 09:19:08 DEBUG rasa.nlu.classifiers.diet_classifier - There is no trained model: component is either not trained or didn't receive enough training data.
2020-04-09 09:19:08 DEBUG rasa.nlu.selectors.response_selector - Adding following selector key to message property: default
2020-04-09 09:19:08 DEBUG rasa.core.processor - Received user message 'Give me all places for Rennes' with intent '{'name': 'places', 'confidence': 0.9888553619384766}' and entities '[{'entity': 'city', 'start': 31, 'end': 38, 'extractor': 'DIETClassifier', 'value': 'Rennes'}]'
2020-04-09 09:19:08 DEBUG rasa.core.processor - Current slot values:
city: Rennes
requested_slot: None
type: None
2020-04-09 09:19:08 DEBUG rasa.core.processor - Logged UserUtterance - tracker now has 5 events.
2020-04-09 09:19:08 DEBUG rasa.core.policies.fallback - NLU confidence threshold met, confidence of fallback action set to core threshold (0.3).
2020-04-09 09:19:08 DEBUG rasa.core.policies.memoization - Current tracker state [None, None, None, {}, {'intent_places': 1.0, 'entity_city': 1.0, 'prev_action_listen': 1.0}]
2020-04-09 09:19:08 DEBUG rasa.core.policies.memoization - Launch DeLorean...
2020-04-09 09:19:08 DEBUG rasa.core.policies.memoization - Current tracker state [None, None, None, {}, {'intent_places': 1.0, 'entity_city': 1.0, 'prev_action_listen': 1.0}]
2020-04-09 09:19:08 DEBUG rasa.core.policies.memoization - There is no memorised next action
2020-04-09 09:19:08 DEBUG rasa.core.policies.form_policy - There is no active form
2020-04-09 09:19:08 DEBUG rasa.core.policies.ensemble - Predicted next action using policy_4_TEDPolicy
2020-04-09 09:19:08 DEBUG rasa.core.processor - Predicted next action 'places_form' with confidence 0.92.
2020-04-09 09:19:08 DEBUG rasa.core.actions.action - Calling action endpoint to run action 'places_form'.
2020-04-09 09:19:08 DEBUG rasa.core.processor - Action 'places_form' ended with events '[BotUttered('Quelle est la valeur du type?', {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, {"city": "Rennes"}, 1586416748.2458563), <rasa.core.events.Form object at 0x7fb3d9a6bcc0>, <rasa.core.events.SlotSet object at 0x7fb3d9ab0da0>, <rasa.core.events.SlotSet object at 0x7fb429a027b8>, <rasa.core.events.SlotSet object at 0x7fb3d9dbb400>]'.
2020-04-09 09:19:08 DEBUG rasa.core.processor - Current slot values:
city: Rennes
requested_slot: type
type: None
2020-04-09 09:19:08 DEBUG rasa.core.policies.fallback - NLU confidence threshold met, confidence of fallback action set to core threshold (0.3).
2020-04-09 09:19:08 DEBUG rasa.core.policies.memoization - Current tracker state [None, None, None, {}, {'intent_places': 1.0, 'entity_city': 1.0, 'prev_action_listen': 1.0}]
2020-04-09 09:19:08 DEBUG rasa.core.policies.memoization - Launch DeLorean...
2020-04-09 09:19:08 DEBUG rasa.core.policies.memoization - Current tracker state [None, None, None, None, {}]
2020-04-09 09:19:08 DEBUG rasa.core.policies.memoization - There is a memorised next action 'action_listen'
2020-04-09 09:19:08 DEBUG rasa.core.policies.form_policy - There is an active form 'places_form'
2020-04-09 09:19:08 DEBUG rasa.core.policies.mapping_policy - There is no mapped action for the predicted intent, 'places'.
2020-04-09 09:19:08 DEBUG rasa.core.policies.ensemble - Predicted next action using policy_2_FormPolicy
2020-04-09 09:19:08 DEBUG rasa.core.processor - Predicted next action 'action_listen' with confidence 1.00.
2020-04-09 09:19:08 DEBUG rasa.core.processor - Action 'action_listen' ended with events '[]'.
2020-04-09 09:19:08 DEBUG rasa.core.lock_store - Deleted lock for conversation '8fcaa471f975433ca5e4c1af83b8ca38'.
Quelle est la valeur du type?
Your input -> tt
2020-04-09 09:19:13 DEBUG rasa.core.tracker_store - Recreating tracker for id '8fcaa471f975433ca5e4c1af83b8ca38'
2020-04-09 09:19:13 DEBUG rasa.nlu.classifiers.diet_classifier - There is no trained model: component is either not trained or didn't receive enough training data.
2020-04-09 09:19:13 DEBUG rasa.nlu.selectors.response_selector - Adding following selector key to message property: default
2020-04-09 09:19:13 DEBUG rasa.core.processor - Received user message 'tt' with intent '{'name': 'deny', 'confidence': 0.5609110593795776}' and entities '[]'
2020-04-09 09:19:13 DEBUG rasa.core.processor - Logged UserUtterance - tracker now has 13 events.
2020-04-09 09:19:13 DEBUG rasa.core.policies.fallback - NLU confidence 0.5609110593795776 is lower than NLU threshold 0.75.
2020-04-09 09:19:13 DEBUG rasa.core.policies.memoization - Current tracker state [None, {}, {'intent_places': 1.0, 'entity_city': 1.0, 'prev_action_listen': 1.0}, {'intent_places': 1.0, 'entity_city': 1.0, 'active_form_places_form': 1.0, 'prev_places_form': 1.0}, {'intent_deny': 1.0, 'prev_action_listen': 1.0, 'active_form_places_form': 1.0}]
2020-04-09 09:19:13 DEBUG rasa.core.policies.memoization - Launch DeLorean...
2020-04-09 09:19:13 DEBUG rasa.core.policies.memoization - Current tracker state [None, None, None, {}, {'intent_deny': 1.0, 'prev_action_listen': 1.0}]
2020-04-09 09:19:13 DEBUG rasa.core.policies.memoization - There is no memorised next action
2020-04-09 09:19:13 DEBUG rasa.core.policies.form_policy - There is an active form 'places_form'
2020-04-09 09:19:13 DEBUG rasa.core.policies.ensemble - Predicted next action using policy_2_FormPolicy
2020-04-09 09:19:13 DEBUG rasa.core.processor - Predicted next action 'places_form' with confidence 1.00.
2020-04-09 09:19:13 DEBUG rasa.core.actions.action - Calling action endpoint to run action 'places_form'.
2020-04-09 09:19:13 ERROR rasa_sdk.endpoint - Failed to extract slot type with action places_form
2020-04-09 09:19:13 ERROR rasa.core.actions.action - Failed to extract slot type with action places_form
2020-04-09 09:19:13 DEBUG rasa.core.policies.fallback - NLU confidence 0.5609110593795776 is lower than NLU threshold 0.75.
2020-04-09 09:19:13 DEBUG rasa.core.policies.memoization - Current tracker state [None, {}, {'intent_places': 1.0, 'entity_city': 1.0, 'prev_action_listen': 1.0}, {'intent_places': 1.0, 'entity_city': 1.0, 'active_form_places_form': 1.0, 'prev_places_form': 1.0}, {'intent_deny': 1.0, 'prev_action_listen': 1.0, 'active_form_places_form': 1.0}]
2020-04-09 09:19:13 DEBUG rasa.core.policies.memoization - Launch DeLorean...
2020-04-09 09:19:13 DEBUG rasa.core.policies.memoization - Current tracker state [None, None, None, {}, {'intent_deny': 1.0, 'prev_action_listen': 1.0}]
2020-04-09 09:19:13 DEBUG rasa.core.policies.memoization - There is no memorised next action
2020-04-09 09:19:13 DEBUG rasa.core.policies.form_policy - There is an active form 'places_form'
2020-04-09 09:19:13 DEBUG rasa.core.policies.ensemble - Execution of 'places_form' was rejected. Setting its confidence to 0.0 in all predictions.
2020-04-09 09:19:13 DEBUG rasa.core.policies.ensemble - Predicted next action using policy_0_FallbackPolicy
2020-04-09 09:19:13 DEBUG rasa.core.processor - Predicted next action 'action_default_fallback' with confidence 1.00.
2020-04-09 09:19:13 DEBUG rasa.core.processor - Action 'action_default_fallback' ended with events '[<rasa.core.events.UserUtteranceReverted object at 0x7fb3d99625c0>]'.
2020-04-09 09:19:13 DEBUG rasa.core.processor - Current slot values:
city: Rennes
requested_slot: type
type: None
2020-04-09 09:19:13 DEBUG rasa.core.policies.fallback - NLU confidence threshold met, confidence of fallback action set to core threshold (0.3).
2020-04-09 09:19:13 DEBUG rasa.core.policies.memoization - Current tracker state [None, None, None, {}, {'intent_places': 1.0, 'entity_city': 1.0, 'prev_action_listen': 1.0}]
2020-04-09 09:19:13 DEBUG rasa.core.policies.memoization - Launch DeLorean...
2020-04-09 09:19:13 DEBUG rasa.core.policies.memoization - Current tracker state [None, None, None, None, {}]
2020-04-09 09:19:13 DEBUG rasa.core.policies.memoization - There is a memorised next action 'action_listen'
2020-04-09 09:19:13 DEBUG rasa.core.policies.form_policy - There is an active form 'places_form'
2020-04-09 09:19:13 DEBUG rasa.core.policies.mapping_policy - There is no mapped action for the predicted intent, 'places'.
2020-04-09 09:19:13 DEBUG rasa.core.policies.ensemble - Predicted next action using policy_2_FormPolicy
2020-04-09 09:19:13 DEBUG rasa.core.processor - Predicted next action 'action_listen' with confidence 1.00.
2020-04-09 09:19:13 DEBUG rasa.core.processor - Action 'action_listen' ended with events '[]'.
2020-04-09 09:19:13 DEBUG rasa.core.lock_store - Deleted lock for conversation '8fcaa471f975433ca5e4c1af83b8ca38'.```
Thanks very much for your help!
Thierry

You don't define slot_mappings, so it's going to assume that it should set the slot from an entity of the same name. But it seems you don't have any annotated type entities, so it can't extract it, and can't set the slot. Either define slot_mappings or add examples of the type entity.

Related

Keycloak 400 bad request when [state] is old

I have a Spring Boot application protected by Keycloak (15.0.2).
Let's say that I have a browser with two tabs opened for the same user and that user is authenticated.
Now I logout in tab 1 and I get the Keycloak login form with the following Url:
http://localhost:8180/auth/realms/AUTOTEST_PG/protocol/openid-connect/auth?response_type=code&client_id=mcd-client&state=e01b8a88-6945-4c56-9ee1-46fb156d33be&login=true&scope=openid
Now I logout in tab 2 and I get the Keycloak login form with the following Url:
http://localhost:8180/auth/realms/AUTOTEST_PG/protocol/openid-connect/auth?response_type=code&client_id=mcd-client&state=b9607387-7a05-4226-8313-4c5c80a1b145&login=true&scope=openid
Next, I tried to login in tab 1 and I get a 400 Bad Request response.
Looking at the Spring Boot log I get the following:
14554520 2021-12-14 00:02:06,100 [ajp-nio-0.0.0.0-8009-exec-8] DEBUG sample.controller.filters.MyKeycloakAuthenticationProcessingFilter ?:? - - - - - attemptAuthentication
14554520 2021-12-14 00:02:06,100 [ajp-nio-0.0.0.0-8009-exec-8] DEBUG sample.security.RealmProvider ?:? - - - - - getRealmName - requestUri: https://10.161.54.36/sso/login?redirect_url=/sctools/&state=e01b8a88-6945-4c56-9ee1-46fb156d33be&session_state=4fe8d159-5ada-4522-a422-07c5a0ce9d15&code=c5dda8ef-0fd9-4323-9922-fa5c73b28f89.4fe8d159-5ada-4522-a422-07c5a0ce9d15.442a440e-6df0-40c7-b8cb-3a52123430a0
14554520 2021-12-14 00:02:06,100 [ajp-nio-0.0.0.0-8009-exec-8] WARN org.keycloak.adapters.OAuthRequestAuthenticator ?:? - - - - - state parameter invalid
14554520 2021-12-14 00:02:06,100 [ajp-nio-0.0.0.0-8009-exec-8] WARN org.keycloak.adapters.OAuthRequestAuthenticator ?:? - - - - - cookie: b9607387-7a05-4226-8313-4c5c80a1b145
14554520 2021-12-14 00:02:06,100 [ajp-nio-0.0.0.0-8009-exec-8] WARN org.keycloak.adapters.OAuthRequestAuthenticator ?:? - - - - - queryParam: e01b8a88-6945-4c56-9ee1-46fb156d33be
14554521 2021-12-14 00:02:06,101 [ajp-nio-0.0.0.0-8009-exec-8] DEBUG sample.controller.filters.MyKeycloakAuthenticationProcessingFilter ?:? - - - - - unsuccessfulAuthentication - committed: true
14554521 2021-12-14 00:02:06,101 [ajp-nio-0.0.0.0-8009-exec-8] DEBUG sample.security.MyKeycloakAuthenticationFailureHandler ?:? - - - - - onAuthenticationFailure org.keycloak.adapters.springsecurity.KeycloakAuthenticationException: Invalid authorization header, see WWW-Authenticate header for details
14554521 2021-12-14 00:02:06,101 [ajp-nio-0.0.0.0-8009-exec-8] DEBUG sample.security.MyKeycloakAuthenticationFailureHandler ?:? - - - - - onAuthenticationFailure - response isCommitted - Status: 400
It seems that the cookie sent and the state value doesn't match and that is the reason why I get a bad request.
How can I login from any of the tabs without getting the Bad Request ?
Note: If I go to the tab2 and login, then I get a message telling me that the user is already login.
UPDATE
When doing the same operation on the Keycloak console, if the user logout from one tab the second tab is automatically logout and the Url doesn't get an state variable it gets a challenge variable, something like:
http://10.161.54.36:8180/auth/realms/master/protocol/openid-connect/auth?client_id=security-admin-console&redirect_uri=http%3A%2F%2F10.161.54.36%3A8180%2Fauth%2Fadmin%2Fmaster%2Fconsole%2F%23%2Frealms&state=c3015df8-4c50-4454-be8d-0555f25e3bd0&response_mode=fragment&response_type=code&scope=openid&nonce=acb2a07d-c4a3-4cd3-8a7a-e23580a97d14&code_challenge=2W6-09eeD_WEwtWct3a5MojpIQJMe-9brcOH-7fbT6A&code_challenge_method=S256
How is it done ?

Kafka Consumer stuck in rebalancing state

We are having a Kafka consumer, which all of a sudden(without any activity) went into a rebalancing state and got stuck. This caused the CPU of the k8 pod to shoot and GC time was also nearly 70-80%. The node didn't recover from that state. Upon deleting all the topics, it recovered after almost 4-5 hours.
Kafka version - 2.1.1
No of topics - 520( with 10 partitions each)
Consumer groups - 1
partition assignment strategy - sticky
Attaching some of the info logs here at that time.
2021-10-12 10:41:21
2021-10-12 05:11:21.160 INFO 6 CID: UID: RID: --- [cation_consumer] s.consumer.internals.AbstractCoordinator : [Consumer clientId=consumer-staging_notification_consumer-10, groupId=staging_notification_consumer] Member consumer-staging_notification_consumer-10-8a7eb399-c4e0-4443-b330-bfac42ca89ae sending LeaveGroup request to coordinator kafka5:9092 (id: 2147483642 rack: null) due to consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.
2021-10-12 05:10:27.340 INFO 6 CID:4967 UID: RID:17c72e2d517-4d5e --- [ntainer#9-0-C-1] s.consumer.internals.ConsumerCoordinator : [Consumer clientId=consumer-staging_notification_consumer-9, groupId=staging_notification_consumer] Giving away all assigned partitions as lost since generation has been reset,indicating that consumer is no longer part of the group
2021-10-12 10:40:27
2021-10-12 05:10:27.340 INFO 6 CID:4967 UID: RID:17c72e2d517-4d5e --- [ntainer#9-0-C-1] s.consumer.internals.ConsumerCoordinator : [Consumer clientId=consumer-staging_notification_consumer-9, groupId=staging_notification_consumer] Lost previously assigned partitions staging_notification_topic_app_low_mobi_3119-4, staging_notification_topic_app_low_mobi_2023-4, staging_notification_topic_app_low_mobi_5170-9, staging_notification_topic_app_low_mobi_3540-9, staging_notification_topic_app_low_mobi_5722-9,
2021-10-12 10:40:37
2021-10-12 05:10:37.247 INFO 6 CID: UID: RID: --- [cation_consumer] s.consumer.internals.AbstractCoordinator : [Consumer clientId=consumer-staging_notification_consumer-12, groupId=staging_notification_consumer] Attempt to heartbeat failed since group is rebalancing
2021-10-12 10:40:37
2021-10-12 05:10:37.183 INFO 6 CID: UID: RID: --- [cation_consumer] s.consumer.internals.AbstractCoordinator : [Consumer clientId=consumer-staging_notification_consumer-10, groupId=staging_notification_consumer] Attempt to heartbeat failed since group is rebalancing
2021-10-12 10:40:57
2021-10-12 05:10:57.435 INFO 6 CID: UID: RID: --- [cation_consumer] s.consumer.internals.AbstractCoordinator : [Consumer clientId=consumer-staging_notification_consumer-10, groupId=staging_notification_consumer] Attempt to heartbeat failed since group is rebalancing
2021-10-12 10:40:57
2021-10-12 05:10:57.435 INFO 6 CID: UID: RID: --- [cation_consumer] s.consumer.internals.AbstractCoordinator : [Consumer clientId=consumer-staging_notification_consumer-12, groupId=staging_notification_consumer] Attempt to heartbeat failed since group is rebalancing
2021-10-12 10:41:56
2021-10-12 05:11:56.922 INFO 6 CID:4967 UID: RID:17c72e2579c-22e6 --- [ntainer#9-3-C-1] s.consumer.internals.ConsumerCoordinator : [Consumer clientId=consumer-staging_notification_consumer-12, groupId=staging_notification_consumer] Failing OffsetCommit request since the consumer is not part of an active group
2021-10-12 10:41:56
2021-10-12 05:11:56.923 ERROR 6 CID:4967 UID: RID:17c72e2579c-22e6 --- [ntainer#9-3-C-1] essageListenerContainer$ListenerConsumer : Consumer exception
2021-10-12 10:41:56
java.lang.IllegalStateException: This error handler cannot process 'org.apache.kafka.clients.consumer.CommitFailedException's; no record information is available
2021-10-12 10:41:56
at org.springframework.kafka.listener.SeekUtils.seekOrRecover(SeekUtils.java:151)
2021-10-12 10:41:56
at org.springframework.kafka.listener.SeekToCurrentErrorHandler.handle(SeekToCurrentErrorHandler.java:113)
2021-10-12 10:41:56
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.handleConsumerException(KafkaMessageListenerContainer.java:1427)
2021-10-12 10:41:56
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1124)
2021-10-12 10:41:56
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2021-10-12 10:41:56
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2021-10-12 10:41:56
at java.lang.Thread.run(Thread.java:748)
2021-10-12 10:41:56
Caused by: org.apache.kafka.clients.consumer.CommitFailedException: Offset commit cannot be completed since the consumer is not part of an active group for auto partition assignment; it is likely that the consumer was kicked out of the group.
2021-10-12 10:41:56
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.sendOffsetCommitRequest(ConsumerCoordinator.java:1134)
2021-10-12 10:41:56
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:999)
2021-10-12 10:41:56
at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1504)
2021-10-12 10:41:56
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doCommitSync(KafkaMessageListenerContainer.java:2396)
2021-10-12 10:41:56
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.commitSync(KafkaMessageListenerContainer.java:2391)
2021-10-12 10:41:56
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.commitIfNecessary(KafkaMessageListenerContainer.java:2377)
2021-10-12 10:41:56
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.processCommits(KafkaMessageListenerContainer.java:2191)
2021-10-12 10:41:56
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1149)
2021-10-12 10:41:56
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1075)
2021-10-12 10:41:56
... 3 common frames omitted
2021-10-12 10:41:58
2021-10-12 05:11:58.442 INFO 6 CID:4967 UID: RID:17c72e2579c-22e6 --- [ntainer#9-3-C-1] s.consumer.internals.ConsumerCoordinator : [Consumer clientId=consumer-staging_notification_consumer-12, groupId=staging_notification_consumer] Giving away all assigned partitions as lost since generation has been reset,indicating that consumer is no longer part of the group
021-10-12 10:58:36
2021-10-12 05:28:36.039 INFO 6 CID:4412 UID: RID:17c72ebadab-64d7 --- [ntainer#3-1-C-1] s.consumer.internals.AbstractCoordinator : [Consumer clientId=consumer-staging_notification_consumer-17, groupId=staging_notification_consumer] Join group failed with org.apache.kafka.common.errors.RebalanceInProgressException: The group is rebalancing, so a rejoin is needed.
2021-10-12 10:58:36
2021-10-12 05:28:36.044 INFO 6 CID:4412 UID: RID:17c72ebadab-64d7 --- [ntainer#3-1-C-1] s.consumer.internals.AbstractCoordinator : [Consumer clientId=consumer-staging_notification_consumer-17, groupId=staging_notification_consumer] (Re-)joining group
021-10-12 10:58:36
2021-10-12 05:28:36.039 INFO 6 CID:4412 UID: RID:17c72ebadab-64d7 --- [ntainer#3-1-C-1] s.consumer.internals.AbstractCoordinator : [Consumer clientId=consumer-staging_notification_consumer-17, groupId=staging_notification_consumer] Join group failed with org.apache.kafka.common.errors.RebalanceInProgressException: The group is rebalancing, so a rejoin is needed.
2021-10-12 10:58:36
2021-10-12 05:28:36.044 INFO 6 CID:4412 UID: RID:17c72ebadab-64d7 --- [ntainer#3-1-C-1] s.consumer.internals.AbstractCoordinator : [Consumer clientId=consumer-staging_notification_consumer-17, groupId=staging_notification_consumer] (Re-)joining group
2021-10-12 11:05:36
2021-10-12 05:35:36.955 INFO 6 CID:3435 UID: RID:17c72f29913-886c --- [ntainer#7-3-C-1] s.consumer.internals.AbstractCoordinator : [Consumer clientId=consumer-staging_notification_consumer-5, groupId=staging_notification_consumer] Join group failed with org.apache.kafka.common.errors.DisconnectException
This was resolved when I deleted the topics.
Similar behaviour is observed when I increase the number of threads in concurrent listener factory, the consumer fails to come up with similar logs of constant rebalancing
The first log indicates that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms (five minutes default).
When this happens, the consumer client will actively initiate a LeaveGroup request to the coordinator to trigger rebalance.
You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.
Of course, a better way is to check the reason for the slow processing of the program and optimize it.

What is the best practice to create a mock service with Citrus Automation Framework for multiple responses?

I've got Component Integration Test suite setup with Citrus Framework. In order to create a mock service for different components. I've been using simple mapping strategy with the help of XPathPayloadMappingKeyExtractor.
The challenge I've got is to create a multiple responses for different soap requests.
I've been struggling to understand the way it's responding. Any experienced person who can help me out on Citrus and Spring please ?
My Context file:
<citrus-ws:client id="ClientEndpoint"
request-url="http://localhost:8085/"/>
<citrus-ws:server id="BS_Customer_Information"
port="8085"
auto-start="true"/>
My Test Method:
#CitrusTest
public void testingServer() {
soap().client(ClientEndpoint)
.send()
.soapAction("urn:RetrieveAddressBookOP_01")
.payload(new ClassPathResource("Requests/Sample1.xml"));
soap().server(BS_Customer_Information)
.receive()
.soapAction("urn:RetrieveAddressBookOP_01");
soap().server(BS_Customer_Information)
.send()
.payload(new ClassPathResource("AtomicResponses/Response1.xml"));
soap().client(ClientEndpoint)
.receive()
.messageType(MessageType.XML);
}
Issue 1: My Soap Client times out and could not get the response
Question 1: Should the server response be parsed into Soap Env and Body ?
Question 2: Why Would it still call go to Channel Endpoint.inbound when I've already defined my response?
Console Log:
2521 DEBUG gEndpointInterceptor| Received SOAP request:
<?xml version="1.0" encoding="UTF-8"?><SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
<SOAP-ENV:Header/>
<SOAP-ENV:Body>
<RetrieveAddressBookOP_01"/>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
2527 DEBUG r.WebServiceEndpoint| Received SOAP request:
SOAPMESSAGE [payload: <?xml version="1.0" encoding="UTF-8"?><au:RetrieveAddressBookOP_01 xmlns:au="au.com.teysau.dataservice"/>][headers: {citrus_message_id=e1311140-6ee3-415a-815e-0d162cbc03c1, citrus_message_timestamp=1490649843807, citrus_soap_action=urn:RetrieveAddressBookOP_01, citrus_http_request_uri=/, citrus_http_context_path=, citrus_http_query_params=, citrus_http_method=POST}][header-data: [<?xml version="1.0" encoding="UTF-8"?><SOAP-ENV:Header xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"/>]][attachments: []]
2527 DEBUG annelEndpointAdapter| Forwarding request to message channel ...
2527 DEBUG t.TestContextFactory| Created new test context - using global variables: '{}'
2527 DEBUG ltCorrelationManager| Saving correlation key for 'citrus_message_correlator_ChannelEndpointAdapter:producer'
2527 DEBUG context.TestContext| Setting variable: citrus_message_correlator_ChannelEndpointAdapter:producer with value: 'citrus_message_id = 'e1311140-6ee3-415a-815e-0d162cbc03c1''
2527 DEBUG .ChannelSyncProducer| Sending message to channel: 'BS_Customer_Information.inbound'
2527 DEBUG .ChannelSyncProducer| Message to send is:
SOAPMESSAGE [payload: <?xml version="1.0" encoding="UTF-8"?><RetrieveAddressBookOP_01"/>][headers: {citrus_message_id=e1311140-6ee3-415a-815e-0d162cbc03c1, citrus_message_timestamp=1490649843807, citrus_soap_action=urn:RetrieveAddressBookOP_01, citrus_http_request_uri=/, citrus_http_context_path=, citrus_http_query_params=, citrus_http_method=POST}][header-data: [<?xml version="1.0" encoding="UTF-8"?><SOAP-ENV:Header xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"/>]][attachments: []]
2527 INFO .ChannelSyncProducer| Message was sent to channel: 'BS_Customer_Information.inbound'
3539 WARN annelEndpointAdapter| Reply timed out after 1000ms. Did not receive reply message on reply channel
3540 DEBUG annelEndpointAdapter| Did not receive reply message - no response is simulated
3540 DEBUG r.WebServiceEndpoint| No reply message from endpoint adapter 'com.consol.citrus.channel.ChannelEndpointAdapter#6dbd0a41'
3540 WARN r.WebServiceEndpoint| No SOAP response for calling client
3542 DEBUG ageDispatcherServlet| Successfully completed request
3546 DEBUG server.Server| RESPONSE / 202 handled=true
3546 DEBUG ver.HttpChannelState| HttpChannelState#244a4722{s=DISPATCHED i=true a=null} unhandle DISPATCHED
3553 DEBUG erver.HttpConnection| org.eclipse.jetty.server.HttpConnection$SendCallback#27d2cb2e[PROCESSING][i=ResponseInfo{HTTP/1.1 202 null,-1,false},cb=org.eclipse.jetty.server.HttpChannel$CommitCallback#2f1ea019] generate: NEED_HEADER (null,[p=0,l=0,c=0,r=0],true)#START
3556 DEBUG erver.HttpConnection| org.eclipse.jetty.server.HttpConnection$SendCallback#27d2cb2e[PROCESSING][i=ResponseInfo{HTTP/1.1 202 null,-1,false},cb=org.eclipse.jetty.server.HttpChannel$CommitCallback#2f1ea019] generate: FLUSH ([p=0,l=114,c=8192,r=114],[p=0,l=0,c=0,r=0],true)#COMPLETING
3556 DEBUG io.WriteFlusher| write: WriteFlusher#76f08437{IDLE} [HeapByteBuffer#2c34f7c0[p=0,l=114,c=8192,r=114]={<<<HTTP/1.1 202 Acce....v20160210)\r\n\r\n>>>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}]
3557 DEBUG io.WriteFlusher| update WriteFlusher#76f08437{WRITING}:IDLE-->WRITING
3562 INFO ent.WebServiceClient| SOAP message was sent to endpoint: 'http://localhost:8085/'
3562 INFO ent.WebServiceClient| Received no SOAP response from endpoint: 'http://localhost:8085/'
3562 INFO citrus.Citrus|
3562 DEBUG citrus.Citrus| TEST STEP 1/4 SUCCESS
3562 INFO citrus.Citrus|
3562 DEBUG citrus.Citrus| TEST STEP 2/4: receive
3563 DEBUG io.ChannelEndPoint| flushed 114 SelectChannelEndPoint#1c2a40b4{/127.0.0.1:54924<->8085,Open,in,out,-,W,1092/30000,HttpConnection}{io=0,kio=0,kro=1}
3566 DEBUG nnel.ChannelConsumer| Receiving message from: BS_Customer_Information.inbound
3567 DEBUG nnel.ChannelConsumer| Received message from: BS_Customer_Information.inbound
3567 DEBUG ltCorrelationManager| Saving correlation key for 'citrus_message_correlator_BS_Customer_Information:consumer'
3567 DEBUG context.TestContext| Setting variable: citrus_message_correlator_BS_Customer_Information:consumer with value: 'citrus_message_id = 'e1311140-6ee3-415a-815e-0d162cbc03c1''
3567 DEBUG ltCorrelationManager| Saving correlated object for 'citrus_message_id = 'e1311140-6ee3-415a-815e-0d162cbc03c1''
3567 DEBUG ageValidatorRegistry| Found 3 message validators for message type: XML
3567 DEBUG mXmlMessageValidator| Start message validation ...
3567 DEBUG mXmlMessageValidator| Start XML message validation
3569 DEBUG mXmlMessageValidator| Starting XML schema validation ...
3569 WARN mXmlMessageValidator| Neither schema instance nor schema repository defined - skipping XML schema validation
3569 INFO mXmlMessageValidator| XML message validation successful: All values OK
3569 DEBUG mXmlMessageValidator| Start message header validation ...
3570 DEBUG io.WriteFlusher| update WriteFlusher#76f08437{IDLE}:WRITING-->IDLE
3570 DEBUG erver.HttpConnection| org.eclipse.jetty.server.HttpConnection$SendCallback#27d2cb2e[PROCESSING][i=ResponseInfo{HTTP/1.1 202 null,-1,false},cb=org.eclipse.jetty.server.HttpChannel$CommitCallback#2f1ea019] generate: DONE ([p=114,l=114,c=8192,r=0],[p=0,l=0,c=0,r=0],true)#END
3570 DEBUG mXmlMessageValidator| Validating header element: citrus_soap_action='urn:RetrieveAddressBookOP_01': OK.
3571 INFO mXmlMessageValidator| Message header validation successful: All properties OK
3571 INFO mXmlMessageValidator| Message validation successful: All values OK
3571 INFO citrus.Citrus|
3571 DEBUG citrus.Citrus| TEST STEP 2/4 SUCCESS
3571 INFO citrus.Citrus|
3571 DEBUG citrus.Citrus| TEST STEP 3/4: send
3571 DEBUG ngCorrelationManager| Get correlation key for 'citrus_message_correlator_BS_Customer_Information:consumer'
3571 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = 'e1311140-6ee3-415a-815e-0d162cbc03c1''
3571 DEBUG .ChannelSyncConsumer| Sending message to reply channel: 'org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#66f66866'
3571 DEBUG .ChannelSyncConsumer| Message to send is:
SOAPMESSAGE [payload: <AddressBook>
<Address>
<Address_Number>14</Address_Number>
<Long_Address>Willis Street</Long_Address>
<Tax_ID>7987398</Tax_ID>
<Alpha_Name>what is alpha</Alpha_Name>
<Code>UN_</Code>
<Mailing_Name>James Bond</Mailing_Name>
<Address_Line1>take the first part of the long address</Address_Line1>
<Address_Line2>You may take the second part not</Address_Line2>
<Postal_Code>8900</Postal_Code>
<City>WLN</City>
<Country>NZ</Country>
<Prefix>233</Prefix>
<Phone_Number>025364</Phone_Number>
<Prefix_2 xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"/>
<Phone_Number2 xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"/>
<userID>s3b9d</userID>
<BatchNumber>jkdhj59</BatchNumber>
<Tansaction_Number>jkd3j29</Tansaction_Number>
<Line_Number>291597</Line_Number>
</Address>
</AddressBook>][headers: {citrus_message_id=fd946a46-f67a-405d-b118-b1b6b3d562c1, citrus_message_timestamp=1490649843443}][attachments: []]
3572 WARN emporaryReplyChannel| Reply message received but the receiving thread has exited due to a timeout:GenericMessage [payload=SOAPMESSAGE [payload: <AddressBook>
<Address>
<Address_Number>14</Address_Number>
<Long_Address>Willis Street</Long_Address>
<Tax_ID>7987398</Tax_ID>
<Alpha_Name>what is alpha</Alpha_Name>
<Code>UN_</Code>
<Mailing_Name>James Bond</Mailing_Name>
<Address_Line1>take the first part of the long address</Address_Line1>
<Address_Line2>You may take the second part not</Address_Line2>
<Postal_Code>8900</Postal_Code>
<City>WLN</City>
<Country>NZ</Country>
<Prefix>233</Prefix>
<Phone_Number>025364</Phone_Number>
<Prefix_2 xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"/>
<Phone_Number2 xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"/>
<userID>s3b9d</userID>
<BatchNumber>jkdhj59</BatchNumber>
<Tansaction_Number>jkd3j29</Tansaction_Number>
<Line_Number>291597</Line_Number>
</Address>
</AddressBook>][headers: {citrus_message_id=fd946a46-f67a-405d-b118-b1b6b3d562c1, citrus_message_timestamp=1490649843443}][attachments: []], headers={id=cfc4eeaa-eac8-8f00-2f45-2e3fcb16fdb6, timestamp=1490649844854}]
3572 INFO .ChannelSyncConsumer| Message was sent to reply channel: 'org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#66f66866'
3572 INFO citrus.Citrus|
3572 DEBUG citrus.Citrus| TEST STEP 3/4 SUCCESS
3572 INFO citrus.Citrus|
3572 DEBUG citrus.Citrus| TEST STEP 4/4: receive
3572 DEBUG ngCorrelationManager| Get correlation key for 'citrus_message_correlator_ClientEndpoint'
3572 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
3572 DEBUG citrus.RetryLogger| Reply message did not arrive yet - retrying in 500ms
3575 DEBUG http.HttpParser| reset HttpParser{s=END,214 of 214}
3576 DEBUG http.HttpParser| END --> START
3576 DEBUG server.HttpChannel| HttpChannelOverHttp#4820529f{r=1,c=false,a=IDLE,uri=} handle exit, result COMPLETE
3576 DEBUG io.ChannelEndPoint| filled 0 SelectChannelEndPoint#1c2a40b4{/127.0.0.1:54924<->8085,Open,in,out,-,-,13/30000,HttpConnection}{io=0,kio=0,kro=1}
3576 DEBUG io.ChannelEndPoint| filled 0 SelectChannelEndPoint#1c2a40b4{/127.0.0.1:54924<->8085,Open,in,out,-,-,13/30000,HttpConnection}{io=0,kio=0,kro=1}
3576 DEBUG http.HttpParser| parseNext s=START HeapByteBuffer#6b681750[p=0,l=0,c=16384,r=0]={<<<>>>POST / HTTP/1.1\r\n...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
3576 DEBUG o.AbstractConnection| fillInterested HttpConnection#2c01adea[FILLING,SelectChannelEndPoint#1c2a40b4{/127.0.0.1:54924<->8085,Open,in,out,-,-,13/30000,HttpConnection}{io=0,kio=0,kro=1}][p=HttpParser{s=START,0 of -1},g=HttpGenerator{s=START},c=HttpChannelOverHttp#4820529f{r=1,c=false,a=IDLE,uri=}]
3576 DEBUG o.AbstractConnection| FILLING-->FILLING_FILL_INTERESTED HttpConnection#2c01adea[FILLING_FILL_INTERESTED,SelectChannelEndPoint#1c2a40b4{/127.0.0.1:54924<->8085,Open,in,out,-,-,13/30000,HttpConnection}{io=0,kio=0,kro=1}][p=HttpParser{s=START,0 of -1},g=HttpGenerator{s=START},c=HttpChannelOverHttp#4820529f{r=1,c=false,a=IDLE,uri=}]
3577 DEBUG o.AbstractConnection| FILLING_FILL_INTERESTED-->FILL_INTERESTED HttpConnection#2c01adea[FILL_INTERESTED,SelectChannelEndPoint#1c2a40b4{/127.0.0.1:54924<->8085,Open,in,out,-,-,13/30000,HttpConnection}{io=0,kio=0,kro=1}][p=HttpParser{s=START,0 of -1},g=HttpGenerator{s=START},c=HttpChannelOverHttp#4820529f{r=1,c=false,a=IDLE,uri=}]
3577 DEBUG electChannelEndPoint| Local interests updating 0 -> 1 for SelectChannelEndPoint#1c2a40b4{/127.0.0.1:54924<->8085,Open,in,out,R,-,0/30000,HttpConnection}{io=1,kio=0,kro=1}
3577 DEBUG io.SelectorManager| Queued change org.eclipse.jetty.io.SelectChannelEndPoint$1#67e66c5c
3578 DEBUG io.SelectorManager| Selector loop woken up from select, 0/1 selected
3578 DEBUG io.SelectorManager| Running change org.eclipse.jetty.io.SelectChannelEndPoint$1#67e66c5c
3578 DEBUG electChannelEndPoint| Key interests updated 0 -> 1 on SelectChannelEndPoint#1c2a40b4{/127.0.0.1:54924<->8085,Open,in,out,R,-,1/30000,HttpConnection}{io=1,kio=1,kro=1}
3578 DEBUG io.SelectorManager| Selector loop waiting on select
4072 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
4072 DEBUG citrus.RetryLogger| Reply message did not arrive yet - retrying in 500ms
4573 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
4573 DEBUG citrus.RetryLogger| Reply message did not arrive yet - retrying in 500ms
5073 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
5073 DEBUG citrus.RetryLogger| Reply message did not arrive yet - retrying in 500ms
5574 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
5574 DEBUG citrus.RetryLogger| Reply message did not arrive yet - retrying in 500ms
6075 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
6075 DEBUG citrus.RetryLogger| Reply message did not arrive yet - retrying in 500ms
6575 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
6575 DEBUG citrus.RetryLogger| Reply message did not arrive yet - retrying in 500ms
7075 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
7075 DEBUG citrus.RetryLogger| Reply message did not arrive yet - retrying in 500ms
7576 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
7576 DEBUG citrus.RetryLogger| Reply message did not arrive yet - retrying in 500ms
8076 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
8076 DEBUG citrus.RetryLogger| Reply message did not arrive yet - retrying in 500ms
8576 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
8580 INFO report.JIRAConsumer| Invoking JIRA API...
works.integration.jira.exceptions.ServiceBindingException: Missing Required Properties - SUMMARY, EXMESSAGE, DETAILEDEXCEPTION, PROJECT
You have to add a fork(true) option to the first `soap().client().send()' action because the Http SOAP protocol is synchronous by nature. The first action in your test waits for a synchronous response and blocks the rest of the test case execution.
Obviously your test needs to receive some other messages before that client response with soap().server(). That is why you need to fork the client send action in the first place so the server actions can perform before the client response has arrived.
In general the SOAP components in Citrus automatically handle SOAP Envelope and SOAP body. So you just need to define the pure body content as payload. SOAP Envelope is added automatically.
Hope this is more clear now.

SonargraphMeasureComputer error in SonarQube background task log

We are using SonarQube 5.4 and we found the following log lines:
....
INFO [c.q.p.s.s.p.e.AbstractSmellMeasureComputer] Computed measure 'SMELL_COUNT_BAD_FRAMEWORK_USAGE': 0)
INFO [c.q.p.s.s.p.e.AbstractSmellMeasureComputer] Computed measure 'SMELL_COUNT_USELESS_TEST': 0)
INFO [c.q.p.s.s.p.e.AbstractSmellMeasureComputer] Computed measure 'SMELL_COUNT_WRONG_LANGUAGE': 0)
INFO [c.q.p.s.s.p.e.AbstractSmellMeasureComputer] Computed measure 'SMELL_COUNT_WRONG_LOGIC': 0)
ERROR [c.h.s.m.SonargraphMeasureComputer] Not processing for module: xxx
ERROR [c.h.s.m.SonargraphMeasureComputer] Not processing for module: xxx
INFO [c.q.p.s.s.p.e.AbstractSmellMeasureComputer] Computing measures for component 'xxx' and level PROJECT)
INFO [c.q.p.s.s.p.e.AbstractSmellMeasureComputer] Computed measure 'SMELL_COUNT': 0)
ERROR [c.h.s.m.SonargraphMeasureComputer] Not processing for module: xxx
ERROR [c.h.s.m.SonargraphMeasureComputer] Not processing for module: xxx
INFO [o.s.s.c.s.ExecuteVisitorsStep] Execution time for each component visitor:
INFO [o.s.s.c.s.ExecuteVisitorsStep] - LoadComponentUuidsHavingOpenIssuesVisitor | time=79ms
INFO [o.s.s.c.s.ExecuteVisitorsStep] - IntegrateIssuesVisitor | time=29920ms
...
We could not find an explanation for the error lines. Does anyone have any ideas?

Apache Storm Flux Simple KafkaSpout --> KafkaBolt NullPointerException

I'm using Apache Storm 0.10.0-beta1 and started converting some topologies to Flux. I decided to start with a simple topology that reads from a Kafka queue and writes to a different Kafka queue. I get this error, which I am having a difficult time figuring out what is wrong. The topology yaml file follows the error.
Parsing file: /Users/frank/src/mapper/mapper.yaml
388 [main] INFO o.a.s.f.p.FluxParser - loading YAML from input stream...
391 [main] INFO o.a.s.f.p.FluxParser - Not performing property substitution.
391 [main] INFO o.a.s.f.p.FluxParser - Not performing environment variable substitution.
466 [main] INFO o.a.s.f.FluxBuilder - Detected DSL topology...
Exception in thread "main" java.lang.NullPointerException
at org.apache.storm.flux.FluxBuilder.canInvokeWithArgs(FluxBuilder.java:561)
at org.apache.storm.flux.FluxBuilder.findCompatibleConstructor(FluxBuilder.java:392)
at org.apache.storm.flux.FluxBuilder.buildObject(FluxBuilder.java:288)
at org.apache.storm.flux.FluxBuilder.buildSpout(FluxBuilder.java:361)
at org.apache.storm.flux.FluxBuilder.buildSpouts(FluxBuilder.java:349)
at org.apache.storm.flux.FluxBuilder.buildTopology(FluxBuilder.java:84)
at org.apache.storm.flux.Flux.runCli(Flux.java:153)
at org.apache.storm.flux.Flux.main(Flux.java:98)
Topology yaml:
name: "mapper-topology"
config:
topology.workers: 1
topology.debug: true
kafka.broker.properties.metadata.broker.list: "localhost:9092"
kafka.broker.properties.request.required.acks: "1"
kafka.broker.properties.serializer.class: "kafka.serializer.StringEncoder"
# component definitions
components:
- id: "topicSelector"
className: "storm.kafka.bolt.selector.DefaultTopicSelector"
constructorArgs:
- "schemaq"
- id: "kafkaMapper"
className: "storm.kafka.bolt.mapper.FieldNameBasedTupleToKafkaMapper"
# spout definitions
spouts:
- id: "kafka-spout"
className: "storm.kafka.SpoutConfig"
parallelism: 1
constructorArgs:
- ref: "zkHosts"
- "mapperq"
- "/mapperq"
- "id-mapperq"
properties:
- name: "forceFromStart"
value: true
- name: "scheme"
ref: "stringMultiScheme"
# bolt definitions
bolts:
- id: "kafka-bolt"
className: "storm.kafka.bolt.KafkaBolt"
parallelism: 1
configMethods:
- name: "withTopicSelector"
args: [ref: "topicSelector"]
- name: "withTupleToKafkaMapper"
args: [ref: "kafkaMapper"]
# streams
streams:
- name: "kafka-spout --> kafka-bolt"
from: "kafka-spout"
to: "kafka-bolt"
grouping:
type: SHUFFLE
And here is the command:
storm jar /Users/frank/src/mapper/target/mapper-0.1.0-SNAPSHOT-standalone.jar org.apache.storm.flux.Flux --local mapper.yaml
spout classname should be storm.kafka.KafkaSpout, not storm.kafka.SpoutConfig. You should define SpoutConfig to "components" section, and let spout refer this.
You can refer https://github.com/apache/storm/blob/master/external/flux/flux-examples/src/main/resources/kafka_spout.yaml to see how to setup KafkaSpout from flux.

Resources