How to get SMTP sesion error with SwiftMailer in Laravel - laravel

I try to send email to fake address. And I can't catch error if this email address is not available and email can not be delivered.
In SMTP-server logs I see:
Fri 2020-12-11 12:08:58.582: 03: --> 250-STARTTLS
Fri 2020-12-11 12:08:58.582: 03: --> 250 SIZE
Fri 2020-12-11 12:08:58.582: 02: <-- MAIL FROM:<my_email#mydomain.com>
Fri 2020-12-11 12:08:58.583: 03: --> 250 2.1.0 Sender OK
Fri 2020-12-11 12:08:58.583: 02: <-- RCPT TO:<somefakeaddress#mydomain.com>
Fri 2020-12-11 12:08:58.584: 03: --> 550 5.1.1 Recipient unknown <mi#f
Fri 2020-12-11 12:08:58.585: 02: <-- RSET
Fri 2020-12-11 12:08:58.585: 03: --> 250 2.0.0 RSET? Well, OK
Fri 2020-12-11 12:08:58.586: 02: <-- QUIT
Fri 2020-12-11 12:08:58.586: 03: --> 221 2.0.0 See ya in cyberspace
Fri 2020-12-11 12:08:58.586: 04: SMTP session terminated (Bytes in/out
But in my Laravel app I can't catch this error even if I use Event Listeners https://laravel.com/docs/7.x/mail#events
Of course, the code
Mail::to('somefakeaddress#mydomain.com')->send(new Mailable($mail_data));
returns nothing.
Is there any idea how to catch the error of the SMTP session in Laravel?

Found this solution:
Add SwiftMailer plugin to mailer
$mailer = Mail::getSwiftMailer();
$logger = new \Swift_Plugins_Loggers_ArrayLogger();
$mailer->registerPlugin(new \Swift_Plugins_LoggerPlugin($logger));
Then we can send mail as usual:
Mail::to(...)->send(new Mailable($data_for_mailable_class));
And after that we can search for any error messages in $logger array.
For test I use dump($logger);
This helped me find a solution:
How to set SwifMailer plugins when using the Laravel mail class?

Related

python 2.7 installing packages using pip on windows returns SSL error

The environment is python 2.7.15 on a Windows 7 PC (Win7 must be used due to user requirements). When attempting to install packages such as plotly and scipy, the below error is returned.
Certificate did not match expected hostname: files.pythonhosted.org. Certifica
te: {'crlDistributionPoints': (u'http://cdp.rapidssl.com/RapidSSLTLSRSACAG1.crl'
,), 'subjectAltName': (('DNS', '*.logicnow.us'), ('DNS', 'logicnow.us')), 'notBe
fore': u'Mar 14 00:00:00 2018 GMT', 'caIssuers': (u'http://cacerts.rapidssl.com/
RapidSSLTLSRSACAG1.crt',), 'OCSP': (u'http://status.rapidssl.com',), 'serialNumb
er': u'035BBD7099460760E357EF1F5E12C586', 'notAfter': 'May 12 12:00:00 2020 GMT'
, 'version': 3L, 'subject': ((('commonName', u'*.logicnow.us'),),), 'issuer': ((
('countryName', u'US'),), (('organizationName', u'DigiCert Inc'),), (('organizat
ionalUnitName', u'www.digicert.com'),), (('commonName', u'RapidSSL TLS RSA CA G1
'),))}
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None))
after connection broken by 'SSLError(CertificateError("hostname 'files.pythonho
sted.org' doesn't match either of '*.logicnow.us', 'logicnow.us'",),)': /package
s/c4/0c/1b8241395302fd66b34a0fe0774ed83632afd14aa5c995b262fb7b3ac540/plotly-2.7.
0.tar.gz
Certificate did not match expected hostname: files.pythonhosted.org. Certifica
te: {'crlDistributionPoints': (u'http://cdp.rapidssl.com/RapidSSLTLSRSACAG1.crl'
,), 'subjectAltName': (('DNS', '*.logicnow.us'), ('DNS', 'logicnow.us')), 'notBe
fore': u'Mar 14 00:00:00 2018 GMT', 'caIssuers': (u'http://cacerts.rapidssl.com/
RapidSSLTLSRSACAG1.crt',), 'OCSP': (u'http://status.rapidssl.com',), 'serialNumb
er': u'035BBD7099460760E357EF1F5E12C586', 'notAfter': 'May 12 12:00:00 2020 GMT'
, 'version': 3L, 'subject': ((('commonName', u'*.logicnow.us'),),), 'issuer': ((
('countryName', u'US'),), (('organizationName', u'DigiCert Inc'),), (('organizat
ionalUnitName', u'www.digicert.com'),), (('commonName', u'RapidSSL TLS RSA CA G1
'),))}
Could not install packages due to an EnvironmentError: HTTPSConnectionPool(host=
'files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/c4
/0c/1b8241395302fd66b34a0fe0774ed83632afd14aa5c995b262fb7b3ac540/plotly-2.7.0.ta
r.gz (Caused by SSLError(CertificateError("hostname 'files.pythonhosted.org' doe
sn't match either of '*.logicnow.us', 'logicnow.us'",),))
I have tried to use the below command which returns the following error.
pip install --trusted-host=pypi.python.org --trusted-host=pypi.org --trusted-host=files.pythonhosted.org plotly
Error:
Collecting plotly
Downloading https://files.pythonhosted.org/packages/c4/0c/1b8241395302fd66b34a
0fe0774ed83632afd14aa5c995b262fb7b3ac540/plotly-2.7.0.tar.gz
- 10kB 1.5MB/s
THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have u
pdated the package versions, please update the hashes. Otherwise, examine the pa
ckage contents carefully; someone may have tampered with them.
plotly from https://files.pythonhosted.org/packages/c4/0c/1b8241395302fd66b3
4a0fe0774ed83632afd14aa5c995b262fb7b3ac540/plotly-2.7.0.tar.gz#sha256=f7305816a4
23fd899da20919282445703118aeece19a54fb97be4cbf302341f8:
Expected sha256 f7305816a423fd899da20919282445703118aeece19a54fb97be4cbf302341f8
Got 5bc26e972b40ab4355857a8140d0ef730b267cfe26e8aff2a8dca148f227e5d7
Is there an alternative to installing packages?

Kafka stream app failing to fetch offsets for partition

I created a kafka cluster with 3 brokers and following details:
Created 3 topics, each one with replication factor=3 and partitions=2.
Created 2 producers each one writing to one of the topics.
Created a Streams application to process messages from 2 topics and write to the 3rd topic.
It was all running fine till now but I suddenly started getting the following warning when starting the Streams application:
[WARN ] 2018-06-08 21:16:49.188 [Stream3-4f7403ad-aba6-4d34-885d-60114fc9fcff-StreamThread-1] org.apache.kafka.clients.consumer.internals.Fetcher [Consumer clientId=Stream3-4f7403ad-aba6-4d34-885d-60114fc9fcff-StreamThread-1-restore-consumer, groupId=] Attempt to fetch offsets for partition Stream3-KSTREAM-OUTEROTHER-0000000005-store-changelog-0 failed due to: Disk error when trying to access log file on the disk.
Due to this warning, Streams application is not processing anything from the 2 topics.
I tried following things:
Stopped all brokers, deleted kafka-logs directory for each broker and restarted the brokers. It didn't solve the issue.
Stopped zookeeper and all brokers, deleted zookeeper logs as well as kafka-logs for each broker, restarted zookeeper and brokers and created the topics again. This too didn't solve the issue.
I am not able to find anything related to this error on official docs or web. Does anyone have an idea of why am I getting this error suddenly?
EDIT:
Out of 3 brokers, 2 brokers(broker-0 and broker-2) continously emit these logs:
Broker-0 logs:
[2018-06-09 02:03:08,750] INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition initial11_topic-1 as the leader reported an error: NOT_LEADER_FOR_PARTITION (kafka.server.ReplicaFetcherThread)
[2018-06-09 02:03:08,750] INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition initial12_topic-0 as the leader reported an error: NOT_LEADER_FOR_PARTITION (kafka.server.ReplicaFetcherThread)
Broker-2 logs:
[2018-06-09 02:04:46,889] INFO [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition initial11_topic-1 as the leader reported an error: NOT_LEADER_FOR_PARTITION (kafka.server.ReplicaFetcherThread)
[2018-06-09 02:04:46,889] INFO [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition initial12_topic-0 as the leader reported an error: NOT_LEADER_FOR_PARTITION (kafka.server.ReplicaFetcherThread)
Broker-1 shows following logs:
[2018-06-09 01:21:26,689] INFO [GroupMetadataManager brokerId=1] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2018-06-09 01:31:26,689] INFO [GroupMetadataManager brokerId=1] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2018-06-09 01:39:44,667] ERROR [KafkaApi-1] Number of alive brokers '0' does not meet the required replication factor '1' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)
[2018-06-09 01:41:26,689] INFO [GroupMetadataManager brokerId=1] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
I again stopped zookeeper and brokers, deleted their logs and restarted. As soon as I create the topics again, I start getting the above logs.
Topic details:
[zk: localhost:2181(CONNECTED) 3] get /brokers/topics/initial11_topic
{"version":1,"partitions":{"1":[1,0,2],"0":[0,2,1]}}
cZxid = 0x53
ctime = Sat Jun 09 01:25:42 EDT 2018
mZxid = 0x53
mtime = Sat Jun 09 01:25:42 EDT 2018
pZxid = 0x54
cversion = 1
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 52
numChildren = 1
[zk: localhost:2181(CONNECTED) 4] get /brokers/topics/initial12_topic
{"version":1,"partitions":{"1":[2,1,0],"0":[1,0,2]}}
cZxid = 0x61
ctime = Sat Jun 09 01:25:47 EDT 2018
mZxid = 0x61
mtime = Sat Jun 09 01:25:47 EDT 2018
pZxid = 0x62
cversion = 1
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 52
numChildren = 1
[zk: localhost:2181(CONNECTED) 5] get /brokers/topics/final11_topic
{"version":1,"partitions":{"1":[0,1,2],"0":[2,0,1]}}
cZxid = 0x48
ctime = Sat Jun 09 01:25:32 EDT 2018
mZxid = 0x48
mtime = Sat Jun 09 01:25:32 EDT 2018
pZxid = 0x4a
cversion = 1
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 52
numChildren = 1
Any clue?
I found out the issue. It was due to following incorrect config in server.properties of broker-1:
advertised.listeners=PLAINTEXT://10.23.152.109:9094
Mistakenly port for advertised.listeners got changed to same as port of advertised.listeners of broker-2.

What is the best practice to create a mock service with Citrus Automation Framework for multiple responses?

I've got Component Integration Test suite setup with Citrus Framework. In order to create a mock service for different components. I've been using simple mapping strategy with the help of XPathPayloadMappingKeyExtractor.
The challenge I've got is to create a multiple responses for different soap requests.
I've been struggling to understand the way it's responding. Any experienced person who can help me out on Citrus and Spring please ?
My Context file:
<citrus-ws:client id="ClientEndpoint"
request-url="http://localhost:8085/"/>
<citrus-ws:server id="BS_Customer_Information"
port="8085"
auto-start="true"/>
My Test Method:
#CitrusTest
public void testingServer() {
soap().client(ClientEndpoint)
.send()
.soapAction("urn:RetrieveAddressBookOP_01")
.payload(new ClassPathResource("Requests/Sample1.xml"));
soap().server(BS_Customer_Information)
.receive()
.soapAction("urn:RetrieveAddressBookOP_01");
soap().server(BS_Customer_Information)
.send()
.payload(new ClassPathResource("AtomicResponses/Response1.xml"));
soap().client(ClientEndpoint)
.receive()
.messageType(MessageType.XML);
}
Issue 1: My Soap Client times out and could not get the response
Question 1: Should the server response be parsed into Soap Env and Body ?
Question 2: Why Would it still call go to Channel Endpoint.inbound when I've already defined my response?
Console Log:
2521 DEBUG gEndpointInterceptor| Received SOAP request:
<?xml version="1.0" encoding="UTF-8"?><SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
<SOAP-ENV:Header/>
<SOAP-ENV:Body>
<RetrieveAddressBookOP_01"/>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
2527 DEBUG r.WebServiceEndpoint| Received SOAP request:
SOAPMESSAGE [payload: <?xml version="1.0" encoding="UTF-8"?><au:RetrieveAddressBookOP_01 xmlns:au="au.com.teysau.dataservice"/>][headers: {citrus_message_id=e1311140-6ee3-415a-815e-0d162cbc03c1, citrus_message_timestamp=1490649843807, citrus_soap_action=urn:RetrieveAddressBookOP_01, citrus_http_request_uri=/, citrus_http_context_path=, citrus_http_query_params=, citrus_http_method=POST}][header-data: [<?xml version="1.0" encoding="UTF-8"?><SOAP-ENV:Header xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"/>]][attachments: []]
2527 DEBUG annelEndpointAdapter| Forwarding request to message channel ...
2527 DEBUG t.TestContextFactory| Created new test context - using global variables: '{}'
2527 DEBUG ltCorrelationManager| Saving correlation key for 'citrus_message_correlator_ChannelEndpointAdapter:producer'
2527 DEBUG context.TestContext| Setting variable: citrus_message_correlator_ChannelEndpointAdapter:producer with value: 'citrus_message_id = 'e1311140-6ee3-415a-815e-0d162cbc03c1''
2527 DEBUG .ChannelSyncProducer| Sending message to channel: 'BS_Customer_Information.inbound'
2527 DEBUG .ChannelSyncProducer| Message to send is:
SOAPMESSAGE [payload: <?xml version="1.0" encoding="UTF-8"?><RetrieveAddressBookOP_01"/>][headers: {citrus_message_id=e1311140-6ee3-415a-815e-0d162cbc03c1, citrus_message_timestamp=1490649843807, citrus_soap_action=urn:RetrieveAddressBookOP_01, citrus_http_request_uri=/, citrus_http_context_path=, citrus_http_query_params=, citrus_http_method=POST}][header-data: [<?xml version="1.0" encoding="UTF-8"?><SOAP-ENV:Header xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"/>]][attachments: []]
2527 INFO .ChannelSyncProducer| Message was sent to channel: 'BS_Customer_Information.inbound'
3539 WARN annelEndpointAdapter| Reply timed out after 1000ms. Did not receive reply message on reply channel
3540 DEBUG annelEndpointAdapter| Did not receive reply message - no response is simulated
3540 DEBUG r.WebServiceEndpoint| No reply message from endpoint adapter 'com.consol.citrus.channel.ChannelEndpointAdapter#6dbd0a41'
3540 WARN r.WebServiceEndpoint| No SOAP response for calling client
3542 DEBUG ageDispatcherServlet| Successfully completed request
3546 DEBUG server.Server| RESPONSE / 202 handled=true
3546 DEBUG ver.HttpChannelState| HttpChannelState#244a4722{s=DISPATCHED i=true a=null} unhandle DISPATCHED
3553 DEBUG erver.HttpConnection| org.eclipse.jetty.server.HttpConnection$SendCallback#27d2cb2e[PROCESSING][i=ResponseInfo{HTTP/1.1 202 null,-1,false},cb=org.eclipse.jetty.server.HttpChannel$CommitCallback#2f1ea019] generate: NEED_HEADER (null,[p=0,l=0,c=0,r=0],true)#START
3556 DEBUG erver.HttpConnection| org.eclipse.jetty.server.HttpConnection$SendCallback#27d2cb2e[PROCESSING][i=ResponseInfo{HTTP/1.1 202 null,-1,false},cb=org.eclipse.jetty.server.HttpChannel$CommitCallback#2f1ea019] generate: FLUSH ([p=0,l=114,c=8192,r=114],[p=0,l=0,c=0,r=0],true)#COMPLETING
3556 DEBUG io.WriteFlusher| write: WriteFlusher#76f08437{IDLE} [HeapByteBuffer#2c34f7c0[p=0,l=114,c=8192,r=114]={<<<HTTP/1.1 202 Acce....v20160210)\r\n\r\n>>>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}]
3557 DEBUG io.WriteFlusher| update WriteFlusher#76f08437{WRITING}:IDLE-->WRITING
3562 INFO ent.WebServiceClient| SOAP message was sent to endpoint: 'http://localhost:8085/'
3562 INFO ent.WebServiceClient| Received no SOAP response from endpoint: 'http://localhost:8085/'
3562 INFO citrus.Citrus|
3562 DEBUG citrus.Citrus| TEST STEP 1/4 SUCCESS
3562 INFO citrus.Citrus|
3562 DEBUG citrus.Citrus| TEST STEP 2/4: receive
3563 DEBUG io.ChannelEndPoint| flushed 114 SelectChannelEndPoint#1c2a40b4{/127.0.0.1:54924<->8085,Open,in,out,-,W,1092/30000,HttpConnection}{io=0,kio=0,kro=1}
3566 DEBUG nnel.ChannelConsumer| Receiving message from: BS_Customer_Information.inbound
3567 DEBUG nnel.ChannelConsumer| Received message from: BS_Customer_Information.inbound
3567 DEBUG ltCorrelationManager| Saving correlation key for 'citrus_message_correlator_BS_Customer_Information:consumer'
3567 DEBUG context.TestContext| Setting variable: citrus_message_correlator_BS_Customer_Information:consumer with value: 'citrus_message_id = 'e1311140-6ee3-415a-815e-0d162cbc03c1''
3567 DEBUG ltCorrelationManager| Saving correlated object for 'citrus_message_id = 'e1311140-6ee3-415a-815e-0d162cbc03c1''
3567 DEBUG ageValidatorRegistry| Found 3 message validators for message type: XML
3567 DEBUG mXmlMessageValidator| Start message validation ...
3567 DEBUG mXmlMessageValidator| Start XML message validation
3569 DEBUG mXmlMessageValidator| Starting XML schema validation ...
3569 WARN mXmlMessageValidator| Neither schema instance nor schema repository defined - skipping XML schema validation
3569 INFO mXmlMessageValidator| XML message validation successful: All values OK
3569 DEBUG mXmlMessageValidator| Start message header validation ...
3570 DEBUG io.WriteFlusher| update WriteFlusher#76f08437{IDLE}:WRITING-->IDLE
3570 DEBUG erver.HttpConnection| org.eclipse.jetty.server.HttpConnection$SendCallback#27d2cb2e[PROCESSING][i=ResponseInfo{HTTP/1.1 202 null,-1,false},cb=org.eclipse.jetty.server.HttpChannel$CommitCallback#2f1ea019] generate: DONE ([p=114,l=114,c=8192,r=0],[p=0,l=0,c=0,r=0],true)#END
3570 DEBUG mXmlMessageValidator| Validating header element: citrus_soap_action='urn:RetrieveAddressBookOP_01': OK.
3571 INFO mXmlMessageValidator| Message header validation successful: All properties OK
3571 INFO mXmlMessageValidator| Message validation successful: All values OK
3571 INFO citrus.Citrus|
3571 DEBUG citrus.Citrus| TEST STEP 2/4 SUCCESS
3571 INFO citrus.Citrus|
3571 DEBUG citrus.Citrus| TEST STEP 3/4: send
3571 DEBUG ngCorrelationManager| Get correlation key for 'citrus_message_correlator_BS_Customer_Information:consumer'
3571 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = 'e1311140-6ee3-415a-815e-0d162cbc03c1''
3571 DEBUG .ChannelSyncConsumer| Sending message to reply channel: 'org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#66f66866'
3571 DEBUG .ChannelSyncConsumer| Message to send is:
SOAPMESSAGE [payload: <AddressBook>
<Address>
<Address_Number>14</Address_Number>
<Long_Address>Willis Street</Long_Address>
<Tax_ID>7987398</Tax_ID>
<Alpha_Name>what is alpha</Alpha_Name>
<Code>UN_</Code>
<Mailing_Name>James Bond</Mailing_Name>
<Address_Line1>take the first part of the long address</Address_Line1>
<Address_Line2>You may take the second part not</Address_Line2>
<Postal_Code>8900</Postal_Code>
<City>WLN</City>
<Country>NZ</Country>
<Prefix>233</Prefix>
<Phone_Number>025364</Phone_Number>
<Prefix_2 xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"/>
<Phone_Number2 xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"/>
<userID>s3b9d</userID>
<BatchNumber>jkdhj59</BatchNumber>
<Tansaction_Number>jkd3j29</Tansaction_Number>
<Line_Number>291597</Line_Number>
</Address>
</AddressBook>][headers: {citrus_message_id=fd946a46-f67a-405d-b118-b1b6b3d562c1, citrus_message_timestamp=1490649843443}][attachments: []]
3572 WARN emporaryReplyChannel| Reply message received but the receiving thread has exited due to a timeout:GenericMessage [payload=SOAPMESSAGE [payload: <AddressBook>
<Address>
<Address_Number>14</Address_Number>
<Long_Address>Willis Street</Long_Address>
<Tax_ID>7987398</Tax_ID>
<Alpha_Name>what is alpha</Alpha_Name>
<Code>UN_</Code>
<Mailing_Name>James Bond</Mailing_Name>
<Address_Line1>take the first part of the long address</Address_Line1>
<Address_Line2>You may take the second part not</Address_Line2>
<Postal_Code>8900</Postal_Code>
<City>WLN</City>
<Country>NZ</Country>
<Prefix>233</Prefix>
<Phone_Number>025364</Phone_Number>
<Prefix_2 xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"/>
<Phone_Number2 xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"/>
<userID>s3b9d</userID>
<BatchNumber>jkdhj59</BatchNumber>
<Tansaction_Number>jkd3j29</Tansaction_Number>
<Line_Number>291597</Line_Number>
</Address>
</AddressBook>][headers: {citrus_message_id=fd946a46-f67a-405d-b118-b1b6b3d562c1, citrus_message_timestamp=1490649843443}][attachments: []], headers={id=cfc4eeaa-eac8-8f00-2f45-2e3fcb16fdb6, timestamp=1490649844854}]
3572 INFO .ChannelSyncConsumer| Message was sent to reply channel: 'org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#66f66866'
3572 INFO citrus.Citrus|
3572 DEBUG citrus.Citrus| TEST STEP 3/4 SUCCESS
3572 INFO citrus.Citrus|
3572 DEBUG citrus.Citrus| TEST STEP 4/4: receive
3572 DEBUG ngCorrelationManager| Get correlation key for 'citrus_message_correlator_ClientEndpoint'
3572 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
3572 DEBUG citrus.RetryLogger| Reply message did not arrive yet - retrying in 500ms
3575 DEBUG http.HttpParser| reset HttpParser{s=END,214 of 214}
3576 DEBUG http.HttpParser| END --> START
3576 DEBUG server.HttpChannel| HttpChannelOverHttp#4820529f{r=1,c=false,a=IDLE,uri=} handle exit, result COMPLETE
3576 DEBUG io.ChannelEndPoint| filled 0 SelectChannelEndPoint#1c2a40b4{/127.0.0.1:54924<->8085,Open,in,out,-,-,13/30000,HttpConnection}{io=0,kio=0,kro=1}
3576 DEBUG io.ChannelEndPoint| filled 0 SelectChannelEndPoint#1c2a40b4{/127.0.0.1:54924<->8085,Open,in,out,-,-,13/30000,HttpConnection}{io=0,kio=0,kro=1}
3576 DEBUG http.HttpParser| parseNext s=START HeapByteBuffer#6b681750[p=0,l=0,c=16384,r=0]={<<<>>>POST / HTTP/1.1\r\n...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
3576 DEBUG o.AbstractConnection| fillInterested HttpConnection#2c01adea[FILLING,SelectChannelEndPoint#1c2a40b4{/127.0.0.1:54924<->8085,Open,in,out,-,-,13/30000,HttpConnection}{io=0,kio=0,kro=1}][p=HttpParser{s=START,0 of -1},g=HttpGenerator{s=START},c=HttpChannelOverHttp#4820529f{r=1,c=false,a=IDLE,uri=}]
3576 DEBUG o.AbstractConnection| FILLING-->FILLING_FILL_INTERESTED HttpConnection#2c01adea[FILLING_FILL_INTERESTED,SelectChannelEndPoint#1c2a40b4{/127.0.0.1:54924<->8085,Open,in,out,-,-,13/30000,HttpConnection}{io=0,kio=0,kro=1}][p=HttpParser{s=START,0 of -1},g=HttpGenerator{s=START},c=HttpChannelOverHttp#4820529f{r=1,c=false,a=IDLE,uri=}]
3577 DEBUG o.AbstractConnection| FILLING_FILL_INTERESTED-->FILL_INTERESTED HttpConnection#2c01adea[FILL_INTERESTED,SelectChannelEndPoint#1c2a40b4{/127.0.0.1:54924<->8085,Open,in,out,-,-,13/30000,HttpConnection}{io=0,kio=0,kro=1}][p=HttpParser{s=START,0 of -1},g=HttpGenerator{s=START},c=HttpChannelOverHttp#4820529f{r=1,c=false,a=IDLE,uri=}]
3577 DEBUG electChannelEndPoint| Local interests updating 0 -> 1 for SelectChannelEndPoint#1c2a40b4{/127.0.0.1:54924<->8085,Open,in,out,R,-,0/30000,HttpConnection}{io=1,kio=0,kro=1}
3577 DEBUG io.SelectorManager| Queued change org.eclipse.jetty.io.SelectChannelEndPoint$1#67e66c5c
3578 DEBUG io.SelectorManager| Selector loop woken up from select, 0/1 selected
3578 DEBUG io.SelectorManager| Running change org.eclipse.jetty.io.SelectChannelEndPoint$1#67e66c5c
3578 DEBUG electChannelEndPoint| Key interests updated 0 -> 1 on SelectChannelEndPoint#1c2a40b4{/127.0.0.1:54924<->8085,Open,in,out,R,-,1/30000,HttpConnection}{io=1,kio=1,kro=1}
3578 DEBUG io.SelectorManager| Selector loop waiting on select
4072 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
4072 DEBUG citrus.RetryLogger| Reply message did not arrive yet - retrying in 500ms
4573 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
4573 DEBUG citrus.RetryLogger| Reply message did not arrive yet - retrying in 500ms
5073 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
5073 DEBUG citrus.RetryLogger| Reply message did not arrive yet - retrying in 500ms
5574 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
5574 DEBUG citrus.RetryLogger| Reply message did not arrive yet - retrying in 500ms
6075 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
6075 DEBUG citrus.RetryLogger| Reply message did not arrive yet - retrying in 500ms
6575 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
6575 DEBUG citrus.RetryLogger| Reply message did not arrive yet - retrying in 500ms
7075 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
7075 DEBUG citrus.RetryLogger| Reply message did not arrive yet - retrying in 500ms
7576 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
7576 DEBUG citrus.RetryLogger| Reply message did not arrive yet - retrying in 500ms
8076 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
8076 DEBUG citrus.RetryLogger| Reply message did not arrive yet - retrying in 500ms
8576 DEBUG ltCorrelationManager| Finding correlated object for 'citrus_message_id = '69b4056a-cae6-4004-a837-41de069865b8''
8580 INFO report.JIRAConsumer| Invoking JIRA API...
works.integration.jira.exceptions.ServiceBindingException: Missing Required Properties - SUMMARY, EXMESSAGE, DETAILEDEXCEPTION, PROJECT
You have to add a fork(true) option to the first `soap().client().send()' action because the Http SOAP protocol is synchronous by nature. The first action in your test waits for a synchronous response and blocks the rest of the test case execution.
Obviously your test needs to receive some other messages before that client response with soap().server(). That is why you need to fork the client send action in the first place so the server actions can perform before the client response has arrived.
In general the SOAP components in Citrus automatically handle SOAP Envelope and SOAP body. So you just need to define the pure body content as payload. SOAP Envelope is added automatically.
Hope this is more clear now.

Getting unknown server error while initializing GSS security context

Working on a cocoa OSX app, Mac bind with active directory and logged in as active-dir user.
Getting the unknown server error while initializing GSS security context: with a PrincipalName provided by the server.
gss_init_sec_context
major: unknown routine error
minor: Server (krbtgt/LOCAL#XYZ.LOCAL)unknown while looking up 'WIN- ****$/xyz.local#LOCAL' (cached result, timeout in 167 sec)
Klist -5 displays these on terminal:
Credentials cache: API:DD4CC511-7BE2-4267-9923-6C8ABCD9297D
Principal: user#XYZ.LOCAL
Issued Expires Principal
Nov 5 17:21:23 2015 Nov 6 03:21:23 2015 krbtgt/XYZ.LOCAL#XYZ.LOCAL
Because off this error, I changed ker5.conf file like this:
[libdefaults]
default_realm = XYZ.LOCAL
renewable = true
forwardable= true
ticket_lifetime = 20d
renew_lifetime = 1d
default_tgs_enctypes = aes256-cts-hmac-sha1-96, aes128-cts-hmac-sha1-96, des3-cbc-sha1, arcfour-hmac-md5
default_tkt_enctypes = aes256-cts-hmac-sha1-96, aes128-cts-hmac-sha1-96, des3-cbc-sha1, arcfour-hmac-md5
[domain_realm]
xyz.com = XYZ.LOCAL
.xyz.com = XYZ.LOCAL

Make Cygnus use WebHDFS to write to local HDFS

I'm trying to make a local Orion+Cygnus persist Orion's data on a local HDFS through WebHDFS.
On Cygnus' instructions on gitub, very little is mentioned about WebHDFS, as the configuration is more about HttpFS.
On the .md OrionHDFSsink it's said that hdfs_port=50070 is for WebHDFS, as indeed my HDFS is. So I would expect by setting the port this way cygnus would automatically use WebHDFS, but on my case it doesn't seem to be working this way.
So, here's my agent_1.conf:
cygnusagent.sources = http-source
cygnusagent.sinks = hdfs-sink
cygnusagent.channels = hdfs-channel
# source configuration
cygnusagent.sources.http-source.channels = hdfs-channel
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnusagent.sources.http-source.port = 5050
cygnusagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.OrionRestHandler
cygnusagent.sources.http-source.handler.notification_target = /notify
cygnusagent.sources.http-source.handler.default_service = def_serv
cygnusagent.sources.http-source.handler.default_service_path = def_servpath
cygnusagent.sources.http-source.handler.events_ttl = 4
cygnusagent.sources.http-source.interceptors = ts gi
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
# OrionHDFSSink configuration
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
cygnusagent.sinks.hdfs-sink.type = com.telefonica.iot.cygnus.sinks.OrionHDFSSink
cygnusagent.sinks.hdfs-sink.hdfs_host = localHDFS.ip
cygnusagent.sinks.hdfs-sink.hdfs_port = 50070
cygnusagent.sinks.hdfs-sink.hdfs_username = HDFSrootUser
cygnusagent.sinks.hdfs-sink.attr_persistence = column
# hdfs-channel configuration
cygnusagent.channels.hdfs-channel.type = memory
cygnusagent.channels.hdfs-channel.capacity = 1000
cygnusagent.channels.hdfs-channel.transactionCapacity = 100
When I update an Entity on Orion, to whom Cygnus is subbed, Cygnus logs the following:
02 Sep 2015 20:09:12,353 INFO [2055470757#qtp-1523539038-0] (com.telefonica.iot.cygnus.handlers.OrionRestHandler.getEvents:150) - Starting transaction (1441217314-956-0000000000)
02 Sep 2015 20:09:12,362 INFO [2055470757#qtp-1523539038-0] (com.telefonica.iot.cygnus.handlers.OrionRestHandler.getEvents:236) - Received data ({ "subscriptionId" : "55e735c9b89e8535f8ca5ef2", "originator" : "localhost", "contextResponses" : [ { "contextElement" : { "type" : "Reading", "isPattern" : "false", "id" : "Reading1.1", "attributes" : [ { "name" : "Cost", "type" : "double", "value" : "32" }, { "name" : "Reading_ID", "type" : "integer", "value" : "14" }, { "name" : "Threshold", "type" : "double", "value" : "30" }, { "name" : "email", "type" : "string", "value" : "arthurmvieira#hotmail.com" } ] }, "statusCode" : { "code" : "200", "reasonPhrase" : "OK" } } ]})
02 Sep 2015 20:09:12,366 INFO [2055470757#qtp-1523539038-0] (com.telefonica.iot.cygnus.handlers.OrionRestHandler.getEvents:258) - Event put in the channel (id=2020008711, ttl=4)
02 Sep 2015 20:09:12,432 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:128) - Event got from the channel (id=2020008711, headers={fiware-servicepath=def_servpath, destination=reading1.1_reading, content-type=application/json, fiware-service=def_serv, ttl=4, transactionId=1441217314-956-0000000000, timestamp=1441217352368}, bodyLength=812)
02 Sep 2015 20:09:12,549 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionHDFSSink.persist:356) - [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file (def_serv/def_servpath/reading1.1_reading/reading1.1_reading.txt), Data ({"recvTime":"2015-09-02T18:09:12.368Z","Cost":"32", "Cost_md":[],"Reading_ID":"14", "Reading_ID_md":[],"Threshold":"30", "Threshold_md":[],"email":"arthurmvieira#hotmail.com", "email_md":[]})
02 Sep 2015 20:09:12,557 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:143) - Persistence error (The /user/root/def_serv/def_servpath/reading1.1_reading directory could not be created in HDFS. HttpFS response: 503 Service unavailable)
02 Sep 2015 20:09:12,558 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:173) - An event was put again in the channel (id=2020008711, ttl=3)
02 Sep 2015 20:09:12,558 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:193) - Finishing transaction (1441217314-956-0000000000)
02 Sep 2015 20:09:13,560 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:128) - Event got from the channel (id=2020008711, headers={fiware-servicepath=def_servpath, destination=reading1.1_reading, content-type=application/json, fiware-service=def_serv, ttl=3, transactionId=1441217314-956-0000000000, timestamp=1441217352368}, bodyLength=812)
02 Sep 2015 20:09:13,574 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionHDFSSink.persist:356) - [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file (def_serv/def_servpath/reading1.1_reading/reading1.1_reading.txt), Data ({"recvTime":"2015-09-02T18:09:12.368Z","Cost":"32", "Cost_md":[],"Reading_ID":"14", "Reading_ID_md":[],"Threshold":"30", "Threshold_md":[],"email":"arthurmvieira#hotmail.com", "email_md":[]})
02 Sep 2015 20:09:13,574 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:143) - Persistence error (The /user/root/def_serv/def_servpath/reading1.1_reading directory could not be created in HDFS. HttpFS response: 503 Service unavailable)
02 Sep 2015 20:09:13,575 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:173) - An event was put again in the channel (id=2020008711, ttl=2)
02 Sep 2015 20:09:13,575 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:193) - Finishing transaction (1441217314-956-0000000000)
02 Sep 2015 20:09:15,576 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:128) - Event got from the channel (id=2020008711, headers={fiware-servicepath=def_servpath, destination=reading1.1_reading, content-type=application/json, fiware-service=def_serv, ttl=2, transactionId=1441217314-956-0000000000, timestamp=1441217352368}, bodyLength=812)
02 Sep 2015 20:09:15,590 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionHDFSSink.persist:356) - [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file (def_serv/def_servpath/reading1.1_reading/reading1.1_reading.txt), Data ({"recvTime":"2015-09-02T18:09:12.368Z","Cost":"32", "Cost_md":[],"Reading_ID":"14", "Reading_ID_md":[],"Threshold":"30", "Threshold_md":[],"email":"arthurmvieira#hotmail.com", "email_md":[]})
02 Sep 2015 20:09:15,599 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:143) - Persistence error (The /user/root/def_serv/def_servpath/reading1.1_reading directory could not be created in HDFS. HttpFS response: 503 Service unavailable)
02 Sep 2015 20:09:15,600 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:173) - An event was put again in the channel (id=2020008711, ttl=1)
02 Sep 2015 20:09:15,600 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:193) - Finishing transaction (1441217314-956-0000000000)
02 Sep 2015 20:09:18,601 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:128) - Event got from the channel (id=2020008711, headers={fiware-servicepath=def_servpath, destination=reading1.1_reading, content-type=application/json, fiware-service=def_serv, ttl=1, transactionId=1441217314-956-0000000000, timestamp=1441217352368}, bodyLength=812)
02 Sep 2015 20:09:18,615 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionHDFSSink.persist:356) - [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file (def_serv/def_servpath/reading1.1_reading/reading1.1_reading.txt), Data ({"recvTime":"2015-09-02T18:09:12.368Z","Cost":"32", "Cost_md":[],"Reading_ID":"14", "Reading_ID_md":[],"Threshold":"30", "Threshold_md":[],"email":"arthurmvieira#hotmail.com", "email_md":[]})
02 Sep 2015 20:09:18,618 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:143) - Persistence error (The /user/root/def_serv/def_servpath/reading1.1_reading directory could not be created in HDFS. HttpFS response: 503 Service unavailable)
02 Sep 2015 20:09:18,621 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:173) - An event was put again in the channel (id=2020008711, ttl=0)
02 Sep 2015 20:09:18,621 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:193) - Finishing transaction (1441217314-956-0000000000)
02 Sep 2015 20:09:22,622 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:128) - Event got from the channel (id=2020008711, headers={fiware-servicepath=def_servpath, destination=reading1.1_reading, content-type=application/json, fiware-service=def_serv, ttl=0, transactionId=1441217314-956-0000000000, timestamp=1441217352368}, bodyLength=812)
02 Sep 2015 20:09:22,635 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionHDFSSink.persist:356) - [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file (def_serv/def_servpath/reading1.1_reading/reading1.1_reading.txt), Data ({"recvTime":"2015-09-02T18:09:12.368Z","Cost":"32", "Cost_md":[],"Reading_ID":"14", "Reading_ID_md":[],"Threshold":"30", "Threshold_md":[],"email":"arthurmvieira#hotmail.com", "email_md":[]})
02 Sep 2015 20:09:22,635 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:143) - Persistence error (The /user/root/def_serv/def_servpath/reading1.1_reading directory could not be created in HDFS. HttpFS response: 503 Service unavailable)
02 Sep 2015 20:09:22,635 WARN [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:163) - The event TTL has expired, it is no more re-injected in the channel (id=2020008711, ttl=0)
02 Sep 2015 20:09:22,635 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:193) - Finishing transaction (1441217314-956-0000000000)
So you can see it's trying to use HttpFS, as it logs the response:
HttpFS response: 503 Service unavailable
...on each writing try.
How should I configure the agent to use WebHDFS?
Thank you
I don't know what was happening, but the configuration mentioned is correct and is working now.
After several tries at rebooting the instance, rewriting the config files and other log errors than the one mentioned, it worked.
At some point Cygnus was trying to write to localhost:50075, instead of {localHDFS.ip}:50070, but that was gone after rebooting cygnus.
All instances are at their latest version (important).
Cygnus configuration for WebHDFS is just about setting the port to 50070, nothing else is required.
Regarding the connections you mention to 50075, they are correct as well, since that's the behaviour of WebHDFS: when you want to upload data to HDFS, first the client (in this case, Cygnus) accesses the Namenode through TCP/50070 port, then the namenode responds with a redirection location pointing to the datanode where the data will be effectively uploaded; such a redirection uses the TCP/50075 port, and thus that datanode:50075 must be accessible by the client (Cygnus). That's why we are using HttpFS in the global instance of Cosmos at FIWARE Lab: HttpFS works as a gateway hiding the details of the datanodes, and a single entry point and port (14000) is required.

Resources