json parse error in ruby - unexpected token at - ruby

I get errors at a lot of places when trying to retrieve ticker symbols for US companies from http://d.yimg.com/autoc.finance.yahoo.com/autoc?callback=YAHOO.Finance.SymbolSuggest.ssCallback&query=Wal-Mart
I have tried to:
resp = Net::HTTP.get_response(URI.parse(url))
data = resp.body
qwe = data.split("symbol")
p qwe[1]
arr1 = data.split("(")
arr2 = arr1[1].split(")")
fnl = arr2[0].gsub(/-/, '')
fnl = fnl.gsub(/\(/, '')
fnl = fnl.gsub(/\)/, '')
fnl = fnl.gsub(/\./, '')
fnl = fnl.gsub('\'', '"')
fnl = fnl.gsub(/([\{|\,}])\s*([a-zA-Z]+):/, '\1 "\2":')
But this doesnt help as i see:
/Library/Ruby/Gems/1.8/gems/json-1.2.0/lib/json/common.rb:123:in `parse': 353: unexpected token at '{"symbol":"BEEV","name": "BENCHMARK ENERGY CORP ' (JSON::ParserError)
Any clues as to what i might be doing wrong?

I don't know why you're doing all those replacements. It works fine once you strip the function call:
>>> pprint.pprint(json.loads(t[39:-1]))
{'ResultSet': {'Query': 'wal-mart',
'Result': [{'exch': 'NYQ',
'exchDisp': 'NYSE',
'name': 'Wal-Mart Stores Inc.',
'symbol': 'WMT',
'type': 'S'},
{'exch': 'MEX',
'exchDisp': 'Mexico',
'name': 'WAL-MART-V',
'symbol': 'WALMEXV.MX',
'type': 'S'},
{'exch': 'TLX',
'name': 'WAL-MART STORES',
'symbol': '984101.TI',
'type': 'S'},
{'exch': 'HAM',
'exchDisp': 'Hamburg',
'name': 'WAL-MART STORES',
'symbol': 'WMT.HM',
'type': 'S'},
{'exch': 'STU',
'exchDisp': 'Stuttgart',
'name': 'WAL-MART-V',
'symbol': '4GN.SG',
'type': 'S'},
{'exch': 'FRA',
'exchDisp': 'Frankfurt',
'name': 'WAL-MART STORES',
'symbol': 'WMT.F',
'type': 'S'},
{'exch': 'FRA',
'exchDisp': 'Frankfurt',
'name': 'WAL-MART-V',
'symbol': '4GN.F',
'type': 'S'},
{'exch': 'BER',
'exchDisp': 'Berlin',
'name': 'WAL-MART STORES',
'symbol': 'WMT.BE',
'type': 'S'},
{'exch': 'STU',
'exchDisp': 'Stuttgart',
'name': 'WAL-MART STORES',
'symbol': 'WMT.SG',
'type': 'S'},
{'exch': 'BUE',
'exchDisp': 'Buenos Aires',
'name': 'WAL-MART STORES INC 2',
'symbol': 'DWMT2.BA',
'type': 'S'}]}}

Related

Inserting data to elasticsearch according to mapping

I have created a mapping:
request_body = {
'settings': {
'number_of_shards': 3,
'number_of_replicas': 2
},
'mappings': {
'dynamic': 'strict',
'properties': {
'content': {'index': True, 'type': 'text'},
'location': {'type': 'geo_point'},
'retweet_count': {'type': 'integer'},
'favorite_count': {'type': 'integer'},
'happened_at': {'type': 'date'},
'author': {
'properties': {
'id': {'type': 'integer'},
'screen_name': {'type': 'text'},
'name': {'type': 'text'},
'description': {'index': True, 'type': 'text'},
'followers_count': {'type': 'integer'},
'friends_count': {'type': 'integer'},
'statuses_count': {'type': 'integer'}
}
},
'country': {
'properties': {
'id': {'type': 'integer'},
'code': {'type': 'text'},
'name': {'type': 'text'}
}
},
'parent_id': {'type': 'integer'},
'mentions': {
'type': 'nested',
'properties': {
'id': {'type': 'integer'},
'screen_name': {'type': 'text'},
'name': {'type': 'text'}
}
},
'hashtags': {
'type': 'nested',
'properties': {
'id': {'type': 'integer'},
'value': {'type': 'text'}
}
}
}
}
}
and then I create an index: es.indices.create(index='pdt', body=request_body), where es is instance of elasticsearch in python.
Then I want to insert a dictionary using bulk insert, the dictionary:
'_id' = {str} '1290712423939346432'
'_source' = {dict}
'content' = {str} 'RT #drfahrettinkoca: YENİ HASTA SAYISINDAKİ YÜKSELME CİDDİ. İki gün arasındaki fark, yakın zamanda ilk kez bu kadar belirgin. Bayram ve tat…'
'location' = {NoneType} None
'retweet_count' = {int} 9556
'favorite_count' = {int} 0
'happened_at' = {str} '2020-08-04 20:13:16+02:00'
'author' = {dict}
'parent_id' = {str} '1290699635003133953'
'mentions' = {list}
'hashtags' = {list}
The only item that is not present is the country, which can be None. I tried with both country = None and
'country': {
'id': None,
'code': None,
'name': None,
}
So I am not sure where the problem could be. Is passing an empty list a problem?
Here is the error code:
('1 document(s) failed to index.', [{'index': {'_index': 'pdt', '_type': 'tweet', '_id': '1290712423939346432', 'status': 400, 'error': {'type': 'illegal_argument_exception', 'reason': 'cannot change object mapping from nested to non-nested'}, 'data': {'content': 'RT #drfahrettinkoca: YENİ HASTA SAYISINDAKİ YÜKSELME CİDDİ. İki gün arasındaki fark, yakın zamanda ilk kez bu kadar belirgin. Bayram ve tat…', 'location': None, 'retweet_count': 9556, 'favorite_count': 0, 'happened_at': '2020-08-04 20:13:16+02:00', 'author': {'id': 794067571226066944, 'screen_name': 'zonguldak_ism', 'name': 'Zonguldak İSM', 'description': 'Adres:Yayla Mah. Ömer Karahasan Sok. No:6 Merkez/ZONGULDAK Telefon:+90 372 253 4607 Fax:+90 372 253 0667', 'followers_count': 703, 'friends_count': 91, 'statuses_count': 1446}, 'parent_id': '1290699635003133953', 'mentions': [{'id': 3679748, 'screen_name': 'drfahrettinkoca', 'name': 'Dr. Fahrettin Koca'}], 'hashtags': []}}}])

What is causing null pointer exception error for data transfer from Oracle to Bigquery using Apache Nifi?

I am currently working with Apache Nifi to transfer data from Oracle to Bigquery.
I have successfully transferred multiple tables from Oracle to Bigquery. One table however is showing an error 'Null pointer exception'.
Null pointer exception usually shows if the Oracle record has null value and the Bigquery table schema does not have 'mode' detail in the json.
I have made sure to get mode and verified if the schema has 'mode', it is still showing the error.
I tried to search here for fix, but I am on the latest version of Apache Nifi (1.10). I have also noticed that the Jobs are not created in Bigquery. Is this problem related to thread? Or is it something else entirely?
Log:
020-01-02 19:12:46,022 ERROR [Timer-Driven Process Thread-13] o.a.n.p.gcp.bigquery.PutBigQueryBatch PutBigQueryBatch[id=bbdb10dd-3194-17d3-a9ab-4e1fcc71bd96] null: java.lang.NullPointerException
java.lang.NullPointerException: null
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:890)
at com.google.cloud.bigquery.Field.<init>(Field.java:199)
at com.google.cloud.bigquery.Field.<init>(Field.java:35)
at com.google.cloud.bigquery.Field$Builder.build(Field.java:193)
at org.apache.nifi.processors.gcp.bigquery.BigQueryUtils.mapToField(BigQueryUtils.java:62)
at org.apache.nifi.processors.gcp.bigquery.BigQueryUtils.listToFields(BigQueryUtils.java:68)
at org.apache.nifi.processors.gcp.bigquery.BigQueryUtils.schemaFromString(BigQueryUtils.java:80)
at org.apache.nifi.processors.gcp.bigquery.PutBigQueryBatch.onTrigger(PutBigQueryBatch.java:288)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Bigquery schema:
[{'mode': 'NULLABLE', #I tried both 'NULLABLE' and 'REQUIRED'
'name': 'ASPC_CUTOFFHISTORYID',
'type': 'STRING'},
{'mode': 'NULLABLE',
'name': 'CDOTYPEID',
'type': 'NUMERIC'},
{'mode': 'NULLABLE',
'name': 'CHANGECOUNT',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'CHANGEHISTORYID',
'type': 'STRING'},
{'mode': 'NULLABLE',
'name': 'DESCRIPTION',
'type': 'STRING'},
{'mode': 'NULLABLE',
'name': 'ICONID',
'type': 'STRING'},
{'mode': 'NULLABLE',
'name': 'ISFROZEN',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'ASPC_CUTOFFHISTORYNAME',
'type': 'STRING'},
{'mode': 'NULLABLE',
'name': 'FACTORYID',
'type': 'STRING'},
{'mode': 'NULLABLE',
'name': 'CUTOFFMONTH',
'type': 'STRING'},
{'mode': 'NULLABLE',
'name': 'PRODUCTIONLINEID',
'type': 'STRING'},
{'mode': 'NULLABLE',
'name': 'OPERATIONID',
'type': 'STRING'},
{'mode': 'NULLABLE',
'name': 'ASPC_INSPECTIONITEMID',
'type': 'STRING'},
{'mode': 'NULLABLE',
'name': 'RESOURCEID',
'type': 'STRING'},
{'mode': 'NULLABLE',
'name': 'ASPC_QA_CONDITIONID',
'type': 'STRING'},
{'mode': 'NULLABLE',
'name': 'ASPC_CHARTTYPEID',
'type': 'STRING'},
{'mode': 'NULLABLE',
'name': 'SIGMATYPE',
'type': 'STRING'},
{'mode': 'NULLABLE',
'name': 'MINOFXBAR',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'MAXOFXBAR',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'MINOFZBAR',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'CLOFR',
'type': 'FLOAT'},
{'mode': 'NULLABLE',
'name': 'STDVALUE',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'CPKVALUE',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'AVGOFXBAR',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'SIOFXBAR',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'OUTLIEROFXBAR',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'RUNOFXBAR',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'TRENDOFXBAR',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'UCLOFXBAR',
'type': 'FLOAT'},
{'mode': 'NULLABLE',
'name': 'CLOFXBAR',
'type': 'FLOAT'},
{'mode': 'NULLABLE',
'name': 'LCLOFXBAR',
'type': 'FLOAT'},
{'mode': 'NULLABLE',
'name': 'TOTALPOINTOFXBAR',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'TOTALOCAPPOINTXBAR',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'MAXOFZBAR',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'AVGOFZBAR',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'MINOFR',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'MAXOFR',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'AVGOFR',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'SIOFR',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'OUTLIEROFR',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'RUNOFR',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'TRENDOFR',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'UCLOFR',
'type': 'FLOAT'},
{'mode': 'NULLABLE',
'name': 'LCLOFR',
'type': 'FLOAT'},
{'mode': 'NULLABLE',
'name': 'TOTALPOINTOFR',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'TOTALOCAPPOINTOFR',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'MINOFS',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'MAXOFS',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'AVGOFS',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'AVGVALUE',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'ISDISABLED',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'CHGCONTROLLINE',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'MINOFW',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'MAXOFW',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'AVGOFW',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'APPLYTODATE',
'type': 'DATE'},
{'mode': 'NULLABLE',
'name': 'AVGOFMIN',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'CLOFMIN',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'LCLOFMIN',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'MAXOFMIN',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'MINOFMIN',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'OUTLIEROFMIN',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'RUNOFMIN',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'SIOFMIN',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'TOTALOCAPPOINTOFMIN',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'TOTALPOINTOFMIN',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'TRENDOFMIN',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'UCLOFMIN',
'type': 'INTEGER'},
{'mode': 'NULLABLE',
'name': 'NOTES',
'type': 'STRING'}]
Oracle:
ASPC_CUTOFFHISTORYID NOT NULL CHAR(16)
CDOTYPEID NUMBER(10)
CHANGECOUNT NUMBER(10)
CHANGEHISTORYID CHAR(16)
DESCRIPTION VARCHAR2(255)
ICONID NUMBER(10)
ISFROZEN NUMBER(10)
ASPC_CUTOFFHISTORYNAME VARCHAR2(50)
FACTORYID CHAR(16)
CUTOFFMONTH VARCHAR2(10)
PRODUCTIONLINEID CHAR(16)
OPERATIONID CHAR(16)
ASPC_INSPECTIONITEMID CHAR(16)
RESOURCEID CHAR(16)
ASPC_QA_CONDITIONID CHAR(16)
ASPC_CHARTTYPEID CHAR(16)
SIGMATYPE VARCHAR2(2)
MINOFXBAR NUMBER
MAXOFXBAR NUMBER
MINOFZBAR NUMBER
CLOFR NUMBER
STDVALUE NUMBER
CPKVALUE NUMBER
AVGOFXBAR NUMBER
SIOFXBAR NUMBER
OUTLIEROFXBAR NUMBER
RUNOFXBAR NUMBER
TRENDOFXBAR NUMBER
UCLOFXBAR NUMBER
CLOFXBAR NUMBER
LCLOFXBAR NUMBER
TOTALPOINTOFXBAR NUMBER
TOTALOCAPPOINTXBAR NUMBER
MAXOFZBAR NUMBER
AVGOFZBAR NUMBER
MINOFR NUMBER
MAXOFR NUMBER
AVGOFR NUMBER
SIOFR NUMBER
OUTLIEROFR NUMBER
RUNOFR NUMBER
TRENDOFR NUMBER
UCLOFR NUMBER
LCLOFR NUMBER
TOTALPOINTOFR NUMBER
TOTALOCAPPOINTOFR NUMBER
MINOFS NUMBER
MAXOFS NUMBER
AVGOFS NUMBER
AVGVALUE NUMBER
ISDISABLED NUMBER(10)
CHGCONTROLLINE NUMBER(10)
MINOFW NUMBER
MAXOFW NUMBER
AVGOFW NUMBER
APPLYTODATE DATE
AVGOFMIN NUMBER
CLOFMIN NUMBER
LCLOFMIN NUMBER
MAXOFMIN NUMBER
MINOFMIN NUMBER
OUTLIEROFMIN NUMBER
RUNOFMIN NUMBER
SIOFMIN NUMBER
TOTALOCAPPOINTOFMIN NUMBER
TOTALPOINTOFMIN NUMBER
TRENDOFMIN NUMBER
UCLOFMIN NUMBER
NOTES VARCHAR2(2000)
I see that you have a DATE field in your BigQuery schema.
I do not know exactly how you are doing this migration but have you tried setting "Use Logical Types" true in your NiFi processor?
EDIT 1.
I replicated your error here and I could solve it by using a schema like this: [ {"mode": "NULLABLE","name": "f0", "type":"STRING"}, {"mode": "NULLABLE", "name": "f1", "type":"INTEGER"}, {"mode": "NULLABLE","name": "f2", "type":"INTEGER"}, {"mode": "NULLABLE","name": "f3", "type":"INTEGER"} ]
Have you tried using double quotes in your schema? Please let me know if it works

How to get data from multiple interactive blocks all at once?

I am creating a slack app where people can schedule when they are on call. I use two date pickers to specify the start date and the end date, and a button to submit the data. What I can't figure out is if the date picker selected_date data can be sent along with the button press, as only initial_date is shown in the block JSON.
Here is the response I get from pressing the button:
[{'action_id': 'hjy',
'action_ts': '1563202745.027278',
'block_id': 'uPBVi',
'text': {'emoji': True, 'text': 'Submit', 'type': 'plain_text'},
'type': 'button',
'value': 'yes'}],
'api_app_id': 'AKL2KTU75',
'channel': {'id': 'CKVM4FJM6', 'name': 'general'},
'container': {'channel_id': 'CKVM4FJM6',
'is_ephemeral': False,
'message_ts': '1563200576.002500',
'type': 'message'},
'message': {'blocks': [{'block_id': 'tgnS',
'text': {'text': 'Hello+<#UKVM4F04Q>!+select+the+days+you+will+be+available+through+below',
'type': 'mrkdwn',
'verbatim': False},
'type': 'section'},
{'block_id': 'bSu1B', 'type': 'divider'},
{'accessory': {'action_id': '5k6C',
'initial_date': '2019-7-15',
'placeholder': {'emoji': True,
'text': 'Select+a+date',
'type': 'plain_text'},
'type': 'datepicker'},
'block_id': 'CcW',
'text': {'text': 'Start+Date',
'type': 'mrkdwn',
'verbatim': False},
'type': 'section'},
{'accessory': {'action_id': 'D5UB',
'initial_date': '2019-7-15',
'placeholder': {'emoji': True,
'text': 'Select+a+date',
'type': 'plain_text'},
'type': 'datepicker'},
'block_id': 'Rpo5',
'text': {'text': 'End+Date',
'type': 'mrkdwn',
'verbatim': False},
'type': 'section'},
{'block_id': 'uPBVi',
'elements': [{'action_id': 'hjy',
'text': {'emoji': True,
'text': 'Submit',
'type': 'plain_text'},
'type': 'button',
'value': 'yes'},
{'action_id': '6t/g',
'text': {'emoji': True,
'text': 'Cancel',
'type': 'plain_text'},
'type': 'button',
'value': 'no'}],
'type': 'actions'}],
'bot_id': 'BKZJLQGG6',
'subtype': 'bot_message',
'text': "This+content+can't+be+displayed.",
'ts': '1563200576.002500',
'type': 'message',
'username': 'pythonapp'},
'response_url': 'https://hooks.slack.com/actions/TKV776BD2/688472980689/5Y52XBs2bUqdnOlwWLq0vWDp',
'team': {'domain': 'testing-o4a6006', 'id': 'TKV776BD2'},
'token': '[leaving out]',
'trigger_id': '694351571380.675245215444.f509bc26fc7ac4c230627e6b32fca516',
'type': 'block_actions',
'user': {'id': 'UKVM4F04Q',
'name': 'noahbraunfeld',
'team_id': 'TKV776BD2',
'username': 'noahbraunfeld'}}
And here is the response I get from the datepickers:
{'actions': [{'action_id': '5k6C',
'action_ts': '1563201722.414594',
'block_id': 'CcW',
'initial_date': '2019-7-15',
'selected_date': '2019-07-14',
'type': 'datepicker'}],
'api_app_id': 'AKL2KTU75',
'channel': {'id': 'CKVM4FJM6', 'name': 'general'},
'container': {'channel_id': 'CKVM4FJM6',
'is_ephemeral': False,
'message_ts': '1563200576.002500',
'type': 'message'},
'message': {'blocks': [{'block_id': 'tgnS',
'text': {'text': 'Hello+<#UKVM4F04Q>!+select+the+days+you+will+be+available+through+below',
'type': 'mrkdwn',
'verbatim': False},
'type': 'section'},
{'block_id': 'bSu1B', 'type': 'divider'},
{'accessory': {'action_id': '5k6C',
'initial_date': '2019-7-15',
'placeholder': {'emoji': True,
'text': 'Select+a+date',
'type': 'plain_text'},
'type': 'datepicker'},
'block_id': 'CcW',
'text': {'text': 'Start+Date',
'type': 'mrkdwn',
'verbatim': False},
'type': 'section'},
{'accessory': {'action_id': 'D5UB',
'initial_date': '2019-7-15',
'placeholder': {'emoji': True,
'text': 'Select+a+date',
'type': 'plain_text'},
'type': 'datepicker'},
'block_id': 'Rpo5',
'text': {'text': 'End+Date',
'type': 'mrkdwn',
'verbatim': False},
'type': 'section'},
{'block_id': 'uPBVi',
'elements': [{'action_id': 'hjy',
'text': {'emoji': True,
'text': 'Submit',
'type': 'plain_text'},
'type': 'button',
'value': 'yes'},
{'action_id': '6t/g',
'text': {'emoji': True,
'text': 'Cancel',
'type': 'plain_text'},
'type': 'button',
'value': 'no'}],
'type': 'actions'}],
'bot_id': 'BKZJLQGG6',
'subtype': 'bot_message',
'text': "This+content+can't+be+displayed.",
'ts': '1563200576.002500',
'type': 'message',
'username': 'pythonapp'},
'response_url': '[leaving out]',
'team': {'domain': 'testing-o4a6006', 'id': 'TKV776BD2'},
'token': '[leaving out]',
'trigger_id': '696646850567.675245215444.317894098ee3aa6938c5c40d4107fb68',
'type': 'block_actions',
'user': {'id': '[leaving out]',
'name': 'noahbraunfeld',
'team_id': 'TKV776BD2',
'username': 'noahbraunfeld'}}

How to append bot's reply into web-chat?

I am unable to get the response from the bot and append it to the web-chat provided by the MS Botframework.
Here's what I have done so far:
Created a bot on https://dev.botframework.com/ (without migration)
Integrated Web Chat and Direct Line
Created a flask listener server and generated HTTPS using ngrok
Gave created server's address as messaging endpoint for the bot
Generated a HTML page using below code (running locally without any server)
<html>
<head>
<link href="https://cdn.botframework.com/botframework-webchat/latest/botchat.css" rel="stylesheet" />
</head>
<body>
<div id="bot"/>
<script src="https://cdn.botframework.com/botframework-webchat/latest/botchat.js"></script>
<script>
BotChat.App({
directLine: { secret: 'direct_line_secret_key' },
user: { id: 'userid' },
bot: { id: 'botid' },
resize: 'detect'
}, document.getElementById("bot"));
</script>
</body>
</html>
Now I am able to send message from the UI as user, and able to capture it in the flask listener server.
But how do I reply from the server so that message will come from bot and append to the UI?
Am I missing something?
I have tried https://directline.botframework.com/v3/directline/conversations/{convoId}/activities to POST a request but that again comes back to server only and does not append to web-chat.
data = json.dumps({"type": "message","from":{"id":"botid"},"text": "Hii!"})
requests.post('https://directline.botframework.com/v3/directline/conversations/' + r['conversation']['id'] + '/activities',
headers={"Authorization": "Bearer " + "secret_key", "Content-Type": "application/json", "Content-Length": "512"},data=data
NOTE: I am using botframework only for the web-chat UI and nothing else.
EDIT 1:
sent this from web-chat ui
http://bcebb07a.ngrok.io/webhook' [POST]>
************************************************************************** {'type': 'message', 'id': '5DvIa5ImiPF4G6WnGlPYyY|0000002',
'timestamp': '2018-06-05T06:51:45.3174659Z', 'serviceUrl':
'https://directline.botframework.com/', 'channelId': 'directline',
'from': {'id': 'sid'}, 'conversation': {'id':
'5DvIa5ImiPF4G6WnGlPYyY'}, 'recipient': {'id':
'one_assist#CrbpWod1mw8', 'name': 'OneAssist'}, 'textFormat': 'plain',
'locale': 'en-US', 'text': 'hello how are you', 'channelData':
{'clientActivityId': '1528180308547.7414264322396316.5'}}
5DvIa5ImiPF4G6WnGlPYyY
#################################################################### {'messages': [{'id':
'5DvIa5ImiPF4G6WnGlPYyY|0000000', 'conversationId':
'5DvIa5ImiPF4G6WnGlPYyY', 'created': '2018-06-05T06:32:50.9651813Z',
'from': 'sid', 'text': 'hello', 'channelData': {'clientActivityId':
'1528180308547.7414264322396316.0'}, 'images': [], 'attachments': []},
{'id': '5DvIa5ImiPF4G6WnGlPYyY|0000001', 'conversationId':
'5DvIa5ImiPF4G6WnGlPYyY', 'created': '2018-06-05T06:47:14.1602925Z',
'from': 'sid', 'text': 'how are you', 'channelData':
{'clientActivityId': '1528180308547.7414264322396316.3'}, 'images':
[], 'attachments': []}, {'id': '5DvIa5ImiPF4G6WnGlPYyY|0000002',
'conversationId': '5DvIa5ImiPF4G6WnGlPYyY', 'created':
'2018-06-05T06:51:45.3174659Z', 'from': 'sid', 'text': 'hello how are
you', 'channelData': {'clientActivityId':
'1528180308547.7414264322396316.5'}, 'images': [], 'attachments':
[]}], 'watermark': '2'}
{ "error": {
"code": "BotError",
"message": "Failed to send activity: bot timed out" }, "httpStatusCode": 504 }
127.0.0.1 - - [05/Jun/2018 12:22:12] "POST /webhook HTTP/1.1" 200 -
************************************************************************** http://bcebb07a.ngrok.io/webhook' [POST]>
************************************************************************** {'type': 'conversationUpdate', 'id': 'D3XJ6CAaVsc', 'timestamp':
'2018-06-05T06:51:56.7508828Z', 'serviceUrl':
'https://directline.botframework.com/', 'channelId': 'directline',
'from': {'id': '5DvIa5ImiPF4G6WnGlPYyY'}, 'conversation': {'id':
'5DvIa5ImiPF4G6WnGlPYyY'}, 'recipient': {'id':
'one_assist#CrbpWod1mw8', 'name': 'OneAssist'}, 'membersAdded':
[{'id': 'botid'}]} 5DvIa5ImiPF4G6WnGlPYyY
#################################################################### {'messages': [{'id':
'5DvIa5ImiPF4G6WnGlPYyY|0000000', 'conversationId':
'5DvIa5ImiPF4G6WnGlPYyY', 'created': '2018-06-05T06:32:50.9651813Z',
'from': 'sid', 'text': 'hello', 'channelData': {'clientActivityId':
'1528180308547.7414264322396316.0'}, 'images': [], 'attachments': []},
{'id': '5DvIa5ImiPF4G6WnGlPYyY|0000001', 'conversationId':
'5DvIa5ImiPF4G6WnGlPYyY', 'created': '2018-06-05T06:47:14.1602925Z',
'from': 'sid', 'text': 'how are you', 'channelData':
{'clientActivityId': '1528180308547.7414264322396316.3'}, 'images':
[], 'attachments': []}, {'id': '5DvIa5ImiPF4G6WnGlPYyY|0000002',
'conversationId': '5DvIa5ImiPF4G6WnGlPYyY', 'created':
'2018-06-05T06:51:45.3174659Z', 'from': 'sid', 'text': 'hello how are
you', 'channelData': {'clientActivityId':
'1528180308547.7414264322396316.5'}, 'images': [], 'attachments':
[]}], 'watermark': '2'}
{ "error": {
"code": "BotError",
"message": "Failed to send activity: bot timed out" }, "httpStatusCode": 504 }
127.0.0.1 - - [05/Jun/2018 12:22:32] "POST /webhook HTTP/1.1" 200 -
# sent this from the server code
************************************************************************** http://bcebb07a.ngrok.io/webhook' [POST]>
************************************************************************** {'type': 'message', 'id': '5DvIa5ImiPF4G6WnGlPYyY|0000003',
'timestamp': '2018-06-05T06:52:16.6925987Z', 'serviceUrl':
'https://directline.botframework.com/', 'channelId': 'directline',
'from': {'id': 'botid'}, 'conversation': {'id':
'5DvIa5ImiPF4G6WnGlPYyY'}, 'recipient': {'id':
'one_assist#CrbpWod1mw8', 'name': 'OneAssist'}, 'text': 'Hii!'}
5DvIa5ImiPF4G6WnGlPYyY
#################################################################### {'messages': [{'id':
'5DvIa5ImiPF4G6WnGlPYyY|0000000', 'conversationId':
'5DvIa5ImiPF4G6WnGlPYyY', 'created': '2018-06-05T06:32:50.9651813Z',
'from': 'sid', 'text': 'hello', 'channelData': {'clientActivityId':
'1528180308547.7414264322396316.0'}, 'images': [], 'attachments': []},
{'id': '5DvIa5ImiPF4G6WnGlPYyY|0000001', 'conversationId':
'5DvIa5ImiPF4G6WnGlPYyY', 'created': '2018-06-05T06:47:14.1602925Z',
'from': 'sid', 'text': 'how are you', 'channelData':
{'clientActivityId': '1528180308547.7414264322396316.3'}, 'images':
[], 'attachments': []}, {'id': '5DvIa5ImiPF4G6WnGlPYyY|0000002',
'conversationId': '5DvIa5ImiPF4G6WnGlPYyY', 'created':
'2018-06-05T06:51:45.3174659Z', 'from': 'sid', 'text': 'hello how are
you', 'channelData': {'clientActivityId':
'1528180308547.7414264322396316.5'}, 'images': [], 'attachments': []},
{'id': '5DvIa5ImiPF4G6WnGlPYyY|0000003', 'conversationId':
'5DvIa5ImiPF4G6WnGlPYyY', 'created': '2018-06-05T06:52:16.6925987Z',
'from': 'botid', 'text': 'Hii!', 'images': [], 'attachments': []}],
'watermark': '3'}
https://directline.botframework.com/v3/directline/conversations/5DvIa5ImiPF4G6WnGlPYyY/activities/
https://directline.botframework.com/v3/conversations/5DvIa5ImiPF4G6WnGlPYyY/activities
{ "error": {
"code": "BotError",
"message": "Failed to send activity: bot timed out" }, "httpStatusCode": 504 }
I had a similar issue while using ngrok and I notice in your logs there are references to 127.0.0.1. Try adding the -host-header parameter to your ngrok command line.
For example I use
ngrok http -host-header=localhost:55486 55486
Where 55486 is the port number my endpoint sits on.

Separator not showing in Adaptive Cards

I've been trying to get the "separator" property of Adaptive cards to work, but it does not seem to be rendering in the BotFramework Emulator.
Here are images to the Emulator and the Visualizer for the same code: Emulator Visualizer
The code in both places is the same, and is as follows:
{
"contentType": "application/vnd.microsoft.card.adaptive",
"content": {
'$schema': 'http://adaptivecards.io/schemas/adaptive-card.json',
'version': '1.0',
'type': 'AdaptiveCard',
'body': [
{
'type': 'TextBlock',
'text': 'Meeting Title',
'weight': 'bolder'
},
{
'type': 'TextBlock',
'text': 'Location',
'separator': true,
'isSubtle': true,
'size': 'small'
},
{
'type': 'TextBlock',
'text': 'Location',
'spacing': 'none'
},
{
'type': 'TextBlock',
'text': 'Organizer',
'separator': true,
'isSubtle': true,
'size': 'small'
},
{
'type': 'TextBlock',
'text': 'Organizer Name',
'spacing': 'none'
},
{
'type': 'TextBlock',
'text': 'Start Time',
'separator': true,
'isSubtle': true,
'size': 'small'
},
{
'type': 'ColumnSet',
'spacing': 'none',
'columns': [
{
'type': 'Column',
'width': 'auto',
'items': [
{
'type': 'TextBlock',
'text': '05:00 PM',
'isSubtle': false,
'weight': 'bolder'
}
]
},
{
'type': 'Column',
'width': 'auto',
'items': [
{
'type': 'TextBlock',
'text': 'May 21'
}
]
},
{
'type': 'Column',
'width': 'auto',
'items': [
{
'type': 'TextBlock',
'text': '2017',
'isSubtle': true,
'weight': 'bolder'
}
]
}
]
},
{
'type': 'TextBlock',
'text': 'End Time',
'separator': true,
'isSubtle': true,
'size': 'small'
},
{
'type': 'ColumnSet',
'spacing': 'none',
'columns': [
{
'type': 'Column',
'width': 'auto',
'items': [
{
'type': 'TextBlock',
'text': '05:30 PM',
'isSubtle': false,
'weight': 'bolder'
}
]
},
{
'type': 'Column',
'width': 'auto',
'items': [
{
'type': 'TextBlock',
'text': 'May 21'
}
]
},
{
'type': 'Column',
'width': 'auto',
'items': [
{
'type': 'TextBlock',
'text': '2017',
'isSubtle': true,
'weight': 'bolder'
}
]
}
]
}
],
'actions': [
{
'type': 'Action.Submit',
'title': 'Accept',
'data':{
'accept': true
}
},
{
'type': 'Action.Submit',
'title': 'Decline',
'data':{
'accept': false
}
}
]
}
}
As seen, The separators appear only in the visualizer, for the same code. Am I missing something?
Instead of separator you can use
Separation = SeparationStyle.Strong
this is working for me
it might be a little bit tricky because the separator documentation is a little bit vague (at least for me).
Take a look here - the syntax purposed to separator doesn't seem to work anywhere.
What I did find:
The spacing property works just fine (at least with these values "none" | "small" | "default" | "medium" | "large" | "extraLarge" | "padding")
It will only works when using in containers (Container, ColumnSet, Column, etc.)
It is also apply (as the name implies) on the outer area of the container (similar to CSS margin property)
Will not work for the first container
You may go to the adaptive cards classic editor and put spacing (let's say "spacing": "large") on the second Container section (there are only 2 of them) and observe the spacing impact by yourself

Resources