How to handle user leaving conversation - botframework

We have welcome examples using OnMembersAddedAsync method but no examples showing how to handle user leaving conversation. I tried to override OnMembersRemovedAsync but it does not seem to be invoked (at least when I use bot framework emulator).
I need to do some cleanup at the event of user leaving/left conversation.
An example or any tips would be appreciated.
Update: I'm using C# and Bot framework v4

This is going to be channel specific as it is dependent on the channel providing a feature that sends an update when the user leaves a conversation. Any other channels, you will need to research.
For Facebook, I was unable to find a scope that covers such an action. These are the available scopes which you can reference more closely here:
messages
message_deliveries
message_echoes
message_reads
messaging_account_linking
messaging_checkout_updates (beta)
messaging_game_plays
messaging_handovers
messaging_optins
messaging_payments(beta)
messaging_policy_enforcement
messaging_postbacks
messaging_pre_checkouts (beta)
messaging_referrals
standby
Web Chat, as a feature, also does not include this. However, given this is a web page, you can utilize the onbeforeunload() window function to dispatch an event. The event listener will make use of Web Chat's store to dispatch either a message or event to the bot. For the sake of clarity, I'm sending different types of data via SEND_MESSAGE and SEND_EVENT.
const store = window.WebChat.createStore( {}, ( { dispatch } ) => next => async action => {
return next( action );
};
window.addEventListener( 'sendEventActivity', ( { data } ) => {
store.dispatch({
type: 'WEB_CHAT/SEND_MESSAGE',
payload: {
text: data
}
} )
,
store.dispatch( {
type: 'WEB_CHAT/SEND_EVENT',
payload: {
name: 'user_event',
value: {
name: 'end_conversation',
value: 'user ended conversation'
},
text: 'The user has left the conversation.'
}
} )
} );
window.onbeforeunload = function() {
const eventSendActivity = new Event( 'sendEventActivity' );
eventSendActivity.data = 'User left conversation';
window.dispatchEvent( eventSendActivity );
}
{ type: 'message',
id: '4uPdpZhlTFfBMziBE7EmEI-f|0000004',
timestamp: 2020-01-10T18:21:26.767Z,
serviceUrl: 'https://directline.botframework.com/',
channelId: 'directline',
from: { id: 'dl_123', name: 'johndoe', role: 'user' },
conversation: { id: '4uPdpZhlTFfBMziBE7EmEI-f' },
recipient: { id: 'botberg#QaeuoeEamLg', name: 'Dungeon Runner' },
textFormat: 'plain',
locale: 'en-US',
text: 'User left conversation',
channelData:
{ clientActivityID: '15786804807910gegwkp2kai',
clientTimestamp: '2020-01-10T18:21:20.792Z' }
}
{ type: 'event',
id: '4uPdpZhlTFfBMziBE7EmEI-f|0000005',
timestamp: 2020-01-10T18:21:26.780Z,
serviceUrl: 'https://directline.botframework.com/',
channelId: 'directline',
from: { id: 'dl_123', name: 'johndoe', role: 'user' },
conversation: { id: '4uPdpZhlTFfBMziBE7EmEI-f' },
recipient: { id: 'botberg#QaeuoeEamLg', name: 'Dungeon Runner' },
locale: 'en-US',
channelData:
{ clientActivityID: '1578680480821h7kgfm9cyz',
clientTimestamp: '2020-01-10T18:21:20.821Z' },
value:
{ name: 'end_conversation', value: 'user ended conversation' },
name: 'user_event'
}
Hope of help!
Update (Aug. 6th, 2021):
As Chrome, and other browsers, disallow blocking / delaying the closing of a window during onbeforeunload(), sending an event using an event handler or listener is usually unreliable, at best. At worst, it just doesn't work.
However, there is another method that does appear work by using 'window.navigator.sendBeacon()' (providing it's supported).
sendBeacon is a low-level, simplified version of fetch that “asynchronously sends a small amount of data over HTTP to a web server”. It only sends as a POST, only takes the URL or URL + data as properties, and doesn’t wait for a response. As the docs state:
The data is sent reliably
It's sent asynchronously
It doesn't impact the loading of the next page
In testing, I have coupled it with a proactive messaging endpoint in my bot and it appears to work perfectly. (My code sample below is in JS, pulled from a project - for reference, here is the proactive messaging C# sample, available in other languages, as well).
When I navigate to another page or close the browser, sendBeacon posts the message to the endpoint creating the proactive message, which, in turn, sends an activity to the bot. When I return to the Web Chat page, the message is now visible in the chat window. Keep in mind, I also have persistence set up in Web Chat allowing me to return to a conversation previously started.
In the below image, I demo loading Web Chat, navigating to my browser’s home page, and then return – the proactive message is now visible in the chat.
(Web Chat) hosting page:
window.onbeforeunload = () => {
let body = { user: { userName: 'McUser', userId: 'abc123' } };
const headers = { type: 'application/json', 'Access-Control-Allow-Origin': '*' };
const blob = new Blob( [ JSON.stringify( body ) ], headers )
navigator.sendBeacon( 'http://localhost:3978/api/notify', blob )
}
Bot's proactive messaging endpoint:
server.post('/api/notify', async (req, res) => {
const userName = req.body.user.userName;
const userId = req.body.user.userId;
for (const conversationReference of Object.values(conversationReferences)) {
await adapter.continueConversation(conversationReference, async (turnContext) => {
await turnContext.sendActivity(`${ userName } (userId: ${ userId }) exited chat.`);
});
}
});

Related

Custom Adapter receives error: [onTurnError] unhandled error: TypeError: bot.reply is not a function

We've created a Custom Botbuilder Adapter to connect to the Vonage API called botbuilder-adapter-vonage-js. To test the Adapter's basic functionality, with a basic bot reply, we send an SMS to the Vonage number and should get an SMS reply back "Hello Back", but instead receive the error below:
[onTurnError] unhandled error: TypeError: bot.reply is not a function
Not sure how to actually debug the Custom Adapter to find where it is broken.
Would be great to find someone familiar with either the Botkit Core library and Botkit Platform Adapters could help with this. I've attached the Express Server (webhook-server.js) below.
// webhook-server.js
require('dotenv').config();
const express = require('express');
const app = express();
const port = 3000;
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
const SendMessagesAPI = require('./Vonage-SEND-messages-api');
const VonageAdapter = require('botbuilder-adapter-vonage-js');
const Botkit = require('botkit');
const {
BotFrameworkAdapter,
InspectionMiddleware,
MemoryStorage,
InspectionState,
UserState,
ConversationState,
} = require('botbuilder');
const { MicrosoftAppCredentials } = require('botframework-connector');
// This bot's main dialog.
const { IntersectionBot } = require('./bot');
const { Message } = require('#vonage/server-sdk');
const creds = {
apiKey: process.env.VONAGE_API_KEY,
apiSecret: process.env.VONAGE_API_SECRET,
applicationId: process.env.VONAGE_APPLICATION_ID,
privateKey: process.env.VONAGE_APPLICATION_PRIVATE_KEY_PATH,
};
const config = {
to_number: process.env.TO_NUMBER,
from_number: process.env.FROM_NUMBER,
// enable_incomplete: true
};
// Create Adapter
const adapter = new VonageAdapter(creds, config);
// Create the Storage provider and the various types of BotState.
const memoryStorage = new MemoryStorage();
const inspectionState = new InspectionState(memoryStorage);
const userState = new UserState(memoryStorage);
const conversationState = new ConversationState(memoryStorage);
// Create and add the InspectionMiddleware to the adapter.
adapter.use(
new InspectionMiddleware(
inspectionState,
userState,
conversationState,
new MicrosoftAppCredentials(
process.env.MicrosoftAppId,
process.env.MicrosoftAppPassword
)
)
);
app.post('/webhooks/dlr', (req, res) => {
res.status(200).end();
});
// Catch-all for errors.
adapter.onTurnError = async (, error) => {
// This check writes out errors to console log .vs. app insights.
// NOTE: In production environment, you should consider logging this to Azure
// application insights. See https://aka.ms/bottelemetry for telemetry
// configuration instructions.
console.error(`\n [onTurnError] unhandled error: ${error}`);
// Send a trace activity, which will be displayed in Bot Framework Emulator
await .sendTraceActivity(
'OnTurnError Trace',
`${error}`,
'https://www.botframework.com/schemas/error',
'TurnError'
);
// Send a message to the user
await .sendActivity('The bot encountered an error or bug.');
await .sendActivity(
'To continue to run this bot, please fix the bot source code.'
);
// Clear out state
await conversationState.clear();
};
// Create the main dialog.
const bot = new IntersectionBot(conversationState, userState);
// Listen for incoming requests.
app.post('/webhooks/inbound', (req, res) => {
console.log('/webhooks/inbound req.body', req.body);
adapter.processActivity(req, res, async () => {
console.log(context);
// [onTurnError] unhandled error: TypeError: Cannot read property 'from' of undefined
// await bot.run();
// [onTurnError] unhandled error: TypeError: .reply is not a function
// await .reply('I heard a message!');
// [onTurnError] unhandled error: TypeError: bot.reply is not a function
await bot.reply('Hello Back!');
});
res.status(200).end();
});
app.post('/webhooks/status', (req, res) => {
res.status(200).end();
});
app.listen(port, () => {
console.log(`🌏 Server running at http://localhost:${port}`);
});
Response
🌏 Server running at http://localhost:3000
/webhooks/inbound req.body {
message_uuid: 'e93a3007-f7a5-436a-8ba7-c46d64343d80',
to: { type: 'sms', number: '12018994297' },
from: { type: 'sms', number: '15754947000' },
timestamp: '2021-08-27T21:14:51.228Z',
usage: { price: '0.0057', currency: 'EUR' },
message: {
content: { type: 'text', text: 'Hello' },
sms: { num_messages: '1' }
},
direction: 'inbound'
}
TurnContext {
_respondedRef: { responded: false },
_turnState: TurnContextStateCollection(2) [Map] {
'httpStatus' => 200,
Symbol(state) => { state: [Object], hash: '{}' }
},
_onSendActivities: [],
_onUpdateActivity: [],
_onDeleteActivity: [],
_turn: 'turn',
_locale: 'locale',
bufferedReplyActivities: [],
_adapter: VonageAdapter {
middleware: MiddlewareSet { middleware: [Array] },
BotIdentityKey: Symbol(BotIdentity),
OAuthScopeKey: Symbol(OAuthScope),
name: 'Vonage Adapter',
middlewares: null,
botkit_worker: [class VonageBotWorker extends BotWorker],
credentials: {
apiKey: '4f2ff535',
apiSecret: 'jtYzPbh3MXr8M1Hr',
applicationId: '978500cf-7ea8-4d7b-ac54-2b42f67b28a2',
privateKey: './private.key'
},
options: {},
to_number: '15754947000',
from_number: '12018994297',
enable_incomplete: undefined,
turnError: [AsyncFunction (anonymous)]
},
_activity: {
id: 'e93a3007-f7a5-436a-8ba7-c46d64343d80',
timestamp: 2021-08-27T21:14:39.573Z,
channelId: 'vonage-sms',
conversation: { id: '15754947000' },
from: { id: '15754947000' },
recipient: { id: '12018994297' },
text: 'Hello',
channelData: {
message_uuid: 'e93a3007-f7a5-436a-8ba7-c46d64343d80',
to: [Object],
from: [Object],
timestamp: '2021-08-27T21:14:51.228Z',
usage: [Object],
message: [Object],
direction: 'inbound'
},
type: 'message'
}
}
[onTurnError] unhandled error: TypeError: bot.reply is not a function
Botkit version:
Messaging Platform: Vonage
Node version: v14.16.1
Os: MAC
It looks like your custom adapter is built using both Botkit and BotFramework similar to other Botkit adapters. However, your bot's implementation aligns more with a bot built only for BotFramework yet you are trying to call the reply() method that belongs to a Botkit bot.
For example, in Botkit's 'botbuilder-adapter-twilio-sms' adapter, you are presented with two ways to utilize the adapter. In the first, under Botkit Basics, an adapter is created which is then consumed by Botkit to form the controller. This then allows you to access the reply() method callable from a Botkit bot.
In the second, under BotBuilder Basics, an adapter is created which is then utilized within the Express server's /api/messages endpoint. Inbound messages are passed to the adapter's processActivity() method where the bot then responds using the sendActivity() method, callable from within a BotFramework adapter's context.
Narrowing down which implementation you intend to use I believe will alleviate the error you are receiving.

Loader icon in Bot Framework Webchat

I am using Bot Framework Webchat. There are few user related data which I am posting using back channel post activity through the store option to greet the user.
<ReactWebChat
activityMiddleware={ activityMiddleware }
directLine={ window.WebChat.createDirectLine( this.state.token ) }
store = {this.handleGetStore()}
styleOptions={styleOptions}
/>
handleGetStore returns the store data:
handleGetStore(){
const store = window.WebChat.createStore({}, ({ dispatch }) => next => action => {
if (action.type === 'DIRECT_LINE/CONNECT_FULFILLED') {
dispatch({
type: 'WEB_CHAT/SEND_EVENT',
payload: {
name: 'userDetail',
value: this.state.userDetail
}
});
}
return next(action);
});
return store;
}
When the connection initiates the loader appears.
After that there is delay of about 3-5 seconds before the welcome message appears and in the meantime the Webchat seems ready for the user.
A slight delay of 3 seconds is acceptable but quite often the delay is upto 10 seconds or more. I understand that this can be slightly improved by using the Always On feature of the App Service and scaling up the plan. Is there a way I can wait for the back channel welcome message to appear and show the loader until then?
Reference: https://github.com/microsoft/BotFramework-WebChat/pull/1866
Unfortunately, the connection status display relies on events received from DirectLineJs and Web Chat does not support customizing its behavior at the moment. That being said, there is a hacky way to accomplish what you're trying to do by dispatching pseudo DirectLine events.
Here are the steps below:
Create a flag that will indicate whether or not the bot has sent a welcome message - received_welcome_message.
When Web Chat dispatches a connection fulfilled event, check the flag
to ensure a welcome message has been received. If the bot has not
sent a welcome message, dispatch the welcome event to the bot and reset the
connection status to fulfilling.
When Web Chat receives an activity
from the bot, check if it is a welcome message. I would recommend
adding a name attribute to message on the bot side to check - await
context.sendActivity({ text: 'Welcome', name: 'welcome'}). If the
activity is a welcome message, dispatch a connection fulfilled event and set the flag to true.
For more details take a look at the code snippets below.
let received_welcome_message = false;
const store = createStore(
{},
({ dispatch}) => next => action => {
if (action.type === 'DIRECT_LINE/CONNECT_FULFILLED') {
if (!received_welcome_message) {
dispatch({
type: 'DIRECT_LINE/CONNECT_FULFILLING'
});
dispatch({
type: 'WEB_CHAT/SEND_EVENT',
payload: { name: 'webchat/join' }
});
return
}
} else if (action.type === 'DIRECT_LINE/INCOMING_ACTIVITY' && action.payload.activity.name === 'welcome') {
received_welcome_message = true;
dispatch({
type: 'DIRECT_LINE/CONNECT_FULFILLED',
});
}
return next(action);
}
);
Edit
A less hacky approach is to dispatch a post activity pending event when the connection to the bot is fulfilled to mimic the bot sending a welcome message. Note, that the bot is unaware of the mimicked activity. See the code snippet below.
const store = createStore(
{},
({ dispatch}) => next => action => {
console.log(action)
if (action.type === 'DIRECT_LINE/CONNECT_FULFILLED') {
dispatch({
type: 'DIRECT_LINE/POST_ACTIVITY_PENDING',
meta: { method: 'keyboard' },
payload: {
activity: {
from: { role: "bot" },
text: "Welcome Message",
textFormat: "plain",
timestamp: new Date().toString(),
type: "message"
}
}
})
}
return next(action);
}
Hope this helps!

Get URL Referer [BotFramework-WebChat]

I'm using a directline inside my website and I was wondering if there is anyway to get the URL of the website inside my bot code. Previously, in v3, I was initializing the chat with:
BotChat.App({
directLine: { secret: "{directline_secret}" },
user: { id: 'You', referrer: window.location.href},
bot: { id: '{bot_id}' },
resize: 'detect'
}, document.getElementById("bot"));
and I was able to get the referrer with this line of code activity.From.Properties["referrer"].ToString(), but in v4 I can't find a way to get the referrer inside the bot.
Can someone help me?
Thanks in advance.
In v4 the value is part of the turnContext.activity (in Node) or turnContext.Activity (in C#) object. Passing the url value as you have done in your question (i.e., as part of the user object) you would access it like so (Node example):
async onTurn(turnContext) {
if (
turnContext.activity.type === "event" && turnContext.activity.name === "eventName"
) {
this.userProfile.location = turnContext.activity.from.referrer;
await console.log(this.userProfile.location);
}
I included a name as well as specified a type in my BotChat.App post to match this event to in the turnContext.activity:
function testMethod(someValue) {
botConnection
.postActivity({
from: { id: 'me', referrer: window.location.href },
name: 'eventName',
type: 'event',
value: someValue
})
.subscribe(function (id) {
console.log('"eventName" sent');
});
};
In this example, the method is tied to a button being pressed on the page.
Hope of help!

BotBuilder backchannel event data into session

TLDR: How can I initialise my conversations with data sent from the backchannel and use that information throughout the conversation with a user?
Using Microsoft botbuilder, a "backchannel" mechanism is provided with which I can send data to and from the bot.
Similar to this answer I am using the backchannel method to send a conversationStarted event to the bot in order to start a conversation as you might expect.
Upon receiving the event, I start a dialog which fires off my proactive messages to the user.
bot.on('event', event => {
if (event.name === 'conversationStarted') {
bot.beginDialog(event.address, determineWhichDialogToStart(event))
}
})
I can then use middleware or event listeners to intercept this event and see its contents. Sure enough I can see the event
{ type: 'event',
name: 'conversationStarted',
text: '',
myMetaProperty: 'foobar',
attachments: [],
entities: [],
address:
{ id: '8f5d3952-df3b-4340-89b4-97360f8d4965',
channelId: 'emulator',
user: { id: '33abc508-86d7-49c1-aa68-00a3491fda12' },
conversation: { id: '4e8a943d-6a45-41f2-aa11-6671cc1ca4f3' },
bot: { id: 'bot' },
serviceUrl: 'http://localhost:3010/' },
source: 'emulator',
agent: 'botbuilder',
user: { id: '33abc508-86d7-49c1-aa68-00a3491fda12' } }
But the events I can listen to to see this event, don't have access to the session because the message hasn't yet been dispatched to one it seems.
If I use the botbuilder middleware, or the routing event, I then see that it's been turned into a message rather than an event, and has lost my meta data passed to the event. (see myMetaProperty)
{ type: 'message',
agent: 'botbuilder',
source: 'emulator',
sourceEvent: {},
address:
{ id: '8f5d3952-df3b-4340-89b4-97360f8d4965',
channelId: 'emulator',
user: { id: '33abc508-86d7-49c1-aa68-00a3491fda12' },
conversation: { id: '4e8a943d-6a45-41f2-aa11-6671cc1ca4f3' },
bot: { id: 'bot' },
serviceUrl: 'http://localhost:3010/' },
text: '',
user: { id: '33abc508-86d7-49c1-aa68-00a3491fda12' } }
I tried getting access to the session in my receive event/middleware per the comment here on Github but whilst I get access to a session, it's not the session.
bot.on('receive', event => {
bot.loadSessionWithoutDispatching(event.address, (error, session) => {
session.conversationData.myMetaProperty = event.myMetaProperty
})
})
This actually ends up starting a new session - having looked into the loadSession/loadSessionWithoutDispatching, they both lead to startSession and therefore me adding data to the session is lost when i try to use it in a dialog
bot.dialog('example', [
(session, args, next) => {
console.log(session.conversationData.myMetaProperty) // nope
console.log(session.cantUseThisEither) // nope
session.send('foo')
next()
}
])
Just to reiterate the question now that there's been a bit of background, how can I initialise my conversations with data sent from the backchannel and use that information throughout the conversation with a user?
I ended up starting a generic dialog which sets up my session and then moves onto the root dialog.
bot.on('event', event => {
if (event.name === 'conversationStarted') {
bot.beginDialog(event.address, 'initialise-conversation', {
myMetaProperty: event.myMetaProperty
});
}
});
bot.dialog('initialise-conversation', (session, args, next) => {
session.conversationData.myMetaProperty = args.myMetaProperty
session.beginDialog('/');
});
Then whenever I need myMetaProperty throughout my conversation I can get it from session.conversationData
--
In response to the comment below, here's some additional information which may help someone.
The shape of the event you receive on the bot must contain an address obviously. For example here's the one I send to my bot.
{ myMetaProperty: 'foo',
name: 'conversationStarted',
type: 'event',
address:
{ id: '4q5aaBw77aNGGvrpkeji8A',
channelId: 'custom-directline',
user: { id: 'c214imy6PCJZY4G1x2AXSD' },
conversation: { id: 'bGvMBjEmFTg3DMxsbvJGjH' },
bot: { id: 'bot' },
serviceUrl: 'https://localhost:3010/' },
source: 'custom-directline',
agent: 'botbuilder',
user: { id: 'c214imy6PCJZY4G1x2AXSD' } }
Start by ensuring that you're sending the address. The key bits of information for routing messages back to an individual are the user and conversation objects.
For example, if you're sending information using the backchannel of the botframework webchat you'd do it as follows.
var user = { id: uuidv4() };
var botConnection = new BotChat.DirectLine({...});
botConnection.postActivity({
type: 'event',
from: user,
name: 'conversationStarted',
myMetaProperty: 'foo',
}).subscribe();
BotChat.App({
user: user,
botConnection: botConnection,
}, document.getElementById("bot"));
This will be picked up by the event listener described above.
Another thing to note is that the emulator (and possibly other clients/channels send a conversationUpdate event which I just passed onto my conversationStarted event when I was using it. Here's the listener to do that for reference in case it's useful.
bot.on('conversationUpdate', message => {
message.membersAdded &&
message.membersAdded
.filter(identity => identity.id === message.address.bot.id)
.forEach(identity => bot.emit('conversationStarted', message));
});
You can pass data via beginDialog(address: IAddress, dialogId: string, dialogArgs?: any, done?: (err: Error) => void): void;, you can pass the data you want to send as the third parameter in this function.
You can refer to the source code at https://github.com/Microsoft/BotBuilder/blob/master/Node/core/src/bots/UniversalBot.ts#L243 for more details.

How can I let users share their location in Bot Framework webchat channel?

I read that you can share location with the backchannel.
I was thinking to use this code to talk directly to the bot:
function postButtonMessage() {
botConnection.postActivity({
entities:{type: "ClientCapabilities", requiresBotState: true, supportsTts: true, supportsListening: true},
from: { id: 'userid', name: 'username' },
name: 'botname',
type: 'message',
value: 'Hi',
textFormat: 'plain'
})
.subscribe(function (id) {
console.log('"buttonClicked" sent');
});
};
But I get an error saying "bad gateway 502", but when I talk through the web channel windows it works perfectly so I know that the direct line key is configured correctly. And when I use the type: event instead of message it works fine and I don't have problems, so I am confused about that.
(Question originally asked at https://github.com/Microsoft/BotFramework-WebChat/issues/778#)
There are two approaches here.
One is to use the backchannel to send an event activity to the bot with whatever data you like. You could do this when a button is clicked, at regular intervals, or whenever the location changes.
var dl = new BotChat.DirectLine({secret});
BotChat.App({
botConnection: dl,
// other Chat props go here
});
function postButtonMessage() {
dl.postActivity({
from: { id: 'userid', name: 'username' },
type: 'event',
name: 'location',
value: { /* location goes here */ }
})
.subscribe(id => {
console.log('"buttonClicked" sent');
});
};
The other is to use client middleware to send that data with every message, by intercepting and modifying each message as it goes out. The advantage to this approach is that each message is 'tagged' with its location. The disadvantage is that you only get location updates when the user sends a message.
var dl = new BotChat.DirectLine({secret});
BotChat.App({
botConnection: {
... dl,
postActivity: activity => dl.postActivity({
... activity,
channelData: { location: /* location goes here */ }
})
},
// other Chat props go here
});
And of course you could do both!

Resources