Why Google Cloud Speech API doesn't transcript the whole audio file? - google-api

I'm trying to transcript a short interview audio file with Google Cloud Speech API (asynchronously) but it only transcribes the first half minute of the recording. I had several attempts with recordings longer than one minute and the results were the same. My question is, how can I achieve the full audio transcription for a given file?
You can find one of my use-cases here:
Upload the audio file:
POST https://speech.googleapis.com/v1beta1/speech:asyncrecognize?key={YOUR_API_KEY}
{
"config": {
"encoding": "LINEAR16",
"sampleRate": 16000,
},
"audio": {
"uri": "gs://protean-blend-146812.appspot.com/record__2017_02_02_12_02_17_greg_16000.wav",
}
}
Got the operation number in the response:
{
"name": "8977932499808116064"
}
Make a request with the operation number:
GET https://speech.googleapis.com/v1beta1/operations/8977932499808116064?key={YOUR_API_KEY}
Got the result:
{
"name": "8977932499808116064",
"metadata": {
"#type": "type.googleapis.com/google.cloud.speech.v1beta1.AsyncRecognizeMetadata",
"progressPercent": 100,
"startTime": "2017-02-02T11:21:41.346784Z",
"lastUpdateTime": "2017-02-02T11:23:03.150491Z"
},
"done": true,
"response": {
"#type": "type.googleapis.com/google.cloud.speech.v1beta1.AsyncRecognizeResponse",
"results": [
{
"alternatives": [
{
"transcript": "McGregor you have any stories about being lost that you have all the good advice well let me know in the Golden Triangle drug trafficking across the border",
"confidence": 0.8535113
}
]
},
{
"alternatives": [
{
"transcript": "we came across this Village very very poor Village People and some of the people there were really unfriendly they just started throwing rocks and my friend and we couldn't talk so we backed away went back quickly up the hill",
"confidence": 0.9027881
}
]
},
{
"alternatives": [
{
"transcript": "and we are wondering you know where to go and luckily I can see in the distance there in one tree",
"confidence": 0.8931573
}
]
}
]
}
}
The links, where I made the requests (at 'Try it!' section):
Upload: https://cloud.google.com/speech/reference/rest/v1beta1/speech/asyncrecognize
Operation response: https://cloud.google.com/speech/reference/rest/v1beta1/operations/get

Related

Search criteria for appointment slots in Epic FHIR

I'm using Epic's FHIR API (with Argonaut) to search for available appointments, in their private sandbox:
POST /api/FHIR/STU3/Appointment/$find
{
"resourceType": "Parameters",
"parameter": [
{
"name": "startTime",
"valueDateTime": "2022-03-22T08:15:00Z"
},
{
"name": "endTime",
"valueDateTime": "2022-04-02T08:15:00Z"
}
]
}
And this gives me some temporary appointments starting at the desired date and time. No problem. However, if I add service-type as a parameter in the body:
POST /api/FHIR/STU3/Appointment/$find
{
"resourceType": "Parameters",
"parameter": [
{
"name": "startTime",
"valueDateTime": "2022-03-23T15:30:00Z"
},
{
"name": "endTime",
"valueDateTime": "2022-04-02T15:30:00Z"
},
{
"name": "service-type",
"valueCodeableConcept": {
"coding": [
{
"system": "urn:oid:1.2.840.114350.1.13.861.1.7.3.808267.11",
"code": "40111223"
}
]
}
}
]
}
I get a slightly different response. I see temporary appointments with the same slot IDs (different appointment IDs of course), but also see an OperationOutcome at the end of the Bundle:
{
"fullUrl": "urn:uuid:00000000-0007-792f-cd9b-f1f44af2c17c",
"resource": {
"resourceType": "OperationOutcome",
"issue": [
{
"severity": "information",
"code": "value",
"details": {
"coding": [
{
"system": "urn:oid:1.2.840.114350.1.13.0.1.7.2.657369",
"code": "59109",
"display": "An element value is invalid."
}
],
"text": "An element value is invalid."
},
"diagnostics": "1.2.840.114350.1.13.861.1.7.3.808267.11",
"location": [
"/f:service-type(oid)"
],
"expression": [
"service-type (oid)"
]
}
]
},
"search": {
"mode": "outcome"
}
}
The service-type system and code I used here were taken right from the example on Epic's page on the $find API. I see similar results when I use service-types from actual Slots in the Epic sandbox. And if I put in a location-reference, it seems to be ignored and appointments at other locations are sometimes returned. The net result of all this is that it seems only the start and end time are honored when finding appointments.
How can I narrow down the results of a $find call using criteria other than the start and end date?
The source of truth for this API should be the fhir.epic.com documentation, but since this is a common question, I'll post some information here:
$find is not a good option for patient open scheduling. I don't know if that is your use case or not, but it is a common question, so I'll address it anyway.
$find invokes a rules engine that is defined by each healthcare organization. The logic of that rules engine is 100% up to the healthcare organization to define. In order for $find to work for you, you'll need very specific pre-coordination to build the logic in that engine.
The Epic sandboxes only have a very basic rules engine built out. It isn't particularly "real-world"-y. And it is probably good that it isn't, as it serves as an early warning that this API may not be the API you are looking for.
The $find API is a way to support cross-organization scheduling. For example, if you want front desk staff to be able to schedule a follow up visit at another organization across town that they have an established business relationship. For example, if a PCP office wants to schedule a dermatology visit for you at an external org. Note that the user in this case is the organization's scheduling staff, not the patient.
Specifically, Epic's $find support is based on Use Case 1 and 2 in the Argonaut Implementation Guide.

Missing elements for a Google Cloud Video Annotation request

I am trying to run annotation on a video using the Google Cloud Video Intelligence API. Annotation requests with just one feature request (i.e., one of "LABEL_DETECTION", "SHOT_CHANGE_DETECTION" or "EXPLICIT_CONTENT_DETECTION"), things work fine. However, when I request an annotation with two or more features at the same time, the response does not always return all the request feature fields. For example, here is a request I ran recently using the API explorer:
{
"features": [
"EXPLICIT_CONTENT_DETECTION",
"LABEL_DETECTION",
"SHOT_CHANGE_DETECTION"
],
"inputUri": "gs://gccl_dd_01/Video1"
}
The operation Id I got back is this: "us-east1.11264560501473964275". When I run a GET with this Id, I have the following response:
200
{
"name": "us-east1.11264560501473964275",
"metadata": {
"#type": "type.googleapis.com/google.cloud.videointelligence.v1.AnnotateVideoProgress",
"annotationProgress": [
{
"inputUri": "/gccl_dd_01/Video1",
"progressPercent": 100,
"startTime": "2018-08-06T17:13:58.129978Z",
"updateTime": "2018-08-06T17:18:01.274877Z"
},
{
"inputUri": "/gccl_dd_01/Video1",
"progressPercent": 100,
"startTime": "2018-08-06T17:13:58.129978Z",
"updateTime": "2018-08-06T17:14:39.074505Z"
},
{
"inputUri": "/gccl_dd_01/Video1",
"progressPercent": 100,
"startTime": "2018-08-06T17:13:58.129978Z",
"updateTime": "2018-08-06T17:16:23.230536Z"
}
]
},
"done": true,
"response": {
"#type": "type.googleapis.com/google.cloud.videointelligence.v1.AnnotateVideoResponse",
"annotationResults": [
{
"inputUri": "/gccl_dd_01/Video1",
"segmentLabelAnnotations": [
...
],
"shotLabelAnnotations": [
...
],
"shotAnnotations": [
...
]
}
]
}
}
The done parameter for the response is set to true, but it does not have any field containing the annotations for Explicit Content.
This issue seems to be occurring at random to my novice eyes. The APIs will return a response with all parameters on some occasions and be missing one on others. I am wondering if there is anything I am missing here or something on my end that is causing this?
I did some tests using just LABEL_DETECTION, just EXPLICIT_CONTENT_DETECTION and using the three of them.
As I am not using videos with explicit content, I don't see any specific field when adding just EXPLICIT_CONTENT_DETECTION:
{
"name": "europe-west1.462458490043912485",
"metadata": {
"#type": "type.googleapis.com/google.cloud.videointelligence.v1.AnnotateVideoProgress",
"annotationProgress": [
{
"inputUri": "/cloud-ml-sandbox/video/chicago.mp4",
"startTime": "2018-08-07T14:18:40.086713Z",
"updateTime": "2018-08-07T14:18:40.230351Z"
}
]
}
}
Can you share a specific video sample, the request.json used and two different outputs, please?

How to know it is auto reply mail in Microsoft Graph?

I am using Microsoft Graph API to get mails.
GET /v1.0/me/messages
It returns
{
"#odata.context": "https://graph.microsoft.com/v1.0/$metadata#users('576552d5-3bc0-42a6-a23d-bfceb405db23')/messages",
"#odata.nextLink": "https://graph.microsoft.com/v1.0/me/messages?$skip=11",
"value": [
{
"#odata.etag": "W/\"HwAAABYAAACpTc/InBsuTYwTUBb+VIb4AACqi2tx\"",
"id": "AAMkADBlZTUwNTkxLWVmODgtNDVhNC1iZjhlLTdjNjA1ODZlMDI2MgBGAAAAAACUbnk-iwQZRbXMgkfKtmYhBwCpTc-InBsuTYwTUBb_VIb4AAAAAAEMAACpTc-InBsuTYwTUBb_VIb4AACqNTk9AAA=",
"createdDateTime": "2017-12-06T21:57:09Z",
"lastModifiedDateTime": "2017-12-06T21:57:19Z",
"changeKey": "HwAAABYAAACpTc/InBsuTYwTUBb+VIb4AACqi8tx",
"categories": [],
"receivedDateTime": "2017-12-06T21:57:09Z",
"sentDateTime": "2017-12-06T21:56:16Z",
"hasAttachments": false,
"internetMessageId": "<e74a536a53d245e49d779d47f774f4a0#CO2PR00MB0214.namprd00.prod.outlook.com>",
"subject": "Automatic reply: Hi",
"bodyPreview": "I am OOF.",
"importance": "normal",
"parentFolderId": "AAMkADBlZTUwNTkxLWVmODgtNDVhNC1iZjhlLTdjNjA2ODZlMDI5MgAuAAAAAACUbnk-iwQZRbXMgkfKtmYhAQCpTc-InBsuTYwTUBb_VIb4AAAAAAEMAAA=",
"conversationId": "AAQkADBlZTUwNTkxLWVmODgtNDVhNC1iZjhlLTdjNjA2ODZlMDI5MgAQAPekscpearpHmBFbhG0DKuc=",
"isDeliveryReceiptRequested": null,
"isReadReceiptRequested": false,
"isRead": true,
"isDraft": false,
"webLink": "https://outlook.office365.com/owa/?ItemID=AAMkADBlZTUwNTkxLWVmODgtNDVhNC1iZjhlLTdjNjA1ODZlMDI5MgBGAAAAAACUbnk%2FiwQZRbXMgkfKtmYhBwCpTc%2FInBsuTYwTUBb%2BVIb4AAAAAAEMAACpTc%2FInBsuTYwTUBb%2BVIb4AACqNTk2AAA%3D&exvsurl=2&viewmodel=ReadMessageItem",
"inferenceClassification": "focused",
"body": {
"contentType": "html",
"content": "hi"
},
"sender": {
"emailAddress": {
"name": "Jack",
"address": "jack#example.com"
}
},
"from": {
"emailAddress": {
"name": "Jack",
"address": "jack#example.com"
}
},
"toRecipients": [
{
"emailAddress": {
"name": "Rose",
"address": "rose#example.com"
}
}
],
"ccRecipients": [],
"bccRecipients": [],
"replyTo": []
}
]
}
I didn't find any field related with determine whether it is an auto reply mail.
Right now I am using
mail.subject.startsWith('Automatic reply:')
to determine whether is auto reply mail in code.
However, it is not reliable. Because sometimes I got mails starting with a different language such as Resposta automática:.
So how to know it is auto reply mail correctly?
As #Horkrine said there is no officially guaranteed way of detecting if an email is an auto reply or not.
But there are two ways that may be useful:
Method 1 : Detect the response time
If you are capable, consider checking the amount of time between the email sent and the response. If that time is within a certain threshold, it is almost certainly an auto reply. Consider a reply received within seconds, for example. This has a lot of correlations with modern-day spam-robot detection techniques.
Method 2 : Keywords
The other way to do it is to look for keywords, just as you are doing now. However, you also have to account for other languages, variations on spelling, misspellings, etc. You will not get everything.
For example:
mail.subject.contains('Automatic') OR mail.subject.contains('Auto-matic') OR mail.subject.contains('Away') OR mail.subject.contains('out of office')
...
OR mail.subject.contains('automática') ...
Rather than typing out such a list, I would recommend doing a quick search on the internet and see if there are any such lists you can copy-paste from, as surely someone has done this sort of thing before and has some free code.
I'm no expert but I don't believe there's any way to determine whether or not an email is an automatic reply unless the email actually contains a string saying "This is an automatic reply" or something.
Just found another interesting API getMailTips, however this can only help determine the auto mail if the other user is Outlook or Office 365 user.
Copy the demo below for convenience.
POST https://graph.microsoft.com/api/beta/users/{id|userPrincipalName}/getMailTips
{
"EmailAddresses": [
"danas#contoso.onmicrosoft.com",
"fannyd#contoso.onmicrosoft.com"
],
"MailTipsOptions": "automaticReplies, mailboxFullStatus"
}
It will return something like
{
"#odata.context":"https://graph.microsoft.com/api/beta/$metadata#Collection(microsoft.graph.mailTips)",
"value":[
{
"emailAddress":{
"name":"",
"address":"danas#contoso.onmicrosoft.com"
},
"automaticReplies":{
"message":"<style type=\"text/css\" style=\"\">\r\n<!--\r\np\r\n\t{margin-top:0;\r\n\tmargin-bottom:0}\r\n-->\r\n</style>\r\n<div dir=\"ltr\">\r\n<div id=\"x_divtagdefaultwrapper\" style=\"font-size:12pt; color:#000000; background-color:#FFFFFF; font-family:Calibri,Arial,Helvetica,sans-serif\">\r\n<p>Hi, I am on vacation right now. I'll get back to you after I return.<br>\r\n</p>\r\n</div>\r\n</div>",
"messageLanguage":{
"locale":"en-US",
"displayName":"English (United States)"
}
},
"mailboxFull":false
},
{
"emailAddress":{
"name":"",
"address":"fannyd#contoso.onmicrosoft.com"
},
"automaticReplies":{
"message":""
},
"mailboxFull":false
}
]
}

Detect the speaker of Google Home or Amazon's Alexa

I would like to detect who is interacting with my agent.
For example I read that Alexa should be able to detect different users. The Google Home advertisement also let me think that it should detect who is talking. So how can I see who is talking?
In slack it seems to be easier since it is well known who is writing. However I cannot see who I get the current user.
I found out how to detect the user in slack: If you implement that hook you will get this example json:
{
"id": "f7912345-e21c-450f-a8ca-d01e38012345",
"timestamp": "2016-12-20T06:53:51.071Z",
"result": {
"source": "agent",
"resolvedQuery": "echo hallo welt",
"speech": "",
"action": "",
"actionIncomplete": false,
"parameters": {
"myInput": "hallo welt"
},
"contexts": [{
"name": "generic",
"parameters": {
"slack_user_id": "U0AT12345",
"myInput": "hallo welt",
"slack_channel": "D3DR12345",
"myInput.original": "hallo welt"
},
"lifespan": 4
}],
"metadata": {
"intentId": "06212345-06a0-40fe-bbeb-9189db412345",
"webhookUsed": "true",
"webhookForSlotFillingUsed": "false",
"intentName": "Response"
},
"fulfillment": {
"speech": "",
"messages": [{
"type": 0,
"speech": ""
}]
},
"score": 0.75
},
"status": {
"code": 200,
"errorType": "success"
},
"sessionId": "10612345-c681-11e6-af08-875120912345",
"originalRequest": {
"source": "slack_testbot",
"data": {
"channel": "D3DR12345",
"match": ["echo hallo welt"],
"text": "echo hallo welt",
"team": "T04H12345",
"type": "message",
"event": "direct_message",
"user": "U0AT12345",
"ts": "1482216830.000005"
}
}
}
So in case of slack you can access result->contexts[0]->paramaters->slack_user_id.
Google Home does not (at least currently) have a way to handle multiple users on the same device.
Google Home keeps improving (even removing development hurdles I've faced with their recent updates). It can now be trained to know your voice vs someone else's voice.
Tomato, tomahto. Google Home now supports multiple users

Can I use actionResulting to link Questionnaire to a CarePlan activity?

I have asked about Questionnaire and CarePlan earlier and someone else has asked another similar question, but my question this time is a bit more specific.
This time I wonder if it's allowed to create a CarePlan using 'detail' rather than reference and then link Questionnaire in to actionResulting. Could I use the actionResulting as a link to the Questionnaire? Would the following resource be valid?
{
"resourceType": "CarePlan",
...,
"activity": [
{
"actionResulting": [
{
"reference": "Questionnaire/copd-form"
}
]
"detail": {
"category": {
"coding": [
{
"system": "http://hl7.org/fhir/care-plan-activity-category",
"code": "observation"
}
]
},
"code": {
"text": "A code for capturing COPD subjective input from patient?"
},
"prohibited": false,
"scheduledTiming": {
"repeat": {
"frequency": 1,
"period": 1,
"periodUnits": "d"
}
},
"performer": [
{
"reference": "Patient/3183"
}
],
"description": "Daily COPD form"
}
}
}
Well, pointing to the Questionnaire would be saying "as a result of this plan, this Questionnaire was designed". I suspect what you'd really want to say is "as a result of this plan, I asked the patient to fill out Questionnaire X" (which would be a DiagnosticOrder) or "as a result of this plan, the patient answered this Questionnaire" (which would be a QuestionnaireResponse). For the first of those, I believe we're still waiting on an extension to allow a DiagnosticRequest to point to the Questionnaire to be filled out.

Resources