Is it possible to train Google Speech API with sample data to help the recognition in my application?
What I mean is an approach like the one provided by wit.ai and described here (even though the example applies to nlp processing). Basically, if you can predict the interactions your users will have with your bot, you can train it to better perform. For instance, I know the subset of cities that will be used, eg: it seems I cannot make the bot understand me when I say Zurich, it becomes Syria or Siberia but I already know that is not possible. So if I, let's say, can upload a list of preferred words to be used first and then if no match is found there fallback to standard recognition or some similar approach I think it will be achieve better results.
Any idea if it is possible and how? I know those APIs are in beta stage and subject to change, but I would still like to give it a try.
I can upload some code sample of what I am currently doing, though it is just sending an audio and analyzing the result so far, so not really close to this problem.
In recognition config you can specify alternatives to return you with maxAlternatives field (up to 30). Once you have 30 alternatives with confidence you'll have Syria with confidence 0.5, Siberia with confidence 0.01 and Zurich with confidence 0.1. Usually the proper answer is present, although it might not be on the top. You can select best alternative according to your current status.
The current Google Cloud Speech-to-Text API allows the user to specify a list of words and phrases that provide hints to the speech recognition task.
From https://cloud.google.com/speech-to-text/docs/basics (mirror):
speechContext - (optional) contains additional contextual information for processing this audio. A context contains the following sub-field:
phrases - contains a list of words and phrases that provide hints to the speech recognition task.
For more details, see: https://cloud.google.com/speech-to-text/docs/basics#phrase-hints (mirror).
Related
I need to create a room based chat app, that needs to support thousands of users at the same time.
The problem is, client wants to filter bad words from the messages.
Standard profanity filter libraries would not be a problem to use if the number of the users is low, but in this case performance of the filtering is the most important thing as the chat is real-time.
Libraries that uses NLP are mostly trained with english language datasets, and it will also not work for me since i need to filter bad words for couple of langugages.
The only remaining thing that comes to my mind is ElasticSearch.
Is there any solution that is good enough and can support large numbers of the users, something that would work like this :
bad_words = [badword1, badword2, badword3...]
Input message : "You are badword1, and i badword2 you!"
Output message: "You are ***, and i *** you"
If the solution can also handle cases where, for the example "badword1" is written as "bad_word1" or something like that, it would be a bonus.
All suggestions are welcomed.
Background
We have an application that is using Google Translation API in order to translate some data to a target language.
For that, we use the /language/translate/v2 call without specifying the source language and Google is detecting the source language and translating in the same call.
Issue
Most of the time we are sending to translate text in the same language that is our target. In example: we receive a row in English (we don't know the source lang) and it needs to be translated into English.
Questions
We know that Google is charging by characters and operation. But, is Google charging double we you call /translate and it has to perform both operations?
Is Google performing a translation when the target lang is the same that was detected?
If so, is Google charging when the source lang is the same that the target lang?
You might already know about the Cloud Translation API > Documentation > Pricing section of the documentation, however I am posting it here so everyone could access.
As it is mentioned in the documentation * Price is per character sent to the API for processing, including whitespace characters. Empty queries are charged for one character.. Which means that Google is charging per character that is sent to the API.
However, it is also specified that Google does not charge extra for language detection when you do not specify the source language for the translate method; the Language Detection price applies to detect method calls.. Which also means that if you don't specify the source language, you will be charged only for the translation process and not the language detection.
We know that Google is charging by characters and operation. But, is Google charging double we you call /translate and it has to perform both operations?
No.
Is Google performing a translation when the target lang is the same that was detected?
No.
If so, is Google charging when the source lang is the same that the target lang?
Good question, I assume so, but cannot say.
It may seem a bit unfair, but much of their cost is development and then handling requests at scale. This goes back to their original motivation for making the API paid - people were calling it in very inefficient ways, for example every time a webpage loaded, which risked making it unstable for other developers or unsustainable for Google.
Similarly, they do not charge twice for translation between two languages that are not English, even though usually they are translating twice under the hood - from the source language to English, and English to the target language. So most of the time, it works in the client's favour.
Personally, if I knew that most of my queries will be no-ops, I would run a client-side lang id library first. Then you can set a threshold, to balance cost and risk. In fact, you could send multiple requests if lang id gives a high score for multiple languages.
Besides the API cost there is latency and machine cost and network cost, and I also bet that you can do better than Google Translate's detection by tuning a bit for your content, assuming it's not open domain.
Background:
I have been digging into the FHIR DSTU2 specification to try and determine what is the most appropriate resource(s) to represent a particular patient's historical list of GPs/PCPs. I am struggling to find an ideal resource to house this information.
The primary criteria I have been using is to identify the proper resource is that it must provide values to associate a patient to a practitioner for a period of time.
Question:
What is the proper resource to represent historical pcp/gp information that can be tied back to a patient resource?
What I have explored:
Here is a list of my possible picks thus far. I paired the resource types with my thought process on why I'm not confident about using it:
Episode of Care - This seems to have the most potential. It has the associations between a patient and a set of doctors for a given time period. However, when I read its description and use-case scenarios, it seems like I would be bastardizing its usage to fit my needs, since it embodies a period of time where a group of related health care activities were performed.
Group - Very generic structure that could fit based on its definition. However, I want to rule out other options before taking this approach.
Care Plan - Similar to Episode of Care rational. It seems like a bastardization to just use this to house PCP/GP history information. The scope of this is much bigger and patient/condition-centric.
I understand that there may not be a clear answer and thus, the question might run the risk of becoming subjective and I apologize in advance if this is the case. Just wondering if anyone can provide concrete evidence of where this information should be stored.
Thanks!
That's not a use-case we've really encountered before. The best possibility is to use the new CareTeam resource (we're splitting out CareTeam from EpisodeOfCare and CarePlan) - take a look at the continuous integration build for a draft.
If you need to use DSTU 2, you could just look at Patient.careProvider and rely on "history" to see changes over time. Or use Basic to look like the new CareTeam resource.
We are currently in the process of evaluating FHIR for use as part of our medical record infrastructure. For the EHR data (Allergies, Visits, Rx, etc..) the HL7 FHIR seems to have an appropriate mapping.
However, lots of data that we deal with is related to personal Fitness - think Fitbit or Apple HealthKit:
Active exercise (aerobic or workout): quantity, energy, heart-rate
Routine activities such as daily steps or water consumption
Sleep patterns/quality (odd case of inter-lapping states within the same timespan)
Other user-provided: emotional rating, eating activity, women's health, UV
While there is the Observation resource, this still seems best fit (!) for the EHR domain. In particular, the user fitness data is not collected during a visit and is not human-verified.
The goal is to find a "standardized FIHR way" to model this sort of data.
Use an Observation (?) with Extensions? Profiles? Domain-specific rules?
FHIR allows extraordinary flexibility, but each extension/profile may increase the cost of being able to exchange the resource directly later.
An explanation on the appropriate use of an FHIR resource - including when to Extend, use Profiles/tags, or encode differentiation via Coded values - would be useful.
Define a new/custom Resource type?
FHIR DSTU2 does not define a way to define a new Resource type. Wanting to do so may indicate that the role of resources - logical concept vs. an implementation interface? - is not understood.
Don't use FHIR at all? Don't use FHIR except on summary interchanges?
It could also be the case that FHIR is not suitable for our messaging format. But would it be any "worse" to go FIHRa <-> FIHRb than x <-> FIHRc when dealing with external interoperability?
The FHIR Registry did not seem to contain any User-Fitness specific Observation Profiles and none of the Proposed Resources seem to add appropriate resource-refinements.
At the end of the day, it would be nice to be able to claim to be able to - with minimal or no translation, ie. in a "standard manner" - be able to exchange User Fitness data as an FHIR stream.
Certainly the intent is to use Observation, and there's lots of projects already doing this.
There's no need for extensions, it's just a straight forward use. Note that this: " In particular the user fitness data is not collected during a visit and is not human-verified" doesn't matter. There's lots of EHR data of dubious provenance...
You just need to use the right codes, and bingo, it all works. I've provided a bit more detail to the answer here:
http://www.healthintersections.com.au/?p=2487
i am thinking of making a website..
bt how can i make sure that when a user who is asking some question is nt using any abusive language or the message is totally subject oriented..
i m nt talking about spams..i know about captcha and all..
what i am asking is how can i keep an eye on human activity[in this case the messages sent] and at the same time providing the user his complete privacy!
One word... manually.
They're on the web, they already don't have complete privacy.
Offer the community the means to police themselves, whether by explicitly appointing moderators (like most bulletin boards), allowing them to decide who they can and cannot see (like social media sites), or collaborative moderation (like here).
You can set up a system where comments/posts must be approved by a moderator before being allowed to be posted. I believe Wordpress can do this.
There are curse-word filtering libraries available in most languages, usually complete with the ability to customize the words that are filtered out.
In order to filter spam, there are things like bayesian spam filters which attempt to determine whether a message is spam based on keywords in the response. This really isn't something you would want to attempt to do yourself.
Another thing to look at is Markov Chains. They are designed to generate strings of seemingly valid text based on the probability that any given word is followed by any other particular word. Using a reverse process you can attempt to determine if a string of text is valid by checking whether the words used are following by other "on-topic" words.
This would be very difficult as well.
In order to keep the privacy of the users, you could use combinations of these three tests to create a threshold. That is, you will examine no messages unless they reach a high curse/spam/off-topic score. At that point, those messages will be manually checked to see if they are appropriate.
There currently is no way to have a 100% automated process that won't block valid messages and let invalid ones through.
how can i keep an eye on human activity
Your answer lies here. I don't quite understand what you're getting at about privacy though.