I set up a scheduling website at work which uses Twilio to send out sms messages with each person's next day assignment information as well as a link to the website where notes and everyone's next-day assignment is posted. Occasionally colleagues complain that they haven't received messages. Recently the problem has gotten much worse though is to be limited to people with AT&T and Sprint. People with Verizon and TMobile are get their messages without a problem.
A few more facts:
All of my code works well in testing and is consistent with Twilio's
instructions.
Twilio's logs list the messages as having been sent.
The people who's sms messages are not delivered tend to be at the end
of the alphabet
Anyone have any ideas what might be going on?
SMS messages the twilio logs show as sent which never arrive at the recipient's phone may be filtered by cell phone carriers as part of an effort to reduce spam sms messages. A website that sends out a batch of messages like the scheduling website you (I) describe may be particularly susceptible to such filtering since spam is also sent out in batches. The fact that recipients at the end of the alphabet tend to be blocked rather than those at the beginning might reflect an algorithm which flags earlier messages as suspicious and blocks subsequent ones that appear similar in that they contain the same website link.
In the past I was able to solve this problem by adding a 1 second delay between messages. That worked fine for my purposes. My site sent out the messages as a background job and with the 1 second delay about 25 sms messages get sent out over 2 minutes.
More recently AT&T and Sprint started blocking many more messages, though not all. At Jan 11, 15:04 PST, Twilio's status was aware of the problem and noted:
Identified - Messages with hyperlinks to goo.gl are being filtered on AT&T and Sprint. We are working with these carriers to address this filtering.
I was able to get around this problem by creating a shortened url using Bitly instead of Google's url shortening service.
Of note, Twilio is a great service but in the past their support folks were unaware that spam filtering could be blocking some of my sms messages. That's my prime motivation for posting about this issue here.
It seems that sites like mine can get caught in the crossfire between sms spammers and wireless carriers attempting to block them. Hopefully some day such sms spam filtering will get more sophisticated and stop blocking the messages sent by sites like mine.
I would love to hear if anyone with more sophisticated insight than mine has any comments.
Currently experiencing a really bad carrier filtering case through Twilio. Most messages sent to Verizon numbers are being blocked. According to Twilio support, Verizon recently implemented additional filtering methods to block A2P (Application-to-Peer) traffic.
Given this article: https://arstechnica.com/information-technology/2015/11/att-verizon-try-to-prevent-ban-on-text-message-blocking/ it seems like Verizon's more aggressive filtering might be related to the recent repeal of Net Neutrality, and their goal is to make more money by forcing businesses to get a short code.
If your using 10 digit phone numbers I Would also look at this
https://www.twilio.com/help/faq/short-codes/why-would-i-want-a-short-code-instead-of-sending-sms-from-a-regular-us-phone-number-or-phone-numbers
Long codes are meant for person-to-person communications, and can send only 1 message per second. For high-volume, application-driven messaging, Twilio recommends using a short code. Short codes can send SMS and MMS at 30 messages per second, and this high throughput is perfect for applications needing to send time-sensitive messages to many users at once. Furthermore, since carriers vet and approve all short codes for their intended use, they are not subject to carrier filtering or suspension for heavy traffic.
Related
Recently I am trying to use MassTransit in our microservice ecosystem.
According to MassTransit vocabulary and from documents my understanding is :
Publish: Sends a message to 1 or many subscribers (Pub/Sub Pattern) to propagate the message.
Send: Used to send messages in fire and forget fashion like publish, but instead It is just used for one receiver. The main difference with Publish is that in Send if your destination didn't receive a message, it would return an exception.
Requests: uses request/reply pattern to just send a message and get a response in a different channel to be able to get response value from the receiver.
Now, my question is according to the Microservice concept, to follow the event-driven design, we use Publish to propagate messages(Events) to the entire ecosystem. but what is exactly the usage (use case) of Send here? Just to get an exception if the receiver doesn't exist?
My next question is that is it a good approach to use Publish, Send and Requests in a Microservices ecosystem at the same time? like publish for propagation events, Send for command (fire and forget), and Requests for getting responses from the destination.
----- Update
I also found here which Chris Patterson clear lots of things. It also helps me a lot.
Your question is not related to MassTransit. MassTransit implements well-known messaging patterns thoughtfully described on popular resources such as Enterprise Integration Patterns
As Eben wrote in his answer, the decision of what pattern to use is driven by intent. There are also technical differences in the message delivery mechanics for each pattern.
Send is for commands, you tell some other service to do something. You do not wait for a reply (fire and forget), although you might get a confirmation of the action success or failure by other means (an event, for example).
It is an implementation of the point-to-point channel, where you also can implement competing consumers to scale the processing, but those will be instances of the same service.
With MassTransit using RabbitMQ it's done by publishing messages to the endpoint exchange rather than to the message type exchange, so no other endpoints will get the message even though they can consume it.
Publish is for events. It's a broadcast type of delivery or fan-out. You might be publishing events to which no one is listening, so you don't really know who will be consuming them. You also don't expect any response.
It is an implementation of the publish-subscribe channel.
MassTransit with RabbitMQ creates exchanges for each message type published and publishes messages to those exchanges. Consumers create bindings between their endpoint exchanges and message exchanges, so each consumer service (different apps) will get those in their independent queues.
Request-response can be used for both commands that need to be confirmed, or for queries.
It is an implementation of the request-reply message pattern.
MassTransit has nice diagrams in the docs explaining the mechanics for RabbitMQ.
Those messaging patterns are frequently used in a complex distributed system in different combinations and variations.
The difference between Send and Publish has to do with intent.
As you stated, Send is for commands and Publish is for events. I worked on a large enterprise system once running on webMethods as the integration engine/service bus and only events were used. I can tell you that it was less than ideal. If the distinction had been there between commands and events it would've made a lot more sense to more people. Anyway, technically one needs a message enqueued and on that level it doesn't matter, which is why a queueing mechanism typically would not care about such semantics.
To illustrate this with a silly example: Facebook places and Event on my timeline that one of my friends is having a birthday on a particular day. I can respond directly (send a message) or I could publish a message on my timeline and hope my friend sees it. Another silly example: You send an e-mail to PersonA and CC 4 others asking "Please produce report ABC". PersonA would be expected to produce the report or arrange for it to be done. If that same e-mail went to all five people as the recipient (no CC) then who gets to do it? I know, even for Publish one could have a 1-1 recipient/topic but what if another endpoint subscribed? What would that mean?
So the sender is responsible, still configurable as subscriptions are, to determine where to Send the message to. For my own service bus I use an implementation of an IMessageRouteProvider interface. A practical example in a system I once developed was where e-mails received had to have their body converted to an image for a content store (IBM FileNet P8 if memory serves). For reasons I will not go into the systems were stopped each night at 20h00 and restarted at 6h00 in the morning. This led to a backlog of usually around 8000 e-mails that had to be converted. The conversion endpoint would process a conversion in about 2 seconds but that still takes a while to work through. In the meantime the web front-end folks could request PDF files for conversion to paged TIFF files. Now, these ended up at the end of the queue and they would have to wait hours for that to come back. The solution was to implement another conversion endpoint, with its own queue, and have the web front-end configured to send the same message type, e.g. ConvertDocumentCommand to that "priority" queue for processing. Pretty easy to do. Now, if that had been a publish how would I do that split? The same event going to 2 different endpoints under different circumstances? Well, you could have another subscription store for your system but now you'd need to maintain both. There could be another answer such as coding this logic into the send bit but that is a design choice and would require coding changes.
In my own Shuttle.Esb service bus I only have Send and Publish. For request/response both the sender and receiver have an inbox and a request would be sent (Send) to the receiver and it in turn could reply (also a Send but uses the sender's URI).
We're having this weird problem with Twilio SMS messages hanging with status.
We have tried sending from different Twilio phone numbers to make sure it isn't a problem with that particular number being blocked and none of them go through.
Our system uses SMS messages in the standard form of two-step authentication with a code and a short message to the user inside the SMS body.
The carrier that the message sent is failing is Tune Talk (A Malaysian one).
The error in the Twilio Logs/Console I see is:
Status: Undelivered
Error: (Error: 30008) Unknown error. None
Message SID if it's in any way useful is: SM1024a2d519cf4f6bbfcbc838587cb2af
Any insight on why this is happening would be greatly welcomed.
We had also the same issue with phones that used to receive messages.
The problem is carrier blocking/filtering.
Every carrier uses different filters.
Some carriers block messages with 90% the same content, others use rate filtering (1 message per second or more) others use a combination. The blocking is not forever thought.
Twilio gives the following possible solutions:
Check that the phone you were sending to is turned on and can receive SMS
Ensure that the phone is not roaming off network. We cannot
guarantee message delivery on roaming phones.
Try sending to other phones who have the same mobile carrier.
If messages to other phones go through, the issue is likely device
related. Try rebooting the device or contact the mobile carrier for
help.
If you are sending SMS from an alphanumeric sender ID, see if using
a Twilio phone number works better. We’ve observed that certain
networks may block alpha sender IDs.
Try sending a shorter message to the phone, with simple content that
does not include any special characters. This would give our support
team an idea as to whether the failure is related to concatenation
or character encoding.
Twilio support can help investigate what went
wrong with our carriers. Please open a support request and include a
minimum of 3 or more message SIDs where a 30008 error was thrown.
Per our carriers' requirements, these SIDs can be no older than 48
hours at most.
Check -> Error 30008
Another solution is to use a 5digit code phone number.
Boris, error 30008 is certainly less descriptive than one would hope. In this case, it would be best to send that Message Sid along to support where we can dig a little deeper into the specifics.
Though it doesn't sound like it in this case, if there were a problem with your code, you could check out a production ready account verification tutorial here.
Came here because of exactly the same problem. I have someone who successfully received SMSes just 12 days ago, using the same Australian number, getting a 30008 for every attempt to send to them today. That's a really average-quality error message right there.
The user states that they ported the number from Telstra to Vodafone, but that was 3 months ago. I'm guessing that the forwarding is broken:
http://www.commsalliance.com.au/__data/assets/pdf_file/0013/2326/G565_2009.pdf
In particular:
1.4.4 Where internationally originated SMS is supported donor routing
must be supported wherever bilateral agreement exists for the
national leg, as international networks are not likely to access an
Australian mobile number portability database prior to routing the
message. However, certain limitations apply – see Appendix A.
Since Twilio aren't sending from an Australian number, they probably aren't looking through the number portability database. This would be my suspect cause for any failure to SMS route to a country with number portability.
I've had some of these errors when doing MMS's.
If I look at the detail in the Twilio console, there is additional detail and a secondary error message "12300 Invalid Content-Type. Attempt to retrieve MediaUrl returned an unsupported Content-Type."
I was putting images up on S3, but not setting the Content-Type of the image when I put it on S3.
In my case, the problem was that I was including a website URL in the message.
Once I removed it (replaced it with a simple name) it worked.
I'm not sure whether the issue was the URL itself (e.g. the network considers it spam) or the fact that it contained characters like /.
Hypothetical: I want to send a single text message to all Verizon phones programmatically. I have multiple email gateways to use (obtained from the all-reliable wikipedia):
number#vtext.com
number#vzwpix.com
number#message.alltel.com
number#text.wireless.alltel.com
number#mms.alltel.net
I don't think that I'm guaranteed that any one of these will work and/or will still be in service (am I?) and I would not like to have to come back and change anything in the code at a later date.
Is there any way that I can make sure that I only send one text message to a given phone number when there are 5 possible gateways?
The only way way (that I know of) to ensure you only deliver one message to the recipient is to try each gateway sequentially until a message sends successfully, like some of the comments mentioned.
However, I've been sending a decent volume (>1500) of messages using #vtext.com lately and haven't noticed any bounces or downtime during sending. I have no way of knowing if every single message was delivered, but none of my test numbers have dropped a message yet. Most US carriers seem to have decent reliability on their gateways these days.
Just remember that SMS is still considered a best-effort service by most carriers. Even if you get your message to their servers successfully, there's no guarantee that the message will make it to its destination.
I'm working on an application that sends and receives SMS messages to and from its users. (Don't worry - it's not spam - every user of the app expects to send and receive these messages).
One key aspect of the app... If a user sends a message to the app, the app then sends that message out to every person on that user's "team". So, the app will be sending a receiving a pretty significant number of messages (I'm hoping for a few thousand users, and 5,000-10,000 SMS messages per day).
I've been experimenting with a number of options:
SMS to email
Connecting a mobile phone to my server
Contracting an SMS gateway
Option 1 is great, since it's free, but it's unreliable (apparently mobile providers queue these messages after SMS messages they can charge for, so they're frequently received late or lost)
Option 2 is also cheap, but the mobile phone can't keep up with the number of messages I'll be sending. Also, the mobile phone provider will consider this volume of messages excessive.
Option 3 is perfect, except that SMS gateway providers charge PER MESSAGE (usually $0.02-$0.06 per), which creates an impossible scaling problem. (Reminds me of the old business adage... "Sure we're losing money on every transaction, but we'll make it up in volume...")
So, long story short - how on EARTH did Twitter pull this off? They've been doing a similar thing (allowing users to exchange SMS messages with the app) since the beginning. Even if they negotiated an INCREDIBLE discount (say, $0.001 per message), they'd be paying an ENORMOUS cost to send the hundreds of millions of messages they handle.
Does anyone have any idea how they did this?
After a bit of Googling, it looks like Twitter has simply been signing deals with cell phone companies in various countries. For example (Twitter blog post link spam incoming):
Australia: http://blog.twitter.com/2009/12/sms-tweets-for-telstra-australia.html
Indonesia: http://blog.twitter.com/2009/11/sms-for-axis-indonesia.html
New Zealand: http://blog.twitter.com/2009/05/hello-new-zealand.html
UK: http://blog.twitter.com/2009/03/full-sms-service-for-vodafone-uk.html
Some more details about SMS in general here.
I am about to start building an SMS alerting function in my web application. The aim is to provide two services:
Host pays - e.g. to send an SMS alerting users to the cancellation of an event
User pays - e.g. to alert that an Email has been sent with, say, details of a new event (apparently a requirement from users when they are away from their EMail systems!)
I also figure that there will be other user-pays scenarios such as if they have secondary mobile phones that they want alerting as the Host will only want to pay for a single phone per user.
From my research:
I can use a 3rd party SMS Gateway provider. Cost is about GBP 0.05 per message. I can either send an EMail to 999999999#TheGatewayProvider.com or I can use an HTTP request with suitable parameters in the URL.
I can send an EMail to the user's network provider (I believe this is only available in the USA)
EDIT: There are variations on how the various providers will handle the From/Subject and Message, so presentation of actual message sent may be hard to predict.
I can set up my own Gateway (which I think is way beyond my ability, and may give grief at our data centre!)
It strikes me that:
EMail we send sometimes get delayed in my server's SMTP queue, let alone any downstream queues. Sending an email to the network provider's SMS gateway seems to often be treated as low priority.
Therefore HTTP to a 3rd party SMS Gateway provider should give me shortest latency (important for "This afternoons event is cancelled because of bad weather")
When I send SMS Text from my mobile occasionally they take days to arrive - I presume this is something we just have to live with?
Having said that we will also have low priority informational messages and sending these by cheapest route is attractive! so I am planning to allow users to enter an EMail address for such messages - the intention being that they will use the email address for their mobile phone company's SMS redirect service, or similar (i.e. the email address of a device, not an inbox).
I also wonder whether allowing the users to enter IM, Twitter or other such is likely to be well used and efficient in practice?
From the 3rd party gateways I have looked at it seems:
Some use higher grade networks than others, this may impact performance? (or is it just marketing hogwash?)
Some provide better feedback than others. I need to keep the arguments over billing - and exactly how many "credits" the client has used up - to a minimum; getting an answerback of "OK" / "Phone number does not exist" therefore strikes me as important. One provider I found creates a text log file daily that can be downloaded and which I could reconcile with my outgoing log.
I would appreciate your opinions and experiences on:
Users will enter their mobile numbers as they know them. Do I need to enforce +9912345... so that I get the country code too?
What happens if the mobile is foreign (I'm based in the UK) Does the recipient pay the international part? or does the gateway provider perhaps have local transmission services?
What do I need to do with non alpha characters? The UK Pound sign "£" and CR / LF spring to mind. If these are encoded might that cause a message right on the length limit to exceed it once encoded (such that I need to build this in to the validation of the message creation form). Are new lines CR+LF or will just CR do?
Do any gateways have simulations? such that I could test my application without incurring any actual costs of SMS texts.
I plan to log the mobile numbers of any failed attempts and flag the user's record so that the next time they login I can encourage them to correct the number.
Any other gotchas and suggestions you have would be much appreciated. Thanks.
The one big gotcha, that has happened to me, although this is not a programming question as such, its just an experience of it, that needs to be taken into account.
When someone sends you a message that is of an urgent meaning, and then the sender (owner of the handset) comes over to you and asked you did you get the message, and this is what irks me, a row can arise because you can claim to not have received it, thus making the sender somewhat feeling miffed/cross/angry that you have "ignored them" or "did not respond" when in fact it is down to the provider's network problems.
And funnily enough, after about an hour after having a row, your handset goes beep, and your message from the sender arrived!!!! This has happened to me several times! But what can I do without ever knowing what is happening on the sender's side of things (crisis, urgent query etc)
So be cautious if sending the message across from ethernet (internet) to the handset as ethernet is not exactly 100% stable (routers go down, dns disappearing, server outages etc) so it is worthwhile to keep this in mind. Like the question is how can you guarantee instant delivery of sms? That's a big question and reliability will have to come in, usually requiring extra effort to do so...
Handsets, most of them usually do, have what is known as a delivery report, some are switched on, some are switched off, that is a useful way of knowing if the recipient's handset is switched on or off, if it's on, you will get an instant delivery report indicating that it was delivered, likewise, if it's off, there will be a noticeable time delay in getting the delivery report, that is dependant on your providers' maximum time for delivery.
Sorry if this seems like a rambling...
Hope this gives you some food for thought,
Best regards,
Tom.
SMS is a telco service, not an Internet service. That comes with some different rules.
For starters, many endpoints are billed/billable, and have contracts with a single service provider. This will include all of your use cases.
Secondly, billing is a contract matter both on the sending and the receiving side. You simply cannot state as a sender that "Host pays", unless you restrict yourself to sending SMS to specific countries. USA is the most famous exception. "Receiver pays" is even worse. Due to SMS spam, telco's will usually allow this kind of traffic only when you have a contract with them.
Third-party SMS operators can deal with many of these problems. It's very easy for them to be more service-oriented than the average telco. They might even be able to deliver international SMS for you.
SMS tends to buffer in the network itself, not necessarily the email gateway. In individual cases, the difference is probably invisible to you. But you would still have delays even if you had a direct SS7 link to the telco.
Real telco's have test gateways, but the terms of use for those I can't give you. The idea though is definitely to be able to test you app at lower costs.
SMS uses its own alphabet, a rather nasty multi-septet encoding (7/14/21 bits!) The quoted 160 character limit comes from a 140 byte payload. This could also be coded as 70 UTF-16 characters.