I'm developing an SMS Gateway and I was thinking about the features that I should implement in it. So I reviewed some SMS gateways (SMSNOW, SMS studio, and Ozeki) for examples.
I almost finished the implementation but I came across a feature in almost all SMS gateways which allows the user to specify TON/NPI for SMSC and ESME.
I understand that when the SMS gateway is preparing to send the sms using the SMPP protocol it would need to parse the sender/reciever number and based on that, specify TON/NPI. And I understand that it's great to allow you to override the defaults and specify your own TON/NPI.
What I don't understand is why all SMS gateways give you the option to specify your own TON/NPI for the SMS gateway when it acts as SMSC (when it receives SMS, not sending them).
What would the SMS gateway do with the specified TON/NPI when it recieves an smpp SMS?
The full specification of an SMS source address involves 3 attributes:
TON
NPI
The actual Address
TON = Type of Number.
This specifies whether the number is a full International Number (i.e. needs the + prefix), or a National Number, or even an alphanumeric sender (i.e. includes up to 11 characters of text instead of a numeric sender, quite useful for advertising).
NPI = Numbering Plan Indicator
This specifies which numbering standard the number is following. It could be the E.164 standard international numbering, or the operator's private numbering (for premium rated services for instance).
The actual Address.
An SMSC would need the TON/NPI to know how to interpret the bytes stored in the Address. If for example the TON is alphanumeric (5), it knows the bytes in the address need to be interpreted as characters and encoded accordingly, while if it is International (1), it would split each byte into 2, with each half byte representing a different digit for the sender.
You can refer to the SMPP 3.4 specification Sections 5.2.5 and 5.2.6 for a full list of these TON/NPI possibilities. The specific SMS gateway you're using might restrict you from using some of them.
TON (Type of Number)
If you wish to specify a special value for the TON, the available options are:
0: Unknown
1: International
2: National
3: Network Specific
4: Subscriber Number
5: Alphanumeric
6: Abbreviated
NPI (Numbering Plan Identification)
For NPI, the available options are:
Unknown = 0
ISDN/telephone numbering plan (E163/E164) = 1
Data numbering plan (X.121) = 3
Telex numbering plan (F.69) = 4
Land Mobile (E.212) =6
National numbering plan = 8
Private numbering plan = 9
ERMES numbering plan (ETSI DE/PS 3 01-3) = 10
Internet (IP) = 13
WAP Client Id (to be defined by WAP Forum) = 18
TON and NPI Defaults
Short Code (3 digits to 8 digits in length)
If the source code/address is a Short Code
TON = 3
NPI = 0
Long Code (10 digits to 15 digits in length, excludes the plus sign)
TON = 1
NPI = 1
Alphanumeric
If the source code/address is Alphanumeric (contains both letters and numbers or only letters)
TON = 5
NPI = 0
Related
I have a case when SFMC JB performed an SMS send to records, that did not have a phone number in DE for Entry Source, but there is a phone number assigned to them in the database.
The Journey settings were set to "Use phone number attribute from entry source - Phone".
Here is an example record in DE:
SubscriberKey | Email | Phone
123 | example#gmail.com | null
Help states that "To send SMS from a journey, the normalized phone number (country code + phone number with no dashes or parentheses) is required in your data extension."
Could you please tell me how to troubleshoot this case, or if you see any other potential things that have given the following output? And how to perform a send so it does not happen again?
Thanks a lot!
I have an EA set in place that loops history trades and builds one large string with trade information. I then send this string every second from MT4 to the python backend using a plain PUSH/PULL pattern.
For whatever reason, the data isn't received on the pull side when the string transferred becomes too long. The backend PULL-socket slices each string and further processes it.
Any chance that the PULL-side is too slow to grab and process all the data which then causes an overflow (so that a delay arises due to the processing part)?
Talking about file sizes we are well below 5kb per second.
This is the PULL-socket, which manipulates the data after receiving it:
while True:
# check 24/7 for available data in the pull socket
try:
msg = zmq_socket.recv_string()
data = msg.split("|")
print(data)
# if data is available and msg is account info, handle as follows
if data[0] == "account_info":
[...]
except zmq.error.Again:
print("\nResource timeout.. please try again.")
sleep(0.000001)
I am a bit curious now since the pull socket seems to not even be able to process a string containing 40 trades with their according information on a single MT4 client - Python connection. I actually planned to set it up to handle more than 5.000 MT4 clients - python backend connections at once.
Q : Any chance that the pull side is too slow to grab and process all the data which then causes an overflow (so that a delay arises due to the processing part)?
Zero chance.
Sending 640 B each second is definitely no showstopper ( 5kb per second - is nowhere near a performance ceiling... )
The posted problem formulation is otherwise undecidable.
Step 1) POSACK/NACK prove whether a PUSH side accepts the payload for sending error-free.
Step 2) prove the PULL side is not to be blamed - [PUSH.send(640*chr(64+i)) for i in range( 10 )] via a python-2-python tcp://-transport-class solo-channel crossing host-to-host hop, over at least your local physical network ( no VMCI/emulated vLAN, no other localhost colocation )
Step 3) if either steps above got POSACK-ed, your next chances are the ZeroMQ configuration space and/or the MT4-based PUSH-side incompatibility, most probably "hidden" inside a (not mentioned) third party ZeroMQ wrapper used / first-party issues with string handling / processing ( which you must have already read about, as it has been so many times observed and mentioned in the past posts about this trouble with well "hidden" MQL4 internal eco-system changes ).
Anyway, stay tuned. ZeroMQ is a sure bet and a truly horsepower for professional and low-latency designs in distributed-system's domain.
I'm using ZeroMQ / ZMQ from Python and Java and have a question. When sending a shorter string, ZMQ uses one byte as described here (http://zguide.zeromq.org/page:all#A-Minor-Note-on-Strings)
Then what goes onto the wire is a length (one byte for shorter
strings) and the string contents as individual characters.
Does anyone know how many bytes are used when sending a longer string?
How many bytes are used for longer string when sending via ZMQ?
That depends on hell more things, than just on the string itself :
Your post refers to indeed historical text - the zguide pages.
While this was sure a very helpful first-read source in the early days of ZeroMQ v.2.x, today, we live with distributed-sysems spanning many versions, from v.2.1+, 3.x, 4.x, 4.2 being so far the last stable API version in 2018-Q2.
No one can a priori guess what API-version was used on the message-sender's side, until a receiver actually sets/accepts the link-setup and .recv()-s the respective message. Relying on a C-lang based s_recv()-helper tricks in post v4.0 API is not a sure direction to follow.
Being in python, many protocol-hardwired details remain beyond your sight, yet there are byte-maps, that get filled under the hood exactly as the benevolent dictatorship, indoctrinated in the published ZeroMQ RFC/ZMTP-specifications, dictates.
If we cannot guess or know beforehand, can we ... ?
Yes, we can experiment. Best setup a controlled distributed-system experiment.
Node A : The Sender
can be pythonic, being a sender:
- setup a REQ-archetype AccessNode ( details in "ZeroMQ Hierarchy in less than a five seconds" ),
- setup .setsockopt( zmq.IDENTITY, ... ) with a randomness-generated static identity,
- setup a .setsockopt( zmq.REQ_RELAXED, 1 ),
- .bind() it to a known endpoint of a transport-class of one's choice
- start an xrange()-generator controlled for L in xrange( 1, int( 1E+9 ) )-loop of .send()-s
- there load a payload .send( r"{0:}|{1:}".format( str( L ), L * r"*" ) )
- handle the respective .recv() for a REP-side "answer",
- check the error-states and adapt the time.sleep()-s / re-loops, according to the sender-side socket capacity and capability to send further payloads
Node B : The Receiver | The MitM-sniffer
ought be as low level as possible, so as to dis-assemble the RFC/ZMTP wire-line protocol, so python gets out of the candidate list. Other option may include a wire-level sniffer(s), if the selected transport-class permits ( an ipc:// or a vmci:// one will not )
setup a ROUTER-archetype AccessNode,
.connect() it to the know Node A's transport-class endpoint used,
start .recv()-ing messages,
If your experiment has correctly acquired / sniffed the wire-level details about the ZMTP-compliant transport sizes of the know payload compositions, your question gets repeatable, verifiable, quantitatively correct records on string-size to message-size mapping-function.
A BONUS POINT: for those indeed interested . . .Next, re-run the controlled white-box distributed-system experiment above, now with having the Node A: The Sender-side extended it's behaviour to also{ randomly | deterministically } { change | alter } its own configuration ( or map both such options onto a pair of the same payload re-.send()-s )with a.setsockopt( zmq.REQ_CORRELATE, { on | off } ) inside its for-loop and record the observed changes in the expected outputs.
This adds a final touch to the definitive answer, as far as the API v.4.2.x permits in the 2018-Q2.
I'm working on an IMAP client using Ruby and Rails. I can successfully import messages, mailboxes, and more... However, after the initial import, how can I detect any changes that have occurred since my last sync?
Currently I am storing the UIDs and UID validity values in the database, comparing them, and searching appropriately. This works, but it doesn't detect deleted messages or changes to message flags, etc.
Do I have to pull all messages every time to detect these changes? How do other IMAP clients do it so quickly (i.e. Apple Mail and Postbox). My script is already taking 10+ seconds per account with very few email addresses:
# select ourself as the current mailbox
#imap_connection.examine(self.location)
# grab all new messages and update them in the database
# if the uid's are still valid, we will just fetch the newest UIDs
# otherwise, we need to search when we last synced, which is slower :(
if self.uid_validity.nil? || uid_validity == self.uid_validity
# for some IMAP servers, if a mailbox is empty, a uid_fetch will fail, so then
begin
messages = #imap_connection.uid_fetch(uid_range, ['UID', 'RFC822', 'FLAGS'])
rescue
# gmail cries if the folder is empty
uids = #imap_connection.uid_search(['ALL'])
messages = #imap_connection.uid_fetch(uids, ['UID', 'RFC822', 'FLAGS']) unless uids.empty?
end
messages.each do |imap_message|
Message.create_from_imap!(imap_message, self.id)
end unless messages.nil?
else
query = self.last_synced.nil? ? ['All'] : ['SINCE', Net::IMAP.format_datetime(self.last_synced)]
#imap_connection.search(query).each do |message_id|
imap_message = #imap_connection.fetch(message_id, ['RFC822', 'FLAGS', 'UID'])[0]
# don't mark the messages as read
##imap_connection.store(message_id, '-FLAGS', [:Seen])
Message.create_from_imap!(imap_message, self.id)
end
end
# now assume all UIDs are valid
self.uid_validity = uid_validity
# now remember that we just fetched all those messages
self.last_synced = Time.now
self.save!
There is an IMAP extension for Quick Flag Changes Resynchronization (RFC-4551). With this extension it is possible to search for all messages that have been changed since the last synchronization (based on some kind of timestamp). However, as far as I know this extension is not widely supported.
There is an informational RFC that describes how IMAP clients should do synchronization (RFC-4549, section 4.3). The text recommends issuing the following two commands:
tag1 UID FETCH <lastseenuid+1>:* <descriptors>
tag2 UID FETCH 1:<lastseenuid> FLAGS
The first command is used to fetch the required information for all unknown mails (without knowing how many mails there are). The second command is used to synchronize the flags for the already seen mails.
AFAIK this method is widely used. Therefore, many IMAP servers contain optimizations in order to provide this information quickly. Typically, the network bandwidth is the limiting factor.
The IMAP protocol is brain dead this way, unfortunately. IDLE really should be able to return this kind of stuff while connected, for example. The FETCH FLAGS suggestion above is the only way to do it.
One thing to be careful of, however, is that UIDs are only valid for a given session per the spec. You should not store them, even if some servers persist them.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
My application sends SMS messages to people, but the numbers entered in as their cell phone are sometimes land lines (this is user error or the user not really knowing if the contact number they have is a cell phone or landline.)
I found a few websites that can tell me if a number is a landline or cell phone, but they do not offer programatic API's. Is anyone aware of a way a web application can figure out if a number can receive SMS messages?
I guess a test SMS message is one way, but my current SMS gateway fails hard when it gets a landline number and doesn't tell me the landline number it tried to send the SMS to. I'll follow this up with my carrier, but I would love an easy way to let the user entering phone numbers in if they are a landline or cell number.
Update:
There are ways to figure this out. Take a look at http://www.phonevalidator.com, they can query a phone number and figure out if it is a landline or cell phone.
You can have JavaScript open a popup with the url:
"http://www.phonevalidator.com/results.aspx?p=" + phoneNumber
It's not a free service, but a company called Targus (not the bag company) has an API for querying phone information. They can tell if it's landline or cell, even address validation. The charge based on how many queries you do (a few cents a query).
http://www.targusinfo.com/
Followup: I contacted TARGUSinfo on Feb. 24, 2011 and was told by their sales rep that they only work with record sets in the hundreds-of-thousands to millions and generally their customers are plugged into their API for real-time access. Differentiating between cell numbers and land line numbers for smaller record sets is "not something they can assist with."
I am afraid the only real way to find it out is contact SMS gateway service providers. A list of them can be found here for example:
http://en.wikipedia.org/wiki/SMS_gateways
Anyway, instead of that I suggest doing the following:
When user inserts cell phone number into his or her profile, send testing SMS to this number with confirmation code. If number is not verified by user, don't bother sending SMS messages to it later on.
Actually there are land line operators that allow receiving SMS. Either by text2voice gateway or phone terminal with extended capability. Moreover, in Europe cell phone operators have started offering "virtual land lines", which are in fact GSM cell phones assigned to one particular base station. But they do follow land line numbering scheme.
Resuming — not allowing sending SMS to land line number is wrong.
The following script returns
646-826-3879 LANDLINE
for the Stack Overflow hot line.
#!/usr/bin/perl -w
use strict;
use warnings;
use LWP::Simple;
use LWP::Simple::Cookies ( autosave => 1, file => "$ENV{'HOME'}/lwp_cookies.dat" );
my $usage = "Usage: $0 <phone_number>\n\nwhere phone_number is on format XXX-XXX-XXXX or XXXXXXXXX";
die $usage unless scalar #ARGV == 1;
my $number = shift #ARGV;
die $usage unless $number =~ /(\d\d\d)-?(\d\d\d)/;
my $NPA = $1;
my $NXX = $2;
#GET /search.asp?frmNPA=646&frmNXX=826&frmCity=&frmState=&frmZip=&frmCounty=&frmCompany=&search.x=0&search.y=0 HTTP/1.0
my $doc = get 'http://www.area-codes.com/search.asp?frmNPA=' . $NPA . '&frmNXX=' . $NXX . '&frmCity=&frmState=&frmZip=&frmCounty=&frmCompany=&search.x=0&search.y=0';
# html format:
#...
# <td><strong>NXX Use Type:</strong></td>
# <td>LANDLINE</td>
#...
my $next = 0;
my $result = "";
grep {
if (/NXX Use Type:/) {
$next = 1;
} else {
if ($next) {
$next = 0;
$result = $_;
}
}
} split(/\n/, $doc);
$result =~ /<[^>]*>(.*)<[^>]*>/;
print "$number\t$1\n";
I'm not sure if you want to do that really... If you want to tell if a number is reserved by a callphone or landline provider, you should be able to find the ranges in some documents from your country's telco supervising entity (not sure who does that in US - it might be http://www.nanpa.com/). Those documents are usually public.
But the problem is that mobile number != able to receive sms. With number porting and all the "unified communication" ideas nowadays you can easily have local numbers redirecting to mobiles, non-geographical numbers handling smses and local "special" numbers rewriting your incoming smses as facebook messages ;) For example my local "landline" number is redirected to a mobile in another country and couple of other locations.
You shouldn't be charged for a message sent to a nonexisting / otherwise strange number, so it might be a good way to check if someone can receive them. If you have a good control over the SMS gateway, you can send a message with delivery report active and expiry == immediate message (forgot the official name). Just send a test / welcome message - if it's not accepted, you can mark the number as unavailable. Otherwise, you can just ask for a "number that accepts SMSes" instead of a "cellphone number".
Trying to combine some answers here...
Daniel mentioned that
You can have JavaScript open a popup with the url:
"http://www.phonevalidator.com/results.aspx?p=" + phoneNumber
Does this still work? I can't seem to reproduce it. If we can get that to work, then maybe we can use a script like the following...
#!/usr/bin/perl -w
use strict;
use warnings;
use LWP::Simple;
my $usage = "Usage: $0 <phone_number>\n\nwhere phone_number is on format XXX-XXX-XXXX or XXXXXXXXX";
die $usage unless scalar #ARGV == 1;
my $number = shift #ARGV;
die $usage unless $number =~ /(\d\d\d)-?(\d\d\d)/;
my $NPA = $1;
my $NXX = $2;
#GET /search.asp?frmNPA=646&frmNXX=826&frmCity=&frmState=&frmZip=&frmCounty=&frmCompany=&search.x=0&search.y=0 HTTP/1.0
my $doc = get 'http://www.phonevalidator.com/results.aspx?p=' . $NPA . $NXX;
# html format:
#...
# <td class="style16">
# Phone Line Type:
# </td>
# <td class="style13">
# <span id="PhoneTypeLabel">LANDLINE</span>
# </td>
#
#...
my $result = "";
grep {
if (/PhoneTypeLabel/) {
$result = $_;
}
} split(/\n/, $doc);
$result =~ /<[^>]*>(.*)<[^>]*>/;
print "$number\t$1\n";
Another option would be to have the user select their cellphone provider from a list. Just a thought.
Why not make a list of the usual format for the landlines and mobile numbers? For example, where I'm located landlines always follow this format:
9xxxx xxxx
and mobiles are always:
04xx xxx xxx
You can easily make a list of the general format of landline/mobile numbers in the area you are serving. Then, when wanting to find out if a US number is landline or mobile, just compare it and see whether it matches the landline format or the mobile number format.
Another thing that is done often to validate phone numbers, is to send a temporary pin via SMS to the user's mobile, and ask them to enter that pin in order to validate their number.
I can imagine doing this, if I had access to the core mobile network. That's an SS7 API, though, not a Web API. I would bet that any service which offers a Web API acts as a proxy to the SS7 network, instead of relying on databases. The only other alternative would be to query the number porability database. Eventually, to terminate a call, all network operators need to determine which other operator they need to connect to.
we have a service which provides fixed line / cellular detection for USA and Canada, and also detects if the number is Live or Dead, which wireless carrier the number is ported to etc. There's a full web API.
If you sign up for an account at www.hlrcheck.com I'll get it validated and add some free credits for you to test. We're in beta at the moment, but full release candidate is imminent.
All solutions mentioned here are static number pattern lookups but the only reliable way to figure out whether a given cell phone number is valid (and ready to receive SMS) is to perform an HLR Lookup. HLR stands for Home Location Register, which is a mobile network operator database with real time subscriber information. There are several API services for HLR lookups, e.g. www.hlr-lookups.com.