Does telnet sub negotiation command without IAC SE is valid command?
No, the IAC, SE sequence indicates the end of the response (and the supplied value).
See Documentation
Like #Robert Bradley says, a sub-option negotiation which does not end with <IAC><SE> is NOT valid.
However there is one unofficial exception, which, because it was broken like that, got replaced, but theoretically could be seen in the wild. That is the original version of the Mud Client Compression Protocol (MCCP) - using sub-option number 85 which uses zlib compression to reduce the amount of data bytes to be sent from a MUD (Multi-User Dungeon) Game Server to the players' Clients. That mistakenly has the Server using <IAC><SB><85><WILL><SE> at the exact point in the data where compression begins. This defect was so significant that the protocol was revised to version 2 - which is otherwise identical except that it uses the sub-option number 86 instead - so that the sequence to begin that (after both server and client have agreed) is the correctly formed: <IAC><SB><86><IAC><SE>.
MUD server and client applications are now to prefer MCCP2 over MCCP1 (and obviously not agree to do the latter if the former has already been negotiated). See also: https://smaugmuds.afkmods.com/mccp/protocol.html.
Related
From section 5.4.4 of SGP.02-v4.2:
The SM-SR has the responsibility to build the final Command script, depending on eUICC capabilities and selected protocol
It is clear that in some cases SM-SR should split the command script that it received in data field of ES3.SendData. This is supported by examples from SGP.11-v4.0: compare step 14 on page 408 where complete profile package #PROFILE_PACKAGE is passed in ES3.SendData. with steps 3, 5 and 7 on pages 211-212 where segments in the form of #PROFILE_PARTi are sent via ES5.
First of all it is not clear what factors limit the capability to send the whole command script to eUICC. This question is important because the answer to it most probably would allow to answer the most important question: how exactly SMSR decides (as in what parameters it takes into account) how to split the command script.
I know about EUICC-CAPABILITIES but they only specify what transports/algorithms/features are supported. They do not give any hint of how large the single segment (that is #PROFILE_PARTi) of the command script built by SMSR can be. Also specification seems to distinguish eUICC capabilities from selected protocol and that's another factor that makes me think that capabilities mentioned in 5.4.4 are not the same as EUICC-CAPABILITIES.
SGP.11-v4.0 shows that even for CAT_TP or HTTPS segmenting should be used and SMSR somehow needs to decide how to split the command script it received.
So questions:
What eUICC capabilities should be taken into account by SMSR?
How exactly SMSR should use the capabilities to decide the size of the script segment (like #PROFILE_PARTi) it can use?
How SMSR can retrieve those capabilities for a given eUICC?
Can they be retrieved via means defined in some GP specification or is this manufacturer specific?
According to section 4.1.3.3 of SGP.02-v4.2 eUICC SHALL support profile command data segments of at least up to 1024 bytes (including tag and length field). Does it mean that:
SMDP will never create profile command data TLVs with code ‘86’ longer than 1024 bytes?
SMSR is compliant if it sends one command TLV at a time? (This way SMSR does not utilize the capabilities of the eUICC but this algorithm will work for any compliant eUICC regardless of eUICC’s capabilities.)
SM-SR:
Take a look into "EUICC-CAPABILITIES", sect. 5.1.1.2.10 for the supported transport features. For your implementation e.g. also the supported buffer size might be interesting to send as less concatenated SMS as possible or have a better TCP payload size. This must be implicitly known by reading the EID and know the features of the eUICC by interpreting the model and manufacturer. Not sure if yome eUICC are offering here more than the minimum.
2.+3.+4. Store the EIS when the eUICC is registered. (the SM-SR could have it in the database). This contains the EUICC-CAPABILITIES.
SM-DP:
Yes. Including the tag and length, MAC and padding. The profile package is sliced in 1020 byte chunks and then prefixed with the 86 tag and the 3 bytes length.
Yes, most likely, but like mentioned above a streaming protocol like CAT_TP or HTTPS does not have the problems of SMS.
SGP.02 - Remote Provisioning Architecture for Embedded UICC Technical Specification (page 255 of v4.0 specifically) says:
the data format provided by the function caller SHALL NOT depend of the selected OTA protocol capabilities (for example SM-DP can consider there is no limit on data length)
and later
The SM-SR has the responsibility to build the final Command script,
depending on eUICC capabilities and selected protocol:
by adding the Command scripting template for definite or indefinite length,
and, if necessary, by segmenting the provided command script into several pieces
and, if necessary, by adding the relevant Script Chaining TLVs.
I understand it that SM-DP can send arbitrary long data parameter to ES3.SendData and SM-SR should send several APDUs in multiple SMSes if data is too large to fit into one. And that is meant by segmenting.
The problem is I can't find a relevant specification that defines how segmenting should be done. And that's the question: where is the segmenting process defined?
I may be wrong but it seems that is not the same as Concatenated Short Messages described in the section 6.3 of ETSI TS 123 048.
It seems that Script Chaining briefly mentioned in ETSI TS 102 226 is somewhat related so pointers to the specification that defines how it works are also very much welcome (TS 102 226 talks about Script Chaining TLVs but not how to use them, at least I'm definitely missing some broader context how it works so any hints are appreciated).
UPDATE:
ES8.EstablishISDPKeySet function requires 3 APDUs to be sent. And they are quite big as they contain keys. From SGP.02-v4.0 table 150 I understand it that they are sent from SM-DP to SM-SR using Expanded Remote Command Format. The script in this format can be rather large as far as I understand (given that SM-DP can assume that there is no limitation on the data length). And it is not clear how SM-SR should segment it or use chaining. I'm just missing the specs where this is described.
The eUICC has a limited internal buffer, i.e. it cannot store a complete profile package of 10kb or more internally. The message must be chunked. If the eUICC supports just e.g. 1 kb then you have to split the APDU commands after at most 3 APDU commands to stay below 1kb. The SGP.02 specification defines to have at least 1024 bytes. A fully featured SM-SR might store some attributes based on the eUICC vendor in the EID to add special handling and patches for certain eUICCs to support a larger buffer size.
Encode each APDUs chunk (1..n APDUs) into the Expanded Remote Command Format (ETSI TS 102 226, sect. 5.2.1) (compact format can only have one response for the last APDU, but if it works, you could save a few bytes)
Encode each Expanded Remote Commands message into a SMS-DELIVER (TS 123 048 and simalliance Interoperability Stepping Stones release 6) This includes the data encryption using the OTA keys (KiC, KID). gsm0348 is a good Java library for this. Pay attention: For each message the OTA counter must be incremented. Select one reference number and keep it for all messages. I guess this is the identifier how the eUICC knows which messages belong together.
If using SMS (I would suggest to use CAT-TP or HTTPs instead -> faster and more reliable) encode it as SMS-PP Download message (TS 131 111). You will using message concatenation here if the payload is more than 140 bytes.
You will receive as response a SendShortMessage (TS 131 111, 6.4.10). Extract the user data again with TS 123 048 and simalliance Interoperability Stepping Stones release 6. You get the SMS Response message. Look into the response packet to get the user data.
Extract the user data as Expanded Remote Response (ETSI TS 102 226)
The eUICC will handle the streamed messages. The concatenated short messages are only used for a message chunk belonging together during the transit.
The best specification that kind of explains how segmenting and script chaining works in details is SGP.11 Remote Provisioning Architecture for Embedded UICC Test Specification.
It does not have a requirements per se but it has the byte level examples of incoming ES3.SendData and examples of the messages received by eUICC. And this allows to deduce the actual behaviour of the SM-SR pretty easily.
And here is more detailed explanation with references to that specification.
Command script
Command script is a list of commands sent in the data fields of the ES3.SendData. It can be (see data field description in table 150 of SGP.02-v4.0) either:
the list of C-APDUs commands (defined in EXPANDED_COMMANDS method on page 595 of SGP.11-v4.0 and uses in step 2 on page 407 of SGP.11-v4.0)
TLV commands that form a SCP03t script (defined in SCP03T_SCRIPT method on page 600 of SGP.11-v4.0 and used in step 14 on page 408 of SGP.11-v4.0)
Script Chaining
Script chaining is the feature used when there is a need to execute several commands sent in multiple transport messages but when some context of the execution needs to preserved after the first command is executed and used for the next one (for example the first command is INSTALL [for personalization] and the second one is STORE DATA). This context is command session defined in section 4 of TS 102 226.
This is used at least in these cases (which is also supported by Annex K in SGP.02):
when the followup command is based on the response from eUICC
when command script is too large and cannot be sent in one piece
Script chaining is performed by SM-SR and is implemented by adding Script Chaining TLV to the command script in Expanded Remote format. See section 5.4.1.2 in TS 102 226.
Examples of script chaining can be found in SGP.11-v4.0.
For the first case see the scenario for EstablishISDRKeyset procedure in section 4.2.10.2.1.1 . For the second case see the scenario for DownloadAndInstallation procedure in section 4.2.18.2.1.1. The byte level representation of the script chaining is described in SCP80_PACKET method on page 601 (see option CHAINING_OPT).
It seems that explicit chaining is applicable only to SMS or CAT_TP transport. For HTTPS the command session coincides with the administrative session as defined in section 3.6 of Remote Application Management over HTTP – Public Release v1.1.3 so there is not need to add explicit Script Chaining TLVs.
Segmenting
When the command script is too large and cannot be sent in one piece SMSR generates multiple command scripts. The commands that SMSR received from SMDP CMD1, CMD2, …, CMDN (see step 14 on page 408 of SGP.11-v4.0) and that form a command script are split into several command scripts. The first one contains some initial portion of the commands CMD1, CMD2, …, CMDi. The second command script contains commands CMDi+1, CMDi+2, …, CMDj. The third one CMDj+1,…,CMDk and so on.
If SMS or CAT_TP is used then for every command a Chaining TLV is added to the beginning with the appropriate value (see section 4.2.18.2.1.1 in SGP.11-v4.0).
And then every new command script is sent in a separate message to eUICC.
I am testing with ab - Apache HTTP server benchmarking tool.
Can I increase the size of the request in Apache (now I see 1 request has the size of 146 bytes).
I tried to increase the size of TCP send/receive buffer (-b option), but It seems does not work. Because I still see the "Total transferred" is 146 bytes.
Do you know any way to increase the size of the request? (change the source code or something).
Or if it is impossible, can you give me a suggestion about some tools which are similar to ab but it can increase the size of the request.
Thank you so much!
Although -b option does seem like it should've worked, I can't say for sure as I haven't used.
Alternatively, have you tried sending a dummy large file in your POST request? that can be accomplished with the -p option followed by a plain text file for instance, that you can either create yourself or Google for something like "generate large file in bytes online" that you can download and pass into the command.
As far as alternatives go, I've heard another open source project called httpperf from HP to be a great option as well, though I doubt we're unable to figure it out how to do it with Apache Benchmark.
I understand the difference between ASCII mode vs binary when it comes to FTP, but what I don't understand is why there is even a need for ASCII mode at all? Is this just a legacy thing that used to save time by eliminating the most significant bit, therefore causing the overall speed of the transfer to increase by 1/8th? Or is there some hidden use for it that I don't know about?
I've encountered many problems because I would forget to switch the mode to bin when transferring text between different OS's. I don't understand why "bin" isn't just the default for everything, especially with today's much faster internet speeds.
Knowwutimean, Vern?
ASCII mode exists so you can get the right answer when you upload a text file to a remote system without having to know what the line termination or character set conventions are for that system. It was more important when transferring text files was more often done via FTP than, say, email.
To address your practical problem: check the documentation for both your FTP client and server(s) to see if there's a way to set ASCII mode by default. Often this is as simple as some kind of "profile" that sends some FTP commands every time you connect.
To address your philosophical problem: FTP is a 40 year old protocol that has its fair share of historical baggage. One day you'll be very glad that some protocol you depend on was standardized long ago and you can still access some old data.
I, for one, vote to eliminate ascii mode from ftp servers. Any EOL translation can be done by applications consuming the files, and many apps today understand both EOL types anyway. At a minimum, I'd like to see servers switch to using binary by default, and only use ascii if requested.
One scenario of practical use of ASCII mode is to upload PHP or Perl or similar scripts from Windows development machine to Unix server. Use of Binary mode would require separate conversion of line ending sequences, while with ASCII mode conversion is performed "automatically".
Update: there's one more scenario that we have come across - when transferring data to/from mainframes that use EBCDIC encoding, ASCII mode tells the server to perform conversion between encodings.
Here's a practical example of a problem that comes from using a binary FTP connection. In php there are two types of comments:
// a single line comment like this
/* a block comment like this */
The block comment has a start and an end. But the single line comment just ends at the end of the line.
If you upload a php file with single line comments using a binary connection, the php will stop running as soon as it hits the single line comment. It doesn't recognise the end of the line as the end of the comment, so it effectively comments out the rest of your php script.
If however you use FTP in ASCII mode, it will correctly read the end of the line and will run your php code as expected.
I am using this line to send a message via a Ruby (1.8.7) socket:
##socket.send login_message, 0
(This works fine)
What is the second parameter for? I can't find the send method in the Ruby API docs...
I first thought that it was some C style length of the message. This is why I used login_message.length as second parameter. That worked but I encountered a strange behavior:
Everything works fine when the second parameter is a odd number. If it's an even number the last character gets lost at receiving on the other side (The other side is a C++ program with a C socket). I inspected the network traffic with Wireshark and noticed that the packets look good. All the data is complete. Why is the last character lost when I receive it?
Thank you
Lennart
This is the flags parameter, the same as the last parameter to the send() system call. Normally it should be 0, but may be something like Socket::MSG_OOB (to send out-of-band data). For Ruby 1.9 this is documented under BasicSocket.