In COSMOS is there an easy way to see the raw bytes being sent over the line by the Command Sender? We want to capture what is sent out and don't know where all the protocol layers get added.
If you go to the Cmd Packets tab in the Command and Telemetry Server you can click View Raw on the command packet you're sending to see the raw bytes. Note that these are the raw bytes in the defined command. If your interface changes the data in some way (puts on a CRC, etc) then the bytes over the line may be different.
The following page describes the COSMOS log format: https://cosmosrb.com/docs/logging/
You can also add the LOG_RAW keyword after your interface definition to log all the raw data going out over the interface. Note that COSMOS does not interpret this log and you'll have to use a hex editor to view it.
Related
I have data in SQLite that is text including a curly apostrophe (U-2019 ’) but the character is typed as in work’s. A browser makes the request for the data over a web socket connection which is configured as chan configure $sock -encoding iso8859-1 -translation crlf -buffering full simply because that is what I set it to before sending the headers to the browser in response to the request to upgrade the connection to a web socket and failed to consider it again thereafter.
I was mistaken it is endoded as chan configure $sock -buffering full -blocking 0 -translation binary.
If I use replace( ..., '’', '’') in the SQL then the browser displays the text with the curly apostrophe rendered normally. And if I follow the example in this SO question and encode the result of the SQL query as set encoded [encoding convertto utf-8 $SQLResult] it renders correctly also.
What is the correct way to handle this? Should the SQL results be endoded to utf-8 or should the web socket be configured differently? If the socket is encoded as utf-8 then it fails on the first message sent from the browser/client.
Thank you.
The data in SQLite should be Unicode (probably UTF-8 but that shouldn't matter; the package binding handles that). That means that the data arrives in Tcl correctly encoded. Tcl's internal encoding schemes are complicated and should be ignored for the purposes of this discussion: pretend that's all Unicode too.
With a websocket, you need to say what sort of messages ("frames") you are sending. For normal payloads, they can be either binary frames or text frames. Binary frames are a bunch of bytes. Text frames are UTF-8 (as specified by the standard). STOMP frames are text frames where the content is JSON (with a few basic rules and some higher-order things over what operations can be stated).
If you are sending text, you should use UTF-8. Ideally by configuring the socket to do that in the part of the websocket client that mediates between the point that receives the text to send (maybe JSON) and the part that writes the frame onto the underlying socket itself. Above that point, you should just work with ordinary Tcl strings. If you're forming JSON messages in Tcl, the rl_json package is recommended, or you can get SQLite to produce JSON directly. (JSON's serialized form is always UTF-8, and it is conceptually Unicode.)
If you are sending binary frames, you handle all the details yourself. Using encoding convertto utf-8 may help with forming the message frame body, or you can use any other encoding scheme you prefer (such as the one you mention in your question).
SGP.02 - Remote Provisioning Architecture for Embedded UICC Technical Specification (page 255 of v4.0 specifically) says:
the data format provided by the function caller SHALL NOT depend of the selected OTA protocol capabilities (for example SM-DP can consider there is no limit on data length)
and later
The SM-SR has the responsibility to build the final Command script,
depending on eUICC capabilities and selected protocol:
by adding the Command scripting template for definite or indefinite length,
and, if necessary, by segmenting the provided command script into several pieces
and, if necessary, by adding the relevant Script Chaining TLVs.
I understand it that SM-DP can send arbitrary long data parameter to ES3.SendData and SM-SR should send several APDUs in multiple SMSes if data is too large to fit into one. And that is meant by segmenting.
The problem is I can't find a relevant specification that defines how segmenting should be done. And that's the question: where is the segmenting process defined?
I may be wrong but it seems that is not the same as Concatenated Short Messages described in the section 6.3 of ETSI TS 123 048.
It seems that Script Chaining briefly mentioned in ETSI TS 102 226 is somewhat related so pointers to the specification that defines how it works are also very much welcome (TS 102 226 talks about Script Chaining TLVs but not how to use them, at least I'm definitely missing some broader context how it works so any hints are appreciated).
UPDATE:
ES8.EstablishISDPKeySet function requires 3 APDUs to be sent. And they are quite big as they contain keys. From SGP.02-v4.0 table 150 I understand it that they are sent from SM-DP to SM-SR using Expanded Remote Command Format. The script in this format can be rather large as far as I understand (given that SM-DP can assume that there is no limitation on the data length). And it is not clear how SM-SR should segment it or use chaining. I'm just missing the specs where this is described.
The eUICC has a limited internal buffer, i.e. it cannot store a complete profile package of 10kb or more internally. The message must be chunked. If the eUICC supports just e.g. 1 kb then you have to split the APDU commands after at most 3 APDU commands to stay below 1kb. The SGP.02 specification defines to have at least 1024 bytes. A fully featured SM-SR might store some attributes based on the eUICC vendor in the EID to add special handling and patches for certain eUICCs to support a larger buffer size.
Encode each APDUs chunk (1..n APDUs) into the Expanded Remote Command Format (ETSI TS 102 226, sect. 5.2.1) (compact format can only have one response for the last APDU, but if it works, you could save a few bytes)
Encode each Expanded Remote Commands message into a SMS-DELIVER (TS 123 048 and simalliance Interoperability Stepping Stones release 6) This includes the data encryption using the OTA keys (KiC, KID). gsm0348 is a good Java library for this. Pay attention: For each message the OTA counter must be incremented. Select one reference number and keep it for all messages. I guess this is the identifier how the eUICC knows which messages belong together.
If using SMS (I would suggest to use CAT-TP or HTTPs instead -> faster and more reliable) encode it as SMS-PP Download message (TS 131 111). You will using message concatenation here if the payload is more than 140 bytes.
You will receive as response a SendShortMessage (TS 131 111, 6.4.10). Extract the user data again with TS 123 048 and simalliance Interoperability Stepping Stones release 6. You get the SMS Response message. Look into the response packet to get the user data.
Extract the user data as Expanded Remote Response (ETSI TS 102 226)
The eUICC will handle the streamed messages. The concatenated short messages are only used for a message chunk belonging together during the transit.
The best specification that kind of explains how segmenting and script chaining works in details is SGP.11 Remote Provisioning Architecture for Embedded UICC Test Specification.
It does not have a requirements per se but it has the byte level examples of incoming ES3.SendData and examples of the messages received by eUICC. And this allows to deduce the actual behaviour of the SM-SR pretty easily.
And here is more detailed explanation with references to that specification.
Command script
Command script is a list of commands sent in the data fields of the ES3.SendData. It can be (see data field description in table 150 of SGP.02-v4.0) either:
the list of C-APDUs commands (defined in EXPANDED_COMMANDS method on page 595 of SGP.11-v4.0 and uses in step 2 on page 407 of SGP.11-v4.0)
TLV commands that form a SCP03t script (defined in SCP03T_SCRIPT method on page 600 of SGP.11-v4.0 and used in step 14 on page 408 of SGP.11-v4.0)
Script Chaining
Script chaining is the feature used when there is a need to execute several commands sent in multiple transport messages but when some context of the execution needs to preserved after the first command is executed and used for the next one (for example the first command is INSTALL [for personalization] and the second one is STORE DATA). This context is command session defined in section 4 of TS 102 226.
This is used at least in these cases (which is also supported by Annex K in SGP.02):
when the followup command is based on the response from eUICC
when command script is too large and cannot be sent in one piece
Script chaining is performed by SM-SR and is implemented by adding Script Chaining TLV to the command script in Expanded Remote format. See section 5.4.1.2 in TS 102 226.
Examples of script chaining can be found in SGP.11-v4.0.
For the first case see the scenario for EstablishISDRKeyset procedure in section 4.2.10.2.1.1 . For the second case see the scenario for DownloadAndInstallation procedure in section 4.2.18.2.1.1. The byte level representation of the script chaining is described in SCP80_PACKET method on page 601 (see option CHAINING_OPT).
It seems that explicit chaining is applicable only to SMS or CAT_TP transport. For HTTPS the command session coincides with the administrative session as defined in section 3.6 of Remote Application Management over HTTP – Public Release v1.1.3 so there is not need to add explicit Script Chaining TLVs.
Segmenting
When the command script is too large and cannot be sent in one piece SMSR generates multiple command scripts. The commands that SMSR received from SMDP CMD1, CMD2, …, CMDN (see step 14 on page 408 of SGP.11-v4.0) and that form a command script are split into several command scripts. The first one contains some initial portion of the commands CMD1, CMD2, …, CMDi. The second command script contains commands CMDi+1, CMDi+2, …, CMDj. The third one CMDj+1,…,CMDk and so on.
If SMS or CAT_TP is used then for every command a Chaining TLV is added to the beginning with the appropriate value (see section 4.2.18.2.1.1 in SGP.11-v4.0).
And then every new command script is sent in a separate message to eUICC.
For example I have protocol buffer file compressed in snappy-format
file.pbuf.sn
how can I view the file's content? Which programms are recommended to work with protocol buffers files?
There's two separate steps here:
un-snappy the file container
process the contents that are presumably protobuf
If you're trying to do this through code, then obviously each will depend on your target language/platform/etc. Presumably "snappy" tools are available from Google (who created "snappy", IIRC).
Once you have he contents, it depends whether it is a .proto schema, binary data contents, JSON data contents, or some combination. If you have a schema for the data, then run it through "protoc" or the language/platform-specific tool of your choice to get the generated code that matches the schema. Then you can run either binary or JASON data through that generated code to get a populated object model.
If you don't have a schema: if it is JSON you should be able to understand the data via the names. Just run it through your chosen JSON tooling
If it is binary data without a schema, things are tougher. Protobuf data doesn't include names and the same values can be encoded in multiple ways (so: the same bytes can have come from multiple sources values). So you'll have to reverse-engineer the meaning of each field. "Protoc" has a schema-less decode mode that might help with this, as does https://protogen.marcgravell.com/decode
I'm using google protocol buffer to serialize equity market data (ie. timestamp, bid,ask fields).
I can store one message into a file and deserialize it without issue.
How can I store multiple messages into a single file? Not sure how I can separate the messages. I need to be able to append new messages to the file on the fly.
I would recommend using the writeDelimitedTo(OutputStream) and parseDelimitedFrom(InputStream) methods on Message objects. writeDelimitedTo writes the length of the message before the message itself; parseDelimitedFrom then uses that length to read only one message and no farther. This allows multiple messages to be written to a single OutputStream to then be parsed separately. For more information, see https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/MessageLite#writeDelimitedTo(java.io.OutputStream)
From the docs:
http://code.google.com/apis/protocolbuffers/docs/techniques.html#streaming
Streaming Multiple Messages
If you want to write multiple messages to a single file or stream, it
is up to you to keep track of where one message ends and the next
begins. The Protocol Buffer wire format is not self-delimiting, so
protocol buffer parsers cannot determine where a message ends on their
own. The easiest way to solve this problem is to write the size of
each message before you write the message itself. When you read the
messages back in, you read the size, then read the bytes into a
separate buffer, then parse from that buffer. (If you want to avoid
copying bytes to a separate buffer, check out the CodedInputStream
class (in both C++ and Java) which can be told to limit reads to a
certain number of bytes.)
Protobuf does not include a terminator per outermost record, so you need to do that yourself. The simplest approach is to prefix the data with the length of the record that follows. Personally, I tend to use the approach of writing a string-header (for an arbitrary field number), then the length as a "varint" - this means the entire document is then itself a valid protobuf, and could be consumed as an object with a "repeated" element, however, just a fixed-length (typically 32-bit little-endian) marker would do just as well. With any such storage, it is appendable as you require.
If you're looking for a C++ solution, Kenton Varda submitted a patch to protobuf around August 2015 that adds support for writeDelimitedTo() and readDelimitedFrom() calls that will serialize/deserialize a sequence of proto messages to/from a file in a way that's compatible with the Java version of these calls. Unfortunately this patch hasn't been approved yet, so if you want the functionality you'll need to merge it yourself.
Another option is Google has open sourced protobuf file reading/writing code through other projects. The or-tools library, for example, contains the classes RecordReader and RecordWriter that serialize/deserialize a proto stream to a file.
If you would like stand-alone versions of these classes that have almost no external dependencies, I have a fork of or-tools that contains only these classes. See: https://github.com/moof2k/recordio
Reading and writing with these classes is straightforward:
File* file = File::Open("proto.log", "w");
RecordWriter writer(file);
writer.WriteProtocolMessage(msg1);
writer.WriteProtocolMessage(msg2);
...
writer.Close();
An easier way is to base64 encode each message and store it as a record per line.
I've checked out a few of the forum posts here and can't find quite what I'm looking for. Suppose you are reading in a text document via Ruby. I understand the stream is essentially the characters coming in byte by byte. What is the purpose/best practice of buffering in this case? My book shows plenty examples of the buffer being utilized, but no real description of what the buffer is or why it even exists. What should I be considering when setting the buffer? For example, the book illustrates the following method as:
read(n, buffer=nil) reads in n bytes, until the bytes are ready
I don't understand what the statement "until the bytes are ready" means. Does the buffer play a role in this? Please feel free to point me to another place where this is explained, I couldn't for the life of me find it on my own.
IO can be not only file, but a network socket. and in networks you regularly have a situation where you are ready to process more data, but the remote side have a pause in data sending.
(You usually see a progress bar or a spinner element in your browser in these cases)
So, if you are using regular files, the bytes are always 'ready'.
The Picaxe book for IO#read says:
Reads at most int bytes from the I/O stream or to the end of file if int is omitted. Returns nil if called at end of file. If buffer (a String) is provided, it is resized accordingly, and input is read directly into it.