I am trying to implement packet sniffing for traffic from an Outlook client to an Exchange server. This communication uses MAPI over HTTPS (Outlook anywhere) as far as I can tell. I don't think I am using MAPI over RPC.
Traffic comes across as TLS which I can decode in wireshark as I have the appropriate certificate loaded. I cannot view the original content of the mail I sent during my test, it is still encoded.
How can I decode messages so I can read the original content in plain text?
I am creating temporary files for each stream that is handled and storing them unchunked and ssl-decoded. Example below. If I view said files in less they appear to be binary:
PROCESSING
DONE
X-StartTime: Fri, 16 Dec 2016 23:17:12 GMT
X-ElapsedTime: 5
^#^#^#^#^#^#^#^#^#^#^#^#^B^#^#^#^#^E^#<9B>^B^A^E<80><9F>^#^D^D^C^A^#^#^#^A^A^Ctestets)^#s^#Y^#^Q^#h^#^V^#p^#^#^A^#^Q^#^B^A^#^E^#^#^O^B^A
^C^#^#9^C^#^O^C^#^E9^^#^X^#^X^#^#:^C^#q:^B^A^#^#^U^D^A=^C^#^X^A^^#8^#^X^#X^#^A`^^#f^A^A^D^#^#>^D
^F^#^H^#^#^#^#/o=ExchangeLabs/ou~^# Administra^#^#^#^#tive Group (FYDIBOHF23SPDLT)/cn=^#^# ^#Recipientsi^#cfb4ddc8c1ba4733a3d23^#^#^#^#4e1321845da-shayne.civi^#S^#h^#a^#y^#X^U^#n^#e^# ^#C^#i^#v^X^#t<98>^#r<88>^#s^X^#^#(^#^^A<80>.^#c^#^^A^Q^#^#<9D>0^#^#w^#^#^#ܧ#B^P^Z^H^#+/<82>*^K^#^#O^Gh<9D> ^D^D<82>^A^#<80>^P^A^##<87>^K #^#w(^#^#^Zd^#g(^#)^Atx^#o^#k^#<89>^Ao^#m#^B?^B ^^A^M
^M^A^#^P^#&^?_^ON<98> h5^GZ]w<83>~^#^B^#^W^R^K5^Ko^C ^F^M/8^#=^#E^#x<88>^#<89>^Bnz^ALH^#b8^A^#u^#i ^#A^##
^Bm<88>^Cn^X^#s^Br^#tX^#v^#Gx^#^ApX^#(^#F^#Y^#D^#I^#B^#O^#bESC^EEHh^#2^#3X^GP<98>^#L^#T^#)(^Cc8^B=^#R^AI^Fp^X^#<99>^Et^B^#c^#f<88>^D4GU^CdX^#8^X^#1x^#a<88>^#7h^B3H^#3^#^B4^A18^#2(^#8h^#5^#a^#-X^B^E<8F> ^A^#^#^P^#^#^G<93>
^W^#^Vs l<82>/<8A>MR]Oˌ^Sx^#^#^DUB^V^#^#^L^#^#^#^#^A^C^#^#^A^#^#^#^#
I have read through all documentation https://msdn.microsoft.com/en-us/library/cc425499(v=exchg.80).aspx about these protocols I could find.
EDIT:
With the use of a program called fiddler and it's Office Inspectors, specifically MAPIInspector, I am able to view almost all the information I need.
https://github.com/OfficeDev/Office-Inspectors-for-Fiddler
I am able to view in plain text: subject, recipient list, sender information, file attachment names, file attachment contents and much more, but I still cannot find the body of the message.
I believe the message is stored in:
ExecuteRequestBody->ROPBuffer->Payload->ROPList->ROPWriteStreamRequest->Data.
I believe the ROPWriteStreamRequest is what I need.
The content is encrypted and/or obfuscated. I was able to find the obfuscation algorithm for RPC which is XOR by 0xA5, but I am not sure if this is done before or after compression. I suspect the compression algorithm is LZ77.
The ExecuteRequestBody->ROPBuffer->Payload->ROPList->ROPWriteStreamRequest->Data saved the attachment content. The email content is stored in RopSetPropertiesRequest->PropertyValues->the 11th TaggedPropertyValue (whose property tag is PidTagBodyHtml). The content is html format.
Related
My question is in the title, this provides context to help you understand my confusion. Everything is sent over https.
My understanding of base 64 encoding is that it is a way of representing binary data as text, such that the text is safe to transmit across networks or the internet because it avoids anything that might be interpreted as a control code by the various possible protocols that might be involved at some point.
Given this understanding, I am confused why everything sent to over the internet is not base 64 encoded. When is it safe not to base 64 encode something before sending it? I understand that not everything understands or expects to receive things in base 64, but my question is why doesn't everything expect and work with this if it is the only way to send data without the possibility it could be interpreted as control codes?
I am designing an Android app and server API such that the app can use the API to send data to the server. There are some potentially large SQLite database files the client will be sending to the server (I know this sounds strange, yes it needs to send the entire database files). They are being gzipped prior to uploading. I know there is also a header that can be used to indicate this: Content-Encoding: gzip. Would it be safe to compress the data and send it with this header without base 64 encoding it? If not, why does such a header exist if it is not safe to use? I mean, if you base 64 encode it first and then compress it, you undo the point of base 64 encoding and it is not at that point base 64 encoded. If you compress it first and then base 64 encode it, that header would no longer be valid as it is not in the compressed format at that point. We actually don't want to use the header because we want to save the files in a compressed state, and using the header will cause the server to decompress it prior to our API code running. I'm only asking this to further clarify why I am confused about whether it is safe to send gzip compressed data without base 64 encoding it.
My best guess is that it depends on if what you are sending is binary data or not. If you are sending binary data, it should be base 64 encoded as the final step before uploading it. But if you are sending text data, you may not need to do this. However it still seems to my logic, this might still depends on the character encoding used. Perhaps some character encodings can result in sending data that could be interpreted as a control code? If this is true, which character encodings are safe to send without base 64 encoding them as the final step prior to sending it? If I am correct about this, it implies you should only use the that gzip header if you are sending compressed text that has not been base 64 encoded. Does compressing it create the possibility of something that could be interpreted as a control code?
I realize this was rather long, so I will repeat my primary questions (the title) here: Is either Gzip compressed binary data or uncompressed text safe to transmit, or should it be base 64 encoded as the final step before sending it? Okay I lied there is one more question involved in this. Would sending gzip compressed text always be safe to send without base 64 encoding it at the end, no matter which character encoding it had prior to compression?
My understanding of base 64 encoding is that it is a way of representing binary data as text,
Specifically, as text consisting of characters drawn from a 64-character set, plus a couple of additional characters serving special purposes.
such that the text is safe to transmit across networks or the internet because it avoids anything that might be interpreted as a control code by the various possible protocols that might be involved at some point.
That's a bit of an overstatement. For two endpoints to communicate with each other, they need to agree on one protocol. If another protocol becomes involved along the way, then it is the responsibility of the endpoints for that transmission to handle any needed encoding considerations for it.
What bytes and byte combinations can successfully be conveyed is a matter of the protocol in use, and there are plenty that handle binary data just fine.
At one time there was also an issue that some networks were not 8-bit clean, so that bytes with numeric values greater than 127 could not be conveyed across those networks, but that is not a practical concern today.
Given this understanding, I am confused why everything sent to over the internet is not base 64 encoded.
Given that the understanding you expressed is seriously flawed, it is not surprising that you are confused.
When is it safe not to base 64 encode something before sending it?
It is not only safe but essential to avoid base 64 encoding when the recipient of the transmission expects something different. The two or more parties to a given transmission must agree about the protocol to be used. That establishes the acceptable parameters of the communication. Although Base 64 is an available option for part or all of a message, it is by no means the only one, nor is it necessarily the best one for binary data, much less for data that are textual to begin with.
I understand that not everything understands or expects to receive things in base 64, but my question is why doesn't everything expect and work with this if it is the only way to send data without the possibility it could be interpreted as control codes?
Because it is not by any means the only way to avoid data being misinterpreted.
They are being gzipped prior to uploading. I know there is also a header that can be used to indicate this: Content-Encoding: gzip. Would it be safe to compress the data and send it with this header without base 64 encoding it?
It would be expected to transfer such data without base-64 encoding it. HTTP(S) handles binary data just fine. The Content-Encoding header tells the recipient how to interpret the message body, and if it specifies a binary content type (such as gzip) then binary data conforming to that content type are what the recipient will expect.
My best guess is that it depends on if what you are sending is binary data or not.
No. These days, for all practical intents and purposes, it depends only on what application-layer protocol you are using for the transmission. If it specifies that some or all of the message is to be base-64 encoded (according to a particular base-64 scheme, as there are more than one) then that's what the sender must do and how the receiver will interpret the message. If the protocol does not specify that, then the sender must not perform base-64 encoding. Some protocols afford the sender the option to make this choice, but those also provide a way for the sender to indicate inside the transmission what choice has been made.
Is either Gzip compressed binary data or uncompressed text safe to transmit, or should it be base 64 encoded as the final step before sending it?
Neither is inherently unsafe to transmit on today's networks. Whether data are base-64 encoded for transmission is a question of agreement between sender and receiver.
Okay I lied there is one more question involved in this. Would sending gzip compressed text always be safe to send without base 64 encoding it at the end, no matter which character encoding it had prior to compression?
The character encoding of the uncompressed text is not a factor in whether a gzipped version can be safely and successfully conveyed. But it probably matters for the receiver or anyone to whom they forward that data to understand the uncompressed text correctly. If you intend to accommodate multiple character encodings then you will want to provide a way to indicate which applies to each text.
Lets say somebody downloads multiple segments of my widevine DRM stream :
Segment01.m4s
Segment02.m4s
Segment...n.m4s
And he somehow got a hold of the decoded initialization header data for that particular stream, what is he able to do with it? Can he use it to decode the segments he downloaded? If not, what are possible ways he could do it, and how do i prevent such thing from happening.
Your question title and your question text ask slightly different things - in the question text you just mention the attacker having the initialisation data, but in the title you also mention the attacker getting hold of the license key itself.
As others have mentioned the initialisation data, which will include the key-id, the initialisation vector etc, is already available in the stream itself.
They key itself is not available - the player must pass the key-id to the DRM system or to the application and it will then request the license key associated with that key-id from the license server. They key which is returned will itself be encrypted so only the DRM system can decrypt it and access it.
If your attacker did have the unencrypted key itself, then they would effectively have bypassed the entire DRM system and they would indeed be able to decode the stream.
We would like to transfer a XML to a WEB API that can accept text as well as binary data.
What is the best way to transfer it in terms of traffic size?
Is it better to transfer it as clear text or as Stream of Binary data?
If you are concerned that the XML data you want to transfer is too large, then you can try using compression, gzip compression being the most popular. Web API has some built-in functionality for this but you could also "roll your own" if you like, for example if you want a different compression algorithm.
Fortunately, there's plenty of code around to help with compressing and decompressing your data stream. Take a look at the following:
MS nuget: https://www.nuget.org/packages/Microsoft.AspNet.WebApi.MessageHandlers.Compression/
http://benfoster.io/blog/aspnet-web-api-compression (blog article with a link to GitHub code)
https://github.com/benfoster/Fabrik.Common/tree/master/src/Fabrik.Common.WebAPI (the GitHub code mentioned above)
(SO) Compression filter for Web API
Finally, you could consider using Expect: 100-Continue. If an API client is about to send a request with a large entity body, like a POST, PUT, or PATCH, they can send “Expect: 100-continue” in their HTTP headers, and wait for a “100 Continue” response before sending their entity body. This allows the API server to verify much of the validity of the request before wasting bandwidth to return an error response (such as a 401 or a 403). Supporting this functionality is not very common, but it can improve API responsiveness and reduce bandwidth in some scenarios. (RFC2616 §8.2.3).
While I appreciate an answer full of links can be problematic if those links go out-of-date or get deleted, explaining Web API compression here is just too large a subject. I hope my answer steers you in a useful direction.
I got a question, since i know that mail providers look up on many different particulars in emails and then decides if email is spam or not,
do you think it's safe to send if i got:
inside the email body, is this assumed for spam?
Or is it safe to use base64 encoded images in email body , instead of using link to the image source?
thanks
Nope, I send a ton of emails with images in the body which regularly have a spamassassin score of 0.3 out of 4. Use your regular <img src="x.jpg"/>. Just make sure you have a good amount of real text.
If you're a 'trusted sender', meaning you send a high volume of emails that people don't mark as spam, you can get away with all images. Examples of companies that do this off the top of my head are Seamless Web and Lego.
I could see base64 code either being stripped or setting off the spam alarm though, based on how many hack scripts there are that use base64 to obfuscate code and then run it.
I'm new to AJAX, but as an overview I'd like to know what formats you can upload and download. Is it limited to JSON or XML or can you even send binary types like MP3 or UTF-8 HTML. And finally, do you have full control over the data, byte for byte in something like a byte array, or is only a string sent/received.
If we are talking about ajax we are talking about javascript? And about XMLHTTPRequest?
The XMLHttpRequest which is only a http request can transfer everything. But there is no byte array in javascript. Only strings, numbers and such. Every thing you get from an ajax call is a piece of text (responseText). That might be parsed into XML (which gives you reponseXML). Special encodings should be more a matter of the http transport.
The binary stuff is not ajax dependent but javascript dependent. There are some weird encodings for strings to deliver byte data inside in javascript (especially for images) but it is not a general solution.
HTML is not a problem and that is the most prominent use case. From this type of request you get an HTML string delivered and that is added to some node in the DOM per innerHTML that parses the HTML.
Since data is transported via HTTP you will have to make sure that you use some kind of encoding. One of the most popular is base64 encoding. You can find more information at: http://www.webtoolkit.info/javascript-base64.html
The methodology is to base64-encode the data you would like to send and then base64-decode the data at the server(or the client) and use the original data as you intended.
You can transfer any type of data either string or bytes
You can send anything you like, the problem may be how to handle it once you get it ;)
Standard HTML is probably the most common type of ajax content in use out there - you can choose character encoding too, although it's always best to stick with one type of encoding.
AJAX simply means you're transferring data asynchronously over HTTP with a JavaScript call. So your script makes a "normal" HTTP request using the XmlHttpRequest() object. However, as the name implies, it's really only suited for text-based data formats since you generally want to perform some action on the client side with the data you got back from the server (not always though, sometimes people just send XmlHttpRequests only to update something on the server).
On a side note, I have never seen an application where sending binary data would have been appropriate anyway.
Most often, people choose to send data over to the server with POST or GET (which is basically a method to transfer name-value pairs inherent to HTTP). For sending more complex data, for example hierarchical structures, they need to be encoded somehow. XML documents can be made natively per JavaScript, sent over to the server and get parsed into whatever data types necessary. But since XML can be a bit of a pain, many devs use JSON encoded data instead because it's easy to generate and easy to parse.
What the server sends back is equally as arbitrary. Usually, you specify a callback function in your Javascript that handles the incoming data. Again, the popular choices are XML and JSON, they parse easily into a document object or an array structure respectively. You could also send plain text or some other packaging but remember that you then have to take care of extracting the usable data from it yourself. Sometimes, it can also be beneficial to send actual HTML fragments to the client to update something on the page directly.
For starters, I suggest you have a look at JQuery. It's a very lightweight framework that abstracts many of evil compatibility stuff and lets you write AJAX requests very nicely.
You can move anything that can be sent over HTTP. There are restrictions about the call being made to the same domain as the page loaded from, but not on the content of the transfer. You can do either GET or POST transactions too.
There is a Digg the Blog entry titled DUI.Stream and MXHR that shows off what they call "Multipart XMLHttpRequests." It is alpha code now, but there is a demo that handles images.