I have built a simple project which use "Winsock" Tool.
When I receive any data I put it in a variable because i cann't put it in a textbox because
it is a file not a text.
But if i send a big file it gets me an error.
"Overflow"
Are there any way to fix this problem ?
A VB variable-length string can only in theory be 2GB in size, it's actual maximum size is depending on available virtual memory which is also limited to 2GB for the entire application. But since VB stores the string in unicode format it means that it can only contain 1GB of text.
(maximum length for string in VB6)
If this is your problem, try splitting incoming data by several strings.
Are you handling the SendComplete event properly before sending more data?
Otherwise you will get a buffer overflow from the WinSock control.
You need to split your data into smaller packets (around 2-5k each should do it) and send each packet individually, then re-construct your packets at the other end. You could add a unique character at the end of the data so that the receiving end know that all the data has been received for that transmission say Chr(0)?
This is quite a simplified solution to this problem - a better method would be to devise a simple protocol for data handshaking so you know each packet has been received.
Related
My question is in the title, this provides context to help you understand my confusion. Everything is sent over https.
My understanding of base 64 encoding is that it is a way of representing binary data as text, such that the text is safe to transmit across networks or the internet because it avoids anything that might be interpreted as a control code by the various possible protocols that might be involved at some point.
Given this understanding, I am confused why everything sent to over the internet is not base 64 encoded. When is it safe not to base 64 encode something before sending it? I understand that not everything understands or expects to receive things in base 64, but my question is why doesn't everything expect and work with this if it is the only way to send data without the possibility it could be interpreted as control codes?
I am designing an Android app and server API such that the app can use the API to send data to the server. There are some potentially large SQLite database files the client will be sending to the server (I know this sounds strange, yes it needs to send the entire database files). They are being gzipped prior to uploading. I know there is also a header that can be used to indicate this: Content-Encoding: gzip. Would it be safe to compress the data and send it with this header without base 64 encoding it? If not, why does such a header exist if it is not safe to use? I mean, if you base 64 encode it first and then compress it, you undo the point of base 64 encoding and it is not at that point base 64 encoded. If you compress it first and then base 64 encode it, that header would no longer be valid as it is not in the compressed format at that point. We actually don't want to use the header because we want to save the files in a compressed state, and using the header will cause the server to decompress it prior to our API code running. I'm only asking this to further clarify why I am confused about whether it is safe to send gzip compressed data without base 64 encoding it.
My best guess is that it depends on if what you are sending is binary data or not. If you are sending binary data, it should be base 64 encoded as the final step before uploading it. But if you are sending text data, you may not need to do this. However it still seems to my logic, this might still depends on the character encoding used. Perhaps some character encodings can result in sending data that could be interpreted as a control code? If this is true, which character encodings are safe to send without base 64 encoding them as the final step prior to sending it? If I am correct about this, it implies you should only use the that gzip header if you are sending compressed text that has not been base 64 encoded. Does compressing it create the possibility of something that could be interpreted as a control code?
I realize this was rather long, so I will repeat my primary questions (the title) here: Is either Gzip compressed binary data or uncompressed text safe to transmit, or should it be base 64 encoded as the final step before sending it? Okay I lied there is one more question involved in this. Would sending gzip compressed text always be safe to send without base 64 encoding it at the end, no matter which character encoding it had prior to compression?
My understanding of base 64 encoding is that it is a way of representing binary data as text,
Specifically, as text consisting of characters drawn from a 64-character set, plus a couple of additional characters serving special purposes.
such that the text is safe to transmit across networks or the internet because it avoids anything that might be interpreted as a control code by the various possible protocols that might be involved at some point.
That's a bit of an overstatement. For two endpoints to communicate with each other, they need to agree on one protocol. If another protocol becomes involved along the way, then it is the responsibility of the endpoints for that transmission to handle any needed encoding considerations for it.
What bytes and byte combinations can successfully be conveyed is a matter of the protocol in use, and there are plenty that handle binary data just fine.
At one time there was also an issue that some networks were not 8-bit clean, so that bytes with numeric values greater than 127 could not be conveyed across those networks, but that is not a practical concern today.
Given this understanding, I am confused why everything sent to over the internet is not base 64 encoded.
Given that the understanding you expressed is seriously flawed, it is not surprising that you are confused.
When is it safe not to base 64 encode something before sending it?
It is not only safe but essential to avoid base 64 encoding when the recipient of the transmission expects something different. The two or more parties to a given transmission must agree about the protocol to be used. That establishes the acceptable parameters of the communication. Although Base 64 is an available option for part or all of a message, it is by no means the only one, nor is it necessarily the best one for binary data, much less for data that are textual to begin with.
I understand that not everything understands or expects to receive things in base 64, but my question is why doesn't everything expect and work with this if it is the only way to send data without the possibility it could be interpreted as control codes?
Because it is not by any means the only way to avoid data being misinterpreted.
They are being gzipped prior to uploading. I know there is also a header that can be used to indicate this: Content-Encoding: gzip. Would it be safe to compress the data and send it with this header without base 64 encoding it?
It would be expected to transfer such data without base-64 encoding it. HTTP(S) handles binary data just fine. The Content-Encoding header tells the recipient how to interpret the message body, and if it specifies a binary content type (such as gzip) then binary data conforming to that content type are what the recipient will expect.
My best guess is that it depends on if what you are sending is binary data or not.
No. These days, for all practical intents and purposes, it depends only on what application-layer protocol you are using for the transmission. If it specifies that some or all of the message is to be base-64 encoded (according to a particular base-64 scheme, as there are more than one) then that's what the sender must do and how the receiver will interpret the message. If the protocol does not specify that, then the sender must not perform base-64 encoding. Some protocols afford the sender the option to make this choice, but those also provide a way for the sender to indicate inside the transmission what choice has been made.
Is either Gzip compressed binary data or uncompressed text safe to transmit, or should it be base 64 encoded as the final step before sending it?
Neither is inherently unsafe to transmit on today's networks. Whether data are base-64 encoded for transmission is a question of agreement between sender and receiver.
Okay I lied there is one more question involved in this. Would sending gzip compressed text always be safe to send without base 64 encoding it at the end, no matter which character encoding it had prior to compression?
The character encoding of the uncompressed text is not a factor in whether a gzipped version can be safely and successfully conveyed. But it probably matters for the receiver or anyone to whom they forward that data to understand the uncompressed text correctly. If you intend to accommodate multiple character encodings then you will want to provide a way to indicate which applies to each text.
I am using HornetQ for email sending.
File attachments are transmitted out-of-band (not as part of the message) using an object storage system. This adds some overhead that I want to avoid for small files by putting them into message properties directly.
I know that I can send huge message bodies, but for large files, object storage works well, this is about small files, and delivery by property value would be very convenient if it works.
What are the considerations for message property values? Can I make them a 100K byte array? Will this slow things down (or even break)?
Headers, Properties and the Body buffer themselves are all combined in relatively straightforward process into the overall buffer for the message, so there should not be significant performance issues from that perspective. You can see the core implementation here:
https://github.com/hornetq/hornetq/blob/master/hornetq-core-client/src/main/java/org/hornetq/core/message/impl/MessageImpl.java
One consideration would be the size of your consumer window size, which by default would only be 1MB. This is the size that will be buffered on the consumer, so if you are sending messages near this size your performance in reading may be much slower as you wait for data at the consumer. This can be changed with the consumer-window-size parameter. See http://docs.jboss.org/hornetq/2.4.0.Final/docs/user-manual/html/flow-control.html#d0e4023 for more information.
Pulling from comments, you'll probably also want to increase your journal size and buffer size. See You'd probably be close to the limits. You would want to site the journal buffer size larger for sure to get some headroom, and probably size up the journal itself as well. http://hornetq.sourceforge.net/docs/hornetq-2.1.1.Final/user-manual/en/html/persistence.html#configuring.message.journal.journal-buffer-size and https://developer.jboss.org/thread/154423
I am using jquery, JSON, and AJAX for a comment system. I am curious, is there a size limit on what you can send through/store with JSON? Like if a user types a large amount and I send it through JSON is there some sort of maximum limit?
Also can any kind of text be sent through JSON. for example sometime I allow users to use html, will this be ok?
JSON is similar to other data formats like XML - if you need to transmit more data, you just send more data. There's no inherent size limitation to the JSON request. Any limitation would be set by the server parsing the request. (For instance, ASP.NET has the "MaxJsonLength" property of the serializer.)
There is no fixed limit on how large a JSON data block is or any of the fields.
There are limits to how much JSON the JavaScript implementation of various browsers can handle (e.g. around 40MB in my experience). See this question for example.
It depends on the implementation of your JSON writer/parser. Microsoft's DataContractJsonSerializer seems to have a hard limit around 8kb (8192 I think), and it will error out for larger strings.
Edit:
We were able to resolve the 8K limit for JSON strings by setting the MaxJsonLength property in the web config as described in this answer: https://stackoverflow.com/a/1151993/61569
Surely everyone's missed a trick here. The current file size limit of a json file is 18,446,744,073,709,551,616 characters or if you prefer bytes, or even 2^64 bytes if you're looking at 64 bit infrastructures at least.
For all intents, and purposes we can assume it's unlimited as you'll probably have a hard time hitting this issue...
Implementations are free to set limits on JSON documents, including the size, so choose your parser wisely. See RFC 7159, Section 9. Parsers:
"An implementation may set limits on the size of texts that it accepts. An implementation may set limits on the maximum depth of nesting. An implementation may set limits on the range and precision of numbers. An implementation may set limits on the length and character contents of strings."
There is really no limit on the size of JSON data to be send or receive. We can send Json data in file too. According to the capabilities of browser that you are working with, Json data can be handled.
If you are working with ASP.NET MVC, you can solve the problem by adding the MaxJsonLength to your result:
var jsonResult = Json(new
{
draw = param.Draw,
recordsTotal = count,
recordsFiltered = count,
data = result
}, JsonRequestBehavior.AllowGet);
jsonResult.MaxJsonLength = int.MaxValue;
What is the requirement? Are you trying to send a large SQL Table as JSON object? I think it is not practical.
You will consume a large chunk of server resource if you push thru with this.
You will also not be able to measure the progress with a progress bar because your App will just wait for the server to reply back which would take ages.
What I recommend to do is to chop the request into batches say first 1000 then request again the next 1000 till you get what you need.
This way you could also put a nice progress bar to know the progress as extracting that amount of data could really take too long.
The maximum length of JSON strings. The default is 2097152 characters, which is equivalent to 4 MB of Unicode string data.
Refer below URL
https://learn.microsoft.com/en-us/dotnet/api/system.web.script.serialization.javascriptserializer.maxjsonlength?view=netframework-4.7.2
I fetch images with open-uri from a remote website and persist them on my local server within my Ruby on Rails application. Most of the images were shown without a problem, but some images just didn't show up.
After a very long debugging-session I finally found out (thanks to this blogpost) that the reason for this is that the class Buffer in the open-uri-libary treats files with less than 10kb in size as IO-objects instead of tempfiles.
I managed to get around this problem by following the answer from Micah Winkelspecht to this StackOverflow question, where I put the following code within a file in my initializers:
require 'open-uri'
# Don't allow downloaded files to be created as StringIO. Force a tempfile to be created.
OpenURI::Buffer.send :remove_const, 'StringMax' if OpenURI::Buffer.const_defined?('StringMax')
OpenURI::Buffer.const_set 'StringMax', 0
This works as expected so far, but I keep wondering, why they put this code into the library in the first place? Does anybody know a specific reason, why files under 10kb in size get treated as StringIO ?
Since the above code practically resets this behaviour globally for my entire application, I just want to make sure that I am not breaking anything else.
When one does network programming, you allocate a buffer of a reasonably large size and send and read units of data which will fit in the buffer. However, when dealing with files (or sometimes things called BLOBs) you cannot assume that the data will fit into your buffer. So, you need special handling for these large streams of data.
(Sometimes the units of data which fit into the buffer are called packets. However, packets are really a layer 4 thing, like frames are at layer 2. Since this is happening a layer 7, they might better be called messages.)
For replies larger than 10K, the open-uri library is setting up the extra overhead to write to a stream objects. When under the StringMax size, it just includes the string in the message, since it knows it can fit in the buffer.
I have a perl script that converts strings to different encodings, like base64, ASCII or hex (both ways). Now I am writing ajax front end for it, and my question is; if I want to automate the detection of the encoding of the string submitted, is it more efficient to perform regex search on the string submitted with javascript before I send it to the server, or is it faster to leave it for the perl script to figure out what type of string?
To clarify, I am asking which of these two is better:
String submitted
Javascript detects the encoding
AJAX submits encoding and the string to perl script
Perl script returns decoded string
or
String submitted
AJAX submits the string to perl script
Perl script detects encoding and returns decoded string
Is there a particular rule of thumb where this type of processing should be performed, and what do you think is better (meaning faster) implementation?
You must validate your data on the server. Period. Otherwise you'll be sailing off into uncharted waters as soon as some two-bit wannabe "hacker" passes you a base64 string and a tag claiming that your javascript thinks it's hex.
Given this, it's up to you whether you want to also detect encoding on the client side. This has some potential benefits, since it allows you to not send data to the server at all if it's encoded in an invalid fashion or to tell the user what encoding was detected and allow them to correct it if it's an ambiguous case (e.g., hex digits are a subset of the base64 character set, so any hex string could potentially be base64). Just remember that, if an encoding gets passed to the server by the client, the server must still sanity-check the received encoding specifier and be prepared to ignore it (or reject the request completely) if it's inappropriate for the corresponding data.
This depends on the scale.
If there will be a LOT of client requests to do this, it's definitely "faster" to do it on the client side (e.g. in JS before the Ajax call), since putting it on the server side causes the server to process ALL those requests whch will compete for server's CPU resources, whereas client side you will only do one detection per client.
If you only anticipate very few concurrent requests, then doing it in Perl is probably marginally faster since Perl's regex implementation is likely better/faster than JavaScript (I don't have any stats to back this up, though) and presumably the server has better CPU.
But I would not really think that the server side margin would be terribly big considering the whole processing shouldn't take that long on either side, so I'd advise to go with client-side checking since that (as per the first paragraph) scales better.
If the performance difference between the two really matters to you a lot, you should actually implement both and benchmark under both the average anticipated and the maximum projected client loads.