I can read files encrypted with ccat file or ccrypt -c file in Bash with ccrypt.
How can I append an encrypted file without doing the decryption process?
You can program this, but you can probably not perform this from the command line.
The description of the protocol can be found here. It uses full block CFB where the previous ciphertext block is encrypted again to create a stream that is XOR'ed with the plaintext.
A quick look at the Wikipedia page shows that you can just grab the last full ciphertext block, use that as IV, skip the bytes of the result used for any partial ciphertext block (if present) and then continue encrypting.
However, you'll have to program that yourself. Good luck!
Related
I'm attempting to stream several large symmetrically encrypted .csv.gpg (40GB+ each) files from S3 to gnupg to an output stream.
I'd like to process the files in chunks using streams so that we never need to download the entire encrypted/decrypted file to disk or memory.
Here's an example using the AWS Ruby S3 SDK to download chunks of the object and pass them to gnupg for decryption, using Ruby 2.5.0 with the ruby-gpgme gem.
crypto = GPGME::Crypto.new
s3_client.get_object(bucket: BUCKET, key: KEY) do |chunk|
crypto.decrypt(chunk, password: PASSWORD, output: $stdout)
end
When running this, I see valid decrypted CSV data in STDOUT (good!) up until it fails at the end of the first chunk:
~/.rvm/gems/ruby-2.5.0/gems/gpgme-2.0.14/lib/gpgme/ctx.rb:435:in `decrypt_verify': Decryption failed (GPGME::Error::DecryptFailed)
This is where I'm stuck.
Can gnupg decrypt chunks at a time or must it read the entire file before writing the output?
Do the chunks need to be of certain size and/or delimited in some way?
Any feedback would be greatly appreciated.
I have thousands (or more) of gzipped files in a directory (on a Windows system) and one of my tools consumes those gzipped files. If it encounters a corrupt gzip file, it conveniently ignores them instead of raising an alarm.
I have been trying to write a Perl program that loops through each file and makes a list of files which are corrupt.
I am using the Compress::Zlib module, and have tried reading the first 1KB of each file, but that did not work since some of the files are corrupted towards the end (verified during the manual extract, alarm raised only towards the end) and reading first 1KB doesn't show a problem. I am wondering if a CRC check of these files will be of any help.
Questions:
Will CRC validation work in this case? If yes, how does it work? Will the true CRC be part of the gzip header, and we are to compare it with the calculated CRC from the file we have? How do I accomplish this in Perl?
Are there any other simpler ways to do this?
In short, the only way to check a gzip file is to decompress it until you get an error, or get to the end successfully. You do not however need to store the result of the decompression.
The CRC stored at the end of a gzip file is the CRC of the uncompressed data, not the compressed data. To use it for verification, you have to decompress all of the data. This is what gzip -t does, decompressing the data and checking the CRC, but not storing the uncompressed data.
Often a corruption in the compressed data will be detected before getting to the end. But if not, then the CRC, as well as a check against an uncompressed length also stored at the end, will with a probability very close to one detect a corrupted file.
The Archive::Zip FAQ gives some very good guidance on this.
It looks like the best option for you is to check the CRC of each member of the archives, and a sample program that does this -- ziptest.pl -- comes with the Archive::Zip module installation.
It should be easy to test the file is not corrupt by just using "gunzip -t" command, gunzip is available for windows as well and should come with gzip package.
In the question Using C++ filestreams (fstream), how can you determine the size of a file?, the top answer is the following C++ snippet:
ifstream file("example.txt", ios::binary | ios::ate);
return file.tellg();
Running it myself I noticed that the size of arbitrarily large files could be determined instantaneously and with a single read operation.
Conventionally I would assume that to determine the size of a file, one would have to move through it byte-by-byte, adding to a byte-counter. How is this achieved instead? Metadata?
The size of the file is embedded in the file metadata in the file system. Different file systems have different ways of storing this information.
Edit Obviously, this is an incomplete answer. When someone will provide an answer where he'll exemplify on a common filesystem like ex3 or ntfs or fat exactly how the file size it's known and stored, i'll delete this answer.
The file size is stored as metadata on most filesystems. In addition to GI Joe's answer above, you can use the stat function on posix systems:
stat(3) manpage
struct stat statbuf;
stat("filename.txt", &statbuf);
printf("The file is %d bytes long\n", statbuf.st_size);
When ios::ate is set, the initial position will be the end of the file, but you are free to seek thereafter.
tellg returns the position of the current character in the input stream. The other key part is ios::binary.
So all it does it seek to the end of the file for you when it opens the filestream and tell you the current position (which is the end of the file). I guess you could say it's a sort of hack in a way, but the logic makes sense.
If you would like to learn how filestreams work at a lower level, please read this StackOverflow question.
I'm creating a program that "hides" an encrypted file at the end of a JPEG. The problem is, when retrieving this encrypted file again, I need to be able to determine when the JPEG it was stored in ends. At first I thought this wouldn't be a problem because I can just run through the file checking for 0xFF and 0xD9, the bytes JPEG uses to end the image. However... I'm noticing that in quite a few JPEGs, this combination of bytes is not exclusive... So my program thinks the image has ended randomly half way through it.
I'm thinking there must be a set way of expressing that a JPEG has finished, otherwise me adding a load of bytes to the end of the file would obviously corrupt it... Is there a practical way to do this?
You should read the JFIF file format specifications
Well, there are always two places in the file that you can find with 100% reliability. The beginning and the end. So, when you add the hidden file, add another 4 bytes that stores the original length of the file and a special signature that's always distinct. When reading it back, first seek to the end - 8 and read that length and signature. Then just seek to that position.
You should read my answer on this question "Detect Eof for JPG images".
You're likely running into the thumbnail in the header, when moving through the file you should find that most marked segments contain a length indicator, here's a reference for which do and which don't. You can skip the bytes within those segments as the true eoi marker will not be within them.
Within the actual jpeg compressed data, any FF byte should be followed either by 00 (the zero byte is then discarded), or by FE to mark a comment (which has a length indicator, and can be skipped as described above).
Theoretically the only way you encounter a false eoi reading in the compressed data is within a comment.
I've been writing a ruby programme that merges the content of two files.
For example if a torrent have been downloaded two times separately, it tries to merge their contents for the blocks which have been completed.
So, I've been looking for a method which modifies a stream only at the place required and saves only that block instead of saving the whole stream again.
I'm reading the file in blocks of 16 KiBs, and how do I "replace" (not append) the content of that 16 KiBs so that only those bytes are written to disk and not the whole file is re-written each time!
Kind of,
#Doesn't exist unfortunately.
#By default it appends instead of replacing, so file size grows.
IO.write(file_name, content, offset, :replace => true)
Is there exists a method which achieves kind of that functionality?
Open the file in "r+b" mode, seek to the location and just write to it:
f=File.new("some.existing.file", "r+b");
f.seek(1024);
f.write("test\n");
f.close()
This will overwrite 5 characters of the file, following offset 1024.
If the file is shorter than your seek offset, an appropriate number of null characters are inserted to the file.