Nowadays I've a process that copies file by file. This is taking a lot of time.
Thought about the following solution:
Compress all files in a serverA and then copy the zip file to serverB;
Decompress this file in serverB;
Is there anyway to decompress this file by command line after the file is copied?
Related
I am trying to concatenate my server log files (gz files named access_log_20191112.gz access_log_20191111.gz etc. ) into one single file (a gz file containing one single file).
Under Windows 10 Powershell command line, I have tried :
cat access*.gz > Global_File.gz
The command is executed, Global_File.gz has a realistic size as viewed in Explorer, but I can't open it with 7-zip. I get the error : Can not open path/Global_File.gz as archive. I have no problem usually opening gz archives with 7-zip.
Any suggestion to correct this approach or an alternate method to concatenate gz files on a windows 10 machine?
Thx
I have a batch file that compresses files into a zip folder so I can email the file. The file runs with no issues. However, when I try to open the zip folder, I get the following error:
'Folder is invalid'
and it won't open. Below is the part of the step that compresses the file on Windows Server 2012:
COMPRESS "%DIR_IP_INTERFACES%\SP_Backups\Outbound_GDC_Req.txt" "%DIR_IP_INTERFACES%\SP_Backups\Outbound_GDC_Req_%DATE:/=%_%TIME::=%.zip"
I need it to zip during the batch file, but I will not be the one unzipping. So, I was looking for a standard way to allow anybody to open/unzip the folder when they receive the email (that doesn't require multiple downloads/changes on each computer that might open these files).
Any ideas?
I have a .bin file that will comprise of 3 files
1. tar.gz file
2. .zip file
3. install.sh file
For now the install.sh file is empty. I am trying to write a shell script that should be able to extract the .zip file and copy the tar.gz file to a specific location when the *.bin file is executed on an Ubuntu machine. There is a Jenkins job that will pull in these 3 files to create the *.bin file
My Question is how do I access the tar.gz and .zip file from my shell script ?
There are two general tricks that I'm aware of for this sort of thing.
The first is to use a file format that will ignore invalid data and find the correct file contents automatically (I believe zip is one such format/tool).
When this is the case you just run the tool on the packed/concatenated file and let the tool do its job.
For formats and tools where that doesn't work and/or isn't possible the general trick is to embed markers in the concatenated file such that the original script ignores the data but can operate on itself to "extract" the embedded data so the other tool can operate on the extracted contents.
I want to transfer too many small files (e.g. 200k files) in a zip file into HDFS from the local machine. When I unzip the zip file and tranfer the files into HDFS, it takes a long time. Is there anyway I can transfer the original zip file into HDFS and unzip it there?
If your file is in GB's then this command would certainly help to avoid out of space errors as there is no need to unzip the file on local filesystem.
put command in hadoop supports reading input from stdin. For reading the input from stdin use '-' as source file.
Compressed filename: compressed.tar.gz
gunzip -c compressed.tar.gz | hadoop fs -put - /user/files/uncompressed_data
Only Disadvantage: The only drawback of this approach is that in HDFS the data will be merged into a single file even though the local compressed file contains more than one file.
http://bigdatanoob.blogspot.in/2011/07/copy-and-uncompress-file-to-hdfs.html
I have a script set up to rotate some log files in windows, and as part of the process I'd like it to automatically compress the rotated file. To do this I use the command
compress source.file destination.file.zip
However, if I try to open the file, I get the message "The Compressed (zipped) Folder is invalid or corrupted"
I've tried compress with -Z, and I get the same message. What am I doing wrong?
compress output is not ZIP file format compatible, it uses the LZW algorithm.
The only way to "open" a compressed file is with uncompress or gunzip.
Windows ports of common Unix commands, including compress and gzip/gunzip available here.
EDIT: To produce ZIP files from the command line in Windows, you can use something like 7-Zip, which includes a command line application (7z.exe). The Unix commands linked above also include zip.exe for manipulating ZIP files from the command line.