I am currently using this command to do a restore in mysql of a gzipped file.
C:\...directory of gzip.exe >gunzip -c filename.gz | mysql -u.. -p.. -P.. -h dbname
I would like to extract files that are located in a directory that is different from the one in which gzip.exe is located.
How should i modify the instruction?
Change to the directory containing the .gz file and either 1) specify the path to gunzip.exe or 2) add the directory containing gunzip.exe to your PATH variable.
"C:\path\to\gunzip.exe" -c filename.gz | mysql -u.. -p.. -P.. -h dbname
Related
example:
C:\Users\Jeffery\Downloads>unzip -o "1942 (Japan, USA).zip" -d %temp% | find ".nes"
inflating: C:/Users/Jeffery/AppData/Local/Temp/1942 (Japan, USA).nes
I want the inflated filename into a variable
SET _game="C:/Users/Jeffery/AppData/Local/Temp/1942 (Japan, USA).nes"
so that I can use it in my launcher
C:/Users/Jeffery/Downloads/nesemu.exe %_game%
I am attempting to export some database data using the BCP Utility.
Here is my batch command so far:
BCP [table] out [file] -c -T -S [server] -t"¶" | 7z.exe a -si [archive name] -sdel
The BCP part works just fine:
BCP [table] out [file] -c -T -S [server] -t"¶"
However, for the 7-Zip part:
7z.exe a -si [archive name] -sdel
It works to a point. The original file is not removed, and I'd also like to encrypt the archive with 128 bit or 256 bit encryption with a password.
Any suggestions?
I found a work around solution with a small VB .NET script.
The script takes in a table name, runs BCP into a text file, runs 7 Zip with encryption options (https://sevenzip.osdn.jp/chm/cmdline/switches/method.htm#Zip), and a password, then deletes the original text file.
These commands are run using the Process() object functions.
That way I can loop through the tables I need placed in files easily.
It is not the Windows batch answer I was looking for, but it works.
Any other suggestions are still welcome.
Thanks!
BCP .... | 7z u -sidirData -pMyPassword -mhe outputFile.7z
^ ^ ^ ^ ^______________ The file that will be generated
| | | |___________________ Encrypt file names
| | |________________________________ Password used for encryption
| |___________________________________________ Name of stored file
|_____________________________________________ update/create container file
Note that there are no spaces between the switches and the values
I want to dump only a specific column on some text file using parquet-tools-1.8.1.jar.But not able to do so. I am trying below command. Please note my column name has forward slash.
parquet-tools-1.8.1.jar dump --column 'dir1/log1/job12121' '/hdfs-path/to/parquet file with space.parquet' > /home/local/parquet/output.text
Run
hadoop jar parquet-tools-1.8.1.jar parquet.tools.Main dump --column 'dir1/log1/job12121' '/hdfs-path/to/parquet file with space.parquet' > /home/local/parquet/output.text
Please use the following:
hadoop jar parquet-tools-1.8.1.jar dump -c dir1 log1 job12121 -m /hdfs-path/to/parquet file with space.parquet >> /home/local/parquet/output.text
Note:No single quotes for input arguments.
How to move the files from local disk to HDFS using Spring XD.
I do not want contents , but to move whole file for archival which saves the file with original name and content.
Here is what i have tried
stream create --name fileapple --definition "file --mode=ref --dir=/Users/dev/code/open/learnspringxd/input --pattern=apple*.txt | WHATTODOHERE"
I can see now with reference the file names with full path are made available , how to move that to HDFS.
You might want to check this which imports data from files to HDFS as a batch job and check if that fits your requirement. You can also check file | hdfs as a stream if that works for you.
example like below will load the file from data folder to HDFS and save the file by date folders(if there are multi records with different date) which by the record column named LastModified, the data file is a json file separate by lines.
file --mode=ref --dir=/Users/dev/code/open/learnspringxd/input --pattern=apple*.txt | hdfs --directory=/user/file_folder --partitionPath=path(dateFormat('yyyy-MM-dd',#jsonPath(payload,'$.LastModified'),'yyyy-MM-dd')) --fileName=output_file_name_prefix --fsUri=hdfs://HDFShostname.company.com:8020 --idleTimeout=30000
I'm using webhdfs to ingest data from Local file system to HDFS. Now I want to ensure integrity of files ingested into HDFS.
How can I make sure transferred files are not corrrupted/altered etc?
I used below webhdfs command to get the checksum of file
curl -i -L --negotiate -u: -X GET "http://$hostname:$port/webhdfs/v1/user/path?op=GETFILECHECKSUM"
How should I use above checksum to ensure the integrity of Ingested files? please suggest
Below is the steps I'm following
>md5sum locale_file
740c461879b484f4f5960aa4f67a145b
>hadoop fs -checksum locale_file
locale_file MD5-of-0MD5-of-512CRC32C 000002000000000000000000f4ec0c298cd6196ffdd8148ae536c9fe
Checksum of file on local system is different than same file on HDFS I need to compare checksum how can I do that?
One way to do that will be to calculate the checksum locally and than match it against the hadoop checksum after you ingest it.
I wrote a library to calculate check sum locally for it, in case any body is interested.
https://github.com/srch07/HDFSChecksumForLocalfile
Try this
curl -i "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=GETFILECHECKSUM"
Refer follow link for full information
https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Get_File_Checksum
It can be done from the console like below
$ md5sum locale_file
740c461879b484f4f5960aa4f67a145b
$ hadoop fs -cat locale_file |md5sum -
740c461879b484f4f5960aa4f67a145b -
You can also verify local file via code
import java.io._
import org.apache.commons.codec.digest.DigestUtils;
val md5sum = DigestUtils.md5Hex("locale_file")
and for the Hadoop
import org.apache.hadoop.fs._
import org.apache.hadoop.io._
val md5sum = MD5Hash.digest(FileSystem.get(hadoopConfiguration).open(new Path("locale_file"))).toString