I am trying to find common names in a file and file name is generated dynamically. But when I try to give the filename using the $ size its not getting replaced tried echo and then eval but get an error as an unexpected token (
The code is as below
hive -e "use $1;show tables;">$1.txt
eval $(echo "comm -12 <(sort -u hub_table_list) <(sort -u $1.txt) >result.txt")
The hive command runs succesfully file is created with the parameter name.
It contains the table names.
All Help appreciated.
Related
I have a sample.hql file which contains below lines.
desc db.table1;
desc db.table2;
desc db.table3;
I am trying to run it from shell command
I want to find out if a particular column is present in the table or not
For eg-If col_1 is present in table1 the output should say col1_1 is found in db.table1
I am not sure how to find it.
I am executing below command
hive -f sample.hql | grep -q "<column_name>"
But I am not sure how to get the db and table name from each executing line.
You can make grep show you before -B and after -A. The below command would show you 10 lines before. This likely will get the job done quick and dirty.
hive -f sample.hql | grep -B 10 -q "<column_name>"
If you wanted to be a little more careful you might try a for loop instead that feeds the lines to hive one at a time. If it finds the column it will echo the table it found the column in. ( the '&&' only executes code if the previous command was successful)
#!/bin/bash
for i in $(cat sample.hql); do
hive -e "$i" | grep -q "<column_name>" && echo $i;
done
I am trying to split a 13Gb file into equal chunks using Linux Bash Shell in windows 10 by:
split -n l/13 myfile.csv
and I am getting the following error:
split: 'xaa' would overwrite input; aborting
the xaa which is created is empty.
I have also tried using:
split -l 9000000 myfile.csv
which wields the same results.
I have used the split command before with similar arguments with no problem.
Any ideas what am I missing?
Thanks in advance
EDIT: even if i provide my own prefix I still get the same error:
split -n l/13 myfile.csv completenewprefix
split: 'completenewprefixaa' would overwrite input; aborting
EDIT2:
ls -di completenewprefixaa myfile.csv
1 completenewprefixaa 1 myfile.csv
findmnt -T .
TARGET SOURCE FSTYPE OPTIONS
/mnt/u U: drvfs rw,relatime,case=off
I have been trying to run this command (that I have run before in a different directory), and everything I've read on the message boards has not solved my unknown issue.
Of note: 1) the files exist in this directory 2) I have proper permissions to move these files around 3) I have run this exact line of code before and it has worked. 4) I tried listing files with and without '' to capture all the files (see below). 5) I also tired to list each file as 'Sample1', but that did not work.
xargs -a [filename.txt] mv -t [new-directory]
I have file beginnings (I have ~5 file for each beginning), and I want to move all the files associated with that beginning.
Example: Sample1.bam Sample1.sorted.bam, etc
The lines in the file are listed as such:
Sample1*
Sample2*
Sample3* ...etc.
What am I doing incorrectly and how can I fix it?
TIA!
When you execute command using 'xargs' arguments are passed directly to the called program ('mv' in your case). Wildcard patterns in the input are not expanded - 'sample1*' is passed as is to "mv", which issue an error message about note having a file named 'sample1*'.
To get file name expansion, you want to use the shell. One way to handle this situation is
xargs -a FILENAME.TXT -I__ sh -c "mv -t NEW-FOLDER -- __"
Security Note: the code provides some protection against command line injection (e.g., file name starting with '-'). However, other possible attacks are possible. Safer version is
cat FILENAME.txt | grep '^[A-Za-z0-9][A-Z-z0-9._-]*$' | xargs I__ sh -c "mv -t NEW-FOLDER -- __"
which will limit the input to file with alphanumeric. The 'grep' patterns can be extend the pattern as needed.
With GNU Parallel you would do something like:
cat FILENAME.txt | parallel mv {} NEW-FOLDER
One of the benefits of GNU Parallel is that it deals correctly with file names like:
My brother's 12" records cost > $1000.txt
I am trying to use "comm" to compare 2 files and save whatever the difference between them is in a third file, like the example below.
if [ -e "$updated_file_name_$day$month$year.txt"];
then
comm -3 < (sort $updated_file_name".txt") < (sort $updated_file_name_$day$month$year.txt) > $updated_file_name_$day$month$year_V$i.txt
else
mv $updated_file_name".txt" $updated_file_name_$day$month$year.txt
fi
It searches or the previous version of a file, if it finds it than both files are supposed to be sorted and compared, than the result would be saved in a third file, else it just saves the new file as the original file for that date.
I used it on the shell console as "bash -c 'comm -3 [rest of the command]" and it worked but for whatever reason it isn't working on my script, i always get the wrong syntax error.
I also tried using "diff instead of "comm" but i get the same, i don`t know what the problem is.
Here is the error message:
bash: -c: line 0: syntax error near unexpected token `('
bash: -c: line 0: `diff < (sort $updated_file_name".txt") < (sort $updated_file_name"_"$day$month$year.txt) > $updated_file_name"_"$day$month$year_V$i".txt"'
Couple of things to address here,
<(), process-substitution is a bash feature, you proabably need to run it with a proper she-bang set i.e. #!/bin/bash or depending upon your distro, where bash is installed
There should be no-spaces present, i.e. using comm -3 <(sort $updated_file_name".txt") <(sort $updated_file_name_$day$month$year.txt) is the correct approach.
Another OP's question was sort was unable to open the file, getting errors during it.
Recommend, properly encasing the variables within {}
comm -3 <(sort ${updated_file_name}".txt") <(sort ${updated_file_name}_${day}${month}${year}".txt")
I'm trying to stack some text files as new columns. The files are named energies_Strength0.0BosonsXXX.txt where XXX is 80,90,100 or 110. When I run the following command:
paste energies_Strength0.0Bosons{110..80..10}.txt | column -s $'\t' -t > energies_Strength0.0.txt
I get the following error:
paste: energies_Strength0.0Bosons{110..80..10}.txt: No such file or directory
paste: energies_Strength0.1Bosons{110..80..10}.txt: No such file or directory
paste: energies_Strength0.05Bosons{110..80..10}.txt: No such file or directory
paste: energies_Strength0.15Bosons{110..80..10}.txt: No such file or directory
This same command works just fine if files are indexed in unit steps. This is, if XXX={80,81,82,...,109,110} and I run the command:
paste energies_Strength0.0Bosons{110..80}.txt | column -s $'\t' -t > energies_Strength0.0.txt
EDIT:
Hello there, I have tried the following lines based on your idea:
#$ -S /bin/bash
LANG=C
for ((i=110; i>=80; i-=10));
do
paste energies_Strength0.0Bosons$i.txt | column -s $'\t' -t > energies_Strength0.0.txt
done
but it only pastes the ...Bosons80.txt file. I need to build an structure like the following:
paste ...80.txt ...90.txt ...100.txt ...110.txt | column -s $'\t' -t > energies_Strength0.0.txt
{110..80..10} syntax is only supported on BASH 4+ VERSIONS.
On OSX your BASH version is 3.2.xx
You can use this alternative arithmetic looping:
for ((i=110; i>=80; i-=10)); do echo $i.txt; done
bash >=4 {100..80..10}
bash <4 you could seq 80 10 100
example:
kent$ seq -f '%g.txt' 80 10 110
80.txt
90.txt
100.txt
110.txt
{A..B} is well-defined Bash syntax called brace expansion, when A and B are integers. {A..B..C} does not follow this pattern, so it's interpreted as a literal.