I am trying to use "comm" to compare 2 files and save whatever the difference between them is in a third file, like the example below.
if [ -e "$updated_file_name_$day$month$year.txt"];
then
comm -3 < (sort $updated_file_name".txt") < (sort $updated_file_name_$day$month$year.txt) > $updated_file_name_$day$month$year_V$i.txt
else
mv $updated_file_name".txt" $updated_file_name_$day$month$year.txt
fi
It searches or the previous version of a file, if it finds it than both files are supposed to be sorted and compared, than the result would be saved in a third file, else it just saves the new file as the original file for that date.
I used it on the shell console as "bash -c 'comm -3 [rest of the command]" and it worked but for whatever reason it isn't working on my script, i always get the wrong syntax error.
I also tried using "diff instead of "comm" but i get the same, i don`t know what the problem is.
Here is the error message:
bash: -c: line 0: syntax error near unexpected token `('
bash: -c: line 0: `diff < (sort $updated_file_name".txt") < (sort $updated_file_name"_"$day$month$year.txt) > $updated_file_name"_"$day$month$year_V$i".txt"'
Couple of things to address here,
<(), process-substitution is a bash feature, you proabably need to run it with a proper she-bang set i.e. #!/bin/bash or depending upon your distro, where bash is installed
There should be no-spaces present, i.e. using comm -3 <(sort $updated_file_name".txt") <(sort $updated_file_name_$day$month$year.txt) is the correct approach.
Another OP's question was sort was unable to open the file, getting errors during it.
Recommend, properly encasing the variables within {}
comm -3 <(sort ${updated_file_name}".txt") <(sort ${updated_file_name}_${day}${month}${year}".txt")
Related
I am trying to split a 13Gb file into equal chunks using Linux Bash Shell in windows 10 by:
split -n l/13 myfile.csv
and I am getting the following error:
split: 'xaa' would overwrite input; aborting
the xaa which is created is empty.
I have also tried using:
split -l 9000000 myfile.csv
which wields the same results.
I have used the split command before with similar arguments with no problem.
Any ideas what am I missing?
Thanks in advance
EDIT: even if i provide my own prefix I still get the same error:
split -n l/13 myfile.csv completenewprefix
split: 'completenewprefixaa' would overwrite input; aborting
EDIT2:
ls -di completenewprefixaa myfile.csv
1 completenewprefixaa 1 myfile.csv
findmnt -T .
TARGET SOURCE FSTYPE OPTIONS
/mnt/u U: drvfs rw,relatime,case=off
I have been trying to run this command (that I have run before in a different directory), and everything I've read on the message boards has not solved my unknown issue.
Of note: 1) the files exist in this directory 2) I have proper permissions to move these files around 3) I have run this exact line of code before and it has worked. 4) I tried listing files with and without '' to capture all the files (see below). 5) I also tired to list each file as 'Sample1', but that did not work.
xargs -a [filename.txt] mv -t [new-directory]
I have file beginnings (I have ~5 file for each beginning), and I want to move all the files associated with that beginning.
Example: Sample1.bam Sample1.sorted.bam, etc
The lines in the file are listed as such:
Sample1*
Sample2*
Sample3* ...etc.
What am I doing incorrectly and how can I fix it?
TIA!
When you execute command using 'xargs' arguments are passed directly to the called program ('mv' in your case). Wildcard patterns in the input are not expanded - 'sample1*' is passed as is to "mv", which issue an error message about note having a file named 'sample1*'.
To get file name expansion, you want to use the shell. One way to handle this situation is
xargs -a FILENAME.TXT -I__ sh -c "mv -t NEW-FOLDER -- __"
Security Note: the code provides some protection against command line injection (e.g., file name starting with '-'). However, other possible attacks are possible. Safer version is
cat FILENAME.txt | grep '^[A-Za-z0-9][A-Z-z0-9._-]*$' | xargs I__ sh -c "mv -t NEW-FOLDER -- __"
which will limit the input to file with alphanumeric. The 'grep' patterns can be extend the pattern as needed.
With GNU Parallel you would do something like:
cat FILENAME.txt | parallel mv {} NEW-FOLDER
One of the benefits of GNU Parallel is that it deals correctly with file names like:
My brother's 12" records cost > $1000.txt
I am trying to find common names in a file and file name is generated dynamically. But when I try to give the filename using the $ size its not getting replaced tried echo and then eval but get an error as an unexpected token (
The code is as below
hive -e "use $1;show tables;">$1.txt
eval $(echo "comm -12 <(sort -u hub_table_list) <(sort -u $1.txt) >result.txt")
The hive command runs succesfully file is created with the parameter name.
It contains the table names.
All Help appreciated.
This should be a no-brainer, but apparently I have no brain today.
I have 50 20-gig logs that contain entries from multiple apps, one of which addes a transaction ID to its log lines. I have 42 transaction IDs I need to review, and I'd like to parse out the appropriate lines into separate files.
To do a single file, the command would be simply,
grep CDBBDEADBEEF2020X02393 server.log* > CDBBDEADBEEF2020X02393.log
that creates a log isolated to that transaction, from all 50 server.logs.
Now, I have a file with 42 txnIDs (shortening to 4 here):
CDBBDEADBEEF2020X02393
CDBBDEADBEEF6548X02302
CDBBDE15644F2020X02354
ABBDEADBEEF21014777811
And I wrote:
#/bin/sh
grep $1 server.\* > $1.log
But that is not working. Changing the shebang to #/bin/bash -xv, gives me this weird output (obviously I'm playing with what the correct escape magic must be):
$ ./xtrakt.sh B7F6E465E006B1F1A
#!/bin/bash -xv
grep - ./server\.\*
' grep - './server.*
: No such file or directory
I have also tried the command line
grep - server.* < txids.txt > $1
But OBVIOUSLY that $1 is pointless and I have no idea how to get a file named per txid using the input redirect form of the command.
Thanks in advance for any ideas. I haven't gone the route of doing a foreach in the shell script, because I want grep to put the original filename in the output lines so I can examine context later if I need to.
Also - it would be great to have the server.* files ordered numerically (server.log.1, server.log.2 NOT server.log.1, server.log.10...)
try this:
while read -r txid
do
grep "$txid" server.* > "$txid.log"
done < txids.txt
and for the file ordering - rename files with one digit to two digit, with leading zeroes, e.g. mv server.log.1 server.log.01.
There are several unix commands that are designed to operate on two files. Commonly such commands allow the contents for one of the "files" to be read from standard input by using a single dash in place of the file name.
I just came across a technique that seems to allow both files to be read from standard input:
comm -12 <(sort file1) <(sort file2)
My initial disbelieving reaction was, "That shouldn't work. Standard input will just have the concatenation of both files. The command won't be able to tell the files apart or even realize that it has been given the contents of two files."
Of course, this construction does work. I've tested it with both comm and diff using bash 3.2.51 on cygwin 1.7.7. I'm curious how and why it works:
Why does this work?
Is this a Bash extension, or is this straight Bourne shell functionality?
This works on my system, but will this technique work on other platforms? (In other words, will scripts written using this technique be portable?)
Bash, Korn shell (ksh93, anyway) and Z shell all support process substitution. These appear as files to the utility. Try this:
$ bash -c 'echo <(echo)'
/dev/fd/63
$ ksh -c 'echo <(echo)'
/dev/fd/4
$ zsh -c 'echo <(echo)'
/proc/self/fd/12
You'll see file descriptors similar to the ones shown.
This is a standard Bash extension. <(sort file1) opens a pipe with the output of the sort file1 command, gives the pipe a temporary file name, and passes that temporary file name on the comm command line.
You can see how it works by getting echo to tell you what's being passed to the program:
echo <(sort file1) <(sort file2)