I have a text file (delete_names.txt) with a name by line. The file have around 200 lines, like this:
ABAA742_2012-01-13_decont.fa
ABAA1502_2014-08-08_decont.fa
I want read each line of the file and concatenate the string ".faa" in the end of each line in order to get:
ABAA742_2012-01-13_decont.fa.faa
ABAA1502_2014-08-08_decont.fa.faa
I try to get the final string, but the sub string is always added in the start.
while read line; do
a='.faa'
echo -E "$line$a";
done < delete_names.txt
output from the code above:
.faa742_2012-01-13_decont.fa
.faa1502_2014-08-08_decont.fa
The final goal, after get the concatenated name is to delete the file inside the directory.
Short sed + xargs approach:
sed 's/$/.faa/' delete_names.txt | xargs -I {} rm {}
Or just with single xargs command:
xargs -a delete_names.txt -I {} rm {}.faa
-a file - read items from file instead of standard input
I frequently use gnu-parallel for those kind of jobs, it could be something like:
cat delete_names.txt | parallel rm -rf "{}a"
what it does is to pipe the each name to the parallel command, and it forks the to one or multiple process. I just added "{}a" but you can also use "{.}.faa" that removes the extension of the your string.
Related
I'm using xargs to try to echo the name of a file, followed by its contents. Here is the command
find output/ -type f | xargs -I {} sh -c "echo {} ; cat {}"
However for some reason, the second replace after cat is not being replaced. Only for some files, so some files do work correctly.
To be clear, I'm not looking for a command that lets me echo the name of a file followed by its contents, I'm trying to understand why this specific command does not work.
Turns out that the command was too long, so it was working with shorter file names and failing for longer ones. From man xargs
-I replstr
Execute utility for each input line, replacing one or more occurrences of replstr in up to replacements (or 5 if no -R flag is specified) arguments to utility with the
entire line of input. The resulting arguments, after replacement is done, will not be allowed to grow beyond 255 bytes; this is implemented by concatenating as much
of the argument containing replstr as possible, to the constructed arguments to utility, up to 255 bytes. The 255 byte limit does not apply to arguments to utility
which do not contain replstr, and furthermore, no replacement will be done on utility itself. Implies -x.
The root cause of the problem is pointed out in Carlos' answer, but without a solution.
After some googling, I couldn't find a way to lift up the 255 characters limit.
So a probable way to workaround it, is to use shell variable as a substitution.
Example:
find . | xargs -I% sh -c 'F="%";iconv -f gb2312 -t utf-8 "$F">"$F.out";mv "$F.out" "$F"'
Remember to use single quotes at the outermost sh -c parameter string, we don't want the $F inside to be replaced by our parent shell.
Is it files with white space in the name that create problems? Try adding \", like this:
find output/ -type f | xargs -I {} sh -c "echo \"{}\" ; cat \"{}\""
This worked for me using Bash.
I can't read or apply any other commands like cat or strings on .txt files because it is not allowed. I need to read a file named flag.txt, but this file is also on the blacklist. So, is there any way to read *.txt using the head command? The head command is allowed.
blacklist=\
'flag\|<\|$\|"\|'"'"'\|'\
'cat\|tac\|*\|?\|less\|more\|pico\|nano\|edit\|hexdump\|xxd\|'\
'sed\|tail\|diff\|grep\|paste\|strings\|bas64\|sort\|uniq\|cut\|awk\|'\
'bzip\|gzip\|xz\|tar\|ar\|'\
'mv\|cp\|ln\|nl\|'\
'python\|perl\|sh\|cc\|g++\|php\|hd\|g++\|gcc\|curl\|tcp\|udp\|'\
'scp\|sftp\|wget\|nc\|netcat'
Thanks
do you want some alternative of the command head *.txt? if so, ls/findand xargs will help, but it can not identify .txt file, it will read all the file under the directory.
ls -1| xargs head
You can use the ` (backtick) in the following way:
head `ls -1`
Backtick has a very special meaning. Everything you type between
backticks is evaluated (executed) by the shell before the main command
So the command will do the following:
`ls -1` - will result with the file names
head - will show the start of the files listed in ls -1
More info about backtick can be found in this answer
If you need a glob that matches flag.txt but can use neither * not the string flag, you can use fl[a]g.txt instead. Then, to print the entire file using head, use -c and pass it the size of the file:
head -c $(stat -c '%s' fl[a]g.txt) fl[a]g.txt
Another approach would be to use the shell to read the file:
while IFS= read -r c; do echo $c; done < fl[a]g.txt
You could also just use paste:
paste fl[a]g.txt
I have file with the name of filesList.txt which contain list of all files which needs to be merged into single file.
filesList.txt
------------------
../../folder/a.js
../../folder/b.js
../../folder/c.js
../../folder/d.js
Current I am running following commands.
cp filesList.txt filesList.sh
chmod 777 filesList.sh
vim filesList.sh
cat
../../folder/a.js
../../folder/b.js
../../folder/c.js
../../folder/d.js
> output.txt
RUN vim command j10 to make above multiline file into single line like this
cat ../../folder/a.js ../../folder/b.js ../../folder/c.js ../../folder/d.js > output.txt
save and quit file within vim using :wq
and run ./fileList.sh to create single output.text file in exact same order files are listed in.
My Question is what command I need to use to create a bash file which create external list of file(filesList.txt) line by line and generate and single file with its contents. So I don't have to conver my filesList.txt file into filesList.sh file each time I need to merge file.
A line-oriented file is a bad choice here (in a "any attacker who can control filenames can inject arbitrary files into your output" sense of bad; you probably don't want to risk that someone who figures out how to create new .js files matching your glob can then introduce /etc/passwd to the list by creating ../../$'\n'/etc/passwd$'\n'/hello.js). Instead, separate values by NULs, and use xargs -0 (a non-POSIX extension, but a popular one provided by major OS vendors) to convert those into arguments.
printf '%s\0' ../../folder/*.js >filesList.nsv # generate file w/ null-separated values
xargs -0 cat <filesList.nsv >output.txt # combine to argument list split on NUL
By the way, if you want to generate your list of files recursively, that first part would become:
find ../../folder -name '*.js' -print0 >filesList.nsv
...and if you don't have any other need for filesList.nsv, I'd just avoid it entirely and generate output.txt directly:
find ../../folder -name '*.js' -exec cat '{}' + >output.txt
If you must use newlines, but you have GNU xargs, at least use xargs -d $'\n' to process them to try to avoid other, quoting-related bugs found in stock xargs or more naive practices in bash:
printf '%s\n' ../../folder/*.js >filesList.txt # generate w/ newline-separated values
xargs -d $'\n' cat <filesList.txt >output.txt # combine on those values
If you don't have GNU xargs, then you can implement this yourself in shell:
# Newline-separated input
while IFS= read -r filename; do
cat "$filename"
done <filesList.txt >output.txt
# ...or NUL-separated input
while IFS= read -r -d '' filename; do
cat "$filename"
done <filesList.txt >output.txt
all the files in my folder are named like this:
aodv-0-0.pcap aodv-1-0.pcap aodv-2-0.pcap aodv-3-0.pcap
etc
Is there any chance to rename them so as they become:
1.pcap 2.pcap 3.pcap 4.pcap
aodv-0-0 ------> 1
aodv-1-0 ------> 2
etc
i tried using "rename" but i don't know how to increment the number in the filename.
#RENAMING the files
printf "...RENAMING the files to get proper a nodeID...\n"
cd /home/marcin/workspace/bake/source/ns-3.19/RESULTS/TRAFFIC_LIGHTS/
rename aodv- "" *.pcap
After using it i get:
aodv-0-0.pcap ---> 0-0.pcap (and i need 1.pcap)
aodv-1-0.pcap ---> 1-0.pcap (and i need 2.pcap)
and then maybe i could use the number(just the number) as a variable "nodeID".
I used "basename" function to get the filename as a variable before but i don't need ".pcap" extension as a variable anymore.
Thanks in advance
ls aodv* | sed -e "p;s/aodv-//" -e "s/-.//" |xargs -n2 mv
For counter increase:
ls aodv*|awk -F\. 'split($1,a,"-"){print $0,a[2]+1""FS""$2}'|xargs -n2 mv
The ls output is piped to sed , then we use the p flag to print the argument without modifications, in other words , the original name of the file .
The next step is use the substitute command to change file extension.
The result is a combined output that consist of a sequence of old_file_name -> new_file_name.
Finally we pipe the resulting feed through xargs to get the effective rename of the files.
http://nixtip.wordpress.com/2010/10/20/using-xargs-to-rename-multiple-files/
I have a file containing valid xmls (one per line) and I want to execute a utility (xpath) on each line one by one.
I tried xargs but that seems doesn't seem to have an option to pass the line as stdin :-
% cat <xmls-file> | xargs -p -t -L1 xpath -p "//Path/to/node"
Cannot open file '//Path/to/node' at /System/Library/Perl/Extras/5.12/XML/XPath.pm line 53.
I also tried parallel --spreadstdin but that doesn't seem to work either :-
% cat <xmls-file> | parallel --spreadstdin xpath -p "//Path/to/node"
junk after document element at line 2, column 0, byte 1607
If you want every line of a file to be split off and made stdin for a utility
you could use a for loop in bash shell:
cat xmls-file | while read line
do ( echo $f > /tmp/input$$;
xpath -p "//Path/to/node" </tmp/input$$
rm -f /tmp/input$$
);
done
The $$ appends the process id number, creating a unique name
I assume xmls-file contains, on each line, what you want iterated into $f and that you want this as stdin for a command line, not as a parameter to the command.
On the other hand, your specification may be incorrect and maybe instead you need each line
to be part of a command. In that case, delete the echo and rm lines, and change the xpath command to include $f wherever the line from the file is needed.
I've not done much XML so the do command may need to be edited.
You are very close with the GNU Parallel version; only -n1 missing:
cat <xmls-file> | parallel -n1 --spreadstdin xpath -p "//Path/to/node"