I have a file that contains two strings in each row which are the two arguments to launch a command (phantomjs) from shell. I would use GNU parallel to launch my command for each row. Now, if I launch manually, I write:
./phantomjs rasterize.js http://www.google.it/ google.png
and everything works. But if I try with parallel, something goes wrong:
parallel --verbose -a myfile --no-run-if-empty ./phantomjs rasterize.js
this is output
./phantomjs rasterize.js http://www.google.it/\ google.png
./phantomjs rasterize.js http://www.anothersite.it/\ pic.png
Usage: rasterize.js URL filename [paperwidth*paperheight|paperformat] [zoom]
paper (pdf output) examples: "5in*7.5in", "10cm*20cm", "A4", "Letter"
Usage: rasterize.js URL filename [paperwidth*paperheight|paperformat] [zoom]
paper (pdf output) examples: "5in*7.5in", "10cm*20cm", "A4", "Letter"
and I can't understand why it doesn't work, and why in verbose mode (first two rows) after link there's \ character.
cat -vet output:
http://www.google.it/ google.png$
http://www.anothersite.it/ pic.png$
$
$
You are pretty close to the answer with why in verbose mode (first two row) after link there's "\" character?
GNU Parallel treats each line as a single argument and will quote any special characters (such as space). This is what you see. This is a feature - not a bug. It means that what ever you put on a single line it will be quoted correctly for use in the shell.
But you do not want the default behaviour. You want GNU Parallel to split the line into 2 arguments. Your column separator is space:
parallel --verbose -a myfile --colsep ' ' --no-run-if-empty ./phantomjs rasterize.js
Related
What does this command mean?
cat /y//.ssh/id_rsa.pub
I know that cat is a concatenate command and .ssh/id_rsa.pub is probably the target file id_rsa.pub in directory .ssh
But what is the /y// all about?
I don't know bash, well out of my comfort zone and when I try to display help, it doesn't help (me):
$ cat --help
Usage: cat [OPTION]... [FILE]...
Concatenate FILE(s) to standard output.
With no FILE, or when FILE is -, read standard input.
-A, --show-all equivalent to -vET
-b, --number-nonblank number nonempty output lines, overrides -n
-e equivalent to -vE
-E, --show-ends display $ at end of each line
-n, --number number all output lines
-s, --squeeze-blank suppress repeated empty output lines
-t equivalent to -vT
-T, --show-tabs display TAB characters as ^I
-u (ignored)
-v, --show-nonprinting use ^ and M- notation, except for LFD and TAB
--help display this help and exit
--version output version information and exit
Examples:
cat f - g Output f's contents, then standard input, then g's contents.
cat Copy standard input to standard output.
GNU coreutils online help: <https://www.gnu.org/software/coreutils/>
Report any translation bugs to <https://translationproject.org/team/>
Full documentation <https://www.gnu.org/software/coreutils/cat>
or available locally via: info '(coreutils) cat invocation'
I want to remove the first two characters of a column in a text file.
I am using the below but this is also truncating the headers.
sed -i 's/^..//' file1.txt
Below is my file:
FileName,Age
./Acct_Bal_Tgt.txt,7229
./IDQ_HB1.txt,5367
./IDQ_HB_LOGC.txt,5367
./IDQ_HB.txt,5367
./IGC_IDQ.txt,5448
./JobSchedule.txt,3851
I want the ./ to be removed from each line in the file name.
Transferring comments to an answer, as requested.
Modify your script to:
sed -e '2,$s/^..//' file1.txt
The 2,$ prefix limits the change to lines 2 to the end of the file, leaving line 1 unchanged.
An alternative is to remove . and / as the first two characters on a line:
sed -e 's%^[.]/%%' file1.txt
I tend to use -e to specify that the script option follows; it isn't necessary unless you split the script over several arguments (so it isn't necessary here where there's just one argument for the script). You could use \. instead of [.]; I'm allergic to backslashes (as you would be if you ever spent time working out whether you needed 8 or 16 consecutive backslashes to get the right result in a troff document).
Advice: Don't use the -i option until you've got your script working correctly. It overwrites your file with the incorrect output just as happily as it will with the correct output. Consequently, if you're asking about how to write a sed script on SO, it isn't safe to be using the -i option. Also note that the -i option is non-standard and behaves differently with different versions of sed (when it is supported at all). Specifically, on macOS, the BSD sed requires a suffix specified; if you don't want a backup, you have to use two arguments: -i ''.
Use this Perl one-liner:
perl -pe 's{^[.]/}{}' file1.txt > output.txt
The Perl one-liner uses these command line flags:
-e : Tells Perl to look for code in-line, instead of in a file.
-p : Loop over the input one line at a time, assigning it to $_ by default. Add print $_ after each loop iteration.
s{^[.]/}{} : Replace a literal dot ([.]) followed by a slash ('/'), found at the beginning of the line (^), with nothing (delete them). This does not modify the header since it does not match the regex.
If you prefer to modify the file in-place, you can use this:
perl -i.bak -pe 's{^[.]/}{}' file1.txt
This creates the backup file file1.txt.bak.
SEE ALSO:
perldoc perlrun: how to execute the Perl interpreter: command line switches
perldoc perlrequick: Perl regular expressions quick start
I'm using the commands given below for splitting my fastq file into two separate paired end reads files:
grep '#.*/1' -A 3 24538_7#2.fq >24538_7#2_1.fq
grep '#.*/2' -A 3 24538_7#2.fq >24538_7#2_2.fq
But it's automatically introducing a -- line separator between the entries. Hence, making my fastq file inappropriate for further processing(because it then becomes an invalid fastq format).
So, I want to get rid of the line separator(--).
PS: I've found the answer for Linux machine but I'm using MacOS, and those didn't work on Mac terminal.
You can use the --no-group-separator option to suppress it (in GNU grep).
Alternatively, you could use (GNU) sed:
sed '\|#.*/1|,+3!d'
deletes all lines other than the one matching #.*/1 and the next three lines.
For macOS sed, you could use
sed -n '\|#.*/1|{N;N;N;p;}'
but this gets unwieldy quickly for more context lines.
Another approach would be to chain grep with itself:
grep '#.*/1' -A 3 file.fq | grep -v "^--"
The second grep selects non-matching (-v) lines that start with -- (though this pattern can sometimes be interpreted as a command line option, requiring some weird escaping like "[-][-]", which is why i put the ^ there).
in applescript editor:
do shell script "grep -w 'SomeText' /tmp/test"
ignores -w
in Bash:
grep -w 'SomeText' /tmp/test
not ignores arguments
But for example arguments -v (negative) works in AppleScript with do shell script
it is happening on both different computers with different systems
how i can use -w argument in grep from applescript?
Thanks!
Regardless of where I run the grep -w ... command from, Terminal or ApplesScript's do shell script command, I get identical output.
The manual page for the -w option in grep states the following:
−w, −−word-regexp
The expression is searched for as a word (as if surrounded by ‘[[:<:]]’ and ‘[[:>:]]’; see re_format(7)).
The manual page for re_format states:
There are two special cases‡ of bracket expressions: the bracket expressions [[:<:]] and [[:>:]] match the null string at the beginning and end of a word respectively. A word is defined as a sequence of word characters which is neither preceded nor followed by word characters. A word character is an alnum character (as defined by ctype(3)) or an underscore.
In Terminal:
Contents of /tmp/test:
$ cat /tmp/test
SomeText
MoreText
ASomeTextZ
Other Text
0 SomeText 1
$
Using grep without -w on /tmp/test:
$ grep 'SomeText' /tmp/test
SomeText
ASomeTextZ
0 SomeText 1
$
As it should, grep finds all three lines containing 'SomeText'.
Using grep with -w on /tmp/test:
$ grep -w 'SomeText' /tmp/test
SomeText
0 SomeText 1
$
As it should, grep -w finds only the lines conforming to what's stated in the manual page excerpts shown above. In this case, only two of the three lines that contain 'SomeText'.
The output of each grep command, show above, when wrapped in a do shell script command in AppleScript are identical, as should be.
In Script Editor:
Because these are the expected results is why I'm adamant about following How to create a Minimal, Complete,and Verifiable example, when asking questions such that you have, in the manner you have!
I'd suggest you show us the actual content of your /tmp/test file and the actual output you get from each of the grep commands, with and without the -w option, from both Terminal and AppleScript's do shell script command.
Although it shouldn't make a difference, nonetheless you should also provide macOS version info so we can test this under the actual version of macOS you're using, so as to see if that's a relevant factor in the equation.
From this question, I found the split utilty, which takes a file and splits it into evenly sized chunks. By default, it outputs these chunks to new files, but I'd like to get it to output them to stdout, separated by a newline (or an arbitrary delimiter). Is this possible?
I tried cat testfile.txt | split -b 128 - /dev/stdout
which fails with the error split: /dev/stdoutaa: Permission denied.
Looking at the help text, it seems this tells split to use /dev/stdout as a prefix for the filename, not to write to /dev/stdout itself. It does not indicate any option to write directly to a single file with a delimiter. Is there a way I can trick split into doing this, or is there a different utility that accomplishes the behavior I want?
It's not clear exactly what you want to do, but perhaps the --filter option to split will help out:
--filter=COMMAND
write to shell COMMAND; file name is $FILE
Maybe you can use that directly. For example, this will read a file 10 bytes at a time, passing each chunk through the tr command:
split -b 10 --filter "tr [:lower:] [:upper:]" afile
If you really want to emit a stream on stdout that has separators between chunks, you could do something like:
split -b 10 --filter 'dd 2> /dev/null; echo ---sep---' afile
If afile is a file in my current directory that looks like:
the quick brown fox jumped over the lazy dog.
Then the above command will result in:
the quick ---sep---
brown fox ---sep---
jumped ove---sep---
r the lazy---sep---
dog.
---sep---
From info page :
`--filter=COMMAND'
With this option, rather than simply writing to each output file,
write through a pipe to the specified shell COMMAND for each
output file. COMMAND should use the $FILE environment variable,
which is set to a different output file name for each invocation
of the command.
split -b 128 --filter='cat ; echo ' inputfile
Here is one way of doing it. You will get each 128 character into variable "var".
You may use your preferred delimiter to print or use it for further processing.
#!/bin/bash
cat yourTextFile | while read -r -n 128 var ; do
printf "\n$var"
done
You may use it as below at command line:
while read -r -n 128 var ; do printf "\n$var" ; done < yourTextFile
No, the utility will not write anything to standard output. The standard specification of it says specifically that standard output in not used.
If you used split, you would need to concatenate the created files, inserting a delimiter in between them.
If you just want to insert a delimiter every N th line, you may use GNU sed:
$ sed '0~3a\-----\' file
This inserts a line containing ----- every 3rd line.
To divide the file into chunks, separated by newlines, and write to stdout, use fold:
cat yourfile.txt | fold -w 128
...will write to stdout in "chunks" of 128 chars.