I have a program (pull) which downloads files and emits their contents (JSON) to stdout, the input of the program is the id of every document I want to download, like so:
pull one two three
>
> { ...one }
> {
...two
}
> { ...three }
However, I now would like to pipe that output to a different file for each file it has emitted, ideally being able to reference the filename by the order of args initially used: one two three.
So, the outcome I am looking for, would something like the below.
pull one two three | > $1.json
>
> saved one.json
> saved two.json
> saved three.json
Is there any way to achieve this or something similar at all?
Update
I just would like to clarify how the program works and why it may not be ideal looping through arguments and executing the program multiple times for each argument declared.
Whenever pull gets executed, it performs two operations:
A: Expensive operation (timely to resolve): This retrieves all documents available in a database where we can lookup items by the argument names provided when invoking pull.
B: Operation specific to the provided argument: after A resolves, we will use its response in order to get the data needed for specifically retrieving the individual document.
This means that, having A+B called multiple times for every argument, wouldn't be ideal as A is an expensive operation.
So instead of having, AB AB AB AB I would like to have ABBBB.
You're doing it the hard way.
for f in one two three; do pull "$f" > "$f.json" & done
Unless something in the script is not compatible with multiple simultaneous copies, this will make the process faster as well. If it is, just change the & to ;.
Update
Try just always writing the individual files. If you also need to be able to send them to stdout, just cat the file afterwards, or use tee when writing it.
If that's not ok, then you will need to clearly identify and parse the data blocks. For example, if the start of a section is THE ONLY place { appears as the first character on a line, that's a decent sentinel value. Split your output to files using that.
For example, throw this into another script:
awk 'NR==FNR { ndx=1; split($0,fn); name=""; next; } /^{/ { name=fn[ndx++]; } { if (length(name)) print $0 > name".json"; }' <( echo "$#" ) <( pull "$#" )
call that script with one two three and it should do what you want.
Explanation
awk '...' <( echo "$#" ) <( pull "$#" )
This executes two commands and returns their outputs as "files", streams of input for awk to process. The first just puts the list of arguments provided on one line for awk to load into an array. The second executes your pull script with those args, which provides the streaming output you already get.
NR==FNR { ndx=1; split($0,fn); name=""; next; }
This tells awk to initialize a file-controlling index, read the single line from the echo command (the args) and split them into an array of filename bases desired, then skip the rest of processing for that record (it isn't "data", it's metadata, and we're done with it.) We initialize name to an empty string so that we can check for length - otherwise those leading blank lines end up in .json, which probably isn't what you want.
/^{/ { name=fn[ndx++]; }
This tells awk each time it sees { as the very first character on a line, set the output filename base to the current index (which we initialized at 1 above) and increment the index for the next time.
{ if (length(name)) print $0 > name".json"; }
This tells awk to print each line to a file named whatever the current index is pointing at, with ".json" appended. if (length(name)) throws away the leading blank line(s) before the first block of JSON.
The result is that each new set will trigger a new filename from your given arguments.
That work for you?
In Use
$: ls *.json
ls: cannot access '*.json': No such file or directory
$: pull one two three # my script to simulate output
{ ...one... }
{
...two...
}
{ ...three... }
$: splitstream one two three # the above command in a file to receive args
$: grep . one* two* three* # now they exist
one.json:{ ...one... }
two.json:{
two.json: ...two...
two.json:}
three.json:{ ...three... }
I'm studying awk pretty fiercely to write a git diffn implementation which will show line numbers for git diff, and I want confirmation on whether or not this Wikipedia page on awk is wrong [Update: I've now fixed this part of that Wikipedia page, but this is what it used to say]:
(pattern)
{
print 3+2
print foobar(3)
print foobar(variable)
print sin(3-2)
}
Output may be sent to a file:
(pattern)
{
print "expression" > "file name"
}
or through a pipe:
(pattern)
{
print "expression" | "command"
}
Notice (pattern) is above the opening brace. I'm pretty sure this is wrong but need to know for certain before editing the page. What I think that page should look like is this:
/regex_pattern/ {
print 3+2
print foobar(3)
print foobar(variable)
print sin(3-2)
}
Output may be sent to a file:
/regex_pattern/ {
print "expression" > "file name"
}
or through a pipe:
/regex_pattern/ {
print "expression" | "command"
}
Here's a test to "prove" it. I'm on Linux Ubuntu 18.04.
1. test_awk.sh
gawk \
'
BEGIN
{
print "START OF AWK PROGRAM"
}
'
Test and error output:
$ echo -e "hey1\nhello\nhey2" | ./test_awk.sh
gawk: cmd. line:3: BEGIN blocks must have an action part
But with this:
2. test_awk.sh
gawk \
'
BEGIN {
print "START OF AWK PROGRAM"
}
'
It works fine!:
$ echo -e "hey1\nhello\nhey2" | ./test_awk.sh
START OF AWK PROGRAM
Another example (fails to provide expected output):
3. test_awk.sh
gawk \
'
/hey/
{
print $0
}
'
Erroneous output:
$ echo -e "hey1\nhello\nhey2" | ./test_awk.sh
hey1
hey1
hello
hey2
hey2
But like this:
4. test_awk.sh
gawk \
'
/hey/ {
print $0
}
'
It works as expected:
$ echo -e "hey1\nhello\nhey2" | ./test_awk.sh
hey1
hey2
Updates: after solving this problem, I just added these sections below:
Learning material:
In the process of working on this problem, I just spent several hours and created these examples: https://github.com/ElectricRCAircraftGuy/eRCaGuy_hello_world/tree/master/awk. These examples, comments, and links would prove useful to anyone getting started learning awk/gawk.
Related:
git diff with line numbers and proper code alignment/indentation
"BEGIN blocks must have an action part" error in awk script
The whole point of me learning awk at all in the first place was to write git diffn. I just got it done: Git diff with line numbers (Git log with line numbers)
I agree with you that the Wikipedia page is wrong. It's right in the awk manual:
A pattern-action statement has the form
pattern { action }
A missing { action } means print the line; a missing pattern always matches. Pattern-action statements are separated by newlines or semicolons.
...
Statements are terminated by semicolons, newlines or right braces.
This the man page for the default awk on my Mac. The same information is in the GNU awk manual, it's just buried a little deeper. And the POSIX specification of awk states
An awk program is composed of pairs of the form:
pattern { action }
Either the pattern or the action (including the enclosing brace characters) can be omitted.
A missing pattern shall match any record of input, and a missing action shall be equivalent to:
{ print }
You can see in you examples that instead of semicolons at the end of statements you can separate them with new lines. When you have
/regex/
{ ...
}
it's equivalent to /regex/; {...} which is equal to /regex/{print $0} {...} as you tested the behavior.
Note that BEGIN and END are special markers and they need action statements explicitly since for BEGIN {print $0} is not possible as the default action. That's why the open curly brace should be on the same line. Perhaps due to convenience but it's all consistent.
I want to calculate this in bash file
files : {
{
file {
name: "Bla,java"
line_changes : [45,146,14]
}
}
{
file {
name: "Foo.java"
line_changed : [7,8,9,10]
}
}
}
so I have this
gitOutput=$(git diff origin/master..origin/mybranch)
echo $gitOuput
My problem is :
The output is sooo, not formatted.
Everything is in one line
I cannot parse it logically...
Like Split by \n or split by "diff --git" etc
Also there are no new line .
So if there are some, it doesnot make sense.
So I want to know, is there any pretty format option for git diff
[UPDATE]
I have tried this weird approach
git diff origin/master..origin/mybranch > data.txt
data=$(cat data.txt)
Output is :
The data.txt is absolutely perfect
but the data var. is all messed up...
is it something related to IFS ???
The short answer is: you should add quotes:
gitOutput="$(git diff origin/master..origin/mybranch)"
echo "$gitOuput"
so that line returns are kept as is. This is usually what you want to do in a general matter when having variables in shell.
For a detailed explanation on the use of quotes, see
https://unix.stackexchange.com/questions/68694/when-is-double-quoting-necessary
I wonder if there is an efficient way to delete the first line in a file if it matches a specified pattern. For example, I have a file with data of the following form:
Date,Open,High,Low,Close,Volume,Adj.Volume
2012-01-27,42.38,42.95,42.27,42.68,2428000,42.68
2012-01-26,44.27,44.85,42.48,42.66,5785700,42.66
.
.
.
I want to delete the first line, only if it contains the text (as shown in the example in the first line), and leave it unchanged if it contains only numbers(as in the rest of the lines). This task is quite easy and I've accomplished it by applying the following peace of code which writes each line to a $newFile as long as it does not include Date pattern:
while( <$origFile> )
{
chomp($_);
print $newFile $_ unless ($_ =~ m/Date/g)
}
So as I mentioned, that makes the job done. However it seems that it's a great waste of resources to read each line in a whole file when it is known that the text can appear only in the first line..
Is there any way to accomplish this task more efficiently?
NOTE: I already found an almost similar question here, but since I want my code to be available on Linux and Windows as well, using sed will not help me here.
Thanks in advance!
$. can be used to determine if are processing the first line of the file.
perl -i.bak -ne'print if $. != 1 || !/^Date/;' file
However it seems that it's a great waste of resources to read each line in a whole file
It's impossible to delete from anywhere but the end of a file. To delete from the start or middle, everything that follows in the file needs to be shifted, which means it must be both read and written.
You can only avoid work if the first line doesn't match (by doing nothing at all). If you need to remove the line, you must copy the whole file.
The Tie::File module is ideal for this. It is very efficient as it does block IO instead of reading a line at a time, and it makes the program very simple to write.
use strict;
use warnings;
use Tie::File;
tie my #data, 'Tie::File', 'mydatafile' or die $!;
shift #data if $data[0] =~ /Date/;
untie #data;
Only do the test on the first line, then just run through the rest of the file without checking:
if (defined( $_ = <$origFile> )) {
if ( ! m/Date/o ) { print $newFile $_ }
my $data;
for (;;) {
my $readRes = read($origFile, $data, 0x10000);
if (!defined $readRes) { die "Can't read: $!" }
if ($readRes == 0) { last }
print $newFile $data;
}
}
I am thinking of using sed for reading .properties file, but was wondering if there is a smarter way to do that from bash script?
This would probably be the easiest way: grep + cut
# Usage: get_property FILE KEY
function get_property
{
grep "^$2=" "$1" | cut -d'=' -f2
}
The solutions mentioned above will work for the basics. I don't think they cover multi-line values though. Here is an awk program that will parse Java properties from stdin and produce shell environment variables to stdout:
BEGIN {
FS="=";
print "# BEGIN";
n="";
v="";
c=0; # Not a line continuation.
}
/^\#/ { # The line is a comment. Breaks line continuation.
c=0;
next;
}
/\\$/ && (c==0) && (NF>=2) { # Name value pair with a line continuation...
e=index($0,"=");
n=substr($0,1,e-1);
v=substr($0,e+1,length($0) - e - 1); # Trim off the backslash.
c=1; # Line continuation mode.
next;
}
/^[^\\]+\\$/ && (c==1) { # Line continuation. Accumulate the value.
v= "" v substr($0,1,length($0)-1);
next;
}
((c==1) || (NF>=2)) && !/^[^\\]+\\$/ { # End of line continuation, or a single line name/value pair
if (c==0) { # Single line name/value pair
e=index($0,"=");
n=substr($0,1,e-1);
v=substr($0,e+1,length($0) - e);
} else { # Line continuation mode - last line of the value.
c=0; # Turn off line continuation mode.
v= "" v $0;
}
# Make sure the name is a legal shell variable name
gsub(/[^A-Za-z0-9_]/,"_",n);
# Remove newlines from the value.
gsub(/[\n\r]/,"",v);
print n "=\"" v "\"";
n = "";
v = "";
}
END {
print "# END";
}
As you can see, multi-line values make things more complex. To see the values of the properties in shell, just source in the output:
cat myproperties.properties | awk -f readproperties.awk > temp.sh
source temp.sh
The variables will have '_' in the place of '.', so the property some.property will be some_property in shell.
If you have ANT properties files that have property interpolation (e.g. '${foo.bar}') then I recommend using Groovy with AntBuilder.
Here is my wiki page on this very topic.
I wrote a script to solve the problem and put it on my github.
See properties-parser
One option is to write a simple Java program to do it for you - then run the Java program in your script. That might seem silly if you're just reading properties from a single properties file. However, it becomes very useful when you're trying to get a configuration value from something like a Commons Configuration CompositeConfiguration backed by properties files. For a time, we went the route of implementing what we needed in our shell scripts to get the same behavior we were getting from CompositeConfiguration. Then we wisened up and realized we should just let CompositeConfiguration do the work for us! I don't expect this to be a popular answer, but hopefully you find it useful.
If you want to use sed to parse -any- .properties file, you may end up with a quite complex solution, since the format allows line breaks, unquoted strings, unicode, etc: http://en.wikipedia.org/wiki/.properties
One possible workaround would using java itself to preprocess the .properties file into something bash-friendly, then source it. E.g.:
.properties file:
line_a : "ABC"
line_b = Line\
With\
Breaks!
line_c = I'm unquoted :(
would be turned into:
line_a="ABC"
line_b=`echo -e "Line\nWith\nBreaks!"`
line_c="I'm unquoted :("
Of course, that would yield worse performance, but the implementation would be simpler/clearer.
In Perl:
while(<STDIN>) {
($prop,$val)=split(/[=: ]/, $_, 2);
# and do stuff for each prop/val
}
Not tested, and should be more tolerant of leading/trailing spaces, comments etc., but you get the idea. Whether you use Perl (or another language) over sed is really dependent upon what you want to do with the properties once you've parsed them out of the file.
Note that (as highlighted in the comments) Java properties files can have multiple forms of delimiters (although I've not seen anything used in practice other than colons). Hence the split uses a choice of characters to split upon.
Ultimately, you may be better off using the Config::Properties module in Perl, which is built to solve this specific problem.
I have some shell scripts that need to look up some .properties and use them as arguments to programs I didn't write. The heart of the script is a line like this:
dbUrlFile=$(grep database.url.file etc/zocalo.conf | sed -e "s/.*: //" -e "s/#.*//")
Effectively, that's grep for the key and filter out the stuff before the colon and after any hash.
if you want to use "shell", the best tool to parse files and have proper programming control is (g)awk. Use sed only simple substitution.
I have sometimes just sourced the properties file into the bash script. This will lead to environment variables being set in the script with the names and contents from the file. Maybe that is enough for you, too. If you have to do some "real" parsing, this is not the way to go, of course.
Hmm, I just run into the same problem today. This is poor man's solution, admittedly more straightforward than clever;)
decl=`ruby -ne 'puts chomp.sub(/=(.*)/,%q{="\1";}).gsub(".","_")' my.properties`
eval $decl
then, a property 'my.java.prop' can be accessed as $my_java_prop.
This can be done with sed or whatever, but I finally went with ruby for its 'irb' which was handy for experimenting.
It's quite limited (dots should be replaced only before '=',no comment handling), but could be a starting point.
#Daniel, I tried to source it, but Bash didn't like dots in variable names.
I have had some success with
PROPERTIES_FILE=project.properties
function source_property {
local name=$1
eval "$name=\"$(sed -n '/^'"$name"'=/,/^[A-Z]\+_*[A-Z]*=/p' $PROPERTIES_FILE|sed -e 's/^'"$name"'=//g' -e 's/"/\\"/g'|head -n -1)\""
}
source_property 'SOME_PROPERTY'
This is a solution that properly parses quotes and terminates at a space when not given quotes. It is safe: no eval is used.
I use this code in my .bashrc and .zshrc for importing variables from shell scripts:
# Usage: _getvar VARIABLE_NAME [sourcefile...]
# Echos the value that would be assigned to VARIABLE_NAME
_getvar() {
local VAR="$1"
shift
awk -v Q="'" -v QQ='"' -v VAR="$VAR" '
function loc(text) { return index($0, text) }
function unquote(d) { $0 = substr($0, eq+2) d; print substr($0, 1, loc(d)-1) }
{ sub(/^[ \t]+/, ""); eq = loc("=") }
substr($0, 1, eq-1) != VAR { next } # assignment is not for VAR: skip
loc("=" QQ) == eq { unquote(QQ); exit }
loc("=" Q) == eq { unquote( Q); exit }
{ print substr($1, eq + 1); exit }
' "$#"
}
This saves the desired variable name and then shifts the argument array so the rest can be passed as files to awk.
Because it's so hard to call shell variables and refer to quote characters inside awk, I'm defining them as awk variables on the command line. Q is a single quote (apostrophe) character, QQ is a double quote, and VAR is that first argument we saved earlier.
For further convenience, there are two helper functions. The first returns the location of the given text in the current line, and the second prints the content between the first two quotes in the line using quote character d (for "delimiter"). There's a stray d concatenated to the first substr as a safety against multi-line strings (see "Caveats" below).
While I wrote the code for POSIX shell syntax parsing, that appears to only differ from your format by whether there is white space around the asignment. You can add that functionality to the above code by adding sub(/[ \t]*=[ \t]*/, "="); before the sub(…) on awk's line 4 (note: line 1 is blank).
The fourth line strips off leading white space and saves the location of the first equals sign. Please verify that your awk supports \t as tab, this is not guaranteed on ancient UNIX systems.
The substr line compares the text before the equals sign to VAR. If that doesn't match, the line is assigning a different variable, so we skip it and move to the next line.
Now we know we've got the requested variable assignment, so it's just a matter of unraveling the quotes. We do this by searching for the first location of =" (line 6) or =' (line 7) or no quotes (line 8). Each of those lines prints the assigned value.
Caveats: If there is an escaped quote character, we'll return a value truncated to it. Detecting this is a bit nontrivial and I decided not to implement it. There's also a problem of multi-line quotes, which get truncated at the first line break (this is the purpose of the "stray d" mentioned above). Most solutions on this page suffer from these issues.
In order to let Java do the tricky parsing, here's a solution using jrunscript to print the keys and values in a bash read-friendy (key, tab character, value, null character) way:
#!/usr/bin/env bash
jrunscript -e '
p = new java.util.Properties();
p.load(java.lang.System.in);
p.forEach(function(k,v) { out.format("%s\t%s\000", k, v); });
' < /tmp/test.properties \
| while IFS=$'\t' read -d $'\0' -r key value; do
key=${key//./_}
printf -v "$key" %s "$value"
printf '=> %s = "%s"\n' "$key" "$value"
done
I found printf -v in this answer by #david-foerster.
To quote jrunscript: Warning: Nashorn engine is planned to be removed from a future JDK release