Here is the bash function to parse a json file use 'jq' command:
jq_fullpath_endkey() {
PATHARRAY=$(jq -c 'paths | select(.[-1] == "'$keyword'")|map(strings |= ".\(.)" | numbers |= "[\(.)]") | join("")' **${news.json}**)
}
The news.json is the json file that contains all the content I'd like to parse with jq.
The function works once I replace ${news.json} with a variable named response which contains news.json content as string.
Below is the command which works:
PATHARRAY=$(jq -c 'paths | select(.[-1] == "'$keyword'")|map(strings |= ".\(.)" | numbers |= "[\(.)]") | join("")'**<<< "$response"**)
My question is how can I use 'json file' as part of the jq cmd ?
I suspect there is something wrong with double/single quote I am using.
I figure it out with "parameter substitution" in bash
myfile="news.json"
PATHARRAY=$(jq -c 'paths | select(.[-1] == "'$keyword'")|map(strings |= ".\(.)" | numbers |= "[\(.)]") | join("")' ${myfile})
Related concepts:
variable substitution
command substitution
I have a text file:
org.jitsi.videobridge.xmpp.user.shard-1.HOSTNAME=localhost
org.jitsi.videobridge.xmpp.user.shard-1.DOMAIN=auth.jc.name.com
org.jitsi.videobridge.xmpp.user.shard-1.USERNAME=name
org.jitsi.videobridge.xmpp.user.shard-1.PASSWORD=Hfr*7462
org.jitsi.videobridge.xmpp.user.shard-1.MUC_JIDS=JvbBredjoy#internal.auth.jc.name.com
org.jitsi.videobridge.xmpp.user.shard-1.MUC_NICKNAME=7896aee5-fgre-4b02-4569-0bcc75ed1d0d
I created a bash script:
#!/bin/bash
DPATH="/etc/jitsi/videobridge/sip-communicator.properties"
k=$(grep -o 'shard-1' $DPATH) # shard ends by a number#
i=$(grep -o 'shard-1' $DPATH | cut -c7)
m=$((i+1))
n="shard-$m"
sed -i "s|${k}|${n}|g" $DPATH
But I get error:
/home/scripts# ./shard_number
./shard_number: line 5: 1
1
1
1
1
1: syntax error in expression (error token is "1
1
1
1
1")
sed: -e expression #1, char 9: unterminated `s' command
Could you please help to solve this issue? Thank you.
If you call your script with bash -x /path/to/your/script or add set -x somewhere at the start of your script (after the #!shebang, but before the commands you want to debug), you will see that your grep commands return not a single 'shard-1' but rather one 'shard-1' per line :
++ grep -o shard-1 /etc/jitsi/videobridge/sip-communicator.properties
+ k='shard-1
shard-1
shard-1
shard-1
shard-1
shard-1'
Once cut, that gives the 1\n1\n1\n1\n1\n string that is mentionned in your error output as an invalid token for the $(( ... )) expression, which also breaks the syntax of your sed substitution :
++ cut -c7
++ grep -o shard-1 /etc/jitsi/videobridge/sip-communicator.properties
+ i='1
1
1
1
1
1'
Make that string a single number (for instance piping your grep into sort -u to unicize all the shards found) and your script will work just fine :
#!/bin/bash
DPATH="/etc/jitsi/videobridge/sip-communicator.properties"
k=$(grep -o 'shard-1' $DPATH | sort -u) # shard ends by a number#
i=$(grep -o 'shard-1' $DPATH | sort -u | cut -c7)
m=$((i+1))
n="shard-$m"
sed -i "s|${k}|${n}|g" $DPATH
You can try it here. Also check this test if you want to see your initial script debugged.
I am trying to extract two #define values from a C header file to use them in a shell script. So I use grep to find them and then print them. However, the variables are sometimes empty.
// main.h
#define DEVICE_NO 1
#define FW_VERSION 1
And the script file is
#!/bin/bash -
read_version()
{
echo $(grep $1 "$projectdir/Inc/main.h" | cut -d ' ' -f 3-)
}
device_no=$(read_version "DEVICE_NO")
fw_version=$(read_version "FW_VERSION")
echo "DEVICE_NO = $device_no, FW_VERSION = $fw_version"
So the expectation is that the output to be:
DEVICE_NO = 1, FW_VERSION = 1
but sometimes it turns to be
5
DEVICE_NO = , FW_VERSION = 1
It randomly misses one or both of the values. The header file does not change so it's not coming from there.
UPDATE
As commented I thought maybe the windows line ending is a problem so I piped the output to tr and removed \r but it did not make any difference. also tried var=$(grep FW_VERSION file); $(echo ${var//[$'\t\r\n']} | cut ... to no avail.
I tried using awk instead of cut but got the same result.
I redirected the error inside the command to the standard output ($(grep $1 file | cut -d ' ' -f 3 2>&1) but did not get any extra information
I split the command to a grep part and a cut, the grep never misses but the output of cut randomly gives an empty string as output.
I still have no idea where that 5 is coming from, there is nothing in cut or awk manuals that throws a 5 to either standard output or stderr.
when I type this directly into the terminal - it works as expected and returns 0 if the app is not running, and 1 if it is.
lsappinfo list | grep -v grep | grep bundleID | grep com.test.myapp | wc -l
However, when I use the code below (swift 3 - macOS), it says it is an unrecognised command?
// DECLARE TASK
let task = Process()
// DEFINE THE PATH
task.launchPath = "/usr/bin/lsappinfo"
// DEFINE THE ARGUMENTS
task.arguments = ["list | grep -v grep | grep bundleID | grep com.test.myapp | wc -l"]
// DECLARE outputPipe
let outputPipe = Pipe()
// RUN THE TASK
task.launch()
// DECLARE data
let data = outputPipe.fileHandleForReading.readDataToEndOfFile()
// DECLARE output AS THE UTF-8 STRING OF THE TERMINAL'S OUTPUT
let output = NSString(data: data, encoding: String.Encoding.utf8.rawValue)
print(output!)
if output == "0" {
print("App is not running!")
} else {
print("App is running!")
}
// PAUSE UNTIL COMPLETED
task.waitUntilExit()
Can someone please tell me where I have gone wrong, as I am new to Swift, and still struggling to get my head around the language / syntax.
Thank you all in advance.
Piping operations are not arguments to the process, they are instructions to a shell to connect several separate processes. If the task construct in swift expects a process and a set of command-line argument, then one way to do this here would maybe be to launch the bash binary, and then put -c "lsappinfo list | grep -v grep | grep bundleID | grep com.test.myapp | wc -l" as the full set of arguments. That way, you would let bash sort out what's needed to accomplish the piping.
I think of xargs as the map function of the UNIX shell. What is the filter function?
EDIT: it looks like I'll have to be a bit more explicit.
Let's say I have to hand a program which accepts a single string as a parameter and returns with an exit code of 0 or 1. This program will act as a predicate over the strings that it accepts.
For example, I might decide to interpret the string parameter as a filepath, and define the predicate to be "does this file exist". In this case, the program could be test -f, which, given a string, exits with 0 if the file exists, and 1 otherwise.
I also have to hand a stream of strings. For example, I might have a file ~/paths containing
/etc/apache2/apache2.conf
/foo/bar/baz
/etc/hosts
Now, I want to create a new file, ~/existing_paths, containing only those paths that exist on my filesystem. In my case, that would be
/etc/apache2/apache2.conf
/etc/hosts
I want to do this by reading in the ~/paths file, filtering those lines by the predicate test -f, and writing the output to ~/existing_paths. By analogy with xargs, this would look like:
cat ~/paths | xfilter test -f > ~/existing_paths
It is the hypothesized program xfilter that I am looking for:
xfilter COMMAND [ARG]...
Which, for each line L of its standard input, will call COMMAND [ARG]... L, and if the exit code is 0, it prints L, else it prints nothing.
To be clear, I am not looking for:
a way to filter a list of filepaths by existence. That was a specific example.
how to write such a program. I can do that.
I am looking for either:
a pre-existing implementation, like xargs, or
a clear explanation of why this doesn't exist
If map is xargs, filter is... still xargs.
Example: list files in the current directory and filter out non-executable files:
ls | xargs -I{} sh -c "test -x '{}' && echo '{}'"
This could be made handy trough a (non production-ready) function:
xfilter() {
xargs -I{} sh -c "$* '{}' && echo '{}'"
}
ls | xfilter test -x
Alternatively, you could use a parallel filter implementation via GNU Parallel:
ls | parallel "test -x '{}' && echo '{}'"
So, youre looking for the:
reduce( compare( filter( map(.. list()) ) ) )
what can be rewiritten as
list | map | filter | compare | reduce
The main power of bash is a pipelining, therefore isn't need to have one special filter and/or reduce command. In fact nearly all unix commands could act in one (or more) functions as:
list
map
filter
reduce
Imagine:
find mydir -type f -print | xargs grep -H '^[0-9]*$' | cut -d: -f 2 | sort -nr | head -1
^------list+filter------^ ^--------map-----------^ ^--filter--^ ^compare^ ^reduce^
Creating a test case:
mkdir ./testcase
cd ./testcase || exit 1
for i in {1..10}
do
strings -1 < /dev/random | head -1000 > file.$i.txt
done
mkdir emptydir
You will get a directory named testcase and in this directory 10 files and one directory
emptydir file.1.txt file.10.txt file.2.txt file.3.txt file.4.txt file.5.txt file.6.txt file.7.txt file.8.txt file.9.txt
each file contains 1000 lines of random strings some lines are contains only numbers
now run the command
find testcase -type f -print | xargs grep -H '^[0-9]*$' | cut -d: -f 2 | sort -nr | head -1
and you will get the largest number-only line from each files like: 42. (of course, this can be done more effectively, this is only for demo)
decomposed:
The find testcase -type f -print will print every plain files so, LIST (and reduced only to files). ouput:
testcase/file.1.txt
testcase/file.10.txt
testcase/file.2.txt
testcase/file.3.txt
testcase/file.4.txt
testcase/file.5.txt
testcase/file.6.txt
testcase/file.7.txt
testcase/file.8.txt
testcase/file.9.txt
the xargs grep -H '^[0-9]*$' as MAP will run a grep command for each file from a list. The grep is usually using as filter, e.g: command | grep, but now (with xargs) changes the input (filenames) to (lines containing only digits). Output, many lines like:
testcase/file.1.txt:1
testcase/file.1.txt:8
....
testcase/file.9.txt:4
testcase/file.9.txt:5
structure of lines: filename colon number, want only numbers so calling a pure filter, what strips out the filenames from each line cut -d: -f2. It outputs many lines like:
1
8
...
4
5
Now the reduce (getting the largest number), the sort -nr sorts all number numerically and reverse order (desc), so its output is like:
42
18
9
9
...
0
0
and the head -1 print the first line (the largest number).
Of course, you can write your own list/filter/map/reduce functions directly with bash programming constructions (loops, conditions and such), or you can employ any fullblown scripting language like perl, special languages like awk, sed "language", or dc (rpn) and such.
Having an special filter command such:
list | filter_command cut -d: -f 2
is simple doesn't needed, because you can use directly the
list | cut
You can have awk do the filter and reduce function.
Filter:
awk 'NR % 2 { $0 = $0 " [EVEN]" } 1'
Reduce:
awk '{ p = p + $0 } END { print p }'
I totally understand your question here as a long time functional programmer and here is the answer: Bash/unix command pipelining isn't as clean as you'd hoped.
In the example above:
find mydir -type f -print | xargs grep -H '^[0-9]*$' | cut -d: -f 2 | sort -nr | head -1
^------list+filter------^ ^--------map-----------^ ^--filter--^ ^compare^ ^reduce^
a more pure form would look like:
find mydir | xargs -L 1 bash -c 'test -f $1 && echo $1' _ | grep -H '^[0-9]*$' | cut -d: -f 2 | sort -nr | head -1
^---list--^^-------filter---------------------------------^^------map----------^^--map-------^ ^reduce^
But, for example, grep also has a filtering capability: grep -q mypattern which simply return 0 if it matches the pattern.
To get a something more like what you want, you simply would have to define a filter bash function and make sure to export it so it was compatible with xargs
But then you get into some problems. Like, test has binary and unary operators. How will your filter function handle this? Hand, what would you decide to output on true for these cases? Not insurmountable, but weird. Assuming only unary operations:
filter(){
while read -r LINE || [[ -n "${LINE}" ]]; do
eval "[[ ${LINE} $1 ]]" 2> /dev/null && echo "$LINE"
done
}
so you could do something like
seq 1 10 | filter "> 4"
5
6
7
8
9
As I wrote this I kinda liked it