Merge two json in bash (no jq) - bash

I have two jsons :
env.json
{
"environment":"INT"
}
roles.json
{
"run_list":[
"recipe[splunk-dj]",
"recipe[tideway]",
"recipe[AlertsSearch::newrelic]",
"recipe[AlertsSearch]"
]
}
expected output should be some thing like this :
{
"environment":"INT",
"run_list":[
"recipe[splunk-dj]",
"recipe[tideway]",
"recipe[AlertsSearch::newrelic]",
"recipe[AlertsSearch]"
]
}
I need to merge these two json (and other like these two) into one single json using only available inbuilt bash commands.
only have sed, cat, echo, tail, wc at my disposal.

Tell whoever put the constraint "bash only" on the project that bash is not sufficient for processing JSON, and get jq.
$ jq --slurp 'add' env.json roles.json

I couldn't use jq either as I was limited due to client's webhost jailing the user on the command line with limited binaries as most discount/reseller web hosting companies do. Luckily they usually have PHP available and you can do a oneliner command like this which something like what I would place in my install/setup bash script for example.
php -r '$json1 = "./env.json";$json2 = "./roles.json";$data = array_merge(json_decode(file_get_contents($json1), true),json_decode(file_get_contents($json2),true));echo json_encode($data, JSON_PRETTY_PRINT);'
For clarity php -r accepts line feeds as well so using this also works.
php -r '
$json1 = "./env.json";
$json2 = "./roles.json";
$data = array_merge(json_decode(file_get_contents($json1), true), json_decode(file_get_contents($json2), true));
echo json_encode($data, JSON_PRETTY_PRINT);'
Output
{
"environment": "INT",
"run_list": [
"recipe[splunk-dj]",
"recipe[tideway]",
"recipe[AlertsSearch::newrelic]",
"recipe[AlertsSearch]"
]
}

A little bit hacky, but hopefully will do.
env_lines=`wc -l < $1`
env_output=`head -n $(($env_lines - 1)) $1`
roles_lines=`wc -l < $2`
roles_output=`tail -n $(($roles_lines - 1)) $2`
echo "$env_output" "," "$roles_output"

Related

How to create a curl loop to make my code more compact?

I am intending to make a small web scraper script:
I have a shell script scrape.sh which I have made executable with chmod 755
curl is used to scrape the data ( I intent to scrape 30,000 url's )
Content of scrape.sh:
curl https://example.com/something/UID1 --output UID1.html
curl https://example.com/something/UID2 --output UID2.html
curl https://example.com/something/UID3 --output UID3.html
curl https://example.com/something/UID4 --output UID4.html
...
curl https://example.com/something/UID30000 --output UID30000.html
Instead of using 30.000 lines of code in my scrape.sh, what is a more compact way of getting this done?
There are multiple ways how to generate sequential numbers in shell.
You can use seq command with for loop:
for id in $(seq 1 100); do
echo "id is ${id}"
done
In many modern shells (like bash, zsh, ...) you can use for loop more idiomatically:
for (( id = 0; id < 100; id++ )); do
echo "id is ${id}"
done
In those shells you can also use brace expansion.
for id in {1..100}; do
echo "id is $id"
done
As a side note, brace expansions are way cooler than this - you can use prefixes (echo abc{1..5}), you can do sequences of letters (echo {a..z}), or even permutations (echo {1..3}{a..c}{1..3}).

Expanding string variables in bash to toggle a value in JSON

I am trying to create a bash function where I can switch environments, below is what I tried. I installed the npm json package globally to edit the relevant file inline, but that may not be needed.
devUrl () { 'https://some-url.com'; }
testUrl () { 'https://some-test-url.com'; }
switchEnv () { 'json -I -f config.json -e "this.url = (this.url == "$1" ? "$2" : "$1")"'; }
alias switch='switchEnv devUrl testUrl';
what am I missing/doing wrong?
I also tried to template the strings devUrl and testUrl inside the double quotes in the switchEnv function, but that's where I got stuck.
Update:
I tried this:
devUrl='https://some-url.com'
testUrl='https://some-test-url.com'
switchEnv() { json -I -f config.json -e "this.url = (this.url == "$devUrl" ? "$testUrl" : "$devUrl")"; }
but got the following error:
this.url = (this.url == https://some-url.com ? https://some-test-url.com : https://some-url.com)
^
SyntaxError: Unexpected token :
at new Function (<anonymous>)
at main (/usr/local/lib/node_modules/json/lib/json.js:1289:27)
it doesn't like the : after https for some reason.
The below is a sample implementation that does what you're looking for; see the notes below for some details on why it was implemented as it was.
# String assignments
devUrl='https://some-url.com'
testUrl='https://some-test-url.com'
configFile="$PWD/config.json"
# Functions
switchEnv() {
local tempfile
tempfile=$(mktemp "$configFile.XXXXXX")
if jq --arg a "$1" \
--arg b "$2" \
'if .url == $a then .url=$b else .url=$a end' <"$configFile" >"$tempfile"; then
mv -- "$tempfile" "$configFile"
else
rm -f -- "$tempfile"
return 1
fi
}
switch() { switchEnv "$devUrl" "$testUrl"; }
Notes:
Unlike aliases, function bodies should be actual code, not strings containing code.
Storing data (as opposed to code) should be done using variables (be they strings or arrays, as appropriate).
Passing data out-of-band from code allows a malicious value of devUrl or testUrl to escape its quoting and run arbitrary json or jq commands. This is wise, in no small part because these languages become more powerful over time: Old versions of jq had no operations that wouldn't run in constant-time, whereas modern versions of the language allow code to be expressed that can be used for denial-of-service attacks; future versions might also add I/O support, allowing malicious code to have a wider array of surprising behaviors.
Now, let's say you were going to ignore the warning (above) about the importance of separating data and code. How would we modify your current code to behave "correctly" (when the strings being handled are non-malicious)?
switchEnv() {
json -I -f config.json -e 'this.url = (this.url == "'"$devUrl"'" ? "'"$testUrl"'" : "'"$devUrl"'")'; }
}

Self-explaining version of bash expressions

Expressions like this are short, but not super-readable:
if [ -f .bash_profile ]; then
...
fi
There are also other possible flags for expressions, for instance:
Description
-d file
True if file is a directory.
-e file
True if file exists.
-f file
True if file exists and is a regular file.
-L file
True if file is a symbolic link.
-z string
True if string is empty. (most innatural IMO)
-n string
True if string is not empty.
... and others...
Are there longer self-explaining versions? Something like:
[ --file-exists .bash_profile ]
This is extremely well documented already. As you can see, there is no long-form version of those conditional expressions.
If you want to use these in a more readable way, you can always create your own functions:
function is_a_file() { test -f "$1"; }
function is_a_dir() { test -d "$1"; }
#etc.
if is_a_file /the/file/name
then
#do something
fi
test is the canonical name for the [ command that is typically used. Its return value becomes the return value of the function, so we can use it in exactly the same way in an if statement.
No, use comments or use something less cryptic like Python

Conditional use of functions?

I created a bash script that parses ASCII files into a comma delimited output. It's worked great. Now, a new file layout for these files is being gradually introduced.
My script has now two parsing functions (one per layout) that I want to call depending on a specific marker that is present in the ASCII file header. The script is structured thusly:
#!/bin/bash
function parseNewfile() {...parse stuff...return stuff...}
function parseOldfile() {...parse stuff...return stuff...}
#loop thru ASCII files array
i=0
while [ $i -lt $len ]; do
#check if file contains marker for new layout
grep CSVHeaderBox output_$i.ASC
#calls parsing function based on exit code
if [ $? -eq 0 ]
then
CXD=`parseNewfile`
else
CXD=`parseOldfile`
fi
echo ${array[$i]}| awk -v cxd=`echo $CXD` ....
let i++
done>>${outdir}/outfile.csv
...
The script does not err out. It always calls the original function "parseOldfile" and ignores the new one. Even when I specifically feed my script with several files with the new layout.
What I am trying to do seem very trivial. What am I missing here?
EDIT: Samples of old and new file layouts.
1) OLD File Layout
F779250B
=====BOX INFORMATION=====
Model = R15-100
Man Date = 07/17/2002
BIST Version = 3.77
SW Version = 0x122D
SW Name = v1b1645
HW Version = 1.1
Receiver ID = 00089787556
=====DISK INFORMATION=====
....
2) NEW File Layout
F779250B
=====BOX INFORMATION=====
Model = HR22-100
Man Date = 07/17/2008
BIST Version = 7.55
SW Version = 0x066D
SW Name = v18m1fgu
HW Version = 2.3
Receiver ID = 028910170936
CSVHeaderBox:Platform,ManufactureDate,BISTVersion,SWVersion,SWName,HWRevision,RID
CSVValuesBox:HR22-100,20080717,7.55,0x66D,v18m1fgu,2.3,028910170936
=====DISK INFORMATION=====
....
This may not solve your problem, but a potential performance boost: instead of
grep CSVHeaderBox output_$i.ASC
#calls parsing function based on exit code
if [ $? -eq 0 ]
use
if grep -q CSVHeaderBox output_$i.ASC
qrep -q will exit successfully on the first match, so it doesn't have to scan the whole file. Plus you don't have to bother with the $? var.
Don't do this:
awk -v cxd=`echo $CXD`
Do this:
awk -v cxd="$CXD"
I'm not sure if this solves the OP's requirement.
What's the need for awk if your function knows how to parse the file?
#/bin/bash
function f1() {
echo "f1() says $#"
}
function f2() {
echo "f2() says $#"
}
FUN="f1"
${FUN} "foo"
FUN="f2"
${FUN} "bar"
I am bit embarrassed to write this but I solved my "problem".
After gedit (I am on Ubuntu) err-ed out several dozen times about "Trailing spaces", I copied and pasted my code into a new file and re-run my script.
It worked.
I have no explanation why.
Thanks to everyone for taking the time.

Run a string as a command within a Bash script

I have a Bash script that builds a string to run as a command
Script:
#! /bin/bash
matchdir="/home/joao/robocup/runner_workdir/matches/testmatch/"
teamAComm="`pwd`/a.sh"
teamBComm="`pwd`/b.sh"
include="`pwd`/server_official.conf"
serverbin='/usr/local/bin/rcssserver'
cd $matchdir
illcommando="$serverbin include='$include' server::team_l_start = '${teamAComm}' server::team_r_start = '${teamBComm}' CSVSaver::save='true' CSVSaver::filename = 'out.csv'"
echo "running: $illcommando"
# $illcommando > server-output.log 2> server-error.log
$illcommando
which does not seem to supply the arguments correctly to the $serverbin.
Script output:
running: /usr/local/bin/rcssserver include='/home/joao/robocup/runner_workdir/server_official.conf' server::team_l_start = '/home/joao/robocup/runner_workdir/a.sh' server::team_r_start = '/home/joao/robocup/runner_workdir/b.sh' CSVSaver::save='true' CSVSaver::filename = 'out.csv'
rcssserver-14.0.1
Copyright (C) 1995, 1996, 1997, 1998, 1999 Electrotechnical Laboratory.
2000 - 2009 RoboCup Soccer Simulator Maintenance Group.
Usage: /usr/local/bin/rcssserver [[-[-]]namespace::option=value]
[[-[-]][namespace::]help]
[[-[-]]include=file]
Options:
help
display generic help
include=file
parse the specified configuration file. Configuration files
have the same format as the command line options. The
configuration file specified will be parsed before all
subsequent options.
server::help
display detailed help for the "server" module
player::help
display detailed help for the "player" module
CSVSaver::help
display detailed help for the "CSVSaver" module
CSVSaver Options:
CSVSaver::save=<on|off|true|false|1|0|>
If save is on/true, then the saver will attempt to save the
results to the database. Otherwise it will do nothing.
current value: false
CSVSaver::filename='<STRING>'
The file to save the results to. If this file does not
exist it will be created. If the file does exist, the results
will be appended to the end.
current value: 'out.csv'
if I just paste the command /usr/local/bin/rcssserver include='/home/joao/robocup/runner_workdir/server_official.conf' server::team_l_start = '/home/joao/robocup/runner_workdir/a.sh' server::team_r_start = '/home/joao/robocup/runner_workdir/b.sh' CSVSaver::save='true' CSVSaver::filename = 'out.csv' (in the output after "runnning: ") it works fine.
You can use eval to execute a string:
eval $illcommando
your_command_string="..."
output=$(eval "$your_command_string")
echo "$output"
I usually place commands in parentheses $(commandStr), if that doesn't help I find bash debug mode great, run the script as bash -x script
don't put your commands in variables, just run it
matchdir="/home/joao/robocup/runner_workdir/matches/testmatch/"
PWD=$(pwd)
teamAComm="$PWD/a.sh"
teamBComm="$PWD/b.sh"
include="$PWD/server_official.conf"
serverbin='/usr/local/bin/rcssserver'
cd $matchdir
$serverbin include=$include server::team_l_start = ${teamAComm} server::team_r_start=${teamBComm} CSVSaver::save='true' CSVSaver::filename = 'out.csv'
./me casts raise_dead()
I was looking for something like this, but I also needed to reuse the same string minus two parameters so I ended up with something like:
my_exe ()
{
mysql -sN -e "select $1 from heat.stack where heat.stack.name=\"$2\";"
}
This is something I use to monitor openstack heat stack creation. In this case I expect two conditions, an action 'CREATE' and a status 'COMPLETE' on a stack named "Somestack"
To get those variables I can do something like:
ACTION=$(my_exe action Somestack)
STATUS=$(my_exe status Somestack)
if [[ "$ACTION" == "CREATE" ]] && [[ "$STATUS" == "COMPLETE" ]]
...
Here is my gradle build script that executes strings stored in heredocs:
current_directory=$( realpath "." )
GENERATED=${current_directory}/"GENERATED"
build_gradle=$( realpath build.gradle )
## touch because .gitignore ignores this folder:
touch $GENERATED
COPY_BUILD_FILE=$( cat <<COPY_BUILD_FILE_HEREDOC
cp
$build_gradle
$GENERATED/build.gradle
COPY_BUILD_FILE_HEREDOC
)
$COPY_BUILD_FILE
GRADLE_COMMAND=$( cat <<GRADLE_COMMAND_HEREDOC
gradle run
--build-file
$GENERATED/build.gradle
--gradle-user-home
$GENERATED
--no-daemon
GRADLE_COMMAND_HEREDOC
)
$GRADLE_COMMAND
The lone ")" are kind of ugly. But I have no clue how to fix that asthetic aspect.
To see all commands that are being executed by the script, add the -x flag to your shabang line, and execute the command normally:
#! /bin/bash -x
matchdir="/home/joao/robocup/runner_workdir/matches/testmatch/"
teamAComm="`pwd`/a.sh"
teamBComm="`pwd`/b.sh"
include="`pwd`/server_official.conf"
serverbin='/usr/local/bin/rcssserver'
cd $matchdir
$serverbin include="$include" server::team_l_start="${teamAComm}" server::team_r_start="${teamBComm}" CSVSaver::save='true' CSVSaver::filename='out.csv'
Then if you sometimes want to ignore the debug output, redirect stderr somewhere.
For me echo XYZ_20200824.zip | grep -Eo '[[:digit:]]{4}[[:digit:]]{2}[[:digit:]]{2}'
was working fine but unable to store output of command into variable.
I had same issue I tried eval but didn't got output.
Here is answer for my problem:
cmd=$(echo XYZ_20200824.zip | grep -Eo '[[:digit:]]{4}[[:digit:]]{2}[[:digit:]]{2}')
echo $cmd
My output is now 20200824

Resources