Extract JSON from a string with plain text and JSON - bash

Given the following string:
[2018-07-30 12:32:26] local.INFO: {"type":"canonical-log-line","http_method":"POST","http_path":"\/foo","http_status":200,"duration":1.720768928527832,"ts":"2018-07-30 12:32:26"}
How would you select only the JSON object? eg:
{"type":"canonical-log-line","http_method":"POST","http_path":"\/foo","http_status":200,"duration":1.720768928527832,"ts":"2018-07-30 12:32:26"}

I would use sed.
string='[2018-07-30 12:32:26] local.INFO: {"type":"canonical-log-line","http_method":"POST","http_path":"\/foo","http_status":200,"duration":1.720768928527832,"ts":"2018-07-30 12:32:26"}'
echo $string | sed 's/^.*{/{/'

This should work (if you can pipe the output into a python variable and use python) as long as your string only contains 1 pair of open/closing brackets:
import json
import subprocess
#command below is the bash command you want to run:
output = subprocess.check_output(command, shell=True).decode()
# overriding output as i dont know what bash command you're running
output = '[2018-07-30 12:32:26] local.INFO: {"type":"canonical-log-line","http_method":"POST","http_path":"\/foo","http_status":200,"duration":1.720768928527832,"ts":"2018-07-30 12:32:26"} '
json_str = output[output.index('{'): output.index('}')+1]
json_obj = json.loads(json_str)
print(json_obj['type'])

Related

How to pass JSON variable to external bash script in Airflow BashOperator

Here I am getting variables from Airflow GUI in form of JSON
example_vars=Variable.get("example_vars", deserialize_json=True)
dag = DAG('NEW_DAG', description='NEW_DAG', schedule_interval=None, default_args = default_args,catchup=False )
shell_command = 'path-to-my-file/my_script.sh '
running_task = BashOperator( task_id='task_to_run', bash_command=shell_command, trigger_rule="all_done",params={"val": example_vars}, dag=dag )
running_task
I tried from this page Airflow BashOperator: Passing parameter to external bash script
in the bash script---->
echo {{ params.val }}
and it prints {{ params.val }} not the json file.
You need
shell_command = 'path-to-my-file/my_script.sh "{{example_vars}}"'
See the examples at https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/bash.html
import json
import shlex
# JSON variable
data = {'key': 'value'}
# Convert JSON variable to string
json_data = json.dumps(data)
# Quote the string to escape any special characters
escaped_json_data = shlex.quote(json_data)
# Pass the quoted string to the bash script
bash_command = './script.sh ' + escaped_json_data
# Create a BashOperator
bash_task = BashOperator(
task_id='bash_task',
bash_command=bash_command,
dag=dag)
In bash script:
#!/bin/bash
json_data=$1

add prefix in shell script to json resonse

I want to add some prefix to the field from output of hitting endpoint.
file.sh
result=$(curl -s <someEndpoint>)
echo $result
I have a cronjob to hit that endpoint regularly;
* * * * * /root/file.sh | jq .field > /root/file.txt 2>&1
I need to add "start" to the .field.
I created another file2.sh:
prefix="start"
val=$(cat /root/file.txt)
concat="${prefix}${val}"
echo $concat
the output looks like:
start"value from file.txt"
I want "start value from file.txt". how can I do that?
It seems like the better solution would be to use jq -r .field to output plaintext instead of a JSON string, but since you explicitly want to work with json, you can use jq again:
$ echo '"string"' | jq '"start" + .'
"startstring"

How to call a variable created in the script in Nextflow?

I have a nextflow script that creates a variable from a text file, and I need to pass the value of that variable to a command line order (which is a bioconda package). Those two processes happen inside the "script" part. I have tried to call the variable using the '$' symbol without any results, I think because using that symbol in the script part of a nextflow script is for calling variables defined in the input part.
To make myself clearer, here is a code sample of what I'm trying to achieve:
params.gz_file = '/path/to/file.gz'
params.fa_file = '/path/to/file.fa'
params.output_dir = '/path/to/outdir'
input_file = file(params.gz_file)
fasta_file = file(params.fa_file)
process foo {
//publishDir "${params.output_dir}", mode: 'copy',
input:
path file from input_file
path fasta from fasta_file
output:
file ("*.html")
script:
"""
echo 123 > number.txt
parameter=`cat number.txt`
create_report $file $fasta --flanking $parameter
"""
}
By doig this the error I recieve is:
Error executing process > 'foo'
Caused by:
Unknown variable 'parameter' -- Make sure it is not misspelt and defined somewhere in the script before using it
Is there any way to call the variable parameter inside the script without Nextflow interpreting it as an input file? Thanks in advance!
The documentation re the script block is useful here:
Since Nextflow uses the same Bash syntax for variable substitutions in
strings, you need to manage them carefully depending on if you want to
evaluate a variable in the Nextflow context - or - in the Bash
environment execution.
One solution is to escape your shell (Bash) variables by prefixing them with a back-slash (\) character, like in the following example:
process foo {
script:
"""
echo 123 > number.txt
parameter="\$(cat number.txt)"
echo "\${parameter}"
"""
}
Another solution is to instead use a shell block, where dollar ($) variables are managed by your shell (Bash interpreter), while exclamation mark (!) variables are handled by Nextflow. For example:
process bar {
echo true
input:
val greeting from 'Hello', 'Hola', 'Bonjour'
shell:
'''
echo 123 > number.txt
parameter="$(cat number.txt)"
echo "!{greeting} parameter ${parameter}"
'''
}
declare "parameter" in the top 'params' section.
params.parameter="1234"
(..)
script:
"""
(...)
create_report $file $fasta --flanking ${params.parameter}
(...)
"""
(...)
and call "nextflow run" with "--parameter 87678"

unix - how to construct IFS for passing in context of file

Am I constructing this properly? I can't figure how to debug this.
Basically, I'm trying to pass in a text file (login.info):
foo
bar
To this shell script to be used as parameters like so:
#/usr/bin/ksh
#Base dir
BASEDIR=/home/scripts
export BASEDIR;
#Key
KEY=$BASEDIR/login.info
export KEY;
IFS="
"
arr=( $(<$KEY) )
echo "username=${arr[0]} password=${arr[1]}"
Getting this error:
./tst.sh[12]: Syntax error at line 12 : `(' is not expected.
It seems like your version of ksh doesn't understand (...) in array assignment. Maybe this will work better:
set -A arr $(cat $KEY)

Calling external script in matlab and capturing output

Hey so I have a bash command that echos a string based on reading some file. Say for simplicity it is like this
for line in `cat file`
do
if [ "$line" == "IwantThisLine" ]
then
echo "True"
fi
done
And I have it saved as its own individual script. Its called readRef.sh. So now I want to call it in matlab and store whatever it outputs in a variable! I am not entirely sure on how to do that, I seem to get an error when using evalc() on a system(). But it could be just me messing up quotations.
I tried something like
evalc(system(['./readRef.sh ' bamfile']))
The "bamfile" is a variable that is just a string to the path of a bamfile.
I get this error.
>> tes = evalc(system(['./readRef.sh ' smplBamFile]))
hg18
??? Undefined function or method 'evalc' for input arguments of type 'double'.
Coincidentally it does spit out "hg18" which is what I want to set the matlab variable to be.
Oh, I see. I don't think you need evalc at all. Reading the system docs you can just do:
[status, result] = system('echo True; echo "I got a loverly bunch of coconuts"')
And result will be
True
I got a loverly bunch of coconuts
So just do:
[status, result] = system(['./readRef.sh ' smplBamFile])
The reason evalc isn't working is that it requires its input to be a Matlab expression in a string, but you are passing it the result of system.
You could try:
evalc("system(['./readRef.sh ' smplBamFile])")
See how I'm passing in the system(...) as a string?
The reason you get this error is because system(...) returns the return-code of the command it ran, not its output. To capture its output, use
[~, output] = system(...)
tes = evalc(output);

Resources