Bash Loop Through URLs Issue - bash

I'm using cURL request to google safe browsing API to check for any of my requested site is malicious or not. When i'm issuing the following command
$curl -H "Content-Type: application/json" -X POST -d ' { "client": { "clientId": "Test", "clientVersion": "1.0.0" }, "threatInfo": { "threatTypes": ["MALWARE", "SOCIAL_ENGINEERING","THREAT_TYPE_UNSPECIFIED","UNWANTED_SOFTWARE","POTENTIALLY_HARMFUL_APPLICATION"], "platformTypes": ["WINDOWS","PLATFORM_TYPE_UNSPECIFIED","ANY_PLATFORM"], "threatEntryTypes": ["URL"], "threatEntries": [ {"url":"hxxp://bookmyroom.pk/assets/timepicker/f.exe"} ] } }' https://safebrowsing.googleapis.com/v4/threatMatches:find?key=AIzaSyD1IMgjaHEza6e9m_jwtjBgPmJX0IMKKIs
Caution: I have "xx"ed instead "tt" for the http string. This site might be a potential malicious site.So do not open it using browser.
I'm getting JSON response as
{
"matches": [
{
"threatType": "UNWANTED_SOFTWARE",
"platformType": "ANY_PLATFORM",
"threat": {
"url": "hxxp://bookmyroom.pk/assets/timepicker/f.exe"
},
"cacheDuration": "300.000s",
"threatEntryType": "URL"
}
]
}
Caution: I have "xx"ed instead "tt" for the http string. This site might be a potential malicious site.So do not open it using browser.
When i'm doing the same in loop
#check.sh
for i in $(cat malsite.txt); do
content="$(curl -H "Content-Type: application/json" -X POST -d ' { "client": { "clientId": "Test", "clientVersion": "1.0.0" }, "threatInfo": { "threatTypes": ["MALWARE", "SOCIAL_ENGINEERING","THREAT_TYPE_UNSPECIFIED","UNWANTED_SOFTWARE","POTENTIALLY_HARMFUL_APPLICATION"], "platformTypes": ["WINDOWS","PLATFORM_TYPE_UNSPECIFIED","ANY_PLATFORM"], "threatEntryTypes": ["URL"], "threatEntries": [ {"url":"$i"} ] } }' https://safebrowsing.googleapis.com/v4/threatMatches:find?key=AIzaSyD1IMgjaHEza6e9m_jwtjBgPmJX0IMKKIs)"
echo "$i"
echo "$content"
done
#malsite.txt
"hxxp://bookmyroom.pk/assets/timepicker/f.exe"
Caution: I have "xx"ed instead "tt"ing for the http string.This site might be a potential malicious site.So do not open it using browser.
I'm not getting any results. It just returns empty result.
#Result of /.check.sh :
{}
Not sure , Where i'm making mistakes. Any thoughts ?

Related

How to send a message with button in shell to Telegram bot?

I'm using this and works. But how to put a button blew the message? I tried a lot and still got no idea.
Sorry i forgot to tell it's about Telegram bot api. I want to send a message with button blew.
function sendadmin(){
if [ "$admin_id" -ne "$chat_id" ]; then
curl -s \
-d parse_mode="MarkdownV2" \
-d text="$stext" \
-d chat_id="$admin_id" \
-d -sendChatAction="videos" \
-d reply_markup="" \
https://api.telegram.org/bot$bot_token/sendMessage
fi
}
stext="||Hello darling||"
sendadmin
I tried solution of this. And it's return Bad Request: can't parse reply keyboard markup JSON object error.
reply=$(cat <<-EOF
{
"inline_keyboard": [
[
{
"text": "Button1",
"callback_data": "lt"
},
{
"text": "Button1",
"callback_data": "rt"
}
],
[
{
"text": "Button3",
"callback_data": "ls"
}
]
]
}
EOF
)
Using car <<EOF EOF to give that json format to a variable and then use it like reply_markup="$reply" . I just found out this way.

Send request to elastic watcher via curl

ELK 7.X
I am trying to create elastic search watcher with curl using the input file. Something like this
curl -X PUT _watcher/watch/cluster_health_watch --data-binary #inputFile
1) What is the file type to be used ? Most of the data is json, but in "actions" field when sending an email, the email body can be HTML !
2) Is there any way that the HTML in the body can be referred from an external file, such that input file can be json ?
Just escaped the double quotes in the html string by adding "\".
Ex:-
<h3 style=\"color:red\"></h3>
"actions": {
"send_email": {
"email": {
"to": "xxxx#gmail.com",
"subject": "My Subject",
"body": {
"html": "<h3 style=\"color:red\"> There was a problem</h3>"
}
}
}
}
curl -X PUT _watcher/watch/cluster_health_watch -H 'Content-Type: application/json' --data-binary #inputFile.json

How to read response of a JIRA REST API

Using the Atlassian documentation, I'm able to create a new JIRA ticket using curl command. I want to understand how to read the response of this command into a variable? Essentially i want to store the JIRA ticket ID into a variable that can be used in other scripts.
Reference Link: https://developer.atlassian.com/server/jira/platform/jira-rest-api-examples/
_SAMPLE_CODE_
curl -D- -u charlie:charlie -X POST --data '{
"fields": {
"project":{
"key": "TEST"
},
"summary": "REST ye merry gentlemen.",
"description": "Creating of an issue using project keys and issue type names using the REST API",
"issuetype": {
"name": "Bug"
}
}
}' -H "Content-Type: application/json" http://localhost:8080/rest/api/2/issue/
_SAMPLE_RESPONSE_
{
"id":"39000",
"key":"TEST-101",
"self":"http://localhost:8080/rest/api/2/issue/39000"
}
From the above log, I want to understand how can we fetch the "key" (i.e. JIRA number or JIRA ID) from response (essentially stdout) into a variable.
you can use jq command line tool to extract value from json:
#!/bin/bash
key=$(curl -s -u charlie:charlie -X POST --data '{
"fields": {
"project":{
"key": "TEST"
},
"summary": "REST ye merry gentlemen.",
"description": "Creating of an issue using project keys and issue type names using the REST API",
"issuetype": {
"name": "Bug"
}
}
}' -H "Content-Type: application/json" http://localhost:8080/rest/api/2/issue/|jq -r .key)
echo "key: $key"
jq command is not installed by default on linux systems.
you will need to install it manually:
sudo apt install jq
sudo yum install jq

Missing newline for adding with bulk API

I want to add the following file to Elasticsearch using the bulk API:
{"_id":{"date":"01-2007","profile":"Da","dgo":"DGO_E_AIEG","consumerType":"residential"},"value":{"min":120.42509,"minKwh":0.20071,"nbItems":6.0}}
using the command
curl -XPOST -H 'Content-Type: application/json' localhost:9200/_bulk --data-binary Downloads/bob/test.json
but I got the following mistake:
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\n]"}],"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\n]"},"status":400}
NB: The file clearly has a empty line at the end
In the docs it says:
NOTE: the final line of data must end with a newline character \n.
There is an example above that of what the document is expected to look like. https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html. Perhaps adding \n at the end of each line would fix the issue.
UPDATE:
There might be something wrong with the way you have placed your data into your JSON file. For example, the following data is in example.json:
{ "index" : { "_index" : "example", "_type" : "doc", "_id" : "1" } }
{ "field1" : "value1" }
<space here>
When running the following curl command, it works:
curl -X POST -H "Content-Type: application/x-ndjson" localhost:9200/_bulk --data-binary "#example.json"
It could be that you're not including something important in your JSON file, or you don't have "#your_file.json", or like the other poster mentioned, you don't have the content-type as application/x-ndjson.
The answer is very simple
{ "index":{ "_index":"schools_gov", "_type":"school", "_id":"1" } }
{ "name":"Model School", "city":"Hyderabad"}
{ "index":{ "_index":"schools_gov", "_type":"school", "_id":"2" } }
{ "name":"Government School", "city":"Pune"}
is not going to work but the below json will work
{ "index":{ "_index":"schools_gov", "_type":"school", "_id":"1" } }
{ "name":"Model School", "city":"Hyderabad"}
{ "index":{ "_index":"schools_gov", "_type":"school", "_id":"2" } }
{ "name":"Government School", "city":"Pune"}
//Give a new line here. Not '\n' but the actual new line.
The HTTP command would be POST http://localhost:9200/schools_gov/_bulk
As the error states, you simply need to add a new line to the end of the file.
If you are on a *nix system, you can do this:
echo "\n" >> Downloads/bob/test.json
Also, as explained in the documentation https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html, the Content-Type should be application/x-ndjson
When sending requests to this endpoint the Content-Type header should
be set to application/x-ndjson
So the command should be:
curl -XPOST -H 'Content-Type: application/x-ndjson' localhost:9200/_bulk --data-binary Downloads/bob/test.json
The error message is very confusing. I typed -data-binary and got the same message. The message sent me to completely wrong direction.

How to update couchbase lite view with Rest API?

How to update couchbase lite view with Rest API ?
From Rest API how to tell indexer that view is updated . I have tried the below code but it did not work.It still returns the old index.
What is the correct way of telling indexer that view is updated so that it can recreate the index.
'PUT'
{db}/_design/todo
{
"_rev":"hf675757577hhfh",
"views":{
"list":{
"map":function(doc){
if(doc.type=='list')
{
emit(doc._id,{"name":doc.name});
}
},
//"version":"1.0" (I have tryied this but not work)
}
}
}
//My view create request was like below:
{db}/_design/todo
{
"views":{
"list":{
"map":function(doc){
if(doc.type=='list')
{
emit(doc._id,{"name":doc.name});
}
},
//"version":"1.0" (I have tryied this but not work)
}
}
}
It looks like you may just have some formatting problems. This shows how to do what you're trying from the command line:
curl -X PUT 'http://localhost:4985/db/_design/todo' --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{ "_rev": "hf675757577hhfh", "views": { "list": { "map": "function(doc) { if (doc.type == \"list\") { emit(doc._id, { \"name\": doc.name }); }}"}}}'
You can test your results with this command:
curl -X GET 'http://localhost:4985/db/_design/todo/_view/list'
You may want to refer to the documentation, which has more examples, at https://developer.couchbase.com/documentation/mobile/current/guides/sync-gateway/views/index.html

Resources