Properly Nesting Arrays Within Objects In Bash - bash

I'm building a Bash shell script to run as a post-build task in Jenkins.
This script sets some information conditionally and then sends it to Slack's Inbound Webhook API.
The information is sent via a cURL request, and the JSON object that I send can accept an attachments key which points to an array of objects.
This is how my code for the attachments array and overall JSON object
attachments="[\"{\"fallback\":\"This\u0020is\u0020a\u0020fallback\u0020message\u0020just\u0020in\u0020case\",\"color\":\"#36a64f\",\"author_name\":\"$author_name\",\"text\":\"$text\"}\"]"
json="{\"channel\":\"$channel\",\"username\":\"$username\",\"icon_emoji\":\"$emoji\",\"attachments\":\"$attachments\"}"
In Jenkins' console output, I'm seeing the following translation in my cURL request:
curl -X POST --data '{"channel":"#jenkinsslacktest","username":"Jenkins-Bot","icon_emoji":":rocket:","attachments":"["{"fallback":"This\u0020is\u0020a\u0020fallback\u0020message\u0020just\u0020in\u0020case","color":"#36a64f","author_name":"TestAuthor","text":":rocket:\u0020:rocket:\u0020SUCCESS!\u0020:rocket:\u0020:rocket:"}"]"}' https://hooks.slack.com/services/link-to-my-webhook
In the way it's formatted, my webhook does not trigger appropriately and no message is sent into my chosen Slack channel.
If I replaced the attachments key/value pair and just placed the text line in its place, this call is successful.
It seems like I'm just not escaping or formatting this attachments value properly. What should I be doing differently?

The only issue here is the JSON syntax of the attachments array. You need to remove the quotes around [ and ] to make it valid JSON syntax. Then it will work.
Here is the correct bash line:
curl -X POST --data '{"channel":"#jenkinsslacktest","username":"Jenkins-Bot","icon_emoji":":rocket:","attachments":[{"fallback":"This\u0020is\u0020a\u0020fallback\u0020message\u0020just\u0020in\u0020case","color":"#36a64f","author_name":"TestAuthor","text":":rocket:\u0020:rocket:\u0020SUCCESS!\u0020:rocket:\u0020:rocket:"}]}' https://hooks.slack.com/services/link-to-my-webhook
You can use a tool like JSONLint or JSONViewer to verify your JSON syntax is correct.

Related

How to send a post from twillio webhook using the body instead the params in the request?

There is a way to config the Twilio webhooks in the conversation product to send a post request to an endpoint and in the body send the information instead in the params?
You would pass a payload of the JSON you want to send in your post body and then pass in a header called x-www-form-urlencoded which tells Twilio that you want the parameters to be sent in the body as form data. I'm not sure if it's limited to only a few parameters or not but I know that it works with \"To\" and \"From\" (as they need to be URL encoded). It would definitely work with MessageSid.
You could also use the \"Bulk\" post body format, which is just JSON. This would allow you to pass more parameters since it's just JSON. (You don't need to url encode them if you do this, so no need to have x-www-form-urlencoded header.)
{
\"To\": \"+15551235555\",
\"From\": \"+15551234567\",
\"Body\": \"A text message\",
...: ...
}
You should be able to send the information you want, along with the headers, from your endpoint and have it pass through Twilio.
Looks like this:
curl -X POST https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Messages -d 'From=%2B15551234567&To=%2B15551235555&Body=Test' -u '{AccountSid}:{AuthToken}'
You can pass any JSON you want as a body using this option but make sure you've set your \"Content-Type\" header to \"application/x-www-form-urlencoded\". This is pretty straightforward and makes it easy to pass in whatever parameters you want.
This isn't limited to text messages! This is exactly how I push data back into a Conversation or Action resource too so it'll work for things like card pushes too! You can use this to programmatically create a response that Twilio will process and then act on in your Conversation or Action instance.
And yeah … if you're going to support a webhook that takes form data then I would suggest adding some basic security checks since anyone could just post random stuff as form data if they wanted and get access to your endpoint. I'd recommend checking the Request Method as well to make sure it's POST.
If you're worried about someone passing in a bad value then you can just check the request body against some regex. I'd recommend checking the Twilio-To and Twilio-From params as well. You could also use the request header too, which is passed along with all webhooks:
header('Access-Control-Allow-Origin: *');
header('Access-Control-Allow-Methods: POST');
header('Access-Control-Allow-Headers: xxx');

Using the content of an HTML file in a JSON string in shell or Makefile (AWS)

I am attempting to use the AWS apigateway CLI to set an Integration Response mapping template for one of my endpoints in my API. This particular case involves using text/html as my Content-Type and as such my mapping template is raw HTML code. Depending on certain circumstances, I would like to use the CLI to set the mapping template to different HTML code. As such, I am attempting to read the HTML content from a specified file into the --response-template JSON string, which will fill in the mapping template.
The command to do this is as follows:
aws apigateway put-integration-response
--rest-api-id $(restApiId) --resource-id $(resourceId) \
--http-method POST --status-code 200 \
--selection-pattern "" \
--response-parameters '{"method.response.header.Content-Type":"'"'"'text/html'"'"'"}' \
--response-templates '{"text/html": "<!DOCTYPE html><html lang="en"><body><p>Hello, world</p></body></html>"}'
I was originally attempting to do this in a Makefile, hence the variables, but I am open to creating a shell script to achieve the same thing. My HTML file is much larger than the example above and contains scripts with functions, brackets, JSON objects, plenty of strings to stylesheets and whatnot - in other words, many characters that need to be escaped. I cannot manually change the HTML file nor am I sure if I can modify it with escape characters since the mapping template needs to be the exact HTML content as that is what is being rendered.
Is it possible for me to place the entire contents of an HTML file into the value portion of that JSON string? If so, how can I get this to work?
Any help is appreciated.
Simon offered a possible solution to this while using the AWS CLI approach but I have solved this in another way.
I decided to use a Python script with the AWS Python SDK, Boto3. I used Python's json library to dump the content of the HTML file into valid JSON format using json.dumps(). I put that as the value to the JSON object that eventually was used in the Boto3 function below and did an inline replacement of the value as I sent it with json.loads(). I followed the top answer from this link on how to convert html source code to json object.
I used the Boto3 put_integration_response() function and sent that JSON object as the --response-template, which worked for me.

Is it possible to receive a gzipped response via elasticsearch-py?

I have an API (via hug) that sits between a UI and Elasticsearch.
The API uses elasticsearch-py to run searches, for example:
es = Elasticsearch([URL], http_compress=True)
#hug.post('/search')
def search(body):
return es.search(index='index', body=body)
This works fine; however, I cannot figure out how to obtain a compressed JSON result.
Elasticsearch is capable of this because a curl test checks out — the following returns a mess of characters to the console instead of JSON and this is what I want to emulate:
curl -X GET -H 'Accept-Encoding: gzip' <URL>/<INDEX>/_search
I've tried the approach here to modify HTTP headers, but interestingly enough the "Accept-Encoding": "gzip" header is already there: it just doesn't appear to be passed to Elastic because the result is always uncompressed.
Lastly, I'm passing http_compress=True when creating the Elastic instance; however, this only compresses the payload — not the result.
Has anyone had a similar struggle and figured it out?

JMeter not attaching contents of binary file to POST data in HTTP Request

I'm attempting to simulate a login call with JMeter 2.11 to a service that uses a binary format. I've created an Http Request with the appropriate settings, except for the body data. I need to POST raw binary data.
According to the docs here, I should be able to set the file path for exactly one file, with no parameter name, and no other content in the Body Data, and have it place the data in the request body.
If it is a POST or PUT or PATCH request and there is a single file whose 'Parameter name' attribute (below) is omitted, then the file is sent as the entire body of the request, i.e. no wrappers are added. This allows arbitrary bodies to be sent. This functionality is present for POST requests after version 2.2, and also for PUT requests after version 2.3.
However, when I run the test, the POST Data is empty.
I have tried the extra set of plugins for JMeter as well, but alas, I'm stuck. The loaded file has 145 bytes of data, and the request shows that the content-length is 0. What am I missing?
The Http Request
The result
Update 1
To clarify, I am NOT attempting to send a file, I'm attempting to send a binary encoded message as raw POST data.
Switch back to Parameters tab not Post body.
See:
http://jmeter.apache.org/usermanual/component_reference.html#HTTP_Request
Yoy could try recoring to see how the resuest look like.
This is my solution,maybe not best, but it works fine:
1st step :
You should write your binary data to a file (assume it's name is
FILENAME);
2nd step :
For your http request sampler,Yout should put ${FILENAME} under file
path in the "Send Files with the request" section (while leaving its
paramter name empty and specifying an encoding (for binary, it is
application/binary)).
Hope it helps.
Refer to this article

How to include an image in an http POST request

I'm trying to make a POST request from the command line to my Flask app, and I want it to include an image. But I don't know how to include it with the command. I've only used strings as data successfully.
So, if my POST request looks like this:
curl -i -H "Content-Type: application/json" -X POST -d '{"username":"user1", "password":"password", "image":##What do I put here?##}' http://localhost:5000/my_app/api/users
I don't know what to put in that image part of the JSON. I'm tagging flask in this question because it might be a specific answer with regards to flask.
I would like to include an actual image here, and then on the Flask side of things, put the image in a folder of the app where all the uploads go, then save the path to the image in the database for later access. But, to do that, I need to know how to send an image in the first place. Any thoughts?
Seems you're mixing things up here.
From your example seems you want to upload an image in a JSON object. This is generally bad for 2 reasons:
Overhead: the image data should be encoded in printable characters, e.g. using base64. This creates a huge overhead on the data itself causing the JSON decoder to slow down.
Testing: You can't try this using curl on the commandline. You should make some command line utility to test the request.
HTTP knows about data uploads. So to mantain the JSON structure without the slowdown of above, you should upload your image as traditional data upload using a field, and another field for the JSON data structure.
Using curl this is achieved with the -F option.
curl -i -Ffiledata=#file.jpg -Fdata='{"username":"user1", "password":"password"}' http://localhost:5000/my_app/api/users
With the above command you are sending an HTTP POST with a file upload named filedata and a 'data` field containing your 'JSON' payload to parse in the receiving view handler.
You must use 2 HTTP fields because in HTTP upload you're using multipart encoding.
Behind the scenes
You're sending the image contents encoded in base64 but since the data is decoded by the application framework knowing exactly what to do (JSON parse must figure it out while parsing) it's a lot faster.

Resources