I have multiple nifi servers that I would like to be able to POST templates to via the REST interface from a script
The "/controller/templates" endpoint appears to be the proper REST endpoint to support POSTing an arbitrary template to my Nifi installation.
The "snippetId" field is what is confusing me, how do I determine "The id of the snippet whose contents will comprise the template"? Does anyone have an example of how I can upload a template "test.xml" to my server without having to use the UI?
The API has moved in 1.0 to:
POST /process-groups/{id}/templates/upload
Example, using Python's requests library:
res = requests.post( "{hostname}/nifi-api/process-groups/{destination_process_group}/templates/upload".format( **args ),
files={"template": open( file_path, 'rb')} )
The provided documentation is somewhat confusing, and the solution I worked out was derived from the nifi api deploy groovy script at https://github.com/aperepel/nifi-api-deploy
Ultimately, to POST a template directly, you can use the following in Python requests
requests.post("%s/nifi-api/controller/templates"%(url,), files={"template":open(filename, 'rb')})
Where filename is the filename of your template and url is the path to your nifi instance. I haven't figured it out in curl directly but this should hopefully get folks with a similar question started!
Edit:
Note that you also can't upload a template with the same name as an existing template. Make sure to delete your existing template before attempting to re-upload. Using the untangle library to parse the XML of the template, the following script works just fine:
import untangle, sys, requests
def deploy_template(filename, url):
p = untangle.parse(filename)
new_template_name=p.template.name.cdata
r=requests.get("%s/nifi-api/controller/templates"%(url,), headers={"Accept":"application/json"})
for each in r.json()["templates"]:
if each["name"]==new_template_name:
requests.delete(each["uri"])
requests.post("%s/nifi-api/controller/templates"%(url,), files={"template":open(filename, 'rb')})
if __name__=="__main__":
deploy_template(sys.argv[1], sys.argv[2])
If you want to POST a template to NiFi via cURL you can use the following command:
curl -iv -F template=#my_nifi_template.xml -X POST http://nifi-host:nifi-port/nifi-api/controller/templates
This will add the template to the NiFi instance with the same name that the template was given when it was generated.
And the -iv is optional - It's just there for debugging purposes.
You can use Nifi Api to upload a template, to do that follow these two steps:
1. Get a token from Nifi Api:
token=$(curl -k -X POST --negotiate -u : https://nifi_hostname:port/nifi-api/access/kerberos)
2. Upload the template file using the token:
curl -k -F template=#template_file.xml -X POST https://nifi_hostname:port/nifi-api/process-groups/Process_group_id/templates/upload -H "Authorization: Bearer $token"
The documentation can be confusing because that endpoint is overloaded and the documentation tool only generates doc for one of them (see NIFI-1113). There is an email thread that addresses the import of a template using curl, so between the above answer and the email thread, hopefully you can find the approach that works for you.
I've implemented a full Python client for doing this in NiPyApi
Key functions for templates are:
[
"list_all_templates", "get_template_by_name", "deploy_template",
"upload_template", "create_pg_snippet", "create_template",
"delete_template", "export_template", 'get_template'
]
The client supports NiFi-1.1.2 - 1.7.1 currently, and NiFi-Registry (which is a lot better than templates for flow deployment)
Related
I have deployed an AWS Lambda function, written in Python, and AWS API Gateway structure to cause POST requests to an API endpoint to be redirected to my function. I want to upload a PDF document to my function and have it store the document in a S3 bucket. The problem I have is that the payload of any POST request to my API is being UTF-8 encoded. I don't want that but can't figure out the magic mojo to disable encoding of the request payload.
I am testing using curl, with the following command line:
curl -XPOST https://xxxxxxxxxx.execute-api.us-west-1.amazonaws.com/test -H 'content-type: application/pdf' --data-binary #document.pdf
UPDATE: I just found the following article describing how API Gateway and Lambda support uploading binary data:
https://aws.amazon.com/blogs/compute/handling-binary-data-using-amazon-api-gateway-http-apis/
This article suggests that all of the complexities that I discussed in the initial formation of my question (still provided below) should not be necessary. All I should need to do to upload binary content to my Lambda function is insure that my request includes an appropriate Content-Type header. I was already doing that, but I massaged my Curl command a bit (modified above) to define my request in exactly the way that is done in this article. I still get UTF-8 encoded data and NOT base-64 encoded data. I tried uploading a jpeg file rather than a PDF so I was doing exactly what was done in the article. Still no love. I don't get it. This article demonstrates exactly what I'm doing. But I don't get the result it suggests I should. Ggggrrrr.
ORIGINAL POST:
I am using Terraform to define my deployment. I want to cause the PDF to not be encoded/mangled at all. This is my first time using API Gateway, and I'm obviously missing some bit of config. The one thing I'm doing specifically right now to say that I want incoming payloads to be treated as binary is via the binary_media_types argument to my API definition in Terraform:
resource aws_api_gateway_rest_api proxy {
...
binary_media_types = [
"application/pdf",
"application/octet-stream",
"*/*"
]
This sets the Binary Media Types configuration associated with the API I've defined. I've confirmed via the AWS Console that this setting is having the desired effect...I can see these types in the console. I should need just the first item in the list, but I've added the others while I try to figure out the problem here. By adding that wildcard item, I believe that it shouldn't matter what the incoming Content-Type is...all payloads should be being treated as binary.
The other bit of config that I know about that might be important is the "integration contentHandling property". Here is the key bit of AWS docs that seems to explain all this:
I think the case that applies to me here is the one I've highlighted, per what I say above. This says to me that I shouldn't need to do anything else, per the "unspecified" value in the table for "contentHandling. I've tried setting the "contentHandling" argument on the integration record of my Terraform config, like this:
resource aws_api_gateway_integration proxy {
...
passthrough_behavior = "WHEN_NO_MATCH"
content_handling = "CONVERT_TO_BINARY"
}
I first tried only specifying the content_handling value. I've also tried setting that value to "CONVERT_TO_TEXT", hoping to then get base64-encoded data. Neither of these has any effect. I've tried adding the passthrough_behavior value as shown. I've also tried replacing "WHEN_NO_MATCH" with "WHEN_NO_TEMPLATES". Nothing I do changes the behavior. I haven't been able to figure out where these settings would show up in the AWS console. If I knew they were necessary, I'd explore this further. But I don't think I need to set these.
What am I missing? How can I POST a PDF document to my AWS Lambda function through API Gateway and have the payload of the request not be converted in any way? TIA!
NOTE: I am aware of this Q/A: PDF Uploaded via AWS API Gateway getting corrupted. The answer there doesn't apply to me, as I need to avoid having to form-encode the upload. The client code that will eventually be doing the upload is set in stone and sends a POST request with a payload that is just the bytes of the PDF.
usualy when I use API's I paste the entire url in browser and it print it out json format using json pro extension in chrome. Like this is a lot easier to copy the path of some data and render it to the page.But my problem is I don't know structure of https. I am not sure where I have to insert the key and secret code. In command line I print all data but I cannot get the path of specific data without using json probextension. Help please. Thank you
I recommend to read the following article:
https://developers.amadeus.com/self-service/apis-docs/guides/authorization-262
and later, review the following repository, you could tinker with the curl request here:
https://github.com/amadeus4dev/amadeus-code-examples
https://github.com/amadeus4dev/amadeus-code-examples/blob/master/airline_code_lookup/v1/get/curl/airline_code_lookup.sh
I'm aware it is possible to generate code snippets from individual Postman requests, but I can't find similar functionality for a entire collection.
I also know that there are tools like newman for running an exported collection in Postman Collection 2.1 json format.
But specifically what I'm looking for is a tool that generates bash code from a collection or from a exported collection in Postman Collection 2.1 json format. This way my co-workers who don't use Postman can replicate the API requests.
Is this option available in Postman and I'm just missing it or are there any tools that do this?
I ended using the Postman SDK and postman-code-gen to create code snippets from Postman Collections that I exported from Postman.
Not really sure why this feature isn't provided already by Postman, but I made a simple CLI tool that does it:
https://github.com/arashout/postman-collection-gen
# cURL request generation
node main.js -c example_collection.json
# curl --location --request GET 'https://v7rr12wbr7.execute-api.us-west-2.amazonaws.com/prod/courses?c0=PHYS153&c1=APSC160&c2=CHEM154&version_key=1.2'
# curl --location --request GET 'https://v7rr12wbr7.execute-api.us-west-2.amazonaws.com/prod/courses?c0=PHYS153'
# How to generate other languages:
node main.js -c example_collection.json -l shell,httpie
node main.js -c example_collection.json -l Swift,URLSession
I'm not sure I understand your question completely.
But I can tell you how I export collection and then run tests.
Export Collection to JSON: choose tab 'Collections' at the left tab of the Postman. Select the 3-dot menu and choose 'Export'.
Export your test environment: Open 'Manage Environments'. Click download.
I don't know a tool that generates bash code, I write it manually.
Write .sh file like the following to run your tests.
"newman run your_test_collection.json -r cli -e your_test_environment.json --reporter-cli-no-assertions --global-var"
Usually, I create different .sh files for different sets of tests.
My question:
I am using SonarQube version 7.1 and trying to extract the metrics and quality gate related to individual projects.
What we have tried
We were using Python SonarQube API to extract these data before our company upgraded to version 7.1. "api/resources" web service Deprecated since sonarqube5.4, so we cannot use it anymore.
I have also tried using getting data using CURL command via Web API using
curl -i -H "Content-Type: application/json" -H "x-api-key:token" -X GET 'http://MY_HOST/api/measures/component?metricKeys=key&component=project_key'
We are able to get a json payload for individual metrics, but involves tedious task of creating the URL every single time.
But I wanted to know if there is a better/smarter way to access these "measures", be it any language or implementation.
You could do this:
Call the API api/metrics/search first to get a (json) list of all the metrics and then iterate over that list and create a comma separated string of all the metric keys.
For example something like this: ncloc,complexity,violations .. as mentioned in the parameters example value in the API documentation here.
Then you could just add this comma separated list to the url as a parameter something like: http://MY_HOST/api/measures/component?metricKeys=ncloc,complexity,violations&component=project_key
and call it once to get the response for all metrics.
Also, I haven't tried this, but as per the latest documentation, the parameter component is optional. So if you omit that, ideally you should get a response with metrics of all the projects.
I am trying to update an ticket and I am trying to do that through Linux. Not sure whether it can be done or not, in the way of searching, I found one blogger sharing, where he/she was also trying to update something
He/She has used the below commands to update the URL.
curl -i -u "script-user:password" -X PUT -d "short_description=Update+me" https:/0005000972
I suspect he/she is trying to update the URL with "Update me".
Is that right? or else Can some one explain what that blogger tried to do??
This is a RESTful request. That is a request using standard HTTP methods (GET, POST, PUT, DELETE) to query a server.
-X PUT specifies the method used
-d "...." specifies the data send to the server
https://... (ill formed in your example) is of course the URL of the target server
Usually, the PUT method is used to replace an existing attribute/value on the server. As the concrete parameters and/or methods available are service dependent, I can only guess here that the intend is to update some attribute named short_description to store the value Update Me (URL encoded -- or more formally x-www-form-urlencoded)
Maybe you should first read a little bit more about those topics, and then, if necessary post an other question describing more in details both the target server and the goal you're trying to achieve on it.