I am working with the confluent kafka, zookeeper in docker. I successfully submit a json file to kafka topic then consume as follow
curl -X POST \
-H "Content-Type: application/json" \
--data '{"name": "quickstart-file-source", "config {"connector.class":"org.apache.kafka.connect.file.FileStreamSourceConnector", "tasks.max":"1", "topic":"quickstart-data", "file": "/tmp/quickstart/input.json"}}' \
http://localhost:28081/connectors
Above curl command has only one json file which executes successfully but I need to post multiple json files. Is there any way to do it?
Here is my kafka connect
docker run -d \
--name=kafka-connect-avro \
--net=host \
-e CONNECT_BOOTSTRAP_SERVERS=localhost:29091 \
-e CONNECT_REST_PORT=28081 \
-e CONNECT_GROUP_ID="quickstart-avro" \
-e CONNECT_CONFIG_STORAGE_TOPIC="quickstart-avro-config" \
-e CONNECT_OFFSET_STORAGE_TOPIC="quickstart-avro-offsets" \
-e CONNECT_STATUS_STORAGE_TOPIC="quickstart-avro-status" \
-e CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1 \
-e CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1 \
-e CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1 \
-e CONNECT_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
-e CONNECT_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
-e CONNECT_INTERNAL_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
-e CONNECT_INTERNAL_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
-e CONNECT_REST_ADVERTISED_HOST_NAME="localhost" \
-e CONNECT_LOG4J_ROOT_LOGLEVEL=DEBUG \
-v /tmp/quickstart/file:/tmp/quickstart \
confluentinc/cp-kafka-connect:latest
Reference link
You could make individual JSON files in the current directory and post them separately in a loop
e.g.
$ ls *.json # list your connectors
payload1.json
payload2.json
And then loop over them
for f in `ls *.json`; do
curl -X POST -H "Content-Type: application/json" \
--data#${f} http://localhost:28081/connectors
done
Or simpler to use cat
Related
I have command I am running with works for one enviornment in a bash script e.g.
so for example
ZONE_ID=prod
#!/bin/bash
curl -X DELETE "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/purge_cache" \
-H "X-Auth-Email: EMAIL" \
-H "X-Auth-Key: AUTH_KEY" \
-H "Content-Type: application/json" \
--data '{"purge_everything":true}'
What i would like to do is change the data item if possible for a different environment depending on the zone_id
ZONE_ID=prod
ZONE_ID=UAT
for example UAT would be
curl -X DELETE "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/purge_cache" \
-H "X-Auth-Email: EMAIL" \
-H "X-Auth-Key: AUTH_KEY" \
-H "Content-Type: application/json" \
--data '{"files":["http://www.example.com/css/styles.css",{"url":"http://www.example.com/cat_picture.jpg","headers":{"Origin":"https://www.cloudflare.com","CF-IPCountry":"US","CF-Device-Type":"desktop"}}]}'
So what I kind of want to do is the following, but this doesnt work
#!/bin/bash
curl -X DELETE "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/purge_cache" \
-H "X-Auth-Email: EMAIL" \
-H "X-Auth-Key: AUTH_KEY" \
-H "Content-Type: application/json" \
if [$ZONE_ID -q prod]; then
--data '{"purge_everything":true}'
else
--data '{"files":["http://www.example.com/css/styles.css",{"url":"http://www.example.com/cat_picture.jpg","headers":{"Origin":"https://www.cloudflare.com","CF-IPCountry":"US","CF-Device-Type":"desktop"}}]}'
One possible way to do it is:
#! /bin/bash
curl -X DELETE "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/purge_cache" \
-H "X-Auth-Email: EMAIL" \
-H "X-Auth-Key: AUTH_KEY" \
-H "Content-Type: application/json" \
--data "$(if [[ $ZONE_ID == prod ]]; then
echo '{"purge_everything":true}'
else
echo '{"files":["http://www.example.com/css/styles.css",{"url":"http://www.example.com/cat_picture.jpg","headers":{"Origin":"https://www.cloudflare.com","CF-IPCountry":"US","CF-Device-Type":"desktop"}}]}'
fi)"
However, a much better option (for clarity and maintainability) is to put the --data argument in a variable.
I would recommend set variable before use it in curl command
example as follow
DATA=""
if [ "$ZONE_ID" = "prod" ]; then
DATA='{"purge_everything":true}'
else
DATA='{"files":["http://www.example.com/css/styles.css",{"url":"http://www.examp
le.com/cat_picture.jpg","headers":{"Origin":"https://www.cloudflare.com","CF-IPC
ountry":"US","CF-Device-Type":"desktop"}}]}'
fi
curl -X DELETE "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/purge_cache"
\
-H "X-Auth-Email: EMAIL" \
-H "X-Auth-Key: AUTH_KEY" \
-H "Content-Type: application/json" \
--data $DATA
Goal:
I'm using a bash CURL script to connect to the Cloudflare APIv4. The goal is to update an A-record. My script:
# Get current public IP
current_ip=curl --silent ipecho.net/plain; echo
# Update A record
curl -X PUT "https://api.cloudflare.com/client/v4/zones/ZONEIDHERE/dns_records/DNSRECORDHERE" \
-H "X-Auth-Email: EMAILHERE" \
-H "X-Auth-Key: AUTHKEYHERE" \
-H "Content-Type: application/json" \
--data '{"id":"ZONEIDHERE","type":"A","name":"example.com","content":"'"${current_ip}"'","zone_name":"example.com"}'
Problem:
The current_ip variable is not printed when I call it in my script. The output will be "content" : "" and not "content" : "1.2.3.4".
I used other stackoverflow posts and I'm trying to follow their examples but I think I'm still doing something wrong, just can't figure out what. :(
Using jq for this, as Charles Duffy's answer suggests, is a very good idea. However, if you can't or do not want to install jq here is what you can do with plain POSIX shell.
#!/bin/sh
set -e
current_ip="$(curl --silent --show-error --fail ipecho.net/plain)"
echo "IP: $current_ip"
# Update A record
curl -X PUT "https://api.cloudflare.com/client/v4/zones/ZONEIDHERE/dns_records/DNSRECORDHERE" \
-H "X-Auth-Email: EMAILHERE" \
-H "X-Auth-Key: AUTHKEYHERE" \
-H "Content-Type: application/json" \
--data #- <<END;
{
"id": "ZONEIDHERE",
"type": "A",
"name": "example.com",
"content": "$current_ip",
"zone_name": "example.com"
}
END
The reliable way to edit JSON from shell scripts is to use jq:
# set shell variables with your contents
email="yourEmail"
authKey="yourAuthKey"
zoneid="yourZoneId"
dnsrecord="yourDnsRecord"
# make sure we show errors; --silent without --show-error can mask problems.
current_ip=$(curl --fail -sS ipecho.net/plain) || exit
# optional: template w/ JSON content that won't change
json_template='{"type": "A", "name": "example.com"}'
# build JSON with content that *can* change with jq
json_data=$(jq --arg zoneid "$zoneid" \
--arg current_ip "$current_ip" \
'.id=$zoneid | .content=$current_ip' \
<<<"$json_template")
# ...and submit
curl -X PUT "https://api.cloudflare.com/client/v4/zones/$zoneid/dns_records/$dnsrecord" \
-H "X-Auth-Email: $email" \
-H "X-Auth-Key: $authKey" \
-H "Content-Type: application/json" \
--data "$json_data"
I am using s-nail to send emails via SMTP by a bash script. Here is an example:
cat data.html | s-nail -v \
-s "Subject Text" \
-r "reply#abc.de" \
-S smtp=smtp.abc.de:587 \
-S smtp-use-starttls \
-S smtp-auth=login \
-S smtp-auth-user="admin#abc.de" \
-S smtp-auth-password="password1234" \
-a foo.img \
foo#bar.com
By default the content type is being set to Text.
How to set the Content-Type to html?
A little late but here you go
-M "text/html"
Goal:
I'm using a bash CURL script to connect to the Cloudflare APIv4. The goal is to update an A-record. My script:
# Get current public IP
current_ip=curl --silent ipecho.net/plain; echo
# Update A record
curl -X PUT "https://api.cloudflare.com/client/v4/zones/ZONEIDHERE/dns_records/DNSRECORDHERE" \
-H "X-Auth-Email: EMAILHERE" \
-H "X-Auth-Key: AUTHKEYHERE" \
-H "Content-Type: application/json" \
--data '{"id":"ZONEIDHERE","type":"A","name":"example.com","content":"'"${current_ip}"'","zone_name":"example.com"}'
Problem:
The current_ip variable is not printed when I call it in my script. The output will be "content" : "" and not "content" : "1.2.3.4".
I used other stackoverflow posts and I'm trying to follow their examples but I think I'm still doing something wrong, just can't figure out what. :(
Using jq for this, as Charles Duffy's answer suggests, is a very good idea. However, if you can't or do not want to install jq here is what you can do with plain POSIX shell.
#!/bin/sh
set -e
current_ip="$(curl --silent --show-error --fail ipecho.net/plain)"
echo "IP: $current_ip"
# Update A record
curl -X PUT "https://api.cloudflare.com/client/v4/zones/ZONEIDHERE/dns_records/DNSRECORDHERE" \
-H "X-Auth-Email: EMAILHERE" \
-H "X-Auth-Key: AUTHKEYHERE" \
-H "Content-Type: application/json" \
--data #- <<END;
{
"id": "ZONEIDHERE",
"type": "A",
"name": "example.com",
"content": "$current_ip",
"zone_name": "example.com"
}
END
The reliable way to edit JSON from shell scripts is to use jq:
# set shell variables with your contents
email="yourEmail"
authKey="yourAuthKey"
zoneid="yourZoneId"
dnsrecord="yourDnsRecord"
# make sure we show errors; --silent without --show-error can mask problems.
current_ip=$(curl --fail -sS ipecho.net/plain) || exit
# optional: template w/ JSON content that won't change
json_template='{"type": "A", "name": "example.com"}'
# build JSON with content that *can* change with jq
json_data=$(jq --arg zoneid "$zoneid" \
--arg current_ip "$current_ip" \
'.id=$zoneid | .content=$current_ip' \
<<<"$json_template")
# ...and submit
curl -X PUT "https://api.cloudflare.com/client/v4/zones/$zoneid/dns_records/$dnsrecord" \
-H "X-Auth-Email: $email" \
-H "X-Auth-Key: $authKey" \
-H "Content-Type: application/json" \
--data "$json_data"
I'm trying to get both the HTTP code and the output of a curl command as part of a shell script, but in turn I'm trying to set both of them as variables. My code (which uses the pushbullet API) looks like this:
CURL_OUTPUT="$(exec 3>&1; \
HTTP_CODE="$(curl -s -S \
-w "%{http_code}" -o >(cat >&3) \
--header 'Authorization: Bearer '"$ACCT_TOKEN" \
-X POST https://api.pushbullet.com/v2/pushes \
--header 'Content-Type: application/json' \
--data-binary "$JSON")" \
)"
which should theoretically set $CURL_OUTPUT to the JSON returned by curl and $HTTP_CODE to the status code I get. Instead, I get only $CURL_OUTPUT; $HTTP_CODE is empty.
However, if I don't do the outer nest, like so:
exec 3>&1; \
HTTP_CODE="$(curl -s -S \
-w "%{http_code}" -o >(cat >&3) \
--header 'Authorization: Bearer '"$ACCT_TOKEN" \
-X POST https://api.pushbullet.com/v2/pushes \
--header 'Content-Type: application/json' \
--data-binary "$JSON")" \
The command works as expected; I get the JSON redirected to stdout, and the status code ends up in $HTTP_CODE.
So is it possible to get both outputs? Can I assign variables within a command substitution line?