Benthos: How to get variable from processor to input? - benthos

Question

i'm new to benthos, hope following configuration to work, i looked at the benthos doc and tried to google, but didn't find an answer, any answer is greatly appreciated
actually, the sign will be the calculated value, but now I'm stuck on the first step, i can't get the sign value to be successfully assigned to the header
input:
processors:
- bloblang: |
meta sign = "1233312312312312"
meta device_id = "31231312312"
http_client:
url: >-
https://test/${!meta("device_id")}
verb: GET
headers:
sign: ${!meta("sign")}
after #Mihai Todor helped, now i have new question.
this config below can work.(first)
input:
http_client:
url: >-
https://test/api
verb: GET
headers:
sign: "signcode"
but this one returned invalid signature error
input:
mapping: root = {}
generate:
count: 1
pipeline:
processors:
- http:
url: >-
https://test/api
verb: GET
headers:
sign: "signcode"
output:
stdout: {}
update(more detail screenshot)
first
second
Finally i got it work with the #Mihai helped.
the reason why i got 'signure is invaild' is because a space character in paramter headers->stringToSign, for reason i need add paramter 'stringTosign' and the value need to be empty, but somehow i copy an invisible space character in it, this will make benthos to add a Content-Length: 1 to the request header(i don't know why), this Content-Length: 1 cause my request always failed with error 'signure invaild'.
after i deleted the space character, all is ok.

Answer 1

Input processors operate on the messages returned by the input, so you can't set metadata that way. Also, metadata is associated with in-flight messages and doesn't persist across messages (use an in-memory cache if you need that).
One workaround is to combine a generate input with an http processor like so:
input:
generate:
mapping: root = ""
interval: 0s
count: 1
processors:
- mapping: |
meta sign = "1233312312312312"
meta device_id = "31231312312"
pipeline:
processors:
- http:
url: >-
https://test/${!meta("device_id")}
verb: GET
headers:
sign: ${!meta("sign")}
output:
stdout: {}
Note that the mapping processor (the replacement for the soon-to-be deprecated bloblang one) can also reside under pipeline.processors, and, if you just need to set those metadata fields, you can also do it inside the mapping field of the generate input (root = {} is implicit).
Update: Following the comments, I ran two configs and used nc to print the full HTTP request each of them make:
generate input with http processor:
input:
generate:
mapping: root = ""
interval: 0s
count: 1
processors:
- mapping: |
meta sign = "1233312312312312"
meta device_id = "31231312312"
pipeline:
processors:
- http:
url: >-
http://localhost:6666/${!meta("device_id")}
verb: GET
headers:
sign: ${!meta("sign")}
output:
stdout: {}
HTTP request dump:
> nc -l localhost 6666
GET /31231312312 HTTP/1.1
Host: localhost:6666
User-Agent: Go-http-client/1.1
Sign: 1233312312312312
Accept-Encoding: gzip
http_client input:
input:
http_client:
url: >-
http://localhost:6666/31231312312
verb: GET
headers:
sign: 1233312312312312
output:
stdout: {}
HTTP request dump:
> nc -l localhost 6666
GET /31231312312 HTTP/1.1
Host: localhost:6666
User-Agent: Go-http-client/1.1
Sign: 1233312312312312
Accept-Encoding: gzip
I used Benthos v4.5.1 on OSX and, for both configs, the request looks identical.
My best guess is that you're seeing a transitive issue on your end (some rate limiting perhaps).