Can not send request with body contain some special character like ß - spring-boot

When I send POST request that body request contains some special character like ß, actually it is a character in German like "Zentrale Bußgeldstelle". And bellow is the information of request I sent by using curl run on Gitbash:
curl --location --request POST 'http://localhost:8080/user/v1/register' \
--header 'Content-Type: application/json' \
--data-raw '{
"username": "ßtest",
"password": "P#ssword"
}'
And I got the bad request status as below:
{
"timestamp":"2020-06-27T05:22:50.676+0000",
"status":400,
"error":"Bad Request",
"message":"JSON parse error: Invalid UTF-8 middle byte 0x74\n at [Source: (PushbackInputStream);
line: 1, column: 18]; nested exception is com.fasterxml.jackson.databind.JsonMappingException: Invalid UTF-8 middle byte 0x74\n at [Source: (PushbackInputStream); line: 1, column: 18]\n at [Source: (PushbackInputStream); line: 1, column: 15] (through reference chain: com.example.springlogin.dto.UserDto[\"username\"])",
"path":"/user/v1/register"
}
I faced that it related to ß character. But the issue not occur when I send that request using postman.I used Springboot to implement that api.
So anyone can help to solve my above issue. Thanks in advance

Related

How to send a POST request with Kotlin/Native using libcurl?

I'm trying to send a JSON through a POST request with a Kotlin/Native application using libcurl. I'm working on a Windows 11 machine, and the endpoint lies under a Spring Boot (version 2.7.8) backend written with Kotlin and Java 11.
The following is the code I wrote to accomplish this task.
import kotlinx.cinterop.*
import libcurl.*
fun main() {
val curl = curl_easy_init()
curl?.let {
var headers: CValuesRef<curl_slist>? = null
headers = curl_slist_append(headers, "Content-Type: application/json")
setCurl(curl, headers)
val res = curl_easy_perform(curl)
if (res != CURLE_OK) println("curl_easy_perform() failed ${curl_easy_strerror(res)?.toKString()}")
curl_easy_cleanup(curl)
curl_slist_free_all(headers)
} ?: println("curl_easy_init() failed to return curl easy handle")
}
private fun setCurl(curl: COpaquePointer?, headers: CPointer<curl_slist>?) {
val body = "{ JSON object }"
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers)
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, body)
curl_easy_setopt(curl, CURLOPT_URL, "http://localhost:8080/dishes/add/")
}
The JSON string I need to send represents a simple Dish object having three fields: name, type and description.
These are the attempts I made to correctly format the JSON string to initialize the val body and the relative outputs that the Spring Boot endpoint return:
{\"name\":\"Fish\",\"type\":\"Second\",\"description\":\"Fry\"} => HTTP 200;
{\"name\":\"Fish fry\",\"type\":\"Second course\",\"description\":\"Lots of fresh fish to delight the palate with an excellent, fried second course\"} => HTTP 400: JSON parse error: Unexpected character (' ' (code 160)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false');
{\\\"name\\\":\\\"Fish fry\\\",\\\"type\\\":\\\"Second course\\\",\\\"description\\\":\\\"Lots of fresh fish to delight the palate with an excellent, fried second course\\\"} => HTTP 400: JSON parse error: Illegal character ((CTRL-CHAR, code 0)): only regular white space (\r, \n, \t) is allowed between tokens;
"\"{\\\"name\\\":\\\"Fish fry\\\",\\\"type\\\":\\\"Second course\\\",\\\"description\\\":\\\"Lots of fresh fish to delight the palate with an excellent, fried second course\\\"}\"" => HTTP 400: JSON parse error: Unexpected character (' ' (code 160)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
From the above tests, I can effectively POST something on the backend only with the first body string, but if I try to send a longer one with even the same format (see the second one), the POST request fails too.
Besides, I also tried to execute a POST request using Postman and the terminal (with the following command: curl -H "Content-Type: application/json" -X POST -d "{\"name\":\"Fish fry\",\"type\":\"Second course\",\"description\":\"Lots of fresh fish to delight the palate with an excellent, fried second course\"}" http://localhost:8080/dishes/add/), and with both, I can correctly send the POSTs. And since using the commands prompt, curl accepts the last body JSON I tried, maybe it could be the right approach to format the string, but I'm not sure.
What am I missing?
Thanks for your precious time!
UPDATE 1:
I just discovered the --libcurl curl parameter, which lets you convert a curl command into libcurl code.
Using this helpful tool, I converted my working cmd POST request
curl -H "Content-Type: application/json" -X POST -d "{\"name\":\"Fish fry\",\"type\":\"Second course\",\"description\":\"Lots of fresh fish to delight the palate with an excellent, fried second course\"}" http://localhost:8080/dishes/add/
Into the following, Kotlin adapted, C snippet:
private fun setCurl(hnd: COpaquePointer?, certPath: String, url: String) {
var headers: CValuesRef<curl_slist>? = null
headers = curl_slist_append(headers, "Content-Type: application/json")
curl_easy_setopt(hnd, CURLOPT_BUFFERSIZE, 102400L)
curl_easy_setopt(hnd, CURLOPT_URL, "http://localhost:8080/dishes/add/")
curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L)
curl_easy_setopt(
hnd,
CURLOPT_POSTFIELDS,
"{\"name\":\"Fish fry\",\"type\":\"Second course\",\"description\":\"Lots of fresh fish to delight the palate with an excellent, fried second course\"}"
)
val postFieldSize: curl_off_t = 138
curl_easy_setopt(hnd, CURLOPT_POSTFIELDSIZE_LARGE, postFieldSize)
curl_easy_setopt(hnd, CURLOPT_HTTPHEADER, headers)
curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.83.1")
curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L)
curl_easy_setopt(hnd, CURLOPT_CUSTOMREQUEST, "POST")
curl_easy_setopt(hnd, CURLOPT_FTP_SKIP_PASV_IP, 1L)
curl_easy_setopt(hnd, CURLOPT_TCP_KEEPALIVE, 1L)
}
But still, my Kotlin/Native application failed to execute the request with Spring Boot returning the same error: Unexpected character (' ' (code 160)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false').
At this time, I ran out of ideas. Please, let me know about any other solutions.
UPDATE 2:
Since the first val body initialization was the only one to be successful (despite the body string having to be small), I started to do some other tests with that type of formatted JSON, so I found that the POST request is successful if the entire string does not cross the 63 chars of length otherwise, the Spring Boot endpoint fires the error regarding the code 160 unexpected character.
The body string I'm currently using is the following one, which length is exactly 63 chars.
{\"name\":\"Fettuccine Alfredo\",\"type\":\"Main co\",\"description\":\"\"}
I don't know why this situation is happening, and I'm very frustrated.
Every tip of advice is much appreciated.

Curl/GraphQL command failing with 200

I am trying to write a shell script that executes a curl against a GraphQL API and I've never interacted with GQL before. I am getting some strange errors and although I understand this community doesn't have access to the GQL server I was hoping someone could take a look at the script and make sure I'm not doing anything flagrantly wrong syntax-wise (both in the shell script layer as well as the GQL query itself).
My script:
#!/bin/bash
BSEE_WEB_SERVER_DNS=https://mybsee.example.com
BSEE_API_KEY=abc123
siteId=1
scanConfigId=456
runScanQuery='mutation CreateScheduleItem { create_schedule_item(input: {site_id: "$siteId" scan_configuration_ids: "$scanConfigId"}) { schedule_item { id } } }'
runScanVariables='{ "input": "site_id": $scanId }}'
runScanOperationName='CreateScheduleItem'
curl -i --request POST \
--url $BSEE_WEB_SERVER_DNS/graphql/v1 \
--header "Authorization: $BSEE_API_KEY" \
--header 'Content-Type: application/json' \
--data '{"query":"$runScanQuery","variables":{$runScanVariables},"operationName":"${runScanOperationName}"}'
And the output when I run the script off the terminal:
HTTP/2 200
<OMITTED RESPONSE HEADERS>
{"errors":[{"message":"Invalid JSON : Unexpected character (\u0027$\u0027 (code 36)): was expecting double-quote to start field name, Line 1 Col 38","extensions":{"code":3}}]}%
I am omitting the HTTP response headers for security and brevity reasons.
I am wondering if my use of quotes/double-quotes is somehow wrong, or if there is anything about the nature of the GQL query itself (via curl) that looks off to anyone.
I verified with the team that manages the server that the HTTP 200 OK response code is correct. 200 shows that the request succeeded to the GQL API, but that GQL is responding with this error to indicate the query itself is incorrect.
We need to modify the GraphQL bits and fix the bash string quoting.
runScanQuery GraphQL operation
Fix the GraphQL syntax. Use a GraphQL operation name CreateScheduleItem with variables $site_id in the arguments input: { site_id: $siteId, scan_configuration_ids: $scanConfigId:
mutation CreateScheduleItem($site_id: String!, $scanConfigId: String!) {
create_schedule_item(
input: { site_id: $siteId, scan_configuration_ids: $scanConfigId }
) {
schedule_item {
id
}
}
}
runScanVariables: JSON
Our mutation expects two variables, which GraphQL will substitute into CreateScheduleItem($site_id: String!, $scanConfigId: String!). Provide the GraphQL variables as JSON. Here is the expected output after bash variable substitution:
{ "$site_id": "1", "$scanConfigId": "456" }
Get the bash quoting right
Finally, translate the inputs into bash-friendly syntax:
runScanQuery='mutation CreateScheduleItem($site_id: String!, $scanConfigId: String!) { create_schedule_item(input: {site_id: $siteId scan_configuration_ids: $scanConfigId}) { schedule_item { id } } }'
runScanVariables='{"$site_id":"'"$siteId"'","$scanConfigId":"'"$scanConfigId"'"}' # no spaces!
runScanOperationName='CreateScheduleItem'
data='{"query":"'"$runScanQuery"'","variables":'$runScanVariables',"operationName":"'"$runScanOperationName"'"}'
Check our bash formats. Paste the terminal output into a code-aware editor like VSCode. Expect the editor to parse the output correctly.
echo $runScanQuery # want string in graphql format
echo $runScanVariables # want JSON
echo $data # want JSON
Edit: add a public API example
Here's a complete working example using the public Star Wars API:
#!/bin/bash
filmId=1
data='{"query":"query Query($filmId: ID) { film(filmID: $filmId) { title }}","variables":{"filmId":"'"$filmId"'"}}'
curl --location --request POST 'https://swapi-graphql.netlify.app/.netlify/functions/index' \
--header 'Content-Type: application/json' \
--data "$data"
Responds with {"data":{"film":{"title":"A New Hope"}}}.
In GraphQL it's normal to always have 200 status code; client must check response body searching for failures.
The reason is simple: In REST, http is part of the protocol and status code has semantics but in GraphQL http is not part of the protocol, you can have GraphQL over serveral transport protocols:
http: typical scenario docs
WebSocket: does not provide any "status code like" payload. sample
MQTT: does not provide any "status code like" payload
...
The only way that server tells you something (even failures) is the body.
In your case I suggest you jq to parse json via bash script searching error property.
Your error is completely unrelated to GraphQL. You really have wrong JSON.
Error message says Unexpected character (\u0027$\u0027 (code 36)): was expecting double-quote to start field name, Line 1 Col 38",
You can replace escaped \u0027 with apostrophe and you will get
Unexpected character ('$' (code 36)): was expecting double-quote to start field name, Line 1 Col 38",
So it hates dollar sign at position 38 in what you send as data to curl
data='{"query":"'"$runScanQuery"'","variables":'$runScanVariables'
^
this
First - all field names and values in JSON should be wrapped with double quotes, not single.
Second - if you want curl to expand env variable, put it to double quotes, not single.

google docs api delete all content

I noticed from the google docs API
I can do
{
"requests": [
{
"deleteContentRange": {
"range": {
"startIndex": 1,
"endIndex": 80
}
}
}
]
}
but if the endindex is greater than the total length of characters in the document, I get the following error:
{
"error": {
"code": 400,
"message": "Invalid requests[0].deleteContentRange: Index 79 must be less than the end index of the referenced segment, 7.",
"status": "INVALID_ARGUMENT"
}
}
but I just want to delete all of the content, even though I don't know the end range value.
So: is it possible to get the endIndex somehow, or delete all content another way?
You want to delete all contents in Google Document using Docs API.
If my understanding is correct, how about this answer? Please think of this as just one of several possible answers.
Issue:
In the current stage, in order to use "DeleteContentRangeRequest", both values of startIndex and endIndex are required. It seems that this is the specification. So in your case, I think that is it possible to get the endIndex somehow, or delete all content another way? leads to the method for resolving your issue.
Flow of workaround:
Here, as the workaround, the following flow is used.
1. Retrieve the object of content from Google Document.
The sample curl command is as follows. When you use this, please set the Document ID. In this case, body.content(startIndex,endIndex) is used as the fields. By this, it is easy to see the response value.
curl \
'https://docs.googleapis.com/v1/documents/###?fields=body.content(startIndex%2CendIndex)' \
--header 'Authorization: Bearer [YOUR_ACCESS_TOKEN]' \
--header 'Accept: application/json'
The response value is like below.
{
"body": {
"content": [
{"endIndex": 1},
{"startIndex": 1, "endIndex": 100},
{"startIndex": 100, "endIndex": 200}
]
}
}
endIndex of the last index of content is the value for this.
2. Retrieve endIndex from the object.
From above response value, it is found that startIndex and endIndex are 1 and 199, respectively. If endIndex is 200, an error occurs. Please be careful this. So please reduce 1 from it.
3. Delete all contents using startIndex and endIndex.
The sample curl command is as follows.
curl --request POST \
'https://docs.googleapis.com/v1/documents/###:batchUpdate' \
--header 'Authorization: Bearer [YOUR_ACCESS_TOKEN]' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--data '{"requests":[{"deleteContentRange":{"range":{"startIndex":1,"endIndex":199}}}]}'
References:
Method: documents.get
Method: documents.batchUpdate
DeleteContentRangeRequest
If I misunderstood your question and this was not the direction you want, I apologize.

Unexpected character (')' (code 41)) in JMeter request

After I record, I used variable and run.
Body data:
{
"data: [
{
"gsVersion":"1.0",
"stepCode":"${stepCode}",
"stepName":"${stepName}",
"familyId":"${familyId}",
"listGeoEventAddOfStep": [
{
"id":0,
"geoEventCode":"${geoEventCode}",
"name":"${geoEventName}")},
"description":"${geoEventDes}"
}
]
}
]
}
However, it gave the response:
{
"timestamp":1514976739620,
"status":400,
"error":"Bad Request",
"exception":"org.springframework.http.converter.HttpMessageNotReadableException",
"message":"Could not read document: Unexpected character (')' (code 41)): was expecting comma to separate Object entries\n at [Source: java.io.PushbackInputStream#4fa24716; line: 1, column: 175]\n at [Source: java.io.PushbackInputStream#4fa24716; line: 1, column: 150] (through reference chain: com.geopost.controller.requestbody.RequestBodyList[\"data\"]->java.util.ArrayList[0]->com.geopost.dto.GeoStepAddDTO[\"listGeoEventAddOfStep\"]->java.util.ArrayList[0]); nested exception is com.fasterxml.jackson.databind.JsonMappingException: Unexpected character (')' (code 41)): was expecting comma to separate Object entries\n at [Source: java.io.PushbackInputStream#4fa24716; line: 1, column: 175]\n at [Source: java.io.PushbackInputStream#4fa24716; line: 1, column: 150] (through reference chain: com.geopost.controller.requestbody.RequestBodyList[\"data\"]->java.util.ArrayList[0]->com.geopost.dto.GeoStepAddDTO[\"listGeoEventAddOfStep\"]->java.util.ArrayList[0])",
"path":"/rest/steps"
}
Extracting the exception message for more clarity:
Could not read document: Unexpected character (')' (code 41)):
was expecting comma to separate Object entries
at [Source: java.io.PushbackInputStream#4fa24716; line: 1, column: 175]
at [Source: java.io.PushbackInputStream#4fa24716; line: 1, column: 150]
(through reference chain: com.geopost.controller.requestbody.RequestBodyList["data"]
->java.util.ArrayList[0]->com.geopost.dto.GeoStepAddDTO["listGeoEventAddOfStep"]
->java.util.ArrayList[0]);
nested exception is com.fasterxml.jackson.databind.JsonMappingException:
Unexpected character (')' (code 41)):
was expecting comma to separate Object entries
at [Source: java.io.PushbackInputStream#4fa24716; line: 1, column: 175]
at [Source: java.io.PushbackInputStream#4fa24716; line: 1, column: 150]
(through reference chain: com.geopost.controller.requestbody.RequestBodyList["data"]
->java.util.ArrayList[0]
->com.geopost.dto.GeoStepAddDTO["listGeoEventAddOfStep"]
->java.util.ArrayList[0])
How do I fix this?
you JSON is invalid, you have for example irrelevant ) sign, a valid JSON can be:
{"data":[{"gsVersion":"1.0","stepCode":"${stepCode}","stepName":"${stepName}","familyId":"${familyId}","listGeoEventAddOfStep":[{"id":0,"geoEventCode":"${geoEventCode}","name":"${geoEventName}"}],"description":"${geoEventDes}"}]}
you can check your json online.

Error handling with curl and elasticsearch

I'm currently developing bash scripts that use elasticsearch and I need a good error-handling.
In this situation I try to add a document to elasticsearch and check if the operation succeeded.
At first I naively tried this :
response=$(curl -XPOST 'http://localhost:9200/indexation/document' -d '
{
"content":"'"$txt"'",,
"date_treatment":"'"$(date +%Y-%m-%d)"'"
}') && echo ok || echo fail
But curl doesn't work that way and still returns success (0 - which is actually logical) even though the json request is obviously incorrect (note the double comma on line 3) and elasticsearch displays errors.
So the answer isn't there. Now I think I should analyze the variable $response to catch errors (grep ?). I post this question to get hints or solutions on the way to do this in a reliable way and to make sure I'm not missing an obvious solution (maybe a curl option I don't know ?).
Additional useful things
Parsing JSON with Unix tools
Examples of the content of $response :
success :
{
"_id": "AVQz7Fg0nF90YvJIX_2C",
"_index": "indexation",
"_shards": {
"failed": 0,
"successful": 1,
"total": 1
},
"_type": "document",
"_version": 1,
"created": true
}
error :
{
"error": {
"caused_by": {
"reason": "json_parse_exception: Unexpected character (',' (code 44)): was expecting either valid name character (for unquoted name) or double-quote (for quoted) to start field name\n at [Source: org.elasticsearch.common.io.stream.InputStreamStreamInput#139163f; line: 3, column: 17]",
"type": "json_parse_exception"
},
"reason": "failed to parse",
"root_cause": [
{
"reason": "json_parse_exception: Unexpected character (',' (code 44)): was expecting either valid name character (for unquoted name) or double-quote (for quoted) to start field name\n at [Source: org.elasticsearch.common.io.stream.InputStreamStreamInput#139163f; line: 3, column: 17]",
"type": "json_parse_exception"
}
],
"type": "mapper_parsing_exception"
},
"status": 400
}
A simple workaround is to use the -f/--fail option.
As per documentation :
(HTTP) Fail silently (no output at all) on server errors. This is
mostly done to better enable scripts etc to better deal with failed
attempts. In normal cases when an HTTP server fails to deliver a
document, it returns an HTML document stating so (which often also
describes why and more). This flag will prevent curl from outputting
that and return error 22.
This method is not fail-safe and there are occasions where
non-successful response codes will slip through, especially when
authentication is involved (response codes 401 and 407).
example:
response=$(curl -XPOST 'http://localhost:9200/indexation/document' -d '
{
"content":"'"$txt"'",,
"date_treatment":"'"$(date +%Y-%m-%d)"'"
}' -f ) && echo ok || echo fail

Resources