In the OpenApi3-spec of my API I have an endpoint returning CSV-data. No my Dredd-Test fails although example and returned are exactly the same. I assume the problem that my API returns CSV with a BOM character.
Now I have no idea How I can encode the character correctly in OpenAPI3-YAML. Can anyone help me out?
I tried \ufeff as I would do in JSON but it didn't work.
responses:
"200":
description: ffffoo
content:
text/csv;charset=UTF-8:
schema:
type: string
example: |-
a;b;c
1;2;3
4;5;5
A literal block scalar does not process escape sequences, you'll need a double-quoted scalar for that:
responses:
"200":
description: ffffoo
content:
text/csv;charset=UTF-8:
schema:
type: string
example: "\ufeff\
a;b;c\n\
1;2;3\n\
4;5;5"
An escaped line break excludes the line break from the content. Unescaped line breaks in double-quoted scalars would be folded into an undesired space. \n\ basically replaces the space that would be generated from the line break with a proper line break.
You can of course do away with the line breaks in the source but I'd say this is more readable.
Related
Sometimes, when processing log events in vector, a log source might have hyphens as key names - for example json structured logs. Assuming it is from a third party and changing them there is not an option, how can we handle these keys?
A sample log message (contrived for demonstration) is:
{
"labels":{"no_hypens":"normal field","this-has-hypens":"this is a test"},
"message":"a message",
"timestamp":"2022-11-01T12:03:00.941866394Z"
}
Note the field labels.this-has-hyphens there.
I managed to put together a test case, and find out the syntax, both in providing test data like this and in extracting the data in VRL:
The test case
---
tests:
- name: hypens
inputs:
- insert_at: hypens
type: log
log_fields:
labels."this-has-hypens": "this is a test"
labels.no_hypens: "normal field"
outputs:
- extract_from: hypens
conditions:
- type: vrl
source: |
assert_eq!("normal field", .no_hypens)
assert_eq!("this is a test", .output_without_hypens)
This will insert and check for two fields from the input data. Note the hyphenated key segment needs to be quoted.
Next the VRL in the transform:
---
transforms:
hypens:
type: remap
inputs:
- route
source: |
log(., "error")
. = {
"no_hypens": .labels.no_hypens,
"output_without_hypens": .labels."this-has-hypens",
}
The log message is there as while debugging this, I had to figure out that the test hyphenated field didn't even get to the transform until I had quotes around it.
Then the field reference itself needs to have quotes too, after the dot (not square brackets).
This will pass the tests, and output the right data.
Using Tavern and trying to run tavern-ci against this yaml:
test_name: tavern poc
- name: list
request:
url: https://xxx.xxx.xxx.us/api/v3/institutions/
method: GET
response:
status_code: 200
headers:
content-type: application/json
save:
body:
content: content
Am getting
E ScannerError: mapping values are not allowed here
E in "/Users/xxx/xxx/xxx/test_poc.tavern.yaml", line 3, column 9
Have tried many of the solutions presented here (most of which are 'put a space after the colon') without joy.
Yamllint gives the same error...
At the root of your YAML document you have a mapping with key test_name and as value the start of a plain scalar tavern .....
The parser expects a key, with the same indent as the first line, on the second line or a continuation of your plain scalar from the first line. The second line is empty, so it continues with the same expectations on the third line. There it finds an - which is further indented than the beginning of test_name, so it is not a key, but part of the plain scalar. Then it finds name also part of a the plain scalar started on the first line and then : (colon + space).
But that colon+space is not allowed in plain scalar, as that gives potential ambiguity with the start of another key-value pair.
The unlikely solution is that you put double quotes before tavern and at the end of the YAML document.
More likely you should include a key for which the structure starting with - name: list this is the value. E.g.:
test_name: tavern poc
stages:
- name: list
request:
(as from the second entry you get from googling "tavern yaml")
I have a string:
string = "\\"
puts string
# => \
I am interpolating this into a new string and sending to a database. However the database (InfluxDB) uses backslashes as escape characters so pushing this string can cause an error.
For example, if I pass the following to Influx it will cause an "unterminated string" error:
insert_cmd = <<-TXT
INSERT INTO my_db.default.my_measurement,my_tag=1 my_val="#{string}"
TXT
My question is how can I replace \ in a string with \\ (two actual backslashes).
I have it working with gsub("\\", "\\\\\\") but I don't understand why this works and the following doesn't:
string.gsub("\\", "\\\\")
# SyntaxError: (irb):10: syntax error, unexpected $undefined, expecting end-of-input
Why doesn't this work? Why does gsub("\\", "\\\\\\") work? Is there a better way?
solved
As I mentioned in a comment, actually I am not manually interpolating into a INSERT INTO string. I am using influxdb-ruby:
INFLUXDB_CLIENT.write_point("things", time: Time.now.to_i, values: { foo: "\\" })
It turns out this is a bug with that gem: https://github.com/influxdata/influxdb-ruby/issues/200
It is fixed in v 0.4.2 and i was using 0.4.1
You just use parameterized query strings:
INSERT INTO my_db.default.my_measurement,my_tag=1 my_val=%{1}
Where when you call it you do this:
influxdb.query("...query...", params: [ string ])
What you did was create a classic injection bug by sending unescaped data into a query. The same principle applies in any database with a plain-text string representation, or even other data formats like HTML and JavaScript.
I have a method Embed.toggler that takes a hash argument. With the following code, I'm trying to use a heredoc in the hash.
Embed.toggler({
title: <<-RUBY
#{entry['time']}
#{entry['group']['who']
#{entry['name']}
RUBY
content: content
})
However, I'm getting the following error trace:
syntax error, unexpected ':', expecting tSTRING_DEND
content: content
^
can't find string "RUBY" anywhere before EOF
syntax error, unexpected end-of-input, expecting tSTRING_CONTENT or tSTRING_DBEG or tSTRING_DVAR or tSTRING_END
title: <<-RUBY
^
How I can avoid getting this error?
Add a comma after your <<-RUBY:
Embed.toggler({
title: <<-RUBY,
#{entry['time']}
#{entry['group']['who']
#{entry['name']}
RUBY
content: content
})
this does work in general. I am not sure why it wasn't working in my code though.
It didn't work because hashes require key/value pair to be separated by a comma, like {title: 'my title', content: 'my content' } and your code just didn't have the comma. It was hard to see that because of the cumbersome HEREDOC syntax.
Do you know if there is a way to perform operations on the string?
You're playing with fire. It's always safer (and usually cleaner) to extract a variable and do post-processing on a variable itself:
title = <<-RUBY
#{entry['time']}
#{entry['group']['who']
#{entry['name']}
RUBY
Embed.toggler(title: title.upcase, content: content)
However, if you feel dangerous today, you can just add operations after opening HEREDOC literal, just as you've added the comma:
Embed.toggler({
title: <<-RUBY.upcase,
#{entry['time']}
#{entry['group']['who']
#{entry['name']}
RUBY
content: content
})
But I discourage you from this because it destroys readability.
I'm trying to make a regex that removes me in my text email: toto#toto.com.
example: I request information on your project email: toto#free.fr
So I did this that captures me "email: toto#toto.com"
message ="I request information on your project email. toto#free.fr"
message.gsub!("/(email: [-a-z0-9_+\.]+\#([-a-z0-9]+\.)+[a-z0-9]{2,4}$)/i")
it returns me nothing, and I wish there was just in the message text.
thanks
Try this. This should work for both uppercase, lowercase and emails appear in the middle of the string.
email = /[A-Za-z]{5}:\s[A-Za-z0-9._%+-]+#[A-Za-z0-9.-]+\.[A-Za-z]{2,4}/
s = "I request information on your project email: toto#free.fr"
s.match(email).pre_match #=> "I request information on your project "
s2 = "This email: blah#bLAH.com is in the middle"
s2.match(email).pre_match #=> "This "
s2.match(email).post_match #=> " is in the middle"
But there are more cases not covered e.g. email: followed by many spaces
Your code has several problems:
You are looking for "email: ...", but you message has "email. ...".
You use gsub!, with one parameter, which is not the classic use case, and returns an Enumerator. The classic use case expects a second parameter, which indicates to what you want to substitute the found matches:
Performs the substitutions of String#gsub in place, returning str, or
nil if no substitutions were performed. If no block and no replacement
is given, an enumerator is returned instead.
You pass a string to the gsub! - "/(email: [-a-z0-9_+\.]+\#([-a-z0-9]+\.)+[a-z0-9]{2,4}$)/i", which is different than sending a regex. To pass a regex, you need to drop the quotes around it: /(email: [-a-z0-9_+\.]+\#([-a-z0-9]+\.)+[a-z0-9]{2,4}$)/i
So a fix to your code would look like this:
message ="I request information on your project email: toto#free.fr"
message.gsub!(/(email: [-a-z0-9_+\.]+\#([-a-z0-9]+\.)+[a-z0-9]{2,4}$)/i, '')
# => "I request information on your project "
Also note I changed your code to use gsub instead of gsub!, since gsub! changes the underlying string, instead of creating a new one, and unless you have a good reason to do that, it is not encouraged to mutate the input arguments...
If you want to remove the email from the text use String#sub
message = "I request information on your project email. toto#free.fr"
message.sub!(/[A-Za-z]{5}:\s[A-Za-z0-9._%+-]+#[A-Za-z0-9.-]+\.[A-Za-z]{2,4}/, '')
# => "I request information on your project "