Sometimes, when processing log events in vector, a log source might have hyphens as key names - for example json structured logs. Assuming it is from a third party and changing them there is not an option, how can we handle these keys?
A sample log message (contrived for demonstration) is:
{
"labels":{"no_hypens":"normal field","this-has-hypens":"this is a test"},
"message":"a message",
"timestamp":"2022-11-01T12:03:00.941866394Z"
}
Note the field labels.this-has-hyphens there.
I managed to put together a test case, and find out the syntax, both in providing test data like this and in extracting the data in VRL:
The test case
---
tests:
- name: hypens
inputs:
- insert_at: hypens
type: log
log_fields:
labels."this-has-hypens": "this is a test"
labels.no_hypens: "normal field"
outputs:
- extract_from: hypens
conditions:
- type: vrl
source: |
assert_eq!("normal field", .no_hypens)
assert_eq!("this is a test", .output_without_hypens)
This will insert and check for two fields from the input data. Note the hyphenated key segment needs to be quoted.
Next the VRL in the transform:
---
transforms:
hypens:
type: remap
inputs:
- route
source: |
log(., "error")
. = {
"no_hypens": .labels.no_hypens,
"output_without_hypens": .labels."this-has-hypens",
}
The log message is there as while debugging this, I had to figure out that the test hyphenated field didn't even get to the transform until I had quotes around it.
Then the field reference itself needs to have quotes too, after the dot (not square brackets).
This will pass the tests, and output the right data.
Related
How to show all the parameter's names as comma-separated strings (concatenated) and assign them to One Variable in YAML Azure so that i can use this variable in several places
i tried using
parameters:
- name: Var1
displayName: Testing
type: string
- name: Var2
displayName: Coding
type: string
- name: Var3
displayName: Marketing
type: string
variables:
- name: allParametersString
${{each item in parameters}}:
value: $allParametersString + ','+ item.displayName
my desired output is upon using $allParametersString I should get
Testing,Coding,Marketing
but this is leaving an error mentioning 'value' is already defined so can anyone help me? regarding this, I am searching for a solution for 2 weeks :(
I found the way of using bash to assign values will work for this i did
variables:
- name: allParametersString
value: ' '
steps:
- ${{ each parameter in parameters }}:
# the below code will help us reassign the values to variables with bash so i am appending all parameters separated by comma
- bash: echo '##vso[task.setvariable variable=allParametersString]$(allParametersString)${{ parameter.Key }}, '
- script:
echo 'printing all parameters names separated by comma .->$(allParametersString)'
Please let me know if I can improve it more
this helps me understand that in order to reassign or assign twice or concatenate the string using YAML this is the one way of doing it
Your current thinking is not feasible.
There are several things that bind you.
1, The first is the processing logic of yml expression in DevOps.
See this official document:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/runs?view=azure-devops#process-the-pipeline
From the first sentence given, we know your yml will be expanded like this:
parameters:
- name: Var1
displayName: Testing
type: string
default: value1
- name: Var2
displayName: Coding
type: string
default: value2
- name: Var3
displayName: Marketing
type: string
default: value3
variables:
- name: allParametersString
value: xxx
value: xxx
value: xxx
variable of yml concept doesn't allow the written method of the above. That's why you encountered error 'value' is already defined.
2, The second is the structure allowed by DevOps yml files.
Every section of yml definition has limited predefined key. This means that compile time cannot find other container to store the variable.
3, I am afraid the usage of yml expression does not support you to do so.
Refer to this:
Each Keyword This tell you the standard usage of 'each':
You can use the each keyword to loop through parameters with the
object type.
JOIN Expression
This is the way the DevOps yml pipeline provides compile-time flattening of data, but it doesn't work in your case.
And there's no such thing as an index or subscript to navigate to the last run.
4, By the way, the item object doesn't have such information 'displayName':
trigger:
- none
parameters:
- name: Var1
displayName: Testing
type: string
default: value1
- name: Var2
displayName: Coding
type: string
default: value2
- name: Var3
displayName: Marketing
type: string
default: value3
steps:
- ${{each item in parameters}}:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "${{ convertToJson(item) }}"
Result:
The DevOps yml pipeline has no built-in features to implement your needs. If you have to do this, a feasible method is to call the API to get the content of the yml file, then parse and get the parameter part (this is actually a self-designed parser), and then combine the acquired parameters into the variable in the script And pass logging command set variable(isoutput=true). In this way, other tasks can use this combined variable.
This can be done, but is overly complicated and you need to consider whether it is necessary to do such a thing.
This the output of my curl
{
"expand": "renderedFields,names,schema,operations,editmeta,changelog,versionedRepresentations",
"id": "240937",
"self": "https://placeholder.atlassian.net/rest/api/latest/issue/240937",
"key": "placeholder-355",
"fields": {
"description": "We need the following page layout changes made (for all user profiles unless specified in front of the item):\n\n# Display {{TestAccount__c}} field on the account object underneath the *Pricing Group* field\n# Display {{blng__LegalEntity__c}} field that sits on {{Order Product}} to be visible on *Order* as well as *Contract* objects _(position doesn’t matter). Read only is fine as long as that works on SF reports if we need to bring this attribute in_\n# Display {{blng__LegalEntity__c}} field that sits on *Invoice Lines* to be visible on the *Invoice* object _(position doesn’t matter). Read only is fine as long as that works on SF reports if we need to bring this attribute in_\n# Display ‘*Notes*’ section on the credit request page - copy the one that exists on the contract object e.g. (sample below)\n!image-20220714-021135.png|width=50%!\n# -Display- *-Agent-* -field on contract object to enable users to see if account has an agent or not- → already in production (not needed)\n# Change default options for *tasks* - the default subjects are:\n#* Call\n#* Send Letter\n#* Send Quote\n#* Other\n# Need to update to the following:\n#* Call\n#* Video Call\n#* Face to Face Meeting\n# Add in *Contact hierarchy* functionality in Salesforce\n## Anyone can update the ‘Reports to’ field\n## Show the following fields on the hierarchy page"
}
}
With jq I'm getting json
jq --raw-output '.fields.description' jira-story.json
Result:
# Display *Pricing Group* field
# Display *Contract*
# Change default options for *tasks* - the default subjects are:
#* Call
#* Send Letter
#* Send Quote
#* Other
# Need to update to the following:
#* Call
#* Video Call
#* Face to Face Meeting
# Add in *Contact hierarchy*
## Anyone can update the ‘Reports to’ field
## Show the following fields on the hierarchy page
I want it nicely format it as
### Display *Pricing Group* field
### Display *Contract*
### Change default options for *tasks* - the default subjects are:
- Call
- Send Letter
- Send Quote
- Other
### Need to update to the following:
- Call
- Video Call
- Face to Face Meeting
### Add in *Contact hierarchy*
- Anyone can update the ‘Reports to’ field
- Show the following fields on the hierarchy page
How can I replace the value in a jq before the output?
"#*" to "-" or "##" to "-" and "#" to "###"
You don't necessarily need to perform the substitution in jq, you can pipe the output to sed:
jq -r '.fields.description' | sed 's/^#[#*]/-/;s/^#/###/'
If you really must do it with a single jq expression, then what user jhnc wrote in the comments:
jq -r '.fields.description | gsub("\n#[#*]";"\n-") | gsub("\n# ","\n### ")'
Attempting to confirm that of all the schema in the head of a page exactly 3 of them should have a specific string within them. These schemas have no tags or sub classes to differentiate themselves from each other, only the text within them. I can confirm that the text exists within any of the schema:
cy.get('head > script[type="application/ld+json"]').should('contain', '"#type":"Product"')
But what I need is to confirm that that string exists 3 times, something like this:
cy.get('head > script[type="application/ld+json"]').contains('"#type":"Product"').should('have.length', 3)
And I can't seem to find a way to get this to work since .filter, .find, .contains, etc don't filter down the way I need them to. Any suggestions? At this point it seems like I either need to import a custom library or get someone to add ids to these specific schema. Thanks!
The first thing to note is that .contains() always yields a single result, even when many element match.
It's not very explicit in the docs, but this is what it says
Yields
.contains() yields the new DOM element it found.
If you run
cy.get('head > script[type="application/ld+json"]')
.contains('"#type":"Product"')
.then(console.log) // logs an object with length: 1
and open up the object logged in devtools you'll see length: 1, but if you remove the .contains('"#type":"Product"') the log will show a higher length.
You can avoid this by using the jQuery :contains() selector
cy.get('script[type="application/ld+json"]:contains("#type\": \"Product")')
.then(console.log) // logs an object with length: 3
.should('have.length', 3);
Note the inner parts of the search string have escape chars (\) for quote marks that are part of the search string.
If you want to avoid escape chars, use a bit of javascript inside a .then() to filter
cy.get('script[type="application/ld+json"]')
.then($els => $els.filter((index, el) => el.innerText.includes('"#type": "Product"')) )
.then(console.log) // logs an object with length: 3
.should('have.length', 3);
Using Tavern and trying to run tavern-ci against this yaml:
test_name: tavern poc
- name: list
request:
url: https://xxx.xxx.xxx.us/api/v3/institutions/
method: GET
response:
status_code: 200
headers:
content-type: application/json
save:
body:
content: content
Am getting
E ScannerError: mapping values are not allowed here
E in "/Users/xxx/xxx/xxx/test_poc.tavern.yaml", line 3, column 9
Have tried many of the solutions presented here (most of which are 'put a space after the colon') without joy.
Yamllint gives the same error...
At the root of your YAML document you have a mapping with key test_name and as value the start of a plain scalar tavern .....
The parser expects a key, with the same indent as the first line, on the second line or a continuation of your plain scalar from the first line. The second line is empty, so it continues with the same expectations on the third line. There it finds an - which is further indented than the beginning of test_name, so it is not a key, but part of the plain scalar. Then it finds name also part of a the plain scalar started on the first line and then : (colon + space).
But that colon+space is not allowed in plain scalar, as that gives potential ambiguity with the start of another key-value pair.
The unlikely solution is that you put double quotes before tavern and at the end of the YAML document.
More likely you should include a key for which the structure starting with - name: list this is the value. E.g.:
test_name: tavern poc
stages:
- name: list
request:
(as from the second entry you get from googling "tavern yaml")
I am trying to support Clojure's ignore text form, #_, (a sort of comment) in VS Code, which uses tmLanguage for its grammar definitions. Since it is common to disable a block of code using #_, I want the disabled block of code to retain its syntax highlighting and just italicize it, indicating its status.
But my lack of skills using tmLanguage seems to stop me. This is one of the failing attempts (a snippet of the cson):
'comment-constants':
'begin': '#_\\s*(?=\'?#?[^\\s\',"\\(\\)\\[\\]\\{\\}]+)'
'beginCaptures':
'0':
'name': 'punctuation.definition.comment.begin.clojure'
'end': '(?=[\\s\',"\\(\\)\\[\\]\\{\\}])'
'name': 'meta.comment-expression.clojure'
'patterns':
[
{
'include': '#constants'
}
]
With constants defining some Clojure constants objects, like keyword:
'keyword':
'match': '(?<=(\\s|\\(|\\[|\\{)):[\\w\\#\\.\\-\\_\\:\\+\\=\\>\\<\\/\\!\\?\\*]+(?=(\\s|\\)|\\]|\\}|\\,))'
'name': 'constant.keyword.clojure'
What I want to happen is that the constants definitions will be used ”inside” the comment. For keywords I have this (failing) spec:
it "tokenizes keywords", ->
tests =
"meta.expression.clojure": ["(:foo)"]
"meta.map.clojure": ["{:foo}"]
"meta.vector.clojure": ["[:foo]"]
"meta.quoted-expression.clojure": ["'(:foo)", "`(:foo)"]
"meta.comment-expression.clojure": ["#_:foo"]
for metaScope, lines of tests
for line in lines
{tokens} = grammar.tokenizeLine line
expect(tokens[1]).toEqual value: ":foo", scopes: ["source.clojure", metaScope, "constant.keyword.clojure"]
(The last test in that list). It fails with this message:
Expected
{ value : ':foo',
scopes : [ 'source.clojure', 'meta.comment-expression.clojure' ] }
to equal
{ value : ':foo',
scopes : [ 'source.clojure', 'meta.comment-expression.clojure', 'constant.keyword.clojure' ] }.
Meaning I am not getting the constant.keyword.clojure scope in place and thus no keyword-colorization for me. 😢
Anyone knows how to do this?
Your keyword regex starts with a lookbehind that requires that there must be a single whitespace, (, [ or { character before keywords. The _ from #_ doesn't meet that requirement.
(?<=(\\s|\\(|\\[|\\{))
You could simply add _ to the list of allowed characters:
(?<=(\\s|\\(|\\[|\\{|_))
Note that this still wouldn't work as-is for your "#_:foo" test case because of the similar lookahead at the end. You could possibly allow $ there, make the match optional, or change the test case.