How can I intervere in the process of parsing the CSV file?
What I intend to do is that if certain values have bad format, I want to correct them. In particular, if there is a value of "1.000-" it results in a NumberFormatException because the minus sign is suffixed instead of being prefixed. Hence, in such a case, I want to switch "1.000-" to "-1.000" and let the parser or CsvToBeanBuilder respectively, continue its work.
Related
I am trying to parameterize my request using CSV Data Set Config. My input includes double quotes("), colon(:) and brackets([])
Eg: fiscal_year ":["2021",2019]"
Had tried with it, but in the actual results its passing as "fiscal_year "":[""2021""
Please share your inputs on what am i missing on the input paramter.
I don't think it's due to double quotes("), colon(:) and brackets([]), CSV stands for comma-separated values so JMeter treats it as a delimiter and reads everything including to the first comma.
So you might want to change the "Delimiter" to something else:
It's hard to come up with a comprehensive solution without seeing at least couple of lines from your CSV file and the way you're parameterizing the HTTP Request with JMeter Variables
In case you have one entry per line in the CSV file it might be easier to go for __StringFromFile() function which reads next line from the file each time it's being called. See Apache JMeter Functions - An Introduction for more information.
Im very new in NiFi..
I get data(FlowFile ?) from my processor "ConsumerKafka", it seems like
So, i have to delete any text before '!',I know a little Python. So with "ExcecuteScript", i want to do something like this
my_string=session.get()
my_string.split('!')[1]
#it return "ZPLR_CHDN_UPN_ECN....."
but how to do it right?
p.s. or, may be, use "substringAfterLast", but how?
Tnanks.
Update:
I have to remove text between '"Tagname":' and '!', how can i do it without regex?
If you simply want to split on a bang (!) and only keep the text after it, then you could achieve this with a SplitContent configured as:
Byte Sequence Format: Text
Byte Sequence: !
Keep Byte Sequence: false
Follow this with a RouteOnAttribute configured as:
Routing Strategy: Route to Property name
Add a new dynamic property called "substring_after" with a value: ${fragment.index:equals(2)}
For your input, this will produce 2 FlowFiles - one with the substring before ! and one with the substring after !. The first FlowFile (substring before) will route out of the RouteOnAttribute to the unmatched relationship, while the second FlowFile (substring after) will route to a substring_after relationship. You can auto-terminate the unmatched relationship to drop the text you don't want.
There are downsides to this approach though.
Are you guaranteed that there is only ever a single ! in the content? How would you handle multiple?
You are doing a substring on some JSON as raw text. Splitting on ! will result in a "} left at the end of the string.
These look like log entries, you may want to consider looking into ConsumeKafkaRecord and utilising NiFi's Record capabilities to interpret and manipulate the data more intelligently.
On scripting, there are some great cookbooks for learning to script in NiFi, start here: https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-1/ta-p/248922
Edit:
Given your update, I would use UpdateRecord with a JSON Reader and Writer, and Replacement Value Strategy set to Record Path Value .
This uses the RecordPath syntax to perform transformations on data within Records. Your JSON Object is a Record. This would allow you to have multiple Records within the same FlowFile (rather than 1 line per FlowFile).
Then, add a dynamic property to the UpdateRecord with:
Name: /Tagname
Value: substringAfter(/Tagname, '!' )
What is this doing?
The Name of the property (/Tagname) is a RecordPath to the Tagname key in your JSON. This tells UpdateRecord where to put the result. In your case, we're replacing the value of an existing key (but it could be also be a new key if you wanted to add one).
The Value of the property is the expression to evaluate to build the value you want to insert. We are using the substringAfter function, which takes 2 parameters. The first parameter is the RecordPath to the Key in the Record that contains the input String, which is also /Tagname (we're replacing the value of Tagname, with a substring of the original Tagname value). The second parameter is the String to split on, which is !.
If your purpose getting the string between ! and "} use ReplaceText with (.*)!(.*)"} , capture second group and replace it with entire content
Please note that this regular expression may not be best for your case but I believe you can find solution for your problem with regular expression
From a csv file, I need to pass
224,329,429
as a single value to one of the parameter in HTTP request.
I have parameterized using CSV data config. But, only 224 is getting passed.
I want 224,329,429 to be treated as a single value.
Please let me know how do I achieve this. Should I change anything in CSV config or CSV file to make this work?
Just use __StringFromFile() function instead of using CSV Data Set Config.
The __StringFromFile() function reads next line from the file each time it's being called so it seems to be a lot easier to stick to it for particular your scenario.
The syntax is as simple as ${__StringFromFile(/path/to/your/file.csv,,,)} and the function can be used anywhere in the script, i.e. directly in the request parameter section.
See Apache JMeter Functions - An Introduction to get started with the JMeter Functions concept and comprehensive information on the above and other JMeter functions.
You should change your delimiter to a not used character e.g. #
In that way you will be able to get full line for every request
Use ${__FileToString(dummy.csv,,payloadvar)} function. It makes the file independent that mean you can use any file extension example: .txt, .csv, .excel etc..
Just keep the string in dummy.csv and it will fetch the whole string.
benefit of using this function is, it will not consider comma's so in case your string has comma separated values then this is the best option.
Just use %2C in the place of comma.
I am working in a simple data loader for text files and would like to add a feature for correctly loading dates into the tables. The problem I have is that I do not know the date format before hand, and it will not be my script doing the inserts - it has to generate insert statements for later use.
The Date.parse is almost what I'd need. If there was a way to grab the format it identified on the string in a way I could use to generate a to_date(...)(Oracle standard) would be perfect.
An example:
My input file:
user_name;birth_date
Sue;20130427
Amy;31/4/1984
Should generate:
insert into my_table values ('Sue', to_date('20130427','yyyymmdd'));
insert into my_table values ('Amy', to_date('31/4/1984','dd/mm/yyyy'));
Note that it is important the original string remains unchanged - so I cannot parse it to a standard format used in the inserts (it is a requirement).
At the moment I am just testing a bunch of regexes and doing some validation, but I was wondering if there was a more robust way.
Suppose (using for example String#scan), you extracted an array of the date strings from a single file. It may be like:
strings = ["20130427", "20130102", ...]
Prepare in advance an array of all formats you can think of. It may be like:
Formats = ["%Y%m%d", "%y%m%d", "%y/%m/%d", "%m/%d/%y", "%d/%m/%y", ...]
Then check all formats that can parse all of the strings:
require "date"
formats =
Formats.select{|format| strings.all?{|s| Date.strptime(s, format) rescue nil}}
If this array formats includes exactly one element, then that means the strings were unambiguously parsed with that format. Using that format, you can go back to the strings and parse them with that format.
Otherwise, either you failed to provide the appropriate format within Formats, or the strings remained ambiguous.
I would use the Chronic gem. It will extract dates in most formats.
It has options to resolve the ambiguity in the xx/xx/xxxx format, but you'd have to specify which to prefer when either match.
I am doing some localization testing and I have to test for strings in both English and Japaneses. The English string might be 'Waiting time is {0} minutes.' while the Japanese string might be '待ち時間は{0}分です。' where {0} is a number that can change over the course of a test. Both of these strings are coming from there respective property files. How would I be able to check for the presence of the string as well as the number that can change depending on the test that's running.
I should have added the fact that I'm checking these strings on a web page which will display in the relevant language depending on the location of where they are been viewed. And I'm using watir to verify the text.
You can read elsewhere about various theories of the best way to do testing for proper language conversion.
One typical approach is to replace all hard-coded text matches in your code with constants, and then have a file that sets the constants which can be updated based on the language in use. (I've seen that done by wrapping the require of that file in a case statement based on the language being tested. Another approach is an array or hash for each value, enumerated by a variable with a name like 'language', which lets the tests change the language on the fly. So validations would look something like this
b.div(:id => "wait-time-message).text.should == WAIT_TIME_MESSAGE[language]
To match text where part is expected to change but fall within a predictable pattern, use a regular expression. I'd recommend a little reading about regular expressions in ruby, especially using unicode regular expressions in ruby, as well as some experimenting with a tool like Rubular to test regexes
In the case above a regex such as:
/Waiting time is \d+ minutes./ or /待ち時間は\d+分です。/
would match the messages above and expect one or more digits in the middle (note that it would fail if no digits appear, if you want zero or more digits, then you would need a * in place of the +
Don't check for the literal string. Check for some kind of intermediate form that can be used to render the final string.
Sometimes this is done by specifying a message and any placeholder data, like:
[ :waiting_time_in_minutes, 10 ]
Where that would render out as the appropriate localized text.
An alternative is to treat one of the languages as a template, something that's more limited in flexibility but works most of the time. In that case you could use the English version as the string that's returned and use a helper to render it to the final page.