We have request to format row like this:
886,89,5052299385882,1,
Problem is last character of row which should be comma, it is export job for integration so this request is dictated from another side. Is there easy way to achieve this with FlatFileItemWriter?
Currently we modeled our java representation of row to have additional string which is always empty, and told field extractor to extract blank filed as last value for row creation but I am searching for a way to append something on each line.
FlatFileItemWriter has a lineAggregator property.
Write your own implementation of LineAggregator interface (a delegation should be enough) and add a comma to returned string.
Related
Im very new in NiFi..
I get data(FlowFile ?) from my processor "ConsumerKafka", it seems like
So, i have to delete any text before '!',I know a little Python. So with "ExcecuteScript", i want to do something like this
my_string=session.get()
my_string.split('!')[1]
#it return "ZPLR_CHDN_UPN_ECN....."
but how to do it right?
p.s. or, may be, use "substringAfterLast", but how?
Tnanks.
Update:
I have to remove text between '"Tagname":' and '!', how can i do it without regex?
If you simply want to split on a bang (!) and only keep the text after it, then you could achieve this with a SplitContent configured as:
Byte Sequence Format: Text
Byte Sequence: !
Keep Byte Sequence: false
Follow this with a RouteOnAttribute configured as:
Routing Strategy: Route to Property name
Add a new dynamic property called "substring_after" with a value: ${fragment.index:equals(2)}
For your input, this will produce 2 FlowFiles - one with the substring before ! and one with the substring after !. The first FlowFile (substring before) will route out of the RouteOnAttribute to the unmatched relationship, while the second FlowFile (substring after) will route to a substring_after relationship. You can auto-terminate the unmatched relationship to drop the text you don't want.
There are downsides to this approach though.
Are you guaranteed that there is only ever a single ! in the content? How would you handle multiple?
You are doing a substring on some JSON as raw text. Splitting on ! will result in a "} left at the end of the string.
These look like log entries, you may want to consider looking into ConsumeKafkaRecord and utilising NiFi's Record capabilities to interpret and manipulate the data more intelligently.
On scripting, there are some great cookbooks for learning to script in NiFi, start here: https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-1/ta-p/248922
Edit:
Given your update, I would use UpdateRecord with a JSON Reader and Writer, and Replacement Value Strategy set to Record Path Value .
This uses the RecordPath syntax to perform transformations on data within Records. Your JSON Object is a Record. This would allow you to have multiple Records within the same FlowFile (rather than 1 line per FlowFile).
Then, add a dynamic property to the UpdateRecord with:
Name: /Tagname
Value: substringAfter(/Tagname, '!' )
What is this doing?
The Name of the property (/Tagname) is a RecordPath to the Tagname key in your JSON. This tells UpdateRecord where to put the result. In your case, we're replacing the value of an existing key (but it could be also be a new key if you wanted to add one).
The Value of the property is the expression to evaluate to build the value you want to insert. We are using the substringAfter function, which takes 2 parameters. The first parameter is the RecordPath to the Key in the Record that contains the input String, which is also /Tagname (we're replacing the value of Tagname, with a substring of the original Tagname value). The second parameter is the String to split on, which is !.
If your purpose getting the string between ! and "} use ReplaceText with (.*)!(.*)"} , capture second group and replace it with entire content
Please note that this regular expression may not be best for your case but I believe you can find solution for your problem with regular expression
http request default. http request config. csv data config used. csv data file used .I wanted to add data from csv file, but if i pass single parameter also its adding "&" character before parameter.
ex: https://www.linkedin.com/start/join?&name=A0A1A0
Just remove this ? sign from the /api/dealer/searchByPostalCode
JMeter automatically detects query string beginning in the URL path and tries to merge parameters with the ones you specify in the path to avoid syntax errors.
Given you don't provide any parameters in the "Path" section you should remove ? from there.
Another possible reason is that your CSV data contains this & character, double check it using Debug Sampler and View Results Tree listener combination.
As per the official documentation:
As a special case, the string "\t" (without quotes) in the delimiter
field is treated as a Tab.
When the end of file (EOF) is reached, and the recycle option is true,
reading starts again with the first line of the file.
Source: http://jmeter.apache.org/usermanual/component_reference.html#CSV_Data_Set_Config
So I'd request you to remove \n in delimiter field from the CSV Data set config and then try.
Would someone help me form a script in Bash to keep only the unique lines, based solely on identifying duplicate values in a single field (the first field)
If I have data like this:
123456,23423,Smith,John,Jacob,Main St.,,Houston,78003<br>
654321,54524,Smith,Jenny,,Main St.,,Houston,78003<br>
332423,9023432,Gonzales,Michael,,Everyman,,Dallas,73423<br>
123456,324324,Bryant,Kobe,,Special St.,,New York,2311<br>
234324,232411,Willis,Bruce,,Sunset Blvd,,Hollywood,90210<br>
438329,34233,Moore,Mike,,Whatever,,Detroit,92343<br>
654321,43234,Smith,Jimbo,,Main St.,,Houston,78003<br>
And I like to only keep the lines which do not have matching first fields
(result would be a file with these contents below, based on above sample)
332423,9023432,Gonzales,Michael,,Everyman,,Dallas,73423<br>
234324,232411,Willis,Bruce,,Sunset Blvd,,Hollywood,90210<br>
438329,34233,Moore,Mike,,Whatever,,Detroit,92343<br>
What would the bash/awk approach be? Thanks in advance.
From a csv file, I need to pass
224,329,429
as a single value to one of the parameter in HTTP request.
I have parameterized using CSV data config. But, only 224 is getting passed.
I want 224,329,429 to be treated as a single value.
Please let me know how do I achieve this. Should I change anything in CSV config or CSV file to make this work?
Just use __StringFromFile() function instead of using CSV Data Set Config.
The __StringFromFile() function reads next line from the file each time it's being called so it seems to be a lot easier to stick to it for particular your scenario.
The syntax is as simple as ${__StringFromFile(/path/to/your/file.csv,,,)} and the function can be used anywhere in the script, i.e. directly in the request parameter section.
See Apache JMeter Functions - An Introduction to get started with the JMeter Functions concept and comprehensive information on the above and other JMeter functions.
You should change your delimiter to a not used character e.g. #
In that way you will be able to get full line for every request
Use ${__FileToString(dummy.csv,,payloadvar)} function. It makes the file independent that mean you can use any file extension example: .txt, .csv, .excel etc..
Just keep the string in dummy.csv and it will fetch the whole string.
benefit of using this function is, it will not consider comma's so in case your string has comma separated values then this is the best option.
Just use %2C in the place of comma.
I am a newbie to Pentaho (installed today). I was able to do basic transformation in Spoon. Now I need to do some stuff, which I can't figure out how.
my input looks like
2012-09-17|garima|fbhjgjhgj87687jghgj88jgjj|garima#1347868164626|::ffff:120.56.132.137
3rd field is an ID, for which I need to get some information from a REST API
http://api.app.com/app/api/v1/feature/fbhjgjhgj87687jghgj88jgjj
What do I need to do in Spoon to get this done?
Also, data return will be in json format. how do I parse that?
You should first get your input with a CSV File Input using | as delimiter. Then you can get the 3rd field as a string.
Next you probably need to remove all spaces from this string with a String operations step. Look at the Remove special character column, and select space.
Then you need to concatenate it with your http address http://api.app.com/app/api/v1/feature/. For this you'll use a Calculator step. At this step first create a new temporary field tmpAddr, with operation Define a constant value for ... (or something like this, sorry my spoon is in portuguese). At the Field A column you'll write your http address. It's a good practice, after you make this work, to set your address as a system variable so if it changes you don't need to replace it everywhere on your transformations (look at menu Edit -> System Variables).
Now on the same Calculator step create another field, let's say MyAddress, with operation A+B. Choose for Field A the field tmpAddr you just created, and for Field B the 3rd field from your input.
Now on your stream you should have the full address as a field MyAddress. Connect a REST client step. Mark Accept URL from field and choose field MyAddress as URL Field Name. Set Application Type to JSON. Set Result Fieldname as MyResult.
If you need further JSON parsing you can add a Json input step. Set Source is defined in a field and select field MyResult as Get Source from field.
An alternate approach is to use the "Replace in String" step to append the string.
Set 'use RegEx' to Y
Set 'Search' to (.*)
Set 'Replace with' to http://api.app.com/app/api/v1/feature/$1
Set 'Whole Word' to Y
The parentheses in the regex set up a capture group that you can then insert into your replacement string with the $X syntax