I just started learning about the custom processor in nifi. I want to understand the specific case of working of onTrigger. I am doing some operations in onTrigger function using the property values which are defined in the nifi flow Processor interface.
Ex: Property value in the custom processor takes a string separated by ',' and in the onTrigger function I write a code which converts the string into an array of String and removes the additional white spaces.
My question is will this operation run every time a flowfile passes through the custom processor or will it be converted only once.
I tried going through the official development docs but could'nt find info on this
The Java code of a processor is compiled when you run a Maven build to produce the NAR file. The code is not compiled by NiFi itself.
You then deploy a NAR file to a NiFi instance by placing it in the lib directory, and then you use components from that NAR in your flow by adding them to the canvas.
Once a component is on the canvas and it is started, then the onTrigger method is called according to the scheduling strategy.
Whatever code is in onTrigger will run for every execution of the processor, so your code to read the property and split the value will run every time.
If the property supports expression language from flow files, then you need to run this code every time in onTrigger because the resulting value could be different for every flow file.
if the property does not support expression language from flow files, then you can instead use a method with #OnScheduled and process the property value into whatever you need, and store in a member variable of the processor, this way it only happens one time.
Related
I would like to create a JMeter test setup with important sampling and test variation parameters entirely controlled by csv-Files (i.e. not modifying the JMX-File). It should be run with maven.
Ideas is as follows:
sources.csv contains
sampleRate;file
40;samplesForController1.csv
30;samplesForController2.csv
5;samplesForController3.csv
...
SampleRate should determine, how often a certain number of tests (defined by parameters in the respective file) should be executed (relative to others).
How could this be achieved? Im asking for the first step here (make sure the files/testSamples are sampled/executed given the indicated sampleRate) as I think I can solve the second part (dealing with parameters in samplesForController1.csv etc.) by myself.
P.S.
I'm struggling with the options presented here: In Jmeter, I have to divide number of thread into the multiple http requests in different percentage but have to keep sequence remain same since
afaics, thread groups cannot be created on-the-fly/dynamically/progammatically
apparently, Throughput Controller needs to know its child element(s probablities) upfront (i.e. not created dynamically), otherweise, sampling is very odd (I could not get it working maintaining the desired sampleRate)
I did not try to integrate jmeter-plugins in the maven build thus far as my impression is, the available plugins/controllers also needs to know their child element upfront
It is possible to create thread groups programmatically, check out:
Five Ways To Launch a JMeter Test without Using the JMeter GUI
jmeter-from-code example project
jmeter-java-dsl project
You can use Switch Controller and use a function like __groovy() to generate the child element index, example implementation can be found in Running JMeter Samplers with Defined Percentage Probability article
It's not a problem to use JMeter Plugins with Maven, see Adding jar's to the /lib/ext directory documentation section for example setup
Using jMeter to set up a soak / load test that needs to run each request with different data. The structure I currently have is
Thread Group (2 Users, 2 Loops)
- Simple controller
-- Java Sampler (Custom Plugin) to convert CSV Formula data into NewThreadData.csv (as variable)
-- Java Sampler (Custom Plugin) to create directory of files created with NewThreadData.csv merged into a template
-- While Controller Condition - js NewThreadData column not = EOF
--- CSV Data Set Config NewThreadData.csv (Recycle False / Stop of EOF False) - filename passed as variable
--- JMS PUblisher with the FileName a variable using the filename column from within NewThreadData.csv
My problem is the on the second loop, the data is updated in NewThreadData.csv, but the CSV in the while loop never runs again.
Appears that the CSV Data Set Config "knows" it has been run, regardless of the actual CSV data.
Questions
How can I get the CSV Data Set Config to be rerun / re executed in this scenario?
Are there undocumented variables or means of getting the config to reprocess?
Is there a way to spawn a new thread on each iteration rather than reusing the existing thread, as the CSV does execute once for each "User"[thread]. I also tried Stop on EOF : True, but that stopped the second loop.
Aim is to eventually ramp up the user count and the number of loops (changing to forever); with there being about about 100 different combinations of data to be inserted on each loop. The formula I am using has time and thread number to give me data uniqueness along with other data that is dynamically created from a formula. Recycle on EOF is not feasible as I need to regenerate the csv contents on each loop. A super-csv I don't think is feasible to cover the load and soak scenarios.
Thanks in anticipation. Andrew
I don't think it's possible to "reset" the CSV Data Set Config, it's a configuration element hence once it reads the file in its current state it will "stick" to its content.
If you're manipulating file content dynamically I would rather recommend going for __CSVRead() function instead which is evaluated in the runtime just where it's placed therefore it doesn't reserve file and it "rewinds" to the beginning of the file when the last line is read.
More information: How to Pick Different CSV Files at JMeter Runtime
CSV Data Set Config element is executed first and only once even though you have placed it within the while controller. Hence CSV Data Set Config element is not suitable for your requirement.
You could use JSSR223 Pre-processor to work with the dynamic CSV files using a supported programming language/script (Java , Groovy)
I am trying to change the value of property "yarn.scheduler.capacity.maximum-am-resource-percent" at the time when job is running in order to provide more resource unit to AM container. My basic logic is, more resource unit to AM, can run more YarnChild, results good throughput.
I am trying to build a jmeter testplan, where all the test values are sent from a csv datafile.I want to add assertions(provided in the datafile) to my HTTP Request at runtime and execute the test. The reason behind doing this is to keep the plan flexible according to the number of assertions. In my case, the assertions are getting added at the runtime; however they fail to get executed. May I know what should be done to get the components added and executed in the same flow?
For example: A part of plan looks like:
XYZ
--HTTP Sampler
-- Response Assertion1
-- Response Assertion2
-- JSON Extractor
where XYZ -->keyword based transaction controller(reusable component)
Everytime I have a request of type XYZ, this chunk of components will get executed. In my case, I do not want to place anything such as Assertions, pre/post processors, extractors in the test plan already. I want to generate these components at run time and execute them (as per my test requisites).
Issue: The problem here is that I cant load the components programmatically and execute them in the same flow. The reason being, the compiler does not know beforehand what all components it needs to execute, so it bypasses the newly added components.
So, I need some alternative solution to execute this.
You can add Response Assertion (or multiple) with Pattern to test filled with a variable as ${testAssert1} and set the variable by default as empty, for example
Put in User Defined Variables name testAssert1 with empty value.
Your assertion(s) will pass until you on run time set the variable with a different value, for example using User Parameters Pre Processor.
In WebSphere Liberty Java Batch,
Is it possible to pass first Step Output to Next step as input parameter.
e.g. First step is Batchlet and second step is chunk. Once first step completes its execution output should be passed to second step runtime..
I'm guessing you are thinking of this in z/OS JCL terms where a step would write output to a temporary dataset that gets passed to a subsequent step. JSR-352 doesn't get into dataset (or file) allocation. That's up to the application code. So you could certainly have a step that wrote output into a file (or dataset) and a later step could certainly read from that same file (or dataset) if it knew the same. You could make the name into a job property that was provided as a property to the batchlet and reader. You could even externalize the value of the job property as a job parameter.
But nothing is going to delete the file for you at the end of the job (like a temporary dataset would get deleted). You'll need to clean up the file yourself.
Is that what you were asking?
You can use the JobContext user data: JobContext.set/getTransientUserData().
This does not however allow you to populate a batch property (via #Inject #BatchProperty) in a parallel way to the manner in which you can supply values from XML via substitution with job parameters.
We have raised an issue to consider an enhancement for the next revision of the Batch specification to allow a property value to be set dynamically from an earlier portion of the execution.
In the meantime there is also the possibility to use CDI bean scopes to share information across steps, but this also is not integrated with batch property injection.