Pentaho Kettle Variable Update - etl

In light words the problem could be stated as "I am setting an environment variable in one transformation t1.ktr and using the variable in another transformation t2.ktr. I want to update the variable every 15 mins without actually stopping the t2.ktr i.e. during the execution of t2.ktr . How can I achieve that ?
To give you an overview :
I am making rest api call using HTTP POST step in my transformation (multi-threaded). To make this rest api call I need to pass a certain Token which gets expired in every 15 mins. I am getting this token via another API call. So in one transformation lets say token.ktr I am getting the token and storing it in environment variable TOKEN via Set Variable step and in next transformation lets call it rest.ktr I am getting this variable via Get Variable step and using it in HTTP POST call.
For 15 min I get proper responses, but after that I get error responses since token gets expired.
Let me know if further clarification is needed.

I've had a similar case that i produced a sequence of KTRs in a Job with Evaluations.
I achieved this with a Job and 2 KTRs
DISCLAIMER - For my case, i have to make several HTTP GET's, not just a single long one, so i can easily loop from one GET to another and check the Expiration of the Token bettween HTTP GETs, this is how i achieved it.
Step 1:
Start your KTR that has the Auth API Call. In this KTR you'll also use a formula step, creating a DateTime row using the NOW() function, this will get the timestamp of when you called the Authorization. This KTR will end by setting the Auth and this timestamp as variables in the Parent job.
KTR1 - Example
Step 2:
In the Job you will call this KTR first, and right after it, you'll use a Set Variables step, i named this variable Expiration, and here you'll set it to OK, signaling Token not expired. Next you'll call the KTR that makes the HTTP GET, using the Token. The result of this KTR will be Success (not expired token, success on HTTP GET, moves on), or Fail (Token expired, set Expiration to "Expired").
JOB Example
Step 3:
After the HTTP GET you need to check to result, whether or not it succeeded, or if the token expired mid way, so you would need to renew token and continue the loop, or end the loop if no more HTTP GETs are needed.
Again, this is for a very especific use case, and i'm sure other people can do better, but, this is my take on the issue, it works for me.

Related

jmeter, regular expression extractor with parallel controller

Sorry for my poor English, and thank you in advance.
I'm trying to make a thread group that makes jmeter logged in our system.
to accomplish this, I need to POST data which contains ID, password, and token.
The token generated every time when the page has opened and hold in hidden value.
So, the usual solution which is like GET response, do the regular expression extractor, and make the value variable and POST it later request doesn't work for me.
since the token become different from when jmeter GET token and POST it.
Then, I found a parallel controller and this might be a solution for me.
but I can't find a way to do it.
also, there are no references in my mother language(Japanese).
I want POST token to certain login action so I need to do regular expression extractor with parallel controller.
then POST the token with ID and password at same time.
I want to know the way of the above or if it's impossible, is there any alternative solution that might work for me.
Thank you a lot for your help.
Don't worry about language at all :)
I will add a solution according to how I understand your problem.
As I understood, your scenario is,
We Request a page
We will get a response with token
We fetch the token and send the next request with the token
We will get a response with a different token
Need to fetch the new token and send another request
In this case, I think you are trying to use the first token in all the requests. Without doing that what if you fetch the token per each request?
The problem with the parallel controller is if you want to fetch the token from a previous request. It might not work.
Please correct me if I haven't understood the question correctly
Parallel Controller is not something you should be using, it's for simulating AJAX requests as JMeter cannot execute JavaScript like browser does and hence cannot run multiple HTTP request samplers in parallel with one virtual user.
Your test plan structure should be much simpler, to wit:
HTTP Request sampler (to get login page)
a Post-Processor (most probably you're looking for CSS Selector Extractor) to get the values from hidden fields
HTTP Request sampler (to perform login)
See Variabilize and Correlate the script chapter for more details on the concept

Avoid REST service to be consumed twice

I have a question about Spring MVC controllers scope and REST services. I have a couple of REST services, wich returns a token in the response so I can later recreate the state of the application, but I don't want the users use the same token twice, so I've decided to save an unique identifier inside the token and also in HttpServletRequest, so I can check it when I get the requests (a new identifier is generated in every request).
So, my questions are: 1) is there any other way to be sure that some user will not use the same token more than once (also considered to save that identifier in DB, but I would have lot of queries to insert, delete, verify, etc).2) is it ok for the controller that receives the requests to be a singleton, or should it be prototype? (considering that the identifier is taken from session and I don't want to mix it between different sessions).
A few words on tokens that are valid only once
It's not possible to achieve it
without keeping the track of the tokens somewhere. This security schema require some trade-offs, deal with it.
Give the user a token and keep the track of it on server side, just like a white list:
When a token is issued, add it to the white list.
When a request comes to the server with a token, check the white list and:
If the token is valid, accept the request and remove the token from the white list.
If the token is invalid, refuse the request by returning a proper status code such as 403.
Also, consider assigning an expiration date to the token and refuse any request that comes to the server with an expired token.
Regarding your performance concerns: Bear in mind that premature optimization is the root of all evil. You shouldn't optimize until you have a performance problem and you have proven that the performance problem comes from the way you store your tokens. You could start storing the tokens in the database and then consider a cache in memory, for example. But always be careful when fixing a problem that you currently don't have.
Working with JWT
If you go for JWT, there are a few Java libraries to issue and validate JWT tokens such as:
jjwt
java-jwt
jose4j
The jti claim should be used to store the token identifier on the token. When validating the token, ensure that it's valid by checking the value of the jti claim against the token identifiers you have on server side.
For the token identifier you could use UUID. In Java, it's as simple as:
String uuid = UUID.randomUUID().toString();
Since HttpSession#getId() is unique, you can use it to create an unique token:
// pseudo code
String token = httpSession.getId() + "-" + System.currentTimeMillis();
You can also create your own counter.
Here my two techniques to prevent it
Disable submit button:
We can disable submit button right before our function call HTTP request and enable it again after finish gets HTTP response. This technique is effective for the process that takes a long time to finish (more than 5 sec.). The user can not click n’ click again because of impatience to get the result. Additionally, we may show a loading box for a good experience.
Issue request token/id:
This technique actually more complicated and difficult to implement, but thanks to a good framework (such as Spring Boot) to make this easier. Before we are going to the code implementation, let’s talk about the mechanism first;
When form page is loaded, issue a new requestId
put issued requestId to HTTP header before calling the backend service
backend service identify a requestId is already registered or not
if requestId is already registered then we can mark as a violation request

Using JMeter regex extraction in a loop controller

Right now, I have a Simple Controller set up to send a HTTP Request, and am extracting the request ID number from the response I get. This is working correctly. However, I want to send 3 such requests, and extract the request ID number from each, and may want to do even more in the future. I'm thinking the best way to set this up would be a loop controller, but I'm not sure how to extend my regex extractor that works for 1 HTTP request to work for multiple HTTP requests. The request ID number will be different each time, so I have to have a different variable for each, but I'm not sure how to make that happen. I was also looking at the ForEach loop, but I don't have any variables I am using as input, so I'm not sure that is the right choice for what I'm looking to do.

why Loadrunner Correlation is getting failed

I want to correlate this 181-418-5889 in the following statement: regSend&transferNumber=181-418-5889".
I used the regular web_reg_save_param: But it failed... any suggestion?
You are using the statement in the wrong location, such as using it just before the request is sent containing the correlated value versus just before the location where the response containing the value is sent to the client
You are not receiving the correct page response and as a result you may not be able to collect the value. The page may be an HTTP 200 page but the content could be completely off. Always check for an appropriate expected result
Your left boundary, right boundary and other parameters are incorrect to collect the value you need
You have not been through training and you are being forced by your management to learn this tool via trial and error
1- I am not using the statement in the wrong location since I did find the needed value I want to correlate via the Tree function and put it just before the statement that hold this value
2- The Page is not an HTTP 200
3- The Left and right boundary are correct since I checked the text if it does exist twice in the response body.
4- I know the tool (Loadrunner) but in fact, the application is developed under ZK platform and I am not sure if ZK and Loadrunner are compatible knowing that I did implement the dtid function in my script to have a static desktop id each time I replay the process.

How to deal with out-of-sequence Ajax requests?

What is the best way deal with out-of-sequence Ajax requests (preferably using a jQuery)?
For example, an Ajax request is sent from the user's browser anytime a field changes. A user may change dog_name to "Fluffy", but a moment later, she changes it to "Spot". The first request is delayed for whatever reason, so it arrives at the server after the second, and her dog ends up being called "Fluffy" instead of "Spot".
I could pass along a client-side timestamp along with each request, and have the server track it as part of each Dog record and disregard earlier requests to change the same field (but only if there is a difference of less than 5 minutes, in case the user changes the time on her machine).
Is this approach sufficiently robust, or is there a better, more standardized approach?
EDIT:
Matt made a great point in his comment. It's much better to serialize requests to change the same field, so is there a standard way of implementing Ajax request queues?
EDIT #2
In response to #cherouvim's comment, I don't think I'd have to lock the form. The field changes to reflect the user's change, a change request is placed into the queue. If a request to change the same field is waiting in the queue, delete that old request. 2 things I still would have to address:
Placing a request into the queue is an asynchronous task. I could have the callback handler from the previous Ajax request send the next request in the queue. Javascript code isn't multi-threaded (or... is it?)
If a request fails, I would need the user interface to reflect the state of the last successful request. So, if the user changes the dog's name to "Spot" and the Ajax request fails, the field would have to be set back to "Fluffy" (the last value successfully committed).
What issues am I missing?
First of all you need to serialize server side processing for each client. If you are programming in Java then synchronizing execution on the http session object is sufficient. Serializing will help in case the second update comes while the first is being processed.
A second enhancement you can implement in your entity updating is http://en.wikipedia.org/wiki/Optimistic_concurrency_control. You add a version property (and column) for your entity. Each time an update happens this is incremented once. In fact the update statement looks like:
update ... set version=6 ... where id=? and version=5;
If affected rows from above pseudoquery query are 0 then someone else has managed to update the entity first. What you do then is up to you. Note that you need to be rendering the version on the html update form of the entity as a hidden parameter and sending it back to the server each time you update. On return you have to write back the updated version.
Generally the first enhancement would be enough. The second one will improve the system in case many people are editing the same entities at the same time. It solves the "lost update" problem.
I would implement a queue on the client side with chaining of successful requests or rollbacks on unsuccessful requests.
You need to define "unsuccessful", be it a timeout or a returned value.

Resources