I have a mapping which I need to be able to run against multiple source schemas (having the same structure), one schema at-a-time. Given the number of schemas, I would rather not set up a session for each schema in order to specify a particular mapping connection, as that will require new sessions to be added as new schemas are added.
Is it possible to set up a workflow in such a way that the data source connection for a mapping within a session is defined (or passed in as a parameter of some sort) at run-time?
Configure the workflow or the session to use a parameter file.
In the session settings, change the 'hard-coded' source connection (option Use object) to Use Connection Variable and enter a variable name with a $DBConnection prefix (e.g. $DBConnectionSource01).
Create a parameter file in an appropriate location with the following contents:
[Global]
$DBConnectionSource01=connection_name
I believe what you are looking for is going to be solved using a parameter file and a bit of shell script (assuming your server is on some Unix flavour).
Setup your workflow to run with a parameter file. Declare a special parameter for Database Connection (staring with $DBConnection) in the parameter file global section. Change your session properties to use that parameter.
You need to create appropriate Relational connection objects for each of the source db/schemas.
Now, write some shell program to dynamically change that parameter file and replace the value of the parameter in parameter file with the new value you want.
Typical sequence of events at run time should look like this -
Whenever you want to run that workflow/session/mapping for a different source, launch the script with appropriate parameter to effect the change in Informatica parameter file.
Shell script runs for launching the job for a given db source. The script run should change the $DBConnection parameter in parameter file
Launch the workflow through pmcmd using the parameter file.
Related
Very new to JMeter (and a long time Stack Overflow listener but first time caller so take it easy on me) so any help would be appreciated.
I am trying to set up a JMeter test that uses multiple dynamically generated access tokens to run across scenarios. I currently have a set of data using the CSV Data Set Config containing login credentials of a user's email and password for example:
email1#email.com,password1
email2#email.com,password2
Next I send a HTTP POST request to the Login service which generates an accessToken. Then, I am using the JSON Extractor to grab the generated accessToken. After that I am using the BeanShell Assertion to store the accessToken property/variable.
My issue seems to lie here in this last step since it will only store the last generated variable instead of each of the generated accessTokens. I want to be able to store/overwrite the grabbed accessToken for each email password combination. I would like it to modify/populate the CSV file like this:
email1#email.com,password1,accessToken1
email2#email.com,password2,accessToken2
I have also tried using the Pre Processor > User Parameters
Screen shot of User Parameters
I would like to have the "userBearerToken" variable update/overwrite along with the tests, but I cannot find a way to do so or if this can even currently be done.
I'm finding it difficult to word what I am trying to ask, but basically I want to store multiple dynamically generated variables (accessTokens that change and time out) and use them in other tests. I don't care which component can handle this (either the CSV or User Parameter), but I need to be able to store these variables with their corresponding email password credentials.
If you want to store the accessToken value into a JMeter property for using in other Thread Group(s) be aware that properties are global for the whole JVM and remain until JMeter is restarted so if you define a single accessToken property - each JMeter Thread (or iteration) will overwrite the value.
The solution is to use current thread (or iteration) number as prefix or postfix, this can be done using either __${__threadNum} function or relevant JMeter Variable depending on how iteration is defined or both.
Example setup:
In first thread group: ${__setProperty(access_token_${__threadNum},bar,)}
In second thread group: ${__P(access_token_${__threadNum},)}
Demo:
If you want to save the values into a file writing into the current one is not the best idea as you can (and most probably will) get malformed file due to a form of a race condition. So I would recommend using Sample Variables property instead.
If you add the next line to user.properties file:
sample_variables=email,password,accessToken
JMeter will store the variables named ${email}, ${password}, and ${accessToken} along with their values in the .jtl results file which is basically CSV file which can be re-used anywhere else.
However if you have a requirement to store only the credentials and the token you can go for the Flexible File Writer plugin and configure it to save the aforementioned variables values into a separate file, the relevant configuration would be as simple as:
variable#0,variable#1,variable#2
You can install Flexible File Writer plugin using JMeter Plugins Manager
I have a requirement to set a parameter say a timestamp in a parameter file using a command task that triggers a Unix shell script that does this task in a workflow and later in the same workflow, use the parameter that is set earlier. Is that possible?
The informatica version used is 9.6
You need to define the Parameter file property on a Session. Not just set the parameter value for a session, but make the session read the paramfile.
With Workflow parmeter file it works like this (briefly speaking):
Workflow is initialized and reads the assigned parameter file. All variables are set.
Upon session initialization all the variables and parameters are already set .
The file is not read again, variables are not re-set.
However, once a parameter file is defined as Session property, it works in a bit different way.
Workflow is initialized and reads the assigned parameter file. All variables are set.
Session is initialized, and it reads its own parameter file, setting all variables and parameters.
In this case, the values defined in session parameter file will be used.
Of course, it is not necessary to use Workflow level parameter file.
It is also possible to have one session that generates the parameter file for the very next session to use it. Provided it is defined as session property, as mentioned above.
I got the solution :). Dont know if there is any other option. But the trick that worked for me is, if you have to set a parameter and then read back the parameter to use it in the same workflow sue to whatever requirement, then you can set the parameter and then use it in a session inside a worklet, and this perfectly works. I am not sure why but this definitely works :)
You can use the Assignment task for this purpose. Set a workflow variable and use it wherever you want in the same workflow.
i have a file 'A' and File 'B' which has business date as a variable both files have different values.I specify file A in session level and file B is specified in workflow level in path.So which file integration service pickup and which file will be used as parameter file?
At session level file 'A' will be used as it will override the workflow level parameter file. For all other session's in which file A is not specified as parameter file 'B' will be used as it is specified at workflow level.
This question is similar to the concept of local and global variables in a way.
Workflow level parameter file will be your global level, and Session level param file will be your local variable.
Thus whenever a session has is defined with a parameter file it will use the same parameter file (in this case A).
If you have sessions that do not have any parameter file defined then those sessions will use B parameter file defined in the workflow level.
I have a common process group that will infer avro schema based on the file i supplied. But I want to set the Avro Record Name to a name corresponding to the filename i am supplying. So I used ${filename}. But the InferAvroSchema got error saying the record name is empty. Note that before this, I already set the property "filename" to the flowfile attribute and it has a value since i tested it using ReplaceText to see if there's value for ${filename}
Unfortunately this looks like a bug in InferAvroSchema. Many of the properties support expression language, but then the processor doesn't evaluate them against the incoming flow file. So it ends up only being able to use a value typed directly into the property (non-EL), or a value from system or environment properties which doesn't really make sense for a lot of these properties.
I created this JIRA for the issue:
https://issues.apache.org/jira/browse/NIFI-2465
The fix is that all of the calls to evaluateAttributeExpressions() should be passing in a flow file like:
context.getProperty(CSV_HEADER_DEFINITION).evaluateAttributeExpressions(inputFlowFile).getValue()
We run our tests in non gui mode and pass in various parameters like Server, port, threads etc. We'd also like to run our test in GUI mode, and be able to change these parameters in the GUI.
What I wanted to do, was use 2x User Defined variable objects, and have one with static data we can edit, and another with parameters. That way the actual test plan doesn't change, but we can set variables as parameters, and just disable the static data one. Or disable the parameterized one when we want to run with static data.
But this doesn't seem to work - no errors nothing.
I do something similar, but I used a UDF for this.
What I did was set up my variables and use default values.
VARNAME VALUE
otl_PROTOCOL ${__P(otl_protocol,https)}
otl_PORT ${__P(otl_port,443)}
otl_THREADS ${__P(otl_threads,1)}
otl_REPS ${__P(otl_reps,1)}
otl_RAMP ${__P(otl_ramp,0)}
I did one for server name as well. This way, I can use the defaults here or I can pass in parameters in either the command line args or in my user.properties. This works very well for me.
How are you passing in values from non-GUI mode? via CSV, using properties, User-Parameters, etc.?
If CSV, this is an easy fix. Simply create a User Defined Variables (UDV) object at the Test Plan Level and "hard code" all your parameters (Server, port, etc.). Keep your CSV config. If you run in GUI mode, disable the CSV and enable the UDV. When you save the file, keep the CSV enabled and the UDV disabled - that way when you run from non-GUI it'll read the CSV file.
If using properties, I would do the following: Create a UDV with all your parameters (Server, etc.). Before entering values, copy the component so you have TWO identical UDVs. In UDV 1, enter the property value from the command line. In UDV 2, enter the "hard coded" values. Throughout your script, replace any references to the properties with variable references. It would look something like this:
UD1: UD2:
Name Value Name Value
SERVER ${__P(Server)} SERVER devdomain.com
PORT ${__P(Port)} PORT 4111
Some HTTP Request
Domain: ${SERVER}
If using user-parameters, it would be the same concept as above.
Actually i am using a function named as regular expression for getting value.
here i have to use variable under variable because there is lots of value with same boundary, so i have selected one paragraph as ${ABC} and fetching value from this paragraph ${XYLEM}. both the variable created as regular expression.tried with ${__V(ABC${XYLEM})};
${__setProperty{Search_Investment_Banking,$Investment_Banking}};