I created PowerBI report which which is connecting to data source via API service. Returning json contains thousands of entities. API service is called via Web.Content function. API service returns always total record count and so we are able to calculate nr. of pages which has to be called to obtain whole dataset. This report is displaying data from our servicedesk app, which is deployed on many servers and for many customers and use Query parameters to connect to any of these servers.
Detail of Power query is below.
Why am I writing here. This report was working without any issue more than 1,5 year but on August 17th one of servers start causing erros in step Pages where are some random lines (pages) with errors - see attached picture labeled "Errors in step Pages". and this is reason that next step Entities (List.Union) in query is stopping refresh and generate errors with message:
Expression.Error: We cannot apply field access to the type List. Details: Value=[List] Key=requests
What is notable
API service si returning records in the same order but faulty lists are random when calling with same parameters
some times is refresh without any error
The same power query called on another server is working correctly , problem is only with one specific server.
This problem started without notice on the most important server after 1,5 year without any problem.
Here is full text power of query for this main source, which is used later in other queries to extract all necessary data. Json is really complicated and I extract from it list of requests, list of solvers, list of solver groups,.... and this base query and its output is input for many referenced queries.
Errors in step Pages
let
BaseAPIUrl = apiurl&"apiservice?", /*apiurl is parameter - name of server e.g. https://xxxx.xxxxxx.sk/ */
EntitiesPerPage = RecordsPerPage, /*RecordsPerPage is parameter and defines nr. of record per page - we used as optimum 200-400 record per pages, but is working also with 4000 record per page*/
ApiToken = FnApiToken(), /*this function is returning apitoken value which is returning value of another api service apiurl&"api/auth/login", which use username and password in body of call to get apitoken */
GetJson = (QParm) => /*definiton general function to get data from data source*/
let
Options =
[ Query= QParm,
Headers=
[
Accept="application/json",
ApiKeyName="apitoken",
Authorization=ApiToken
]
],
RawData = Web.Contents(BaseAPIUrl, Options),
Json = Json.Document(RawData)
in Json,
GetEntityCount = () => /*one times called function to get nr of records using GetJson, which is returned as a part of each call*/
let
QParm = [pp="1", pg="1" ],
Json = GetJson(QParm),
Count = Json[totalRecord]
in
Count,
GetPage = (Index) => /*repeatadly called function to get each page of json using GetJson*/
let
PageNr = Text.From(Index+1),
PerPage = Text.From(EntitiesPerPage),
QParm = [pg = PageNr, pp=PerPage],
Json = GetJson(QParm),
Value = Json[data][requests]
in Value,
EntityCount = List.Max({ EntitiesPerPage, GetEntityCount() }), /*setup of nr. of records to variable*/
PageCount = Number.RoundUp(EntityCount / EntitiesPerPage), /*setup of nr. of pages */
PageIndices = { 0 .. PageCount - 1 },
Pages = List.Transform(PageIndices, each GetPage(_) /*Function.InvokeAfter(()=>GetPage(_),#duration(0,0,0,1))*/), /*here we call for each page GetJson function to get whole dataset - there is in comment test with delay between getpages but was not neccessary*/
Entities = List.Union(Pages),
Table = Table.FromList(Entities, Splitter.SplitByNothing(), null, null, ExtraValues.Error)
I also tried another way of appending pages to list using List.Generate. This is also bringing random errors in list but
it is bringing possibility to transform to table in contrast with original way with using List.Transform, but other referenced queries are failing and contains on the last row errors
When I am exploring content of faulty page/list extracting it via Add as New Query there are always all record without any fail.....
Source = List.Generate( /*another way to generate list of all pages*/
() => [Page = 0, ReqPageData = GetPage(0) ],
each [Page] < PageCount,
each [ReqPageData = GetPage( [Page] ),
Page = [Page] + 1 ],
each [ReqPageData]
),
#"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error), /*here i am able to generate table from list in contrast when is used List.Generate*/
#"Expanded Column1" = Table.ExpandListColumn(#"Converted to Table", "Column1"), /*here aj can expand list to column*/
#"Removed Errors" = Table.RemoveRowsWithErrors(#"Expanded Column1", {"Column1"}) /*here i try to exclude errors, but i dont know what happend and which records (if any) are excluded*/
Extracting errored page
and finnaly I am tottaly clueless not able to find the cause of this behavior on this specific server. I tested to call pages which are errored via POSTMAN, I discused this issue with author of API service and He also tried to call this API service with all parameters but server is returning every page OK, only Power query is not able to List.Transform ...
I will be grateful and appreciate any tips or advice or if somebody solved the same issue in the past ....
Kuby
No, each error line of list in step List.Transform coud by extracted as new query and there are all records from one page OK. hmmmm
Finnaly, problem described in this issue was caused by "corrupted" content of returning json. The provider of core system informed me that they found bug and after fixing on the side of servisdesk is everything OK again. I tried to find problem in Power query and problem was in servisdesk. :(
Is there a way to add each session Id that is retrieved from a Loop Controller to a list and assign it to a property for use in the following thread group? Below I used a couple of Dummy Sampler to explain my requirement.
I had 3 users stored in a list to retrieve 3 session ids in the setUp Thread Group.
JSR223 PreProcessor
List usernames = Arrays.asList('Peter', 'Alex', 'Mary');
props.put('accounts', usernames);
I was able to read a username from this property to get a session id in the response accordingly per iteration in the Loop Controller.
"sessionId": "this_is_my_session_id-${__groovy(props.get('accounts').get(${__jm__LoopController__idx} % 3),)}-${__jm__LoopController__idx} "
I parsed the 3 session ids out by a JSR223 PostProcessor
import groovy.json.JsonOutput
import groovy.json.JsonSlurper
def jsonSlurper = new JsonSlurper();
def response = jsonSlurper.parseText(prev.getResponseDataAsString());
def json = JsonOutput.toJson(response.sessionId)
def sessionId = new JsonSlurper().parseText(json)
log.info('The session id is:' + sessionId)
ArrayList<String> sessionIds = new ArrayList<String>();
props.put("sessionIds", sessionIds.add(sessionId))
I needed to add these 3 session ids to a list and assign it to a property so that I can use one session id inside the property per VU/thread in the following Thread Group. But it didn't work as expected. It threw error saying No such property: sessionIds
${__groovy(props.get(sessionIds).get(${__jm__UseSession__idx} % 3),)}
We don't know what do you "expect"
Most probably the problem is here:
props.put("sessionIds", sessionIds.add(sessionId))
Collection.add() function returns a boolean value so it puts true to the sessionIds property instead of the real value of the ArrayList.
So I believe you need to change it to something like:
sessionIds.add(sessionId)
props.put("sessionIds", sessionIds)
if you're going to run the JSR223 Test Element in the loop you can also reconsider the way you're initializing the sessionIds and implement the following logic:
If sessionIds property exists - read its value
If it doesn't exist - create a new ArrayList
Something like:
ArrayList<String> sessionIds = props.get("sessionIds") ?: new ArrayList<String>()
More information on Groovy scripting in JMeter: Apache Groovy: What Is Groovy Used For?
getproperty values passed from Thread Group 1 to Thread group2
Result from BeanShell assertion
Step 1- USing jdbc request to get data from database with 2 columns and multiple rows.
Step 2 - From ThreadGroup 1, Set property to the database results using ${__setProperty(StateCodeProperty,${stateDetails})};
Step 3 - Access in Thread Group 2 by get property using beanshell assertion- String result = (vars.get("${__property(StateCodeProperty)}")); I need help on how to separate the columns and use it in api call. –
In any case if you want to access the DB results in different Thread group then you can try to do something like this inside beanshell assertion (not sure though) -
ArrayList results = ${__property(StateCodeProperty)}; //it should return the object as an arraylist
for (int i; i < results.size(); i++) {
if (results.get(i).get("statecode").equals("NY")) { //iterating the results, 'statecode' is the name of your 1st column, similarly you can do for 'State'
//Do your comparisons or whatever you like here
}
}
I'm using jMeter 3.2 to write some tests. I have a CSV file with test account info. Each row contains login info for a user. Each user needs to request a token that is used on later requests.
My test plan:
The get token request retrieves a token. The login requests logs in the user and returns another token. Select customer card selects a customer and returns the final token. The code for the postprocesser is (I'm not experienced in this, so any advice is appreciated):
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
// Check if our map already exists
if (props.get("map") == null) {
JSONObject obj = new JSONObject();
obj.put("${department}", new String(data));
log.info("Adding department to map. Department: ${department}. Token: " + new String(data));
props.put("map", obj.toJSONString());
} else {
// Retrieve the current map
map = props.get("map");
JSONParser parser = new JSONParser();
JSONObject jobj = (JSONObject) parser.parse(map);
// Add the new department (with it's token) to the map
jobj.put("${department}", new String(data));
log.info("Updating map for department. Department: ${department}. Token: " + new String(data));
props.put("map", jobj.toJSONString());
}
Attempt 1:
I'm setting up a once only controller to log in a user and retrieve the token.
Now lets say I have 10 lines in my CSV file but in my test I only want to use 3 users and loop 10 times. What happens is that 3 login requests are sent (one for each user). This works fine for the first iteration. At the 2. iteration the 3 threads will use row 4-6 which doesn't have a token and thereby fail.
Attempt 2:
I'm using an if controller to check whether the token has been set or not. I haven't got this working at all. I added a beanshell preprocessor to the controller where I attempt to retrieve the token. If it's null or empty I set the token variable to "". In the if controller I check for this value. But yeah. No luck yet.
Attempt 3
In Beanshell check if the token is created already. If not, call the test fragment that retrieves it. Unfortunately this seems not possible.
It might be worth noting that I store my tokens in a property, so that all threads can access it.
Please let me know if you need more information.
I figured out a solution. In essence what I tried to do is to store a token for each row in the data file.
I did this by creating a setUp Thread Group which is executed before other thread groups. In this I loop through the data and store a token for each. Now all other thread groups may access these as they run.
I have some functional tests created via JMeter. It is pretty huge but i can't handle one simple check.
I generate properties using BSF pre processor with help of JS. Parameter (lets call it "payment_fee") should be generated only if other parameter (lets call it "role") has a value = 1 .In this case we post pre generated integer into payment_fee and everything works well. But if role =2 then we should post nothing into payment_fee.
The problem is, i don't know how to say to JMeter: In case if role = 1 use variable with pre generated payment_fee but if role = 2, you shouldn't use this variable so just post an empty value for payment_fee . Server waits for an integer so empty string or NULL had been rejected.
For more clarification:
I will try to explain more clear.
Here is a part of my code
var role = Math.floor(Math.random()*3+1)
var paymentType = ["creditcard","cash"]
var randomPay = installerType[Math.floor(Math.random()*installerType.length)];
var payment = "";
var paymentFee;
if (role == 1){
payment+=randomPay,
paymentFee = Math.floor((Math.random() * 999) + 1) / 10.00
}
vars.put("role", role);
vars.put("payment", payment);
vars.put("paymentFee", paymentFee);
And if role == 1 i should post paymentFee value. Like this - http://prntscr.com/b50kk1 BUT! if role == 2 || role == 3 I should remove this value, so it should be like this http://prnt.sc/b50l82
I don't fully understand what you're trying to do as your 2 statements clash:
But if role =2 then we should post nothing into payment_fee
Server waits for an integer so empty string or NULL had been rejected
You should know few bits about JMeter properties:
Properties are global for the whole JVM. Once you set property it will "live" until you exit JMeter.
Properties can be accesses by all threads of all Thread Groups.
So it might be the case when you define property earlier or by another thread and expect it to be not set later on.
Also BSF PreProcessor and JavaScript isn't the best combination from performance perspective, consider switching to JSR223 PreProcessor and Groovy language.