I am using apdex report and the current JTL size is >3GB and i see the following issue while generating the report. Could someone suggest a workaround?
CMD>jmeter.bat -g .\Result1.jtl -o .\Report1\
An error occurred: Error while processing samples: Consumer failed with message :Consumer failed with message :Consumer failed with message :Consumer failed with message :Consumer failed with message :Mismatch between expected number of columns:17 and columns in CSV file:30, check your jmeter.save.saveservice.* configuration or check if line 98797899 in 'temp\latencyVsRequest\latencyVsRequest**************-0' is complete
errorlevel=1
Jmeter.log :
ERROR o.a.j.r.c.ReportGeneratorConfiguration: Error parsing property jmeter.reportgenerator.start_date with value: yyyyMMddHHmmss using format: yyyyMMddHHmmss
java.text.ParseException: Unparseable date: "yyyyMMddHHmmss"
at java.text.DateFormat.parse(DateFormat.java:396) ~[?:?]
at org.apache.jmeter.report.config.ReportGeneratorConfiguration.loadFromProperties(ReportGeneratorConfiguration.java:677) [ApacheJMeter_core.jar:5.2.1]
at org.apache.jmeter.report.dashboard.ReportGenerator.(ReportGenerator.java:154) [ApacheJMeter_core.jar:5.2.1]
at org.apache.jmeter.JMeter.start(JMeter.java:542) [ApacheJMeter_core.jar:5.2.1]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:564) ~[?:?]
at org.apache.jmeter.NewDriver.main(NewDriver.java:253) [ApacheJMeter.jar:5.2.1]
ERROR o.a.j.r.c.ReportGeneratorConfiguration: Error parsing property jmeter.reportgenerator.end_date with value: yyyyMMddHHmmss using format: yyyyMMddHHmmss
java.text.ParseException: Unparseable date: "yyyyMMddHHmmss"
at java.text.DateFormat.parse(DateFormat.java:396) ~[?:?]
at org.apache.jmeter.report.config.ReportGeneratorConfiguration.loadFromProperties(ReportGeneratorConfiguration.java:686) [ApacheJMeter_core.jar:5.2.1]
at org.apache.jmeter.report.dashboard.ReportGenerator.(ReportGenerator.java:154) [ApacheJMeter_core.jar:5.2.1]
at org.apache.jmeter.JMeter.start(JMeter.java:542) [ApacheJMeter_core.jar:5.2.1]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:564) ~[?:?]
at org.apache.jmeter.NewDriver.main(NewDriver.java:253) [ApacheJMeter.jar:5.2.1]
TimeStamp* INFO o.a.j.r.c.ReportGeneratorConfiguration: Will use date range start date: yyyyMMddHHmmss, end date: yyyyMMddHHmmss
2020-12-19 10:32:42,607 INFO o.a.j.r.c.ReportGeneratorConfiguration: Property 'jmeter.reportgenerator.graph.totalTPS.exclude_controllers' not found, using default value 'false' instead.
Your question doesn't have enough details in order to provide the solution, if you need input from us you need to provide:
1st line of your .jtl file
2nd line of your .jtl file
Line 98797899 of your .jtl file
Line 98797900 of your .jtl file (if exists)
According to the error you're getting, the columns number doesn't match the header/current JMeter configuration so I can think of the following options:
Your test result is really incomplete, if this is the case - either re-run your test and wait for graceful shutdown of all threads or delete the last line of your .jtl file using your favourite text editor. If your test ends prematurely due to error which you don't expect - make sure to check jmeter.log for the execution and follow recommendations from 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure
If you're merging 2 different .jtl result files make sure that JMeter Results File Configuration is the same for both, if there is an extra column in one of them which you can omit - delete this column and amend the configuration
As the last resort you can try generating tables and charts using JMeterPluginsCMD Command Line Tool which is more "tolerant" to the results file configuration and tries to "guess" the format from the file itself
Related
When I run a distributed test with JMeter, I get the following error on some slave clients. For this reason, for example, if I send 10000 requests, only 5000 requests go.
2023-02-07 17:35:55,125 ERROR o.a.j.s.BatchSampleSender: sampleOccurred
java.rmi.MarshalException: error marshalling arguments; nested exception is:
java.net.SocketException: Broken pipe (Write failed)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:160) ~[?:?]
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:217) ~[?:?]
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:162) ~[?:?]
at com.sun.proxy.$Proxy19.processBatch(Unknown Source) ~[?:?]
at org.apache.jmeter.samplers.BatchSampleSender.sampleOccurred(BatchSampleSender.java:182) ~[ApacheJMeter_core.jar:5.5]
at org.apache.jmeter.samplers.DataStrippingSampleSender.sampleOccurred(DataStrippingSampleSender.java:106) ~[ApacheJMeter_core.jar:5.5]
at org.apache.jmeter.samplers.RemoteListenerWrapper.sampleOccurred(RemoteListenerWrapper.java:94) ~[ApacheJMeter_core.jar:5.5]
at org.apache.jmeter.threads.ListenerNotifier.notifyListeners(ListenerNotifier.java:58) ~[ApacheJMeter_core.jar:5.5]
at org.apache.jmeter.threads.JMeterThread.notifyListeners(JMeterThread.java:1037) ~[ApacheJMeter_core.jar:5.5]
at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:591) ~[ApacheJMeter_core.jar:5.5]
at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:501) ~[ApacheJMeter_core.jar:5.5]
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:268) ~[ApacheJMeter_core.jar:5.5]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
Caused by: java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method) ~[?:?]
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110) ~[?:?]
at java.net.SocketOutputStream.write(SocketOutputStream.java:150) ~[?:?]
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) ~[?:?]
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:127) ~[?:?]
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1883) ~[?:?]
at java.io.ObjectOutputStream$BlockDataOutputStream.writeByte(ObjectOutputStream.java:1921) ~[?:?]
at java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1582) ~[?:?]
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:352) ~[?:?]
at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:293) ~[?:?]
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:155) ~[?:?]
... 12 more
I looked in the etc/ssh/ssh_config settings but it doesn't seem to be because of it.
It looks like JMeter slaves cannot report results back to master due to networking issues. If you're using SSH tunnels for establishing the connectivity you might want to check SSH server logs for any suspicious entries.
From JMeter perspective you can check the following:
RMI ports which are used for bi-directional communication should be open and accessible
You can also play with batching properties like num_sample_threshold and asynch.batch.queue.size and try to setting it to lower value so sending results would occur more often and the size of the chunk would be less.
More information:
Configuring JMeter
Apache JMeter Properties Customization Guide
I'm trying to use the JMeterPluginsCMD on Windows to generate load test result graphs after a JMeter run. I installed the plugin from the JMeter Plugins Manager.
When I run a command, I get an error message. What can I do about this error message?
`JMeterPluginsCMD.bat --generate-png "test.png" --input-jtl "...\test-3.jtl" --plugin-type ResponseTimesOverTime --width 800 --height 600`
The plugin JMeterPluginsCMD.bat existed in JMETER_home/bin
I am getting the following error:
2021-11-17 09:54:35,131 WARN o.a.j.u.JMeterUtils: Exception 'Cannot invoke "java.util.Properties.getProperty(String, String)" because "org.apache.jmeter.util.JMeterUtils.appProperties" is null' occurred when fetching String property:'sampleresult.default.encoding', defaulting to: ISO-8859-1
2021-11-17 09:54:35,137 WARN o.a.j.u.JMeterUtils: Exception 'Cannot invoke "java.util.Properties.getProperty(String)" because "org.apache.jmeter.util.JMeterUtils.appProperties" is null' occurred when fetching String property:'jmeterPlugin.prefixPlugins'
2021-11-17 09:54:35,138 INFO k.a.j.PluginsCMDWorker: Using JMeterPluginsCMD v. N/A
2021-11-17 09:54:35,139 INFO o.a.j.u.JMeterUtils: Setting Locale to en_US
2021-11-17 09:54:35,147 INFO k.a.j.JMeterPluginsUtils: Loading user properties from: E:\Outcode\ProtocolNow\apache-jmeter-5.3\bin\user.properties
2021-11-17 09:54:35,148 INFO k.a.j.JMeterPluginsUtils: Loading system properties from: E:\Outcode\ProtocolNow\apache-jmeter-5.3\bin\system.properties
ERROR: java.lang.ExceptionInInitializerError
*** Problem's technical details go below ***
Home directory was detected as: E:\Outcode\ProtocolNow\apache-jmeter-5.3\lib
Exception in thread "main" java.lang.ExceptionInInitializerError
at com.thoughtworks.xstream.XStream.setupConverters(XStream.java:990)
at com.thoughtworks.xstream.XStream.<init>(XStream.java:593)
at com.thoughtworks.xstream.XStream.<init>(XStream.java:515)
at com.thoughtworks.xstream.XStream.<init>(XStream.java:484)
at com.thoughtworks.xstream.XStream.<init>(XStream.java:430)
at com.thoughtworks.xstream.XStream.<init>(XStream.java:383)
at org.apache.jmeter.save.SaveService$XStreamWrapper.<init>(SaveService.java:82)
at org.apache.jmeter.save.SaveService$XStreamWrapper.<init>(SaveService.java:80)
at org.apache.jmeter.save.SaveService.<clinit>(SaveService.java:113)
at org.apache.jmeter.save.CSVSaveService.processSamples(CSVSaveService.java:144)
at org.apache.jmeter.reporters.ResultCollector.loadExistingFile(ResultCollector.java:380)
at kg.apc.jmeter.PluginsCMDWorker.doJob(PluginsCMDWorker.java:138)
at kg.apc.cmdtools.ReporterTool.processParams(ReporterTool.java:255)
at kg.apc.cmdtools.PluginsCMD.processParams(PluginsCMD.java:62)
at kg.apc.cmdtools.PluginsCMD.processParams(PluginsCMD.java:21)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at kg.apc.cmd.UniversalRunner.main(UniversalRunner.java:115)
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field private final java.util.Comparator java.util.TreeMap.comparator accessible: module java.base does not "opens java.util" to unnamed module #75c072cb
at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
at com.thoughtworks.xstream.core.util.Fields.locate(Fields.java:40)
at com.thoughtworks.xstream.converters.collections.TreeMapConverter.<clinit>(TreeMapConverter.java:50)
... 20 more
The right place to seek for help with this regards is JMeter Plugins Support Forum
Coming back to your question:
In the JMeterPluginsCMD.bat change this line:
java -jar %~dp0\..\lib\cmdrunner-2.2.jar --tool Reporter %*
to this one:
java --add-opens=java.base/java.util=ALL-UNNAMED -jar %~dp0\..\lib\cmdrunner-2.2.jar --tool Reporter %*
Be aware that according to 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure article you should always be using the latest version of JMeter so consider upgrading to JMeter 5.4.1 (or whatever is the latest stable version which is available at JMeter Downloads page) on next available opportunity.
I'm trying to use the JMeterPluginsCMD on Windows to generate load test result graphs after a JMeter run. I installed the plugin manually because I can't use the JMeter Plugins Manager. (I don't have the network information I need to configure it). When I run a simple command .\JMeterPluginsCMD.bat, I get an error message. What can I do about this error message?
Here's the error message ERROR: java.lang.IllegalArgumentException: Missing plugin type specification
Here's the full error output:
~\Desktop\apache-jmeter-5.1.1\bin> .\JMeterPluginsCMD.bat
2019-08-12 12:10:29,564 WARN o.a.j.u.JMeterUtils: Exception 'null' occurred when fetching String property:'sampleresult.default.encoding', defaulting to: ISO-8859-1
2019-08-12 12:10:29,571 WARN o.a.j.u.JMeterUtils: Exception 'null' occurred when fetching String property:'jmeterPlugin.prefixPlugins'
2019-08-12 12:10:29,572 INFO k.a.j.PluginsCMDWorker: Using JMeterPluginsCMD v. N/A
2019-08-12 12:10:29,574 INFO o.a.j.u.JMeterUtils: Setting Locale to en_US
2019-08-12 12:10:29,578 INFO k.a.j.JMeterPluginsUtils: Loading user properties from: C:\Path\apache-jmeter-5.1.1\bin\user.properties
2019-08-12 12:10:29,582 INFO k.a.j.JMeterPluginsUtils: Loading system properties from: C:\Path\apache-jmeter-5.1.1\bin\system.properties
ERROR: java.lang.IllegalArgumentException: Missing plugin type specification
*** Problem's technical details go below ***
Home directory was detected as: C:\Path\apache-jmeter-5.1.1\lib
Exception in thread "main" java.lang.IllegalArgumentException: Missing plugin type specification
at kg.apc.jmeter.PluginsCMDWorker.checkParams(PluginsCMDWorker.java:78)
at kg.apc.jmeter.PluginsCMDWorker.doJob(PluginsCMDWorker.java:104)
at kg.apc.cmdtools.ReporterTool.processParams(ReporterTool.java:255)
at kg.apc.cmdtools.PluginsCMD.processParams(PluginsCMD.java:62)
at kg.apc.cmdtools.PluginsCMD.processParams(PluginsCMD.java:21)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at kg.apc.cmd.UniversalRunner.main(UniversalRunner.java:115)
~\Desktop\apache-jmeter-5.1.1\bin>
You are missing plugin-type parameter, example:
JMeterPluginsCMD.bat --generate-png test.png --input-jtl results.jtl --plugin-type ResponseTimesOverTime --width 800 --height 600
Your plugin jar should be available in lib/ext folder
Make note that you need to install corresponding plugins set to have Extras plugins available in CMD.
Relatively new to the pig/hadoop ecosystem and encountering a frustrating issue when trying to execute a simple DUMP. I am trying to call the below pig script (the file is local, not HFDS, so I am opening the pig shell using pig -x local).
REGISTER utils.py USING jython AS utils;
events = LOAD '../test/events.csv' USING PigStorage(',') AS (patientid:int, eventid:chararray, eventdesc:chararray, timestamp:chararray, value:float);
events = FOREACH events GENERATE patientid, eventid, ToDate(timestamp, 'yyyy-MM-dd') AS etimestamp, value;
DUMP events;
However, when doing this, I receive the following error messages (failed job summary below, full PIG stack trace at bottom):
Input(s): Failed to read data from "file:///bootcamp/test/events.csv"
Output(s): Failed to produce result in "file/tmp/temp/305054006/tmp-908064458"
Pig Stack Trace:
ERROR 1066: Unable to open iterator for alias events. Backend error : java.lang.IllegalStateException: Job in state DEFINE instead of RUNNING
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias events. Backend error : java.lang.IllegalStateException: Job in state DEFINE instead of RUNNING
at org.apache.pig.PigServer.openIterator(PigServer.java:925)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:746)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
at org.apache.pig.Main.run(Main.java:558)
at org.apache.pig.Main.main(Main.java:170)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: java.lang.IllegalStateException: Job in state DEFINE instead of RUNNING
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.getStats(MapReduceLauncher.java:822)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:452)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:280)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1390)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375)
at org.apache.pig.PigServer.storeEx(PigServer.java:1034)
at org.apache.pig.PigServer.store(PigServer.java:997)
at org.apache.pig.PigServer.openIterator(PigServer.java:910)
... 13 more
Caused by: java.lang.IllegalStateException: Job in state DEFINE instead of RUNNING
at org.apache.hadoop.mapreduce.Job.ensureState(Job.java:294)
at org.apache.hadoop.mapreduce.Job.getTaskReports(Job.java:540)
at org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.getTaskReports(HadoopShims.java:235)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.getStats(MapReduceLauncher.java:801)
...20 more
I have seen similar issues in regards to failed jobs, but sadly I haven't managed to hunt down a resolution as of yet.
EDIT: I should mention that when following the PIG tutorial at the below link, I was encountering the same issue.
http://www.sunlab.org/teaching/cse8803/fall2016/lab/hadoop-pig/
So, I found I was able to "DUMP" the file by doing the following:
tmp = events 100000; --any int larger than number of rows
dump tmp;
I had seen a similar issue on here, and was able to resolve by running as root.
I am very new to cassandra and using 1.2.10. I have a primary key column which is of timestamp datatype. Now I am trying retrieve the data for the date ranges. Since we know we can't use between in cassandra, I am using greater than(<) and less than(>) to get the date ranges. This perfectly seems to work in cassandra's cqlsh. But with pig_cassandra integration, here is the load function.
cql://<keyspace>/<columnfamily>?where_clause=time1%3E1357054841000590+and+time1%3C1357121822000430"
And here is the error it throws.
2013-12-18 04:32:51,196 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2997: Unable to recreate exception from backed error: java.lang.RuntimeException
at org.apache.cassandra.hadoop.cql3.CqlPagingRecordReader$RowIterator.executeQuery(CqlPagingRecordReader.java:651)
at org.apache.cassandra.hadoop.cql3.CqlPagingRecordReader$RowIterator.computeNext(CqlPagingRecordReader.java:352)
at org.apache.cassandra.hadoop.cql3.CqlPagingRecordReader$RowIterator.computeNext(CqlPagingRecordReader.java:275)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.apache.cassandra.hadoop.cql3.CqlPagingRecordReader.getProgress(CqlPagingRecordReader.java:181)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.getProgress(PigRecordReader.java:169)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.getProgress(MapTask.java:514)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:539)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: InvalidRequestException(why:Invalid restrictions found on time1)
at org.apache.cassandra.thrift.Cassandra$prepare_cql3_query_result.read(Cassandra.java:39567)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.apache.cassandra.thrift.Cassandra$Client.recv_prepare_cql3_query(Cassandra.java:1625)
at org.apache.cassandra.thrift.Cassandra$Client.prepare_cql3_query(Cassandra.java:1611)
at org.apache.cassandra.hadoop.cql3.CqlPagingRecordReader$RowIterator.prepareQuery(CqlPagingRecordReader.java:591)
at org.apache.cassandra.hadoop.cql3.CqlPagingRecordReader$RowIterator.executeQuery(CqlPagingRecordReader.java:621)
... 17 more
Any help is highly appreciated!!!
We seem to have fixed the problem by modifying the cassandra source, compiling and then adding the obtained jar file in the lib folder.