Have created a unit test using EvoSuite and want to use maven build to do code coverage analysis on this unit test. When I run maven build I get ...
JMockit Coverage got reloaded through custom class loader org.evosuite.runtime.instrumentation.EvoClassLoader;
and
Class org.xyz.MyClass$1 was not instrumented by EvoSuite. This could happen if you are running JUnit tests in a way that is not handled by EvoSuite, in which some classes are loaded be reflection before the tests are run. Consult the EvoSuite documentation for possible workarounds for this issue.
So seems like code that is breaking is...
if (!InstrumentedClass.class.isAssignableFrom(clazz)) {
String msg = "Class " + clazz.getName() + " was not instrumented by EvoSuite. " +
"This could happen if you are running JUnit tests in a way that is not handled by EvoSuite, in " +
"which some classes are loaded be reflection before the tests are run. Consult the EvoSuite documentation " +
"for possible workarounds for this issue.";
logger.error(msg);
problem = true;
//throw new IllegalStateException(msg); // throwing an exception might be a bit too extreme }
Is it possible to run the EvoSuite unit test through JMockit? Any suggestions?
before reading this question, I had never heard of JMockit, so it is not something that was under the radar in EvoSuite... :(
anyway, I just added now some documentation about it at:
http://www.evosuite.org/documentation/measuring-code-coverage/
you can read there why you get 0% coverage, and possible workarounds for it.
Related
I posted this: Maven JUnit 5: Run all unit tests by default or run only a specific test suite from the command line .
I'm simply posting it again with almost exactly the same title because nothing about my base problem has changed, but I will include what I tried to do with Tags, which are not working for me.
We are upgrading our services to a relatively recent SpringBoot version, along with JUnit 5. We normally run our builds with just "mvn package", which runs all of our unit tests by default.
We also have "component tests", which are run from a separate command line, and which are all specified in a single test suite with a specific name. Before JUnit 5, we would run this with something like:
mvn -Dtest=ComponentTestSuite test
This was working fine with JUnit 4.
The class looks like this:
#Suite
#SelectClasses(<testclassname>.class)
public class ComponentTestSuite {
}
With JUnit 5, this ends up saying "No tests found".
We are using v2.3.12 of Spring-Boot, which by default includes a somewhat older version of JUnit 5. I am overriding those defaults and including v1.8.2 of the junit-platform components and v5.8.2 of the jupiter components.
For the service that I'm testing this with, the test suite only has a single component test (which is unusual). I WAS able to simply replace "ComponentTestSuite" with the name of the component test class, and that would run that single component test.
I noticed this thread: Junit5 test suites with #Suite annotation doesn't execute tests with mvn test command .
Unfortunately, what I found was that changing "-Dtest" to "-Dinclude" simply ran all my default unit tests and ignored the test suite.
So, I was told that I'm "not supposed to do it this way", although it's not clear to me exactly which parts I'm not supposed to do, but I am supposed to use "Tags". Ok, so I tried to use tags.
I also noticed this article: https://www.baeldung.com/junit-filtering-tests , although that article appears to use some deprecated mechanisms, like "#RunWith(JUnitPlatform.class)", so I'm not sure how much I can believe from that article.
I've been trying several variations, and my last attempt is this:
#Suite
#Tag("ComponentTest")
#IncludeTags("ComponentTest")
public class ComponentTestSuite {
}
And I added #Tag("ComponentTest") to my one component test class.
This still doesn't work. I try to run mvn -Dtest=ComponentTestSuite test, and it says "No tests found". If I instead try mvn -Dtests=ComponentTest test that just runs all of my unit tests and ignores my component test suite. If I instead run mvn test -Dgroups=ComponentTest, that gives me another variation of "No tests found" in that it doesn't print that, it just literally executes no tests without giving an error.
Looking at that suite class now, it does seem nonsensical, as it doesn't really "contain" anything anymore, which is unfortunate. It was good to have that list of component test classes in one place. Now, it seems like the suite class is pointless, but I can't get anything to work anyway.
Note that I haven't yet added a "groups" element to my surefire config, and I haven't edited all of my unit tests to add a #Tag("UnitTest") annotation. I've only edited the suite class and the one component test.
Update:
To address a comment, I am specifying the following dependencies (among others):
Version 1.8.2 of
junit-platform-suite
junit-platform-suite-api
junit-platform-launcher
junit-platform-commons
junit-platform-engine
junit-platform-runner
junit-platform-suite-engine
Version 5.8.2 of
junit-jupiter
junit-bom (probably unnecessary)
The entire suite class is this:
import org.junit.jupiter.api.Tag;
import org.junit.platform.suite.api.IncludeTags;
import org.junit.platform.suite.api.SelectClasses;
import org.junit.platform.suite.api.Suite;
#Suite
#Tag("ComponentTest")
#IncludeTags("ComponentTest")
public class ComponentTestSuite {
}
I have a project, with tests implemented by junit 4 and 5. Some of the tests require a database to be present, is there any way I can mark these specific tests (is TestSuite the answer?), so gradle continue the build even if those tests fail?
I don't want to skip other tests, but just these specific tests.
And I am using junit 5 vintage, so my test task runs both junit 4 and junit 5 tests together.
Honestly, I wouldn't do this. What is the point of a test if you ignore it when it fails? You might as well delete the test right away. Can't you delay test execution if setup is slow?
That said...
gradle --continue will continue execution when a task fails, collecting all the errors. You may still need to ignore Gradle's exit code (e.g., in a build pipeline).
You can also use the ignoreFailures test property to always ignore failing tests.
Both may be a bit too broad. Depending on your build script, you could add have a separate test target for database tests and add ignoreFailures only to that.
If you want to handle this in JUnit, you could look into the Assume class which lets you skip a test if a certain condition is (or is not) given:
import static org.junit.Assume.*;
public class SomeDatabaseTest {
#Test
public void someThing() {
assumeTrue(Database.isAvailable());
// actual test goes here
}
}
Add this block of code to your module or project level build.gradle file:
project.gradle.teskGraph.whenReady {
connectedDebugAndroidTest {
ignoreFailures = true
}
}
You can use any of the test tasks instead of connectedDebugAndroidTest. You can also specify #Test for only those tests annotated as such. See Android documentation here for example of tests.
My project has two different cucumber test and each one needs a different spring context configuration.
The problem I have is that when I run each test individually from Intellij, they load the right Spring Context and the tests are passing, but I press run all test, none of them are passing.
Running a maven test, both test are passing.
this is my code:
#RunWith(FCSCucumber.class)
#Cucumber.Options(strict = true
// , tags = {"~#ignore"}
// , tags = {"#Only"}
, glue ="feature.scenario1"
, features = "src/test/resources/feature/scenario1/"
)
#FCSApplicationProperties(commonProps="config/scenario1/exec.common.properties",
environmentProps="src/test/resources/scenario1-test.properties")
public class TesScenario1Features {
}
#ContextConfiguration("/cucumber-scenario1.xml")
public class scenario1Steps {
......
}
#RunWith(FCSCucumber.class)
#Cucumber.Options(strict = true
// , tags = {"~#ignore"}
// , tags = {"#Only"}
, glue ="feature.scenario2"
, features = "src/test/resources/feature/scenario2/"
)
#FCSApplicationProperties(commonProps="config/scenario2/exec.common.properties",
environmentProps="src/test/resources/scenario2-test.properties")
public class TesScenario2Features {
}
#ContextConfiguration("/cucumber-scenario2.xml")
public class scenario2Steps {
......
}
Thank you very much for your help
The issue is that the IntelliJ cucumber plugin is using the cucumber cli to run tests, without using the JUnit runner at all. This causes several limitations, like requiring the spring annotations on the step definition classes instead of the runner, or by default requiring the steps definitions to be in the same package as the scenario files.
In your example I would actually expect also running a single test to fail, unless the correct application properties are also referenced by the /cucumber-scenario{1,2}.xml files.
The only option I see with the standard cucumber implementation would be to extract the tests into separate projects.
I'm actually working on an alternative implementation of cucumber with an improved spring integration that you might want to try. It's not fully integrated with IntelliJ yet though.
I've tried just about every configuration I can think of (and reviewed some answers on StackOverflow), but all of our tests show the 'Failed to load ApplicationContext' error when run through Hudson. What is interesting is that some tests appear to run and pass, while some run and fail (as expected), but regardless I'm always getting the errors list for all tests. Here is the basic configuration:
#ContextConfiguration(locations = "classpath:/MyTest-context.xml")
#RunWith(SpringJUnit4ClassRunner.class)
public class MyTest {
#Autowired
private ApplicationContext applicationContext;
public MyTest() {}
#Test
public void doSomething() {
// Implementation...
}
}
UPDATE:
There appears to be a duplicate set of tests running, one for Emma coverage reporting, and the other the normal tests. It is when the tests run for Emma coverage that they are showing the errors. If I turn off the "emma:emma package" goal so those don't run then I don't get the errors, and the tests appear to run fine. I'm not sure if that helps any.
The answer ended up being close to what gontard was pointing to, which is an issue that was hidden by the way Emma's classloader works. Between my local JUnit tests, what was running in our DEV environment, and what was running in Hudson with Emma, all of them have a different way in which the classloader orders the loading of libraries and classes. I ended up reviewing the stack trace on the test results, and it turns out on my local, a new version of a library was loaded via the POM, but in Hudson Emma was loading an old version of a library first. I had to find and remove the old version, and everything now works fine.
I am using Emma for code-coverage for my project. In my JUnit test case, I am calling a method but in code coverage report, it shows no coverage for that method which is getting called from testCase.
Check these points:
Set a breakpoint in the method to make sure it is really called.
You can tell emma to omit some classes. Check the config.
Is the class properly instrumented?