Cucumber Guice / Injector seems not to be thread-safe (Parallel execution / ExecutorService) - thread-safety

[long description warning]
I'm running some cucumber tests which have to be executed intercalated a defined server - for instance:
a.feature -> JBoss Server 1 | b.feature -> JBoss Serv. 2 | c.feature -> JB1 | etc.
For that, I created a hypothetical ExecutorService like this:
final ExecutorService executorService = Executors.newFixedThreadPool(2); //numberOfServers
for (Runnable task : tasks) {
executorService.execute(task);
}
executorService.shutdown();
try {
executorService.awaitTermination(1000, TimeUnit.SECONDS);
} catch (InterruptedException e) {
//doX();
}
The way that I manage about how will be the server chosen as liable to execute is:
inside of my Runnable class created for the executorService, I pass as a parameter a instanceId to a TestNG (XmlTest class) as below:
#Override
public void run() {
setupTest().run();
}
private TestNG setupTest() {
TestNG testNG = new TestNG();
XmlSuite xmlSuite = new XmlSuite();
XmlTest xmlTest = new XmlTest(xmlSuite);
xmlTest.setName(//irrelevant);
xmlTest.addParameter("instanceId", String.valueOf(instanceId));
xmlTest.setXmlClasses(..........);
testNG.setXmlSuites(..........);
return testNG;
}
Then, I get this just fine in a class that extends TestNgCucumberAdaptor:
#BeforeTest
#Parameters({"instanceId"})
public void setInstanceId(#Optional("") String instanceId) {
if (!StringUtils.isEmpty(instanceId)) {
this.instanceId = Integer.valueOf(instanceId);
}
}
And inside a #BeforeClass I'm populating a Pojo with this instanceId and setting the Pojo in a threadLocal attribute of another class. So far, so good.
public class CurrentPojoContext {
private static final ThreadLocal<PojoContext> TEST_CONTEXT = new ThreadLocal<PojoContext>();
...
public static PojoContext getContext(){
TEST_CONTEXT.get();
}
Now the problem really starts - I'm using Guice (Cucumber guice as well) in a 3rd class, injecting this pojo object that contains the instanceId. The example follows:
public class Environment {
protected final PojoContext pojoContext;
#Inject
public Environment() {
this.pojoContext = CurrentPojoContext.getContext();
}
public void foo() {
print(pojoContext.instanceId); // output: 1
Another.doSomething(pojoContext);
}
class Another{
public String doSomething(PojoContext p){
print(p.instanceId); // output: 2
}
}
}
Here it is not every time like this the outputs (1 and 2) but from time to time, I realized that the execution of different threads is messing with the attribute pojoContext. I know that is a little confusing, but my guess is that the Guice Injector is not thread-safe for this scenario - it might be a long shot, but I'd appreciate if someone else takes a guess.
Regards

Well, just in order to provide a solution for someone else, my solution was the following:
Create a class that maintains a Map with an identifier (unique and thread-safe one) as the key and a Guice Injector as value;
Inside my instantiation of Guice injector, I created my own module
Guice.createInjector(Stage.PRODUCTION, MyOwnModules.SCENARIO, new RandomModule());
and for this module:
public class MyOwnModules {
public static final Module SCENARIO = new ScenarioModule(MyOwnCucumberScopes.SCENARIO);
}
the scope defined here provides the following:
public class MyOwnCucumberScopes {
public static final ScenarioScope SCENARIO = new ParallelScenarioScope();
}
To sum up, the thread-safe will be in the ParallelScenarioScope:
public class ParallelScenarioScope implements ScenarioScope {
private static final Logger LOGGER = Logger.getLogger(ParallelScenarioScope.class);
private final ThreadLocal<Map<Key<?>, Object>> threadLocalMap = new ThreadLocal<Map<Key<?>, Object>>();
#Override
public <T> Provider<T> scope(final Key<T> key, final Provider<T> unscoped) {
return new Provider<T>() {
public T get() {
Map<Key<?>, Object> scopedObjects = getScopedObjectMap(key);
#SuppressWarnings("unchecked")
T current = (T) scopedObjects.get(key);
if (current == null && !scopedObjects.containsKey(key)) {
current = unscoped.get();
scopedObjects.put(key, current);
}
return current;
}
};
}
protected <T> Map<Key<?>, Object> getScopedObjectMap(Key<T> key) {
Map<Key<?>, Object> map = threadLocalMap.get();
if (map == null) {
throw new OutOfScopeException("Cannot access " + key + " outside of a scoping block");
}
return map;
}
#Override
public void enterScope() {
checkState(threadLocalMap.get() == null, "A scoping block is already in progress");
threadLocalMap.set(new ConcurrentHashMap<Key<?>, Object>());
}
#Override
public void exitScope() {
checkState(threadLocalMap.get() != null, "No scoping block in progress");
threadLocalMap.remove();
}
private void checkState(boolean expression, String errorMessage) {
if (!expression) {
LOGGER.info("M=checkState, Will throw exception: " + errorMessage);
throw new IllegalStateException(errorMessage);
}
}
}
Now the gotcha is just to be careful regarding the #ScenarioScoped and the code will work as expected.

Related

How to start multiple boot apps for end-to-end tests?

I'd like to write end-to-end tests to validate two boot apps work well together with various profiles.
What already works:
create a third maven module (e2e) for end-to-end tests, in addition to the two tested apps (authorization-server and resource-server)
write tests using TestResTemplate
Test work fine if I start authorization-server and resource-server manually.
What I now want to do is automate the tested boot apps startup and shutdown with the right profiles for each test.
I tried:
adding maven dependencies to tested apps in e2e module
using SpringApplication in new threads for each app to start
But I face miss-configuration issues as all resources and dependencies end in the same shared classpath...
Is there a way to sort this out?
I'm also considering starting two separate java -jar ... processes, but then, how to ensure tested apps fat-jars are built before 2e2 unit-tests run?
Current app start/shutdown code sample which fails as soon as I had maven dependency to second app to start:
private Service startAuthorizationServer(boolean isJwtActive) throws InterruptedException {
return new Service(
AuthorizationServer.class,
isJwtActive ? new String[]{ "jwt" } : new String[]{} );
}
private static final class Service {
private ConfigurableApplicationContext context;
private final Thread thread;
public Service(Class<?> appClass, String... profiles) throws InterruptedException {
thread = new Thread(() -> {
SpringApplication app = new SpringApplicationBuilder(appClass).profiles(profiles).build();
context = app.run();
});
thread.setDaemon(false);
thread.start();
while (context == null || !context.isRunning()) {
Thread.sleep(1000);
};
}
#PreDestroy
public void stop() {
if (context != null) {
SpringApplication.exit(context);
}
if (thread != null) {
thread.interrupt();
}
}
}
I think your case, running the two applications via a docker compose can be a good idea.
This article shows how you can set up some integration tests using a docker compose image: https://blog.codecentric.de/en/2017/03/writing-integration-tests-docker-compose-junit/
Also, take a look at this post from Martin Fowler: https://martinfowler.com/articles/microservice-testing/
I got things working with second solution:
end-to-end tests projects has no other maven dependency than what is required to run spring-tests with TestRestClient
test config initialises environment, running mvn packageon required modules in separate processes
test cases run (re)start apps with chosen profiles in separate java -jar ... processes
Here is the helper class I wrote for this (taken from there):
class ActuatorApp {
private final int port;
private final String actuatorEndpoint;
private final File jarFile;
private final TestRestTemplate actuatorClient;
private Process process;
private ActuatorApp(File jarFile, int port, TestRestTemplate actuatorClient) {
this.port = port;
this.actuatorEndpoint = getBaseUri() + "actuator/";
this.actuatorClient = actuatorClient;
this.jarFile = jarFile;
Assert.isTrue(jarFile.exists(), jarFile.getAbsolutePath() + " does not exist");
}
public void start(List<String> profiles, List<String> additionalArgs) throws InterruptedException, IOException {
if (isUp()) {
stop();
}
this.process = Runtime.getRuntime().exec(appStartCmd(jarFile, profiles, additionalArgs));
Executors.newSingleThreadExecutor().submit(new ProcessStdOutPrinter(process));
for (int i = 0; i < 10 && !isUp(); ++i) {
Thread.sleep(5000);
}
}
public void start(String... profiles) throws InterruptedException, IOException {
this.start(Arrays.asList(profiles), List.of());
}
public void stop() throws InterruptedException {
if (isUp()) {
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON_UTF8);
headers.setAccept(List.of(MediaType.APPLICATION_JSON_UTF8));
actuatorClient.postForEntity(actuatorEndpoint + "shutdown", new HttpEntity<>(headers), Object.class);
Thread.sleep(5000);
}
if (process != null) {
process.destroy();
}
}
private String[] appStartCmd(File jarFile, List<String> profiles, List<String> additionalArgs) {
final List<String> cmd = new ArrayList<>(
List.of(
"java",
"-jar",
jarFile.getAbsolutePath(),
"--server.port=" + port,
"--management.endpoint.heath.enabled=true",
"--management.endpoint.shutdown.enabled=true",
"--management.endpoints.web.exposure.include=*",
"--management.endpoints.web.base-path=/actuator"));
if (profiles.size() > 0) {
cmd.add("--spring.profiles.active=" + profiles.stream().collect(Collectors.joining(",")));
}
if (additionalArgs != null) {
cmd.addAll(additionalArgs);
}
return cmd.toArray(new String[0]);
}
private boolean isUp() {
try {
final ResponseEntity<HealthResponse> response =
actuatorClient.getForEntity(actuatorEndpoint + "health", HealthResponse.class);
return response.getStatusCode().is2xxSuccessful() && response.getBody().getStatus().equals("UP");
} catch (ResourceAccessException e) {
return false;
}
}
public static Builder builder(String moduleName, String moduleVersion) {
return new Builder(moduleName, moduleVersion);
}
/**
* Configure and build a spring-boot app
*
* #author Ch4mp
*
*/
public static class Builder {
private String moduleParentDirectory = "..";
private final String moduleName;
private final String moduleVersion;
private int port = SocketUtils.findAvailableTcpPort(8080);
private String actuatorClientId = "actuator";
private String actuatorClientSecret = "secret";
public Builder(String moduleName, String moduleVersion) {
this.moduleName = moduleName;
this.moduleVersion = moduleVersion;
}
public Builder moduleParentDirectory(String moduleParentDirectory) {
this.moduleParentDirectory = moduleParentDirectory;
return this;
}
public Builder port(int port) {
this.port = port;
return this;
}
public Builder actuatorClientId(String actuatorClientId) {
this.actuatorClientId = actuatorClientId;
return this;
}
public Builder actuatorClientSecret(String actuatorClientSecret) {
this.actuatorClientSecret = actuatorClientSecret;
return this;
}
/**
* Ensures the app module is found and packaged
* #return app ready to be started
* #throws IOException if module packaging throws one
* #throws InterruptedException if module packaging throws one
*/
public ActuatorApp build() throws IOException, InterruptedException {
final File moduleDir = new File(moduleParentDirectory, moduleName);
packageModule(moduleDir);
final File jarFile = new File(new File(moduleDir, "target"), moduleName + "-" + moduleVersion + ".jar");
return new ActuatorApp(jarFile, port, new TestRestTemplate(actuatorClientId, actuatorClientSecret));
}
private void packageModule(File moduleDir) throws IOException, InterruptedException {
Assert.isTrue(moduleDir.exists(), "could not find module. " + moduleDir + " does not exist.");
String[] cmd = new File(moduleDir, "pom.xml").exists() ?
new String[] { "mvn", "-DskipTests=true", "package" } :
new String[] { "./gradlew", "bootJar" };
Process mvnProcess = new ProcessBuilder().directory(moduleDir).command(cmd).start();
Executors.newSingleThreadExecutor().submit(new ProcessStdOutPrinter(mvnProcess));
Assert.isTrue(mvnProcess.waitFor() == 0, "module packaging exited with error status.");
}
}
private static class ProcessStdOutPrinter implements Runnable {
private InputStream inputStream;
public ProcessStdOutPrinter(Process process) {
this.inputStream = process.getInputStream();
}
#Override
public void run() {
new BufferedReader(new InputStreamReader(inputStream)).lines().forEach(System.out::println);
}
}
public String getBaseUri() {
return "https://localhost:" + port;
}
}

ServiceTrackerCustomizer service reference returns null

I have a persistence bundle containing a Product entity. Since I want to trigger updates on my Solr server on demand, when I'm altering an instance, I introduced an entity listener:
#Entity
public class Product {
// id and fields
#PrePersist
public void onPersist () {
ProductAudit.onPersist( this );
}
// other lifecycle callbacks
}
public class ProductAudit {
private static final Set<ProductListener> listeners = new HashSet<>();
// static addListener/removeListener
public void onPersist (Product p) {
listeners.forEach(l -> {
l.onPersist(p);
}
}
}
To integrate my persistence unit into my OSGi environment, I wrote an adapter bundle leveraging a service tracker:
#Component(
name="com.acme.product.audit",
immediate=true)
public class OsgiProductListener implements ProductListener {
private ServiceTracker<ProductListener,ProductListener> tracker;
#Activate
public void activate (BundleContext context) {
tracker = new ServiceTracker<>(context, ProductListener.class, new ServiceTrackerCustomizer<ProductListener, ProductListener>() {
#Override
public ProductListener addingService(ServiceReference<ProductListener> reference) {
if (validProps(reference)) {
ProductListener l = reference.getBundle().getBundleContext().getService(reference); // null
ProductListener l = context.getService(reference); // null too
System.out.println("Adding " + l);
ProductAudit.addListener(l);
return l;
}
return null;
}
// modifiedService, removedService
private boolean validProps(ServiceReference<?> reference) {
// check for enabling flag
}
});
tracker.open();
}
Now I have two provider bundles - a test bundle and an indexer bundle. When I deploy both, on console is printed:
> Activating com.acme.persistence.audit component
...
> Adding com.acme.persistence.test.ProductListenerImpl#1337
> Adding null
All three bundles use the same API version, else my tracker wouldnt track them.
So why I am getting null, if I try to obtain the ProductListener service?

Singleton, cannot go in the runnable task

I'm trying to run a task automatically (all the 30s).
For that, I built a singleton :
public class PortalSingleton {
private static final Logger LOG = LoggerFactory.getLogger(PortalSingleton.class);
private static final int INITIAL_DELAY = 0;
private static final int DELAY = 30;
private static volatile ScheduledExecutorService instance;
private static HomepageView homeView = new HomepageView();
private PortalSingleton() {}
public static final void refreshGridHomePageAutomatically() {
Runnable task = () -> UI.getCurrent().access(() -> {
homeView.refreshGrid();
LOG.info("The grid has been refreshed Automatically");
});
getInstance().scheduleWithFixedDelay(task, INITIAL_DELAY, DELAY, TimeUnit.SECONDS);
}
public final static ScheduledExecutorService getInstance() {
if (instance == null) {
synchronized (ScheduledExecutorService.class) {
if (instance == null) {
instance = Executors.newScheduledThreadPool(1);
}
}
}
return instance;
}
}
But, I didn't have any issue/error AND I didn't have my log msg and my grid hasn't been refreshed..
The behavior expected is :
my grid refresh
see the log msg
Even if I delete the line homeView.refreshGrid();, I don't have my log msg...
What did I do wrong?
Thanks,
EDIT : I call it by doing : PortalSingleton.refreshGridHomePageAutomatically();
EDIT2, thanks #Holger :
public class PortalSingleton {
private static final Logger LOG = LoggerFactory.getLogger(PortalSingleton.class);
private static final int INITIAL_DELAY = 0;
private static final int DELAY = 30;
private static final ScheduledExecutorService instance = Executors.newScheduledThreadPool(1);
private static HomepageView homeView = new HomepageView();
private PortalSingleton() {
}
public static final void refreshGridHomePageAutomatically() {
Runnable task = () -> UI.getCurrent().access(() -> {
homeView.refreshGrid();
LOG.info("The grid has been refreshed Automatically");
});
try {
getInstance().scheduleWithFixedDelay(task, INITIAL_DELAY, DELAY, TimeUnit.SECONDS);
} catch (Exception e) {
LOG.error("error" + e);
}
}
public final static ScheduledExecutorService getInstance() {
return instance;
}
}
When you schedule an action, you do not get a feedback when an exception occurs. Instead, it will just stop executing it:
ScheduledExecutorService.scheduleWithFixedDelay(…):
…If any execution of the task encounters an exception, subsequent executions are suppressed.
Therefore, you will have to use a try … catch block in the action itself to report it, e.g. in the lambda expression defining your Runnable:
Runnable task = () -> {
try { UI.getCurrent().access(…); }
catch (Exception e) { LOG.error("error" + e); }
};
It looks suspicious to me that you are calling UI.getCurrent() from a non-UI thread which I suspect to return null causing a NullPointerException when trying to invoke a method on it.

Customize SLF4J Logger

I'm trying to find a nice way to add a prefix to my logs without passing it on every calls, without instanciate Logger again.
The purpose is to trace Rest calls individually.
(The prefix would be re-generated on each call using UUID)
This would be like
#RestController
class MyClass {
//Here the prefix is initialise once
//default value is X
Logger LOG = LoggerFactory.getLogger(MyClass.class);
#RequestMapping("/a")
void methodA() {
LOG.debug("foo");
}
#RequestMapping("/b")
void methodB() {
LOG.setPrefix("B");
LOG.debug("bar");
}
with this output
[...] [prefix X] foo
[...] [prefix B] bar
As you've said you're using Logback, here's a couple options to do the kind of thing you're trying to do:
Markers
Each log entry can have a "marker" established for it. (The best documentation I've seen for it is in the SLF4J FAQ.) Something like:
class MyClass {
Marker methodBMarker = MarkerFactory.getMarker("B");
Logger logger = LoggerFactory.getLogger(MyClass.class);
…
void methodB() {
logger.debug(methodBMarker, "bar");
}
}
You would need to update all log entries in each method to use the appropriate marker. You can then put %marker in your layout to put the log entry's marker into the log.
MDC
The other option is to use the "Mapped Diagnostic Context" functionality to specify the current "context" for each log entry.
class MyClass {
Logger logger = LoggerFactory.getLogger(MyClass.class);
…
void methodB() {
MDC.put("method", "b");
try {
…
logger.debug("bar");
…
} finally {
MDC.clear();
}
}
}
You would then use %mdc{method} in your layout to output that particular MDC value. Note that MDC is really intended to be used for per-thread values like something web-connection-specific, so it's important to ensure that it's cleared out of what you don't want when you're leaving the context you want the value logged in.
Please see http://www.slf4j.org/extensions.html#event_logger for an example of how to use the MDC. You do not have to use the EventLogger. Once you set things in the MDC they are present in every log record.
A Marker does not meet your criteria since it has to be specified on every call.
Here's my MDC implementation explained to share my experiments with MDC.
//In this abstract class i'm defining initLogData methods to set MDC context
//It would be inherited by Controller and other classes who needs logging with traced transactions
public abstract class AbstractService {
protected LogData initLogData() {
return LogData.init();
}
protected LogData initLogData(String tName) {
return LogData.init(tName);
}
}
//LogData holds the MDC logic
public class LogData {
private final static int nRandom = 8;
//this keys are defined in logback pattern (see below)
private final static String tIdKey = "TID";
private final static String tNameKey = "TNAME";
//Transaction id
private String tId;
//Transaction name
private String tName;
public String getTId() {
return tId;
}
public void setTId(String tId) {
this.tId = tId;
}
public String gettName() {
return tName;
}
public void settName(String tName) {
this.tName = tName;
}
//random transaction id
//I'm not using uuid since its too longs and perfect unicity is not critical here
public String createTId(){
Random r = new Random();
StringBuilder sb = new StringBuilder();
while(sb.length() < nRandom){
sb.append(Integer.toHexString(r.nextInt()));
}
return sb.toString().substring(0, nRandom);
}
//private constructors (use init() methods to set LogData)
private LogData(String tId, String tName) {
this.tId = tId;
this.tName = tName;
}
private LogData(String tName) {
this.tId = createTId();
this.tName = tName;
}
private LogData() {
this.tId = createTId();
}
//init MDC with cascading calls processing (using same id/name within same context
//even if init() is called again)
public static LogData init(String tName) {
String previousTId = MDC.get(tIdKey);
String previousTName = MDC.get(tNameKey);
MDC.clear();
LogData logData = null;
if(previousTId != null) {
logData = new LogData(previousTId, previousTName);
} else {
logData = new LogData(tName);
}
MDC.put(tIdKey, logData.getTId());
MDC.put(tNameKey, logData.gettName());
return logData;
}
//init MDC without cascading calls management (new keys are generated for each init() call)
public static LogData init() {
MDC.clear();
LogData logData = new LogData();
MDC.put(tIdKey, logData.getTId());
return logData;
}
}
//logback.xml : values to include in log pattern
[%X{TID}] [%X{TNAME}]
#RestController
#RequestMapping("/test")
public class RestControllerTest extends AbstractRestService {
private final Logger LOG = LoggerFactory.getLogger(ServiceRestEntrypointStatus.class);
#RequestMapping(value="/testA")
public void testA() {
initLogData("testA");
LOG.debug("This is A");
}
#RequestMapping(value="/testB")
public void testB() {
initLogData("testA");
LOG.debug("This is B");
}
#RequestMapping(value="/testC")
public void testC() {
initLogData("testC");
LOG.debug("This is C");
testA();
testB();
}
}
Calling RestControllerTest mapped /test/testA produces :
[fdb5d310] [testA] This is A
Calling /test/testC produces (id and name are kept even if initLogData is called in sub methods):
[c7b0af53] [testC] This is C
[c7b0af53] [testC] This is A
[c7b0af53] [testC] This is B

Spring Mvc with Thread

Hi My thread class is showing null pointer exception please help me to resolve
#Component
public class AlertsToProfile extends Thread {
public final Map<Integer, List<String>> userMessages = Collections.synchronizedMap(new HashMap<Integer, List<String>>());
#Autowired
ProfileDAO profileDAO;
private String categoryType;
private String dataMessage;
public String getCategoryType() {
return categoryType;
}
public void setCategoryType(String categoryType) {
this.categoryType = categoryType;
}
public String getDataMessage() {
return dataMessage;
}
public void setDataMessage(String dataMessage) {
this.dataMessage = dataMessage;
}
public void run() {
String category=getCategoryType();
String data= getDataMessage();
List<Profile> all = profileDAO.findAll();
if (all != null) {
if (category == "All" || category.equalsIgnoreCase("All")) {
for (Profile profile : all) {
List<String> list = userMessages.get(profile.getId());
if (list == null ) {
ArrayList<String> strings = new ArrayList<String>();
strings.add(data);
userMessages.put(profile.getId(), strings);
} else {
list.add(data);
}
}
}
}
}
and my service method is as follows
#Service
public class NoteManager
{
#Autowired AlertsToProfile alertsToProfile;
public void addNote(String type, String message, String category) {
alertsToProfile.setCategoryType(category);
String data = type + "," + message;
alertsToProfile.setDataMessage(data);
alertsToProfile.start();
System.out.println("addNotes is done");
}
But when i call start() method am getting null pointer exception please help me. I am new to spring with thread concept
It pretty obvious: you instantiate your thread directly, as opposed to letting spring create AlertsToProfile and auto wire your instance.
To fix this, create a Runnable around your run() method and embed that into a method, something like this:
public void startThread() {
new Thread(new Runnable() {
#Override
public void run() {
// your code in here
}}).start();
}
you will want to bind the Thread instance to a field in AlertsToProfile in order to avoid leaks and stop the thread when you're done.

Resources