I try to write an spring-boot integration test using the docker-compose module of testcontainers. I get the following exception on
startup:
java.lang.ExceptionInInitializerError
Caused by: com.github.dockerjava.api.exception.InternalServerErrorException:
{"message":"Get https://quay.io/v2/: dial tcp: lookup quay.io on 192.168.65.1:53: no such host"}
I already tried to add our company http proxy using with Env but it doesn't work.
#RunWith(SpringRunner.class)
#SpringBootTest
public class FtpExportIntegrationTest {
#ClassRule
public static DockerComposeContainer environment =
new DockerComposeContainer(new File("src/test/resources/docker-compose.yml"))
.withExposedService("search-kafka", 9092)
.withEnv("HTTP_PROXY", "http://proxy.mycompany.com:8080")
.withEnv("HTTPS_PROXY", "http://proxy.mycompany.com:8080")
.withEnv("http_proxy", "http://proxy.mycompany.com:8080")
.withEnv("https_proxy", "http://proxy.mycompany.com:8080");
You need to set it in your Docker daemon settings, not container's.
Related
So I read that Spring Boot now supports PEM since 2.7.0
https://docs.spring.io/spring-boot/docs/2.7.0-SNAPSHOT/reference/htmlsingle/#howto.webserver.configure-ssl 17.3.7. Configure SSL
So I am using PEM generated by certbot.
My application.properties
spring.jpa.generate-ddl=true
spring1.jpa.hibernate.ddl-auto=update
spring.jpa.show-sql=false
spring.jpa.properties.hibernate.format_sql=false
server.port=443
server.ssl.certificate=fullchain1.pem
server.ssl.certificate.certificate-private-key=privkey1.pem
server.ssl.trust-certificate=fullchain1.pem
When I launch I get
org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Could not load key store 'null'
Caused by: org.springframework.boot.web.server.WebServerException: Could not load key store 'null'
Caused by: java.lang.IllegalArgumentException: Resource location must not be null
As per the documentation SSL configuration springboot
UPDATE:
Adding the content from link to directly in the answer, as link can
get updates
SSL can be configured declaratively by setting the various
server.ssl.* properties, typically in application.properties or
application.yml. The following example shows setting SSL properties
using a Java KeyStore file:
server.port=8443
server.ssl.key-store=classpath:keystore.jks
server.ssl.key-store-password=secret
server.ssl.key-password=another-secret
The following example shows setting SSL properties using PEM-encoded
certificate and private key files:
server.port=8443
server.ssl.certificate=classpath:my-cert.crt
server.ssl.certificate-private-key=classpath:my-cert.key
server.ssl.trust-certificate=classpath:ca-cert.crt
Your properties are not correctly declared,
server.ssl.certificate.certificate-private-key=privkey1.pem should be changed to server.ssl.certificate-private-key=privkey1.pem
So this workaround works
#Configuration
public class SSLConfig {
#Bean
public ConfigurableServletWebServerFactory webServerFactory() throws Exception {
TomcatServletWebServerFactory factory = new TomcatServletWebServerFactory();
Ssl ssl = new Ssl();
ssl.setEnabled(true);
ssl.setCertificate("cert1.pem");
ssl.setCertificatePrivateKey("privkey1.pem");
ssl.setKeyStoreType("PKCS12");
ssl.setKeyStorePassword(""); // without this decrytption fails
factory.setSsl(ssl);
factory.setPort(443);
return factory;
}
}
server.ssl.key-store=file:///Users/...
Have you tried this way of setting the path? First, make sure your application up with the correct path, then dig into the next step.
I get NoSuchBeanDefinitionException when trying to deploy my spring app in a docker to PCF. The docker image is one that I've built and runs just fine outside of PCF.
I can run the same docker image locally connecting to the same config server. The config server is the only configuration provided when starting the app either locally or in PCF.
The same application runs just fine in PCF also when just deployed as a jar rather than a docker image.
The class with the problem is:
#Component
#EnableRabbit
#Slf4j
class ListenerProblemListener extends Listener {
#Autowired
ListenerProblemListener(ErrorServiceConfiguration errorServiceConfiguration,
MessageQueueLibrary messageQueueLibrary,
CachingConnectionFactory cachingConnectionFactory,
ProblemService problemService,
Sender sender) {
super(cachingConnectionFactory, messageQueueLibrary, problemService, sender)
log.info("Connecting to queues ${errorServiceConfiguration.allProblemQueues}")
}
}
I get the following exception only when attempting this in PCF.
2019-10-14T11:29:49.795-05:00 [APP/PROC/WEB/0] [OUT] [WARN ] 2019-10-14 16:29:49.794 - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException:
Error creating bean with name 'listenerProblemListener' defined in URL [jar:file:/app.jar!/BOOT-INF/classes!/com/identifix/crawlererrorservice/listener/ListenerProblemListener.class]:
Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException:
No qualifying bean of type 'org.springframework.amqp.rabbit.connection.CachingConnectionFactory' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {}
Turns out when running in docker in PCF I needed to set the env var SPRING_RABBITMQ_HOST. I'm not sure why providing it in the config server wasn't enough yet.
Previously everything worked properly. Today I configured Spring Batch together with my Spring Boot application and faced an issue with application.properties.
I have following properties encrypted with Jasypt:
spring.profiles.active=https
ENVIRONMENT=h2
#aws sqs
aws.sqs.account.access.key=ENC(kjsdh456fgkjhdfsgkjhdfg)
#queue message listener
queue.message.listener.task.executor.threads.number=1
queue.message.listener.task.executor.max.concurrent.consumers=1
Now, in order to configure Spring Batch I added
ENVIRONMENT=h2
to application.properties file.
also, I have added batch-h2.properties file:
# Placeholders batch.* for H2 database:
batch.jdbc.driver=org.h2.Driver
batch.jdbc.url=jdbc:h2:~/testdb;CIPHER=AES;AUTO_SERVER=TRUE;DB_CLOSE_ON_EXIT=FALSE
batch.jdbc.user=sa
batch.jdbc.password="sa sa"
batch.jdbc.testWhileIdle=false
batch.jdbc.validationQuery=
batch.drop.script=classpath:/org/springframework/batch/core/schema-drop-h2.sql
batch.schema.script=classpath:/org/springframework/batch/core/schema-h2.sql
batch.business.schema.script=classpath:/business-schema-h2.sql
batch.database.incrementer.class=org.springframework.jdbc.support.incrementer.H2SequenceMaxValueIncrementer
batch.database.incrementer.parent=sequenceIncrementerParent
batch.lob.handler.class=org.springframework.jdbc.support.lob.DefaultLobHandler
batch.grid.size=2
batch.jdbc.pool.size=6
batch.verify.cursor.position=true
batch.isolationlevel=ISOLATION_SERIALIZABLE
batch.table.prefix=BATCH_
and after that I continuously receiving following exception:
Caused by: java.lang.IllegalArgumentException: Could not resolve placeholder 'aws.sqs.account.access.key' in string value "${aws.sqs.account.access.key}"
aws.sqs.account.access.key property now cannot be resolved.
I'm injecting this property into my configuration:
#Configuration
public class SQSConfig {
#Value("${aws.sqs.account.access.key}")
private String accessKey;
How to fix it ?
I know that spring automatically expose JMX beans. I was able to access it locally using VisualVM.
However on prod how I can connect to remotely to the app using it's JMX beans? Is there a default port or should I define anything in addition?
Thanks,
ray.
By default JMX is automatically accessible locally, so running jconsole locally would detect all your local java apps without port exposure.
To access an app via JMX remotely you have to specify an RMI Registry port. The thing to know is that when connecting, JMX initializes on that port and then establishes a data connection back on a random high port, which is a huge problem if you have a firewall in the middle. ("Hey sysadmins, just open up everything, mkay?").
To force JMX to connect back on the same port as you've established, you have a couple of the following options. Note: you can use different ports for JMX and RMI or you can use the same port.
Option 1: Command line
-Dcom.sun.management.jmxremote.port=$JMX_REGISTRY_PORT
-Dcom.sun.management.jmxremote.rmi.port=$RMI_SERVER_PORT
If you're using Spring Boot you can put this in your (appname).conf file that lives alongside your (appname).jar deployment.
Option 2: Tomcat/Tomee configuration
Configure a JmxRemoteLifecycleListener:
Maven Jar:
<dependency>
<groupId>org.apache.tomcat</groupId>
<artifactId>tomcat-catalina-jmx-remote</artifactId>
<version>8.5.9</version>
<type>jar</type>
</dependency>
Configure your server.xml:
<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener"
rmiRegistryPortPlatform="10001" rmiServerPortPlatform="10002" />
Option 3: configure programmatically
#Configuration
public class ConfigureRMI {
#Value("${jmx.rmi.host:localhost}")
private String rmiHost;
#Value("${jmx.rmi.port:1099}")
private Integer rmiPort;
#Bean
public RmiRegistryFactoryBean rmiRegistry() {
final RmiRegistryFactoryBean rmiRegistryFactoryBean = new RmiRegistryFactoryBean();
rmiRegistryFactoryBean.setPort(rmiPort);
rmiRegistryFactoryBean.setAlwaysCreate(true);
return rmiRegistryFactoryBean;
}
#Bean
#DependsOn("rmiRegistry")
public ConnectorServerFactoryBean connectorServerFactoryBean() throws Exception {
final ConnectorServerFactoryBean connectorServerFactoryBean = new ConnectorServerFactoryBean();
connectorServerFactoryBean.setObjectName("connector:name=rmi");
connectorServerFactoryBean.setServiceUrl(String.format("service:jmx:rmi://%s:%s/jndi/rmi://%s:%s/jmxrmi", rmiHost, rmiPort, rmiHost, rmiPort));
return connectorServerFactoryBean;
}
}
The trick, you'll see, is the serviceUrl in which you specify both the jmx:rmi host/port and the jndi:rmi host/port. If you specify both, you won't get the random high "problem".
Edit: For JMX remoting to work, you'll need to make a decision about authenticating. It's better to do it in 3 distinct steps: 1) basic setup with -Dcom.sun.management.jmxremote.authenticate=false then 2) add a password file (-Dcom.sun.management.jmxremote.password.file). See here for instructions. + -Dcom.sun.management.jmxremote.ssl=false and then 3) set up SSL.
Add the following JVM Properties in "$JAVA_OPTS" (in your application):
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=<PORT_NUMBER> -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=<HOST'S_IP>
In the Jconsole/Visual VM use the following to connect:
service:jmx:rmi:///jndi/rmi://<HOST'S_IP>:<PORT_NUMBER>/jmxrmi
It doesn't enable security, but will help you to connect to the remote server.
A tested approach on Java 1.8.0_71 and Spring Boot(1.3.3.RELEASE).
Append below parameters to JVM arguments for monitored JVM.
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12348 -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.rmi.port=12349 -Dcom.sun.management.jmxremote.password.file=/somewhere/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/somewhere/jmx/jmxremote.access
The com.sun.management.jmxremote.port is used to define the fixed RMI registry port, and the com.sun.management.jmxremote.rmi.port is used to instruct JVM to use fixed RMI port, but NOT use random one.
By setting this, I am able to connect JVM client from remote host to the monitored JVM via a firewall just opening 12348 and 12349 port.
I tested using java -jar cmdline-jmxclient-0.10.3.jar user:pwd hostip:12348 on a remote machine, which generates below output(shortened just for demonstration).
java.lang:type=Runtime
java.lang:name=PS Scavenge,type=GarbageCollector
Tomcat:J2EEApplication=none,J2EEServer=none,WebModule=//localhost/,j2eeType=Filter,name=requestContextFilter
java.nio:name=mapped,type=BufferPool
Tomcat:host=localhost,type=Host
java.lang:name=Compressed Class Space,type=MemoryPool
.......
The jar is downloaded from Here.
Another alternative
Reference for jmxremote.password and jmxremote.access files
import java.util.HashMap;
import java.util.Map;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.DependsOn;
import org.springframework.jmx.support.ConnectorServerFactoryBean;
import org.springframework.remoting.rmi.RmiRegistryFactoryBean;
#Configuration
public class ConfigureRMI {
#Value("${jmx.rmi.password.file:/tmp/jmxremote.password}")
private String passwordFile;
#Value("${jmx.rmi.access.file:/tmp/jmxremote.access}")
private String accessFile;
#Value("${jmx.rmi.port:19999}")
private Integer rmiPort;
#Bean
public RmiRegistryFactoryBean rmiRegistry() {
final RmiRegistryFactoryBean rmiRegistryFactoryBean = new RmiRegistryFactoryBean();
rmiRegistryFactoryBean.setPort(rmiPort);
rmiRegistryFactoryBean.setAlwaysCreate(true);
return rmiRegistryFactoryBean;
}
#Bean
#DependsOn("rmiRegistry")
public ConnectorServerFactoryBean connectorServerFactoryBean() throws Exception {
final ConnectorServerFactoryBean connectorServerFactoryBean = new ConnectorServerFactoryBean();
connectorServerFactoryBean.setObjectName("connector:name=rmi");
Map<String, Object> properties = new HashMap<>();
properties.put("jmx.remote.x.password.file", passwordFile);
properties.put("jmx.remote.x.access.file", accessFile);
connectorServerFactoryBean.setEnvironmentMap(properties);
connectorServerFactoryBean.setServiceUrl(String.format("service:jmx:rmi:///jndi/rmi://:%s/jmxrmi", rmiPort));
return connectorServerFactoryBean;
}
}
I am attempting to get spring cloud to work with messaging using auto configure.
My properties file contains:
cloud.aws.credentials.accessKey=xxxxxxxxxx
cloud.aws.credentials.secretKey=xxxxxxxxxx
cloud.aws.region.static=us-west-2
My Configuration class is as follows:
#EnableSqs
#ComponentScan
#EnableAutoConfiguration
public class Application {
public static void main(String[] args) throws Exception {
SpringApplication.run(Application.class, args);
}
}
My Listener class:
#RestController
public class OrderListener {
#MessageMapping("orderQueue")
public void orderListener(Order order){
System.out.println("Order Name " + order.getName());
System.out.println("Order Url" + order.getUrl());
}
}
However, when I run this. I get the following error:
org.springframework.context.ApplicationContextException: Failed to start bean 'simpleMessageListenerContainer'; nested exception is org.springframework.messaging.core.DestinationResolutionException: The specified queue does not exist for this wsdl version. (Service: AmazonSQS; Status Code: 400; Error Code: AWS.SimpleQueueService.NonExistentQueue; Request ID: cc8cb199-be88-5993-bd58-fca3c9f17110); nested exception is com.amazonaws.services.sqs.model.QueueDoesNotExistException: The specified queue does not exist for this wsdl version. (Service: AmazonSQS; Status Code: 400; Error Code: AWS.SimpleQueueService.NonExistentQueue; Request ID: cc8cb199-be88-5993-bd58-fca3c9f17110)
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:176)
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:51)
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:346)
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:149)
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:112)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:770)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:140)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:483)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:118)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:691)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:321)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:961)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:950)
at com.releasebot.processor.Application.main(Application.java:40)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
Caused by: org.springframework.messaging.core.DestinationResolutionException: The specified queue does not exist for this wsdl version. (Service: AmazonSQS; Status Code: 400; Error Code: AWS.SimpleQueueService.NonExistentQueue; Request ID: cc8cb199-be88-5993-bd58-fca3c9f17110); nested exception is com.amazonaws.services.sqs.model.QueueDoesNotExistException: The specified queue does not exist for this wsdl version. (Service: AmazonSQS; Status Code: 400; Error Code: AWS.SimpleQueueService.NonExistentQueue; Request ID: cc8cb199-be88-5993-bd58-fca3c9f17110)
at org.springframework.cloud.aws.messaging.support.destination.DynamicQueueUrlDestinationResolver.resolveDestination(DynamicQueueUrlDestinationResolver.java:81)
at org.springframework.cloud.aws.messaging.support.destination.DynamicQueueUrlDestinationResolver.resolveDestination(DynamicQueueUrlDestinationResolver.java:37)
at org.springframework.messaging.core.CachingDestinationResolverProxy.resolveDestination(CachingDestinationResolverProxy.java:88)
at org.springframework.cloud.aws.messaging.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:300)
at org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer.start(SimpleMessageListenerContainer.java:38)
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:173)
... 18 common frames omitted
Anyone else run across this? Any help would be greatly appreciated
This error means that the specified queue orderQueue does not exist on region us-west-2. Just create it and it should work.
Btw, there's no need to add _#EnableSqs_ when using _#EnableAutoConfiguration_.
Alain's answer is correct. This error indicates that the queue doesn't exist in the region us-west-2. One of the reasons could be that the AWS Java SDK uses us-east-1 as the default region. From AWS documentation http://docs.aws.amazon.com/java-sdk/latest/developer-guide/java-dg-region-selection.html
The AWS SDK for Java uses us-east-1 as the default region if you do not specify a region in your code. However, the AWS Management Console uses us-west-2 as its default. Therefore, when using the AWS Management Console in conjunction with your development, be sure to specify the same region in both your code and the console.
You can set the region or end point specifically in the client using setRegion() or setEndpoint() methods of AmazonSQSClientobject. See http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#setEndpoint-java.lang.String-
For a list of region and endpoints see http://docs.aws.amazon.com/general/latest/gr/rande.html#sqs_region
Try using URL of the queue, instead of the name.
I've impacted the same issue while trying to get-queue-url using command line. Look what I had:
A client error (AWS.SimpleQueueService.NonExistentQueue) occurred when calling the GetQueueUrl operation: The specified queue does not exist for this wsdl version.
Had to run this:
$aws configure
And under prompt 'Default region name[...]:' entered the region than my queue belongs to. Then the error disappeared.
So double check your configs ;)
Make sure you are using the correct AWS Profile. This was the problem in my case. If you have multiple profiles you can get all:
cat ~/.aws/credentials
and then you can set a different profile:
export AWS_PROFILE=<profile_name>
Without specifying this, the current profile by default is default, so make sure you take that into account.
I also had this error after re-creating a queue of the same name, creating a new queue was the immediate solution, since I wasn't sure how long it would take until the new ARN would attach.
If the queue name already exist and you still get this error make sure to run aws configure
and set the region to us-east-1.
In my case the queue name existed but it was still throwing up that error, because the region was set to us-east-2. I am not sure why though, but seems like changing the region to us-east-1 to the default region fixed it for me.