I am trying to run a simple unit test using #EmbeddedKafka Annotation.
As a reference, I am following the below spring documentation
https://docs.spring.io/spring-kafka/reference/html/#embedded-kafka-annotation
#RunWith(SpringRunner.class)
#DirtiesContext
#EmbeddedKafka(brokerProperties = "log.dir=/kafka-logs", partitions = 1,
topics = {
"dare_policy_created"})
#Slf4j
public class ConsumerTest {
#Autowired
private EmbeddedKafkaBroker embeddedKafka;
#Test
public void someTest() {
Map<String, Object> consumerProps = KafkaTestUtils.consumerProps("testGroup", "true", this.embeddedKafka);
consumerProps.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
ConsumerFactory<Integer, String> cf = new DefaultKafkaConsumerFactory<>(consumerProps);
Consumer<Integer, String> consumer = cf.createConsumer();
this.embeddedKafka.consumeFromAnEmbeddedTopic(consumer, "dare_policy_created");
ConsumerRecords<Integer, String> replies = KafkaTestUtils.getRecords(consumer);
//assertThat(replies.count()).isGreaterThanOrEqualTo(1);
}
}
I was trying to define the log.dir #EmbeddedKafka(brokerProperties = "log.dir= ") because I was getting an error when running the Test.
I tried :
log.dir=/kafka-logs
log.dir=real_path_to_my_project/kafka-logs
...
But every time I run the test I get this error :
kafka.server.LogDirFailureChannel.error - Failed to create or validate data directory /kafka-logs java.io.IOException: Failed to load /kafka-logs during broker startup
kafka.log.LogManager.fatal - Shutdown broker because none of the specified log dirs from /kafka-logs can be created or validated
I was able to resolve the issue by removing an explicit dependency on kafka-client.
I had below dependency in my pom
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>2.3.0</version>
</dependency>
Related
I want to upload a multipart file to AWS S3. So, i have to convert it.
But new File method needs a local location to get the file.
I am able to do in local. But running this code in every machine seems like a issue.
Please find both scenarios.
Working
private File convertMultiPartToFile(MultipartFile multipartFile) throws IOException {
File convFile = new File("C:\\Users\\" + multipartFile.getOriginalFilename());
multipartFile.transferTo(convFile);
return convFile;
}
Not working
private File convertMultiPartToFile(MultipartFile multipartFile) throws IOException {
File convFile = new File(multipartFile.getOriginalFilename());
multipartFile.transferTo(convFile);
return convFile;
}
Error received :
java.io.FileNotFoundException: newbusiness.jpg (Access is denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
You could use Spring Content S3. This will hide the implementation details so you don't need to worry about them.
There are Spring Boot starter alternatives but as you are not using Spring Boot add the following dependency to your pom.xml
pom.xml
<dependency>
<groupId>com.github.paulcwarren</groupId>
<artifactId>spring-content-s3</artifactId>
<version>0.0.11</version>
</dependency>
Add the following configuration that creates a SimpleStorageResourceLoader bean:
#Configuration
#EnableS3Stores
public class S3Config {
#Autowired
private Environment env;
public Region region() {
return Region.getRegion(Regions.fromName(env.getProperty("AWS_REGION")));
}
#Bean
public BasicAWSCredentials basicAWSCredentials() {
return new BasicAWSCredentials(env.getProperty("AWS_ACCESS_KEY_ID"), env.getProperty("AWS_SECRET_KEY"));
}
#Bean
public AmazonS3 client(AWSCredentials awsCredentials) {
AmazonS3Client amazonS3Client = new AmazonS3Client(awsCredentials);
amazonS3Client.setRegion(region());
return amazonS3Client;
}
#Bean
public SimpleStorageResourceLoader simpleStorageResourceLoader(AmazonS3 client) {
return new SimpleStorageResourceLoader(client);
}
}
Create a "Store":
S3Store.java
public interface S3Store extends Store<String> {
}
Autowire this store into where you need to upload resources:
#Autowired
private S3Store store;
WritableResource r = (WritableResource)store.getResource(getId());
InputStream is = // plug your input stream in here
OutputStream os = r.getOutputStream();
IOUtils.copy(is, os);
is.close();
os.close();
When your application starts it will see the dependency on spring-content-s3 and your S3Store interface and inject an implementation for you, therefore, you don't need to worry about implementing this yourself.
IF you writing some sort of web application or microservice and you need a REST API then you can also add this dependency:
<dependency>
<groupId>com.github.paulcwarren</groupId>
<artifactId>spring-content-rest</artifactId>
<version>0.0.11</version>
</dependency>
Update your S3Config.java as follows:
#Configuration
#EnableS3Stores
#Import(RestConfiguration.class)
public class S3Config {
...
Update your store as follows:
S3Store.java
#StoreRestResource(path="s3docs")
public interface S3Store extends Store<String> {
}
Now when your application starts it will see your Store interface and also inject an #Controller implementation that will forward REST request onto your store. This replaces the autowiring code above obviously.
Then:
curl -X POST /s3docs/example-doc
with a multipart/form-data request will store the image in s3.
curl /s3docs/example-doc
will fetch it again and so on. This controller supports full CRUD and video streaming by the way.
If you want to associate this "content" with JPA Entity or something like that then you can have your S3Store extend AssociateStore or ContentStore and you have additional methods available that provide for associations.
There are a couple of getting started guides here. The s3 reference guide is here. And there is a tutorial video here. The coding bit starts about 1/2 way through.
HTH
Since it needs a temporary location to place files. Below code worked after deploying war on AWS.
private File convertMultiPartToFile(MultipartFile multipartFile) throws IOException {
File convFile = new File(System.getProperty("java.io.tmpdir") + System.getProperty("file.separator") +
multipartFile.getOriginalFilename());
multipartFile.transferTo(convFile);
return convFile;
}
You have problems with relative Paths
You can do this
public class UploadStackoverflow {
private String location = "upload-dir";
private Path rootLocation;
public File convertFile(MultipartFile file) throws IOException {
rootLocation = Paths.get(location);
Files.createDirectories(rootLocation);
String filename = StringUtils.cleanPath(file.getOriginalFilename());
InputStream inputStream = file.getInputStream();
Files.copy(inputStream, this.rootLocation.resolve(filename),
StandardCopyOption.REPLACE_EXISTING);
return new File(this.rootLocation.resolve(filename).toAbsolutePath().toString());
}
}
Based on this answer and the comments I implemented the code to receive the scores of an elastic search query.
public class CustomizedHotelRepositoryImpl implements CustomizedHotelRepository {
private final ElasticsearchTemplate elasticsearchTemplate;
#Autowired
public CustomizedHotelRepositoryImpl(ElasticsearchTemplate elasticsearchTemplate) {
super();
this.elasticsearchTemplate = elasticsearchTemplate;
}
#Override
public Page<Hotel> findHotelsAndScoreByName(String name) {
QueryBuilder queryBuilder = QueryBuilders.boolQuery()
.should(QueryBuilders.queryStringQuery(name).lenient(true).defaultOperator(Operator.OR).field("name"));
NativeSearchQuery nativeSearchQuery = new NativeSearchQueryBuilder().withQuery(queryBuilder)
.withPageable(PageRequest.of(0, 100)).build();
DefaultEntityMapper mapper = new DefaultEntityMapper();
ResultsExtractor<Page<Hotel>> rs = new ResultsExtractor<Page<Hotel>>() {
#Override
public Page<Hotel> extract(SearchResponse response) {
ArrayList<Hotel> hotels = new ArrayList<>();
SearchHit[] hits = response.getHits().getHits();
for (SearchHit hit : hits) {
try {
Hotel hotel = mapper.mapToObject(hit.getSourceAsString(), Hotel.class);
hotel.setScore(hit.getScore());
hotels.add(hotel);
} catch (IOException e) {
e.printStackTrace();
}
}
return new PageImpl<>(hotels, PageRequest.of(0, 100), response.getHits().getTotalHits());
}
};
return elasticsearchTemplate.query(nativeSearchQuery, rs);
}
}
As you can see I needed to create a new instance of DefaultEntityMapper mapper = new DefaultEntityMapper(); which should not be the case because it should be possible to #Autowire EntityMapper. If I do so, I get the exception that there is no bean.
Description:
Field entityMapper in com.example.elasticsearch5.es.cluster.repository.impl.CustomizedCluserRepositoryImpl required a bean of type 'org.springframework.data.elasticsearch.core.EntityMapper' that could not be found.
Action:
Consider defining a bean of type 'org.springframework.data.elasticsearch.core.EntityMapper' in your configuration.
So does anybody know if its possible to autowire EntityMapper directly or does it needs to create the bean manually using #Bean annotation.
I use spring-data-elasticsearch-3.0.2.RELEASE.jar where the core package is inside.
My pom.xml:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-elasticsearch</artifactId>
</dependency>
I checked out the source code of spring-data-elasticsearch. There is no bean/comoponent definition for EntityMapper. It seems this answer is wrong. I test it on my project and get the same error.
Consider defining a bean of type 'org.springframework.data.elasticsearch.core.EntityMapper' in your configuration.
I couldn't find any other option by except defining a #Bean
I have a restful service calling an external service using Spring Cloud Feign client
#FeignClient(name = "external-service", configuration = FeignClientConfig.class)
public interface ServiceClient {
#RequestMapping(value = "/test/payments", method = RequestMethod.POST)
public void addPayment(#Valid #RequestBody AddPaymentRequest addPaymentRequest);
#RequestMapping(value = "/test/payments/{paymentId}", method = RequestMethod.PUT)
public ChangePaymentStatusResponse updatePaymentStatus(#PathVariable("paymentId") String paymentId,
#Valid #RequestBody PaymentStatusUpdateRequest paymentStatusUpdateRequest);
}
I noticed the following failure 3-4 times in the last 3 months in my log file:
json.ERROR_RESPONSE_BODY:Connection refused executing POST
http://external-service/external/payments json.message:Send Payment
Add Payment Failure For other reason: {ERROR_RESPONSE_BODY=Connection
refused executing POST http://external-service/external/payments,
EVENT=ADD_PAYMENT_FAILURE, TRANSACTION_ID=XXXXXXX} {}
json.EVENT:ADD_PAYMENT_FAILURE
json.stack_trace:feign.RetryableException: Connection refused
executing POST http://external-service/external/payments at
feign.FeignException.errorExecuting(FeignException.java:67) at
feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:104)
at
feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:76)
at
feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103)
Is it possible to add Spring Retry on a Feign client.
What I wanted to annotate the addPayment operation with
#Retryable(value = {feign.RetryableException.class }, maxAttempts = 3, backoff = #Backoff(delay = 2000, multiplier=2))
But this is not possible, what other options do I have?
You can add a Retryer in the FeignClientConfig
#Configuration
public class FeignClientConfig {
#Bean
public Retryer retryer() {
return new Custom();
}
}
class Custom implements Retryer {
private final int maxAttempts;
private final long backoff;
int attempt;
public Custom() {
this(2000, 3);
}
public Custom(long backoff, int maxAttempts) {
this.backoff = backoff;
this.maxAttempts = maxAttempts;
this.attempt = 1;
}
public void continueOrPropagate(RetryableException e) {
if (attempt++ >= maxAttempts) {
throw e;
}
try {
Thread.sleep(backoff);
} catch (InterruptedException ignored) {
Thread.currentThread().interrupt();
}
}
#Override
public Retryer clone() {
return new Custom(backoff, maxAttempts);
}
}
Updated with sample Retryer example config based on the Retryer.Default.
If you are using ribbon you can set properties, you can use below properties for retry:
myapp.ribbon.MaxAutoRetries=5
myapp.ribbon.MaxAutoRetriesNextServer=5
myapp.ribbon.OkToRetryOnAllOperations=true
Note: "myapp" is your service id.
Checkout this Github implementation for working example
Just new a contructor Default
#Configuration
public class FeignClientConfig {
#Bean
public Retryer retryer() {
return new Retryer.Default(100, 2000, 3);
}
}
Adding this if it can help someone. I was getting connection reset using feign, as some unknown process was running on that port.
Try changing the port. Refer this to find the process running on a port
I prepared a blog post about using Spring Retry with Feign Client methods. You may consider checking the Post. All steps have been explained in the post.
This is my config. Test OK in spring boot 2.2.0.RELEASE
spring cloud Hoxton.M3.
feign.hystrix.enabled=true
MY-SPRING-API.ribbon.MaxAutoRetries=2
MY-SPRING-API.ribbon.MaxAutoRetriesNextServer=2
MY-SPRING-API.ribbon.OkToRetryOnAllOperations=true
MY-SPRING-API.ribbon.retryableStatusCodes=404,500
feign.client.config.PythonPatentClient.connectTimeout=500
feign.client.config.PythonPatentClient.readTimeout=500
hystrix.command.PythonPatentClient#timeTest(String).execution.isolation.thread.timeoutInMilliseconds=5000
java code is :
#FeignClient(name = "MY-SPRING-API",configuration = {PythonPatentConfig.class},fallbackFactory = FallBack.class)
public interface PythonPatentClient
#RequestLine("GET /test?q={q}")
void timeTest(#Param("appNo") String q);
Controller is :
#RequestMapping(value = "/test",method = {RequestMethod.POST,RequestMethod.GET})
public Object test() throws InterruptedException {
log.info("========important print enter test========");
TimeUnit.SECONDS.sleep(10L);
pom.xml additon add:
<dependency>
<groupId>org.springframework.retry</groupId>
<artifactId>spring-retry</artifactId>
</dependency>
optional:
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-aspects</artifactId>
</dependency>
#EnableRetry
#SpringBootApplication
public class ApiApplication
this is document :
https://docs.spring.io/spring-cloud-netflix/docs/2.2.10.RELEASE/reference/html/#retrying-failed-requests
https://github.com/spring-projects/spring-retry
https://github.com/spring-cloud/spring-cloud-netflix/
I resolved that by creating a wrapper on top of ServiceClient
#Configuration
public class ServiceClient {
#Autowired
ServiceFeignClient serviceFeignClient;
#Retryable(value = { ClientReprocessException.class }, maxAttemptsExpression = "#{${retryMaxAttempts}}", backoff = #Backoff(delayExpression = "#{${retryDelayTime}}"))
public void addPayment( AddPaymentRequest addPaymentRequest){
return serviceFeignClient.addPayment(addPaymentRequest);
}
}
I am trying to add JMX metric in my Dropwizard application, But when I am trying make it up locally it gives me error. I added following in my Application class in initalize method :
guiceBundle.addModule(new MetricsInstrumentationModule(bootstrap.getMetricRegistry()))
bootstrap.addBundle(new JmxMetricsBundle());
and in guice module file, I have added provider:
#Provides
#Singleton
MetricRegistry providesMetricRegistry(Environment environment) {
return environment.metrics();
}
added below dependency in pom.xml
<dependency>
<groupId>com.palominolabs.metrics</groupId>
<artifactId>metrics-guice</artifactId>
<version>3.1.3</version>
</dependency>
In my class I am using it below way:
private Timer getTimer(String name) {
return metricRegistry
.timer(MetricRegistry.name(DocumentService.class, name));
}
public void method(){
final Timer.Context context = getTimer("methodMetric").time();
try{
//do something
}finally {
context.stop();
}
}
but when I am making my service up it is throwing following error:
DEBUG [2017-06-02 09:01:58,059] com.codahale.metrics.JmxReporter:
Unable to register timer! javax.management.InstanceAlreadyExistsException:
metrics:name=io.dropwizard.jetty.MutableServletContextHandler.connect-requests
DEBUG [2017-06-02 09:01:58,066] com.codahale.metrics.JmxReporter:
Unable to register gauge! javax.management.InstanceAlreadyExistsException:
metrics:name=io.dropwizard.jetty.MutableServletContextHandler.percent-
4xx-1m
And similar error messages, although my application is getting up successfully. I have no idea why this error is occurring, is it some localhost setup issue which will not happen on production or some genuine issue which I am missing. please help.
EDIT:
JmxMetricsBundle class:
public class JmxMetricsBundle implements Bundle {
public void initialize(Bootstrap<?> bootstrap) {
JmxReporter reporter = JmxReporter.forRegistry(bootstrap.getMetricRegistry()).build();
reporter.start();
}
#Override
public void run(Environment environment) {
}
}
I want to be able to use a test properties files and only override a few properties. Having to override every single property will get ugly fast.
This is the code I am using to test my ability to mock properties and use existing properties in a test case
#RunWith(SpringRunner.class)
#SpringBootTest(classes = MyApp.class)
#TestPropertySource(
locations = { "classpath:myapp-test.properties" },
properties = { "test.key = testValue" })
public class EnvironmentMockedPropertiesTest {
#Autowired private Environment env;
// #MockBean private Environment env;
#Test public void testExistingProperty() {
// some.property=someValue
final String keyActual = "some.property";
final String expected = "someValue";
final String actual = env.getProperty(keyActual);
assertEquals(expected, actual);
}
#Test public void testMockedProperty() {
final String keyMocked = "mocked.test.key";
final String expected = "mockedTestValue";
when(env.getProperty(keyMocked)).thenReturn(expected);
final String actual = env.getProperty(keyMocked);
assertEquals(expected, actual);
}
#Test public void testOverriddenProperty() {
final String expected = "testValue";
final String actual = env.getProperty("test.key");
assertEquals(expected, actual);
}
}
What I find is:
#Autowired private Environment env;
testExistingProperty() and testOverriddenProperty() pass
testMockedProperty() fails
#MockBean private Environment env;
testMockedProperty() passes
testExistingProperty() and testOverriddenProperty() fail
Is there a way to achieve what I am aiming for?
Dependencies:
<spring.boot.version>1.4.3.RELEASE</spring.boot.version>
...
<!-- Spring -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot</artifactId>
<version>${spring.boot.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-autoconfigure</artifactId>
<version>${spring.boot.version}</version>
</dependency>
<!-- Starter for testing Spring Boot applications with libraries including JUnit,
Hamcrest and Mockito -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<version>${spring.boot.version}</version>
</dependency>
Ok i have made this work, you need to use Mockito to accompish what you are looking for:
Maven Dependency
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>2.6.4</version>
</dependency>
Test Class Set up
import static org.mockito.Mockito.*;
import static org.springframework.test.util.AopTestUtils.getTargetObject;
#RunWith(SpringRunner.class)
#SpringBootTest(classes = MyApp.class)
#TestPropertySource(
locations = { "classpath:myapp-test.properties" },
properties = { "test.key = testValue" })
public class AnswerTest {
// This will be only for injecting, we will not be using this object in tests.
#Autowired
private Environment env;
// This is the reference that will be used in tests.
private Environment envSpied;
// Map of properties that you intend to mock
private Map<String, String> mockedProperties;
#PostConstruct
public void postConstruct(){
mockedProperties = new HashMap<String, String>();
mockedProperties.put("mocked.test.key_1", "mocked.test.value_1");
mockedProperties.put("mocked.test.key_2", "mocked.test.value_2");
mockedProperties.put("mocked.test.key_3", "mocked.test.value_3");
// We use the Spy feature of mockito which enabled partial mocking
envSpied = Mockito.spy((Environment) getTargetObject(env));
// We mock certain retrieval of certain properties
// based on the logic contained in the implementation of Answer class
doAnswer(new CustomAnswer()).when(envSpied).getProperty(Mockito.anyString());
}
Test case
// Testing for both mocked and real properties in same test method
#Test public void shouldReturnAdequateProperty() {
String mockedValue = envSpied.getProperty("mocked.test.key_3");
String realValue = envSpied.getProperty("test.key");
assertEquals(mockedValue, "mocked.test.value_3");
assertEquals(realValue, "testValue");
}
Implementation of Mockito's Answer interface
// Here we define what should mockito do:
// a) return mocked property if the key is a mock
// b) invoke real method on Environment otherwise
private class CustomAnswer implements Answer<String>{
#Override
public String answer(InvocationOnMock invocationOnMock) throws Throwable {
Object[] arguments = invocationOnMock.getArguments();
String parameterKey = (String) arguments[0];
String mockedValue = mockedProperties.get(parameterKey);
if(mockedValue != null){
return mockedValue;
}
return (String) invocationOnMock.callRealMethod();
}
}
}
Try it out, and let me know if all is clear here.