Upload data file as byte array with Feign - spring

How can I send file in Feign as byte array?
#RequestLine("POST /api/files/{num}/push")
#Headers({"Content-Type: application/zip"})
void pushFile(#Param("num") String num, #Param("file") byte[] file);
This is not working and passing the data in form of json with top element named file.
What can I do to properly receive array of bytes on the other side using this controller method parameter annotation?
#RequestBody byte[] file

You can try OpenFeign/feign-form, simple example:
pom.xml dependencies
<dependencies>
<!--feign dependencies-->
<dependency>
<groupId>io.github.openfeign.form</groupId>
<artifactId>feign-form</artifactId>
<version>3.8.0</version>
</dependency>
<dependency>
<groupId>io.github.openfeign</groupId>
<artifactId>feign-core</artifactId>
<version>10.1.0</version>
</dependency>
<!--jetty to dependencies to check feign request-->
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
<version>9.4.3.v20170317</version>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-servlet</artifactId>
<version>9.4.3.v20170317</version>
</dependency>
</dependencies>
FeignUploadFileExample.java:
import feign.*;
import feign.codec.EncodeException;
import feign.codec.Encoder;
import org.eclipse.jetty.http.HttpStatus;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.servlet.ServletContextHandler;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.*;
import java.lang.reflect.Type;
import java.util.Map;
import static java.nio.charset.StandardCharsets.UTF_8;
public class FeignUploadFileExample {
public static void main(String[] args) throws Exception {
// start jetty server to check feign request
startSimpleJettyServer();
SimpleUploadFileApi simpleUploadFileApi = Feign.builder()
.encoder(new SimpleFileEncoder())
.target(SimpleUploadFileApi.class, "http://localhost:8080/upload");
// upload file bytes (simple string bytes)
byte[] fileBytes = "Hello World".getBytes();
String response = simpleUploadFileApi.uploadFile(fileBytes);
System.out.println(response);
}
public static final String FILE_PARAM = "file";
// encode #Param("file") to request body bytes
public static class SimpleFileEncoder implements Encoder {
public void encode(Object object, Type type, RequestTemplate template)
throws EncodeException {
template.body(Request.Body.encoded(
(byte[]) ((Map) object).get(FILE_PARAM), UTF_8));
}
}
// feign interface to upload file
public interface SimpleUploadFileApi {
#RequestLine("POST /upload")
#Headers("Content-Type: application/zip")
String uploadFile(#Param(FILE_PARAM) byte[] file);
}
// embedded jetty server
public static void startSimpleJettyServer() throws Exception {
Server server = new Server(8080);
ServletContextHandler handler = new ServletContextHandler(server, "/upload");
handler.addServlet(SimpleBlockingServlet.class, "/");
server.start();
}
// simple servlet get request and return info of received data
public static class SimpleBlockingServlet extends HttpServlet {
protected void doPost(
HttpServletRequest request,
HttpServletResponse response) throws IOException {
String data = new BufferedReader(
new InputStreamReader(request.getInputStream())).readLine();
response.setStatus(HttpStatus.OK_200);
response.getWriter().println("Request header 'Content-Type': " +
request.getHeaders("Content-Type").nextElement());
response.getWriter().println("Request downloaded file data: " + data);
}
}
}
response output:
Request header 'Content-Type': application/zip
Request downloaded file data: Hello World
also #RequestBody it's annotation for REST json body, for files:
#RequestParam("file") MultipartFile file
take a look at Spring Boot Uploading Files

You can use FormData from feign-form to upload file and specify Content-type and name:
<dependency>
<groupId>io.github.openfeign.form</groupId>
<artifactId>feign-form</artifactId>
<version>3.8.0</version>
</dependency>
#RequestLine("POST /api/files/{num}/push")
void pushFile(#Param("num") String num, #Param("file") FormData file);
Usage example:
byte[] bytes = getFileContent();
FormData file = new FormData("application/zip", "example.zip", bytes);
client.pushFile(num, file);
Or in case you are using spring-cloud-starter-openfeign:
#PostMapping("/api/files/{num}/push")
void pushFile(#PathVariable String num, #RequestPart FormData file);
Tested this for spring-cloud-starter-openfeign but I assume it works without spring considering FormData class lives in form-data dependency.

Related

sleuth does not show Trace Id / Span Id in logs while WebClient Rest call

On rest api call with Webclient, few default logs are printed like below but sleuth doesn't add tracid with it. see below:
2022-08-10 10:18:26.123 DEBUG [cib_bulk,,] 1 --- [or-http-epoll-1] r.netty.http.client.HttpClientConnect : [7c54bef8-1, L:/1.1.1.:60568 - R:xyz.c11.1.1.:443] Handler is being applied: {uri=xyz.c/services/productInventory/v2/product/search/count?abc=2346&status=ACTIVE, method=GET}
only application name is attached here [cib_bulk,,]. But in entire application, when I log manually through logger, then sleuth attach traceid and span id.
#Bean
public WebClient webClientWithTimeout() {
String baseUrl = environment.getProperty("cibase.productapi.service.url");
LOG.info("Base Url of Product Inventory Service: {}",baseUrl);
String username = environment.getProperty("cibase.productapi.basicauth.username");
String password = environment.getProperty("cibase.productapi.basicauth.password");
String trackingid = environment.getProperty("cibase.productapi.basicauth.trackingid");
String trackingIdValue = environment.getProperty("cibase.productapi.basicauth.trackingid.value");
HttpClient httpClient = HttpClient.create();
Builder builder =
WebClient.builder()
.codecs(configurer -> configurer.defaultCodecs().maxInMemorySize(IN_MEMORY_SIZE))
.filter(basicAuthentication(username, password));
if(trackingid != null){
builder.defaultHeader(trackingid, trackingIdValue);
}
return builder.baseUrl(baseUrl).clientConnector(new ReactorClientHttpConnector(httpClient)).build();
}
=============
List<Product> productList = webClient
.get()
.uri(uriBuilder -> uriBuilder.path(MessageConstants.PRODUCT_INVENTORY_API_URL).replaceQuery(queryString).build())
.retrieve()
.bodyToFlux(Product.class)
.collectList()
.retryWhen(retryConfiguration())
.block();
=====
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-sleuth</artifactId>
<version>3.1.1</version>
</dependency>
I found the solution. Just use below code to print Trace id and spanId in logging. This code is also useful to print REST call's request and response body in pretty format.
import static org.springframework.web.reactive.function.client.ExchangeFilterFunctions.basicAuthentication;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.env.Environment;
import org.springframework.http.client.reactive.ReactorClientHttpConnector;
import org.springframework.web.reactive.function.client.WebClient;
import org.springframework.web.reactive.function.client.WebClient.Builder;
import brave.http.HttpTracing;
import io.netty.handler.logging.LogLevel;
import reactor.netty.http.brave.ReactorNettyHttpTracing;
import reactor.netty.http.client.HttpClient;
import reactor.netty.transport.logging.AdvancedByteBufFormat;
#Configuration
public class WebClientConfiguration {
public static final Logger LOG = LoggerFactory.getLogger(WebClientConfiguration.class);
#Autowired
private Environment environment;
private static final int IN_MEMORY_SIZE = -1; // unlimited in-memory
/* Step 1: This bean is responsible to add Sleuth generated TraceId and SpanId in logs*/
#Bean
ReactorNettyHttpTracing reactorNettyHttpTracing(final HttpTracing httpTracing) {
return ReactorNettyHttpTracing.create(httpTracing);
}
#Bean
public WebClient webClientWithTimeout(final ReactorNettyHttpTracing reactorNettyHttpTracing) {
String baseUrl = environment.getProperty("cibase.productapi.service.url");
LOG.info("Base Url of Product Inventory Service: {}",baseUrl);
String username = environment.getProperty("cibase.productapi.basicauth.username");
String password = environment.getProperty("cibase.productapi.basicauth.password");
String trackingid = environment.getProperty("cibase.productapi.basicauth.trackingid");
String trackingIdValue = environment.getProperty("cibase.productapi.basicauth.trackingid.value");
// wiretap used to log request and response body
HttpClient httpClient = HttpClient.create().wiretap(this.getClass().getCanonicalName(), LogLevel.DEBUG, AdvancedByteBufFormat.TEXTUAL);
Builder builder =
WebClient.builder()
.codecs(configurer -> configurer.defaultCodecs().maxInMemorySize(IN_MEMORY_SIZE))
.filter(basicAuthentication(username, password));
if(trackingid != null){
builder.defaultHeader(trackingid, trackingIdValue);
}
/* step 2. reactorNettyHttpTracing object used here */
return builder
.baseUrl(baseUrl)
.clientConnector(new ReactorClientHttpConnector(reactorNettyHttpTracing.decorateHttpClient(httpClient)))
.build(); // here we have used the above bean
}
}

Spring Feign: Could not extract response: no suitable HttpMessageConverter found for response type

I am trying to get a Spring Cloud Netflix Feign client to fetch a bit of JSON over HTTP and convert it to an object. I keep getting this error instead:
org.springframework.web.client.RestClientException: Could not extract response: no suitable HttpMessageConverter found for response type [class io.urig.checkout.Book] and content type [application/json;charset=UTF-8]
Here's the bit of JSON returned from the remote service:
{
"id": 1,
"title": "Moby Dick",
"author": "Herman Melville"
}
Here's the corresponding class I'm trying to deserialize to:
package io.urig.checkout;
public class Book {
private long id;
private String title;
private String author;
public Book() {}
public Book(long id, String title, String author) {
super();
this.id = id;
this.title = title;
this.author = author;
}
public long getId() {
return id;
}
public void setId(long id) {
this.id = id;
}
public String getTitle() {
return title;
}
public void setTitle(String title) {
this.title = title;
}
public String getAuthor() {
return author;
}
public void setAuthor(String author) {
this.author = author;
}
}
And here's my Feign client:
package io.urig.checkout;
import java.util.Optional;
import org.springframework.cloud.openfeign.FeignClient;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import io.urig.checkout.Book;
#FeignClient(name="inventory", url="http://localhost:8080/")
public interface InventoryClient {
#RequestMapping(method = RequestMethod.GET, value = "books/{bookId}")
public Optional<Book> getBookById(#PathVariable(value="bookId") Long bookId);
}
What do I need to do to get this to work?
I don't know Feign, but when I've had "no suitable HttpMessageConverter found..." errors in the past, it's because the content type has not been registered. Perhaps you need to add this to the RequestMapping:
consumes = "application/json"
All I can suggest is to try to confirm if Feign configuration has MappingJackson2HttpMessageConverter registered as a converter for Book. Not sure if this is something that should work out of the box with Feign, or if you have to do it manually. I see an example on Feign's GitHub that has:
GitHub github = Feign.builder()
.encoder(new JacksonEncoder())
.decoder(new JacksonDecoder())
.target(GitHub.class, "https://api.github.com");
Have you created configuration using Feign.builder() or some equivalent configuration files?
You will need to ensure that you have at least one JSON library on your classpath. Feign supports both GSON and Jackson and Spring Cloud OpenFeign will autoconfigure the SpringEncoder and SpringDecoder instances with the appropriate MessageConverter if they are found on your classpath. Ensure that you have at least one of the following in your pom.xml or build.gradle
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.9.4</version>
</dependency>
or
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.2</version>
</dependency>
Once they are found, Spring will register the appropriate MessageConverter
I think your problem is the response type. Try converting it to Book from Optional. If you want to return an Optional than you should provide your custom converter.
Sorry, for too late answer.
Had the same problem.
Just add two parameters to your #RequestMapping -
consumes = "application/json", produces = "application/json"
In your code this will look like this -
package io.urig.checkout;
import java.util.Optional;
import org.springframework.cloud.openfeign.FeignClient;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import io.urig.checkout.Book;
#FeignClient(name="inventory", url="http://localhost:8080/")
public interface InventoryClient {
#RequestMapping(method = RequestMethod.GET, value = "books/{bookId}", consumes = "application/json", produces = "application/json")
public Optional<Book> getBookById(#PathVariable(value="bookId") Long bookId);
}
Thanks to all who tried to help!
As it turned out my issue was a defective Maven dependency, probably corrupted during download or installation. Having entirely deleted the .m2/repository folder on my machine and then updating Maven dependencies for the project the issue is now gone.
I am late here but I would like to add one more point. In my case I observed Spring Feign client returns this exception when you specified the return type as a specific model/entity class and that entity is not found.
You should check the response for the another service which you are calling and see what response it returns in case the entity is not found, or in case an exception is thrown.
So in case an entity is not found or any exception is thrown and that response does not match to what you have specified in return type then this exception is thrown in the client service.

configuring the encrypted database password in the spring datasource [duplicate]

I have the task of obfuscating passwords in our configuration files. While I don't think this is the right approach, managers disagree...
So the project I am working on is based on Spring Boot and we are using YAML configuration files. Currently the passwords are in plain text:
spring:
datasource:
url: jdbc:sqlserver://DatabaseServer
driverClassName: com.microsoft.sqlserver.jdbc.SQLServerDriver
username: ele
password: NotTheRealPassword
The idea is to have some special syntax that supports an obfuscated or encrypted password:
spring:
datasource:
url: jdbc:sqlserver://DatabaseServer
driverClassName: com.microsoft.sqlserver.jdbc.SQLServerDriver
username: ele
password: password(Tm90VGhlUmVhbFBhc3N3b3Jk)
In order for this to work I want to parse the property values using a regular expression and if it matches replace the value with the deobfuscated/decrypted value.
But how do I intercept the property value?
If finally got this to work. (Mainly thanks to stephane-deraco on github)
Key to the solution is a class that implements ApplicationContextInitializer<ConfigurableApplicationContext>. I called it PropertyPasswordDecodingContextInitializer.
The main problem was to get spring to use this ApplicationContextInitializer. Important information can be found in the reference. I chose the approach using a META-INF/spring.factories with following content:
org.springframework.context.ApplicationContextInitializer=ch.mycompany.myproject.PropertyPasswordDecodingContextInitializer
The PropertyPasswordDecodingContextInitializer uses a PropertyPasswordDecoder and an implementing class, currently for simplicity a Base64PropertyPasswordDecoder.
PropertyPasswordDecodingContextInitializer.java
package ch.mycompany.myproject;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.springframework.context.ApplicationContextInitializer;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.core.env.CompositePropertySource;
import org.springframework.core.env.ConfigurableEnvironment;
import org.springframework.core.env.EnumerablePropertySource;
import org.springframework.core.env.MapPropertySource;
import org.springframework.core.env.PropertySource;
import org.springframework.stereotype.Component;
#Component
public class PropertyPasswordDecodingContextInitializer implements ApplicationContextInitializer<ConfigurableApplicationContext> {
private static final Pattern decodePasswordPattern = Pattern.compile("password\\((.*?)\\)");
private PropertyPasswordDecoder passwordDecoder = new Base64PropertyPasswordDecoder();
#Override
public void initialize(ConfigurableApplicationContext applicationContext) {
ConfigurableEnvironment environment = applicationContext.getEnvironment();
for (PropertySource<?> propertySource : environment.getPropertySources()) {
Map<String, Object> propertyOverrides = new LinkedHashMap<>();
decodePasswords(propertySource, propertyOverrides);
if (!propertyOverrides.isEmpty()) {
PropertySource<?> decodedProperties = new MapPropertySource("decoded "+ propertySource.getName(), propertyOverrides);
environment.getPropertySources().addBefore(propertySource.getName(), decodedProperties);
}
}
}
private void decodePasswords(PropertySource<?> source, Map<String, Object> propertyOverrides) {
if (source instanceof EnumerablePropertySource) {
EnumerablePropertySource<?> enumerablePropertySource = (EnumerablePropertySource<?>) source;
for (String key : enumerablePropertySource.getPropertyNames()) {
Object rawValue = source.getProperty(key);
if (rawValue instanceof String) {
String decodedValue = decodePasswordsInString((String) rawValue);
propertyOverrides.put(key, decodedValue);
}
}
}
}
private String decodePasswordsInString(String input) {
if (input == null) return null;
StringBuffer output = new StringBuffer();
Matcher matcher = decodePasswordPattern.matcher(input);
while (matcher.find()) {
String replacement = passwordDecoder.decodePassword(matcher.group(1));
matcher.appendReplacement(output, replacement);
}
matcher.appendTail(output);
return output.toString();
}
}
PropertyPasswordDecoder.java
package ch.mycompany.myproject;
public interface PropertyPasswordDecoder {
public String decodePassword(String encodedPassword);
}
Base64PropertyPasswordDecoder.java
package ch.mycompany.myproject;
import java.io.UnsupportedEncodingException;
import org.apache.commons.codec.binary.Base64;
public class Base64PropertyPasswordDecoder implements PropertyPasswordDecoder {
#Override
public String decodePassword(String encodedPassword) {
try {
byte[] decodedData = Base64.decodeBase64(encodedPassword);
String decodedString = new String(decodedData, "UTF-8");
return decodedString;
} catch (UnsupportedEncodingException e) {
throw new RuntimeException(e);
}
}
}
Mind you, the ApplicationContext has not finished initialized at this stage, so autowiring or any other bean related mechanisms won't work.
Update: Included #jny's suggestions.
I used #Daniele Torino's answer and made several minor changes.
First, thanks to his link to the options on how to make spring recognize Initializer, I chose to do it in the Application:
public static void main(String[] args) throws Exception {
SpringApplication application=new SpringApplication(Application.class);
application.addInitializers(new PropertyPasswordDecodingContextInitializer());
application.run(args);
}
Second, IDEA told me that that else if (source instanceof CompositePropertySource) { is redundant and it is because CompositePropertySource inherits from EnumerablePropertySource.
Third, I beleive there is a minor bug: it messes up the order of property resolution. If you have one encoded property in environment, and another one in application.properties file the environment value will be overwritten with the application.properties value.
I changed the logic to insert the decodedProperties right before encoded:
for (PropertySource<?> propertySource : environment.getPropertySources()) {
Map<String, Object> propertyOverrides = new LinkedHashMap<>();
decodePasswords(propertySource, propertyOverrides);
if (!propertyOverrides.isEmpty()) {
environment.getPropertySources().addBefore(propertySource.getName(), new MapPropertySource("decoded"+propertySource.getName(), propertyOverrides));
}
}
Just use https://github.com/ulisesbocchio/jasypt-spring-boot, works out of the box
Inspired by #gogstad. Here is my major action in the spring boot project to encrypted my username and password and decrypted them in the project to work with tomcat:
1. In pom.xml file
<dependency>
<groupId>com.github.ulisesbocchio</groupId>
<artifactId>jasypt-spring-boot</artifactId>
<version>1.12</version>
</dependency>
…
<build>
<resources>
<resource>
<directory>src/main/java</directory>
<includes>
<include>**/*.properties</include>
<include>**/*.xml</include>
</includes>
<targetPath>${project.build.directory}/classes</targetPath>
</resource>
<resource>
<directory>src/main/resources</directory>
<includes>
<include>**/*.properties</include>
</includes>
<targetPath>${project.build.directory}/classes</targetPath>
</resource>
</resources>
…
</build>
2. In App.java (Note:to deploy the decryted springboot on tomcat, you should add the #ServletComponentScan annotation and extends the SpringBootServletInitializer)
#SpringBootApplication
#ServletComponentScan
#EnableEncryptableProperties
#PropertySource(name="EncryptedProperties", value = "classpath:config/encrypted.properties")
public class App extends SpringBootServletInitializer {
public static void main(String[] args) throws Exception {
SpringApplication.run(App.class, args);
}
}
3. Encrypted your username and password and fill the application.properties file with the result:
java -cp ~/.m2/repository/org/jasypt/jasypt/1.9.2/jasypt-1.9.2.jar org.jasypt.intf.cli.JasyptPBEStringEncryptionCLI input="mypassword" password=mykey algorithm=PBEWithMD5AndDES
output is like the demo below:
java -cp ~/.m2/repository/org/jasypt/jasypt/1.9.2/jasypt-1.9.2.jar org.jasypt.intf.cli.JasyptPBEStringEncryptionCLI input="mypassword" password=mykey algorithm=PBEWithMD5AndDES
----ENVIRONMENT-----------------
Runtime: Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.45-b02
----ARGUMENTS-------------------
algorithm: PBEWithMD5AndDES
input: mypassword
password: mykey
----OUTPUT----------------------
5XNwZF4qoCKTO8M8KUjRprQbivTkmI8H
4. under the directory src/main/resources/config add two properties file:
a. application.properties
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://xxx
spring.datasource.username=ENC(xxx)
spring.datasource.password=ENC(xxx)
mybatis.mapper-locations=classpath:*/mapper/*.xml
mybatis.type-aliases-package=com.xx.xxx.model
logging.level.com.xx.xxx: DEBUG
b. encrypted.properties
jasypt.encryptor.password=mykey
Use spring cloud config server
Define encrypt.key=MySecretKey
Post message to encrypt https://config-server/encrypt
Define password now like
app.password={cipher}encryptedvalue
Use #Value("${app.password}") in code
and spring boot should give you decrypted value

CXF InInterceptor not firing

I have created web service. It works fine. Now I'm trying to implement authentication to it. I'm using CXF interceptors for that purpose. For some reason interceptors won't fire. What am I missing? This is my first web service.
import javax.annotation.Resource;
import javax.inject.Inject;
import javax.jws.WebMethod;
import javax.jws.WebParam;
import javax.jws.WebService;
import javax.xml.ws.WebServiceContext;
import org.apache.cxf.interceptor.InInterceptors;
#WebService
#InInterceptors(interceptors = "ws.BasicAuthAuthorizationInterceptor")
public class Service {
#WebMethod
public void test(#WebParam(name = "value") Integer value) throws Exception {
System.out.println("Value = " + value);
}
}
-
package ws;
import java.io.IOException;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
import org.apache.cxf.binding.soap.interceptor.SoapHeaderInterceptor;
import org.apache.cxf.configuration.security.AuthorizationPolicy;
import org.apache.cxf.endpoint.Endpoint;
import org.apache.cxf.interceptor.Fault;
import org.apache.cxf.message.Exchange;
import org.apache.cxf.message.Message;
import org.apache.cxf.transport.Conduit;
import org.apache.cxf.ws.addressing.EndpointReferenceType;
public class BasicAuthAuthorizationInterceptor extends SoapHeaderInterceptor {
#Override
public void handleMessage(Message message) throws Fault {
System.out.println("**** GET THIS LINE TO CONSOLE TO SEE IF INTERCEPTOR IS FIRING!!!");
AuthorizationPolicy policy = message.get(AuthorizationPolicy.class);
// If the policy is not set, the user did not specify credentials.
// 401 is sent to the client to indicate that authentication is required.
if (policy == null) {
sendErrorResponse(message, HttpURLConnection.HTTP_UNAUTHORIZED);
return;
}
String username = policy.getUserName();
String password = policy.getPassword();
// CHECK USERNAME AND PASSWORD
if (!checkLogin(username, password)) {
System.out.println("handleMessage: Invalid username or password for user: "
+ policy.getUserName());
sendErrorResponse(message, HttpURLConnection.HTTP_FORBIDDEN);
}
}
private boolean checkLogin(String username, String password) {
if (username.equals("admin") && password.equals("admin")) {
return true;
}
return false;
}
private void sendErrorResponse(Message message, int responseCode) {
Message outMessage = getOutMessage(message);
outMessage.put(Message.RESPONSE_CODE, responseCode);
// Set the response headers
#SuppressWarnings("unchecked")
Map<String, List<String>> responseHeaders = (Map<String, List<String>>) message
.get(Message.PROTOCOL_HEADERS);
if (responseHeaders != null) {
responseHeaders.put("WWW-Authenticate", Arrays.asList(new String[] { "Basic realm=realm" }));
responseHeaders.put("Content-Length", Arrays.asList(new String[] { "0" }));
}
message.getInterceptorChain().abort();
try {
getConduit(message).prepare(outMessage);
close(outMessage);
} catch (IOException e) {
e.printStackTrace();
}
}
private Message getOutMessage(Message inMessage) {
Exchange exchange = inMessage.getExchange();
Message outMessage = exchange.getOutMessage();
if (outMessage == null) {
Endpoint endpoint = exchange.get(Endpoint.class);
outMessage = endpoint.getBinding().createMessage();
exchange.setOutMessage(outMessage);
}
outMessage.putAll(inMessage);
return outMessage;
}
private Conduit getConduit(Message inMessage) throws IOException {
Exchange exchange = inMessage.getExchange();
EndpointReferenceType target = exchange.get(EndpointReferenceType.class);
Conduit conduit = exchange.getDestination().getBackChannel(inMessage, null, target);
exchange.setConduit(conduit);
return conduit;
}
private void close(Message outMessage) throws IOException {
OutputStream os = outMessage.getContent(OutputStream.class);
os.flush();
os.close();
}
}
I'm fighting with this for few days now. Don't know what to google any more. Help is appreciated.
I've found solution. I was missing the following line in MANIFEST.MF file in war project:
Dependencies: org.apache.cxf
Maven wasn't includint this line by himself so I had to find workaround. I found about that here. It says: When using annotations on your endpoints / handlers such as the Apache CXF ones (#InInterceptor, #GZIP, ...) remember to add the proper module dependency in your manifest. Otherwise your annotations are not picked up and added to the annotation index by JBoss Application Server 7, resulting in them being completely and silently ignored.
This is where I found out how to change MANIFEST.MF file.
In short, I added custom manifest file to my project and referenced it in pom.xml. Hope this helps someone.
The answer provided by Felix is accurate. I managed to solve the problem using his instructions. Just for completion here is the maven config that lets you use your own MANIFEST.MF file placed in the META-INF folder.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<configuration>
<archive>
<manifestFile>src/main/resources/META-INF/MANIFEST.MF</manifestFile>
</archive>
</configuration>
</plugin>
and here is the relevant content of the content of the MANIFEST.MF file I was using.
Manifest-Version: 1.0
Description: yourdescription
Dependencies: org.apache.ws.security,org.apache.cxf

MRUnit with Avro NullPointerException in Serialization

I'm trying to test a Hadoop .mapreduce Avro job using MRUnit. I am receiving a NullPointerException as seen below. I've attached a portion of the pom and source code. Any assistance would be appreciated.
Thanks
The error I'm getting is :
java.lang.NullPointerException
at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:73)
at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:91)
at org.apache.hadoop.mrunit.internal.io.Serialization.copyWithConf(Serialization.java:104)
at org.apache.hadoop.mrunit.TestDriver.copy(TestDriver.java:608)
at org.apache.hadoop.mrunit.MapDriverBase.setInputKey(MapDriverBase.java:64)
at org.apache.hadoop.mrunit.MapDriverBase.setInput(MapDriverBase.java:104)
at org.apache.hadoop.mrunit.MapDriverBase.withInput(MapDriverBase.java:218)
at org.lab41.project.mapreduce.ParseMetadataAsTextIntoAvroTest.testMap(ParseMetadataAsTextIntoAvroTest.java:115)
.....
pom snippet:
<dependency>
<groupId>org.apache.mrunit</groupId>
<artifactId>mrunit</artifactId>
<version>0.9.0-incubating</version>
<classifier>hadoop2</classifier>
<scope>test</scope>
</dependency>
<avro.version>1.7.4</avro.version>
<hadoop.version>2.0.0-mr1-cdh4.1.3</hadoop.version>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>${avro.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>${hadoop.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro-mapred</artifactId>
<version>${avro.version}</version>
<classifier>hadoop2</classifier>
</dependency>
Here is an excerpt of the test :
import static org.junit.Assert.*;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.CharBuffer;
import java.nio.charset.Charset;
import java.nio.charset.CharsetEncoder;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import org.apache.avro.mapred.AvroKey;
import org.apache.avro.hadoop.io.AvroSerialization;
import org.apache.avro.mapred.AvroValue;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mrunit.mapreduce.MapDriver;
import org.apache.hadoop.mrunit.types.Pair;
import org.junit.After;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
import org.lab41.project.domain.DataRecord;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class ParseMetadataAsTextIntoAvroTest {
Logger logger = LoggerFactory
.getLogger(ParseMetadataAsTextIntoAvroTest.class);
private MapDriver<LongWritable, Text, AvroKey<Long>, AvroValue<DataRecord>> mapDriver;
#BeforeClass
public static void setUpClass() {
}
#AfterClass
public static void tearDownClass() {
}
#Before
public void setUp() throws IOException {
ParseMetadataAsTextIntoAvroMapper mapper = new ParseMetadataAsTextIntoAvroMapper();
mapDriver = new MapDriver<LongWritable, Text, AvroKey<Long>, AvroValue<DataRecord>>();
mapDriver.setMapper(mapper);
mapDriver.getConfiguration().setStrings("io.serializations", new String[]{
AvroSerialization.class.getName()
});
}
#Test
public void testMap() throws ParseException, IOException {
Text testInputText = new Text(test0);
DataRecord record = new DataRecord();
….
AvroKey<Long> expectedPivot = new AvroKey<Long>(1L);
AvroValue<DataRecord> expectedRecord = new AvroValue<DataRecord>(record);
mapDriver.withInput(new Pair<LongWritable, Text>(new LongWritable(1), testInputText));
mapDriver.withOutput(new Pair<AvroKey<Long>, AvroValue<DataRecord>>(expectedPivot, expectedRecord));
mapDriver.runTest();
}
}
In order to get this to work you have add the AvroSerializatio to the default serailizations. You also have to configure AvroSerializationn.
#Before
public void setUp() throws IOException {
ParseMetadataAsTextIntoAvroMapper mapper = new ParseMetadataAsTextIntoAvroMapper();
mapDriver = new MapDriver<LongWritable, Text, AvroKey<Long>, AvroValue<NetworkRecord>>();
mapDriver.setMapper(mapper);
//Copy over the default io.serializations. If you don't do this then you will
//not be able to deserialize the inputs to the mapper
String[] strings = mapDriver.getConfiguration().getStrings("io.serializations");
String[] newStrings = new String[strings.length +1];
System.arraycopy( strings, 0, newStrings, 0, strings.length );
newStrings[newStrings.length-1] = AvroSerialization.class.getName();
//Now you have to configure AvroSerialization by sepecifying the key
//writer Schema and the value writer schema.
mapDriver.getConfiguration().setStrings("io.serializations", newStrings);
mapDriver.getConfiguration().setStrings("avro.serialization.key.writer.schema", Schema.create(Schema.Type.LONG).toString(true));
mapDriver.getConfiguration().setStrings("avro.serialization.value.writer.schema", NetworkRecord.SCHEMA$.toString(true));
}
This also solve the problem, with merits of shorter and more clear code.
MapDriver driver = MapDriver.newMapDriver(your mapper);
Configuration conf = driver.getConfiguration();
AvroSerialization.addToConfiguration(conf);
AvroSerialization.setKeyWriterSchema(conf, your schema);
AvroSerialization.setKeyReaderSchema(conf, your schema);
Job job = new Job(conf);
job.set... your job settings;
AvroJob.set... your avro job settings;
It may be bug of mrunit, that don't set the io.serializations right
Instead it should have been set by job.setInputFormatClass(AvroKeyInputFormat.class) I think.
You have to add AvroSerialization to the default serializations and configure AvroSerialization.
#Before
public void setUp() throws IOException {
ParseMetadataAsTextIntoAvroMapper mapper = new ParseMetadataAsTextIntoAvroMapper();
mapDriver = new MapDriver<LongWritable, Text, AvroKey<Long>, AvroValue<NetworkRecord>>();
mapDriver.setMapper(mapper);
Configuration configuration = mapDriver.getConfiguration();
// Add AvroSerialization to the configuration
// (copy over the default serializations for deserializing the mapper inputs)
String[] serializations = configuration.getStrings(CommonConfigurationKeysPublic.IO_SERIALIZATIONS_KEY);
String[] newSerializations = Arrays.copyOf(serializations, serializations.length + 1);
newSerializations[serializations.length] = AvroSerialization.class.getName();
configuration.setStrings(CommonConfigurationKeysPublic.IO_SERIALIZATIONS_KEY, newSerializations);
//Configure AvroSerialization by specifying the key writer and value writer schemas
AvroSerialization.setKeyWriterSchema(configuration, Schema.create(Schema.Type.LONG));
AvroSerialization.setValueWriterSchema(configuration, NetworkRecord.SCHEMA$)
}
Answered here: https://issues.apache.org/jira/browse/MRUNIT-181 specifically: https://cwiki.apache.org/confluence/display/MRUNIT/MRUnit+with+Avro

Resources