Linebreaks ("\n" or "\r") are been stripped out from gzip response body, when using response compression configuration (as below) in Spring Cloud Open Feign. There are no errors raised. Linebreaks are just been replace by an empty string "". The response has the correct "content-encoding: gzip" header, and a well formed gzipped body content.
Does someone has a clue? It seems a issue for me as I opened here spring-cloud-openfeign/issue400
feign.compression.response.enabled: true
feign.compression.response.useGzipDecoder: true
# Same behaviour using Apache Http as client
feign.httpclient.enabled: true
SpringCloudFeignClient.java
package springcloudfeigngzip;
import org.springframework.cloud.openfeign.FeignClient;
import org.springframework.web.bind.annotation.GetMapping;
#FeignClient(name = "SpringCloudFeignClient", url = "http://localhost:8082")
public interface SpringCloudFeignClient {
#GetMapping(value = "/gzip")
String getGzippedString();
}
ApplicationTest.java:
package springcloudfeigngzip;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.cloud.contract.wiremock.AutoConfigureWireMock;
import static com.github.tomakehurst.wiremock.client.WireMock.*;
import static org.junit.jupiter.api.Assertions.assertEquals;
#SpringBootTest
#AutoConfigureWireMock(port = 8082)
class ApplicationTest {
#Autowired
private SpringCloudFeignClient springCloudFeignClient;
#Test
void success_GzipOneLine() {
stubFor(get(urlEqualTo("/gzip")).withHeader("Accept-Encoding", containing("gzip"))
.willReturn(aResponse().withStatus(200).withBody("lineone")));
String response = springCloudFeignClient.getGzippedString();
assertEquals("lineone", response); //success
}
#Test
void fail_GzipLineBreak() {
stubFor(get(urlEqualTo("/gzip")).withHeader("Accept-Encoding", containing("gzip"))
.willReturn(aResponse().withStatus(200).withBody("lineone\nlinetwo")));
String response = springCloudFeignClient.getGzippedString();
assertEquals("lineone\nlinetwo", response); //fail!
}
}
bootstrap.yml:
feign:
compression:
response:
enabled: true
useGzipDecoder: true
Versions:
org.springframework.boot: 2.3.3.RELEASE
org.springframework.cloud: Hoxton.SR8
Full project here:
https://github.com/spring-cloud/spring-cloud-openfeign/files/5147346/spring-cloud-feign-gzip.zip
Related
On rest api call with Webclient, few default logs are printed like below but sleuth doesn't add tracid with it. see below:
2022-08-10 10:18:26.123 DEBUG [cib_bulk,,] 1 --- [or-http-epoll-1] r.netty.http.client.HttpClientConnect : [7c54bef8-1, L:/1.1.1.:60568 - R:xyz.c11.1.1.:443] Handler is being applied: {uri=xyz.c/services/productInventory/v2/product/search/count?abc=2346&status=ACTIVE, method=GET}
only application name is attached here [cib_bulk,,]. But in entire application, when I log manually through logger, then sleuth attach traceid and span id.
#Bean
public WebClient webClientWithTimeout() {
String baseUrl = environment.getProperty("cibase.productapi.service.url");
LOG.info("Base Url of Product Inventory Service: {}",baseUrl);
String username = environment.getProperty("cibase.productapi.basicauth.username");
String password = environment.getProperty("cibase.productapi.basicauth.password");
String trackingid = environment.getProperty("cibase.productapi.basicauth.trackingid");
String trackingIdValue = environment.getProperty("cibase.productapi.basicauth.trackingid.value");
HttpClient httpClient = HttpClient.create();
Builder builder =
WebClient.builder()
.codecs(configurer -> configurer.defaultCodecs().maxInMemorySize(IN_MEMORY_SIZE))
.filter(basicAuthentication(username, password));
if(trackingid != null){
builder.defaultHeader(trackingid, trackingIdValue);
}
return builder.baseUrl(baseUrl).clientConnector(new ReactorClientHttpConnector(httpClient)).build();
}
=============
List<Product> productList = webClient
.get()
.uri(uriBuilder -> uriBuilder.path(MessageConstants.PRODUCT_INVENTORY_API_URL).replaceQuery(queryString).build())
.retrieve()
.bodyToFlux(Product.class)
.collectList()
.retryWhen(retryConfiguration())
.block();
=====
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-sleuth</artifactId>
<version>3.1.1</version>
</dependency>
I found the solution. Just use below code to print Trace id and spanId in logging. This code is also useful to print REST call's request and response body in pretty format.
import static org.springframework.web.reactive.function.client.ExchangeFilterFunctions.basicAuthentication;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.env.Environment;
import org.springframework.http.client.reactive.ReactorClientHttpConnector;
import org.springframework.web.reactive.function.client.WebClient;
import org.springframework.web.reactive.function.client.WebClient.Builder;
import brave.http.HttpTracing;
import io.netty.handler.logging.LogLevel;
import reactor.netty.http.brave.ReactorNettyHttpTracing;
import reactor.netty.http.client.HttpClient;
import reactor.netty.transport.logging.AdvancedByteBufFormat;
#Configuration
public class WebClientConfiguration {
public static final Logger LOG = LoggerFactory.getLogger(WebClientConfiguration.class);
#Autowired
private Environment environment;
private static final int IN_MEMORY_SIZE = -1; // unlimited in-memory
/* Step 1: This bean is responsible to add Sleuth generated TraceId and SpanId in logs*/
#Bean
ReactorNettyHttpTracing reactorNettyHttpTracing(final HttpTracing httpTracing) {
return ReactorNettyHttpTracing.create(httpTracing);
}
#Bean
public WebClient webClientWithTimeout(final ReactorNettyHttpTracing reactorNettyHttpTracing) {
String baseUrl = environment.getProperty("cibase.productapi.service.url");
LOG.info("Base Url of Product Inventory Service: {}",baseUrl);
String username = environment.getProperty("cibase.productapi.basicauth.username");
String password = environment.getProperty("cibase.productapi.basicauth.password");
String trackingid = environment.getProperty("cibase.productapi.basicauth.trackingid");
String trackingIdValue = environment.getProperty("cibase.productapi.basicauth.trackingid.value");
// wiretap used to log request and response body
HttpClient httpClient = HttpClient.create().wiretap(this.getClass().getCanonicalName(), LogLevel.DEBUG, AdvancedByteBufFormat.TEXTUAL);
Builder builder =
WebClient.builder()
.codecs(configurer -> configurer.defaultCodecs().maxInMemorySize(IN_MEMORY_SIZE))
.filter(basicAuthentication(username, password));
if(trackingid != null){
builder.defaultHeader(trackingid, trackingIdValue);
}
/* step 2. reactorNettyHttpTracing object used here */
return builder
.baseUrl(baseUrl)
.clientConnector(new ReactorClientHttpConnector(reactorNettyHttpTracing.decorateHttpClient(httpClient)))
.build(); // here we have used the above bean
}
}
I'm calling the Post method of org.apache.http.client.fluent.Request class, and it's blocking - the next line of code is not executed. I'm getting logging that says that I got a successful response back too! My log messages (below) confirm that the message came back with an OK status (200).
>> ...with body JSON:
<< status: 200
So what happened? More importantly, what can I do to protect myself?
P.S. I've used this code thousands of times and this has, to my knowledge, never happened before.
Context
This is a Spring BOOT application (version 1.3.5). I'm using the RESTTemplate to communicate internally among microservices, but I'm using the apache fluent client to talk to an outside web service. I'm using org.apache.httpcomponents (version 4.5.2).
My code:
import java.io.IOException;
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Iterator;
import java.util.List;
import java.util.stream.Collectors;
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang.StringUtils;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.HttpStatus;
import org.apache.http.StatusLine;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpResponseException;
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.fluent.Request;
import org.apache.http.entity.ContentType;
import org.json.JSONObject;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.eneura.devcom.microservices.exceptions.ExternalWebServiceCommunicationFailureException;
import com.eneura.devcom.microservices.exceptions.InvalidArgumentException;
import com.eneura.devcom.microservices.exceptions.PreconditionsFailureException;
final private String internalDoPostSingleCall(
String subUrlPath, String jsonString, List<String> parmList
)
throws InvalidArgumentException, ExternalWebServiceCommunicationFailureException, HttpResponseException
{
String bodyArg = (StringUtils.isNotBlank(jsonString)) ? jsonString : ""; //don't pass in null or blanks or ... just fold down to ""
String url = computeUrl(subUrlPath, parmList);
logger.info(">> sending POST " + url);
logger.info(">> ...with body JSON: " + bodyArg);
String res;
try {
res = Request
.Post( url )
.bodyString(
bodyArg, ContentType.APPLICATION_JSON
).execute()
.handleResponse(new ResponseHandler<String>() {
public String handleResponse(
final HttpResponse response) throws IOException {
StatusLine statusLine = response.getStatusLine();
HttpEntity entity = response.getEntity();
logger.info("<< status: " + statusLine.getStatusCode() );
if (statusLine.getStatusCode() >= 300) {
throw new HttpResponseException(statusLine
.getStatusCode(), statusLine
.getReasonPhrase());
}
String content =
(null == entity) ? (
"" // ok not to have any content back.
) : (
IOUtils.toString(entity.getContent(), "UTF-8")
);
return content;
}
});
} catch (HttpResponseException e) {
throw e;
} catch (IOException e) {
String msg =
"failed to perform POST to ExternalWebService - exception: " + e.getLocalizedMessage();
throw new ExternalWebServiceCommunicationFailureException(msg);
}
logger.info("<< body returned for POST (len="+ res.length() +"): " + res);
return res;
}
Log messages:
2020-03-14 18:46:14.119 INFO 15213 --- [http-nio-6444-exec-9] c.e.d.m.s.avcourier.AVRestDelegate : >> sending POST https://externalwebservice.com/api/v1/operations/a0a01212445566770011001130304040/cancel
2020-03-14 18:46:14.119 INFO 15213 --- [http-nio-6444-exec-9] c.e.d.m.s.avcourier.AVRestDelegate : >> ...with body JSON:
2020-03-14 18:46:14.393 INFO 15213 --- [http-nio-6444-exec-9] c.e.d.m.s.avcourier.AVRestDelegate : << status: 200
2020-03-14 18:50:07.589 INFO 15213 --- [AsyncResolver-bootstrap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver : Resolving eureka endpoints via configuration
2020-03-14 18:55:07.590 INFO 15213 --- [AsyncResolver-bootstrap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver : Resolving eureka endpoints via configuration
2020-03-14 19:00:07.591 INFO 15213 --- [AsyncResolver-bootstrap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver : Resolving eureka endpoints via configuration
I am using spring boot to interact with pubsub topic.
My config class for this connection look like this:
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.cloud.gcp.pubsub.core.PubSubTemplate;
import org.springframework.cloud.gcp.pubsub.core.publisher.PubSubPublisherTemplate;
import org.springframework.cloud.gcp.pubsub.support.PublisherFactory;
import org.springframework.cloud.gcp.pubsub.support.converter.SimplePubSubMessageConverter;
import org.springframework.util.Assert;
import org.springframework.util.concurrent.ListenableFuture;
import org.springframework.util.concurrent.SettableListenableFuture;
import com.google.api.core.ApiFuture;
import com.google.api.core.ApiFutureCallback;
import com.google.api.core.ApiFutures;
import com.google.pubsub.v1.PubsubMessage;
public abstract class PubSubPublisher {
private static final Logger LOGGER = LoggerFactory.getLogger(PubSubPublisher.class);
private final PubSubTemplate pubSubTemplate;
protected PubSubPublisher(PubSubTemplate pubSubTemplate) {
this.pubSubTemplate = pubSubTemplate;
}
protected abstract String topic(String topicName);
public ListenableFuture<String> publish(String topicName, String message) {
LOGGER.info("Publishing to topic [{}]. Message: [{}]", topicName, message);
return pubSubTemplate.publish(topicName, message);
}
}
And I am calling this at my service, like this:
publisher.publish(topic-name, payload);
This publish method is async one, which always pass on did not wait for acknowldgrment. I make add get after publish for wait until it get the response from pubsub.
But I wanted to know if in case my topic is not already present and i try to push some message, it should throw some error like resource not found, considering using default async method only.
Might be implementing the callback would help but i am unable to do that in my code. And the current override publish method which use callback is just throwing the WARN not exception i wanted that to be exception. that is the reason i wanted to implement the callback.
You can check if Topic already present
from google.cloud import pubsub_v1
project_id = "projectname"
topic_name = "unknowTopic"
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(project_id, topic_name)
try:
response = publisher.get_topic(topic_path)
except Exception as e:
print(e)
This returns the error as
404 Resource not found (resource=unknowTopic).
I am facing some issues while pinging the server With RIDC code for oracle UCM 10 g and i am getting HTTP/1.1 401 Authorization Required exception.
import java.util.logging.Level;
import java.util.logging.Logger;
import oracle.stellent.ridc.IdcClient;
import oracle.stellent.ridc.IdcClientException;
import oracle.stellent.ridc.IdcClientManager;
import oracle.stellent.ridc.IdcContext;
import oracle.stellent.ridc.model.DataBinder;
import oracle.stellent.ridc.protocol.ServiceResponse;
public class PingGuest {
IdcClientManager manager = new IdcClientManager();
IdcClient idcClient;
public PingGuest() throws IdcClientException {
this.idcClient = manager.createClient("Server Address");
idcClient.getConfig ().setProperty ("http.library", "apache4");
IdcContext userPasswordContext = new IdcContext("user", "pass");
DataBinder dataBinder = this.idcClient.createBinder ();
dataBinder.putLocal ("IdcService", "PING_SERVER");
ServiceResponse response = idcClient.sendRequest (userPasswordContext, dataBinder);
}
public static void main(String args[]){
try {
new PingGuest();
} catch (IdcClientException ex) {
Logger.getLogger(PingGuest.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
This is the error.
oracle.stellent.ridc.protocol.http.HttpProtocolException: Http status: HTTP/1.1 401 Authorization Required
Any kind of help is appreciated.
You have omitted your server address. I understand why you omitted it, but getting that wrong might cause the issue you are seeing. If you've read newer documentation, keep in mind that it differs a bit in the older versions.
I have created web service. It works fine. Now I'm trying to implement authentication to it. I'm using CXF interceptors for that purpose. For some reason interceptors won't fire. What am I missing? This is my first web service.
import javax.annotation.Resource;
import javax.inject.Inject;
import javax.jws.WebMethod;
import javax.jws.WebParam;
import javax.jws.WebService;
import javax.xml.ws.WebServiceContext;
import org.apache.cxf.interceptor.InInterceptors;
#WebService
#InInterceptors(interceptors = "ws.BasicAuthAuthorizationInterceptor")
public class Service {
#WebMethod
public void test(#WebParam(name = "value") Integer value) throws Exception {
System.out.println("Value = " + value);
}
}
-
package ws;
import java.io.IOException;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
import org.apache.cxf.binding.soap.interceptor.SoapHeaderInterceptor;
import org.apache.cxf.configuration.security.AuthorizationPolicy;
import org.apache.cxf.endpoint.Endpoint;
import org.apache.cxf.interceptor.Fault;
import org.apache.cxf.message.Exchange;
import org.apache.cxf.message.Message;
import org.apache.cxf.transport.Conduit;
import org.apache.cxf.ws.addressing.EndpointReferenceType;
public class BasicAuthAuthorizationInterceptor extends SoapHeaderInterceptor {
#Override
public void handleMessage(Message message) throws Fault {
System.out.println("**** GET THIS LINE TO CONSOLE TO SEE IF INTERCEPTOR IS FIRING!!!");
AuthorizationPolicy policy = message.get(AuthorizationPolicy.class);
// If the policy is not set, the user did not specify credentials.
// 401 is sent to the client to indicate that authentication is required.
if (policy == null) {
sendErrorResponse(message, HttpURLConnection.HTTP_UNAUTHORIZED);
return;
}
String username = policy.getUserName();
String password = policy.getPassword();
// CHECK USERNAME AND PASSWORD
if (!checkLogin(username, password)) {
System.out.println("handleMessage: Invalid username or password for user: "
+ policy.getUserName());
sendErrorResponse(message, HttpURLConnection.HTTP_FORBIDDEN);
}
}
private boolean checkLogin(String username, String password) {
if (username.equals("admin") && password.equals("admin")) {
return true;
}
return false;
}
private void sendErrorResponse(Message message, int responseCode) {
Message outMessage = getOutMessage(message);
outMessage.put(Message.RESPONSE_CODE, responseCode);
// Set the response headers
#SuppressWarnings("unchecked")
Map<String, List<String>> responseHeaders = (Map<String, List<String>>) message
.get(Message.PROTOCOL_HEADERS);
if (responseHeaders != null) {
responseHeaders.put("WWW-Authenticate", Arrays.asList(new String[] { "Basic realm=realm" }));
responseHeaders.put("Content-Length", Arrays.asList(new String[] { "0" }));
}
message.getInterceptorChain().abort();
try {
getConduit(message).prepare(outMessage);
close(outMessage);
} catch (IOException e) {
e.printStackTrace();
}
}
private Message getOutMessage(Message inMessage) {
Exchange exchange = inMessage.getExchange();
Message outMessage = exchange.getOutMessage();
if (outMessage == null) {
Endpoint endpoint = exchange.get(Endpoint.class);
outMessage = endpoint.getBinding().createMessage();
exchange.setOutMessage(outMessage);
}
outMessage.putAll(inMessage);
return outMessage;
}
private Conduit getConduit(Message inMessage) throws IOException {
Exchange exchange = inMessage.getExchange();
EndpointReferenceType target = exchange.get(EndpointReferenceType.class);
Conduit conduit = exchange.getDestination().getBackChannel(inMessage, null, target);
exchange.setConduit(conduit);
return conduit;
}
private void close(Message outMessage) throws IOException {
OutputStream os = outMessage.getContent(OutputStream.class);
os.flush();
os.close();
}
}
I'm fighting with this for few days now. Don't know what to google any more. Help is appreciated.
I've found solution. I was missing the following line in MANIFEST.MF file in war project:
Dependencies: org.apache.cxf
Maven wasn't includint this line by himself so I had to find workaround. I found about that here. It says: When using annotations on your endpoints / handlers such as the Apache CXF ones (#InInterceptor, #GZIP, ...) remember to add the proper module dependency in your manifest. Otherwise your annotations are not picked up and added to the annotation index by JBoss Application Server 7, resulting in them being completely and silently ignored.
This is where I found out how to change MANIFEST.MF file.
In short, I added custom manifest file to my project and referenced it in pom.xml. Hope this helps someone.
The answer provided by Felix is accurate. I managed to solve the problem using his instructions. Just for completion here is the maven config that lets you use your own MANIFEST.MF file placed in the META-INF folder.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<configuration>
<archive>
<manifestFile>src/main/resources/META-INF/MANIFEST.MF</manifestFile>
</archive>
</configuration>
</plugin>
and here is the relevant content of the content of the MANIFEST.MF file I was using.
Manifest-Version: 1.0
Description: yourdescription
Dependencies: org.apache.ws.security,org.apache.cxf