Google Cloud Run - Can't send outbound requests - spring-boot

Question

I have deployed a container in Google Cloud Run. It's an app built in Spring boot and Java.
I have exposed an endpoint(rest controller) to Cloud run container which further invokes a REST api to hit an external endpoint and at that point it fails. I am hitting the exposed endpoint from Postman.
Container is able to listen to incoming requests and even able to connect to Cloud SQL. Problem is - it cant hit an external API.
I even added an outbound rule in firewall to allow all the traffic but no success. Later, tried to turn the logs ON for firewall but no logs were generated.
Would anyone have any idea what could be the issue here ?
Please see the logs below and exception
2021-09-30T14:33:06.955652Z2021-09-30 14:33:06.955 WARN 1 --- [nio-8080-exec-3] io.netty.util.internal.MacAddressUtil :
Failed to find a usable hardware address from the network interfaces; using random bytes: 61:71:ba:a4:d5:85:ee:11
021-09-30T14:33:08.094246Z2021-09-30 14:33:08.093 DEBUG 1 --- [or-http-epoll-2] r.netty.http.client.HttpClientConnect : [id:93c6dc9b-1, L:/169.... - R:www.homedepot.ca/23.211.51.167:443] Handler is being applied: {uri=https://www.homedepot.ca/ method=GET}
2021-09-30T15:13:41.887524Z org.springframework.web.reactive.function.client.WebClientRequestException: handshake timed out after 10000ms;
nested exception is io.netty.handler.ssl.SslHandshakeTimeoutException: handshake timed out after 10000ms at
org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141)
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
2021-09-30T15:13:41.887759Z at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:414)
2021-09-30T15:13:41.887776Z at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:251)
2021-09-30T15:13:41.887805Z at reactor.core.publisher.EmitterProcessor.drain(EmitterProcessor.java:491)
2021-09-30T15:13:41.887813Z at reactor.core.publisher.EmitterProcessor.tryEmitNext(EmitterProcessor.java:299)
2021-09-30T15:13:41.887822Z at reactor.core.publisher.SinkManySerialized.tryEmitNext(SinkManySerialized.java:100)
2021-09-30T15:13:41.887859Z at reactor.core.publisher.InternalManySink.emitNext(InternalManySink.java:27)
2021-09-30T15:13:41.887873Z at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:190)
2021-09-30T15:13:41.887893Z at reactor.core.publisher.MonoCreate$MonoSink.error(MonoCreate.java:189)
2021-09-30T15:13:41.887903Z at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect$ClientTransportSubscriber.onError(HttpClientConnect.java:304)
2021-09-30T15:13:41.887916Z at reactor.core.publisher.MonoCreate$MonoSink.error(MonoCreate.java:189)
2021-09-30T15:13:41.887934Z at reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.onUncaughtException(PooledConnectionProvider.java:218)
2021-09-30T15:13:41.887958Z at reactor.netty.resources.PooledConnectionProvider$PooledConnection.onUncaughtException(PooledConnectionProvider.java:467)
2021-09-30T15:13:41.887979Z at reactor.netty.channel.ChannelOperationsHandler.exceptionCaught(ChannelOperationsHandler.java:129)
2021-09-30T15:13:41.887993Z at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
2021-09-30T15:13:41.888011Z at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
2021-09-30T15:13:41.888032Z at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273)
2021-09-30T15:13:41.888049Z at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedChannelDuplexHandler.java:424)
2021-09-30T15:13:41.982305Z at io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:92)
2021-09-30T15:13:41.982894Z2021-09-30 15:13:41.281 WARN 1 --- [or-http-epoll-3] r.netty.http.client.HttpClientConnect : [id:c351aec9, L:/169.... - R:www.homedepot.ca/96.7.86.161:443] The connection observed an error
Code in Spring boot that is making the call
WebClient webClient =
WebClient.builder()
.defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
.build();
Mono<HDCanadaProdResp> webClientResponse =
webClient.get().uri(stockApiurl).retrieve().bodyToMono(HDCanadaProdResp.class);
webClientResponse.block();
All the code works in local system in a Docker Container, not sure what's the issue on Cloud run.
Here is the Dockerfile
FROM maven:3.6.3-openjdk-11-slim as builder
WORKDIR /app
COPY pom.xml .
RUN mvn dependency:go-offline
COPY src/ /app/src/
RUN mvn package -DskipTests
FROM openjdk:11-jre-slim
COPY --from=builder /app/target/*.jar /app.jar
#PORT for cloud run, coz cloud maps to this port
ENV PORT 8080
ENV HOST 0.0.0.0
CMD ["java", "-jar", "/app.jar"]
Update on Nov 7, 2021
I changed my spring Boot code where I was using WebClient API to make rest call and now the error has changed.
old code
WebClient webClient =
WebClient.builder()
.defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
.defaultHeader(HttpHeaders.ACCEPT_ENCODING, "gzip, deflate, br")
.defaultHeader(HttpHeaders.ACCEPT_LANGUAGE, "en-CA,en-US;q=0.7,en;q=0.3")
.defaultHeader(HttpHeaders.HOST, "www.homedepot.ca")
.build();
Mono<HDCanadaRecomProdResp> webClientResponse =
webClient.get().uri(prodApiurl).retrieve().bodyToMono(HDCanadaRecomProdResp.class);
return webClientResponse.block();
New code
HttpComponentsClientHttpRequestFactory requestFactory =
new HttpComponentsClientHttpRequestFactory();
requestFactory.setConnectTimeout(60 * 1000);
requestFactory.setReadTimeout(60 * 1000);
RestTemplate restTemplate = new RestTemplate(requestFactory);
ResponseEntity<HDCanadaRecomProdResp> response = null;
try {
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
headers.set(HttpHeaders.ACCEPT_ENCODING, "gzip, deflate, br");
headers.set(HttpHeaders.ACCEPT_LANGUAGE, "en-CA,en-US;q=0.7,en;q=0.3");
headers.set(HttpHeaders.HOST, "www.homedepot.ca");
headers.setAccept(Arrays.asList(MediaType.ALL));
HttpEntity<Map<String, String>> entity = new HttpEntity<>(headers);
response =
restTemplate.exchange(prodApiurl, HttpMethod.GET, entity, HDCanadaRecomProdResp.class);
}
And now the error has changed to
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://www.homedepot.ca/api/rec/v1/recommendations/fbt/products/1001200972/store/7301":
Read timed out; nested exception is java.net.SocketTimeoutException: Read timed out at
org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:785) at
org.springframework.web.client.RestTemplate.execute(RestTemplate.java:711) at
org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:602) at
com.pricetracker.productfetcher.client.HomeDepotCAClient.getProductDetails(HomeDepotCAClient.java:68) at
com.pricetracker.productfetcher.client.HomeDepotCAClient.getProductDetails(HomeDepotCAClient.java:24) at
com.pricetracker.productfetcher.service.ProductSearchServiceImpl.getProductFromWS(ProductSearchServiceImpl.java:116) at
com.pricetracker.productfetcher.service.ProductSearchServiceImpl.getProduct(ProductSearchServiceImpl.java:82) at
com.pricetracker.productfetcher.service.ProductSearchServiceImpl.productSearch(ProductSearchServiceImpl.java:61) at
com.pricetracker.productfetcher.controller.ProductsController.searchProduct(ProductsController.java:51) at
com.pricetracker.productfetcher.controller.ProductsController$$FastClassBySpringCGLIB$$37eced0e.invoke(<generated>) at
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:779) at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) at
org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:123) at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:692) at
com.pricetracker.productfetcher.controller.ProductsController$$EnhancerBySpringCGLIB$$ec439be9.searchProduct(<generated>) at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at
java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197) at
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141) at
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) at
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1064) at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) at

Answer 1

Use strace for this kind of Cloud Run related issues. Also view the logs in the Cloud Logging section of the GCP Console (not in the "Logs'' tab of the Cloud Run section), you can look for "Container Sandbox" with a "DEBUG" severity in the varlog/system logs. Also try running your application locally on Docker using these instructions and verify if your applications boots fine locally. The container must listen for requests on 0.0.0.0 on the port to which requests are sent. By default, requests are sent to 8080. Add the --min-instances in cloudbuild.yaml file and give it some value for now and then try.
If you see this log (which was not visible earlier, as it was coming during startup of container) Debug 2021-11-07T12:39:14.876726806ZContainer Sandbox: Unsupported syscall setsockopt(0xa,0x11,0x67,0x3ed0ac62bffc,0x4,0x1a) and you are able to run the setup locally, that means there is no issue with the docker image or your configurations. Unsupported syscall- This message indicates that system calls may not be supported in this Cloud Run (fully managed) as container instances are sandboxed using the gVisor container runtime sandbox.
See socket(7) - Linux manual page 8 for details. So gvisor does not support this socket option. Have a look at this Github issue which talks about the same error message you received.
It's a known issue in Cloud Run fully managed. You can try Cloud Run for Anthos with the same configurations and container setup (with less allocated resources).