I need help to understand why I can't connect to my container. Neither 5672 or 15672 ports will respond. Docker shows the container up and running but no communication whatsoever can reach it. I thought it was a firewall issue, ran these commands with no avail:
netsh advfirewall firewall add rule name="RabbitMQ Management" dir=in action=allow protocol=TCP localport=15672
netsh advfirewall firewall add rule name="RabbitMQ" dir=in action=allow protocol=TCP localport=5672
This is my command:
docker run -d -p 15672:15672 -p 5672:5672 -p 15692:15692 --name leap-rabbit rabbitmq:3.10.1-management-alpine
This is the startup response from the container:
leap-rabbitmq | 2022-05-20 17:27:52.859804+00:00 [info] <0.221.0> Running boot step cluster_name defined by app rabbit
leap-rabbitmq | 2022-05-20 17:27:52.860046+00:00 [info] <0.221.0> Initialising internal cluster ID to 'rabbitmq-cluster-id-ZJg5LIrVcEx8-VxjZ46hNw'
leap-rabbitmq | 2022-05-20 17:27:52.900450+00:00 [info] <0.221.0> Running boot step direct_client defined by app rabbit
leap-rabbitmq | 2022-05-20 17:27:52.900598+00:00 [info] <0.221.0> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management
leap-rabbitmq | 2022-05-20 17:27:52.900832+00:00 [info] <0.678.0> Resetting node maintenance status
leap-rabbitmq | 2022-05-20 17:27:52.917408+00:00 [info] <0.737.0> Management plugin: HTTP (non-TLS) listener started on port 15672
leap-rabbitmq | 2022-05-20 17:27:52.917548+00:00 [info] <0.765.0> Statistics database started.
leap-rabbitmq | 2022-05-20 17:27:52.917617+00:00 [info] <0.764.0> Starting worker pool 'management_worker_pool' with 3 processes in it
leap-rabbitmq | 2022-05-20 17:27:52.927564+00:00 [info] <0.779.0> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
leap-rabbitmq | 2022-05-20 17:27:52.927840+00:00 [info] <0.678.0> Ready to start client connection listeners
leap-rabbitmq | 2022-05-20 17:27:52.929580+00:00 [info] <0.823.0> started TCP listener on [::]:5672
leap-rabbitmq | completed with 4 plugins.
leap-rabbitmq | 2022-05-20 17:27:53.013224+00:00 [info] <0.678.0> Server startup complete; 4 plugins started.
leap-rabbitmq | 2022-05-20 17:27:53.013224+00:00 [info] <0.678.0> * rabbitmq_prometheus
leap-rabbitmq | 2022-05-20 17:27:53.013224+00:00 [info] <0.678.0> * rabbitmq_management
leap-rabbitmq | 2022-05-20 17:27:53.013224+00:00 [info] <0.678.0> * rabbitmq_web_dispatch
leap-rabbitmq | 2022-05-20 17:27:53.013224+00:00 [info] <0.678.0> * rabbitmq_management_agent
And this is the docker inspect output:
"HostConfig": {
"Binds": null,
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "default",
"PortBindings": {
"15672/tcp": [
{
"HostIp": "",
"HostPort": "15672"
}
],
"15692/tcp": [
{
"HostIp": "",
"HostPort": "15692"
}
],
"5672/tcp": [
{
"HostIp": "",
"HostPort": "5672"
}
]
},
Related
I have following deployments and service but still not able to access the service on my Windows machine with minikube installed on it.
# kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
helloworld-dep 3/3 3 3 9h
# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
greet-api-demo 1/1 Running 2 (23m ago) 6d9h 172.17.0.5 minikube <none> <none>
helloworld-dep-f8586dd84-28dxg 1/1 Running 1 (23m ago) 9h 172.17.0.6 minikube <none> <none>
helloworld-dep-f8586dd84-p4pg9 1/1 Running 1 (23m ago) 9h 172.17.0.7 minikube <none> <none>
helloworld-dep-f8586dd84-tgmcj 1/1 Running 1 (23m ago) 9h 172.17.0.8 minikube <none> <none>
# kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
helloworld-service NodePort 10.96.44.199 <none> 80:30008/TCP 4m1s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 191d
# kubectl logs helloworld-dep-f8586dd84-28dxg
Setting Active Processor Count to 8
Calculating JVM memory based on 11276448K available memory
For more information on this calculation, see https://paketo.io/docs/reference/java-reference/#memory-calculator
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx10889346K -XX:MaxMetaspaceSize=79901K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 11276448K, Thread Count: 50, Loaded Class Count: 11693, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 124 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=8 -XX:MaxDirectMemorySize=10M -Xmx10889346K -XX:MaxMetaspaceSize=79901K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v3.0.0)
2022-12-23T03:38:53.397Z INFO 1 --- [ main] com.playground.k8s.K8sDemoApplication : Starting K8sDemoApplication using Java 17.0.5 with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
2022-12-23T03:38:53.403Z INFO 1 --- [ main] com.playground.k8s.K8sDemoApplication : No active profile set, falling back to 1 default profile: "default"
2022-12-23T03:39:14.389Z INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2022-12-23T03:39:14.648Z INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2022-12-23T03:39:14.650Z INFO 1 --- [ main] o.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/10.1.1]
2022-12-23T03:39:15.883Z INFO 1 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2022-12-23T03:39:15.888Z INFO 1 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 21396 ms
2022-12-23T03:39:19.693Z INFO 1 --- [ main] o.s.b.a.w.s.WelcomePageHandlerMapping : Adding welcome page template: index
2022-12-23T03:39:23.699Z INFO 1 --- [ main] o.s.b.a.e.web.EndpointLinksResolver : Exposing 1 endpoint(s) beneath base path '/actuator'
2022-12-23T03:39:25.262Z INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2022-12-23T03:39:25.397Z INFO 1 --- [ main] com.playground.k8s.K8sDemoApplication : Started K8sDemoApplication in 35.178 seconds (process running for 38.144)
Tried minikube ip, clusterip and node ip but none of them worked !
# curl http://192.168.49.2:30008/actuator
curl: (28) Failed to connect to 192.168.49.2 port 30008 after 21061 ms: Timed out
# curl http://10.96.44.199:30008/actuator
curl: (28) Failed to connect to 10.96.44.199 port 30008 after 21034 ms: Timed out
# curl http://172.17.0.6:30008/actuator
curl: (28) Failed to connect to 172.17.0.6 port 30008 after 21052 ms: Timed out
Not sure from where you are truing to hit the curl but you will be only able to curl those IP if truing inside of Kubernetes cluster.
Cluster IP is internal, you will be able to access it internal or internal services can use it to access it.
You can run one container extra to test this
kubectl run -it --rm --image=curlimages/curl curly -- sh
after command try curl <clusterIP> or curl helloworld-service
If you want to try from outside from Host machine you can check URL with
minikube service helloworld-service --url
try once curl localhost:30008/actuator
For node port you can get minikube IP with command
minikube ip
curl MinikubeIP:30008
I am using Tesseract for Java for OCR text extraction, which makes JNI native calls to Leptonica system library.
When running a Quarkus application in JVM mode in a Linux-based host it resolves JNI method calls correctly,
and all works as expected. But when running as a native image, an UnsatisfiedLinkError exception is thrown when making a call to pixReadMem metohd, which is not found.
The stack trace when running the native image is:
2022-12-16 08:20:29,736 ERROR [org.jbo.res.rea.com.cor.AbstractResteasyReactiveContext] (executor-thread-0) Request failed: java.lang.UnsatisfiedLinkError: Error looking up function 'pixReadMem': com.sun.jna.Native.findSymbol(JLjava/lang/String;)J [symbol: Java_com_sun_jna_Native_findSymbol or Java_com_sun_jna_Native_findSymbol__JLjava_lang_String_2]
at com.sun.jna.Function.<init>(Function.java:252)
at com.sun.jna.NativeLibrary.getFunction(NativeLibrary.java:600)
at com.sun.jna.NativeLibrary.getFunction(NativeLibrary.java:576)
at com.sun.jna.NativeLibrary.getFunction(NativeLibrary.java:562)
at com.sun.jna.Library$Handler.invoke(Library.java:243)
at com.sun.proxy.$Proxy107.pixReadMem(Unknown Source)
at com.itextpdf.pdfocr.tesseract4.TesseractOcrUtil.readPix(TesseractOcrUtil.java:662)
at com.itextpdf.pdfocr.tesseract4.TesseractOcrUtil.readPix(TesseractOcrUtil.java:641)
at com.itextpdf.pdfocr.tesseract4.ImagePreprocessingUtil.preprocessImage(ImagePreprocessingUtil.java:166)
at com.itextpdf.pdfocr.tesseract4.Tesseract4LibOcrEngine.getOcrResultForSinglePage(Tesseract4LibOcrEngine.java:314)
at com.itextpdf.pdfocr.tesseract4.Tesseract4LibOcrEngine.doTesseractOcr(Tesseract4LibOcrEngine.java:189)
at com.itextpdf.pdfocr.tesseract4.AbstractTesseract4OcrEngine.processInputFiles(AbstractTesseract4OcrEngine.java:494)
at com.itextpdf.pdfocr.tesseract4.AbstractTesseract4OcrEngine.doImageOcr(AbstractTesseract4OcrEngine.java:232)
at es.gen.quarkus.ocr.service.TesseractService.extratTextFromJPG(TesseractService.java:42)
at es.gen.quarkus.ocr.service.TesseractService_ClientProxy.extratTextFromJPG(Unknown Source)
at es.gen.quarkus.ocr.service.OCRService.extractTextFromPDF(OCRService.java:34)
at es.gen.quarkus.ocr.service.OCRService_ClientProxy.extractTextFromPDF(Unknown Source)
at es.gen.quarkus.ocr.resource.OCRResource.extractTextFromPDF(OCRResource.java:46)
at es.gen.quarkus.ocr.resource.OCRResource$quarkusrestinvoker$extractTextFromPDF_21209ffb320090ffe15503899b13ececc74cb601.invoke(Unknown Source)
at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:114)
at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:145)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:576)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base#11.0.17/java.lang.Thread.run(Thread.java:829)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:775)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:203)`
```
I have tried to set Java system properties with no success:
- `java.library.path` (the current value includes the linux system folders).
- `jna.debug_load=true` does not log any information.
Add this to properties
quarkus.native.additional-build-args=--initialize-at-build-time=java.awt.image.DirectColorModel --trace-class-initialization=sun.awt.image.IntegerInterleavedRaster\\,java.awt.image.DataBufferByte\\,java.awt.Rectangle\\,java.awt.image.SinglePixelPackedSampleModel\\,sun.java2d.StateTrackableDelegate$2\\,java.awt.image.PackedColorModel\\,java.awt.image.DirectColorModel\\,java.awt.image.BufferedImage\\,java.awt.image.DataBuffer\\,java.awt.Toolkit\\,java.awt.color.ColorSpace$BuiltInSpace\\,sun.awt.image.IntegerComponentRaster\\,sun.awt.image.SunWritableRaster\\,java.awt.image.ComponentSampleModel\\,java.awt.image.ColorModel\\,java.awt.image.BandedSampleModel\\,java.awt.image.WritableRaster\\,java.awt.Image\\,java.awt.image.SampleModel\\,java.awt.image.Raster\\,java.awt.image.DataBufferInt\\,sun.java2d.StateTrackableDelegate\\,sun.awt.image.ByteBandedRaster
install libtesseract5 because latest code uses that. this is how I installed it
sudo add-apt-repository ppa:alex-p/tesseract-ocr5
sudo apt update
sudo apt install libtesseract5
then I manage to get the runner executable. no need to run install just mvn package -Pnative will do fine.
When I ran in my local and send a pdf, got another exception.
/quarkus-ocr$ ./target/quarkus-ocr-1.0.0-SNAPSHOT-runner
__ ____ __ _____ ___ __ ____ ______
--/ __ \/ / / / _ | / _ \/ //_/ / / / __/
-/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/
2022-12-20 19:09:30,419 INFO [com.zhe.qua.ocr.res.OCRResource] (executor-thread-0) Extracting text from PDF: Profile.pdf (58739 bytes)
java.io.IOException: Error: Could not find referenced cmap stream Identity-H
at org.apache.fontbox.cmap.CMapParser.getExternalCMap(CMapParser.java:492)
at org.apache.fontbox.cmap.CMapParser.parsePredefined(CMapParser.java:99)
at org.apache.pdfbox.pdmodel.font.CMapManager.getPredefinedCMap(CMapManager.java:55)
at org.apache.pdfbox.pdmodel.font.PDType0Font.readEncoding(PDType0Font.java:287)
at org.apache.pdfbox.pdmodel.font.PDType0Font.<init>(PDType0Font.java:204)
at org.apache.pdfbox.pdmodel.font.PDFontFactory.createFont(PDFontFactory.java:97)
at org.apache.pdfbox.pdmodel.PDResources.getFont(PDResources.java:146)
at org.apache.pdfbox.contentstream.operator.text.SetFontAndSize.process(SetFontAndSize.java:66)
at org.apache.pdfbox.contentstream.PDFStreamEngine.processOperator(PDFStreamEngine.java:966)
at org.apache.pdfbox.contentstream.PDFStreamEngine.processStreamOperators(PDFStreamEngine.java:541)
at org.apache.pdfbox.contentstream.PDFStreamEngine.processStream(PDFStreamEngine.java:516)
at org.apache.pdfbox.contentstream.PDFStreamEngine.processPage(PDFStreamEngine.java:155)
at org.apache.pdfbox.text.LegacyPDFStreamEngine.processPage(LegacyPDFStreamEngine.java:155)
at org.apache.pdfbox.text.PDFTextStripper.processPage(PDFTextStripper.java:363)
at org.apache.pdfbox.text.PDFTextStripper.processPages(PDFTextStripper.java:291)
at org.apache.pdfbox.text.PDFTextStripper.writeText(PDFTextStripper.java:238)
at org.apache.pdfbox.text.PDFTextStripper.getText(PDFTextStripper.java:202)
at com.zheta.quarkus.ocr.service.OCRService.extractTextFromPDF(OCRService.java:30)
at com.zheta.quarkus.ocr.service.OCRService_ClientProxy.extractTextFromPDF(Unknown Source)
at com.zheta.quarkus.ocr.resource.OCRResource.extractTextFromPDF(OCRResource.java:43)
at com.zheta.quarkus.ocr.resource.OCRResource$quarkusrestinvoker$extractTextFromPDF_924fb8ae275125151f52704028fdf25205ebf12f.invoke(Unknown Source)
at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:114)
at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:145)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:576)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base#17.0.5/java.lang.Thread.run(Thread.java:833)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:775)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:203)
2022-12-20 19:09:30,429 INFO [com.zhe.qua.ocr.res.OCRResource] (executor-thread-0) Extracted text from PDF: Profile.pdf (58739 bytes) finished in 10 ms
build log to show versions
mintozzy#laptop:~/tmp/quarkus-ocrmvn clean package -Pnativer
[INFO] Scanning for projects...
[INFO]
[INFO] -------------------< com.zheta.quarkus:quarkus-ocr >--------------------
[INFO] Building quarkus-ocr 1.0.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) # quarkus-ocr ---
[INFO] Deleting /home/mintozzy/tmp/quarkus-ocr/target
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) # quarkus-ocr ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO]
[INFO] --- quarkus-maven-plugin:2.14.3.Final:generate-code (default) # quarkus-ocr ---
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) # quarkus-ocr ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 7 source files to /home/mintozzy/tmp/quarkus-ocr/target/classes
[INFO]
[INFO] --- quarkus-maven-plugin:2.14.3.Final:generate-code-tests (default) # quarkus-ocr ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) # quarkus-ocr ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/mintozzy/tmp/quarkus-ocr/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) # quarkus-ocr ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:3.0.0-M7:test (default-test) # quarkus-ocr ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) # quarkus-ocr ---
[INFO] Building jar: /home/mintozzy/tmp/quarkus-ocr/target/quarkus-ocr-1.0.0-SNAPSHOT.jar
[INFO]
[INFO] --- quarkus-maven-plugin:2.14.3.Final:build (default) # quarkus-ocr ---
[INFO] [io.quarkus.deployment.pkg.steps.JarResultBuildStep] Building native image source jar: /home/mintozzy/tmp/quarkus-ocr/target/quarkus-ocr-1.0.0-SNAPSHOT-native-image-source-jar/quarkus-ocr-1.0.0-SNAPSHOT-runner.jar
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Building native image from /home/mintozzy/tmp/quarkus-ocr/target/quarkus-ocr-1.0.0-SNAPSHOT-native-image-source-jar/quarkus-ocr-1.0.0-SNAPSHOT-runner.jar
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on GraalVM 22.3.0 Java 17 EE (Java Version 17.0.5+9-LTS-jvmci-22.3-b07)
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] /home/mintozzy/.sdkman/candidates/java/graalvm-ee-java17-22.3.0/bin/native-image -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dlogging.initial-configurator.min-level=500 -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Duser.language=en -J-Duser.country=GB -J-Dfile.encoding=UTF-8 --features=io.quarkus.runner.Feature,io.quarkus.runtime.graal.ResourcesFeature,io.quarkus.runtime.graal.DisableLoggingFeature,io.quarkus.awt.runtime.graal.AwtFeature,io.quarkus.awt.runtime.graal.DarwinAwtFeature -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -J--add-opens=java.base/java.io=ALL-UNNAMED -J--add-opens=java.base/java.lang.invoke=ALL-UNNAMED -J--add-opens=java.base/java.util=ALL-UNNAMED -H:+CollectImageBuildStatistics -H:ImageBuildStatisticsFile=quarkus-ocr-1.0.0-SNAPSHOT-runner-timing-stats.json -H:BuildOutputJSONFile=quarkus-ocr-1.0.0-SNAPSHOT-runner-build-output-stats.json --initialize-at-build-time=java.awt.image.DirectColorModel --trace-class-initialization=sun.awt.image.IntegerInterleavedRaster,java.awt.image.DataBufferByte,java.awt.Rectangle,java.awt.image.SinglePixelPackedSampleModel,sun.java2d.StateTrackableDelegate\$2,java.awt.image.PackedColorModel,java.awt.image.DirectColorModel,java.awt.image.BufferedImage,java.awt.image.DataBuffer,java.awt.Toolkit,java.awt.color.ColorSpace\$BuiltInSpace,sun.awt.image.IntegerComponentRaster,sun.awt.image.SunWritableRaster,java.awt.image.ComponentSampleModel,java.awt.image.ColorModel,java.awt.image.BandedSampleModel,java.awt.image.WritableRaster,java.awt.Image,java.awt.image.SampleModel,java.awt.image.Raster,java.awt.image.DataBufferInt,sun.java2d.StateTrackableDelegate,sun.awt.image.ByteBandedRaster -H:+AllowFoldMethods -J-Djava.awt.headless=true --no-fallback --link-at-build-time -H:+ReportExceptionStackTraces -H:-AddAllCharsets --enable-url-protocols=http -H:NativeLinkerOption=-no-pie -H:-UseServiceLoaderFeature -H:+StackTrace -J--add-exports=org.graalvm.sdk/org.graalvm.nativeimage.impl=ALL-UNNAMED -J--add-exports=org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk=ALL-UNNAMED quarkus-ocr-1.0.0-SNAPSHOT-runner -jar quarkus-ocr-1.0.0-SNAPSHOT-runner.jar
========================================================================================================================
GraalVM Native Image: Generating 'quarkus-ocr-1.0.0-SNAPSHOT-runner' (executable)...
========================================================================================================================
[1/7] Initializing... (11.3s # 0.19GB)
Version info: 'GraalVM 22.3.0 Java 17 EE'
Java version info: '17.0.5+9-LTS-jvmci-22.3-b07'
C compiler: gcc (linux, x86_64, 9.4.0)
Garbage collector: Serial GC
5 user-specific feature(s)
- io.quarkus.awt.runtime.graal.AwtFeature
- io.quarkus.awt.runtime.graal.DarwinAwtFeature
- io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
- io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase for the [org.jboss.threads] categories
- io.quarkus.runtime.graal.ResourcesFeature: Register each line in META-INF/quarkus-native-resources.txt as a resource on Substrate VM
# Printing class initialization configuration to: /home/mintozzy/tmp/quarkus-ocr/target/quarkus-ocr-1.0.0-SNAPSHOT-native-image-source-jar/reports/class_initialization_configuration_20221220_193027.csv
# Printing class initialization configuration to: /home/mintozzy/tmp/quarkus-ocr/target/quarkus-ocr-1.0.0-SNAPSHOT-native-image-source-jar/reports/class_initialization_configuration_20221220_193129.csv
# Printing class initialization dependencies to: /home/mintozzy/tmp/quarkus-ocr/target/quarkus-ocr-1.0.0-SNAPSHOT-native-image-source-jar/reports/class_initialization_dependencies_20221220_193205.dot
# Printing class initialization report to: /home/mintozzy/tmp/quarkus-ocr/target/quarkus-ocr-1.0.0-SNAPSHOT-native-image-source-jar/reports/class_initialization_report_20221220_193205.csv
[2/7] Performing analysis... [*********] (98.7s # 2.15GB)
15,243 (88.71%) of 17,182 classes reachable
25,842 (62.65%) of 41,250 fields reachable
88,340 (59.44%) of 148,629 methods reachable
516 classes, 118 fields, and 2,372 methods registered for reflection
179 classes, 1,537 fields, and 2,092 methods registered for JNI access
7 native libraries: dl, freetype, m, pthread, rt, stdc++, z
[3/7] Building universe... (9.6s # 2.71GB)
[4/7] Parsing methods... [***] (11.5s # 2.63GB)
[5/7] Inlining methods... [***] (4.4s # 1.94GB)
[6/7] Compiling methods... [**************] (214.3s # 1.47GB)
[7/7] Creating image... (13.4s # 2.06GB)
46.92MB (44.56%) for code area: 51,972 compilation units
55.45MB (52.66%) for image heap:1,114,462 objects and 302 resources
2.93MB ( 2.78%) for other data
105.29MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 packages in code area: Top 10 object types in image heap:
3.17MB com.oracle.svm.core.code 12.28MB o.a.c.imaging.common.itu_t4.HuffmanTree$Node
1.66MB sun.security.ssl 9.53MB byte[] for code metadata
1.37MB java.util 4.56MB byte[] for general heap data
1.12MB com.oracle.svm.core.jni 4.53MB byte[] for embedded resources
986.77KB sun.font 3.77MB byte[] for java.lang.String
971.69KB io.netty.buffer 3.75MB java.lang.String
914.40KB com.sun.crypto.provider 2.74MB java.lang.Class
881.87KB java.lang.invoke 2.48MB java.lang.Object[]
718.86KB java.lang 1.60MB int[]
641.56KB io.vertx.core.http.impl 1.58MB java.util.HashMap$Node
34.23MB for 623 more packages 7.67MB for 3524 more object types
------------------------------------------------------------------------------------------------------------------------
16.3s (4.3% of total time) in 146 GCs | Peak RSS: 4.75GB | CPU load: 6.64
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
/home/mintozzy/tmp/quarkus-ocr/target/quarkus-ocr-1.0.0-SNAPSHOT-native-image-source-jar/quarkus-ocr-1.0.0-SNAPSHOT-runner (executable)
/home/mintozzy/tmp/quarkus-ocr/target/quarkus-ocr-1.0.0-SNAPSHOT-native-image-source-jar/quarkus-ocr-1.0.0-SNAPSHOT-runner-build-output-stats.json (json)
/home/mintozzy/tmp/quarkus-ocr/target/quarkus-ocr-1.0.0-SNAPSHOT-native-image-source-jar/quarkus-ocr-1.0.0-SNAPSHOT-runner-timing-stats.json (raw)
/home/mintozzy/tmp/quarkus-ocr/target/quarkus-ocr-1.0.0-SNAPSHOT-native-image-source-jar/quarkus-ocr-1.0.0-SNAPSHOT-runner.build_artifacts.txt (txt)
========================================================================================================================
Finished generating 'quarkus-ocr-1.0.0-SNAPSHOT-runner' in 6m 14s.
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] objcopy --strip-debug quarkus-ocr-1.0.0-SNAPSHOT-runner
[INFO] [io.quarkus.deployment.QuarkusAugmentor] Quarkus augmentation completed in 381602ms
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 06:25 min
[INFO] Finished at: 2022-12-20T19:36:34Z
[INFO] ------------------------------------------------------------------------
what libraries it depends
mintozzy#laptop:~/tmp/quarkus-ocr$ ldd ./target/quarkus-ocr-1.0.0-SNAPSHOT-runner
linux-vdso.so.1 (0x00007fff0f5ff000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fdbd2522000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fdbd23d3000)
libfreetype.so.6 => /lib/x86_64-linux-gnu/libfreetype.so.6 (0x00007fdbd2314000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fdbd22f8000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fdbd22d5000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fdbd22cf000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fdbd22b2000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdbd20c0000)
/lib64/ld-linux-x86-64.so.2 (0x00007fdbd2720000)
libpng16.so.16 => /lib/x86_64-linux-gnu/libpng16.so.16 (0x00007fdbd2088000)
I am unable to shutdown my activemq gracefully after enabling jmx. Please help and tell me what am I doing wrong. Here is what I am trying to do.
start activemq:-
[mwapp#JMNGD1BAO150V02 ~]$ /app/apache-activemq-5.14.0/bin/activemq start xbean:/app/apache-activemq-5.14.0/conf/activemq-security.xml
INFO: Loading '/app/apache-activemq-5.14.0//bin/env'
INFO: Using java '/usr/java/jre1.7.0_79//bin/java'
INFO: Starting - inspect logfiles specified in logging.properties and log4j.properties to get details
INFO: pidfile created : '/app/apache-activemq-5.14.0//data/activemq.pid' (pid '16917')
activemq.log:- To me it's looking fine
2017-10-12 13:48:18,936 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1#2142b533: startup date [Thu Oct 12 13:48:18 IST 2017]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2017-10-12 13:48:20,008 | INFO | Loading properties file from URL [file:/app/apache-activemq-5.14.0//conf/credentials-enc.properties] | org.jasypt.spring31.properties.EncryptablePropertyPlaceholderConfigurer | main
2017-10-12 13:48:20,975 | INFO | Loaded the Bouncy Castle security provider. | org.apache.activemq.broker.BrokerService | main
2017-10-12 13:48:21,283 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[/jms_nas/kahadb] | org.apache.activemq.broker.BrokerService | main
2017-10-12 13:48:21,308 | INFO | JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi | org.apache.activemq.broker.jmx.ManagementContext | JMX connector
2017-10-12 13:48:21,594 | INFO | KahaDB is version 6 | org.apache.activemq.store.kahadb.MessageDatabase | main
2017-10-12 13:48:21,653 | INFO | Recovering from the journal #1105:27118028 | org.apache.activemq.store.kahadb.MessageDatabase | main
2017-10-12 13:48:21,657 | INFO | Recovery replayed 58 operations from the journal in 0.046 seconds. | org.apache.activemq.store.kahadb.MessageDatabase | main
2017-10-12 13:48:21,719 | INFO | PListStore:[/app/apache-activemq-5.14.0/data/localhost/tmp_storage] started | org.apache.activemq.store.kahadb.plist.PListStoreImpl | main
2017-10-12 13:48:21,903 | INFO | Apache ActiveMQ 5.14.0 (localhost, ID:JMNGD1BAO150V02-59661-1507796301746-0:1) is starting | org.apache.activemq.broker.BrokerService | main
2017-10-12 13:48:22,786 | INFO | Listening for connections at: ssl://JMNGD1BAO150V02:61616?needClientAuth=true&maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.TransportServerThreadSupport | main
2017-10-12 13:48:22,787 | INFO | Connector ssl started | org.apache.activemq.broker.TransportConnector | main
2017-10-12 13:48:22,787 | INFO | Apache ActiveMQ 5.14.0 (localhost, ID:JMNGD1BAO150V02-59661-1507796301746-0:1) started | org.apache.activemq.broker.BrokerService | main
2017-10-12 13:48:22,787 | INFO | For help or more information please see: http://activemq.apache.org | org.apache.activemq.broker.BrokerService | main
2017-10-12 13:48:22,788 | WARN | Store limit is 102400 mb (current store usage is 1397 mb). The data directory: /jms_nas/kahadb only has 91534 mb of usable space. - resetting to maximum available disk space: 91534 mb | org.apache.activemq.broker.BrokerService | main
2017-10-12 13:48:23,646 | INFO | No Spring WebApplicationInitializer types detected on classpath | /admin | main
2017-10-12 13:48:23,755 | INFO | ActiveMQ WebConsole available at http://localhost:8161/ | org.apache.activemq.web.WebConsoleStarter | main
2017-10-12 13:48:23,755 | INFO | ActiveMQ Jolokia REST API available at http://localhost:8161/api/jolokia/ | org.apache.activemq.web.WebConsoleStarter | main
2017-10-12 13:48:23,799 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
2017-10-12 13:48:24,068 | INFO | No Spring WebApplicationInitializer types detected on classpath | /api | main
2017-10-12 13:48:24,185 | INFO | jolokia-agent: Using policy access restrictor classpath:/jolokia-access.xml | /api | main
stop activemq:-
[mwapp#JMNGD1BAO150V02 ~]$ /app/apache-activemq-5.14.0/bin/activemq stop xbean:/app/apache-activemq-5.14.0/conf/activemq-security.xml
INFO: Loading '/app/apache-activemq-5.14.0//bin/env'
INFO: Using java '/usr/java/jre1.7.0_79//bin/java'
INFO: Waiting at least 30 seconds for regular process termination of pid '16917' :
Java Runtime: Oracle Corporation 1.7.0_79 /usr/java/jre1.7.0_79
Heap sizes: current=63488k free=61608k max=932352k
JVM args: -Xms64M -Xmx1G -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=/app/apache-activemq-5.14.0//conf/login.config -Dactivemq.classpath=/app/apache-activemq-5.14.0//conf:/app/apache-activemq-5.14.0//../lib/: -Dactivemq.home=/app/apache-activemq-5.14.0/ -Dactivemq.base=/app/apache-activemq-5.14.0/ -Dactivemq.conf=/app/apache-activemq-5.14.0//conf -Dactivemq.data=/app/apache-activemq-5.14.0//data
Extensions classpath:
[/app/apache-activemq-5.14.0/lib,/app/apache-activemq-5.14.0/lib/camel,/app/apache-activemq-5.14.0/lib/optional,/app/apache-activemq-5.14.0/lib/web,/app/apache-activemq-5.14.0/lib/extra]
ACTIVEMQ_HOME: /app/apache-activemq-5.14.0
ACTIVEMQ_BASE: /app/apache-activemq-5.14.0
ACTIVEMQ_CONF: /app/apache-activemq-5.14.0/conf
ACTIVEMQ_DATA: /app/apache-activemq-5.14.0/data
Connecting to JMX URL: service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
ERROR: java.lang.NullPointerException
java.lang.NullPointerException
at org.apache.activemq.console.command.AbstractCommand.handleException(AbstractCommand.java:167)
at org.apache.activemq.console.command.AbstractJmxCommand.execute(AbstractJmxCommand.java:390)
at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)
at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)
at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
at org.apache.activemq.console.Main.main(Main.java:115)
...............................
INFO: Regular shutdown not successful, sending SIGKILL to process
INFO: sending SIGKILL to pid '16917'
As you can see, the system is forcefully shutting down using pid which is not expected. I am currently using apache-activemq-5.14.0 and the configuration file will look something like below. I am not sure why activemq gave two separate file to enable JMX i.e. env and activemq-security.xml. Or the env file has some different role to play. I read the documentation, and I got more confuse when they mention from V.5.12.0 onwards they are supporting OCSP. Do I need to enable that too?
${ACTIVEMQ_HOME}/bin/env
#!/bin/sh
# Active MQ installation dirs
# ACTIVEMQ_HOME="<Installationdir>/"
# ACTIVEMQ_BASE="$ACTIVEMQ_HOME"
# ACTIVEMQ_CONF="$ACTIVEMQ_BASE/conf"
# ACTIVEMQ_DATA="$ACTIVEMQ_BASE/data"
# ACTIVEMQ_TMP="$ACTIVEMQ_BASE/tmp"
ACTIVEMQ_OPTS_MEMORY="-Xms64M -Xmx1G"
if [ -z "$ACTIVEMQ_OPTS" ] ; then
ACTIVEMQ_OPTS="$ACTIVEMQ_OPTS_MEMORY -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=$ACTIVEMQ_CONF/login.config"
fi
#ACTIVEMQ_OPTS="$ACTIVEMQ_OPTS -Dorg.apache.activemq.audit=true"
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.port=1099"
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_CONF}/jmx.password"
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_CONF}/jmx.access"
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.ssl=true"
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote"
#ACTIVEMQ_SUNJMX_CONTROL="--jmxurl service:jmx:rmi:///jndi/rmi://127.0.0.1:1099/jmxrmi --jmxuser controlRole --jmxpassword abcd1234"
ACTIVEMQ_SUNJMX_CONTROL=""
if [ -z "$ACTIVEMQ_QUEUEMANAGERURL" ]; then
ACTIVEMQ_QUEUEMANAGERURL="--amqurl tcp://localhost:61616"
fi
if [ -z "$ACTIVEMQ_SSL_OPTS" ] ; then
#ACTIVEMQ_SSL_OPTS="-Djava.security.properties=$ACTIVEMQ_CONF/java.security"
ACTIVEMQ_SSL_OPTS=""
fi
#ACTIVEMQ_DEBUG_OPTS="-Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005"
if [ -z "$ACTIVEMQ_KILL_MAXSECONDS" ]; then
ACTIVEMQ_KILL_MAXSECONDS=30
fi
ACTIVEMQ_USER=""
# ACTIVEMQ_PIDFILE="$ACTIVEMQ_DATA/activemq.pid"
JAVA_HOME="/usr/java/jre1.7.0_79/"
${ACTIVEMQ_HOME}/conf/activemq-security.xml
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.data}">
----------------------------------------------
<managementContext>
<!--managementContext createConnector="true" connectorPort="1099"/-->
<managementContext createConnector="true">
<property xmlns="http://www.springframework.org/schema/beans" name="environment">
<map xmlns="http://www.springframework.org/schema/beans">
<entry xmlns="http://www.springframework.org/schema/beans" key="jmx.remote.x.password.file" value="${activemq.base}/conf/jmx.password"/>
<entry xmlns="http://www.springframework.org/schema/beans" key="jmx.remote.x.access.file" value="${activemq.base}/conf/jmx.access"/>
</map>
</property>
</managementContext>
</managementContext>
----------------------------------------------
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook"/>
</shutdownHooks>
</broker>
We tried to submit a simple SparkPI example onto Spark on Yarn. The bat is written as below:
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --num-executors 3 --driver-memory 4g --executor-memory 1g --executor-cores 1 .\examples\target\spark-examples_2.10-1.4.0.jar 10
pause
Our HDFS and Yarn works well. We are using Hadoop 2.7.0 and Spark 1.4.1. We have only 1 node that acts as both NameNode and DataNode.
When we execute it, it fails with log says the following:
2015-08-21 11:07:22,044 DEBUG [main] | ===============================================================================
2015-08-21 11:07:22,044 DEBUG [main] | Yarn AM launch context:
2015-08-21 11:07:22,044 DEBUG [main] | user class: org.apache.spark.examples.SparkPi
2015-08-21 11:07:22,044 DEBUG [main] | env:
2015-08-21 11:07:22,044 DEBUG [main] | CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__hadoop_conf__<CPS>{{PWD}}/__spark__.jar<CPS>%HADOOP_HOME%\etc\hadoop<CPS>%HADOOP_HOME%\share\hadoop\common\*<CPS>%HADOOP_HOME%\share\hadoop\common\lib\*<CPS>%HADOOP_HOME%\share\hadoop\mapreduce\*<CPS>%HADOOP_HOME%\share\hadoop\mapreduce\lib\*<CPS>%HADOOP_HOME%\share\hadoop\hdfs\*<CPS>%HADOOP_HOME%\share\hadoop\hdfs\lib\*<CPS>%HADOOP_HOME%\share\hadoop\yarn\*<CPS>%HADOOP_HOME%\share\hadoop\yarn\lib\*<CPS>%HADOOP_MAPRED_HOME%\share\hadoop\mapreduce\*<CPS>%HADOOP_MAPRED_HOME%\share\hadoop\mapreduce\lib\*
2015-08-21 11:07:22,060 DEBUG [main] | SPARK_YARN_CACHE_FILES_FILE_SIZES -> 165181064,1420218
2015-08-21 11:07:22,060 DEBUG [main] | SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1440062075415_0026
2015-08-21 11:07:22,060 DEBUG [main] | SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE
2015-08-21 11:07:22,060 DEBUG [main] | SPARK_USER -> msrabi
2015-08-21 11:07:22,060 DEBUG [main] | SPARK_YARN_MODE -> true
2015-08-21 11:07:22,060 DEBUG [main] | SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1440126441200,1440126441575
2015-08-21 11:07:22,060 DEBUG [main] | SPARK_YARN_CACHE_FILES -> hdfs://msra-sa-44:9000/user/msrabi/.sparkStaging/application_1440062075415_0026/spark-assembly-1.4.0-hadoop2.7.0.jar#__spark__.jar,hdfs://msra-sa-44:9000/user/msrabi/.sparkStaging/application_1440062075415_0026/spark-examples_2.10-1.4.0.jar#__app__.jar
2015-08-21 11:07:22,060 DEBUG [main] | resources:
2015-08-21 11:07:22,060 DEBUG [main] | __app__.jar -> resource { scheme: "hdfs" host: "msra-sa-44" port: 9000 file: "/user/msrabi/.sparkStaging/application_1440062075415_0026/spark-examples_2.10-1.4.0.jar" } size: 1420218 timestamp: 1440126441575 type: FILE visibility: PRIVATE
2015-08-21 11:07:22,060 DEBUG [main] | __spark__.jar -> resource { scheme: "hdfs" host: "msra-sa-44" port: 9000 file: "/user/msrabi/.sparkStaging/application_1440062075415_0026/spark-assembly-1.4.0-hadoop2.7.0.jar" } size: 165181064 timestamp: 1440126441200 type: FILE visibility: PRIVATE
2015-08-21 11:07:22,060 DEBUG [main] | __hadoop_conf__ -> resource { scheme: "hdfs" host: "msra-sa-44" port: 9000 file: "/user/msrabi/.sparkStaging/application_1440062075415_0026/__hadoop_conf__7908628615251032149.zip" } size: 82888 timestamp: 1440126441794 type: ARCHIVE visibility: PRIVATE
2015-08-21 11:07:22,060 DEBUG [main] | command:
2015-08-21 11:07:22,075 DEBUG [main] | {{JAVA_HOME}}/bin/java -server -Xmx4096m -Djava.io.tmpdir={{PWD}}/tmp '-Dspark.app.name=org.apache.spark.examples.SparkPi' '-Dspark.executor.memory=1g' '-Dspark.driver.memory=4g' '-Dspark.master=yarn-cluster' -Dspark.yarn.app.container.log.dir=<LOG_DIR> org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.SparkPi' --jar file:/D:/sp/./examples/target/spark-examples_2.10-1.4.0.jar --arg '10' --executor-memory 1024m --executor-cores 1 --num-executors 3 1> <LOG_DIR>/stdout 2> <LOG_DIR>/stderr
2015-08-21 11:07:22,075 DEBUG [main] | ===============================================================================
...........(omitting some lines)......
2015-08-21 11:07:23,231 INFO [main] | Application report for application_1440062075415_0026 (state: ACCEPTED)
2015-08-21 11:07:23,247 DEBUG [main] |
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1440126442169
final status: UNDEFINED
tracking URL: http://msra-sa-44:8088/proxy/application_1440062075415_0026/
user: msrabi
2015-08-21 11:07:24,263 TRACE [main] | 1: Call -> MSRA-SA-44/10.190.173.181:8032: getApplicationReport {application_id { id: 26 cluster_timestamp: 1440062075415 }}
2015-08-21 11:07:24,263 DEBUG [IPC Parameter Sending Thread #0] | IPC Client (443384617) connection to MSRA-SA-44/10.190.173.181:8032 from msrabi sending #37
2015-08-21 11:07:24,263 DEBUG [IPC Client (443384617) connection to MSRA-SA-44/10.190.173.181:8032 from msrabi] | IPC Client (443384617) connection to MSRA-SA-44/10.190.173.181:8032 from msrabi got value #37
2015-08-21 11:07:24,263 DEBUG [main] | Call: getApplicationReport took 0ms
2015-08-21 11:07:24,263 TRACE [main] | 1: Response <- MSRA-SA-44/10.190.173.181:8032: getApplicationReport {application_report { applicationId { id: 26 cluster_timestamp: 1440062075415 } user: "msrabi" queue: "default" name: "org.apache.spark.examples.SparkPi" host: "N/A" rpc_port: -1 yarn_application_state: ACCEPTED trackingUrl: "http://msra-sa-44:8088/proxy/application_1440062075415_0026/" diagnostics: "" startTime: 1440126442169 finishTime: 0 final_application_status: APP_UNDEFINED app_resource_Usage { num_used_containers: 1 num_reserved_containers: 0 used_resources { memory: 4608 virtual_cores: 1 } reserved_resources { memory: 0 virtual_cores: 0 } needed_resources { memory: 4608 virtual_cores: 1 } memory_seconds: 0 vcore_seconds: 0 } originalTrackingUrl: "N/A" currentApplicationAttemptId { application_id { id: 26 cluster_timestamp: 1440062075415 } attemptId: 1 } progress: 0.0 applicationType: "SPARK" }}
2015-08-21 11:07:24,263 INFO [main] | Application report for application_1440062075415_0026 (state: ACCEPTED)
.......(omitting some lines where the state are all ACCEPTED and final status are all UNDEFINED).....
2015-08-21 11:07:30,359 INFO [main] | Application report for application_1440062075415_0026 (state: FAILED)
2015-08-21 11:07:30,359 DEBUG [main] |
client token: N/A
diagnostics: Application application_1440062075415_0026 failed 2 times due to AM Container for appattempt_1440062075415_0026_000002 exited with exitCode: 1
For more detailed output, check application tracking page:http://msra-sa-44:8088/cluster/app/application_1440062075415_0026Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1440062075415_0026_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Shell output: 1 file(s) moved.
And then we opened stderr, it says:
Error: Could not find or load main class 'Dspark.app.name=org.apache.spark.examples.SparkPi'
It's so strange, this should be a parameter passed to java, and it seems that java recognized it as the main class. There should be a main class parameter in the command section of the log, but there is not.
How can that happen? What should we do to know what's wrong with it?
Thank you!
We solved this problem.
The root cause is that when generating the java command line, our Spark uses single quote('-Dxxxx') to wrap the parameters. Single quote works only in Linux. On Windows, the parameters are either not wrapped, or wrapped with double quotes("-Dxxxx"). The only way to solve this is to edit the source code of Spark and re-compile it.
It seems that this is currently an issue of Spark. (https://issues.apache.org/jira/browse/SPARK-5754)
Preface: I have tried for several days to solve this problem on my own before bothering any of you nice people. Apologies in advance.
My problem: Tests hang for my custom source module as well as the spring-xd-samples project rss-feed-source.
I am running spring-xd 1.1.1.RELEASE on Java 1.7 OSX 10.9.5 and see the same result with mvn in the terminal as well as in Spring Tool Suite. It may be worth mentioning that I have no issue running spring's xd-amdin, xd-container, xd-singlenode, or xd-shell executables.
FeedSourceModuleIntegrationTest is where the several second hang occurs.
/**
* Start the single node container, binding random unused ports, etc. to not conflict with any other instances
* running on this host. Configure the ModuleRegistry to include the project module.
*/
#BeforeClass
public static void setUp() {
RandomConfigurationSupport randomConfigSupport = new RandomConfigurationSupport();
application = new SingleNodeApplication().run(); // Hang occurs here!
SingleNodeIntegrationTestSupport singleNodeIntegrationTestSupport = new SingleNodeIntegrationTestSupport(application);
singleNodeIntegrationTestSupport.addModuleRegistry(new SingletonModuleRegistry(ModuleType.source, "feed"));
}
FeedConfigurationTest Works just fine!
Here is the terminal output:
(may need to scroll)
$ cd rss-feed-source
$ mvn test
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building rss-feed-source 1.0.0.BUILD-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) # rss-feed-source ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) # rss-feed-source ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /Users/food_meister/Documents/spring-xd-samples/rss-feed-source/target/classes
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) # rss-feed-source ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) # rss-feed-source ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 2 source files to /Users/food_mesiter/Documents/spring-xd-samples/rss-feed-source/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.18:test (default-test) # rss-feed-source ---
[INFO] Surefire report directory: /Users/food_meister/Documents/spring-xd-samples/rss-feed-source/target/surefire-reports
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running com.acme.FeedConfigurationTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.749 sec - in com.acme.FeedConfigurationTest
Running com.acme.FeedSourceModuleIntegrationTest
_____ __ _______
/ ___| (-) \ \ / / _ \
\ `--. _ __ _ __ _ _ __ __ _ \ V /| | | |
`--. \ '_ \| '__| | '_ \ / _` | / ^ \| | | |
/\__/ / |_) | | | | | | | (_| | / / \ \ |/ /
\____/| .__/|_| |_|_| |_|\__, | \/ \/___/
| | __/ |
|_| |___/
1.1.1.RELEASE eXtreme Data
Started : SingleNodeApplication
Documentation: https://github.com/spring-projects/spring-xd/wiki
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.026 sec - in com.acme.FeedSourceModuleIntegrationTest
[2015-04-11 23:14:01,577] WARN Failed to unregister MBean InMemoryDataTree (org.apache.zookeeper.jmx.MBeanRegistry:117)
[2015-04-11 23:14:01,578] WARN Error during unregister (org.apache.zookeeper.jmx.MBeanRegistry:134)
javax.management.InstanceNotFoundException: org.apache.ZooKeeperService:name0=StandaloneServer_port-1,name1=InMemoryDataTree
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
at org.apache.zookeeper.jmx.MBeanRegistry.unregister(MBeanRegistry.java:115)
at org.apache.zookeeper.jmx.MBeanRegistry.unregister(MBeanRegistry.java:132)
at org.apache.zookeeper.server.ZooKeeperServer.unregisterJMX(ZooKeeperServer.java:465)
at org.apache.zookeeper.server.ZooKeeperServer.shutdown(ZooKeeperServer.java:458)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:116)
at org.springframework.xd.dirt.zookeeper.EmbeddedZooKeeper$ServerRunnable.run(EmbeddedZooKeeper.java:240)
at java.lang.Thread.run(Thread.java:745)
Results :
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15.890 s
[INFO] Finished at: 2015-04-11T23:14:01-07:00
[INFO] Final Memory: 35M/455M
[INFO] ------------------------------------------------------------------------
It's not clear what you are asking...
Running com.acme.FeedSourceModuleIntegrationTest
...
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.026 sec - in com.acme.FeedSourceModuleIntegrationTest
Indicates the test ran ok (no errors or failures).
When I run it in STS, yes there's a delay during the startup (it takes a while to start the single node) but the test runs to successful completion, giving me a green bar.