I had set up a carte cluster (1 master and 2 slaves) and run a job on carte cluster with spoon. But when I ran with kitchen command or carte http access, it ran as standalone (just run in master node).
Did I miss anything in the configuration? Or doesn't it support cluster mode?
Here was what I tried:
my config:
ran in spoon with "Enviroment Type -- Local"
master output:
2017/11/28 04:47:09 - RepositoriesMeta - Reading repositories XML file: /root/.kettle/repositories.xml
Tue Nov 28 04:47:09 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2017/11/28 04:47:10 - sortcluster111 (master) - Dispatching started for transformation [sortcluster111 (master)]
Tue Nov 28 04:47:10 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Tue Nov 28 04:47:10 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2017/11/28 04:47:10 - output.0 - Connected to database [102] (commit=1000)
2017/11/28 04:47:10 - input.0 - Finished reading query, closing connection.
2017/11/28 04:47:10 - input.0 - Finished processing (I=47, O=0, R=0, W=47, U=0, E=0)
2017/11/28 04:47:10 - input.0 - Server socket accepted for port [40001], reading from server Dynamic slave [kettleslave02:8083]
2017/11/28 04:47:10 - input.0 - Server socket accepted for port [40000], reading from server Dynamic slave [kettleslave01:8082]
2017/11/28 04:47:10 - output.0 - Finished processing (I=47, O=47, R=0, W=47, U=0, E=0)
slave01 output:
2017/11/28 04:47:09 - RepositoriesMeta - Reading repositories XML file: /root/.kettle/repositories.xml
Tue Nov 28 04:47:09 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2017/11/28 04:47:10 - sortcluster111 (cluster1:Dynamic slave [kettleslave01:8082]) - Dispatching started for transformation [sortcluster111 (cluster1:Dynamic slave [kettleslave01:8082])]
2017/11/28 04:47:10 - sort.0 - Server socket accepted for port [40000], reading from server kettlemaster01
2017/11/28 04:47:10 - sort.0 - Finished processing (I=24, O=0, R=0, W=24, U=0, E=0)
slave02 output:
2017/11/28 04:47:09 - RepositoriesMeta - Reading repositories XML file: /root/.kettle/repositories.xml
2017/11/28 04:47:09 - General - Unable to connect to the repository with name 'Mysqlrep'
2017/11/28 04:47:10 - sortcluster111 (cluster1:Dynamic slave [kettleslave02:8083]) - Dispatching started for transformation [sortcluster111 (cluster1:Dynamic slave [kettleslave02:8083])]
2017/11/28 04:47:10 - sort.0 - Server socket accepted for port [40000], reading from server kettlemaster01
2017/11/28 04:47:10 - sort.0 - Finished processing (I=23, O=0, R=0, W=23, U=0, E=0)
ran with kitchen:
kitchen.sh -rep=Mysqlrep -user=admin -pass=admin -job trans1
master output:
2017/11/28 04:10:19 - trans1 - Starting entry [sorttrans]
2017/11/28 04:10:19 - sorttrans - Loading transformation from repository [sortcluster111] in directory [/]
2017/11/28 04:10:19 - sorttrans - Using run configuration [cluster config]
2017/11/28 04:10:19 - sorttrans - Using legacy execution engine
2017/11/28 04:10:19 - sortcluster111 - Dispatching started for transformation [sortcluster111]
Tue Nov 28 04:10:19 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Tue Nov 28 04:10:19 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2017/11/28 04:10:19 - output.0 - Connected to database [102] (commit=1000)
2017/11/28 04:10:19 - input.0 - Finished reading query, closing connection.
2017/11/28 04:10:19 - input.0 - Finished processing (I=47, O=0, R=0, W=47, U=0, E=0)
2017/11/28 04:10:19 - sort.0 - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0)
2017/11/28 04:10:19 - output.0 - Finished processing (I=0, O=47, R=47, W=47, U=0, E=0)
2017/11/28 04:10:19 - trans1 - Starting entry [finish]
2017/11/28 04:10:19 - trans1 - Finished job entry [finish] (result=[true])
2017/11/28 04:10:19 - trans1 - Finished job entry [sorttrans] (result=[true])
2017/11/28 04:10:19 - trans1 - Finished job entry [SQL] (result=[true])
2017/11/28 04:10:19 - trans1 - Job execution finished
2017/11/28 04:10:19 - Kitchen - Finished!
2017/11/28 04:10:19 - Kitchen - Start=2017/11/28 04:10:00.586, Stop=2017/11/28 04:10:19.739
2017/11/28 04:10:19 - Kitchen - Processing ended after 19 seconds.
has no output in the slave
Regards
John
There is a bug in new versions of pdi:
The option "Run this transformation in a clustered mode?" not exists, so to fix that you need open the job xml file and remove the property run_configuration and set cluster to Y from entry's of transformation that you wanna run in clustered mode.
Hope this helps.
First, start carte on distant server:
./carte.sh hostname port
check in your browser: hostname:port/kettle/status/ respond with an empty status page
Second, define carte in spoon:
in spoon view (left panel): click on Slave server, then new
server name = carte
hostname = previously defined hostname
port = previously defined port
web app name = blank (important!)
username = cluster
password = cluster
Is the master = Yes
Third, configure spoon run:
in spoon view (left panel): click on Run configuration, then new
Name: carte/servername
Engine: Pentaho (default)
Slave server: Yes
Location: carte (server name as defined in second step)
Send resource to this server: Yes (unless you read the doc)
Forth, run the transformaton/job
the Run configuration drop-down box let you choose between Pentaho local or carte/hostname, or any configuration defined in third step.
if you disabled the Run option panel, then Drop-down the Run tool or Top menu/Action/Run option or F8. And check the Always show dialog on run, you'll need it in the future.
Fifth
in your browser: hostname:port/kettle/status/
have fun.
Related
Introduction:
I'm new to Jmeter, so I'm working with basic tasks but I have a mentor. We are working with distributed architecture, one master and one slave in my local network executing a real world working testplan.
Problem:
No response from slave:
We have installed and configured Jmeter 5.3 and java version "1.8.0_271" in both slave and master with windows 10 machines. Each machine can ping the other. When the testplan is executed in the master, from the JMeter GUI or CLI, the slave recognize the order and start the process, it also notify the end, but there is no response in the .jtl file another than the file header. We have tried with RMI enabled and disabled. The logs don't show errors.
System specs:
OS W10, Jmeter 5.3, Java 1.8, Jmeter plugin: jmeter-plugins-manager-1.4.jar (Plugins installed: Custom Thread Groups, 3 Basic Graphs, Console Status Logger), Windows Firewall is disabled in both machines, Jmeter properties (Master remote_hosts: Slave ip, Slave remote_hosts: 127.0.0.1)
JMeter GUI execution log (Master):
2020-12-11 11:33:45,446 INFO o.a.j.e.DistributedRunner: Configuring remote engine: 192.168.1.135
2020-12-11 11:33:45,446 INFO o.a.j.r.RmiUtils: Disabling SSL for RMI as server.rmi.ssl.disable is set to 'true'
2020-12-11 11:33:45,624 INFO o.a.j.e.DistributedRunner: Starting distributed test with remote engines: [192.168.1.135] # Fri Dec 11 11:33:45 CET 2020 (1607682825623)
2020-12-11 11:33:45,624 INFO o.a.j.e.ClientJMeterEngine: running clientengine run method
2020-12-11 11:33:45,629 INFO o.a.j.r.RmiUtils: Disabling SSL for RMI as server.rmi.ssl.disable is set to 'true'
2020-12-11 11:33:45,629 INFO o.a.j.r.RmiUtils: Disabling SSL for RMI as server.rmi.ssl.disable is set to 'true'
2020-12-11 11:33:45,630 INFO o.a.j.s.BatchSampleSender: Using batching (client settings) for this run. Thresholds: num=100, time=60000
2020-12-11 11:33:45,630 INFO o.a.j.s.DataStrippingSampleSender: Using DataStrippingSampleSender for this run
2020-12-11 11:33:45,630 INFO o.a.j.r.RmiUtils: Disabling SSL for RMI as server.rmi.ssl.disable is set to 'true'
2020-12-11 11:33:45,630 INFO o.a.j.r.RmiUtils: Disabling SSL for RMI as server.rmi.ssl.disable is set to 'true'
2020-12-11 11:33:45,630 INFO o.a.j.s.BatchSampleSender: Using batching (client settings) for this run. Thresholds: num=100, time=60000
2020-12-11 11:33:45,630 INFO o.a.j.s.DataStrippingSampleSender: Using DataStrippingSampleSender for this run
2020-12-11 11:33:45,630 INFO o.a.j.r.RmiUtils: Disabling SSL for RMI as server.rmi.ssl.disable is set to 'true'
2020-12-11 11:33:45,630 INFO o.a.j.r.RmiUtils: Disabling SSL for RMI as server.rmi.ssl.disable is set to 'true'
2020-12-11 11:33:45,630 INFO o.a.j.r.RmiUtils: Disabling SSL for RMI as server.rmi.ssl.disable is set to 'true'
2020-12-11 11:33:45,630 INFO o.a.j.r.RmiUtils: Disabling SSL for RMI as server.rmi.ssl.disable is set to 'true'
2020-12-11 11:33:45,641 INFO o.a.j.r.RmiUtils: Disabling SSL for RMI as server.rmi.ssl.disable is set to 'true'
2020-12-11 11:33:45,641 INFO o.a.j.r.RmiUtils: Disabling SSL for RMI as server.rmi.ssl.disable is set to 'true'
2020-12-11 11:33:45,641 INFO o.a.j.s.BatchSampleSender: Using batching (client settings) for this run. Thresholds: num=100, time=60000
2020-12-11 11:33:45,641 INFO o.a.j.s.DataStrippingSampleSender: Using DataStrippingSampleSender for this run
2020-12-11 11:33:48,060 INFO o.a.j.e.ClientJMeterEngine: sent test to 192.168.1.135 basedir='.'
2020-12-11 11:33:48,060 INFO o.a.j.e.ClientJMeterEngine: Sending properties {}
2020-12-11 11:33:48,075 INFO o.a.j.e.ClientJMeterEngine: sent run command to 192.168.1.135
2020-12-11 11:33:48,075 INFO o.a.j.e.DistributedRunner: Remote engines have been started:[192.168.1.135]
2020-12-11 11:33:48,721 INFO o.a.j.g.u.JMeterMenuBar: setRunning(true, 192.168.1.135)
2020-12-11 11:35:49,030 INFO o.a.j.g.u.JMeterMenuBar: setRunning(false, 192.168.1.135)
JMeter CLI interface:
Creating summariser <summary>
Created the tree successfully using fileExample.jmx
Configuring remote engine: 192.168.1.135
Starting distributed test with remote engines: [192.168.1.135] # Fri Dec 11 12:41:46 CET 2020 (1607686906826)
Remote engines have been started:[192.168.1.135]
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
summary = 0 in 00:00:00 = ******/s Avg: 0 Min: 9223372036854775807 Max: -9223372036854775808 Err: 0 (0.00%)
Tidying up remote # Fri Dec 11 12:43:51 CET 2020 (1607687031041)
... end of run
Guides and articles read:
Our own guide, which has worked in the past (installed in production), based in:
https://jmeter.apache.org/usermanual/jmeter_distributed_testing_step_by_step.html
https://jmeter.apache.org/usermanual/remote-test.html
JMeter Summary report in distributed mode
https://loadium.com/blog/jmeter-distributed-testing-step-by-step/
https://cwiki.apache.org/confluence/display/jmeter/JMeterFAQ#How_to_do_remote_testing_the_.27proper_way.27.3F
Question:
So the question is basically what can be wrong to make the server recognize the start calls, send the end signal, but don't write the .jtl file.
EDIT WITH SOLUTION
The .csv file in what the test are based must be in the slave inside the JMeter bin folder /bin/data/yourFile.csv
How about slave log? Given the symptoms I can think of 3 possible reasons:
Your test relies on external data file, i.e. CSV file or .properties file, if this is the case - you need to copy all the dependent files to all the slave machines
Your test relies on a plugin which is not installed on the slave(s), either copy your JMeter installation from master to all the slaves or use JMeter Plugins Manager to install the missing plugins
Your RMI configuration is not correct, i.e. port which is used for communication from the slave to the master is random by default, you might want to set it explicitly and open it in firewall
In order to be able to tell more I need to see your jmeter-server.log file (it's being generated in "bin" folder of your JMeter installation given you launch the slave process via jmeter-server.bat file, if you use other approach you can set the desired log file name/location via -j command-line argument
We are facing the below issue while migrating.
Spring Batch 4.0.x from 3.0.x
Spring Boot 2.0.x from 1.5.x
Spring core 5.0.x from 4.3.x
After restarting the application, I could see after a min it throws the logs as SpringBootJPAHikariCP - Shutdown initiated....
But when we trigger the Job Hikari starts the new connections and working as expected.
But I am not sure why the first connection is shutting down after a minute.
I understood that SpringBoot2.0 uses the Hikari for the DB Pool, but not sure why I am seeing Shut down info logs after one min of ideal time.
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.0.4.RELEASE)
2018-09-07 17:16:38,117 -0700,threadId=1,level=INFO ,logger=,msg="The following profiles are active: local"
2018-09-07 17:16:39,495 -0700,threadId=1,level=WARN ,logger=HikariConfig.1081,msg="SpringBootJPAHikariCP - idleTimeout is close to or more than maxLifetime, disabling it."
2018-09-07 17:16:39,495 -0700,threadId=1,level=INFO ,logger=HikariDataSource.110,msg="SpringBootJPAHikariCP - Starting..."
Fri Sep 07 17:16:39 PDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-09-07 17:16:39,695 -0700,threadId=1,level=INFO ,logger=HikariDataSource.123,msg="SpringBootJPAHikariCP - Start completed."
Fri Sep 07 17:16:39 PDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Fri Sep 07 17:16:39 PDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Fri Sep 07 17:16:39 PDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Fri Sep 07 17:16:39 PDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Fri Sep 07 17:16:39 PDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Fri Sep 07 17:16:39 PDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Fri Sep 07 17:16:39 PDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Fri Sep 07 17:16:39 PDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Fri Sep 07 17:16:39 PDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-09-07 17:16:40,002 -0700,threadId=1,level=INFO ,logger=Http11NioProtocol.180,msg="Initializing ProtocolHandler ["http-nio-8080"]"
2018-09-07 17:16:40,010 -0700,threadId=1,level=INFO ,logger=StandardService.180,msg="Starting service [Tomcat]"
2018-09-07 17:16:40,011 -0700,threadId=1,level=INFO ,logger=StandardEngine.180,msg="Starting Servlet Engine: Apache Tomcat/8.5.32"
2018-09-07 17:16:40,014 -0700,threadId=54,level=INFO ,logger=AprLifecycleListener.180,msg="The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/Users/xxxxx/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.]"
2018-09-07 17:16:40,117 -0700,threadId=54,level=INFO ,logger=[/].180,msg="Initializing Spring embedded WebApplicationContext"
2018-09-07 17:16:40,310 -0700,threadId=54,level=INFO ,logger=ReceiverConfig$$EnhancerBySpringCGLIB$$8c1192a0.68,msg="creating activeMQConnectionFactory"
2018-09-07 17:16:40,786 -0700,threadId=1,level=WARN ,logger=Flyway.53,msg="Flyway.setCallbacks(FlywayCallback) has been deprecated and will be removed in Flyway 6.0. Use Flyway.setCallbacks(Callback) instead."
2018-09-07 17:16:40,804 -0700,threadId=1,level=INFO ,logger=VersionPrinter.49,msg="Flyway Community Edition 5.1.4 by Boxfuse"
2018-09-07 17:16:40,807 -0700,threadId=1,level=INFO ,logger=DatabaseFactory.49,msg="Database: jdbc:mysql://127.0.0.1:3306/invoicing (MySQL 5.7)"
2018-09-07 17:16:40,853 -0700,threadId=1,level=INFO ,logger=DbValidate.49,msg="Successfully validated 2 migrations (execution time 00:00.015s)"
2018-09-07 17:16:40,861 -0700,threadId=1,level=INFO ,logger=DbMigrate.49,msg="Current version of schema `invoicing`: 1.0.13.1"
2018-09-07 17:16:40,862 -0700,threadId=1,level=INFO ,logger=DbMigrate.49,msg="Schema `invoicing` is up to date. No migration necessary."
2018-09-07 17:16:43,166 -0700,threadId=1,level=ERROR,logger=,msg="[ThreadID-1] Warning: big time skew between machina and client: service=1536365803166 client=1536365760000 diffSecs=43"
2018-09-07 17:16:44,425 -0700,threadId=1,level=WARN ,logger=JpaBaseConfiguration$JpaWebConfiguration$JpaWebMvcConfiguration.235,msg="spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning"
2018-09-07 17:16:45,814 -0700,threadId=85,level=INFO ,logger=FailoverTransport.1052,msg="Successfully connected to ssl://qal.message-preprod.a.intuit.com:61617"
2018-09-07 17:16:46,100 -0700,threadId=1,level=INFO ,logger=Http11NioProtocol.180,msg="Starting ProtocolHandler ["http-nio-8080"]"
2018-09-07 17:16:46,120 -0700,threadId=1,level=INFO ,logger=NioSelectorPool.180,msg="Using a shared selector for servlet write/read"
2018-09-07 17:16:46,133 -0700,threadId=1,level=INFO ,logger=LateFeeProcessor.59,msg="Started LateFeeProcessor in 12.283 seconds (JVM running for 13.152)"
2018-09-07 17:16:46,429 -0700,threadId=19,level=INFO ,logger=[/].180,msg="Initializing Spring FrameworkServlet 'dispatcherServlet'"
2018-09-07 17:17:47,868 -0700,threadId=93,level=INFO ,logger=HikariDataSource.381,msg="SpringBootJPAHikariCP - Shutdown initiated..."
2018-09-07 17:17:47,876 -0700,threadId=93,level=INFO ,logger=HikariDataSource.383,msg="SpringBootJPAHikariCP - Shutdown completed."
I have some issue and so far found that somehow spring issues shutdown command to Hikari thinking that its disposable bean.
I'm trying to setup Teamcity on an EC2 Ubuntu instance.
I've:
Downloaded the package through wget http://download.jetbrains.com/teamcity/TeamCity-9.0.4.tar.gz. Unpackaged it.
Unpackaged it: tar -xvzf TeamCity-9.0.4.tar.gz.
sh Teamcity/bin/teamcity-server.sh start.
I can't access the web-app, and when I check the logs, says:
INFO - jetbrains.buildServer.STARTUP - Server shutdown has been requested
WARN - jetbrains.buildServer.STARTUP - Could not determine the local IP address: ip-172-30-1-83: ip-172-30-1-83: unknown error
INFO - jetbrains.buildServer.STARTUP - Server IP address: -unknown-
INFO - jetbrains.buildServer.STARTUP - Server shutdown complete
Complete log:
INFO - jetbrains.buildServer.STARTUP - Starting TeamCity server
INFO - jetbrains.buildServer.STARTUP - Server shutdown has been requested
INFO - jetbrains.buildServer.STARTUP - TeamCity version: 9.0.4 (build 32407), data format version 709
INFO - jetbrains.buildServer.STARTUP - OS: Linux, User: ubuntu, JRE: 1.8.0_45-b14 Java HotSpot(TM) 64-Bit Server VM
INFO - jetbrains.buildServer.STARTUP - JVM parameters: -Djava.util.logging.config.file=/home/ubuntu/TeamCity/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xmx512m -XX:MaxPermSize=270m -Dlog4j.configuration=file:/home/ubuntu/TeamCity/bin/../conf/teamcity-server-log4j.xml -Dteamcity_logs=../logs/ -Djsse.enableSNIExtension=false -Djava.awt.headless=true -Djava.endorsed.dirs=/home/ubuntu/TeamCity/endorsed -Dcatalina.base=/home/ubuntu/TeamCity -Dcatalina.home=/home/ubuntu/TeamCity -Djava.io.tmpdir=/home/ubuntu/TeamCity/temp
INFO - jetbrains.buildServer.STARTUP - Garbage collection: Copy, MarkSweepCompact
WARN - jetbrains.buildServer.STARTUP - Could not determine the local IP address: ip-172-30-1-83: ip-172-30-1-83: unknown error
INFO - jetbrains.buildServer.STARTUP - Web server version: Apache Tomcat/7.0.59
INFO - jetbrains.buildServer.STARTUP - Server IP address: -unknown-
INFO - jetbrains.buildServer.STARTUP - Servlet real path: /home/ubuntu/TeamCity/webapps/ROOT/
INFO - jetbrains.buildServer.STARTUP - Current stage: TeamCity server is shutting down
INFO - jetbrains.buildServer.STARTUP - Stopping the main application
INFO - jetbrains.buildServer.STARTUP - The main servlet is not initialized, nothing to shut down.
INFO - jetbrains.buildServer.STARTUP - Server shutdown complete
Any hints on how to solve it?
I have Teamcity's web-app working on my local machine (also Ubuntu, and the same Teamcity package).
Read the shell script TeamCity/bin/teamcity-server.sh, it is written with limitation, the script can't run out of the application folder.
export CATALINA_OPTS
CATALINA_HOME=./..
CATALINA_BASE=./..
So you should cd to the bin folder and run :
cd Teamcity/bin
sh teamcity-server.sh start
Let me know if it fixs your issue or not.
I setup Cassandra, OpsCenter and the needed DataStax agent on my EC2 Amazon machine. At the moment it's only one machine.
Everything seems to be running fine, except the node list is empty and so are the keyspaces in the Opscenter. The cassandra, datastax and opscenter logs show no errors and I followed the installation / configuration carefully. Then tried all the suggested fixes.
My guess is the problem lies in the communication between the agent and opscenter.
After a while these requests fail:
etc/cassandra/cassandra.yaml: (simplified)
cluster_name: 'CassandraCluster'
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "1.2.3.4"
listen_address: 1.2.3.4
rpc_address: 0.0.0.0
endpoint_snitch: Ec2Snitch
etc/opscenter/opscenterd.conf: (simplified)
[webserver]
port = 81
interface = 0.0.0.0
[authentication]
enabled = False
[stat_reporter]
[agents]
use_ssl = false
var/lib/datastax-agent/conf/address.yaml: (simplified)
stomp_interface: 1.2.3.4
local_interface: 1.2.3.4
use_ssl: 0
nodetool status output:
Note: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: eu-west_1_cassandra
===============================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 1.2.3.4 2.06 MB 256 100.0% 8a121c12-7cbf-4a2a-b111-4ad111c111d8 1a
Nothing really strange shows up in the log except for the repetitive occurence of the following line in the agent.log:
INFO [install-location-finder] 2015-03-11 15:26:04,690 New JMX connection (127.0.0.1:7199)
INFO [install-location-finder] 2015-03-11 15:27:04,698 New JMX connection (127.0.0.1:7199)
INFO [install-location-finder] 2015-03-11 15:28:04,709 New JMX connection (127.0.0.1:7199)
INFO [install-location-finder] 2015-03-11 15:29:04,716 New JMX connection (127.0.0.1:7199)
INFO [install-location-finder] 2015-03-11 15:30:04,724 New JMX connection (127.0.0.1:7199)
INFO [install-location-finder] 2015-03-11 15:31:04,731 New JMX connection (127.0.0.1:7199)
To supply all the info here are the logs:
opscenterd.log
agent.log
cassandra/system.log
In certain environments the persistent connection between the browser and opscenterd may fail. We're working on implementing a more robust connection that will work in all environments, but in the meantime you can use the following workaround:
http://www.datastax.com/documentation/opscenter/5.1/opsc/troubleshooting/opscTroubleshootingZeroNodes.html
Minimal configuration that I find working was setting this options below for address.yaml
stomp_interface: [opscenter-ip]
stomp_port: 61620
use_ssl: 0
cassandra_conf: /etc/cassandra/cassandra.yaml
jmx_host: [cassandra-node-ip]
jmx_port: 7199
Make sure you have sysstat installed also.
I am setting up a test environment using apache accumulo ver 1.4.0, hadoop ver 0.20.2 and zookeeper ver 3.3.3. Please see below for the issue.
Hadoop and Zookeeper work great together, but when I start accumulo using the procedures on apache incubator I get the following zookeeper stream of info and a warn:
2011-12-08 20:13:56,601 - INFO [main:QuorumPeerConfig#90] - Reading configuration from: /home/hadoop/zookeeper-3.3.3/bin/../conf/zoo.cfg
2011-12-08 20:13:56,603 - WARN [main:QuorumPeerMain#105] - Either no config or no quorum defined in config, running in standalone mode
2011-12-08 20:13:56,616 - INFO [main:QuorumPeerConfig#90] - Reading configuration from: /home/hadoop/zookeeper-3.3.3/bin/../conf/zoo.cfg
2011-12-08 20:13:56,617 - INFO [main:ZooKeeperServerMain#94] - Starting server
2011-12-08 20:13:56,626 - INFO [main:Environment#97] - Server environment:zookeeper.version=3.3.3-1073969, built on 02/23/2011 22:27 GMT
2011-12-08 20:13:56,627 - INFO [main:Environment#97] - Server environment:host.name.paz
2011-12-08 20:13:56,627 - INFO [main:Environment#97] - Server environment:java.version=1.6.0_26
2011-12-08 20:13:56,628 - INFO [main:Environment#97] - Server environment:java.vendor=Sun Microsystems Inc.
2011-12-08 20:13:56,629 - INFO [main:Environment#97] - Server environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.26/jre
2011-12-08 20:13:56,629 - INFO [main:Environment#97] - Server environment:java.class.path=/home/hadoop/zookeeper-3.3.3/bin/../build/classes:/home/hadoop/zookeeper-3.3.
3/bin/../build/lib/*.jar:/home/hadoop/zookeeper-3.3.3/bin/../zookeeper-3.3.3.jar:/home/hadoop/zookeeper-3.3.3/bin/../lib/log4j-1.2.15.jar:/home/hadoop/zookeeper-3.3.3/b
in/../lib/jline-0.9.94.jar:/home/hadoop/zookeeper-3.3.3/bin/../src/java/lib/*.jar:/home/hadoop/zookeeper-3.3.3/bin/../conf:
2011-12-08 20:13:56,630 - INFO [main:Environment#97] - Server environment:java.library.path=/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/i386/server:/usr/lib/jvm/java-6-su
n-1.6.0.26/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
2011-12-08 20:13:56,630 - INFO [main:Environment#97] - Server environment:java.io.tmpdir=/tmp
2011-12-08 20:13:56,631 - INFO [main:Environment#97] - Server environment:java.compiler=<NA>
2011-12-08 20:13:56,631 - INFO [main:Environment#97] - Server environment:os.name=Linux
2011-12-08 20:13:56,632 - INFO [main:Environment#97] - Server environment:os.arch=i386
2011-12-08 20:13:56,633 - INFO [main:Environment#97] - Server environment:os.version=3.0.0-13-generic
2011-12-08 20:13:56,633 - INFO [main:Environment#97] - Server environment:user.name=hadoop
2011-12-08 20:13:56,634 - INFO [main:Environment#97] - Server environment:user.home=/home/hadoop
2011-12-08 20:13:56,634 - INFO [main:Environment#97] - Server environment:user.dir=/home/hadoop
2011-12-08 20:13:56,641 - INFO [main:ZooKeeperServer#663] - tickTime set to 2000
2011-12-08 20:13:56,641 - INFO [main:ZooKeeperServer#672] - minSessionTimeout set to -1
2011-12-08 20:13:56,642 - INFO [main:ZooKeeperServer#681] - maxSessionTimeout set to -1
2011-12-08 20:13:56,661 - INFO [main:NIOServerCnxn$Factory#143] - binding to port 0.0.0.0/0.0.0.0:2181
2011-12-08 20:13:56,691 - INFO [main:FileSnap#82] - Reading snapshot /home/hadoop/zoo/dataDir/version-2/snapshot.0
2011-12-08 20:13:56,708 - INFO [main:FileTxnSnapLog#208] - Snapshotting: 4e
2011-12-08 20:14:52,147 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory#251] - Accepted socket connection from /0:0:0:0:0:0:0:1:40694
2011-12-08 20:14:52,153 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn#777] - Client attempting to establish new session at /0:0:0:0:0:0:0:1:40694
2011-12-08 20:14:52,154 - INFO [SyncThread:0:FileTxnLog#197] - Creating new log file: log.4f
2011-12-08 20:14:52,410 - INFO [SyncThread:0:NIOServerCnxn#1580] - Established session 0x13420623ee70000 with negotiated timeout 30000 for client /0:0:0:0:0:0:0:1:4069
4
2011-12-08 20:14:52,959 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory#251] - Accepted socket connection from /127.0.0.1:38446
2011-12-08 20:14:52,962 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn#777] - Client attempting to establish new session at /127.0.0.1:38446
2011-12-08 20:14:53,007 - INFO [SyncThread:0:NIOServerCnxn#1580] - Established session 0x13420623ee70001 with negotiated timeout 30000 for client /127.0.0.1:38446
2011-12-08 20:14:59,932 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn#634] - EndOfStreamException: Unable to read additional data from client session
id 0x13420623ee70000, likely client has closed socket
and when I start the accumulo shell I get the following errors:
18 12:44:38,746 [impl.ServerClient] WARN : Failed to find an available server in the list of servers: []
18 12:44:38,846 [impl.ServerClient] WARN : Failed to find an available server in the list of servers: []
18 12:44:38,947 [impl.ServerClient] WARN : Failed to find an available server in the list of servers: []
18 12:44:39,048 [impl.ServerClient] WARN : Failed to find an available server in the list of servers: []
18 12:44:39,148 [impl.ServerClient] WARN : Failed to find an available server in the list of servers: []
18 12:44:39,249 [impl.ServerClient] WARN : Failed to find an available server in the list of servers: []
18 12:44:39,350 [impl.ServerClient] WARN : Failed to find an available server in the list of servers: []
18 12:44:39,450 [impl.ServerClient] WARN : Failed to find an available server in the list of servers: []
Corrected tserver memory settings to not exceed waht is allowed by the JVM. Tserver does not crash and error resolved.
Answer came from accumulo-incubator user list and is reposted at the bottom.
Basically, when going in and modifying memory settings to run in pseudo-distributed mode on a laptop I made incorrect modifications to the accumulo-site.xml and accumulo-env.sh files concerning the tablet server memory usage. The clue to the error was found in the /home/hadoop/accumulo/logs/tserver*.log file:
20 18:20:00,951 [tabletserver.NativeMap] ERROR: Failed to load native
map library
/home/hadoop/accumulo/lib/native/map/libNativeMap-Linux-i386-32.so
java.lang.UnsatisfiedLinkError: Can't load library:
/home/hadoop/accumulo/lib/native/map/libNativeMap-Linux-i386-32.so
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1706)
at java.lang.Runtime.load0(Runtime.java:770)
at java.lang.System.load(System.java:1003)
at
org.apache.accumulo.server.tabletserver.NativeMap.loadNativeLib(NativeMap.java:144)
at
org.apache.accumulo.server.tabletserver.NativeMap.<clinit>(NativeMap.java:156)
at
org.apache.accumulo.server.tabletserver.TabletServerResourceManager.<init>(TabletServerResourceManager.java:123)
at
org.apache.accumulo.server.tabletserver.TabletServer.config(TabletServer.java:2959)
at
org.apache.accumulo.server.tabletserver.TabletServer.main(TabletServer.java:3085)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.accumulo.start.Main$1.run(Main.java:89)
at java.lang.Thread.run(Thread.java:662)
20 18:20:00,999 [tabletserver.TabletServer] ERROR: Uncaught exception in
TabletServer.main, exiting
java.lang.IllegalArgumentException: Maximum tablet server map memory
134,217,728 and block cache sizes 186,646,528 is too large for this JVM
configuration 132,579,328
at
org.apache.accumulo.server.tabletserver.TabletServerResourceManager.<init>(TabletServerResourceManager.java:134)
at
org.apache.accumulo.server.tabletserver.TabletServer.config(TabletServer.java:2959)
at
org.apache.accumulo.server.tabletserver.TabletServer.main(TabletServer.java:3085)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.accumulo.start.Main$1.run(Main.java:89)
at java.lang.Thread.run(Thread.java:662)
Specific help text:
"The native libraries are not loading, which is shifting in-memory
map into the java workspace. Add to that your block cache size, and your
specifications for memory use are higher than the JVM will be allowed to
allocate. The tablet servers complain, and exit. You should see these
complaints on the accumulo monitor web pages.
You may find a benefit to rebuilding the native map library, which will
move the allocation of the in-memory map to outside the JVM. This is
not required.
The size of any memory dedicated to cache must be smaller than the size
of the JVM, which must include substantial working space for RPC calls
and garbage collection over time."
If you run the Zookeeper client (/home/hadoop/zookeeper-3.3.3/bin/zkCli.sh) and do an ls of
/accumulo/<instance uuid>/tservers
I assume you won't see any servers listed. You should see one or more tablet servers listed if Accumulo was initialized properly. Are you sure you ran the Accumulo init script after setting your Zookeeper servers in the accumulo-site.xml per the instructions?
Make sure you set "instance.zookeeper.host" to the location of your zookeeper node in the accumulo-site.xml file.
Additionally, check the logs for your tservers and loggers. If you have other configuration issues, they will not go live, which will cause the master to report having difficulties finding tservers.
Go into $ACCUMULO_HOME/conf . End the files masters,slaves and tracers to contain one single line that reads "localhost" (I assume you are doing single node)