I have been trying to configure Flink UI for Flink application running locally in IDE. So I've set up relevant dependencies and execution environment. But Flink UI still does not work. What might be the reason of it?
By http://localhost:8081 there is a message
{"errors":["Not found."]}
Execution environment setup:
val conf = new Configuration()
conf.setBoolean(ConfigConstants.LOCAL_START_WEBSERVER, true);
val env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(conf)
Dependency setup in build.sbt:
"org.apache.flink" %% "flink-runtime-web" % "1.9.1"
Related
I am measuring the run-times of a spark job with different resource configurations and need to compare the run time of each stage. I can see them in UI only when the job is running.
I run my job on a Hadoop cluster and use Yarn as the resource manager.
Is there any way to keep each stage's run-time? Is there any log for them?
UPDATE:
I read the monitoring document which is mentioned in the comment and add the following lines but it doesn't work:
in spark-defaults.conf :
spark.eventLog.enabled true
spark.eventLog.dir hdfs:///[nameNode]:8020/[PathToSparkEventLogDir]
spark.history.fs.logDirectory
hdfs:///[nameNode]:8020/[PathTosparkLogDirectory]
in spark-env.sh:
export SPARK_PUBLIC_DNS=[nameNode]
SPARK_HISTORY_OPTS="-Dspark.eventLog.enabled=true"
SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=$sparkHistoryDir"
SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.provider=org.apache.spark.deploy.history.FsHistoryProvider"
SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.cleaner.enabled=true"
SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.cleaner.interval=7d"
It looks for /tmp/spark-events/ folder and when I create it and start the history server, it doesn't show any complete or incomplete application.
Note I tried the logDirectory value without port number too but it didn't work.
I could run the Spark History Server and see the history of completed and incompleted applications by applying the following commands:
Set the public DNS value in conf/spark-env.sh
export SPARK_PUBLIC_DNS= NameNode-IP
Add these properties to SparkConf in my Java code:
SparkConf conf = new SparkConf()
.set("spark.eventLog.enabled", "true")
.set("spark.eventLog.dir", "hdfs:///user/[user-path]/sparkEventLog")
.set("spark.history.fs.logDirectory", "hdfs:///user/[user-path]/sparkEventLog")
Create the property file ( spark/conf/history.properties ) containg the following lines
spark.eventLog.enabled true
spark.eventLog.dir hdfs:///user/[user-path]/sparkEventLog
spark.history.fs.logDirectory hdfs:///user/[user-path]/sparkEventLog
Start the history server:
./sbin/start-history-server.sh --properties-file ./conf/history.properties
Note: The properties eventLog.dir and eventLog.dir should have the save values.
I'm trying to create Spring Application without referring to any external files. This is supposed to be a module that you'd then include as a dependency, configure and use to plug in the service into an existing ecosystem. This is how I'm doing that:
Map<String, Object> properties = new HashMap<>();
properties.put("server.address", "0.0.0.0")
properties.put("server.port", 8080)
properties.put("spring.profiles.active", "cloud")
properties.put("spring.application.name", "someApp")
properties.put("spring.cloud.config.failFast", true)
properties.put("spring.cloud.config.discovery.enabled", true)
properties.put("spring.cloud.config.discovery.serviceId", "config")
properties.put("eureka.instance.preferIpAddress", true)
properties.put("eureka.instance.statusPageUrlPath", "/health")
new SpringApplicationBuilder()
.bannerMode(Banner.Mode.OFF)
.properties(properties)
.sources(SpringConfiguration.class)
.web(false)
.registerShutdownHook(true)
.build()
I then go on to provide Eureka default zone in the run command, via environmental variables:
--env eureka_client_serviceUrl_defaultZone='http://some-host:8765/eureka/' --env SPRING_CLOUD_CONFIG_LABEL='dev' --env SPRING_CLOUD_INETUTILS_PREFERRED_NETWORKS='10.0'
Application registers successfully in Eureka, but unfortunately it tries to fetch the config prior to that and it's looking for it under the default URL (http://localhost:8888) instead of fetching config server IP from the registry. And yes, it does work if I put all of those properties in the bootstrap.yml file. Can I somehow make it work without using file-resources?
You are passing the properties using SpringApplicationBuilder which is responsible for SpringApplication and ApplicationContext instances.
From the documentation , the properties provided here will be part of ApplicationContext NOT the BootstrapContext. ApplicationContext is the child of BootstrapContext.
You can read more about the Bootstrap Context here -
http://cloud.spring.io/spring-cloud-commons/1.3.x/single/spring-cloud-commons.html#_the_bootstrap_application_context
Bootstrap.yml/properties is used to configure your Bootstrap Context.
You can look at these properties to change the name or location of the file -
spring.cloud.bootstrap.name - bootstrap(default)
spring.cloud.bootstrap.location
You will have to use a file resource(yml or properties).
I'm building a Play! Framework (2.5) Java application and I want to connect it to ElasticSearch using Ebean.
First, I followed the steps here and successfully connected to a dummy jdbc (Added plugin, modified build.sbt, added ebean.default="models.*", default.driver = org.h2.Driver, and default.url = "jdbc:h2:mem:play" to application.conf). So far so good.
Now, I want to switch my connection to ElasticSearch instead of jdbc, so I'm following the steps here, but I can't figure it out for the life of me. What I've done so far:
Added to build.sbt:
libraryDependencies ++= Seq( "org.avaje.ebeanorm" % "avaje-ebeanorm-elastic" % "1.3.1" )
Added to application.conf:
ebean.docstore.url="http://localhost:9200"
ebean.docstore.active=true
ebean.docstore.generateMapping=true
ebean.docstore.dropCreate=true`
Removed from application.conf
default.driver = org.h2.Driver
default.url = "jdbc:h2:mem:play"
I cannot get this code to run, because I keep getting this error:
RuntimeException: com.typesafe.config.ConfigException$WrongType: application.conf # file:/C:/Users/../conf/application.conf: 357-360: docstore has type OBJECT rather than LIST
I would greatly appreciate any pointers in the right direction. Thanks!
You can either :
Override your ebean configuration programmaticaly
Add your configuration in a ebean property file
1 - Programmatically
package models;
import com.avaje.ebean.config.ServerConfig;
import com.avaje.ebean.event.ServerConfigStartup;
public class ConfigOverride implements ServerConfigStartup {
public void onStart(ServerConfig serverConfig) {
serverConfig.getDocStoreConfig().setUrl("http://localhost:9200");
serverConfig.getDocStoreConfig().setActive(true);
serverConfig.getDocStoreConfig().setGenerateMapping(false);
serverConfig.getDocStoreConfig().setDropCreate(false);
serverConfig.getDocStoreConfig().setCreate(false);
serverConfig.getDocStoreConfig().setPathToResources("conf");
}
}
2 - Configuration
You have to put your ebean.docstore properties in a ebean.properties in conf directory. But unfortunatly, I didn't manage to make the application work in stage mode
Source: https://raw.githubusercontent.com/playframework/play-ebean/master/docs/manual/working/javaGuide/main/sql/JavaEbean.md
FYI : I opened an issue here :
https://github.com/playframework/play-ebean/issues/104
I'm trying to connect my application to Shared Database 5MB using grails tutorial. I'm using grails 2.1.0. On my local machine when I run heroku config I can see that DATABASE_URL is set, but after executing git push... my build stops with error:
Error packaging application: Error loading DataSource.groovy: null (Use --stacktrace to see the full trace)
Failed to build app
Heroku push rejected, failed to compile Grails app
So I've added logging to see how environment looks on heroku. I've added println System.env to Datasource.groovy and I cannot find any DATABASE_URL:
.[OLDPWD:/app/tmp/repo.git/.cache, SHELL:/bin/bash, SHLVL:2, GIT_DIR:., JAVA_HOME:/usr/lib/jvm/java-6-openjdk, PATH:/app/tmp/repo.git/.cache/.grails/bin::/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin, SSH_CLIENT:XXX, MAIL:XXX, USER:XXX, LOG_TOKEN:t.9e79a5be-dfa2-446e-9f83-f6b4ebbc4eb7, GRAILS_HOME:/app/tmp/repo.git/.cache/.grails, PWD:XXX, HOME:/app/, LOGNAME:u11846, _:/app/tmp/repo.git/.cache/.grails/bin/grails, SSH_CONNECTION:XXX, LD_LIBRARY_PATH:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/../lib/amd64]
I have checked with heroku addons that for my application shared database is installed:
heroku addons
=== XXX Configured Add-ons
shared-database:5mb
and when I execute heroku config
heroku config
=== Config Vars for XXX
DATABASE_URL: postgres://XXX
JAVA_OPTS: -Xmx384m -Xss512k -XX:+UseCompressedOops
SHARED_DATABASE_URL: postgres://XXX
Datasource.groovy:
production {
dataSource {
dbCreate = "update"
driverClassName = "org.postgresql.Driver"
dialect = org.hibernate.dialect.PostgreSQLDialect
println System.env
println System.env.DATABASE_URL
uri = new URI(System.env.DATABASE_URL)
println uri
url = "jdbc:postgresql://" + uri.host + uri.path
username = uri.userInfo.split(":")[0]
password = uri.userInfo.split(":")[1]
}
}
In BuildConfig.groovy I've added
runtime 'postgresql:postgresql:8.4-702.jdbc3'
I can launch application locally with DATABASE_URL set to local postgresql. I'm building war and running it on jetty.
Can anyone tell me what am I missing?
The application environment variables are not available at build time. Your code needs to be able to handle that accordingly. Or you can enable the new user_env_compile Heroku Labs feature that does enable the environment variables to be there at build time: https://devcenter.heroku.com/articles/labs-user-env-compile
I would like to be able to package my jpa-ejb-web project as a standalone application, by using Glassfish embedded API.
To use the JPA layer, I need to deploy the sun-resource.xml configuration, which should be possible with the asadmin command add-resources path\to\sun-resources.xml. I've this code to do it:
String command = "add-resources";
ParameterMap params = new ParameterMap();
params.add("", "...\sun-resources.xml" );
CommandRunner runner = server.getHabitat().getComponent(CommandRunner.class);
ActionReport report = server.getHabitat().getComponent(ActionReport.class);
runner.getCommandInvocation(command, report).parameters(params).execute();
but Glassfish refuses it with:
15-Jul-2010 16:34:12 org.glassfish.admin.cli.resources.AddResources execute
SEVERE: Something went wrong in add-resources
java.lang.Exception: ...\gfembed6930201441546233570tmp\lib\dtds\sun-resources_1_4.dtd (The system cannot find the path specified)
at org.glassfish.admin.cli.resources.ResourcesXMLParser.initProperties(ResourcesXMLParser.java:163)
at org.glassfish.admin.cli.resources.ResourcesXMLParser.<init>(ResourcesXMLParser.java:109)
at org.glassfish.admin.cli.resources.ResourcesManager.createResources(ResourcesManager.java:67)
at org.glassfish.admin.cli.resources.AddResources.execute(AddResources.java:106)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:305)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:320)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1176)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$900(CommandRunnerImpl.java:83)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1235)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1224)
at javaapplication4.Main.main(Main.java:55)
and indeed, there is no lib directory on the indicated path ...
is there something wrong in my code? (I use glassfish-embedded-all-3.0.1.jar)
Thanks
I solved it by specifying an Embedded File System for the embedded Glassfish, and prepopulated the /path/to/my/glassfish/lib/dtds folder with the files which were missing.
EmbeddedFileSystem.Builder efsb = new EmbeddedFileSystem.Builder();
efsb.autoDelete(false);
efsb.installRoot(new File("/path/to/my/glassfish"), true);
EmbeddedFileSystem efs = efsb.build();
Server.Builder builder = new Server.Builder("test");
builder.embeddedFileSystem(efs);
builder.logger(true);
Server server = builder.build();
server.addContainer(ContainerBuilder.Type.all);
server.start();
and asking Glassfish not to delete the folder at the end of the execution.
I'm not sure this is possible, Running asadmin Commands Using the Sun GlassFish Embedded Server API doesn't mention such a use case (passing a sun-resources.xml).
But I would use a preconfigured domain.xml instead of trying to deploy a sun-resource.xml file, the result should be similar. From the Sun GlassFish Enterprise Server v3 Embedded Server Guide:
Using an Existing domain.xml File
Using an existing domain.xml file
avoids the need to configure embedded
Enterprise Server programmatically in
your application. Your application
obtains domain configuration data from
an existing domain.xml file. You can
create this file by using the
administrative interfaces of an
installation of nonembedded Enterprise
Server. To specify an existing
domain.xml file, invoke the
installRoot, instanceRoot, or
configurationFile method of the
EmbeddedFileSystem.Builder class or
a combination of these methods.
The documentation provides code samples showing how to do this (should be pretty straightforward).