Hadoop itself (i.e hdfs and map/reduce) is working - and brew-installed hive as well (which validates the first two). Just the GUI is weird: see the attached screenshot: the webserver is up but the JobTracker is not providing any useful info
jobtracker.jsp not found:
Anyone home at all on port 50030 (yes!)
What about namenode ui?
I was getting the same error on hadoop-2.0.0-cdh4.5.0 installation. After spending some time looking into the code, I managed to get this working by applying following patch to hadoop-common-2.0.0-cdh4.5.0.jar:
diff --git a/org/apache/hadoop/http/HttpServer.java b/org/apache/hadoop/http/HttpServer.java
index 25b4c75..1896623 100644
--- a/org/apache/hadoop/http/HttpServer.java
+++ b/org/apache/hadoop/http/HttpServer.java
## -237,6 +237,10 ## public class HttpServer implements FilterContainer {
webAppContext.setDisplayName(name);
webAppContext.setContextPath("/");
webAppContext.setWar(appDir + "/" + name);
+
+ webAppContext.setDescriptor(appDir + "/" + name + "/WEB-INF/web.xml");
+ webAppContext.setResourceBase(appDir + "/" + name);
+
webAppContext.getServletContext().setAttribute(CONF_CONTEXT_ATTRIBUTE, conf);
webAppContext.getServletContext().setAttribute(ADMINS_ACL, adminsAcl);
addNoCacheFilter(webAppContext);
basically the jobtracker/tasktracker webapp initialization using Jetty WebAppContext was missing the web.xml location, adding it there using setDescriptor() and setting resourceBase correctly using setResourceBase() solves the problem.
Related
I need to generate the dynamic file path in the setup thread group like below.
def result_file = new File(org.apache.jmeter.services.FileServer.getFileServer().getBaseDir() + File.separator + 'transactions_passed_' + new Date().format('MM_dd_yyyy_HH_mm_ss') + '.csv');
props.put("result_file", result_file);
Now I want to pass that file path as a filename value of Flexible File Writer plugin of jmeter so that variables are stored inside it.
Not able to make it work. Kindly help. Thanks
I have tried below options:
Filename: ${__groovy(props.get("result_file").text)}
tried to use preprocessor and set the value:
vars.put("result_file", '${__FileToString(props.get("result_file"),,)}');
Also tried to use below groovy script in the FileName field of Flexible File Writer, however it throws an exception of FileNotFound exception:
${__groovy(new File(org.apache.jmeter.services.FileServer.getFileServer().getBaseDir() + System.getProperty('file.separator') + 'transactions_passed_' + new Date().format('MM_dd_yyyy_HH_mm_ss') + '.csv').text)}
I want to use DYNAMIC FILE PATH (which I am setting as property in setup thread group) in the FILENAME field of FLEXIBLE FILE WRITER
The approach with __groovy() function should work however you need to remove this .text bit from there
${__groovy(new File(org.apache.jmeter.services.FileServer.getFileServer().getBaseDir() + System.getProperty('file.separator') + 'transactions_passed_' + new Date().format('MM_dd_yyyy_HH_mm_ss') + '.csv'))}
because .text returns you the file contents and as the file doesn't exist - you're getting this FileNotFound error. More information on Groovy scripting in JMeter - Apache Groovy: What Is Groovy Used For?
We have deployed an ear file to our WebSphere instance. By default, all *.jar & *.war modules are mapped to the jvm.
We would like to map all the *.war modules also to the web server.
I have created the below jython script to map the additional modules to the web server:
modules = AdminApp.listModules('${p:appName}', '-server')
splitted = modules.splitlines()
for moduleLine in splitted:
print "Mapping module: " + moduleLine
appName, moduleUri, target = moduleLine.split("#")
print appName
print moduleUri
print target
if moduleUri.find('.war') >= 0:
print "It's a war: " + moduleUri
module, webXml = moduleUri.split("+")
print module
print webXml
AdminApp.edit('${p:appName}', ['-MapModulesToServers', [[module, module + ',' + webXml, target]]])
The above script works when the name of the module is the same name as referenced in the uri. However, in some cases, the web.xml contains another name as 'display name'. When we do a -MapModulesToServers, it seems to look at the display name of the module, not the uri.
For example:
In the WebSphere console, we would have the following line:
Module URI Module type
Demo be.fictive.company.demo.war,WEB-INF/web.xml Web Module
The 'AdminApp.listModules' method is returning the uri name (be.fictive.company.demo.war), whereas I need the name of the module (Demo).
Am I missing something or is there another way to retrieve the module name, so I can use the AdminApp.edit('${p:appName}', ['-MapModulesToServers', [[module, module + ',' + webXml, target]]]) to update the targets?
The administrative scripting console is not available, so I don't see a way to retrieve the commands that have been issued.
You can use the wildcat to match the name of the module. Replace
AdminApp.edit('${p:appName}', ['-MapModulesToServers', [[module, module + ',' + webXml, target]]])
with
AdminApp.edit('${p:appName}', ['-MapModulesToServers', [['.*', module + ',' + webXml, target]]])
To render my own spring boot starter guide, I followed these 2 pages:
https://github.com/spring-guides/getting-started-guides/wiki/Testing-edits-to-your-guide-and-macros
https://github.com/spring-guides/getting-started-guides/wiki/Run-sagan-to-test-your-own-content
I can run Sagan, but not with my own guide. Particularly, first wiki page says:
Example 1. sagan-site/src/main/resources/application.yml
...
guides:
owner:
name: ${GITHUB_GUIDES_OWNER_NAME:/*your githubusername*/}
type: ${GITHUB_GUIDES_OWNER_TYPE:users}
...
But nowhere in sagan's code it uses GITHUB_GUIDES_OWNER_NAME.
I did find sagan/sagan-renderer/src/main/resources/application.yml, which defines the organization property, but editing like this didn't get sagan to render my guide:
sagan:
renderer:
guides:
organization: ge0ffrey
# github:
...
# token: ${GITHUB_ACCESS_TOKEN:}
I got sagan to render my guide by hacking the sagan code (see below). Basically, get the guides list from spring-guides, but get your guide from your repository (ge0ffrey in my case).
For what it's worth (not much), here's the patch diff of my local hack:
Index: sagan-renderer/src/main/java/sagan/renderer/guides/GuideRenderer.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- sagan-renderer/src/main/java/sagan/renderer/guides/GuideRenderer.java (revision 68d9ced83d8ee97c7972a0d603a9077299c0435f)
+++ sagan-renderer/src/main/java/sagan/renderer/guides/GuideRenderer.java (date 1578485243882)
## -41,7 +41,7 ##
GuideContentResource guideContent = new GuideContentResource();
guideContent.setName(guideName);
String repositoryName = type.getPrefix() + guideName;
- String org = this.properties.getGuides().getOrganization();
+ String org = "ge0ffrey";
String tempFilePrefix = org + "-" + repositoryName;
File unzippedRoot = null;
Index: sagan-renderer/src/main/java/sagan/renderer/guides/GuidesController.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- sagan-renderer/src/main/java/sagan/renderer/guides/GuidesController.java (revision 68d9ced83d8ee97c7972a0d603a9077299c0435f)
+++ sagan-renderer/src/main/java/sagan/renderer/guides/GuidesController.java (date 1578484725643)
## -53,6 +53,13 ##
.toResources(this.githubClient.fetchOrgRepositories(properties.getGuides().getOrganization()))
.stream().filter(guide -> !guide.getType().equals(GuideType.UNKNOWN))
.collect(Collectors.toList());
+ Repository optaplannerGuide = new Repository(15L, "gs-constraint-solving-ai-optaplanner",
+ "spring-guides/gs-constraint-solving-ai-optaplanner", "OptaPlanner Guide description",
+ "https://github.com/ge0ffrey/gs-constraint-solving-ai-optaplanner/",
+ "git#github.com:ge0ffrey/gs-constraint-solving-ai-optaplanner.git",
+ "https://github.com/ge0ffrey/gs-constraint-solving-ai-optaplanner.git",
+ "git://github.com/ge0ffrey/gs-constraint-solving-ai-optaplanner.git", null);
+ guideResources.add(new GuideResource(optaplannerGuide));
Resources<GuideResource> resources = new Resources<>(guideResources);
for (GuideType type : GuideType.values()) {
## -70,7 +77,7 ##
if (GuideType.UNKNOWN.equals(guideType)) {
return ResponseEntity.notFound().build();
}
- Repository repository = this.githubClient.fetchOrgRepository(properties.getGuides().getOrganization(),
+ Repository repository = this.githubClient.fetchOrgRepository("ge0ffrey",
guideType.getPrefix() + guide);
GuideResource guideResource = this.guideAssembler.toResource(repository);
if (guideResource.getType().equals(GuideType.UNKNOWN)) {
Note that just overriding the organization in application.properties doesn't work, because it needs to get a full listing of all guides first. Maybe it works if you clone other repositories too (no idea which ones).
Are there any algorithms in Apache Spark to find out the frequent patterns in a text file. I tried following example but always end up with this error:
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:
/D:/spark-1.3.1-bin-hadoop2.6/bin/data/mllib/sample_fpgrowth.txt
Can anyone help me solve this problem?
import org.apache.spark.mllib.fpm.FPGrowth
val transactions = sc.textFile("...").map(_.split(" ")).cache()
val model = new FPGrowth()
model.setMinSupport(0.5)
model.setNumPartitions(10)
model.run(transactions)
model.freqItemsets.collect().foreach {
itemset => println(itemset.items.mkString("[", ",", "]") + ", " + itemset.freq)
}
try this
file://D:/spark-1.3.1-bin-hadoop2.6/bin/data/mllib/sample_fpgrowth.txt
or
D:/spark-1.3.1-bin-hadoop2.6/bin/data/mllib/sample_fpgrowth.txt
if not work, replace / with //
I assume you are running spark on windows.
Use file path like
D:\spark-1.3.1-bin-hadoop2.6\bin\data\mllib\sample_fpgrowth.txt
NOTE : Escape "\" if necessary .
H2 database with custom function alias defined as:
create alias to_date as $$
java.util.Date toDate(java.lang.String dateString, java.lang.String pattern) {
try {
return new java.text.SimpleDateFormat(javaPattern).parse(dateString);
} catch(java.text.ParseException e) {
throw new java.lang.RuntimeException(e);
}
}
$$;
H2 initialized as:
jdbc:h2:mem:testdb;INIT=runscript from 'classpath:create_alias.sql
This is used in tests, executed for multiple projects concurrently on a Jenkins instance. Sometimes such tests would fail with following error:
Could not get JDBC Connection; nested exception is org.h2.jdbc.JdbcSQLException: Syntax error in SQL statement "javac: file not found: org/h2/dynamic/TO_DATE.java
Usage: javac <options> <source files>
use -help for a list of possible options
"; SQL statement:
create alias to_date as $$
java.util.Date toDate(java.lang.String dateString, java.lang.String pattern) {
....
My guess is that org.h2.util.SourceCompiler is assuming that there is only one instance of h2 running at the time and writes the generated Java source to 'java.io.tmpdir', which is shared among all processes running under same account. I propose following fix:
Index: SourceCompiler.java
===================================================================
--- SourceCompiler.java (revision 5086)
+++ SourceCompiler.java (working copy)
## -40,7 +40,15 ##
*/
final HashMap<String, Class<?>> compiled = New.hashMap();
- private final String compileDir = Utils.getProperty("java.io.tmpdir", ".");
+ private final String compileDir;
+
+ {
+ // use random folder under java.io.tmpdir so multiple h2 could compile at the same time
+ // without overwriting each other files
+ File tmp = File.createTempFile("h2tmp", ".tmp");
+ tmp.mkdir();
+ compileDir = tmp.getAbsolutePath();
+ }
static {
Class<?> clazz;
Should I open the support ticket or there are workarounds for this issue?
You can use javax.tools.JavaCompiler API and provide your own implementation for in-memory JavaFileManager to completely avoid creating those temp files.
BTW, Janino also support javax.tools.JavaCompiler API.
I had the same problem running multiple Jenkins executors and having Arquillian/Wildfly/H2 integration tests configuration. I found a workaround by setting java.io.tmpdir property to the build directory in the test standalone.xml.