I have been searching in mrunit documentation but hasnt been able to find it so far..
How do i pass configuration parameters in my mrunit.
So for example, if i take the wordcount example.
Lets say, in my driver code I am setting this parameter...
conf.set("delimiter",args[2])
And in my mapper code I am calling this as:
String delimiter = conf.get("delimiter");
String [] tokens = value.toString().split(delimiter);
for (String token:tokens)
context.write(token,one);
How do I set up this configuration parameter.
I have been looking into this example:
https://github.com/wpm/Hadoop-Word-Count/blob/master/src/test/java/wpmcn/hadoop/WordCountTest.java
Thanks
Use MapDriver.withConfiguration
Configuration conf = new Configuration();
conf.set("delimiter", someValue);
myMapDriver.withConfiguration(conf);
I had similar problem and I solved it as given in below code.
mapDriver.withInput(key, value);
mapDriver.getConfiguration().set("my.config.param", "my.config.param.value");
.....
.....
mapDriver.run();
Please note that mapDriver.getContext().getConfiguration may not work in this case, because the context object is a mocked object in the API.
Related
I have a Map in EL as ${map} and I am trying to get the value of it using a key which is by itself also an EL variable ${key} with the value "1000".
Using ${map["1000"]} works, but ${map["$key"]} does not work. What am I doing wrong and how can I get the Map value using a variable as key?
$ is not the start of a variable name, it indicates the start of an expression. You should use ${map[key]} to access the property key in map map.
You can try it on a page with a GET parameter, using the following query string for example ?whatEver=something
<c:set var="myParam" value="whatEver"/>
whatEver: <c:out value="${param[myParam]}"/>
This will output:
whatEver: something
See: https://stackoverflow.com/tags/el/info and scroll to the section "Brace notation".
I have faced this issue before. This typically happens when the key is not a String. The fix is to cast the key to a String before using the key to get a value from the map
Something like this:
<c:set var="keyString">${someKeyThatIsNotString}</c:set>
<c:out value="${map[keyString]}"/>
Hope that helps
You can put the key-value in a map on Java side and access the same using JSTL on JSP page as below:
Prior java 1.7:
Map<String, String> map = new HashMap<String, String>();
map.put("key","value");
Java 1.7 and above:
Map<String, String> map = new HashMap<>();
map.put("key","value");
JSP Snippet:
<c:out value="${map['key']}"/>
My five cents. Now I am working with EL 3.0 (jakarta impl) and I can access map value using three ways:
1. ${map.someKey}
2. ${map['someKey']}
3. ${map[someVar]} //if someVar == 'someKey'
I think that you should access your map something like:
${map.key}
and check some tutorials about jstl like 1 and 2 (a little bit outdated, but still functional)
Regarding above subject, is there any way to get the value of a field from a pipe. And use that value outside the pipe's scope in Hadoop Cascading? The data has delimiter as '|':
first_name|description
Binod|nothing
Rohit|nothing
Ramesh|abc
From above pipe I need to get a value from the description, whatever that is 'nothing' or 'abc'.
Hadoop Cascading is developed with a concept of creating real case scenario by flowing data between pipe and executing parallely it over Map-Reduce Hadoop system.
Execution of java program is unnecessary to depend with rest of the cascading flow (from creating source tap to sink tap), and what Hadoop Cascading does is: it executes those two different processes in different independent JVM instances and they will be unable to share their values.
Following code and its output shows brief hints:
System.out.println("Before Debugging");
m_eligPipe = new Each(m_eligPipe, new Fields("first_name"), new Debug("On Middle", true));
System.out.println("After Debugging");
Expected ouput:
Before Debugging
On Middle: ['first_name']
On Middle: ['Binod']
On Middle: ['Rohit']
On Middle: ['Ramesh']
After Debugging
Actual output:
Before Debugging
After Debugging
...
...
On Middle: ['first_name']
On Middle: ['Binod']
On Middle: ['Rohit']
On Middle: ['Ramesh']
I don't understand what you are trying to say. Do you to mean to extract the value of field ${description} outside the scope of the pipe. If possible something like this in pseudo code.
str = get value of description in inputPipe (which is in the scope of the job rather than function or buffer)
I assume this is what you want: you have a pipe with one field, that is the concatenation of ${first_name} and ${description}. And you want the output to be a pipe with field that is ${description}.
If so, this is what I'd do: implement a function that extracts description and have your flow execute it.
You function (let's call it ExtractDescriptionFunction) should override method operate with something like this:
#Override
public void operate(FlowProcess flowProcess, FunctionCall<Tuple> functionCall) {
TupleEntry arguments = functionCall.getArguments();
String concatenation = arguments.getString("$input_field_name");
String[] values = concatenation.split("\\|"); // you might want to have some data sanity check here
String description = values[1];
Tuple tuple = functionCall.getContext();
tuple.set(0, description);
functionCall.getOutputCollector().add(tuple);
}
Then, in your flow definition, add this:
Pipe outputPipe = new Each(inputPipe, new ExtractDescriptionFunction());
Hope this helps.
I need to get a list of all existing servers in the application.conf file, I take a look to EBean class, but i only found how to get an specific server Ebean.getServer("test"), also this returns an EbeanServer object, and i need a string value.
This is part of my application.conf:
db.default.driver=oracle.jdbc.OracleDriver
db.default.url="jdbc:oracle:thin:#//178.20.26.25:1521/orcl"
db.default.user="TEST1"
db.default.password="test1"
db.test.driver=oracle.jdbc.OracleDriver
db.test.url="jdbc:oracle:thin:#//178.20.26.26:1521/orcl"
db.test.user="TEST"
db.test.password="test"
ebean.default="models.*"
ebean.test="models.*"
My expected output is a list that contains (default,test). Does anybody know a way to get this without parsing hole file?
Thanks in advance.
Following code will give set instead of list:
Map<String, String> map = (Map<String, String>) play.Play.application().configuration().getObject("db");
Set<String> keys = map.keySet();
If you want to do it in type safe way and get rid of compiler warning:
Set<String> keys = play.Play.application().configuration().getConfig("db").subKeys();
Both examples will return subkeys of db key which is [default, test].
I am using Velocity 1.7 to format string and I had some trouble with default values. Velocity by itself has no special syntax for case when value is not set and we want to use some another, default value.
By the means of Velocity it looks like:
#if(!${name})Default John#else${name}#end
which is unconveniant for my case.
After googling I've found DisplayTool, according to documentation it will look like:
$display.alt($name,"Default John")
So I added maven dependency but not sure how to add DisplayTool to my method and it is hard to found instructions for this.
Maybe somebody can help with advice or give useful links?..
My method:
public String testVelocity(String url) throws Exception{
Velocity.init();
VelocityContext context = getVelocityContext();//gets simple VelocityContext object
Writer out = new StringWriter();
Velocity.evaluate(context, out, "testing", url);
logger.info("got first results "+out);
return out.toString();
}
When I send
String url = "http://www.test.com?withDefault=$display.alt(\"not null\",\"exampleDefaults\")&truncate=$display.truncate(\"This is a long string.\", 10)";
String result = testVelocity(url);
I get "http://www.test.com?withDefault=$display.alt(\"not null\",\"exampleDefaults\")&truncate=$display.truncate(\"This is a long string.\", 10)" without changes, but should get
"http://www.test.com?withDefault=not null&truncate=This is...
Please tell me what I am missing. Thanks.
The construction of the URL occurs in your Java code, before you invoke Velocity, so Velocity isn't going to evaluate $display.alt(\"not null\",\"exampleDefaults\"). That syntax will be valid only in a Velocity template (which typically have .vm extensions).
In the Java code, there's no need to use the $ notation, you can just call the DisplayTool methods directly. I've not worked with DisplayTool before, but it's probably something like this:
DisplayTool display = new DisplayTool();
String withDefault = display.alt("not null","exampleDefaults");
String truncate = display.truncate("This is a long string.", 10);
String url = "http://www.test.com?"
+ withDefault=" + withDefault
+ "&truncate=" + truncate;
It might be better, though, to call your DisplayTool methods directly from the Velocity template. That's what is shown in the example usage.
I have the following code:
File overitimeFile = new File(filePath+overtimeFileName);
FlatFileItemWriter<OvertimeSAPExport> overtimeItemWriter =
new FlatFileItemWriter<OvertimeSAPExport>();
overtimeItemWriter.setResource(new FileSystemResource(overitimeFile));
overtimeItemWriter.setShouldDeleteIfExists(true);
PassThroughLineAggregator<OvertimeSAPExport> lineAggregator =
new PassThroughLineAggregator<OvertimeSAPExport>();
overtimeItemWriter.setLineAggregator(lineAggregator);
overtimeItemWriter.open(new ExecutionContext());
List<OvertimeSAPExport> overtimeList = overtimeDAO.getSapOvertimeData(locationId, month);
overtimeItemWriter.write(overtimeList);
I have implemented the toString method for OvertimeSAPExport and when I debug I can see that it enters the toString once for each record in the list and gets te correct string from it.
It also creates the file without problems and throws no exceptions my way, but when I look at the file, it's empty.
Could someone PLEASE show me where my mistake is?
Try overtimeItemWriter.close(); and see if the file is flushed on disk. You also need to validate if a transaction is ongoing that postponed the writing.