Are there any algorithms in Apache Spark to find out the frequent patterns in a text file. I tried following example but always end up with this error:
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:
/D:/spark-1.3.1-bin-hadoop2.6/bin/data/mllib/sample_fpgrowth.txt
Can anyone help me solve this problem?
import org.apache.spark.mllib.fpm.FPGrowth
val transactions = sc.textFile("...").map(_.split(" ")).cache()
val model = new FPGrowth()
model.setMinSupport(0.5)
model.setNumPartitions(10)
model.run(transactions)
model.freqItemsets.collect().foreach {
itemset => println(itemset.items.mkString("[", ",", "]") + ", " + itemset.freq)
}
try this
file://D:/spark-1.3.1-bin-hadoop2.6/bin/data/mllib/sample_fpgrowth.txt
or
D:/spark-1.3.1-bin-hadoop2.6/bin/data/mllib/sample_fpgrowth.txt
if not work, replace / with //
I assume you are running spark on windows.
Use file path like
D:\spark-1.3.1-bin-hadoop2.6\bin\data\mllib\sample_fpgrowth.txt
NOTE : Escape "\" if necessary .
Related
I need to generate the dynamic file path in the setup thread group like below.
def result_file = new File(org.apache.jmeter.services.FileServer.getFileServer().getBaseDir() + File.separator + 'transactions_passed_' + new Date().format('MM_dd_yyyy_HH_mm_ss') + '.csv');
props.put("result_file", result_file);
Now I want to pass that file path as a filename value of Flexible File Writer plugin of jmeter so that variables are stored inside it.
Not able to make it work. Kindly help. Thanks
I have tried below options:
Filename: ${__groovy(props.get("result_file").text)}
tried to use preprocessor and set the value:
vars.put("result_file", '${__FileToString(props.get("result_file"),,)}');
Also tried to use below groovy script in the FileName field of Flexible File Writer, however it throws an exception of FileNotFound exception:
${__groovy(new File(org.apache.jmeter.services.FileServer.getFileServer().getBaseDir() + System.getProperty('file.separator') + 'transactions_passed_' + new Date().format('MM_dd_yyyy_HH_mm_ss') + '.csv').text)}
I want to use DYNAMIC FILE PATH (which I am setting as property in setup thread group) in the FILENAME field of FLEXIBLE FILE WRITER
The approach with __groovy() function should work however you need to remove this .text bit from there
${__groovy(new File(org.apache.jmeter.services.FileServer.getFileServer().getBaseDir() + System.getProperty('file.separator') + 'transactions_passed_' + new Date().format('MM_dd_yyyy_HH_mm_ss') + '.csv'))}
because .text returns you the file contents and as the file doesn't exist - you're getting this FileNotFound error. More information on Groovy scripting in JMeter - Apache Groovy: What Is Groovy Used For?
I'm using ml-gradle to run a block of XQuery to update the MarkLogic database. The problem I am running into is I need to wrap all of the code in quotes, but since the code itself has quotes in it I am running into some errors when I try to declare variables i.e. let $config. Does anyone know a way around this? I was thinking I could concatenate all of the code into one big string so it ignores the first and last quotation.
task addCron(type: com.marklogic.gradle.task.ServerEvalTask) {
xquery = "xquery version \"1.0-ml\";\n" +
"import module namespace admin = \"http://marklogic.com/xdmp/admin\" at \"/MarkLogic/admin.xqy\";\n" +
"declare namespace group = \"http://marklogic.com/xdmp/group\";\n" +
" let $config := admin:get-configuration()\n" +
It bombs out when it is trying to declare $config as a variable. With the error:
> Could not get unknown property 'config' for task ':
Here is an example that works
task setSchemasPermissions(type: com.marklogic.gradle.task.ServerEvalTask) {
doFirst {
println "Changing permissions in " + mlAppConfig.schemasDatabaseName + " for:"
}
xquery = "xdmp:invoke('/admin/fix-permissions.xqy', (), map:entry('database', xdmp:database('" + mlAppConfig.schemasDatabaseName + "')))"
}
Here is some documentation for ServerEvalTask: https://github.com/marklogic-community/ml-gradle/wiki/Writing-your-own-task
I suspect you are hitting some string template mechanism in Groovy/Gradle. Try escaping the $ sign as well.
Note that you can use both single and double quotes in XQuery code.
HTH!
firstly,thanks for attention
i defined ftp adapter in my spring integration project and used mv command to move files in ftp server,directory structure is:
ftp-root
-----------Directory1\
-----------------in\
---------------------------file.in
-----------------out\
i want to move file file.in in ftp-root\Directory1\in\ directory to move ftp-root\Directory1\out\ with .out.rpt extension ftp-root\Directory1\out\a.out
i used int-ftp:outbound-gateway adapter to run mv command on ftp server,my code is:
<int-ftp:outbound-gateway id="gatewayMv"
session-factory="ftpSessionFactory"
expression="payload.remoteDirectory + '/' + payload.filename"
request-channel="mvChannel"
command="mv"
rename-expression="payload.remoteDirectory + '/' + payload.filename "
reply-channel="aggregateResultsChannel"/>
how to use SpEL expression to replace in with out in rename-expression option?
rename-expression="payload.remoteDirectory + '/' + payload.filename.replaceFirst('in', 'out')"
In most cases SpEL work like the regular Java. Since filename is a String you can apply for it any string operation.
thanks for #Artem Bilan ane #Gary for answer
the other way to working with string
define a bean as bellow :
public class StringUtil {
public String replacement(String value,String var1,String var2) {
return value.replace(var1,var2);
}
}
and in expression option set to:
rename-expression="#stringUtil.replacement(payload.remoteDirectory + '/' + payload.filename,'in','out')"
I am trying to read the contents of the file from a local disk as follows :
content = File.read("C:\abc.rb","r")
when I execute the rb file I get an exception as Error: No such file or directory .What am I missing in this?
In a double quoted string, "\a" is a non-printable bel character. Similar to how "\n" is a newline. (I think these originate from C)
You don't have a file with name "C:<BEL>bc.rb" which is why you get the error.
To fix, use single quotes, where these interpolations don't happen:
content = File.read('C:\abc.rb')
content = File.read("C:\/abc.rb","r")
First of all:
Try using:
Dir.glob(".")
To see what's in the directory (and therefore what directory it's looking at).
open("C:/abc.rb", "rb") { |io| a = a + io.read }
EDIT: Unless you're concatenating files together, you could write it as:
data = File.open("C:/abc.rb", "rb") { |io| io.read }
I am attempting to crawl through my FTP site with ftp.list(parent_path)
Whenever the parent_path variable contains a space, I get the following error
Ftp LIST exception: Net::FTPPermError detail: 550 /Download/Dimension: The system cannot find the file specified.
Ftp LIST exception: the parent_path (if present) was : /Download/Dimension Data
Here is my code snippet
begin
#logger.error("on #{ip} : " + ftp.system())
entry_list = parent_path ? ftp.list("#{parent_path}") : ftp.list
rescue => detail
retries_count += 1
#logger.error("on #{ip} : Ftp LIST exception: " + detail.class.to_s + " detail: " + detail.to_s)
#logger.error("on #{ip} : Ftp LIST exception: the parent_path (if present) was : " + parent_path)
I have tried escaping the spaces with a \ and I tried using %20, not sure what else to try...
Any ideas, thoughts, suggestions, etc, on how to get ftp.list to honor or escape the spaces is greatly appreciated!
Are you using Windows? This problem comes up when the ftp site's OS is Windows. I turned all of my spaces to underscore. I wish there was a better solution.