I use MultipleOutputs to output data to some absolute paths, instead of a path relative to OutputPath.
Then, i get the error:
Error: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException): Failed to create file [/test/convert.bak/326/201505110030/326-m-00035] for [DFSClient_attempt_1425611626220_29142_m_000035_1001_-370311306_1] on client [192.168.7.146], because this file is already being created by [DFSClient_attempt_1425611626220_29142_m_000035_1000_-53988495_1] on [192.168.7.149] at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2320) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2083) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2012) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1963) at
https://issues.apache.org/jira/browse/MAPREDUCE-6357
Output files must in ${mapred.output.dir} 。
The design and implementation dosn't support outputing data to files out of ${mapred.output.dir}.
By looking into stack trace error, it seems that output file is already created.
If you want to write your data into multiple files, then try to generate those file name dynamically and use those files name as shown in code taken from Hadoop Definitive Guide
String basePath = String.format("%s/%s/part", parser.getStationId(), parser.getYear());
multipleOutputs.write(NullWritable.get(), value, basePath);
I hope this will help.
As it clearly suggests that the path you are trying to create,already exists. So try to do a check before creating that path whether that path exists or not.If exists, then delete that path.
FileSystem hdfs;
Path path = new Path (YourHadoopPath);
if (hdfs.exists(path)) {
hdfs.delete(path);
}
Related
I am implementing a diff package generation task in my project's gradle build from the git command line output. Currently I have a method which will give me a list of changed files from git diff --name-only. What I would like to do is create a directory structure in a new directory which matches the paths of each file. For example: inputting the string repo/dir/file.java would create in an output directory if not already created and inside it the directories head/repo/dir with the current file.java and prev/repo/dir with the previous file.java.
My current plan is to split the string repo/dir/file.java on the forward slash, and create directories until the last element of the split result, then write the file there. but nothing I have been able to come up with in gradle is nice or clean. I am wondering if there is a nicer way to create directories from a string like that.
My current plan is to split the string repo/dir/file.java on the forward slash, and create directories until the last element of the split result
Rather than splitting your string manually, you could try using File.mkdirs():
File newDirectoryStructureParent = new File('some/path/to/parent/dir')
def s = 'repo/dir/file.java'
def newContainer = new File(s, newDirectoryStructureParent).getParent()
newContainer.mkdirs()
everyone
In this part of my code you can just work around Path not File!
At the first you can define Path and second need check that path exist or not, if not mkdirs can make it ;)
Its help when you unknown about that path exist or not /
File fullPath = new File('/tmp/Test1')
if (!fullPath.exists())
fullPath.mkdirs()
I am looking for a way to add File(s) to an existing directory that has a random name as part of a Visual Studio Setup Project and I hoped someone might be able to help me solve this puzzle please.
I have been attempting to obtain the discovered path property of the directory using a Launch Condition; Unfortunately this method returns the full file path including the filename, which cannot be used as a directory property.
The directory in question takes the form [AppDataFolder]Company\Product\aaaaaaaaaaaa\
where aaaaaaaaaaaa is a random installation string.
Within the Launch Condition Setup I check for the directory's existence by searching for a file that would appear inside it,
Search Target Machine
(Name): File marker
Filename: sample.txt
Folder: [AppDataFolder]Company\Product\
Property: DIRFILE
Launch Condition
(Name): File marker exists
Condition: DIRFILE
In the Setup Project I add the file I wish to insert, with the details
Condition: DIRFILE
Folder: 'Installation folder'
Then in File System Setup I add a new folder entry for the random directory aaaaaaaaaaaa
(Name): Installation folder
Condition: DIRFILE
DefaultLocation: [DIRFILE]\..\ *Incorrect*
Property [DIRLOCATION]
As you can see the installer detects the existence of the marker file but, instead of placing my file at the same location, when using [DIRFILE] the installer would incorrectly try and insert it INTO the file;
This is because the file path was returned
[AppDataFolder]Company\Product\aaaaaaaaaaaa\sample.txt
where I instead need the directory path
[AppDataFolder]Company\Product\aaaaaaaaaaaa
Therefore I was wondering if it was possible to return the directory the file was found in from Search Target Machine (as opposed to the file location of the file), if I could extract the directory path by performing a string replace of the filename on the file location DIRFILE within the DefaultLocation field in File System Setup, or if perhaps there is even another method I am missing?
I'm also very interested in a simple solution for this, inside the setup project.
The way I did solve it was to install the files to a temporary location and then copy them to the final location in an AfterInstall event handler. Not a very elegant solution! Since it no longer care about the user selected target path I removed that dialog. Also I needed to take special care when uninstalling.
public override void OnAfterInstall(IDictionary savedState)
{
base.OnAfterInstall(savedState);
// Get original file folder
string originDir = Context.Parameters["targetdir"];
// Get new file folder based on the dir of sample.txt
string newDir = Path.GetDirectoryName(Context.Parameters["dirfile"]);
// Application executable file name
// (or loop for all files on the path instead)
string filename = "ApplicationName.exe";
// Move or copy the file
File.Move(Path.Combine(originDir, filename), Path.Combine(newDir, filename)));
}
Whenever I use the -lpf parameter with the pmcmd command, the workflow runs perfectly fine but when I add the same path in the Parameter FileName under Workflow 'Properties' and try to execute the workflow from the Workflow Manager, I get an error saying that parameter file is not found.
Now, the path which I am giving for '-lpf' is :
/apps/config/informatica/param.txt.
I don't understand why it works when I am overriding the parameter file name, whereas it doesn't work when I add it in the workflow properties (the file is not found).
By default, is any Informatica Environment variable set which needs to be changed and what's the default path of the parameter file on server and can this be changed?
Could you provide the log file?
Assuming I did understand this:
when you run the workflow with the parameter file -lpf that has this path:
/apps/config/informatica/param.txt
it does work, instead when you run it manually does not.
it could be so simply that manually you have to put instead of the extended path the string $PMSourceFileDir\ in the Source file Directory or to put it better: Source file Directory = $PMSourceFileDir\.
That because $PMSourceFileDir refer to the Informatica server initialization, as it is a server variable.
Instead with a parameter file usually is used to override that "deafult" path.
I am creating files from within Ruby scripts and adding stuff to them. But where are these files stored that I am creating?
I'm very new to this, sorry!
The files are created at whatever location you specified. For instance:
f = File.new("another_test.txt","w+")
that will create the file in the current working directory. You specify the path along with the file name. For example:
f = File.new("~/Desktop/another_test.txt","w+") # will create the file on the desktop.
For more details, check the File documentation.
Updated:
Included mu is too short correction.
I'm using JDBC to create a directory object in the database.
i.e create or replace directory "dir" as 'c:\temp';
My question is:
I want to pass the path at runtime
Or at least specify path relative to current one. I mean : create or replace "dir" as './../tempdir'
Is there a way to do this or is specifying absolute path the only way out.
From Oracle Doc:
path_name Specify the full path name of the operating system directory
of the server where the files are located.
I think this is because there is no such thing as "current directory". (You are in database, not in command line :) )
You can post a question with your problem and you'll get help from community.