I'm finding that a mapreduce job cant start due to some funniness in the RawLocalFileSystem, it appears.
How can I debug this error ? It appears that there is no trace of the directory or command which is associated with the NativeIO chmod exception.
One option will be of course to bundle a jar into my classpath with a custom RawLocalFileSystem implementation, but that seems like overkill.
13/07/11 18:39:43 ERROR security.UserGroupInformation: PriviledgedActionException as:root cause:ENOTDIR: Not a directory
ENOTDIR: Not a directory
at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)
This was an interesting bug: It turned out that i indeed had a FILE created which existed, already, in a place where a directory was needed to be created.
That is , some how, I had a file called "tmp" in my file system implementation's root directory !
In any case, the confusion was due to scarce error reporting by the hadoop NativeIO class.
I think it is a failure which should be reported and logged better by the underlying NativeIO class .
Related
I'm using Hortonworks sandbox and trying to run a simple pig script. There appear to be annoying error related to "file does not exist".
Below is the script:
REGISTER '/piggybank.jar';
inp = load '/my.csv' USING org.apache.pig.piggybank.storage.CSVExcelStorage..
ERROR 2997: Encountered IOException. File does not exist:
hdfs://sandbox.hortonworks.com:8020/tmp/udfs/ '/piggybank.jar'
However, my jar is present at the root(/) and I have given proper permission as well. Don't know why the path is pointing to /tmp/udfs....
Can anyone provide some suggestion?
Do not place the path within quotes. Also provide full URI of the Jar file location.
REGISTER hdfs://sandbox.hortonworks.com:8020/piggybank.jar;
Refer REGISTER (a jar/script).
Following an install of Eclipse 3.7.2 on my Ubuntu 12.04 development machine, I have been unable to execute any ODI packages/interfaces/procedures. On execution (for both simulated and actual runs), an error is thrown (java trace below). I am not sure if it's anything to do with the Eclipse install, but it seems likely. Does anyone have an idea how to fix this?
Also, when launching ODI from the terminal using 'bash odi', the following error is displayed in the terminal:
2013-08-15 14:43:46.162 ERROR Error during RuntimeClassLoader initialization. ODI will start without RuntimeClassLoader
Error output:
oracle.odi.core.exception.OdiRuntimeException: Error during Code Interpretor creation
at com.sunopsis.dwg.codeinterpretor.SnpCodeInterpretor.getInstance(SnpCodeInterpretor.java:209)
at com.sunopsis.dwg.codeinterpretor.SnpGeneratorSQLCIT.<init>(SnpGeneratorSQLCIT.java:300)
at com.sunopsis.graphical.dialog.SnpsDialogExecution.doPackageExecuter(SnpsDialogExecution.java:907)
at oracle.odi.ui.action.SnpsPopupActionExecuteHandler.actionPerformed(SnpsPopupActionExecuteHandler.java:68)
at oracle.odi.ui.SnpsActionControler.handleEvent(SnpsActionControler.java:75)
at oracle.ide.controller.IdeAction.performAction(IdeAction.java:529)
at oracle.ide.controller.IdeAction.actionPerformedImpl(IdeAction.java:884)
at oracle.ide.controller.IdeAction.actionPerformed(IdeAction.java:501)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1995)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2318)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:387)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:242)
at javax.swing.AbstractButton.doClick(AbstractButton.java:357)
at javax.swing.plaf.basic.BasicMenuItemUI.doClick(BasicMenuItemUI.java:809)
at javax.swing.plaf.basic.BasicMenuItemUI$Handler.mouseReleased(BasicMenuItemUI.java:850)
at java.awt.Component.processMouseEvent(Component.java:6297)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3275)
at java.awt.Component.processEvent(Component.java:6062)
at java.awt.Container.processEvent(Container.java:2039)
at java.awt.Component.dispatchEventImpl(Component.java:4660)
at java.awt.Container.dispatchEventImpl(Container.java:2097)
at java.awt.Component.dispatchEvent(Component.java:4488)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4575)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4236)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4166)
at java.awt.Container.dispatchEventImpl(Container.java:2083)
at java.awt.Window.dispatchEventImpl(Window.java:2489)
at java.awt.Component.dispatchEvent(Component.java:4488)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:674)
at java.awt.EventQueue.access$400(EventQueue.java:81)
at java.awt.EventQueue$2.run(EventQueue.java:633)
at java.awt.EventQueue$2.run(EventQueue.java:631)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:98)
at java.awt.EventQueue$3.run(EventQueue.java:647)
at java.awt.EventQueue$3.run(EventQueue.java:645)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:644)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
Caused by: org.apache.bsf.BSFException: unable to load language: beanshell
at org.apache.bsf.BSFManager.loadScriptingEngine(BSFManager.java:718)
at com.sunopsis.dwg.codeinterpretor.SnpCodeInterpretor.loadEngine(SnpCodeInterpretor.java:85)
at com.sunopsis.dwg.codeinterpretor.SnpCodeInterpretor.<init>(SnpCodeInterpretor.java:75)
at com.sunopsis.dwg.codeinterpretor.SnpCodeInterpretor.getInstance(SnpCodeInterpretor.java:184)
... 45 more
After digging around for about a day on this issue, I brazenly tried running ODI as the root user on the off chance that this was a permissions issue. I started ODI from the command line (using 'bash odi') for greater verbosity, and it loaded without the error mentioned above. Something gave me the impression that this wasn't a permissions issue, but one related to the user settings.
To rectify the issue, I removed my user's odi settings folder (renaming it, for safety):
mv ~/.odi ~/.backup_odi
Then I started ODI from the terminal under my own user (i.e. not root) - there were no errors! None of my connections were available in the new settings folder though. This I fixed by closing ODI and entering the following:
cp ~/.backup_odi/oracledi/snps_login_work.xml ~/.odi/oracledi/
If anybody else encounters this issue, I hope you find this post quicker than it took me to fix it!
org.apache.bsf.BSFException: unable to load language: beanshell
The exception was thrown because bsh-2.Ob4.jar was not in the classpath and it is a dependent jar of bsf.jar
I have found the mapping sdk code samples, however without any documentation whatsoever.
Currently working with 9.0, I am looking for more info.
For one of the current issues, when I try to save the mapping to repository, (with pcconfig.properties lying in the same folder where the xml file is being generated), I am getting the following error:
Written the file..
Caught an exception in run() method
java.io.IOException: Cannot run program ""C:\Informatica\pmrep"" (in directory "C:\Informatica"): CreateProcess error=2, The system cannot find the file specified
java.io.IOException: Cannot run program ""C:\Informatica\pmrep"" (in directory "C:\Informatica"): CreateProcess error=2, The system cannot find the file specified
com.informatica.powercenter.sdk.mapfwk.exception.MapFwkOutputException: Error saving to repository : Failed to connect to repository
at com.informatica.powercenter.sdk.mapfwk.xml.XMLWriter.save(Unknown Source)
at com.informatica.powercenter.sdk.mapfwk.repository.Repository.save(Unknown Source)
at TestRaghavExample.generateOutput(TestRaghavExample.java:259)
at TestRaghavExample.create(TestRaghavExample.java:64)
at TestRaghavExample.main(TestRaghavExample.java:272)
Caught an exception in run() method
This is my initial example.. trying to find my way through the API..
The path to the pmrep utility doesn't look correct. It is normally found at the path C:\Informatica\<version>\server\bin on Windows.
Check the value of PC_CLIENT_INSTALL_PATH within pcconfig.properties is correct.
I am trying to load large data to HDFS and I sometimes get the error below. any idea why?
The error:
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /data/work/20110926-134514/_temporary/_attempt_201109110407_0167_r_000026_0/hbase/site=3815120/day=20110925/107-107-3815120-20110926-134514-r-00026 File does not exist. Holder DFSClient_attempt_201109110407_0167_r_000026_0 does not have any open files.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1557)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1548)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:1603)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:1591)
at org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:675)
at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
at org.apache.hadoop.ipc.Client.call(Client.java:1107)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
at $Proxy1.complete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.complete(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3566)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3481)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:966)
at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.close(SequenceFile.java:1297)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:78)
at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs$RecordWriterWithCounter.close(MultipleOutputs.java:303)
at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:456)
at com.my.hadoop.platform.sortmerger.MergeSortHBaseReducer.cleanup(MergeSortHBaseReducer.java:145)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:178)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:572)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:414)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
I managed to fix the problem:
When the job ends he deletes /data/work/ folder. If few jobs are running in parallel the deletion will also delete the files of the another job. actually I need to delete /data/work/.
In other words this exception is thrown when the job try to access to files which are not existed anymore
I meet the same problem when i use spark streaming to saveAsHadoopFile to Hadoop(2.6.0-cdh5.7.1), of course i use MultipleTextOutputFormat to write different data to different path. Sometimes the exception what Zohar said would happen. The reason is as Matiji66 say:
another program read,write and delete this tmp file cause this error.
but the root reason he didn't talk about is the hadoop speculative:
Hadoop doesn’t try to diagnose and fix slow running tasks, instead, it tries to detect them and runs backup tasks for them.
So the really reason is that, your task execute slow, then hadoop run another task to do the same thing(in my case is to save data to a file on hadoop), when one task of the two task finished, it will delete the temp file, and the other after finished, it will delete the same file, then it does not exists, so the exception
does not have any open files
happened
you can fix it by close the speculative of spark and hadoop:
sparkConf.set("spark.speculation", "false");
sparkConf.set("spark.hadoop.mapreduce.map.speculative", "false");
sparkConf.set("spark.hadoop.mapreduce.reduce.speculative", "false")
For my case, another program read,write and delete this tmp file cause this error.
Try to avoid this.
ROOT CAUSE
Storage policy was set on staging directory and hence MAPREDUCE job failed.
<property>
<name>yarn.app.mapreduce.am.staging-dir</name>
<value>/user</value>
</property>
RESOLUTION
Setup staging directory for which storage policy is not setup. I.e. modify yarn.app.mapreduce.am.staging-dir in yarn-site.xml
<property>
<name>yarn.app.mapreduce.am.staging-dir</name>
<value>/tmp</value>
</property>
I use Sqoop to import into HDFS and have same error. By the help of previous answers I have realized that I needed to remove last "/" from --target-dir /dw/data/
I used --target-dir /dw/data
works fine
I encountered this problem when I changed my program to use saveAsHadoopFile method to improve performance, in which scenario I can't make use of DataFrame API directly. see the problem
The reason why this would happen is basically what Zohar said, the saveAsHadoopFile method with MultipleTextOutputFormat actually doesn't allow multiple programs concurrently running to save files to the same directory. Once a program finished, it would delete the common _temporary directory the others still need, I am not sure if it's a bug in M/R API. (2.6.0-cdh5.12.1)
You can try this solution below if you can't redesign your program:
This is the source code of FileOutputCommitter in M/R API: (you must download an according version)
package org.apache.hadoop.mapreduce.lib.output;
public class FileOutputCommitter extends OutputCommitter {
private static final Log LOG = LogFactory.getLog(FileOutputCommitter.class);
/**
* Name of directory where pending data is placed. Data that has not been
* committed yet.
*/
public static final String PENDING_DIR_NAME = "_temporary";
Changes:
"_temporary"
To:
System.getProperty("[the property name you like]")
Compiles the single Class with all required dependencies, then creates a jar with the three output class files and places the jar to you classpath. (make it before the original jar)
Or, you can simply put the source file to your project.
Now, you can config the temp directory for each program by setting a different system property.
Hope it can help you.
I'm using Teamcity 5 for our CI environment. It's a great tool, but I've been struggling with one thing: the stdout_yyyyMMdd.log file in the \TeamCity\logs folder grows to ridiculous sizes. Is there a way to turn it off?
Places I've looked so far:
Jetbrains: Nothing on stdout;
Google for "tomcat stdout logs": the first few links don't really address the issue.
Edit:
At KIR's suggestion, I actually looked to see what's in stdout. It's the same exception message repeated over and over again:
[2010-12-01 08:57:21,268] WARN - jetbrains.buildServer.SERVER - java.io.FileNotFoundException: <...Path...>\.BuildServer\system\caches\search\_8p.prx (The system cannot find the file specified)
[2010-12-01 08:57:21,315] ERROR - erverSide.search.SearchService - SearchService.enqueueHistory
java.io.FileNotFoundException: <...Path...>\.BuildServer\system\caches\search\_8p.prx (The system cannot find the file specified)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(Unknown Source)
at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:78)
at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:108)
at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:65)
at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:132)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:599)
at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:104)
at org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:27)
at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:74)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:704)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:69)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:476)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:314)
at jetbrains.buildServer.serverSide.search.SearchService.getIndexSearcher(SearchService.java:451)
at jetbrains.buildServer.serverSide.search.SearchService.enqueueHistory(SearchService.java:515)
at jetbrains.buildServer.serverSide.search.BackgroundIndexer.run(BackgroundIndexer.java:32)
at java.lang.Thread.run(Unknown Source)
Any idea what this file is?
If you're running TC on unix you could use logrotate: http://linuxcommand.org/man_pages/logrotate8.html (Obviously, this is a workaround but it should be effective.)
This guy has a windows equivalent that may do the trick too: http://www.datori.org/?p=7
Remove .BuildServer\system\caches\search directory and restart TeamCity. May be this would help.
The problem is caused by someone or something deleting the Lucene Index in Team city.
Everytime you hit a page after this it will log in stdout that it couldnt find the file.
If you clear out the whole folder which should be
%USERPROFILE%.BuildServer\system\caches\search\
See http://confluence.jetbrains.net/display/TCD5/TeamCity+Data+Directory for more information on where to find the folder.
And restart Teamcity it will recreate the index on start up and stop logging the error message.
Oh and search should start working again too.