How to run Hadoop NodeManager as privileged account for Windows - windows

Question

I am working on the following versions:
OS- Windows 8
Hadoop- 2.3 (Single node cluster), Pig- 0.12.1, Hive- 0.13
I could start Namenode, Datanode, ResourceManager and NodeManager using windows command prompt (without admin privileges). When I try to run mapreduce program, Mapping task is not start, It show always as "Mapping-0%".
note:
If I start 'NodeManager' from windows command prompt with Administrator privileges then MapReduce program works fine.
NodeManager Log:
14/10/07 14:44:18 INFO http.HttpServer2: adding path spec: /node/*
14/10/07 14:44:18 INFO http.HttpServer2: adding path spec: /ws/*
14/10/07 14:44:18 INFO http.HttpServer2: Jetty bound to port 8042
14/10/07 14:44:18 INFO mortbay.log: jetty-6.1.26
14/10/07 14:44:18 INFO mortbay.log: Extract jar:file:/C:/kaveen/BDSDK/1
.1.0.8/SDK/Hadoop/share/hadoop/yarn/hadoop-yarn-common-2.3.0.jar!/webapps/node t
o C:\Users\kaveen\AppData\Local\Temp\Jetty_localhost_8042_node____j463i9\webapp
14/10/07 14:44:19 INFO mortbay.log: Started SelectChannelConnector#localhost:804
2
14/10/07 14:44:19 INFO webapp.WebApps: Web app /node started at 8042
14/10/07 14:44:19 INFO webapp.WebApps: Registered webapp guice modules
14/10/07 14:44:19 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0
:8031
14/10/07 14:44:19 INFO nodemanager.NodeStatusUpdaterImpl: Registering with RM us
ing finished containers :[]
14/10/07 14:44:19 INFO security.NMContainerTokenSecretManager: Rolling master-ke
y for container-tokens, got key with id 1336526610
14/10/07 14:44:19 INFO security.NMTokenSecretManagerInNM: Rolling master-key for
nm-tokens, got key with id :-93940432
14/10/07 14:44:19 INFO nodemanager.NodeStatusUpdaterImpl: Registered with Resour
ceManager as 127.0.0.1:49344 with total resource of <memory:8192, vCores:8>
14/10/07 14:44:19 INFO nodemanager.NodeStatusUpdaterImpl: Notifying ContainerMan
ager to unblock new container-requests
ResourceManager Log:
14/10/07 14:44:19 INFO util.RackResolver: Resolved 127.0.0.1 to /default-rack
14/10/07 14:44:19 INFO resourcemanager.ResourceTrackerService: NodeManager from
node 127.0.0.1(cmPort: 49344 httpPort: 8042) registered with capability: <memory
:8192, vCores:8>, assigned nodeId 127.0.0.1:49344
14/10/07 14:44:19 INFO rmnode.RMNodeImpl: 127.0.0.1:49344 Node Transitioned from
NEW to RUNNING
14/10/07 14:44:19 INFO capacity.CapacityScheduler: Added node 127.0.0.1:49344 cl
usterResource: <memory:16384, vCores:16>
14/10/07 14:44:19 INFO rmnode.RMNodeImpl: Node 127.0.0.1:49344 reported UNHEALTH
Y with details: 1/1 local-dirs turned bad: /tmp/hadoop-kaveen/nm-local-dir;1/1 l
og-dirs turned bad: C:/kaveen/BDSDK/1.1.0.8/SDK/Hadoop/logs/userlogs
14/10/07 14:44:19 INFO rmnode.RMNodeImpl: 127.0.0.1:49344 Node Transitioned from
RUNNING to UNHEALTHY
14/10/07 14:44:19 INFO capacity.CapacityScheduler: Removed node 127.0.0.1:49344
clusterResource: <memory:8192, vCores:8>
At cluster Node "http://localhost:8088/cluster/nodes" Look like below sceenshots.
No Nodes are created.
Do anyone know how to run 'NodeManager' without Admin privileges on windows?
Thanks in Advance...

Answer 1

A quick fix would be to run C:hadoop\sbin\start-dfs and C:hadoop\sbin\start-yarn under Administrator privileges.

Answer 2

local-dirs turned bad: /tmp/hadoop-kaveen/nm-local-dir;1/1
log-dirs turned bad: C:/kaveen/BDSDK/1.1.0.8/SDK/Hadoop/logs/userlogs
Your NM does not have a valid local-dirs, you have left Linux style local-dirs that are obviously not going to work on Windows. log-dirs are also incorrect, NM probably does not have the required privileges on it.
There is absolutely no need to elevate NM. Instead, fix local-dirs in yarn-site.xml:
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>C:/kaveen/BDSDK/1.1.0.8/SDK/Hadoop/local</value>
</property>
Then make sure the dirs exists and grant full control to NM:
c:\>md C:\kaveen\BDSDK\1.1.0.8\SDK\Hadoop\local
c:\>md C:\kaveen\BDSDK\1.1.0.8\SDK\Hadoop\logs
c:\>cacls C:\kaveen\BDSDK\1.1.0.8\SDK\Hadoop\local /E /T /G <nm account>:F
c:\>cacls C:\kaveen\BDSDK\1.1.0.8\SDK\Hadoop\logs/E /T /G <nm account>:F
Substitute with the appropriate account.