I am having a Micro-Service Platform having multiple Micro-Services connected to each other, Platform uses Kerberos for authentication of Micro-Services. In One of Micro-Service Node hadoop is installed which uses separate KDC for Hadoop cluster authentication.
Lets say platform domain is "idm.com" and hadoop domain is "hadoop.com".
Resource Manager is running on one node. I have configure HTTP principal for spnego in core-site.xml using "hadoop.http.authentication.kerberos.principal" property to "HTTP/master.hadoop.com#HADOOP.COM" and nodes Hostname is "hadoopmaster.idm.com".
I do Kinit and acquire root user ticket from TGS. When I tried to do curl using "curl -k -v --negotiate -u : https://master.hadoop.com:8090/cluster" It shows GSS Exception: No valid credentials provided.
If I see klist it shows two ticket one krbtgt and second "HTTP/hadoopmaster.idm.com#HADOOP.COM"(I have added this principal in kdc database). First krbtgt i got using kinit and second HTTP one i Got it automatically after doing curl before curl the ticket was not there. Krb client acquired another for using HTTP service.
After some debugging I noticed the problem/behaviour is I got ticket for HTTP/hadoopmaster.idm.com#HADOOP.COM where I have configure hadoop to use HTTP/master.hadoop.com#HADOOP.COM. If we configure hadoop to use "HTTP/hadoopmaster.idm.com#HADOOP.COM" then ui is accessible.
I have added both FQDNs to /etc/hosts file.
It seems when I do curl using any of the FQDNs I got the HTTP ticket of the first entry in /etc/hosts file.
For example if
...
10.7.0.5 hadoopmaster.idm.com
10.7.0.5 master.hadoop.com
...
now if i do curl i will get HTTP/hadoopmaster.idm.com#HADOOP.COM in klist.
and if /etc/hosts looks like this
...
10.7.0.5 master.hadoop.com
10.7.0.5 hadoopmaster.idm.com
...
Now if i do curl i will get HTTP/master.hadoop.com in klist
In both the cases if i configure the hadoop property to the same i got using curl then UI will be accessible and other wise it will shows 403 GSSException which i guess means curl used spnego but didn't get valid credentials.
And if it matches with the hadoop's configured principal then it will work.
It looks like Hostname is causing problem is there any way to map this hostname or is there any kerberos config which can map this or any property which will give me exact ticket with exact hostname i have specified in curl despite of hadoop configurations.
Related
For copying the file from HDFS to S3 bucket I used the command
hadoop distcp -Dfs.s3a.access.key=ACCESS_KEY_HERE\
-Dfs.s3a.secret.key=SECRET_KEY_HERE /path/in/hdfs s3a:/BUCKET NAME
But the access key and sectet key are visible here which are not secure .
Is there any method to provide credentials from file .
I dont want to edit config file ,which is one of the method I came across .
I also faced the same situation, and after got temporary credentials from matadata instance. (in case you're using IAM User's credential, please notice that the temporary credentials mentioned here is IAM Role, which attach to EC2 server not human, refer http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html)
I found only specifying the credentials in the hadoop distcp cmd will not work.
You also have to specify a config fs.s3a.aws.credentials.provider. (refer http://hortonworks.github.io/hdp-aws/s3-security/index.html#using-temporary-session-credentials)
Final command will look like below
hadoop distcp -Dfs.s3a.aws.credentials.provider="org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider" -Dfs.s3a.access.key="{AccessKeyId}" -Dfs.s3a.secret.key="{SecretAccessKey}" -Dfs.s3a.session.token="{SessionToken}" s3a://bucket/prefix/file /path/on/hdfs
Recent (2.8+) versions let you hide your credentials in a jceks file; there's some documentation on the Hadoop s3 page there. That way: no need to put any secrets on the command line at all; you just share them across the cluster and then, in the distcp command, set hadoop.security.credential.provider.path to the path, like jceks://hdfs#nn1.example.com:9001/user/backup/s3.jceks
Fan: if you are running in EC2, the IAM role credentials should be automatically picked up from the default chain of credential providers: after looking for the config options & env vars, it tries a GET of the EC2 http endpoint which serves up the session credentials. If that's not happening, make sure that com.amazonaws.auth.InstanceProfileCredentialsProvider is on the list of credential providers. It's a bit slower than the others (and can get throttled), so best to put near the end.
Amazon allows to generate temporary credentials that you can retrieve from http://169.254.169.254/latest/meta-data/iam/security-credentials/
you can read from there
An application on the instance retrieves the security credentials provided by the role from the instance metadata item iam/security-credentials/role-name. The application is granted the permissions for the actions and resources that you've defined for the role through the security credentials associated with the role. These security credentials are temporary and we rotate them automatically. We make new credentials available at least five minutes prior to the expiration of the old credentials.
The following command retrieves the security credentials for an IAM role named s3access.
$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access
The following is example output.
{
"Code" : "Success",
"LastUpdated" : "2012-04-26T16:39:16Z",
"Type" : "AWS-HMAC",
"AccessKeyId" : "AKIAIOSFODNN7EXAMPLE",
"SecretAccessKey" : "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
"Token" : "token",
"Expiration" : "2012-04-27T22:39:16Z"
}
For applications, AWS CLI, and Tools for Windows PowerShell commands that run on the instance, you do not have to explicitly get the temporary security credentials — the AWS SDKs, AWS CLI, and Tools for Windows PowerShell automatically get the credentials from the EC2 instance metadata service and use them. To make a call outside of the instance using temporary security credentials (for example, to test IAM policies), you must provide the access key, secret key, and the session token. For more information, see Using Temporary Security Credentials to Request Access to AWS Resources in the IAM User Guide.
if you do not want to use access and secret key (or show them on your scripts) and if your EC2 instance has access to S3 then you can use the instance credentials
hadoop distcp \
-Dfs.s3a.aws.credentials.provider="com.amazonaws.auth.InstanceProfileCredentialsProvider" \
/hdfs_folder/myfolder \
s3a://bucket/myfolder
Not sure if it is because of a version difference, but to use "secrets from credential providers" the -Dfs flag would not work for me, I had to use the -D flag as shown on the hadoop version 3.1.3 "Using_secrets_from_credential_providers" docs.
First I saved my AWS S3 credentials in a Java Cryptography Extension KeyStore (JCEKS) file.
hadoop credential create fs.s3a.access.key \
-provider jceks://hdfs/user/$USER/s3.jceks \
-value <my_AWS_ACCESS_KEY>
hadoop credential create fs.s3a.secret.key \
-provider jceks://hdfs/user/$USER/s3.jceks \
-value <my_AWS_SECRET_KEY>
Then the following distcp command format worked for me.
hadoop distcp \
-D hadoop.security.credential.provider.path=jceks://hdfs/user/$USER/s3.jceks \
/hdfs_folder/myfolder \
s3a://bucket/myfolder
I need to run curl commands from a Windows server to an API on a Linux box. When I am on my workstation I just run a kinit and give my user name and pwd for the Kerberos realm.
I have automated scripts in which I need to run a kinit before running the Powershell script. I do not want any manual intervention. I am trying to create a keytab file with my Kerberos account but it is not working -
I am receiving the error message "No key table entry found for user#domain.net while getting initial credentials.
I really don't understand much about Kerberos, and whether the process cares that I am on a Windows server tryng to use a keytab file for this purpose.
Can someone help? I have been going a bit nuts with this...
Thanks for any help!
Look into your keytab if the principal there is the one you are going to use. klist -k -t <keytab>
You are obviously doing wrong. You have to obtain a curl binary for Windows which says with curl --version:
curl 7.46.0 ...
Protocols: http https ...
Features: SSPI Kerberos SPNEGO ...
That's it. Enable SPNEGO auth with curl --negotiate -u : <URL> and you are done. No keytab necessary on Windows.
I am having a problem with my server and so far couldn't find any solution for this. When I try to add a server from a server manager (windows server 2012) I can see only the kerberos security error. Both servers are in the same domain(i have tried from several servers from domain and got the same error).
The strange thing is when I unjoin the problematic server from domain and rejoin it with another name it works normally. But the problem is to make it work with existing name. Anyhelp will be highly appreciated
thanks in advance.
Late reply, but I've just encountered the same error and hope this solution proves useful to others.
Situation: I had to wipe and reinstall a virtual server on which I'd previously had to set some Service Principal Names, and some SPNs for a service account. Turns out the SPNs were still there for the old server/account and I had to remove them.
I recommend checking for and removing rogue SPNs to resolve this. Use the following commands in an elevated command prompt:
setspn -l <servername/username>
In my case I had problems with MBAM, the Bitlocker admin tool, so for example I used:
setspn -l mbam01
Which gave me the output (changed names to protect the innocent):
Registered ServicePrincipalNames for CN=MBAM01,OU=Member Servers,DC=corp,DC=domainname,DC=com:
termserv/mbam01.corp.domainname.com
termserv/mbam01
http/mbam01.corp.domainname.com
http/mbam01
HOST/MBAM01
HOST/mbam01.corp.domainname.com
This will list the SPNs associated with the server or user account. Then you remove the errant SPNs with this command:
setspn -d <listed service> <servername/username>
In my case it turned out the mbamapppool user had http/mbam01 and http/mbam01.corp.domainname.com associated with it, causing Server Manager to fail to poll the server. I removed the http/ refs from the user and then added them to the server with the following commands:
setspn -d http/mbam01 corp\mbamapppooluser
setspn -d http/mbam.corp.domainname.com corp\mbamapppooluser
setspn -s http/mbam01 mbam01
setspn -s http/mbam01.corp.domainname.com mbam01
I then refreshed Server Manager and it polled the server successfully, and the Kerberos Security Error had gone.
I have an access to a preconfigured Windows XP 32-bit workstation.
It's under firewall and to get outside it uses http proxy server.
To get proxy settings I need host, port, username and password.
http: // username : password # server : port
I'm able to find the proxy url and port (from PAC - automatic config file),
but I don't know how to get username/password. Is there a way to read it somehow? At least username? I might get the admin rights.
Do you know any tool that can help?
I only found these MS docs about some methods to get proxy config connection, but I don't know how to determinate what's the provided credentials:
WinHttpGetIEProxyConfigForCurrentUser function
WINHTTP_CURRENT_USER_IE_PROXY_CONFIG structure
WinHttpGetProxyForUrl function
I've also found this lib, but it's rather for parsing PAC:
http://code.google.com/p/pacparser/
In windows you cannot get password.
To get connect throw a proxy you have to use SSPI, check libcurl, you can get a connected socket throw proxy, build it with enable-sspi and openssl.
Iterating on Fernando Sanchez's answer and Robert's comment, you need to authenticate via SSPI. In my case I've connected using NTLM using this link
With curl using ntlm :
curl.exe --proxy <proxy_name>:<proxy_port> --proxy-ntlm -U : https://www.google.com
Also to partially answer your question the -U : means from curl man page :
If you use a Windows SSPI-enabled curl binary and do either Negotiate
or NTLM authentication then you can tell curl to select the user name
and password from your environment by specifying a single colon with
this option: "-U :".
You can get the proxy name and port from the windows registry (regedit). Quote from Robert's comment:
[HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet
Settings]
It can be present either in a manual way using REG_SZ ProxyServer or using an autoconfiguration script using REG_SZ AutoConfigURL = https://<configuration_url>
If the curl command above fails with Failure when receiving data from the peer maybe you have a curl release without NTLM, SSPI or OpenSSL. Try with the latest curl release
I'm trying to run MPI programs on 8 machines, but I get the error
connect to address 127.0.0.1 port 544: Connection refused
Trying krb4 rsh...
connect to address 127.0.0.1 port 544: Connection refused
trying normal rsh (/usr/bin/rsh)
lagrid02: Connection refused
When I run it with a machinefile option, I get the error lagrid03: No route to host where lagrid03 is the neighbouring node connected to master node.
How should I rectify this ?
Regarding your first error, is rsh running on (all) the machine(s)? You'll need rsh or password-less ssh configured (and ask your mpi job launcher use ssh) before you can start jobs on different machines.
The second error indicates that there is no way to reach the machine lagrid03 with the current network config. I guess you have a /etc/hosts entry with the IP addresses for lagrid03, but you do not have an interface configured in that network. For a more detailed answer you'll need to post details about your network configuration.
The issue is with authentication, if you go into the /etc/pam.d/rsh file and move rlogin and rsh to the top and make it look like this, it would work just fine.
/* For root login to succeed here with pam_securetty, "rsh" must be listed in /etc/securetty.*/
auth required pam_nologin.so
auth required pam_securetty.so
auth required pam_env.so
auth required pam_rhosts_auth.so
account include system-auth
session optional pam_keyinit.so force revoke
session include system-auth