I'm trying to use WebUpd8 team's oracle-java8-installer to install Java 8 on my Ubuntu 14.04 computers. Some of them could succeed but others failed. After some debugging, I realized it was caused by the HTTP proxy setting. I'll provide more details below, but basically my questions are: Why does the use of http_proxy cause the problem? I believe it's must be related to how an HTTP proxy works, but since I have little experience in that, could someone tell me what knowledge I should learn to understand this issue?
Here are more details.
Under the hood, the oracle-java8-installer uses wget to download the jdk-8u181 package. So I can reproduce the issue with the steps below:
Install apt-cacher-ng: sudo apt-get install apt-cacher-ng
You don't have to configure anything in the APT configuration to reproduce this problem. apt-cacher-ng uses localhost:3142 by default to cache the packages.
Run http_proxy="http://localhost:3142" wget --continue --no-check-certificate -O jdk-8u181-linux-x64.tar.gz --header "Cookie: oraclelicense=a" http://download.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jdk-8u181-linux-x64.tar.gz
Here are some notes:
The http://localhost:3142 is configured for apt-cacher-ng. Those machines that failed had apt-cacher-ng installed before I tried to install jdk-8u181.
The Cookie: oraclelicense=a is to indicate the user has accepted the license.
If you run the last command, the download of the jdk-8u181-linux-x64.tar.gz is finished instantly. There is a line saying "Proxy request sent, awaiting response... 200 OK". But if you open the received ".tar.gz", you'll see it's merely an HTML page that contains error information.
If you remove the http_proxy environment variable and run:
wget --continue --no-check-certificate -O jdk-8u181-linux-x64.tar.gz --header "Cookie: oraclelicense=a" http://download.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jdk-8u181-linux-x64.tar.gz
You will have the full package downloaded correctly.
My best guess is that an HTTP proxy works with wget if the target URL is the final URL, so the proxy would cache it in its storage. Conceptually, it's like a key-value store:
proxy['URL'] = result
However, in this case, the target URL (http://download.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jdk-8u181-linux-x64.tar.gz) actually returns a "302" code and a "Location" header field for the new URL. This can be seen from the output:
ywen#ubuntu:~$ wget --continue --no-check-certificate -O
jdk-8u181-linux-x64.tar.gz --header "Cookie: oraclelicense=a"
http://download.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jdk-8u181-linux-x64.tar.gz
--2018-08-01 11:10:04-- http://download.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jdk-8u181-linux-x64.tar.gz
Resolving download.oracle.com (download.oracle.com)... 23.32.72.143
Connecting to download.oracle.com
(download.oracle.com)|23.32.72.143|:80... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location:
https://edelivery.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jdk-8u181-linux-x64.tar.gz
[following]
--2018-08-01 11:10:04-- https://edelivery.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jdk-8u181-linux-x64.tar.gz
Resolving edelivery.oracle.com (edelivery.oracle.com)...
23.216.148.161, 2001:559:19:3081::2d3e, 2001:559:19:3086::2d3e
Connecting to edelivery.oracle.com
(edelivery.oracle.com)|23.216.148.161|:443... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location:
http://download.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jdk-8u181-linux-x64.tar.gz?AuthParam=1533136324_72efc4e6208a5a7fc1cbba0527c741b6
[following]
--2018-08-01 11:10:04-- http://download.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jdk-8u181-linux-x64.tar.gz?AuthParam=1533136324_72efc4e6208a5a7fc1cbba0527c741b6
Connecting to download.oracle.com
(download.oracle.com)|23.32.72.143|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 185646832 (177M) [application/x-gzip]
Saving to: ‘jdk-8u181-linux-x64.tar.gz’
Handling the redirection is out of the capability of a proxy (Am I right??), therefore those machines set with the HTTP proxies failed.
I'm having an issue with the Shield plugin for Elasticsearch. I have installed elasticsearch so that it runs as a service, and I can see that it is running on port 9200.
If I enter curl http://localhost:9200 in my Mac terminal I get the correct elasticsearch output...
Recently I have been trying to upgrade from basic authentication to shield authentication. I installed shield and license using the following commands:
/usr/share/elasticsearch/bin/plugin install license/latest
/usr/share/elasticsearch/bin/plugin install shield/latest
This successfull installs shield and license plugins. They are contained within my /usr/share/elasticsearch/plugins directory on my Vagrant box.
So, now if I try to use curl -XGET http://localhost:9200 again, I get the following response:
{"error":{"root_cause":[{"type":"security_exception","reason":"missing
authentication token for REST request
[/]","header":{"WWW-Authenticate":"Basic
realm=\"shield\""}}],"type":"security_exception","reason":"missing
authentication token for REST request
[/]","header":{"WWW-Authenticate":"Basic
realm=\"shield\""}},"status":401}
This is good. I can not reach elasticsearch without proper authentication. So now I want to create a user to authenticate to elasticsearch. I use the command:
sudo ./esusers useradd testuser -p password -r admin
This creates a user 'testuser' with password 'password' with admin privileges.
I can verify that this user has been created using the following command:
sudo ./esusers list
This returns:
testuser : admin
So I now try to run the curl command again with this user:
curl -u testuser:password -XGET http://localhost:9200
But I get the same error message as before when I tried without the admin user.
What is the issue here? Why is my admin user not authenticating?
You said that your verified the user is created using sudo ./esusers list but still the curl command fails. Chances are that you are using custom directory for elasticsearch even though the home dir is /usr/share/elasticsearch. And due to this, the esusers creates the users in /etc/elasticsearch/shield/ directory but they need to be copied to the custom dir, in case you are using one.
I know you've solved this differently but I'm answering this in the hope that it helps someone else in future. Can you confirm regarding the custom dir? For e.g in my case, the custom dir was /data/elasticsearch and the home dir was /usr/share/elasticsearch
I have 2 Linux Servers (with LAMP):
Web Server with SSL (https://www.example.com)
Admin Server (needs to connect to Web Server, via https)
When i connect from Admin Server (to Web Server) via curl command. It is refusing. Then when i use curl with --caeert option, its going through. Like this:
# curl --cacert CAchain.crt -I https://www.example.com
HTTP/1.1 200 OK
..
I'm getting 200 OK only because of --cacert CAchain.crt.
Then obviously i need the pure/basic curl command without defining the --cacert, to be working. Like:
# curl -I https://www.example.com
HTTP/1.1 200 OK
..
So that my Admin Application will for sure be able to connect to it (via https).
But now, when i connect to https://www.example.com from Admin Server (via its Application), it is bouncing back. Not able to reach, with SSL.
How do i make my Linux (RHEL) to install the client's CA-CERT inside, in order automatically AVOID defining the cert file. So that any communications to "https://www.example.com" via CURL or Web Browser (from Admin), can just then successfully go through. (Is it something like, we make "SSH without Keys" logic? But how, please?)
You need to add the CA cert to somewhere that curl can use it - it looks like you're just keeping it in your local directory (which isn't where curl looks for it - typically in some /etc/pki/ssl/ca-bundle.crt-type location). There's a handful of ways to do this. I don't have much experience doing it in RHEL (or CentOS), but have done it for Debian.
This ServerFault Post might help.
Likewise, This Post might help you install/import the CA cert properly.
I have an access to a preconfigured Windows XP 32-bit workstation.
It's under firewall and to get outside it uses http proxy server.
To get proxy settings I need host, port, username and password.
http: // username : password # server : port
I'm able to find the proxy url and port (from PAC - automatic config file),
but I don't know how to get username/password. Is there a way to read it somehow? At least username? I might get the admin rights.
Do you know any tool that can help?
I only found these MS docs about some methods to get proxy config connection, but I don't know how to determinate what's the provided credentials:
WinHttpGetIEProxyConfigForCurrentUser function
WINHTTP_CURRENT_USER_IE_PROXY_CONFIG structure
WinHttpGetProxyForUrl function
I've also found this lib, but it's rather for parsing PAC:
http://code.google.com/p/pacparser/
In windows you cannot get password.
To get connect throw a proxy you have to use SSPI, check libcurl, you can get a connected socket throw proxy, build it with enable-sspi and openssl.
Iterating on Fernando Sanchez's answer and Robert's comment, you need to authenticate via SSPI. In my case I've connected using NTLM using this link
With curl using ntlm :
curl.exe --proxy <proxy_name>:<proxy_port> --proxy-ntlm -U : https://www.google.com
Also to partially answer your question the -U : means from curl man page :
If you use a Windows SSPI-enabled curl binary and do either Negotiate
or NTLM authentication then you can tell curl to select the user name
and password from your environment by specifying a single colon with
this option: "-U :".
You can get the proxy name and port from the windows registry (regedit). Quote from Robert's comment:
[HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet
Settings]
It can be present either in a manual way using REG_SZ ProxyServer or using an autoconfiguration script using REG_SZ AutoConfigURL = https://<configuration_url>
If the curl command above fails with Failure when receiving data from the peer maybe you have a curl release without NTLM, SSPI or OpenSSL. Try with the latest curl release
Cntlm is an NTLM / NTLM Session Response / NTLMv2 authenticating HTTP proxy intended to help you break free from the chains of Microsoft proprietary world.
I have my proxy URL in the following format:
http://user:passwords#my_proxy_server.com:80
And I have to provide this information to cntlm. Its config file cntlm.ini has following structure and parameters:
Username
Domain
Password
Proxy
I am not sure, how to break up my original proxy property to fill these four options?
Update your user, domain, and proxy information in cntlm.ini, then test your proxy with this command (run in your Cntlm installation folder):
cntlm -c cntlm.ini -I -M http://google.ro
It will ask for your password, and hopefully print your required authentication information, which must be saved in your cntlm.ini
Sample cntlm.ini:
Username user
Domain domain
# provide actual value if autodetection fails
# Workstation pc-name
Proxy my_proxy_server.com:80
NoProxy 127.0.0.*, 192.168.*
Listen 127.0.0.1:54321
Listen 192.168.1.42:8080
Gateway no
SOCKS5Proxy 5000
# provide socks auth info if you want it
# SOCKS5User socks-user:socks-password
# printed authentication info from the previous step
Auth NTLMv2
PassNTLMv2 98D6986BCFA9886E41698C1686B58A09
Note: on linux the config file is cntlm.conf
The solution takes two steps!
First, complete the user, domain, and proxy fields in cntlm.ini. The username and domain should probably be whatever you use to log in to Windows at your office, eg.
Username employee1730
Domain corporate
Proxy proxy.infosys.corp:8080
Then test cntlm with a command such as
cntlm.exe -c cntlm.ini -I -M http://www.bbc.co.uk
It will ask for your password (again whatever you use to log in to Windows_). Hopefully it will print 'http 200 ok' somewhere, and print your some cryptic tokens authentication information. Now add these to cntlm.ini, eg:
Auth NTLM
PassNT A2A7104B1CE00000000000000007E1E1
PassLM C66000000000000000000000008060C8
Finally, set the http_proxy environment variable in Windows (assuming you didn't change with the Listen field which by default is set to 3128) to the following
http://localhost:3128
Without any configuration, you can simply issue the following command (modifying myusername and mydomain with your own information):
cntlm -u myusername -d mydomain -H
or
cntlm -u myusername#mydomain -H
It will ask you the password of myusername and will give you the following output:
PassLM 1AD35398BE6565DDB5C4EF70C0593492
PassNT 77B9081511704EE852F94227CF48A793
PassNTLMv2 A8FC9092D566461E6BEA971931EF1AEC # Only for user 'myusername', domain 'mydomain'
Then create the file cntlm.ini (or cntlm.conf on Linux using default path) with the following content (replacing your myusername, mydomain and A8FC9092D566461E6BEA971931EF1AEC with your information and the result of the previous command):
Username myusername
Domain mydomain
Proxy my_proxy_server.com:80
NoProxy 127.0.0.*, 192.168.*
Listen 127.0.0.1:5865
Gateway yes
SOCKS5Proxy 5866
Auth NTLMv2
PassNTLMv2 A8FC9092D566461E6BEA971931EF1AEC
Then you will have a local open proxy on local port 5865 and another one understanding SOCKS5 protocol at local port 5866.
Here is a guide on how to use cntlm
What is cntlm?
cntlm is an NTLM/NTLMv2 authenticating HTTP proxy
It takes the address of your proxy and opens a listening socket, forwarding each request to the parent proxy
Why cntlm?
Using cntlm we make it possible to run tools like choro, pip3, apt-get from a command line
pip3 install requests
choco install git
The main advantage of cntlm is password protection.
With cntlm you can use password hashes.
So NO PLAINTEXT PASSWORD in %HTTP_PROXY% and %HTTPS_PROXY% environment variables
Install cntlm
You can get the latest cntlm release from sourceforge
Note! Username and domain
My username is zezulinsky
My domain is local
When I run commands I use zezulinsky#local
Place your username when you run commands
Generate password hash
Run a command
cntlm -u zezulinsky#local -H
Enter your password:
Password:
As a result you are getting hashed password:
PassLM AB7D42F42QQQQ407552C4BCA4AEBFB11
PassNT PE78D847E35FA7FA59710D1231AAAF99
PassNTLMv2 46738B2E607F9093296AA4C319C3A259
Verify your generated hash is valid
Run a command
cntlm -u zezulinsky#local -M http://google.com
Enter your password
Password:
The result output
Config profile 1/4... OK (HTTP code: 301)
----------------------------[ Profile 0 ]------
Auth NTLMv2
PassNTLMv2 46738B2E607F9093296AA4C319C3A259
------------------------------------------------
Note! check that PassNTLMv2 hash is the same
The resulting hash is the same for both commands
PassNTLMv2 46738B2E607F9093296AA4C319C3A259
Change configuration file
Place generated hashes into the cntlm.ini configuration file
C:\Program Files (x86)\Cntlm\cntlm.ini
Here is how your cntlm.ini should look like
Username zezulinsky
Domain local
PassLM AB7D42F42QQQQ407552C4BCA4AEBFB11
PassNT PE78D847E35FA7FA59710D1231AAAF99
PassNTLMv2 46738B2E607F9093296AA4C319C3A259
Proxy PROXYSERVER:8080
NoProxy localhost, 127.0.0.*
Listen 3128
Note! newline at the end of cntlm.ini
It is important to add a newline at the end of the cntlm.ini configuration file
Set your environment variables
HTTPS_PROXY=http://localhost:3128
HTTP_PROXY=http://localhost:3128
Check that your cntlm works
Stop all the processes named cntlm.exe with process explorer
Run the command
cntlm -u zezulinsky#local -H
The output looks like
cygwin warning:
MS-DOS style path detected: C:\Program Files (x86)\Cntlm\cntlm.ini
Preferred POSIX equivalent is: /Cntlm/cntlm.ini
CYGWIN environment variable option "nodosfilewarning" turns off this warning.
Consult the user's guide for more details about POSIX paths:
http://cygwin.com/cygwin-ug-net/using.html#using-pathnames
section: local, Username = 'zezulinsky'
section: local, Domain = 'local'
section: local, PassLM = 'AB7D42F42QQQQ407552C4BCA4AEBFB11'
section: local, PassNT = 'PE78D847E35FA7FA59710D1231AAAF99'
section: local, PassNTLMv2 = '46738B2E607F9093296AA4C319C3A259'
section: local, Proxy = 'PROXYSERVER:8080'
section: local, NoProxy = 'localhost, 10.*, 127.0.0.*
section: local, Listen = '3128'
Default config file opened successfully
cntlm: Proxy listening on 127.0.0.1:3128
Adding no-proxy for: 'localhost'
Adding no-proxy for: '10.*'
Adding no-proxy for: '127.0.0.*'
cntlm: Workstation name used: MYWORKSTATION
cntlm: Using following NTLM hashes: NTLMv2(1) NT(0) LM(0)
cntlm: PID 1234: Cntlm ready, staying in the foreground
Open a new cmd and run a command:
pip3 install requests
You should have requests python package installed
Restart your machine
Congrats, now you have cntlm installed and configured
Just to add , if you are performing a "pip" operation , you might need to add and additional "--proxy=localhost:port_number"
e.g pip install --proxy=localhost:3128 matplotlib
Visit this link to see full details.
Once you generated the file, and changed your password, you can run as below,
cntlm -H
Username will be the same. it will ask for password, give it, then copy the PassNTLMv2, edit the cntlm.ini, then just run the following
cntlm -v
Thank you Sasha Zezulinsky.
In windows:
I used
SET HTTPS_PROXY=http://localhost:3128
When it was set to
SET HTTPS_PROXY=http://127.0.0.1:3128, it never worked.
Below posts are also very helpful.
How to use pip on windows behind an authenticating proxy
For me just using cntlm -H wasn't generating the right hash, but it does with the command below providing the user name.
If you need to generate a new password hash for cntlm, because you have change it or you've been forced to update it, you can just type the below command and update your cntlm.conf configuration file with the output:
$ cntlm -u test -H
Password:
PassLM D2AABAF8828482D5552C4BCA4AEBFB11
PassNT 83AC305A1582F064C469755F04AE5C0A
PassNTLMv2 4B80D9370D353EE006D714E39715A5CB # Only for user 'test', domain ''