curl -O "https://www.apache.org/dist/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz"
This is the command I am using from terminal to download maven but it's either timed out or curl: (7) Failed to connect to www.apache.org port 443: Operation timed out.
If I use browser to download, no issue.
My assumption is the ssl connection or certificate issue. Any idea how can I resolve the curl issue.
Please take note, I am using this in a Dockerfile to create docker image and here is that:
FROM ******/mule-42x:v2.2.1
ENV MAVEN_VERSION 3.6.3
RUN mkdir -p /opt/maven \
&& cd /opt/maven \
&& curl -O "https://www.apache.org/dist/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz" \
&& tar xzvf "apache-maven-$MAVEN_VERSION-bin.tar.gz" \
&& rm "apache-maven-$MAVEN_VERSION-bin.tar.gz"
ENV MAVEN_HOME "/opt/maven/apache-maven-$MAVEN_VERSION"
ENV PATH=$MAVEN_HOME/bin:$PATH
I've tried to run the command presented in the question and encountered the following errors:
curl -O "https://www.apache.org/dist/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz"
curl tries to verify the SSL certificate but fails with the following message:
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
So I've added -k flag as the message suggests.
Now this works, however the http call returns and HTML page with 302 (redirect) to https://downloads.apache.org/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
So the command that has worked for me is:
curl -O -k https://downloads.apache.org/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
An Important side note:
I'm assuming that you've configured the network right and it has all the proper proxy definitions if you're running behind the proxy in your organization, otherwise you should define proxy first.
All in all I suggest you running this command 'manually' first (from the command line not as a part of the build I mean) outside the docker on the machine where you run the docker build and only when you make sure it works run it in a docker file.
Related
I am trying to install and setup Elasticsearch on a remote Machine running CenOS7 via this guide. Following the install and start instructions for RPM I entered the cURL command given on the guide to check its up and running: curl --cacert $ES_PATH_CONF/certs/http_ca.crt -u elastic https://localhost:9200
I was met with the following message:
curl: (77) Problem with the SSL CA cert (path? access rights?)
I tried to go to the domain with the following command:
curl https://localhost:9200/
And was met with this:
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
So I tried the previous command once adding a '-k' but was met with this error:
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
I was able to complete the guide on my MacBook but have almost no experience working with remote machines or in CentOS 7. How do I successfully cURL the cluster information?
We have a superset docker containers which is using keycloak as identity broker. All this setup is working fine on http. Further, we have installed ssl certificate on keycloak and same is also working fine. Our superset and keycloak integration code changes look exactly like its mentioned in the answer here.
Now, when we changed auth uris from http to https in superset/docker/pythonpath_dev/client_secret.json, we are getting below error after the login flow is redirected from keycloak to superset.
Forbidden
'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)'
We also tried installing root certificates on superset by mounting them on /usr/local/share/ca-certificates and then executing update-ca-certificates in the container, but still there was no help. Any idea how this can be resolved?
Thanks #sventorben for the tip. Indeed it was python which was not able to read my ca files. Since I am new to this, I would detail out all the steps followed. However, some of these steps might be redundant.
After I received my root as well intermediary CA files, I first converted them to PEM format as they were in DER format using openssl.
openssl x509 -inform DER -in myintermediary.cer -out myintermediary.crt
openssl x509 -inform DER -in myroot.cer -out myroot.crt
Then, I mounted these files to my superset container at path /usr/local/share/ca-certificates/
Then, I logged into my container and executed update-ca-certificates command and verified that 2 new pem files got added at /etc/ss/certs/ path i.e. myroot.pem and intermediary.pem.
Then, I added these CA files to python certifi inside my container. To find out the path of cacert.pem, I executed below commands into python terminal.
import certifi
certifi.where()
exit()
Here, second command gave me the path of cacert.pem which was like /usr/local/lib/python3.7/site-pacakges/certifi/cacert.pem.
After this, i appended my ca files at the end of cacert.pem
cat /etc/ssl/certs/myroot.pem /etc/ssl/certs/intermediary.pem >> /usr/local/lib/python3.7/site-pacakges/certifi/cacert.pem
In the end i logged out of my container and restarted it.
docker-compose stop
docker-compose up -d
Note:
I feel step 3 is redundant as python does not read CA files from there. However, i still did it and I am in no mood of reverting and test it out again.
Also, this was my temporary fix as executing the commands inside the container is not useful as they are ephermal.
Update:
Below are the steps followed for production deployment.
Convert root certificates in PEM format using openssl.
Concat both PEM files into a new PEM file which will be installed as bundle. Lets say, the new PEM file is mycacert.pem and same is mounted at /app/docker/.
Create one sh file called start.sh and write 2 commands as below.
cat /app/docker/mycacert.pem >> /usr/local/lib/python3.7/site-pacakges/certifi/cacert.pem
gunicorn --bind 0.0.0.0:8088 --access-logfile - --error-logfile - --workers 5 --worker-class gthread --threads 4 --timeout 200 --limit-request-line 4094 --limit-request-field_size 8190 'superset.app:create_app()'
Modify docker-compose.yml and change command as below.
command: ["/app/docker/start.sh"]
Restart superset container.
docker-compose stop
docker-compose up -d
gitlab-ci-multi-runner register
gave me
couldn't execute POST against https://xxxx/ci/api/v1/runners/register.json:
Post https://xxxx/ci/api/v1/runners/register.json:
x509: cannot validate certificate for xxxx because it doesn't contain any IP SANs
Is there a way to disable certification validation?
I'm using Gitlab 8.13.1 and gitlab-ci-multi-runner 1.11.2.
Based on Wassim's answer, and gitlab documentation about tls-self-signed and custom CA-signed certificates, here's to save some time if you're not the admin of the gitlab server but just of the server with the runners (and if the runner is run as root):
SERVER=gitlab.example.com
PORT=443
CERTIFICATE=/etc/gitlab-runner/certs/${SERVER}.crt
# Create the certificates hierarchy expected by gitlab
sudo mkdir -p $(dirname "$CERTIFICATE")
# Get the certificate in PEM format and store it
openssl s_client -connect ${SERVER}:${PORT} -showcerts </dev/null 2>/dev/null | sed -e '/-----BEGIN/,/-----END/!d' | sudo tee "$CERTIFICATE" >/dev/null
# Register your runner
gitlab-runner register --tls-ca-file="$CERTIFICATE" [your other options]
Update 1: CERTIFICATE must be an absolute path to the certificate file.
Update 2: it might still fail with custom CA-signed because of gitlab-runner bug #2675
In my case I got it working by adding the path to the .pem file as following:
sudo gitlab-runner register --tls-ca-file /my/path/gitlab/gitlab.myserver.com.pem
Often, gitlab-runners are hosted in a docker container. In that case, one needs to make sure that the tls-ca-file is available in the container.
Ok I followed step by step this post http://moonlightbox.logdown.com/posts/2016/09/12/gitlab-ci-runner-register-x509-error and then it worked like a charm.
To prevent dead link I copy the steps below:
First edit ssl configuration on the GitLab server (not the runner)
vim /etc/pki/tls/openssl.cnf
[ v3_ca ]
subjectAltName=IP:192.168.1.1 <---- Add this line. 192.168.1.1 is your GitLab server IP.
Re-generate self-signed certificate
cd /etc/gitlab/ssl
sudo openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout /etc/gitlab/ssl/192.168.1.1.key -out /etc/gitlab/ssl/192.168.1.1.crt
sudo openssl dhparam -out /etc/gitlab/ssl/dhparam.pem 2048
sudo gitlab-ctl restart
Copy the new CA to the GitLab CI runner
scp /etc/gitlab/ssl/192.168.1.1.crt root#192.168.1.2:/etc/gitlab-runner/certs
Thanks #Moon Light #Wassim Dhif
The following steps worked in my environment. (Ubuntu)
Download certificate
I did not have access to the gitlab server. Therefore,
Open https://some-host-gitlab.com in browser (I use chrome).
View site information, usually a green lock in URL bar.
Download/Export certificate by navigating to certificate information(chrome, firefox has this option)
In gitlab-runner host
Rename the downloaded certificate with .crt
$ mv some-host-gitlab.com some-host-gitlab.com.crt
Register the runner now with this file
$ sudo gitlab-runner register --tls-ca-file /path/to/some-host-gitlab.com.crt
I was able to register runner to a project.
Currently there is no possibility to run the multi runner with an insecure ssl option.
There is currently an open issue at GitLab about that.
Still you should be able to get your certificate, make it a PEM file and give it to the runner command using --tls-ca-file
To craft the PEM file use openssl.
openssl x509 -in mycert.crt -out mycert.pem -outform PEM
In my setup the following the following worked as well. It's just important that IP/Name used for creating certificate matches IP/Name used for registering the runner.
gitlab-runner register --tls-ca-file /my/path/gitlab/gitlab.myserver.com.pem
Furthermore, it could be necessary to add a line for hostname lookup to the runners config.toml file also (section [runners.docker]):
extra_hosts = ["git.domain.com:192.168.99.100"]
see also https://gitlab.com/gitlab-org/gitlab-runner/issues/2209
In addition, there could be some network-trouble if for gitlab/gitlab-runner network-mode host is used, it has to be added to the config.toml as well, as it starts additional containers, which otherwise could have a problem to connect to the gitlab-host ((section [runners.docker]):
network_mode="host"
Finally, there might be an issue with the self-signed SSL-Cert (https://gitlab.com/gitlab-org/gitlab-runner/issues/2659).
A dirty workaround is to add
environment = ["GIT_SSL_NO_VERIFY=true"]
to the [[runners]] section.
I am trying to setup my dart http server to run only with https. So I gather I need to use HttpServer.bindSecure but I'm not clear from the description what needs to be passed in as certificateName and whether requestClientCertificate being true makes it more or less secure, or has no impact on security what so ever. The small sample code at the top of the HttpServer page passes in certificateName: 'localhost_cert' but before that it does something with a database, but doesn't seem to use it in anyway. Can anyone explain in more detail what these values are and what they need to be in order to make them secure?
The requestClientCertificate parameter of bindSecure is used to specify a client certificate. Client certificates are used by servers to identify and authorize clients, which appears not to be the objective of this question. It should be noted that there is a known issue with using client certificates in Dart on IE9 and Windows 7.
The certificateName parameter is used to specify the nickname of a certificate that exists in your certificate database. You specify the certificate nickname using the -n <nickname> option when importing a certificate to your database using certutil.
Use the following steps to:
Install the NSS utility (including certutil),
Create a new certificate database in directory <dir> with a password <password>, and
Import your self-signed or purchased certificate identified by nickname <host> such that it can be used to create an HTTPS server using the following sample code. Though the nickname can be chosen arbitrarily, we use the host name in this example. These steps have been confirmed working in Ubuntu 14.04 and Dart SDK 1.6 through (currently last stable version) 1.8.3.
Install the NSS utility
sudo apt-get install libnss3-tools
cd to the directory that will contain your certificate database
cd <dir>
Create a password file to use with the certificate database:
echo "<password>" > pwdfile
Create the certificate database
certutil -N -d 'sql:./' -f pwdfile
Either:
Generate a self-signed certificate:
certutil -S -s "cn=<host>" -n "self signed for dart" -x -t "C,C,C" -m 1000 -v 120 -d "sql:./" -k rsa -g 2048 -f pwdfile
where <host> is the host ("common name") for which to generate a certificate, for example "localhost"
Or, purchase a certificate by first creating a signing request for a real domain <host>, for example "myhost.com":
certutil -R -s "CN=<host>, O=None, L=San Diego, ST=California, C=US" -a -g 2048 -o <host>.csr -d "sql:./"
Then specify the content of file <host>.csr when prompted for a CSR upon purchasing a certificate from a signing authority.
Copy the purchased certificate to a file named <host>.crt
Import the certificate to the database
certutil -A -n <host> -t "p,p,p" -i <host>.crt -d "sql:./"
If necessary to use an intermediate certificate, it can be imported as such:
certutil -A -n my_intermediate_certificate -t "p,p,p" -i intermediate.crt -d "sql:./"
where "intermediate.crt" is the intermediate certificate file downloaded from the signing authority.
Verify that the certificates exist in the database
certutil -L -n <host> -d "sql:./"
certutil -L -n my_intermediate_certificate -d "sql:./"
To use this certificate and create an HTTPS server, do the following:
// Initialize secure socket to use certificate database (note: replace `<dir>`
// with the absolute path to the certificate database directory, and `<password>`
// with the value chosen above)
SecureSocket.initialize(database: "<dir>", password: "<password>");
// Bind secure HTTP server to specified host and port (typically 443)
HttpServer.bindSecure("<host>", 443, certificateName: "<host>")
.then((HttpServer httpServer) {
// Listen for incoming requests
httpServer.listen((HttpRequest httpRequest) {
// TODO: process request
});
})
.catchError((error) {
// TODO: handle error
});
Update
I don't have enough reputation points to respond to the comments, so here are additional details that may help answer the questions: Client certificates are not used to encrypt client-server communication and are not needed in the common scenario of establishing secure communication between a web browser and a webserver via HTTPS. The steps outlined above show how to create an HTTPS server in Dart using bindSecure.
I have downloaded cURL for Windows from here. I selected the Win64 - Generic version without SSL. I try to run curl https://www.google.com from the command line and I get the following error: curl: (1) Protocol https not supported or disabled in libcurl.
Okay! Next, I decide to download the version with SSL and install that. I try to run the same command as above and I get following error:
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle" of Certificate
Authority (CA) public keys (CA certs). If the default bundle file isn't adequate, you
can specify an alternate file using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
What should I do next?
Here's what I did to solve the problem.
I got the Bundle of CA root certificates that Mozilla uses from here. I copied the data from the page and saved it in a new file called curl-ca-bundle.crt in the folder where the curl.exe file was. That solved the problem.
Of course, these instructions are also found on the cURL documentation page for SSL certificates, precisely instructions 4 and 5.
Hope that helps.