I often use grep for openssl results when testing TLS security. For example:
$ openssl s_client -tls1_2 -connect 172.11.15.32:443 </dev/null | grep 'IS s'
depth=0 C = US, ST = asd, O = Billing, CN = asdasd, emailAddress = root#asdasd
verify error:num=18:self signed certificate
verify return:1
depth=0 C = US, ST = asd, O = Billing, CN = asdasd, emailAddress = root#asdasd
verify return:1
DONE
Secure Renegotiation IS supported
However, the issue is, that no matter what I grep for, output always contains these (or similar) lines in the beginning:
depth=0 C = US, ST = asd, O = Billing, CN = asdasd, emailAddress = root#asdasd
verify error:num=18:self signed certificate
verify return:1
depth=0 C = US, ST = asd, O = Billing, CN = asdasd, emailAddress = root#asdasd
verify return:1
Is it possible to somehow suppress these messages and receive only grep results as one would expect?
As indicated in comments, the problem is that the command openssl displays part of its output through stderr. Then, this will show no matter what you pipe.
So if you want to just show what grep has filtered to you, you have to previously redirect stderr to /dev/null so that it does not "jump the pipe":
openssl ... 2>/dev/null | grep 'IS s'
# ^^^^^^^^^^^
See another example of this:
$ touch hello
$ ls hello adlsfjaskldf
ls: cannot access adlsfjaskldf: No such file or directory # stderr
hello # stdout
Let's grep, where everything appears:
$ ls hello adlsfjaskldf | grep hello
ls: cannot access adlsfjaskldf: No such file or directory # stderr
hello # stdout
Let's grep but redirect stderr beforehand:
$ ls hello adlsfjaskldf 2>/dev/null | grep hello
hello # no "ls: cannot access..." here
Related
How to send an email with attachment using openssl s_client command as:
openssl s_client -crlf -quiet -starttls smtp -connect $SMTPHostName:587
using bash script
You can pipe smtp commands into the openssl s_client with
$ openssl s_client -crlf -quiet -starttls smtp -connect smtp.xs4all.nl:587 <<< QUIT
Result:
depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
verify return:1
depth=0 CN = smtp.xs4all.nl
verify return:1
250 8BITMIME
221 2.0.0 Bye
Available commands can be shown with 'help'.
The sequence of commands is usually something like this:
helo mail.example.com
data
To: dummy#dummy.nl
From: dummy#example.com
Date: Fri, 15 dec 2021 14:00:00 -0200
Message-ID: <xxxx.xxxx.xx#mail.example.com>
This is a test
.
NB. There may be an authentication needed.
To add an attachment something like the following needs to be added to the email.
Content-Type: image/png;
name="bullet.png"
Content-Transfer-Encoding: x-uuencode
begin 664 bullet.png
MB5!...
<snipped>
'
end
The file bullet.png was encoded with uuencode to ascii and added to the message. You can use other MIME formats.
Commands can vary from mailserver to mailserver depending on what for mailextensions and RFC's they support
I am trying to download some files with snakemake. The files (http://snpeff.sourceforge.net/SnpSift.html#dbNSFP) I would like to download are on a google site/drive and my usual wget approach does not work. I found a bash function that does the job (https://www.zachpfeffer.com/single-post/wget-a-Google-Drive-file):
function gdrive_download () { CONFIRM=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate "https://docs.google.com/uc?export=download&id=$1" -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p') wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$CONFIRM&id=$1" -O $2 rm -rf /tmp/cookies.txt }
gdrive_download 120aPYqveqPx6jtssMEnLoqY0kCgVdR2fgMpb8FhFNHo test.txt
I have tested this function with my ids in a plain bash script and was able to download all the files. To add a bit to the complexity, I must use a workplace template, and incorporate the function into it.
rule dl:
params:
url = 'ftp://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_{genome}/{afile}'
output:
'data/{genome}/{afile}'
params:
id1 = '0B7Ms5xMSFMYlOTV5RllpRjNHU2s',
f1 = 'dbNSFP.txt.gz'
shell:
"""CONFIRM=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate "https://docs.google.com/uc?export=download&id={{params.id1}}" -O- | sed -rn "s/.*confirm=([0-9A-Za-z_]+).*/\1\n/p") && wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$CONFIRM&id={{params.id1}}" -O {{params.f1}} && rm -rf /tmp/cookies.txt"""
#'wget -c {params.url} -O {output}'
rule checksum:
input:
i = 'data/{genome}/{afile}'
output:
o = temp('tmp/{genome}/{afile}.md5')
shell:
'md5sum {input} > {output}'
rule file_size:
input:
i = 'data/{genome}/{afile}'
output:
o = temp('tmp/{genome}/{afile}.size')
shell:
'du -csh --apparent-size {input} > {output}'
rule file_info:
"""md5 checksum and file size"""
input:
md5 = 'tmp/{genome}/{afile}.md5',
s = 'tmp/{genome}/{afile}.size'
output:
o = temp('tmp/{genome}/info/{afile}.csv')
run:
with open(input.md5) as f:
md5, fp = f.readline().strip().split()
with open(input.s) as f:
size = f.readline().split()[0]
with open(output.o, 'w') as fout:
print('filepath,size,md5', file=fout)
print(f"{fp},{size},{md5}", file=fout)
rule manifest:
input:
expand('tmp/{genome}/info/{suffix}.csv', genome=('GRCh37','GRCh38'), suffix=('dbNSFP.txt.gz', 'dbNSFP.txt.gz.tbi'))
#expand('tmp/{genome}/info/SnpSift{suffix}.csv', genome=('GRCh37','GRCh38'), suffix=('dbNSFP.txt.gz', 'dbNSFP.txt.gz.tbi'))
output:
o = 'MANIFEST.csv'
run:
pd.concat([pd.read_csv(afile) for afile in input]).to_csv(output.o, index=False)
There are four downloadable files for which I have ids (I only show one in params), however I don't know how to call the bash functions as written by ZPfeffer for all the ids I have with snakemake. Additionally, when I run this script, there are several errors, the most pressing being
sed: -e expression #1, char 31: unterminated `s' command
I am far from a snakemake expert, any assistance on how to modify my script to a) call the functions with 4 different ids, b) remove the sed error, and c) verify whether this is the correct url format (currently url = 'https://docs.google.com/uc?export/{afile}) will be greatly appreciated.
You would want to use raw string literal so that snakemake doesn't escape special characters, such as backslash in sed command. For example (notice r in front of shell command):
rule foo:
shell:
r"sed d\s\"
You could use --printshellcmds or -p to see how exactly shell: commands get resolved by snakemake.
Here is how I "solved" it:
import pandas as pd
rule dl:
output:
'data/{genome}/{afile}'
shell:
"sh download_snpsift.sh"
rule checksum:
input:
i = 'data/{genome}/{afile}'
output:
o = temp('tmp/{genome}/{afile}.md5')
shell:
'md5sum {input} > {output}'
rule file_size:
input:
i = 'data/{genome}/{afile}'
output:
o = temp('tmp/{genome}/{afile}.size')
shell:
'du -csh --apparent-size {input} > {output}'
rule file_info:
"""md5 checksum and file size"""
input:
md5 = 'tmp/{genome}/{afile}.md5',
s = 'tmp/{genome}/{afile}.size'
output:
o = temp('tmp/{genome}/info/{afile}.csv')
run:
with open(input.md5) as f:
md5, fp = f.readline().strip().split()
with open(input.s) as f:
size = f.readline().split()[0]
with open(output.o, 'w') as fout:
print('filepath,size,md5', file=fout)
print(f"{fp},{size},{md5}", file=fout)
rule manifest:
input:
expand('tmp/{genome}/info/{suffix}.csv', genome=('GRCh37','GRCh38'), suffix=('dbNSFP.txt.gz', 'dbNSFP.txt.gz.tbi'))
output:
o = 'MANIFEST.csv'
run:
pd.concat([pd.read_csv(afile) for afile in input]).to_csv(output.o, index=False)
And here is the bash script.
function gdrive_download () {
CONFIRM=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate "https://docs.google.com/uc?export=download&id=$1" -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')
wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$CONFIRM&id=$1" -O $2
rm -rf /tmp/cookies.txt
}
gdrive_download 0B7Ms5xMSFMYlSTY5dDJjcHVRZ3M data/GRCh37/dbNSFP.txt.gz
gdrive_download 0B7Ms5xMSFMYlOTV5RllpRjNHU2s data/GRCh37/dbNSFP.txt.gz.tbi
gdrive_download 0B7Ms5xMSFMYlbTZodjlGUDZnTGc data/GRCh38/dbNSFP.txt.gz
gdrive_download 0B7Ms5xMSFMYlNVBJdFA5cFZRYkE data/GRCh38/dbNSFP.txt.gz.tbi
How do I identify a certificate with a missing or incorrect value when it does not match my requirements?
Ideally if a cert is queried and there was not an answer, I'd like it to print a - for the corresponding certificate so I can look into the failure.
I'm creating a list containing specific content of PEM certificate files. The list will contain the CN or email value used in the cert.
To get the CN value I run:
openssl x509 -noout -subject -in certificate.pem | sed -n '/^subject/s/^.*CN=//p'
example.com
To get the email value I run:
openssl x509 -noout -email -in certificate2.pem
user#example.com
I tried this but it is not working as I thought:
while read common_names; do
openssl x509 -noout -email -in $common_names
if [[ -z $common_names ]] ; then
echo ""$common_names" Not valid smime cert"
fi
done < /user/audit/smime/smime_list.txt > /user/audit/smime/smime_cert_common_name.txt
The contents of /user/audit/smime/smime_list.txt
/var/certs/example1.com.crt
/var/certs/example2.com.crt
/var/certs/example3.com.crt
/var/certs/example4.com.crt
/var/certs/example5.com.crt
Current broken Output:
Cert_Name Common_Name Days_Expired
examle1.com.crt user#examle1.com 30
examle2.com.crt user#examle2.com 30
examle3.com.crt 30
examle4.com.crt 30
examle5.com.crt 30
Expected and Output without issues:
Cert_Name Common_Name Days_Expired
example.com.crt example.com 30
Expected output with an issue:
exception from a certificate with a missing Common_Name (CN)
Cert_Name Common_Name Days_Expired
example.com.crt - 30
or
Cert_Name Common_Name Days_Expired
example.com.crt N/A 30
PS. I already have the days expired from a previous block.
May i suggest this approach? First create list(an array) of certs.
certs=(
certificate1.pem
certificate2.pem
certificate3.pem
# and so on
)
or
certs=( $(ls /path/*.pem) )
And process it
XY () { printf "\e[${2};${1}H${3}"; } # use this function to print in columns
Y=1 # set start Y(lines) to 1 and print 1st row
XY 0 $Y "Cert_Name"; XY 10 $Y "Common_Name"; XY 20 $Y "Days_Expired"
for cert in "${carts[#]}"; {
((Y++)) # inc Y to print lines
common_name=$( code to get Common Name )
day_expired=$( code to get Days Expired )
XY 2 $Y "${cert:-'N/A'}"; XY 12 $Y "${common_name:-'N/A'}"; XY 22 $Y "${day_expired:-'N/A'}"
}
I used Ivan's suggestion but inside the array I added:
common_name=$(openssl x509 -noout -email -in $cert)
if [[ -z "$common_name" ]] ; then
common_name="-"
fi
and
day_left=$(/root/server-fixes/ssl-cert-check -b -c $cert | awk '{print $6}')
if [[ -z "$day_left" ]] ; then
day_left="-"
fi
to set the variable to my desired value if the result is null
I have the following encryption/decryption using OpenSSL (under Linux in my example):
$ echo test | openssl des -a -K 79616d7379616d73 -iv 1234567890ABCDEF
+ax5NT+Pjh0=
$ echo +ax5NT+Pjh0= | openssl des -a -d -K 79616d7379616d73 -iv 1234567890ABCDEF
test
All good. I need to translate it in Ruby code. As far as I did:
#!/usr/bin/env ruby
require 'openssl'
key = "\x79\x61\x6d\x73\x79\x61\x6d\x73"
iv = "\x12\x34\x56\x78\x90\xAB\xCD\xEF"
todecode = "+ax5NT+Pjh0="
def decode(encryptedString, key, iv)
decrypt = OpenSSL::Cipher::Cipher.new('des-cbc')
decrypt.decrypt
decrypt.key = key
decrypt.iv = iv
decrypt.update(encryptedString) + decrypt.final
end
decoded = decode(todecode, key, iv)
puts decoded
It throws me the following error:
decode.rb:14:in `final': wrong final block length (OpenSSL::Cipher::CipherError)`
What am I doing wrong? Did I select the wrong encryption or wrong use of key/iv?
Seems I forgot to base64_decode the string.
todecode = Base64::decode64("+ax5NT+Pjh0=")
I'm working with OpenSSL and need a sane default list of CAs. I'm using Mozilla's list of trusted CAs, as bundled by cURL. However, I need to split this bundle of CA certs, because the OpenSSL documentation says:
If CApath is not NULL, it points to a directory containing CA certificates in PEM format. The files each contain one CA certificate. The files are looked up by the CA subject name hash value, which must hence be available.
For example, using the ca-bundle.crt file directly works fine:
openssl-1.0.1g> ./apps/openssl s_client -connect www.google.com:443 -CAfile /home/user/certs/ca-bundle.crt
...
Verify return code: 0 (ok)
---
DONE
But specifying the directory containing the ca-bundle.crt file does not work:
openssl-1.0.1g> ./apps/openssl s_client -connect www.google.com:443 -CApath /opt/aspera/certs
Verify return code: 20 (unable to get local issuer certificate)
---
DONE
I presume this is because my folder doesn't adhere to what the documentation asks for (namely, a directory containing CA certs in PEM format, with each file containing one cert, named by hash value). My directory just has the single bundle of certs.
How can I split my bundle of certs to adhere to OpenSSL's request that each cert be in an individual file? Bonus points if the hashing can be done too (though if needed I could write a script to do that myself if all the certs are in individual files).
You can split the bundle with awk, like this, in an appropriate directory:
awk 'BEGIN {c=0;} /BEGIN CERT/{c++} { print > "cert." c ".pem"}' < ca-bundle.pem
Then, create the links OpenSSL wants by running the c_rehash utility that comes with OpenSSL:
c_rehash .
Note: use 'gawk' on non linux-platforms - as above relies on a GNU specific feature.
Just to give an alternative; facing the same issue I ended up with csplit:
csplit -k -f bar foo.pem '/END CERTIFICATE/+1' {10}
If you want to get a single certificate out of a multi-certificate PEM, try:
$ awk '/subject.*CN=host.domain.com/,/END CERTIFICATE/' INPUT.PEM
source
The following Ruby-script will split the bundle (with one or more certificates in it) into files named after the hashes -- side-stepping the c_rehash step in most cases.
To use, cd into the right directory (such as /etc/ssl/certs/) and run the script with the path to your certificate bundle as the sole argument. For example: ruby /tmp/split-certificates.rb ca-root-nss.crt.
#!/usr/bin/env ruby
require 'openssl'
blob = IO.binread(ARGV[0]) # Read the entire file at once
DELIMITER = "\n-----END CERTIFICATE-----\n"
blobs = blob.split(DELIMITER)
blobs.each do |blob|
blob.strip!
blob += DELIMITER # Does not break DER
begin
cert = OpenSSL::X509::Certificate.new blob
rescue
puts "Skipping what seems like junk"
next
end
begin
# XXX Need to handle clashes, suffix other than 0
filename=sprintf("%x.0", cert.subject.hash)
File.open(filename,
File::WRONLY|File::CREAT|File::EXCL) do |f|
f.write(blob)
end
rescue Errno::EEXIST
puts "#{filename} already exists, skipping"
end
end
Here is mine in Perl (so much code, but I like gonzo programming):
#!/usr/bin/perl -w
# -------
# Split "certificate bundles" like those found in /etc/pki/tls/certs into
# individual files and append the X509 cleartext description to each file.
#
# The file to split is given on the command line or piped via STDIN.
#
# Files are simply created in the current directory!
#
# Created files are named "certificate.XX" or "trusted-certificate.XX",
# with XX an index value.
#
# If a file with the same name as the output file already exists, it is not
# overwritten. Instead a new name with a higher index is tried.
#
# This works for bundles of both trusted and non-trusted certificates.
#
# See http://tygerclan.net/?q=node/49 for another program of this kind,
# which sets the name of the split-off files in function of the subject
# -------
my #lines = <> or die "Could not slurp: $!";
my $state = "outside"; # reader state machine state
my $count = 0; # index of the certificate file we create
my $fh; # file handle of the certificate file we create
my $fn; # file name of the certificate file we create
my $trusted; # either undef or "TRUSTED" depend on type of certificate
for my $line (#lines) {
chomp $line;
if ($state eq "outside") {
if ($line =~ /^(-----BEGIN (TRUSTED )?CERTIFICATE-----)\s*$/) {
my $marker = $1;
$trusted = $2;
$state = "inside";
my $created = 0;
my $prefix = "";
if ($trusted) {
$prefix = "trusted-"
}
while (!$created) {
$fn = "${prefix}certificate.$count";
$count++;
if (-f $fn) {
# print STDERR "File '$fn' exists; increasing version number to $count\n";
}
else {
print STDERR "Certificate data goes to file '$fn'\n";
open($fh,">$fn") || die "Could not create file '$fn': $!\n";
$created = 1;
print $fh "$marker\n"
}
}
}
else {
print STDERR "Skipping line '$line'\n"
}
}
else {
if ($line =~ /^(-----END (TRUSTED )?CERTIFICATE-----)\s*$/) {
my $marker = $1;
my $trustedCheck = $2;
if (!((($trusted && $trustedCheck) || (!$trusted && !$trustedCheck)))) {
die "Trusted flag difference detected\n"
}
$state = "outside";
print $fh "$marker\n";
print STDERR "Closing file '$fn'\n";
close $fh;
# Append x509 cleartext output by calling openssl tool
`openssl x509 -noout -text -in '$fn' >> '$fn'`;
if ($? != 0) {
die "Could not run 'openssl x509' command: $!\n";
}
}
else {
print $fh "$line\n"
}
}
}
if ($state eq "inside") {
die "Last certificate was not properly terminated\n";
}