How to submit wget jobs through a bash script to condor? - bash

Question

I have a problem submitting wget jobs to condor.
I can use wget to download a file from a url using command line.
$ wget https://wordpress.org/latest.zip
--2019-01-24 16:43:42-- https://wordpress.org/latest.zip
Resolving wordpress.org... 198.143.164.252
Connecting to wordpress.org|198.143.164.252|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11383968 (11M) [application/zip]
Saving to: “latest.zip”
100%[======================================>] 11,383,968 --.-K/s in 0.09s
2019-01-24 16:43:43 (117 MB/s) - “latest.zip” saved [11383968/11383968]
But if I save the command to a bash script file "test.sh", like this:
#!/bin/sh
wget https://wordpress.org/latest.zip
And then submit it to condor:
#!/usr/bin/env condor_submit
Executable = test.sh
Universe = vanilla
output = tmp.out
error = tmp.error
Log = tmp.log
Queue 1
It will have error as "Connection time out."
--2019-01-24 16:53:50-- https://wordpress.org/latest.zip
Resolving wordpress.org... 198.143.164.252
Connecting to wordpress.org|198.143.164.252|:443... failed: Connection timed out
But test.sh works well from command line as follows:
$./test.sh
I changed "tesh.sh" to:
#!/bin/sh
wget --debug --verbose https://wordpress.org/latest.zip
And the output is:
Setting --verbose (verbose) to 1
DEBUG output created by Wget 1.12 on linux-gnu.
--2019-01-24 17:25:58-- https://wordpress.org/latest.zip
Resolving wordpress.org... 198.143.164.252
Caching wordpress.org => 198.143.164.252
Connecting to wordpress.org|198.143.164.252|:443... Closed fd 3
failed: Connection timed out.

Answer 1

I suspect the problem is with "https": the SSL/TLS handshake is failing.
If you connect to an HTTPS site with a web browser, you use the browser's trust store (very often, this is one-in-the-same as your desktop's Windows trust store).
SUGGESTIONS:
Verify your "test.sh" works from the command line.
Modify your script to run wget --debug --trace ..., and check the output.
If only for troubleshooting purposes, try --no-check-certificate, too.
Look here for more details:
GNU Wget 1.18 Manual
ADDENDUM:
If you run wget --debug --verbose ..., you should see something like this:
$ wget --debug --verbose https://wordpress.org/latest.zip
Setting --verbose (verbose) to 1
Setting --verbose (verbose) to 1
DEBUG output created by Wget 1.19.4 on linux-gnu.
Reading HSTS entries from /home/xxxxx/.wget-hsts
URI encoding = ‘UTF-8’
Converted file name 'latest.zip' (UTF-8) -> 'latest.zip' (UTF-8)
--2019-01-24 15:49:12-- https://wordpress.org/latest.zip
Resolving wordpress.org (wordpress.org)... 198.143.164.252
Caching wordpress.org => 198.143.164.252
Connecting to wordpress.org (wordpress.org)|198.143.164.252|:443... connected.
Created socket 3.
Releasing 0x000055afcdc64650 (new refcount 1).
Initiating SSL handshake.
Handshake successful; connected socket 3 to SSL handle 0x000055afcdc64750
certificate:
subject: CN=*.wordpress.org,OU=Domain Control Validated
issuer: CN=Go Daddy Secure Certificate Authority - G2,OU=http://certs.godaddy.com/repository/,O=GoDaddy.com\\, Inc.,L=Scottsdale,ST=Arizona,C=US
X509 certificate successfully verified and matches host wordpress.org
---request begin---
GET /latest.zip HTTP/1.1
User-Agent: Wget/1.19.4 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: wordpress.org
Connection: Keep-Alive
...
If you DON'T see ANY of this ... I'd contact your network administrator about a firewall or proxy that might be blocking your Condor app.