Download the files that are bigger than a certain size - bash

Question

I have Files.txt, which consists of a list of URLs of Excel files:
http://www.bcheadset.ca/applicant.xlsx
http://www.bcheadset.ca/fte.xlsx
http://www.iutox.org/TRTF_Matrix2012_Oct.xlsx
http://www.journalprices.com/2013_FinalSummaryForWeb.xlsx
http://www.camts.org/7__2013_Aircraft_Checklist.xlsx
http://www.nanotr11.org/poster_program.xlsx
http://www.vliruos.be/media/6352100/nss2015_annex_3_budget.xlsx
...
What I want to do is to download (by wget) the files that are bigger than 10Mo.
To verify if a file is bigger than 10Mo before dowloading it, we could use curl --head. For instance, curl --head http://www.bcheadset.ca/fte.xlsx returns
HTTP/1.1 200 OK
Content-Length: 3158847
Content-Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Last-Modified: Mon, 27 Jul 2015 22:16:45 GMT
Accept-Ranges: bytes
ETag: "ffb49fecb9c8d01:c05"
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Date: Tue, 06 Oct 2015 15:14:27 GMT
And from Content-Length, we can verify that it is bigger than 10Mo.
My question is how to automate this by a piece of Bash or Perl. Especially I don't know how to extract the Content-Length and does a comparison...
Could anyone help?

Answer 1

You can use LWP and perl and send a HEAD request, fetching response headers. Something like this:
#!/usr/bin/env perl
use strict;
use warnings;
use LWP;
use Data::Dumper;
my $request = HTTP::Request -> new ( HEAD => 'http://www.google.com' );
my $agent = LWP::UserAgent -> new;
my $response = $agent -> request ( $request );
print $response -> header ( 'Content-Length');
And then use LWP to do a GET instead of a HEAD you want that file.

Answer 2

In order to adapt to various HTTP forms and their content some type of pattern matching may be prudent. Here is a rough example of how to use glob matching in shell to accomplish this:
#!/bin/sh
BUFFER=$(curl --head http://www.bcheadset.ca/fte.xlsx )
HOLD=""
TAKENEXT=0
for i in $BUFFER
do case "$i" in
Content-Length:)
TAKENEXT=1
;;
*) if [ "$TAKENEXT" -eq 1 ]
then HOLD="$i"
break
fi
;;
esac
done
printf "SIZE was: %s\n" "$HOLD"
Here is a simpler method using grep:
SIZE=$(curl --head http://www.bcheadset.ca/fte.xlsx | grep 'Content-Length:' | awk '{print $2}')
printf "%s\n" "$SIZE"
.