How can I extract URLs from the source code of a webpage? - bash

Question

I am trying to setup a stream for some scanners I have found on Broadcastify. The problem is that the URLs they use are dynamic and only the same for a few hours a time. I would like to create a shell script that can simply scan the page from which the stream is accessed (which does have a static URL) and return the current URL of the stream, which can then be fed to the audio player.
For instance, right now the following stream at https://www.broadcastify.com/listen/feed/30185/web has a stream at http://audio12.broadcastify.com/kq2ydfr1jz98shw.mp3
However, that stream link will only work for a short period of time. I need an MP3 stream like the one above.
I only have minor experience with shell scripting, so I'm wondering what the best approach would be here. Specifically, my first problem is if I simply "View page source" and search for "mp3", there are no results. I can only find the URL by inspecting element (F12 developer tools) and, in Chrome for instance, going to Application → Frames → Media. I thought I could do a "view frame source" on the audio player in the past but that option isn't there now.
I imagine I could use grep if I was able to CURL the source code, but I'm not sure what I would need to CURL here, if that makes sense.
UPDATE
Thanks mk12 for the insight. Based off that, here is my shell script:
#!/bin/bash
curl "https://www.broadcastify.com/listen/feed/$1/web" | grep webAuth > /var/tmp/broadcastifyauth$1.txt
pta=`cat /var/tmp/broadcastifyauth$1.txt | sed -i 's/$.ajaxSetup({ headers: { "webAuth": "//g' /var/tmp/broadcastifyauth$1.txt`
pta=`cat /var/tmp/broadcastifyauth$1.txt | sed -i 's/" }});//g' /var/tmp/broadcastifyauth$1.txt`
auth=`cat /var/tmp/broadcastifyauth$1.txt`
echo $auth
curl "https://www.broadcastify.com/listen/webpl.php?feedId=$1" --request POST --header "webAuth: $auth" --data 't=14' >/var/tmp/broadcastify$1.txt
pta=`cat /var/tmp/broadcastify$1.txt | grep -o 'http://[^"]*' > /var/tmp/broadcastify$1.b.txt`
pta=`cat /var/tmp/broadcastify$1.b.txt`
echo $pta
#pta=`cat /var/tmp/broadcastify$1.txt | sed -n '/<audio/s/^.*<audio width="300px" id="mePlayer_$1" src="\([^"]*\)".*/\1/p' > /var/tmp/broadcastify$1.b.txt`
#ptb=`cat /var/tmp/broadcastify$1.b.txt`
#echo $ptb
Here is its output:
root#na01:/etc/asterisk/scripts/music# ./broadcastify.sh 30185
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 9175 100 9175 0 0 51843 0 --:--:-- --:--:-- --:--:-- 52130
74f440ad812f0cc2192ab782e27608cc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 946 0 942 100 4 3851 16 --:--:-- --:--:-- --:--:-- 3844
http://relay.broadcastify.com/b94hfrp5k1s0tvy.mp3?xan=DCJP4HvtwMoXdH9HvtwMJ5vv342DfleDptcoX3dH9H48vtwMJ
Works!

Answer 1

The mp3 URL is not present in the original HTML document — it's added to the DOM later by JavaScript code. That's why you can't find it in "View page source," but you can with "Inspect element."
If you run curl https://www.broadcastify.com/listen/feed/30185/web, you will see the following somewhere in the middle:
<div id="fp" width="300px"></div>
<script>
$.ajaxSetup({ headers: { "webAuth": "74f440ad812f0cc2192ab782e27608cc" }});
$('#fp').load('/listen/webpl.php?feedId=30185',{t:14});
</script>
Note in particular that it loads content (using jQuery .load) into the initially empty <div id="fp"> just above. When you use "Inspect element" to find the audio player, you'll find it gets placed inside that div.
Before trying to reproduce this request with curl, I looked in the Network tab of the developer tools to see what the browser did. Filtering for "listen," I found the webpl.php request. Here is the relevant information from the "Headers" tab:
URL: https://www.broadcastify.com/listen/webpl.php?feedId=30185
Request
POST /listen/webpl.php HTTP/1.1
Content-Type: application/x-www-form-urlencoded
webAuth: 74f440ad812f0cc2192ab782e27608cc
Query String Parameters
feedId: 30185
Request Data
MIME Type: application/x-www-form-urlencoded
t: 14
Let's reproduce this request with curl:
curl 'https://www.broadcastify.com/listen/webpl.php?feedId=30185' \
--request POST \
--header 'webAuth: 74f440ad812f0cc2192ab782e27608cc' \
--data 't=14'
Here's the result:
<script src="/scripts/me_4.2.9/mediaelement-and-player.min.js"></script>
<link rel="stylesheet" href="/scripts/me_4.2.9/mediaelementplayer.min.css"/>
<audio width="300px" id="mePlayer_30185" src="http://relay.broadcastify.com/9wzfd3hrpyctvqx.mp3?xan=DCJP4HvtwMoXdH9HvtwMJ5vv342DfleDptcoX3dH9H48vtwMJ" type="audio/mp3" controls="controls"
autoplay="true">
</audio>
<script>
$('audio').mediaelementplayer({
features: ['playpause', 'current', 'volume'],
error: function () {
alert("Feed has disconnected from the server. This could be due to a power outage, network connection problem, or server problem. Click OK to restart the player. If the player fails to connect then the feed might be down for an extended timeframe.");
location.reload();
}
});
</script>
<br />
<div class="c">If the feed does not automatically play, click or touch the play icon in the player above.</div>
There's your mp3 link, in the src attribute of the <audio> tag. If we try to get it:
$ curl http://relay.broadcastify.com/9wzfd3hrpyctvqx.mp3?xan=DCJP4HvtwMoXdH9HvtwMJ5vv342DfleDptcoX3dH9H48vtwMJ
Moved Temporarily. Redirecting to http://audio13.broadcastify.com/9wzfd3hrpyctvqx.mp3?nocache=2623053&xan=DCJP4HvtwMoXdH9HvtwMJ5vv342DfleDptcoX3dH9H48vtwMJ
If you try to access that URL (or the original one with -L, instructing curl to follow redirects), the mp3 stream will start printing to your terminal as a bunch of nonsense characters.
So, your shell script should hit the /listen/webpl.php endpoint instead of trying to scrape the web player HTML page. Or perhaps just scrape the page to first get a webAuth token.
Update
In response to your update with the shell script, here is a simplified script that does the same thing and also strips the "Moved Temporarily" prefix to just get the audio url. Note that there's no need to use a temporary file, and the $(...) syntax is preferred over the `...` syntax:
#!/bin/bash
# I always start my scripts with this. See https://sipb.mit.edu/doc/safe-shell/
set -eufo pipefail
auth=$(curl -s "https://www.broadcastify.com/listen/feed/$1/web" \
| grep webAuth \
| head -n 1 \
| sed 's/^.*"webAuth": "//;s/".*$//')
relay_url=$(curl -s "https://www.broadcastify.com/listen/webpl.php?feedId=$1" \
-H "webAuth: $auth" -d 't=14' \
| grep -o 'http://[^"]*')
audio_url=$(curl -s "$relay_url" | cut -d' ' -f5)
echo "$audio_url"