I am writing a program in bash and want to use curl to display the contents of a website. The website url is http://examplewebsite.com/example.php?pass=hello hello:world. Using:
curl http://examplewebsite.com/example.php?pass=hello hello:world
However this returns:
Couldn't resolve host 'hello:world'
The error was caused by the space between “hello” and “world” that caused bash to split the link into two different tokens, which curl interpreted as two different links.
You should at least quote the URL parameters, as Explosion Pills did.
However, it is not a good style to pass arguments directly like that, because you might end up with some characters that need escaping (space is one of those, but it seems to be handled automatically by curl).
To do this, you can use the --data-urlencode:
curl "http://examplewebsite.com/example.php" --data-urlencode "pass=hello hello:world" --get
(--data-urlencode switches the method to POST instead of GET, so you can use --get to switch back to GET)
Just wrap the URL in a string:
curl "http://examplewebsite.com/example.php?pass=hello hello:world"
Your mileage may vary as to whether this works properly or not, so you should also URL encode the value:
curl "http://examplewebsite.com/example.php?pass=hello%22hello%3Aworld"
Related
As of 4 days ago, you were able to send a GET request to or visit https://video.google.com/timedtext?lang=en&v={youtubeVideoId} and receive an xml response containing the caption track of a given youtube video. Does anyone know if this support has been removed, because as of tonight, it no longer provides the xml response with the captions, the page is simply empty for every video. There were numerous videos this worked for 4 days ago that no longer work. Thanks in advance
Captions in default language (single available or English it seems):
To get captions of a YouTube video just use this Linux command (using curl and base64):
curl -s 'https://www.youtube.com/youtubei/v1/get_transcript?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8' -H 'Content-Type: application/json' --data-raw "{\"context\":{\"client\":{\"clientName\":\"WEB\",\"clientVersion\":\"2.2021111\"}},\"params\":\"$(printf '\n\x0bVIDEO_ID' | base64)\"}"
Change the VIDEO_ID parameter with the one interesting you.
Note: the key isn't a YouTube Data API v3 one, it is the first public (tested on some computers in different countries) one coming if you curl https://www.youtube.com/ | grep AIzaSy
Note: If interested in how I reverse-engineered this YouTube feature, say it in the comments and I would write a paragraph to explain
Captions in desired language if available:
YouTube made things tricky maybe to lose you at this step, so follow me: the only thing we have to change is the params value which is base64 encoded data which is in addition to weird characters also containing base64 data which also contains weird characters.
Get the language initials like "ru" for russian
Encode \n\x00\x12\x02LANGUAGE_INITIALS\x1a\x00 in base64 with for instance A=$(printf '\n\x00\x12\x02LANGUAGE_INITIALS\x1a\x00' | base64) (don't forget to change LANGUAGE_INITIALS to your language initials wanted ru for instance). The result for ru is CgASAnJ1GgA=
Encode the result as a URL by replacing the = to %3D with for instance B=$(printf %s $A | jq -sRr #uri). The result for ru is CgASAnJ1GgA%3D
Only if using shell commands: replace the single % to two % with for instance C=$(echo $B | sed 's/%/%%/'). The result for ru is CgASAnJ1GgA%%3D
Encode \n\x0bVIDEO_ID\x12\x0e$C (don't forget to change VIDEO_ID to your video id, with $C the result of the previous step) with for instance D=$(printf "\n\x0bVIDEO_ID\x12\x0e$C" | base64). The result for ru and lo0X2ZdElQ4 is CgtsbzBYMlpkRWxRNBIOQ2dBU0FuSjFHZ0ElM0Q=
Use this params value from the Captions in default language section: curl -s 'https://www.youtube.com/youtubei/v1/get_transcript?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8' -H 'Content-Type: application/json' --data-raw "{\"context\":{\"client\":{\"clientName\":\"WEB\",\"clientVersion\":\"2.2021111\"}},\"params\":\"$D\"}"
I recommend that anyone who uses python to try the module youtube_transcript_api. I used to send GET request to https://video.google.com/timedtext?lang=en&v={videoId}, but now the page is blank. The following is the code example. In addition, this method does not need api key.
from youtube_transcript_api import YouTubeTranscriptApi
srt = YouTubeTranscriptApi.get_transcript("videoId",languages=['en'])
Old API currently returns 404 on every request. And YouTube right now uses new version of this API:
https://www.youtube.com/api/timedtext?v={youtubeVideoId}&asr_langs=de%2Cen%2Ces%2Cfr%2Cid%2Cit%2Cja%2Cko%2Cnl%2Cpt%2Cru%2Ctr%2Cvi&caps=asr&exp=xftt%2Cxctw&xoaf=5&hl=en&ip=0.0.0.0&ipbits=0&expire=1637102374&sparams=ip%2Cipbits%2Cexpire%2Cv%2Casr_langs%2Ccaps%2Cexp%2Cxoaf&signature=0BEBD68A2638D8A18A5BC78E1851D28300247F93.7D5E6D26397D8E8A93F65CCA97260D090C870462&key=yt8&kind=asr&lang=en&fmt=json3
The main problem with this API is to calculate the signature field of request. Unfortunately I couldn't find its algorithm. Maybe someone can reverse engineered it form YouTube player.
The YouTube API change around captions caused me a lot of hassle, which I circumvented through use of youtube-dl, which has won GitHub legal support and is now again available for download/clone.
The software is available as source or binary download for all major platforms, details on their GitHub page, linked above.
Sample use is this simple:
youtube-dl --write-sub --sub-lang en --skip-download --sub-format vtt https://www.youtube.com/watch?v=E-lZ8lCG7WY
I'm trying to make a bash script to send a GET requests
also I'm trying to provide a URL access after changing some parameter value
problem:
when I hover over the link it will only underline URL without parameters values(check the following screen)
when I entered the URL it will redirect me to:
http://testphp.vulnweb.com/artists.php?artist=
I'm wondering if there is a solution for this case or not
Thank you in advance
Note that it's the terminal and not your script that decides what's considered part of a URL for the purpose of selecting or clicking.
But what I can suggest is to Use the URL encoding for those special characters instead of explicit special characters:
%27 = Single Quote **'**
%3C = Less than **<**
%3E = More than **>**
For example
URL=http://testphp.vulnweb.com/artists.php?artist=
PARTIAL_URL=$URL%3Cscript%3E
That will be shown http://testphp.vulnweb.com/artists.php?artist=<script>
Maybe that would do the trick
I want to send a curl post request with query params. One of these params is an array of strings. An example for such a string (using Postman chrome extension):
http://www.example.com/my_rest_endpoint?start=123456&duration=11&array_param=["activity level"]
Now, how can I translate this request to a curl command? no matter what I do, I can't seem to send the last parameter, array_param to curl. I've tried escaping the quotes and/or brackets with \, tried using -i flag and other various options, all with no success.
Solution was very close to what benaryorg suggested: Brackets and spaces needs to be placed with %5B, %5D and + according to the url encoding. The quotes were escaped using \". The final result is:
http://www.example.com/my_rest_endpoint?start=123456&duration=11&array_param=%5B\"Activity+Level\"%5D".
Have you tried placing ' around the parameters instead of escaping?
Also if that does not work, you could try to escape them like this example from Wikipedia.
You can find the codes for encoding here.
I've been trying to get an response over http with curl. The response is in json format and contains numbers
when I get the reply there are fields with numeric values but the floating point has been changed as follows:
"value": 2.7123123E7 instead of just "value": 27123123
why is this happening and how I can disable it? I do not want to parse the file second time and do the change, but just disable this behavior. For example my web browser where I submit the same query does not has this behavior but I cannot use my browser because the data I want to gather (response) is very big and it stucks :S
Thank you
It looks like jq will do this for you if you want a simple filter to convert the notation:
$ echo '{"value":2.7123123E7}' | jq '.'
{
"value": 27123123
}
See the manual for more info. So, a simple parsing would just be to pipe the output of curl through jq.
I am trying to dump url on the terminal that needs to be clickable, and the url comes with a query parameter. For example --
google='https://www.google.com/search?q='
orgname='foo bar'
gsearch=$google\'$orgname\'
echo "details: $orgname ($gsearch)"
But the problem is that the clickable link totally omits everything after the q=, i.e. does not include the string 'foo bar', please see the image below --
How do I make a clickable link that includes the query (i.e. the whole url in the braces above)?
Please also note that I am adding quote in the search parameter since the it may contain spaces.
Single quotes are not valid in URLs. Use the URL encoding %27 instead:
google='https://www.google.com/search?q='
orgname='foo'
gsearch=$google%27$orgname%27
echo "details: $orgname ($gsearch)"
Note that it's the terminal and not your script that decides what's considered part of a URL for the purpose of selecting or clicking. The above results in
https://www.google.com/search?q=%27foo%27
which is more clickable in most terminals. The script can't specify what's the extent of the URL except through expressing it in such a standard way that each individual terminal emulator has a decent chance of recognizing it.
PS: I don't think Google cares about surrounding single quotes.