I've been trying to get an response over http with curl. The response is in json format and contains numbers
when I get the reply there are fields with numeric values but the floating point has been changed as follows:
"value": 2.7123123E7 instead of just "value": 27123123
why is this happening and how I can disable it? I do not want to parse the file second time and do the change, but just disable this behavior. For example my web browser where I submit the same query does not has this behavior but I cannot use my browser because the data I want to gather (response) is very big and it stucks :S
Thank you
It looks like jq will do this for you if you want a simple filter to convert the notation:
$ echo '{"value":2.7123123E7}' | jq '.'
{
"value": 27123123
}
See the manual for more info. So, a simple parsing would just be to pipe the output of curl through jq.
Related
As of 4 days ago, you were able to send a GET request to or visit https://video.google.com/timedtext?lang=en&v={youtubeVideoId} and receive an xml response containing the caption track of a given youtube video. Does anyone know if this support has been removed, because as of tonight, it no longer provides the xml response with the captions, the page is simply empty for every video. There were numerous videos this worked for 4 days ago that no longer work. Thanks in advance
Captions in default language (single available or English it seems):
To get captions of a YouTube video just use this Linux command (using curl and base64):
curl -s 'https://www.youtube.com/youtubei/v1/get_transcript?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8' -H 'Content-Type: application/json' --data-raw "{\"context\":{\"client\":{\"clientName\":\"WEB\",\"clientVersion\":\"2.2021111\"}},\"params\":\"$(printf '\n\x0bVIDEO_ID' | base64)\"}"
Change the VIDEO_ID parameter with the one interesting you.
Note: the key isn't a YouTube Data API v3 one, it is the first public (tested on some computers in different countries) one coming if you curl https://www.youtube.com/ | grep AIzaSy
Note: If interested in how I reverse-engineered this YouTube feature, say it in the comments and I would write a paragraph to explain
Captions in desired language if available:
YouTube made things tricky maybe to lose you at this step, so follow me: the only thing we have to change is the params value which is base64 encoded data which is in addition to weird characters also containing base64 data which also contains weird characters.
Get the language initials like "ru" for russian
Encode \n\x00\x12\x02LANGUAGE_INITIALS\x1a\x00 in base64 with for instance A=$(printf '\n\x00\x12\x02LANGUAGE_INITIALS\x1a\x00' | base64) (don't forget to change LANGUAGE_INITIALS to your language initials wanted ru for instance). The result for ru is CgASAnJ1GgA=
Encode the result as a URL by replacing the = to %3D with for instance B=$(printf %s $A | jq -sRr #uri). The result for ru is CgASAnJ1GgA%3D
Only if using shell commands: replace the single % to two % with for instance C=$(echo $B | sed 's/%/%%/'). The result for ru is CgASAnJ1GgA%%3D
Encode \n\x0bVIDEO_ID\x12\x0e$C (don't forget to change VIDEO_ID to your video id, with $C the result of the previous step) with for instance D=$(printf "\n\x0bVIDEO_ID\x12\x0e$C" | base64). The result for ru and lo0X2ZdElQ4 is CgtsbzBYMlpkRWxRNBIOQ2dBU0FuSjFHZ0ElM0Q=
Use this params value from the Captions in default language section: curl -s 'https://www.youtube.com/youtubei/v1/get_transcript?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8' -H 'Content-Type: application/json' --data-raw "{\"context\":{\"client\":{\"clientName\":\"WEB\",\"clientVersion\":\"2.2021111\"}},\"params\":\"$D\"}"
I recommend that anyone who uses python to try the module youtube_transcript_api. I used to send GET request to https://video.google.com/timedtext?lang=en&v={videoId}, but now the page is blank. The following is the code example. In addition, this method does not need api key.
from youtube_transcript_api import YouTubeTranscriptApi
srt = YouTubeTranscriptApi.get_transcript("videoId",languages=['en'])
Old API currently returns 404 on every request. And YouTube right now uses new version of this API:
https://www.youtube.com/api/timedtext?v={youtubeVideoId}&asr_langs=de%2Cen%2Ces%2Cfr%2Cid%2Cit%2Cja%2Cko%2Cnl%2Cpt%2Cru%2Ctr%2Cvi&caps=asr&exp=xftt%2Cxctw&xoaf=5&hl=en&ip=0.0.0.0&ipbits=0&expire=1637102374&sparams=ip%2Cipbits%2Cexpire%2Cv%2Casr_langs%2Ccaps%2Cexp%2Cxoaf&signature=0BEBD68A2638D8A18A5BC78E1851D28300247F93.7D5E6D26397D8E8A93F65CCA97260D090C870462&key=yt8&kind=asr&lang=en&fmt=json3
The main problem with this API is to calculate the signature field of request. Unfortunately I couldn't find its algorithm. Maybe someone can reverse engineered it form YouTube player.
The YouTube API change around captions caused me a lot of hassle, which I circumvented through use of youtube-dl, which has won GitHub legal support and is now again available for download/clone.
The software is available as source or binary download for all major platforms, details on their GitHub page, linked above.
Sample use is this simple:
youtube-dl --write-sub --sub-lang en --skip-download --sub-format vtt https://www.youtube.com/watch?v=E-lZ8lCG7WY
I am working on a performance script in Jmeter that contains a number of http requests. One of the parameters I pass in my request will always be formatted as follows:
{"a":"transition9","ap":"203867"}
Everything about the above remains constant with the exception of "ap". I need to pull "ap" from regular expression extractor, which I can do.
So at the end of the day the above will actually look something like this.
{"a":"transition9","ap":"${regexExtractedValue}"}
Here is the really tricky part. If I can achieve the above, I then need to base64 encode the value, which I know can be done using ${__base64Encode(test string)}. See https://jmeter-plugins.org/wiki/Functions/#base64Encodesupfont-color-gray-size-1-since-1-2-0-font-sup.
I have tried a number of approaches, which mainly have involved splitting up the hardcoded values and trying to combine them with the dynamic values, but the comma seems to throw it off. An example of something I have tried.
prefix = eyJhIjoidHJhbnNpdGlvbjkiLCJhcCI6Ij
ap = ${__base64Encode(203867"})
Then you would combine the 2 and the value being passed into the param would look something like this
{"stuff":"thing","__Action":"${prefix}${app}","__Scroll":"base64:MA=="}
This yields strange results. Is there a way to get what I need here?
In the parameter value I used this format:
${prefix}${__base64Encode("${post}"})}
Using JMeter to support functional API testing and have run across a problem with reading data from a CSV file. The data from the file is used in building a POST data body which contains something like this:
"wibbles" : ${wibble-var},
${wibble-var} is read from a CSV file and has the format :
["wibble1","wibble2","wibble3"]
... there are over 1000 wibble values in the list.
If "wibbles" : ["wibble1","wibble2","wibble3"]... is hard-coded into the POST body, then JMeter is happy, builds the POST request and does the business, but it's proved impossible to create a CSV file with even the 3 value example above, that JMeter will parse. JMeter skips the thread containing the 'CSV read' without building the POST request or sending it, so there's no response to examine, and a Debug Sampler is similarly skipped. I've heard rumours that doubling up the quotes can work but haven't been able to find the right syntax. Can anyone throw any light on this issue? Thanks
Double quotes will work if you can get "wibble1,wibble2,wibble3" & if you set Allow quoted data to true in CSV data set config
You can get this value and then use beanshell preprocessor to convert to the format "wibble1","wibble2","wibble3".
If you want to get in this format "wibble1","wibble2","wibble3" directly, you can use \t as the delimiter & modify the data in the CSV file accordingly.
Trial and error led to the following solution.
The format of the single data variable I needed to parse is ["value1","value2","value3"] (i.e a JSON array.) and this is exactly what the CSV file contained (with a header name of course on the first row), including the [ and ] brackets.
I modified the parameterised POST body to:
"wibbles": [${wibble-var}],
-- that is, I moved the square brackets out of the CSV file so that the CSV file now just contained the quoted elements of the array:
"value1","value2","value3" etc
I then set the delimiter in the CSV Data Set Config to |
And Allow Quoted Data to FALSE. <--- This was a bit counter intuitive but without it JMeter would not read the whole comma separated list of 2000 quoted strings as a single variable.
With these changes in place the script executed correctly.
Thanks again for the responses, I will definitely look at the __String functions mentioned.
I would go for the following options:
If your "wibbles" are a single string which you need to pass a a JSON Array it might be a lot easier to access them via __StringFromFile() or __FileToString() functions like:
"wibbles" : ${_StringFromFile(/path/to/file/containing/wibbles,,,)},
If you need to access individual "wibbles" and your CSV file is basically a JSON file:
Add HTTP Request Sampler to your test plan (before one which sends these "wibbles") and configure it as follows:
Protocol: file
Path: c:/testdata/yourfile.csv
Add JSON Path PostProcessor and use a JSON Path query to store the "wibbles" into a JMeter Variable(s)
I'm trying to fetch the yammer Followers using below rest API.
https://www.yammer.com/api/v1/users.json
Api contains details for each user. From this I need to extract followers count alone.
{"type":"user","id":1517006975,"network_id":461,"stats":{"following":0,"followers":0,"updates":0}}
Rate limit for per page is 50, as we have 100 000+ users I need to iterate 2000+ times to get the whole dump which is actually slow.
So I need method to directly extract the necessary data.
I am using shell script + pentaho .
I think you have two options.
If you are bound to shell, you could run the json response through a series of sed silliness to get to a list that you can then parse more effectively with shell tools. something like: curl http://foo.com | sed 's/,/\n/g'
will get you something more row based, and then you can start to parse it out from there using more sed or awk or cut and tr.
look at jq? it is a statically linked standalone c binary that allows really nice filtering of json
I am writing a program in bash and want to use curl to display the contents of a website. The website url is http://examplewebsite.com/example.php?pass=hello hello:world. Using:
curl http://examplewebsite.com/example.php?pass=hello hello:world
However this returns:
Couldn't resolve host 'hello:world'
The error was caused by the space between “hello” and “world” that caused bash to split the link into two different tokens, which curl interpreted as two different links.
You should at least quote the URL parameters, as Explosion Pills did.
However, it is not a good style to pass arguments directly like that, because you might end up with some characters that need escaping (space is one of those, but it seems to be handled automatically by curl).
To do this, you can use the --data-urlencode:
curl "http://examplewebsite.com/example.php" --data-urlencode "pass=hello hello:world" --get
(--data-urlencode switches the method to POST instead of GET, so you can use --get to switch back to GET)
Just wrap the URL in a string:
curl "http://examplewebsite.com/example.php?pass=hello hello:world"
Your mileage may vary as to whether this works properly or not, so you should also URL encode the value:
curl "http://examplewebsite.com/example.php?pass=hello%22hello%3Aworld"