val=$(curl https://www.geeksforgeeks.org | tr '\n' ' ' | tr -s [[:space::]] | grep -E client_id -e publication_id)
I have a requirement to fetch the particular value from curl command:
Sample I/P:
client_id:"15647",publication_id:["151","152","153"]
Output:
publication_id:["151,"152","153"]
I want to use grep command to fetch publication_id values from curl command in unix and store in variable
One solution with nodejs, xidel, xpath and puppeteer, as said in comments, you need a js aware tool:
Javascript wget.js code (this is only to mimic wget, it's output only HTML, and the one generated by js too):
const puppeteer = require('puppeteer');
(async () => {
var url = process.argv[2];
const browser = await puppeteer.launch({headless: true});
const page = await browser.newPage();
await page.goto(url, { waitUntil: 'networkidle2' });
const html = await page.evaluate(() => document.documentElement.outerHTML);
console.log(html);
browser.close();
})()
code:
node wget.js 'https://www.geeksforgeeks.org' |
xidel -e '//div[#id="g_id_onload"]/#data-client_id'
Output:
388036620207-3uolk1hv6ta7p3r9l6s3bobifh086qe1.apps.googleusercontent.com
Related
I have Puppeteer installed in a Docker container and I want to access it from another container via SSH. The command I'm using looks like this:
$ sshpass -proot ssh root#puppeteer node -e "$(cat index.js)"
bash: -c: line 0: unexpected EOF while looking for matching `''
bash: -c: line 1: syntax error: unexpected end of file
index.js looks like this:
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({
args: [
'--no-sandbox',
'--disable-setuid-sandbox'
]
});
const page = await browser.newPage();
await page.goto('https://example.com');
await page.screenshot({path: 'example.png'});
await browser.close();
})();
I can't figure out why it isn't working. Just for comparison (I can't use this approach in the final code) I tried it through docker-compose and it worked:
docker-compose exec puppeteer node -e "$(cat ../index.js)"
I know that I could first copy index.js to the puppeteer container using scp and then access it directly, but in general it seems that multiline arguments should be possible, how to make them work here?
I have a working curl command that I'd like to split out to make it easier to read.
curl d "valuea=1234&valueb=4567&valuec=87979&submit=Set" -XPOST "http://$ipa/school.cgi" > /dev/null
I've tried several ways to do this, but none seems to work.
curl -d "valuea=1234"\
-d "valueb=4567"\
-d "valuec=87979"\
-d "submit=Set"\
-XPOST "http://$ipa/school.cgi"
curl -d "valuea=1234\
valueb=4567\
valuec=87979\
submit=Set"\
-XPOST "http://$ipa/school.cgi"
Can someone advise how to do it ?
Thanks
The first approach is right, I experienced problems with spreading commands on multiple lines on some environments which were trimming whitespaces, hence it's a good idea to add a space before the backslashes:
curl -d "valuea=1234" \
-d "valueb=4567" \
-d "valuec=87979" \
-d "submit=Set" \
-XPOST "http://$ipa/school.cgi"
Eventually try it against a simple service which will inform you on what's receiving, like this one:
// echorequest.js
const http = require('http');
const hostname = '0.0.0.0';
const port = 3001;
const server = http.createServer((req, res) => {
console.log(`\n${req.method} ${req.url}`);
console.log(req.headers);
res.statusCode = 200;
res.setHeader('Content-Type', 'text/plain');
let data = '';
req.on('data', function(chunk) {
data += chunk
});
req.on('end', function() {
console.log('BODY: ' + data);
res.end(data + "\n");
});
});
server.listen(port, hostname, () => {
console.log(`Server running at http://localhost:${port}/`);
});
... run by node echorequest.js (& change target of the command: -XPOST "localhost:3001")
Also second approach could works by adding missing '&'.
For example
POST:
curl http://url -d \
"valuea=1234&\
valueb=4567&\
valuec=87979&\
submit=Set"
GET:
curl "http://url?valuea=1234&\
valueb=4567&\
valuec=87979&\
submit=Set"
If you need a service on the fly where to test your requests, try https://requestbin.fullcontact.com/
I am writing a network monitoring script in bash. The base command I am using is ettercap -T -M ARP -i en1 // //. Then I pipe egrep --color 'Host:|GET' into it.
A sample output I am getting looks like this:
GET /images/srpr/logo11w.png HTTP/1.1.
Host: www.google.com.
GET /en-us/us/products HTTP/1.1.
Host: www.caselogic.com.
My desired output is this:
Title: logo11w.png
URL: www.google.com/images/srpr/logo11w.png HTTP/1.1.
Title: Products - Case Logic
URL: www.caselogic.com/en-us/us/products
Things to notice: HTTP/1.1. and the . at the end of the host are gone. They also are formed into one URL and there is a blank line after each Title/URL listing. I attempted forming them into one URL by parsing the commands output into a variable with
var=`sudo ettercap -T -M ARP -i en1 // // | egrep --color 'Host:|GET'` | echo $var
but obviously that doesn't work because the input to the variable is a command the isn't done until the user requests a stop (CTRL + C).
To get the title of an HTML page, I use the command wget -qO- 'https://url.goes/here' | perl -l -0777 -ne 'print $1 if /<title.*?>\s*(.*?)\s*<\/title/si'. If it is something that doesn't have a title, such as an image, no title is fine.
Any help is greatly appreciated, and sorry if what I wrote is hard to read, feel free to ask questions.
Try this:
title_host.pl
#!/usr/bin/env perl
use warnings;
use strict;
use WWW::Mechanize;
my $mech = WWW::Mechanize->new();
my ($get,$host,$title);
while (<>) {
if (m|^GET (\S+) |) {
$get = $1;
} elsif ( m|^Host: (\S+)\.| ) {
$host = $1;
} else {
# Unrecognized line...reset
$get = $host = $title = '';
}
if ($get and $host) {
my ($title) = $get =~ m|^.*\/(.+?)$|; # default title
my $url = 'http://' . $host . $get;
$mech->get($url);
if ($mech->success) {
# HTML may have title, images will not
$title = $mech->title() || $title;
}
print "Title: $title\n";
print "URL: $url\n";
print "\n";
$get = $host = $title = '';
}
}
input
GET /images/srpr/logo11w.png HTTP/1.1.
Host: www.google.com.
GET /en-us/us/products HTTP/1.1.
Host: www.caselogic.com.
now just pipe your input into the perl script:
cat input | perl title_host.pl
output:
Title: logo11w.png
URL: http://www.google.com/images/srpr/logo11w.png
Title: Products - Case Logic
URL: https://www.caselogic.com/en-us/us/products
My question is similar to this one: How to detect if my shell script is running through a pipe?. The difference is that the shell script I’m working on is written in Node.js.
Let’s say I enter:
echo "foo bar" | ./test.js
Then how can I get the value "foo bar" in test.js?
I’ve read Unix and Node: Pipes and Streams but that only seems to offer an asynchronous solution (unless I’m mistaken). I’m looking for a synchronous solution. Also, with this technique, it doesn’t seem very straightforward to detect if the script is being piped or not.
TL;DR My question is two-fold:
How to detect if a Node.js script is running through a shell pipe, e.g. echo "foo bar" | ./test.js?
If so, how to read out the piped value in Node.js?
I just found out a simpler answer to part of my question.
To quickly and synchronously detect if piped content is being passed to the current script in Node.js, use the process.stdin.isTTY boolean:
$ node -p -e 'process.stdin.isTTY'
true
$ echo 'foo' | node -p -e 'process.stdin.isTTY'
undefined
So, in a script, you could do something like this:
if (process.stdin.isTTY) {
// handle shell arguments
} else {
// handle piped content (see Jerome’s answer)
}
The reason I didn’t find this before is because I was looking at the documentation for process, where isTTY is not mentioned at all. Instead, it’s mentioned in the TTY documentation.
Pipes are made to handle small inputs like "foo bar" but also huge files.
The stream API makes sure that you can start handling data without waiting for the huge file to be totally piped through (this is better for speed & memory). The way it does this is by giving you chunks of data.
There is no synchronous API for pipes. If you really want to have the whole piped input in your hands before doing something, you can use
note: use only node >= 0.10.0 because the example uses the stream2 API
var data = '';
function withPipe(data) {
console.log('content was piped');
console.log(data.trim());
}
function withoutPipe() {
console.log('no content was piped');
}
var self = process.stdin;
self.on('readable', function() {
var chunk = this.read();
if (chunk === null) {
withoutPipe();
} else {
data += chunk;
}
});
self.on('end', function() {
withPipe(data);
});
test with
echo "foo bar" | node test.js
and
node test.js
It turns out that process.stdin.isTTY is not reliable because you can spawn a child process that is not a TTY.
I found a better solution here using file descriptors.
You can test to see if your program with piped in or out with these functions:
function pipedIn(cb) {
fs.fstat(0, function(err, stats) {
if (err) {
cb(err)
} else {
cb(null, stats.isFIFO())
}
})
}
function pipedOut(cb) {
fs.fstat(1, function(err, stats) {
if (err) {
cb(err)
} else {
cb(null, stats.isFIFO())
}
})
}
pipedIn((err, x) => console.log("in", x))
pipedOut((err, x) => console.log("out", x))
Here's some tests demonstrating that it works.
❯❯❯ node pipes.js
in false
out false
❯❯❯ node pipes.js | cat -
in false
out true
❯❯❯ echo 'hello' | node pipes.js | cat -
in true
out true
❯❯❯ echo 'hello' | node pipes.js
in true
out false
❯❯❯ node -p -e "let x = require('child_process').exec(\"node pipes.js\", (err, res) => console.log(res))"
undefined
in false
out false
❯❯❯ node -p -e "let x = require('child_process').exec(\"echo 'hello' | node pipes.js\", (err, res) => console.log(res))"
undefined
in true
out false
❯❯❯ node -p -e "let x = require('child_process').exec(\"echo 'hello' | node pipes.js | cat -\", (err, res) => console.log(res))"
undefined
in true
out true
❯❯❯ node -p -e "let x = require('child_process').exec(\"node pipes.js | cat -\", (err, res) => console.log(res))"
undefined
in false
out true
If you need to pipe into nodejs using an inline --eval string in bash, cat works too:
$ echo "Hello" | node -e "console.log(process.argv[1]+' pipe');" "$(cat)"
# "Hello pipe"
You need to check stdout (not stdin like suggested elsewhere) like this:
if (process.stdout.isTTY) {
// not piped
} else {
// piped
}
How can I paste to codepad.org from the commandline using curl?
here's a Python script
import urllib
import urllib2
url = 'http://codepad.org'
content=open("myscript.py").read()
values = {'lang' : 'Python',
'code' : content,
'submit':'Submit'}
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page = response.read()
for href in the_page.split("</a>"):
if "Link:" in href:
ind=href.index('Link:')
found = href[ind+5:]
for i in found.split('">'):
if '<a href=' in i:
print "The link: " ,i.replace('<a href="',"").strip()
output
$ python python.py
The link: http://codepad.org/W0G8HIzM
Yes, you can do it with curl. Assuming your code is Python and in myfile.python, you can do it like this:
$ curl -d "lang=Python&submit=Submit" --data-urlencode code#myfile.py codepad.org
(Edited to make it work.)
You can also use reval:
reval test.py
reval -l ruby -e 'p 2+2'
reval prog.hs -p # make private