Related
I'm reading the goaccess man page but I'm missing simple examples. I have a customised nginx with the following config:
log_format timed_combined '$remote_addr - $remote_user [$time_local] '
'$ssl_protocol/$ssl_cipher '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" '
'$request_time $upstream_response_time $pipe';
Here is an example log entry:
66.249.76.120 - - [20/Dec/2016:19:04:03 +0100]
TLSv1.2/ECDHE-RSA-AES128-GCM-SHA256 "GET / HTTP/1.1" 200 27232
"-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)" 0.026 0.026 .
How do I have to configure the .goaccessrc to read that format?
You can add this to your config file or ~/.goaccessrc
log-format %h %^[%d:%t %^] %^"%r" %s %b "%R" "%u" %T %^
date-format %d/%b/%Y
time-format %H:%M:%S
I am currently using goaccess-1.0.2. I have installed it on an Amazon Linux box. The box which it resides has customized logs that were forwarded from an Apache WebApp Server. What I have tried to accomplish but can't seem to figure out is how to get GoAccess to parse our customized log.
Here is an example of the custom forwarded WebApp Log entry:
Jun 24 00:00:41 directory1 httpd-access: 55.117.170.95 www.URLaddress.com - [24/Jun/2016:00:00:41 -0700] "GET /sites/all/themes/somthing_on_demand/js/fancybox/jquery.fancybox-1.3.4.css HTTP/1.1" 304 - "ht
tps://www.IPaddress.com/my_account/yum" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" "SESSb9948a0b21e4d377a7d82f6adbf86c91=l
on7pgjlikml7q4tq954ejiao1; cookie_js=1; __utma=23285183.1119616966.1452095139.1468883973.1468963151.39; __utmb=23285183.500.10.1468963151; __utmc=23285183; __utmz=23285183.1468963151.39.39.utmcsr=fyi.URLaddress.com|utm
ccn=(r/INFOSEC-MAXLEN-256" "-" 57630
Here are a few log-formats I have tried:
log-format %^ %^ %^ "%h %^ %u %t \"%r\" %>s %b \"%R\" \"%u\" \"%^\" \"%^\" %D"
log-format "%h %{Host}i %{SSL_CLIENT_S_DN_CN}x %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%{SHORT_COOKIE}e\" \"%{X-Forwarded-For}i\" %D"
log-format "%h %{Host}i %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%{SHORT_COOKIE}e\" \"%{X-Forwarded-For}i\" %D"
I thought I would ignore the date and time format using %^ then use date format %m %d and time format %T .
I am very new at this and could really use help. Thank you for your feedback in advance.
Please try this, it works for me:
goaccess -f access.log --log-format='%^:%^:%^: %h %v %^[%d:%t %^] "%r" %s %b "%R" "%u" "%^" "%^" %D' --date-format='%d/%b/%Y' --time-format='%T'
I use curl to get http headers to find http status code and also return response. I get the http headers with the command
curl -I http://localhost
To get the response, I use the command
curl http://localhost
As soon as use the -I flag, I get only the headers and the response is no longer there. Is there a way to get both the http response and the headers/http status code in in one command?
I was able to get a solution by looking at the curl doc which specifies to use - for the output to get the output to stdout.
curl -o - -I http://localhost
To get the response with just the http return code, I could just do
curl -o /dev/null -s -w "%{http_code}\n" http://localhost
the verbose mode will tell you everything
curl -v http://localhost
I found this question because I wanted independent access to BOTH the response and the content in order to add some error handling for the user.
Curl allows you to customize output. You can print the HTTP status code to std out and write the contents to another file.
curl -s -o response.txt -w "%{http_code}" http://example.com
This allows you to check the return code and then decide if the response is worth printing, processing, logging, etc.
http_response=$(curl -s -o response.txt -w "%{http_code}" http://example.com)
if [ $http_response != "200" ]; then
# handle error
else
echo "Server returned:"
cat response.txt
fi
The %{http_code} is a variable substituted by curl. You can do a lot more, or send code to stderr, etc. See curl manual and the --write-out option.
-w, --write-out
Make curl display information on stdout after a completed
transfer. The format is a string that may contain plain
text mixed with any number of variables. The format can be
specified as a literal "string", or you can have curl read
the format from a file with "#filename" and to tell curl
to read the format from stdin you write "#-".
The variables present in the output format will be
substituted by the value or text that curl thinks fit, as
described below. All variables are specified as
%{variable_name} and to output a normal % you just write
them as %%. You can output a newline by using \n, a
carriage return with \r and a tab space with \t.
The output will be written to standard output, but this
can be switched to standard error by using %{stderr}.
https://man7.org/linux/man-pages/man1/curl.1.html
I use this command to print the status code without any other output. Additionally, it will only perform a HEAD request and follow the redirection (respectively -I and -L).
curl -o -I -L -s -w "%{http_code}" http://localhost
This makes it very easy to check the status code in a health script:
sh -c '[ $(curl -o -I -L -s -w "%{http_code}" http://localhost) -eq 200 ]'
The -i option is the one that you want:
curl -i http://localhost
-i, --include Include protocol headers in the output (H/F)
Alternatively you can use the verbose option:
curl -v http://localhost
-v, --verbose Make the operation more talkative
I have used this :
request_cmd="$(curl -i -o - --silent -X GET --header 'Accept: application/json' --header 'Authorization: _your_auth_code==' 'https://example.com')"
To get the HTTP status
http_status=$(echo "$request_cmd" | grep HTTP | awk '{print $2}')
echo $http_status
To get the response body I've used this
output_response=$(echo "$request_cmd" | grep body)
echo $output_response
This command
curl http://localhost -w ", %{http_code}"
will get the comma separated body and status; you can split them to get them out.
You can change the delimiter as you like.
This is a way to retrieve the body "AND" the status code and format it to a proper json or whatever format works for you. Some may argue it's the incorrect use of write format option but this works for me when I need both body and status code in my scripts to check status code and relay back the responses from server.
curl -X GET -w "%{stderr}{\"status\": \"%{http_code}\", \"body\":\"%{stdout}\"}" -s -o - “https://github.com” 2>&1
run the code above and you should get back a json in this format:
{
"status" : <status code>,
"body" : <body of response>
}
with the -w write format option, since stderr is printed first, you can format your output with the var http_code and place the body of the response in a value (body) and follow up the enclosing using var stdout. Then redirect your stderr output to stdout and you'll be able to combine both http_code and response body into a neat output
To get response code along with response:
$ curl -kv https://www.example.org
To get just response code:
$ curl -kv https://www.example.org 2>&1 | grep -i 'HTTP/1.1 ' | awk '{print $3}'| sed -e 's/^[ \t]*//'
2>&1: error is stored in output for parsing
grep: filter the response code line from output
awk: filters out the response code from response code line
sed: removes any leading white spaces
For programmatic usage, I use the following :
curlwithcode() {
code=0
# Run curl in a separate command, capturing output of -w "%{http_code}" into statuscode
# and sending the content to a file with -o >(cat >/tmp/curl_body)
statuscode=$(curl -w "%{http_code}" \
-o >(cat >/tmp/curl_body) \
"$#"
) || code="$?"
body="$(cat /tmp/curl_body)"
echo "statuscode : $statuscode"
echo "exitcode : $code"
echo "body : $body"
}
curlwithcode https://api.github.com/users/tj
It shows following output :
statuscode : 200
exitcode : 0
body : {
"login": "tj",
"id": 25254,
...
}
My way to achieve this:
To get both (header and body), I usually perform a curl -D- <url> as in:
$ curl -D- http://localhost:1234/foo
HTTP/1.1 200 OK
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: application/json
Date: Wed, 29 Jul 2020 20:59:21 GMT
{"data":["out.csv"]}
This will dump headers (-D) to stdout (-) (Look for --dump-header in man curl).
IMHO also very handy in this context:
I often use jq to get that json data (eg from some rest APIs) formatted. But as jq doesn't expect a HTTP header, the trick is to print headers to stderr using -D/dev/stderr. Note that this time we also use -sS (--silent, --show-errors) to suppress the progress meter (because we write to a pipe).
$ curl -sSD/dev/stderr http://localhost:1231/foo | jq .
HTTP/1.1 200 OK
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: application/json
Date: Wed, 29 Jul 2020 21:08:22 GMT
{
"data": [
"out.csv"
]
}
I guess this also can be handy if you'd like to print headers (for quick inspection) to console but redirect body to a file (eg when its some kind of binary to not mess up your terminal):
$ curl -sSD/dev/stderr http://localhost:1231 > /dev/null
HTTP/1.1 200 OK
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: application/json
Date: Wed, 29 Jul 2020 21:20:02 GMT
Be aware: This is NOT the same as curl -I <url>! As -I will perform a HEAD request and not a GET request (Look for --head in man curl. Yes: For most HTTP servers this will yield same result. But I know a lot of business applications which don't implement HEAD request at all ;-P
A one-liner, just to get the status-code would be:
curl -s -i https://www.google.com | head -1
Changing it to head -2 will give the time as well.
If you want a while-true loop over it, it would be:
URL="https://www.google.com"
while true; do
echo "------"
curl -s -i $URL | head -2
sleep 2;
done
Which produces the following, until you do cmd+C (or ctrl+C in Windows).
------
HTTP/2 200
date: Sun, 07 Feb 2021 20:03:38 GMT
------
HTTP/2 200
date: Sun, 07 Feb 2021 20:03:41 GMT
------
HTTP/2 200
date: Sun, 07 Feb 2021 20:03:43 GMT
------
HTTP/2 200
date: Sun, 07 Feb 2021 20:03:45 GMT
------
HTTP/2 200
date: Sun, 07 Feb 2021 20:03:47 GMT
------
HTTP/2 200
date: Sun, 07 Feb 2021 20:03:49 GMT
A clear one to read using pipe
function cg(){
curl -I --silent www.google.com | head -n 1 | awk -F' ' '{print $2}'
}
cg
# 200
Welcome to use my dotfile script here
Explanation
--silent: Don't show progress bar when using pipe
head -n 1: Only show the first line
-F' ': separate text by columns using separator space
'{print $2}': show the second column
Some good answers here, but like the OP I found myself wanting, in a scripting context, all of:
any response body returned by the server, regardless of the response status-code: some services will send error details e.g. in JSON form when the response is an error
the HTTP response code
the curl exit status code
This is difficult to achieve with a single curl invocation and I was looking for a complete solution/example, since the required processing is complex.
I combined some other bash recipes on multiplexing stdout/stderr/return-code with some of the ideas here to arrive at the following example:
{
IFS= read -rd '' out
IFS= read -rd '' http_code
IFS= read -rd '' status
} < <({ out=$(curl -sSL -o /dev/stderr -w "%{http_code}" 'https://httpbin.org/json'); } 2>&1; printf '\0%s' "$out" "$?")
Then the results can be found in variables:
echo out $out
echo http_code $http_code
echo status $status
Results:
out { "slideshow": { "author": "Yours Truly", "date": "date of publication", "slides": [ { "title": "Wake up to WonderWidgets!", "type": "all" }, { "items": [ "Why <em>WonderWidgets</em> are great", "Who <em>buys</em> WonderWidgets" ], "title": "Overview", "type": "all" } ], "title": "Sample Slide Show" } }
http_code 200
status 0
The script works by multiplexing the output, HTTP response code and curl exit status separated by null characters, then reading these back into the current shell/script. It can be tested with curl requests that would return a >=400 response code but also produce output.
Note that without the -f flag, curl won't return non-zero error codes when the server returns an abnormal HTTP response code i.e. >=400, and with the -f flag, server output is suppresses on error, making use of this flag for error-detection and processing unattractive.
Credits for the generic read with IFS processing go to this answer: https://unix.stackexchange.com/a/430182/45479 .
In my experience we usually use curl this way
curl -f http://localhost:1234/foo || exit 1
curl: (22) The requested URL returned error: 400 Bad Request
This way we can pipe the curl when it fails, and it also shows the status code.
Append a line "http_code:200" at the end, and then grep for the keyword "http_code:" and extract the response code.
result=$(curl -w "\nhttp_code:%{http_code}" http://localhost)
echo "result: ${result}" #the curl result with "http_code:" at the end
http_code=$(echo "${result}" | grep 'http_code:' | sed 's/http_code://g')
echo "HTTP_CODE: ${http_code}" #the http response code
In this case, you can still use the non-silent mode / verbose mode to get more information about the request such as the curl response body.
Wow so many answers, cURL devs definitely left it to us as a home exercise :) Ok here is my take - a script that makes the cURL working as it's supposed to be, i.e.:
show the output as cURL would.
exit with non-zero code in case of HTTP response code not in 2XX range
Save it as curl-wrapper.sh:
#!/bin/bash
output=$(curl -w "\n%{http_code}" "$#")
res=$?
if [[ "$res" != "0" ]]; then
echo -e "$output"
exit $res
fi
if [[ $output =~ [^0-9]([0-9]+)$ ]]; then
httpCode=${BASH_REMATCH[1]}
body=${output:0:-${#httpCode}}
echo -e "$body"
if (($httpCode < 200 || $httpCode >= 300)); then
# Remove this is you want to have pure output even in
# case of failure:
echo
echo "Failure HTTP response code: ${httpCode}"
exit 1
fi
else
echo -e "$output"
echo
echo "Cannot get the HTTP return code"
exit 1
fi
So then it's just business as usual, but instead of curl do ./curl-wrapper.sh:
So when the result falls in 200-299 range:
./curl-wrapper.sh www.google.com
# ...the same output as pure curl would return...
echo $?
# 0
And when the result is out of in 200-299 range:
./curl-wrapper.sh www.google.com/no-such-page
# ...the same output as pure curl would return - plus the line
# below with the failed HTTP code, this line can be removed if needed:
#
# Failure HTTP response code: 404
echo $?
# 1
Just do not pass "-w|--write-out" argument since that's what added inside the script
I used the following way of getting both return code as well as response body in the console.
NOTE - use tee which append the output into a file as well as to the console, which solved my purpose.
Sample CURL call for reference:
curl -s -i -k --location --request POST ''${HOST}':${PORT}/api/14/project/'${PROJECT_NAME}'/jobs/import' \
--header 'Content-Type: application/yaml' \
--header 'X-Rundeck-Auth-Token: '${JOB_IMPORT_TOKEN}'' \
--data "$(cat $yaml_file)" &>/dev/stdout | tee -a $response_file
return_code=$(cat $response_file | head -3 | tail -1 | awk {'print $2'})
if [ "$return_code" != "200" ]; then
echo -e "\Job import api call failed with rc: $return_code, please rerun or change pipeline script."
exit $return_code
else
echo "Job import api call completed successfully with rc: $return_code"
fi
Hope this would help a few.
To capture only response:
curl --location --request GET "http://localhost:8000"
To capture the response and its statuscode:
curl --location --request GET "http://localhost:8000" -w "%{http_code}"
To capture the response in a file:
curl --location --request GET "http://localhost:8000" -s -o "response.txt"
while : ; do curl -sL -w "%{http_code} %{url_effective}\\n" http://host -o /dev/null; done
This works for me:
curl -Uri 'google.com' | select-object StatusCode
i want the output of the script need to be write it into the file i.e QR_bar_code_$fdt.sh
here my script file qr_script.sh
dt=$(date "+%d/%B/%Y")
prev=$(date --date yesterday "+%d/%B/%Y")
fdt=$(date --date yesterday "+%d-%m-%Y")
echo "$fdt"
echo "Date :$prev"
grep -F "$prev" access_log|grep -F 'GET /?p='
echo "count :$(grep -F "$prev" access_log|grep -F 'GET /?p='|wc -l)"
cat >QR_bar_code_$fdt.sh
exec >>QR_bar_code_$fdt.sh
I am not sure about last 2 lines.
my problem is when i run the script the file created but the output is not append to newly created file and script execution waits after creation of file when i write something that will be appends to the file but not the output generated by the script
what my newly created file must be like output of the qr_script.sh
that will be like this
Date :13/May/2014
183.82.99.35 - - [13/May/2014:03:03:16 -0400] "GET /?p=135873 HTTP/1.1" 301 281
183.82.99.35 - - [13/May/2014:03:03:49 -0400] "GET /?p=134201 HTTP/1.1" 301 281
183.82.99.35 - - [13/May/2014:03:04:06 -0400] "GET /?p=134201 HTTP/1.1" 301 281
183.82.99.35 - - [13/May/2014:03:04:25 -0400] "GET /?p=134201 HTTP/1.1" 301 281
count :4
if anybody have any idea about my problem.Help Me to resolve it
Thanks in Advance
Try
prev=$(date --date yesterday "+%d/%B/%Y")
fdt=$(date --date yesterday "+%d-%m-%Y")
echo "$fdt" >QR_bar_code_$fdt.sh
echo "Date :$prev" >>QR_bar_code_$fdt.sh
grep -F "$prev" access_log|grep -F 'GET /?p=' >>QR_bar_code_$fdt.sh
echo "count :$(grep -F "$prev" access_log|grep -F 'GET /?p='|wc -l)" >>QR_bar_code_$fdt.sh
I'm trying to set up a minimal web server using netcat (nc). When the browser calls up localhost:1500, for instance, it should show the result of a function (date in the example below, but eventually it'll be a python or c program that yields some data).
My little netcat web server needs to be a while true loop in bash, possibly as simple as this:
while true ; do echo -e "HTTP/1.1 200 OK\n\n $(date)" | nc -l -p 1500 ; done
When I try this the browser shows the currently available data during the moment when nc starts. I want the browser displays the data during the moment the browser requests it, though. How can I achieve this?
Try this:
while true ; do nc -l -p 1500 -c 'echo -e "HTTP/1.1 200 OK\n\n $(date)"'; done
The -cmakes netcat execute the given command in a shell, so you can use echo. If you don't need echo, use -e. For further information on this, try man nc. Note, that when using echo there is no way for your program (the date-replacement) to get the browser request. So you probably finally want to do something like this:
while true ; do nc -l -p 1500 -e /path/to/yourprogram ; done
Where yourprogram must do the protocol stuff like handling GET, sending HTTP 200 etc.
I had the problem where I wanted to return the result of executing a bash command:
$ while true; do { echo -e 'HTTP/1.1 200 OK\r\n'; sh test; } | nc -l 8080; done
NOTE:
This command was taken from: http://www.razvantudorica.com/08/web-server-in-one-line-of-bash
This executes a bash script and returns the result to a browser client connecting to the server running this command on port 8080.
My script does this:
$ nano test
#!/bin/bash
echo "************PRINT SOME TEXT***************\n"
echo "Hello World!!!"
echo "\n"
echo "Resources:"
vmstat -S M
echo "\n"
echo "Addresses:"
echo "$(ifconfig)"
echo "\n"
echo "$(gpio readall)"
and my web browser is showing
************PRINT SOME TEXT***************
Hello World!!!
Resources:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 314 18 78 0 0 2 1 306 31 0 0 100 0
Addresses:
eth0 Link encap:Ethernet HWaddr b8:27:eb:86:e8:c5
inet addr:192.168.1.83 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:27734 errors:0 dropped:0 overruns:0 frame:0
TX packets:26393 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1924720 (1.8 MiB) TX bytes:3841998 (3.6 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
GPIOs:
+----------+-Rev2-+------+--------+------+-------+
| wiringPi | GPIO | Phys | Name | Mode | Value |
+----------+------+------+--------+------+-------+
| 0 | 17 | 11 | GPIO 0 | IN | Low |
| 1 | 18 | 12 | GPIO 1 | IN | Low |
| 2 | 27 | 13 | GPIO 2 | IN | Low |
| 3 | 22 | 15 | GPIO 3 | IN | Low |
| 4 | 23 | 16 | GPIO 4 | IN | Low |
| 5 | 24 | 18 | GPIO 5 | IN | Low |
| 6 | 25 | 22 | GPIO 6 | IN | Low |
| 7 | 4 | 7 | GPIO 7 | IN | Low |
| 8 | 2 | 3 | SDA | IN | High |
| 9 | 3 | 5 | SCL | IN | High |
| 10 | 8 | 24 | CE0 | IN | Low |
| 11 | 7 | 26 | CE1 | IN | Low |
| 12 | 10 | 19 | MOSI | IN | Low |
| 13 | 9 | 21 | MISO | IN | Low |
| 14 | 11 | 23 | SCLK | IN | Low |
| 15 | 14 | 8 | TxD | ALT0 | High |
| 16 | 15 | 10 | RxD | ALT0 | High |
| 17 | 28 | 3 | GPIO 8 | ALT2 | Low |
| 18 | 29 | 4 | GPIO 9 | ALT2 | Low |
| 19 | 30 | 5 | GPIO10 | ALT2 | Low |
| 20 | 31 | 6 | GPIO11 | ALT2 | Low |
+----------+------+------+--------+------+-------+
Add -q 1 to the netcat command line:
while true; do
echo -e "HTTP/1.1 200 OK\n\n $(date)" | nc -l -p 1500 -q 1
done
The problem you are facing is that nc does not know when the web client is done with its request so it can respond to the request.
A web session should go something like this.
TCP session is established.
Browser Request Header: GET / HTTP/1.1
Browser Request Header: Host: www.google.com
Browser Request Header: \n #Note: Browser is telling Webserver that the request header is complete.
Server Response Header: HTTP/1.1 200 OK
Server Response Header: Content-Type: text/html
Server Response Header: Content-Length: 24
Server Response Header: \n #Note: Webserver is telling browser that response header is complete
Server Message Body: <html>sample html</html>
Server Message Body: \n #Note: Webserver is telling the browser that the requested resource is finished.
The server closes the TCP session.
Lines that begin with "\n" are simply empty lines without even a space and contain nothing more than a new line character.
I have my bash httpd launched by xinetd, xinetd tutorial. It also logs date, time, browser IP address, and the entire browser request to a log file, and calculates Content-Length for the Server header response.
user#machine:/usr/local/bin# cat ./bash_httpd
#!/bin/bash
x=0;
Log=$( echo -n "["$(date "+%F %T %Z")"] $REMOTE_HOST ")$(
while read I[$x] && [ ${#I[$x]} -gt 1 ];do
echo -n '"'${I[$x]} | sed -e's,.$,",'; let "x = $x + 1";
done ;
); echo $Log >> /var/log/bash_httpd
Message_Body=$(echo -en '<html>Sample html</html>')
echo -en "HTTP/1.0 200 OK\nContent-Type: text/html\nContent-Length: ${#Message_Body}\n\n$Message_Body"
To add more functionality, you could incorporate.
METHOD=$(echo ${I[0]} |cut -d" " -f1)
REQUEST=$(echo ${I[0]} |cut -d" " -f2)
HTTP_VERSION=$(echo ${I[0]} |cut -d" " -f3)
If METHOD = "GET" ]; then
case "$REQUEST" in
"/") Message_Body="HTML formatted home page stuff"
;;
/who) Message_Body="HTML formatted results of who"
;;
/ps) Message_Body="HTML formatted results of ps"
;;
*) Message_Body= "Error Page not found header and content"
;;
esac
fi
Happy bashing!
Another way to do this
while true; do (echo -e 'HTTP/1.1 200 OK\r\n'; echo -e "\n\tMy website has date function" ; echo -e "\t$(date)\n") | nc -lp 8080; done
Let's test it with 2 HTTP request using curl
In this example, 172.16.2.6 is the server IP Address.
Server Side
admin#server:~$ while true; do (echo -e 'HTTP/1.1 200 OK\r\n'; echo -e "\n\tMy website has date function" ; echo -e "\t$(date)\n") | nc -lp 8080; done
GET / HTTP/1.1 Host: 172.16.2.6:8080 User-Agent: curl/7.48.0 Accept:
*/*
GET / HTTP/1.1 Host: 172.16.2.6:8080 User-Agent: curl/7.48.0 Accept:
*/*
Client Side
user#client:~$ curl 172.16.2.6:8080
My website has date function
Tue Jun 13 18:00:19 UTC 2017
user#client:~$ curl 172.16.2.6:8080
My website has date function
Tue Jun 13 18:00:24 UTC 2017
user#client:~$
If you want to execute another command, feel free to replace $(date).
I had the same need/problem but nothing here worked for me (or I didn't understand everything), so this is my solution.
I post my minimal_http_server.sh (working with my /bin/bash (4.3.11) but not /bin/sh because of the redirection):
rm -f out
mkfifo out
trap "rm -f out" EXIT
while true
do
cat out | nc -l 1500 > >( # parse the netcat output, to build the answer redirected to the pipe "out".
export REQUEST=
while read -r line
do
line=$(echo "$line" | tr -d '\r\n')
if echo "$line" | grep -qE '^GET /' # if line starts with "GET /"
then
REQUEST=$(echo "$line" | cut -d ' ' -f2) # extract the request
elif [ -z "$line" ] # empty line / end of request
then
# call a script here
# Note: REQUEST is exported, so the script can parse it (to answer 200/403/404 status code + content)
./a_script.sh > out
fi
done
)
done
And my a_script.sh (with your need):
#!/bin/bash
echo -e "HTTP/1.1 200 OK\r"
echo "Content-type: text/html"
echo
date
mkfifo pipe;
while true ;
do
#use read line from pipe to make it blocks before request comes in,
#this is the key.
{ read line<pipe;echo -e "HTTP/1.1 200 OK\r\n";echo $(date);
} | nc -l -q 0 -p 8080 > pipe;
done
Here is a beauty of a little bash webserver, I found it online and forked a copy and spruced it up a bit - it uses socat or netcat I have tested it with socat -- it is self-contained in one-script and generates its own configuration file and favicon.
By default it will start up as a web enabled file browser yet is easily configured by the configuration file for any logic. For files it streams images and music (mp3's), video (mp4's, avi, etc) -- I have tested streaming various file types to Linux,Windows and Android devices including a smartwatch!
I think it streams better than VLC actually. I have found it useful for transferring files to remote clients who have no access beyond a web browser e.g. Android smartwatch without needing to worry about physically connecting to a USB port.
If you want to try it out just copy and paste it to a file named bashttpd, then start it up on the host with $> bashttpd -s
Then you can go to any other computer (presuming the firewall is not blocking inbound tcp connections to port 8080 -- the default port, you can change the port to whatever you want using the global variables at the top of the script). http://bashttpd_server_ip:8080
#!/usr/bin/env bash
#############################################################################
###########################################################################
### bashttpd v 1.12
###
### Original author: Avleen Vig, 2012
### Reworked by: Josh Cartwright, 2012
### Modified by: A.M.Danischewski, 2015
### Issues: If you find any issues leave me a comment at
### http://scriptsandoneliners.blogspot.com/2015/04/bashttpd-self-contained-bash-webserver.html
###
### This is a simple Bash based webserver. By default it will browse files and allows for
### retrieving binary files.
###
### It has been tested successfully to view and stream files including images, mp3s,
### mp4s and downloading files of any type including binary and compressed files via
### any web browser.
###
### Successfully tested on various browsers on Windows, Linux and Android devices (including the
### Android Smartwatch ZGPAX S8).
###
### It handles favicon requests by hardcoded favicon image -- by default a marathon
### runner; change it to whatever you want! By base64 encoding your favorit favicon
### and changing the global variable below this header.
###
### Make sure if you have a firewall it allows connections to the port you plan to
### listen on (8080 by default).
###
### By default this program will allow for the browsing of files from the
### computer where it is run.
###
### Make sure you are allowed connections to the port you plan to listen on
### (8080 by default). Then just drop it on a host machine (that has bash)
### and start it up like this:
###
### $192.168.1.101> bashttpd -s
###
### On the remote machine you should be able to browse and download files from the host
### server via any web browser by visiting:
###
### http://192.168.1.101:8080
###
#### This program requires (to work to full capacity) by default:
### socat or netcat (w/ '-e' option - on Ubuntu netcat-traditional)
### tree - useful for pretty directory listings
### If you are using socat, you can type: bashttpd -s
###
### to start listening on the LISTEN_PORT (default is 8080), you can change
### the port below.
### E.g. nc -lp 8080 -e ./bashttpd ## <-- If your nc has the -e option.
### E.g. nc.traditional -lp 8080 -e ./bashttpd
### E.g. bashttpd -s -or- socat TCP4-LISTEN:8080,fork EXEC:bashttpd
###
### Copyright (C) 2012, Avleen Vig <avleen#gmail.com>
###
### Permission is hereby granted, free of charge, to any person obtaining a copy of
### this software and associated documentation files (the "Software"), to deal in
### the Software without restriction, including without limitation the rights to
### use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
### the Software, and to permit persons to whom the Software is furnished to do so,
### subject to the following conditions:
###
### The above copyright notice and this permission notice shall be included in all
### copies or substantial portions of the Software.
###
### THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
### IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
### FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
### COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
### IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
### CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
###
###########################################################################
#############################################################################
### CHANGE THIS TO WHERE YOU WANT THE CONFIGURATION FILE TO RESIDE
declare -r BASHTTPD_CONF="/tmp/bashttpd.conf"
### CHANGE THIS IF YOU WOULD LIKE TO LISTEN ON A DIFFERENT PORT
declare -i LISTEN_PORT=8080
## If you are on AIX, IRIX, Solaris, or a hardened system redirecting
## to /dev/random will probably break, you can change it to /dev/null.
declare -a DUMP_DEV="/dev/random"
## Just base64 encode your favorite favicon and change this to whatever you want.
declare -r FAVICON="AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAADg4+3/srjc/5KV2P+ortn/xMrj/6Ch1P+Vl9f/jIzc/3572f+CgNr/fnzP/3l01f+Ih9r/h4TZ/8fN4//P1Oj/3uPr/7O+1v+xu9X/u8XY/9bi6v+UmdD/XV26/3F1x/+GitT/VVXC/3x/x/+HjNT/lp3Z/6633f/E0eD/2ePr/+bt8v/U4+v/0uLp/9Xj6//Z5e3/oKbX/0pJt/9maML/cHLF/3p8x//T3+n/3Ofu/9vo7//W5Oz/0uHq/9zn7f/j6vD/1OLs/8/f6P/R4Oj/1OPr/7jA4f9KSbf/Skm3/3p/yf/U4ez/1ePq/9rn7//Z5e3/0uHp/87e5//a5Ov/5Ovw/9Hf6v/T4uv/1OLp/9bj6/+kq9r/Skq3/0pJt/+cotb/zdnp/9jl7f/a5u//1+Ts/9Pi6v/O3ub/2uXr/+bt8P/Q3un/0eDq/9bj7P/Z5u7/r7jd/0tKt/9NTLf/S0u2/8zW6v/c5+//2+fv/9bj6//S4un/zt3m/9zm7P/k7PD/1OPr/9Li7P/V5Oz/2OXt/9jl7v+HjM3/lZvT/0tKt/+6w+L/2ebu/9fk7P/V4+v/0uHq/83d5v/a5ev/5ezw/9Pi6v/U4+z/1eXs/9bj6//b5+//vsjj/1hYvP9JSLb/horM/9nk7P/X5e3/1eTs/9Pi6v/P3uf/2eXr/+Tr7//O3+n/0uLr/9Xk7P/Y5e3/w8/k/7XA3/9JR7f/SEe3/2lrw//G0OX/1uLr/9Xi7P/T4ev/0N/o/9zn7f/k7PD/zN3p/8rd5v/T4ur/1ePt/5We0/+0w9//SEe3/0pKt/9OTrf/p7HZ/7fD3//T4uv/0N/o/9Hg6f/d5+3/5ezw/9Li6//T4uv/2ubu/8PQ5f9+hsr/ucff/4eOzv+Ei8z/rLja/8zc6P/I1+b/0OLq/8/f6P/Q4Oj/3eft/+bs8f/R4On/0+Lq/9Tj6v/T4Ov/wM7h/9Df6f/M2uf/z97q/9Dg6f/Q4On/1OPr/9Tj6//S4ur/0ODp/93o7f/n7vH/0N/o/8/f5//P3+b/2OXt/9zo8P/c6fH/zdjn/7fB3/+3weD/1eLs/9nn7//V5Oz/0+Lr/9Pi6//e6O7/5u3x/9Pi6v/S4en/0uLp/9Tj6//W4+v/3Ojw/9rm7v9vccT/wcvm/9rn7//X5Oz/0uHq/9Hg6f/S4er/3uju/+bt8f/R4On/0uHp/9Xk6//Y5u7/1OTs/9bk7P/W5Ov/XFy9/2lrwf/a5+//1uPr/9Pi6v/U4er/0eHq/93o7v/v8vT/5ezw/+bt8f/o7vL/6e/z/+jv8v/p7/L/6e/y/9XZ6//IzOX/6e7y/+nv8v/o7vL/5+7x/+ft8f/r8PP/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
declare -i DEBUG=1
declare -i VERBOSE=0
declare -a REQUEST_HEADERS
declare REQUEST_URI=""
declare -a HTTP_RESPONSE=(
[200]="OK"
[400]="Bad Request"
[403]="Forbidden"
[404]="Not Found"
[405]="Method Not Allowed"
[500]="Internal Server Error")
declare DATE=$(date +"%a, %d %b %Y %H:%M:%S %Z")
declare -a RESPONSE_HEADERS=(
"Date: $DATE"
"Expires: $DATE"
"Server: Slash Bin Slash Bash"
)
function warn() { ((${VERBOSE})) && echo "WARNING: $#" >&2; }
function chk_conf_file() {
[ -r "${BASHTTPD_CONF}" ] || {
cat >"${BASHTTPD_CONF}" <<'EOF'
#
# bashttpd.conf - configuration for bashttpd
#
# The behavior of bashttpd is dictated by the evaluation
# of rules specified in this configuration file. Each rule
# is evaluated until one is matched. If no rule is matched,
# bashttpd will serve a 500 Internal Server Error.
#
# The format of the rules are:
# on_uri_match REGEX command [args]
# unconditionally command [args]
#
# on_uri_match:
# On an incoming request, the URI is checked against the specified
# (bash-supported extended) regular expression, and if encounters a match the
# specified command is executed with the specified arguments.
#
# For additional flexibility, on_uri_match will also pass the results of the
# regular expression match, ${BASH_REMATCH[#]} as additional arguments to the
# command.
#
# unconditionally:
# Always serve via the specified command. Useful for catchall rules.
#
# The following commands are available for use:
#
# serve_file FILE
# Statically serves a single file.
#
# serve_dir_with_tree DIRECTORY
# Statically serves the specified directory using 'tree'. It must be
# installed and in the PATH.
#
# serve_dir_with_ls DIRECTORY
# Statically serves the specified directory using 'ls -al'.
#
# serve_dir DIRECTORY
# Statically serves a single directory listing. Will use 'tree' if it is
# installed and in the PATH, otherwise, 'ls -al'
#
# serve_dir_or_file_from DIRECTORY
# Serves either a directory listing (using serve_dir) or a file (using
# serve_file). Constructs local path by appending the specified root
# directory, and the URI portion of the client request.
#
# serve_static_string STRING
# Serves the specified static string with Content-Type text/plain.
#
# Examples of rules:
#
# on_uri_match '^/issue$' serve_file "/etc/issue"
#
# When a client's requested URI matches the string '/issue', serve them the
# contents of /etc/issue
#
# on_uri_match 'root' serve_dir /
#
# When a client's requested URI has the word 'root' in it, serve up
# a directory listing of /
#
# DOCROOT=/var/www/html
# on_uri_match '/(.*)' serve_dir_or_file_from "$DOCROOT"
# When any URI request is made, attempt to serve a directory listing
# or file content based on the request URI, by mapping URI's to local
# paths relative to the specified "$DOCROOT"
#
#unconditionally serve_static_string 'Hello, world! You can configure bashttpd by modifying bashttpd.conf.'
DOCROOT=/
on_uri_match '/(.*)' serve_dir_or_file_from
# More about commands:
#
# It is possible to somewhat easily write your own commands. An example
# may help. The following example will serve "Hello, $x!" whenever
# a client sends a request with the URI /say_hello_to/$x:
#
# serve_hello() {
# add_response_header "Content-Type" "text/plain"
# send_response_ok_exit <<< "Hello, $2!"
# }
# on_uri_match '^/say_hello_to/(.*)$' serve_hello
#
# Like mentioned before, the contents of ${BASH_REMATCH[#]} are passed
# to your command, so its possible to use regular expression groups
# to pull out info.
#
# With this example, when the requested URI is /say_hello_to/Josh, serve_hello
# is invoked with the arguments '/say_hello_to/Josh' 'Josh',
# (${BASH_REMATCH[0]} is always the full match)
EOF
warn "Created bashttpd.conf using defaults. Please review and configure bashttpd.conf before running bashttpd again."
# exit 1
}
}
function recv() { ((${VERBOSE})) && echo "< $#" >&2; }
function send() { ((${VERBOSE})) && echo "> $#" >&2; echo "$*"; }
function add_response_header() { RESPONSE_HEADERS+=("$1: $2"); }
function send_response_binary() {
local code="$1"
local file="${2}"
local transfer_stats=""
local tmp_stat_file="/tmp/_send_response_$$_"
send "HTTP/1.0 $1 ${HTTP_RESPONSE[$1]}"
for i in "${RESPONSE_HEADERS[#]}"; do
send "$i"
done
send
if ((${VERBOSE})); then
## Use dd since it handles null bytes
dd 2>"${tmp_stat_file}" < "${file}"
transfer_stats=$(<"${tmp_stat_file}")
echo -en ">> Transferred: ${file}\n>> $(awk '/copied/{print}' <<< "${transfer_stats}")\n" >&2
rm "${tmp_stat_file}"
else
## Use dd since it handles null bytes
dd 2>"${DUMP_DEV}" < "${file}"
fi
}
function send_response() {
local code="$1"
send "HTTP/1.0 $1 ${HTTP_RESPONSE[$1]}"
for i in "${RESPONSE_HEADERS[#]}"; do
send "$i"
done
send
while IFS= read -r line; do
send "${line}"
done
}
function send_response_ok_exit() { send_response 200; exit 0; }
function send_response_ok_exit_binary() { send_response_binary 200 "${1}"; exit 0; }
function fail_with() { send_response "$1" <<< "$1 ${HTTP_RESPONSE[$1]}"; exit 1; }
function serve_file() {
local file="$1"
local CONTENT_TYPE=""
case "${file}" in
*\.css)
CONTENT_TYPE="text/css"
;;
*\.js)
CONTENT_TYPE="text/javascript"
;;
*)
CONTENT_TYPE=$(file -b --mime-type "${file}")
;;
esac
add_response_header "Content-Type" "${CONTENT_TYPE}"
CONTENT_LENGTH=$(stat -c'%s' "${file}")
add_response_header "Content-Length" "${CONTENT_LENGTH}"
## Use binary safe transfer method since text doesn't break.
send_response_ok_exit_binary "${file}"
}
function serve_dir_with_tree() {
local dir="$1" tree_vers tree_opts basehref x
## HTML 5 compatible way to avoid tree html from generating favicon
## requests in certain browsers, such as browsers in android smartwatches. =)
local no_favicon=" <link href=\"data:image/x-icon;base64,${FAVICON}\" rel=\"icon\" type=\"image/x-icon\" />"
local tree_page=""
local base_server_path="/${2%/}"
[ "$base_server_path" = "/" ] && base_server_path=".."
local tree_opts="--du -h -a --dirsfirst"
add_response_header "Content-Type" "text/html"
# The --du option was added in 1.6.0. "/${2%/*}"
read _ tree_vers x < <(tree --version)
tree_page=$(tree -H "$base_server_path" -L 1 "${tree_opts}" -D "${dir}")
tree_page=$(sed "5 i ${no_favicon}" <<< "${tree_page}")
[[ "${tree_vers}" == v1.6* ]]
send_response_ok_exit <<< "${tree_page}"
}
function serve_dir_with_ls() {
local dir="$1"
add_response_header "Content-Type" "text/plain"
send_response_ok_exit < \
<(ls -la "${dir}")
}
function serve_dir() {
local dir="$1"
# If `tree` is installed, use that for pretty output.
which tree &>"${DUMP_DEV}" && \
serve_dir_with_tree "$#"
serve_dir_with_ls "$#"
fail_with 500
}
function urldecode() { [ "${1%/}" = "" ] && echo "/" || echo -e "$(sed 's/%\([[:xdigit:]]\{2\}\)/\\\x\1/g' <<< "${1%/}")"; }
function serve_dir_or_file_from() {
local URL_PATH="${1}/${3}"
shift
URL_PATH=$(urldecode "${URL_PATH}")
[[ $URL_PATH == *..* ]] && fail_with 400
# Serve index file if exists in requested directory
[[ -d "${URL_PATH}" && -f "${URL_PATH}/index.html" && -r "${URL_PATH}/index.html" ]] && \
URL_PATH="${URL_PATH}/index.html"
if [[ -f "${URL_PATH}" ]]; then
[[ -r "${URL_PATH}" ]] && \
serve_file "${URL_PATH}" "$#" || fail_with 403
elif [[ -d "${URL_PATH}" ]]; then
[[ -x "${URL_PATH}" ]] && \
serve_dir "${URL_PATH}" "$#" || fail_with 403
fi
fail_with 404
}
function serve_static_string() {
add_response_header "Content-Type" "text/plain"
send_response_ok_exit <<< "$1"
}
function on_uri_match() {
local regex="$1"
shift
[[ "${REQUEST_URI}" =~ $regex ]] && \
"$#" "${BASH_REMATCH[#]}"
}
function unconditionally() { "$#" "$REQUEST_URI"; }
function main() {
local recv=""
local line=""
local REQUEST_METHOD=""
local REQUEST_HTTP_VERSION=""
chk_conf_file
[[ ${UID} = 0 ]] && warn "It is not recommended to run bashttpd as root."
# Request-Line HTTP RFC 2616 $5.1
read -r line || fail_with 400
line=${line%%$'\r'}
recv "${line}"
read -r REQUEST_METHOD REQUEST_URI REQUEST_HTTP_VERSION <<< "${line}"
[ -n "${REQUEST_METHOD}" ] && [ -n "${REQUEST_URI}" ] && \
[ -n "${REQUEST_HTTP_VERSION}" ] || fail_with 400
# Only GET is supported at this time
[ "${REQUEST_METHOD}" = "GET" ] || fail_with 405
while IFS= read -r line; do
line=${line%%$'\r'}
recv "${line}"
# If we've reached the end of the headers, break.
[ -z "${line}" ] && break
REQUEST_HEADERS+=("${line}")
done
}
if [[ ! -z "{$1}" ]] && [ "${1}" = "-s" ]; then
socat TCP4-LISTEN:${LISTEN_PORT},fork EXEC:"${0}"
else
main
source "${BASHTTPD_CONF}"
fail_with 500
fi
LOL, a super lame hack, but at least curl and firefox accepts it:
while true ; do (dd if=/dev/zero count=10000;echo -e "HTTP/1.1\n\n $(date)") | nc -l 1500 ; done
You better replace it soon with something proper!
Ah yes, my nc were not exactly the same as yours, it did not like the -p option.
If you're using Apline Linux, the BusyBox netcat is slightly different:
while true; do nc -l -p 8080 -e sh -c 'echo -e "HTTP/1.1 200 OK\n\n$(date)"'; done
And another way using printf:
while true; do nc -l -p 8080 -e sh -c "printf 'HTTP/1.1 200 OK\n\n%s' \"$(date)\""; done
while true; do (echo -e 'HTTP/1.1 200 OK\r\nConnection: close\r\n';) | timeout 1 nc -lp 8080 ; done
Closes connection after 1 sec, so curl doesn't hang on it.
Type in nc -h and see if You have -e option available. If yes, You can create a script, for example:
script.sh
echo -e "HTTP/1.1 200 OK\n\n $(date)"
and run it like this:
while true ; do nc -l -p 1500 -e script.sh; done
Note that -e option needs to be enabled at compilation to be available.
I think the problem that all the solution listed doesn't work, is intrinsic in the nature of http service, the every request established is with a different client and the response need to be processed in a different context, every request must fork a new instance of response...
The current solution I think is the -e of netcat but I don't know why doesn't work... maybe is my nc version that I test on openwrt...
with socat it works....
I try this https://github.com/avleen/bashttpd
and it works, but I must run the shell script with this command.
socat tcp-l:80,reuseaddr,fork EXEC:bashttpd &
The socat and netcat samples on github doesn't works for me, but the socat that I used works.
Actually, the best way to close gracefully the connection is to send the Content-Length header like following. Client (like curl will close the connection after receiving the data.
DATA="Date: $(date)";
LENGTH=$(echo $DATA | wc -c);
echo -e "HTTP/1.1 200 OK\nContent-Length: ${LENGTH}\n\n${DATA}" | nc -l -p 8000;
On OSX you can use :
while true; do echo -e "HTTP/1.1 200 OK\n\n $(date)" | nc -l localhost 1500 ; done