I have a file with a list of users. I would like to query our company's ldap to check if the users on my list are still existing accounts on the company's ldap server.
The bash script would essentially, use the file to use the names to check with ldaps 'cn', then possibly output/print to the results to identify which names no longer exist.
It sounds simple, and I'm familiar with doing basic ldapsearch commands, but not sure how I would begin scripting this out.
Appreciate all the help!
I have done this exact task and my approach is this: Do the ldapsearch query and get all emails for valid users in ldap. Convert to lower case, sort, remove duplicates and store in a file. Do the same with your list of users you want to check. Then use comm to find any emails that are not in the list from LDAP. This method should be the fastest unless you have a large number of LDAP records. Here is the code:
LDAP_SERVER="ldap://YOUR.LDAP.SERVER:389"
LDAP_USER="QUERY_USER_NAME"
LDAP_PASSWORD="QUERY_PASSWORD"
ldapsearch -E pr=1000/noprompt -LLL -o ldif-wrap=no -x \
-b 'dc=example,dc=com' \
-H $LDAP_SERVER -D $LDAP_USER -w $LDAP_PASSWORD \
'(&(objectCategory=person)(objectClass=user)(Mail=*))' mail |\
awk '/^mail:/{print $2}' |\
tr '[:upper:]' '[:lower:]' |\
sort -u >ldap_emails
cat user_list |\
tr '[:upper:]' '[:lower:]' |\
sort -u >user_emails
comm -13 ldap_emails user_emails
Related
I've hit a wall after trying to find a solution to this issue. I have a file of 1000 lines from a csv that I need to use to create users within centos.
csv file is structured as username, first, last, gender, dob, country, color, fruits, OS, shell, permission
lines 601 - 1000 | Add users with the following requirements:
username
comment
shell
primary group must be their color
I have my shell script like this:
cat file.csv | cut -d, -f7 | tail -400 | while read group; do groupadd "$group"; done
cat file.csv | cut -d, -f1-3,7,10 | tail -400 | while read username first last color shell; do useradd "$username" -c "$first $last" -g "$color" -s "$shell"; done
When I run the first script, I get "groupapp: group 'color' already exists." I think mainly because it added the group the first time and when it went through the new lines, it stated it already exists. When I verify using /etc/group I do see the groups listed in there.
Now when I run the second script, I get "useradd: group ' ' does not exist". Like I stated when I looked in /etc/group the groups were there. What am I doing wrong?
For the "group already exists" problem, you could add sort -u into the pipeline.
For the useradd problem, see what the output of cut is:
$ echo '1,2,3,4,5,6,7,8,9,10' | cut -d, -f1-3,7,10
1,2,3,7,10
To split that into separate variables with the read command, you need to specify that the delimiter is comma:
cut -d, -f1-3,7,10 file.csv \
| tail -400 \
| while IFS="," read -r username first last color shell; do
useradd "$username" -c "$first $last" -g "$color" -s "$shell"
done
Notes:
cut can read files, don't need cat
you almost always want read -r
Is there any bash command/script in Linux so we can extract the active domains from a long list,
example, I have a csv file (domains.csv) there are 55 million domains are listed horizontally, we need only active domains in a csv file (active.csv)
Here active mean a domain who has a web page at least, not a domain who is expired or not expired. example whoisdatacenter.info is not expired but it has no webpage, we consider it as non-active.
I check google and stack website. I saw we can get domain by 2 ways. like
$ curl -Is google.com | grep -i location
Location: http://www.google.com/
or
nslookup google.com | grep -i name
Name: google.com
but I got no idea how can I write a program in bash for this for 55 million domains.
below commands, won't give any result so I come up that nsloop and curl is wayway to get result
$ nslookup whoisdatacenter.info | grep -i name
$ curl -Is whoisdatacenter.info | grep -i location
1st 25 lines
$ head -25 domains.csv
"
"0----0.info"
"0--0---------2lookup.com"
"0--0-------free2lookup.com"
"0--0-----2lookup.com"
"0--0----free2lookup.com"
"0--1.xyz"
"0--123456789.com"
"0--123456789.net"
"0--6.com"
"0--7.com"
"0--9.info"
"0--9.net"
"0--9.world"
"0--a.com"
"0--a.net"
"0--b.com"
"0--m.com"
"0--mm.com"
"0--reversephonelookup.com"
"0--z.com"
"0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0.com"
"0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0.com"
"0-0-0-0-0-0-0-0-0-0-0-0-0-10-0-0-0-0-0-0-0-0-0-0-0-0-0.info"
code I am running
while read line;
do nslookup "$line" | awk '/Name/';
done < domains.csv > active3.csv
the result I am getting
sh -x ravi2.sh
+ read line
+ nslookup ''
+ awk /Name/
nslookup: '' is not a legal name (unexpected end of input)
+ read line
+ nslookup '"'
+ awk /Name/
+ read line
+ nslookup '"0----0.info"'
+ awk /Name/
+ read line
+ nslookup '"0--0---------2lookup.com"'
+ awk /Name/
+ read line
+ nslookup '"0--0-------free2lookup.com"'
+ awk /Name/
+ read line
+ nslookup '"0--0-----2lookup.com"'
+ awk /Name/
+ read line
+ nslookup '"0--0----free2lookup.com"'
+ awk /Name/
still, active3.csv is empty
below . the script is working, but something stopping the bulk lookup, either it's in my host or something else.
while read line
do
nslookup $(echo "$line" | awk '{gsub(/\r/,"");gsub(/.*-|"$/,"")} 1') | awk '/Name/{print}'
done < input.csv >> output.csv
The bulk nslookup show such error in below
server can't find facebook.com\013: NXDOMAIN
[Solved]
Ravi script is working perfectly fine, I was running in my MAC which gave Nslookup Error, I work in CentOS Linux server, Nslookup work great with Ravi script
Thanks a lot!!
EDIT: Please try my EDIT solution as per OP's shown samples.
while read line
do
nslookup $(echo "$line" | awk '{gsub(/\r/,"");gsub(/.*-|"$/,"")} 1') | awk '/Name/{found=1;next} found && /Address/{print $NF}'
done < "Input_file"
Could you please try following.
OP has control M characters in her Input_file so run following command too remove them first:
tr -d '\r' < Input_file > temp && mv temp Input_file
Then run following code:
while read line
do
nslookup "$line" | awk '/Name/{found=1;next} found && /Address/{print $NF}'
done < "Input_file"
I am assuming that since you are passing domain name you need to get their address(IP address) in output. Also since you are using a huge Input_file so it may be a bit slow in providing output, but trust me this is a simpler way.
nslookup simply indicates whether or not the domain name has a record in DNS. Having one or more IP addresses does not automatically mean you have a web site; many IP addresses are allocated for different purposes altogether (but might coincidentally host a web site for another domain name entirely!)
(Also, nslookup is not particularly friendly to scripting; you will want to look at dig instead for automation.)
There is no simple way to visit 55 million possible web sites in a short time, and probably you should not be using Bash if you want to. See e.g. https://pawelmhm.github.io/asyncio/python/aiohttp/2016/04/22/asyncio-aiohttp.html for an exposition of various approaches based on Python.
The immediate error message indicates that you have DOS carriage returns in your input file; this is a common FAQ which is covered very well over at Are shell scripts sensitive to encoding and line endings?
You can run multiple curl instances in parallel but you will probably eventually saturate your network -- experiment with various degrees of parallelism -- maybe split up your file into smaller pieces and run each piece on a separate host with a separate network connection (perhaps in the cloud) but to quickly demonstrate,
tr -d '\r' <file |
xargs -P 256 -i sh -c 'curl -Is {} | grep Location'
to run 256 instances of curl in parallel. You will still need to figure out which output corresponds to which input, so maybe refactor to something like
tr -d '\r' <file |
xargs -P 256 -i sh -c 'curl -Is {} | sed -n "s/Location/{}:&/p"'
to print the input domain name in front of each output.
(Maybe also note that just a domain name is not a complete URL. curl will helpfully attempt to add a "http://" in front and then connect to that, but that still doesn't give you an accurate result if the domain only has a "https://" website and no redirect from the http:// one.)
If you are on a Mac, where xargs doesn't understand -i, try -I {} or something like
tr -d '\r' <file |
xargs -P 256 sh -c 'for url; do curl -Is "$url" | sed -n "s/Location/{}:&/p"; done' _
The examples assume you didn't already fix the DOS carriage returns once and for all; you probably really should (and consider dropping Windows from the equation entirely).
I need to retrieve the "message" variable from the website and store it in a variable.
Im new to shell script so im not sure how to do it. Ive been trying various ways for a while but they dont seem to work.
This is the output of the website example.web:8080/rest/message
[{"id":33,"message":"Dash","lastUpdated":1569922857154,"userName":null}]
#!/bin/bash
message=$( curl -# -L "http://example.web:8080/rest/message}")
username=$(
<<< "${message}" \
grep -P -o -e '(?<=<li>message: <strong>)(.*?)(?=<\/strong><\/li>)' |
head -n 1
)
I need the message variable "Dash" To be stored which can be printer later on.
Though I am not sure if output you are getting is pure json or not, if you are ok with awk could you please try following(If you have proper json then please use jq or parser which is specially made for json parsing).
Your_command | awk -v s1="\"" '
match($0,s1 "message" s1 ":" s1 "[^\"]*"){
num=split(substr($0,RSTART,RLENGTH),array,"\"")
print array[num]
}'
EDIT: If you have jq tool with you, could you please try following(not tested).
curl -# -L "http://example.web:8080/rest/message" | jq '.[] | .message'
I have an env.txt file in the following format:
lDRIVER={ODBC Driver 13 for SQL Server};
PORT=1433;
SERVER=serveename;
DATABASE=db;
UID=username;
PWD=password!
I have a git bash script (.sh) that requires the UID and PWD from that file. I was thinking about getting it by the last/second last line number. How do I do this/ is there a better way (say looking for UID and PWD and assigning the git bash variable that way)
There's lots of ways to do this. You could use awk which I would personally use since it's sort of like an x-acto knife for this type of thing:
uid=$(awk -F"[=;]" '/UID/{print $2}' env.txt)
pwd=$(awk -F"[=;]" '/PWD/{print $2}' env.txt)
Or grep and sed. sed is nice because it allows you to get very specific about the piece of info you want to cut from the line, but it's regex which has its learning curve:
uid=$(grep "UID" env.txt | sed -r 's/^.*=(.*)(;|$)/\1/g' )
pwd=$(grep "PWD" env.txt | sed -r 's/^.*=(.*)(;|$)/\1/g' )
As #JamesK noted in the comments you can use sed and have it do the search instead of grep. This is super nice and I would definitely choose this instead of the grep | sed.
uid=$(sed -nr '/UID/s/^.*=(.*)(;|$)/\1/gp' )
pwd=$(sed -nr '/PWD/s/^.*=(.*)(;|$)/\1/gp' )
Or grep and cut. Bleh... we can all do better, but sometimes we just want to grep and cut and not have to think about it:
uid=$(grep "UID" env.txt | cut -d"=" -f2 | cut -d";" -f1)
pwd=$(grep "PWD" env.txt | cut -d"=" -f2 | cut -d";" -f1)
I definitely wouldn't go by line number though. That looks like and odbc.ini file and the order in which the parameters are listed in each odbc entry are irrelevant.
First rename PWD to something like PASSWORD. PWD is a special variable used by the shell. Even better is to use lowercase variable names for all your own variables.
When the password is without special characters (spaces, $, ), you can
source env.txt
When the password has something special, consider editing the env.txt:
lDRIVER="{ODBC Driver 13 for SQL Server}"
PORT="1433"
SERVER="serveename"
DATABASE="db"
UID="username"
PASSWORD="password!"
When you are only interested in lowercase uid and passwd, consider selecting only the interesting fields and change the keywords to lowercase
source <(sed -rn '/^(UID|PWD)=/ s/([^=]*)/\L\1/p' env.txt)
I would like to have a shell script use 'ldapsearch' to compare UIDs listed in a text file with those on a remote LDAP directory.
I'm no shell script expert, and would appreciate any assistance. The following loops through a text file given as an argument, but what I need is to echo when a UID in my text file does not exist in the LDAP.
#!/bin/sh
for i in `cat $1`;
do ldapsearch -x -H ldaps://ldap-66.example.com -b ou=People,dc=crm,dc=example,dc=com uid=$i | grep uid: | awk '{print $2}';
echo $i
done
Try:
#!/bin/sh
url="ldaps://ldap-66.example.com"
basedn="ou=People,dc=crm,dc=example,dc=com"
for i in `cat $1`; do
if ldapsearch -x -H "$url" -b "$basedn" uid=$i uid > /dev/null
then
# Do nothing
true
else
echo $i
fi
done
Anttix, this is the final version that worked. Thanks again. Does this mark the question as accepted?
#!/bin/sh
url="ldaps://ldap-66.example.com"
basedn="ou=People,dc=crm,dc=example,dc=com"
for i in `cat $1`; do
if ldapsearch -x -H "$url" -b "$basedn" uid=$i | grep uid: | awk '{print $2}' > /dev/null
then
# Do nothing
true
else
echo $i
fi
done
#!/bin/sh
uids="${1:?missing expected file argument}"
URL='ldaps://ldap-66.example.com'
BASE='ou=People,dc=crm,dc=example,dc=com'
ldapsearch \
-x -LLL -o ldif-wrap=no -c \
-H "$URL" \
-b "$BASE" \
-f "$uids" \
'(uid=%s)' uid |
sed \
-r -n \
-e 's/^uid: //p' \
-e 's/^uid:: (.+)/echo "\1"|base64 -d/ep' |
grep \
-F -x -v \
-f - \
"$uids"
Explanation
for the ldapsearch part
You can invoke ldapsearch -f just once to perform multiple searches and read them from a file $uids.
The (uid=%s) is used as a template for each line, where the %s is filled in for each line.
The -c is needed to continue even on errors, e.g. when a UID is not found.
The -x enforces simple authentication with should be considered insecure. Better use SASL.
The -LLL will remove un-needed cruft from the ldapsearch output.
The -o ldif-wrap=no will prevent lines longer than 79 characters from being wrapped - otherwise grep may only pick up the first part of your user names.
The uid tells ldapsearch to only return that attribute and skip all the other attributes we're not interested in; saves some network bandwidth and processing time.
for the sed part
The -r enabled extended regular expressions turning +, (…) into operators; otherwise they have to be pre-fixed with a back-slash \.
The -n tells sed to do not print each line by default but only when told to do so using an explicit print command p.
The s/uid: // will strip that prefix from lines containing it.
The s/uid:: …/…/ep part is needed for handling non-ASCII-characters like Umlauts as ldapsearch encodes them using base64.
The suffix …p will print only those lines which were substituted.
for the grep part
The -F tells grep to not use regular expression pattern matching, but plain text; this becomes important if your names contain ., *, parenthesis, ….
The -x enforces whole line matching so searching for foo will not also match foobar.
The -f - tells grep to read the patterns from STDIN, that is the existing UIDs found by ldapsearch.
The -v inverts the search and will filter out those existing users.
The $uids will again read the list of all requested users. After removing the existing users the list of missing users will remain.
Issues with previous solutions
The previous solutions all had certain kinds of issues:
missing grep
missing quoting which breaks as soon as blanks are involved
use-less use of cat
use-less use of grep when awk is also used
inefficient because for each UID many processes get fork()ed
inefficient because for each UID a separate search is performed
did not work for long user names
did not work for user names containing regular expression meta characters
did not work for user names containing Umlauts