mongoexport recently hours datas from mongodb - bash

I am a new beginner of mongodb. I want to export some data in recent hours from my databases. So, I think I need to write mongoexport command and include date range in --query options to do it.
I write a bash file like this and try to run it:
#!/bin/bash
mongoexport --host localhost:27017 --db copy --collection txt --csv --fields x1,x2,x3...,date --query '{ "date" : {$gt:new Date(new Date() - 1000*60*60*3)} }' --out home/data.csv
But I get the results says:
connected to: localhost:27017
assertion: 16619 code FailedToParse: FailedToParse: Expecting '}' or ',': offset:25 of:{ "date" : {$gt:new Date(new Date() - 1000*60*60*3)} }
It sees connect to localhost but cannot output the data. If I remove --query option, this can run successfully and get the whole data, but I must need the query to subset the data in recently 3 hours.
Any ideas and help will be highly appreciated. Thank you and Best.

with mongoexport you have to provide the Date object a timestamp.
An explaination is answered here: MongoDb timestamp
What you can write as script is something like this (I'm rather rusty with bash, surely can be improved to stay on a single line):
timestamp=$(date +%s)
let total=$timestamp*1000-3600*1000*3
mongoexport --host localhost:27017 --db copy --collection txt --csv --fields x1,x2,x3...,date --query '{ "date" : {$gt:new Date('$total')} }' --out home/data.csv

Related

Insert Error while trying to load data to postgres from oracle using Kafka-Connect-API

I'm trying to transfer data from oracle to postgres using kafka jdbc connect API
I don't wanna create a new table at postgres automatically. so I created my own table at postgres before creating a connectors.
so these are my configuration:-
Source:-(Ora XE 11g)
curl -i -X PUT http://localhost:8083/connectors/ora_1/config
-H "Content-Type: application/json"
-d '{
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"connection.url":"jdbc:oracle:thin:#//localhost:1521/db",
"connection.user":"user",
"connection.password":"passwd",
"mode":"incrementing",
"incrementing.column.name": "<column_name>",
"topic.prefix":"ora_",
"table.whitelist":"<table_name>",
"poll.interval.ms":10000
}'
Target:- (Postgres)
curl -X PUT http://localhost:8083/connectors/pg_1/config \
-H "Content-Type: application/json" \
-d '{
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"connection.url":"jdbc:postgresql://localhost:5432/db",
"connection.user":"user",
"connection.password":"passwd",
"tasks.max" : "1",
"topics":"<topic_name>",
"insert.mode":"insert",
"table.name.format":"<table_name>"
}'
If we see in the image
In insert statement the columns are coming within the double quotes which is leading me to an insert error. When I tried by creating the table with columns enclosing in double quotes at postgres then it's working fine.
So, Anyone can please help me out how to avoid those double quotes for those columns as I don't wanna create table automatically and don't wanna create table by defining it's columns in double quotes.....
I used "quote.sql.identifiers":"never" at the source and sink configuration. now it is working fine.
you can have a look here:-
https://github.com/confluentinc/kafka-connect-jdbc/issues/455

How to add sysdate from bcp

I have a .csv file with the following sample data format:
REFID|PARENTID|QTY|DESCRIPTION|DATE
AA01|1234|1|1st item|null
AA02|12345|2|2nd item|null
AA03|12345|3|3rd item|null
AA04|12345|4|4th item|null
To load the above file into a table I am using below BCP command:
/bcp $TABLE_NAME in $FILE_NAME -S $DB_SERVER -t "|" -F 1 -U $DB_USERNAME -d $DB_NAME
What i am trying to look here is like below (adding sysdate instead of null from bcp)
AA01|1234|1|1st item|3/16/2020
AA02|12345|2|2nd item|3/16/2020
AA03|12345|3|3rd item|3/16/2020
AA04|12345|4|4th item|3/16/2020
Update : I was able to exclude header with #Jamie answer by -F 1 option, but looking for some help on inserting date with bcp. Tried looking some old Q&A, but no luck so far..
To exclude a single header record, you can use the -F option. This will tell BCP which line in the file is the first line to begin loading from. For your sample, -F2 should work fine. However, your command has other issues. See comments.
There is no way to introduce new data using the BCP command as you stated. BCP cannot introduce a date value while copying data into your table. To accomplish this I suggest a default for your date column or to first load the raw data into a table without the date column then you can introduce the date value as you see fit in late processing.

How to do inventory sync as soon as the inventory is created in ansible tower through script (in an automated fashion)

I wanted to understand how to do inventory sync as soon as the inventory is created using schedules (something like below) :
tower-cli schedule create --name sync_schedule --inventory-source 182 --rrule "DTSTART:20190507T183000Z RRULE:FREQ=DAILY;INTERVAL=1;COUNT=1"
Wanted to figure out rrule to sync once as soon as it is created.
Solution is to set "update_on_launch" value to true in API call:
curl --silent -lk -u $user:$password -H 'Content-Type:application/json' -XPOST -d '{"name": '\"$project_name\"',"inventory": '\"$inventory_id\"',"source_script": 4,"credential": null,"overwrite_vars": true,"overwrite": true,"update_on_launch": true,"verbosity": 1,"update_cache_timeout": 0,"source_regions":"","source_vars":"---\nTERRAFORM_ENV: stl/'$project_name'","source":"custom"}' https://awx.com/api/v2/inventory_sources/

ldapsearch - how to display "lastLogonTimestamp"

I want to search Active Directory for inactive users that have no login for x days/months. I've got such a ldapsearch query:
ldapsearch -h domain.test -p 389 -D "cn=login,ou=test,dc=domain,dc=test" -w "passwd" -s sub -b "ou=Test,dc=domain,dc=test" "(&(objectCategory=person)(objectClass=user)(userAccountControl:1.2.840.113556.1.4.803:=2))"
It gives me the list of all inactive users in domain.test with all attributes.
I would like to add a filter for searching users that have no login for x days/months, and it would be great if the result was the list of sAMAccountNames (inactive user and lastLogonTimestamp >e.g. 3months).
I'm aware that LastLogonTimestamp is not the real time of last user logon, but in this case it's not so important.
EDIT: now I only need to know if there is a way to show attribute like "lastLogonTimestamp" in the output of above ldapsearch query?
ANSWER: Attribute lastLogonTimestamp was not set for each object in the output of above ldapsearch query. I haven't noticed that. So grep displayed it:
ldapsearch -h domain.test -p 389 -D "cn=login,ou=test,dc=domain,dc=test" -w "passwd" -s sub -b "ou=Test,dc=domain,dc=test" "(&(objectCategory=person)(objectClass=user)(userAccountCont‌​rol:1.2.840.113556.1‌​.4.803:=2))" | grep -i lastlogontimestamp
EDIT:
I though that it will be ok to compare lastlogontimestamps - it isn't because lastlogontimestamp values are not comparable. The only way is to convert to date format first, and then compare to get users that lastlogon was eg. before 01/06/2017. And here's the question: how to convert windows lastlogontimestamp to date in bash?
Please let me know if it's the correct way to do it.
Any advice very appreciated.

user account creation date in LDAP

I would like to write a shell script to do the following , would advise how to make it ? very thanks
Connect LDAP server ( Id : user , password : pass )
check the field user_account_create_date in the LDAP server, the format of data in this field is ABC20130922 (September 22, 2013)
find the record in this field to check last 8 digits ,
Pseudocode:
if the date is within 7 days: # account is created within 7 days
then do
...
else do
...
I have a script as below which connect ldap, and check if the user account is created within 7 days ( the script is not complete yet)
timestamp = date --date="-7 days" +%Y%m%d%k%m%SZ
ldapsearch -h 192.168.1.100 -p 389 -D cn=admin,o=services -w pass -x "(&(objectclass=*)(createTimestamp>=$timestamp))"
then
What I hope to do now is to modify the above script so that createTimestamp get the creation date string which can be used to compare with timestamp.
Something close to this should work:
ndays=7
timestamp=`date --date="-$ndays days" +'%Y%m%d'"100000Z"`
ldapsearch -x -LLL -h yourhost.yourdoamin.com -p 389 -b "ou=people,dc=yourdomain,dc=com" -D cn=admin,....,dc=yourdomain,dc=com -w yourpassword "(&(objectclass=inetorgperson)(modifytimestamp>=$timestamp))" modifytimestamp
I used modifyTimeStamp for testing as no one had been created lately on my home system.
-jim
Ensure that the timestamp has the correct syntax (generalized time). When using a value in an assertion (filter), the value must be the syntax of the attribute. Also, the objectClass=* component of the filter is not needed: all entries have at least one objectClass.

Resources