I am trying to run a bash script which should load data into jena. This script comes from a github repository and was allegedly working on the owner's machine but on mine it won't run, even though I followed the instructions. So let me first describe what the script does based on my understanding: It should load .nt data (RDF data) into Jena using docker by using the docker image of jena, named stain/jena. Here is the script:
#/bin/bash
files=$(echo $(pwd)/rawdata-bearb/hour/alldata.IC.nt/*.nt | sed "s%$(pwd)/rawdata-bearb/hour/alldata.IC.nt%/var/data/in%g")
mkdir output # added
for file in $files; do
v=$(echo $file | sed "s/^.*\/\([0-9][0-9]*\)\.nt$/\1-1/" | bc)
echo "$v"
mkdir -p /var/data/out/ic/$v
time docker run \
-it \
--rm \
-v $(pwd)/tdb-bearb-hour/:/var/data/out/ \
-v $(pwd)/rawdata-bearb/hour/alldata.IC.nt/:/var/data/in/ \
stain/jena /jena/bin/tdbloader2 \
--sort-args "-S=16G" \
--loc /var/data/out/ic/$v $file \
> output/load-bearb-hour-ic-$v-.txt
done
However, when I execute the script, I get following message from the saved log file:
13:12:46 INFO -- TDB Bulk Loader Start
mkdir: cannot create directory ‘/var/data/out/ic/0’: No such file or directory
13:12:46 ERROR Failed during data phase
According to the tdbloader2 manual the --loc parameter should create the directory if it does not exist
-- loc: Sets the location in which the database should be created.
This location must be a directory and must be empty,
if a non-existent path is specified it will be created as a new directory.
I created the directories /var/data/out/ic/0 - /var/data/out/ic/10 manually and re-executed the script. Still, I got the same error message. My first guess was that tdbloader2 or docker use the mkdir command without the -p parameter but since I manually created the directories, thus, they existed before the execution and I still got the same error, it must be something else. I am kindly asking for your help
Related
I have the following code on cron job, it runs but the code does not really do what it supposed to. It does not create the directory plus is does not do anything in the code. Please help check if the way I pointed to the directory is wrong.
#!/bin/bash
NAMEDATE=`date +%F_%H-%M`_`whoami`
NAMEDATE2=`date `
mkdir ~/home/u3811*****/domains/website.com/public_html/cron/backup/files/$NAMEDATE -m 0755
mysqldump -u u3811*****_boss -p"*******" u3811*****_data | gzip ~/home/u3811*****/domains/website.com/public_html/cron/backup/files/$NAMEDATE/db.sql.gz
echo "This is the database backup for website.com on $NAMEDATE2" |
mailx -a ~/home/u3811*****/domains/website.com/public_html/cron/backup/files/$NAMEDATE/db.sql.gz -s "website.com Database attached" -- mail#gmail.com
chmod -R 0644 ~/home/u3811*****/domains/website.com/public_html/cron/backup/files/$NAMEDATE/*
exit 0
Your NAMEDATE variable needs to be modified a bit, as shown below, for more information about variables in bash see this link
NAMEDATE=$(date +%F_%H-%M"_"$(whoami))
When you issue the mkdir command you will need to pass the -p option to create the complete directory structure if it doesn't exists.
mkdir -p ~/home/u3811numbers/domains/website.com/public_html/cron/backup/files/$NAMEDATE -m 0755
Also, the ~ character on Linux based distributions is used as a shortcut for the home directory of the user that invokes it so, in the line below the result is /home//home/u3811*****/domains/website.com/public_html/cron/backup/files/2020-09-04_23-13_ you can read more about it in here
In you last command before the exit, you might need to pass a wildcard (*) to avoid removing the executable bit on the directory, see below
chmod -R 0644 ~/home/u3811*****/domains/website.com/public_html/cron/backup/files/$NAMEDATE/
The final version of your script will look something like this.
#!/bin/bash
NAMEDATE=$(date +%F_%H-%M"_"$(whoami))
NAMEDATE2=date
mkdir -p ~/home/u3811******/domains/website.com/public_html/cron/backup/files/$NAMEDATE -m 0755
mysqldump -u u3811*****_boss -p"******" u3811*****_data | gzip > ~/home/u3811*****/domains/website.com/public_html/cron/backup/files/$NAMEDATE/db.sql.gz
echo "This is the database backup for website.com on $NAMEDATE2" | mailx -a ~/home/u3811*****/domains/website.com/public_html/cron/backup/files/$NAMEDATE/db.sql.gz -s "website.com Database attached" -- mail#gmail.com
chmod -R 0644 ~/home/u3811*****/domains/website.com/public_html/cron/backup/files/$NAMEDATE/*
To debug a bash script you can always pass the -x flag for more information take a look at this article
I am new to Docker, Debezium, Bash, and Kafka. I am attempting to run the Debezium tutorial/example for MSSQL Server on Windows 10 here:
https://github.com/debezium/debezium-examples/blob/master/tutorial/README.md#using-sql-server
I am able to start the topology, per step one. However, when I go to step two and execute the following command:
cat debezium-sqlserver-init/inventory.sql | docker exec -i tutorial_sqlserver_1 bash -c '/opt/mssql-tools/bin/sqlcmd -U sa -P $SA_PASSWORD'
I get the following error:
bash: C:/Program: No such file or directory
I do not have the foggiest idea why it would even drag C:/Program in to this. I do not see it in the command nor do I see it in the *.sql file. Does anyone know why this is happening and what the fix is?
Note 1: I am already in the current directory where this command should be runnable and there are no spaces in the folder/file path
Note 2: I am running the commands in Git Bash
When using set -x to log how the command is run, there's still no C:/Program anywhere in it, as can be seen by the following log:
$ cat debezium-sqlserver-init/inventory.sql | docker exec -i tutorial_sqlserver_1 bash -c '/opt/mssql-tools/bin/sqlcmd -U sa -P $SA_PASSWORD'
+ cat debezium-sqlserver-init/inventory.sql
+ docker exec -i tutorial_sqlserver_1 bash -c '/opt/mssql-tools/bin/sqlcmd -U sa -P $SA_PASSWORD'
bash: C:/Program: No such file or directory
I had a similar problem yesterday, the solution was adding a backslash before the absolute path, like :
cat debezium-sqlserver-init/inventory.sql | docker exec -i tutorial_sqlserver_1 bash -c '\/opt/mssql-tools/bin/sqlcmd -U sa -P $SA_PASSWORD'
\/opt/mssql-tools/bin/sqlcmd prevents conversion to Windows path.
I want to be able to specify a directory with fastq files with a script that will loop through all the files and do some stuff. Here is my attempt:
threads=24
current_path=`pwd`
input_file=${current_path}/raw/
files=${current_path}/raw/*
for file in ${files}; do
output_file=${current_path}/${file}_out/
mkdir -m gu=wrx,o=rx ${output_file}
spades.py \
--s1 ${input_file}${file} \
-t ${threads} \
--plasmid \
--careful \
-o ${output_file}
done
So in this script I get an error: cannot make directory, directory does not exist The script generates a /home folder. I don't know if I am specifying the files incorrectly or if I am using the for loop incorrectly.
Thank you!
you concatenate full path to file with folder in line
output_file=${current_path}/${file}_out/
it should be
output_file=${file}_out/
In a directory I have a config file with my db variables.
This file (db/database.ini) looks like this:
[PostgreSQL]
host=localhost
database=...
user=postgres
password=...
I have another file (db/create_stmts.sql) where I have all my raw create table statements, and i am trying to experiment the use of a Makefile to have a command like this:
make create-db from_file=db/create_stmts.sql
In order not to repeat myself, I thought of tailing the variables of db/database.ini to a file which I would then source, creating shell variables to pass to psql in the make file.
Here's my plan:
make-db:
# from_file: path to .sql file with all create statements to create the database where to insert
# how to run: make create-db from_file={insert path to sql file}
file_path=$(PWD)/file.sh
tail -n4 db/database.ini > file.sh && . $(file_path)
# -U: --user
# -d: --database
# -q: --quiet
# -f: --file
psql -U $(user) -d $(database) -q -f $(from_file) && rm file.sh
Which I run by: make create-db from_file=db/create_stmts.sql
Which gives me this message - from which i kindof understand that the sourcing just did not work.
#from_file: path to .sql file with all create statements to create the database where to insert
# how to run: make create-db from_file={insert path to sql file}
file_path=/home/gabriele/Desktop/TIUK/companies-house/file.sh
tail -n4 db/database.ini > file.sh && .
# -U: --user
# -d: --database
# -q: --quiet
# -f: --file
psql -U -d -q -f db/schema_tables.sql && rm file.sh
psql: FATAL: Peer authentication failed for user "-d"
Makefile:3: recipe for target 'create-db' failed
make: *** [create-db] Error 2
Any help?
Another solution, perhaps simpler to understand:
make-db:
file_path=$$PWD/file.sh; \
tail -n4 db/database.ini > file.sh && . $$file_path; \
psql -U $$user -d $$database -q -f $$from_file && rm file.sh
Note using ; and \ to convince make to run all commands in a single shell, and using $$ to escape the $ and use shell variable references.
The error is in the text, namely
psql -U -d -q -f db/schema_tables.sql && rm file.sh
This happens because the variables $(user) and $(database) aren't set. Every line within a target is executed in a sub shell. There is now way to use source like you would in a regular script.
You could create a file named database.mk in which you define these variables and use include database.mk at the top of your makefile to include them:
Makefile
CONFILE ?= database
include $(CONFILE).mk
test:
#echo $(user)
#echo $(database)
database.mk
user := user
database := data
If you want to parse the ini file you could do that as such
CONFILE := db/database.ini
make-db: _setup_con
echo $(user) $(database)
# your target
_setup_con:
$(eval user=$(shell grep "user=" $(CONFILE) | grep -Eo "[^=]*$$"))
$(eval database=$(shell grep "database=" $(CONFILE) | grep -Eo "[^=]*$$"))
# and so forward
I would make it more Make-way by using feature of automatic Makefile generation. Given that a configuration file is a simple properties file, its syntax is easily parseable by Make, it's sufficient to just get the lines with variables, i.e.:
include database.mk
database.mk: db/database.ini
grep -E '^\w+=\w+$$' $< > $#
.PHONY: create-db
create-db: $(from_file)
psql -U $(user) -d $(database) -q -f $<
Some additional notes:
create-db should be made .PHONY to avoid situation when nothing is done due to somebody creating (accidentally or not) a file named create-db,
by making create-db depending on from_file one can get a clean and readable error from make that a file does not exist instead of possibly cryptic error later.
I am trying to create a shell script that will check for a new file then cp to a Docker Container. The code I have so far is...
#!/bin/sh
source="/var/www/html/"
dest="dev_ubuntu:/var/www/html/"
inotifywait -m "/var/www/html" -e create -e moved_to |
while read file; do
sudo docker cp /var/www/html/$file dev_ubuntu:/var/www/html
done
But this code gives the following error:
Setting up watches.
Watches established.
"docker cp" requires exactly 2 argument(s).
See 'docker cp --help'.
Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
Copy files/folders between a container and the local filesystem
What am I doing wrong?
Do you have spaces in your file names? Use double quotes to avoid separating filenames by words:
echo $file
sudo docker cp "$file" dev_ubuntu:"$file"
I've also echoed the file name to see what is happening.