I'm upgrading to Airflow 2. From the below code in my entrypoint.sh, I had airflow connections --delete --conn_id, but I've changed it to the below according to the docs (https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html)
Now I'm getting the following error for each connection:
airflow command error: unrecognized arguments: airflow_db airflow.. aws_default... azure_container_instances_default .. azure_cosmos_default
delete_default_connections() {
declare -a DEFAULT_CONNECTIONS=(
"airflow_db"
"aws_default"
"azure_container_instances_default"
"azure_cosmos_default"
)
for CONN in "${DEFAULT_CONNECTIONS[#]}" do
su -c "airflow connections delete conn_id $CONN" airflow done }
Change the command to the following, conn_id is a positional argument.
delete_default_connections() {
declare -a DEFAULT_CONNECTIONS=(
"airflow_db"
"aws_default"
"azure_container_instances_default"
"azure_cosmos_default"
)
for CONN in "${DEFAULT_CONNECTIONS[#]}" do
su -c "airflow connections delete $CONN" airflow done }
Related
I created a container for an Oracle Express database following these instructions, with the following command:
docker run -d -e ORACLE_PWD="root" --name testdb -p 5500:5500 -p 8080:8080 -p 1521:1521 container-registry.oracle.com/database/express:21.3.0-xe
What does work
I managed to access the database from within the container with this command:
docker exec -it testdb sqlplus system/root#//localhost:1521/XE
I also managed to access to access the Oracle Enterprise Manager on localhost:5500/em using these credentials:
Username: system
Password: root
Container Name: <blank>
What doesn't work
I fail to connect using IntelliJ, and therefore the underlying JDBC library. I use the following options:
For Password, I used root again, the JDBC URL is as follows:
jdbc:oracle:thin:#localhost:1521:XE
When I click on Test connection, IntelliJ tries to connect for about a minute, before showing the following error
I did a test on my MacOS
# fire up the database. Hint, use gvenzl images instead. Much faster!
docker run -d -e ORACLE_PWD="root" --name testdb -p 5500:5500 -p 8081:8080 -p 1521:1521 container-registry.oracle.com/database/express:21.3.0-xe
# I have sqlplus installed locally on my MacOS
echo 'select dummy from dual;' | sqlplus -S system/"root"#localhost/XE
D
-
X
echo 'select dummy from dual;' | sqlplus -S system/"root"#localhost:XE
ERROR:
ORA-12545: Connect failed because target host or object does not exist
SP2-0306: Invalid option.
# so, how is JDBC behaving taking the connect string as argument
java -cp .:./ojdbc8-19.6.0.0.jar OracleJDBC "jdbc:oracle:thin:#localhost:1521:XE"
X
java -cp .:./ojdbc8-19.6.0.0.jar OracleJDBC "jdbc:oracle:thin:#localhost:XE"
java.sql.SQLRecoverableException: IO Error: Invalid number format for port number
java -cp .:./ojdbc8-19.6.0.0.jar OracleJDBC "jdbc:oracle:thin:#localhost/XE"
X
Note. Port is not needed, defaults to 1521
cat OracleJDBC.java
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.SQLException;
import java.sql.Statement;
import java.sql.ResultSet;
public class OracleJDBC {
public static void main(String[] argv) {
try {
Class.forName("oracle.jdbc.driver.OracleDriver");
} catch (ClassNotFoundException e) {
System.out.println("Where is your Oracle JDBC Driver?");
e.printStackTrace();
return;
}
Connection connection = null;
String query = "select dummy from dual";
try {
connection = DriverManager.getConnection(argv[0], "system","root");
Statement stmt = connection.createStatement();
ResultSet rows = stmt.executeQuery(query);
while (rows.next()) {
System.out.println(rows.getString("dummy"));
}
} catch (SQLException e) {
e.printStackTrace();
}
}
}
I cannot explain why you get the error. Can it be the JDBC version you are using? I have just proven, your connection should work. That said, you are not supposed to connect using the SID construct (:SID) anymore. You will hit the root-container and not where you are supposed to store your data - in a pluggable database. The express-edition comes with the default pluggable database "XEPDB1".
echo 'select name from v$pdbs;' | sqlplus -S system/"root"#localhost/XE
NAME
------------------------------
PDB$SEED
XEPDB1
This should be your connect string:
echo 'select dummy from dual;' | sqlplus -S system/"root"#localhost/XEPDB1
D
-
X
From here you create your app schema and user so you no longer will use the power-user 'system'.
Best of luck!
I worked perfectly well when I used the same configuration, but with this image instead of the official one.
Thanks to Bjarte Brandt for pointing me to this image.
I am trying to execute SFTP command with proxy option, but it is giving error. Script sample is as below
Start
sftpCommand="sftp -o ConnectTimeout=3 -o "ProxyCommand=/usr/bin/nc --proxy-type http --proxy 192.168.20.98:3128 --proxy-auth pxuser:Password#1 %h %p" -oPort=22 user-sftp2#202.89.99.20"
/usr/bin/expect << EOD
set timeout -1
spawn /bin/sh -c $sftpCommand
expect {
"*?assword:*" {
send "$sftpPassword\r"
}
"*>*" {
send "put $datewise_dir $remoteDir\r"
}
"*>*" {
send "bye\r"
}
}
EOD
End
above script is giving error like
line 6: {--proxy-type}: command not found
spawn /bin/sh -c
/bin/sh: -c: option requires an argument
In your first line defining sftpCommand you use double quotes " inside a string which is itself enclosed in double quotes - this nesting will not work. You can fix this by changing the outer quotes to single ones ', i.e.
sftpCommand='sftp -o ConnectTimeout=3 -o "ProxyCommand=/usr/bin/nc --proxy-type http --proxy 192.168.20.98:3128 --proxy-auth pxuser:Password#1 %h %p" -oPort=22 user-sftp2#202.89.99.20'
I am trying to login into a server 100.18.10.182 and triggering my spark submit job in the server 100.18.10.36 from .182 server in Apache Airflow. I have used BashOperator (a shell script to ssh into 100.18.10.182 server) and for the spark submit job, I have used SparkSubmitOperator as a downstream to BashOperator.
I am able to execute the BashOperator successfully but the SparkOperator fails with:
Cannot execute: Spark submit
I think this is because I am unable to pass the session of my SSH (of .182 server) into the next SparkSubmitOperator or it may be due to some other issue related to --jars or --packages, not sure here.
I was thinking to use xcom_push to push some data from my BashOperator and xcom_pull into the SparkSubmitOperator but not sure how to pass it in a way that my server is logged in and then my SparkSubmitOperator gets triggered from that box itself?
Airflow dag code:
t2 = BashOperator(
task_id='test_bash_operator',
bash_command="/Users/hardikgoel/Downloads/Work/airflow_dir/shell_files/airflow_prod_ssh_script.sh ",
dag=dag)
t2
t3_config = {
'conf': {
"spark.yarn.maxAppAttempts": "1",
"spark.yarn.executor.memoryOverhead": "8"
},
'conn_id': 'spark_default',
'packages': 'com.sparkjobs.SparkJobsApplication',
'jars': '/var/spark/spark-jobs-0.0.1-SNAPSHOT-1/spark-jobs-0.0.1-SNAPSHOT.jar firstJob',
'driver_memory': '1g',
'total_executor_cores': '21',
'executor_cores': 7,
'executor_memory': '48g'
}
t3 = SparkSubmitOperator(
task_id='t3',
**t3_config)
t2 >> t3
Shell Script code:
#!/bin/bash
USERNAME=hardikgoel
HOSTS="100.18.10.182"
SCRIPT="pwd; ls"
ssh -l ${USERNAME} ${HOSTS} "${SCRIPT}"
echo "SSHed successfully"
if [ ${PIPESTATUS[0]} -eq 0 ]; then
echo "successfull"
fi
20201118:18:13:10:008884 gpexpand:master27:gpadmin-[INFO]:-Configuring new segments (primary)
20201118:18:13:10:008884 gpexpand:master27:gpadmin-[INFO]:-{'slave34': '/opt/data/gpdatap3/gpseg17:6002:true:false:19:17::-1:', 'slave33': '/opt/data/gpdatap3/gpseg16:6002:true:false:18:16::-1:', 'slave32': '/opt/data/gpdatap3/gpseg15:6002:true:false:17:15::-1:', 'slave31': '/opt/data/gpdatap3/gpseg14:6002:true:false:16:14::-1:', 'slave28': '/opt/data/gpdatap3/gpseg12:6002:true:false:14:12::-1:', 'slave29': '/opt/data/gpdatap3/gpseg13:6002:true:false:15:13::-1:'}
20201118:18:13:42:008884 gpexpand:master27:gpadmin-[ERROR]:-gpexpand failed: ExecutionError: 'Error Executing Command: ' occurred. Details: 'ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=60 slave33 ". /usr/local/greenplum-db-6.9.0/greenplum_path.sh; env GPSESSID=0000000000 GPERA=None $GPHOME/bin/pg_ctl -D /opt/data/gpdatap3/gpseg16 -l /opt/data/gpdatap3/gpseg16/pg_log/startup.log -w -t 600 -o \" -p 6002 -c gp_role=utility -M \" start 2>&1"' cmd had rc=1 completed=True halted=False
stdout='waiting for server to start.... stopped waiting
pg_ctl: could not start server
Examine the log output.
'
stderr=''
Exiting...
20201118:18:13:42:008884 gpexpand:master27:gpadmin-[ERROR]:-Please run 'gpexpand -r' to rollback to the original state.
20201118:18:13:42:008884 gpexpand:master27:gpadmin-[INFO]:-Shutting down gpexpand...
Any help?
Thanks a lot
guys,i have a problem on using xinetd,the error message is 'xinetd[20126]: execv( /home/fulu/download/mysqlchk_status2.sh ) failed: Exec format error (errno = 8)'
the system operation is : CentOS release 6.2;
i installed the xinetd by the command 'sudo yum install xinetd'
i edited the /etc/services, add my port 6033 for my service named 'mysqlchk'
the service 'mysqlchk' in /etc/xinetd.d/mysqlchk is
service mysqlchk
{
disable = no
flags = REUSE
socket_type = stream
port = 6033
wait = no
user = fulu
server = /home/fulu/download/mysqlchk_status2.sh
log_on_failure += USERID
}
the shell file /home/fulu/download/mysqlchk_status2.sh content is
echo 'test'
6.i can run the command /home/fulu/download/mysqlchk_status2.sh straightly and get the result 'test'
when i telnet 127.0.0.1 6033,i get the output
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
Connection closed by foreign host.
then i tail the log file /var/log/messages,it shows
Apr 22 22:01:47 AY1304111122016 xinetd[20001]: START: mysqlchk pid=20126 from=127.0.0.1
Apr 22 22:01:47 AY1304111122016 xinetd[20126]: execv( /home/fulu/download/mysqlchk_status2.sh ) failed: Exec format error (errno = 8)
Apr 22 22:01:47 AY1304111122016 xinetd[20001]: EXIT: mysqlchk status=0 pid=20126 duration=0(sec)
i don't know why,can anybody help me ?
I'm sorry, after questioning it i suddenly found the answer. If you want the shell to be run in other program you need add '#!/bin/echo' at the first line of the shell file (of course the echo can be changed)