I need to run Apache Airflow in a corporate network. For that I need to set "http_proxy", "https_proxy" and "no_proxy" in any machine I want to use internet.
Right now, the VM that I'm using to run Airflow stores these env. variables in /etc/profile.
I can run Python scripts that make HTTP requests to external websites with ease, when I run them on the terminal, but when I run them inside a DAG, it breaks because it couldn't resolve/access the address.
It seems that Airflow runs scripts in an isolated environment. I am currently using CeleryExecutor.
Firstly, I've accessed all the environment variables with a print(environ). I got this:
environ({'LANG': 'en_US.UTF-8', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin', 'HOME': '/home/airflow', 'LOGNAME': 'airflow', 'USER': 'airflow', 'SHELL': '/bin/bash', 'INVOCATION_ID': '5c777ce3b07748309b972d877a0545ea', 'JOURNAL_STREAM': '9:37430', 'AIRFLOW_CONFIG': '/opt/airflow/airflow.cfg', 'AIRFLOW_HOME': '/opt/airflow', '_MP_FORK_LOGLEVEL_': '20', '_MP_FORK_LOGFILE_': '', '_MP_FORK_LOGFORMAT_': '[%(asctime)s: %(levelname)s/%(processName)s] %(message)s', 'CELERY_LOG_LEVEL': '20', 'CELERY_LOG_FILE': '', 'CELERY_LOG_REDIRECT': '1', 'CELERY_LOG_REDIRECT_LEVEL': 'WARNING', 'AIRFLOW_CTX_DAG_OWNER': 'airflow', 'AIRFLOW_CTX_DAG_ID': 'primeiro-teste', 'AIRFLOW_CTX_TASK_ID': 'extract', 'AIRFLOW_CTX_EXECUTION_DATE': '2022-12-13T16:18:17.185417+00:00', 'AIRFLOW_CTX_DAG_RUN_ID': 'manual__2022-12-13T16:18:17.185417+00:00'})
There is no proxy variables, so the script cannot access outside information.
I've even debugged within a DAG which were the DNS servers, to see if they were correct. The result was positive.
The only way I got the script to work was by getting these environ variables defined before running an HTTP request:
os.environ['HTTP_PROXY'] = os.environ['http_proxy'] = os.environ['HTTPS_PROXY'] = os.environ['https_proxy'] = "PROXY STRING"
I was hoping to find a way to get these variables defined for all DAGs, but when I set them like Tomasz, I can't seem to use them if they don't start with the "AIRFLOW" prefix.
Creating an environment file and putting it in some location is not sufficient. You have to tell Airflow about the location of that file when it starts, however you do that (e.g. systemd).
Airflow gets its environment variables very specifically. When Airflow starts you need to reference the environment file created for Airflow. When you run Airflow using systemd you can specify which EnvironmentFile that you would like Airflow to use, under the [Service] section of the unit file. Environment variables not defined within that file will not be picked up by Airflow. Your unit files may look different to mine but here is mine as an example:
[Unit]
Description=Airflow webserver daemon
After=network.target mysqld.service rabbitmq-server.service
Wants=mysqld.service rabbitmq-server.service
[Service]
EnvironmentFile=/prod/airflow/airflow.env
User=airflow
Group=airflow
Type=simple
ExecStart=/usr/bin/bash -c "source /prod/airflow/airflow_38_venv/bin/activate ; /prod/airflow/airflow_38_venv/bin/airflow webserver -p 7635 --pid /prod/airflow/run/webserver.pid"
Restart=on-failure
RestartSec=5s
PrivateTmp=true
[Install]
WantedBy=multi-user.target
EnvironmentFile can point to any location/filename that the user running Airflow has read access to. The suggested filename and location are /etc/sysconfig/airflow but as you can see mine is different than what is recommended.
Here is what the body of my EnvironmentFile looks like, edited to remove specific details. Again, yours will probably look different.
$ cat /prod/airflow/airflow.env
# This file is the environment file for Airflow. Put this file in /etc/sysconfig/airflow per default
# configuration of the systemd unit files.
#
AIRFLOW_CONFIG=/prod/airflow/airflow.cfg
AIRFLOW_HOME=/prod/airflow
http_proxy=http://something.proxyserver.com:80
https_proxy=http://something.proxyserver.com:80
no_proxy=*.google.com,127.0.0.1
HTTP_PROXY=http://something.proxyserver.com:80
HTTPS_PROXY=http://something.proxyserver.com:80
NO_PROXY=*.google.com,127.0.0.1
I have an opensplice publisher on Ubuntu 20.04 that is started via systemd.
If the publisher starts via systemd then the data is not pubished, but also no errors are reported or present in the opensplice log files.
The publisher works if I run it from a command line or if I stop and restart the service.
The QoS are the same for the publisher and subscriber.
The publisher and subscriber applications are running on different machines.
There are no other participants on the network. All the machines are rebooted and the order of reboot does not change the observed behaviour.
The systemd service is:
[Unit]
Description=Publisher Process
Documentation=
After=network.target
StartLimitIntervalSec=0
[Service]
Type=simple
WorkingDirectory=/opt/publisher/bin
ExecStart=/opt/publisher/bin/publisher.sh
Restart=always
RestartSec=2
[Install]
WantedBy=multi-user.target
The publisher.sh is:
#!/bin/bash
cd /opt/publisher/bin
source bashrc_local
# We just keep running the application (in case of a crash)
while true; do
./publisher
sleep 15
done
I have a work around that feels a little bit naff.
#!/bin/bash
cd /opt/publisher/bin
source bashrc_local
timeout 30 ./remote_processor
killall remote_processor
# We just keep running the application (in case of a crash)
while true; do
./publisher
sleep 15
done
Any ideas on how I can remove my work around?
Edit 16 Sept 22
The issue appears to be systemd start order and dependencies as I have run into the same issue with a program publishing data via UDP which is not using DDS.
Changing the dependencies so the services are started just before the user login does not help.
check your environment variables as systemd will not run with the same environment as your bash console
in particular have you set the OSPL_URI variable to point at the config?
if using the commercial version, OSPL_HOME and ADLINK_LICENSE will also need to be set
Does the PATH variable include your OSPL shared libraries?
These are all setup by running the $OSPL_HOME\release.com script in your bash session
I tend to manually add the required ones to the service file
e.g.
Environment=OSPL_URI=file:///opt/ospl.xml
I'm using Ubuntu 18.04.4 LTS which uses systemd, but the squid package packaged with this version of Ubuntu is configured to start via init.d. It starts and runs via systemctl start squid.service if I start it manually after the system has booted.
However, I'm using a ZFS mount point ("/media") to store the cache data, and during the boot process squid is starting before this mount point is active. Consequently I'm getting the error "Failed to verify one of the swap directories". Full output of systemctl status squid is here
I'd like to tell systemd to wait until after media.mount has completed in the most minimally invasive way possible (e.g. without modifying the /etc/init.d/squid file that is maintained by the package). To that end I created the /etc/systemd/system/squid.service.d/override.conf file like so:
% cat /etc/systemd/system/squid.service.d/override.conf
[Unit]
Wants=network.target network-online.target nss-lookup.target media.mount
After=network.target network-online.target nss-lookup.target media.mount
[Install]
WantedBy=multi-user.target
But squid is still starting too early.
Is what I want to do possible? Or do I have to bite the bullet and define a native /etc/systemd/system/squid.service file and remove the /etc/init.d/squid init script?
I have a strange problem with Ubuntu 16 and a systemd unit file. I have a service which reads a directory from the local filesystem. The directory is read from an environment variable. Now when I start the service manually (as in: in a ssh session), everything works fine. But when I start the service with the unit file from below, the service is unable to open the storage directory. The error I get is: could nog read contents of storage" message="open /srv/services/poddy/storage: no such file or directory.
Now my question is: does systemd kind of "sandbox" the services?
[Unit]
Description=Poddy service
After=network.target
[Service]
Type=simple
User=myusername
Group=myusername
WorkingDirectory=/srv/services/poddy
ExecStart=/srv/services/poddy/poddy
Restart=always
RestartSec=5
StartLimitInterval=60s
StartLimitBurst=3
Environment=PODDY_STORAGE="/srv/services/poddy/storage"
Environment=PODDY_PORT=8085
[Install]
WantedBy=multi-user.target
Well, I solved it myself. It turns out that quoting the value of an environment var in the systemd unit file eventually double-escaped the value.
So, changing this:
Environment=PODDY_STORAGE="/srv/services/poddy/storage"
into:
Environment=PODDY_STORAGE=/srv/services/poddy/storage
solved my problem :).
I am new to systemd service scripts. I am trying to start my application from systemd service scripts. My application is a process that in turn invokes multiple process that includes Qt GUI as one of its child. But the service downt starting up my application.
This is how my service looks like:
[Unit]
Description=/etc/rc.d/rc.local Compatibility
ConditionFileIsExecutable=/etc/rc.d/rc.local
After=network.target
[Service]
Type=forking
ExecStart=/etc/rc.d/rc.local start
SysVStartPriority=99
rc.local script looks like:
#!/bin/bash
export DISPLAY=:0
sleep 5
cd /var/MINC3/apps
./PMonTsk
So when try to run the command "systemctl start rc-local.service", the command executes the script but doesnt invoke my application. If I replace some other QT GUI sample application in the plcae of my application in rc.local, it is working fine. Please help me on sorting this issue.
If you add
[Install]
WantedBy=multi-user.target
I think it will work ;)
I found solution for the above problem. I modified my service in the following way. It works fine after the modification.
[Unit]
Description=/etc/rc.d/rc.local Compatibility
ConditionFileIsExecutable=/etc/rc.d/rc.local
After=network.target
[Service]
Type=forking
ExecStart=/etc/rc.d/rc.local start
ControlGroup=cpu:/
SysVStartPriority=99