I am connecting to a database using a quite standard syntax.
How to change the default timeout value here?
Does it need to be set at the driver or at the jdbc level?
jaydebeapi documentation does not mention that.
Source for jaydebeapi-connection.py
https://community.denodo.com/kb/view/document/How%20to%20connect%20to%20Denodo%20from%20Python%20-%20a%20starter%20for%20Data%20Scientists?category=Northbound+Connections
## script name: jaydebeapi-connection.py
## Importing the main library used to connect to Denodo via JDBC
import jaydebeapi as dbdriver
## Importing the gethostname function from socket to
## put the hostname in the useragent variable
from socket import gethostname
# Connection parameters of the Denodo Server that we are connecting to
denodoserver_name = "denodoserver"
# This is the standard port for jdbc connections
denodoserver_jdbc_port = "9999"
denodoserver_database = "distributed_tpcds"
denodoserver_uid = "tpcds_usr"
denodoserver_pwd = "tpcds_usr"
denododriver_path = "/opt/denodo/8.0/tools/client-drivers/jdbc/denodo-vdp-jdbcdriver.jar"
## Create the useragent as the concatenation of
## the client hostname and the python library used
client_hostname = gethostname()
useragent = "%s-%s" % (dbdriver.__name__,client_hostname)
## Creating a variable with the connection uri. We add here the UserAgent
## so the query can be better identified on the server. To append parameters you
## can use the syntax <param_name>=<param_value> and separate them with '&'.
## The full list of accepted parameters is available here
## https://community.denodo.com/docs/html/browse/7.0/vdp/developer/
## access_through_jdbc/parameters_of_the_jdbc_connection_url/
## parameters_of_the_jdbc_connection_url
conn_uri = "jdbc:vdb://%s:%s/%s?userAgent=%s" % (denodoserver_name,
denodoserver_jdbc_port, denodoserver_database,
useragent)
cnxn = dbdriver.connect( "com.denodo.vdp.jdbc.Driver",
conn_uri,
driver_args = {"user": denodoserver_uid,
"password": denodoserver_pwd},
jars = denododriver_path
)
## Query to be sent to the Denodo VDP Server
query = "select * from bv_store_returns"
## Define a cursor and execute the results
cur = cnxn.cursor()
cur.execute(query)
## Finally fetch the results. `results` is a list of tuples,
## If you don't want to load all the records in memory,
## you may want to use cur.fetchone() or cur.fetchmany()
results = cur.fetchall()
# >> len(results)
# 287514
# >> type(results)
# list
# >> type(results[0])
# tuple
# >> results[0]
# (2451794, 40096, 1, 7157, 910283, 6421, 37312, ...)
You can actually do it via changing the JDBC driver configuration (adding the driver parameter 'queryTimeout') or via the CONTEXT clause of the query itself (https://community.denodo.com/docs/html/browse/latest/en/vdp/vql/queries_select_statement/context_clause/context_clause)
Related
I have a basic Gatling script on EC2 instance from which I want to push the results into an Influx database instance. I can successfully run a Gatling script and Influx is also running.
My Gatling configuration is the following:
data {
writers = [console, graphite] # The list of DataWriters to which Gatling write simulation data (currently supported : console, file, graphite)
console {
#light = false # When set to true, displays a light version without detailed request stats
#writePeriod = 5 # Write interval, in seconds
}
file {
#bufferSize = 8192 # FileDataWriter's internal data buffer size, in bytes
}
leak {
#noActivityTimeout = 30 # Period, in seconds, for which Gatling may have no activity before considering a leak may be happening
}
graphite {
light = false # only send the all* stats
host = "ec2-35-181-26-79.eu-west-3.compute.amazonaws.com" # The host where the Carbon server is located
port = 2003 # The port to which the Carbon server listens to (2003 is default for plaintext, 2004 is default for pickle)
protocol = "tcp" # The protocol used to send data to Carbon (currently supported : "tcp", "udp")
rootPathPrefix = "gatling" # The common prefix of all metrics sent to Graphite
bufferSize = 8192 # Internal data buffer size, in bytes
writePeriod = 1 # Write period, in seconds
}
And for Influx, I've setup a Telegraf with the following configuration
[[outputs.influxdb_v2]]
## The URLs of the InfluxDB cluster nodes.
##
## Multiple URLs can be specified for a single cluster, only ONE of the
## urls will be written to each interval.
## urls exp: http://127.0.0.1:8086
urls = ["http://ec2-35-181-26-79.eu-west-3.compute.amazonaws.com:8086"]
## Token for authentication.
token = "$INFLUX_TOKEN"
## Organization is the name of the organization you wish to write to; must exist.
organization = "Test"
## Destination bucket to write into.
bucket = "Test"
[[inputs.socket_listener]]
## URL to listen on
service_address = "tcp://:2003"
data_format = "graphite"
## Content encoding for message payloads, can be set to "gzip" to or
## "identity" to apply no encoding.
# content_encoding = "identity"
templates = [
"gatling.*.*.*.* measurement.simulation.request.status.field",
"gatling.*.users.*.* measurement.simulation.measurement.request.field"
]
With both Telegraf (with this configuration) and Influx running, I don't see any data pushed into the 'Test' bucket. Moreover I don't get any errors that could help me debugging.
Any help would be much appreciated. Thanks.
Everyone, I am using Airflow(v.1.10.12) for one of the project to schedule the Jobs daily. Wanted to execute multiple DAG’s(14) in parallel. Have updated the concurrency parameters in cfg file. However, have observed the behavior that: Scheduler starts the execution of tasks, assigns the tasks to queue but after certain amount of time it pauses and restart after 5 minutes. Due to this behavior the time taken by all Dag is huge and also some of tasks are getting failed. Can someone help me with the understanding why the Scheduler gets halt and if we should modify some parameter or shift to some other Airflow version?
Below is the configuration file
[core]
dags_folder = /home/airflow/dags
base_log_folder = /home/airflow/logs
remote_logging = False
remote_log_conn_id =
remote_base_log_folder =
encrypt_s3_logs = False
logging_level = INFO
fab_logging_level = WARN
logging_config_class =
colored_console_log = True
colored_log_format = [%%(blue)s%%(asctime)s%%(reset)s] {%%(blue)s%%(filename)s:%%(reset)s%%(lineno)d} %%(log_color)s%%(levelname)s%%(reset)s - %%(log_color)s%%(message)s%%(reset)s
colored_formatter_class = airflow.utils.log.colored_log.CustomTTYColoredFormatter
log_format = [%%(asctime)s] {%%(filename)s:%%(lineno)d} %%(levelname)s - %%(message)s
simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s
log_filename_template = {{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log
log_processor_filename_template = {{ filename }}.log
dag_processor_manager_log_location = /home/airflow/logs/dag_processor_manager/dag_processor_manager.log
hostname_callable = socket:getfqdn
default_timezone = utc
executor = CeleryExecutor
sql_alchemy_conn = postgresql+psycopg2://devairflow:airflow#localhost:5432/pcfdb
sql_engine_encoding = utf-8
sql_alchemy_pool_enabled = True
sql_alchemy_pool_size = 0
sql_alchemy_max_overflow = -1
# The SqlAlchemy pool recycle is the number of seconds a connection
# can be idle in the pool before it is invalidated. This config does
# not apply to sqlite. If the number of DB connections is ever exceeded,
# a lower config value will allow the system to recover faster.
sql_alchemy_pool_recycle = 1800
sql_alchemy_pool_pre_ping = True
sql_alchemy_schema =
parallelism = 50
dag_concurrency = 50
dags_are_paused_at_creation = True
max_active_runs_per_dag = 50
load_examples = False
load_default_connections = True
plugins_folder = /home/airflow/plugins
fernet_key = M4dpP6f2Hd5p3N--CxtIoUo9XaSDifA42MPLs1UR7-g=
donot_pickle = False
dagbag_import_timeout = 41460
dag_file_processor_timeout = 60
task_runner = StandardTaskRunner
default_impersonation =
security =
secure_mode = False
unit_test_mode = False
enable_xcom_pickling = True
killed_task_cleanup_time = 60
dag_run_conf_overrides_params = False
worker_precheck = False
dag_discovery_safe_mode = True
default_task_retries = 0
store_serialized_dags = False
min_serialized_dag_update_interval = 30
max_num_rendered_ti_fields_per_task = 100
# On each dagrun check against defined SLAs
check_slas = True
[secrets]
# Full class name of secrets backend to enable (will precede env vars and metastore in search path)
# Example: backend = airflow.contrib.secrets.aws_systems_manager.SystemsManagerParameterStoreBackend
backend =
# The backend_kwargs param is loaded into a dictionary and passed to __init__ of secrets backend class.
# See documentation for the secrets backend you are using. JSON is expected.
# Example for AWS Systems Manager ParameterStore:
# ``{"connections_prefix": "/airflow/connections", "profile_name": "default"}``
backend_kwargs =
[cli]
# In what way should the cli access the API. The LocalClient will use the
# database directly, while the json_client will use the api running on the
# webserver
api_client = airflow.api.client.local_client
# If you set web_server_url_prefix, do NOT forget to append it here, ex:
# ``endpoint_url = http://localhost:8080/myroot``
# So api will look like: ``http://localhost:8080/myroot/api/experimental/...``
endpoint_url = http://localhost:8080
[debug]
# Used only with DebugExecutor. If set to True DAG will fail with first
# failed task. Helpful for debugging purposes.
fail_fast = False
[api]
# How to authenticate users of the API. See
# https://airflow.apache.org/docs/stable/security.html for possible values.
# ("airflow.api.auth.backend.default" allows all requests for historic reasons)
auth_backend = airflow.api.auth.backend.deny_all
[lineage]
# what lineage backend to use
backend =
[atlas]
sasl_enabled = False
host =
port = 21000
username =
password =
[operators]
# The default owner assigned to each new operator, unless
# provided explicitly or passed via ``default_args``
default_owner = airflow
default_cpus = 1
default_ram = 1024
default_disk = 1024
default_gpus = 0
[hive]
# Default mapreduce queue for HiveOperator tasks
default_hive_mapred_queue =
[webserver]
# The base url of your website as airflow cannot guess what domain or
# cname you are using. This is used in automated emails that
# airflow sends to point links to the right web server
base_url = http://localhost:8080
# Default timezone to display all dates in the RBAC UI, can be UTC, system, or
# any IANA timezone string (e.g. Europe/Amsterdam). If left empty the
# default value of core/default_timezone will be used
# Example: default_ui_timezone = America/New_York
default_ui_timezone = UTC
# The ip specified when starting the web server
web_server_host = 0.0.0.0
# The port on which to run the web server
web_server_port = 8080
# Paths to the SSL certificate and key for the web server. When both are
# provided SSL will be enabled. This does not change the web server port.
web_server_ssl_cert =
# Paths to the SSL certificate and key for the web server. When both are
# provided SSL will be enabled. This does not change the web server port.
web_server_ssl_key =
# Number of seconds the webserver waits before killing gunicorn master that doesn't respond
web_server_master_timeout = 41460
# Number of seconds the gunicorn webserver waits before timing out on a worker
web_server_worker_timeout = 41460
# Number of workers to refresh at a time. When set to 0, worker refresh is
# disabled. When nonzero, airflow periodically refreshes webserver workers by
# bringing up new ones and killing old ones.
worker_refresh_batch_size = 1
# Number of seconds to wait before refreshing a batch of workers.
worker_refresh_interval = 30
# If set to True, Airflow will track files in plugins_folder directory. When it detects changes,
# then reload the gunicorn.
reload_on_plugin_change = False
# Secret key used to run your flask app
# It should be as random as possible
secret_key = temporary_key
# Number of workers to run the Gunicorn web server
workers = 4
# The worker class gunicorn should use. Choices include
# sync (default), eventlet, gevent
worker_class = sync
# Log files for the gunicorn webserver. '-' means log to stderr.
access_logfile = -
# Log files for the gunicorn webserver. '-' means log to stderr.
error_logfile = -
# Expose the configuration file in the web server
expose_config = True
# Expose hostname in the web server
expose_hostname = True
# Expose stacktrace in the web server
expose_stacktrace = True
# Set to true to turn on authentication:
# https://airflow.apache.org/security.html#web-authentication
authenticate = False
# Filter the list of dags by owner name (requires authentication to be enabled)
filter_by_owner = False
# Filtering mode. Choices include user (default) and ldapgroup.
# Ldap group filtering requires using the ldap backend
#
# Note that the ldap server needs the "memberOf" overlay to be set up
# in order to user the ldapgroup mode.
owner_mode = user
# Default DAG view. Valid values are:
# tree, graph, duration, gantt, landing_times
dag_default_view = tree
# "Default DAG orientation. Valid values are:"
# LR (Left->Right), TB (Top->Bottom), RL (Right->Left), BT (Bottom->Top)
dag_orientation = LR
# Puts the webserver in demonstration mode; blurs the names of Operators for
# privacy.
demo_mode = False
# The amount of time (in secs) webserver will wait for initial handshake
# while fetching logs from other worker machine
log_fetch_timeout_sec = 5
# Time interval (in secs) to wait before next log fetching.
log_fetch_delay_sec = 2
# Distance away from page bottom to enable auto tailing.
log_auto_tailing_offset = 30
# Animation speed for auto tailing log display.
log_animation_speed = 1000
# By default, the webserver shows paused DAGs. Flip this to hide paused
# DAGs by default
hide_paused_dags_by_default = False
# Consistent page size across all listing views in the UI
page_size = 100
# Use FAB-based webserver with RBAC feature
rbac = False
# Define the color of navigation bar
navbar_color = #007A87
# Default dagrun to show in UI
default_dag_run_display_number = 25
# Enable werkzeug ``ProxyFix`` middleware for reverse proxy
enable_proxy_fix = False
# Number of values to trust for ``X-Forwarded-For``.
# More info: https://werkzeug.palletsprojects.com/en/0.16.x/middleware/proxy_fix/
proxy_fix_x_for = 1
# Number of values to trust for ``X-Forwarded-Proto``
proxy_fix_x_proto = 1
# Number of values to trust for ``X-Forwarded-Host``
proxy_fix_x_host = 1
# Number of values to trust for ``X-Forwarded-Port``
proxy_fix_x_port = 1
# Number of values to trust for ``X-Forwarded-Prefix``
proxy_fix_x_prefix = 1
# Set secure flag on session cookie
cookie_secure = False
# Set samesite policy on session cookie
cookie_samesite =
# Default setting for wrap toggle on DAG code and TI log views.
default_wrap = False
# Allow the UI to be rendered in a frame
x_frame_enabled = True
# Send anonymous user activity to your analytics tool
# choose from google_analytics, segment, or metarouter
# analytics_tool =
# Unique ID of your account in the analytics tool
# analytics_id =
# Update FAB permissions and sync security manager roles
# on webserver startup
update_fab_perms = True
# Minutes of non-activity before logged out from UI
# 0 means never get forcibly logged out
force_log_out_after = 0
# The UI cookie lifetime in days
session_lifetime_days = 30
[email]
email_backend = airflow.utils.email.send_email_smtp
[smtp]
# If you want airflow to send emails on retries, failure, and you want to use
# the airflow.utils.email.send_email_smtp function, you have to configure an
# smtp server here
#smtp_host = localhost
# SMTP Address
# 192.168.152.213
# SMTP Port
# 25
# User Name
# etf#csopasset.com
smtp_host = *.*.*.*
smtp_starttls = True
smtp_ssl = False
# smtp_user = etf#csopasset.com
# smtp_password = etfGen2013
smtp_port = 25
smtp_mail_from = etf#***.com
[sentry]
# Sentry (https://docs.sentry.io) integration
sentry_dsn =
[celery]
# This section only applies if you are using the CeleryExecutor in
# ``[core]`` section above
# The app name that will be used by celery
celery_app_name = airflow.executors.celery_executor
# The concurrency that will be used when starting workers with the
# ``airflow celery worker`` command. This defines the number of task instances that
# a worker will take, so size up your workers based on the resources on
# your worker box and the nature of your tasks
worker_concurrency = 50
# The maximum and minimum concurrency that will be used when starting workers with the
# ``airflow celery worker`` command (always keep minimum processes, but grow
# to maximum if necessary). Note the value should be max_concurrency,min_concurrency
# Pick these numbers based on resources on worker box and the nature of the task.
# If autoscale option is available, worker_concurrency will be ignored.
# http://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html#cmdoption-celery-worker-autoscale
# Example: worker_autoscale = 16,12
# worker_autoscale =
# When you start an airflow worker, airflow starts a tiny web server
# subprocess to serve the workers local log files to the airflow main
# web server, who then builds pages and sends them to users. This defines
# the port on which the logs are served. It needs to be unused, and open
# visible from the main web server to connect into the workers.
worker_log_server_port = 8793
# The Celery broker URL. Celery supports RabbitMQ, Redis and experimentally
# a sqlalchemy database. Refer to the Celery documentation for more
# information.
# http://docs.celeryproject.org/en/latest/userguide/configuration.html#broker-settings
#roker_url = sqla+mysql://airflow:airflow#localhost:3306/airflow
broker_url = amqp://guest:guest#localhost:5672//
# The Celery result_backend. When a job finishes, it needs to update the
# metadata of the job. Therefore it will post a message on a message bus,
# or insert it into a database (depending of the backend)
# This status is used by the scheduler to update the state of the task
# The use of a database is highly recommended
# http://docs.celeryproject.org/en/latest/userguide/configuration.html#task-result-backend-settings
#result_backend = db+mysql://airflow:airflow#localhost:3306/airflow
result_backend = db+postgresql+psycopg2://devairflow:airflow#localhost:5432/pcfdb
# Celery Flower is a sweet UI for Celery. Airflow has a shortcut to start
# it ``airflow flower``. This defines the IP that Celery Flower runs on
flower_host = 0.0.0.0
# The root URL for Flower
# Example: flower_url_prefix = /flower
flower_url_prefix =
# This defines the port that Celery Flower runs on
flower_port = 5555
# Securing Flower with Basic Authentication
# Accepts user:password pairs separated by a comma
# Example: flower_basic_auth = user1:password1,user2:password2
flower_basic_auth =
# Default queue that tasks get assigned to and that worker listen on.
default_queue = default
# How many processes CeleryExecutor uses to sync task state.
# 0 means to use max(1, number of cores - 1) processes.
sync_parallelism = 0
# Import path for celery configuration options
celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
# In case of using SSL
ssl_active = False
ssl_key =
ssl_cert =
ssl_cacert =
# Celery Pool implementation.
# Choices include: prefork (default), eventlet, gevent or solo.
# See:
# https://docs.celeryproject.org/en/latest/userguide/workers.html#concurrency
# https://docs.celeryproject.org/en/latest/userguide/concurrency/eventlet.html
pool = prefork
# The number of seconds to wait before timing out ``send_task_to_executor`` or
# ``fetch_celery_task_state`` operations.
operation_timeout = 50
[celery_broker_transport_options]
# This section is for specifying options which can be passed to the
# underlying celery broker transport. See:
# http://docs.celeryproject.org/en/latest/userguide/configuration.html#std:setting-broker_transport_options
# The visibility timeout defines the number of seconds to wait for the worker
# to acknowledge the task before the message is redelivered to another worker.
# Make sure to increase the visibility timeout to match the time of the longest
# ETA you're planning to use.
# visibility_timeout is only supported for Redis and SQS celery brokers.
# See:
# http://docs.celeryproject.org/en/master/userguide/configuration.html#std:setting-broker_transport_options
# Example: visibility_timeout = 21600
# visibility_timeout =
[dask]
# This section only applies if you are using the DaskExecutor in
# [core] section above
# The IP address and port of the Dask cluster's scheduler.
cluster_address = 127.0.0.1:8786
# TLS/ SSL settings to access a secured Dask scheduler.
tls_ca =
tls_cert =
tls_key =
[scheduler]
# Task instances listen for external kill signal (when you clear tasks
# from the CLI or the UI), this defines the frequency at which they should
# listen (in seconds).
job_heartbeat_sec = 5
# The scheduler constantly tries to trigger new tasks (look at the
# scheduler section in the docs for more information). This defines
# how often the scheduler should run (in seconds).
scheduler_heartbeat_sec = 5
# After how much time should the scheduler terminate in seconds
# -1 indicates to run continuously (see also num_runs)
run_duration = -1
# The number of times to try to schedule each DAG file
# -1 indicates unlimited number
num_runs = -1
# The number of seconds to wait between consecutive DAG file processing
processor_poll_interval = 1
# after how much time (seconds) a new DAGs should be picked up from the filesystem
min_file_process_interval = 0
# How often (in seconds) to scan the DAGs directory for new files. Default to 5 minutes.
dag_dir_list_interval = 300
# How often should stats be printed to the logs. Setting to 0 will disable printing stats
print_stats_interval = 30
# If the last scheduler heartbeat happened more than scheduler_health_check_threshold
# ago (in seconds), scheduler is considered unhealthy.
# This is used by the health check in the "/health" endpoint
scheduler_health_check_threshold = 30
child_process_log_directory = /home/airflow/logs/scheduler
# Local task jobs periodically heartbeat to the DB. If the job has
# not heartbeat in this many seconds, the scheduler will mark the
# associated task instance as failed and will re-schedule the task.
scheduler_zombie_task_threshold = 1800
# Turn off scheduler catchup by setting this to False.
# Default behavior is unchanged and
# Command Line Backfills still work, but the scheduler
# will not do scheduler catchup if this is False,
# however it can be set on a per DAG basis in the
# DAG definition (catchup)
catchup_by_default = True
# This changes the batch size of queries in the scheduling main loop.
# If this is too high, SQL query performance may be impacted by one
# or more of the following:
# - reversion to full table scan
# - complexity of query predicate
# - excessive locking
# Additionally, you may hit the maximum allowable query length for your db.
# Set this to 0 for no limit (not advised)
max_tis_per_query = 512
# Statsd (https://github.com/etsy/statsd) integration settings
statsd_on = False
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
# If you want to avoid send all the available metrics to StatsD,
# you can configure an allow list of prefixes to send only the metrics that
# start with the elements of the list (e.g: scheduler,executor,dagrun)
statsd_allow_list =
# The scheduler can run multiple threads in parallel to schedule dags.
# This defines how many threads will run.
max_threads = 2
authenticate = False
# Turn off scheduler use of cron intervals by setting this to False.
# DAGs submitted manually in the web UI or with trigger_dag will still run.
use_job_schedule = True
# Allow externally triggered DagRuns for Execution Dates in the future
# Only has effect if schedule_interval is set to None in DAG
allow_trigger_in_future = False
[ldap]
# set this to ldaps://<your.ldap.server>:<port>
uri =
user_filter = objectClass=*
user_name_attr = uid
group_member_attr = memberOf
superuser_filter =
data_profiler_filter =
bind_user = cn=Manager,dc=example,dc=com
bind_password = insecure
basedn = dc=example,dc=com
cacert = /etc/ca/ldap_ca.crt
search_scope = LEVEL
# This setting allows the use of LDAP servers that either return a
# broken schema, or do not return a schema.
ignore_malformed_schema = False
[mesos]
# Mesos master address which MesosExecutor will connect to.
master = localhost:5050
# The framework name which Airflow scheduler will register itself as on mesos
framework_name = Airflow
# Number of cpu cores required for running one task instance using
# 'airflow run <dag_id> <task_id> <execution_date> --local -p <pickle_id>'
# command on a mesos slave
task_cpu = 1
# Memory in MB required for running one task instance using
# 'airflow run <dag_id> <task_id> <execution_date> --local -p <pickle_id>'
# command on a mesos slave
task_memory = 256
# Enable framework checkpointing for mesos
# See http://mesos.apache.org/documentation/latest/slave-recovery/
checkpoint = False
# Failover timeout in milliseconds.
# When checkpointing is enabled and this option is set, Mesos waits
# until the configured timeout for
# the MesosExecutor framework to re-register after a failover. Mesos
# shuts down running tasks if the
# MesosExecutor framework fails to re-register within this timeframe.
# Example: failover_timeout = 604800
# failover_timeout =
# Enable framework authentication for mesos
# See http://mesos.apache.org/documentation/latest/configuration/
authenticate = False
# Mesos credentials, if authentication is enabled
# Example: default_principal = admin
# default_principal =
# Example: default_secret = admin
# default_secret =
# Optional Docker Image to run on slave before running the command
# This image should be accessible from mesos slave i.e mesos slave
# should be able to pull this docker image before executing the command.
# Example: docker_image_slave = puckel/docker-airflow
# docker_image_slave =
[kerberos]
ccache = /tmp/airflow_krb5_ccache
# gets augmented with fqdn
principal = airflow
reinit_frequency = 3600
kinit_path = kinit
keytab = airflow.keytab
[github_enterprise]
api_rev = v3
[admin]
# UI to hide sensitive variable fields when set to True
hide_sensitive_variable_fields = True
[elasticsearch]
# Elasticsearch host
host =
# Format of the log_id, which is used to query for a given tasks logs
log_id_template = {dag_id}-{task_id}-{execution_date}-{try_number}
# Used to mark the end of a log stream for a task
end_of_log_mark = end_of_log
# Qualified URL for an elasticsearch frontend (like Kibana) with a template argument for log_id
# Code will construct log_id using the log_id template from the argument above.
# NOTE: The code will prefix the https:// automatically, don't include that here.
frontend =
# Write the task logs to the stdout of the worker, rather than the default files
write_stdout = False
# Instead of the default log formatter, write the log lines as JSON
json_format = False
# Log fields to also attach to the json output, if enabled
json_fields = asctime, filename, lineno, levelname, message
[elasticsearch_configs]
use_ssl = False
verify_certs = True
Below are the Worker logs:
Worker logs File
Airflow 2.0 has massive scheduler improvements. I you should upgrade to 2.0 and enjoy the new scheduler.
If the last scheduler heartbeat happened more than scheduler_health_check_threshold ago (in seconds), scheduler is considered unhealthy. This is used by the health check in the “/health” endpoint
Reference: Airflow Documentation
Make sure scheduler health check thresh hold is higher than scheduler heartbeat sec.
scheduler_health_check_threshold = 300
scheduler_heartbeat_sec = 60
We are always getting error:
Caused by: net.opentsdb.uid.NoSuchUniqueName: No such name for 'metrics': 'test' at net.opentsdb.uid.UniqueId$1GetIdCB.call(UniqueId.java:450) ~[tsdb-2.4.0.jar:] at net.opentsdb.uid.UniqueId$1GetIdCB.call(UniqueId.java:447) ~[tsdb-2.4.0.jar:] ... 34 common frames omitted
ERROR [AsyncHBase I/O Worker #13] UniqueId: Failed attempt #1 to assign an UID for metrics:test at step #2
org.hbase.async.RemoteException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.shaded.protobufotobufUtil$ClassLoaderHolder
I saw that usual error on Web is if missing some of three parameters:
tsd.core.auto_create_metrics = true
tsd.core.auto_create_tagks = true
tsd.core.auto_create_tagvs = true
We are sending data to the Open TSDB.
echo "put test 1548838714 1 tag1=1" | nc 192.168.150.101 4243
Also we notice sometimes error while trying to execute echo command (if OpenTSDB is being run using the build/tsdb tsd and not through the /etc/init.d/opentsdb (by using the command service opentsdb start ):
This is configuration file:
# --------- NETWORK ----------
# The TCP port TSD should use for communications
# *** REQUIRED ***
tsd.network.port = 4243
# The IPv4 network address to bind to, defaults to all addresses
# tsd.network.bind = 0.0.0.0
# Enables Nagel's algorithm to reduce the number of packets sent over the
# network, default is True
#tsd.network.tcpnodelay = true
# Determines whether or not to send keepalive packets to peers, default
# is True
#tsd.network.keepalive = true
# Determines if the same socket should be used for new connections, default
# is True
#tsd.network.reuseaddress = true
# Number of worker threads dedicated to Netty, defaults to # of CPUs * 2
#tsd.network.worker_threads = 8
# Whether or not to use NIO or tradditional blocking IO, defaults to True
#tsd.network.async_io = true
# ----------- HTTP -----------
# The location of static files for the HTTP GUI interface.
# *** REQUIRED ***
tsd.http.staticroot = /opt/opentsdb-2.4.0/build/staticroot/
# Where TSD should write it's cache files to
# *** REQUIRED ***
tsd.http.cachedir = /opt/opentsdb-2.4.0/build/CACHE
# --------- CORE ----------
# Whether or not to automatically create UIDs for new metric types, default
# is False
tsd.core.auto_create_metrics = true
# --------- STORAGE ----------
# Whether or not to enable data compaction in HBase, default is True
#tsd.storage.enable_compaction = true
# How often, in milliseconds, to flush the data point queue to storage,
# default is 1,000
# tsd.storage.flush_interval = 1000
# Name of the HBase table where data points are stored, default is "tsdb"
tsd.storage.hbase.data_table = tsdb
# Name of the HBase table where UID information is stored, default is "tsdb-uid"
tsd.storage.hbase.uid_table = tsdb-uid
# Path under which the znode for the -ROOT- region is located, default is "/hbase"
tsd.storage.hbase.zk_basedir = /hbase-unsecure
# A comma separated list of Zookeeper hosts to connect to, with or without
# port specifiers, default is "localhost"
#tsd.storage.hbase.zk_quorum = localhost
tsd.storage.hbase.zk_quorum = namenode1.local,namenode2.local
tsd.http.request.enable_chunked = true
tsd.http.request.max_chunk = 16000
tsd.storage.fix_duplicates = true
tsd.storage.max_tags = 45
tsd.storage.uid.width.metric = 4
tsd.storage.uid.width.tagk = 4
tsd.storage.uid.width.tagv = 4
tsd.core.uid.random_metrics = true
tsd.core.auto_create_tagks = true
tsd.core.auto_create_tagvs = true
We are trying to build an incoming request validation platform using HAProxy+Lua.
Our use-case is to create a LUA scripts that will essentially make a socket call to a Validation API, and based on the response
from Validation API we want to redirect the request to a backend API, and if the validation fails we would want to return
the request right from the LUA script. For example, for 200 response we would want to redirect the request to backend api, and for 404 we would want
to return the request. From the documentation, I understand that there are various default functions available
with Lua-Haproxy integration.
core.register_action() --> I'm using this. Take TXN as input
core.register_converters() --> Essentially used for string manipulations.
core.register_fetches() --> Takes TXN as input and returns string; Mainly used for representing dynamic backend profiles in haproxy config
core.register_init() --> Used for initialization
core.register_service() --> You have to return the response mandatorily while using this function, which doesn't satisfy our requirements
core.register_task() --> For using normal functions. No mandatory input class. TXN is required to fetch header details from request
I have tried all of the functions from above list, I understand that core.register_service is basically to return a response from the
Lua script. However, what is problematic is, we must send the response from the LUA script and it will not redirect the request to BACKEND.
Currently, I am using core.register_action to interrupt the requests, but I'm not able to return the request using this function. Here's
what my code looks like:
local http_socket = require("socket.http")
local pretty_print = require("pl.pretty")
function add_http_request_header(txn, name, value)
local headerName = name
local headerValue = value
txn.http:req_add_header(headerName, headerValue)
end
function call_validation_api()
local request, code, header = http_socket.request {
method = "GET", -- Validation API Method
url = "http://www.google.com/" -- Validation API URL
}
-- Using core.log; Print in some cases is a blocking operation http://www.arpalert.org/haproxy-lua.html#h203
core.Info( "Validation API Response Code: " .. code )
pretty_print.dump( header )
return code
end
function failure_response(txn)
local response = "Validation Failed"
core.Info(response)
txn.res:send(response)
-- txn:close()
end
core.register_action("validation_action", { "http-req", "http-res" }, function(txn)
local validation_api_code = call_validation_api()
if validation_api_code == 200 then
core.Info("Validation Successful")
add_http_request_header(txn, "test-header", "abcdefg")
pretty_print.dump( txn.http:req_get_headers() )
else
failure_response(txn) --->>> **HERE I WANT TO RETURN THE RESPONSE**
end
end)
Following is the configuration file entry:
frontend http-in
bind :8000
mode http
http-request lua.validation_action
#Capturing header of the incoming request
capture request header test-header len 64
#use_backend %[lua.fetch_req_params]
default_backend app
backend app
balance roundrobin
server app1 127.0.0.1:9999 check
Any help is much appreciated in achieving this functionality. Also, I understand that SOCKET call from Lua script is a blocking call, which is opposite to HAProxy's default nature of keep-alive connection. Please feel free to suggest any other utility to achieve this functionality, if you have already used it.
Ok I have figured out the answer to this question:
I created 2 backends for success and failure of requests, and based on the response I am returning 2 different strings. In "failure_backend", I have called a different service, which essentially is a core.register_service and can return the response. I'm pasting code for both the configuration file and lua script
HAProxy conf file:
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
maxconn 4000
user haproxy
group haproxy
daemon
#lua file load
lua-load /home/aman/coding/haproxy/http_header.lua
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
retries 3
timeout http-request 90s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend http-in
bind :8000
mode http
use_backend %[lua.validation_fetch]
default_backend failure_backend
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend success_backend
balance roundrobin
server app1 172.23.12.94:9999 check
backend failure_backend
http-request use-service lua.failure_service
# For displaying HAProxy statistics.
frontend stats
bind :8888
default_backend stats
backend stats
stats enable
stats hide-version
stats realm Haproxy Statistics
stats uri /haproxy/stats
stats auth aman:rjil#123
Lua script:
local http_socket = require("socket.http")
local pretty_print = require("pl.pretty")
function add_http_request_header(txn, name, value)
local headerName = name
local headerValue = value
txn.http:req_add_header(headerName, headerValue)
end
function call_validation_api()
local request, code, header = http_socket.request {
method = "GET", -- Validation API Method
url = "http://www.google.com/" -- Validation API URL
}
-- Using core.log; Print in some cases is a blocking operation http://www.arpalert.org/haproxy-lua.html#h203
core.Info( "Validation API Response Code: " .. code )
pretty_print.dump( header )
return code
end
function failure_response(txn)
local response = "Validation Failed"
core.Info(response)
return "failure_backend"
end
-- Decides back-end based on Success and Failure received from validation API
core.register_fetches("validation_fetch", function(txn)
local validation_api_code = call_validation_api()
if validation_api_code == 200 then
core.Info("Validation Successful")
add_http_request_header(txn, "test_header", "abcdefg")
pretty_print.dump( txn.http:req_get_headers() )
return "success_backend"
else
failure_response(txn)
end
end)
-- Failure service
core.register_service("failure_service", "http", function(applet)
local response = "Validation Failed"
core.Info(response)
applet:set_status(400)
applet:add_header("content-length", string.len(response))
applet:add_header("content-type", "text/plain")
applet:start_response()
applet:send(response)
end)
Let's say I'm using Terraform to provision two machines inside AWS:
An EC2 Machine running NodeJS
An RDS instance
How does the NodeJS code obtain the address of the RDS instance?
You've got a couple of options here. The simplest one is to create a CNAME record in Route53 for the database and then always point to that CNAME in your application.
A basic example would look something like this:
resource "aws_db_instance" "mydb" {
allocated_storage = 10
engine = "mysql"
engine_version = "5.6.17"
instance_class = "db.t2.micro"
name = "mydb"
username = "foo"
password = "bar"
db_subnet_group_name = "my_database_subnet_group"
parameter_group_name = "default.mysql5.6"
}
resource "aws_route53_record" "database" {
zone_id = "${aws_route53_zone.primary.zone_id}"
name = "database.example.com"
type = "CNAME"
ttl = "300"
records = ["${aws_db_instance.default.endpoint}"]
}
Alternative options include taking the endpoint output from the aws_db_instance and passing that into a user data script when creating the instance or passing it to Consul and using Consul Template to control the config that your application uses.
You may try Sparrowform - a lightweight provision tool for Terraform based instances, it's capable to make an inventory of Terraform resources and provision related hosts, passing all the necessary data:
$ terrafrom apply # bootstrap infrastructure
$ cat sparrowfile # this scenario
# fetches DB address from terraform cache
# and populate configuration file
# at server with node js code:
#!/usr/bin/env perl6
use Sparrowform;
$ sparrowfrom --ssh_private_key=~/.ssh/aws.pem --ssh_user=ec2 # run provision tool
my $rdb-adress;
for tf-resources() -> $r {
my $r-id = $r[0]; # resource id
if ( $r-id 'aws_db_instance.mydb') {
my $r-data = $r[1];
$rdb-address = $r-data<address>;
last;
}
}
# For instance, we can
# Install configuration file
# Next chunk of code will be applied to
# The server with node-js code:
template-create '/path/to/config/app.conf', %(
source => ( slurp 'app.conf.tmpl' ),
variables => %(
rdb-address => $rdb-address
),
);
# sparrowform --ssh_private_key=~/.ssh/aws.pem --ssh_user=ec2 # run provisioning
PS. disclosure - I am the tool author