Celery beat (Redbeat) on heroku : logs not showing - heroku

I have a Django app with a periodic task running on Celery.
Procfile
worker: celery -A myproject worker --loglevel=INFO
beat : celery -A myproject beat --scheduler redbeat.RedBeatScheduler
I've tried with both RedBeatScheduler and django_celery_beat.schedulers:DatabaseScheduler, same result.
settings.py
...
CELERY_BROKER_URL = os.environ.get("REDIS_URL") or 'redis://localhost:6379/0'
CELERY_BACKEND_URL = os.environ.get("REDIS_URL") or 'redis://localhost:6379/1'
CELERY_RESULT_BACKEND = os.environ.get("REDIS_URL") or 'redis://localhost:6379/0'
CELERY_REDBEAT_REDIS_URL = os.environ.get("REDIS_URL") or "redis://localhost:6379/1"
celery.py
...
app = Celery("myproject")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()
app.conf.beat_schedule = {
'chord_scraping_00': {
'task': 'myapp.tasks.chord_scraping',
'schedule': crontab(minute="0-8"),
'args': ("00",),
}
}
Locally, it all works just fine.
But when I deploy it to Heroku, I don't see any logs using heroku logs --tail
I do get this log repeatedly :
app[heroku-redis]: source=REDIS addon=redis-rugged-87179 sample#active-connections=6 sample#load-avg-1m=0.335 sample#load-avg-5m=0.23 sample#load-avg-15m=0.23 sample#read-iops=0 sample#write-iops=0.15417 sample#memory-total=15629036kB sample#memory-free=6105444kB sample#memory-cached=4767492kB sample#memory-redis=434704bytes sample#hit-rate=0.39511 sample#evicted-keys=0
But there is no log app[beat] as I would expect. Locally I get in Celery's logs :
INFO/MainProcess] Received task: myapp.tasks.chord_scraping[932eba58-102d-4db2-a274-722200a76d45]
Is the periodic task even launching ? Where should the log appear if not in the logs tail?

To be sure that your beat wokrs and for test purpose i suggest that you lunch it every 2 or 3 minutes crontab(minute='*/3').
To check the worker logs:
In herokuapp admin checkout you app and then:
under More choose : View logs
under All Processes choose worker

Related

Can't get tagging to work with cypress-cucumber-preprocessor

I am currently having issues with using tagging with the cypress-cucumber-preprocessor package. I know that the cypress-tags has been removed and made redundant so I'm trying to set up tagging using the new syntax but to no avail.
Here is my feature:
Feature: duckduckgo.com
Rule: I am on a desktop
Scenario: visiting the frontpage
When I visit <site>
Then I should see a search bar
#google
Examples:
| site |
| google.com |
#duckduckgo
Examples:
| site |
| duckduckgo.com |
And my step definitions:
import { When, Then } from "#badeball/cypress-cucumber-preprocessor";
When(`I visit` + url, () => {
if(url === 'duckduckgo.com') return cy.visit("https://www.duckduckgo.com");
if(url === 'google.com') return cy.visit("https://www.google.com");
});
Then("I should see a search bar", () => {
cy.get("input").should(
"have.attr",
"placeholder",
"Search the web without being tracked"
);
});
When I try to run my tests with npx cypress run --env tags="#google", it gives me an error saying url in my steps definitions isn't defined. What am I doing wrong?
Try to add script with this command in package.json file this way:
"scripts": {
"open:test": "npm run clean:report && cypress open --env configFile=test,TAGS=#test" (or any tag you need)
}
And then use it as:
npn run open:test
The main difference besides rapping it into script is not using quotes, maybe this will help you

Possible reasons for groovy program running as kubernetes job dumping threads during execution

I have a simple groovy script that leverages the GPars library's withPool functionality to launch HTTP GET requests to two internal API endpoints in parallel.
The script runs fine locally, both directly as well as a docker container.
When I deploy it as a Kubernetes Job (in our internal EKS cluster: 1.20), it runs there as well, but the moment it hits the first withPool call, I see a giant thread dump, but the execution continues, and completes successfully.
NOTE: Containers in our cluster run with the following pod security context:
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 1000
Environment
# From the k8s job container
groovy#app-271df1d7-15848624-mzhhj:/app$ groovy --version
WARNING: Using incubator modules: jdk.incubator.foreign, jdk.incubator.vector
Groovy Version: 4.0.0 JVM: 17.0.2 Vendor: Eclipse Adoptium OS: Linux
groovy#app-271df1d7-15848624-mzhhj:/app$ ps -ef
UID PID PPID C STIME TTY TIME CMD
groovy 1 0 0 21:04 ? 00:00:00 /bin/bash bin/run-script.sh
groovy 12 1 42 21:04 ? 00:00:17 /opt/java/openjdk/bin/java -Xms3g -Xmx3g --add-modules=ALL-SYSTEM -classpath /opt/groovy/lib/groovy-4.0.0.jar -Dscript.name=/usr/bin/groovy -Dprogram.name=groovy -Dgroovy.starter.conf=/opt/groovy/conf/groovy-starter.conf -Dgroovy.home=/opt/groovy -Dtools.jar=/opt/java/openjdk/lib/tools.jar org.codehaus.groovy.tools.GroovyStarter --main groovy.ui.GroovyMain --conf /opt/groovy/conf/groovy-starter.conf --classpath . /tmp/script.groovy
groovy 116 0 0 21:05 pts/0 00:00:00 bash
groovy 160 116 0 21:05 pts/0 00:00:00 ps -ef
Script (relevant parts)
#Grab('org.codehaus.gpars:gpars:1.2.1')
import static groovyx.gpars.GParsPool.withPool
import groovy.json.JsonSlurper
final def jsl = new JsonSlurper()
//...
while (!(nextBatch = getBatch(batchSize)).isEmpty()) {
def devThread = Thread.start {
withPool(poolSize) {
nextBatch.eachParallel { kw ->
String url = dev + "&" + "query=$kw"
try {
def response = jsl.parseText(url.toURL().getText(connectTimeout: 10.seconds, readTimeout: 10.seconds,
useCaches: true, allowUserInteraction: false))
devResponses[kw] = response
} catch (e) {
println("\tFailed to fetch: $url | error: $e")
}
}
}
}
def stgThread = Thread.start {
withPool(poolSize) {
nextBatch.eachParallel { kw ->
String url = stg + "&" + "query=$kw"
try {
def response = jsl.parseText(url.toURL().getText(connectTimeout: 10.seconds, readTimeout: 10.seconds,
useCaches: true, allowUserInteraction: false))
stgResponses[kw] = response
} catch (e) {
println("\tFailed to fetch: $url | error: $e")
}
}
}
}
devThread.join()
stgThread.join()
}
Dockerfile
FROM groovy:4.0.0-jdk17 as builder
USER root
RUN apt-get update && apt-get install -yq bash curl wget jq
WORKDIR /app
COPY bin /app/bin
RUN chmod +x /app/bin/*
USER groovy
ENTRYPOINT ["/bin/bash"]
CMD ["bin/run-script.sh"]
The bin/run-script.sh simply downloads the above groovy script at runtime and executes it.
wget "$GROOVY_SCRIPT" -O "$LOCAL_FILE"
...
groovy "$LOCAL_FILE"
As soon as the execution hits the first call to withPool(poolSize), there's a giant thread dump, but execution continues.
I'm trying to figure out what could be causing this behavior. Any ideas 🤷🏽‍♂️?
Thread dump
For posterity, answering my own question here.
The issue turned out to be this log4j2 JVM hot-patch that we're currently leveraging to fix the recent log4j2 vulnerability. This agent (running as a DaemonSet) patches all running JVMs in all our k8s clusters.
This, somehow, causes my OpenJDK 17 based app to thread dump. I found the same issue with an ElasticSearch 8.1.0 deployment as well (also uses a pre-packaged OpenJDK 17). This one is a service, so I could see a thread dump happening pretty much every half hour! Interestingly, there are other JVM services (and some SOLR 8 deployments) that don't have this issue 🤷🏽‍♂️.
Anyway, I worked with our devops team to temporarily exclude the node that deployment was running on, and lo and behold, the thread dumps disappeared!
Balance in the universe has been restored 🧘🏻‍♂️.

Laravel sail artisan test is getting killed

I am trying to run a test using sail artisan test --filter=RestaurantAuthTest --verbose, the process is getting killed and I don't know why, I am getting the error below. I am using Laravel Sail on WSL; I am using PHP 8.0.7 and Laravel 8.40.. I haven't had much luck finding related issues online.
⨯ restaurant auth
---
• Tests\Feature\RestaurantAuthTest > restaurant auth
PHPUnit\Framework\Exception
Killed
at vendor/phpunit/phpunit/src/Util/PHP/AbstractPhpProcess.php:270
266▕
267▕ if (!empty($stderr)) {
268▕ $result->addError(
269▕ $test,
➜ 270▕ new Exception(trim($stderr)),
271▕ $time
272▕ );
273▕ } else {
274▕ set_error_handler(
1 vendor/phpunit/phpunit/src/Util/PHP/AbstractPhpProcess.php:187
PHPUnit\Util\PHP\AbstractPhpProcess::processChildResult()
2 vendor/phpunit/phpunit/src/Framework/TestCase.php:883
PHPUnit\Util\PHP\AbstractPhpProcess::runTestJob()
3 vendor/phpunit/phpunit/src/Framework/TestSuite.php:677
PHPUnit\Framework\TestCase::run()
4 vendor/phpunit/phpunit/src/Framework/TestSuite.php:677
PHPUnit\Framework\TestSuite::run()
5 vendor/phpunit/phpunit/src/Framework/TestSuite.php:677
PHPUnit\Framework\TestSuite::run()
6 vendor/phpunit/phpunit/src/TextUI/TestRunner.php:667
PHPUnit\Framework\TestSuite::run()
7 vendor/phpunit/phpunit/src/TextUI/Command.php:143
PHPUnit\TextUI\TestRunner::run()
8 vendor/phpunit/phpunit/src/TextUI/Command.php:96
PHPUnit\TextUI\Command::run()
9 vendor/phpunit/phpunit/phpunit:61
PHPUnit\TextUI\Command::main()
Tests: 1 failed
Time: 395.42s
Thanks

Parse-Server: How can I create a log file on the outside?

I use Parse-Server on Heroku.
I can see server log via parse dashboard. However, it disappears at deployment time.
Is it possible to write logs to external storage?
You can access to your logs like that: heroku logs -n 1500
If you want more than 1500 log lines, you should use an addon or Log Drains
So you can redirect your log to a file or an external storage, for example you can write:
heroku logs -n 1500 --app application_name >> file_logs.txt
There also use FileLoggerAdapter from parse you can use it like that:
fileLogger.info('info content', {} => {...});
fileLogger.info('error content', {} => {...});
fileLogger.query({
level: 'error',
size: 10,
from: Date.now() - (30 * 24 * 60 * 60 * 1000),
until: Date.now(),
order: 'desc'
}, {} => {...})

Writing Parquet file in standalone mode works..multiple worker mode fails

In Spark, version 1.6.1 (code is in Scala 2.10), I am trying to write a data frame to a Parquet file:
import sc.implicits._
val triples = file.map(p => _parse(p, " ", true)).toDF()
triples.write.mode(SaveMode.Overwrite).parquet("hdfs://some.external.ip.address:9000/tmp/table.parquet")
When I do it in development mode, everything works fine. It also works fine if I setup a master and one worker in standalone mode in a docker environment (separate docker containers) on the same machine. It fails when I try to execute it on a cluster (1 master, 5 workers). If I set it up local on the master it also works.
When I try to execute it, I get following stacktrace:
{
"duration": "18.716 secs",
"classPath": "LDFSparkLoaderJobTest2",
"startTime": "2016-07-18T11:41:03.299Z",
"context": "sql-context",
"result": {
"errorClass": "org.apache.spark.SparkException",
"cause": "Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 6, curry-n3): java.lang.NullPointerException
at org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:147)
at org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:113)
at org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:112)
at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.close(ParquetRelation.scala:101)
at org.apache.spark.sql.execution.datasources.DefaultWriterContainer.abortTask$1(WriterContainer.scala:294)
at org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:271)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:150)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:150)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)\n\nDriver stacktrace:",
"stack":[
"org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)",
"org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)",
"org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)",
"scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)",
"scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)",
"org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)",
"org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)",
"org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)",
"scala.Option.foreach(Option.scala:236)",
"org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)",
"org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)",
"org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)",
"org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)",
"org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)",
"org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)",
"org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)",
"org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)",
"org.apache.spark.SparkContext.runJob(SparkContext.scala:1922)",
"org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:150)",
"org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)",
"org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)",
"org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56)",
"org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:108)",
"org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58)",
"org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56)",
"org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70)",
"org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)",
"org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130)",
"org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)",
"org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130)",
"org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:55)",
"org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55)",
"org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:256)",
"org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:148)",
"org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:139)",
"org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:334)",
"LDFSparkLoaderJobTest2$.readFile(SparkLoaderJob.scala:55)",
"LDFSparkLoaderJobTest2$.runJob(SparkLoaderJob.scala:48)",
"LDFSparkLoaderJobTest2$.runJob(SparkLoaderJob.scala:18)",
"spark.jobserver.JobManagerActor$$anonfun$spark$jobserver$JobManagerActor$$getJobFuture$4.apply(JobManagerActor.scala:268)",
"scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)",
"scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)",
"java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)",
"java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)",
"java.lang.Thread.run(Thread.java:745)"
],
"causingClass": "org.apache.spark.SparkException",
"message": "Job aborted."
},
"status": "ERROR",
"jobId": "54ad3056-3aaa-415f-8352-ca8c57e02fe9"
}
Notes:
The job is submitted via the Spark Jobserver.
The file that needs to be converted to a Parquet file is 15.1 MB in size.
Question:
Is there something I am doing wrong (I followed the docs)
Or is there another way I can create the Parquet file, so all my workers have access to it?
In your stand alone setup only one worker is working with ParquetRecordWriter. so it worked fine.
In case of real test i.e. cluster (1 master, 5 workers). with ParquetRecordWriter it will fail since you are concurrently writing with multiple workers...
pls try below.
import sc.implicits._
val triples = file.map(p => _parse(p, " ", true)).toDF()
triples.write.mode(SaveMode.Append).parquet("hdfs://some.external.ip.address:9000/tmp/table.parquet")
pls. see SaveMode.Append "append" When saving a DataFrame to a data source, if data/table already exists, contents of the DataFrame are expected to be appended to existing data.
I had not exactly the same, but similar issues writing dataframes to parquet files in cluster mode. Those problems disappeared when deleteing the file, just before writing, using this convenience function 'write(..)' :
import org.apache.hadoop.fs.FileSystem
import org.apache.hadoop.fs.Path
..
def main(arg: Array[String]) {
..
val fs = FileSystem.get(sc.hadoopConfiguration)
..
def write(df:DataFrame, fn:String ) = {
val op1=s"hdfs:///user/you/$fn"
fs.delete(new Path(op1))
df.write.parquet(op1)
}
Give it a go, tell me if it works for you...

Resources