I hit a strange problem with potentially broken stale frameworks garbage collection on mesos slaves (1.9.0-2.0.1.debian8).
As per documentation stale frameworks should be automatically deleted if are not accessed for predefined time period, however while in slave log I could see slave is running out of storage, there are still framework folders dated more than week old.
I have following values set for garbage collection:
--disk_watch_interval="1mins"
--gc_delay="2days"
--gc_disk_headroom="0.2"
$ date
Thu Nov 7 13:32:50 UTC 2019
$ tail -20 /srv/mesos/agent/logs/mesos-slave.INFO |grep 'Max allowed'
I1107 13:32:33.576203 27465 slave.cpp:7406] Current disk usage 71.83%. Max allowed age: 3.923527125292778hrs
$ ls -l --time=atime /srv/mesos/agent/slaves/bcb34071-9fd8-481a-8d0c-767ef0628216-S2/frameworks/ |grep Oct|tail -5
drwxr-xr-x 3 root root 4096 Oct 28 12:23 fbf26a3e-bb25-4f8f-b9fa-d089c46f7ebd-0004
drwxr-xr-x 3 root root 4096 Oct 28 12:23 fbf26a3e-bb25-4f8f-b9fa-d089c46f7ebd-0005
drwxr-xr-x 3 root root 4096 Oct 28 12:23 fbf26a3e-bb25-4f8f-b9fa-d089c46f7ebd-0007
drwxr-xr-x 3 root root 4096 Oct 28 12:23 fbf26a3e-bb25-4f8f-b9fa-d089c46f7ebd-0008
drwxr-xr-x 3 root root 4096 Oct 28 12:23 fbf26a3e-bb25-4f8f-b9fa-d089c46f7ebd-0009
Any idea for a reason that can happen and a way to force garbage collection (except adjusting gc_delay to zero and restarting mesos slave)?
What frameworks are you using? Maybe you need to set -gc_non_executor_container_sandboxes
A new --gc_non_executor_container_sandboxes agent flag has been added to garbage collect the sandboxes of nested containers, which includes the tasks groups launched by the default executor. We recommend enabling the flag if you have frameworks that launch multiple task groups on the same default executor instance.
Related
I start off by deleting Gradle's build cache
$ rm -rf $GRADLE_HOME/caches/build-cache-*
I manually confirm this directory is gone. I also clean to ensure nothing from incremental build will interfere
$ ./gradlew clean
I then run a build w/ build-cache enabled:
$ ./gradlew --build-cache --scan app:assembleDebug
...
238 actionable tasks: 172 executed, 49 from cache, 17 up-to-date
Checking the generated report I see that some tasks are still marked as FROM-CACHE. How is this happening?
FWIW I'm using Gradle 5.6.3. I also don't use a remote build cache.
UPDATE: after clearing build cache, here's what's left in the cache dir:
$ pwd
/Users/jasonatwood/.gradle/caches
$ ls -al
total 24
drwx------ 11 jasonatwood staff 352 Dec 13 18:11 .
drwxr-xr-x 9 jasonatwood staff 288 Oct 15 17:24 ..
drwxr-xr-x 14 jasonatwood staff 448 Oct 29 16:07 5.4.1
drwxr-xr-x 15 jasonatwood staff 480 Nov 26 18:25 5.6.3
drwxr-xr-x 35 jasonatwood staff 1120 Dec 11 14:14 jars-3
drwxr-xr-x 5 jasonatwood staff 160 Oct 15 17:44 journal-1
drwxr-xr-x 7 jasonatwood staff 224 Oct 15 17:50 modules-2
drwxr-xr-x 5 jasonatwood staff 160 Oct 15 17:50 transforms-2
-rw-r--r-- 1 jasonatwood staff 39 Oct 15 17:27 transforms-2.lock
-rw------- 1 jasonatwood staff 28 Oct 22 16:34 user-id.txt
-rw-r--r-- 1 jasonatwood staff 17 Dec 13 18:00 user-id.txt.lock
If I understand correctly, I don't need to delete any of these.
The local task cache in the .gradle directory within the root project might interfere.
In order to disable the local/global caching altogether, use gradle.properties:
org.gradle.caching=false
If you remove the local build cache and don't use a remove cache, Gradle should indeed not be able to fetch anything cached on the next build. However, I don't think you actually removed anything.
The GRADLE_HOME environment variable, if you have it, should points to an unpacked local Gradle distribution. It could be something like /Users/jasonatwood/gradle-6.0-all. It would be wrong to set it to a path that is not a Gradle distribution.
The Gradle build cache resides in /Users/jasonatwood/.gradle/caches/build-cache-1.
When you try to remove it with $ rm -rf $GRADLE_HOME/caches/build-cache-*, you are using GRADLE_HOME as if it is pointing to the root of your home folder (/Users/jasonatwood), which it most likely isn't.
So instead, use this for removing it: $ rm -rf ~/.gradle/caches/build-cache-*
Or, at the very least, verify that the cache folder has indeed been removed.
If I run a --scan and look at the FROM-CACHE tasks in the generated HTML report.
I then look at the build cache keys for those FROM-CACHE tasks. I then go find those keys on my local machine. I see they are generated at the same time as the rest of the build. So I'm guessing that Gradle is using build cache within a build and not just across builds.
Sometimes there is a need to invalidate cache every time you restart the IDEA. This may be due to shared indexes reuse. What I did Settings --> Tools --> Shared Indexes -> Download Automatically and it resolved all the pain.
Earlier the setting was Don't download use Local.
Any custom metasploit module I create isn't getting loaded.
I tried both of these demos:
https://www.offensive-security.com/metasploit-unleashed/building-module/ https://github.com/rapid7/metasploit-framework/wiki/Loading-External-Modules
and got the same result that the modules were NOT found.
Before posting here, I checked these out:
https://forums.kali.org/showthread.php?28940-Metasploit-modules-not-loading!
https://www.offensive-security.com/metasploit-unleashed/modules-and-locations
and from SO:
I can't get new modules to load in metasploit
How to add module in Metasploit?
no help
Just working with the github example, on the Kali host, I do indeed have the file in the right location (according to the demo):
root#kali:~/.msf4/modules/exploits/test# ls -al
total 12
drwxr-xr-x 2 root root 4096 Mar 19 13:59 .
drwxr-xr-x 3 root root 4096 Mar 19 13:58 ..
-rw-r--r-- 1 root root 9 Mar 19 13:59 test_module.rb
I then ran reload_all and when using this command:
use exploit/test/test_module it returns with Failed to load module.
I also tried to manually load that path and it failed too:
msf > loadpath ~/.msf4/modules/
Loaded 0 modules:
Posting the answer for anyone who may come across this. I'm kinda new to this; didn't know where metasploit info gets logged.
It's in: ~/.msf4/logs/framework.log
The log told me I had a syntax error... must have happened from copying and pasting the text from the browser. Cleaned that up and everything works.
I have IntelliJ CE and I just imported a Go project that I cloned from GitHub. On my Mac, I have some folder organization where I group projects that I work on based on the technologies. For example., in my Projects folder on my Mac, I have the following sub folders:
- go-projects
- scala-projects
- rust-projects
- ruby-projects
So I obviously checked out the Go project in the go-projects folder which looks like this:
Joes-MacBook-Pro:go-projects joe$ ll
total 0
drwxr-xr-x 3 joe staff 102B Oct 28 07:51 bin/
drwxr-xr-x 19 joe staff 646B Oct 24 06:48 golang-restful-starter-kit/
drwxr-xr-x 3 joe staff 102B Oct 28 07:51 pkg/
drwxr-xr-x 3 joe staff 102B Oct 27 10:23 src/
And if I get into the src folder, it looks like this:
Joes-MacBook-Pro:go-projects joe$ cd src
total 0
drwxr-xr-x 6 joe staff 204B Nov 7 19:14 github.com/
Joes-MacBook-Pro:src joe$ cd github.com/
total 0
drwxr-xr-x 3 joe staff 102B Oct 28 08:20 btcsuite/
drwxr-xr-x 16 joe staff 544B Nov 7 19:16 eth-client/
drwxr-xr-x 3 joe staff 102B Oct 27 10:24 ethereum/
drwxr-xr-x 3 joe staff 102B Oct 28 07:51 tools/
Joes-MacBook-Pro:github.com joe$
Here is my Go related environment settings:
# For Go projects, we set the GOPATH
export GOROOT="/usr/local/go"
export GOPATH="/Users/joe/Projects/Private/go-projects"
export GODEPS="/Users/joe/Projects/Private/go-projects/bin/godep"
Now I import one of the Go project into IntelliJ and I get to see the following error saying that it is not able to resolve a directory as it can be seen in the screenshot below!
How can I get rid of this error? Any ideas?
I just realized that when I clone the project, I should not do a git clone but rather do:
go get -u github.com/golang/lint/golint
Not sure why this is so, but after doing this my IntelliJ was able to resolve every file on my project!
For golang projects, it's good to have Goland installed ( also a product of JetBrains )Goland Download. Moreover for the above concern you can use:
go get *the_project_to_clone*
I stuck with this problem in my project, so, I tried it with an empty test project. Here is what I do.
Create a new Cocoa application project in Xcode.
Add a new command-line tool target to it.
Add this target to the dependencies of the cocoa app.
In cocoa app target build phases add Copy files build phase, Destination = Executables, add executable product to the list.
Now Run cocoa application. No problem, application launched. Navigate app bundle in Finder, Show package contents, executable here in Contents/MacOS. This is what I expect to be.
The problem occurs when I archive cocoa application, it fails with the following error:
(null): error: /Users/prudnikov/Library/Developer/Xcode/DerivedData/TestAppWithExecutable-hiysgipuqcktgoayxrgjfrtcxubl/Build/Intermediates/ArchiveIntermediates/TestAppWithExecutable/BuildProductsPath/Release/TestAppExecutable: No such file or directory
Here is a screenshot
Here is content of this directory
prudnikov#Vladimirs-iMac ~ $ ls -al /Users/prudnikov/Library/Developer/Xcode/DerivedData/TestAppWithExecutable-hiysgipuqcktgoayxrgjfrtcxubl/Build/Intermediates/ArchiveIntermediates/TestAppWithExecutable/BuildProductsPath/Release/
total 8
drwxr-xr-x 5 prudnikov staff 170B Apr 26 21:59 ./
drwxr-xr-x# 3 prudnikov staff 102B Apr 26 21:59 ../
drwxr-xr-x 3 prudnikov staff 102B Apr 26 21:59 TestAppExecutable.dSYM/
lrwxr-xr-x 1 prudnikov staff 239B Apr 26 21:59 TestAppWithExecutable.app# -> /Users/prudnikov/Library/Developer/Xcode/DerivedData/TestAppWithExecutable-hiysgipuqcktgoayxrgjfrtcxubl/Build/Intermediates/ArchiveIntermediates/TestAppWithExecutable/InstallationBuildProductsLocation/Applications/TestAppWithExecutable.app
drwxr-xr-x 3 prudnikov staff 102B Apr 26 21:59 TestAppWithExecutable.app.dSYM/
Archiving executable — no problems. All build settings are defaults.
I've also tried to change build configuration from Debug to Release for Run action of Cocoa application scheme. It runs successfully in this case as well.
This is very strange for me. Please advice how to do it?
UPDATE: It seems that I have problem with OS. People tried the same example and it worked. Then I trien to create a new account on my Mac and do the same there and it worked as well. What global configuration can cause this?
I don't think this is an Answer but it's too long for a comment.
Check the parent directories
gdunham: ~/Library/Developer/Xcode/DerivedData$ ls -laeG /Users/gdunham/Library/Developer/Xcode/DerivedData | grep -e " \\.$"
drwxr-xr-x 31 gdunham staff 1054 Apr 15 14:29 /Users/gdunham/Library/Developer/Xcode/DerivedData
drwxr-xr-x 13 gdunham staff 442 May 1 19:56 /Users/gdunham/Library/Developer/Xcode
drwxr-xr-x 4 gdunham staff 136 Mar 8 2012 /Users/gdunham/Library/Developer/
drwx------# 66 gdunham staff 2244 Apr 2 22:43 /Users/gdunham/Library
0: group:everyone deny delete
drwxr-xr-x+ 62 gdunham staff 2108 Apr 30 14:33 /Users/gdunham
0: group:everyone deny delete
drwxr-xr-x 7 root admin 238 Feb 20 14:16 /Users
drwxr-xr-x 34 root wheel 1224 Apr 20 09:26 /
Also look at your group membership
gdunham: ~/Library/Developer/Xcode/DerivedData$ groups
staff com.apple.access_screensharing com.apple.access_screensharing-disabled everyone _appstore localaccounts _appserverusr admin _appserveradm _lpadmin _lpoperator _developer com.apple.sharepoint.group.1
"The application has exited during startup (i.e. during the evaluation of config/environment.rb). The error message may have been written to the web server's log file. Please check the web server's log file (i.e. not the (Rails) application's log file) to find out why the application exited."
I'm getting the above error message when I try to access my test site.
Permission for config/* is
-rw-r--r-- 1 www-data root 2011 Aug 27 20:17 environment.rb
drwxr-xr-x 3 root root 4096 Aug 27 20:17 environments
drwxr-xr-x 3 root root 4096 Aug 27 20:17 initializers
drwxr-xr-x 3 root root 4096 Aug 27 20:17 locales
-rw-r--r-- 1 root root 2431 Aug 27 20:17 routes.rb
I've tried "root root 2011 Aug 27 20:17 environment.rb" as well but no success either.
I'm running Debian GNU/Linux and have other Rails sites running fine on the same server, I have tried to replicate the setup from one of the other sites, but it's not having any effect.
I encountered this same issue, and after killing passenger and apache ended up finding:
http://geminstallthat.wordpress.com/2009/03/01/passenger-with-rails-23-application-missingsourcefile/
Essentially the issue is for some versions of libapache2-mod-passenger the app/controllers/application.rb change to app/controllers/application_controller.rb (in Rails 2.3) explodes things (technical term).