I was running my Acceptance test using IntelliJ
Unfortunately sometimes I get this Exception in IntelliJ executing specific task as this:
Caused by: java.lang.IllegalArgumentException: Could not resolve placeholder 'SOME_VARIABLE' in value "classpath:/${SOME_VARIABLE}/acceptance-test.properties"
at org.springframework.util.PropertyPlaceholderHelper.parseStringValue(PropertyPlaceholderHelper.java:180)
at org.springframework.util.PropertyPlaceholderHelper.replacePlaceholders(PropertyPlaceholderHelper.java:126)
at org.springframework.core.env.AbstractPropertyResolver.doResolvePlaceholders(AbstractPropertyResolver.java:239)
at org.springframework.core.env.AbstractPropertyResolver.resolveRequiredPlaceholders(AbstractPropertyResolver.java:210)
at org.springframework.core.env.AbstractEnvironment.resolveRequiredPlaceholders(AbstractEnvironment.java:630)
at org.springframework.context.annotation.ConfigurationClassParser.processPropertySource(ConfigurationClassParser.java:461)
at org.springframework.context.annotation.ConfigurationClassParser.doProcessConfigurationClass(ConfigurationClassParser.java:280)
at org.springframework.context.annotation.ConfigurationClassParser.processConfigurationClass(ConfigurationClassParser.java:250)
at org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:207)
at org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:175)
... 67 more
But, I would like to execute it using my terminal in macOS.
I know that, I can't to do it passing all variables as this manner!
SPRING_PROFILES_ACTIVE=xyz SPRING_CLOUD_CONFIG_URI=http://config-server EUREKA_SERVICE_URL=http=//eureka-server VAULT_PORT=8200 VAULT_AUTHENTICATION=TOKEN VAULT_TRUST_STORE_PWD=xxxxx VAULT_TRUST_STORE=file:/Users/joseluisbz/blabla.ts gradle app:acceptanceTest --stacktrace
Is it possible passing the same .env file in terminal?
Passing ENV values to gradle test is 2 layered
using ./gradle.properties or ~/.gradle/gradle.properties to setup the basic ENV variables. see https://docs.gradle.org/current/userguide/build_environment.html#sec:gradle_configuration_properties
in the test { } block setup map of env values using either systemProperties or environment as per your need. see https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test
I have created a custom environment variable as below in my environment-develop.properties file. I have set the value of
aws.access-key-id=${AWS_ACCESS_KEY_ID}
Trying to get the value as below:
System.out.println(environment.getProperty("aws.access-key-id"));
But this is not working. Error is:
Caused by: java.lang.IllegalArgumentException: Could not resolve placeholder 'AWS_ACCESS_KEY_ID' in string value "${AWS_ACCESS_KEY_ID}"
How to obtain this value (set to abcd in Intellij's environment variable configuration)?
I am able to run my pipelines using the kedro run command without issue. For some reason though I can't access my context and catalog from Jupyter Notebook anymore. When I run kedro jupyter notebook and start a new (or existing) notebook using my project name when selecting "New", I get the errors following errors:
context
NameError: name 'context' is not defined
catalog.list()
NameError: name 'catalog' is not defined
EDIT:
After running the magic command %kedro_reload I can see that my ProjectContext init_spark_session is looking for files in project_name/notebooks instead of project_name/src. I tried changing the working directory in my Jupyter Notebook session with %cd ../src and os.ch_dir('../src') but kedro still looks in the notebooks folder:
%kedro_reload
java.io.FileNotFoundException: File file:/Users/user_name/Documents/app_name/kedro/notebooks/dist/project_name-0.1-py3.8.egg does not exist
_spark_session.sparkContext.addPyFile() is looking in the wrong place. When I comment out this line from my ProjectContext this error goes away but I receive another one about not being able to find my Oracle driver when trying to load a dataset from the catalog:
df = catalog.load('dataset')
java.lang.ClassNotFoundException: oracle.jdbc.driver.OracleDriver
EDIT 2:
For reference:
kedro/src/project_name/context.py
def init_spark_session(self) -> None:
"""Initialises a SparkSession using the config defined in project's conf folder."""
# Load the spark configuration in spark.yaml using the config loader
parameters = self.config_loader.get("spark*", "spark*/**")
spark_conf = SparkConf().setAll(parameters.items())
# Initialise the spark session
spark_session_conf = (
SparkSession.builder.appName(self.package_name)
.enableHiveSupport()
.config(conf=spark_conf)
)
_spark_session = spark_session_conf.getOrCreate()
_spark_session.sparkContext.setLogLevel("WARN")
_spark_session.sparkContext.addPyFile(f'src/dist/project_name-{__version__}-py3.8.egg')
kedro/conf/base/spark.yml:
# You can define spark specific configuration here.
spark.driver.maxResultSize: 8g
spark.hadoop.fs.s3a.impl: org.apache.hadoop.fs.s3a.S3AFileSystem
spark.sql.execution.arrow.pyspark.enabled: true
# https://kedro.readthedocs.io/en/stable/11_tools_integration/01_pyspark.html#tips-for-maximising-concurrency-using-threadrunner
spark.scheduler.mode: FAIR
# JDBC driver
spark.jars: drivers/ojdbc8-21.1.0.0.jar
I think a combination of this might help you:
Generally, let's try to avoid manually interfering with the current working directory, so let's remove os.chdir in your notebook. Construct an absolute path where possible.
In your init_spark_session, when addPyFile, use absolute path instead. self.project_path points to the root directory of your Kedro project, so you can use it to construct the path to your PyFile accordingly, e.g. _spark_session.sparkContext.addPyFile(f'{self.project_path}/src/dist/project_name-{__version__}-py3.8.egg')
Not sure why you would need to add the PyFile though, but maybe you have a specific reason.
I have created these variables with their respective values, but even after doing this, command prompt is not recognizing mvn command
System Variables:
name: M2_HOME
value: C:\Program Files\apache-maven-3.2.1
name: M2
value: %M2_HOME%/bin;
name: path
value: %JAVA_HOME%\bin; %M2_HOME%\bin;
From Maven documentation page:
Maven, like many cross-platform tools, can encounter problems when
there are space characters in important pathnames. The instructions
below will remind you of this for several particular items.
You have Maven installed in C:\Program Files\apache-maven-3.2.1
Try to reinstall it to, for example, C:\maven, or change your current variable M2_HOME to C:\PROGRA~1\apache-maven-3.2.1
I had the same issue.
What I did to resolve this is creating M2_HOME & MAVEN_HOME.
Also, edit Path variable value.
New ->C:\Program Files\Apache Software Foundation\apache-maven-3.6.0\bin
where Apache Software Foundation is a folder I created to extract the apache-maven-3.6.0-bin.zip file.
Also, make sure Java is working fine and JAVA_HOME is properly set.
try :
to restart the command prompt
to start the command prompt with run/cmd instead of shift+right-click -"open command window here"
to put the system path between double quote to manage espace character
to replace C:\Program Files\ per C:\PROGRA~1\
edit
this is my own config :
MAVEN_HOME="C:\Program Files\apache-maven-3.2.1"
PATH=XXX;%MAVEN_HOME%\bin;XXX
Hello I had the same problem and I figured it out that I was including some space characters on the "path" next to the ; just by removing the space character I was able to have everything working properly"
this is the settings for my maven.
name: MVN_HOME
value: C:\Program Files\apache-maven-3.2.1
name: path -> add the following value(just the maven part)
value: %JAVA_HOME%\bin;%MVN_HOME%\bin; -> this should have no spaces
I have a very difficult time to solve this problem. It did not work no matter how I set up the environment variables in the control panel of Win7.
My final solution is to set the variables via the cmd.
set M2_HOME "D:\Program Files\Maven\apache-maven-3.3.3"
set M2=%M2_HOME%\bin
set path=%path%;%M2%
I think you have a typo:
change
name: M2
value: %M2_HOME%/bin;
to
name: M2
value: %M2_HOME%\bin;
if it doesn't work change %M2_HOME%\bin with the path.
in this case:
C:\Program Files\apache-maven-3.2.1\bin
Make sure to set the path under the System Variables and not User Variables.
In my case, for reasons I cannot explain, it made a difference to windows 7 if the MAVEN_HOME environment variable was defined in "System Variables" vs "User Variables".
I originally had both JAVA_HOME and MAVEN_HOME as "User Variables" and "Java -version" worked but windows could not find mvn (My path variable looked like this in both cases "Path=C:\Program Files\Java\jre1.8.0_144\bin;E:\Clif\Java\apache-maven-3.6.0\bin;")
Once I changed both Environment variables from "User Variables" to "System Variables" windows was able to find mvn.
I was using %MAVEN_HOME% in the variable path and it was giving problem, even though echo %MAVEN_HOME% command was showing the right path.
I changed it to C:\Program Files\apache-maven-3.6.1\bin in the path (Depends on your version of-course) and it worked like a charm.
I have a sample from Gaelyk (called Bloogie) and it is using gradlew.
I am behind a proxy.
I've read gradle docs and found this:
gradle.properties
systemProp.http.proxyHost=www.somehost.org
systemProp.http.proxyPort=8080
systemProp.http.proxyUser=userid
systemProp.http.proxyPassword=password
But I have no clue how to put this info into the wrapper gradlew. Any idea?
All you have to do is to create a file called gradle.properties (with the properties you mentioned above) and place it under your gradle user home directory (which defaults to USER_HOME/.gradle) OR in your project directory.
Gradle (the wrapper too!!!) automatically picks up gradle.properties files if found in the user home directory or project directories.
For more info, read the Gradle user guide, especially at section 12.3: Accessing the web via a proxy
If you need https access behind a proxy, please consider defining also the same set of properties for systemProp.https.
systemProp.https.proxyHost=www.somehost.org
systemProp.https.proxyPort=8080
See Can't build Android app using crashlytics behind VPN and proxy for more information.
Add the below in your gradle.properties file and in your gradle/wrapper/gradle-wrapper.properties file if you are downloading the wrapper over a proxy
If you want to set these properties globally then add it in USER_HOME/.gradle/gradle.properties file
## Proxy setup
systemProp.proxySet=true
systemProp.http.keepAlive=true
systemProp.http.proxyHost=host
systemProp.http.proxyPort=port
systemProp.http.proxyUser=username
systemProp.http.proxyPassword=password
systemProp.http.nonProxyHosts=local.net|some.host.com
systemProp.https.keepAlive=true
systemProp.https.proxyHost=host
systemProp.https.proxyPort=port
systemProp.https.proxyUser=username
systemProp.https.proxyPassword=password
systemProp.https.nonProxyHosts=local.net|some.host.com
## end of proxy setup
Use this in prompt line:
gradle -Dhttp.proxyHost=*** -Dhttp.proxyPort=*** -Dhttp.proxyUser=**** -Dhttp.proxyPassword=****
Works here!
I could not get the proxy property to work until I set the https proxy:
systemProp.https.proxyHost=www.somehost.org
systemProp.https.proxyPort=8080
However I had to use the http property for user name and password:
systemProp.http.proxyUser=userid
systemProp.http.proxyPassword=password
This problem with the Gradle Wrapper has been fixed with Gradle 1.0-milestone-8. Give it a shot.
after of this JDK update, I couldn't use gradlew behind a proxy again.
and finally I found a JDK has disabled Basic authentication for HTTPS tunneling by default.
so I have to add this property for gradle.properties in addition to proxy settings
systemProp.jdk.http.auth.tunneling.disabledSchemes=""
I hope it would be helpful for someone who struggle same problem
To add more nuances, for my case, when I have multiple gradle.properties files in both USER_HOME/.gradle and the project root, I encountered the authenticationrequired 407 error, with the bellow log: CONNECT refused by proxy: HTTP/1.1 407 authenticationrequired
This caused my systemProp.https.proxyPassword and systemProp.http.proxyPasswordblank in the gradle.properties file under USER_HOME/.gradle, while the gradle.properties file under the project root remained password info. Not sure the exact reason, But when I remove one gradle.properties in the project root and keep the file in the USER_HOME/.gradle, my case is resolved.
I had same problem and first thing I did was to create gradle.properties. I had not such as file so I should create it with following content:
systemProp.http.proxyHost=proxy
systemProp.http.proxyPort=port
systemProp.http.nonProxyHosts=domainname|localhost
systemProp.https.proxyHost=proxy
systemProp.https.proxyPort=port
systemProp.https.nonProxyHosts=domainname|localhost
When I added them gradlew command works properly behind corporate proxy. I hope that it can be useful.
I was found that reading of properties from gradle.properties can be incorrect. In case line contains trail white space, gradle cannot find proxy. check your proxy file and cut whitespace at the end of line. Can be help
This was not working for me at first.
In my case, I had created what I thought was a USER_HOME/.gradle/gradle.properties file but ended up with a gradle.properties.txt file.
From the terminal window an ls command will show the full file names in the .gradle folder.
Then mv gradle.properties.txt gradle.properties
I have the same proxy issue while working with Cordova project.
To fix the issue, I have created a new gradle.properties file under the android folder of my Cordova project (hello/platforms/android), and added the code from your question
systemProp.http.proxyHost=proxy.yourproxysite.com
systemProp.http.proxyPort=8080
systemProp.http.proxyUser=yourusername
systemProp.http.proxyPassword=password
Setting SSl proxy worked for me.
systemProp.http.proxyHost=proxy.yourproxysite.com
systemProp.http.proxyPort=8080
systemProp.https.proxyHost=proxy.yourproxysite.com
systemProp.https.proxyPort=8080
An excerpted answer from the linked thread below. It shows how to do
this more programtically. Hope it helps
task setHttpProxyFromEnv {
def map = ['HTTP_PROXY': 'http', 'HTTPS_PROXY': 'https']
for (e in System.getenv()) {
def key = e.key.toUpperCase()
if (key in map) {
def base = map[key]
//Get proxyHost,port, username, and password from http system properties
// in the format http://username:password#proxyhost:proxyport
def (val1,val2) = e.value.tokenize( '#' )
def (val3,val4) = val1.tokenize( '//' )
def(userName, password) = val4.tokenize(':')
def url = e.value.toURL()
//println " - systemProp.${base}.proxy=${url.host}:${url.port}"
System.setProperty("${base}.proxyHost", url.host.toString())
System.setProperty("${base}.proxyPort", url.port.toString())
System.setProperty("${base}.proxyUser", userName.toString())
System.setProperty("${base}.proxyPassword", password.toString())
}
}
}
See this thread for more
After lots of struggling with this and banging my head against a wall, because nothing on my system was using a proxy: it turned out that my ** Android Emulator instance ** itself was secretly/silently setting a proxy for me via Android Emulator > Settings > Proxy and had applied these settings when playing around with it weeks earlier in order to troubleshoot an issue with Expo.
If anyone is having this issue, make sure you check 100% to see if indeed no custom proxy settings are being used via: ./gradlew installDebug --info --debug --stacktrace and searching for proxyHost in the log output to make sure of this. It may be your emulator.
The following applies when your gradle archive is mirrored behind the firewall (like mine..):
For some reason, I needed both of these lines:
gradle.properties:
systemProp.http.nonProxyHosts=*.localserver.co
systemProp.https.nonProxyHosts=*.localserver.co
EVEN though my download line started with https, such as below:
gradle-wrapper.properties:
distributionUrl=https\://s.localserver.co/gradle-7.0.1-bin.zip
It wasn't working in ANY other way... except only it worked if I used export JAVA_OPTS=-Dhttp.nonProxyHosts=localserver.co|etc.
Even though my environment variable no_proxy was already correctly set, it wasn't working without the two values in the above properties.
systemProp.http.proxyUser=userId
systemProp.http.proxyPassword=password
same with https......