I am trying to use an environment variable which contains the file path, in NiFi Expression language. When I just give environment variable ${MY_VAR} for the file name, it is not getting resolved.
But if there is a space or some other character before the NiFi EL, it is being evaluated, but that will make an invalid file name. How to have this resolved without any extra characters.
So as per observation, when the whole content is just an EL it is not getting evaluated otherwise, it is.
PS: I am trying give the file name in Truststore Filename field in default StandardSSLContextService
You are encountering a bug in Apache NiFi. As Bryan pointed out, the truststore location field is not intended to support the Expression Language. The problem is stemming from an incorrect line at StandardSSLContextService:183 where the .evaluateExpressionAttributes() method is being invoked in the custom file validator, despite the comment a few lines above explaining the entire reason for this custom validator is because the default supports EL.
I ran a remote debugger and was able to evaluate ${TRUSTSTORE_LOCATION} using the environment variable TRUSTSTORE_LOCATION (which must be set before NiFi is started, because the VariableRegistry.ENVIRONMENT_SYSTEM_REGISTRY is only created by NotificationServiceManager which is invoked in RunNiFi:270). I was able to watch it correctly evaluate the environment variable to a file path, but then it was concatenated to the current running directory (.../nifi-assembly/target/nifi-1.4.0-SNAPSHOT-bin/nifi-1.4.0-SNAPSHOT/TRUSTSTORE_LOCATION) rather than being treated as an absolute/relative path because the property descriptor is unaware that this is the result of EL evaluation.
I have filed NIFI-4274 to document this bug. A workaround is to use a hard-coded truststore location in NiFi and use symlinks on your filesystem to point that at the actual file location if you do not know it at configure time.
Related
So I have this flow step and I want to replace hardcoded value of file identifier
with variable.
(Point where I call variable)
(How variable looks like)
Hardcoded path works, I tried it.
I want to set hardcoded path into variable and use it in steps like variable.
There are two operations for retrieving file contents, you need to use the right one when you want to use.
I created a basic variable with a file name ...
... and then tested with both types of Get file content operations in the SharePoint group.
You need to use the Get file content using path operation.
New to NiFi, struggling with some basics and would be grateful for any assistance...
I need to get files from a remote server and store them on a NiFi server. Using GetSFTP and PutFile this works ok. But I wish to keep the relative paths.
ie. on remote server /data/hosts/host01/.... would be copied into /imports/host01/... on NiFi
I have this working by using /imports/$path:substring(11)} as the Directory value in PutFile. I have tried using a parameter and a variable to store the Remote Path value and tried using the :length(var1) function of the Parameter and/or Variable within the the above substring function but nothing seems to work other than hard coding the substring length.
Is their an easier way to keep relative paths with get and put processors??
GetFile and PutFile require absolute paths for the directories. If you know the relationship between the root directory and the destinations, you can construct the full path using Expression Language.
I am creating an extract from sequential file. I created a parameter with the correct file location and when I try to "View Data", it says it can't find the file. If I hard code the location it finds the file and I am able to "View Data".
example:
#filedirectory# = aaa/bbb/ccc/
so my entry for "File" is #filedirectory#filename.txt and this does not work
however, the following does work
aaa/bbb/ccc/filename.txt
Any ideas what would cause this?
Try using the absolute path. Start with a / and the root directory.
Second point is that the parameter itself does not have "#". The "#" are only needed to reference it - in the Sequential File stage. So name it filedirectory when you define it in the job.
Recommendation:
As filedirectory will probably be used throughout your project I recommand using a ParameteSet.
I had the same issue, and it happened I forgot to include the directory parameter before the file parameter. Hope this information helps someone.
In WebSphere Application Server 8.5.5 I'm having problems setting the native_stdout and native_stderr log paths from wsadmin jython script.
Everytime I set one path, the other is blanked out. i.e. set to null.
I presume I need to form a line of code that sets both paths simultaneously but have been unable to create the correct syntax, if this is at all possible.
The code to produce one path change looks like:
srv = AdminConfig.getid('/Cell:mycell/Node:mynode/Server:server1/')
processDef = AdminConfig.list('JavaProcessDef', srv)
AdminConfig.modify(processDef, [['ioRedirect', [['stderrFilename', '${LOG_ROOT}/server1/new_stderr.log']]]])
AdminConfig.save()
This works fine but when I then use the same script to set the stdoutFilename, it blanks the stderrFilename previously set.
It seems to make no difference at which point I commit the changes using save.
Any help with this issue would be greatly received!
You're right, even though the manual elsewhere warns that AdminConfig.modify() might append to lists when you don't expect it to. I observed the same thing.
Here's the pattern you want to be safe, clear it out and set both properties at the same time:
AdminConfig.modify(processDef, [['ioRedirect', []]])
AdminConfig.modify(processDef, [['ioRedirect', [['stderrFilename', '${SERVER_LOG_ROOT}/my_stderr.log'], ['stdoutFilename', '${SERVER_LOG_ROOT}/my_stdout.log']]]])
After this, you can do this for a simple verification:
print AdminConfig.showall(processDef)
But watch the output closely as it is not pretty-printed
The HSQLDB documentation says something to the effect of:
Special care may be needed w.r.t. file path specifications containing whitespace, mixed-case, special characters and/or reserved file names. Please read your OS file system documentation.
However, they provide ZERO documentation or examples on the syntax for actually using file names with reserved characters or spaces. When I use a file name with a space in it, HSQLDB appears to simply hang my entire application.
I've tried escaping with URL syntax ("%20") which does not work because HSQLDB interprets that literally. I've tried surrounding various portions of the JDBC URL with single and double quotes. Most result in a failure to access the database error. At least they don't simply hang.
Does anyone know the proper way to reference an embedded HSQLDB file when the file path has spaces?
When I use a file name with a space in it, HSQLDB appears to simply hang my entire application.
For what it's worth, my Windows (Vista) test box has no problems with spaces in the database path or the database name
String connStr = "jdbc:hsqldb:file:C:/Users/Public/test/HSQLDB test/my db";
try (Connection con = DriverManager.getConnection(connStr, "sa", "")) {
results in this
Tested using hsqldb-2.3.1.