I'm trying to insert Maxmind Data into a Clickhouse Dictionary but defining it source as a local file where I can running my Client from.
so to define my dictionary I use the query:
CREATE DICTIONARY usage_analytics.city_locations(
geoname_id UInt64 DEFAULT 0,
...
...
...
...
)
PRIMARY KEY geoname_id
SOURCE(File(path '/home/ubuntu/maxmind_csv/GeoLite2-City-Locations-en.csv' format 'CSVWithNames'))
SETTINGS(format_csv_allow_single_quotes = 0)
LAYOUT(HASHED())
LIFETIME(300);
yet I keep getting hit with the error of:
Failed to load dictionary 'usage_analytics.city_locations': std::exception. Code: 1001, type: std::__1::__fs::filesystem::filesystem_error, e.what() = filesystem error: in canonical: No such file or directory [\home/ubuntu/maxmind_csv/GeoLite2-City-Locations-en.csv] [/],
According to the documentation, I have to use its absolute path, which I did by using readlink, and still it cannot detect my file. I am running a clickhouse client from a remote machine and have the files on the remote machine. Am I suppose to have my files else where or what?
It looks like this file is not available, to fix it need to to set right ownership for file:
chown clickhouse:clickhouse /home/ubuntu/maxmind_csv/GeoLite2-City-Locations-en.csv
# chown -R clickhouse:clickhouse /home/ubuntu/maxmind_csv
.XML dictionary allows to read files from any folder.
SQL dictionary does not.
https://clickhouse.tech/docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-sources/#dicts-external_dicts_dict_sources-local_file
When dictionary with source FILE is created via DDL command (CREATE DICTIONARY ...), the source file needs to be located in user_files directory, to prevent DB users accessing arbitrary file on ClickHouse node.
/etc/clickhouse-server/config.xml
<!-- Directory with user provided files that are accessible by 'file' table function. -->
<user_files_path>/var/lib/clickhouse/user_files/</user_files_path>
Related
The sample Docker configuration section of stack.yaml gives:
# Location of database used to track image usage, which `stack docker cleanup`
# uses to determine which images should be kept. On shared systems, it may
# be useful to override this in the global configuration file so that
# all users share a single database.
database-path: "~/.stack/docker.db"
However when I put this in the stack.yaml for a new project and stack setup I get:
Aeson exception:
Error in $.docker['database-path']: failed to parse field 'docker': failed to parse field 'database-path': InvalidAbsFile "~/.stack/docker.db"
See http://docs.haskellstack.org/en/stable/yaml_configuration/
This is the only reference I could find to database-path, without digging in to the code.
Is database-path required?
If so: How do I initialize a .db file (to mitigate InvalidAbsFile "~/.stack/docker.db")?
It is not a matter of initialization of the database. The problem is that it does not expand the ~, so you need to use /home/dukedave/.stack/docker.db
it seems my SAS cannot access all folders in the path, i am running sas9.4 on windows10.
in sas9.4, when i run below code:
data _null_;
format x $10000.;
x=sysget('path');
put x=;
run;
it returns full path of various folders:
x=C:\Program Files\SASHome\SASFoundation\9.4;C:\ProgramData\Oracle\Java\javapath;C:\Program Files\MySQL\MySQL Connector C 6.1\lib; C:\xampp\mysql\bin;C:\xampp\mysql\lib; ......(and other folders' path)
then, in sas, when i run
libname mysqllib mysql user=xxxxx .....
return errors:
ERROR: The SAS/ACCESS Interface to MYSQL cannot be loaded. The
libmysql code appendage could not be loaded.
Actually, I already have libmysql.dll under my ...\MySQL\MySQL Connector C 6.1\ (i.e. the 3rd folder in the path)
anyway, i try putting the libmysql.dll under the first folder in path (i.e. C:\Program Files\SASHome\SASFoundation\9.4;) and run again this "libname mysqllib mysql user=xxx" ..... , it works successfully!
Then i deleted this dll under above \9.4 and now try putting this libmysql.dll in the 2nd folder in path (i.e. C:\ProgramData\Oracle\Java\javapath;), and run this "libname mysqllib mysql user=xxx", it fails again.
So, i doubt SAS cannot successfully load in other folders in the path, except only able to load the 1st folder, am i right?
any suggestion? thanks very much.
I am following the tutorial at
https://cloud.google.com/appengine/docs/java/endpoints/calling-from-ios
and when I get to step 5 and Open a new Terminal window to invoke ServiceGenerator. I get the error message in my terminal saying..
Barrys-MacBook-Pro:~ barrymadej$ /Users/barrymadej/Library/Developer/Xcode/DerivedData/ServiceGenerator-avaeguyitgyhxpcnaejpgzvxezei/Build/Products/Debug/ServiceGenerator \
/Users/barrymadej/Documents/AndroidStudioProjects/StudentProgressTrackerDatabaseAndCloud/backend/build/discovery-docs/myApi-v2-rpc.discovery /
ERROR: An output directory is required.
Usage: ServiceGenerator [FLAGS] [ARGS]
Required Flags:
--outputDir PATH
The destination directory for writing the generated files.
Optional Flags:
--discoveryService URL
Instead of discovery's default URL, use the specified URL as the
location to send the JSON-RPC requests. This is useful for running
against a custom or prerelease server.
--gtlFrameworkName NAME
Will generate sources that include GTL's headers as if they are in a
framework with the given name. If you are using GTL via CocoaPods,
you'll likely want to pass "GoogleAPIClient" as the value for this.
--apiLogDir DIR
Write out a file into DIR for each JSON API description processed. These
can be useful for reporting bugs if generation fails with an error.
--httpLogDir PATH
Turn on the HTTP fetcher logging and set it to write to PATH. This can
be useful for diagnosing errors on discovery fetches.
--generatePreferred
Causes the list of services to be collected, and all preferred services
to be generated.
--httpHeader NAME:VALUE
Causes the given NAME/VALUE pair to be added as an HTTP header on *all*
HTTP requests made by the generator. Can be used repeatedly to provide
additional header pairs.
--formattedName SERVICE:VERSION=NAME
Causes the given SERVICE:VERSION pair to override its service name in
files, classes, etc. with NAME. If :VERSION is omitted the override is
for any version of the service. Can be used repeatedly to provide
several maps when generating a few things in a single run.
--addServiceNameDir yes|no Default: no
Causes the generator to add a directory with the service name in the
outputDir for the files. This is useful for generating multiple
services.
--generatedDir yes|no Default: no
Causes a directory in outputDir called "Generated" to be created and
used to contain the generated files.
--removeUnknownFiles yes|no Default: no
By default, the generator will report unknown files in the output
directory, as commonly happens when classes go away in a new API
version. This option causes the generator to also remove the unknown
files.
--rootURLOverrides yes|no Default: yes
Causes any API root URL for a Google sandbox server to be replaced with
the googleapis.com root instead.
--verbose
Generate more verbose output. Can be used more than once.
Arguments:
Multiple arguments can be given on the command line.
service:version
The description of the given [service]/[version] pair is fetched and the
files for it are generated. When using --generatePreferred version can
be '-' to skip generating the name service.
http[s]://url/to/rpc_description_json
A URL to download containing the description of a service to generate.
path/to/rpc_description.json
The path to a text file containing the description of a service to
generate.
ServiceGenerator path:
/Users/barrymadej/Library/Developer/Xcode/DerivedData/ServiceGenerator-avaeguyitgyhxpcnaejpgzvxezei/Build/Products/Debug/ServiceGenerator
ERROR: There was one or more errors; check the full output for details.
Barrys-MacBook-Pro:~ barrymadej$ --outputDir
-bash: --outputDir: command not found
Barrys-MacBook-Pro:~ barrymadej$ /Users/barrymadej/Documents/AndroidStudioProjects/StudentProgressTrackerDatabaseAndCloud/API
You should generate a REST discovery document and use the new Objective C client instead. The client library you're trying to use is deprecated anyway. It looks like it didn't work because you specified the flag without the rest of the command, though.
I would like to delete a file after all the rows in the file have been processed.
My streams look like
source (file --fixedDelay=0 --outputType=text/plain --dir=XXX) |
splitter --expression=payload.split('\\n') |
transform -> filter -> sink
My files are stored in a directory, that is being watched by the file module. I would like that each file is deleted after it has been processed.
Thanks.
Indeed it is surprising that the file source does not have an option for deleting the file after processing, which can be confirmed by looking at the configuration file in xd/modules/source/file/config/file.xml (as of version 1.1.0).
While the file source does not have this option, the sftp source does have it. Hence you could use the sftp source. This will require an ssh server on the machine where spring xd is installed. Does this help?
You may also want to add your own custom source module by configuring a file transformer. File transformers support a delete-files="true" option:
http://docs.spring.io/spring-integration/reference/html/files.html
I'm trying to automate process of deployment and I want to upload some files to WAS using wsadmin (jython). My question is if it is possible to upload file from my standalone wsadmin to remote WAS Server. And if so, is it possible to upload file somewhere out of application (fe. /opt/IBM/WebSphere/AppServer/temp)? I don't want to upload it to specific profile, but to server root.
When I'm deploying application it is copying war/ear file to WAS, so is it there some mechani to upload separate file?
many thanks
AntAgent allows you to upload any file, provided that the content of the file can fit in memory:
https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.javadoc.doc/web/mbeanDocs/AntAgent.html
In wsadmin you'll need to use invoke_jmx method of AdminControl object.
from java.lang import String
import jarray
fileContent = 'hello!'
antAgent = AdminControl.makeObjectName(AdminControl.queryNames('WebSphere:*,type=AntAgent,process=dmgr'))
str = String(fileContent)
bytes = str.getBytes()
AdminControl.invoke_jmx(antAgent, 'putScript', [String('hello.txt'),bytes], jarray.array(['java.lang.String', '[B'], String))
Afterwards you'll find 'hello.txt' file in WAS profile's temp directory. You may use relative paths as well.