Attempting a Pyflink stream setup from ADLS and currently trying to read a json file using the StreamExecutionEnvironment.from_source() method.
Here's how the code looks like:
from flink.plan.Environment import get_environment
from pyflink.datastream.functions import SourceFunction
from pyflink.datastream import StreamExecutionEnvironment, RuntimeExecutionMode
from pyflink.datastream.connectors.file_system import (FileSource, StreamFormat, FileSink,
OutputFileConfig, RollingPolicy, BucketAssigner)
from pyflink.common import WatermarkStrategy, Encoder, Types
from azure.storage.filedatalake import FileSystemClient
file_system = FileSystemClient.from_connection_string(connection_str, file_system_name="my_fs")
# setting the stream environment object
env = StreamExecutionEnvironment.get_execution_environment()
env.set_runtime_mode(RuntimeExecutionMode.STREAMING)
env.add_jars("file:///opt/flink/plugins/azure/flink-azure-fs-hadoop-1.16.0.jar")
env.add_classpaths("file:///opt/flink/plugins/azure/flink-azure-fs-hadoop-1.16.0.jar")
file_client = file_system.get_file_client(my_file)
input_path = 'abfss://' + file_client.url[8:]
print('URL is ===== >>>>',file_client.url)
# Source
ds = env.from_source(
source=FileSource.for_record_stream_format(StreamFormat.text_line_format(),
input_path)
.process_static_file_set().build(),
watermark_strategy=WatermarkStrategy.for_monotonous_timestamps(),
source_name="file_source"
)
ds.sink_to(
sink=FileSink.for_row_format(
base_path=output_path,
encoder=Encoder.simple_string_encoder())
.with_bucket_assigner(BucketAssigner.base_path_bucket_assigner())
.build())
ds.print()
env.execute()
I am getting below error:
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'abfs'. The scheme is directly supported by Flink through the following plugin(s): flink-fs-azure-hadoop. Please ensure that each plugin resides within its own subfolder within the plugins directory. See https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/plugins/ for more information. If you want to use a Hadoop file system for that scheme, please add the scheme to the configuration fs.allowed-fallback-filesystems. For a full list of supported file systems, please see https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/.
Jar file has already been added in to the plugins folder as given in the documentation:
https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/plugins/
Also, storage account key is also added into the config.yaml file.
Alternatively tried adding the source as DataStream using :
ds = env.read_text_file(input_path)
I have a CSV file I generate through code. I want to import the generated CSV file into neo4j using the following cypher query.
LOAD CSV WITH HEADERS FROM 'file:////Users/{user}/Desktop/neo4j-importer/tmp/temp_data.csv'
I have changed the following config varables
Commenting out dbms.directories.import=import.
And set dbms.security.allow_csv_import_from_file_urls=true
Problem is I get thrown the following error:
Neo.ClientError.Statement.ExternalResourceFailed:
Couldn't load the external resource at:
file:/Users/{user}/Library/Application%20Support/Neo4j%20Desktop/Application/neo4jDatabases/database-c517b267-220d-4b7a-be26-813d5b64a51a/installation-3.5.3/import/Users/{user}/Desktop/neo4j-importer/tmp/temp_data.csv
I mean it is partly right just not the /Users/{user}/Library/Application%20Support/Neo4j%20Desktop/Application/neo4jDatabases/database-c517b267-220d-4b7a-be26-813d5b64a51a/installation-3.5.3/import/ bit... Any suggestions on how to fix this weird file pathing problem?
Try changing the config setting to point to the directory with your imports:
dbms.directories.import=/Users/{user}/Desktop/neo4j-importer/tmp
and then changing the Cypher query to just specify the CSV file:
LOAD CSV WITH HEADERS FROM 'file:///temp_data.csv'
...
I tried to import bulk image dataflow profile,
this is my export format:
this is my import format :
this is my excel format in which I export,
then I import the exported excel file and get the following error:
Step1: Create import folder inside media folder
Step2: manage you csv file , Only add "/imagename.jpg" in your images
columns
I'm trying to automate process of deployment and I want to upload some files to WAS using wsadmin (jython). My question is if it is possible to upload file from my standalone wsadmin to remote WAS Server. And if so, is it possible to upload file somewhere out of application (fe. /opt/IBM/WebSphere/AppServer/temp)? I don't want to upload it to specific profile, but to server root.
When I'm deploying application it is copying war/ear file to WAS, so is it there some mechani to upload separate file?
many thanks
AntAgent allows you to upload any file, provided that the content of the file can fit in memory:
https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.javadoc.doc/web/mbeanDocs/AntAgent.html
In wsadmin you'll need to use invoke_jmx method of AdminControl object.
from java.lang import String
import jarray
fileContent = 'hello!'
antAgent = AdminControl.makeObjectName(AdminControl.queryNames('WebSphere:*,type=AntAgent,process=dmgr'))
str = String(fileContent)
bytes = str.getBytes()
AdminControl.invoke_jmx(antAgent, 'putScript', [String('hello.txt'),bytes], jarray.array(['java.lang.String', '[B'], String))
Afterwards you'll find 'hello.txt' file in WAS profile's temp directory. You may use relative paths as well.
I am working on a project using Python 2.7 and PySide 1.1.2. My code is working without any problem on my GNU/Linux but I want to distribute for Windows ( 7 and 8 ) as well. I can't expect users to install Python and PySide, so I decided to use py2exe (I also tried cx_freeze and pyinstaller).
First of all, here is my file tree: My Project on GitHub
I created a setup.py, here it is:
# -*- coding: utf-8 -*-
from distutils.core import setup
import py2exe
setup(
console=['bin/metusuite.py'],
name='metusuite',
version='0.1',
author='H. Gökhan Sarı',
author_email='me#th0th.me',
packages=['metusuite_libs'],
package_dir={'metusuite_libs': 'metusuite_libs'},
package_data={'metusuite_libs': ['ui/*', 'images/*']},
scripts=['bin/metusuite.py'],
url='https://github.com/th0th/metusuite/',
license='LICENSE.txt',
description='METU Suite.',
long_description=open('README.md').read(),
)
When I run
setup.py py2exe
it successfully builds metusuite.exe in 'dist' folder, however, since application depends on external user interface files -created with Qt Designer- and it can't find them, I get an error:
Designer: An error has occurred while reading the UI file at line 1, column 0: Premature end of document.
Traceback (most recent call last):
File "metusuite.py", line 38, in <module>
File "metusuite_libs\msCafeteriaMenu.pyc", line 37, in __init__
File "metusuite_libs\msCafeteriaMenu.pyc", line 17, in __init__
RuntimeError: Unable to open/read ui device
And I couldn't figure out how am I supposed to add *.ui files (also there are some .png icons) into that structure. I was thinking of converting .ui files to Python code, then I would have encounter same issue when I need to add some icons.
Hence, how can I add my ui and png files in py2exe structure? Or is there any alternative method for what I am trying to accomplish?
Well, I think you could do one of two realistic things:
Compile your .ui files to .py files using pyside-uic and modify your code to do conditional loading of the py files for the user interface and place the png files in a Qt Resource file
Create a Qt Resource file with your ui files inside of it, compile that with pyside-rcc, and then load the ui files using QtUiTools or some similar process
pyside-uic
I greatly prefer using the pyside-uic method for loading ui files because it is the most straightforward way of loading ui files into a program that correlates with my knowledge of Qt in C++. pyside-uic is included with the PySide applications and for me it is found in the Scripts directory of my Python installation, e.g. C:\Python27\Scripts\pyside-uic.exe. Taking a note from how C++ compilation handles ui files, I typically compile my ui files to have a name like ui_[Name of the ui file].py:
C:\Python27\Scripts\pyside-uic mainwindow.ui > ui_mainwindow.py
Inside of that resulting .py file, pyside-uic creates a class named the same name as the base class of the ui file prepended with Ui_. So, for instance, if you created a mainwindow.ui that contained the definition for a class named MainWindow, the created class would be Ui_MainWindow. If the ui file defined a class named SourceWindow, the class within the .py file Ui_SourceWindow. In Qt Designer you set the class name by setting objectName in the root element of the object tree (in the upper right of the window).
With your files cafeteria_menu,ui and dialog_login.ui, you would get derived classes Ui_cafeteria_menu and Ui_dialog_login.
Once you have the .py file generated, it can be used by importing it into the definition file for your widget and used using the setupUi method of the class in the Ui file
from PySide import QtCore, QtGui
from ui_mainwindow import Ui_MainWindow
class MainWindow(QtGui.QMainWindow):
def __init__(self, parent=None):
super(MainWindow, self).__init__(parent)
self.ui = Ui_MainWindow()
self.ui.setupUi(self)
Once you have ui defined for the class, all of the connections and ui elements for the widget need to be accessed through self.ui
self.ui.lineEdit.textChanged[str].connect(self.processText)
Since you would have to put your .png files in a Qt Resource File, I'll talk about it in the next section.
pyside-rcc
Like pyside-uic, pyside-rcc is included with the PySide application, although mine is in the site-packages directory of Python instead of in Scripts (if it's in the same place for you, you can always copy it).
C:\Python27\lib\site-packages\PySide\pyside-rcc.exe
Before you can compile the Qt Resource File, you have to first create it using one of the Qt Tools. I use Qt Creator since it can perform almost all of the functions related to Qt in one application. The documentation for the Qt Resource System shows that the resource file is really just an XML file that defines file paths and internal paths for the resource system. You can set up and organize the files however you want but when it comes to compile, all of the files defined in the Resource File must be in the same directory or a sub-directory of the file. Once you have the Resource File defined, you need to use pyside-rcc.exe to compile it into a .py file. I typically name the resource file the same as the project and keep everything in one resource file to make dealing with the resources more concise.
C:\Python27\lib\site-packages\PySide\pyside-rcc.exe -py2 MyProject.qrc > MyProject_Resources.py
The -py2 switch defines that the output from the file should be formatted for Python 2.x. If you are using Python 3.x or plan to use it in the future, you can use the -py3 switch and the outcome will be compatible with Python 3.x.
Putting it all together
Since you are already loading the ui files directly QUiLoader, you just need to refactor your QUiLoader statements to load a QFile that opens the ui resource from the resource system. To use the files from the resource system, all you need to do is import your Resource .py file, the one generated from pyside-rcc, into the main script file of your program and the last line in the resource file is a call to qInitResources() which initializes the resources to be used in the entire program. To load a file using QFile, use a path that starts with ":" and then references the paths that were defined in the Resource File. You could create a file msResources.qrc that has ui and images that has your ui and png files defined as sub categories.
So, if your resource file looks something like this
/ui
cafeteria_menu.ui
dialog_login.ui
/images
cafeteria-menu.png
exit.png
logo.png
mail-fetch.ong
And, if you want to load those files, you just need to create a QIcon or QFile like so:
cafeteriaMenuIcon = QtGui.QIcon(":/images/cafeteria-menu.png")
cafeteriaMenuUi = QtCore.QFile(":/ui/cafeteria_menu.ui")
In use in your code for GUICafeteriaMenu in msCafeteriaMenu, I would just change the __init__ method for GuiCafeteriaMenu to load and use the ui file from the resources:
uiFile = QFile(":/ui/cafeteria_menu.ui")
uiFile.open(QFile.ReadOnly)
UiLoader.load(uiFile, self)
uiFile.close()
I would probably place the output from pyside-rcc into the metsuite_libs package into something like msResources.py and import the msResources file in the __init__.py file as part of your package. That way, once you have the .py files created and imported into your program, the extra file would be encapsulated in your package and you will not need to change your setup.py file. Before you do the py2exe conversion, running the refactored program should work just fine normally. Additionally, no matter how you handle the ui files, you will always need to use a Resource File to be able to package icons into the program. For portability reasons, using resource files for icons is probably a good habit to get in to.