I'm new to spring batch and I'm using below code to read files from unix server in spring batch application.
public class test2
{
static String user = "myusername";
static String host = "user#remotehost";
static String password = "mypasswd";
static String port = 22;
public static void main(String[] args) throws FileNotFoundException, IOException, JSchException
{
JSch jsch=new JSch();
Session session=jsch.getSession(user, host, 22);
session.setPassword(password);
Properties config = new Properties ();
config.put("StrictHostKeyChecking", "no");
session.setConfig(config);
session.connect();
Channel channel=session.openChannel("sftp");
channel.connect();
ChannelSftp channelsftp = (ChannelSftp)channel;
channelSftp.cd("unix directory");
List<ChannelSftp.LsEntry> list = channelSftp.ls("*"+fileName+"*");
// Processing the files here
}
}
Is there any other way to read files from unix using any item readers. Please suggest
Thanks in Advance
Related
I am kind of running into this interesting problem. I wanted to delete the jetty working directory before jetty restarts next time. The application has 2 war files and this is implemented using embed jetty. Name of the working folder looks like below:
jetty-0_0_0_0-10000-oaxui_war-_ui_oax-any-4214704288653178451
jetty-0_0_0_0-10000-oaxservice_war-_api_oax-any-1938823993160354573
Options 1: We have run.sh file which actually starts JettyServer.So I thought to place the below code in the file just before that.
echo "now deleting"
DELTEMP="rm -rf /tmp/jetty*"
exec $DELTEMP
echo "deleted....."
Result: It actually deletes the jetty working directory but does not let it create one working directory also.
Option 2: Create a temp directory in the location provided by us as below and then later delete this folder using the run.sh and above command but the path will be custom. Unfortunately, this also didn't help as the working directory was not being created in the first place.
private HandlerCollection getWebAppHandlers() throws SQLException, NamingException{
//Setting the war and context path for the service layer: oaxservice
File tempDir1 = new File("/faw/service/appshell/tempdir/");
File tempDir2 = new File("/faw/service/appshell/tempdir/");
WebAppContext serviceWebapp = new WebAppContext();
tempDir1.mkdirs();
serviceWebapp.setTempDirectory(tempDir1);
serviceWebapp.setWar(APPSHELL_API_WAR_FILE_PATH);
serviceWebapp.setContextPath(APPSHELL_API_CONTEXT_PATH);
//setting the war and context path for the UI layer: oaxui
WebAppContext uiWebapp = new WebAppContext();
tempDir2.mkdirs();
uiWebapp.setTempDirectory(tempDir2);
uiWebapp.setWar(APPSHELL_UI_WAR_FILE_PATH);
uiWebapp.setContextPath(APPSHELL_UI_CONTEXT_PATH);
uiWebapp.setAllowNullPathInfo(true);
uiWebapp.setInitParameter("org.eclipse.jetty.servlet.Default.dirAllowed", "false");
//set error page handler for the UI context
uiWebapp.setErrorHandler(new CustomErrorHandler());
//handling the multiple war files using HandlerCollection.
HandlerCollection handlerCollection = new HandlerCollection();
handlerCollection.setHandlers(new Handler[]{serviceWebapp, uiWebapp});
return handlerCollection;
}
Below is the complete JettyServer.java file
public class JettyServer {
private final static Logger logger = Logger.getLogger(JettyServer.class.getName());
private static final int JETTY_PORT = 10000;
private static final String JETTY_REALM_PROPERTIES_FILE_NAME = "realm.properties";
private static final String JETTY_REALM_NAME = "myrealm";
private static final String APPSHELL_WAR_FOLDER = "/faw/service/appshell/target/";
private static final String APPSHELL_UI_WAR_FILE_PATH = APPSHELL_WAR_FOLDER+"oaxui.war";
private static final String APPSHELL_API_WAR_FILE_PATH = APPSHELL_WAR_FOLDER+"oaxservice.war";
private static final String APPSHELL_API_CONTEXT_PATH = "/api/oax";
private static final String APPSHELL_UI_CONTEXT_PATH = "/ui/oax";
private static final String JETTY_CONFIG_FOLDER = "/faw/service/appshell/config/";
private static final String JETTY_CONFIG_FILE = JETTY_CONFIG_FOLDER+"datasource.properties";
private static final String JETTY_CONFIG_DATASOURCE_URL = "datasource.url";
private static final String JETTY_CONFIG_APPSHELL_SCHEMA_NAME = "appshell.datasource.username";
private static final String JETTY_CONFIG_APPSHELL_SCHEMA_PWD = "appshell.datasource.password";
private static final String JETTY_CONFIG_APPSHELL_DS_INITIAL_POOL_SIZE = "appshell.datasource.initialPoolSize";
private static final String JETTY_CONFIG_APPSHELL_DS_MAX_POOL_SIZE = "appshell.datasource.maxPoolSize";
private static final String JETTY_CONFIG_FAWCOMMON_SCHEMA_NAME = "fawcommon.datasource.username";
private static final String JETTY_CONFIG_FAWCOMMON_SCHEMA_PWD = "fawcommon.datasource.password";
private static final String JETTY_CONFIG_FAWCOMMON_DS_INITIAL_POOL_SIZE = "fawcommon.datasource.initialPoolSize";
private static final String JETTY_CONFIG_FAWCOMMON_DS_MAX_POOL_SIZE = "fawcommon.datasource.maxPoolSize";
private static final int INITIAL_POOL_SIZE_DEFAULT = 0;
private static final int MAX_POOL_SIZE_DEFAULT = 50;
private static final String JNDI_NAME_FAWAPPSHELL = "jdbc/CXOMetadataDatasource";
private static final String JNDI_NAME_FAWCOMMON = "jdbc/FawCommonDatasource";
private static final String JETTY_REQUEST_LOG_FILE_NAME = "/faw/logs/appshell/applogs/jetty-request.yyyy_mm_dd.log";
private static final String JETTY_REQUEST_LOG_FILE_NAME_DATE_FORMAT = "yyyy_MM_dd";
private static final boolean JETTY_REQUEST_LOG_FILE_APPEND = true;
private static final int JETTY_REQUEST_LOG_FILE_RETAIN_DAYS = 31;
private static final String JETTY_STDOUT_LOG_FILE_NAME = "/faw/logs/appshell/applogs/jetty-out.yyyy_mm_dd.log";
private static final int MAX_REQUEST_HEADER_SIZE = 65535;
private static final String SSL_SERVER_KEY_STROKE_PATH="/faw/tmp/customscript/certs/server.jks";
private static final String SSL_TRUST_KEY_STROKE_PATH="/faw/tmp/customscript/certs/trust.jks";
private static final String SSL_KEY_STROKE_KEY="changeit";
public static QueuedThreadPool threadPool;
public JettyServer() {
try {
//Redirect system out and system error to our print stream.
RolloverFileOutputStream os = new RolloverFileOutputStream(JETTY_STDOUT_LOG_FILE_NAME, true);
PrintStream logStream = new PrintStream(os);
System.setOut(logStream);
System.setErr(logStream);
Server server = new Server(JETTY_PORT);
server.addBean(getLoginService());
//Set SSL context
if (isIDCSEnvironment) {
try {
logger.info("Configuring Jetty SSL..");
HttpConfiguration http_config = new HttpConfiguration();
http_config.setSecureScheme("https");
http_config.setSecurePort(JETTY_PORT);
SslContextFactory sslContextFactory = new SslContextFactory.Server();
sslContextFactory.setKeyStorePath(SSL_SERVER_KEY_STROKE_PATH);
sslContextFactory.setCertAlias("server");
sslContextFactory.setKeyStorePassword(SSL_KEY_STROKE_KEY);
sslContextFactory.setTrustStorePath(SSL_TRUST_KEY_STROKE_PATH);
sslContextFactory.setTrustStorePassword(SSL_KEY_STROKE_KEY);
HttpConfiguration https_config = new HttpConfiguration(http_config);
https_config.addCustomizer(new SecureRequestCustomizer());
ServerConnector https = new ServerConnector(server, new SslConnectionFactory(sslContextFactory, HttpVersion.HTTP_1_1.asString()), new HttpConnectionFactory(https_config));
https.setPort(JETTY_PORT);
server.setConnectors(new Connector[]{https});
logger.info("Jetty SSL successfully configured..");
} catch (Exception e){
logger.severe("Error configuring Jetty SSL.."+e);
throw e;
}
}
Configuration.ClassList classlist = Configuration.ClassList.setServerDefault(server);
classlist.addAfter("org.eclipse.jetty.webapp.FragmentConfiguration",
"org.eclipse.jetty.plus.webapp.EnvConfiguration",
"org.eclipse.jetty.plus.webapp.PlusConfiguration");
//register oaxui and oaxservice web apps
HandlerCollection webAppHandlers = getWebAppHandlers();
for (Connector c : server.getConnectors()) {
c.getConnectionFactory(HttpConnectionFactory.class).getHttpConfiguration().setRequestHeaderSize(MAX_REQUEST_HEADER_SIZE);
c.getConnectionFactory(HttpConnectionFactory.class).getHttpConfiguration().setSendServerVersion(false);
}
threadPool = (QueuedThreadPool) server.getThreadPool();
// request logs
RequestLogHandler requestLogHandler = new RequestLogHandler();
AsyncRequestLogWriter asyncRequestLogWriter = new AsyncRequestLogWriter(JETTY_REQUEST_LOG_FILE_NAME);
asyncRequestLogWriter.setFilenameDateFormat(JETTY_REQUEST_LOG_FILE_NAME_DATE_FORMAT);
asyncRequestLogWriter.setAppend(JETTY_REQUEST_LOG_FILE_APPEND);
asyncRequestLogWriter.setRetainDays(JETTY_REQUEST_LOG_FILE_RETAIN_DAYS);
asyncRequestLogWriter.setTimeZone(TimeZone.getDefault().getID());
requestLogHandler.setRequestLog(new AppShellCustomRequestLog(asyncRequestLogWriter));
webAppHandlers.addHandler(requestLogHandler);
StatisticsHandler statisticsHandler = new StatisticsHandler();
statisticsHandler.setHandler(new AppshellStatisticsHandler());
webAppHandlers.addHandler(statisticsHandler);
// set handler
server.setHandler(webAppHandlers);
//start jettyMetricsPsr
JettyMetricStatistics.logJettyMetrics();
// set error handler
server.addBean(new CustomErrorHandler());
// GZip Handler
GzipHandler gzip = new GzipHandler();
server.setHandler(gzip);
gzip.setHandler(webAppHandlers);
//setting server attribute for datasources
server.setAttribute("fawappshellDS", new Resource(JNDI_NAME_FAWAPPSHELL, DatasourceUtil.getFawAppshellDatasource()));
server.setAttribute("fawcommonDS", new Resource(JNDI_NAME_FAWCOMMON, DatasourceUtil.getCommonDatasource()));
//new Resource(server, JNDI_NAME_FAWAPPSHELL, getFawAppshellDatasource());
//new Resource(server, JNDI_NAME_FAWCOMMON, getFawCommonDatasource());
Map<String, String> configDetails;
if (isIDCSEnvironment) {
configDetails = DatasourceUtil.getConfigMap();
} else {
configDetails = DatasourceUtil.configMapInternal;
}
if (isIDCSEnvironment && configDetails.containsKey(PODDB_ATP_ENABLED) && BooleanUtils.toBoolean(configDetails.get(PODDB_ATP_ENABLED))){
configDetails.put(DatabagProperties.LCM_MASTER_PROPERTY,"true");
try {
DBUtils.migrateDBATP(configDetails, DatasourceUtil.getFawAppshellDatasourceATP());
configDetails.remove(DatabagProperties.LCM_MASTER_PROPERTY, "true");
} catch(SQLException e){
logger.info("Exception while executing DBUtils.migrateDBATP.");
if(LCMUtils.isATPDBDown(e)) {
logger.info("Redis flow starts....Skipping consumption from Redis for now");
}
else{
logger.info("This is not eligible for redis flow. Actual Exception : "+ e);
throw e;
}
}
} else if (isIDCSEnvironment && configDetails.containsKey(PODDB_ATP_ENABLED) && !BooleanUtils.toBoolean(configDetails.get(PODDB_ATP_ENABLED))){
try{
DBUtils.migrateDB();
} catch (FawLCMPluginException e) {
logger.info(" This is not eligible for Redis consumption and exception is : " + e);
}
}
//For Dev env..
if (!isIDCSEnvironment && configDetails.containsKey(PODDB_ATP_ENABLED) && BooleanUtils.toBoolean(configDetails.get(PODDB_ATP_ENABLED))){
configDetails.put(DatabagProperties.LCM_MASTER_PROPERTY,"true");
try{
DBUtils.migrateDBATP(configDetails,DatasourceUtil.getDevFawAppshellDatasourceATP());
configDetails.remove(DatabagProperties.LCM_MASTER_PROPERTY,"true");
}
catch(SQLException e){
logger.info("Exception while executing DBUtils.migrateDBATP for dev environment.");
if(LCMUtils.isATPDBDown(e)) {
logger.info("Redis flow starts....Skipping consumption from Redis for now");
} else{
logger.info("This is not eligible for redis flow. Actual Exception for dev: "+ e);
}
}
} else if (!isIDCSEnvironment && configDetails.containsKey(PODDB_ATP_ENABLED) && !BooleanUtils.toBoolean(configDetails.get(PODDB_ATP_ENABLED))){
try {
DBUtils.migrateDB();
}catch (FawLCMPluginException e) {
logger.info(" This is not eligible for Redis consumption and exception in dev is : " + e);
}
}
server.start();
server.join();
} catch (Exception e) {
e.printStackTrace();
}
}
private HandlerCollection getWebAppHandlers() throws SQLException, NamingException{
//Setting the war and context path for the service layer: oaxservice
File tempDir1 = new File("/faw/service/appshell/tempdir/");
File tempDir2 = new File("/faw/service/appshell/tempdir/");
WebAppContext serviceWebapp = new WebAppContext();
tempDir1.mkdirs();
serviceWebapp.setTempDirectory(tempDir1);
serviceWebapp.setWar(APPSHELL_API_WAR_FILE_PATH);
serviceWebapp.setContextPath(APPSHELL_API_CONTEXT_PATH);
//setting the war and context path for the UI layer: oaxui
WebAppContext uiWebapp = new WebAppContext();
tempDir2.mkdirs();
uiWebapp.setTempDirectory(tempDir2);
uiWebapp.setWar(APPSHELL_UI_WAR_FILE_PATH);
uiWebapp.setContextPath(APPSHELL_UI_CONTEXT_PATH);
uiWebapp.setAllowNullPathInfo(true);
uiWebapp.setInitParameter("org.eclipse.jetty.servlet.Default.dirAllowed", "false");
//set error page handler for the UI context
uiWebapp.setErrorHandler(new CustomErrorHandler());
//handling the multiple war files using HandlerCollection.
HandlerCollection handlerCollection = new HandlerCollection();
handlerCollection.setHandlers(new Handler[]{serviceWebapp, uiWebapp});
return handlerCollection;
}
/**
* The name of the LoginService needs to correspond to what is configured a webapp's web.xml which is
* <realm-name>myrealm</realm-name> and since it has a lifecycle of its own, we register it as a bean
* with the Jetty server object so it can be started and stopped according to the lifecycle of the server itself.
*
* #return the login service instance
* #throws FileNotFoundException In case realmProps is null
*/
public LoginService getLoginService() throws IOException {
URL realmProps = JettyServer.class.getClassLoader().getResource(JETTY_REALM_PROPERTIES_FILE_NAME);
if (realmProps == null)
throw new FileNotFoundException("Unable to find " + JETTY_REALM_PROPERTIES_FILE_NAME);
return new HashLoginService(JETTY_REALM_NAME, realmProps.toExternalForm());
}
public static void main(String[] args) {
new JettyServer();
}
}
Run.sh file :
#!/bin/bash
set -eu # Exit on error
set -o pipefail # Fail a pipe if any sub-command fails.
VERSION=1.0
cd "$(dirname "$0")"
RED='\033[0;31m'
ORANGE='\033[0;33m'
GREEN='\033[0;32m'
NC='\033[0m' # No Color
function die() {
printf "${RED}%s\n${NC}" "$1" >&2
exit 1
}
function warn() {
printf "${ORANGE}%s\n${NC}" "$1"
}
function info() {
printf "${GREEN}%s\n${NC}" "$1"
}
CURR_PID=$$
mem_args=8192
info "mem args "$mem_args
export MAX_MEM=$mem_args
#put the managed server specific variable setting below, then export the variables
USER_MEM_ARGS="-Xms4096m -Xmx"$MAX_MEM"m -XX:MetaspaceSize=1024M -XX:MaxMetaspaceSize=1024m"
export USER_MEM_ARGS
info "USER_MEM_ARGS= ${USER_MEM_ARGS}"
#FAW_JAVA_OPTIONS="-Djava.util.logging.config.file=/faw/service/appshell/config/ucp_log.properties -Doracle.jdbc.fanEnabled=false $USER_MEM_ARGS -Xloggc:/faw/logs/appshell/applogs/gc.jetty.log -XX:+HeapDumpOnOutOfMemoryError -XshowSettings:vm -XX:+PrintCodeCache -XX:ReservedCodeCacheSize=512M -XX:+UnlockCommercialFeatures -XX:+FlightRecorder -XX:FlightRecorderOptions=maxage=30m,defaultrecording=true,stackdepth=1024,dumponexit=true,dumponexitpath=/faw/logs/jetty/jetty_1618996595.jfr -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:+ExitOnOutOfMemoryError -XX:+DisableExplicitGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=4 -XX:GCLogFileSize=5M -XX:HeapDumpPath=/faw/logs/appshell_hprof-dumps_`date +%x_%r|awk -F" " '{print $1}'|sed 's/\//_/g'`.hprof"
FAW_JAVA_OPTIONS="-Djava.util.logging.config.file=/faw/service/appshell/config/ucp_log.properties -Doracle.jdbc.fanEnabled=false $USER_MEM_ARGS -Xloggc:/faw/logs/appshell/applogs/gc.jetty.log -XX:+HeapDumpOnOutOfMemoryError -XshowSettings:vm -XX:+PrintCodeCache -XX:ReservedCodeCacheSize=512M -XX:+UnlockCommercialFeatures -XX:+FlightRecorder -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:+ExitOnOutOfMemoryError -XX:+DisableExplicitGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=4 -XX:GCLogFileSize=5M -XX:HeapDumpPath=/faw/logs/appshell_hprof-dumps_`date +%x_%r|awk -F" " '{print $1}'|sed 's/\//_/g'`.hprof"
echo "now deleting"
DELTEMP="rm -rf /tmp/jetty*"
exec $DELTEMP
echo "deleted....."
info "FAW_JAVA_OPTIONS= ${FAW_JAVA_OPTIONS}"
if [[ -e "/faw/environment" ]]; then
sh /faw/tmp/customscript/setUpTLSCerts.sh
fi
CMD="java ${FAW_JAVA_OPTIONS} -cp /faw/service/appshell/target:/faw/service/appshell/libs/* oracle.biapps.cxo.docker.jetty.server.JettyServer --lib /faw/service/appshell/libs"
ulimit -c 0
# Now execute it
info "Executing: $CMD"
exec $CMD
Any help is appreciated. Thank you.
Thank you "Joakim Erdfelt" for the input. Got it. The below solution works fine for me in embedded jetty. I will be creating a custom delete method for deleting the directory first and then creating the jetty working directory.
public void cleanJettyWorkingDirectory(){
final File folder = new File(JETTY_TEMP_WORKING_DIRECTORY);
final File[] files = folder.listFiles((dir, name) -> name.matches( "oax.*" ));
if (files!=null && files.length > 0) {
for ( final File file : files ) {
try {
FileUtils.deleteDirectory(file);
logger.info("Cleaning of the directory is completed.");
} catch (IOException e) {
logger.info("Unable to delete the Jetty working directory");
}
}
}else{
logger.info("No working directory file is present starting with oax for deletion.");
}
}
Also while setting the war , we will do below to create directories:
try {
cleanJettyWorkingDirectory();
logger.info("starting to create temp working directory for oax service ");
java.nio.file.Path oaxServicePath = Files.createTempDirectory("oaxservice");
String oaxServiceTempPath = oaxServicePath.toString();
logger.info("starting to create temp working directory for oax ui ");
java.nio.file.Path uiPath = Files.createTempDirectory("oaxui");
String oaxUIServiceTempPath = uiPath.toString();
File oaxServiceTempDir = new File(oaxServiceTempPath);
File oaxUITempDir = new File(oaxUIServiceTempPath);
serviceWebapp.setTempDirectory(oaxServiceTempDir);
uiWebapp.setTempDirectory(oaxUITempDir);
logger.info("The process of creating working directory is completed.");
} catch (IOException e) {
logger.log(Level.SEVERE, "Exception while creating directories.",e);
}
*** Update: I have changed my approach as described in my answer to the question, due to which the original issue reported becomes moot. ***
I'm trying to develop a Nifi application that provides a WebSocket interface to Kakfa. I could not accomplish this using the standard Nifi components as I have tried below (it may not make sense but intuitively this is what I want to accomplish):
I have now created a custom Processor "ReadFromKafka" that I intend to use as shown in the image below. "ReadFromKafka" would use the same implementation as the standard "PutWebSocket" component but would read messages from a Kafka Topic and send as response to the WebSocket client.
I have provided a code snippet of the implementation below:
#SystemResourceConsideration(resource = SystemResource.MEMORY)
public class ReadFromKafka extends AbstractProcessor {
public static final PropertyDescriptor PROP_WS_SESSION_ID = new PropertyDescriptor.Builder()
.name("websocket-session-id")
.displayName("WebSocket Session Id")
.description("A NiFi Expression to retrieve the session id. If not specified, a message will be " +
"sent to all connected WebSocket peers for the WebSocket controller service endpoint.")
.required(true)
.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.defaultValue("${" + ATTR_WS_SESSION_ID + "}")
.build();
public static final PropertyDescriptor PROP_WS_CONTROLLER_SERVICE_ID = new PropertyDescriptor.Builder()
.name("websocket-controller-service-id")
.displayName("WebSocket ControllerService Id")
.description("A NiFi Expression to retrieve the id of a WebSocket ControllerService.")
.required(true)
.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.defaultValue("${" + ATTR_WS_CS_ID + "}")
.build();
public static final PropertyDescriptor PROP_WS_CONTROLLER_SERVICE_ENDPOINT = new PropertyDescriptor.Builder()
.name("websocket-endpoint-id")
.displayName("WebSocket Endpoint Id")
.description("A NiFi Expression to retrieve the endpoint id of a WebSocket ControllerService.")
.required(true)
.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.defaultValue("${" + ATTR_WS_ENDPOINT_ID + "}")
.build();
public static final PropertyDescriptor PROP_WS_MESSAGE_TYPE = new PropertyDescriptor.Builder()
.name("websocket-message-type")
.displayName("WebSocket Message Type")
.description("The type of message content: TEXT or BINARY")
.required(true)
.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
.defaultValue(WebSocketMessage.Type.TEXT.toString())
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.build();
public static final Relationship REL_SUCCESS = new Relationship.Builder()
.name("success")
.description("FlowFiles that are sent successfully to the destination are transferred to this relationship.")
.build();
public static final Relationship REL_FAILURE = new Relationship.Builder()
.name("failure")
.description("FlowFiles that failed to send to the destination are transferred to this relationship.")
.build();
private static final List<PropertyDescriptor> descriptors;
private static final Set<Relationship> relationships;
static{
final List<PropertyDescriptor> innerDescriptorsList = new ArrayList<>();
innerDescriptorsList.add(PROP_WS_SESSION_ID);
innerDescriptorsList.add(PROP_WS_CONTROLLER_SERVICE_ID);
innerDescriptorsList.add(PROP_WS_CONTROLLER_SERVICE_ENDPOINT);
innerDescriptorsList.add(PROP_WS_MESSAGE_TYPE);
descriptors = Collections.unmodifiableList(innerDescriptorsList);
final Set<Relationship> innerRelationshipsSet = new HashSet<>();
innerRelationshipsSet.add(REL_SUCCESS);
innerRelationshipsSet.add(REL_FAILURE);
relationships = Collections.unmodifiableSet(innerRelationshipsSet);
}
#Override
public Set<Relationship> getRelationships() {
return relationships;
}
#Override
public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
return descriptors;
}
#Override
public void onTrigger(final ProcessContext context, final ProcessSession processSession) throws ProcessException {
final FlowFile flowfile = processSession.get();
if (flowfile == null) {
return;
}
final String sessionId = context.getProperty(PROP_WS_SESSION_ID)
.evaluateAttributeExpressions(flowfile).getValue();
final String webSocketServiceId = context.getProperty(PROP_WS_CONTROLLER_SERVICE_ID)
.evaluateAttributeExpressions(flowfile).getValue();
final String webSocketServiceEndpoint = context.getProperty(PROP_WS_CONTROLLER_SERVICE_ENDPOINT)
.evaluateAttributeExpressions(flowfile).getValue();
final String messageTypeStr = context.getProperty(PROP_WS_MESSAGE_TYPE)
.evaluateAttributeExpressions(flowfile).getValue();
final WebSocketMessage.Type messageType = WebSocketMessage.Type.valueOf(messageTypeStr);
if (StringUtils.isEmpty(sessionId)) {
getLogger().debug("Specific SessionID not specified. Message will be broadcast to all connected clients.");
}
if (StringUtils.isEmpty(webSocketServiceId)
|| StringUtils.isEmpty(webSocketServiceEndpoint)) {
transferToFailure(processSession, flowfile, "Required WebSocket attribute was not found.");
return;
}
final ControllerService controllerService = context.getControllerServiceLookup().getControllerService(webSocketServiceId);
if (controllerService == null) {
getLogger().debug("ControllerService is NULL");
transferToFailure(processSession, flowfile, "WebSocket ControllerService was not found.");
return;
} else if (!(controllerService instanceof WebSocketService)) {
getLogger().debug("ControllerService is not instance of WebSocketService");
transferToFailure(processSession, flowfile, "The ControllerService found was not a WebSocket ControllerService but a "
+ controllerService.getClass().getName());
return;
}
...
processSession.getProvenanceReporter().send(updatedFlowFile, transitUri.get(), transmissionMillis);
processSession.transfer(updatedFlowFile, REL_SUCCESS);
processSession.commit();
} catch (WebSocketConfigurationException|IllegalStateException|IOException e) {
// WebSocketConfigurationException: If the corresponding WebSocketGatewayProcessor has been stopped.
// IllegalStateException: Session is already closed or not found.
// IOException: other IO error.
getLogger().error("Failed to send message via WebSocket due to " + e, e);
transferToFailure(processSession, flowfile, e.toString());
}
}
private FlowFile transferToFailure(final ProcessSession processSession, FlowFile flowfile, final String value) {
flowfile = processSession.putAttribute(flowfile, ATTR_WS_FAILURE_DETAIL, value);
processSession.transfer(flowfile, REL_FAILURE);
return flowfile;
}
}
I have deployed the custom processor and when I connect to it using the Chrome "Simple Web Socket Client" I can see the following message in the logs:
ControllerService found was not a WebSocket ControllerService but a com.sun.proxy.$Proxy75
I'm using the exact same code as in PutWebSocket and can't figure out why it would behave any different when I use my custom Processor. I have configured "JettyWebSocketServer" as the ControllerService under "ListenWebSocket" as shown in the image below.
Additional exception details seen in the log are provided below:
java.lang.ClassCastException: class com.sun.proxy.$Proxy75 cannot be cast to class org.apache.nifi.websocket.WebSocketService (com.sun.proxy.$Proxy75 is in unnamed module of loader org.apache.nifi.nar.InstanceClassLoader #35c646b5; org.apache.nifi.websocket.WebSocketService is in unnamed module of loader org.apache.nifi.nar.NarClassLoader #361abd01)
I ended up modifying my flow to utilize out-of-box ListenWebSocket, PutWebSocket Processors, and a custom "FetchFromKafka" Processor that is a modified version of ConsumeKafkaRecord. With this I'm able to provide a WebSocket interface to Kafka. I have provided a screenshot of the updated flow below. More work needs to be done with the custom Processor to support multiple sessions.
I have a folder structure /data/reports on a file system, which contains all reports.
How can I configure a SpringBoot application to serve the contents of this file sytem.
Currently I have tried few options, but none working
#Configuration
#EnableWebMvc
public class AppConfig implements WebMvcConfigurer {
#Value(value = "${spring.resources.static-locations:#{null}}")
private String fileSystem;
#Override
public void addResourceHandlers(ResourceHandlerRegistry registry) {
registry
.addResourceHandler("/data/reports/**")
.addResourceLocations(fileSystem)
.setCachePeriod(3600)
.resourceChain(true)
.addResolver(new PathResourceResolver());
}
}
and in application.properties I have defined
spring.resources.static-locations=file:///data/reports
server.servlet.jsp.init-parameters.listings=true
But in both cases, when I try
http://host:port/application/data/reports
I'm getting 404
What am I missing ?
Based on the suggestions given, I realized that one mistake I'm doing is to access the reports via
http://host:port/application/data/reports
instead of
http://host:port/data/reports
if I use application in the request, those calls will go through RequestDispatcher and will try to find for a matching RequestMapping, which does not exist. I think I'm convinced so far.
But the problem I'm seeing now is, I'm getting SocketTimeoutException while trying to read from the resource listed in the URL. I had put some breakpoints in Spring source "ResourceHttpMessageConverter.java"
protected void writeContent(Resource resource, HttpOutputMessage outputMessage)
throws IOException, HttpMessageNotWritableException {
try {
InputStream in = resource.getInputStream(); //It is timing out here
try {
StreamUtils.copy(in, outputMessage.getBody());
}
catch (NullPointerException ex) {
// ignore, see SPR-13620
}
The resource is a small text file with 1 line "Hello World". Yet it is timing out.
The resource in the above class is a FileUrlResource opened on file:///c:/data/reports/sample.txt
On the other hand, I tried to read that resource as
File file = new File("c:/data/reports/sample.txt");
System.out.println(file.exists());
URL url = file.toURI().toURL();
URLConnection con = url.openConnection();
InputStream is = con.getInputStream(); //This works
Thanks
So I've created a batch job which generates reports (csv files). I have been able to generate the the files seamlessly using FlatFileItemWriter but my end goal is to create an InputStream to call a rest service which will store the document or a byte array to store it in the database.
public class CustomWriter implements ItemWriter<Report> {
#Override
public void write(List<? extends Report> reports) throws Exception {
reports.forEach(this::writeDataToFile);
}
private void writeDataToFile(final Report data) throws Exception {
FlatFileItemWriter writer = new FlatFileItemWriter();
writer.setResource(new FileSystemResource("C:/reports/test-report.csv"));
writer.setLineAggregator(getLineAggregator();
writer.afterPropertiesSet();
writer.open(new ExecutionContext());
writer.write(data);
writer.close();
}
private DelimitedLineAggregator<Report> getLineAggregator(final Report report) {
DelimitedLineAggregator<Report> delimitedLineAgg = new DelimitedLineAggregator<Report>();
delimitedLineAgg.setDelimiter(",");
delimitedLineAgg.setFieldExtractor(getFieldExtractor());
return delimitedLineAgg;
}
private FieldExtractor<Report> getFieldExtractor() {
BeanWrapperFieldExtractor<Report> fieldExtractor = new BeanWrapperFieldExtractor<Report>();
fieldExtractor.setNames(COLUMN_HEADERS.toArray(new String[0]));
return fieldExtractor;
}
}
One way I could do this is to store the file locally temporarily and create a new step to pick the generated files up and do the sending/storing but I would really like to skip this step and send/store it in the first step.
How do I go about doing this?
I am trying to use Spring Integration with Spring batch.I am transforming my message Message to JobLaunchRequest
public class FileMessageToJobRequest {
private Job job;
private String fileParameterName;
public void setFileParameterName(String fileParameterName) {
this.fileParameterName = fileParameterName;
}
public void setJob(Job job) {
this.job = job;
}
public JobLaunchRequest toRequest(Message<File> message) throws IOException {
JobParametersBuilder jobParametersBuilder = new JobParametersBuilder();
String fileName = message.getPayload().getName();
String currentDateTime = LocalDateTime.now().toString();
logger.info("currentDateTime"+currentDateTime);
jobParametersBuilder.addString(this.fileName, fileName);
jobParametersBuilder.addString(this.currentDateTime, currentDateTime);
return new JobLaunchRequest(job, jobParametersBuilder.toJobParameters());
}
}
here is my Integration flow
return IntegrationFlows.
from(Files.inboundAdapter(new File("landing"))
.preventDuplicates() .patternFilter("*.txt")
, e -> e.poller(Pollers.fixedDelay(2500) .maxMessagesPerPoll(15)
.taskExecutor(getFileProcessExecutor())))
.transform(fileMessageToJobRequest)
.handle(jobLaunchingGw(jobLauncher)).get();
#Bean
public MessageHandler jobLaunchingGw(JobLauncher jobLauncher)
{
return new JobLaunchingGateway(jobLauncher);
}
For some reason one of the files of the two that print the same time is not getting processed if two files print the same currentDateTime.The files are most likely to come at the same time.I am using ThreadPoolTaskExecutor corepool size,max pool size ,queue capacity is 15
String currentDateTime = LocalDateTime.now().toString();
the file is not getting processed.I am assuming this is some kind of race condition.I even tried synchronized on the JobLaunchRequest.Is this a race condition?Why would a message drop?
I am using Spring Integration 4.2.6 all latest versions(almost) of Spring.
My reference for this config is this question
Java DSL config