Spring boot SFTP, dynamic directory in SFTP - spring-boot

I tried to upload files to dynamic directory to SFTP. When I uploaded some files, the first file always uploaded to the last directory. Then after that rest file will be uploaded to the correct directory. When I did debug mode, I saw that every first file would be uploaded to temporaryDirectory which is the code already set up by spring. I don't know how to set the value of this temporaryDirectory to the right value. Please, help me to solve the problem.
Or maybe you guys have other way to upload and create proper dynamic directory. Please let me know.
Here is the code:
private String sftpRemoteDirectory = "documents/"
#MessagingGateway
public interface UploadGateway {
#Gateway(requestChannel = "toSftpChannel")
void upload(File file, #Header("dirName") String dirName);
}
#Bean
#ServiceActivator(inputChannel = "toSftpChannel")
public MessageHandler handler() {
SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory());
SimpleDateFormat formatter = new SimpleDateFormat("yyMMdd");
String newDynamicDirectory = "E" + formatter.format(new Date())+String.format("%04d",Integer.parseInt("0001") + 1);
handler.setRemoteDirectoryExpression(new LiteralExpression(sftpRemoteDirectory + newDynamicDirectory));
handler.setFileNameGenerator(message -> {
String dirName = (String) message.getHeaders().get("dirName");
handler.setRemoteDirectoryExpression(new LiteralExpression(sftpRemoteDirectory + dirName));
handler.setAutoCreateDirectory(true);
if (message.getPayload() instanceof File) {
return (((File) message.getPayload()).getName());
} else {
throw new IllegalArgumentException("File expected as payload!");
}
});
return handler;
}

You are using a LiteralExpression, evaluated just once, you need an expression that's evaluated at runtime.
handler.setRemoteDirectoryExpressionString("'" + sftpRemoteDirectory/ + "'" + headers['dirName']);

Related

Can I use SpringBatch to transfer file(video/txt) present of http and read it and write it on my pc?

Right now i am trying to write an http file(sample:"https://www.w3.org/TR/PNG/iso_8859-1.txt") to my local pc using springBatch. But unable to read the file.
My Reader file is as follows:
public FlatFileItemReader<String> reader(){
FlatFileItemReader<String> reader = new FlatFileItemReader<String>();
reader.setResource(new UrlResource("https://www.w3.org/TR/PNG/iso_8859-1.txt"));
reader.setRecordSeparatorPolicy(new RecordSeparatorPolicy() {
#Override
public boolean isEndOfRecord(String s) {
if (s.length() == 100)
return true;
return false;
}
#Override
public String postProcess(String s) {
return s;
}
#Override
public String preProcess(String s) {
return s;
}
});
return reader;
}
I am reading 100 length chunks from the input and passing it to writer. But i am getting this error:
org.springframework.batch.item.file.FlatFileParseException: Unexpected end of file before record complete
at org.springframework.batch.item.file.FlatFileItemReader.applyRecordSeparatorPolicy(FlatFileItemReader.java:303) ~[spring-batch-infrastructure-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.batch.item.file.FlatFileItemReader.readLine(FlatFileItemReader.java:220) ~[spring-batch-infrastructure-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.batch.item.file.FlatFileItemReader.doRead(FlatFileItemReader.java:178) ~[spring-batch-infrastructure-4.2.4.RELEASE.jar:4.2.4.RELEASE]
...
How to read complete .txt file from http and process and write it to desire location?
Also what if instead of txt file, I will pass video file to reader(What should be FlatFileItemReader<???> type in this case?
You don't need that custom record separator policy, you can use a PassThroughLineMapper instead.
Also what if instead of txt file, I will pass video file to reader(What should be FlatFileItemReader type in this case?
You can use byte[] as item type or use the SimpleBinaryBufferedReaderFactory.

spring cloud stream file source app - History of Processed files and polling files under sub directory

I'm building a data pipeline with Spring Cloud Stream File Source app at the start of the pipeline. I need some help with working around some missing features
My File source app (based on org.springframework.cloud.stream.app:spring-cloud-starter-stream-source-file) works perfectly well excepting missing features that I need help with. I need
To delete files after polled and messaged
Poll into the subdirectories
With respect to item 1, I read that the delete feature doesn't exist in the file source app (it is available on sftp source). Every time the app is restarted, the files that were processed in the past will be re-picked, can the history of files processed made permanent? Is there an easy alternative?
To support those requirements you definitely need to modify the code of the mentioned File Source project: https://docs.spring.io/spring-cloud-stream-app-starters/docs/Einstein.BUILD-SNAPSHOT/reference/htmlsingle/#_patching_pre_built_applications
I would suggest to fork the project and poll it from GitHub as is, since you are going to modify existing code of the project. Then you follow instruction in the mentioned doc how to build the target binder-specific artifact which will be compatible with SCDF environment.
Now about the questions:
To poll sub-directories for the same file pattern, you need to configure a RecursiveDirectoryScanner on the Files.inboundAdapter():
/**
* Specify a custom scanner.
* #param scanner the scanner.
* #return the spec.
* #see FileReadingMessageSource#setScanner(DirectoryScanner)
*/
public FileInboundChannelAdapterSpec scanner(DirectoryScanner scanner) {
Note that all the filters must be configured on this DirectoryScanner instead.
There is going to be a warning otherwise:
// Check that the filter and locker options are _NOT_ set if an external scanner has been set.
// The external scanner is responsible for the filter and locker options in that case.
Assert.state(!(this.scannerExplicitlySet && (this.filter != null || this.locker != null)),
() -> "When using an external scanner the 'filter' and 'locker' options should not be used. " +
"Instead, set these options on the external DirectoryScanner: " + this.scanner);
To keep track of the files, it is better to consider to have a FileSystemPersistentAcceptOnceFileListFilter based on the external persistence store for the ConcurrentMetadataStore implementation: https://docs.spring.io/spring-integration/reference/html/#metadata-store. This must be used instead of that preventDuplicates(), because FileSystemPersistentAcceptOnceFileListFilter ensure only once logic for us as well.
Deleting file after sending might not be a case, since you may just send File as is and it is has to be available on the other side.
Also, you can add a ChannelInterceptor into the source.output() and implement its postSend() to perform ((File) message.getPayload()).delete(), which is going to happen when the message has been successfully sent to the binder destination.
#EnableBinding(Source.class)
#Import(TriggerConfiguration.class)
#EnableConfigurationProperties({FileSourceProperties.class, FileConsumerProperties.class,
TriggerPropertiesMaxMessagesDefaultUnlimited.class})
public class FileSourceConfiguration {
#Autowired
#Qualifier("defaultPoller")
PollerMetadata defaultPoller;
#Autowired
Source source;
#Autowired
private FileSourceProperties properties;
#Autowired
private FileConsumerProperties fileConsumerProperties;
private Boolean alwaysAcceptDirectories = false;
private Boolean deletePostSend;
private Boolean movePostSend;
private String movePostSendSuffix;
#Bean
public IntegrationFlow fileSourceFlow() {
FileInboundChannelAdapterSpec messageSourceSpec = Files.inboundAdapter(new File(this.properties.getDirectory()));
RecursiveDirectoryScanner recursiveDirectoryScanner = new RecursiveDirectoryScanner();
messageSourceSpec.scanner(recursiveDirectoryScanner);
FileVisitOption[] fileVisitOption = new FileVisitOption[1];
recursiveDirectoryScanner.setFilter(initializeFileListFilter());
initializePostSendAction();
IntegrationFlowBuilder flowBuilder = IntegrationFlows
.from(messageSourceSpec,
new Consumer<SourcePollingChannelAdapterSpec>() {
#Override
public void accept(SourcePollingChannelAdapterSpec sourcePollingChannelAdapterSpec) {
sourcePollingChannelAdapterSpec
.poller(defaultPoller);
}
});
ChannelInterceptor channelInterceptor = new ChannelInterceptor() {
#Override
public void postSend(Message<?> message, MessageChannel channel, boolean sent) {
if (sent) {
File fileOriginalFile = (File) message.getHeaders().get("file_originalFile");
if (fileOriginalFile != null) {
if (movePostSend) {
fileOriginalFile.renameTo(new File(fileOriginalFile + movePostSendSuffix));
} else if (deletePostSend) {
fileOriginalFile.delete();
}
}
}
}
//Override more interceptor methods to capture some logs here
};
MessageChannel messageChannel = source.output();
((DirectChannel) messageChannel).addInterceptor(channelInterceptor);
return FileUtils.enhanceFlowForReadingMode(flowBuilder, this.fileConsumerProperties)
.channel(messageChannel)
.get();
}
private void initializePostSendAction() {
deletePostSend = this.properties.isDeletePostSend();
movePostSend = this.properties.isMovePostSend();
movePostSendSuffix = this.properties.getMovePostSendSuffix();
if (deletePostSend && movePostSend) {
String errorMessage = "The 'delete-file-post-send' and 'move-file-post-send' attributes are mutually exclusive";
throw new IllegalArgumentException(errorMessage);
}
if (movePostSend && (movePostSendSuffix == null || movePostSendSuffix.trim().length() == 0)) {
String errorMessage = "The 'move-post-send-suffix' is required when 'move-file-post-send' is set to true.";
throw new IllegalArgumentException(errorMessage);
}
//Add additional validation to ensure the user didn't configure a file move that will result in cyclic processing of file
}
private FileListFilter<File> initializeFileListFilter() {
final List<FileListFilter<File>> filtersNeeded = new ArrayList<FileListFilter<File>>();
if (this.properties.getFilenamePattern() != null && this.properties.getFilenameRegex() != null) {
String errorMessage = "The 'filename-pattern' and 'filename-regex' attributes are mutually exclusive.";
throw new IllegalArgumentException(errorMessage);
}
if (StringUtils.hasText(this.properties.getFilenamePattern())) {
SimplePatternFileListFilter patternFilter = new SimplePatternFileListFilter(this.properties.getFilenamePattern());
if (this.alwaysAcceptDirectories != null) {
patternFilter.setAlwaysAcceptDirectories(this.alwaysAcceptDirectories);
}
filtersNeeded.add(patternFilter);
} else if (this.properties.getFilenameRegex() != null) {
RegexPatternFileListFilter regexFilter = new RegexPatternFileListFilter(this.properties.getFilenameRegex());
if (this.alwaysAcceptDirectories != null) {
regexFilter.setAlwaysAcceptDirectories(this.alwaysAcceptDirectories);
}
filtersNeeded.add(regexFilter);
}
FileListFilter<File> createdFilter = null;
if (!Boolean.FALSE.equals(this.properties.isIgnoreHiddenFiles())) {
filtersNeeded.add(new IgnoreHiddenFileListFilter());
}
if (Boolean.TRUE.equals(this.properties.isPreventDuplicates())) {
filtersNeeded.add(new AcceptOnceFileListFilter<File>());
}
if (filtersNeeded.size() == 1) {
createdFilter = filtersNeeded.get(0);
} else {
createdFilter = new CompositeFileListFilter<File>(filtersNeeded);
}
return createdFilter;
}
}

eclipse scout image change

I am trying to change image inside Image view.
I know that getTestImageField().setImageId(Icons.Logo); would not work, because it would not refresh renderer.
Because I need to use setImage(), I need a way to get Image from Icons class.
As Patrick suggested I try
final IconProviderService provider = SERVICES.getService(IconProviderService.class);
final IconSpec ic = provider.getIconSpec(AbstractIcons.StatusError);
final byte[] content = ic.getContent();
but my problem is that ic is always null.
While I debug this I notice that inside IconProviderService.class in line 57 :
#Override
protected URL findResource(String fullPath) {
URL[] entries = FileLocator.findEntries(m_hostBundle, new Path(fullPath));
if (entries != null && entries.length > 0) {
URL url = entries[entries.length - 1];
if (LOG.isDebugEnabled()) {
LOG.debug("find image " + fullPath + " in bundle " + m_hostBundle.getSymbolicName() + "->" + url);
}
return url;
}
return null;
}
URL[] entries is always empty no matter witch icon I try to present.
After further debugging I found out that FileLocator tries to find fragments from bundle, and then look for the path inside this fragments. (line 242)
Bundle[] fragments = activator.getFragments(b);
but Bundle[] fragments is always null.
Normally my bundle b is (Bundle) EquinoxBundle : org.eclipse.scout.rt.ui.rap.mobile_4.0.100.20140829-1424.
I want to try with different bundle so I do :
final BundleContext context = Activator.getDefault().getBundle().getBundleContext();
for (final Bundle b : context.getBundles()) {
final IconProviderService provider = SERVICES.getService(IconProviderService.class);
provider.setHostBundle(b);
final IconSpec ic = provider.getIconSpec(AbstractIcons.StatusError);
if (ic != null) {
final byte[] content = ic.getContent();
imageField().setImage(content);
}
}
but fragments (from above code) is always null.
You can obtain the image content (byte[]) that you can set on the image field as follows:
IconProviderService provider = SERVICES.getService(IconProviderService.class);
byte[] content = provider.getIconSpec(Icons.YourIconName).getContent();
getImageField().setImage(content);
I quickly checked it and it works for me.
Please ensure that the icon is available and you set up the icon provider service as explained in this Wiki Article

Upload a image in a porltet Liferay

I am doing a portlet to create banners. I preferences I made the form with: input type="file" and the form nctype='multipart/form-data'
In the processAction I get the image, but I don't know how save it in the server, because I only get save in temporal instance portlet, but if I restart the server I lose the image.
This is my code to save the image:
private boolean uploadFile( ActionRequest request, ActionResponse response) throws ValidatorException, IOException, ReadOnlyException {
try {
// Si la request es del tipo multipart ...
if (PortletFileUpload.isMultipartContent(request)) {
DiskFileItemFactory diskFileItemFactory = new DiskFileItemFactory();
PortletFileUpload servletFileUpload = new PortletFileUpload(diskFileItemFactory);
servletFileUpload.setSizeMax(81920); // bytes
List fileItemsList = servletFileUpload.parseRequest(request);
Iterator it = fileItemsList.iterator();
while (it.hasNext()){
FileItem fileItem = (FileItem)it.next();
if (fileItem.isFormField()){
}
else{
String nombreCampo = fileItem.getFieldName();
String nombreArchivo = fileItem.getName();
String extension = nombreArchivo.substring(nombreArchivo.indexOf("."));
PortletContext context = request.getPortletSession().getPortletContext();
String path = context.getRealPath("/images");
File archivo = new File(path + "/" + nombreArchivo);
PortletContext pc = request.getPortletSession().getPortletContext();
fileItem.write(archivo);
}
}
}
} catch (Exception e) {}
return true;
}
I don't know if I am doing something wrong or this isn't the correct way.
Any idea?
Thanks in advance
EDIT:
Finally I tried do it with DLFolderLocalServiceUtil and DLFileEntryLocalServiceUtil, but it doesn't work correctly. When I load the page you can see the image, but after, when the page is load completely, the image disappears.
I don't know if it is because I don't create fine the fileEntry or the url is wrong.
This is my code:
long folderId = CounterLocalServiceUtil.increment(DLFolder.class.getName());
DLFolder folder = DLFolderLocalServiceUtil.createDLFolder(folderId);
long userId = themeDisplay.getUserId();
long groupId = themeDisplay.getScopeGroupId();
folder.setUserId(userId);
folder.setGroupId(groupId);
folder.setName("Banner image " + nombreArchivo+String.valueOf(folderId));
DLFolderLocalServiceUtil.updateDLFolder(folder);
ServiceContext serviceContext= ServiceContextFactory.getInstance(DLFileEntry.class.getName(), request);
File myfile = new File(nombreArchivo);
fileItem.write(myfile);
List<DLFileEntryType> tip = DLFileEntryTypeLocalServiceUtil.getFileEntryTypes(DLUtil.getGroupIds(themeDisplay));
DLFileEntry DLfileEntry = DLFileEntryLocalServiceUtil.addFileEntry(userId, groupId, 0, folderId, null, MimeTypesUtil.getContentType(myfile), nombreArchivo, "Image banner_"+nombreArchivo, "", tip.get(0).getFileEntryTypeId(), null, myfile, fileItem.getInputStream(), myfile.getTotalSpace(), serviceContext);
FileVersion fileVersion = null;
//FileEntry fileEntry = DLAppServiceUtil.getFileEntry(groupId, folderId, nombreArchivo);
//String path = DLUtil.getPreviewURL(fileEntry, fileVersion, themeDisplay, "&imagePreview=1");
String path1 = themeDisplay.getPortalURL()+"/c/document_library/get_file?uuid="+DLfileEntry.getUuid()+"&groupId="+themeDisplay.getScopeGroupId();
String path = "/documents/" + DLfileEntry.getGroupId() + "/" + DLfileEntry.getFolderId() + "/" + DLfileEntry.getTitle()+"/"+DLfileEntry.getUuid();
System.out.println("path " + path);
System.out.println("path " + path1);
prefs.setValue(nombreCampo, path);
And this is the output:
path /documents/10180/0/cinesa888.png/f24e6da2-0be8-47ad-a3b5-a4ab0d41d17f
path http://localhost:8080/c/document_library/get_file?uuid=f24e6da2-0be8-47ad-a3b5-a4ab0d41d17f&groupId=10180
I tried to get the url like lpratlong said (DLUtil) but when I tried to get the FileEntry with DLAppServiceUtil.getFileEntry(..) I have an error that says no exist FileEntry.
I don't know what I am doing wrong.. Any idea?
Thanks.
You can use Liferay API to store the file in the Document Library : take a look in DLFolder and DLFileEntry API (for exemple, DLFileEntryLocalServiceUtil will show you allowed local operations).
These API will allowed you to store your file in your file system (in the "data" folder of your Liferay installation) and to store reference of your file in Liferay database.

How to set the default namespace or how to define the #key values when using google-api and using Atom serialization/parser

I'm having trouble with Atom parsing/serializing - clearly something related to the namespace and the default alias - but I can;t figure out what I'm doing wrong.
I have two methods - one that I'm trying to do a GET and see if an album is defined and what that tries to do a POST to create the album (if it does not exist).
The GET I managed to get working - although there too I'm pretty sure I am doing something wrong because it is different from the PicasaAndroidSample. Specifically, if I define:
public class EDAlbum {
#Key("atom:title")
public String title;
#Key("atom:summary")
public String summary;
#Key("atom:gphoto:access")
public String access;
#Key("atom:category")
public EDCategory category = EDCategory.newKind("album");
}
Then the following code does indeed get all the albums:
PicasaUrl url = PicasaUrl.relativeToRoot("feed/api/user/default");
HttpRequest request = EDApplication.getRequest(url);
HttpResponse res = request.execute();
EDAlbumFeed feed = res.parseAs(EDAlbumFeed.class);
boolean hasEDAlbum = false;
for (EDAlbum album : feed.items) {
if (album.title.equals(EDApplication.ED_ALBUM_NAME)) {
hasEDAlbum = true;
break;
}
}
But - if instead I have:
public class EDAlbum {
#Key("title")
public String title;
#Key("summary")
public String summary;
#Key("gphoto:access")
public String access;
#Key("category")
public EDCategory category = EDCategory.newKind("album");
}
Then the feed has an empty collection - i.e. the parser does not know that this is Atom (my guess).
I can live with the android:title in my classes - I don;t get it, but it works.
The problem is that I can't get the POST to wok (to create the album). This code is:
EDAlbum a = new EDAlbum();
a.access = "public";
a.title = EDApplication.ED_ALBUM_NAME;
a.summary = c.getString(R.string.ed_album_summary);
AtomContent content = new AtomContent();
content.entry = a;
content.namespaceDictionary = EDApplication.getNamespaceDictionary();
PicasaUrl url = PicasaUrl.relativeToRoot("feed/api/user/default");
HttpRequest request = EDApplication.postRequest(url, content);
HttpResponse res = request.execute();
The transport and namespace are:
private static final HttpTransport transport = new ApacheHttpTransport(); // my libraries don;t include GoogleTransport.
private static HttpRequestFactory createRequestFactory(final HttpTransport transport) {
return transport.createRequestFactory(new HttpRequestInitializer() {
public void initialize(HttpRequest request) {
AtomParser parser = new AtomParser();
parser.namespaceDictionary = getNamespaceDictionary();
request.addParser(parser);
}
});
}
public static XmlNamespaceDictionary getNamespaceDictionary() {
if (nsDictionary == null) {
nsDictionary = new XmlNamespaceDictionary();
nsDictionary.set("", "http://www.w3.org/2005/Atom");
nsDictionary.set("atom", "http://www.w3.org/2005/Atom");
nsDictionary.set("exif", "http://schemas.google.com/photos/exif/2007");
nsDictionary.set("gd", "http://schemas.google.com/g/2005");
nsDictionary.set("geo", "http://www.w3.org/2003/01/geo/wgs84_pos#");
nsDictionary.set("georss", "http://www.georss.org/georss");
nsDictionary.set("gml", "http://www.opengis.net/gml");
nsDictionary.set("gphoto", "http://schemas.google.com/photos/2007");
nsDictionary.set("media", "http://search.yahoo.com/mrss/");
nsDictionary.set("openSearch", "http://a9.com/-/spec/opensearch/1.1/");
nsDictionary.set("xml", "http://www.w3.org/XML/1998/namespace");
}
return nsDictionary;
}
If I use
#Key("title")
public String title;
then I get an exception that it does not have a default namespace:
W/System.err( 1957): java.lang.IllegalArgumentException: unrecognized alias: (default)
W/System.err( 1957): at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
W/System.err( 1957): at com.google.api.client.xml.XmlNamespaceDictionary.getNamespaceUriForAliasHandlingUnknown(XmlNamespaceDictionary.java:288)
W/System.err( 1957): at com.google.api.client.xml.XmlNamespaceDictionary.startDoc(XmlNamespaceDictionary.java:224)
and if I use
#Key("atom:title")
public String title;
then it does serialize but each element has the atom: prefix and the call fails - when I to a tcpdump on it I see something like
.`....<? xml vers
ion='1.0 ' encodi
ng='UTF- 8' ?><at
om:entry xmlns:a
tom="htt p://www.
w3.org/2 005/Atom
"><atom: category
scheme= "http://
schemas. google.c
om/g/200 5#kind"
term="ht tp://sch
emas.goo gle.com/
photos/2 007#albu
m" /><at om:gphot
o:access >public<
/atom:gp hoto:acc
....
What do I need to do different in order to use
#Key("title")
public String title;
and have both the GET and the POST manage the namespace?
It looks you are adding either duplicate dictonary keys or keys that are not understood by the serializer.
Use the following instead.
static final XmlNamespaceDictionary DICTIONARY = new XmlNamespaceDictionary()
.set("", "http://www.w3.org/2005/Atom")
.set("activity", "http://activitystrea.ms/spec/1.0/")
.set("georss", "http://www.georss.org/georss")
.set("media", "http://search.yahoo.com/mrss/")
.set("thr", "http://purl.org/syndication/thread/1.0");
Removing the explicit set for the "atom" item namespace solved this issue for me.

Resources