We are running Java-Cucumber-Maven automated test scripts from TeamCity. Up to now this has worked very well but our current test requires that several .CSV files are downloaded for analysis by the script. There's a
ReadCsvFiles.java
file in our code and that dictates where the downloaded files are located....easy for running tests locally but we're having real problems with the path to the build agent in TeamCity..
public class ReadCsvFiles {
//public static String sFileLocation = "C:\\Users\\ankit.bisht\\Downloads\\";
public static String sFileLocation = "C:\\BuildAgent\\temp\\buildTmp\\";
//public static String sFileLocation = "//users/redmayned//Downloads/";
public static void depositsHeldReports() throws Throwable {
String csvFile = sFileLocation + "DepositsHeld.csv";
CSVReader reader = null;
try {
reader = new CSVReader(new FileReader(csvFile));
String[] column;
boolean doesExist = false;
while ((column = reader.readNext()) != null) {
The commented out entries are for local windows and local OSX. The middile one is just one of many options we've tried...all resulting in an error message;
java.io.FileNotFoundException: C:\BuildAgent\temp\buildTmp\DepositsHeld.csv (The system cannot find the file specified)
Can anyone save my sanity please?
Thanks
David
Related
I want to test my simple class that connected to public Ftp and get list of fileNames from there.
And i found very strange issue - navigation on real server and on test conteiner with same dirs are not same
there is a dir structure
Well if i connect to real server and use ftp.changeWorkingDirectory("/out/published")
everything is good - directory changed and i retrive file from there
if i trying to do same in testconteiner ftp.changeWorkingDirectory("/out/published") it return false ( cant find this directory) and if i remove slash like this ftp.changeWorkingDirectory("out/published") it will work
and second issue if i will do ftp.listnames("/out/published/tverskaya_obl/purchaseNotice/daily) on real server it will work good, but in testconteiner it work only if i use this
ftp.listNames("tverskaya_obl/purchaseNotice/daily"
)
is there any way to fix it
here is the code to create testcontainer and folders
#TestInstance(TestInstance.Lifecycle.PER_CLASS)
#SpringBootTest(properties = "ftp.host = localhost")
class PurchaseFZ223FtpServiceTest {
private static final int PORT = 21;
private static final String USER = "";
private static final String PASSWORD = "";
private static final int PASSIVE_MODE_PORT = 21000;
#Autowired
FtpService ftpService;
private static final FixedHostPortGenericContainer ftp = new FixedHostPortGenericContainer<>(
"delfer/alpine-ftp-server:latest")
.withFixedExposedPort(PASSIVE_MODE_PORT, PASSIVE_MODE_PORT)
.withExposedPorts(PORT)
.withEnv("USERS", USER + "|" + PASSWORD)
.withEnv("MIN_PORT", String.valueOf(PASSIVE_MODE_PORT))
.withEnv("MAX_PORT", String.valueOf(PASSIVE_MODE_PORT));
#BeforeAll
void init() throws NoSuchFieldException, IllegalAccessException, IOException {
ftp.start();
FTPClient client =new FTPClient();
client.connect("localhost", ftp.getMappedPort(PORT));
client.enterLocalPassiveMode();
client.login(USER, PASSWORD);
client.makeDirectory("out");
client.makeDirectory("out/published");
client.makeDirectory("out/published/tverskaya_obl");
client.makeDirectory("out/published/tverskaya_obl/purchaseNotice");
client.makeDirectory("out/published/tverskaya_obl/purchaseNotice/daily");
client.changeWorkingDirectory("out/published");
client.storeFile("list_regions.txt", new ByteArrayInputStream("tverskaya_obl".getBytes(StandardCharsets.UTF_8)));
client.changeWorkingDirectory("tverskaya_obl/purchaseNotice/daily");
client.storeFile("purchaseNotice_Tverskaya_obl_20221209_000000_20221209_235959_daily_001.xml.txt", new ByteArrayInputStream("test".getBytes(StandardCharsets.UTF_8)));
}
Finally found what is going wrong
When testconteiner create ftp server it just add two folders like this
ftp/yourLoginName
so my mistake was to ask list of file name with invalid path
valid path for me to get list of fileNames is /ftp/***/out/published/tverskaya_obl/purchaseNotice/daily
same with change working dir
if you dnt know how to check real path you are at just do this
ftp.printWorkingDirectory()
I just deployed my first microservice. My microservice is working fine. All routes are working. But the service class inside the microservice is not working properly. The service class is not reading data from the CSV file.
Below is the code I am using to read data from CSV file.
public class ReadCsvUtil {
private static final Logger LOGGER = LoggerFactory.getLogger(ReadCsvUtil.class);
public List<String[]> readData() throws IOException {
String file = ".\\src\\main\\resources\\pensioners.csv";
List<String[]> content = new ArrayList<>();
try(BufferedReader br = new BufferedReader(new FileReader(file))) {
String line = "";
while ((line = br.readLine()) != null) {
content.add(line.split(","));
}
} catch (FileNotFoundException e) {
LOGGER.debug(e.getMessage());
}
return content;
}
}
The service class invokes the above function to get details of all the people.
The above code is working fine on my desktop and I am able to get details but code is not working on AWS. Also, I tried to remove the CSV and manually enter the values and it's working in AWS. So I am 99% sure there is some problem in reading the CSV files.
Is there anyway I can fix this?
If the path to the file one directory up, instead of
String file = ".\\src\\main\\resources\\pensioners.csv";
try,
String file = "..\src\main\resources\pensioners.csv";
I presume the AWS sever is ubuntu and your local is windows OS.
I am building a spring boot application where i need to read json files for my component tests. I have a utility method which takes the name of the file and reads the content using ResourceUtils. Here is the code:
public static String getContent(String path) throws IOException {
File file = ResourceUtils.getFile(MyTest.class.getResource(path));
String content = new String(Files.readAllBytes(file.toPath()));
return content;
}
The checkmarx is reporting the above code as "This may cause a Path
Traversal vulnerability."
How to fix this?
Thanks
See this example for path traversal vulnerability Path Traversal
To fix this change it something like
private static final String BASE_PATH ="/yourbasepath/somewherewherefileisstored";
public static String getContent(String path) throws IOException {
File file = new File(BASE_PATH, path);
if (file.getCanonicalPath().startsWith(BASE_PATH)){
String content = new String(Files.readAllBytes(file.toPath()));
return content;
}
else{
//throw some error
}
}
I have an hdfs path hdfs://host1:8899/path/to/file. I want to strip the host1 and port programmatically. As result, it should be hdfs:/path/tofile. Is there any helper method can do that?
"Is there any helper method can do that?"
Doesn't really take much to create your own. Just use the basic String class utility functions like split(), indexOf(), substring(), etc.
Something like this would do (with Java, though most languages have those methods):
public class TestPath {
public static void main(String[] args) throws Exception {
String path = "hdfs://localhost:9000/path/to/file";
System.out.println(getPathWithoutHostAndPort(path));
}
public static String getPathWithoutHostAndPort(String path) {
String[] array = path.split("(//)");
int indexOfFirstSlash = array[1].indexOf("/");
StringBuilder builder = new StringBuilder();
builder.append(array[0]).append(array[1].substring(indexOfFirstSlash));
return builder.toString();
}
}
Result: hdfs:/path/to/file
I have a simple console app as below (un-relevant code removed for simplicity)
[ImportMany(typeof(ILogger))]
public IEnumerable<ILogger> _loggers {get;set;}
public interface ILogger
{
void Write(string message);
}
[Export(typeof(ILogger))]
public class ConsoleLogger : ILogger
{
public void Write(string message)
{
Console.WriteLine(message);
}
}
[Export(typeof(ILogger))]
public class DebugLogger : ILogger
{
public void Write(string message)
{
Debug.Print(message);
}
}
The code that initialize the catalog is below
(1) var catalog = new AggregateCatalog();
(2) catalog.Catalogs.Add(new DirectoryCatalog(AppDomain.CurrentDomain.BaseDirectory));
(3) //var catalog = new AssemblyCatalog(Assembly.GetExecutingAssembly());
var container = new CompositionContainer(catalog);
var batch = new CompositionBatch();
batch.AddPart(this);
container.Compose(batch);
If the catalog is initialized trough lines 1-2, nothing got loaded into _logger
If the catalog is initialized trough line 3, both logger got loaded into _logger
What's the issue with AggregateCatalog approach?
Thanks
It should work the way you are using it.
However, on line 2 you are creating a DirectoryCatalog, and on line 3 an AssemblyCatalog. Does it work as expected if you change line two into
catalog.Catalogs.Add(new AssemblyCatalog(Assembly.GetExecutingAssembly()));
I found the problem.
It seems DirectoryCatalog(path) search only in DLL by default, and my test program was a console application. And the exports were in EXE (not DLL) so they weren't loaded.
On the other hand, AssemblyCatalog(Assembly.GetExecutingAssembly()), obviously loaded exports from current assembly(which is the EXE).
The solution is to use the other constructor of DirectoryCatalog(path, searchPattern) , and use "*.*" for second param. And it works