I want to test my simple class that connected to public Ftp and get list of fileNames from there.
And i found very strange issue - navigation on real server and on test conteiner with same dirs are not same
there is a dir structure
Well if i connect to real server and use ftp.changeWorkingDirectory("/out/published")
everything is good - directory changed and i retrive file from there
if i trying to do same in testconteiner ftp.changeWorkingDirectory("/out/published") it return false ( cant find this directory) and if i remove slash like this ftp.changeWorkingDirectory("out/published") it will work
and second issue if i will do ftp.listnames("/out/published/tverskaya_obl/purchaseNotice/daily) on real server it will work good, but in testconteiner it work only if i use this
ftp.listNames("tverskaya_obl/purchaseNotice/daily"
)
is there any way to fix it
here is the code to create testcontainer and folders
#TestInstance(TestInstance.Lifecycle.PER_CLASS)
#SpringBootTest(properties = "ftp.host = localhost")
class PurchaseFZ223FtpServiceTest {
private static final int PORT = 21;
private static final String USER = "";
private static final String PASSWORD = "";
private static final int PASSIVE_MODE_PORT = 21000;
#Autowired
FtpService ftpService;
private static final FixedHostPortGenericContainer ftp = new FixedHostPortGenericContainer<>(
"delfer/alpine-ftp-server:latest")
.withFixedExposedPort(PASSIVE_MODE_PORT, PASSIVE_MODE_PORT)
.withExposedPorts(PORT)
.withEnv("USERS", USER + "|" + PASSWORD)
.withEnv("MIN_PORT", String.valueOf(PASSIVE_MODE_PORT))
.withEnv("MAX_PORT", String.valueOf(PASSIVE_MODE_PORT));
#BeforeAll
void init() throws NoSuchFieldException, IllegalAccessException, IOException {
ftp.start();
FTPClient client =new FTPClient();
client.connect("localhost", ftp.getMappedPort(PORT));
client.enterLocalPassiveMode();
client.login(USER, PASSWORD);
client.makeDirectory("out");
client.makeDirectory("out/published");
client.makeDirectory("out/published/tverskaya_obl");
client.makeDirectory("out/published/tverskaya_obl/purchaseNotice");
client.makeDirectory("out/published/tverskaya_obl/purchaseNotice/daily");
client.changeWorkingDirectory("out/published");
client.storeFile("list_regions.txt", new ByteArrayInputStream("tverskaya_obl".getBytes(StandardCharsets.UTF_8)));
client.changeWorkingDirectory("tverskaya_obl/purchaseNotice/daily");
client.storeFile("purchaseNotice_Tverskaya_obl_20221209_000000_20221209_235959_daily_001.xml.txt", new ByteArrayInputStream("test".getBytes(StandardCharsets.UTF_8)));
}
Finally found what is going wrong
When testconteiner create ftp server it just add two folders like this
ftp/yourLoginName
so my mistake was to ask list of file name with invalid path
valid path for me to get list of fileNames is /ftp/***/out/published/tverskaya_obl/purchaseNotice/daily
same with change working dir
if you dnt know how to check real path you are at just do this
ftp.printWorkingDirectory()
Related
Quick summary: How can I change the default port of the minio client running in my test container?
I want to use minio as a testcontainer in my application which already working when I start it locally. here is the codesnippet I use to run the testcontainer:
public class MinioContainer extends GenericContainer<MinioContainer> {
private static final int DEFAULT_PORT = 9000;
private static final String DEFAULT_IMAGE = "/minio/minio";
private static final String DEFAULT_TAG = "latest";
private static final String MINIO_ACCESS_KEY = "MINIO_ACCESS_KEY";
private static final String MINIO_SECRET_KEY = "MINIO_SECRET_KEY";
private static final String DEFAULT_STORAGE_DIRECTORY = "/data";
private static final String HEALTH_ENDPOINT = "/minio/health/ready";
public MinioContainer() {
this(DEFAULT_IMAGE + ":" + DEFAULT_TAG);
}
public MinioContainer(String image) {
super(image == null ? DEFAULT_IMAGE + ":" + DEFAULT_TAG : image);
Network network = Network.newNetwork();
withNetwork(network);
withNetworkAliases("minio-" + Base58.randomString(6));
addExposedPort(DEFAULT_PORT);
withEnv(MINIO_ACCESS_KEY, "access_key");
withEnv(MINIO_SECRET_KEY, "secret_key");
withCommand("server", DEFAULT_STORAGE_DIRECTORY);
setWaitStrategy(new HttpWaitStrategy()
.forPort(DEFAULT_PORT)
.forPath(HEALTH_ENDPOINT)
.withStartupTimeout(Duration.ofMinutes(1)));
}
public String getHostAddress() {
return getHost() + ":" + getMappedPort(DEFAULT_PORT);
}
}
As soon as I deploy this on our cluster, where also an minio container is running at port 9000, it shows this error message in the console:
io.minio.errors.ErrorResponseException: The Access Key Id you provided does not exist in our records.
at some.package.MinioTest.setup(MinioTest.java:58)
In my test i am running a SpringBootTest using this container and injecting my minio client. I also configured a test application yaml so I can run my test with an active test profile. The error happens on following code snippet:
private final String BUCKET = "bucket";
....
#BeforeEach
void setup() {
boolean bucketExists = minioClient.bucketExists(BucketExistsArgs.builder().bucket(BUCKET).build());
...
}
Is there a way to change the DEFAULT_PORT on my MinioContainer so it is not the same port as the minio container already running on my cluster? I am not able to get my tests running on our pipeline because of this issue, which is only happening on our cluster.
As soon as I change the DEFAULT_PORT to something different than 9000 on my MinioContainer, the Container stops working because it is not able to find the HEALTH_ENDPOINT and therefor the whole container just stops working.
I hope I explained my problem clear enough. If not please tell me so I can try to explain it clearer. I am already completely frustrated with this issue.
BR
Null
I found the solution for my problem. Minio supports the following command:
"server /data --address :9100"
Now I was able to generate my testcontainer now like this:
public static GenericContainer<?> minioContainer = new GenericContainer<>(MINIO_IMAGE)
.withCommand("server /data --address :9100")
.withExposedPorts(9100);
Now the MinioContainer in my Test runs with Port 9100.
Hopefully I was able to help someone else with this issue.
I just deployed my first microservice. My microservice is working fine. All routes are working. But the service class inside the microservice is not working properly. The service class is not reading data from the CSV file.
Below is the code I am using to read data from CSV file.
public class ReadCsvUtil {
private static final Logger LOGGER = LoggerFactory.getLogger(ReadCsvUtil.class);
public List<String[]> readData() throws IOException {
String file = ".\\src\\main\\resources\\pensioners.csv";
List<String[]> content = new ArrayList<>();
try(BufferedReader br = new BufferedReader(new FileReader(file))) {
String line = "";
while ((line = br.readLine()) != null) {
content.add(line.split(","));
}
} catch (FileNotFoundException e) {
LOGGER.debug(e.getMessage());
}
return content;
}
}
The service class invokes the above function to get details of all the people.
The above code is working fine on my desktop and I am able to get details but code is not working on AWS. Also, I tried to remove the CSV and manually enter the values and it's working in AWS. So I am 99% sure there is some problem in reading the CSV files.
Is there anyway I can fix this?
If the path to the file one directory up, instead of
String file = ".\\src\\main\\resources\\pensioners.csv";
try,
String file = "..\src\main\resources\pensioners.csv";
I presume the AWS sever is ubuntu and your local is windows OS.
I am building a spring boot application where i need to read json files for my component tests. I have a utility method which takes the name of the file and reads the content using ResourceUtils. Here is the code:
public static String getContent(String path) throws IOException {
File file = ResourceUtils.getFile(MyTest.class.getResource(path));
String content = new String(Files.readAllBytes(file.toPath()));
return content;
}
The checkmarx is reporting the above code as "This may cause a Path
Traversal vulnerability."
How to fix this?
Thanks
See this example for path traversal vulnerability Path Traversal
To fix this change it something like
private static final String BASE_PATH ="/yourbasepath/somewherewherefileisstored";
public static String getContent(String path) throws IOException {
File file = new File(BASE_PATH, path);
if (file.getCanonicalPath().startsWith(BASE_PATH)){
String content = new String(Files.readAllBytes(file.toPath()));
return content;
}
else{
//throw some error
}
}
We are running Java-Cucumber-Maven automated test scripts from TeamCity. Up to now this has worked very well but our current test requires that several .CSV files are downloaded for analysis by the script. There's a
ReadCsvFiles.java
file in our code and that dictates where the downloaded files are located....easy for running tests locally but we're having real problems with the path to the build agent in TeamCity..
public class ReadCsvFiles {
//public static String sFileLocation = "C:\\Users\\ankit.bisht\\Downloads\\";
public static String sFileLocation = "C:\\BuildAgent\\temp\\buildTmp\\";
//public static String sFileLocation = "//users/redmayned//Downloads/";
public static void depositsHeldReports() throws Throwable {
String csvFile = sFileLocation + "DepositsHeld.csv";
CSVReader reader = null;
try {
reader = new CSVReader(new FileReader(csvFile));
String[] column;
boolean doesExist = false;
while ((column = reader.readNext()) != null) {
The commented out entries are for local windows and local OSX. The middile one is just one of many options we've tried...all resulting in an error message;
java.io.FileNotFoundException: C:\BuildAgent\temp\buildTmp\DepositsHeld.csv (The system cannot find the file specified)
Can anyone save my sanity please?
Thanks
David
Can anyone tell why this didn't work? The code works great when I run it from my unit tests. Security gets setup perfectly and our service works great just like I expect.
However, when I deployed it to our application server (weblogic), my service fails every time because my tokens are not getting setup. I got it working by setting up the tokens every time my send(final ServiceAPInvoice invoice) method gets called.
My question is why does the tokens not get setup by my constructor when this is deployed in our Weblogic environment? What causes this issue? OAuthSecurityContextHolder is a static class. Is that playing into my issue? Will I still run into issues if I setup the tokens each time my send method is called? I haven't noticed any issues yet but have not done any load testing
I am using Spring's OAuthRestTemplate (1.0) and I have non-expiring tokens that I need to setup.
Here is where the magic happens. I had to rename the code slightly to make it generic so hopefully I don't have any typos:
public class ServiceRestTemplate {
private final OAuthRestTemplate serviceOAuthRestTemplate;
private final String apUri;
private final String arUri;
private final String tokenValue;
private final String tokenSecret;
public ServiceRestTemplate(final OAuthRestTemplate serviceOAuthRestTemplate,
final String apUri,
final String arUri,
final String tokenValue,
final String tokenSecret) {
this.serviceOAuthRestTemplate = serviceOAuthRestTemplate;
this.apUri = apUri;
this.arUri = arUri;
this.tokenSecret = tokenSecret;
this.tokenValue = tokenValue;
setContext(tokenValue, tokenSecret); // I expected this to be enough to setup my tokens 1 time
}
private void setContext(final String tokenValue, final String tokenSecret) {
final OAuthConsumerToken accessToken = new OAuthConsumerToken();
accessToken.setAccessToken(true);
accessToken.setResourceId(serviceOAuthRestTemplate.getResource().getId());
accessToken.setValue(tokenValue);
accessToken.setSecret(tokenSecret);
final OAuthSecurityContextImpl securityContext = new OAuthSecurityContextImpl();
if (securityContext.getAccessTokens() == null) {
securityContext.setAccessTokens(new HashMap<String, OAuthConsumerToken>());
}
if (!securityContext.getAccessTokens().containsKey(accessToken.getResourceId())) {
securityContext.getAccessTokens().put(accessToken.getResourceId(), accessToken);
}
OAuthSecurityContextHolder.setContext(securityContext);
}
#Override
public ServiceWebResponse send(final ServiceAPInvoice invoice) {
setContext(this.tokenValue, this.tokenSecret); // This line of code is the workaround to fixed my issue.
final ServiceWebResponse serviceResponse = serviceOAuthRestTemplate.postForObject(apUri,
invoice,
ServiceWebResponse.class);
return serviceResponse;
}
}