I'm trying to import data to parse.com so I can test my application (I'm new to parse and I've never used json before).
Can you please give me an example of a json file that I can use to import binary files (images) ?
NB : I'm trying to upload my data in bulk directry from the Data Browser. Here is a screencap : i.stack.imgur.com/bw9b4.png
In parse docs i think 2 sections could help you out depend on whether you want to use REST api of the android sdk.
rest api - see section on POST, uploading files that can be upload to parse using REST POST.
SDk - see section on "files"
code for Rest includes following:
use some HttpClient implementation having "ByteArrayEntity" class or something
Map your image to bytearrayEntity and POST it with the correct headers for Mime/Type in httpclient...
case POST:
HttpPost httpPost = new HttpPost(url); //urlends "audio OR "pic"
httpPost.setProtocolVersion(new ProtocolVersion("HTTP", 1,1));
httpPost.setConfig(this.config);
if ( mfile.canRead() ){
FileInputStream fis = new FileInputStream(mfile);
FileChannel fc = fis.getChannel(); // Get the file's size and then map it into memory
int sz = (int)fc.size();
MappedByteBuffer bb = fc.map(FileChannel.MapMode.READ_ONLY, 0, sz);
data2 = new byte[bb.remaining()];
bb.get(data2);
ByteArrayEntity reqEntity = new ByteArrayEntity(data2);
httpPost.setEntity(reqEntity);
fis.close();
}
,,,
request.addHeader("Content-Type", "image/*") ;
pseudocode for post the runnable to execute the http request
The only binary data allowed to be loaded to parse.com are images. In other cases like files or streams .. etc the most suitable solution is to store a link to the binary data in another dedicated storage for such type of information.
Related
I set up an Elastic Cloud to offload my local elasticsearch config (as one does), but for reasons unknown to me, I can't get it to show any logs in Elastic Cloud, despite it working fine locally.
The code I got: (modified for privacy reasons)
//var uri = new Uri("http://localhost:9200"); // old one
var uri = new Uri("https://my-server.kb.eastus2.azure.elastic-cloud.com:9243");
var sinkOptions = new ElasticsearchSinkOptions(uri)
{
AutoRegisterTemplate = true,
ModifyConnectionSettings = x => x.BasicAuthentication("elastic", "the password I was given"),
IndexFormat = $"test-logs-{env.EnvironmentName?.ToLower().Replace('.', '-')}-{DateTime.Now:yyyy-MM}",
};
Log.Logger = new LoggerConfiguration()
.ReadFrom.Configuration(config)
.Enrich.FromLogContext()
.Enrich.WithMachineName()
.WriteTo.Console()
.WriteTo.Elasticsearch(sinkOptions)
.Enrich.WithProperty("Environment", env.EnvironmentName)
.CreateLogger();
There are two possible reasons I can think of that might be the cause of this not working:
The credentials are wrong
The Uri is wrong
Every solution I've been given so far has provided the data in this fashion, and nowhere does it say what the URI I'm supposed to use looks like.
I get no errors.
I get no warnings.
I get no logs.
What am I doing wrong here?
The issue was using the incorrect uri. I wrote
my-server.kb.eastus2.azure.elastic-cloud.com:9243 rather than
my-server.es.eastus2.azure.elastic-cloud.com:9243.
Note the very tiny difference that is kb vs es in the url
I have been trying to upload an image in chunks with client side streaming using grpcurl. The service is working without error except that at the server, image data received is 0 bytes.
The command I am using is:
grpcurl -proto image_service.proto -v -d # -plaintext localhost:3010 imageservice.ImageService.UploadImage < out
This link mentions that the chunk data should be base64 encode and so the contents of my out file are:
{"chunk_data": "<base64 encoded image data>"}
This is exactly what I am trying to achieve, but using grpcurl.
Please tell what is wrong in my command and what is the best way to achieve streaming via grpcurl.
I have 2 more questions:
Does gRPC handles the splitting of data into chunks?
How can I first send a meta-data chunk (ImageInfo type) and then the actual image data via grpcurl?
Here is my proto file:
syntax = "proto3";
package imageservice;
import "google/protobuf/wrappers.proto";
option go_package = "...";
service ImageService {
rpc UploadImage(stream UploadImageRequest) returns (UploadImageResponse) {}
}
message UploadImageRequest {
oneof data {
ImageInfo info = 1;
bytes chunk_data = 3;
};
}
message ImageInfo {
string unique_id = 1;
string image_type = 2;
}
message UploadImageResponse {
string url = 1;
}
Interesting question. I've not tried streaming messages with (the excellent) grpcurl.
The documentation does not explain how to do this but this issue shows how to stream using stdin.
I recommend you try it that way first to ensure that works for you.
If it does, then bundling various messages into a file (out) should also work.
Your follow-on questions suggest you're doing this incorrectly.
chunk_data is the result of having split the file into chunks; i.e. each of these base64-encoded strings should be a subset of your overall image file (i.e. a chunk).
your first message should be { "info": "...." }, subsequent messages will be { "chunk_data": "<base64-encoded chunk>" } until EOF.
In a spring boot application I read an image file from a remote service, which returns byte array and in headers I can check what is file extension:
ResponseEntity<byte[]> result = restTemplate.exchange(url, HttpMethod.GET, entity, byte[].class);
Now I want to put this byte array in a S3 bucket in a folder which I decide during run time, for example folder name can base don current timestamp.
I checked AmazonS3 class, but it doesnt seem to have any such API which can help me?
How can this be done?
As per example from documentation:
https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/examples-s3-objects.html#upload-object
// Put Object. here 'bytes' is byte array.
PutObjectResponse response = s3.putObject(PutObjectRequest.builder().bucket(bucketName).key(filePathLocation).build(),RequestBody.fromBytes(bytes));
You can use the MinIO java S3 client. Here you can find the documentation.
The code will look something like the following one:
MinioClient minioClient =
MinioClient.builder()
.endpoint("https://play.min.io")
.credentials("Q3AM3UQ867SPQQA43P2F", "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG")
.build();
StringBuilder builder = new StringBuilder();
for (int i = 0; i < 1000; i++) {
builder.append(
"Sphinx of black quartz, judge my vow: Used by Adobe InDesign to display font samples. ");
builder.append("(29 letters)\n");
builder.append(
"Jackdaws love my big sphinx of quartz: Similarly, used by Windows XP for some fonts. ");
builder.append("(31 letters)\n");
builder.append(
"Pack my box with five dozen liquor jugs: According to Wikipedia, this one is used on ");
builder.append("NASAs Space Shuttle. (32 letters)\n");
builder.append(
"The quick onyx goblin jumps over the lazy dwarf: Flavor text from an Unhinged Magic Card. ");
builder.append("(39 letters)\n");
builder.append(
"How razorback-jumping frogs can level six piqued gymnasts!: Not going to win any brevity ");
builder.append("awards at 49 letters long, but old-time Mac users may recognize it.\n");
builder.append(
"Cozy lummox gives smart squid who asks for job pen: A 41-letter tester sentence for Mac ");
builder.append("computers after System 7.\n");
builder.append(
"A few others we like: Amazingly few discotheques provide jukeboxes; Now fax quiz Jack! my ");
builder.append("brave ghost pled; Watch Jeopardy!, Alex Trebeks fun TV quiz game.\n");
builder.append("---\n");
// Create a InputStream for object upload.
ByteArrayInputStream bais = new ByteArrayInputStream(builder.toString().getBytes("UTF-8"));
// Create object 'my-objectname' in 'my-bucketname' with content from the input stream.
minioClient.putObject(
PutObjectArgs.builder().bucket("my-bucketname").object("my-objectname").stream(
bais, bais.available(), -1)
.build());
bais.close();
System.out.println("my-objectname is uploaded successfully");
The full code can be found here.
Checkout the AWS JAVA SDK:
Here the getting started section:
https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/getting-started.html
In order to use in Spring context use the Maven dependency:
https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/setup-project-maven.html
Uploading an object to S3 Bucket:
https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-s3-objects.html#upload-object
import com.amazonaws.AmazonServiceException;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
System.out.format("Uploading %s to S3 bucket %s...\n", file_path, bucket_name);
final AmazonS3 s3 = AmazonS3ClientBuilder.standard().withRegion(Regions.DEFAULT_REGION).build();
try {
s3.putObject(bucket_name, key_name, new File(file_path));
} catch (AmazonServiceException e) {
System.err.println(e.getErrorMessage());
System.exit(1);
I have an endpoint that downloads an xlsx file. In my test, I need to check the content of the file (not comparing the file with another file, but reading the content and checking). I am using karate framework for testing and I am trying to use apache POI for working with the excel sheet. However, the response I get from karate when calling the download endpoint is a String. For creating an excel file with POI I need an InputStream or the path to the actual file. I have tried the conversion, but it does not work.
I guess I am missing some connection here, or maybe the conversion is bad, I am new to karate and to the whole thing.
I appreciate any help, thanks!
Given url baseUrl
Given path downloadURI
When method GET
Then status 200
And match header Content-disposition contains 'attachment'
And match header Content-disposition contains 'example.xlsx'
And match header Content-Type == 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
* def value= FileChecker.createExcelFile(response)
* print value
And the Java code:
public static String createExcelFile(String excel) throws IOException, InvalidFormatException {
InputStream stream = IOUtils.toInputStream(excel, Charset.forName("UTF-8"));
Workbook workbook = WorkbookFactory.create(stream);
return ("Workbook has " + workbook.getNumberOfSheets() + " Sheets : ");
}
When running the scenario, I get the following error:
javascript evaluation failed: FileChecker.createExcelFile(response), java.io.IOException: Failed to read zip entry source
When testing the same endpoint in Postman, I am getting a valid excelsheet.
In Karate 0.9.X onwards you have a responseBytes variable which will be raw bytes, which may be what you need.
* def value = FileChecker.createExcelFile(responseBytes)
And you can change your method signature to be:
public static String createExcelFile(byte[] excel) {}
You should be easily able to convert a byte array to an InputStream and take it from there.
P.S. just saying that it "works in Postman" is not helpful :P
TO download zip file from Karate tests as binary bite array
Scenario: To verify and get the ADCI Uri from deployment
Given url basicURL + DeployUri +ArtifactUri
And headers {authorization:'#(authToken)',accept:'application/json',tenant:'#(tenantUUId)',Content-Type:'application/zip'}
When method get
Then status 200
And def responsebytes = responseBytes
I'm trying to access url and then parse it's contents based on tags.
My code:
page = requests.get('https://support.apple.com/downloads/')
self.tree = html.fromstring(page.content)
names = self.tree.xpath("//span[#class='truncate_name']//text()")
Problem: variable page is containing data that of url 'https://support.apple.com/'
I'm new to python 2.7. The whole encoding issues in file. I'm using unicode-escape as my default encoding. Encoding on resource at https://support.apple.com/downloads/ is utf-8 whereas encoding of resource at https://support.apple.com/ is variable. Is this has something to do with the problem? Please suggest solution for this.
It has nothing to do with encoding , what you are looking for is dynamically created so not in the source you get back. A series of ajax calls populates the data. To get the product names etc.. from the carousel where you see the span.truncate_name in your browser:
params = {"page": "products",
"locale": "en_US",
"doctype": "DOWNLOADS",
}
js = requests.get("https://km.support.apple.com/kb/index", params=params).content
Normally we could call .json() on the response object but in this case we need to use "unicode_escape" then call loads:
from json import loads, dumps
js2 = loads(js.decode("unicode_escape"))
print(js2)
Which gives you a huge dict of data like:
{u'products': [{u'name': u'Servers and Enterprise', u'urlpath': u'serversandenterprise', u'order': u'', u'products': .............
You can see the request in chrome tools:
We leave off callback:ACDownloadSearch.customCallBack as we want to get back valid json.