I am trying to download image/file with rust, works like a wget.
I followed this https://rust-lang-nursery.github.io/rust-cookbook/web/clients/download.html#download-a-file-to-a-temporary-directory
But a temp file isn't created, not sure why. If you remove the temp folder part, a png file is created but cannot be read. Seems corrupted.
How do I download a binary file with Rust? I could not find a good solution online.
Found the solution.
Use .bytes instead of .text(), and has to wrap the bytes inside std::io::Cursor.
Since temp file is removed on exit, I commented one line out to see the resulting image in cwd.
let mut content = Cursor::new(response.bytes().await?);
copy(&mut content, &mut dest)?;
Complete Solution
use error_chain::error_chain;
use std::io::copy;
use std::fs::File;
use tempfile::Builder;
use std::io::Cursor;
error_chain! {
foreign_links {
Io(std::io::Error);
HttpRequest(reqwest::Error);
}
}
#[tokio::main]
async fn main() -> Result<()> {
let tmp_dir = Builder::new().prefix("example").tempdir()?;
let target = "https://www.rust-lang.org/logos/rust-logo-512x512.png";
let response = reqwest::get(target).await?;
let mut dest = {
let fname = response
.url()
.path_segments()
.and_then(|segments| segments.last())
.and_then(|name| if name.is_empty() { None } else { Some(name) })
.unwrap_or("tmp.bin");
println!("file to download: '{}'", fname);
// let fname = tmp_dir.path().join(fname);
println!("will be located under: '{:?}'", fname);
File::create(fname)?
};
// let content = response.text().await?;
// copy(&mut content.as_bytes(), &mut dest)?;
let mut content = Cursor::new(response.bytes().await?);
copy(&mut content, &mut dest)?;
Ok(())
}
Related
I'm struggling to get a string of the prefix of a path. All I want to do is get the "D:" from the string "D:\Household\Internet\September_2022_Statement.pdf".
If I follow the example for std:path:Component in here, I can see the value I want but don't know how to get to it.
The code I am running is:
let filepath = "D:\\Household\\Internet\\September_2022_Statement.pdf";
let path = Path::new(filepath);
let components = path.components().collect::<Vec<_>>();
for value in &components {
println!("{:?}", value);
}
The output I get is:
Prefix(PrefixComponent { raw: "D:", parsed: Disk(68) })
RootDir
Normal("Household")
Normal("Internet")
Normal("September_2022_Statement.pdf")
How do I get the raw value "D:" from Prefix(PrefixComponent { raw: "D:", parsed: Disk(68) })?
Looks like components is an iterator of instances of the Component enum, which is declared as
pub enum Component<'a> {
Prefix(PrefixComponent<'a>),
RootDir,
CurDir,
ParentDir,
Normal(&'a OsStr),
}
Since you know that the drive is a Prefix, you can test for that.
let filepath = "D:\\Household\\Internet\\September_2022_Statement.pdf";
let path = Path::new(filepath);
let components = path.components().collect::<Vec<_>>();
for value in &components {
if let std::path::Component::Prefix(prefixComponent) = value {
return Some(value.as_os_str());
// instead of returning you could set this to a mutable variable
// or you could just check the first element of `components`
}
}
None // if this were a function that returned Option<String>
The example from the Rust docs is
use std::path::{Component, Path, Prefix};
use std::ffi::OsStr;
let path = Path::new(r"c:\you\later\");
match path.components().next().unwrap() {
Component::Prefix(prefix_component) => {
assert_eq!(Prefix::Disk(b'C'), prefix_component.kind());
assert_eq!(OsStr::new("c:"), prefix_component.as_os_str());
}
_ => unreachable!(),
}
Thanks to #Samathingamajig and #PitaJ, I was able to achieve what I needed. Obviously, I am a Rust newbie so I'm sure there is a cleaner way of doing this, but combining the help suggested, this works for me:
let filepath = "D:\\Household\\Internet\\September_2022_Statement.pdf";
let path = Path::new(fileName);
let components = path.components().collect::<Vec<_>>();
let mut os_string = OsString::new();
match path.components().next().unwrap() {
Component::Prefix(prefix_component) => {
os_string = prefix_component.as_os_str().into();
}
_ => todo!()
}
println!("{}", os_string.to_str().unwrap());
Resulting output:
D:
I try to use pty as backend to build an emulated terminal on web application which supports local shells. And I use rust nix.openpty to get slave and master file descriptor on linux. Like this:
unsafe fn spawn_pty_with_shell(default_shell: String) -> bipty {
match openpty(None, None) {
// Spawn successful.
Ok(pty_res) => {
let master = pty_res.master;
let slave = pty_res.slave;
println!("master fd: {}, slave fd: {}", &master, &slave);
// If the result is a child process, spawn a new shell.
let builder = Command::new(&default_shell)
// Get input from the slave file descriptor.
.stdin(Stdio::from_raw_fd(slave))
.stdout(Stdio::from_raw_fd(slave))
.stderr(Stdio::from_raw_fd(slave))
.spawn()
.expect("failed to spawn");
// wait for 2s and then exit.
std::thread::sleep(std::time::Duration::from_millis(2000));
bipty {
process: builder,
mfd: master,
sfd: slave,
}
},
Err(e) => {
panic!("failed to fork {:?}", e);
}
}
}
And the I successfully implement writing to the master file functionality and it works fine. Code below:
// Execute a command with user input, by flushing master file descripter.
fn pty_execute(mfd: RawFd, command: &str) {
let mut master_file = unsafe { File::from_raw_fd(mfd) };
// Write command to the file descriptor.
write!(master_file, "{}", command).unwrap();
master_file.flush().unwrap();
}
Then the problems occurs when I try to get the output from the master file:
// Error: Bad file descriptor code 9.
// fn read_from_master(mfd: RawFd) -> String {
// let mut master_file = unsafe { File::from_raw_fd(mfd) };
// let mut read_buffer = String::new();
// master_file.read_to_string(&mut read_buffer).unwrap(); // Execution stops here.
// format!("Master file content: {}", read_buffer)
// }
fn read_from_master_fd(mfd: RawFd) -> String {
let mut read_buffer: Vec<u8> = vec![];
loop {
match read_from_fd(mfd) {
Some(mut read_bytes) => {
read_buffer.append(&mut read_bytes);
}
None => {
break;
}
}
}
format!("{:?}", String::from_utf8(read_buffer).unwrap())
}
Here I implement 2 methods, the first read_from_master will panic and show Bad file descriptor, code 9, and the second read_from_master_fd will stuck and return nothing (The terminal will not give you any output and stuck, exit by pressing Ctrl+C). After all, I find no way to read content correctly from the master file and pass it as String for frontend display.
I found most rust tutorials online are either only about writing or only about reading. And I find it quite confusing to implement them correctly at the same time. It would be so great if someone willing to give some hints.
My vision is to build a rust pty backend for xtermjs (as emulated terminal frontend).
Bests.
Edited this question to use a simpler version of the code.
The TestPDF is all text and about 300 pages. As the loop runs it crashes after consuming 2gb of memory. I don’t need the value in the print statement after it’s printed. However the code keeps it in memory. How to clear the memory allocation of the contents of the print statement before the loop closes?
func loadPDFDocument(){
let documentURL = Bundle.main.url(forResource: "TestPDF", withExtension: "pdf")!
if let document = PDFDocument(url: documentURL) {
for page in 1...document.pageCount {
DispatchQueue.global().async {
print(document.page(at: page)!.string!)
}
}
}
}
Solutions I have tried include autoreleasepool and creating a new PDFDocument object in for each loop and using that. That second option does free the memory but is seriously too slow.
func loadPDFDocument(){
let documentURL = Bundle.main.url(forResource: "TestPDF", withExtension: "pdf")!
if let document = PDFDocument(url: documentURL) {
for page in 1...document.pageCount {
DispatchQueue.global().async {
let innerDocument = PDFDocument(url: documentURL)!
print(innerDocument.page(at: page)!.string!)
}
}
}
}
my solution so far has been to reload the PDFDocument in didReceiveMemoryWarning
so I have a global variable
var document = PDFDocument()
use it
let pdfURL = ...
document = PDFDocument(url: pdfURL)!
then if low memory
override func didReceiveMemoryWarning() {
let pdfURL = ...
document = PDFDocument(url: pdfURL)!
}
I'm using this code to download an mp4 file:
func downloadImageFile() {
let myURLstring = getImageURLCM()
let myFilePathString = "/Users/jack/Desktop/Comics/"+getTitle()+".mp4"
let url = NSURL(string: myURLstring)
let dataFromURL = NSData(contentsOfURL: url!)
let fileManager = NSFileManager.defaultManager()
fileManager.createFileAtPath(myFilePathString, contents: dataFromURL, attributes: nil)
}
But I've noticed that the file actually gets loaded up into my RAM first, before the NSFileManager saves it to my hard drive (based on the Xcode debug session). That's tolerable for smaller files, but most of the files I want to download with this will be at least 1GB.
My main question is: how to make this more RAM friendly?
I've also noticed that I get the spinning wheel of death until the download is finished, so if advice on fixing that would be appreciated as well.
You would be better to go with the system managed download in NSURLSession, specifically NSURLDownloadTask. This way you don't have to worry about memory management of large downloads. From NSURLSession swift file
/*
* download task convenience methods. When a download successfully
* completes, the NSURL will point to a file that must be read or
* copied during the invocation of the completion routine. The file
* will be removed automatically.
*/
func downloadTaskWithURL(url: NSURL, completionHandler: (NSURL?, NSURLResponse?, NSError?) -> Void) -> NSURLSessionDownloadTask?
Example of Use below - copy and paste into new Swift Playground:
import UIKit
import XCPlayground
func downloadFile(filePath: String) {
let url = NSURL(string: filePath)
if let unwrappedURL = url {
let downloadTask = NSURLSession.sharedSession().downloadTaskWithURL(unwrappedURL) { (urlToCompletedFile, reponse, error) -> Void in
// unwrap error if present
if let unwrappedError = error {
print(unwrappedError)
}
else {
if let unwrappedURLToCachedCompletedFile = urlToCompletedFile {
print(unwrappedURLToCachedCompletedFile)
// Copy this file to your destinationURL with
//NSFileManager.defaultManager().copyItemAtURL
}
}
}
downloadTask?.resume()
}
}
downloadFile("http://devstreaming.apple.com/videos/wwdc/2015/711y6zlz0ll/711/711_networking_with_nsurlsession.pdf?dl=1")
XCPSetExecutionShouldContinueIndefinitely()
Simple Example on github here - https://github.com/serendipityapps/NSURLSessionDownloadTaskExample
dataFromURL.writeToFile(myFilePathString, atomically: true)
This is the snippet I use, it writes the loaded data into the file at the given path.
Selecting images in Photos.app to pass to an action extension seems to yield paths to images on disk (e.g.: file:///var/mobile/Media/DCIM/109APPLE/IMG_9417.JPG). Is there a way to get the corresponding ALAsset or PHAsset?
The URL looks like it corresponds to the PHImageFileURLKey entry you get from calling PHImageManager.requestImageDataForAsset. I'd hate to have to iterate through all PHAssets to find it.
I did what I didn't want to do and threw this dumb search approach together. It works, although it's horrible, slow and gives me memory issues when the photo library is large.
As a noob to both Cocoa and Swift I'd appreciate refinement tips. Thanks!
func PHAssetForFileURL(url: NSURL) -> PHAsset? {
var imageRequestOptions = PHImageRequestOptions()
imageRequestOptions.version = .Current
imageRequestOptions.deliveryMode = .FastFormat
imageRequestOptions.resizeMode = .Fast
imageRequestOptions.synchronous = true
let fetchResult = PHAsset.fetchAssetsWithOptions(nil)
for var index = 0; index < fetchResult.count; index++ {
if let asset = fetchResult[index] as? PHAsset {
var found = false
PHImageManager.defaultManager().requestImageDataForAsset(asset,
options: imageRequestOptions) { (_, _, _, info) in
if let urlkey = info["PHImageFileURLKey"] as? NSURL {
if urlkey.absoluteString! == url.absoluteString! {
found = true
}
}
}
if (found) {
return asset
}
}
}
return nil
}
So this is commentary on ("refinement tips") to your auto-answer. SO comments don't cut it for code samples, so here we go.
You can replace your for-index loop with a simpler for-each loop. E.g. something like:
for asset in PHAsset.fetchAssetsWithOptions(nil)
As of the last time I checked, the key in info["PHImageFileURLKey"] is undocumented. Be apprised. I don't think it will get you rejected, but the behavior could change at any time.