I'm working with large data in wcf (gigs). I need to return this to the client and the previous code wrote a memory stream and then returned this to the client as a property on a streamed data contract.
The problem is that the data has grown and the memory stream now uses all available memory and eventually fails before all the data is written to the stream.
Is it possible to write the data to the stream on-demand, so the server is writing it was the client is requesting it?
Here's our code;
internal StreamedFileResponse CreateStream()
{
var stream = new MemoryStream();
using (var writer = new CsvWriter(stream, Encoding.UTF8))
{
writer.Write(rowFactory.CreateHeader());
foreach (var source in sources)
{
var row = rowFactory.Create(source);
if (row != null)
{
writer.Write(row);
}
}
}
stream.Seek(0, SeekOrigin.Begin);
return new StreamedFileResponse { Data = stream, Length = stream.Length };
}
[MessageContract(WrapperNamespace = "http://www.contoso.com/services/contract/reports/streamed")]
public class StreamedFileResponse : IDisposable
{
[MessageBodyMember]
public Stream Data { get; set; }
[MessageHeader(MustUnderstand = true)]
public long Length { get; set; }
public void Dispose()
{
Data.Dispose();
}
}
Related
I have the following two methods that handles taking photos from a camera and picking photos from a library. They're both similar methods as at the end of each method, I get an ImageSource back from the Stream and I pass it onto another page which has an ImageSource binding ready to be set. These two method work perfectly. The next step now is to save the Image in SQLite so I can show the images in a ListView later on. My question for the XamGods (Xamarin Pros =), what is the best way to save image in SQLite in 2019? I have been in the forums for hours and I still don't have a tunnel vision on what I want to do. I can either
Convert Stream into an array of bytes to save in Sqlite.
Convert ImageSource into an array of bytes (messy/buggy).
Somehow retrieve the actual Image selected/taken and convert that into an array of bytes into SQLite
I'm sorry if my question is general, but Xamarin does not provide a clear-cut solution on how to save images in SQLite and you can only find bits and pieces of solutions throughout the forums listed below.
How to save and retrieve Image from Sqlite
Load Image from byte[] array.
Creating a byte array from a stream
Thank you in advance!
private async Task OnAddPhotoFromCameraSelected()
{
Console.WriteLine("OnAddPhotoFromCameraSelected");
var photo = await Plugin.Media.CrossMedia.Current.TakePhotoAsync(new Plugin.Media.Abstractions.StoreCameraMediaOptions() { });
var stream = photo.GetStream();
photo.Dispose();
if (stream != null)
{
ImageSource cameraPhotoImage = ImageSource.FromStream(() => stream);
var parms = new NavigationParameters();
parms.Add("image", cameraPhotoImage);
var result = await NavigationService.NavigateAsync("/AddInspectionPhotoPage?", parameters: parms);
if (!result.Success)
{
throw result.Exception;
}
}
}
private async Task OnAddPhotoFromLibrarySelected()
{
Console.WriteLine("OnAddPhotoFromLibrarySelected");
Stream stream = await DependencyService.Get<IPhotoPickerService>().GetImageStreamAsync();
if (stream != null)
{
ImageSource selectedImage = ImageSource.FromStream(() => stream);
var parms = new NavigationParameters();
parms.Add("image", selectedImage);
parms.Add("stream", stream);
var result = await NavigationService.NavigateAsync("/AddInspectionPhotoPage?", parameters: parms);
if (!result.Success)
{
throw result.Exception;
}
}
}
As Jason said that you can save image path into sqlite database, but if you still want to save byte[] into sqlite database, you need to convert stream into byte[] firstly:
private byte[] GetImageBytes(Stream stream)
{
byte[] ImageBytes;
using (var memoryStream = new System.IO.MemoryStream())
{
stream.CopyTo(memoryStream);
ImageBytes = memoryStream.ToArray();
}
return ImageBytes;
}
Then load byte[] from sqlite, converting into stream.
public Stream BytesToStream(byte[] bytes)
{
Stream stream = new MemoryStream(bytes);
return stream;
}
For simple sample, you can take a look:
Insert byte[] in sqlite:
private void insertdata()
{
var path = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), "sqlite1.db3");
using (var con = new SQLiteConnection(path))
{
Image image = new Image();
image.Content = ConvertStreamtoByte();
var result = con.Insert(image);
sl.Children.Add(new Label() { Text = result > 0 ? "insert successful insert" : "fail insert" });
}
}
Loading image from sqlite:
private void getdata()
{
var path = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), "sqlite1.db3");
using (var con = new SQLiteConnection(path))
{
var image= con.Query<Image>("SELECT content FROM Image ;").FirstOrDefault();
if(image!=null)
{
byte[] b = image.Content;
Stream ms = new MemoryStream(b);
image1.Source = ImageSource.FromStream(() => ms);
}
}
}
Model:
public class Image
{
[PrimaryKey, AutoIncrement]
public int Id { get; set; }
public string FileName { get; set; }
public byte[] Content { get; set; }
}
I'm developing the service within ASP.NET Boilerplate engine and getting the error from the subject. The nature of the error is not clear, as I inheriting from ApplicationService, as documentation suggests. The code:
namespace MyAbilities.Api.Blob
{
public class BlobService : ApplicationService, IBlobService
{
public readonly IRepository<UserMedia, int> _blobRepository;
public BlobService(IRepository<UserMedia, int> blobRepository)
{
_blobRepository = blobRepository;
}
public async Task<List<BlobDto>> UploadBlobs(HttpContent httpContent)
{
var blobUploadProvider = new BlobStorageUploadProvider();
var list = await httpContent.ReadAsMultipartAsync(blobUploadProvider)
.ContinueWith(task =>
{
if (task.IsFaulted || task.IsCanceled)
{
if (task.Exception != null) throw task.Exception;
}
var provider = task.Result;
return provider.Uploads.ToList();
});
// store blob info in the database
foreach (var blobDto in list)
{
SaveBlobData(blobDto);
}
return list;
}
public void SaveBlobData(BlobDto blobData)
{
UserMedia um = blobData.MapTo<UserMedia>();
_blobRepository.InsertOrUpdateAndGetId(um);
CurrentUnitOfWork.SaveChanges();
}
public async Task<BlobDto> DownloadBlob(int blobId)
{
// TODO: Implement this helper method. It should retrieve blob info
// from the database, based on the blobId. The record should contain the
// blobName, which should be returned as the result of this helper method.
var blobName = GetBlobName(blobId);
if (!String.IsNullOrEmpty(blobName))
{
var container = BlobHelper.GetBlobContainer();
var blob = container.GetBlockBlobReference(blobName);
// Download the blob into a memory stream. Notice that we're not putting the memory
// stream in a using statement. This is because we need the stream to be open for the
// API controller in order for the file to actually be downloadable. The closing and
// disposing of the stream is handled by the Web API framework.
var ms = new MemoryStream();
await blob.DownloadToStreamAsync(ms);
// Strip off any folder structure so the file name is just the file name
var lastPos = blob.Name.LastIndexOf('/');
var fileName = blob.Name.Substring(lastPos + 1, blob.Name.Length - lastPos - 1);
// Build and return the download model with the blob stream and its relevant info
var download = new BlobDto
{
FileName = fileName,
FileUrl = Convert.ToString(blob.Uri),
FileSizeInBytes = blob.Properties.Length,
ContentType = blob.Properties.ContentType
};
return download;
}
// Otherwise
return null;
}
//Retrieve blob info from the database
private string GetBlobName(int blobId)
{
throw new NotImplementedException();
}
}
}
The error appears even before the app flow jumps to 'SaveBlobData' method. Am I missed something?
Hate to answer my own questions, but here it is... after a while, I found out that if UnitOfWorkManager is not available for some reason, I can instantiate it in the code, by initializing IUnitOfWorkManager in the constructor. Then, you can simply use the following construction in your Save method:
using (var unitOfWork = _unitOfWorkManager.Begin())
{
//Save logic...
unitOfWork.Complete();
}
I am new to Web API and REST services and looking to build a simple REST server which accepts file uploads. I found out grapevine which is simple and easy to understand. I couldn't find any file upload example?
This is an example using System.Web.Http
var streamProvider = new MultipartFormDataStreamProvider(ServerUploadFolder);
await Request.Content.ReadAsMultipartAsync(streamProvider);
but the grapevine Request property does not have any method to do that. Can someone point me to an example?
If you are trying to upload a file as a binary payload, see this question/answer on GitHub.
If you are trying to upload a file from a form submission, that will be a little bit trickier, as the multi-part payload parsers haven't been added yet, but it is still possible.
The following code sample is complete untested, and I just wrote this off the top of my head, so it might not be the best solution, but it's a starting point:
public static class RequestExtensions
{
public static IDictionary<string, string> ParseFormUrlEncoded(this IHttpRequest request)
{
var data = new Dictionary<string, string>();
foreach (var tuple in request.Payload.Split('='))
{
var parts = tuple.Split('&');
var key = Uri.UnescapeDataString(parts[0]);
var val = Uri.UnescapeDataString(parts[1]);
if (!data.ContainsKey(key)) data.Add(key, val);
}
return data;
}
public static IDictionary<string, FormElement> ParseFormData(this IHttpRequest request)
{
var data = new Dictionary<string, FormElement>();
var boundary = GetBoundary(request.Headers.Get("Content-Type"));
if (boundary == null) return data;
foreach (var part in request.Payload.Split(new[] { boundary }, StringSplitOptions.RemoveEmptyEntries))
{
var element = new FormElement(part);
if (!data.ContainsKey(element.Name)) data.Add(element.Name, element);
}
return data;
}
private static string GetBoundary(string contenttype)
{
if (string.IsNullOrWhiteSpace(contenttype)) return null;
return (from part in contenttype.Split(';', ',')
select part.TrimStart().TrimEnd().Split('=')
into parts
where parts[0].Equals("boundary", StringComparison.CurrentCultureIgnoreCase)
select parts[1]).FirstOrDefault();
}
}
public class FormElement
{
public string Name => _dispositionParams["name"];
public string FileName => _dispositionParams["filename"];
public Dictionary<string, string> Headers { get; private set; }
public string Value { get; }
private Dictionary<string, string> _dispositionParams;
public FormElement(string data)
{
var parts = data.Split(new [] { "\r\n\r\n", "\n\n" }, StringSplitOptions.None);
Value = parts[1];
ParseHeaders(parts[0]);
ParseParams(Headers["Content-Disposition"]);
}
private void ParseHeaders(string data)
{
Headers = data.TrimStart().TrimEnd().Split(new[] {"\r\n", "\n"}, StringSplitOptions.RemoveEmptyEntries).Select(header => header.Split(new[] {':'})).ToDictionary(parts => parts[0].TrimStart().TrimEnd(), parts => parts[1].TrimStart().TrimEnd());
}
private void ParseParams(string data)
{
_dispositionParams = new Dictionary<string, string>();
foreach (var part in data.Split(new[] {';'}))
{
if (part.IndexOf("=") == -1) continue;
var parts = part.Split(new[] {'='});
_dispositionParams.Add(parts[0].TrimStart(' '), parts[1].TrimEnd('"').TrimStart('"'));
}
}
}
If you are looking for something async to use immediately, you can try to implement the answer to this stackoverflow question, which has not been tested by me.
I'm trying to port an Android app with a Realm to Xamarin so it'll be also available for iOS devices. In Android, I have several JSON files with some necessary initial data, e.g. cities.json, and I import it at the beginning with realm.createOrUpdateAllFromJson(Class<E> clazz, InputStream in) method, like this:
private void loadInitialCities(Realm realm) {
InputStream stream = context.getAssets().open("data/cities.json");
realm.createOrUpdateAllFromJson(City.class, stream);
}
I also find this method very useful when retrieving data from a web service in form of JSON.
Now with Xamarin I don't see any equivalent to such method. Is there any method to achieve this? Or at least a workaround/tool to create a RealmObject from a JSON in C#?
I wrote my own extension methods for doing this (yes, I miss the built-in helper methods also).
https://github.com/sushihangover/Realm.Json.Extensions
Here is a basic example of how I do it:
JSON Model:
[
{
"name": "Alabama",
"abbreviation": "AL"
},
{
"name": "Alaska",
"abbreviation": "AK"
},
~~~~
]
Realm Model:
public class State : RealmObject
{
public string name { get; set; }
public string abbreviation { get; set; }
}
Xamarin.Android asset and Newtonsoft Streaming reader:
var config = RealmConfiguration.DefaultConfiguration;
config.SchemaVersion = 1;
using (var theRealm = Realm.GetInstance("StackOverflow.realm"))
using (var assetStream = Assets.Open("States.json"))
using (var streamReader = new StreamReader(assetStream))
using (var jsonTextReader = new JsonTextReader(streamReader))
{
var serializer = new JsonSerializer();
if (!jsonTextReader.Read() || jsonTextReader.TokenType != JsonToken.StartArray)
throw new Exception("Bad Json, start of array missing");
while (jsonTextReader.Read())
{
if (jsonTextReader.TokenType == JsonToken.EndArray)
break;
var state = serializer.Deserialize<State>(jsonTextReader);
theRealm.Write(() =>
{
var realmState = theRealm.CreateObject<State>();
realmState.abbreviation = state.abbreviation;
realmState.name = state.name;
});
}
}
Update: One of my extensions methods:
Extension Method Usage:
using (var theRealm = Realm.GetInstance("StackOverflow.realm"))
using (var assetStream = Assets.Open("States.json"))
{
theRealm.JsonArrayToRealm<State>(assetStream);
}
Extension Method:
Note: This uses AutoMapper to copy RealmObject and avoid reflection, also using Newtonsoft.Json.
public static class RealmDoesJson
{
public static void JsonArrayToRealm<T>(this Realm realm, Stream stream) where T : RealmObject
{
Mapper.Initialize(cfg =>
{
cfg.CreateMap<T, T>();
});
using (var streamReader = new StreamReader(stream))
using (var jsonTextReader = new JsonTextReader(streamReader))
{
var serializer = new JsonSerializer();
if (!jsonTextReader.Read() || jsonTextReader.TokenType != JsonToken.StartArray)
throw new Exception("MALFORMED JSON, Start of Array missing");
while (jsonTextReader.Read())
{
if (jsonTextReader.TokenType == JsonToken.EndArray)
break;
var jsonObject = serializer.Deserialize<T>(jsonTextReader);
realm.Write(() => // inside while loop / single object transaction for memory manangement reasons...
{
var realmObject = realm.CreateObject(typeof(T).Name);
Mapper.Map<T, T>(jsonObject, realmObject);
});
}
}
}
}
I want to use protobuf-net to serialize stock market data. I'm playing around with following message model:
1st message: Meta Data describing what data to expect and some other info.
2nd message: DataBegin
3rd message: DataItem
4th message: DataItem
...
nth message: EndData
Here's an example of a Data Item:
class Bar{
DateTime DateTime{get;set;}
float Open{get;set}
float High{get;set}
float Low{get;set}
float Close{get;set}
intVolume{get;set}
}
Right now I'm using TypeModel.SerializeWithLengthPrefix(...) to serialize each message (TypeModel is compiled). Which works great, but it's about 10x slower than serializing each message manually using a BinaryWriter. What matters here of course is not the meta data but the serialization of each DataItem. I have a lot of data and in some cases it's read/written to a file and there performance is crucial.
What would be a good way of increasing the performance of the serialization and deserialization of each DataItem?
Should I use ProtoWriter directly here? If yes how would I do this (i'm a bit new to Protocol Buffers).
Yes, if your data is a very simple set of homogeneous records, with no additional requirements (for example, it doesn't need to be forwards compatible or version elegantly, or be usable from clients that don't fully know all the data), doesn't need to be conveniently portable, and you don't mind implementing all the serialization manually, then yes: you can do it more efficiently manually. In a quick test:
protobuf-net serialize: 55ms, 3581680 bytes
protobuf-net deserialize: 65ms, 100000 items
BinaryFormatter serialize: 443ms, 4200629 bytes
BinaryFormatter deserialize: 745ms, 100000 items
manual serialize: 26ms, 2800004 bytes
manual deserialize: 32ms, 100000 items
The extra space is presumably the field markers (which you don't need if you are packing the records manually and don't need to worry about different versions of the API in use at the same time).
I certainly don't reproduce "10x"; I get 2x, which isn't bad considering the things that protobuf offers. And is certainly a lot better than BinaryFormatter, which is more like 20x! Here's some of the features:
version tolerance
portability
schema usage
no manual code
inbuilt support for sub-objects and collections
support for omitting default values
support for common .NET scenarios (serialization callbacks; conditional serialization patterns, etc)
inheritance (protobuf-net only; not part of the standard protobuf spec)
It sounds like in your scenario manual serialization is the thing to do; that's fine - I'm not offended ;p the purpose of a serialization library is to address the more general problem in a way that doesn't need manual code writing.
My test rig:
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using ProtoBuf;
using ProtoBuf.Meta;
using System.Runtime.Serialization.Formatters.Binary;
public static class Program
{
static void Main()
{
var model = RuntimeTypeModel.Create();
model.Add(typeof(BarWrapper), true);
model.Add(typeof(Bar), true);
model.CompileInPlace();
var data = CreateBar(100000).ToList();
RunTest(model, data);
}
private static void RunTest(RuntimeTypeModel model, List<Bar> data)
{
using(var ms = new MemoryStream())
{
var watch = Stopwatch.StartNew();
model.Serialize(ms, new BarWrapper {Bars = data});
watch.Stop();
Console.WriteLine("protobuf-net serialize: {0}ms, {1} bytes", watch.ElapsedMilliseconds, ms.Length);
ms.Position = 0;
watch = Stopwatch.StartNew();
var bars = ((BarWrapper) model.Deserialize(ms, null, typeof (BarWrapper))).Bars;
watch.Stop();
Console.WriteLine("protobuf-net deserialize: {0}ms, {1} items", watch.ElapsedMilliseconds, bars.Count);
}
using (var ms = new MemoryStream())
{
var bf = new BinaryFormatter();
var watch = Stopwatch.StartNew();
bf.Serialize(ms, new BarWrapper { Bars = data });
watch.Stop();
Console.WriteLine("BinaryFormatter serialize: {0}ms, {1} bytes", watch.ElapsedMilliseconds, ms.Length);
ms.Position = 0;
watch = Stopwatch.StartNew();
var bars = ((BarWrapper)bf.Deserialize(ms)).Bars;
watch.Stop();
Console.WriteLine("BinaryFormatter deserialize: {0}ms, {1} items", watch.ElapsedMilliseconds, bars.Count);
}
byte[] raw;
using (var ms = new MemoryStream())
{
var watch = Stopwatch.StartNew();
WriteBars(ms, data);
watch.Stop();
raw = ms.ToArray();
Console.WriteLine("manual serialize: {0}ms, {1} bytes", watch.ElapsedMilliseconds, raw.Length);
}
using(var ms = new MemoryStream(raw))
{
var watch = Stopwatch.StartNew();
var bars = ReadBars(ms);
watch.Stop();
Console.WriteLine("manual deserialize: {0}ms, {1} items", watch.ElapsedMilliseconds, bars.Count);
}
}
static IList<Bar> ReadBars(Stream stream)
{
using(var reader = new BinaryReader(stream))
{
int count = reader.ReadInt32();
var bars = new List<Bar>(count);
while(count-- > 0)
{
var bar = new Bar();
bar.DateTime = DateTime.FromBinary(reader.ReadInt64());
bar.Open = reader.ReadInt32();
bar.High = reader.ReadInt32();
bar.Low = reader.ReadInt32();
bar.Close = reader.ReadInt32();
bar.Volume = reader.ReadInt32();
bars.Add(bar);
}
return bars;
}
}
static void WriteBars(Stream stream, IList<Bar> bars )
{
using(var writer = new BinaryWriter(stream))
{
writer.Write(bars.Count);
foreach (var bar in bars)
{
writer.Write(bar.DateTime.ToBinary());
writer.Write(bar.Open);
writer.Write(bar.High);
writer.Write(bar.Low);
writer.Write(bar.Close);
writer.Write(bar.Volume);
}
}
}
static IEnumerable<Bar> CreateBar(int count)
{
var rand = new Random(12345);
while(count-- > 0)
{
var bar = new Bar();
bar.DateTime = new DateTime(
rand.Next(2008,2011), rand.Next(1,13), rand.Next(1, 29),
rand.Next(0,24), rand.Next(0,60), rand.Next(0,60));
bar.Open = (float) rand.NextDouble();
bar.High = (float)rand.NextDouble();
bar.Low = (float)rand.NextDouble();
bar.Close = (float)rand.NextDouble();
bar.Volume = rand.Next(-50000, 50000);
yield return bar;
}
}
}
[ProtoContract]
[Serializable] // just for BinaryFormatter test
public class BarWrapper
{
[ProtoMember(1, DataFormat = DataFormat.Group)]
public List<Bar> Bars { get; set; }
}
[ProtoContract]
[Serializable] // just for BinaryFormatter test
public class Bar
{
[ProtoMember(1)]
public DateTime DateTime { get; set; }
[ProtoMember(2)]
public float Open { get; set; }
[ProtoMember(3)]
public float High { get; set; }
[ProtoMember(4)]
public float Low { get; set; }
[ProtoMember(5)]
public float Close { get; set; }
// use zigzag if it can be -ve/+ve, or default if non-negative only
[ProtoMember(6, DataFormat = DataFormat.ZigZag)]
public int Volume { get; set; }
}