Deserialization fail after switching PrefixStyle to Fixed32 - protocol-buffers

I have a test suite for a communication library I'm developing using protobuf-net that is running okay. All tests pass. But if I change the PrefixStyle from Base128 to Fixed32, the deserialization fail.
The exception I receive from the TryDeserializeWithLengthPrefix function is:
System.ArgumentNullException was caught
Message="Value cannot be null.\r\nParameter name: type"
Source="protobuf-net"
ParamName="type"
Everything just works if I simple keep the PrefixStyle.Base128 when serializing and deserializing the message.
Does anyone know what may be happening?

Hohum, yup looks like a bug (now logged); repeatable example below. I'll see if I can fix it on the train (shortly). Sorry 'bout that:
using System;
using System.IO;
using ProtoBuf;
[ProtoContract]
public class Strange // test entity
{
[ProtoMember(1)]
public string Foo { get; set; } // test prop
[ProtoMember(2)]
public int Bar { get; set; } // test prop
static void Main() {
var original = new Strange { Foo = "abc", Bar = 123 };
// serialize and deserialize with base-128
using (MemoryStream ms = new MemoryStream()) {
Serializer.SerializeWithLengthPrefix(ms, original, PrefixStyle.Base128,1);
ms.Position = 0;
object obj;
Serializer.NonGeneric.TryDeserializeWithLengthPrefix(ms,
PrefixStyle.Base128, i => typeof(Strange),out obj);
var clone = (Strange)obj;
Console.WriteLine("Foo via Base128: " + clone.Foo); // works fine
Console.WriteLine("Bar via Base128: " + clone.Bar);
}
// serialize and deserialize with fixed-32
using (MemoryStream ms = new MemoryStream())
{
Serializer.SerializeWithLengthPrefix(ms, original, PrefixStyle.Fixed32,1);
ms.Position = 0;
object obj;
// BOOM here; oh how embarrassing
Serializer.NonGeneric.TryDeserializeWithLengthPrefix(ms,
PrefixStyle.Fixed32, i => typeof(Strange), out obj);
var clone = (Strange)obj;
Console.WriteLine("Foo via Fixed32: " + clone.Foo);
Console.WriteLine("Bar via Fixed32: " + clone.Bar);
}
}
}

Related

Error "Must set UnitOfWorkManager before use it"

I'm developing the service within ASP.NET Boilerplate engine and getting the error from the subject. The nature of the error is not clear, as I inheriting from ApplicationService, as documentation suggests. The code:
namespace MyAbilities.Api.Blob
{
public class BlobService : ApplicationService, IBlobService
{
public readonly IRepository<UserMedia, int> _blobRepository;
public BlobService(IRepository<UserMedia, int> blobRepository)
{
_blobRepository = blobRepository;
}
public async Task<List<BlobDto>> UploadBlobs(HttpContent httpContent)
{
var blobUploadProvider = new BlobStorageUploadProvider();
var list = await httpContent.ReadAsMultipartAsync(blobUploadProvider)
.ContinueWith(task =>
{
if (task.IsFaulted || task.IsCanceled)
{
if (task.Exception != null) throw task.Exception;
}
var provider = task.Result;
return provider.Uploads.ToList();
});
// store blob info in the database
foreach (var blobDto in list)
{
SaveBlobData(blobDto);
}
return list;
}
public void SaveBlobData(BlobDto blobData)
{
UserMedia um = blobData.MapTo<UserMedia>();
_blobRepository.InsertOrUpdateAndGetId(um);
CurrentUnitOfWork.SaveChanges();
}
public async Task<BlobDto> DownloadBlob(int blobId)
{
// TODO: Implement this helper method. It should retrieve blob info
// from the database, based on the blobId. The record should contain the
// blobName, which should be returned as the result of this helper method.
var blobName = GetBlobName(blobId);
if (!String.IsNullOrEmpty(blobName))
{
var container = BlobHelper.GetBlobContainer();
var blob = container.GetBlockBlobReference(blobName);
// Download the blob into a memory stream. Notice that we're not putting the memory
// stream in a using statement. This is because we need the stream to be open for the
// API controller in order for the file to actually be downloadable. The closing and
// disposing of the stream is handled by the Web API framework.
var ms = new MemoryStream();
await blob.DownloadToStreamAsync(ms);
// Strip off any folder structure so the file name is just the file name
var lastPos = blob.Name.LastIndexOf('/');
var fileName = blob.Name.Substring(lastPos + 1, blob.Name.Length - lastPos - 1);
// Build and return the download model with the blob stream and its relevant info
var download = new BlobDto
{
FileName = fileName,
FileUrl = Convert.ToString(blob.Uri),
FileSizeInBytes = blob.Properties.Length,
ContentType = blob.Properties.ContentType
};
return download;
}
// Otherwise
return null;
}
//Retrieve blob info from the database
private string GetBlobName(int blobId)
{
throw new NotImplementedException();
}
}
}
The error appears even before the app flow jumps to 'SaveBlobData' method. Am I missed something?
Hate to answer my own questions, but here it is... after a while, I found out that if UnitOfWorkManager is not available for some reason, I can instantiate it in the code, by initializing IUnitOfWorkManager in the constructor. Then, you can simply use the following construction in your Save method:
using (var unitOfWork = _unitOfWorkManager.Begin())
{
//Save logic...
unitOfWork.Complete();
}

Is there any straightforward way to populate and update a Realm-Xamarin from JSON?

I'm trying to port an Android app with a Realm to Xamarin so it'll be also available for iOS devices. In Android, I have several JSON files with some necessary initial data, e.g. cities.json, and I import it at the beginning with realm.createOrUpdateAllFromJson(Class<E> clazz, InputStream in) method, like this:
private void loadInitialCities(Realm realm) {
InputStream stream = context.getAssets().open("data/cities.json");
realm.createOrUpdateAllFromJson(City.class, stream);
}
I also find this method very useful when retrieving data from a web service in form of JSON.
Now with Xamarin I don't see any equivalent to such method. Is there any method to achieve this? Or at least a workaround/tool to create a RealmObject from a JSON in C#?
I wrote my own extension methods for doing this (yes, I miss the built-in helper methods also).
https://github.com/sushihangover/Realm.Json.Extensions
Here is a basic example of how I do it:
JSON Model:
[
{
"name": "Alabama",
"abbreviation": "AL"
},
{
"name": "Alaska",
"abbreviation": "AK"
},
~~~~
]
Realm Model:
public class State : RealmObject
{
public string name { get; set; }
public string abbreviation { get; set; }
}
Xamarin.Android asset and Newtonsoft Streaming reader:
var config = RealmConfiguration.DefaultConfiguration;
config.SchemaVersion = 1;
using (var theRealm = Realm.GetInstance("StackOverflow.realm"))
using (var assetStream = Assets.Open("States.json"))
using (var streamReader = new StreamReader(assetStream))
using (var jsonTextReader = new JsonTextReader(streamReader))
{
var serializer = new JsonSerializer();
if (!jsonTextReader.Read() || jsonTextReader.TokenType != JsonToken.StartArray)
throw new Exception("Bad Json, start of array missing");
while (jsonTextReader.Read())
{
if (jsonTextReader.TokenType == JsonToken.EndArray)
break;
var state = serializer.Deserialize<State>(jsonTextReader);
theRealm.Write(() =>
{
var realmState = theRealm.CreateObject<State>();
realmState.abbreviation = state.abbreviation;
realmState.name = state.name;
});
}
}
Update: One of my extensions methods:
Extension Method Usage:
using (var theRealm = Realm.GetInstance("StackOverflow.realm"))
using (var assetStream = Assets.Open("States.json"))
{
theRealm.JsonArrayToRealm<State>(assetStream);
}
Extension Method:
Note: This uses AutoMapper to copy RealmObject and avoid reflection, also using Newtonsoft.Json.
public static class RealmDoesJson
{
public static void JsonArrayToRealm<T>(this Realm realm, Stream stream) where T : RealmObject
{
Mapper.Initialize(cfg =>
{
cfg.CreateMap<T, T>();
});
using (var streamReader = new StreamReader(stream))
using (var jsonTextReader = new JsonTextReader(streamReader))
{
var serializer = new JsonSerializer();
if (!jsonTextReader.Read() || jsonTextReader.TokenType != JsonToken.StartArray)
throw new Exception("MALFORMED JSON, Start of Array missing");
while (jsonTextReader.Read())
{
if (jsonTextReader.TokenType == JsonToken.EndArray)
break;
var jsonObject = serializer.Deserialize<T>(jsonTextReader);
realm.Write(() => // inside while loop / single object transaction for memory manangement reasons...
{
var realmObject = realm.CreateObject(typeof(T).Name);
Mapper.Map<T, T>(jsonObject, realmObject);
});
}
}
}
}

install nopcommerce with oracle

hello I downloaded the solution nopCommerce an e-commerce open source which achievement operate and install without problems with MSSQLSERVER database however I would like to implement with ORACLEdatabase
Official Site http://www.nopcommerce.com/
I have been guiding me this post
http://www.nopcommerce.com/boards/t/17712/mysql-support.aspx
I have tried to follow the steps indicated for mysql and adapt to oracle yet one of the first things that tells me is the creation of two classes
OracleConnectionFactory:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Data.Entity.Infrastructure;
using System.Data.Common;
using Oracle.DataAccess.Client;
namespace Nop.Data
{
public class OracleConnectionFactory : IDbConnectionFactory
{
private readonly string _baseConnectionString;
private Func<string, DbProviderFactory> _providerFactoryCreator;
public OracleConnectionFactory()
{
}
public OracleConnectionFactory(string baseConnectionString)
{
this._baseConnectionString = baseConnectionString;
}
public DbConnection CreateConnection(string nameOrConnectionString)
{
string connectionString = nameOrConnectionString;
bool treatAsConnectionString = nameOrConnectionString.IndexOf('=') >= 0;
if (!treatAsConnectionString)
{
OracleConnectionStringBuilder builder = new OracleConnectionStringBuilder(this.BaseConnectionString);
//MySqlConnectionStringBuilder builder = new MySqlConnectionStringBuilder(this.BaseConnectionString);
//builder.Server = nameOrConnectionString;
connectionString = builder.ConnectionString;
}
DbConnection connection = null;
try
{
connection = this.ProviderFactory("Oracle.DataAccess.Client").CreateConnection();
connection.ConnectionString = connectionString;
}
catch
{
//connection = new MySqlConnection(connectionString);
connection = new OracleConnection(connectionString);
}
return connection;
}
public string BaseConnectionString
{
get
{
return this._baseConnectionString;
}
}
internal Func<string, DbProviderFactory> ProviderFactory
{
get
{
Func<string, DbProviderFactory> func1 = this._providerFactoryCreator;
return delegate(string name)
{
return DbProviderFactories.GetFactory(name);
};
}
set
{
this._providerFactoryCreator = value;
}
}
}
}
OracleProvider :
using System;
using System.Collections.Generic;
using System.Data.Common;
using System.Data.Entity;
using System.Data.Entity.Infrastructure;
using System.Data.SqlClient;
using System.IO;
using System.Text;
using System.Web.Hosting;
using Nop.Data.Initializers;
using Oracle.DataAccess.Client;
using Nop.Core.Data;
namespace Nop.Data
{
public class OracleDataProvider : IDataProvider
{
#region Utilities
protected virtual string[] ParseCommands(string filePath, bool throwExceptionIfNonExists)
{
if (!File.Exists(filePath))
{
if (throwExceptionIfNonExists)
throw new ArgumentException(string.Format("Specified file doesn't exist - {0}", filePath));
else
return new string[0];
}
var statements = new List<string>();
using (var stream = File.OpenRead(filePath))
using (var reader = new StreamReader(stream))
{
var statement = "";
while ((statement = readNextStatementFromStream(reader)) != null)
{
statements.Add(statement);
}
}
return statements.ToArray();
}
protected virtual string readNextStatementFromStream(StreamReader reader)
{
var sb = new StringBuilder();
string lineOfText;
while (true)
{
lineOfText = reader.ReadLine();
if (lineOfText == null)
{
if (sb.Length > 0)
return sb.ToString();
else
return null;
}
//MySql doesn't support GO, so just use a commented out GO as the separator
if (lineOfText.TrimEnd().ToUpper() == "-- GO")
break;
sb.Append(lineOfText + Environment.NewLine);
}
return sb.ToString();
}
#endregion
#region Methods
public virtual void InitConnectionFactory()
{
//var connectionFactory = new SqlConnectionFactory();
var connectionFactory = new OracleConnectionFactory();
//TODO fix compilation warning (below)
#pragma warning disable 0618
Database.DefaultConnectionFactory = connectionFactory;
}
/// <summary>
/// Initialize database
/// </summary>
public virtual void InitDatabase()
{
InitConnectionFactory();
SetDatabaseInitializer();
}
/// <summary>
/// Set database initializer
/// </summary>
public virtual void SetDatabaseInitializer()
{
//pass some table names to ensure that we have nopCommerce 2.X installed
var tablesToValidate = new[] { "Customer", "Discount", "Order", "Product", "ShoppingCartItem" };
//custom commands (stored proedures, indexes)
var customCommands = new List<string>();
//use webHelper.MapPath instead of HostingEnvironment.MapPath which is not available in unit tests
customCommands.AddRange(ParseCommands(HostingEnvironment.MapPath("~/App_Data/Install/SqlServer.Indexes.sql"), false));
//use webHelper.MapPath instead of HostingEnvironment.MapPath which is not available in unit tests
customCommands.AddRange(ParseCommands(HostingEnvironment.MapPath("~/App_Data/Install/SqlServer.StoredProcedures.sql"), false));
var initializer = new CreateTablesIfNotExist<NopObjectContext>(tablesToValidate, customCommands.ToArray());
Database.SetInitializer(initializer);
}
/// <summary>
/// A value indicating whether this data provider supports stored procedures
/// </summary>
public virtual bool StoredProceduredSupported
{
get { return true; }
}
/// <summary>
/// Gets a support database parameter object (used by stored procedures)
/// </summary>
/// <returns>Parameter</returns>
public virtual DbParameter GetParameter()
{
//return new SqlParameter();
return new OracleParameter();
}
#endregion
}
}
also i installed the managed nuget package like this link said
http://www.oracle.com/webfolder/technetwork/tutorials/obe/db/dotnet/CodeFirst/index.html
oracle odp.net managed driver
in nop.data and nop.web
I appreciate any help freshened up the steps I need to do or that I may be going
one of the first thing i trying is the Oracle provider recognize and achieve connect to my database
It was an interesting question. Using nop commerce with Oracle is technically possible, but it would be a very wild ride for you.
Good news first. Nop Commerce is architectured based on repository pattern. Basically Nop.Data abstracts all the SQL Server related data access. You would need to re-write this almost entirely.
Not so good news next. Search, Paging, Catalog listing uses a stored procedure in SQL Server. You may need to re-write it completely. And most of the time, you are on your own. Unless if you are confident with Oracle and .Net EF, it would be a really wild ride.
I would say, its less problem if you want to stick with SQL Server. I understand sometime you may not make technical decision. So you can explain clearly about complexity and effort needed to migrate to the person makes that decision.
Source: NopCommerce developer for last 2.5years.

Entity Framework, Code First and Full Text Search

I realize that a lot of questions have been asked relating to full text search and Entity Framework, but I hope this question is a bit different.
I am using Entity Framework, Code First and need to do a full text search. When I need to perform the full text search, I will typically have other criteria/restrictions as well - like skip the first 500 rows, or filter on another column, etc.
I see that this has been handled using table valued functions - see http://sqlblogcasts.com/blogs/simons/archive/2008/12/18/LINQ-to-SQL---Enabling-Fulltext-searching.aspx. And this seems like the right idea.
Unfortunately, table valued functions are not supported until Entity Framework 5.0 (and even then, I believe, they are not supported for Code First).
My real question is what are the suggestions for the best way to handle this, both for Entity Framework 4.3 and Entity Framework 5.0. But to be specific:
Other than dynamic SQL (via System.Data.Entity.DbSet.SqlQuery, for example), are there any options available for Entity Framework 4.3?
If I upgrade to Entity Framework 5.0, is there a way I can use table valued functions with code first?
Thanks,
Eric
Using interceptors introduced in EF6, you could mark the full text search in linq and then replace it in dbcommand as described in http://www.entityframework.info/Home/FullTextSearch:
public class FtsInterceptor : IDbCommandInterceptor
{
private const string FullTextPrefix = "-FTSPREFIX-";
public static string Fts(string search)
{
return string.Format("({0}{1})", FullTextPrefix, search);
}
public void NonQueryExecuting(DbCommand command, DbCommandInterceptionContext<int> interceptionContext)
{
}
public void NonQueryExecuted(DbCommand command, DbCommandInterceptionContext<int> interceptionContext)
{
}
public void ReaderExecuting(DbCommand command, DbCommandInterceptionContext<DbDataReader> interceptionContext)
{
RewriteFullTextQuery(command);
}
public void ReaderExecuted(DbCommand command, DbCommandInterceptionContext<DbDataReader> interceptionContext)
{
}
public void ScalarExecuting(DbCommand command, DbCommandInterceptionContext<object> interceptionContext)
{
RewriteFullTextQuery(command);
}
public void ScalarExecuted(DbCommand command, DbCommandInterceptionContext<object> interceptionContext)
{
}
public static void RewriteFullTextQuery(DbCommand cmd)
{
string text = cmd.CommandText;
for (int i = 0; i < cmd.Parameters.Count; i++)
{
DbParameter parameter = cmd.Parameters[i];
if (parameter.DbType.In(DbType.String, DbType.AnsiString, DbType.StringFixedLength, DbType.AnsiStringFixedLength))
{
if (parameter.Value == DBNull.Value)
continue;
var value = (string)parameter.Value;
if (value.IndexOf(FullTextPrefix) >= 0)
{
parameter.Size = 4096;
parameter.DbType = DbType.AnsiStringFixedLength;
value = value.Replace(FullTextPrefix, ""); // remove prefix we added n linq query
value = value.Substring(1, value.Length - 2);
// remove %% escaping by linq translator from string.Contains to sql LIKE
parameter.Value = value;
cmd.CommandText = Regex.Replace(text,
string.Format(
#"\[(\w*)\].\[(\w*)\]\s*LIKE\s*#{0}\s?(?:ESCAPE N?'~')",
parameter.ParameterName),
string.Format(#"contains([$1].[$2], #{0})",
parameter.ParameterName));
if (text == cmd.CommandText)
throw new Exception("FTS was not replaced on: " + text);
text = cmd.CommandText;
}
}
}
}
}
static class LanguageExtensions
{
public static bool In<T>(this T source, params T[] list)
{
return (list as IList<T>).Contains(source);
}
}
For example, if you have class Note with FTS-indexed field NoteText:
public class Note
{
public int NoteId { get; set; }
public string NoteText { get; set; }
}
and EF map for it
public class NoteMap : EntityTypeConfiguration<Note>
{
public NoteMap()
{
// Primary Key
HasKey(t => t.NoteId);
}
}
and context for it:
public class MyContext : DbContext
{
static MyContext()
{
DbInterception.Add(new FtsInterceptor());
}
public MyContext(string nameOrConnectionString) : base(nameOrConnectionString)
{
}
public DbSet<Note> Notes { get; set; }
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
modelBuilder.Configurations.Add(new NoteMap());
}
}
you can have quite simple syntax to FTS query:
class Program
{
static void Main(string[] args)
{
var s = FtsInterceptor.Fts("john");
using (var db = new MyContext("CONNSTRING"))
{
var q = db.Notes.Where(n => n.NoteText.Contains(s));
var result = q.Take(10).ToList();
}
}
}
That will generate SQL like
exec sp_executesql N'SELECT TOP (10)
[Extent1].[NoteId] AS [NoteId],
[Extent1].[NoteText] AS [NoteText]
FROM [NS].[NOTES] AS [Extent1]
WHERE contains([Extent1].[NoteText], #p__linq__0)',N'#p__linq__0 char(4096)',#p__linq__0='(john)
Please notice that you should use local variable and cannot move FTS wrapper inside expression like
var q = db.Notes.Where(n => n.NoteText.Contains(FtsInterceptor.Fts("john")));
I have found that the easiest way to implement this is to setup and configure full-text-search in SQL Server and then use a stored procedure. Pass your arguments to SQL, allow the DB to do its job and return either a complex object or map the results to an entity. You don't necessarily have to have dynamic SQL, but it may be optimal. For example, if you need paging, you could pass in PageNumber and PageSize on every request without the need for dynamic SQL. However, if the number of arguments fluctuates per query, it will be the optimal solution.
As the other guys mentioned, I would say start using Lucene.NET
Lucene has a pretty high learning curve, but I found an wrapper for it called "SimpleLucene", that can be found on CodePlex
Let me quote a couple of codeblocks from the blog to show you how easy it is to use. I've just started to use it, but got the hang of it really fast.
First, get some entities from your repository, or in your case, use Entity Framework
public class Repository
{
public IList<Product> Products {
get {
return new List<Product> {
new Product { Id = 1, Name = "Football" },
new Product { Id = 2, Name = "Coffee Cup"},
new Product { Id = 3, Name = "Nike Trainers"},
new Product { Id = 4, Name = "Apple iPod Nano"},
new Product { Id = 5, Name = "Asus eeePC"},
};
}
}
}
The next thing you want to do is create an index-definition
public class ProductIndexDefinition : IIndexDefinition<Product> {
public Document Convert(Product p) {
var document = new Document();
document.Add(new Field("id", p.Id.ToString(), Field.Store.YES, Field.Index.NOT_ANALYZED));
document.Add(new Field("name", p.Name, Field.Store.YES, Field.Index.ANALYZED));
return document;
}
public Term GetIndex(Product p) {
return new Term("id", p.Id.ToString());
}
}
and create an search index for it.
var writer = new DirectoryIndexWriter(
new DirectoryInfo(#"c:\index"), true);
var service = new IndexService();
service.IndexEntities(writer, Repository().Products, ProductIndexDefinition());
So, you now have an search-able index. The only remaining thing to do is.., searching! You can do pretty amazing things, but it can be as easy as this: (for greater examples see the blog or the documentation on codeplex)
var searcher = new DirectoryIndexSearcher(
new DirectoryInfo(#"c:\index"), true);
var query = new TermQuery(new Term("name", "Football"));
var searchService = new SearchService();
Func<Document, ProductSearchResult> converter = (doc) => {
return new ProductSearchResult {
Id = int.Parse(doc.GetValues("id")[0]),
Name = doc.GetValues("name")[0]
};
};
IList<Product> results = searchService.SearchIndex(searcher, query, converter);
The example here http://www.entityframework.info/Home/FullTextSearch is not complete solution. You will need to look into understand how the full text search works. Imagine you have a search field and the user types 2 words to hit search. The above code will throw an exception. You need to do pre-processing on the search phrase first to pass it to the query by using logical AND or OR.
for example your search phrase is "blah blah2" then you need to convert this into:
var searchTerm = #"\"blah\" AND/OR \"blah2\" ";
Complete solution would be:
value = Regex.Replace(value, #"\s+", " "); //replace multiplespaces
value = Regex.Replace(value, #"[^a-zA-Z0-9 -]", "").Trim();//remove non-alphanumeric characters and trim spaces
if (value.Any(Char.IsWhiteSpace))
{
value = PreProcessSearchKey(value);
}
public static string PreProcessSearchKey(string searchKey)
{
var splitedKeyWords = searchKey.Split(null); //split from whitespaces
// string[] addDoubleQuotes = new string[splitedKeyWords.Length];
for (int j = 0; j < splitedKeyWords.Length; j++)
{
splitedKeyWords[j] = $"\"{splitedKeyWords[j]}\"";
}
return string.Join(" AND ", splitedKeyWords);
}
this methods uses AND logic operator. You might pass that as an argument and use the method for both AND or OR operators.
You must escape none-alphanumeric characters otherwise it would throw exception when a user enters alpha numeric characters and you have no server site model level validation in place.
I recently had a similar requirement and ended up writing an IQueryable extension specifically for Microsoft full text index access, its available here IQueryableFreeTextExtensions

How to squeeze the most performance out of probuf-net

I want to use protobuf-net to serialize stock market data. I'm playing around with following message model:
1st message: Meta Data describing what data to expect and some other info.
2nd message: DataBegin
3rd message: DataItem
4th message: DataItem
...
nth message: EndData
Here's an example of a Data Item:
class Bar{
DateTime DateTime{get;set;}
float Open{get;set}
float High{get;set}
float Low{get;set}
float Close{get;set}
intVolume{get;set}
}
Right now I'm using TypeModel.SerializeWithLengthPrefix(...) to serialize each message (TypeModel is compiled). Which works great, but it's about 10x slower than serializing each message manually using a BinaryWriter. What matters here of course is not the meta data but the serialization of each DataItem. I have a lot of data and in some cases it's read/written to a file and there performance is crucial.
What would be a good way of increasing the performance of the serialization and deserialization of each DataItem?
Should I use ProtoWriter directly here? If yes how would I do this (i'm a bit new to Protocol Buffers).
Yes, if your data is a very simple set of homogeneous records, with no additional requirements (for example, it doesn't need to be forwards compatible or version elegantly, or be usable from clients that don't fully know all the data), doesn't need to be conveniently portable, and you don't mind implementing all the serialization manually, then yes: you can do it more efficiently manually. In a quick test:
protobuf-net serialize: 55ms, 3581680 bytes
protobuf-net deserialize: 65ms, 100000 items
BinaryFormatter serialize: 443ms, 4200629 bytes
BinaryFormatter deserialize: 745ms, 100000 items
manual serialize: 26ms, 2800004 bytes
manual deserialize: 32ms, 100000 items
The extra space is presumably the field markers (which you don't need if you are packing the records manually and don't need to worry about different versions of the API in use at the same time).
I certainly don't reproduce "10x"; I get 2x, which isn't bad considering the things that protobuf offers. And is certainly a lot better than BinaryFormatter, which is more like 20x! Here's some of the features:
version tolerance
portability
schema usage
no manual code
inbuilt support for sub-objects and collections
support for omitting default values
support for common .NET scenarios (serialization callbacks; conditional serialization patterns, etc)
inheritance (protobuf-net only; not part of the standard protobuf spec)
It sounds like in your scenario manual serialization is the thing to do; that's fine - I'm not offended ;p the purpose of a serialization library is to address the more general problem in a way that doesn't need manual code writing.
My test rig:
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using ProtoBuf;
using ProtoBuf.Meta;
using System.Runtime.Serialization.Formatters.Binary;
public static class Program
{
static void Main()
{
var model = RuntimeTypeModel.Create();
model.Add(typeof(BarWrapper), true);
model.Add(typeof(Bar), true);
model.CompileInPlace();
var data = CreateBar(100000).ToList();
RunTest(model, data);
}
private static void RunTest(RuntimeTypeModel model, List<Bar> data)
{
using(var ms = new MemoryStream())
{
var watch = Stopwatch.StartNew();
model.Serialize(ms, new BarWrapper {Bars = data});
watch.Stop();
Console.WriteLine("protobuf-net serialize: {0}ms, {1} bytes", watch.ElapsedMilliseconds, ms.Length);
ms.Position = 0;
watch = Stopwatch.StartNew();
var bars = ((BarWrapper) model.Deserialize(ms, null, typeof (BarWrapper))).Bars;
watch.Stop();
Console.WriteLine("protobuf-net deserialize: {0}ms, {1} items", watch.ElapsedMilliseconds, bars.Count);
}
using (var ms = new MemoryStream())
{
var bf = new BinaryFormatter();
var watch = Stopwatch.StartNew();
bf.Serialize(ms, new BarWrapper { Bars = data });
watch.Stop();
Console.WriteLine("BinaryFormatter serialize: {0}ms, {1} bytes", watch.ElapsedMilliseconds, ms.Length);
ms.Position = 0;
watch = Stopwatch.StartNew();
var bars = ((BarWrapper)bf.Deserialize(ms)).Bars;
watch.Stop();
Console.WriteLine("BinaryFormatter deserialize: {0}ms, {1} items", watch.ElapsedMilliseconds, bars.Count);
}
byte[] raw;
using (var ms = new MemoryStream())
{
var watch = Stopwatch.StartNew();
WriteBars(ms, data);
watch.Stop();
raw = ms.ToArray();
Console.WriteLine("manual serialize: {0}ms, {1} bytes", watch.ElapsedMilliseconds, raw.Length);
}
using(var ms = new MemoryStream(raw))
{
var watch = Stopwatch.StartNew();
var bars = ReadBars(ms);
watch.Stop();
Console.WriteLine("manual deserialize: {0}ms, {1} items", watch.ElapsedMilliseconds, bars.Count);
}
}
static IList<Bar> ReadBars(Stream stream)
{
using(var reader = new BinaryReader(stream))
{
int count = reader.ReadInt32();
var bars = new List<Bar>(count);
while(count-- > 0)
{
var bar = new Bar();
bar.DateTime = DateTime.FromBinary(reader.ReadInt64());
bar.Open = reader.ReadInt32();
bar.High = reader.ReadInt32();
bar.Low = reader.ReadInt32();
bar.Close = reader.ReadInt32();
bar.Volume = reader.ReadInt32();
bars.Add(bar);
}
return bars;
}
}
static void WriteBars(Stream stream, IList<Bar> bars )
{
using(var writer = new BinaryWriter(stream))
{
writer.Write(bars.Count);
foreach (var bar in bars)
{
writer.Write(bar.DateTime.ToBinary());
writer.Write(bar.Open);
writer.Write(bar.High);
writer.Write(bar.Low);
writer.Write(bar.Close);
writer.Write(bar.Volume);
}
}
}
static IEnumerable<Bar> CreateBar(int count)
{
var rand = new Random(12345);
while(count-- > 0)
{
var bar = new Bar();
bar.DateTime = new DateTime(
rand.Next(2008,2011), rand.Next(1,13), rand.Next(1, 29),
rand.Next(0,24), rand.Next(0,60), rand.Next(0,60));
bar.Open = (float) rand.NextDouble();
bar.High = (float)rand.NextDouble();
bar.Low = (float)rand.NextDouble();
bar.Close = (float)rand.NextDouble();
bar.Volume = rand.Next(-50000, 50000);
yield return bar;
}
}
}
[ProtoContract]
[Serializable] // just for BinaryFormatter test
public class BarWrapper
{
[ProtoMember(1, DataFormat = DataFormat.Group)]
public List<Bar> Bars { get; set; }
}
[ProtoContract]
[Serializable] // just for BinaryFormatter test
public class Bar
{
[ProtoMember(1)]
public DateTime DateTime { get; set; }
[ProtoMember(2)]
public float Open { get; set; }
[ProtoMember(3)]
public float High { get; set; }
[ProtoMember(4)]
public float Low { get; set; }
[ProtoMember(5)]
public float Close { get; set; }
// use zigzag if it can be -ve/+ve, or default if non-negative only
[ProtoMember(6, DataFormat = DataFormat.ZigZag)]
public int Volume { get; set; }
}

Resources