Facebook Twitter Gplus LinkedIn RSS
formats

Silverlight Binary Serialization

Download

For the latest information, version and more complete documentation, please click here.

Overview

SilverlightSerializer is a binary serializer that works in .NET and Silverlight, allowing you to move your objects between the two instances or store and retrieve them locally in either.  It is highly performant and produces small files which are suitable for further compression if you need to.

It can serialize just about anything, including complex multi-dimensional arrays of complex objects. It can only serialize public fields and properties, unless you write a custom serializer for a particular type.

The serializer can also produce Checksums that can be used to compare different objects to see if they have the same meaning, or to use as file names to see if you have a cached version already available.

There is now a Silverlight Serializer page on this blog where I will keep a version history and add revisions. This page has additional instructions on how to use the serializer and configure your classes, I suggest you take a look at that as I’m going to stop updating this post and start putting information on there.  Feel free to comment in either location however.

Commentary

I finally got fed up with the terrible performance of the DataContract serializer, especially for shared class libraries and was forced to write my own.

The serializer I have created doesn’t need to know about KnownTypes, is capable of suppressing a property or field with a [DoNotSerialize] attribute and writes to a byte array which is much smaller than the bloated XML format favoured by DataContractSerializer. It also handles object references, which you can’t make DCS do in a shared library.

The serialization won’t waste space and therefore doesn’t serialize properties/fields that have their initial values from the constructor. It won’t bother serializing empty collections etc and it uses tokens for type names and property names to save space; though I’m also using SharpZip to compress the output further in a wrapper I’ve put around the serializer in my own code (not shown here, or in the example project). This is sometimes necessary as it does store the AssemblyQualifiedName of the types in the graph (which is the trade off for not marking everything with KnownType in advance). I might look at a way of adding that functionality myself later, but my version doesn’t have problems with conflicting names that DCS frequently pops up.

So you can find the project here if you are interested, feel free to use or modify it in any way you like.

Here’s an example of using it:

        public class outer
        {
            public enum MyModes
            {
                One,
                Two,
                Three
            };
            public string Name { get; set; }
            public Dictionary Dict { get; set; }
            public BaseItem[] Array { get; set; }
            public MyModes Mode { get; set; }
            public outer()
            {
                Name = string.Empty;
                Dict = new Dictionary();
                Array = new BaseItem[10];
            }
        }

Calling the serializer:

             var s = new outer();
            var br = new Branch();  //One of my complex objects
            br.Elements.Add(new FilterBranch());
            s.Dict["Hello"] = br;
            s.Array[2] = new Campaign();
            s.Mode = outer.MyModes.Two;

            var bytes = SilverlightSerializer.Serialize(s);
            var item = SilverlightSerializer.Deserialize(bytes)  as outer;

Update: if you want to share classes between Silverlight and .NET4 make sure you read this post first!

Significant update:

Thanks to Peter and Steve I have modified the serializer so that it handles arrays with more than 2 dimensions and can construct some further types of object.

I’ve also updated it so that you can write your own custom serializers for your types and indeed for system types as well.  The way you do this is by creating a class that implements ISerializeObject and decorate it with a SerializerAttribute which declares the type that the item can serialize (one class is allowed to serialize more than one type, by using the attribute more than once).  The methods of ISerializeObject will then be called for your type – this allows you to have objects without a parameterless constructor be serialized in the normal way.  You must register each assembly that contains these decorated extensions by calling SilverlightSerializer.RegisterSerializationAssembly – if you are calling it from the assembly that contains your extension you don’t need to pass a parameter.

If you want to make your serializer store private properties you can implement the interface on the actual item you want to serialize (still decorating it with the attribute) and therefore have access to them.

Here’s a class without a paramterless constructor and containing a complex array:

public class myClass
{
   public int Parameter;
   public string[,,] Myarray = new string[3,3,3];
   public myClass(int aParameter)
   {
      Parameter = aParameter;
   }
}

Now here’s the definition of a helper class to serialize it:

[Serializer(typeof(myClass))]
public class SerializeMyClass : ISerializeObject
{
   public object[] Serialize(object target)
   {
        return new object[]
           {
               ((myClass) target).Parameter,
               SilverlightSerializer.Serialize(((myClass)target).Myarray)
            };
   }

   public object Deserialize(object[] data)
   {
       var parm = (int) data[0];
       return new myClass(parm)
          {
              Myarray = SilverlightSerializer.Deserialize((byte[]) data[1]) as string[,,]
          };
    }
}

Serializing it might look something like this (though you would probably register your serialization extensions during some init phase):

SilverlightSerializer.RegisterSerializationAssembly();
var cls = new myClass(101);
cls.Myarray[0, 1, 1] = "Hello Mum";
var d = SilverlightSerializer.Serialize(cls);
var x = SilverlightSerializer.Deserialize(d);

 

Further Update

I’ve fixed a number of bugs with serializing Enums when they were stored in an object property and handle Decimals properly.

Silverlight Serializer In Action

Peter Bromberg has come up with a great article on EggheadCafe that combines my serializer with a compression library and some utility classes plus examples. Check it out here.

The serializer has been linked into a persistent storage library for client side isolated storage by Ashiq Alibhai that you can find here.

42 Responses

  1. Asif

    Hi,

    I am trying to use this in my Silverlight project where I want to return binary serialized object from my WCF RIA service layer. I am facing an issue here…

    In service layer SilverlightSerializer able to serialize with no issue.

    In my ViewModel, where I handle the callback from the service function by using InvokeOperation. The byte[] array is returned but when I try to de-serialize this byte[], I get an error…

    ——————————————————————————————————————————————————————————-
    Cannot reference type MyProject.Web.SLDataSet.DataSetData, MyProject.Web, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null in this context
    ——————————————————————————————————————————————————————————-

    Kindly guide me how can I fix this issue, It will be of a great help.

    Thanks

    • whydoidoit

      It appears that you need to add the reference to this SLDataSet to both sides of the project (the Silverlight and the WCF side).

      • Asif

        Thanks for the answer, I have a reference added of my object. Please look at the code..

        - WCF –
        byte[] ret = SilverlightSerializer.Serialize(dsd);

        - ViewModel –
        var dsd = SilverlightSerializer.Deserialize(iv.Value as byte[]);

        Exception thrown on the following line of the code of SilverlightSerializer’s SerializationUnits.cs

        if (tp == null)
        throw new ArgumentException(string.Format(“Cannot reference type {0} in this context”, typeName));
        SilverlightSerializer.KnownTypes.Add(tp.TypeHandle);

  2. show…

    [...]Silverlight Binary Serialization « Mike Talbot's Blog[...]…

  3. nicks

    hey hi MIke,

    After reading your article and scratching our head finally we understood compression and implemented it successfully in our academic project.
    The compression ratio was about average 85%.
    Thanks a lot for your post.Really appreciable

    Regards
    Shilpesh and Nikhil.

    • You are very welcome :)

      I did think about adding a version with compression here, but figured it was one of those things that people have a preference on so I didn’t want to force another library.

  4. Hello.

    I am trying to use your serializer with EF and I am getting an error that i am not being able to solve. In this particular procedure gives an error of index out of range:

    //Gets the name from the stream
    public void DeserializeGetName(Entry entry)
    {
    if (entry.MustHaveName)
    {
    ushort id = this.ReadSimpleValue();
    entry.Name = SilverlightSerializer.PropertyIds[id]; // <= here is the error – id = 25 but PropertyIds only has 20 records …

    }
    }

    The error is this :

    {System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
    Parameter name: index
    at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource)
    at System.ThrowHelper.ThrowArgumentOutOfRangeException()
    at System.Collections.Generic.List`1.get_Item(Int32 index)
    at Serialization.BinarySerializer.DeserializeGetName(Entry entry)
    at Serialization.SilverlightSerializer.DeserializeObject(Entry entry, IStorage storage)
    at Serialization.SilverlightSerializer.DeserializeProperties(IStorage storage, Type itemType, Object o)
    at Serialization.SilverlightSerializer.DeserializeObjectAndProperties(Object o, Type itemType, IStorage storage)
    at Serialization.SilverlightSerializer.DeserializeObject(Entry entry, IStorage storage)
    at Serialization.SilverlightSerializer.Deserialize(Stream inputStream, Object instance)
    at Serialization.SilverlightSerializer.Deserialize(Byte[] bytes)
    at Serialization.PayloadHelper.CompressedBytesToObject[T](Byte[] compressed)
    at ESUX.WebPresentation.Silverlight.Modules.PrescriptionReconciliation.ViewModels.ReconciliationModuleLoading.prescriptionReconciliationServiceClient_GetCompressedDrugListCompleted(Object sender, GetCompressedDrugListCompletedEventArgs e)
    at ESUX.WebPresentation.Silverlight.Modules.PrescriptionReconciliation.PrescriptionReconciliationClient.PrescriptionReconciliationServiceClient.OnGetCompressedDrugListCompleted(Object state)}

    the object that it is trying to deserialize is a collection (List).

    Can you please help me?

    Thanks!

  5. Matt

    Hi Mike,

    Is this compatible with SL 4 RIA. I am having a hard time making it work.
    I am not able to get the typeName of my object when desirializing .

    Matt

  6. Hi I try implement in my code but i have error in:

    private static object CreateObject(Type itemType)
    {
    try
    {
    return Activator.CreateInstance(itemType);
    }
    catch (Exception)
    {
    return itemType.GetConstructor(new Type[] { }).Invoke(new object[] { });

    }

    }

    the message error is:

    System.NullReferenceException was unhandled by user code
    Message=Object reference not set to an instance of an object.
    Source=Serialization
    StackTrace:
    at Serialization.SilverlightSerializer.CreateObject(Type itemType) in C:\Users\franciscor\Documents\Mis Proyectos\GeoViewer\lw_GeoViewer\Serialization\SilverlightSerializer.cs:line 1109
    at Serialization.SilverlightSerializer.SerializeObjectAndProperties(Object item, Type itemType, BinaryWriter writer) in C:\Users\franciscor\Documents\Mis Proyectos\GeoViewer\lw_GeoViewer\Serialization\SilverlightSerializer.cs:line 1092
    at Serialization.SilverlightSerializer.SerializeObject(Object item, BinaryWriter writer, Type propertyType) in C:\Users\franciscor\Documents\Mis Proyectos\GeoViewer\lw_GeoViewer\Serialization\SilverlightSerializer.cs:line 941
    at Serialization.SilverlightSerializer.SerializeList(IList item, Type tp, BinaryWriter writer) in C:\Users\franciscor\Documents\Mis Proyectos\GeoViewer\lw_GeoViewer\Serialization\SilverlightSerializer.cs:line 957
    at Serialization.SilverlightSerializer.SerializeObject(Object item, BinaryWriter writer, Type propertyType) in C:\Users\franciscor\Documents\Mis Proyectos\GeoViewer\lw_GeoViewer\Serialization\SilverlightSerializer.cs:line 935
    at Serialization.SilverlightSerializer.Serialize(Object item, Stream outputStream) in C:\Users\franciscor\Documents\Mis Proyectos\GeoViewer\lw_GeoViewer\Serialization\SilverlightSerializer.cs:line 799
    at Serialization.SilverlightSerializer.Serialize(Object item) in C:\Users\franciscor\Documents\Mis Proyectos\GeoViewer\lw_GeoViewer\Serialization\SilverlightSerializer.cs:line 834
    at lw_GeoViewer.Web.Model.PayloadHelper.ObjectToCompressedBytes(Object obj) in C:\Users\franciscor\Documents\Mis Proyectos\GeoViewer\lw_GeoViewer\lw_GeoViewer.Web\Model\PayloadHelper.cs:line 14
    at lw_GeoViewer.Web.Service1.OnloadEntityGeometryTest(Int64 sEntityId) in C:\Users\franciscor\Documents\Mis Proyectos\GeoViewer\lw_GeoViewer\lw_GeoViewer.Web\Service1.svc.cs:line 166
    at SyncInvokeOnloadEntityGeometryTest(Object , Object[] , Object[] )
    at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
    at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
    InnerException:

    • Ok well it’s trying to create a new instance of your object, but it can’t. This could be due to it not being built in a shared library or some other reason the DLL isn’t available. Can you create an instance of the object type causing the exception (itemType) in your application? That should point you in the right direction.

  7. Peter

    Hay Mike,

    I noticed a post of you on the Silverlight forum about accessing private fields and the security issues.
    I found a way around for this which also may improves the speed of your serializer.

    The Silverlight library supports expression trees that can generate runtime field getters and setters.


    ParameterExpression obj = Expression.Parameter(typeof(object), "obj");
    ParameterExpression value = Expression.Parameter(typeof(object), "value");

    Expression getField = Expression.Convert(Expression.Field(Expression.Convert(obj, field.DeclaringType), field), typeof(object));
    Func getter = Expression.Lambda<Func>(getField, obj).Compile();

    Expression setField = Expression.Assign(Expression.Field(Expression.Convert(obj, field.DeclaringType), field), Expression.Convert(value, field.FieldType));
    Action setter = Expression.Lambda<Action>(setField, new ParameterExpression[] { obj, value }).Compile();

    This wonderfully allows you to get and set a field :)
    There is one downside. Somehow the security issues still remain for types inside the Silverlight CLR.

    Besides all that, the expression trees can also be used to get and set properties.
    The great advantage of this, is the execution speed.

    Peter

    • I looked at that method, and it does work really well – however I found that it is even faster if you get the GetMethod and SetMethod for properties directly and access those through a generic custom delegate – I’m working to get this code ready for an upload right now.

  8. Paul G

    Hi & thanks for sharing your great serializer.

    I did have a small ‘head sctratcher’ initially.
    I am using seperate applications, .net on the server & Silverlight on the client. I realised (after some head scratching) that the serialized object definitions must be in the same assembly shared by both client & server.
    It will not work if you share the source file of your objects (as i was) and compile them into seperate net & silverlight assemblies, as the serializer seems to use fully qualified assemby names.

    It’s not a problem, it just took me a while to figure it out, so i thought i’d share it in case anyone else bumps up against same thing.

    Cheers & thanks again, Paul

    • Yes, that is a good point it does require that. That’s the advantage of DataContract serialization – you establish the contract using simpler names, this does cause some problems of course with collisions, especially if you are using POCOs anywhere in your graph. I think there has to be a good half way house.

      For the moment, you could implement a handler for the MapMissingType event, parse out the type name and supply your compatible type on each end.

  9. Brent

    Thanks for the great work! The only addition I would make to your exposed static functions (my own personal preference) are these:

    public static T DeSerialize(byte[] array) where T : class {
    var result = Deserialize(array) as T;
    return result;
    }

    public static T DeSerialize(Stream stream) where T : class {
    var result = Deserialize(stream) as T;
    return result;
    }

    • Yes, that’s a good idea. In my project that uses SilverlightSerializer a wrapper that integrates with the compression library does something similar – but you are right, it makes sense to have it in here. Will update next chance I get.

    • I’ve now added the methods you suggested to the latest version of the serializer. Thanks again.

      • Glad you like the addtions. One more addition I’ve made that I’d thought I’d share. I am using your library to mainly store off pixel data for bitmaps (I work with very large, hi def radiology images), which are of course int arrays. I changed the following in bold to optimize the Deserialization of int arrays:

        		private static object DeserializeArray(Type itemType, BinaryReader reader, int count) {
        			// If the count is -1 at this point, then it is being called from the
        			// deserialization of a multi-dimensional array - so we need
        			// to read the size of the array
        			if(count == -1) {
        				count = reader.ReadInt32();
        			}
        
        			//Get the expected element type
        			var elementType = itemType.GetElementType();
        			//Optimize for byte arrays
        			if(elementType == typeof(byte)) {
        				var ret = reader.ReadBytes(count);
        				_loadedObjects.Add(ret);
        				return ret;
        			}
        			//Optimize for int arrays
        			if(elementType == typeof(int)) {
        				int[] ret = new int[reader.BaseStream.Length / 4];
        				int index = 0;
        				var length = reader.BaseStream.Length;
        				while(reader.BaseStream.Position != length) {
        					ret[index++] = reader.ReadInt32();
        				}
        				_loadedObjects.Add(ret);
        				return ret;
        			}
        
        			//Create an array of the correct type
        			var array = Array.CreateInstance(elementType, count);
        			_loadedObjects.Add(array);
        			//Check whether the array contains primitives, if it does we don't
        			//need to store the type of each member
        			if(IsSimpleType(elementType))
        				for(var l = 0; l < count; l++) {
        					array.SetValue(ReadValue(reader, elementType), l);
        				}
        			else
        				for(var l = 0; l < count; l++) {
        					array.SetValue(DeserializeObject(reader, elementType), l);
        				}
        			return array;
        		}
        

        And for the serialization part:

        		private static void SerializeArray(Array item, Type tp, BinaryWriter writer) {
        			var length = item.Length;
        
        			writer.Write(length);
        
        			var propertyType = tp.GetElementType();
        			//Special optimization for arrays of byte
        			if(propertyType == typeof(byte)) {
        				writer.Write((byte[])item, 0, length);
        			}
        			//Special optimization for arrays of int
        			if(propertyType == typeof(int)) {
        				foreach(var i in (item as int[])) {
        					writer.Write(i);
        				}
        			}
        
        			//Special optimization for arrays of simple types
        			//which don't need to have the entry type stored
        			//for each item
        			else if(IsSimpleType(propertyType))
        				for(var l = 0; l < length; l++) {
        					WriteValue(writer, item.GetValue(l));
        				}
        			else
        				for(var l = 0; l < length; l++) {
        					SerializeObject(item.GetValue(l), writer, propertyType);
        				}
        		}
        

        …in case anyone else finds that useful. I found a significant performance increase adding these bits (1000% or so). I would guess the cost for reflection on the simple type array is pretty high, but I didn’t dig into the performance problems that much… Maybe just a corner case, I don’t know.

  10. Jaime Bula

    Thanks for sharing this, this is quite solid, moving 10.000 rows now is a piece ok cake! This should be comitted to the .NET Stack! Thumbs UP!

  11. Johan J v Rensburg

    Excellent thank you very much for sharing this saved me a ton of work..

    Johan

  12. Andrus

    How much this increases the speed of data retrieval if used in web applicaton ? Where to find comaprison this with WCF binary transfer protocol ?

    • The speed difference is down largely to the network performance between the client and server, so it is only possible to measure in specific cases. Peter Bromberg provides some samples in his article linked from the blog above.

      This isn’t an either/or with binary transport for WCF, I use that in combination with the serializer in my application. The binary transport is much more efficient than normal text based communication and does reduce the need to use further compression library to compact the results of my serializer.

      You should use the serializer if you want to create isolated storage out of objects or so pass large complicated object graphs to the server, especially if you want to be dealing with actual instances of your objects complete with code rather than proxies.

      Hope that helps!

  13. jdev

    Does this serialize Dictionary collections? I quick test returned a null Dictionary on deserialization.

    Thanks.

    • I certainly serialized dictionaries frequently in my application that uses this class, I tested a basic straight serialization of a Dictionary and that worked fine too. Perhaps you could post an example in case there’s some case that I have missed?

  14. Steve

    Excellent piece of code!

    BTW, was SerializeMultiDimensionArray intended to handle arbitrary rank arrays? It appears to work fine with 2-dimensional arrays, but it causes an exception with a 3-dimensional array.

    It appears where you copy the source array to baseArray, indexes only contains 2 elements rather than rank elements (3 in my case). This causes an exception in accessing the source array elements with GetValue(indexes).


    int[] indexes = new int[] { l, arrayStartIndex };
    baseArray.SetValue(item.GetValue(indexes), arrayStartIndex);

    Also, thanks so much for sharing this!

    • You’re right, that was my intention but it looks like I’ve just written it so it will work for 2 dimensions. Thanks for spotting that! I’ll get to it…

    • Ok, I’ve updated the code so it now persists multi dimensional arrays properly…

    • Peter

      Hi Mike,

      Your serializer looks like a solid piece of code.
      Very nice!

      Although I’ve got a little question.
      What’s with the DoNotChecksum attribute?
      Isn’t it the same as the DoNotSerialize attribute?

      • Ah, that’s a left over from the project that the code is a part of!

        To create a key that represents one of my objects I serialize all of the properties – but I leave out things like a GUID ID that doesn’t change the meaning. I do this by serializing the object and then use an algorithm to create a 36 character key, DoNotChecksum indicates that an field or property should not be serialized if Checksum mode is on, but should normally be serialized when I need an exact copy of the object.

  15. [...] I’ve realized that I didn’t detail how to ensure you can serialize your own classes between Silverlight and .NET 4 using my SilverlightSerializer class.  [...]

  16. Jonas Kello

    Great work! I removed the dependencies on System and System.Core so I was able to use it in SL3. I also found a small bug that you might want to be aware of. In the function ReadValue() there are these lines:

    if (tp == typeof(float))
    return reader.ReadDouble();

    I changed it to:

    if (tp == typeof(float))
    return reader.ReadSingle();

    because a float is a System.Single in C#.

    • Thanks for the catch on the Single!

      I am using this in Silverlight 3! Not sure why you needed to remove those dependencies – they are available in the shared libraries, but if it works without them – great :)

  17. Hi Mike,

    Amazing work! I’ve integrated your serializer into my Persistent Storage library to allow arbitrary saving of objects using Isolated Storage.

    Thanks for all your hard work!

  18. [...] of the issues that came up is serialization in Silverlight. I cannot say enough about Mike Talbot’s Silverlight serializer, which somehow (magically?) serializes any and every object, without needing any class-level or [...]

  19. Bassvix

    I have to test for “GetList” WCF RIA Services return, but don´t work.

    public byte[] GetList(…)
    {

    }

    Don´t work.

    “Type ‘Byte’ is not a valid entity type. Entity types cannot be a primitive type or a simple type like string or Guid. )

    • I just use a straightforward WCF service, I think if you are using RIA services then you are already projecting database entities to the client side. I’m not sure what the correct way of asking for a blob is from RIA services, but that’s what you need, something that can be represented as a byte array. Of course you could also just use Base64 encoding to turn the byte array into a string and use that to forward information around the place. The syntax of that is Convert.ToBase64String( myArray );

  20. [...] UPDATE: Another post on this blog lists my solution to this. Posted by whydoidoit Filed in Uncategorized ·Tags: [...]

Home Project With Code Silverlight Binary Serialization