Detect The maximum array length quota <xxxx> has been exceeded while reading XML data errors

Thanks Jacques.

I'll have to have a think about this one, as it's there for protection against DoS (see here), so ignoring (or setting a larger default) would defeat the purpose of the protection. It'll also likely also change once we move to REST/OData (in which case the limits are likely to be a bit more sensible as it allows better paging).

For reference, the sample stack trace:

System.ServiceModel.Dispatcher.NetDispatcherFaultException: The formatter threw an exception while trying to deserialize the message: There was an error while trying to deserialize parameter http://tempuri.org/:RetrieveResult. The InnerException message was 'There was an error deserializing the object . The maximum array length quota (512000) has been exceeded while reading XML data. This quota may be increased by changing the MaxArrayLength property on the XmlDictionaryReaderQuotas object used when creating the XML reader. Line 1, position 3492.'. Please see InnerException for more details. ---> System.Runtime.Serialization.SerializationException: There was an error deserializing the object . The maximum array length quota (512000) has been exceeded while reading XML data. This quota may be increased by changing the MaxArrayLength property on the XmlDictionaryReaderQuotas object used when creating the XML reader. Line 1, position 3492. ---> System.Xml.XmlException: The maximum array length quota (512000) has been exceeded while reading XML data. This quota may be increased by changing the MaxArrayLength property on the XmlDictionaryReaderQuotas object used when creating the XML reader. Line 1, position 3492. at System.Xml.XmlExceptionHelper.ThrowXmlException(XmlDictionaryReader reader, String res, String arg1, String arg2, String arg3)
 at System.Xml.XmlDictionaryReader.ReadContentAsBytes(Boolean base64, Int32 maxByteArrayContentLength)
 at System.Xml.XmlDictionaryReader.ReadContentAsBase64(Int32 maxByteArrayContentLength, Int32 maxInitialCount)
 at System.Xml.XmlDictionaryReader.ReadElementContentAsBase64()
 at ReadBinaryValueFromXml(XmlReaderDelegator , XmlObjectSerializerReadContext , XmlDictionaryString[] , XmlDictionaryString[] )
 at System.Runtime.Serialization.ClassDataContract.ReadXmlValue(XmlReaderDelegator xmlReader, XmlObjectSerializerReadContext context)
 at System.Runtime.Serialization.XmlObjectSerializerReadContextComplex.InternalDeserializeInSharedTypeMode(XmlReaderDelegator xmlReader, Int32 declaredTypeID, Type declaredType, String name, String ns)
 at System.Runtime.Serialization.XmlObjectSerializerReadContextComplex.InternalDeserialize(XmlReaderDelegator xmlReader, Int32 declaredTypeID, RuntimeTypeHandle declaredTypeHandle, String name, String ns)
 at ReadKeyValuePairOfAdapterEntityValueCollectionKeyanyTypeTl2z7kj3FromXml(XmlReaderDelegator , XmlObjectSerializerReadContext , XmlDictionaryString[] , XmlDictionaryString[] )
 at System.Runtime.Serialization.ClassDataContract.ReadXmlValue(XmlReaderDelegator xmlReader, XmlObjectSerializerReadContext context)
 at System.Runtime.Serialization.XmlObjectSerializerReadContextComplex.InternalDeserializeInSharedTypeMode(XmlReaderDelegator xmlReader, Int32 declaredTypeID, Type declaredType, String name, String ns)
 at System.Runtime.Serialization.XmlObjectSerializerReadContextComplex.InternalDeserialize(XmlReaderDelegator xmlReader, Int32 declaredTypeID, RuntimeTypeHandle declaredTypeHandle, String name, String ns)
 at ReadArrayOfKeyValuePairOfAdapterEntityValueCollectionKeyanyTypeTl2z7kj3FromXml(XmlReaderDelegator , XmlObjectSerializerReadContext , XmlDictionaryString , XmlDictionaryString , CollectionDataContract )
 at System.Runtime.Serialization.CollectionDataContract.ReadXmlValue(XmlReaderDelegator xmlReader, XmlObjectSerializerReadContext context)
 at System.Runtime.Serialization.XmlObjectSerializerReadContextComplex.InternalDeserializeInSharedTypeMode(XmlReaderDelegator xmlReader, Int32 declaredTypeID, Type declaredType, String name, String ns)
 at System.Runtime.Serialization.XmlObjectSerializerReadContextComplex.InternalDeserialize(XmlReaderDelegator xmlReader, Int32 declaredTypeID, RuntimeTypeHandle declaredTypeHandle, String name, String ns)
 at ReadArrayOfKeyValueOfAdapterEntityValueCollectionKeyanyTypeTl2z7kj3FromXml(XmlReaderDelegator , XmlObjectSerializerReadContext , XmlDictionaryString[] , XmlDictionaryString[] )
 at System.Runtime.Serialization.ClassDataContract.ReadXmlValue(XmlReaderDelegator xmlReader, XmlObjectSerializerReadContext context)
 at System.Runtime.Serialization.XmlObjectSerializerReadContextComplex.InternalDeserializeInSharedTypeMode(XmlReaderDelegator xmlReader, Int32 declaredTypeID, Type declaredType, String name, String ns)
 at System.Runtime.Serialization.XmlObjectSerializerReadContextComplex.InternalDeserialize(XmlReaderDelegator xmlReader, Int32 declaredTypeID, RuntimeTypeHandle declaredTypeHandle, String name, String ns)
 at ReadAdapterEntityFromXml(XmlReaderDelegator , XmlObjectSerializerReadContext , XmlDictionaryString[] , XmlDictionaryString[] )
 at System.Runtime.Serialization.ClassDataContract.ReadXmlValue(XmlReaderDelegator xmlReader, XmlObjectSerializerReadContext context)
 at System.Runtime.Serialization.XmlObjectSerializerReadContextComplex.InternalDeserializeInSharedTypeMode(XmlReaderDelegator xmlReader, Int32 declaredTypeID, Type declaredType, String name, String ns)
 at System.Runtime.Serialization.XmlObjectSerializerReadContextComplex.InternalDeserialize(XmlReaderDelegator xmlReader, Int32 declaredTypeID, RuntimeTypeHandle declaredTypeHandle, String name, String ns)
 at ReadArrayOfAdapterEntityFromXml(XmlReaderDelegator , XmlObjectSerializerReadContext , XmlDictionaryString , XmlDictionaryString , CollectionDataContract )
 at System.Runtime.Serialization.CollectionDataContract.ReadXmlValue(XmlReaderDelegator xmlReader, XmlObjectSerializerReadContext context)
 at System.Runtime.Serialization.XmlObjectSerializerReadContextComplex.InternalDeserializeInSharedTypeMode(XmlReaderDelegator xmlReader, Int32 declaredTypeID, Type declaredType, String name, String ns)
 at System.Runtime.Serialization.XmlObjectSerializerReadContextComplex.InternalDeserialize(XmlReaderDelegator xmlReader, Type declaredType, String name, String ns)
 at System.Runtime.Serialization.NetDataContractSerializer.InternalReadObject(XmlReaderDelegator xmlReader, Boolean verifyObjectName)
 at System.Runtime.Serialization.XmlObjectSerializer.ReadObjectHandleExceptions(XmlReaderDelegator reader, Boolean verifyObjectName, DataContractResolver dataContractResolver)
 --- End of inner exception stack trace ---
 at System.Runtime.Serialization.XmlObjectSerializer.ReadObjectHandleExceptions(XmlReaderDelegator reader, Boolean verifyObjectName, DataContractResolver dataContractResolver)
 at System.Runtime.Serialization.NetDataContractSerializer.ReadObject(XmlDictionaryReader reader, Boolean verifyObjectName)
 at System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter.PartInfo.ReadObject(XmlDictionaryReader reader, XmlObjectSerializer serializer)
 at System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter.DeserializeParameterPart(XmlDictionaryReader reader, PartInfo part, Boolean isRequest)
 --- End of inner exception stack trace ---
 
 Server stack trace: 
 at System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter.DeserializeParameterPart(XmlDictionaryReader reader, PartInfo part, Boolean isRequest)
 at System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter.DeserializeBody(XmlDictionaryReader reader, MessageVersion version, String action, MessageDescription messageDescription, Object[] parameters, Boolean isRequest)
 at System.ServiceModel.Dispatcher.OperationFormatter.DeserializeBodyContents(Message message, Object[] parameters, Boolean isRequest)
 at System.ServiceModel.Dispatcher.OperationFormatter.DeserializeReply(Message message, Object[] parameters)
 at System.ServiceModel.Dispatcher.ProxyOperationRuntime.AfterReply(ProxyRpc& rpc)
 at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
 at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
 at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
 at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
 
 Exception rethrown at [0]: 
 at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
 at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
 at Unify.Framework.Linq.IQueryRemoteHandler.Retrieve(SerializableExpression expression)
 at Unify.Framework.Linq.InterLinqQuery`1.GetEnumerator()
 at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
 at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
 at Unify.Connect.Web.IdentityBrokerEntitySearchController.CurrentEntities(EntityRetrievalInformation information)
 at Unify.Connect.Web.IdentityBrokerEntitySearchController.SearchEntities(Guid partitionId, Nullable`1 pageSize, Nullable`1 pageNumber, String groupColumn, Nullable`1 ascending, String searchContext)
 at Unify.Connect.Web.ConnectorController.SearchEntities(Guid partitionId, Nullable`1 pageSize, Nullable`1 pageNumber, String groupColumn, Nullable`1 ascending, String searchContext)
 at lambda_method(Closure , ControllerBase , Object[] )
 at System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters)
 at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters)
 at System.Web.Mvc.Async.AsyncControllerActionInvoker.<BeginInvokeSynchronousActionMethod>b__36(IAsyncResult asyncResult, ActionInvocation innerInvokeState)
 at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult`2.CallEndDelegate(IAsyncResult asyncResult)
 at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult)
 at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3c()
 at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass45.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3e()
 at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(IAsyncResult asyncResult)
 at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass1e.<>c__DisplayClass28.<BeginInvokeAction>b__19()
 at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass1e.<BeginInvokeAction>b__1b(IAsyncResult asyncResult)

I probably also need to clarify that this is a UI issue only, and the standards for the WS calls should be manually managed, for the exact reasons you have mentioned Adam.

In my experience, this really starts becoming an issue when client importing large photos. In the case of Workday, they (WD) state that they have a 1mb limit in their help file, but in reality the limit is actually 10mb, which is stupidly large for a cloud service, given that DD has 34k users.

The client have suggested that we look at the following ways to enhance the UI:

• Provide a flag to exclude photo's (and other binary fields) from the read operation in the UI as it is usually the culprit and is not in a readable format anyway ;or
• render the binary data (which is probably much harder given that all binary data isn't photo's); or
• Find a way to reduce the reader in capping the binary field to only read the first 1024 bits of the binary field.

Provide a flag to exclude photo's (and other binary fields) from the read operation in the UI as it is usually the culprit and is not in a readable format anyway

Would be very difficult with the WCF LINQ provider. Will be possible (probably without any extra work) when we move to OData.

render the binary data (which is probably much harder given that all binary data isn't photo's);

This has already been done in v5.1. But due to the first point, doesn't help solve the exception.

Find a way to reduce the reader in capping the binary field to only read the first 1024 bits of the binary field.

Same as the first issue.