Bugs
0
Fixed
CSV Connector fails export
Exports to the CSV Connector are failing with the following:
Save entities [Count:1] to connector CSV Connector 2 failed with reason The process cannot access the file 'C:\Shared\Identity Broker 4\CSV Test\Talent2 Person Connector5.csv' because it is being used by another process.. Duration: 00:00:00 Error details: System.IO.IOException: The process cannot access the file 'C:\Shared\Identity Broker 4\CSV Test\Talent2 Person Connector5.csv' because it is being used by another process. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options) at System.IO.StreamWriter.CreateFile(String path, Boolean append) at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize) at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding) at System.IO.File.AppendAllLines(String path, IEnumerable`1 contents) at Unify.Product.IdentityBroker.CsvConnector.AddEntities(IEnumerable`1 entities) at Unify.Product.IdentityBroker.ConnectorToWritingConnectorBridge.AddEntities(IEnumerable`1 entities) at Unify.Product.IdentityBroker.EventNotifierWritingConnectorDecorator.AddEntities(IEnumerable`1 entities) at Unify.Product.IdentityBroker.Adapter.AddEntities(IEnumerable`1 entities, Boolean reflect) at Unify.Product.IdentityBroker.Adapter.AddEntity(IAdapterEntity entity) at Unify.Product.IdentityBroker.AdapterNotifierDecoratorBase`1.AddEntity(IAdapterEntity entity) at Unify.Product.IdentityBroker.LDIFAdapterBase.ExportChanges(ExportedLDIFForAdapter exportedLdifForAdapter) at Unify.Product.IdentityBroker.LDIFAdapterServiceHostDecoratorBase.ExportChanges(ExportedLDIFForAdapter exportedLdifForAdapter) at SyncInvokeExportChanges(Object , Object[] , Object[] ) at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs) at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage41(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(MessageRpc& rpc) at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
There were a series of improvements made to the CSV connector that will need to be re-evaluated and completely regression tested. These changes were made in attempts to remove the likelihood of locks on a file being breached on the same thread by reducing the number of reads/writes to the file.
These included:
- Preventing the collection of IConnectorEntities being completely evaluated before being written back.
- Reducing the reads on the same file to 1 instead of 3 on Add entities.
Customer support service by UserEcho
Moved to beta.
Matthew, when you get a chance, can you please let me know what would be involved in fixing this?
Thanks.
Unsure what was happening here. Currently have exports to CSV working without throwing the above error. Marking as resolved, will reopen if this appears again.
I've been testing with CSV exports with success. Issue closed.
Tony, I have pushed a possible fix for the file locking issue. I noticed that the locking was on the second call in that method, so I removed the use of the first call.
Not sure where your Mutex stuff had gone. I didn't add it in case you had a check-in that you hadn't pushed. Would still be a good idea to have that to stop imports and exports from clashing.
I didn't push the Mutex work as I couldn't prove that it had an affect in solving this case, and I thought it made sense to take the path of least resistance with changes to connectors at this stage.
After discussions we've agreed the this should be applied to the CSV connector before RC, as it is a problem when an export/import happens at the same time.
Additionally I'll need to take a look at UpdateConfiguration and mutex use.
There have been a number of changes in a separate
IDB-380branch that will need to be merged back into default when the changes can be confirmed valid.Tony, please merge back work from this branch so that we can close this one off.
We can reopen if it reoccurs in the wild. I think the next bet would be to retry the write (a few times) with a delay.
Remaining work entails testing/confirming changes in the separated branch and merging back in.
Remaining work entails fixing/confirming deletes for this connector (the heading section is no re-written). Otherwise, imports and add/update have been confirmed, with times in the region of:
7-14seconds for 10k imports, dependant on changes processing required
0.004 seconds per add
~0.9 seconds per update
Delete Entities has been fixed/confirmed, with a delete entity call taking ~0.004 seconds
Reassigned for confirmation of completion - to be confirmed during next regression test cycle.
I have confirmed adds now work in prdgrp-test1 (where they weren't before).
Reopening
An update created a .orig file in the same directory that was not cleared, and following an update to the user "100736", the file went from
ID,Multi,OtherField
2,"3,4,5",Something
3,"2",Something Else Other
100736,"2",Another
to
ID,Multi,OtherField
100736,2,Carmin
D,Multi,OtherField
2,"3,4,5",Something
3,2,Something Else Other
It looks like the headings have been copied after the updated line (see the third line above), but the first letter was cut off
A second export attempt after fixing the file removed the .orig file, but the file still became malformed. Subsequent attempts readd (and then remove) the .orig file
Reassigned for confirmation of completion.
I need to ensure no orphaned ".orig" files can remaing.
Reassigned for confirmation of completion. A series of unit tests have been added at the CSVConnectorTestFixture.
Confirmed multiple times during test harness testing during HP TRIM regression.
When presenting the test harness to Shane I noticed that the add operation wasn't behaving correctly, as a result of a follow-up investigation I determined this operation was broken for the edge-case where a newline was not specified for the last character.
Reassigned for confirmation of completion.
Confirmed in latest installer.