Diagnosing an Ektron eSync Relationship

Ektron’s eSync service is a great product when it’s working but getting the initial configuration right can be a bit fiddly.  I’ve compiled a quick diagnostic procedure to resolve the most common issues, as well as some of the more common issues.
Note: The Ektron Windows Service is usually in: C:\Program Files\Ektron\EktronWindowsService30. After any configuration change restart the Ektron Windows Service on every node in the relationship.

1. Restart the Ektron Windows Service on every node within the relationship
2. Ensure that the Ektron license is valid for every database within the relationship
3. Ensure that you can access http://<DOMAINNAME>/workarea/ServerControlWS.asmx on each node of the relationship.  In load balanced environments ensure that you test each node explicitly.
   1. Check that this is the same as the WSPath in the AppSettings
   2. Check that this is the same in the sitedb.config file in the Ektron Windows Service folder
4. Ensure that each node is accessible on port 8732 from the next nodes in the chain
   1. From the command line enter ‘telnet servername 8732’ (in Server 2008/Windows 7 you will need to install the telnet client)
   2. If a connection is successful, you fill see a blank screen press ctrl+c to exit and restart the service you just connected to
5. Ensure that the SearchConfigUI tool can properly iterate through all of the IIS websites (it will group results by unique connectionstrings).
   1. If the SearchConfigUI fails on an IIS 7 box ensure that IIS 6 Management Tools and Metabase compatability have been installed
6. Ensure that all servers are using the same version of Ektron
   1. Compare the filesize (and versions) of the following files within the Windows Service folder
      • Ektron.ASM.EktronServices30.exe
      • Ektron.ASM.AssetConfig.dll
      • Ektron.FileSync.Common.dll
      • Ektron.FileSync.Framework.dll
      • Ektron.Sync.Communication.dll
      • Ektron.Sync.SyncServices.dll
7. Ensure that the website is built against the same version of ektron as installed on each of the nodes
   1. Check that the common libraries within the website bin folder and windows service folders match
8. Check that the website bin directories are the same size on all nodes within the relationship
9. Check the Windows Service Error Log for indications of a Transport Layer Error (log files are within the /logs folder)
   1. The following errors can indicate a DNS or Proxy issue:
      • The remote server returned an unexpected response: (407) Proxy Authentication Required.
      • The request failed with HTTP status 403: Forbidden
      • The HTTP request was forbidden with client authentication scheme 'Anonymous'
10. Ensure all nodes of the eSync relationship got all counterpart certificates installed (using Security Configurator)?
    1. The Ektron Windows Service directory will contain all of the *_SyncClient.* files for each node and the *_SyncServer.* files for the local machine
    2. The website root direction should contain the local machines *_SyncClient.* files and the timestamps should match the equivalent certificates in the Windows Service directory
    3. Check that local machines certificates are referenced within the ektron.serviceModel section of the web.config
    4. The Ektron.ASM.EktronServices30.exe.config file within the Windows Service folder should contain an element called publicCertKeys the has a reference to a client certificate for each server in the eSync relationship.  These should be present on every machine in the relationship (they may be in a different order).
    5. The website on each machine should have an AppSetting setting called EncodedValue this should be identical to the value stored within the Service Config file for the same machine.  If not, copy from the Windows Service config into the AppSetting not vice-versa.

Common eSync Errors

(as seen on Synchronisation Progress screen)

Object Reference Not Set To An Instance of An Object

1. Check that the client certificate is present in the root of the website
2. Check that local machines certificates are referenced within the ektron.serviceModel section of the web.config
3. Check the WSPath is correct and acccessble
4. Ensure the AppPool is runnning under the Network Service identity

The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state

1. Restart all services
2. Check that licenses are installed and valid
3. Check that certificates are installed
4. Check the WSPath is correct and acccessble
5. Compare the EncodedValue setting in the web.config with the value stored in the Ektron Service config

SyncDatabaseFailed failed with message: Max index do not match. Local index:X, Remote index:Y

1. Decide which server has the correct index (ie if one server has other eSync profiles that are working ok)
2. On the server that has the wrong index open the ServerInfo.xml (within the x:\sync folder)
3. Change all of the MaxId="X" values to the correct value
4. Open the Ektron database for the website with the incorrect index and modify the 'share_index' field of the settings table to the correct value
5. Restart all Ektron Windows Services within the relationship

Invalid method invocation. Remote position is NOT staged.

1. Stop all Ektron Windows Services within the relationship
2. Create a copy of the x:\sync folder (where x is the drive letter for your workarea website)
3. Delete all of the contents of the x:\sync folder
4. Restart all of the Ektron Windows Services within the relationship
5. Rerun synchronisation to rebuild relationship

SyncAssetLibraryFailed failed with message: Object reference not set to an instance of an object.

1. Ensure that the ‘StorageLocation’ attribute of ‘DocumentManagerData’ is set to a unique location for each environment (this is within the AssetManagement.config) (if hosted on the same server)

No CMS400.NET sites were found at this location (when configuring a new relationship)

1. Ensure that the certificates have been correctly installed on each node of the relationship.

The Remote Position is NOT Staged (when performaning an intitial sync)

1. If a previous relationship existed between the two servers, ensure that the records relating to them have been removed from the [dbo].[scheduler] database table.
2. Remove all folders relating to the previous relationship from the x:\sync folder (and in all child folders) 
3. Try running the sync from the target website
4. If problem persists, recreate the target min database – a previous initial sync attempt has semi-populated the target database beyond economical repair.

eSync Status is ‘Running’ Even When There Is No Sync In Progress

1. Ensure that all Ektron services related to the website (development machines, build servers, etc, etc) have exactly the same version.
2. Manually set the status to ‘completed’ in the [dbo].[scheduler] table and restart the services.
3. If the problem recurs using SQL Profiler to find which server is updating the status (using the additional  HostName column).  The problem  command starts with ‘exec cms_updateschedulerun’

Service Not Listening on port 8732 and EktronL2 Event Log contains ‘Service initialized successfully’ and ‘Service stopped successfully’

1. Ensure that the workarea can be opened without an error (cmslogin.aspx and /workarea/servercontrolws.asmx)
2. Restart service
3. Ensure ‘Service started successfully’ message is recorded

SyncDatabaseFailed failed with message: The message with Action 'http://tempuri.org/ISyncService/InitSession' cannot be processed at the receiver, due to a ContractFilter mismatch at the EndpointDispatcher

1. Ensure that the same esync service version is installed on all nodes of the relationship

Failed to execute the command 'DeleteCommand' for table 'content_folder_tbl' (or similar)

1. There’s been an irresolvable conflict, try reversing the conflict policy temporarily and resyncing after a successful sync, the original policy can be restored.

Initial Sync Fails with a Database Constraint Error

The initial sync fails with a constraint error similar to:

• Violation of PRIMARY KEY constraint 'PK_history_meta_tbl'
• The ALTER TABLE statement conflicted with the FOREIGN KEY constraint "folder_taxonomy_tbl_fk2"
• Violation of PRIMARY KEY constraint 'PK_content_meta_tbl'

1. Remove the damaged sync profile (see below)
2. Stop users from editing content whilst the initial sync is run

Initial Sync Fails with:  Could not find file ‘….\uploadedFiles\\metaconfig.doc'

1. Copy ‘metaconfig.doc’ from /assets into the uploadedFiles and uploadedImages
2. Run SearchConfigUI and rebuild catalogs
3. Re-run the intitial sync (it should pick up where it left off!).  You shouldn’t need to remove the relationship and rebuild.

Out of Memory Exceptions

1. Ensure that all servers have adequate memory
2. Reduce the batch size of the Ektron Windows Service by adding batchsize=100 to the 'DatabaseRuntime’ element within DbSync.config in the Ektron Windows Service folder.
3. Ensure that all servers in the relationship have the same batchsize configured.
4. Restart Ektron Service
5. Rerun sync

The system can not find the file specified

1. Ensure that the three *_SyncClient.* files the the local machine
2. Ensure that local machines certificates are referenced within the ektron.serviceModel section of the web.config
3. If the certificates have been accidently deleted, you can copy them from the Ektron Windows Service folder into the root folder.

Removing a Damaged eSync Relationship

If an initial sync has failed it can leave a partially populated sync relationship that will need to be removed.  Each time this is run the Max Server index will be incremented.
The procedure to do this is:

1. Stop Ektron Service
2. Deleting the target database
3. Remove the corresponding entries from 'scheduler' table in the sending database
4. Remove entries from 'X:\sync\ServerInfo.xml' on both servers
5. Remove any folders (for the target environment) in X:\sync folder
6. Remote entries from sitedb.conf for connections on both servers
7. Remove contents of 'assets' and 'privateAssets' on target server
8. Reinstall min database (using Site Setup)
9. Recreate indexies using SearchConfigUI on target server
10. Configure and run initial sync