This Knowledgebase article provides internal information and remedial actions about Neverfail Heartbeat 'Out of Disk' behavior on the passive server in a SQL Server environment.
When 'MaxDiskUsage' or 'OutOfDiskSpace' is reached on a passive server with SQL Server as a protected application and the queue set to a high value, the following Exception is logged:
Exception in CommsMgr [L9] Exceeded the maximum disk usage(NFChannelExceeded MaxDiskUsageException)or
[L20]Out of disk space (NFChannelOutOfDiskSpaceException)
To determine the cause of the excessive workload, please check the following:
If an Exceeded Max Disk Usage Exception is encountered on the passive server, investigation of the passive server (safe) queue to determine the files contained within the queue may help identify files that should not be present. To check the queue, observe the files being closed when stopping replication.
Example: If you observe an application used to backup the data functions and find folders containing temporary backup data included in your protected set, this could be the cause excess replication traffic. Removing the filter should find that normal database move operations can be performed without generating the massive amounts of traffic.
- Check to see if a rebuilding of the clustered index on the databases is scheduled. If that is the case, modify the schedule so that the task is run for a longer period (if possible) or run different tasks for different databases.
- If the customer is using SQL Server Full-Text Catalogs/Indexes, building the catalogs can cause the size of the passive server (safe) queue to grow faster. This is because the catalog build process is disk and processor intensive, especially for 'Apply'. This task is disk and processor intensive on both servers (even Microsoft indicates that it is resource-intensive), and generates a high replication traffic for a small amount of protected catalog data.
If the customer is protecting Full Text Catalogs (..\FTDATA\**), and have enough free disk space, they should do the following:
- Check if the times when Heartbeat runs out of space for the passive server (safe) queue coincide with the Full Text Catalog build.
- If disk free space is available (> 10 GB), adjust the queue to a higher value.
- Before increasing the queue sizes, always make sure that in the location where this is stored has enough free disk space to cover the new queue size, otherwise you will end up in an Out Of Disk Space scenario.
- Check the Maintenance tasks running on the server and their scheduling.
- Ensure that database backups are not included in the 'Protected' set.
INFO838884[ControllerStop](com.neverfail.newapply.ApplyEntry) - clearOpenFiles - openFiles: Filename: E:\confirmit_sql_data\survey_p9651571.mdf Closed FileId: 11276
It is possible that other activity (e.g. database maintenance) may cause intense disk/processor activity. In the case of Full Text Catalogues, the queue builds fast and drains slowly; but does eventually drain.