Explanation of the Verify/Synchronization Process and LBM

Follow

Summary

This Knowledgebase article explains how the verify/synchronization process operates with the Low Bandwidth Module (LBM).


More Information

Low Bandwidth

The FSM has two sets of known configuration values. One set for low bandwidth ( < 10Mbit/s) and another set for high bandwidth ( > 10Mbit/s). The configuration of the FSM can be switched from one configuration value to the other using the Configure Server wizard and selecting or clearing the 'Low Bandwidth Optimization' check box.

Note: Each time the 'OK' button is pressed on the Configure Server wizard, the values listed below will be reset, even if the check box selection has not changed.

The 'Low Bandwidth Optimization' configuration values are provided below:

Low Bandwidth (LB)

SectionLength = 1048576 (1MB)

MaxVolumeOfWaitingRequests = 3145728 (3MB)

VerifySize = 100 (100 bytes)

High Bandwidth (HB)

SectionLength == 10485760 (10MB)

MaxVolumeOfWaitingRequest = 31457280 (30MB)

VerifySize = 4096 (4KB)

Explanation of Variables:

Section

When the FSM picks up a file for verify or sync it determines the length of the file and breaks it into sections; breaking it into a section means that if the verify or synchronization of the section fails due to concurrent access, etc., then only the section will need to be retried as opposed to the entire file.

MaxVolumeOfWaitingRequests

The maxVolumeOfWaitingRequests identifies the number of bytes of data the FSM will send without waiting for a response from the passive server. Once MaxVolumeOfWaitingRequests has been sent and no reply has been received, the FSM waits for a response. In LBM, ensuring this value is relatively small means the FSM does not fill the replication buffer and ensures stopping times during verify/sync operations are not excessive.

VerifySize

VerifySize identifies the smallest file the FSM will verify, i.e. if the file is smaller than VerifySize, then the FSM will only synchronize the file such that a request to verify a 99 byte file will result in a synchronize.

NoOfSections in File = File length / SectionLength

Each section is broken into chunks where the ChunkSize is (1MB)

NoOfChunks = Section / ChunkSize

Each section is treated as a separate file with the FSM performing the following operations on each section:

  • startup
  • processAttributes
  • processSecurity
  • processTimeStamp
  • processReparsePoint
  • processSize
  • processCompressionFormat
  • verifyChunk x noOfChunks or syncChunk x noOfChunks
  • cleanup

The objective of these Low Bandwidth Settings are:

  • Reduce the FSMS utilization of the replication buffer, hence leaving more of it for replication traffic. This is achieved by the smaller MaxVolumeOfWaitingRequests and the smaller section length.
  • Reduce the time it takes the system to stop. Often there have been cases in the past where due to the FSMS utilization of the replication buffer, waiting to stop took several hours. Ensuring that the FSM only has 3MB of verify or sync work in the air at any one time ensures that there is not a large backlog of work at the time the user attempts to stop.

Comparison of Verify With LBW and Without LBW assuming a single 60GB file:

Verify

High Bandwidth : NoOfSections = 60GB / 10MB = 6144
Low Bandwidth : NoOfSections = 60GB / 1MB = 61440

If we assume that the HeaderOverhead per file is 1024 byes this overhead includes the filename, attributes, security information etc and we assume that the size of the verify check sum for 1MB is 512 bytes. These values were selected arbitrarily to demonstrate the principle.

Data Sent LB = NoOfSections (HeaderOverhead + VerifyCheckSum)
Data Sent LB = 61440 (1024 + 512) = 90MB

Data Sent HB = NoOfSections (HedaerOverhead + 10 * VerifyCheckSum)
Data Sent HB = 6144 (1024 + 512 X 10) = 36MB

Therefore, during verification of a large file, there will be approximately 3X more data sent with LBM in this example.

Sync

Data sent LB = NoOfSections (HeaderOverHead + SectionLength)
Data Sent LB = 61440 (1024 + 1048576) = 60.06GB

Data Sent HB = NoOfSections (HeaderOverHead + SectionLength)
Data Sent HB = 6144 (1024 + 10 * 1048576) = 60.006GB

Therefore, during synchronization due to the amount of data being sent the extra overhead is negligible.


Applies To

All Versions


Related Information

None

KBID-1277

0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.