We have a very fast change rate. So much so that often we have to reserve 150 to 180% of the volume size in the replication destination group. So a 10TB volume would need 15 to 18TB reserved in replication. One obvious strategy we're tackling is how to fix this issue if possible.
However in the meantime I am constantly dealing with alerts that the reservation amount is not high enough and thus replication is paused. It could be that there's not enough space delegated for replication to the group or that the individual volume does not have enough space reserved.
Currently the only way I know of it increase those two amounts is to log into a machine and open the group manager (requiring java) and increase the numbers. So I am wondering:
Does anyone have a way to increase the delegated amount number in the replication group for a production group, remotely. And:
Does anyone have a way to increase the reserved replication amount for a volume (done in the production group, remotely.
I'm thinking things like powershell scripts, batch files, windows tasks, etc. These are things I can probably find a tool that will let me do this remotely from my phone. Otherwise I'm forced to take my laptop with me absolutely everywhere I go so that I can do this remotely. But not everywhere has cell phone signals for a MiFi device to use. And I'd rather not have my backpack on my back when I got out to dinner with my wife!
Ultimately I need to get more staffing so that people can be on call for stuff like this, but that's not happening anytime soon.
Ideally the tools would calculate how much the delegation amount has to be increased and do it if there's room, or it would calculate how much to increase the reserved amount by. I know the amount it needs can be found via snmp.
Super ideally equallogic would change these games so that it takes as much as it needs when it needs it and then gives it back when it doesn't. Most modern quality SANs do this, but with Equallogic it requires so much more management time for people with high change rates.
There are some sample scrips in PERL and Python on the EQL website. They're with the Firmware downloads, called "Host Scripting Tools". This will make it easier to create scripts to make the changes more easily
There is a Windows registry change that will stop windows from updating the last access date/time when it reads a file/directory. So every read involves a small write. Which can add up, especially in virtualized environments. Where all the C: drives changes add up quickly. Also in VMware you can put the VMware swapfile on a partition that's not being replicated. This cuts down on snapshot/replication amounts.
This is from the EQL KB.
If you find that your snapshot reserve space is filling up faster than you expect it to, or your replication jobs are much larger than you expected them to be, it really means that your volume is being written to more than you think it is. There are a number of reasons why your filesystems may be getting modified much more than you think. This list is representative rather than exhaustive - there are probably many more packages that make quiet but significant changes to a volume.
o Microsoft Exchange Server has an option called mailbox management that can rearrange the location of mailboxes on the volume, causing lots of unexpected modifications to a volume. You can disable it in the SETOP Properties dialog.
Here is an article explaining how to set up mailbox management for Exchange 2003: searchexchange.techtarget.com/.../0,289142,sid43_gci1119801,00.html
For Exchange 2007, the operation is to do daily defragmentation. technet.microsoft.com/.../aa997972.aspx
o Defragmenting a disk will essentially modify every block on the volume that has data on it. Any process that defragments in the background will cause you to fill up your snapshot reserve (and increase the size of your replication jobs) at an alarming rate. Some Logical Volume Managers (LVMs) do this. However, defragmenting occasionally can actually shrink average snapshot size over time, at the cost of a spike in snapshot size at the time the defrag happens. A single defragmentation job when your snapshots seem to large can help you decide if a regular defragmentation regimen is appropriate.
o We have found that on Windows file servers, the system automatically updates the "Last Access Time" field of the directory entry for each file touched, which can prove to have a very relevant impact on snapshot and replication utilization.
To disable Last Access Time handling on an NTFS filesystem, add the following key to the Registry on your server:
And set it to 1
It requires a reboot of the host to take effect. This article in the Microsoft KB describes this parameter:
Snapshots and replications taken after the next reboot of that host may be noticeably smaller, and you may see increased performance.
o W2indows VSS snapshots could be utilized without the EqualLogic VSS hardware provider software installed. The Microsoft VSS provider uses a snapshot repository file to store the copy-on-write data, and by default this repository lives on the disk itself. So if you are using the MS VSS provider for Previous Versions functionality, or for VSS backups, that will cause unexpectedly large writes to your disk even though the actual data in use is not growing. Two options to continue using VSS but not have the cost in writes are:
If you are using VSS for backups, install the Host Integration Tools to enable array-based snapshots for VSS instead of local repository files.
If you are using VSS for Previous Versions functionality, create a separate volume to hold the VSS repository file, so that activity in the VSS repository does not cause additional I/O to your original volume
o Other operating systems also implement snapshotting on local disks using software. Similarly to Windows VSS, other operating systems that implement snapshots in software will create significant additional writes while the snapshots exist. This exists, for instance, in VMWare ESX. Whether or not you can adjust that effect is dependent on the operating system. However you may not have th eoption of not using OS-based snapshots.
o Some virus scanners and backup programs modify the directory entries of every file they touch, usually setting the "Last Access" time for each file and/or the "archive bit" in the directory entries. Although these scans aren't modifying the files themselves, directories occupy disk space as well, and modifying them causes snapshot reserve to be used. Because directories are normally distributed randomly across the disk, any task that routinely accesses every file in a filesystem, and is not careful about how it does it, will use up a lot more snapshot reserve than might be expected.
For backup software, you should have the backup job not modify the Archive Bit, but instead use a database catalog or modification time to track file changes, if the software supports such an option.
One possible approach to minimizing the impact of directory scans like that is to rearrange the filesystem to put all the directories in the same area of the disk. Some defragmenting tools can do this, though you will need to specifically look for this feature. Windows defrag may not support this option.
Remember that disk defragmentation can cause a spike in snapshot utilization because of the number of blocks it writes to as it moves data around. Directory defragmentation is not normally something that would need to be run often, but the operator will need to find a balance between the impact of the defragmentation itself, and the impact of the directory scan.
Snapshots only occupy space when data is modified. The space occupied by a snapshot is the space on the original volume that has been modified since the snapshot. So the moment a snapshot is made, it occupies very little space, since it's just a pointer to the original volume. As modifications to the original volume are made, snapshot space is used up keeping a copy of the original data.
When a second snapshot is made, it is made relative to the first snapshot. Space is allocated to the second snapshot whenever the first snapshot is changed - the second snapshot contains a copy of the first snapshot's data before the change. A third snapshot is made relative to the second. Deleting a snapshot doesn't release all of its reserve space, since snapshot reserve can be a part of multiple snapshots, depending on when the data was modified.
When you select the 5.2.2 firmware, it's right there.
Login first then go to this link.
5.2.2 download page, on the right hand side.
Host scripting tools for Windows
Host Scripting tools for Linux
Do you know if the windows server 2008 r2 (/windows 7) auto-defrag feature could be touching all volumes and how to disable that? I was told we no longer need to defrag in vista/windows7/windowsserver2008/r2 because it does it automatically. for a volume on a san there's no benefit to this. I imagine that could affect replication as well.
I don't know how to use perl or python, particularly from a windows system as I am no familiar with Linux enough to dare touch it yet. Do you have scripts to do those things (Manage reservations and delegation numbers) from a windows box?
The nice thing about PERL and Python is that they are well supported under Windows. There are complete development environments for them, so Linux knowledge is not in any way required.
The samples on the website show you how to get in, issue commands and logout. Once you connect you basically are issuing the standard CLI commands. The sample scripts show you how to create snapshots, etc..
I don't have any sample code specific to what you are doing.
Re: Auto-defrag. ABSOLUTELY! That touches MANY blocks as do AV programs. Again with AV just doing READS there are small associated WRITES to go with them. I totally space the auto defrag service. And for RAIDed devices there's no real benefit to that. So that's something else to turn off.
TweatUI has an option to do that. Doing a Google search will likely yield more suggestions.
Thanks! DO you know if that can be turned on/off per volume? I like it for the boot volume and possibly the database volume (though probably not), but for the SAN I'd like it off. We don't have the third party defrag tools you mentioned.
I don't know sorry. I suspect it's system wide.
I can't find those scripts. can you show me where? I looked at 5.2.2 firmware download section and other places and don't see them.
Thanks. Do you recommend perl or python? I've never used either. I've done some vbs, powershell, and .net stuff.
Python. It will be more 'like' VBS than PERL. There are some nice free IDEs for Python, you can use SSH. IMO the text handling is much easier as well.
If Anyone has any scripts they care to share that'd be great. I'm hoping for one of two things:
1. an automated means of this happening when it needs more space instead of it pausing rep until someone reacts, which could be in the middle of the night.
2. a secure web page where I can choose a group, volume and replication reserve number to increase to and submit. would also include the ability to choose the replication group and increase the delegated space which often has to be done first.