Nabble - Network Appliance - Toasters

Re: Determining the current 'snapmirror throttle' value?

2009-11-09T16:58:52Z

Re: Determining the current 'snapmirror throttle' value? That only works if the snapmirror was scheduled. If it was started manually (or a scheduled snapmirror was aborted and restarted manually) I don’t know of any way to determine the speed. On the other hand, the /etc/log/auditlog* contains the manually executed commands, and should go back at least 5 weeks, so the command should be in there.

On 11/9/09 4:35 PM, "Roach, Steve" <Steven.Roach@...> wrote:

So based on that KB article doing the following from the command line should show what the throttle is set for:

NetApp:> rdfile /etc/snapmirror.conf

-----Original Message-----
From: Webster, Stetson
Sent: Monday, November 09, 2009 5:09 PM
To: John Stoffel; toasters@...
Subject: RE: Determining the current 'snapmirror throttle' value?

Not exactly your specific question, but this will take you there.

How to remove SnapMirror throttling on the fly:
https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb20418

Good luck.

Stetson Webster
Professional Services Consultant
Virtualization and Consolidation
NCIE-SAN, NCIE-B&R, SCSN-E, VCP
NetApp
919.250.0052 Direct Phone
stetson@...
Learn more: netapp.com/and

-----Original Message-----
From: John Stoffel [john.stoffel@...]
Sent: Monday, November 09, 2009 3:48 PM
To: toasters@...
Subject: Determining the current 'snapmirror throttle' value?

Guys,

Does anyone know how to determine what the current setting of
'snapmirror throttle N destination' actually is? I've tried looking
at the output of:

   snapmirror status -l
   snapmirror throttle

We're looking to dynamically throttle and un-throttle connections, and
it would be nice to confirm what the actual setting is without have to
guess and just set the value blindly.

We're running 7.2.6.1 and moving to 7.3.1.1 soon, or something newer.

Thanks,
John
    John Stoffel - Senior Staff Systems Administrator - System LSI Group
  Toshiba America Electronic Components, Inc. - http://www.toshiba.com/taec
             john.stoffel@... - 508-486-1087

Re: Determining the current 'snapmirror throttle' value?

2009-11-09T16:58:52Z

So based on that KB article doing the following from the command line should show what the throttle is set for:

NetApp:> rdfile /etc/snapmirror.conf

-----Original Message-----
From: Webster, Stetson
Sent: Monday, November 09, 2009 5:09 PM
To: John Stoffel; toasters@...
Subject: RE: Determining the current 'snapmirror throttle' value?

Not exactly your specific question, but this will take you there.

How to remove SnapMirror throttling on the fly:
https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb20418

Good luck.

Stetson Webster
Professional Services Consultant
Virtualization and Consolidation
NCIE-SAN, NCIE-B&R, SCSN-E, VCP
NetApp
919.250.0052 Direct Phone
stetson@...
Learn more: netapp.com/and

-----Original Message-----
From: John Stoffel [john.stoffel@...]
Sent: Monday, November 09, 2009 3:48 PM
To: toasters@...
Subject: Determining the current 'snapmirror throttle' value?

Guys,

Does anyone know how to determine what the current setting of
'snapmirror throttle N destination' actually is? I've tried looking
at the output of:

   snapmirror status -l
   snapmirror throttle

We're looking to dynamically throttle and un-throttle connections, and
it would be nice to confirm what the actual setting is without have to
guess and just set the value blindly.

We're running 7.2.6.1 and moving to 7.3.1.1 soon, or something newer.

Thanks,
John
    John Stoffel - Senior Staff Systems Administrator - System LSI Group
  Toshiba America Electronic Components, Inc. - http://www.toshiba.com/taec
             john.stoffel@... - 508-486-1087

Re: Determining the current 'snapmirror throttle' value?

2009-11-09T16:58:52Z

So based on that KB article doing the following from the command line should show what the throttle is set for:

NetApp:> rdfile /etc/snapmirror.conf

-----Original Message-----
From: Webster, Stetson
Sent: Monday, November 09, 2009 5:09 PM
To: John Stoffel; toasters@...
Subject: RE: Determining the current 'snapmirror throttle' value?

Not exactly your specific question, but this will take you there.

How to remove SnapMirror throttling on the fly:
https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb20418

Good luck.

Stetson Webster
Professional Services Consultant
Virtualization and Consolidation
NCIE-SAN, NCIE-B&R, SCSN-E, VCP
NetApp
919.250.0052 Direct Phone
stetson@...
Learn more: netapp.com/and

-----Original Message-----
From: John Stoffel [john.stoffel@...]
Sent: Monday, November 09, 2009 3:48 PM
To: toasters@...
Subject: Determining the current 'snapmirror throttle' value?

Guys,

Does anyone know how to determine what the current setting of
'snapmirror throttle N destination' actually is? I've tried looking
at the output of:

   snapmirror status -l
   snapmirror throttle

We're looking to dynamically throttle and un-throttle connections, and
it would be nice to confirm what the actual setting is without have to
guess and just set the value blindly.

We're running 7.2.6.1 and moving to 7.3.1.1 soon, or something newer.

Thanks,
John
    John Stoffel - Senior Staff Systems Administrator - System LSI Group
  Toshiba America Electronic Components, Inc. - http://www.toshiba.com/taec
             john.stoffel@... - 508-486-1087

Re: Determining the current 'snapmirror throttle' value?

2009-11-09T16:58:52Z

So based on that KB article doing the following from the command line should show what the throttle is set for:

NetApp:> rdfile /etc/snapmirror.conf

-----Original Message-----
From: Webster, Stetson
Sent: Monday, November 09, 2009 5:09 PM
To: John Stoffel; toasters@...
Subject: RE: Determining the current 'snapmirror throttle' value?

Not exactly your specific question, but this will take you there.

How to remove SnapMirror throttling on the fly:
https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb20418

Good luck.

Stetson Webster
Professional Services Consultant
Virtualization and Consolidation
NCIE-SAN, NCIE-B&R, SCSN-E, VCP
NetApp
919.250.0052 Direct Phone
stetson@...
Learn more: netapp.com/and

-----Original Message-----
From: John Stoffel [john.stoffel@...]
Sent: Monday, November 09, 2009 3:48 PM
To: toasters@...
Subject: Determining the current 'snapmirror throttle' value?

Guys,

Does anyone know how to determine what the current setting of
'snapmirror throttle N destination' actually is? I've tried looking
at the output of:

   snapmirror status -l
   snapmirror throttle

We're looking to dynamically throttle and un-throttle connections, and
it would be nice to confirm what the actual setting is without have to
guess and just set the value blindly.

We're running 7.2.6.1 and moving to 7.3.1.1 soon, or something newer.

Thanks,
John
    John Stoffel - Senior Staff Systems Administrator - System LSI Group
  Toshiba America Electronic Components, Inc. - http://www.toshiba.com/taec
             john.stoffel@... - 508-486-1087

RE: Determining the current 'snapmirror throttle' value?

2009-11-09T16:35:56Z

So based on that KB article doing the following from the command line should show what the throttle is set for:

NetApp:> rdfile /etc/snapmirror.conf

-----Original Message-----
From: Webster, Stetson
Sent: Monday, November 09, 2009 5:09 PM
To: John Stoffel; toasters@...
Subject: RE: Determining the current 'snapmirror throttle' value?

Not exactly your specific question, but this will take you there.

How to remove SnapMirror throttling on the fly:
https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb20418

Good luck.

Stetson Webster
Professional Services Consultant
Virtualization and Consolidation
NCIE-SAN, NCIE-B&R, SCSN-E, VCP
NetApp
919.250.0052 Direct Phone
stetson@...
Learn more: netapp.com/and

-----Original Message-----
From: John Stoffel [mailto:john.stoffel@...]
Sent: Monday, November 09, 2009 3:48 PM
To: toasters@...
Subject: Determining the current 'snapmirror throttle' value?

Guys,

Does anyone know how to determine what the current setting of
'snapmirror throttle N destination' actually is? I've tried looking
at the output of:

snapmirror status -l
snapmirror throttle

We're looking to dynamically throttle and un-throttle connections, and
it would be nice to confirm what the actual setting is without have to
guess and just set the value blindly.

We're running 7.2.6.1 and moving to 7.3.1.1 soon, or something newer.

Thanks,
John
John Stoffel - Senior Staff Systems Administrator - System LSI Group
Toshiba America Electronic Components, Inc. - http://www.toshiba.com/taec
john.stoffel@... - 508-486-1087

RE: Determining the current 'snapmirror throttle' value?

2009-11-09T16:35:56Z

RE: Determining the current 'snapmirror throttle' value?

2009-11-09T16:35:56Z

RE: Determining the current 'snapmirror throttle' value?

2009-11-09T16:35:56Z

RE: Determining the current 'snapmirror throttle' value?

2009-11-09T16:35:56Z

RE: Determining the current 'snapmirror throttle' value?

2009-11-09T14:09:23Z

Not exactly your specific question, but this will take you there.

How to remove SnapMirror throttling on the fly:
https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb20418

Good luck.

Stetson Webster
Professional Services Consultant
Virtualization and Consolidation
NCIE-SAN, NCIE-B&R, SCSN-E, VCP
NetApp
919.250.0052 Direct Phone
stetson@...
Learn more: netapp.com/and

-----Original Message-----
From: John Stoffel [mailto:john.stoffel@...]
Sent: Monday, November 09, 2009 3:48 PM
To: toasters@...
Subject: Determining the current 'snapmirror throttle' value?

Guys,

Does anyone know how to determine what the current setting of
'snapmirror throttle N destination' actually is? I've tried looking
at the output of:

snapmirror status -l
snapmirror throttle

We're looking to dynamically throttle and un-throttle connections, and
it would be nice to confirm what the actual setting is without have to
guess and just set the value blindly.

We're running 7.2.6.1 and moving to 7.3.1.1 soon, or something newer.

Thanks,
John
John Stoffel - Senior Staff Systems Administrator - System LSI Group
Toshiba America Electronic Components, Inc. - http://www.toshiba.com/taec
john.stoffel@... - 508-486-1087

Re: Determining the current 'snapmirror throttle' value?

2009-11-09T14:09:12Z

>>>>> "John" == John Stoffel <john.stoffel@...> writes:

John> Guys,

John> Does anyone know how to determine what the current setting of
John> 'snapmirror throttle N destination' actually is? I've tried looking
John> at the output of:

John> snapmirror status -l
John> snapmirror throttle

John> We're looking to dynamically throttle and un-throttle connections, and
John> it would be nice to confirm what the actual setting is without have to
John> guess and just set the value blindly.

John> We're running 7.2.6.1 and moving to 7.3.1.1 soon, or something newer.

I guess I wasn't quite clear enough in my request. Let me backup a
bit and clarify.

- We're running multiple snapmirrors from filerA to filerB.
- I have all my volumes configured with kbs=2000 in snapmirror.conf
- I need to dynamically limit the per-volume bandwidth, mostly to just
unlimit the transfer rate for each volume at certain times of the day.

I can just do:

snapmirror throttle 0 <volume>

without a problem. But without looking at the log file
(/vol/root/etc/log/snapmirror) I can't tell what the *current*
snapmirror throttle setting is as a check.

Some people have pointed out the 'options replication.throttle.enable'
option, but that's a system-wide setting, not a per-volume setting.

Though I admit it might be better to limit the system's max-bandwith,
instead of individual bandwidth settings. Esp since I just want to
limit network impact during the production hours on weekdays. So this
replication limit might work quite well for this.

So while it doesn't answer my question directly, it might just be the
answer I need.

Thanks,
John

Determining the current 'snapmirror throttle' value?

2009-11-09T12:47:57Z

Guys,

Does anyone know how to determine what the current setting of
'snapmirror throttle N destination' actually is? I've tried looking
at the output of:

snapmirror status -l
snapmirror throttle

We're looking to dynamically throttle and un-throttle connections, and
it would be nice to confirm what the actual setting is without have to
guess and just set the value blindly.

We're running 7.2.6.1 and moving to 7.3.1.1 soon, or something newer.

Thanks,
John
John Stoffel - Senior Staff Systems Administrator - System LSI Group
Toshiba America Electronic Components, Inc. - http://www.toshiba.com/taec
john.stoffel@... - 508-486-1087

Determining the current 'snapmirror throttle' value?

2009-11-09T12:47:57Z

Determining the current 'snapmirror throttle' value?

2009-11-09T12:47:57Z

Determining the current 'snapmirror throttle' value?

2009-11-09T12:47:57Z

Determining the current 'snapmirror throttle' value?

2009-11-09T12:47:57Z

Re: SnapDrive 6 & Windows 2008 - can't see or connect disks in SD but can see them in Disk Management MMC - only mount read-only

2009-11-09T09:26:14Z

Thanks Charles,

Olaf from Netapp set me on the right track yesterday:

I used a combination of these articles to set me on the right track

http://communities.netapp.com/thread/3313

https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb41190

I still see four 'phantom' drives in Disk Management MMC (+ the real
four) but at least the four that I couldn't mount via SnapDrive or
would only mount read-only are now OK.

Thanks to everyone for their help.

Raj.

On 11/10/09, Ledbetter, Charles <Charles.Ledbetter@...> wrote:

> Sounds to me like the NTFS volume inside the LUN is set to Hidden and/or
> Readonly. There's an old Veritas bug that would cause this.
>
> Look for a utility called "vflag.exe" from Microsoft. Mount the LUN
> manually and run "vflag -display *" and look for one marked
> hidden/readonly.
>
> Use vflag to clear it. You can also use Diskpart.exe to do the same
> thing.
>
> -Charles
>
>
>
> -----Original Message-----
> From: Raj Patel [mailto:phigmov@...]
> Sent: Saturday, November 07, 2009 9:27 PM
> To: toasters@...
> Subject: SnapDrive 6 & Windows 2008 - can't see or connect disks in SD
> but can see them in Disk Management MMC - only mount read-only
>
>
> Hi,
>
> Another question - I have a datawarehouse box - has 4 luns -> SD 6.1 and
> Server 2k8 both 64bit.
>
> I can't seem to reconnect the LUN's using the SD wizard - however they
> do end up appearing in the Server Disk Management MMC.
>
> Catch is they will only mount read-only (but at least the data is fine).
>
> In iSCSI session management it will see the connected LUN's related to
> the igroup.
>
> One of two things tends to happen - if I try to use SD I either get a
> 120 sec time out or I get an error about the lun already being mapped
> (presumably by the built in iSCSI initiator).
>
> As per a previous post - this box was fine until a controller/ontap
> upgrade.
>
> I've tried zapping the igroup and mappings on the SAN and removing all
> traces from SD and the MS iSCSI initiator but I can't seem to overcome
> this problem.
>
> Any ideas ? Worst case I can carve off a whole new set of LUN's and move
> the data but I really don't want to have to do that if I don't have to.
>
> Thanks in advance,
> Raj.
>

RE: SnapDrive 6 & Windows 2008 - can't see or connect disks in SD but can see them in Disk Management MMC - only mount read-only

2009-11-09T07:47:53Z

Sounds to me like the NTFS volume inside the LUN is set to Hidden and/or
Readonly. There's an old Veritas bug that would cause this.

Look for a utility called "vflag.exe" from Microsoft. Mount the LUN
manually and run "vflag -display *" and look for one marked
hidden/readonly.

Use vflag to clear it. You can also use Diskpart.exe to do the same
thing.

-Charles

-----Original Message-----
From: Raj Patel [mailto:phigmov@...]
Sent: Saturday, November 07, 2009 9:27 PM
To: toasters@...
Subject: SnapDrive 6 & Windows 2008 - can't see or connect disks in SD
but can see them in Disk Management MMC - only mount read-only

Hi,

Another question - I have a datawarehouse box - has 4 luns -> SD 6.1 and
Server 2k8 both 64bit.

I can't seem to reconnect the LUN's using the SD wizard - however they
do end up appearing in the Server Disk Management MMC.

Catch is they will only mount read-only (but at least the data is fine).

In iSCSI session management it will see the connected LUN's related to
the igroup.

One of two things tends to happen - if I try to use SD I either get a
120 sec time out or I get an error about the lun already being mapped
(presumably by the built in iSCSI initiator).

As per a previous post - this box was fine until a controller/ontap
upgrade.

I've tried zapping the igroup and mappings on the SAN and removing all
traces from SD and the MS iSCSI initiator but I can't seem to overcome
this problem.

Any ideas ? Worst case I can carve off a whole new set of LUN's and move
the data but I really don't want to have to do that if I don't have to.

Thanks in advance,
Raj.

SnapDrive 6 & Windows 2008 - can't see or connect disks in SD but can see them in Disk Management MMC - only mount read-only

2009-11-07T18:26:48Z

Hi,

Another question - I have a datawarehouse box - has 4 luns -> SD 6.1
and Server 2k8 both 64bit.

I can't seem to reconnect the LUN's using the SD wizard - however they
do end up appearing in the Server Disk Management MMC.

Catch is they will only mount read-only (but at least the data is fine).

In iSCSI session management it will see the connected LUN's related to
the igroup.

One of two things tends to happen - if I try to use SD I either get a
120 sec time out or I get an error about the lun already being mapped
(presumably by the built in iSCSI initiator).

As per a previous post - this box was fine until a controller/ontap upgrade.

I've tried zapping the igroup and mappings on the SAN and removing all
traces from SD and the MS iSCSI initiator but I can't seem to overcome
this problem.

Any ideas ? Worst case I can carve off a whole new set of LUN's and
move the data but I really don't want to have to do that if I don't
have to.

Thanks in advance,
Raj.

Re: Controller upgrade complete but . . .

2009-11-07T14:01:58Z

> I believe that happens if you first bring the aggregates online in maintenance mode,
> rather than in normal mode. Maintenance mode will strip all the mappings, etc.

Is that by design or a bug ? Seems a little bit lame if its
intentional behaviour.

Luckily reconnecting the luns and establishing the initiator security
groups only took an extra few hours.

Off to enjoy the rest of my Sunday :-)

Cheers,
Raj.

On 11/8/09, Webster, Stetson <Stetson.Webster@...> wrote:

> I believe that happens if you first bring the aggregates online in
> maintenance mode, rather than in normal mode. Maintenance mode will strip
> all the mappings, etc.
>
>
> Stetson Webster
> Professional Services Consultant
> Virtualization and Consolidation
> NCIE-SAN, NCIE-B&R, SCSN-E, VCP
> NetApp
> 919.250.0052 Direct Phone
> stetson@...
> Learn more: netapp.com/and
>
>
> -----Original Message-----
> From: Raj Patel [mailto:phigmov@...]
> Sent: Saturday, November 07, 2009 12:58 PM
> To: Learmonth, Peter
> Cc: toasters@...
> Subject: Re: Controller upgrade complete but . . .
>
> Hi Peter,
>
> That sounds fairly plausible - the upgrade itself was performed by a
> vendor and the upgrade steps we used in our change control look like
> they were generated by a NetApp wizard. They certainly didn't mention
> pulling disks out first (which I remember from your post last week
> concerning the signature issue).
>
> We reset the iqn - that was fine.
>
> Losing the igroup LUN security stuff was more of a hassle - I could
> understand resetting the iqn based on new hardware but losing that
> information completely seemed kind of weird (and annoying).
>
> I'm still unsure how it relates back to the SD for Windows - having
> reset the iqn and recreated the igroup mappings on the SAN I would
> have thought SD would just reconnect to the appropriate LUN's on the
> correct drive mappings.
>
> Oh well - looks like the simplest option is just to reconnect the
> LUN's manually.
>
> Cheers,
> Raj.
>
> On 11/8/09, Learmonth, Peter <Peter.Learmonth@...> wrote:
>> It sounds to me like the root volume from the factory was used and the
>> setup routine was run, rather than removing the new drives, connecting
>> the old shelves and making the original root vol work on the new filer,
>> then adding the new drives back in to use as spares or new aggr(s). Is
>> that correct?
>>
>> One method to make the original root vol work as the root vol on the new
>> filer consists of:
>> - Disconnect or remove any new disks from the factory - at least any
>> that contain a root aggr/vol.
>> - Connect the original disks with the original aggrs and vols.
>> - Boot into maintenance mode and reassign the old disks to the new
>> sysid.
>> - Reboot and verify the old disks are running as the root vol, and that
>> hostname, IP, IQN, WWN, etc. are all as expected
>> - Connect or insert the new disks.
>> - Clobber the factory root vol/aggr and use the disks for new aggrs or
>> add to old aggrs as desired and supported.
>>
>> What happens when you connect all (old and new) the disks at the same
>> time is that the disks from the factory are already owned by the new
>> controller, and have a valid root vol. So it ignores the original root
>> vol which it doesn't actually own. So you power it all up and it runs
>> setup as a new filer, and things like IQN and FC WWN are generated fresh
>> using the NVRAM serial number. New NVRAM card + running setup on a new
>> filer = new IQN and WWN. Even though you gave it the same IP, hostname,
>> etc. the IQN and WWN are not user-determined by default.
>>
>> Once generated, these identities are kept on disk. So, if you later
>> swap NVRAM or the whole controller but keep the existing root vol, the
>> identity is kept.
>>
>> Another option is to use "config dump" and "config restore"
>>
>> I can't answer the SnapDrive questions as I haven't specialized in that
>> for a few years.
>>
>> Peter
>>
>>
>> -----Original Message-----
>> From: Raj Patel [mailto:phigmov@...]
>> Sent: Saturday, November 07, 2009 1:40 AM
>> To: toasters@...
>> Subject: Controller upgrade complete but . . .
>>
>> Hi all,
>>
>> Hoping you can save me some legwork -
>>
>> We've upgraded our 270c to a 2050ha - data looks good. We did run into
>> the lun signaturing issue with VMWare but forewarned we were able to
>> place the old sigs back onto the luns.
>>
>> My biggest hassle is that the Windows boxes with iSCSI attached LUN's
>> appear to have lost their luns (ie SQL & Exchange). So the IQN string
>> changed - we set it back to what it should be - but SnapDrive appears
>> to have lost its drive mappings and the SAN lost its iSCSI initiator
>> groups (luckily we can put them back manually from previous
>> autosupports).
>>
>> So wondering a couple of things
>>
>> 1. whats the fastest way to get SnapDrive to remember its luns (even
>> the MS iSCSI initiator seems to know what it thinks it should be
>> seeing but SD 6 has lost all its config) ? I know I can manually
>> re-connect them all but with DBA's and Exchange Admins having weird
>> and wonderful drive mappings this is going to be time-consuming.
>>
>> 2. why did the SAN lose its initiator group config (across all luns
>> and including chap security for our dmz esx servers) and why did
>> SnapDrive lose its config (ie the SAN it connects to, the luns, igroup
>> and drive mappings) as a result of a controller and ontap upgrade ?
>>
>> Next time I'll be sure to screenshot every lun mapping in Windows but
>> surely I shouldn't need to do this ?
>>
>> Any post-upgrade tips to make my Sunday reconnecting systems a little
>> more pleasant will be gratefully received.
>>
>> Cheers,
>> Raj.
>>
>

RE: Controller upgrade complete but . . .

2009-11-07T10:14:49Z

I believe that happens if you first bring the aggregates online in maintenance mode, rather than in normal mode. Maintenance mode will strip all the mappings, etc.

Stetson Webster
Professional Services Consultant
Virtualization and Consolidation
NCIE-SAN, NCIE-B&R, SCSN-E, VCP
NetApp
919.250.0052 Direct Phone
stetson@...
Learn more: netapp.com/and

-----Original Message-----
From: Raj Patel [mailto:phigmov@...]
Sent: Saturday, November 07, 2009 12:58 PM
To: Learmonth, Peter
Cc: toasters@...
Subject: Re: Controller upgrade complete but . . .

Hi Peter,

That sounds fairly plausible - the upgrade itself was performed by a
vendor and the upgrade steps we used in our change control look like
they were generated by a NetApp wizard. They certainly didn't mention
pulling disks out first (which I remember from your post last week
concerning the signature issue).

We reset the iqn - that was fine.

Losing the igroup LUN security stuff was more of a hassle - I could
understand resetting the iqn based on new hardware but losing that
information completely seemed kind of weird (and annoying).

I'm still unsure how it relates back to the SD for Windows - having
reset the iqn and recreated the igroup mappings on the SAN I would
have thought SD would just reconnect to the appropriate LUN's on the
correct drive mappings.

Oh well - looks like the simplest option is just to reconnect the
LUN's manually.

Cheers,
Raj.

On 11/8/09, Learmonth, Peter <Peter.Learmonth@...> wrote:

> It sounds to me like the root volume from the factory was used and the
> setup routine was run, rather than removing the new drives, connecting
> the old shelves and making the original root vol work on the new filer,
> then adding the new drives back in to use as spares or new aggr(s). Is
> that correct?
>
> One method to make the original root vol work as the root vol on the new
> filer consists of:
> - Disconnect or remove any new disks from the factory - at least any
> that contain a root aggr/vol.
> - Connect the original disks with the original aggrs and vols.
> - Boot into maintenance mode and reassign the old disks to the new
> sysid.
> - Reboot and verify the old disks are running as the root vol, and that
> hostname, IP, IQN, WWN, etc. are all as expected
> - Connect or insert the new disks.
> - Clobber the factory root vol/aggr and use the disks for new aggrs or
> add to old aggrs as desired and supported.
>
> What happens when you connect all (old and new) the disks at the same
> time is that the disks from the factory are already owned by the new
> controller, and have a valid root vol. So it ignores the original root
> vol which it doesn't actually own. So you power it all up and it runs
> setup as a new filer, and things like IQN and FC WWN are generated fresh
> using the NVRAM serial number. New NVRAM card + running setup on a new
> filer = new IQN and WWN. Even though you gave it the same IP, hostname,
> etc. the IQN and WWN are not user-determined by default.
>
> Once generated, these identities are kept on disk. So, if you later
> swap NVRAM or the whole controller but keep the existing root vol, the
> identity is kept.
>
> Another option is to use "config dump" and "config restore"
>
> I can't answer the SnapDrive questions as I haven't specialized in that
> for a few years.
>
> Peter
>
>
> -----Original Message-----
> From: Raj Patel [mailto:phigmov@...]
> Sent: Saturday, November 07, 2009 1:40 AM
> To: toasters@...
> Subject: Controller upgrade complete but . . .
>
> Hi all,
>
> Hoping you can save me some legwork -
>
> We've upgraded our 270c to a 2050ha - data looks good. We did run into
> the lun signaturing issue with VMWare but forewarned we were able to
> place the old sigs back onto the luns.
>
> My biggest hassle is that the Windows boxes with iSCSI attached LUN's
> appear to have lost their luns (ie SQL & Exchange). So the IQN string
> changed - we set it back to what it should be - but SnapDrive appears
> to have lost its drive mappings and the SAN lost its iSCSI initiator
> groups (luckily we can put them back manually from previous
> autosupports).
>
> So wondering a couple of things
>
> 1. whats the fastest way to get SnapDrive to remember its luns (even
> the MS iSCSI initiator seems to know what it thinks it should be
> seeing but SD 6 has lost all its config) ? I know I can manually
> re-connect them all but with DBA's and Exchange Admins having weird
> and wonderful drive mappings this is going to be time-consuming.
>
> 2. why did the SAN lose its initiator group config (across all luns
> and including chap security for our dmz esx servers) and why did
> SnapDrive lose its config (ie the SAN it connects to, the luns, igroup
> and drive mappings) as a result of a controller and ontap upgrade ?
>
> Next time I'll be sure to screenshot every lun mapping in Windows but
> surely I shouldn't need to do this ?
>
> Any post-upgrade tips to make my Sunday reconnecting systems a little
> more pleasant will be gratefully received.
>
> Cheers,
> Raj.
>

Re: Controller upgrade complete but . . .

2009-11-07T09:57:55Z

Hi Peter,

That sounds fairly plausible - the upgrade itself was performed by a
vendor and the upgrade steps we used in our change control look like
they were generated by a NetApp wizard. They certainly didn't mention
pulling disks out first (which I remember from your post last week
concerning the signature issue).

We reset the iqn - that was fine.

Losing the igroup LUN security stuff was more of a hassle - I could
understand resetting the iqn based on new hardware but losing that
information completely seemed kind of weird (and annoying).

I'm still unsure how it relates back to the SD for Windows - having
reset the iqn and recreated the igroup mappings on the SAN I would
have thought SD would just reconnect to the appropriate LUN's on the
correct drive mappings.

Oh well - looks like the simplest option is just to reconnect the
LUN's manually.

Cheers,
Raj.

On 11/8/09, Learmonth, Peter <Peter.Learmonth@...> wrote:

RE: Controller upgrade complete but . . .

2009-11-07T09:23:30Z

It sounds to me like the root volume from the factory was used and the
setup routine was run, rather than removing the new drives, connecting
the old shelves and making the original root vol work on the new filer,
then adding the new drives back in to use as spares or new aggr(s). Is
that correct?

One method to make the original root vol work as the root vol on the new
filer consists of:
- Disconnect or remove any new disks from the factory - at least any
that contain a root aggr/vol.
- Connect the original disks with the original aggrs and vols.
- Boot into maintenance mode and reassign the old disks to the new
sysid.
- Reboot and verify the old disks are running as the root vol, and that
hostname, IP, IQN, WWN, etc. are all as expected
- Connect or insert the new disks.
- Clobber the factory root vol/aggr and use the disks for new aggrs or
add to old aggrs as desired and supported.

What happens when you connect all (old and new) the disks at the same
time is that the disks from the factory are already owned by the new
controller, and have a valid root vol. So it ignores the original root
vol which it doesn't actually own. So you power it all up and it runs
setup as a new filer, and things like IQN and FC WWN are generated fresh
using the NVRAM serial number. New NVRAM card + running setup on a new
filer = new IQN and WWN. Even though you gave it the same IP, hostname,
etc. the IQN and WWN are not user-determined by default.

Once generated, these identities are kept on disk. So, if you later
swap NVRAM or the whole controller but keep the existing root vol, the
identity is kept.

Another option is to use "config dump" and "config restore"

I can't answer the SnapDrive questions as I haven't specialized in that
for a few years.

Peter

-----Original Message-----
From: Raj Patel [mailto:phigmov@...]
Sent: Saturday, November 07, 2009 1:40 AM
To: toasters@...
Subject: Controller upgrade complete but . . .

Hi all,

Hoping you can save me some legwork -

We've upgraded our 270c to a 2050ha - data looks good. We did run into
the lun signaturing issue with VMWare but forewarned we were able to
place the old sigs back onto the luns.

My biggest hassle is that the Windows boxes with iSCSI attached LUN's
appear to have lost their luns (ie SQL & Exchange). So the IQN string
changed - we set it back to what it should be - but SnapDrive appears
to have lost its drive mappings and the SAN lost its iSCSI initiator
groups (luckily we can put them back manually from previous
autosupports).

So wondering a couple of things

1. whats the fastest way to get SnapDrive to remember its luns (even
the MS iSCSI initiator seems to know what it thinks it should be
seeing but SD 6 has lost all its config) ? I know I can manually
re-connect them all but with DBA's and Exchange Admins having weird
and wonderful drive mappings this is going to be time-consuming.

2. why did the SAN lose its initiator group config (across all luns
and including chap security for our dmz esx servers) and why did
SnapDrive lose its config (ie the SAN it connects to, the luns, igroup
and drive mappings) as a result of a controller and ontap upgrade ?

Next time I'll be sure to screenshot every lun mapping in Windows but
surely I shouldn't need to do this ?

Any post-upgrade tips to make my Sunday reconnecting systems a little
more pleasant will be gratefully received.

Cheers,
Raj.

Controller upgrade complete but . . .

2009-11-07T01:40:04Z

Hi all,

Hoping you can save me some legwork -

We've upgraded our 270c to a 2050ha - data looks good. We did run into
the lun signaturing issue with VMWare but forewarned we were able to
place the old sigs back onto the luns.

My biggest hassle is that the Windows boxes with iSCSI attached LUN's
appear to have lost their luns (ie SQL & Exchange). So the IQN string
changed - we set it back to what it should be - but SnapDrive appears
to have lost its drive mappings and the SAN lost its iSCSI initiator
groups (luckily we can put them back manually from previous
autosupports).

So wondering a couple of things

1. whats the fastest way to get SnapDrive to remember its luns (even
the MS iSCSI initiator seems to know what it thinks it should be
seeing but SD 6 has lost all its config) ? I know I can manually
re-connect them all but with DBA's and Exchange Admins having weird
and wonderful drive mappings this is going to be time-consuming.

2. why did the SAN lose its initiator group config (across all luns
and including chap security for our dmz esx servers) and why did
SnapDrive lose its config (ie the SAN it connects to, the luns, igroup
and drive mappings) as a result of a controller and ontap upgrade ?

Next time I'll be sure to screenshot every lun mapping in Windows but
surely I shouldn't need to do this ?

Any post-upgrade tips to make my Sunday reconnecting systems a little
more pleasant will be gratefully received.

Cheers,
Raj.

RE: SMVI / VMWare Experiences...

2009-11-04T08:47:29Z

If you don't get VMWare snapshots in your SMVI process then your backups
would be inconsistent. So conceptually the restore would be akin to a
traditional physical server backup. The state of the VM will be unknown
and thus there could be a fsck/chkdsk process that would need to occur.
I find it perfectly acceptable for systems to be backed up
"inconsistent"; this is the old methodology for backups. The only
problem you run into is application awareness (i.e. VMWare pre/post
snapshot scripts to quiesce applications or databases).

We sent one of our VMs that was consistently erroring with snapshot
backups to NetApp; they were able to recreate the problem in their lab
with our VM.

On a side note:
VM disk alignment is HUGE, make sure you're aligned (I bet you're
hearing a lot of this; it can really make a performance difference). I
recommend the tools from NetApp: mbrscan/mbralign.

-----Original Message-----
From: Steffen Kammerer [mailto:steffen.kammerer@...]
Sent: Wednesday, November 04, 2009 1:00 AM
To: Ken Williams; Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

Thanks for all your answers...

We made some tests yesterday with esx4 and cloning and snapshotting...

It seems that the challenge is not because of SMVI. If we try to clone
these machines which failed to create a snapshot (with the error below)
we get the same failure.

The error appears after 5 seconds...

But if do not quiesce we maybe get inconsistent snapshots... do you have
any experience restoring not quiesced snapshots??

Thanks and best regards,

Steffen

-----Original Message-----
From: Ken Williams [mailto:kwillia@...]
Sent: Tuesday, November 03, 2009 6:52 PM
To: Ken Williams; Steffen Kammerer; Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

Data ONTap version should read 7.3.1.1p7.

-----Original Message-----
From: Ken Williams
Sent: Tuesday, November 03, 2009 9:21 AM
To: 'Steffen Kammerer'; Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

Yes, we have the same issue.

Came down to a few things:

1. Update to 7.3.2p7. There are algorithm changes to WAFL that help with
VMFS/VMDK reading/writing. I saw a HUGE performance change with this.

2. Check your CPU (systat 0), ours was pegged due to NDMP backups;
changing the times helped out a bunch.

3. Disable File Sync in VMWare tools on each guest. The File Sync driver
is problematic and not recommended. This is on each guest in add/remove
programs for VMWare tools.

4. VMWare admitted this is a problem; most users accept the work around
to not do quiesced backups. There is a checkbox in SMVI that will allow
you to not do VMWare level snaps.

Otherwise try snaping smaller groups (10 max) of VMs. We're at about 80%
success right now; not great but moving in the right direction.

-----Original Message-----
From: Steffen Kammerer [mailto:steffen.kammerer@...]
Sent: Tuesday, November 03, 2009 6:41 AM
To: Ken Williams; Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

Hi there,

We have the same issue with SMVI 2.0 on nfs datastores... on some VMs we
get the following error:

Cannot create a quiesced snapshot because the create snapshot
operation exceeded the time limit for holding off I/O in the
frozen virtual machine.

Does anybody approach the same error?!

Thanks and best regards,

Steffen

-----Original Message-----
From: owner-toasters@... [mailto:owner-toasters@...]
On Behalf Of Ken Williams
Sent: Monday, September 14, 2009 6:51 PM
To: Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

Sounds like whatever user-defined script you have is failing sometimes?
Or perhaps it's a VMWare tools issue.

We've been able to track our issue down to the Guest OS level (win2k3
specifically). Looks like its an issue with VSS or LUN alignment.

I would recommend ensuring your LUNs are aligned (use the VMWare host
util kit, mbrscan / mbralign). There is detailed documentation on the
NOW.netapp.com site.

-----Original Message-----
From: Nick Silkey [mailto:nick@...]
Sent: Friday, September 11, 2009 7:15 PM
To: Ken Williams
Cc: toasters@...
Subject: Re: SMVI / VMWare Experiences...

Ken --

We too are experiencing issues with SMVI 1.2 bombing out when attempting
to perform a VMware quiesce snap on _some_ RHEL5.3 32-bit VMs. A couple
of notables:

- These problematic VMs have a 100% success rate at taking VMware
quiesce snaps within vCenter, independent of SMVI.
- The problem is 100% reproducible during night, day, etc.
- We will deploy several VMs at a crack, all the same build. When the
next SMVI schedule hits, some fail while others succeed. Bizarre.
- Over time (weve been experiencing this issue for several weeks now),
the 'problem' VMs change. Example: VMs abc and xyz will fail for days;
without intervention, VM abc will stop failing while VM xyz continues to
fail ... even if theyre part of the same deploy base template/kickstart.
- We are nowhere near our snap limit on the volumes.
- These problematic VMs only bomb when attempting a quiesce.
Non-quiesce SMVI snaps work like a champ.

Been working with NetApp and VMware for some time now. Were at ESX
3.5u4+ to an 3160-R5 @ 7.2.6.1P3 via NFS + vCenter 4.0 + synch
SnapMirror to another 3160-R5 @ 7.2.6.1P3. The only thing revealing is
SMVI + vCenter logs of "cannot create a quiesced snapshot because the
(user-supplied) custom prefreeze script in the virtual machine exited
with a nonzero return code".

--
Nick

On Wed, Aug 26, 2009 at 5:32 PM, Ken Williams <kwillia@...> wrote:
> I'm looking for some experiences people out there may have with SMVI
> with NetApp. We're currently experiencing major issues with SMVI
> snapshots failing. I've had open tickets with NetApp/VMWare/Microsoft
> for 3 months and still have yet to have a solution.
>
> My environment looks like such:
>
> 6 x HP DL380 G5 (32gb Ram) in a ESX Cluster Dual Emulex 10000 Cards in

> each host.
> Cisco MDS SAN
> Netapp FAS3070 Cluster ~9tb aggregate for VMWare.
> VMFS Datastores ~10-15 VMs per datastore. ~50gb per VM.
> ASIS Turned on
> Volume and LUNspace reservation turned off OnTap 7.2.5.1 Windows 2003
> Guest OS.
>
> I cant see us reaching any limitation on the Filers or the SAN. Yet we

> have random VMs failing snapshots every night. Are other people seeing

> these issues? (I've gone through the gamut of troubleshooting, version

> management of ESX/VMWareTools/etc). Snapshots timeout and fail at the
> VMWare/Guest level, not at the Netapp snapshot level.
>
> We want to have SMVI function with VSS enabled.
>
> Has anyone had failing snapshots been able to resolve a similar issue?

> Or does anyone have SMVI working properly that we could use as a
> reference to compare configuration?
>
> __________________________________________________________
> Ken Williams
> Storage Administrator, Business Technology Operations Sacramento
> Municipal Utility District
> E-Mail: kwillia@...
> Phone: (916) 732-6744
> Cell: (916) 240-4213

RE: SMVI / VMWare Experiences...

2009-11-04T00:59:37Z

Thanks for all your answers...

We made some tests yesterday with esx4 and cloning and snapshotting...

It seems that the challenge is not because of SMVI. If we try to clone these machines which failed to create a snapshot (with the error below) we get the same failure.

The error appears after 5 seconds...

But if do not quiesce we maybe get inconsistent snapshots... do you have any experience restoring not quiesced snapshots??

Thanks and best regards,

Steffen

-----Original Message-----
From: Ken Williams [mailto:kwillia@...]
Sent: Tuesday, November 03, 2009 6:52 PM
To: Ken Williams; Steffen Kammerer; Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

Data ONTap version should read 7.3.1.1p7.

-----Original Message-----
From: Ken Williams
Sent: Tuesday, November 03, 2009 9:21 AM
To: 'Steffen Kammerer'; Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

Yes, we have the same issue.

Came down to a few things:

1. Update to 7.3.2p7. There are algorithm changes to WAFL that help with
VMFS/VMDK reading/writing. I saw a HUGE performance change with this.

2. Check your CPU (systat 0), ours was pegged due to NDMP backups;
changing the times helped out a bunch.

3. Disable File Sync in VMWare tools on each guest. The File Sync driver
is problematic and not recommended. This is on each guest in add/remove
programs for VMWare tools.

4. VMWare admitted this is a problem; most users accept the work around
to not do quiesced backups. There is a checkbox in SMVI that will allow
you to not do VMWare level snaps.

Otherwise try snaping smaller groups (10 max) of VMs. We're at about 80%
success right now; not great but moving in the right direction.

-----Original Message-----
From: Steffen Kammerer [mailto:steffen.kammerer@...]
Sent: Tuesday, November 03, 2009 6:41 AM
To: Ken Williams; Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

Hi there,

We have the same issue with SMVI 2.0 on nfs datastores... on some VMs we
get the following error:

Cannot create a quiesced snapshot because the create snapshot
operation exceeded the time limit for holding off I/O in the
frozen virtual machine.

Does anybody approach the same error?!

Thanks and best regards,

Steffen

-----Original Message-----
From: owner-toasters@... [mailto:owner-toasters@...]
On Behalf Of Ken Williams
Sent: Monday, September 14, 2009 6:51 PM
To: Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

Sounds like whatever user-defined script you have is failing sometimes?
Or perhaps it's a VMWare tools issue.

We've been able to track our issue down to the Guest OS level (win2k3
specifically). Looks like its an issue with VSS or LUN alignment.

I would recommend ensuring your LUNs are aligned (use the VMWare host
util kit, mbrscan / mbralign). There is detailed documentation on the
NOW.netapp.com site.

-----Original Message-----
From: Nick Silkey [mailto:nick@...]
Sent: Friday, September 11, 2009 7:15 PM
To: Ken Williams
Cc: toasters@...
Subject: Re: SMVI / VMWare Experiences...

Ken --

We too are experiencing issues with SMVI 1.2 bombing out when attempting
to perform a VMware quiesce snap on _some_ RHEL5.3 32-bit VMs. A couple
of notables:

- These problematic VMs have a 100% success rate at taking VMware
quiesce snaps within vCenter, independent of SMVI.
- The problem is 100% reproducible during night, day, etc.
- We will deploy several VMs at a crack, all the same build. When the
next SMVI schedule hits, some fail while others succeed. Bizarre.
- Over time (weve been experiencing this issue for several weeks now),
the 'problem' VMs change. Example: VMs abc and xyz will fail for days;
without intervention, VM abc will stop failing while VM xyz continues to
fail ... even if theyre part of the same deploy base template/kickstart.
- We are nowhere near our snap limit on the volumes.
- These problematic VMs only bomb when attempting a quiesce.
Non-quiesce SMVI snaps work like a champ.

Been working with NetApp and VMware for some time now. Were at ESX
3.5u4+ to an 3160-R5 @ 7.2.6.1P3 via NFS + vCenter 4.0 + synch
SnapMirror to another 3160-R5 @ 7.2.6.1P3. The only thing revealing is
SMVI + vCenter logs of "cannot create a quiesced snapshot because the
(user-supplied) custom prefreeze script in the virtual machine exited
with a nonzero return code".

--
Nick

On Wed, Aug 26, 2009 at 5:32 PM, Ken Williams <kwillia@...> wrote:
> I'm looking for some experiences people out there may have with SMVI
> with NetApp. We're currently experiencing major issues with SMVI
> snapshots failing. I've had open tickets with NetApp/VMWare/Microsoft
> for 3 months and still have yet to have a solution.
>
> My environment looks like such:
>
> 6 x HP DL380 G5 (32gb Ram) in a ESX Cluster Dual Emulex 10000 Cards in

> each host.
> Cisco MDS SAN
> Netapp FAS3070 Cluster ~9tb aggregate for VMWare.
> VMFS Datastores ~10-15 VMs per datastore. ~50gb per VM.
> ASIS Turned on
> Volume and LUNspace reservation turned off OnTap 7.2.5.1 Windows 2003
> Guest OS.
>
> I cant see us reaching any limitation on the Filers or the SAN. Yet we

> have random VMs failing snapshots every night. Are other people seeing

> these issues? (I've gone through the gamut of troubleshooting, version

> management of ESX/VMWareTools/etc). Snapshots timeout and fail at the
> VMWare/Guest level, not at the Netapp snapshot level.
>
> We want to have SMVI function with VSS enabled.
>
> Has anyone had failing snapshots been able to resolve a similar issue?

RE: SMVI / VMWare Experiences...

2009-11-03T09:52:05Z

Data ONTap version should read 7.3.1.1p7.

-----Original Message-----
From: Ken Williams
Sent: Tuesday, November 03, 2009 9:21 AM
To: 'Steffen Kammerer'; Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

Yes, we have the same issue.

Came down to a few things:

1. Update to 7.3.2p7. There are algorithm changes to WAFL that help with
VMFS/VMDK reading/writing. I saw a HUGE performance change with this.

2. Check your CPU (systat 0), ours was pegged due to NDMP backups;
changing the times helped out a bunch.

3. Disable File Sync in VMWare tools on each guest. The File Sync driver
is problematic and not recommended. This is on each guest in add/remove
programs for VMWare tools.

4. VMWare admitted this is a problem; most users accept the work around
to not do quiesced backups. There is a checkbox in SMVI that will allow
you to not do VMWare level snaps.

Otherwise try snaping smaller groups (10 max) of VMs. We're at about 80%
success right now; not great but moving in the right direction.

-----Original Message-----
From: Steffen Kammerer [mailto:steffen.kammerer@...]
Sent: Tuesday, November 03, 2009 6:41 AM
To: Ken Williams; Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

Hi there,

We have the same issue with SMVI 2.0 on nfs datastores... on some VMs we
get the following error:

Cannot create a quiesced snapshot because the create snapshot
operation exceeded the time limit for holding off I/O in the
frozen virtual machine.

Does anybody approach the same error?!

Thanks and best regards,

Steffen

-----Original Message-----
From: owner-toasters@... [mailto:owner-toasters@...]
On Behalf Of Ken Williams
Sent: Monday, September 14, 2009 6:51 PM
To: Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

Sounds like whatever user-defined script you have is failing sometimes?
Or perhaps it's a VMWare tools issue.

We've been able to track our issue down to the Guest OS level (win2k3
specifically). Looks like its an issue with VSS or LUN alignment.

I would recommend ensuring your LUNs are aligned (use the VMWare host
util kit, mbrscan / mbralign). There is detailed documentation on the
NOW.netapp.com site.

-----Original Message-----
From: Nick Silkey [mailto:nick@...]
Sent: Friday, September 11, 2009 7:15 PM
To: Ken Williams
Cc: toasters@...
Subject: Re: SMVI / VMWare Experiences...

Ken --

We too are experiencing issues with SMVI 1.2 bombing out when attempting
to perform a VMware quiesce snap on _some_ RHEL5.3 32-bit VMs. A couple
of notables:

- These problematic VMs have a 100% success rate at taking VMware
quiesce snaps within vCenter, independent of SMVI.
- The problem is 100% reproducible during night, day, etc.
- We will deploy several VMs at a crack, all the same build. When the
next SMVI schedule hits, some fail while others succeed. Bizarre.
- Over time (weve been experiencing this issue for several weeks now),
the 'problem' VMs change. Example: VMs abc and xyz will fail for days;
without intervention, VM abc will stop failing while VM xyz continues to
fail ... even if theyre part of the same deploy base template/kickstart.
- We are nowhere near our snap limit on the volumes.
- These problematic VMs only bomb when attempting a quiesce.
Non-quiesce SMVI snaps work like a champ.

Been working with NetApp and VMware for some time now. Were at ESX
3.5u4+ to an 3160-R5 @ 7.2.6.1P3 via NFS + vCenter 4.0 + synch
SnapMirror to another 3160-R5 @ 7.2.6.1P3. The only thing revealing is
SMVI + vCenter logs of "cannot create a quiesced snapshot because the
(user-supplied) custom prefreeze script in the virtual machine exited
with a nonzero return code".

--
Nick

On Wed, Aug 26, 2009 at 5:32 PM, Ken Williams <kwillia@...> wrote:
> I'm looking for some experiences people out there may have with SMVI
> with NetApp. We're currently experiencing major issues with SMVI
> snapshots failing. I've had open tickets with NetApp/VMWare/Microsoft
> for 3 months and still have yet to have a solution.
>
> My environment looks like such:
>
> 6 x HP DL380 G5 (32gb Ram) in a ESX Cluster Dual Emulex 10000 Cards in

> each host.
> Cisco MDS SAN
> Netapp FAS3070 Cluster ~9tb aggregate for VMWare.
> VMFS Datastores ~10-15 VMs per datastore. ~50gb per VM.
> ASIS Turned on
> Volume and LUNspace reservation turned off OnTap 7.2.5.1 Windows 2003
> Guest OS.
>
> I cant see us reaching any limitation on the Filers or the SAN. Yet we

> have random VMs failing snapshots every night. Are other people seeing

> these issues? (I've gone through the gamut of troubleshooting, version

> management of ESX/VMWareTools/etc). Snapshots timeout and fail at the
> VMWare/Guest level, not at the Netapp snapshot level.
>
> We want to have SMVI function with VSS enabled.
>
> Has anyone had failing snapshots been able to resolve a similar issue?

RE: SMVI / VMWare Experiences...

2009-11-03T09:21:18Z

Yes, we have the same issue.

Came down to a few things:

1. Update to 7.3.2p7. There are algorithm changes to WAFL that help with
VMFS/VMDK reading/writing. I saw a HUGE performance change with this.

2. Check your CPU (systat 0), ours was pegged due to NDMP backups;
changing the times helped out a bunch.

3. Disable File Sync in VMWare tools on each guest. The File Sync driver
is problematic and not recommended. This is on each guest in add/remove
programs for VMWare tools.

4. VMWare admitted this is a problem; most users accept the work around
to not do quiesced backups. There is a checkbox in SMVI that will allow
you to not do VMWare level snaps.

Otherwise try snaping smaller groups (10 max) of VMs. We're at about 80%
success right now; not great but moving in the right direction.

-----Original Message-----
From: Steffen Kammerer [mailto:steffen.kammerer@...]
Sent: Tuesday, November 03, 2009 6:41 AM
To: Ken Williams; Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

Hi there,

We have the same issue with SMVI 2.0 on nfs datastores... on some VMs we
get the following error:

Cannot create a quiesced snapshot because the create snapshot
operation exceeded the time limit for holding off I/O in the
frozen virtual machine.

Does anybody approach the same error?!

Thanks and best regards,

Steffen

-----Original Message-----
From: owner-toasters@... [mailto:owner-toasters@...]
On Behalf Of Ken Williams
Sent: Monday, September 14, 2009 6:51 PM
To: Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

Sounds like whatever user-defined script you have is failing sometimes?
Or perhaps it's a VMWare tools issue.

We've been able to track our issue down to the Guest OS level (win2k3
specifically). Looks like its an issue with VSS or LUN alignment.

I would recommend ensuring your LUNs are aligned (use the VMWare host
util kit, mbrscan / mbralign). There is detailed documentation on the
NOW.netapp.com site.

-----Original Message-----
From: Nick Silkey [mailto:nick@...]
Sent: Friday, September 11, 2009 7:15 PM
To: Ken Williams
Cc: toasters@...
Subject: Re: SMVI / VMWare Experiences...

Ken --

We too are experiencing issues with SMVI 1.2 bombing out when attempting
to perform a VMware quiesce snap on _some_ RHEL5.3 32-bit VMs. A couple
of notables:

- These problematic VMs have a 100% success rate at taking VMware
quiesce snaps within vCenter, independent of SMVI.
- The problem is 100% reproducible during night, day, etc.
- We will deploy several VMs at a crack, all the same build. When the
next SMVI schedule hits, some fail while others succeed. Bizarre.
- Over time (weve been experiencing this issue for several weeks now),
the 'problem' VMs change. Example: VMs abc and xyz will fail for days;
without intervention, VM abc will stop failing while VM xyz continues to
fail ... even if theyre part of the same deploy base template/kickstart.
- We are nowhere near our snap limit on the volumes.
- These problematic VMs only bomb when attempting a quiesce.
Non-quiesce SMVI snaps work like a champ.

Been working with NetApp and VMware for some time now. Were at ESX
3.5u4+ to an 3160-R5 @ 7.2.6.1P3 via NFS + vCenter 4.0 + synch
SnapMirror to another 3160-R5 @ 7.2.6.1P3. The only thing revealing is
SMVI + vCenter logs of "cannot create a quiesced snapshot because the
(user-supplied) custom prefreeze script in the virtual machine exited
with a nonzero return code".

--
Nick

On Wed, Aug 26, 2009 at 5:32 PM, Ken Williams <kwillia@...> wrote:
> I'm looking for some experiences people out there may have with SMVI
> with NetApp. We're currently experiencing major issues with SMVI
> snapshots failing. I've had open tickets with NetApp/VMWare/Microsoft
> for 3 months and still have yet to have a solution.
>
> My environment looks like such:
>
> 6 x HP DL380 G5 (32gb Ram) in a ESX Cluster Dual Emulex 10000 Cards in

> each host.
> Cisco MDS SAN
> Netapp FAS3070 Cluster ~9tb aggregate for VMWare.
> VMFS Datastores ~10-15 VMs per datastore. ~50gb per VM.
> ASIS Turned on
> Volume and LUNspace reservation turned off OnTap 7.2.5.1 Windows 2003
> Guest OS.
>
> I cant see us reaching any limitation on the Filers or the SAN. Yet we

> have random VMs failing snapshots every night. Are other people seeing

> these issues? (I've gone through the gamut of troubleshooting, version

> management of ESX/VMWareTools/etc). Snapshots timeout and fail at the
> VMWare/Guest level, not at the Netapp snapshot level.
>
> We want to have SMVI function with VSS enabled.
>
> Has anyone had failing snapshots been able to resolve a similar issue?

RE: SMVI / VMWare Experiences...

2009-11-03T07:42:31Z

Crud. Make sure the partition offset/4096 is an integer, not even.
Sorry.

And I'm not too certain about the 4096 #, read the TR :)

-----Original Message-----
From: owner-toasters@... [mailto:owner-toasters@...]
On Behalf Of Page, Jeremy
Sent: Tuesday, November 03, 2009 10:38 AM
To: Webster, Stetson; Steffen Kammerer; Ken Williams; Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

If your VMs are Windows it's relatively simple to do a WMI scan against
them for the partition offset and then divide by 4096 (?) and make sure
it's an even #. If not your not set up ideally. Something like below
(please excuse my scriptomatic generated code).

Set objWMIService = GetObject("winmgmts:\\" & strComputer &
"\root\CIMV2")
Set colItems = objWMIService.ExecQuery("SELECT * FROM
Win32_DiskPartition", "WQL", _
wbemFlagReturnImmediately +
wbemFlagForwardOnly)

For Each objItem In colItems
WScript.Echo "StartingOffset: " & objItem.StartingOffset
Next

-----Original Message-----
From: owner-toasters@... [mailto:owner-toasters@...]
On Behalf Of Webster, Stetson
Sent: Tuesday, November 03, 2009 10:26 AM
To: Steffen Kammerer; Ken Williams; Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

This is very commonly an alignment issue. Snapshots exacerbate the
already intense I/O caused by misalignment. Essentially, the existence
of VMware snapshots (although they are 100% successful), cause the I/O
on the system to intensify while ONTAP is actually trying to quiesce for
a snapshot. So we are having a bad situation that intensifies at the
wrong time.

Have you tried to take the NetApp snapshot manually (sans SMVI) after
the VMware snapshots have all completed? My guess is that you will find
that the snapshot takes a very long time to complete (if it does).

Here is a document that discusses block alignment and the breadth of
it's impact:

Best Practices for File System Alignment in Virtual Environments:
http://www.netapp.com/us/library/technical-reports/tr-3747.html

We have a tool called 'mbrscan' for identifying misalignment which is a
part of our ESX Host Utilities available here:

FC Host Utilities for ESX(r):
http://now.netapp.com/NOW/download/software/sanhost_esx/ESX

We also have this KB article for identifying misalignment:

How to diagnose misaligned I/O on Windows hosts:
https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb36108

Finally, while mbralign is now part of the ESX Host Utilities, there is
good, detailed documentation here as you navigate to the download page:

mbralign:
http://now.netapp.com/NOW/download/tools/mbralign

Stetson M. Webster
Professional Services Consultant
NCIE-SAN, NCIE-B&R, SNIA-SCSN-E
NetApp Professional Services - East
919.250.0052 Mobile
Stetson.Webster@...
Learn how: netapp.com/guarantee

-----Original Message-----
From: Steffen Kammerer [mailto:steffen.kammerer@...]
Sent: Tuesday, November 03, 2009 9:41 AM
To: Ken Williams; Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

Hi there,

We have the same issue with SMVI 2.0 on nfs datastores... on some VMs we
get the following error:

Cannot create a quiesced snapshot because the create snapshot
operation exceeded the time limit for holding off I/O in the
frozen virtual machine.

Does anybody approach the same error?!

Thanks and best regards,

Steffen

-----Original Message-----
From: owner-toasters@... [mailto:owner-toasters@...]
On Behalf Of Ken Williams
Sent: Monday, September 14, 2009 6:51 PM
To: Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

Sounds like whatever user-defined script you have is failing sometimes?
Or perhaps it's a VMWare tools issue.

We've been able to track our issue down to the Guest OS level (win2k3
specifically). Looks like its an issue with VSS or LUN alignment.

I would recommend ensuring your LUNs are aligned (use the VMWare host
util kit, mbrscan / mbralign). There is detailed documentation on the
NOW.netapp.com site.

-----Original Message-----
From: Nick Silkey [mailto:nick@...]
Sent: Friday, September 11, 2009 7:15 PM
To: Ken Williams
Cc: toasters@...
Subject: Re: SMVI / VMWare Experiences...

Ken --

We too are experiencing issues with SMVI 1.2 bombing out when attempting
to perform a VMware quiesce snap on _some_ RHEL5.3 32-bit VMs. A couple
of notables:

- These problematic VMs have a 100% success rate at taking VMware
quiesce snaps within vCenter, independent of SMVI.
- The problem is 100% reproducible during night, day, etc.
- We will deploy several VMs at a crack, all the same build. When the
next SMVI schedule hits, some fail while others succeed. Bizarre.
- Over time (weve been experiencing this issue for several weeks now),
the 'problem' VMs change. Example: VMs abc and xyz will fail for days;
without intervention, VM abc will stop failing while VM xyz continues to
fail ... even if theyre part of the same deploy base template/kickstart.
- We are nowhere near our snap limit on the volumes.
- These problematic VMs only bomb when attempting a quiesce.
Non-quiesce SMVI snaps work like a champ.

Been working with NetApp and VMware for some time now. Were at ESX
3.5u4+ to an 3160-R5 @ 7.2.6.1P3 via NFS + vCenter 4.0 + synch
SnapMirror to another 3160-R5 @ 7.2.6.1P3. The only thing revealing is
SMVI + vCenter logs of "cannot create a quiesced snapshot because the
(user-supplied) custom prefreeze script in the virtual machine exited
with a nonzero return code".

--
Nick

On Wed, Aug 26, 2009 at 5:32 PM, Ken Williams <kwillia@...> wrote:
> I'm looking for some experiences people out there may have with SMVI
> with NetApp. We're currently experiencing major issues with SMVI
> snapshots failing. I've had open tickets with NetApp/VMWare/Microsoft
> for 3 months and still have yet to have a solution.
>
> My environment looks like such:
>
> 6 x HP DL380 G5 (32gb Ram) in a ESX Cluster Dual Emulex 10000 Cards in

> each host.
> Cisco MDS SAN
> Netapp FAS3070 Cluster ~9tb aggregate for VMWare.
> VMFS Datastores ~10-15 VMs per datastore. ~50gb per VM.
> ASIS Turned on
> Volume and LUNspace reservation turned off OnTap 7.2.5.1 Windows 2003
> Guest OS.
>
> I cant see us reaching any limitation on the Filers or the SAN. Yet we

> have random VMs failing snapshots every night. Are other people seeing

> these issues? (I've gone through the gamut of troubleshooting, version

> management of ESX/VMWareTools/etc). Snapshots timeout and fail at the
> VMWare/Guest level, not at the Netapp snapshot level.
>
> We want to have SMVI function with VSS enabled.
>
> Has anyone had failing snapshots been able to resolve a similar issue?

Please be advised that this email may contain confidential information.
If you are not the intended recipient, please do not read, copy or
re-transmit this email. If you have received this email in error,
please notify us by email by replying to the sender and by telephone
(call us collect at +1 202-828-0850) and delete this message and any
attachments. Thank you in advance for your cooperation and assistance.

In addition, Danaher and its subsidiaries disclaim that the content of
this email constitutes an offer to enter into, or the acceptance of,
any
contract or agreement or any amendment thereto; provided that the
foregoing disclaimer does not invalidate the binding effect of any
digital or other electronic reproduction of a manual signature that is
included in any attachment to this email.

Please be advised that this email may contain confidential information.
If you are not the intended recipient, please do not read, copy or
re-transmit this email. If you have received this email in error,
please notify us by email by replying to the sender and by telephone
(call us collect at +1 202-828-0850) and delete this message and any
attachments. Thank you in advance for your cooperation and assistance.

In addition, Danaher and its subsidiaries disclaim that the content of
this email constitutes an offer to enter into, or the acceptance of,
any
contract or agreement or any amendment thereto; provided that the
foregoing disclaimer does not invalidate the binding effect of any
digital or other electronic reproduction of a manual signature that is
included in any attachment to this email.

RE: SMVI / VMWare Experiences...

2009-11-03T07:42:31Z

RE: SMVI / VMWare Experiences...

2009-11-03T07:38:09Z

If your VMs are Windows it's relatively simple to do a WMI scan against
them for the partition offset and then divide by 4096 (?) and make sure
it's an even #. If not your not set up ideally. Something like below
(please excuse my scriptomatic generated code).

Set objWMIService = GetObject("winmgmts:\\" & strComputer &
"\root\CIMV2")
Set colItems = objWMIService.ExecQuery("SELECT * FROM
Win32_DiskPartition", "WQL", _
wbemFlagReturnImmediately +
wbemFlagForwardOnly)

For Each objItem In colItems
WScript.Echo "StartingOffset: " & objItem.StartingOffset
Next

-----Original Message-----
From: owner-toasters@... [mailto:owner-toasters@...]
On Behalf Of Webster, Stetson
Sent: Tuesday, November 03, 2009 10:26 AM
To: Steffen Kammerer; Ken Williams; Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

This is very commonly an alignment issue. Snapshots exacerbate the
already intense I/O caused by misalignment. Essentially, the existence
of VMware snapshots (although they are 100% successful), cause the I/O
on the system to intensify while ONTAP is actually trying to quiesce for
a snapshot. So we are having a bad situation that intensifies at the
wrong time.

Have you tried to take the NetApp snapshot manually (sans SMVI) after
the VMware snapshots have all completed? My guess is that you will find
that the snapshot takes a very long time to complete (if it does).

Here is a document that discusses block alignment and the breadth of
it's impact:

Best Practices for File System Alignment in Virtual Environments:
http://www.netapp.com/us/library/technical-reports/tr-3747.html

We have a tool called 'mbrscan' for identifying misalignment which is a
part of our ESX Host Utilities available here:

FC Host Utilities for ESX(r):
http://now.netapp.com/NOW/download/software/sanhost_esx/ESX

We also have this KB article for identifying misalignment:

How to diagnose misaligned I/O on Windows hosts:
https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb36108

Finally, while mbralign is now part of the ESX Host Utilities, there is
good, detailed documentation here as you navigate to the download page:

mbralign:
http://now.netapp.com/NOW/download/tools/mbralign

Stetson M. Webster
Professional Services Consultant
NCIE-SAN, NCIE-B&R, SNIA-SCSN-E
NetApp Professional Services - East
919.250.0052 Mobile
Stetson.Webster@...
Learn how: netapp.com/guarantee

-----Original Message-----
From: Steffen Kammerer [mailto:steffen.kammerer@...]
Sent: Tuesday, November 03, 2009 9:41 AM
To: Ken Williams; Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

Hi there,

We have the same issue with SMVI 2.0 on nfs datastores... on some VMs we
get the following error:

Cannot create a quiesced snapshot because the create snapshot
operation exceeded the time limit for holding off I/O in the
frozen virtual machine.

Does anybody approach the same error?!

Thanks and best regards,

Steffen

-----Original Message-----
From: owner-toasters@... [mailto:owner-toasters@...]
On Behalf Of Ken Williams
Sent: Monday, September 14, 2009 6:51 PM
To: Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

Sounds like whatever user-defined script you have is failing sometimes?
Or perhaps it's a VMWare tools issue.

We've been able to track our issue down to the Guest OS level (win2k3
specifically). Looks like its an issue with VSS or LUN alignment.

I would recommend ensuring your LUNs are aligned (use the VMWare host
util kit, mbrscan / mbralign). There is detailed documentation on the
NOW.netapp.com site.

-----Original Message-----
From: Nick Silkey [mailto:nick@...]
Sent: Friday, September 11, 2009 7:15 PM
To: Ken Williams
Cc: toasters@...
Subject: Re: SMVI / VMWare Experiences...

Ken --

We too are experiencing issues with SMVI 1.2 bombing out when attempting
to perform a VMware quiesce snap on _some_ RHEL5.3 32-bit VMs. A couple
of notables:

- These problematic VMs have a 100% success rate at taking VMware
quiesce snaps within vCenter, independent of SMVI.
- The problem is 100% reproducible during night, day, etc.
- We will deploy several VMs at a crack, all the same build. When the
next SMVI schedule hits, some fail while others succeed. Bizarre.
- Over time (weve been experiencing this issue for several weeks now),
the 'problem' VMs change. Example: VMs abc and xyz will fail for days;
without intervention, VM abc will stop failing while VM xyz continues to
fail ... even if theyre part of the same deploy base template/kickstart.
- We are nowhere near our snap limit on the volumes.
- These problematic VMs only bomb when attempting a quiesce.
Non-quiesce SMVI snaps work like a champ.

Been working with NetApp and VMware for some time now. Were at ESX
3.5u4+ to an 3160-R5 @ 7.2.6.1P3 via NFS + vCenter 4.0 + synch
SnapMirror to another 3160-R5 @ 7.2.6.1P3. The only thing revealing is
SMVI + vCenter logs of "cannot create a quiesced snapshot because the
(user-supplied) custom prefreeze script in the virtual machine exited
with a nonzero return code".

--
Nick

On Wed, Aug 26, 2009 at 5:32 PM, Ken Williams <kwillia@...> wrote:
> I'm looking for some experiences people out there may have with SMVI
> with NetApp. We're currently experiencing major issues with SMVI
> snapshots failing. I've had open tickets with NetApp/VMWare/Microsoft
> for 3 months and still have yet to have a solution.
>
> My environment looks like such:
>
> 6 x HP DL380 G5 (32gb Ram) in a ESX Cluster Dual Emulex 10000 Cards in

> each host.
> Cisco MDS SAN
> Netapp FAS3070 Cluster ~9tb aggregate for VMWare.
> VMFS Datastores ~10-15 VMs per datastore. ~50gb per VM.
> ASIS Turned on
> Volume and LUNspace reservation turned off OnTap 7.2.5.1 Windows 2003
> Guest OS.
>
> I cant see us reaching any limitation on the Filers or the SAN. Yet we

> have random VMs failing snapshots every night. Are other people seeing

> these issues? (I've gone through the gamut of troubleshooting, version

> management of ESX/VMWareTools/etc). Snapshots timeout and fail at the
> VMWare/Guest level, not at the Netapp snapshot level.
>
> We want to have SMVI function with VSS enabled.
>
> Has anyone had failing snapshots been able to resolve a similar issue?

RE: SMVI / VMWare Experiences...

2009-11-03T07:26:10Z

This is very commonly an alignment issue. Snapshots exacerbate the
already intense I/O caused by misalignment. Essentially, the existence
of VMware snapshots (although they are 100% successful), cause the I/O
on the system to intensify while ONTAP is actually trying to quiesce for
a snapshot. So we are having a bad situation that intensifies at the
wrong time.

Have you tried to take the NetApp snapshot manually (sans SMVI) after
the VMware snapshots have all completed? My guess is that you will find
that the snapshot takes a very long time to complete (if it does).

Here is a document that discusses block alignment and the breadth of
it's impact:

Best Practices for File System Alignment in Virtual Environments:
http://www.netapp.com/us/library/technical-reports/tr-3747.html

We have a tool called 'mbrscan' for identifying misalignment which is a
part of our ESX Host Utilities available here:

FC Host Utilities for ESX(r):
http://now.netapp.com/NOW/download/software/sanhost_esx/ESX

We also have this KB article for identifying misalignment:

How to diagnose misaligned I/O on Windows hosts:
https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb36108

Finally, while mbralign is now part of the ESX Host Utilities, there is
good, detailed documentation here as you navigate to the download page:

mbralign:
http://now.netapp.com/NOW/download/tools/mbralign

Stetson M. Webster
Professional Services Consultant
NCIE-SAN, NCIE-B&R, SNIA-SCSN-E
NetApp Professional Services - East
919.250.0052 Mobile
Stetson.Webster@...
Learn how: netapp.com/guarantee

-----Original Message-----
From: Steffen Kammerer [mailto:steffen.kammerer@...]
Sent: Tuesday, November 03, 2009 9:41 AM
To: Ken Williams; Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

Hi there,

We have the same issue with SMVI 2.0 on nfs datastores... on some VMs we
get the following error:

Cannot create a quiesced snapshot because the create snapshot
operation exceeded the time limit for holding off I/O in the
frozen virtual machine.

Does anybody approach the same error?!

Thanks and best regards,

Steffen

-----Original Message-----
From: owner-toasters@... [mailto:owner-toasters@...]
On Behalf Of Ken Williams
Sent: Monday, September 14, 2009 6:51 PM
To: Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

Sounds like whatever user-defined script you have is failing sometimes?
Or perhaps it's a VMWare tools issue.

We've been able to track our issue down to the Guest OS level (win2k3
specifically). Looks like its an issue with VSS or LUN alignment.

I would recommend ensuring your LUNs are aligned (use the VMWare host
util kit, mbrscan / mbralign). There is detailed documentation on the
NOW.netapp.com site.

-----Original Message-----
From: Nick Silkey [mailto:nick@...]
Sent: Friday, September 11, 2009 7:15 PM
To: Ken Williams
Cc: toasters@...
Subject: Re: SMVI / VMWare Experiences...

Ken --

We too are experiencing issues with SMVI 1.2 bombing out when attempting
to perform a VMware quiesce snap on _some_ RHEL5.3 32-bit VMs. A couple
of notables:

- These problematic VMs have a 100% success rate at taking VMware
quiesce snaps within vCenter, independent of SMVI.
- The problem is 100% reproducible during night, day, etc.
- We will deploy several VMs at a crack, all the same build. When the
next SMVI schedule hits, some fail while others succeed. Bizarre.
- Over time (weve been experiencing this issue for several weeks now),
the 'problem' VMs change. Example: VMs abc and xyz will fail for days;
without intervention, VM abc will stop failing while VM xyz continues to
fail ... even if theyre part of the same deploy base template/kickstart.
- We are nowhere near our snap limit on the volumes.
- These problematic VMs only bomb when attempting a quiesce.
Non-quiesce SMVI snaps work like a champ.

Been working with NetApp and VMware for some time now. Were at ESX
3.5u4+ to an 3160-R5 @ 7.2.6.1P3 via NFS + vCenter 4.0 + synch
SnapMirror to another 3160-R5 @ 7.2.6.1P3. The only thing revealing is
SMVI + vCenter logs of "cannot create a quiesced snapshot because the
(user-supplied) custom prefreeze script in the virtual machine exited
with a nonzero return code".

--
Nick

On Wed, Aug 26, 2009 at 5:32 PM, Ken Williams <kwillia@...> wrote:
> I'm looking for some experiences people out there may have with SMVI
> with NetApp. We're currently experiencing major issues with SMVI
> snapshots failing. I've had open tickets with NetApp/VMWare/Microsoft
> for 3 months and still have yet to have a solution.
>
> My environment looks like such:
>
> 6 x HP DL380 G5 (32gb Ram) in a ESX Cluster Dual Emulex 10000 Cards in

> each host.
> Cisco MDS SAN
> Netapp FAS3070 Cluster ~9tb aggregate for VMWare.
> VMFS Datastores ~10-15 VMs per datastore. ~50gb per VM.
> ASIS Turned on
> Volume and LUNspace reservation turned off OnTap 7.2.5.1 Windows 2003
> Guest OS.
>
> I cant see us reaching any limitation on the Filers or the SAN. Yet we

> have random VMs failing snapshots every night. Are other people seeing

> these issues? (I've gone through the gamut of troubleshooting, version

> management of ESX/VMWareTools/etc). Snapshots timeout and fail at the
> VMWare/Guest level, not at the Netapp snapshot level.
>
> We want to have SMVI function with VSS enabled.
>
> Has anyone had failing snapshots been able to resolve a similar issue?

RE: SMVI / VMWare Experiences...

2009-11-03T06:41:04Z

Hi there,

We have the same issue with SMVI 2.0 on nfs datastores... on some VMs we get the following error:

Cannot create a quiesced snapshot because the create snapshot
operation exceeded the time limit for holding off I/O in the
frozen virtual machine.

Does anybody approach the same error?!

Thanks and best regards,

Steffen

-----Original Message-----
From: owner-toasters@... [mailto:owner-toasters@...] On Behalf Of Ken Williams
Sent: Monday, September 14, 2009 6:51 PM
To: Nick Silkey
Cc: toasters@...
Subject: RE: SMVI / VMWare Experiences...

Sounds like whatever user-defined script you have is failing sometimes?
Or perhaps it's a VMWare tools issue.

We've been able to track our issue down to the Guest OS level (win2k3
specifically). Looks like its an issue with VSS or LUN alignment.

I would recommend ensuring your LUNs are aligned (use the VMWare host
util kit, mbrscan / mbralign). There is detailed documentation on the
NOW.netapp.com site.

-----Original Message-----
From: Nick Silkey [mailto:nick@...]
Sent: Friday, September 11, 2009 7:15 PM
To: Ken Williams
Cc: toasters@...
Subject: Re: SMVI / VMWare Experiences...

Ken --

We too are experiencing issues with SMVI 1.2 bombing out when attempting
to perform a VMware quiesce snap on _some_ RHEL5.3 32-bit VMs. A couple
of notables:

- These problematic VMs have a 100% success rate at taking VMware
quiesce snaps within vCenter, independent of SMVI.
- The problem is 100% reproducible during night, day, etc.
- We will deploy several VMs at a crack, all the same build. When the
next SMVI schedule hits, some fail while others succeed. Bizarre.
- Over time (weve been experiencing this issue for several weeks now),
the 'problem' VMs change. Example: VMs abc and xyz will fail for days;
without intervention, VM abc will stop failing while VM xyz continues to
fail ... even if theyre part of the same deploy base template/kickstart.
- We are nowhere near our snap limit on the volumes.
- These problematic VMs only bomb when attempting a quiesce.
Non-quiesce SMVI snaps work like a champ.

Been working with NetApp and VMware for some time now. Were at ESX
3.5u4+ to an 3160-R5 @ 7.2.6.1P3 via NFS + vCenter 4.0 + synch
SnapMirror to another 3160-R5 @ 7.2.6.1P3. The only thing revealing is
SMVI + vCenter logs of "cannot create a quiesced snapshot because the
(user-supplied) custom prefreeze script in the virtual machine exited
with a nonzero return code".

--
Nick

On Wed, Aug 26, 2009 at 5:32 PM, Ken Williams <kwillia@...> wrote:
> I'm looking for some experiences people out there may have with SMVI
> with NetApp. We're currently experiencing major issues with SMVI
> snapshots failing. I've had open tickets with NetApp/VMWare/Microsoft
> for 3 months and still have yet to have a solution.
>
> My environment looks like such:
>
> 6 x HP DL380 G5 (32gb Ram) in a ESX Cluster Dual Emulex 10000 Cards in

> each host.
> Cisco MDS SAN
> Netapp FAS3070 Cluster ~9tb aggregate for VMWare.
> VMFS Datastores ~10-15 VMs per datastore. ~50gb per VM.
> ASIS Turned on
> Volume and LUNspace reservation turned off OnTap 7.2.5.1 Windows 2003
> Guest OS.
>
> I cant see us reaching any limitation on the Filers or the SAN. Yet we

> have random VMs failing snapshots every night. Are other people seeing

> these issues? (I've gone through the gamut of troubleshooting, version

> management of ESX/VMWareTools/etc). Snapshots timeout and fail at the
> VMWare/Guest level, not at the Netapp snapshot level.
>
> We want to have SMVI function with VSS enabled.
>
> Has anyone had failing snapshots been able to resolve a similar issue?

SUMMARY: trouble accessing nfs .snapshot dirs via cifs shares

2009-11-01T23:34:56Z

Hi Everyone, First off I'd like to thank everyone whom wrote back to me privately with some suggestions on how to fix this issue. (Original question at bottom of message.) Though I'm glad people wrote to me privately I wish they would reply back to the mailing list as well as I often find answers to questions I have in mailing list archives.

I also wanted to write back to say that I finally have gotten the issue resolved, though after having spent way too much time on it. (Netapp support was seemingly unable to help me resolve this issue.)

Kudo's to Nick Bernstein for having a suggestion which was very close to the actual fix. Though I must admit that I didn't think his suggestion was close to the issue I was having and thus didn't look into much until the very end of my research. (A.K.A. a lot of wasted time on my part :( )

Here is Nicks response:

When you first access a volume over nfs, it does a create/convert
unicode on the volume. Snapshots are read-only, so they can't have the
same conversion applied. This is normally the cause when you can
access the normal volume, but not the snapshots for it. Once new snaps
are taken you should be able to access the new ones.

The fix to the "Access Denied" error when trying to access a snapvault destination qtree with unix security via cifs is to set the options convert_ucode and create_ucode to on *before* you initiate the snapvault relationship with "snapvault start" otherwise the qtree is created without the unicode encoding and since the snapvault dest. is read only changing the ucode options after the qtree is created doesn't help and thus you have no access to any of your data (including new snapshots) via cifs, though nfs access is still fine.

Hope this helps someone save some time in the future.

What eventually tipped me off was this knowledgebase article, which is written for snapmirror but is similiar enough to the issue that I was having with snapvault that I was able to figure it out.

https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb23656

Romeo

Original Question:

On Thu, Oct 22, 2009 at 11:32 PM, Romeo Theriault <romeotheriault@...> wrote:

Hello, We have a VMware environment with nfs as the transport protocol. What I'm testing out is snapvaulting the vmware datastores to another filer and then creating cifs shares on the snapvaulted volumes (qtrees), so the windows client vm's can access their vmdk files to restore files. I should state that I'm a newbie to cifs on the netapps as we are a unix shop. But, I have the snapvault setup all working. Where I'm running into problems is accessing the directories in the shares when I map a drive in windows to the share.

I configured cifs to authenticate against:

(3) Windows Workgroup authentication using the filer's local user accounts

So I created a user 'vmrestore' though the useradmin tool and user vmrestore is part of a group cifsonly which has role none assigned to it. I then added an entry to /etc/passwd for the vmrestore user. At this point I can create a share to the vmware .snapshot directory on the snapvault destination and give the vmrestore user access to the share via:

filer> cifs access vmsnaps vmrestore rwx

I can now connect to the share fine but cannot access any of the shares folders as I get a "Access is denied" error. I've been reading the netapp docs and googling around for a while now but am having a difficult time understanding how I need to configure the local user account so I can access the folders/files on the unix security style volumes.

I've tried mapping the vmrestore user to the root user via the usermap.cfg file. Thinking that since the vm files are owned by root maybe the vmrestore user has to be mapped to a user with a uid of 0 to have access to them. I've also tried the --forcegroup option on the share.

Basically, I'm not having any luck and wanted to see if anyone has run into this (or just knows what I'm doing wrong).

Any help is very much appreciated.

Thanks,

--
Romeo Theriault
System Administrator
Information Technology Services

--
Romeo Theriault
System Administrator
Information Technology Services

Re: Controller upgrade - ESX LUN re-signaturing

2009-10-30T03:09:26Z

I wrote a simple little perl script with automates the process of changing the serial numbers on the luns back to their pre-head upgrade serials, which you can find on the now site here: http://communities.netapp.com/docs/DOC-2369

There is also a vb script one here: http://communities.netapp.com/docs/DOC-1754

which I basically imitated when making my perl version.

Romeo

On Thu, Oct 29, 2009 at 12:58 PM, Raj Patel <phigmov@...> wrote:

Hi all,

We're planning to upgrade our controller (a 270c to a 2050ha) - my
understanding is the iSCSI LUN's will retain their identity (ie the
existing IQN strings will remain the same) however we will probably
run into the ESX re-signaturing issue.

Is there a way to avoid this as part of the controller upgrade or do I
need to go into each connected ESX host and change the value of
LVM.EnableResignature to 1 ?

Are there any other potential 'gotchas' about an in-place head
replacement ? The process itself looks fairly straightforward if all
the steps are followed through in sequence.

Thanks in advance,
Raj.

--
Romeo Theriault
System Administrator
Information Technology Services