High CPU utilizations and file I/O

[Jedan58]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Hi all,

 

From some days the CPU of a 3020C reaches 100% slowing down all processes.

Maybe some area of a 3020HA is stressed on I/O.

 

There’s a VMware on NFS farm using both servers and VDI also! The customer states that in the NFS volume for VMware there’s an high activity of continous destruction and building of VMware vdisk…

 

 

Is there some tool that can let me understand which volume is under stress and in case move it on the other head?

Or maybe this filer is undersized for the job that has been asked…

 

Regards,

 

Dott. Giacomo Milazzo

Technical Account Manager

Sinergy SpA
Filiale di Roma
+   00148. viale Castello della Magliana, 38

'   ([phone]  0665970252

7    [phone]
- Giacomo.Milazzo@...

 

 

[Jedan58]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Thanks for info, for now.

I’ve discovered a tool I didn’t know. LogicMonitor seems really a good tool.

 

Together with Operation Manager I remember that also DFM could do the job. Right? So, nothing free (execpt CLI and lot of values to read on the putty log!! J)

 

Is there somebody that remember if the new System Manager can help? I remember that there’s some ‘stats&perf” area…

 

Regards,

 

Da: Steve Francis [mailto:sfrancis@...]
Inviato: giovedì 1 ottobre 2009 17.50
A: Milazzo Giacomo
Cc: toasters@...
Oggetto: Re: High CPU utilizations and file I/O

 

Not sure if by 'tool' you meant commercial product, but LogicMonitor certainly does that.
It will discover all the volumes, and which aggregate they are on, and construct graphs for each aggregate showing IO operations of each volume (as well as detailed volume graphs showing read, write and other operations/sec, and throughput, and latency values per volume.)
Plus a lot of other NetApp alerting and monitoring (space/inode per volume, shelf electronics, spare drives, autosupport success, individual CPU stats, physical disk utilization, etc).

Let me know if you want more information.


Milazzo Giacomo wrote:

Hi all,

 

From some days the CPU of a 3020C reaches 100% slowing down all processes.

Maybe some area of a 3020HA is stressed on I/O.

 

There’s a VMware on NFS farm using both servers and VDI also! The customer states that in the NFS volume for VMware there’s an high activity of continous destruction and building of VMware vdisk…

 

 

Is there some tool that can let me understand which volume is under stress and in case move it on the other head?

Or maybe this filer is undersized for the job that has been asked…

 

Regards,

 

Dott. Giacomo Milazzo

Technical Account Manager

Sinergy SpA
Filiale di Roma
+   00148. viale Castello della Magliana, 38

'   ([phone]  0665970252

7    [phone]
- Giacomo.Milazzo@...

 

 

 

--

Steve Francis		LogicMonitor LLC
sfrancis@... www.logicmonitor.com		Ph: 1 888 41 LOGIC x500 Ph: 1 805 698 0770

 

[Page, Jeremy]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Show stats will let you look at per volume IO.

stats show -i 15 -e volume:(volumes):(nfs)

 

Replace volumes with something that matches all the volume names, -e means it’s looked at as a regular expression.

 

Jeremy Page____________________            

Systems Architect

* email:Jeremy.Page@... - ( phone: [phone] - 6 fax: [phone] - È cell: 336.601.7274

---

From: owner-toasters@... [mailto:owner-toasters@...] On Behalf Of Milazzo Giacomo
Sent: Thursday, October 01, 2009 11:28 AM
To: toasters@...
Subject: High CPU utilizations and file I/O

 

Hi all,

 

From some days the CPU of a 3020C reaches 100% slowing down all processes.

Maybe some area of a 3020HA is stressed on I/O.

 

There’s a VMware on NFS farm using both servers and VDI also! The customer states that in the NFS volume for VMware there’s an high activity of continous destruction and building of VMware vdisk…

 

 

Is there some tool that can let me understand which volume is under stress and in case move it on the other head?

Or maybe this filer is undersized for the job that has been asked…

 

Regards,

 

Dott. Giacomo Milazzo

Technical Account Manager

Sinergy SpA
Filiale di Roma
+   00148. viale Castello della Magliana, 38

'   ([phone]  0665970252

7    [phone]
- Giacomo.Milazzo@...

 

 

Please be advised that this email may contain confidential information.
If you are not the intended recipient, please do not read, copy or
re-transmit this email. If you have received this email in error,
please notify us by email by replying to the sender and by telephone
(call us collect at [phone]) and delete this message and any
attachments. Thank you in advance for your cooperation and assistance.

In addition, Danaher and its subsidiaries disclaim that the content of
this email constitutes an offer to enter into, or the acceptance of,
any
contract or agreement or any amendment thereto; provided that the
foregoing disclaimer does not invalidate the binding effect of any
digital or other electronic reproduction of a manual signature that is
included in any attachment to this email.

[Jedan58]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Good question!

I’ll investigate

 

 

Da: Jeff Mohler [mailto:speedtoys.racing@...]
Inviato: giovedì 1 ottobre 2009 19.04
A: Milazzo Giacomo
Cc: toasters@...
Oggetto: Re: High CPU utilizations and file I/O

 

Let me ask one question.

Are the VMDK's aligned?


If Yes, continue with volume stats.

If No, stop everything, there's your problem, no way around it.

On Thu, Oct 1, 2009 at 8:28 AM, Milazzo Giacomo <G.Milazzo@...> wrote:

Hi all,

 

From some days the CPU of a 3020C reaches 100% slowing down all processes.

Maybe some area of a 3020HA is stressed on I/O.

 

There’s a VMware on NFS farm using both servers and VDI also! The customer states that in the NFS volume for VMware there’s an high activity of continous destruction and building of VMware vdisk…

 

 

Is there some tool that can let me understand which volume is under stress and in case move it on the other head?

Or maybe this filer is undersized for the job that has been asked…

 

Regards,

 

Dott. Giacomo Milazzo

Technical Account Manager

Sinergy SpA
Filiale di Roma
+   00148. viale Castello della Magliana, 38

'   ([phone]  0665970252

7    [phone]
- Giacomo.Milazzo@...

 

 

--
No Signature Required
Save The Bits, Save The World!

[romeotheriault]

Is there some tool that can let me understand which volume is under stress and in case move it on the other head?

I wrote a tool called "topvol" a while ago that does just this. You can find it here:

http://communities.netapp.com/docs/DOC-1262

I also have an updated version that allows you to filter on aggregate too, if you'd be interested.

--
Romeo Theriault
System Administrator
Information Technology Services

[Maxwell Reid]

Hi Romeo,

I think everyone would be interested in the updated version!

~Max

On Oct 1, 2009, at 8:54 PM, Romeo Theriault wrote:

Is there some tool that can let me understand which volume is under stress and in case move it on the other head?

I wrote a tool called "topvol" a while ago that does just this. You can find it here:

http://communities.netapp.com/docs/DOC-1262

I also have an updated version that allows you to filter on aggregate too, if you'd be interested.

--
Romeo Theriault
System Administrator
Information Technology Services

[Jedan58]

Some parts of this message have been removed. Learn more about Nabble's security policy.

It’s a great thing to see that a “simple” question on a common issue could push so many answers!

Great community this one!

 

Thanks,

 

Da: Maxwell Reid [mailto:max.reid@...]
Inviato: venerdì 2 ottobre 2009 7.17
A: Romeo Theriault
Cc: Milazzo Giacomo; toasters@...
Oggetto: Re: High CPU utilizations and file I/O

 

 

 

Hi Romeo,

 

I think everyone would be interested in the updated version!

 

~Max

 

 

 

On Oct 1, 2009, at 8:54 PM, Romeo Theriault wrote:

Is there some tool that can let me understand which volume is under stress and in case move it on the other head?

 

I wrote a tool called "topvol" a while ago that does just this. You can find it here:

http://communities.netapp.com/docs/DOC-1262

I also have an updated version that allows you to filter on aggregate too, if you'd be interested.

--
Romeo Theriault
System Administrator
Information Technology Services

 

[romeotheriault]

I think everyone would be interested in the updated version!

Great! Ok, I'll put up the version that can also filter on aggregates on Monday. I hadn't put it up earlier mainly because it takes a bit more setup to get working. I used the netapp perl api to access the aggregate information so it also requires a user account with the appropriate api permissions and the associated netapp perl api modules. I just haven't gotten around to figuring out if I can legally bundle the netapp perl modules and just generally putting it in a easily usable package for the general public. I'll try to figure out the details on monday and see what I can get out, with or without the netapp modules.

--
Romeo Theriault
System Administrator
Information Technology Services

[Raj Patel-2]

As a matter of interest how much latency is to much (I know, how many
angels can you fit on a pinhead) ?

We've had a few DBA types come and comment on long SQL queue lengths
associated with disk i/o - the SQL luns are connected via iSCSI.

As far as the end user is concerned performance seems fine for things
like SharePoint and CRM. Even Exchange. However a few apps with high
i/o (mail archiving to database and SCOM which seems to thrash the
database) do seem to suffer a little in terms of slow response time.

Do people have tips for optimising iSCSI performance (targets are ESX
servers, Exchange on physical hardware and SQL on a mix of physical
and virtual servers) ?

What key counters should I keep an eye on ?

Fibre Channel is still prohibitively expensive plus 10GbE seems very
promising and more pervasive than a few years ago.

Cheers,
Raj.

On 10/3/09, Romeo Theriault <romeotheriault@...> wrote:

>>
>> I think everyone would be interested in the updated version!
>>
>
> Great! Ok, I'll put up the version that can also filter on aggregates on
> Monday. I hadn't put it up earlier mainly because it takes a bit more setup
> to get working. I used the netapp perl api to access the aggregate
> information so it also requires a user account with the appropriate api
> permissions and the associated netapp perl api modules. I just haven't
> gotten around to figuring out if I can legally bundle the netapp perl
> modules and just generally putting it in a easily usable package for the
> general public. I'll try to figure out the details on monday and see what I
> can get out, with or without the netapp modules.
>
> --
> Romeo Theriault
> System Administrator
> Information Technology Services
>

[romeotheriault]

I think everyone would be interested in the updated version!

Ok, I've put up the newer version of topvol which allows you to optionally filter the volumes based on aggregates. Here is an example of using topvol without aggregate filtering:

./topvol filername 4 5 ops

and a example of filtering on an aggregate called aggrsata750. This will only show you the volumes that are in aggrsata750.

./topvol filername:aggrsata750 4 5 ops

You can find the download here:

http://communities.netapp.com/docs/DOC-1262

Please let me know if you have any questions or comments.

Enjoy,

--
Romeo Theriault
System Administrator
Information Technology Services

[Raj Patel-2]

Thanks for all the advice - thanks to Romeo Theriaults awesome tool
I'll run some stats over time and see how things look.

I'll probably give Logicmonitor a quick look and see how it compares to DFM.

Cheers,
Raj.


On 10/7/09, Kevin M. Parker <kevin@...> wrote:

> Raj,
> I'll share knowledge I've received specific to NFS environments (and
> this may be specific to VMware over NFS - sorry I can't confirm). The
> way I understand it is that clients start experiencing negative
> performance after 20ms disk latency. I do not know if that same latency
> applies to iSCSI traffic.
>
> As it relates to the disks making up your aggregates, I can offer the
> following numbers as they relate to disk IOPS, which is also somewhat
> related to the original request in this thread. Numbers below are the
> IOPS that disk can sustain before experiencing a >= 20ms delay.
> FC@10KRPM: 120IOPS
> FC@15KRPM: 220IOPS
> SATA: 40IOPS
>
> I had looked on NOW for this type info but did not come up w/anything
> quickly - perhaps you could reach out to your SE.
>
> -/-
> Kevin Parker
> 919.521.8413
> http://theparkerz.com
> Blackle.com - Saving energy one search at a time.
>
>
> -----Original Message-----
> From: owner-toasters@... [mailto:owner-toasters@...]
> On Behalf Of Raj Patel
> Sent: Monday, October 05, 2009 12:59 AM
> To: Romeo Theriault
> Cc: Maxwell Reid; vera.lee@...; Milazzo Giacomo;
> toasters@...
> Subject: Re: High CPU utilizations and file I/O
>
> As a matter of interest how much latency is to much (I know, how many
> angels can you fit on a pinhead) ?
>
> We've had a few DBA types come and comment on long SQL queue lengths
> associated with disk i/o - the SQL luns are connected via iSCSI.
>
> As far as the end user is concerned performance seems fine for things
> like SharePoint and CRM. Even Exchange. However a few apps with high
> i/o (mail archiving to database and SCOM which seems to thrash the
> database) do seem to suffer a little in terms of slow response time.
>
> Do people have tips for optimising iSCSI performance (targets are ESX
> servers, Exchange on physical hardware and SQL on a mix of physical
> and virtual servers) ?
>
> What key counters should I keep an eye on ?
>
> Fibre Channel is still prohibitively expensive plus 10GbE seems very
> promising and more pervasive than a few years ago.
>
> Cheers,
> Raj.
>
> On 10/3/09, Romeo Theriault <romeotheriault@...> wrote:
>>>
>>> I think everyone would be interested in the updated version!
>>>
>>
>> Great! Ok, I'll put up the version that can also filter on aggregates
> on
>> Monday. I hadn't put it up earlier mainly because it takes a bit more
> setup
>> to get working. I used the netapp perl api to access the aggregate
>> information so it also requires a user account with the appropriate
> api
>> permissions and the associated netapp perl api modules. I just haven't
>> gotten around to figuring out if I can legally bundle the netapp perl
>> modules and just generally putting it in a easily usable package for
> the
>> general public. I'll try to figure out the details on monday and see
> what I
>> can get out, with or without the netapp modules.
>>
>> --
>> Romeo Theriault
>> System Administrator
>> Information Technology Services
>>
>