Christian Bilien’s Oracle performance and tuning blog

March 23, 2007

Storage array bottlenecks

Filed under: Storage — christianbilien @ 8:39 pm

Even if the internal communications of a switch may be “oversubscribed” when the aggregate speed of the ports exceeds the internal bandwidth of the device, many switch devices (at least the director class) are “non-blocking”, meaning that all ports can operate at full speed simultaneously. I’ll write a post one day on SAN bottlenecks, but for now here is a view of the main hardware bottlenecks encountered in storage arrays:

Host ports:

The available bandwidth is 2 or 4Gb/s (200 or 400MB/s – FC frames are 10 bytes long -) per port. As load balancing software (Powerpath, MPXIO, DMP, etc.) are most of the times used both for redundancy and load balancing, I/Os coming from a host can take advantage of an aggregated bandwidth of two ports. However, reads can use only one path, but writes are duplicated, i.e. a host write ends up as one write on each host port.

Below is an example of a couple of host ports on an EMC DMX1000 (2Gb/s host ports).

fa4a0.giffa13a0.gif

Thanks to Powerpath, the load is well spread over the two ports. Both ports are running at about half of the bandwidth (but queuing theory shows that queues would start to be non negligible when the available bandwidth reaches 50%).

Array service processors

Depending on the hardware maker, service processors may be either bound Not all SPs are bound toto specific volumes, or may be accessed by any of the array SP. I wrote a blog entry sometimes ago about SP binding. Higher end arrays such as DMX and XP are not do not bind Luns to SPs, whilst Clariion and EVA do.

Back end controllers

Back end controllers are the access point for disk FC loops. Backend controllers also have a given throughput, usually limited anyway by the fact that at a given point in time, the dual ported disks within the FC-AL loop only allow 2 senders and 2 receivers. Below is a DMX1000 controller utilization rate, where almost all disks are running at a minimum of 60% of their available bandwidth, with 30 RAID 10 disks in each loop.

 controller1.gif

 

Disks

From a host standpoint, disks can sustain a much higher utilization rate when they are behind a cache than when they are accessed directly: remember than a disk running at a utilization rate of 50% will queue on average one I/O out of 2 (seen from the host, the I/O service time will be on average 50% higher than the disk service). It is not uncommon to measure disks utilization rates of nearly 100%. This will only becomes a problem when the array cache stops buffering the I/Os because space is exhausted.

 disk1.gif

 

 

3 Comments »

  1. I’ve never believed in usefulness of load-balancing MPIO.

    >However, reads can use only one path, but writes are duplicated, i.e. a host write ends up as one write on each host port.

    (from the linked post):

    >Because write I/Os always involves both controller for cache mirroring, write performance will be the same regardless of which controller receives the I/O.

    Nicely said.
    I usually use these two arguments to persuade customers *not* to use MPIO for load balancing. Sadly, many customers believe it’s a very useful feature.

    Comment by sean — May 5, 2007 @ 1:10 pm

  2. Hi Sean,

    Thank you for your comment.

    Christian

    Comment by christianbilien — May 8, 2007 @ 7:53 am

  3. hi christian,

    it would be nice if you also include how
    you get those statistics from your graph..

    Comment by Karl Arao — January 9, 2009 @ 11:20 am


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: