Christian Bilien’s Oracle performance and tuning blog

March 5, 2007

HP-UX Processor Load Balancing on SMPs (2/2)

Filed under: HP-UX — christianbilien @ 3:18 pm

The first post I wrote on CPU scheduling was describing the circumstances in which thread stealing would be considered. This post will go further down the road: once there is enough cpu idleness, or all CPUs are starving threads, context switches may occur. This post describes the rules enforced by the HP-UX scheduler in versions 11.11 and later.

Before digging into the subject, locality domains (LDOM) should be explained. A locality domain is basically a cell. A cell is made of four single or dual core processors, as well as its own memory. The reason locality domain exist is the inter cell bus latency which may greatly impact memory access time. I’ll write a post one day about HP-UX partitioning which will go a bit more into details.

  • A mundane_balance() iteration is run within each LDOM. Each processor is assigned a score based on load average AND starvation (remember from post 1 that starvation occurs when a thread assigned to a given processor has not been running for ‘a long time’). According to the HP system internals course, starvation is given more importance than load average, which makes sense as a cpu hog will be able to run 80 to 100ms before giving up the processor to another thread. In any case, an idle processor is always one of the best processors.
  • In 11.22, the locality domain balancer is called to potentially move a thread from one domain to another.

The outcome is a pair of “best” and “worst” processors. If the pair has lightly loaded cpus, with a load average of less than 0.2, the system is considered to be well balanced and nothing is done. A thread running on the “worst” processor is otherwise selected (it must not be a real time or locked thread), removed from the run queue and inserted into the “best” processor run queue.

How is this “next” thread selected?

The selection is based on the virtual address of the kthread structure: the purpose of the algorithm is to ensure each thread is cycled through.



  1. […] second post will consider Cell-local vs Interleaved […]

    Pingback by Memory partitioning strategy: avoiding design traps on high end HP-UX systems (1/2) « Christian Bilien’s Oracle performance and tuning blog — April 12, 2007 @ 7:49 pm

  2. […] A processor is in a state of starvation if it has one or more threads on its run queue that hasn’t executed for a long time (this “long time” varies with the CPU load). Only if there are no processors suffering from starvation, or all processors have starving threads (or could be forced into that condition), does HP-UX considers looking for balancing (see next post). […]

    Pingback by HP-UX Processor Load Balancing on SMPs (1/2) « Christian Bilien’s Oracle performance and tuning blog — May 24, 2007 @ 7:19 pm

  3. this is the advence knowledge about processor thing.. nice article dude…

    Comment by Athlon 64 — January 30, 2010 @ 1:01 pm

  4. Thx, i read most articles in hp-ux category, and i learned something new.

    Comment by Gasha — April 21, 2010 @ 2:15 pm

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

Blog at

%d bloggers like this: