hwpmc(4)

NAME

hwpmc - Hardware Performance Monitoring Counter support

SYNOPSIS

options HWPMC_HOOKS
device hwpmc
Additionally, for i386 systems:
device apic

DESCRIPTION

The hwpmc driver virtualizes the hardware performance moni

toring facilities in modern CPUs and provides support for using these fa

cilities from
user level processes.

The driver supports multi-processor systems.

PMCs are allocated using the PMC_OP_PMCALLOCATE request. A

successful
PMC_OP_PMCALLOCATE request will return an integer handle

(typically a
small integer) to the requesting process. Subsequent opera

tions on the
allocated PMC use this handle to denote the specific PMC. A

process that
has successfully allocated a PMC is termed an ``owner pro

cess''.

PMCs may be allocated to operate in process-private or in

system-wide
modes.

Process-private In process-private mode, a PMC is ac

tive only when a

thread belonging to a process it is at

tached to is
scheduled on a CPU.

System-wide In system-wide mode, a PMC operates in

dependently of

processes and measures hardware events

for the system as a whole.

The hwpmc driver supports the use of hardware PMCs for

counting or for
sampling:

Counting In counting modes, the PMCs count hardware

events. These

counts are retrievable using the PMC_OP_PM

CREAD system call
on all architectures, though some architec

tures like the
i386 and amd64 offer faster methods of reading

these
counts.

Sampling In sampling modes, where PMCs are configured

to sample the

CPU instruction pointer after a configurable

number of
hardware events have been observed. These in

struction
pointer samples are directed to a log file for

subsequent
analysis.

These modes of operation are orthogonal; a PMC may be con

figured to operate in one of four modes:

Process-private, counting

These PMCs count hardware events whenever a thread

in their
attached process is scheduled on a CPU. These PMCs

normally
count from zero, but the initial count may be set

using the
PMC_OP_SETCOUNT operation. Applications can read

the value of
the PMC anytime using the PMC_OP_PMCRW operation.

Process-private, sampling

These PMCs sample the target processes instruction

pointer after
they have seen the configured number of hardware

events. The
PMCs only count events when a thread belonging to

their attached
process is active. The desired frequency of sam

pling is set
using the PMC_OP_SETCOUNT operation prior to start

ing the PMC.
Log files are configured using the PMC_OP_CONFIG

URELOG operation.

System-wide, counting

These PMCs count hardware events seen by them inde

pendent of the
processes that are executing. The current count on

these PMCs
can be read using the PMC_OP_PMCRW request. These

PMCs normally
count from zero, but the initial count may be set

using the
PMC_OP_SETCOUNT operation.

System-wide, sampling

These PMCs will periodically sample the instruction

pointer of
the CPU they are allocated on, and will write the

sample to a log
for further processing. The desired frequency of

sampling is set
using the PMC_OP_SETCOUNT operation prior to start

ing the PMC.
Log files are configured using the PMC_OP_CONFIG

URELOG operation.

System-wide statistical sampling can only be enabled

by a process
with super-user privileges.

Processes are allowed to allocate as many PMCs are the hard

ware and current operating conditions permit. Processes may mix alloca

tions of system-wide and process-private PMCs. Multiple processes are

allowed to be
concurrently using the facilities of the hwpmc driver.

Allocated PMCs are started using the PMC_OP_PMCSTART opera

tion, and
stopped using the PMC_OP_PMCSTOP operation. Stopping and

starting a PMC
is permitted at any time the owner process has a valid han

dle to the PMC.

Process-private PMCs need to be attached to a target process

before they
can be used. Attaching a process to a PMC is done using the
PMC_OP_PMCATTACH operation. An already attached PMC may be

detached from
its target process using the converse PMC_OP_PMCDETACH oper

ation. Issuing a PMC_OP_PMCSTART operation on an as yet unattached PMC

will cause it
to be attached to its owner process. The following rules

determine
whether a given process may attach a PMC to another target

process:
+o A non-jailed process with super-user privileges is al

lowed to attach

to any other process in the system.

+o Other processes are only allowed to attach to targets

that they would

be able to attach to for debugging (as determined by

p_candebug(9)).

PMCs are released using PMC_OP_PMCRELEASE. After a success

ful
PMC_OP_PMCRELEASE operation the handle to the PMC will be

come invalid.

Modifier Flags

The PMC_OP_PMCALLOCATE operation supports the following

flags that modify
the behavior of an allocated PMC:

PMC_F_DESCENDANTS

This modifier is valid only for a PMC being allocat

ed in processprivate mode. It signifies that the PMC will track

hardware
events for its target process and the target's cur

rent and future
descendants.

PMC_F_KGMON

This modifier is valid only for a PMC being allocat

ed in systemwide sampling mode. It signifies that the PMC's

sampling interrupt is to be used to drive kernel profiling via kg

mon(8).

PMC_F_LOG_PROCCSW

This modifier is valid only for a PMC being allocat

ed in processprivate mode. When this modifier is present, at ev

ery context
switch, hwpmc will log a record containing the num

ber of hardware
events seen by the target process when it was sched

uled on the
CPU.

PMC_F_LOG_PROCEXIT

This modifier is valid only for a PMC being allocat

ed in processprivate mode. With this modifier present, hwpmc

will maintain
per-process counts for each target process attached

to a PMC. At
process exit time, a record containing the target

process' PID
and the accumulated per-process count for that pro

cess will be
written to the configured log file.

Modifiers PMC_F_LOG_PROCEXIT and PMC_F_LOG_PROCCSW may be

used in combination with modifier PMC_F_DESCENDANTS to track the be

haviour of complex
pipelines of processes. PMCs with modifiers PMC_F_LOG_PRO

CEXIT and
PMC_F_LOG_PROCCSW cannot be started until their owner pro

cess has configured a log file.

Signals

The hwpmc driver may deliver signals to processes that have

allocated
PMCs:

SIGIO A PMC_OP_PMCRW operation was attempted on a pro

cess-private

PMC that does not have attached target process

es.

SIGBUS The hwpmc driver is being unloaded from the ker

nel.

PROGRAMMING API

The recommended way for application programs to use the fa

cilities of the
hwpmc driver is using the API provided by the pmc(3) li

brary.

The hwpmc driver operates using a system call number that is

dynamically
allotted to it when it is loaded into the kernel.

The hwpmc driver supports the following operations:

PMC_OP_CONFIGURELOG

Configure a log file for sampling mode PMCs.

PMC_OP_FLUSHLOG

Transfer buffered log data inside hwpmc to a config

ured output
file. This operation returns to the caller after

the write operation has returned.

PMC_OP_GETCPUINFO

Retrieve information about the number of CPUs on the

system and
the number of hardware performance monitoring coun

ters available
per-CPU.

PMC_OP_GETDRIVERSTATS

Retrieve module statistics (for analyzing the behav

ior of hwpmc
itself).

PMC_OP_GETMODULEVERSION

Retrieve the version number of API.

PMC_OP_GETPMCINFO

Retrieve information about the current state of the

PMCs on a
given CPU.

PMC_OP_PMCADMIN

Set the administrative state (i.e., whether enabled

or disabled)
for the hardware PMCs managed by the hwpmc driver.

PMC_OP_PMCALLOCATE

Allocate and configure a PMC. On successful alloca

tion, a handle
to the PMC (a small integer) is returned.

PMC_OP_PMCATTACH

Attach a process mode PMC to a target process. The

PMC will be
active whenever a thread in the target process is

scheduled on a
CPU.

If the PMC_F_DESCENDANTS flag had been specified at

PMC allocation time, then the PMC is attached to all current

and future
descendants of the target process.

PMC_OP_PMCDETACH

Detach a PMC from its target process.

PMC_OP_PMCRELEASE

Release a PMC.

PMC_OP_PMCRW

Read and write a PMC. This operation is valid only

for PMCs configured in counting modes.

PMC_OP_SETCOUNT

Set the initial count (for counting mode PMCs) or

the desired
sampling rate (for sampling mode PMCs).

PMC_OP_PMCSTART

Start a PMC.

PMC_OP_PMCSTOP

Stop a PMC.

PMC_OP_WRITELOG

Insert a timestamped user record into the log file.

i386 Specific API

Some i386 family CPUs support the RDPMC instruction which

allows a user
process to read a PMC value without needing to invoke a

PMC_OP_PMCRW
operation. On such CPUs, the machine address associated

with an allocated PMC is retrievable using the PMC_OP_PMCX86GETMSR sys

tem call.

PMC_OP_PMCX86GETMSR

Retrieve the MSR (machine specific register) number

associated
with the given PMC handle.

The PMC needs to be in process-private mode and al

located without
the PMC_F_DESCENDANTS modifier flag, and should be

attached only
to its owner process at the time of the call.

amd64 Specific API

AMD64 CPUs support the RDPMC instruction which allows a user

process to
read a PMC value without needing to invoke a PMC_OP_PMCRW

operation. The
machine address associated with an allocated PMC is retriev

able using the
PMC_OP_PMCX86GETMSR system call.

PMC_OP_PMCX86GETMSR

Retrieve the MSR (machine specific register) number

associated
with the given PMC handle.

The PMC needs to be in process-private mode and al

located without
the PMC_F_DESCENDANTS modifier flag, and should be

attached only
to its owner process at the time of the call.

SYSCTL VARIABLES AND LOADER TUNABLES

The behavior of hwpmc is influenced by the following

sysctl(8) and
loader(8) tunables:

kern.hwpmc.debugflags (string, read-write)

(Only available if the hwpmc driver was compiled

with -DDEBUG.)
Control the verbosity of debug messages from the

hwpmc driver.

kern.hwpmc.hashsize (integer, read-only)

The number of rows in the hash tables used to keep

track of owner
and target processes. The default is 16.

kern.hwpmc.logbuffersize (integer, read-only)

The size in kilobytes of each log buffer used by

hwpmc's logging
function. The default buffer size is 4KB.

kern.hwpmc.mtxpoolsize (integer, read-only)

The size of the spin mutex pool used by the PMC

driver. The
default is 32.

kern.hwpmc.nbuffers (integer, read-only)

The number of log buffers used by hwpmc for logging.

The default
is 16.

kern.hwpmc.nsamples (integer, read-only)

The number of entries in the per-CPU ring buffer

used during sampling. The default is 16.

security.bsd.unprivilegeds_yspmcs (boolean, read-write)

If set to non-zero, allow unprivileged processes to

allocate system-wide PMCs. The default value is 0.

security.bsd.unprivilegedp_rocd_ebug (boolean, read-write)

If set to 0, the hwpmc driver will only allow privi

leged processes to attach PMCs to other processes.

These variables may be set in the kernel environment using

kenv(1) before
hwpmc is loaded.

SECURITY CONSIDERATIONS

PMCs may be used to monitor the actual behaviour of the sys

tem on hardware. In situations where this constitutes an undesirable

information
leak, the following options are available:

1. Set the sysctl(8) tunable

security.bsd.unprivilegeds_yspmcs to 0.This ensures that unprivileged processes cannot allo

cate system-wide
PMCs and thus cannot observe the hardware behavior of

the system as
a whole. This tunable may also be set at boot time us

ing loader(8),
or with kenv(1) prior to loading the hwpmc driver into

the kernel.

2. Set the sysctl(8) tunable

security.bsd.unprivilegedp_rocd_ebug to 0. This will ensure that an unprivileged process cannot

attach a PMC to
any process other than itself and thus cannot observe

the hardware
behavior of other processes with the same credentials.

System administrators should note that on IA-32 platforms

FreeBSD makes
the content of the IA-32 TSC counter available to all pro

cesses via the
RDTSC instruction.

IMPLEMENTATION NOTES

SMP Symmetry

The kernel driver requires all physical CPUs in an SMP sys

tem to have
identical performance monitoring counter hardware.

x86 TSC Handling
Historically, on the x86 architecture, FreeBSD has permitted

user processes running at a processor CPL of 3 to read the TSC using

the RDTSC
instruction. The hwpmc driver preserves this semantics.

Intel P4/HTT HandlingOn CPUs with HTT support, Intel P4 PMCs are capable of qual

ifying only a
subset of hardware events on a per-logical CPU basis. Con

sequently, if
HTT is enabled on a system with Intel Pentium P4 PMCs, then

the hwpmc
driver will reject allocation requests for process-private

PMCs that
request counting of hardware events that cannot be counted

separately for
each logical CPU.

Intel Pentium-Pro Handling Writing a value to the PMC MSRs found ing Intel Pentium-Pro

style PMCs
(found in Intel Pentium Pro, Pentium II, Pentium III, Pen

tium M and
Celeron processors) will replicate bit 31 of the value being

written into
the upper 8 bits of the MSR, bringing down the usable width

of these PMCs
to 31 bits. For process-virtual PMCs, the hwpmc driver im

plements a
workaround in software and makes the corrected 64 bit count

available via
the PMC_OP_RW operation. Processes that intend to use RDPMC

instructions
directly or that intend to write values larger than 2^31 in

to these PMCs
with PMC_OP_RW need to be aware of this hardware limitation.

DIAGNOSTICS

hwpmc: [class/npmc/capabilities]... Announce the presence

of npmc PMCs
of class class, with capabilities described by bit string

capabilities.

hwpmc: kernel version (0x%x) does not match module version

(0x%x). The
module loading process failed because a version mismatch was

detected
between the currently executing kernel and the module being

loaded.

hwpmc: this kernel has not been compiled with 'options

HWPMC_HOOKS'.

module loading process failed because the currently execut

ing kernel was
not configured with the required configuration option

HWPMC_HOOKS

hwpmc: tunable hashsize=%d must be greater than zero. A

negative value
was supplied for tunable kern.hwpmc.hashsize.

hwpmc: tunable logbuffersize=%d must be greater than zero.

A negative
value was supplied for tunable kern.hwpmc.logbuffersize.

hwpmc: tunable nlogbuffers=%d must be greater than zero. A

negative
value was supplied for tunable kern.hwpmc.nlogbuffers.

hwpmc: tunable nsamples=%d out of range. The value for tun

able
kern.hwpmc.nsamples was negative or greater than 65535.

COMPATIBILITY

The hwpmc driver is currently under development. The API

and ABI documented in this manual page may change in the future. The

recommended
method of accessing this driver is using the pmc(3) API.

ERRORS

A command issued to the hwpmc driver may fail with the fol

lowing errors:

[EBUSY] A PMC_OP_CONFIGURELOG operation was re

quested while an
existing log was active.

[EBUSY] A DISABLE operation was requested using

the
PMC_OP_PMCADMIN request for a set of

hardware
resources currently in use for process

private PMCs.

[EBUSY] A PMC_OP_PMCADMIN operation was requested

on an active
system mode PMC.

[EBUSY] A PMC_OP_PMCATTACH operation was request

ed for a tar
get process that already had another PMC

using the
same hardware resources attached to it.

[EBUSY] A PMC_OP_PMCRW request writing a new val

ue was issued
on a PMC that was active.

[EBUSY] A PMC_OP_PMCSETCOUNT request was issued

on a PMC that
was active.

[EDOOFUS] A PMC_OP_PMCSTART operation was requested

without alog file being configured for a PMC allo

cated with
PMC_F_LOG_PROCCSW and PMC_F_LOG_PROCEXIT

modifiers.

[EEXIST] A PMC_OP_PMCATTACH request was reissued

for a target
process that already is the target of

this PMC.

[EFAULT] A bad address was passed in to the driv

er.

[EINVAL] A process specified an invalid PMC han

dle.

[EINVAL] An invalid CPU number was passed in for a
PMC_OP_GETPMCINFO operation.

[EINVAL] An invalid CPU number was passed in for a
PMC_OP_PMCADMIN operation.

[EINVAL] An invalid operation request was passed

in for a
PMC_OP_PMCADMIN operation.

[EINVAL] An invalid PMC ID was passed in for a

PMC_OP_PMCADMIN
operation.

[EINVAL] A suitable PMC matching the parameters

passed in to a
PMC_OP_PMCALLOCATE request could not be

allocated.

[EINVAL] An invalid PMC mode was requested during

a
PMC_OP_PMCALLOCATE request.

[EINVAL] An invalid CPU number was specified dur

ing a
PMC_OP_PMCALLOCATE request.

[EINVAL] A CPU other than PMC_CPU_ANY was speci

fied in aPMC_OP_ALLOCATE request for a process

private PMC.

[EINVAL] A CPU number of PMC_CPU_ANY was specified

in a
PMC_OP_ALLOCATE request for a system-wide

PMC.

[EINVAL] The pmf_lags argument to an PMC_OP_PMCALLO

CATE request
contained unknown flags.

[EINVAL] A PMC allocated for system-wide operation

was speci
fied with a PMC_OP_PMCATTACH request.

[EINVAL] The pmp_id argument to a PMC_OP_PMCATTACH

request
specified an illegal process ID.

[EINVAL] A PMC_OP_PMCDETACH request was issued for

a PMC not
attached to the target process.

[EINVAL] Argument pmf_lags to a PMC_OP_PMCRW request

contained
illegal flags.

[EINVAL] A PMC_OP_PMCX86GETMSR operation was re

quested for a
PMC not in process-virtual mode, or for a

PMC that is
not solely attached to its owner process,

or for a PMC
that was allocated with flag PMC_F_DE

SCENDANTS.

[EINVAL] (On Intel Pentium 4 CPUs with HTT sup

port) An alloca
tion request for a process-private PMC

was issued for
an event that does not support counting

on a per-logical CPU basis.

[ENOMEM] The system was not able to allocate ker

nel memory.

[ENOSYS] (i386 architectures) A PMC_OP_PMCX86GETM

SR operation
was requested for hardware that does not

support reading PMCs directly with the RDPMC instruc

tion.

[ENXIO] A PMC_OP_GETPMCINFO operation was re

quested for a dis
abled CPU.

[ENXIO] A system-wide PMC on a disabled CPU was

requested to
be allocated with PMC_OP_PMCALLOCATE.

[ENXIO] A PMC_OP_PMCSTART or PMC_OP_PMCSTOP re

quest was issued
for a system-wide PMC that was allocated

on a currently disabled CPU.

[EOPNOTSUPP] A PMC_OP_PMCALLOCATE request was issued

for PMC capa
bilities not supported by the specified

PMC class.

[EPERM] A PMC_OP_PMCADMIN request was issued by a

process
without super-user privilege or by a

jailed super-user
process.

[EPERM] A PMC_OP_PMCATTACH operation was issued

for a target
process that the current process does not

have permission to attach to.

[EPERM] (i386 and amd64 architectures) A

PMC_OP_PMCATTACH
operation was issued on a PMC whose MSR

has been
retrieved using PMC_OP_PMCX86GETMSR.

[ESRCH] A process issued a PMC operation request

without hav
ing allocated any PMCs.

[ESRCH] A process issued a PMC operation request

after the PMCwas detached from all of its target pro

cesses.

[ESRCH] A PMC_OP_PMCATTACH request specified a

non-existent
process ID.

[ESRCH] The target process for a PMC_OP_PMCDETACH

operation is
not being monitored by the hwpmc driver.

SEE ALSO

kenv(1), pmc(3), pmclog(3), kgmon(8), kldload(8), pmccon

trol(8),
pmcstat(8), sysctl(8), p_candebug(9)

HISTORY

The hwpmc driver first appeared in FreeBSD 6.0.

BUGS

The driver samples the state of the kernel's logical proces

sor support at
the time of initialization (i.e., at module load time). On

CPUs supporting logical processors, the driver could misbehave if logi

cal processors
are subsequently enabled or disabled while the driver is ac

tive.

On the i386 architecture, the driver requires that the local

APIC on the
CPU be enabled for sampling mode to be supported. Many sin

gle-processor
motherboards keep the APIC disabled in BIOS; on such systems

hwpmc will
not support sampling PMCs.

BSD September 28, 2005

BSD