geom(4)

NAME

GEOM - modular disk I/O request transformation framework.

DESCRIPTION

The GEOM framework provides an infrastructure in which

"classes" can perform transformations on disk I/O requests on their path from

the upper
kernel to the device drivers and back.

Transformations in a GEOM context range from the simple geo

metric displacement performed in typical disk partitioning modules

over RAID algorithms and device multipath resolution to full blown crypto

graphic protection of the stored data.

Compared to traditional "volume management", GEOM differs

from most and
in some cases all previous implementations in the following

ways:

+o GEOM is extensible. It is trivially simple to write a

new class of

transformation and it will not be given stepchild treat

ment. If
someone for some reason wanted to mount IBM MVS

diskpacks, a class
recognizing and configuring their VTOC information would

be a trivial
matter.

+o GEOM is topologically agnostic. Most volume management

implementa

tions have very strict notions of how classes can fit

together, very
often one fixed hierarchy is provided for instance sub

disk - plex volume.

Being extensible means that new transformations are treated

no differently than existing transformations.

Fixed hierarchies are bad because they make it impossible to

express the
intent efficiently. In the fixed hierarchy above it is not

possible to
mirror two physical disks and then partition the mirror into

subdisks,
instead one is forced to make subdisks on the physical vol

umes and to
mirror these two and two resulting in a much more complex

configuration.
GEOM on the other hand does not care in which order things

are done, the
only restriction is that cycles in the graph will not be al

lowed.

TERMINOLOGY and TOPOLOGY

GEOM is quite object oriented and consequently the terminol

ogy borrows a
lot of context and semantics from the OO vocabulary:

A "class", represented by the data structure g_class imple

ments one particular kind of transformation. Typical examples are MBR

disk partition,
BSD disklabel, and RAID5 classes.

An instance of a class is called a "geom" and represented by

the data
structure "g_geom". In a typical i386 FreeBSD system, there

will be one
geom of class MBR for each disk.

A "provider", represented by the data structure

"g_provider", is the
front gate at which a geom offers service. A provider is "a

disk-like
thing which appears in /dev" - a logical disk in other

words. All
providers have three main properties: name, sectorsize and

size.

A "consumer" is the backdoor through which a geom connects

to another
geom provider and through which I/O requests are sent.

The topological relationship between these entities are as

follows:

+o A class has zero or more geom instances.

+o A geom has exactly one class it is derived from.

+o A geom has zero or more consumers.

+o A geom has zero or more providers.

+o A consumer can be attached to zero or one providers.

+o A provider can have zero or more consumers attached.

All geoms have a rank-number assigned, which is used to de

tect and prevent loops in the acyclic directed graph. This rank number

is assigned
as follows:

1. A geom with no attached consumers has rank=1

2. A geom with attached consumers has a rank one higher

than the high
est rank of the geoms of the providers its consumers

are attached
to.

SPECIAL TOPOLOGICAL MANEUVERS

In addition to the straightforward attach, which attaches a

consumer to a
provider, and detach, which breaks the bond, a number of

special topological maneuvers exists to facilitate configuration and to im

prove the
overall flexibility.

TASTING is a process that happens whenever a new class or

new provider is
created and it provides the class a chance to automatically

configure an
instance on providers, which it recognize as its own. A

typical example
is the MBR disk-partition class which will look for the MBR

table in the
first sector and if found and validated it will instantiate

a geom to
multiplex according to the contents of the MBR.

A new class will be offered to all existing providers in

turn and a new
provider will be offered to all classes in turn.

Exactly what a class does to recognize if it should accept

the offered
provider is not defined by GEOM, but the sensible set of op

tions are:

+o Examine specific data structures on the disk.

+o Examine properties like sectorsize or mediasize for the

provider.

+o Examine the rank number of the provider's geom.

+o Examine the method name of the provider's geom.

ORPHANIZATION is the process by which a provider is removed

while it
potentially is still being used.

When a geom orphans a provider, all future I/O requests will

"bounce" on
the provider with an error code set by the geom. Any con

sumers attached
to the provider will receive notification about the orpha

nization when
the eventloop gets around to it, and they can take appropri

ate action at
that time.

A geom which came into being as a result of a normal taste

operation
should selfdestruct unless it has a way to keep functioning

lacking the
orphaned provider. Geoms like diskslicers should therefore

selfdestruct
whereas RAID5 or mirror geoms will be able to continue, as

long as they
do not loose quorum.

When a provider is orphaned, this does not necessarily re

sult in any
immediate change in the topology: any attached consumers are

still
attached, any opened paths are still open, any outstanding

I/O requests
are still outstanding.

The typical scenario is
+o A device driver detects a disk has departed and

orphans the
provider for it.

+o The geoms on top of the disk receive the orpha

nization event
and orphans all their providers in turn.

Providers, which are
not attached to, will typically self-destruct

right away. This
process continues in a quasi-recursive fashion un

til all relevant pieces of the tree has heard the bad news.

+o Eventually the buck stops when it reaches geom_dev

at the top
of the stack.

+o Geom_dev will call destroy_dev(9) to stop any more

request fromcoming in. It will sleep until all (if any) out

standing I/O
requests have been returned. It will explicitly

close (ie:
zero the access counts), a change which will prop

agate all the
way down through the mesh. It will then detach

and destroy its
geom.

+o The geom whose provider is now attached will de

stroy theprovider, detach and destroy its consumer and de

stroy its geom.

+o This process percolates all the way down through

the mesh,
until the cleanup is complete.

While this approach seems byzantine, it does provide the

maximum flexibility and robustness in handling disappearing devices.

The one absolutely crucial detail to be aware is that if the

device
driver does not return all I/O requests, the tree will not

unravel.

SPOILING is a special case of orphanization used to protect

against stale
metadata. It is probably easiest to understand spoiling by

going through
an example.

Imagine a disk, "da0" on top of which a MBR geom provides

"da0s1" and
"da0s2" and on top of "da0s1" a BSD geom provides "da0s1a"

through
"da0s1e", both the MBR and BSD geoms have autoconfigured

based on data
structures on the disk media. Now imagine the case where

"da0" is opened
for writing and those data structures are modified or over

written: Now
the geoms would be operating on stale metadata unless some

notification
system can inform them otherwise.

To avoid this situation, when the open of "da0" for write

happens, all
attached consumers are told about this, and geoms like MBR

and BSD will
selfdestruct as a result. When "da0" is closed again, it

will be offered
for tasting again and if the data structures for MBR and BSD

are still
there, new geoms will instantiate themselves anew.

Now for the fine print:

If any of the paths through the MBR or BSD module were open,

they would
have opened downwards with an exclusive bit rendering it im

possible to
open "da0" for writing in that case and conversely the re

quested exclusive bit would render it impossible to open a path through

the MBR geom
while "da0" is open for writing.

From this it also follows that changing the size of open ge

oms can only
be done with their cooperation.

Finally: the spoiling only happens when the write count goes

from zero to
non-zero and the retasting only when the write count goes

from non-zero
to zero.

INSERT/DELETE are a very special operation which allows a

new geom to be
instantiated between a consumer and a provider attached to

each other and
to remove it again.

To understand the utility of this, imagine a provider with

being mounted
as a file system. Between the DEVFS geoms consumer and its

provider we
insert a mirror module which configures itself with one mir

ror copy and
consequently is transparent to the I/O requests on the path.

We can now
configure yet a mirror copy on the mirror geom, request a

synchronization, and finally drop the first mirror copy. We have now

in essence
moved a mounted file system from one disk to another while

it was being
used. At this point the mirror geom can be deleted from the

path again,
it has served its purpose.

CONFIGURE is the process where the administrator issues in

structions for
a particular class to instantiate itself. There are multi

ple ways to
express intent in this case, a particular provider can be

specified with
a level of override forcing for instance a BSD disklabel

module to attach
to a provider which was not found palatable during the TASTE

operation.

Finally IO is the reason we even do this: it concerns itself

with sending
I/O requests through the graph.

I/O REQUESTS represented by struct bio, originate at a con

sumer, are
scheduled on its attached provider, and when processed, re

turned to the
consumer. It is important to realize that the struct bio

which enters
through the provider of a particular geom does not "come out

on the other
side". Even simple transformations like MBR and BSD will

clone the
struct bio, modify the clone, and schedule the clone on

their own consumer. Note that cloning the struct bio does not involve

cloning the
actual data area specified in the IO request.

In total four different IO requests exist in GEOM: read,

write, delete,
and get attribute.

Read and write are self explanatory.

Delete indicates that a certain range of data is no longer

used and that
it can be erased or freed as the underlying technology sup

ports. Technologies like flash adaptation layers can arrange to erase

the relevant
blocks before they will become reassigned and cryptographic

devices may
want to fill random bits into the range to reduce the amount

of data
available for attack.

It is important to recognize that a delete indication is not

a request
and consequently there is no guarantee that the data actual

ly will be
erased or made unavailable unless guaranteed by specific ge

oms in the
graph. If "secure delete" semantics are required, a geom

should be
pushed which converts delete indications into (a sequence

of) write
requests.

Get attribute supports inspection and manipulation of out

of-band
attributes on a particular provider or path. Attributes are

named by
ascii strings and they will be discussed in a separate sec

tion below.

(stay tuned while the author rests his brain and fingers:

more to come.)

DIAGNOSTICS

Several flags are provided for tracing GEOM operations and

unlocking protection mechanisms via the kern.geom.debugflags sysctl. All

of these
flags are off by default, and great care should be taken in

turning them
on.

0x01 (G_T_TOPOLOGY)
Provide tracing of topology change events.

0x02 (G_T_BIO)
Provide tracing of buffer I/O requests.

0x04 (G_T_ACCESS)
Provide tracing of access check controls.

0x08 (unused)

0x10 (allow foot shooting)
Allow writing to Rank 1 providers. This would, for

example, allow
the super-user to overwrite the MBR on the root disk

or write random sectors elsewhere to a mounted disk. The implica

tions are
obvious.

0x20 (G_T_DETAILS)
This appears to be unused at this time.

0x40 (G_F_DISKIOCTL)
This appears to be unused at this time.

0x80 (G_F_CTLDUMP)
Dump contents of gctl requests.

HISTORY

This software was developed for the FreeBSD Project by Poul

Henning Kamp
and NAI Labs, the Security Research Division of Network As

sociates, Inc.
under DARPA/SPAWAR contract N66001-01-C-8035 ("CBOSS"), as

part of the
DARPA CHATS research program.

The first precursor for GEOM was a gruesome hack to Minix

1.2 and was
never distributed. An earlier attempt to implement a less

general scheme
in FreeBSD never succeeded.

AUTHORS

Poul-Henning Kamp <phk@FreeBSD.org>

BSD March 27, 2002

BSD