lket(5)

NAME

LKET - Linux Kernel Event Trace tool based on SystemTap

DESCRIPTION

The Linux Kernel Event Trace (LKET) tool is an extension

to the tapsets library available on SystemTap. Its goal is to

utilize the dynamic probing capabilities provided through System

Tap to create a set of standard hooks that probe pre-defined

places in the kernel. It can be used to collect important infor

mation that can be used as a starting point to analyze a perfor

mance problem in the system.

The LKET tapsets are designed to only trace the events se

lected by the user. Once the data has been collected, it is then

post-processed according to the need of the user. Trace data can

be processed in various different ways to generate simple to com

plex reports.

EVENT HOOKS

The following sections enumerate the variety of event

hooks implemented in LKET and their trace data format. The trace

data generated by different event hooks contain common data as

well as some data specific to that event hook.

the INT8, INT16, INT32, INT64 and STRING appeared in trace

data format represents 8-bit, 16-bit, 32-bit, 64-bit binary data

and NULL-terminated string respectively.

The data common(i.e. common_data in the following subse

cions) to all event hooks is:

timestamp(INT64),(tid<<32 | groupID<<24

hookID<<16 | cpu_id<<8)(INT64)

Each event hook group is a collection of those hooks that

have similarities of what they could trace. And the ID of each

event hook (HookID) is defined in the context of its correspond

ing group.

EVENT REGISTER

Event register is not actually an event. It is used to log

the metadata of the trace data, including the extra trace data

appended by user. See EVENT REGISTER and CUSTOMIZED TRACE DATA

for more details.

register_sys_event

This is a function used to register event hooks

available in LKET. It should be called from regis

ter_event.stp:register_sys_events().

register_user_event

This is a function used to log the metadata of the

extra trace data appended by user for a specific event. It

should be called in the probe register_event

SYSTEM CALLS

You could use addevent.syscall to trace the entry and re

turn of all system calls. It contains two sub event hooks:

addevent.syscall.entry

Trace entry of all system calls.

Data format is:

common_data, syscall_name(STRING)

addevent.syscall.return

Trace return of all system calls.

Data format is:

common_data, syscall_name(STRING)

PROCESS CREATION

This group contains three sub event hooks. You could use

addevent.process to trace fork and execve of processes(note that

process_snapshot() won't be included).

process_snapshot()

This event hook isn't a probe definition but a

function. It is called by LKET silently to take a snapshot of all

running processes.

Data format is:

common_data, tid(INT32), pid(INT32), ppid(INT32),

process_name(STRING)

addevent.process.fork

Trace fork of processes

Data format is:

common_data, new_tid(INT32), new_pid(INT32),

ppid(INT32)

addevent.process.execve

Trace execve of new processes

Data format is:

common_data, pid(INT32), new_process_name(STRING)

SIGNAL

You could use addevent.signal to trace signal activities.

It contains the following events:

addevent.signal.send.entry

Trace when a signal is sent to a process

Data format is: common_data, sig(INT8),

shared(INT8), send2queue(INT8), pid(INT32)

addevent.signal.send.return

Trace when returning from sending signal

Data format is: common_data, return(INT8)

addevent.signal.syskill.entry

Trace when sys_kill is called to send a signal to a

process.

Data format is: common_data, pid(INT32), sig(INT8)

addevent.signal.syskill.return

Trace when return from sys_kill

Data format is: common_data, return(INT8)

addevent.signal.systgkill.entry

Trace when sys_tgkill is called to send a signal to

one specific thread

Data format is: common_data, tid(INT32),

pid(INT32), sig(INT8)

addevent.signal.systgkill.return

Trace when returning from sys_tgkill

Data format is: common_data, return(INT8)

addevent.signal.systkill.entry

Trace when sys_tkill is called to send a signal to

a single process.

Data format is: common_data, pid(INT32), sig(INT8)

addevent.signal.systkill.return

Trace when returning from sys_tkill.

Data format is: common_data, return(INT8)

addevent.signal.pending.entry

Trace when examine the set of signals that are

pending for delivery.

Data format is: common_data, sigset_addr(INT32),

setsize(INT32)

addevent.signal.pending.return

Trace when returning from signal.pending

Data format is: common_data, return(INT8)

addevent.signal.do_action.entry

Trace when a thread is about to examine and change

a signal action

Data format is: common_data, sig(INT8),

handler(INT64)

addevent.signal.do_action.return

Trace when returning from signal.do_action

Data format is: common_data, return(INT8)

addevent.signal.procmask.entry

Trace when a thread is about to examine and change

blocked signals

Data format is: common_data, how(INT8),

sigset(INT64)

addevent.signal.procmask.return

Trace when returning from signal.procmask

Data format is common_data, return(INT8)

addevent.signal.flush.entry

Trace when flush all pending signals for a task

Data format is: common_data, pid(INT32)

IO SCHEDULER ACTIVITIES

You could use addevent.ioscheduler to trace the IO sched

uler activities. It contains three sub event hooks:

addevent.ioscheduler.elv_add_request

Trace when a request is added to the request queue

Data format is:

common_data, elevator_name(STRING),

disk_major(INT8), disk_minor(INT8), request_addr(INT64),

request_flags(INT64)

addevent.ioscheduler.elv_next_request.entry

Trace when try to retrieve a request from request

queue

Data format is:

common_data, elevator_name(STRING)

addevent.ioscheduler.elv_next_request.return

Trace when return from retrieving a request from

request queue

Data format is:

common_data, disk_major(INT8), disk_minor(INT8),

request_addr(INT64), request_flags(INT64)

addevent.ioscheduler.elv_completed_request

Trace when a request is completed

Data format is:

common_data, elevator_name(STRING),

disk_major(INT8), disk_minor(INT8), request_addr(INT64),

request_flags(INT64)

TASK SCHEDULE ACTIVITIES

You could use addevent.tskdispatch to trace the task

scheduler activities. It contains two sub event hooks:

addevent.tskdispatch.ctxswitch

Trace the process context switch

Data format is:

common_data, prev_pid(INT32), next_pid(INT32),

prev_state(INT8)

addevent.tskdispatch.cpuidle

Trace when cpu goes idle

Data format is:

common_data, current_pid(INT32)

SCSI ACTIVITIES

You could use addevent.scsi to trace the scsi layer activ

ities. It contains four sub event hooks:

addevent.scsi.ioentry

mid-layer prepares a IO request

Data format is:

common_data, disk_major(INT8), disk_minor(INT8),

device_state(INT8), request_addr(INT64)

addevent.scsi.iodispatching

Dispatch a command to the low-level driver

Data format is:

common_data, host(INT8), channel(INT8), lun(INT8),

dev_id(INT8), device_state(INT8), data_direction(INT8),

reqbuf_addr(INT64), reqbuf_len(INT32), request_addr(INT64)

addevent.scsi.iodone

I/O is done by low-level driver

Data format is:

common_data, host(INT8), channel(INT8), lun(INT8),

dev_id(INT8), device_state(INT8), data_direction(INT8),

request_addr(INT64)

addevent.scsi.iocompleted

mid-layer processed the completed IO

Data format is:

common_data, host(INT8), channel(INT8), lun(INT8),

dev_id(INT8), device_state(INT8), data_direction(INT8),

request_addr(INT64), bytes_done(INT32)

PAGE FAULT

You could use addevent.pagefault to trace page fault

events. It contains only one sub event hooks:

addevent.pagefault

Data format is:

common_data, memory_address(INT64),

write_access(INT8)

NETWORK DEVICE ACTIVITIES

You could use addevent.netdev to trace the network device

activities. It contains two sub event hooks:

addevent.netdev.receive

network device receives a packet

Data format is:

common_data, netdev_name(STRING),

data_length(INT32), protocol(INT16), buffer_length(INT32)

addevent.netdev.transmit

A packet will be sent out by network device

Data format is:

common_data, netdev_name(STRING),

data_length(INT32), protocol(INT16), buffer_length(INT32)

IO SYSCALLS

You could use addevent.iosyscall to trace the detail ac

tivities of io related system calls. It contains 16 entry hooks

and 16 corresponding return hooks.

All the return hooks will only log the common_data and the

return value. So in the following subsections, only the entry

hooks will be listed:

addevent.iosyscall.open.entry

the entry of sys_open

Data format is:

common_data, filename(STRING), flags(INT32),

mode(INT32)

addevent.iosyscall.close.entry

the entry of sys_close

Data format is:

common_data, fd(INT64)

addevent.iosyscall.read.entry

the entry of sys_read

Data format is:

common_data, fd(INT64), buf_addr(INT64),

count(INT64)

addevent.iosyscall.write.entry

the entry of sys_write

Data format is:

common_data, fd(INT64), buf_addr(INT64),

count(INT64)

addevent.iosyscall.readv.entry

the entry of sys_readv

Data format is:

common_data, fd(INT64), vector_addr(INT64),

count(INT64)

addevent.iosyscall.writev.entry

the entry of sys_writev

Data format is:

common_data, fd(INT64), vector_addr(INT64),

count(INT64)

addevent.iosyscall.pread64.entry

the entry of sys_pread64

Data format is:

common_data, fd(INT64), buff_addr(INT64),

count(INT64), offset(INT64)

addevent.iosyscall.pwrite64.entry

the entry of sys_pwrite64

Data format is:

common_data, fd(INT64), buff_addr(INT64),

count(INT64), offset(INT64)

addevent.iosyscall.readahead.entry

the entry of sys_readahead

Data format is:

common_data, fd(INT64), offset(INT64), count(INT64)

addevent.iosyscall.senfile.entry

the entry of sys_sendfile and sys_sendfile64

Data format is:

common_data, out_fd(INT64), in_fd(INT64),

offset_uaddr(INT64), count(INT64)

addevent.iosyscall.lseek.entry

the entry of sys_lseek

Data format is:

common_data, fd(INT64), offset(INT64), whence(INT8)

addevent.iosyscall.llseek.entry

the entry of sys_llseek

Data format is:

common_data, fd(INT64), offset_high(INT64),

offset_low(INT64), result_addr(INT64), whence(INT8)

addevent.iosyscall.sync.entry

the entry of sys_sync

Data format is:

common_data

addevent.iosyscall.fsync.entry

the entry of sys_fsync

Data format is:

common_data, fd(INT64)

addevent.iosyscall.fdatasync.entry

the entry of sys_fdatasync

Data format is:

common_data, fd(INT64)

addevent.iosyscall.flock.entry

the entry of sys_flock

Data format is:

common_data, fd(INT64), operation(INT32)

Asynchronous IO

You could use addevent.aio to trace the detail activities

of AIO related calls(most of them are AIO system calls). It con

tains 6 entry hooks and 6 corresponding return hooks.

All the return hooks will only log the common_data and the

return value. So in the following subsections, only the entry

hooks will be listed:

addevent.aio.io_setup.entry

Fired by calling io_setup from user space. The cor

responding system call is sys_io_setup, which will create an

aio_context capable of receiving at least maxevents.

Data format is:

common_data, nr_events(INT32), ctxp_uaddr(INT64)

addevent.aio.io_submit.entry

Fired by calling io_submit from user space. The

corresponding system call is sys_io_submit which will queue the

nr iocbs pointed to by iocbpp_uaddr for processing.

Data format is:

common_data, ctx_id(INT64), nr(INT32),

iocbpp_uaddr(INT64)

addevent.aio.io_submit_one.entry

Called by sys_io_submit. It will iterate iocbpp and

process them one by one

Data format is:

common_data, ctx(INT64), user_iocb_uaddr(INT64),

aio_lio_opcode(INT16), aio_reqprio(INT16), aio_fildes(INT32),

aio_buf(INT64), aio_nbytes(INT64), aio_offset(INT64)

addevent.aio.io_getevents.entry

Fired by calling io_getevents from user space. The

corresponding system call is sys_io_getevents, which will attempt

to read at least min_nr events and up to nr events from the com

pletion queue for the aio_context specified by ctx_id.

Data format is:

common_data, ctx_id(INT64), min_nr(INT32),

nr(INT32), events_uaddr(INT64), tv_sec(INT32), tv_nsec(INT32)

addevent.aio.io_destroy.entry

Fired by calling io_destroy from user space. The

corresponding system call is sys_io_destroy, which will destroy

the aio_context specified.

Data format is:

common_data, ctx(INT64)

addevent.aio.io_cancel.entry

Fired by calling io_cancel from user space. The

corresponding system call is sys_io_cancel, which will attempt to

cancel an iocb previously passed to io_submit.

Data format is:

common_data, ctx_id(INT64), iocb_uaddr(INT64),

result_uaddr(INT64)

SUNRPC

You could use addevent.sunrpc to trace the details of SUN

RPC activities. It is now divided into three groups: high-level

client operation event hooks (addevent.sunrpc.clnt), high-level

server operation event hooks (addevent.sunrpc.svc) and RPC sched

uler operation event hooks (addevent.sunrpc.sched). It contains

19 entry hooks and 19 corresponding return hooks.

All the return hooks will only log the common_data and the

return value. So in the following subsections, only the entry

hooks will be listed:

addevent.sunrpc.clnt.create_client.entry

Fires when an RPC client is to be created

Data format is:

common_data, servername(STRING), prog(INT64),

vers(INT8), prot(INT16), port(INT16), authflavor(INT8)

addevent.sunrpc.clnt.clone_client.entry

Fires when the RPC client structure is to be cloned

Data format is:

common_data, servername(STRING), prog(INT64),

vers(INT8), prot(INT16), port(INT16), authflavor(INT8)

addevent.sunrpc.clnt.shutdown_client.entry

Fires when an RPC client is to be shut down

Data format is:

common_data, servername(STRING), prog(INT64),

clones(INT16), tasks(INT16), rpccnt(INT32)

addevent.sunrpc.clnt.bind_new_program.entry

Fires when a new RPC program is to be bound an ex

isting client

Data format is:

common_data, servername(STRING), old_prog(INT64),

old_vers(INT8), prog(INT64), vers(INT8)

addevent.sunrpc.clnt.call_sync.entry

Fires when an RPC procedure is to be called syn

chronously

Data format is:

common_data, servername(STRING), prog(INT64),

vers(INT8), proc(INT64), flags(INT64)

addevent.sunrpc.clnt.call_async.entry

Fires when an RPC procedure is to be called asyn

chronously

Data format is:

common_data, servername(STRING), prog(INT64),

vers(INT8), proc(INT64), flags(INT64)

addevent.sunrpc.clnt.restart_call.entry

Fires when want to restart a task

Data format is:

common_data, tk_pid(INT64), tk_flags(INT64)

addevent.sunrpc.svc.register.entry

Fires when an RPC service is to be registered with

the local portmapper

Data format is:

common_data, sv_name(STRING), prog(INT64),

prot(INT16), port(INT32)

addevent.sunrpc.svc.create.entry

Fires when an RPC service is to be created

Data format is:

common_data, prog(INT64), pg_nvers(INT8),

bufsize(INT32)

addevent.sunrpc.svc.destroy.entry

Fires when an RPC service is to be destroyed

Data format is:

common_data, sv_name(STRING), sv_prog(INT64),

sv_nrthreads(INT32)

addevent.sunrpc.svc.process.entry

Fires when an RPC request is to be processed

Data format is:

common_data, sv_name(STRING), sv_prog(INT64),

peer_ip(INT64), rq_xid(INT64), rq_prog(INT64), rq_vers(INT8),

rq_proc(INT8)

addevent.sunrpc.svc.authorise.entry

Fires when an RPC request is to be authorised

Data format is:

common_data, sv_name(STRING), peer_ip(INT64),

rq_xid(INT64), rq_prog(INT64), rq_vers(INT8), rq_proc(INT64)

addevent.sunrpc.svc.recv.entry

Fires when receiving the next request on any socket

Data format is:

common_data, sv_name(STRING), timeout(INT64)

addevent.sunrpc.svc.send.entry

Fires when want to return reply to the client

Data format is:

sv_name(STRING), peer_ip(INT64), rq_xid(INT64),

rq_prog(INT64), rq_vers(INT8), rq_proc(INT64)

addevent.sunrpc.svc.drop.entry

Fires when a request is to be dropped

Data format is:

common_data, sv_name(STRING), peer_ip(INT64),

rq_xid(INT64), rq_prog(INT64), rq_vers(INT8), rq_proc(INT64)

addevent.sunrpc.sched.new_task.entry

Fires when creating a new task for the specified

client

Data format is:

common_data, xid(INT64), prog(INT64), vers(INT8),

prot(INT64), flags(INT64)

addevent.sunrpc.sched.release_task.entry

Fires when releasing a task

Data format is:

common_data, xid(INT64), prog(INT64), vers(INT8),

prot(INT64), flags(INT64)

addevent.sunrpc.sched.execute.entry

Fires when an RPC request is to be executed

Data format is:

common_data, xid(INT64), prog(INT64), vers(INT8),

prot(INT64), tk_pid(INT64), tk_flags(INT64)

addevent.sunrpc.sched.delay.entry

Fires when want to delay an RPC request

Data format is:

common_data, xid(INT64), prog(INT64),

tk_pid(INT64), tk_flags(INT64), delay(INT64)

NFS

You could use addevent.nfs to trace the detail activities

of nfs on client side. It divided into three groups: nfs file

operation event hooks(addevent.nfs.fop), nfs address space opera

tion event hooks(addevent.nfs.aop), nfs proc event hooks(adde

vent.nfs.proc). It contains 36 entry hooks and 33 corresponding

return hooks

All the return hooks will only log the common_data and the

return value. So in the following subsections, only the entry

hooks will be listed:

addevent.nfs.fop.llseek.entry

the entry of nfs_file_llseek

Data format is:

common_data, major_device(INT8),

minor_devide(INT8), fileid(INT32), offset(INT32), origin(INR8)

addevent.nfs.fop.read.entry

the entry of do_sync_read

Data format is:

common_data, major_device(INT8),

minor_devide(INT8), fileid(INT32), buf_addr(INT64), count(INT64)

, offset(INT64)

addevent.nfs.fop.write.entry

the entry of do_sync_write

Data format is:

common_data, major_device(INT8),

minor_devide(INT8), fileid(INT32), buf_addr(INT64), count(INT64)

, offset(INT64)

addevent.nfs.fop.aio_read.entry

the entry of nfs_file_read

Data format is:

common_data, major_device(INT8),

minor_devide(INT8), fileid(INT32), buf_addr(INT64), count(INT64)

, offset(INT64)

addevent.nfs.fop.aio_write.entry

the entry of nfs_file_read

Data format is:

common_data, major_device(INT8),

minor_devide(INT8), fileid(INT32), buf_addr(INT64), count(INT64)

, offset(INT64)

addevent.nfs.fop.mmap.entry

the entry of nfs_file_mmap

Data format is:

common_data, major_device(INT8),

minor_devide(INT8), fileid(INT32), vm_start(INT64), vm_end(INT64)

, vm_flags(INT32)

addevent.nfs.fop.open.entry

the entry of nfs_file_open

Data format is:

common_data, major_device(INT8),

minor_devide(INT8), fileid(INT32), flag(INT32), filename(STRING)

addevent.nfs.fop.flush.entry

the entry of nfs_file_flush

Data format is:

common_data, major_device(INT8),

minor_devide(INT8), fileid(INT32), ndirty(INT32)

addevent.nfs.fop.release.entry

the entry of nfs_file_release

Data format is:

common_data, major_device(INT8),

minor_devide(INT8), fileid(INT32), mode(INT16)

addevent.nfs.fop.fsync.entry

the entry of nfs_fsync

Data format is:

common_data, major_device(INT8),

minor_devide(INT8), fileid(INT32), ndirty(INT32)

addevent.nfs.fop.lock.entry

the entry of nfs_lock

Data format is:

common_data, major_device(INT8),

minor_devide(INT8), fileid(INT32), fl_start(INT64),

fl_end(INT64), fl_type(INT8), fl_flag(INT8), cmd(INT32)

addevent.nfs.fop.sendfile.entry

the entry of nfs_file_sendfile

Data format is:

common_data, major_device(INT8),

minor_devide(INT8), fileid(INT32), count(INT64), ppos(INT64)

addevent.nfs.fop.checkflags.entry

the entry of nfs_check_flags

Data format is:

flag(INT32)

addevent.nfs.aop.readpage.entry

the entry of nfs_readpage

Data format is:

fileid(INT64), rsize(INT32), page_address(INT64),

page_index(INT64)

addevent.nfs.aop.readpages.entry

the entry of nfs_readpages

Data format is:

fileid(INT64), rpages(INT32), nr_pages(INT32)

addevent.nfs.aop.writepage.entry

the entry of nfs_writepage

Data format is:

fileid(INT64), wsize(INT32), page_address(INT64),

page_index(INT64)

addevent.nfs.aop.writepages.entry

the entry of nfs_writepages

Data format is:

fileid(INT64), wpages(INT32), nr_to_write(INT64)

addevent.nfs.aop.prepare_write.entry

the entry of nfs_prepare_write

Data format is:

fileid(INT64), page_address(INT64),

page_index(INT64)

addevent.nfs.aop.commit_write.entry

the entry of nfs_commit_write

Data format is:

fileid(INT64), page_address(INT64),

page_index(INT64),offset(INT32),count(INT32)

addevent.nfs.aop.set_page_dirty.entry

the entry of __set_page_dirty_nobuffers

Data format is:

page_address(INT64), page_flag(INT8)

addevent.nfs.aop.release_page.entry

the entry of nfs_release_page

Data format is:

page_address(INT64), page_index(INT64)

addevent.nfs.proc.lookup.entry

the entry of nfs_proc_lookup , nfs3_proc_lookup and

nfs4_proc_lookup

Data format is:

major_dev(INT8), minor_dev(INT8), fileid(INT64),

version(INT8), filename(STRING)

addevent.nfs.proc.read.entry

the entry of nfs_proc_read, nfs3_proc_read and

nfs4_proc_read

Data format is:

major_dev(INT8), minor_dev(INT8), fileid(INT64),

version(INT8), count(INT32),offset(INT64)

addevent.nfs.proc.write.entry

the entry of nfs_proc_write, nfs3_proc_write and

nfs4_proc_write

Data format is:

major_dev(INT8), minor_dev(INT8), fileid(INT64),

version(INT8), count(INT32),offset(INT64)

addevent.nfs.proc.commit.entry

Fires when client writes the buffered data to

disk,the buffered data is asynchronously written by client before

. The commit function works in sync way,not exist in NFSV2

the entry of nfs3_proc_commit and nfs4_proc_commit

Data format is:

major_dev(INT8), minor_dev(INT8), fileid(INT64),

version(INT8), count(INT32),offset(INT64)

addevent.nfs.proc.read_setup.entry

The read_setup function is used to setup a read rpc

task,not do a real read operation

the entry of nfs_proc_read_setup,

nfs3_proc_read_setup and nfs4_proc_read_setup

Data format is:

major_dev(INT8), minor_dev(INT8), fileid(INT64),

version(INT8), count(INT32),offset(INT64)

addevent.nfs.proc.write_setup.entry

The write_setup function is used to setup a write

rpc task,not do a real write operation

the entry of nfs_proc_write_setup,

nfs3_proc_write_setup and nfs4_proc_write_setup

Data format is:

major_dev(INT8), minor_dev(INT8), fileid(INT64),

version(INT8), how(INT8), count(INT32),offset(INT64)

addevent.nfs.proc.commit_setup.entry

The commit_setup function is used to setup a commit

rpc task,not do a real commit operation.It is not exist in NFSV2

the entry of nfs3_proc_commit_setup and

nfs4_proc_commit_setup

Data format is:

major_dev(INT8), minor_dev(INT8), fileid(INT64),

version(INT8), how(INT8), count(INT32),offset(INT64)

addevent.nfs.proc.read_done.entry

Fires when a read reply is received or some read

error occur (timeout or socket shutdown)

the entry of nfs_read_done, nfs3_read_done and

nfs4_read_done

Data format is:

major_dev(INT8), minor_dev(INT8), fileid(INT64),

version(INT8), status(INT32), count(INT32)

addevent.nfs.proc.write_done.entry

Fires when a write reply is received or some write

error occur (timeout or socket shutdown)

the entry of nfs_write_done, nfs3_write_done and

nfs4_write_done

Data format is:

major_dev(INT8), minor_dev(INT8), fileid(INT64),

version(INT8), status(INT32), count(INT32)

addevent.nfs.proc.commit_done.entry

Fires when a commit reply is received or some com

mit operation error occur (timeout or socket shutdown)

the entry of nfs_commit_done, nfs3_commit_done and

nfs4_commit_done

Data format is:

major_dev(INT8), minor_dev(INT8), fileid(INT64),

version(INT8), status(INT32), count(INT32)

addevent.nfs.proc.open.entry

the entry of nfs_open

Data format is:

major_dev(INT8), minor_dev(INT8), fileid(INT64),

version(INT8), filename(STRING), flag(INT32), mode(INT32)

addevent.nfs.proc.release.entry

the entry of nfs_release

Data format is:

major_dev(INT8), minor_dev(INT8), fileid(INT64),

version(INT8), filename(STRING), flag(INT32), mode(INT32)

addevent.nfs.proc.create.entry

the entry of nfs_proc_create, nfs3_proc_create,

nfs4_proc_create

Data format is:

major_dev(INT8), minor_dev(INT8), fileid(INT64),

version(INT8), filename(STRING), mode(INT32)

addevent.nfs.proc.rename.entry

the entry of nfs_proc_rename, nfs3_proc_rename,

nfs4_proc_rename

Data format is:

version(INT8), major_old(INT8), minor_old(INT8),

old_fileid(INT64), old_name(STRING), major_new(INT8),

minor_new(INT8), new_fileid(INT64), new_name(STRING)

addevent.nfs.proc.remove.entry

the entry of nfs_proc_remove, nfs3_proc_remove,

nfs4_proc_remove

Data format is:

major_dev(INT8), minor_dev(INT8), fileid(INT64),

version(INT8), filename(STRING)

NFSD

You could use addevent.nfsd to trace the detail activities

of nfs on server side. It divided into two groups: nfsd opera

tion event hooks(addevent.nfsd.op), nfsd proc event hooks(adde

vent.nfsd.proc). It contains 19 entry hooks and 19 corresponding

return hooks

All the return hooks will only log the common_data and the

return value. So in the following subsections, only the entry

hooks will be listed:

addevent.nfsd.dispatch.entry

Fires when server receives a NFS operation from

client

the entry of nfsd_dispatch

Data format is:

proto(INT8), version(INT8), xid(INT32),

proc(INT32),client_ip(INT32)

addevent.nfsd.open.entry

the entry of nfsd_open

Data format is:

fh_size(INT8), fhandle0(INT64), fhandle1(INT64),

fhandle2(INT64), type(INT32), access(INT32)

addevent.nfsd.read.entry

the entry of nfsd_read

Data format is:

fh_size(INT8), fhandle0(INT64), fhandle1(INT64),

fhandle2(INT64), count(INT64), offset(INT64), iov_len(INT64),

vlen(INT64)

addevent.nfsd.write.entry

the entry of nfsd_write

Data format is:

fh_size(INT8), fhandle0(INT64), fhandle1(INT64),

fhandle2(INT64), count(INT64), offset(INT64), iov_len(INT64),

vlen(INT64)

addevent.nfsd.lookup.entry

the entry of nfsd_lookup

Data format is:

fh_size(INT8), fhandle0(INT64), fhandle1(INT64),

fhandle2(INT64), filename(STRING)

addevent.nfsd.commit.entry

the entry of nfsd_commit

Data format is:

fh_size(INT8), fhandle0(INT64), fhandle1(INT64),

fhandle2(INT64), count(INT64), offset(INT64)

addevent.nfsd.create.entry

Fires when client creates a file(regular,dir,de

vice,fifo) on server side, sometimes nfsd will call nfsd_cre

ate_v3 instead of this function

the entry of nfsd_create

Data format is:

fh_size(INT8), fhandle0(INT64), fhandle1(INT64),

fhandle2(INT64), filename(STRING), type(INT32), iap_valid(INT16),

iap_mode(INT32)

addevent.nfsd.createv3.entry

Fires when client creates a regular file or set

file attributes on server side, only called by nfsd3_proc_create

and nfsd4_open(op_claim_type is NFS4_OPEN_CLAIM_NULL)

the entry of nfsd_create_v3

Data format is:

fh_size(INT8), fhandle0(INT64), fhandle1(INT64),

fhandle2(INT64), filename(STRING), createmode(INT8),

iap_valid(INT16), iap_mode(INT32)

addevent.nfsd.unlink.entry

the entry of nfsd_unlink

Data format is:

fh_size(INT8), fhandle0(INT64), fhandle1(INT64),

fhandle2(INT64), filename(STRING), type(INT32)

addevent.nfsd.rename.entry

the entry of nfsd_rename

Data format is:

old_fhsize(INT8), old_fh0(INT64), old_fh1(INT64),

old_fh2(INT64), old_name(STRING) new_fhsize(INT8),

new_fh0(INT64), new_fh1(INT64), new_fh2(INT64), new_name(STRING)

addevent.nfsd.close.entry

the entry of nfsd_close

Data format is:

filename(STRING)

addevent.nfsd.proc.lookup.entry

the entry of nfsd_proc_lookup, nfsd3_proc_lookup

Data format is:

fh_size(INT8), fhandle0(INT64), fhandle1(INT64),

fhandle2(INT64), version(INT8) filename(STRING)

addevent.nfsd.proc.read.entry

the entry of nfsd_proc_read, nfsd3_proc_read

Data format is:

fh_size(INT8), fhandle0(INT64), fhandle1(INT64),

fhandle2(INT64), version(INT8) count(INT64), offset(INT64),

iov_len(INT64), vlen(INT64)

addevent.nfsd.proc.write.entry

the entry of nfsd_proc_write, nfsd3_proc_write

Data format is:

fh_size(INT8), fhandle0(INT64), fhandle1(INT64),

fhandle2(INT64), version(INT8) count(INT64), offset(INT64),

iov_len(INT64), vlen(INT64)

addevent.nfsd.proc.commit.entry

the entry of nfsd_proc_commit, nfsd3_proc_commit

Data format is:

fh_size(INT8), fhandle0(INT64), fhandle1(INT64),

fhandle2(INT64), version(INT8) count(INT64), offset(INT64)

addevent.nfsd.proc.commit.entry

the entry of nfsd4_proc_compound

Data format is:

number(INT32)

addevent.nfsd.proc.remove.entry

the entry of nfsd4_proc_compound

Data format is:

fh_size(INT8), fhandle0(INT64), fhandle1(INT64),

fhandle2(INT64), version(INT8) filename(STRING)

addevent.nfsd.proc.rename.entry

the entry of nfsd_proc_rename, nfsd3_proc_rename

Data format is:

old_fhsize(INT8), old_fh0(INT64), old_fh1(INT64),

old_fh2(INT64), old_name(STRING) new_fhsize(INT8),

new_fh0(INT64), new_fh1(INT64), new_fh2(INT64), new_name(STRING)

addevent.nfsd.proc.create.entry

the entry of nfsd_proc_create, nfsd3_proc_create

Data format is:

fh_size(INT8), fhandle0(INT64), fhandle1(INT64),

fhandle2(INT64), version(INT8) filename(STRING)

TRACE DATA FORMAT

By default, LKET will log the trace data in binary format.

To get a better performance for binary tracing, the "-b"

option should be turned on for stap and thus -M option has to be

added to stop staprun merging per-cpu files.

You could use the command lket-b2a to convert the binary

trace data generated by LKET into readable data in ascii format.

lket-b2a uses the pre-cpu binary trace data files as in

puts, and generates an output file named lket.out You should use

"stap -b -M" with LKET to get those pre-cpu files (stpd_cpu*) be

fore using it.

If you want LKET to log trace data in ASCII format direct

ly, you should:

stap -D ASCII_TRACE ...

EVENT REGISTER

LKET provides a way to log the metadata of the trace data

by events registering.

Two function is provided:

register_sys_event(grpid:long, hookid:long,

fmt:string, names:string)

register_user_event(grpid:long, hookid:long,

fmt:string, names:string)

The grpid and hookid is the groupid and hookid of the

event you want to register.

fmt contains a set of fomat tokens seperated by ":". The

valid format tokens are: UINT8, UINT16, UINT32, UINT64 and STRING

which represents 8-bit, 16-bit, 32-bit, 64-bit binary data and

NULL-terminated respectively.

names contains a set of names seperated by ":". The names

contains in names should match the format tokens contains in fmt

register_sys_event is used to register the newly added

event hooks. For example, supposing you want to add a new event

hook to trace the entry of sys_open, and you want this event hook

to log the fd, flag and mode paremeters for you. You should add:

register_sys_event(GROUP_IOSYSCALL,

HOOKID_IOSYSCALL_OPEN_ENTRY,

"STRING:INT32:INT32", "filename:flags:mode")

into the function register_sys_events in LKET/regis

ter_event.stp

register_user_event is used for user to add extra trace

data for a event hook. See the section CUSTOMIZED TRACE DATA for

more detail

CUSTOMIZED TRACE DATA

LKET defines a set of event hooks and will log the prede

fined trace data for you, but what if you want to trace extra da

ta for that event?

LKET provides a way to do this without modifying the codes

in the tapset of that event hook. You can simply use printf to

trace extra data. For example, supposing you want to trace

sk_buff->mac_len and sk_buff->priority besides the sk_buff->len,

sk_buff->protocol and sk_buff->truesize for the netdev event

hooks:

probe register_event
{

register_user_event(GROUP_NETDEV, HOOKID_NET

DEV_TRANSMIT,

"INT32:INT32", "mac_len:priority")

}
probe addevent.netdev.transmit
{

printf("%4b%4b", $skb->mac_len, $skb->priority)

}

EXAMPLES

Here are some examples of using LKET:

To turn on all event hooks:

stap -e "probe addevent.* {}" -bM

To probe syscall:

stap -e "probe addevent.syscall {}" -bM

To only probe syscall.entry:

stap -e "probe addevent.syscall.entry {}" -bM

To probe netdev transmition and log extra data of mac_len

and priority:

stap -e "probe addevent.netdev.transmit { printf(

SEE ALSO

stap(1) lket-b2a(5)

IBM 2006-10-28

LKET(5)