Matthew Veety




iomsg protocol



The iomsg protocol is a protocol to allow reading and writing of data
off of a pool of disks. The protocol was designed to be easily
implemented in concurrent way using whatever physical medium is
available. Currently the disk manager dm(8) uses it internally to
serialize access to file descriptors, to generalize access to the
devices, both physical and virtual, and to allow access across the
network to the device pool.

An iomsg has a few fields:

enum { // for type
IOMSGREAD = 1,
IOMSGWRITE = 2,
IOMSGOPEN = 3,
IOMSGCLOSE = 4,
IOMSGSYNC = 5,
IOMSGREPLY = 100,
IOMSGERROR = 101,
};
struct IOMsg {
int type;
long len;
void data[];
vlong offset;
};

The type field is the type of iomsg it is: read, write, sync, open,
close, reply, or error. The data is a freeform field which can be
data to be written (IOMSGWRITE), data that has been read (IOMSGREPLY),
or what to open (IOMSGOPEN). The len field is either how long the
data field is or how long the data field should be in the reply. The
offset is the read or write offset in terms of the start of the
channel (see: read(2)).

IOMsg's go across and are bound to a channel. This channel has a
source/sink at one end where data goes or comes from and the client at
the other where data is used or produced. In the dm(8) analogy, the
disk pool is the source/sink and the client is the mount server at the
other end. A channel can only have a single source/sink but many
clients, but in practice normally only have one client because of
inherent issues with serializing access to filesystems and keeping the
ODS consistent in Plan 9 filesystems.

We have had much success in using iomsg with multiple different
transmission layers. Obviously the best performance exists on our
infiniband links (which we've found can even be faster then
libthread's Channel type), but we get pretty good performance(tm)
using gigabit to our large memory cpu/fileserver that actually handles
the archival filesystems that are on the disk server. Two of our
group members use xdm(8) at home on their personal systems with good
performance over pipes serving normal home things like media files,
music, and such. There are obvious deficiencies in our
implementation, specifically in terms of having many clients using
many links. xdm(8) improves some of this over dm(8), but for most
users dm(8) will be good enough as they won't be doing stupid things,
serving massive files (Nik's porn collection), or having ~1000 clients
slamming a dm(8) server.


Messages:

IOMSGREAD:
IOMsg {
type = IOMSGREAD
data = nil
len = length to be read
offset = offset from 0 of the read
}
IOMsg success {
type = IOMSGREPLY
data = data read
len = amount read
offset = undefined
}
IOMsg error {
type = IOMSGERROR
data = errstr
len = length of data
offset = status number
}

IOMSGWRITE:
IOMsg {
type = IOMSGWRITE
data = data to be written
len = length of data
offset = offset from 0 of the write
}
IOMsg success {
type = IOMSGREPLY
data = nil
len = amount writen
offset = undefined
}
IOMsg error {
type = IOMSGERROR
data = errstr
len = length of data
offset = status number
}

IOMSGOPEN:
IOMsg {
type = IOMSGOPEN
data = string defining what to open
len, offset = undefined
}
IOMsg success {
type = IOMSGREPLY
data, len = zeroed
offset = undefined
}
IOMsg error {
type = IOMSGERROR
data = errstr
len = length of data
offset = status number
}

IOMSGCLOSE:
IOMsg {
type = IOMSGCLOSE
data, len = zeroed
offset = undefined
}
no reply

IOMSGSYNC:
IOMsg {
type = IOMSGSYNC
data, len = zeroed
offset = undefined
}
no reply


EOF