1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
|
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="generator" content="AsciiDoc 9.0.0rc1">
<title>Rust based vhost-user I2C backend</title>
</head>
<body>
<h1>Rust based vhost-user I2C backend</h1>
<p>
</p>
<a name="preamble"></a>
<p>There is a growing trend towards virtualization in areas other than the
traditional server environment. The server environment is uniform in nature, but
as we move towards a richer ecosystem in automotive, medical, general mobile and
the IoT spaces, more device abstractions, and way richer organizations are
needed. <a href="https://www.linaro.org/projects/#automotive_STR">Linaro’s Project
Stratos</a> is working towards developing hypervisor agnostic abstract devices
leveraging virtio and extending hypervisor interfaces and standards to allow all
architectures.</p>
<p>The Virtual Input/Output device (Virtio) standard provides an open interface for
guest virtual machines (VMs) to access simplified "virtual" devices, such as
network adapters and block devices, in a paravirtualized environment. Virtio
provides a straightforward, efficient, standard and extensible mechanism for
virtual devices, rather than a per-environment or per-OS mechanism.</p>
<p>The backend (BE) virtio driver, implemented in the hypervisor running on the host,
exposes the virtio device to the guest OS through a transport method, like PCI
or MMIO. The virtio device, by design, looks like a physical device to the guest
OS, which implements a frontend (FE) virtio driver compatible with the virtio
device exposed by the hypervisor. The virtio device and driver communicate based
on a set of predefined protocols as defined by the
<a href="https://github.com/oasis-tcs/virtio-spec">virtio specification</a>, which is
maintained by <a href="https://www.oasis-open.org/org/">OASIS</a>. The FE driver can
implement zero or more Virtual queues (virtqueues), as defined by the virtio
specification. The virtqueues are the mechanism of bulk data transport between
FE (guest) and BE (host) drivers. These are normally implemented as standard
ring buffers in the guest physical memory by the FE drivers. The BE drivers
parse the virtqueues to obtain the request descriptors, process them and queue
the response descriptors back to the virtqueue.</p>
<p>The FE virtio driver at the guest and the virtio specification are normally
independent of where the virtqueue processing happens at the host, in-kernel or
userspace. The vhost protocol allows the virtio virtqueue processing at the
host to be offloaded to another element, a user process or a kernel module. The
vhost protocol when implemented in userspace is called as "vhost-user". Since
Linaro’s Project Stratos is targeting hypervisor agnostic BE drivers, engineers
at Linaro decided to work over the existing vhost-user protocol. This article
focuses on the Rust based vhost-user implementation for I2C devices.</p>
<hr>
<h2><a name="_virtio_i2c_specification"></a>Virtio I2C Specification</h2>
<p>The Virtio
<a href="https://github.com/oasis-tcs/virtio-spec/blob/master/virtio-i2c.tex">specification</a>
for I2C and the Linux
<a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/i2c/busses/i2c-virtio.c">i2c-virtio</a>
driver are upstreamed by Jie Deng (Intel), who tested his work with the
<a href="https://projectacrn.org">ACRN</a> hypervisor for IoT development. Both
specification and driver received updates later on by Viresh Kumar (Linaro), to
improve buffer management and allow zero-length transactions. Lets go through
the I2C virtio specification briefly.</p>
<p>virtio-i2c is a virtual I2C adapter device, which provides a way to flexibly
organize and use the host I2C controlled devices from the guest. All
communication between the FE and BE drivers happens over the "requestq"
virtqueue. It is also mandatory for both the sides to implement the
<code>VIRTIO_I2C_F_ZERO_LENGTH_REQUEST</code> feature, which allows zero-length transfers
(like SMBus Quick) to take place. The I2C requests always originate at the guest
FE driver, where the FE driver puts one or more I2C requests, represented by the
<code>struct virtio_i2c_req</code>, on the requestq virtqueue. The I2C requests may or may
not be be interdependent. If multiple requests are received together, then the
host BE driver must process the requests in the order they are received on the
virtqueue.</p>
<table border="0" bgcolor="#e8e8e8" width="100%" cellpadding="4"><tr><td>
<pre><code>struct virtio_i2c_req {
struct virtio_i2c_out_hdr out_hdr;
u8 buf[];
struct virtio_i2c_in_hdr in_hdr;
};</code></pre>
</td></tr></table>
<p>Each I2C virtio request consists of an <code>out_hdr</code> (set by the FE driver), followed by
an optional buffer of some length (set by FE or BE driver based on if the
transaction is write or read), followed by an <code>in_hdr</code> (set by the BE driver). The
buffer is not sent for zero-length requests, like for the SMBus Quick command
where no data is required to be sent or received.</p>
<table border="0" bgcolor="#e8e8e8" width="100%" cellpadding="4"><tr><td>
<pre><code>struct virtio_i2c_out_hdr {
le16 addr;
le16 padding;
le32 flags;
};</code></pre>
</td></tr></table>
<p>The <code>out_hdr</code> is represented by the <code>struct virtio_i2c_out_hdr</code>. The <code>addr</code>
field of the header is the address of the I2C controlled device. Both 7-bit and
10-bit address modes are supported by the specification (though only 7-bit mode
is supported by the current implementation of the Linux FE driver). The <code>flags</code>
field is used to mark a request "Read or write" (<code>VIRTIO_I2C_FLAGS_M_RD</code> (bit
1)) or to show dependency between multiple requests
(<code>VIRTIO_I2C_FLAGS_FAIL_NEXT</code> (bit 0)).</p>
<p>As described earlier, the <code>buf</code> is optional. For "write" transactions, it is
pre-filled by the FE driver and read by the BE driver. For "read" transactions,
it is filled by the BE driver and read by the FE driver after the response is
received.</p>
<table border="0" bgcolor="#e8e8e8" width="100%" cellpadding="4"><tr><td>
<pre><code>struct virtio_i2c_in_hdr {
u8 status;
};</code></pre>
</td></tr></table>
<p>The <code>in_hdr</code> is represented by the <code>struct virtio_i2c_in_hdr</code> and is used by the
host BE driver to notify the guest with the status of the transfer with
<code>VIRTIO_I2C_MSG_OK</code> or <code>VIRTIO_I2C_MSG_ERR</code>.</p>
<p>Please refer the Virtio I2C
<a href="https://github.com/oasis-tcs/virtio-spec/blob/master/virtio-i2c.tex">specification</a>
of more details.</p>
<hr>
<h2><a name="_rust_based_i2c_backend"></a>Rust based I2C backend</h2>
<p>Rust is the next big thing disrupting the Linux world and most of us are already
aware of the <a href="https://github.com/Rust-for-Linux">Rust for Linux</a> project
slowly making its way into the Linux kernel. Rust is a multi-paradigm,
general-purpose programming language designed for performance and safety. It
brings a lot of benefits to the table, especially
<a href="https://en.wikipedia.org/wiki/Memory_safety">memory-safety</a> and safe
<a href="https://en.wikipedia.org/wiki/Concurrency_(computer_science)">concurrency</a>.
It was an easy choice to pick for developing hypervisor agnostic I2C BE driver.</p>
<p>The <a href="https://github.com/rust-vmm">rust-vmm</a> project, an open-source
initiative, was started back in late 2018, with the aim to share virtualization
packages. The rust-vmm project lets one build custom
<a href="https://en.wikipedia.org/wiki/Hypervisor">Virtual Machine Monitors (VMMs)
and hypervisors</a>. This empowers other projects to quickly develop virtualization
solutions, by reusing the components provided by rust-vmm, and better focus on
key differentiators of their products. The rust-vmm project is organized as a
shared ownership project that so far includes contributions from Alibaba, AWS,
Cloud Base, Google, Intel, Linaro, Red Hat and other individual contributors.
The components provided by rust-vmm are already used by several projects, like
Amazon’s <a href="https://github.com/firecracker-microvm/firecracker">Firecracker</a>
and Intel’s <a href="https://github.com/cloud-hypervisor/cloud-hypervisor">Cloud
Hypervisor</a>. The rust-vmm project currently roughly 30 repositories (or Rust
crates, equivalent of a C library), where each crate plays a special role in the
development of a fully functioning VMM.</p>
<p>One such component provided by the rust-vmm project is the
<a href="https://crates.io/crates/vhost-user-backend">vhost-user-backend</a> crate,
which has recently made its way to <a href="https://crates.io/">crates.io</a>, the Rust
community’s crate registry. The vhost-user-backend crate provides a framework to
implement the vhost-user backend services. It provides necessary public APIs to
support vhost-user backends, like a daemon control object (<code>VhostUserDaemon</code>) to
start and stop the service daemon, a vhost-user backend trait
(<code>VhostUserBackendMut</code>) to handle vhost-user control messages and virtio
messages, and a vring access trait (<code>VringT</code>) to access virtio queues.</p>
<p>A separate Rust workspace,
<a href="https://github.com/rust-vmm/vhost-device">vhost-device</a>, is recently created
in the rust-vmm project, to host per-device vhost-user backend crates. The only
crate merged there as for now is for the I2C device, while there are others
getting developed and reviewed as we speak, like GPIO, RNG, VSOCK, SCSI, and
<a href="https://en.wikipedia.org/wiki/Replay_Protected_Memory_Block">RPMB</a>.</p>
<p>The I2C vhost-device binary-crate (generates an executable upon build),
developed by Viresh Kumar (Linaro), supports sharing host I2C busses (Adaptors)
and client devices with multiple guest VMs at the same time with a single
instance of an always running backend daemon. Once the vhost-device crate is
compiled with <code>cargo build --release</code>, it generates the
<code>target/release/vhost-device-i2c</code> executable. The <code>vhost-device-i2c</code> daemon
communicates with guest VMs over Unix domain sockets, a unique socket for each
VM.</p>
<p>The daemon accepts three arguments:</p>
<ul>
<li>
<p>
-s, --socket-path: Path of the vhost-user Unix domain sockets. This is
suffixed with 0,1,2..socket_count-1 by the daemon to obtain actual socket
paths.
</p>
</li>
<li>
<p>
-c, --socket-count: Number of sockets (guests) to connect to. This parameter
is optional and defaults to 1.
</p>
</li>
<li>
<p>
-l, --device-list: List of I2C bus and clients in the format
<bus>:<client_addr>[:<client_addr>][,<bus>:<client_addr>[:<client_addr>]]
</p>
</li>
</ul>
<p>As an example, consider the following command:</p>
<table border="0" bgcolor="#e8e8e8" width="100%" cellpadding="4"><tr><td>
<pre><code>./vhost-device-i2c -s ~/i2c.sock -c 6 -l 6:32:41,9:37:6</code></pre>
</td></tr></table>
<p>This will start the I2C backend daemon, which will create 6 Unix domain sockets
(<sub>/i2c.sock0, .., </sub>/i2c.sock5), in order to communicate with 6 guest VMs, where
communication with each VM happens in parallel with the help of a separate
native OS thread. Once the threads are created by the daemon, the threads wait
for a VM to start communicating on the thread’s designated socket. Later, when a
VM shuts down, the respective thread starts waiting for a new VM to communicate
on the same socket path. The daemon is also passed a list of host I2C busses and
client devices, which are shared by all the VMs. The daemon can be modified
later on, if required, to allow specific devices to be accessed only by a
particular VM, this feature isn’t added in the current version of the daemon. In
the above example, the devices shared by the host with the daemon are: devices
with address 32 and 41 attached to I2C bus 6, 37 and 6 attached to I2C bus 9.
The daemon extensively validates the device-list at initialization to avoid any
failures later.</p>
<p>The <code>vhost-user-i2c</code> daemon supports both I2C and SMBus protocols, only basic
SMBus commands up to word-transfer. The backend provides the <code>pub trait
I2cDevice</code>, a public Rust trait, which can be implemented for different host
environments to provide access to the underlying I2C busses and devices. This is
currently implemented only for the Linux userspace, where the I2C busses and
devices are accessed via the <code>/dev/i2c-X</code> I2C device files. For the above
example, the backend daemon will look for <code>/dev/i2c-6</code> and <code>/dev/i2c-9</code> device
files. The users may need to load the <code>i2c-dev</code> kernel module, if not loaded
already, for these device files to be available under <code>/dev/</code>. For a different
host environment, like a bare-metal type 1 hypervisor, we need to add another
implementation of the trait depending on how the I2C busses and devices are
accessed.</p>
<p>The <code>vhost-user-i2c</code> backend is truly a hypervisor agnostic solution that works
with any hypervisor which understands the vhost-user protocol. It has been
extensively tested with QEMU for example, with Linux userspace environment. Work
is in progress to make Xen hypervisor vhost-user protocol compatible. Once that
is done, we will be able to use the same <code>vhost-user-i2c</code> executable with both
QEMU and Xen, for example, under the same host environments.</p>
<p>Support for virtio-i2c is already merged in QEMU source, boilerplate stuff to
create the virtio-i2c device in the guest kernel, and the virtio-i2c device can
be created in the guest kernel by adding following command line arguments to
your QEMU command:</p>
<p><code>-chardev socket,path=~/i2c.sock0,id=vi2c -device vhost-user-i2c-device,chardev=vi2c,id=i2c</code></p>
<p></p>
<p></p>
<hr><p><small>
Last updated
2022-01-05 12:58:45 IST
</small></p>
</body>
</html>
|
