|
160
|
1 Filesystem
|
|
|
2 ==========
|
|
|
3
|
|
|
4 Simple filesystem read/write is achieved using the ``uv_fs_*`` functions and the
|
|
|
5 ``uv_fs_t`` struct.
|
|
|
6
|
|
|
7 .. note::
|
|
|
8
|
|
|
9 The libuv filesystem operations are different from :doc:`socket operations
|
|
|
10 <networking>`. Socket operations use the non-blocking operations provided
|
|
|
11 by the operating system. Filesystem operations use blocking functions
|
|
|
12 internally, but invoke these functions in a `thread pool`_ and notify
|
|
|
13 watchers registered with the event loop when application interaction is
|
|
|
14 required.
|
|
|
15
|
|
|
16 .. _thread pool: https://docs.libuv.org/en/v1.x/threadpool.html#thread-pool-work-scheduling
|
|
|
17
|
|
|
18 All filesystem functions have two forms - *synchronous* and *asynchronous*.
|
|
|
19
|
|
|
20 The *synchronous* forms automatically get called (and **block**) if the
|
|
|
21 callback is null. The return value of functions is a :ref:`libuv error code
|
|
|
22 <libuv-error-handling>`. This is usually only useful for synchronous calls.
|
|
|
23 The *asynchronous* form is called when a callback is passed and the return
|
|
|
24 value is 0.
|
|
|
25
|
|
|
26 Reading/Writing files
|
|
|
27 ---------------------
|
|
|
28
|
|
|
29 A file descriptor is obtained using
|
|
|
30
|
|
|
31 .. code-block:: c
|
|
|
32
|
|
|
33 int uv_fs_open(uv_loop_t* loop, uv_fs_t* req, const char* path, int flags, int mode, uv_fs_cb cb)
|
|
|
34
|
|
|
35 ``flags`` and ``mode`` are standard
|
|
|
36 `Unix flags <https://man7.org/linux/man-pages/man2/open.2.html>`_.
|
|
|
37 libuv takes care of converting to the appropriate Windows flags.
|
|
|
38
|
|
|
39 File descriptors are closed using
|
|
|
40
|
|
|
41 .. code-block:: c
|
|
|
42
|
|
|
43 int uv_fs_close(uv_loop_t* loop, uv_fs_t* req, uv_file file, uv_fs_cb cb)
|
|
|
44
|
|
|
45
|
|
|
46 Filesystem operation callbacks have the signature:
|
|
|
47
|
|
|
48 .. code-block:: c
|
|
|
49
|
|
|
50 void callback(uv_fs_t* req);
|
|
|
51
|
|
|
52 Let's see a simple implementation of ``cat``. We start with registering
|
|
|
53 a callback for when the file is opened:
|
|
|
54
|
|
|
55 .. rubric:: uvcat/main.c - opening a file
|
|
|
56 .. literalinclude:: ../../code/uvcat/main.c
|
|
|
57 :language: c
|
|
|
58 :linenos:
|
|
|
59 :lines: 41-53
|
|
|
60 :emphasize-lines: 4, 6-7
|
|
|
61
|
|
|
62 The ``result`` field of a ``uv_fs_t`` is the file descriptor in case of the
|
|
|
63 ``uv_fs_open`` callback. If the file is successfully opened, we start reading it.
|
|
|
64
|
|
|
65 .. rubric:: uvcat/main.c - read callback
|
|
|
66 .. literalinclude:: ../../code/uvcat/main.c
|
|
|
67 :language: c
|
|
|
68 :linenos:
|
|
|
69 :lines: 26-39
|
|
|
70 :emphasize-lines: 2,8,12
|
|
|
71
|
|
|
72 In the case of a read call, you should pass an *initialized* buffer which will
|
|
|
73 be filled with data before the read callback is triggered. The ``uv_fs_*``
|
|
|
74 operations map almost directly to certain POSIX functions, so EOF is indicated
|
|
|
75 in this case by ``result`` being 0. In the case of streams or pipes, the
|
|
|
76 ``UV_EOF`` constant would have been passed as a status instead.
|
|
|
77
|
|
|
78 Here you see a common pattern when writing asynchronous programs. The
|
|
|
79 ``uv_fs_close()`` call is performed synchronously. *Usually tasks which are
|
|
|
80 one-off, or are done as part of the startup or shutdown stage are performed
|
|
|
81 synchronously, since we are interested in fast I/O when the program is going
|
|
|
82 about its primary task and dealing with multiple I/O sources*. For solo tasks
|
|
|
83 the performance difference usually is negligible and may lead to simpler code.
|
|
|
84
|
|
|
85 Filesystem writing is similarly simple using ``uv_fs_write()``. *Your callback
|
|
|
86 will be triggered after the write is complete*. In our case the callback
|
|
|
87 simply drives the next read. Thus read and write proceed in lockstep via
|
|
|
88 callbacks.
|
|
|
89
|
|
|
90 .. rubric:: uvcat/main.c - write callback
|
|
|
91 .. literalinclude:: ../../code/uvcat/main.c
|
|
|
92 :language: c
|
|
|
93 :linenos:
|
|
|
94 :lines: 17-24
|
|
|
95 :emphasize-lines: 6
|
|
|
96
|
|
|
97 .. warning::
|
|
|
98
|
|
|
99 Due to the way filesystems and disk drives are configured for performance,
|
|
|
100 a write that 'succeeds' may not be committed to disk yet.
|
|
|
101
|
|
|
102 We set the dominos rolling in ``main()``:
|
|
|
103
|
|
|
104 .. rubric:: uvcat/main.c
|
|
|
105 .. literalinclude:: ../../code/uvcat/main.c
|
|
|
106 :language: c
|
|
|
107 :linenos:
|
|
|
108 :lines: 55-
|
|
|
109 :emphasize-lines: 2
|
|
|
110
|
|
|
111 .. warning::
|
|
|
112
|
|
|
113 The ``uv_fs_req_cleanup()`` function must always be called on filesystem
|
|
|
114 requests to free internal memory allocations in libuv.
|
|
|
115
|
|
|
116 Filesystem operations
|
|
|
117 ---------------------
|
|
|
118
|
|
|
119 All the standard filesystem operations like ``unlink``, ``rmdir``, ``stat`` are
|
|
|
120 supported asynchronously and have intuitive argument order. They follow the
|
|
|
121 same patterns as the read/write/open calls, returning the result in the
|
|
|
122 ``uv_fs_t.result`` field. The full list:
|
|
|
123
|
|
|
124 .. rubric:: Filesystem operations
|
|
|
125 .. code-block:: c
|
|
|
126
|
|
|
127 int uv_fs_close(uv_loop_t* loop, uv_fs_t* req, uv_file file, uv_fs_cb cb);
|
|
|
128 int uv_fs_open(uv_loop_t* loop, uv_fs_t* req, const char* path, int flags, int mode, uv_fs_cb cb);
|
|
|
129 int uv_fs_read(uv_loop_t* loop, uv_fs_t* req, uv_file file, const uv_buf_t bufs[], unsigned int nbufs, int64_t offset, uv_fs_cb cb);
|
|
|
130 int uv_fs_unlink(uv_loop_t* loop, uv_fs_t* req, const char* path, uv_fs_cb cb);
|
|
|
131 int uv_fs_write(uv_loop_t* loop, uv_fs_t* req, uv_file file, const uv_buf_t bufs[], unsigned int nbufs, int64_t offset, uv_fs_cb cb);
|
|
|
132 int uv_fs_copyfile(uv_loop_t* loop, uv_fs_t* req, const char* path, const char* new_path, int flags, uv_fs_cb cb);
|
|
|
133 int uv_fs_mkdir(uv_loop_t* loop, uv_fs_t* req, const char* path, int mode, uv_fs_cb cb);
|
|
|
134 int uv_fs_mkdtemp(uv_loop_t* loop, uv_fs_t* req, const char* tpl, uv_fs_cb cb);
|
|
|
135 int uv_fs_mkstemp(uv_loop_t* loop, uv_fs_t* req, const char* tpl, uv_fs_cb cb);
|
|
|
136 int uv_fs_rmdir(uv_loop_t* loop, uv_fs_t* req, const char* path, uv_fs_cb cb);
|
|
|
137 int uv_fs_scandir(uv_loop_t* loop, uv_fs_t* req, const char* path, int flags, uv_fs_cb cb);
|
|
|
138 int uv_fs_scandir_next(uv_fs_t* req, uv_dirent_t* ent);
|
|
|
139 int uv_fs_opendir(uv_loop_t* loop, uv_fs_t* req, const char* path, uv_fs_cb cb);
|
|
|
140 int uv_fs_readdir(uv_loop_t* loop, uv_fs_t* req, uv_dir_t* dir, uv_fs_cb cb);
|
|
|
141 int uv_fs_closedir(uv_loop_t* loop, uv_fs_t* req, uv_dir_t* dir, uv_fs_cb cb);
|
|
|
142 int uv_fs_stat(uv_loop_t* loop, uv_fs_t* req, const char* path, uv_fs_cb cb);
|
|
|
143 int uv_fs_fstat(uv_loop_t* loop, uv_fs_t* req, uv_file file, uv_fs_cb cb);
|
|
|
144 int uv_fs_rename(uv_loop_t* loop, uv_fs_t* req, const char* path, const char* new_path, uv_fs_cb cb);
|
|
|
145 int uv_fs_fsync(uv_loop_t* loop, uv_fs_t* req, uv_file file, uv_fs_cb cb);
|
|
|
146 int uv_fs_fdatasync(uv_loop_t* loop, uv_fs_t* req, uv_file file, uv_fs_cb cb);
|
|
|
147 int uv_fs_ftruncate(uv_loop_t* loop, uv_fs_t* req, uv_file file, int64_t offset, uv_fs_cb cb);
|
|
|
148 int uv_fs_sendfile(uv_loop_t* loop, uv_fs_t* req, uv_file out_fd, uv_file in_fd, int64_t in_offset, size_t length, uv_fs_cb cb);
|
|
|
149 int uv_fs_access(uv_loop_t* loop, uv_fs_t* req, const char* path, int mode, uv_fs_cb cb);
|
|
|
150 int uv_fs_chmod(uv_loop_t* loop, uv_fs_t* req, const char* path, int mode, uv_fs_cb cb);
|
|
|
151 int uv_fs_utime(uv_loop_t* loop, uv_fs_t* req, const char* path, double atime, double mtime, uv_fs_cb cb);
|
|
|
152 int uv_fs_futime(uv_loop_t* loop, uv_fs_t* req, uv_file file, double atime, double mtime, uv_fs_cb cb);
|
|
|
153 int uv_fs_lutime(uv_loop_t* loop, uv_fs_t* req, const char* path, double atime, double mtime, uv_fs_cb cb);
|
|
|
154 int uv_fs_lstat(uv_loop_t* loop, uv_fs_t* req, const char* path, uv_fs_cb cb);
|
|
|
155 int uv_fs_link(uv_loop_t* loop, uv_fs_t* req, const char* path, const char* new_path, uv_fs_cb cb);
|
|
|
156 int uv_fs_symlink(uv_loop_t* loop, uv_fs_t* req, const char* path, const char* new_path, int flags, uv_fs_cb cb);
|
|
|
157 int uv_fs_readlink(uv_loop_t* loop, uv_fs_t* req, const char* path, uv_fs_cb cb);
|
|
|
158 int uv_fs_realpath(uv_loop_t* loop, uv_fs_t* req, const char* path, uv_fs_cb cb);
|
|
|
159 int uv_fs_fchmod(uv_loop_t* loop, uv_fs_t* req, uv_file file, int mode, uv_fs_cb cb);
|
|
|
160 int uv_fs_chown(uv_loop_t* loop, uv_fs_t* req, const char* path, uv_uid_t uid, uv_gid_t gid, uv_fs_cb cb);
|
|
|
161 int uv_fs_fchown(uv_loop_t* loop, uv_fs_t* req, uv_file file, uv_uid_t uid, uv_gid_t gid, uv_fs_cb cb);
|
|
|
162 int uv_fs_lchown(uv_loop_t* loop, uv_fs_t* req, const char* path, uv_uid_t uid, uv_gid_t gid, uv_fs_cb cb);
|
|
|
163 int uv_fs_statfs(uv_loop_t* loop, uv_fs_t* req, const char* path, uv_fs_cb cb);
|
|
|
164
|
|
|
165
|
|
|
166 .. _buffers-and-streams:
|
|
|
167
|
|
|
168 Buffers and Streams
|
|
|
169 -------------------
|
|
|
170
|
|
|
171 The basic I/O handle in libuv is the stream (``uv_stream_t``). TCP sockets, UDP
|
|
|
172 sockets, and pipes for file I/O and IPC are all treated as stream subclasses.
|
|
|
173
|
|
|
174 Streams are initialized using custom functions for each subclass, then operated
|
|
|
175 upon using
|
|
|
176
|
|
|
177 .. code-block:: c
|
|
|
178
|
|
|
179 int uv_read_start(uv_stream_t*, uv_alloc_cb alloc_cb, uv_read_cb read_cb);
|
|
|
180 int uv_read_stop(uv_stream_t*);
|
|
|
181 int uv_write(uv_write_t* req, uv_stream_t* handle,
|
|
|
182 const uv_buf_t bufs[], unsigned int nbufs, uv_write_cb cb);
|
|
|
183
|
|
|
184 The stream based functions are simpler to use than the filesystem ones and
|
|
|
185 libuv will automatically keep reading from a stream when ``uv_read_start()`` is
|
|
|
186 called once, until ``uv_read_stop()`` is called.
|
|
|
187
|
|
|
188 The discrete unit of data is the buffer -- ``uv_buf_t``. This is simply
|
|
|
189 a collection of a pointer to bytes (``uv_buf_t.base``) and the length
|
|
|
190 (``uv_buf_t.len``). The ``uv_buf_t`` is lightweight and passed around by value.
|
|
|
191 What does require management is the actual bytes, which have to be allocated
|
|
|
192 and freed by the application.
|
|
|
193
|
|
|
194 .. ERROR::
|
|
|
195
|
|
|
196 **THIS PROGRAM DOES NOT ALWAYS WORK, NEED SOMETHING BETTER**
|
|
|
197
|
|
|
198 To demonstrate streams we will need to use ``uv_pipe_t``. This allows streaming
|
|
|
199 local files [#]_. Here is a simple tee utility using libuv. Doing all operations
|
|
|
200 asynchronously shows the power of evented I/O. The two writes won't block each
|
|
|
201 other, but we have to be careful to copy over the buffer data to ensure we don't
|
|
|
202 free a buffer until it has been written.
|
|
|
203
|
|
|
204 The program is to be executed as::
|
|
|
205
|
|
|
206 ./uvtee <output_file>
|
|
|
207
|
|
|
208 We start off opening pipes on the files we require. libuv pipes to a file are
|
|
|
209 opened as bidirectional by default.
|
|
|
210
|
|
|
211 .. rubric:: uvtee/main.c - read on pipes
|
|
|
212 .. literalinclude:: ../../code/uvtee/main.c
|
|
|
213 :language: c
|
|
|
214 :linenos:
|
|
|
215 :lines: 62-80
|
|
|
216 :emphasize-lines: 4,5,15
|
|
|
217
|
|
|
218 The third argument of ``uv_pipe_init()`` should be set to 1 for IPC using named
|
|
|
219 pipes. This is covered in :doc:`processes`. The ``uv_pipe_open()`` call
|
|
|
220 associates the pipe with the file descriptor, in this case ``0`` (standard
|
|
|
221 input).
|
|
|
222
|
|
|
223 We start monitoring ``stdin``. The ``alloc_buffer`` callback is invoked as new
|
|
|
224 buffers are required to hold incoming data. ``read_stdin`` will be called with
|
|
|
225 these buffers.
|
|
|
226
|
|
|
227 .. rubric:: uvtee/main.c - reading buffers
|
|
|
228 .. literalinclude:: ../../code/uvtee/main.c
|
|
|
229 :language: c
|
|
|
230 :linenos:
|
|
|
231 :lines: 19-22,44-60
|
|
|
232
|
|
|
233 The standard ``malloc`` is sufficient here, but you can use any memory allocation
|
|
|
234 scheme. For example, node.js uses its own slab allocator which associates
|
|
|
235 buffers with V8 objects.
|
|
|
236
|
|
|
237 The read callback ``nread`` parameter is less than 0 on any error. This error
|
|
|
238 might be EOF, in which case we close all the streams, using the generic close
|
|
|
239 function ``uv_close()`` which deals with the handle based on its internal type.
|
|
|
240 Otherwise ``nread`` is a non-negative number and we can attempt to write that
|
|
|
241 many bytes to the output streams. Finally remember that buffer allocation and
|
|
|
242 deallocation is application responsibility, so we free the data.
|
|
|
243
|
|
|
244 The allocation callback may return a buffer with length zero if it fails to
|
|
|
245 allocate memory. In this case, the read callback is invoked with error
|
|
|
246 UV_ENOBUFS. libuv will continue to attempt to read the stream though, so you
|
|
|
247 must explicitly call ``uv_close()`` if you want to stop when allocation fails.
|
|
|
248
|
|
|
249 The read callback may be called with ``nread = 0``, indicating that at this
|
|
|
250 point there is nothing to be read. Most applications will just ignore this.
|
|
|
251
|
|
|
252 .. rubric:: uvtee/main.c - Write to pipe
|
|
|
253 .. literalinclude:: ../../code/uvtee/main.c
|
|
|
254 :language: c
|
|
|
255 :linenos:
|
|
|
256 :lines: 9-13,23-42
|
|
|
257
|
|
|
258 ``write_data()`` makes a copy of the buffer obtained from read. This buffer
|
|
|
259 does not get passed through to the write callback trigged on write completion. To
|
|
|
260 get around this we wrap a write request and a buffer in ``write_req_t`` and
|
|
|
261 unwrap it in the callbacks. We make a copy so we can free the two buffers from
|
|
|
262 the two calls to ``write_data`` independently of each other. While acceptable
|
|
|
263 for a demo program like this, you'll probably want smarter memory management,
|
|
|
264 like reference counted buffers or a pool of buffers in any major application.
|
|
|
265
|
|
|
266 .. WARNING::
|
|
|
267
|
|
|
268 If your program is meant to be used with other programs it may knowingly or
|
|
|
269 unknowingly be writing to a pipe. This makes it susceptible to `aborting on
|
|
|
270 receiving a SIGPIPE`_. It is a good idea to insert::
|
|
|
271
|
|
|
272 signal(SIGPIPE, SIG_IGN)
|
|
|
273
|
|
|
274 in the initialization stages of your application.
|
|
|
275
|
|
|
276 .. _aborting on receiving a SIGPIPE: http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod#The_special_problem_of_SIGPIPE
|
|
|
277
|
|
|
278 File change events
|
|
|
279 ------------------
|
|
|
280
|
|
|
281 All modern operating systems provide APIs to put watches on individual files or
|
|
|
282 directories and be informed when the files are modified. libuv wraps common
|
|
|
283 file change notification libraries [#fsnotify]_. This is one of the more
|
|
|
284 inconsistent parts of libuv. File change notification systems are themselves
|
|
|
285 extremely varied across platforms so getting everything working everywhere is
|
|
|
286 difficult. To demonstrate, I'm going to build a simple utility which runs
|
|
|
287 a command whenever any of the watched files change::
|
|
|
288
|
|
|
289 ./onchange <command> <file1> [file2] ...
|
|
|
290
|
|
|
291 .. note::
|
|
|
292
|
|
|
293 Currently this example only works on OSX and Windows.
|
|
|
294 Refer to the `notes of uv_fs_event_start`_ function.
|
|
|
295
|
|
|
296 .. _notes of uv_fs_event_start: https://docs.libuv.org/en/v1.x/fs_event.html#c.uv_fs_event_start
|
|
|
297
|
|
|
298 The file change notification is started using ``uv_fs_event_init()``:
|
|
|
299
|
|
|
300 .. rubric:: onchange/main.c - The setup
|
|
|
301 .. literalinclude:: ../../code/onchange/main.c
|
|
|
302 :language: c
|
|
|
303 :linenos:
|
|
|
304 :lines: 26-
|
|
|
305 :emphasize-lines: 15
|
|
|
306
|
|
|
307 The third argument is the actual file or directory to monitor. The last
|
|
|
308 argument, ``flags``, can be:
|
|
|
309
|
|
|
310 .. code-block:: c
|
|
|
311
|
|
|
312 /*
|
|
|
313 * Flags to be passed to uv_fs_event_start().
|
|
|
314 */
|
|
|
315 enum uv_fs_event_flags {
|
|
|
316 UV_FS_EVENT_WATCH_ENTRY = 1,
|
|
|
317 UV_FS_EVENT_STAT = 2,
|
|
|
318 UV_FS_EVENT_RECURSIVE = 4
|
|
|
319 };
|
|
|
320
|
|
|
321 ``UV_FS_EVENT_WATCH_ENTRY`` and ``UV_FS_EVENT_STAT`` don't do anything (yet).
|
|
|
322 ``UV_FS_EVENT_RECURSIVE`` will start watching subdirectories as well on
|
|
|
323 supported platforms.
|
|
|
324
|
|
|
325 The callback will receive the following arguments:
|
|
|
326
|
|
|
327 #. ``uv_fs_event_t *handle`` - The handle. The ``path`` field of the handle
|
|
|
328 is the file on which the watch was set.
|
|
|
329 #. ``const char *filename`` - If a directory is being monitored, this is the
|
|
|
330 file which was changed. Only non-``null`` on Linux and Windows. May be ``null``
|
|
|
331 even on those platforms.
|
|
|
332 #. ``int events`` - one of ``UV_RENAME`` or ``UV_CHANGE``, or a bitwise OR of
|
|
|
333 both.
|
|
|
334 #. ``int status`` - If ``status < 0``, there is an :ref:`libuv error<libuv-error-handling>`.
|
|
|
335
|
|
|
336 In our example we simply print the arguments and run the command using
|
|
|
337 ``system()``.
|
|
|
338
|
|
|
339 .. rubric:: onchange/main.c - file change notification callback
|
|
|
340 .. literalinclude:: ../../code/onchange/main.c
|
|
|
341 :language: c
|
|
|
342 :linenos:
|
|
|
343 :lines: 9-24
|
|
|
344
|
|
|
345 ----
|
|
|
346
|
|
|
347 .. [#fsnotify] inotify on Linux, FSEvents on Darwin, kqueue on BSDs,
|
|
|
348 ReadDirectoryChangesW on Windows, event ports on Solaris, unsupported on Cygwin
|
|
|
349 .. [#] see :ref:`pipes`
|
