1 <meta HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=utf-8">
4 # Copyright Neil Brown ©2015-2019 <neil@brown.name>
5 # May be distributed under terms of GPLv2 - see file:COPYING
8 Edlib - a library for building a document editor
9 ==============================================
11 Edlib is an extensible document editor. It is inspired in part by
12 emacs, both by its strengths and its weaknesses.
14 emacs provides a programming language — E-lisp — for configuring and
15 extending the editor. Edlib doesn't. It allows various pre-existing
16 languages to be used to configure and extend the editor. It does
17 this by providing a library of core editing tools and providing
18 bindings to various languages.
20 At least, that is the plan. At time if writing, edlib only provides
21 bindings for C and Python. Other languages should be fairly easy.
23 The particular value-add of edlib over emacs (apart from the obvious
24 “NIH” issues) is that both document storage and document rendering are
25 fully extensible. A document is not always a text buffer, it could
26 also be a mem-mapped files, a directory, or an internal data
28 Any document can have multiple views, and each view can
29 show very different things: scriptable code is run whenever
30 rendering is required. This should make implementing documents with
31 non-trivial structures a lot easier.
33 Edlib is designed to have well defined abstractions that can be
34 exported to various languages for them to manipulate. They include
35 primarily commands, panes, and marks, and also attributes, documents,
36 displays, events, and keymaps.
41 Commands are the single mechanism by which control is transferred from
42 one part of the editor implementation to another. They provide the
43 only way for code in one implementation language to invoke code
44 written in another language, and they are the preferred way to
45 interact between different modules even in the same language.
47 All commands receive the same types of arguments and produce an integer
48 result. The arguments include two panes (“home” and “focus”),
49 two marks (“mark” and “mark2”),
50 three strings (“key”, “str”, “str2”),
51 two numbers (“num” and “num2”),
52 a co-ordinate pair (“x”, “y”) and two commands (“comm1” and “comm2”).
53 Extra result values can be effected by passing them to a call to
54 the “comm2” argument - i.e the command can be used as a call-back.
56 Each “pane” has a dedicate command which handles messages sent to
57 the pane, as will be described later. Commands can also be passed
58 to other commands, which can either call them directly (like the
59 call-back mentioned above) or store them for later use, or
62 Three of the arguments provided to a command have very special
63 meanings. One of the strings, known as “key”, identifies what action
64 should be performed. The pane handler will normally use this key to
65 select some other command to actually handle the message. Other
66 commands may ignore the key, or use it however they please.
68 When a message is sent to a pane and the handler command is called,
69 the “home” argument is set to the pane that owns the handle, so it acts
70 a bit like the “self” argument in some object-oriented languages.
71 One of the primary uses of “home” is to access “home->data” which is a
72 private data structure owned by the pane. The command associated with
73 a particular pane is typically the only code which can understand the data.
75 Finally the “comm1” argument passed to a command always identifies
76 exactly the command that is being run. “comm1” is a pointer to a
77 structure containing a pointer to the function being called. This
78 structure can be embedded is some other data structure which contains
79 context for the command. This context might be read-only, to refine
80 the behaviour of the command, or read/write to provide storage for the
81 command to use. For example, the call-back commands described earlier
82 are normally embedded in a data structure that can store the extra
85 Apart from the return value of zero which indicates “command not
86 understood”, a command can return a positive result on success or a
87 negative result indicating lack of success. Known error codes include:
89 - Enoarg : missing argument
90 - Einval : something is wrong with the context of the request
91 - Efail : request makes sense, but didn't work
92 - Enosup: request makes sense, but isn't allowed for some reason
93 - Efalse: Not really an error, just a Boolean status
98 A pane combines an optional rectangular area of display with some
99 data storage and some functionality. As such it can receive mouse and
100 keyboard events, can draw on a display, and can send commands to other
103 All panes are arranged as a tree with all but the root having a parent
104 and many having siblings and children. When a pane represents a
105 rectangle of display all children are restricted to just that
106 rectangle or less. Often a child will cover exactly the same area as
107 its parent. In other cases several children will share out the area.
109 An “event” is a set of arguments to a command which is being sent to
110 a pane. Events are often generated at a leaf of the tree of panes
111 (i.e. a pane with no children). They travel up the tree towards the
112 root until they find a pane which can handle them. That pane (or its
113 handler command) might handle the event by generating other events.
114 They will typically start looking for a handler at the same leaf which
115 is available as the “focus” argument. For this reason branches of the
116 pane tree usually have more generic panes closer to the root and more
117 special-purpose panes near the leaves.
119 It is quite normal for there to be panes in the tree that are not
120 directly involves in displaying anything - these are just useful
121 containers for data and functionality. Documents, described below,
122 exist as panes that are not directly displayed. Instead there are
123 display pane which link to the document and display its content.
125 As well as a dedicated command (the “handler”) and private data, each
128 - x,y co-ordinates together with width and height. The co-ordinates
129 are relative to the parent, and by recursive addition can be made
131 - a “z” value which indicates display priority with respect to
132 siblings. When siblings overlap, the sibling with the higher “z”
133 value will be drawn “over” siblings with a lower “z” value, including
134 all the children of that sibling (independent of their z value).
135 - a selected child referred to as the “focus”. Keyboard input at a
136 display is forwarded down the chain of focus links until it reaches
137 a leaf pane. This is where handling of the keystroke starts.
138 - a set of “damaged” flags which record if any changes have been made
139 which might affect the display.
140 - an arbitrary set of attributes with assigned values.
142 Each pane may also request notifications from other panes. These
143 include, but are not limited to, a notification when the pane is
144 destroyed and a notification when a document attached to the pane
145 changes. These notifications are effected by calling the pane's command
146 with a key like “Notify:Close” and with the second pane argument
147 (the “focus”) set to the pane which is sending the notification.
152 A document provides access to whatever data is being edited, or just
153 being displayed. There can be multiple implementations for documents
154 but they all have a common interface.
156 A “document” is assumed to be a linear sequence of elements each of
157 which presents as a single character and may have some attributes
158 associated with it. For a “text” document, the characters are
159 typically Unicode characters stored as UTF-8 and the attributes, if
160 any, are hints for parsing or display.
161 For a “directory” document, the elements are entries in the directory
162 and the associated character reflects the type of entry. The
163 attributes contain other information such as file name, size,
166 A document is represented by a non-display pane. These panes are
167 typically collected together as children of a “document-list” pane
168 which can be asked to add or find documents. To display a
169 document, a document-display pane is normally created. This contains,
170 in its private data, a reference to the document pane and a “point”
171 (see below) indicating where changes will happen. Events that arrive
172 at the document-display pane will typically be forwarded to the
173 document, though they maybe be handled directly by the display pane.
178 An attribute is a simple name=value pair, both being strings.
179 Attributes can be associated with various other objects, including
180 marks, panes, and elements in a document. Parsing code can
181 annotate a buffer with attributes, and rendering code can use these
182 attributes to guide rendering. e.g. parsing code can attach
183 “spelling=wrong” or “spelling=doubtful” and rendering can underline in
184 red or whatever is appropriate.
189 A “mark” identifies a location in a document. The location is between
190 two elements in the document, or at the start or end, and the mark
191 remains at that location despite any edits that do not affect
192 neighbouring elements.
194 Marks come in three different sorts: ungrouped, grouped, and points.
195 All of these appear in document-order in a linked list, all have a
196 sequence number in the list so ordering-tests are easy, and each can
197 have a set of attributes attached.
199 An ungrouped mark has no property beyond the above. A grouped marked
200 is included in a second linked list with all the other marks in the
201 same group. This group is owned by a specific pane and keeps
202 information relevant to the task of that pane. A pane responsible for
203 rendering part of a document might have marks identifying the start
204 and end of the visible portion, and maybe even the start of each line
205 in the visible portion. A grouped mark also has a reference to an
206 arbitrary data structure which is understood by the pane which owns
209 A “point” is a special grouped-mark which is included in all of the
210 other lists of grouped marks. This is achieved by using the external
211 reference to hold an auxiliary data structure which is linked in to
212 all of the lists. Every document-display pane owns a point. This
213 point is usually where changes to the document happen. When the
214 notification mechanism mentioned earlier tells other panes of a chance
215 to the document, the point where the change happened is also reported.
216 From this point it is easy to find and update nearby marks of any
219 An example use is to have a group of marks which are used to track
220 line numbers. “line-count” marks are placed every 500 lines (or so)
221 with an attribute recording exactly how many lines between this and
222 the next “line-count” mark. When a change happens, the recorded line
223 count on the preceding mark is cleared. When a line count or line
224 number is needed, the list of “line-count” marks is walked from the
225 start. If any has its count cleared, the lines in that section are
226 counted and the record is updated. Otherwise all that is required is
227 simply adding up a few numbers.
229 Marks could be used by a parser to identify key locations which would
230 allow a renderer to find the important content quickly if it was only
231 rendering a partial view - such as the headings in outline mode.
236 A “display” is just a pane which can create an image somehow, and
237 responds to commands like “pane-clear”, “text-display”, and
238 “image-display”. Displays are typically quite close to the root of the
239 “pane” tree, but this is not a requirement.
241 A display is also expected to call “Keystroke” and “Mouse-event”
242 commands in response to appropriate events. These will propagate
243 towards the root and normally hit an input-management pane which will
244 find the appropriate target leaf, will convert to a full event,
245 e.g. adding a repeat count or indication of a prefix key, and will
246 submit the new event at the target.
251 A keymap is a mapping from command names to commands. While a
252 pane hander could use any mapping it likes, the keymap implemented in
253 edlib has one small advantage in that a range of strings can be mapped to
254 a command, then exceptions can be recorded.
256 The handler for a pain typically looks up the passed “key” in a
257 keymap, locates the target command, and passes control to that command.
262 Now that we have plenty of context, it is time to revisit commands to
263 discuss how they are called. It is possible to invoke a specific
264 command directly if you have a reference to it but most often a more
265 general mechanism is used to find the appropriate command. The most
266 common mechanism is to pass and "Event" (i.e. a set of arguments) to a
267 “home” pane. The handler for that pane and each ancestor will be tried
268 in turn until a handler returns a non-zero value (i.e. it accepts the event),
269 or until the root pane has been tried. Very often the starting home pane
270 will also be the focus pane so when the two are the same it is not necessary
273 The other common mechanism is a “notification event” which follows the
274 "notifier" chain from a pane. This chain lists a number of panes which
275 have requested notifications. When calling notifiers, all target panes have their
276 handler called and if any return a non-zero value, the over-all return
277 value will be non-zero. More precisely it will be the value returned
278 which has the largest absolute value.
280 Each handler can perform further lookup however it likes. It may
281 just compare the “key” against a number of supported keys, or it might
282 perform a lookup in a key-table. One particularly useful approach is
283 to look up all commands with a prefix matching the key and call all of
284 them in order until one returns a non-zero value. This can be used to
285 allow multiple handlers to register for a service where each will
286 handle different instances. For example when choosing a document type
287 to open a given file, all document types will be tried but some would
288 be expected to return zero. e.g. if the file is actually a directory,
289 everything but the directory document type would reject the request.
291 Another example worth understanding is the document-display pane
292 type. When this receives an event it will handle it directly if it
293 understands the key, otherwise if it starts with “doc:” it will pass it
294 to the document pane. If that doesn't recognise the event it will
295 continue up the tree from the document-display pane.
297 Like document-display, other pane types are free to direct events
298 elsewhere as appropriate. The “input” handler takes keystroke events
299 and redirects them to the current focus pane, and take mouse events
300 and redirects them to the pane with the greatest 'z' depth which covers
307 These are the basic common objects which can (I hope) be used to build
308 a rich document editor. There needs to be lots of extensions of course
309 to make them useful. The current extensions that are available include:
315 A text document stores text in various linked data structures designed
316 to make simple edits easy and to support unlimited undo/redo. There
317 are a number of allocations, and a list of “chunks” which each
318 identify a start and end in one of those allocations. Edits can
319 add text to the last allocation, can change the endpoints of a chuck,
320 and can insert new chunks or delete old chunks.
322 Each chunk has a list of attributes each with an offset into the allocation,
323 so they each apply to a single character, though are often interpreted
324 to apply to follow characters as well.
329 A directory document contains a list of directory entries from a
330 directory and provides a variety of attributes for each entry. The
331 directory can be re-read at any time with incremental changes made to
337 There is typically one pane of this type and it registers an
338 “doc:appeared-” handler with the root pane to get notified when documents
339 are created. It will reparent the document so that it becomes a
340 child of a separate “collection” pane. Then all documents can be found in the
343 The “documents” pane presents as a document which can be viewed and
344 appears as a list of document names. Various keystroke events allow
345 documents to be opened, deleted, etc.
347 Rendering virtual document
348 --------------------------
350 Working directory with the "directory" or "documents" document is sometimes
351 a bit awkward as a mark can only point to a while directory entry or
352 document. When these are displayed one-per-line it isn't possible to move
353 the cursor within that line, which can feel strange, particularly if the line
354 is wider than that display pane. Also selecting an copying
355 text cannot select part of a line.
357 To overcome these problems, a "rendering" document can be layered over
358 the main document. It presents a virtual document which contains
359 all the character that are use to display (to render) the underlying
360 document. This then "feels" more like a regular document and can respond
361 to select and copy just like a text document.
363 Multipart document, and "crop" filter
364 -------------------------------------
366 Multipart is another virtual document, which appears to contain
367 the content of a sequence of other documents. This allows a sequence
368 of documents to appear to be combined into one. This can co-operate
369 with the "crop" filter which limits access to a given document to the
370 section between two marks. By combining multipart and crop,
371 one document can be divided up and re-assembled in any order, or
372 parts of multiple documents can be merged.
374 Email document; base64, qprint, utf8, rfc822header filters
375 ----------------------------------------------------------
377 The Email document handler uses crop and multipart and other tools to
378 present an email message as a readable document. The different
379 parts of an email message (header, body, attachments) are identified
380 and cropped out with appropriate filters attached.
385 "notmuch" as a email indexing and management tools. The notmuch
386 email reader provides one document which displays various
387 saved searches with a count of the number of items, and another
388 document type which can show a summary line for each message
389 found by a given search. They work with the email document
390 pane to allow reading and managing an e-mail mailbox.
395 The “ncurses” display can draw text on a terminal window and can set
396 various attributes such as colour, bold, underline etc. It also
397 receives keyboard and mouse input and sends “Mouse-event” or
398 “Keystroke” command up to the input manage.
400 There can be multiple ncurses displays, each attached to a different terminal.
405 This is a display module written in python and using pygtk for
408 When a “text” or “clear” request is made on a pane, the module
409 allocates a pixmap (arranging for it to be destroyed when the pane is
410 closed) and performs the drawings there. When a refresh is required,
411 the various pixmaps are combined and drawn to the target window.
413 Variable width fonts are supported as are images. An image is
414 typically the only thing drawn in a pane, so sub-panes must be used to
415 draw images within a document.
417 Lines-Renderer and render-line filter.
418 --------------------------------------
420 The lines renderer is designed to work with any document that presents
421 as a list of lines. Lines that are wider than the pane can either be
422 truncated (with side-scrolling) or wrapped. The lines renderer moves
423 backwards and forwards from the cursor “point” to determine which lines
424 should be drawn and sends a “render-line” command to get the
425 displayed text for those lines.
427 There is “render-line” pane type which provides the “render-line”
428 function for a simple text document. It looks for particular
429 attributes in the document and on marks in the document and interprets
430 them to allow highlighting different parts of the text. It returns a
431 line with markup which the line renderer understands.
433 Attribute Format Renderer
434 -------------------------
436 The attribute formatter is given a format for each line into which it
437 interpolates attributes from each element in the document. These
438 lines are provided in response to “render-line” requests so that if
439 the line-render and the attribute formatter are both stacked on a
440 document, then the attributes of elements in the document can be
443 The format specification is found by requesting the attribute
444 “Line-format” from panes starting at the focus and moving towards the root.
449 The “completion” render is a filter. In response to a “render-line”
450 call it calls “render-line” on its parent and only returns lines that
451 start with a given prefix, or contain a given substring. It can also
452 add highlights to rendered text to distinguish the common prefix from
455 A prefix is set by a “set-prefix” command. The response to this
456 indicates if the selected lines have a longer common prefix, and if
457 there is in fact only a single matching line. This supports the
458 implementation of filename, document name, command name completion
459 etc. To complete from a set of names, you just need to provide a
460 document which responds to “render-line” with the various options to
466 The HEX renderer provides an alternate “render-line” for a document
467 which starts each line at a multiple of 16 bytes from the start of the
468 document, and formats the next 16 bytes as hex and ASCII. Each
469 rendered line starts with the byte offset of the start of the line.
471 Presentation renderer
472 ---------------------
474 The presentation rendering accepts a text document and interprets
475 it as describing pages of a presentation in a language similar
476 to MarkDown. It generates a marked-up rendering the various
477 lines of text which are given to the lines renderer to produce
478 a single page of the presentation.
480 There is a partner "markdown" mode which interacts with a presentation
481 pane and can ask it to move forward or backward one page, or to
482 redraw some other arbitrary page.
487 The “tile” handler takes a pane (typically the root pane of a display)
488 and divides it up into 1 or more non-overlapping tiles. Tiles are
489 grouped in horizontal and vertical stacks. Tiles can be split, can be
490 discarded, or can be resized. Any of these operations may affect other
493 The leaves of the tile tree need to have some other pane attached. The
494 tiler doesn't render anything itself, not even borders. That is left
500 A “view” draws borders around a pane and provides a child pane which
501 is slightly smaller to allow for those borders (so it should probably
504 The borders can contain scroll-bars, a document name, or other
505 information provided by child panes.
510 The popup manager places a small window with an elevated “z” value
511 somewhere relevant on the display and can provide a simple text document
512 for text entry - or can use a provided document. Various key strokes
513 are captured to allow the popup to be aborted, or to send the content
514 of the mini document to the originating pane.
516 The popup requests notifications from that pane so that if it is
517 closed, the popup automatically closes too.
522 The line-counter uses the model described earlier of placing marks
523 every few hundred lines in a document and using them to expedite line
524 counting. This is implemented as a pane which attaches directly to
530 “Keymap” pane allows both global and local keys (or arbitrary
531 commands) to be defined. The global mappings are handled at a pane
532 which must be stacked before the tiler. A pane to handle local
533 mappings is added on demand at the current focus pane.
538 “search” provides a global command rather than a pane. This command
539 can perform a reg-ex search through a document.
544 “Messageline” trims the bottom line off a pane (providing a pane which is
545 slightly smaller) and will display messages in this pane until the
546 next keyboard command.
551 A pane of this module stores some state related to the current input
552 context, including a modifier prefix and a repeat count.
554 When a “keystroke” command is received the prefix is added to the key
555 and this is sent as a command to the focus. The command includes the
556 repeat count, which gets cleared.
558 Commands are provided to set a new prefix or repeat count. So for
559 example “Meta-1” might multiply the repeat count in the command by 10,
560 add 1, and then ask “input” to set that as the new repeat count for
566 This pane makes it easy to run a command in the background and capture
567 the output in a text document, which can then be displayed like
573 This allows make or grep to be running with the output captured and parsed.
574 A simple keystroke then causes the editor to open a file mentioned
575 in the output, and to go to the identified line.
580 The viewer pane suppresses any commands that would modify the document,
581 and repurposes some of them to make it easier to move around a document
582 being viewed - so 'space' pages forward, and 'backspace' pages backwards.
587 A history pane stores data a bit like a document, but provides
588 access in a different way. Individual lines can be recalled and
589 new lines can be appended. It makes it easy to provide a history
590 of lines of text entered for some purpose, such as running a shell
591 command or an editor command selected by name.
596 Similar in principle to "history", a copybuf pane can store a series
597 of arbitrary slabs of text. It is used to provide copy/paste functionality.
602 The server pane listens on a socket for request to open a file,
603 or to create an ncurses pane on the current terminal.
604 It also reports to the request when the file has been edited,
605 or when the ncurses pane is closed.
610 This provides a set of named commands which can be given to “keymap”
611 as a global key map. In provides a number of emacs-like bindings.
617 This pane capture various editing command and tailors them
618 to suite editing C or Python code. It helps with correct
619 indenting, highlight matching brackets, and will eventually do
625 This module allows python code to be run, provides an interface to
626 panes, marks, and commands, and allows commands to be called on a
627 given pane. It also allows commands to be defined in python that can
628 be called from other modules just like any other command. It is a
629 complete two-way interface between python and other languages to
630 access the core edlib functionality.
635 edlib needs an event loop to wait for input, capture signals, and run
636 tasks. Any module can register an event loop by registering handlers
637 for various “event:*” events with the root pane. When pygtk is being
638 used, the glib event loop must be used. Otherwise some other event
639 loop is needed. To this end, the libevent module registers a
640 low-priority set of event handler which use libevent.
642 I'm not entirely happy about this arrangement. In particular I would
643 like to be able to have multiple event loops running in separate
644 threads. So expect things to change here.
649 The [TO-DO list](DOC/TODO.md) is now a separate document.
git / edlib.git / blob