1 <meta HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=utf-8">
4 # Copyright Neil Brown ©2015-2021 <neil@brown.name>
5 # May be distributed under terms of GPLv2 - see file:COPYING
8 Edlib - a library for building a document editor
9 ==============================================
11 Edlib is an extensible document editor. It is inspired in part by
12 Emacs, both by its strengths and its weaknesses.
14 Emacs provides a programming language — E-lisp — for configuring and
15 extending the editor. Edlib doesn't. It allows various pre-existing
16 languages to be used to configure and extend the editor. It does
17 this by providing a library of core editing tools and providing
18 bindings to various languages.
20 At least, that is the plan. At time if writing, edlib only provides
21 bindings for C and Python. Other languages should be fairly easy.
23 The particular value-add of edlib over Emacs (apart from the obvious
24 “NIH” issues) is that both document storage and document rendering are
25 fully extensible. A document is not always a text buffer, it could
26 also be a mem-mapped files, a directory, or an internal data
28 Any document can have multiple views, and each view can
29 show very different things: scriptable code is run whenever
30 rendering is required. This should make implementing documents with
31 nontrivial structures a lot easier.
33 Edlib is designed to have well defined abstractions that can be
34 exported to various languages for them to manipulate. They include
35 primarily commands, panes, and marks, and also attributes, documents,
36 displays, events, and keymaps.
41 Commands are the single mechanism by which control is transferred from
42 one part of the editor implementation to another. They provide the
43 only way for code in one implementation language to invoke code
44 written in another language, and they are the preferred way to
45 interact between different modules even in the same language.
47 All commands receive the same types of arguments and produce an integer
48 result. The arguments include two panes (“home” and “focus”),
49 two marks (“mark” and “mark2”),
50 three strings (“key”, “str”, “str2”),
51 two numbers (“num” and “num2”),
52 a coordinate pair (“x”, “y”) and two commands (“comm1” and “comm2”).
53 Extra result values can be effected by passing them to a call to
54 the “comm2” argument - i.e the command can be used as a callback.
56 Each “pane” has a dedicated command which handles messages sent to
57 the pane, as will be described later. Commands can also be passed
58 to other commands, which can either call them directly (like the
59 callback mentioned above) or store them for later use, or
62 Three of the arguments provided to a command have very special
63 meanings. One of the strings, known as “key”, identifies what action
64 should be performed. The pane handler will normally use this key to
65 select some other command to actually handle the message. Other
66 commands may ignore the key, or use it however they please.
68 When a message is sent to a pane and the handler command is called,
69 the “home” argument is set to the pane that owns the handle, so it acts
70 a bit like the “self” argument in some object-oriented languages.
71 One of the primary uses of “home” is to access “home->data” which is a
72 private data structure owned by the pane. The command associated with
73 a particular pane is typically the only code which can understand the data.
75 Finally the “comm1” argument passed to a command always identifies
76 exactly the command that is being run. “comm1” is a pointer to a
77 structure containing a pointer to the function being called. This
78 structure can be embedded is some other data structure which contains
79 context for the command. This context might be read-only, to refine
80 the behaviour of the command, or read/write to provide storage for the
81 command to use. For example, the call-back commands mentioned earlier
82 are normally embedded in a data structure that can store the extra
85 Apart from the return value of zero (Efallthrough) which indicates
86 “command not understood”, a command can return a positive result on
87 success or a negative result indicating lack of success. Known error
90 - Enoarg : missing argument
91 - Einval : something is wrong with the context of the request
92 - Efail : request makes sense, but didn't work
93 - Enosup: request makes sense, but isn't allowed for some reason
94 - Efalse: Not really an error, just a Boolean status
99 A pane combines an optional rectangular area of display with some
100 data storage and some functionality. As such it can receive mouse and
101 keyboard events, can draw on a display, and can send commands to other
104 All panes are arranged as a tree with all but the root having a parent
105 and many having siblings and children. When a pane represents a
106 rectangle of display all children are restricted to just that
107 rectangle or less. Often a child will cover exactly the same area as
108 its parent. In other cases several children will share out the area.
110 An “event” is a set of arguments to a command which is being sent to
111 a pane. Events are often generated at a leaf of the tree of panes
112 (i.e. a pane with no children). They travel up the tree towards the
113 root until they find a pane which can handle them. That pane (or its
114 handler command) might handle the event by generating other events.
115 They will typically start looking for a handler at the same leaf which
116 is available as the “focus” argument. For this reason branches of the
117 pane tree usually have more generic panes closer to the root and more
118 special-purpose panes near the leaves.
120 It is quite normal for there to be panes in the tree that are not
121 directly involved in displaying anything - these are just useful
122 containers for data and functionality. Documents, described below,
123 exist as panes that are not directly displayed. Instead there are
124 display panes which link to the document and display its content.
126 As well as a dedicated command (the “handler”) and private data, each
129 - x,y coordinates together with width and height. The coordinates
130 are relative to the parent, and by recursive addition can be made
132 - a “z” value which indicates display priority with respect to
133 siblings. When siblings overlap, the sibling with the higher “z”
134 value will be drawn “over” siblings with a lower “z” value, including
135 all the children of that sibling (independent of their z value).
136 - a selected child referred to as the “focus”. Keyboard input at a
137 display is forwarded down the chain of focus links until it reaches
138 a leaf pane. This is where handling of the keystroke starts.
139 - a set of “damaged” flags which record if any changes have been made
140 which might affect the display.
141 - an arbitrary set of attributes with assigned values.
143 Each pane may also request notifications from other panes. These
144 include, but are not limited to, a notification when the pane is
145 destroyed and a notification when a document attached to the pane
146 changes. These notifications are effected by calling the pane's command
147 with a key like “Notify:Close” and with the second pane argument
148 (the “focus”) set to the pane which is sending the notification.
153 A document provides access to whatever data is being edited, or just
154 being displayed. There can be multiple implementations for documents
155 but they all have a common interface.
157 A “document” is assumed to be a linear sequence of elements each of
158 which presents as a single character and may have some attributes
159 associated with it. For a “text” document, the characters are
160 typically Unicode characters stored as UTF-8 and the attributes, if
161 any, are hints for parsing or display.
162 For a “directory” document, the elements are entries in the directory
163 and the associated character reflects the type of entry. The
164 attributes contain other information such as file name, size,
167 A document is represented by a non-display pane. These panes are
168 typically collected together as children of a “document-list” pane
169 which can be asked to add or find documents. To display a
170 document, a document-display pane is normally created. This contains,
171 in its private data, a reference to the document pane and a “point”
172 (see below) indicating where changes will happen. Events that arrive
173 at the document-display pane will typically be forwarded to the
174 document, though they maybe be handled directly by the display pane.
179 An attribute is a simple name=value pair, both being strings.
180 Attributes can be associated with various other objects, including
181 marks, panes, and elements in a document. Parsing code can
182 annotate a buffer with attributes, and rendering code can use these
183 attributes to guide rendering. e.g. parsing code can attach
184 “spelling=wrong” or “spelling=doubtful” and rendering can underline in
185 red or whatever is appropriate.
190 A “mark” identifies a location in a document. The location is between
191 two elements in the document, or at the start or end, and the mark
192 remains at that location despite any edits that do not affect
193 neighbouring elements.
195 Marks come in three different sorts: ungrouped, grouped, and points.
196 All of these appear in document-order in a linked list, all have a
197 sequence number in the list so ordering-tests are easy, and each can
198 have a set of attributes attached.
200 An ungrouped mark has no property beyond the above. A grouped marked
201 is included in a second linked list with all the other marks in the
202 same group. This group is owned by a specific pane and keeps
203 information relevant to the task of that pane. A pane responsible for
204 rendering part of a document might have marks identifying the start
205 and end of the visible portion, and maybe even the start of each line
206 in the visible portion. A grouped mark also has a reference to an
207 arbitrary data structure which is understood by the pane which owns
210 A “point” is a special grouped-mark which is included in all of the
211 other lists of grouped marks. This is achieved by using the external
212 reference to hold an auxiliary data structure which is linked in to
213 all of the lists. Every document-display pane owns a point. This
214 point is usually where changes to the document happen. When the
215 notification mechanism mentioned earlier tells other panes of a change
216 to the document, the point where the change happened is also reported.
217 From this point it is easy to find and update nearby marks of any
220 An example use is to have a group of marks which are used to track
221 line numbers. “line-count” marks are placed every 500 lines (or so)
222 with an attribute recording exactly how many lines between this and
223 the next “line-count” mark. When a change happens, the recorded line
224 count on the preceding mark is cleared. When a line count or line
225 number is needed, the list of “line-count” marks is walked from the
226 start. If any has its count cleared, the lines in that section are
227 counted and the record is updated. Otherwise all that is required is
228 simply adding up a few numbers.
230 Marks could be used by a parser to identify key locations which would
231 allow a renderer to find the important content quickly if it was only
232 rendering a partial view - such as the headings in outline mode.
237 A “display” is just a pane which can create an image somehow, and
238 responds to commands like “Draw:clear”, “text-display”, and
239 “image-display”. Displays are typically quite close to the root of the
240 “pane” tree, but this is not a requirement.
242 A display is also expected to call “Keystroke” and “Mouse-event”
243 commands in response to appropriate events. These will propagate
244 towards the root and normally hit an input-management pane which will
245 find the appropriate target leaf, will convert to a full event,
246 e.g. adding a repeat count or indication of a prefix key, and will
247 submit the new event at the target.
252 A keymap is a mapping from command names to commands. While a
253 pane handler could use any mapping it likes, the keymap implemented in
254 edlib has one small advantage in that a range of strings can be mapped to
255 a command, then exceptions can be recorded.
257 The handler for a pain typically looks up the passed “key” in a
258 keymap, locates the target command, and passes control to that command.
263 Now that we have plenty of context, it is time to revisit commands to
264 discuss how they are called. It is possible to invoke a specific
265 command directly if you have a reference to it but most often a more
266 general mechanism is used to find the appropriate command. The most
267 common mechanism is to pass an “Event” (i.e. a set of arguments) to a
268 “home” pane. The handler for that pane and each ancestor will be tried
269 in turn until a handler returns a non-zero (non-Efallthrough) value
270 (i.e. it accepts the event), or until the root pane has been tried.
271 Very often the starting home pane will also be the focus pane so when
272 the two are the same it is not necessary to specify both.
274 The other common mechanism is a “notification event” which follows the
275 “notifier” chain from a pane. This chain lists a number of panes which
276 have requested notifications. When calling notifiers, all target panes
277 have their handler called and if any return a non-zero value, the
278 over-all return value will be non-zero. More precisely it will be the
279 value returned which has the largest absolute value.
281 Each handler can perform further lookup however it likes. It may
282 just compare the “key” against a number of supported keys, or it might
283 perform a lookup in a key-table. One particularly useful approach is
284 to look up all commands with a prefix matching the key and call all of
285 them in order until one returns a non-zero value. This can be used to
286 allow multiple handlers to register for a service where each will
287 handle different instances. For example when choosing a document type
288 to open a given file, all document types will be tried but some would
289 be expected to return zero. e.g. if the file is actually a directory,
290 everything but the directory document type would reject the request.
292 Another example worth understanding is the document-display pane
293 type. When this receives an event it will handle it directly if it
294 understands the key, otherwise if it starts with “doc:” it will pass it
295 to the document pane. If that doesn't recognise the event it will
296 continue up the tree from the document-display pane.
298 Like document-display, other pane types are free to direct events
299 elsewhere as appropriate. The “input” handler takes keystroke events
300 and redirects them to the current focus pane, and take mouse events
301 and redirects them to the pane with the greatest 'z' depth which covers
308 These are the basic common objects which can (I hope) be used to build
309 a rich document editor. There needs to be lots of extensions of course
310 to make them useful. The current extensions that are available include:
316 A text document stores text in various linked data structures designed
317 to make simple edits easy and to support unlimited undo/redo. There are
318 a number of allocations for text, and a list of “chunks” which each
319 identify a start and end in one of those allocations. Edits can add
320 text to the last allocation, can change the endpoints of a chuck, and
321 can insert new chunks or delete old chunks.
323 Each chunk has a list of attributes each with an offset into the allocation,
324 so they each apply to a single character, though are often interpreted
325 to apply to follow characters as well.
330 A directory document contains a list of directory entries from a
331 directory and provides a variety of attributes for each entry. The
332 directory can be re-read at any time with incremental changes made to
338 There is typically one pane of this type and it registers a
339 “doc:appeared-” handler with the root pane to get notified when
340 documents are created. It will reparent the document so that it becomes
341 a child of a separate “collection” pane. Then all documents can be
342 found in the list of children.
344 The “documents” pane presents as a document which can be viewed and
345 appears as a list of document names. Various keystroke events allow
346 documents to be opened, deleted, etc.
348 Multipart document, and “crop” filter
349 -------------------------------------
351 Multipart is a virtual document, which appears to contain the content of
352 a sequence of other documents. This allows a sequence of documents to
353 appear to be combined into one. This can co-operate with the “crop”
354 filter which limits access to a given document to the section between
355 two marks. By combining multipart and crop, one document can be divided
356 up and re-assembled in any order, or parts of multiple documents can be
359 Email document; base64, qprint, utf8, rfc822header filters
360 ----------------------------------------------------------
362 The Email document handler uses crop and multipart and other tools to
363 present an email message as a readable document. The different
364 parts of an email message (header, body, attachments) are identified
365 and cropped out with appropriate filters attached.
370 “notmuch” as a email indexing and management tools. The notmuch email
371 reader provides one document which displays various saved searches with
372 a count of the number of items, and another document type which can show
373 a summary line for each message found by a given search. They work with
374 the email document pane to allow reading and managing an e-mail mailbox.
379 The “ncurses” display can draw text on a terminal window and can set
380 various attributes such as colour, bold, underline etc. It also
381 receives keyboard and mouse input and sends “Mouse-event” or “Keystroke”
382 command up to the input manage.
384 There can be multiple ncurses displays, each attached to a different
390 This is a display module written in python and using python-gtk for
393 When a “text” or “clear” request is made on a pane, the module
394 allocates a pixmap (arranging for it to be destroyed when the pane is
395 closed) and performs the drawings there. When a refresh is required,
396 the various pixmaps are combined and drawn to the target window.
398 Variable width fonts are supported as are images. An image is
399 typically the only thing drawn in a pane, so sub-panes must be used to
400 draw images within a document.
402 Lines-Renderer and render-line filter.
403 --------------------------------------
405 The lines renderer is designed to work with any document that presents
406 as a list of lines. Lines that are wider than the pane can either be
407 truncated (with side-scrolling) or wrapped. The lines renderer moves
408 backwards and forwards from the cursor “point” to determine which lines
409 should be drawn and sends a “render-line” command to get the
410 displayed text for those lines.
412 There is “render-line” pane type which provides the “render-line”
413 function for a simple text document. It looks for particular
414 attributes in the document and on marks in the document and interprets
415 them to allow highlighting different parts of the text. It returns a
416 line with markup which the line renderer understands.
418 Attribute Format Renderer
419 -------------------------
421 The attribute formatter assumes each element in the document should be
422 displayed as a line, and it is given a format description which
423 describes which attributes of that element are used to create the
424 desired line. These lines of provided in a way that “render-line” can
425 access and display them.
427 There are two modes. On the simple mode, each line appears to be a
428 single object and it is not possible to move the cursor within that line
429 or highlight arbitrary substrings such as with selections or search.
431 The second mode is more functional but only available if the underlying
432 document is written to collaborate by sharing reference information. In
433 this case the document uses just a pointer to identify each element, and
434 the format rendered can use an "offset" field that is in every mark to
435 allow marks to be positioned with the formatted line. The net results
436 if that it is possible to move around character in a line, and to select
437 or search, even though these characters are not strictly part of the
440 The format specification is found by requesting the attribute
441 “line-format” from panes starting at the focus and moving towards the root.
446 The “completion” render is a filter. In response to a “render-line”
447 call it calls “render-line” on its parent and only returns lines that
448 start with a given prefix, or contain a given substring. It can also
449 add highlights to rendered text to distinguish the common prefix from
452 A prefix is set by a “set-prefix” command. The response to this
453 indicates if the selected lines have a longer common prefix, and if
454 there is in fact only a single matching line. This supports the
455 implementation of filename, document name, command name completion
456 etc. To complete from a set of names, you just need to provide a
457 document which responds to “render-line” with the various options to
463 The HEX renderer provides an alternate “render-line” for a document
464 which starts each line at a multiple of 16 bytes from the start of the
465 document, and formats the next 16 bytes as hex and ASCII. Each
466 rendered line starts with the byte offset of the start of the line.
468 Presentation renderer
469 ---------------------
471 The presentation rendering accepts a text document and interprets it as
472 describing pages of a presentation in a language similar to MarkDown.
473 It generates a marked-up rendering the various lines of text which are
474 given to the lines renderer to produce a single page of the
477 There is a partner “markdown” mode which interacts with a presentation
478 pane and can ask it to move forward or backward one page, or to redraw
479 some other arbitrary page.
484 The “tile” handler takes a pane (typically the root pane of a display)
485 and divides it up into 1 or more non-overlapping tiles. Tiles are
486 grouped in horizontal and vertical stacks. Tiles can be split, can be
487 discarded, or can be resized. Any of these operations may affect other
490 The leaves of the tile tree need to have some other pane attached. The
491 tiler doesn't render anything itself, not even borders. That is left
497 A “view” draws borders around a pane and provides a child pane which
498 is slightly smaller to allow for those borders (so it should probably
501 The borders can contain scroll-bars, a document name, or other
502 information provided by child panes.
507 The popup manager places a small window with an elevated “z” value
508 somewhere relevant on the display and can provide a simple text document
509 for text entry - or can use a provided document. Various key strokes
510 are captured to allow the popup to be aborted, or to send the content
511 of the mini document to the originating pane.
513 The popup requests notifications from that pane so that if it is
514 closed, the popup automatically closes too.
519 The line-counter uses the model described earlier of placing marks
520 every few hundred lines in a document and using them to expedite line
521 counting. This is implemented as a pane which attaches directly to
527 “Keymap” pane allows global keys or arbitrary commands to be defined.
528 The global mappings are handled at a pane which must be stacked before
534 “search” provides a global command rather than a pane. This command
535 can perform a reg-ex search through a document.
540 “Messageline” trims the bottom line off a pane (providing a pane which
541 is slightly smaller) and will display messages in this pane. They can
542 be “modal” messages which disappear on the next keystroke, normal
543 messages which remain for a period of time and recorded in a
544 “*Messages*” document, or broadcast messages which are like normal
545 messages, but are sent to all active Messageline panes.
550 A pane of this module stores some state related to the current input
551 context, including a modifier prefix and a repeat count.
553 When a “keystroke” command is received the prefix is added to the key
554 and this is sent as a command to the focus. The command includes the
555 repeat count, which gets cleared.
557 Commands are provided to set a new prefix or repeat count. So for
558 example “Alt-1” might multiply the repeat count in the command by 10,
559 add 1, and then ask “input” to set that as the new repeat count for
565 This pane makes it easy to run a command in the background and capture
566 the output in a text document, which can then be displayed like
572 This allows “make” or “grep” to be run with the output captured and
573 parsed. A simple keystroke then causes the editor to open a file
574 mentioned in the output, and to go to the identified line.
579 The viewer pane suppresses any commands that would modify the document,
580 and repurposes some of them to make it easier to move around a document
581 being viewed - so 'space' pages forward, and 'backspace' pages backwards.
586 A history pane stores data a bit like a document, but provides
587 access in a different way. Individual lines can be recalled and
588 new lines can be appended. It makes it easy to provide a history
589 of lines of text entered for some purpose, such as running a shell
590 command or an editor command selected by name.
595 Similar in principle to “history”, a copybuf pane can store a series
596 of arbitrary slabs of text. It is used to provide copy/paste functionality.
601 The server pane listens on a socket for request to open a file, or to
602 create an ncurses pane on the current terminal.
603 It also reports to the requester when the file has been edited, or when
604 the ncurses pane is closed.
609 This provides a set of named commands which can be given to “keymap” as
610 a global key map. In provides a number of Emacs-like bindings.
616 This pane capture various editing command and tailors them to suite
617 editing C or Python code. It helps with correct indenting, highlight
618 matching brackets, and will eventually do a lot more.
623 This module allows python code to be run, provides an interface to
624 panes, marks, and commands, and allows commands to be called on a
625 given pane. It also allows commands to be defined in python that can
626 be called from other modules just like any other command. It is a
627 complete two-way interface between python and other languages to
628 access the core edlib functionality.
633 edlib needs an event loop to wait for input, capture signals, and run
634 tasks. Any module can register an event loop by registering handlers
635 for various “event:*” events with the root pane. When pygtk is being
636 used, the glib event loop must be used. Otherwise some other event
637 loop is needed. To this end, the libevent module registers a
638 low-priority set of event handler which use libevent.
640 I'm not entirely happy about this arrangement. In particular I would
641 like to be able to have multiple event loops running in separate
642 threads. So expect things to change here.
647 The [TO-DO list](DOC/TODO.md) is now a separate document.
git / edlib.git / blob