1 <meta HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=utf-8">
4 # Copyright Neil Brown ©2015 <neil@brown.name>
5 # May be distrubuted under terms of GPLv2 - see file:COPYING
8 Edlib - a library for building a document editor
9 ==============================================
11 edlib is an extensible document editor. It is inspired in part by
12 emacs, both by its strengths and its weaknesses.
14 emacs provides a programming language — E-lisp — for configuring and
15 extending the editor. edlib doesn't. It allows various pre-existing
16 languages to be used to configure and extend the editor. It does
17 this by providing a library of core editing tools and providing
18 bindings to various languages.
20 At least, that is the plan. At time if writing, only provides
21 bindings for C and Python, and these are incomplete. Other languages
22 should be fairly easy.
24 The particular value-add of edlib over emacs (apart from the obvious
25 “NIH” issues) is that both document storage and document rendering are
26 fully extensible. A document is not always a text buffer, it could
27 also be a mem-mapped files, a directory, or an internal data
29 Any document can have multiple views, and each view can
30 show very different things: scriptable code is run whenever
31 rendering is required. This should make implementing documents with
32 non-trivial structures a lot easier.
34 edlib is designed to have well defined abstractions that can be
35 exported to various languages for them to manipulate. They include
36 primarily commands, panes and marks, and also
37 attributes, documents, displays, events, and keymaps.
42 Commands are the single mechanism by which control is transferred from
43 one part of the editor implementation to another. They provide the
44 only way for code in one implementation language to invoke code
45 written in another language, and they are the preferred way to
46 interact between different modules even in the same language.
48 All commands receive the same types of arguments and produce an integer
49 result. The arguments include two panes, two marks, three
50 strings, two numbers, two co-ordinate pairs and one command. Extra
51 result values can be effected by passing them to a call to the
54 Commands can be used internally to a module and externally for
55 communication between modules. Externally visible commands appear
56 primarily in two sorts of places. Each “pane” has a command which
57 performs all the actions required of, or provided by, that pane. A
58 pane may also maintain a set of commands with assigned names that
59 are provided to it from elsewhere. In particular the root pane
60 maintains a set of global named commands.
62 Two of the arguments provided to a command have very special
63 meanings. One of the strings, known as “key”, identifies what action
64 should be performed. For commands that have been assigned a name, the
65 “key” string passed will always be exactly the name by which the
66 command was found. For commands which are assigned to a pane, the
67 “key” could be anything. The command must determine if it understands
68 that key. If it does it should perform the action (possibly
69 calling some other command internally). If not it should return 0.
70 This return code usually causes the caller to look elsewhere for a
71 command to perform the desired action.
73 The other special argument is one of the panes, called “home”. This
74 always identifies the pane in which the command was found, either as a
75 dedicated 'pane' command or as a named command. One of the primary
76 uses of “home” is to access “home->data” which a private data
77 structure owned by the pane. The command associated with a
78 particular pane is typically the only code which can understand the
81 Apart from the return value of zero which indicates “command not
82 understood”, a command can return a positive result on success or a
83 negative result indicating lack of success. In particular, “-1” is a
84 generic failure return code, and “-2” indicates that the command will
85 complete asynchronously. Commands only return this if they are
86 specifically documented to do say. It is usually only relevant if a
87 callback was passed as the “command” argument and it indicates that
88 the callback has not yet been called, but otherwise no error has been
94 A pane combines a rectangular area of display with some
95 functionality. As such it can receive mouse and keyboard events,
96 can draw on a display, and can send commands to other panes. As
97 previously mentioned, panes can also store module-specific data.
99 All panes are arranged as a tree with all but the root having a parent
100 and many having siblings and children. The “root” pane does not
101 correspond to any display, but is still implemented as a pane for
102 consistency. The children of the root are the different display
103 windows, each of which can use a distinct display technology such as
104 “ncurses” or “gtk” or “qt” or “fbdev” or anything else. The root pane
105 also has some “virtual display” panes which are used to gather other
106 special panes - see “Documents” below.
108 As well as a dedicated command and private data, each pane has:
110 - x,y co-ordinates together with width and height. The co-ordinates
111 are relative to the parent, and by recursive addition can be made
113 - a 'z' value which indicates display priority with respect to
114 siblings. When siblings overlap, the sibling with the higher “z”
115 value will be draw “over” siblings with a lower “z” value, including
116 all the children of that sibling (independent of their z value).
117 - a selected child referred to as the 'focus'. Keyboard input at a
118 display is forwarded down the chain of focus links until it reaches
119 a leaf pane. This is where handling of the keystroke starts.
120 - a set of 'damaged' flag which record if any changes have been made
121 which might affect the display.
122 - an arbitrary set of attributes with assigned values.
124 Each pane may also request notifications from other panes. These
125 include, but are not limited to, a notification when the pane is
126 destroyed and a notification when a document attached to the pane
127 changes. This notifications are effected by calling the panes command
128 with a key like “notify:close” and with the second pane argument
129 (known as 'focus') set to the pane which is sending the notification.
135 A document provides access to whatever data is being edited. There
136 can be multiple implementations of a document but they call have a
139 A “document” is assumed to be a linear sequence of elements each of
140 which presents as a single character and may have some attributes
141 associated with it. For a “text” document, the characters are
142 typically Unicode characters stored as UTF-8 and the attributes, if
143 any, are hints for parsing or display.
144 For a “directory” document, the elements are entries in the directory
145 and the associated character reflects the type of entry. The
146 attributes contain the useful information such as file name, size,
149 Documents cannot be accessed directly, but must be accessed through a
150 “document access” pane. There can be several panes which access the
151 one document though there is always one special pane in a “virtual”
152 display. There panes in this virtual display contain all of the
153 active documents so a list of documents can be obtained by iterating
154 over the children of this virtual display. There is a singleton
155 document types called “*Documents*” which lists all active documents.
156 By opening a pane on this document, the set of active documents can be
159 It is (will be?) possible for different panes to provide slightly
160 different access styles to the same document. For example one pane
161 might present a file as a sequence of bytes while another presents the
162 same file as a sequence of unicode characters. Similarly one
163 interface may accept arbitrary insert/delete operations while another
164 converts all changes to over-writes (maybe).
170 An attribute is a simple name/value pair, both being strings.
171 Attributes can be associated with various other objects, including
172 mark, panes, and elements in a document. Parsing code can
173 annotate a buffer with attributes, and rendering code can use these
174 attributes to guide rendering. e.g. parsing code can attach
175 “spelling=wrong” or “spelling=doubtful” and rendering can underline in
178 Currently most attributes have to be stored using a particular
179 implementation of attribute storage. I'm not sure I want that in the
180 longer term. I'm considering having the pane command support a
181 "attribute:get" command though there are still unresolved issues with
187 A “mark” identifies a location in a document. The location is between
188 two elements in the document, or at the start or end, and the mark
189 remains at that location despite any edits that do not affect
190 neighbouring element.
192 Marks come in three different sorts: ungrouped, grouped, and points.
193 All of these appear in document-order a single linked list, all have a
194 sequence number in the list so ordering-tests are easy, and each can
195 have a set of attributes attached.
197 As well as identifying a location in a document, a mark can identify a
198 location in the display of that document location. When a single
199 element in a document is displayed using multiple characters (as for
200 example a directory entry might be), the “rendering position” or
201 “rpos” can record where in those multiple character the mark really
202 belong. I'm not yet sure how useful this is, but it seems like a good
205 An ungrouped mark has no property beyond the above. A grouped marked
206 is included in a second linked list with all the other marks in the
207 same group. This group is owned by a specific pane and keeps
208 information relevant to the task of that pane. A pane responsible for
209 rendering part of a document might have marks identifying the start
210 and end of the visible portion, and maybe even the start of teach line
211 in the visible portion. An ungrouped mark also has a reference to an
212 arbitrary data structure which is understood by the pane which owns
215 A “point” is a special grouped-mark which is included in all of the
216 other lists of grouped marks. This is achieved by using the external
217 reference to hold an auxiliary data structure which is linked in to
218 all of the lists. Every pane which views a document owns a point.
219 This point is usually where changes to the document happen. When the
220 notification mechanism mentioned earlier tells other panes of a chance
221 to the document, the point where the change happened is also
222 reported. From this point it is easy to find and update nearby marks
225 An example use is to have a group of marks which are used to track line
226 numbers. “line-count” marks are placed every 500 lines (or so) with
227 an attribute recording exactly how many lines between this and the
228 next “line-count” mark. When a change happens, the recorded line
229 count is cleared. When a line count or line number is needed, the
230 list of “line-count” marks is walked from the start. If any has its
231 count cleared, the lines in that section are counted and the record is
232 updated. Otherwise all that is required is simply adding up a few
235 Marks could be used by a parser to identify key locations which would
236 allow a renderer to find the important content quickly if it was only
237 rendering a partial view - such as the headings in outline mode.
243 A “display” is just a pane which can create an image somehow, and
244 responds to commands like “draw:clear” and “draw:text”. Displays are
245 typically just below the root of the 'pane' tree, but this is not a
248 A display is also expected to call "Keystroke" and "Mouse-event"
249 commands in response to appropriate events.
254 A keymap is a mapping from command names to commands. In many cases a
255 similar data structure such as a Python “dict” could be used. The
256 keymap implemented in edlib has one small advantage in that a range of
257 strings can be mapped to a command, then exception can be recorded.
259 Keymaps are a bit like attributes in that the concept is valuable but
260 it isn't yet clear how central a particular implementation should be.
265 Now that we have plenty of context, it is time to revisit commands to
266 discuss how they are called. It is possible to invoke a specific
267 command directly but most often a more general mechanism is used to
268 find the appropriate command. There are three such mechanisms which
269 each search the pane tree testing different commands until one accepts
270 the given “key” name. They each find a starting pane in a different
271 way, and then try that pane and then its parent and so on up the tree
272 until a pane accepts the command (returning non-zero) or until the
275 There are three ways to choose the starting point, which is included
276 in the arguments to every command called as the “focus” point.
278 The first way is for the caller to explicitly set it. This is not a
279 very common approach but is needed when a pane acts like a “filter”.
280 The handler for a particular command might call that command on the
281 parent, then process the result in some way and return that result to
282 the caller. This can also be used when the appropriate “focus” has
283 already been found. If the handler for some command wants to call
284 some other command it will typically pass the “focus” of the first as
285 the “focus” to start searching for the second.
287 The second way is to search out to a leaf of the tree following the
288 “focus” links in each pane. The search typically starts at a
289 “Display” pane where a keystroke is generated,
291 The third way is to search out to a leaf of the tree which contains
292 some particular x,y co-ordinate. The pane with the highest 'z' value
293 which contains the co-ordinate will be chosen. This is typically used
294 to find the correct handler for a mouse event.
299 These are the basic common objects which can (I hope) be used to build
300 a rich document editor. There need to be lots of extensions of course
301 to make them useful. The current extensions that are available include:
306 A text document stores text in various linked data structured designed
307 to make simple edits easy and to support unlimited undo/redo. There
308 are a number of allocations, and a list of “chunks” which each
309 identify a start and end in one of those allocations. Edits can
310 add text to the last allocation, can change the endpoints of a chuck,
311 and can insert new chunks or delete old chunks.
313 Each chunk has a list of attributes each with an offset into the allocation.
318 A directory document contains a list of directory entries from a
319 directory and provides a variety of attributes for each entry. The
320 directory can be re-read at any time with incremental changes made to
326 The 'ncurses' display can draw text on a terminal window and can set
327 various attributes such as colour, bold, underline etc. It also
328 receives keyboard and mouse input and sends 'Mouse-event' or
329 'Keystroke' commands ... somewhere.
334 The line renderer is designed to work with any document that presents
335 as a list of lines. Lines that are wider than the pane can either be
336 truncated (with side-scrolling) or wrapped. The line renderer moves
337 back and forwards from the cursor “point” to determine which lines
338 should be drawn and sends a “render-line” command to get the
339 displayed text for those lines.
341 The “text” document provides direct support for “render-line”, but
342 needs a better way of provides stable limited-sized lines when at file
343 doesn't contain as many newline characters as we would like.
345 Attribute Format Renderer
346 -------------------------
348 The attribute formatter is given a format for each line into which it
349 interpolates attributes from each element in the document. These
350 lines are provided in response to “render-line” requests so that if
351 the line-render and the attribute formatter are both stacked on a
352 document, then the attributes of elements in the document can be
355 The format specification is found by requesting the attribute
356 “Line-format” from panes starting at the focus and moving towards the root.
361 The “completion” render is a filter. In response to a “render-line”
362 call it calls “render-line” on its parent and only returns lines that
363 start with a given prefix. It can also add highlights to rendered
364 text to distinguish the common prefix from the remainder.
366 A prefix is set by a “set-prefix” command. The response to this
367 indicates if the selected lines have a longer common prefix, and if
368 there is in fact only a single matching line. This supports the
369 implementation of filename, document name, command name completion
370 etc. To complete from a set of names, you just need to provide a
371 document which responds to “render-line” with the various options to
377 The HEX renderer provides an alternate “render-line” for a document
378 which starts each line at a multiple of 16 bytes from the start of the
379 document, and formats the next 16 bytes as hex and ASCII. Each
380 rendered line start with the byte offset of the start of the line.
385 The “tile” handler takes a pane (typically the root pane of a display)
386 and divides it up into 1 or more non-overlapping tiles. Tiles are
387 grouped in horizonal and vertical stacks. Tiles can be split, can be
388 discarded, or can be resized. Any of these operations may affect other
391 The leaves of the tile tree need to have some other pane attached. The
392 tiler doesn't render anything itself, not even borders. That is left
398 A “view” draws borders around a pane and provides a child pane which
399 is slightly smaller to allow for those borders (so it should probably
402 The borders can contain scroll-bars, a document name, or other
403 information provided by child panes.
408 The popup manager places a small window with an elevated 'z' value
409 somewhere relevant on the display and can provide a simple text document
410 for text entry - or can use a provided document. Various key strokes
411 are captured to allow the popup to be aborted, or to send the content
412 of the mini document to the originating pane.
414 The popup requests notifications from that pane so that if it is
415 closed, the popup automatically closes too.
420 The line-counter uses the model described earlier of placing marks
421 every few hundred lines in a document and using them to expedite line
422 counting. This currently isn't implemented as a pane. I wonder if it
428 “Keymap” allows both global and local keys (or arbitrary commands) to
429 be defined. The global mappings are handled at a pane which must be
430 stacked before the tiler. A pane to handle local mappings is added on
431 demand at the current focus pane.
436 “search” is like line-counter in that it provides a global command
437 rather than a pane. This command can perform a reg-ex search through
438 a document. Currently it only searches the per-element characters, so
439 it isn't useful on directory listings. It should be extended to work
440 with the results of “render-line”.
445 “Messageline” trims the bottom line off a pane (providing a pane which is
446 slightly smaller) and will displays messages in this pane until the
447 next keyboard command.
452 A pane of this module store some state related to the current input
453 context, including a modifier prefix and a repeat count.
455 When a “keystroke” command is received the prefix is added to the key
456 and this is sent as a command to the focus. The command includes the
457 repeat count, which gets cleared.
459 Commands are provided to set a new prefix or repeat count. So for
460 example “Meta-1” might multiply the repeat count in the command by 10,
461 add 1, and then ask 'input' to set that as the new repeat count for
467 This provides a set of named commands which can be given to “keymap”
468 as a global key map. In provides a number of emacs-like bindings.
473 This module allows python code to be run, provide an interface to
474 panes, marks, and commands, and allows commands to be called on a
475 given pane. It also allows commands to be defined in python that can
476 be called from other modules just like any other command. It is, or
477 will be, a complete two-way interface between python and other
478 languages to access the core edlib functionality.
484 This is an under-development display module written in python and
485 using pygtk for drawing.
487 When a “text” or “clear” request is made on a pane, the module
488 allocates a pixmap (arranging for it to be destroyed when the pane is
489 closed) and performs the drawings there. When a refresh is required,
490 the various pixmaps are combined and drawn to the target window.
492 There is currently no attempt to handle variable-width fonts. That
499 There is still so very much to do. At time of writing a lot of things
500 work and I can load and save files with filename completion, and I can
501 browse directories and search in text files. This is encouraging but
502 it is barely a start.
504 This a list of just some of the things I want to work on soon. You
505 might noticed that the above texts might suggest that some of them are
506 done already. In those cases I was being a little ahead of myself above.
508 - Generic notifications between panes: currently only a document can
511 - change commands that return a non-integer to do so by calling
512 the call-back command.
514 - The “complete” popup should be positioned above/below the file name,
515 not over the top of it. And typing should increase/decrease the
518 - render-lines should always re-render the line containing point, so
519 the location of 'point' can affect the rendering.
521 - allow searching in the rendered output as well as in the document
523 - support case-insensitive search an literal (non-regex) search.
525 - Create an append-only limited size document for a log of messages
526 and a log of keystrokes.
528 - use above to allow keyboard macros.
530 - create a 'mmap' document type so I can edit a block device without
533 - create a 'reflection' document so I can view the internal data structures.
535 - Generalize the “event loop” interface so that glib events can be
536 used when gtk is active.
538 - create clean threading interfaces so that I can have different event
539 loops in different threads and suitable locking so commands can be
540 called from any event loop.
542 - Support write-file (providing a file name) - currently I only save
543 to the file I loaded from.
545 - improve format options for status line.
551 - lots of work on pygtk interface
553 - allow a second (and more) ncurses display to be created.
555 - determine how best to support variable-sized fonts, both
556 non-constant-width and requested font size changing.
558 - improve ncurses code for choosing colours.
564 - spell check - async?
569 I have big dreams for edlib. How much will ever be reality is an open
570 question, but dreams are nice. Here are some, big an little.
575 I loved using emacs for reading mail, but it became too clumsy. I
576 want to go back to using the one editor for everything.
578 I imagine using notmuch as an indexing backend and viewing everything
579 via edlib. Viewing HTML would certainly be an interesting challenge
580 but I can probably live without that if everything else works well.
582 Attached images, including PDF, is important. I would certainly like
583 to be able to show images in a pane but have not designed anything for
589 I only recently discovered “emacsclient -t”. I think I like it. I
590 certainly want edlib to have a “server” mode and to be able to perform
591 editing in arbitrary terminal windows where I do other work.
593 calculator in floating pane
594 ---------------------------
596 I very often use 'bc' for hex/decimal conversion etc. But it is a
597 little clumsy. I want an easy calculator on my desktop with base
598 conversion. I'd like to use edlib. I imagine a small pop-up
599 appearing which automatically converts whatever I type.
604 Many years ago I started writing a spreadsheet program in emacs-lisp.
605 The document was a 'LaTeX' document with specially marked “tabular”
606 sections. Each cell was on a line by itself. It consisted of the
607 current appearance of the text, and then a comment containing the
608 formula and formatting rules.
610 The e-lisp code would hide the comment and the newlines so that it
611 looked like a table. When the focus entered a cell, it would switch
612 to displaying the formula, or optionally the formatting. Any change
613 to formula would propagate around the table.
615 It never worked very well. I felt I spent most of my time fighting
616 with emacs rather than writing useful code.
618 If I can make a spreadsheet work at least reasonably well with edlib,
619 then edlib will be a success.
624 Markdown? Commonmark? Some sort of markup is useful and there are
625 few if any that are both easy to write and highly functional.
627 I like writing with markdown, but it is far from complete. So I'd
628 like to create a mark-down mode that format as I type and which can
629 render to PDF. Then I would start extending it. Some extensions like
630 table are fairly standard - maybe ASCII-Math too.
632 Description-lists are an obvious (to me) omission. Anchors for links,
633 structure tags (author, title), foot notes, insets for
634 table/figure/equation etc, with labeling. Index, contents...
636 Figure Drawings? There are lots of drawing languages. It would be
637 nice to find a simple but versatile one to embed in markright.
640 Outline code rendering.
641 -----------------------
643 I like the principle of outlines and only showing the heading of
644 nearby sections, but the detail of the current section. I've always
645 found them a bit clumsy to use. I want the rendering to automatically
646 do what I want, partly depending on where the cursor is, partly
649 So when I am editing I want to see a lot of immediately surrounding
650 text, but I also want to see nearby minor headings and all major
652 If headings are short enough and there are enough of them, the having
653 several to a line would be a good idea - maybe even abbreviated. If I
654 click on an abbreviated heading in the middle of a line at the top of
655 the screen, then that section should open up, and the nearby sections
656 should get a bit more space.
658 When searching, I probably don't need as much context of the current
659 point, so less of the current section would be displayed, more of
660 surrounding sections. If those surrounding sections also contained
661 matches, then that would be the part of those sections that was shown.
663 I would like to implement this sort of view for markright mode, but
664 more importantly I want to implement it for C-mode. When editing C
665 code (which I do a lot) I want the few lines at the top and bottom of
666 the view to just list some function names. Then maybe a few lines
667 with function signature one the whole line.
669 Certainly the start of the current function would appear somewhere no
670 matter where in the function I am editing, and as many of the
671 variables as possible. If I am in an 'if' statement in a 'for' look,
672 then the loop header and the if condition would be displayed if at all
678 This is something that is very clumsy with emacs, and should be very
679 easy with edlib. A 'hexedit' view shows the hex value of each byte in
680 the file in a nice regular pattern. For ASCII chars, it also shows
681 the character separately.
686 I never quite got into using shell-mode in emacs - it never felt quite
687 as raw as an xterm. And the infinite history bothered me - possibly
690 Still, a shell mode for edlib might be a useful thing, though having
691 edlib easily available in any terminal window might be just as good.
693 If I did a shell mode, I would capture the output of each command into
694 a separate buffer, and display all those buffers in the one view.
695 Sufficiently old buffers would be discarded. Output from any recent
696 command could easily be saved or piped. It would be possible to
697 arrange for some interactive commands to also send output of each
698 command to a separate buffer.
700 Auto paging would be disabled (where possible) and edlib would page
701 output as needed. This means that `cat bigfile` could move the whole
702 file into a buffer, which wouldn't be good. If a 'less' command could
703 give the filename to edlib and let it display, that might be nice.
git / edlib.git / blob