Simple Dillo plugins

text only version | previous version

[see also(related and less ancient)]

dpi1-v1.0 (draft 10, with implementation)by: Jorge Arellano, Eric GaudetJul 26, 2003/*  This  document  is an hybrid between a draft and a doc. This isbecause  the full dpi1 spec isn't completely finished yet, but wealready have a working framework to develop with. */                     ---------------------                     Simple plugins (dpi1)                     ---------------------------------Introduction------------   Hopefully,  dillo  will  have  two  types of plugins: one of asimple  nature  (described  in this document), that operates in asimilar  way  of  CGIs  (with  some extensions), and another thatintegrates interaction with the rendering engine and its widgets.   The  fact  is  that  our  current  rendering  system,  and itsassociated  internal  layers  is  still under construction, makesalmost  impossible  to  begin  working  on dpi2, and on the otherhand, gives an excellent oportunity to focus on dpi1 (that's alsoexpected to be easier to program with).   We  expect  dpi1 to be incrementally extensible, but also verysimple in its concept at the same time.   Please bear the simplicity concept in mind while studying thisdocument. That's the basis for a powerful and clean dpi spec.----------Motivation----------   Dillo aims to be a small, fast and efficient web browser. Thiskind of PI is designed to push out of the main code features thatusually  are optional, and also some others that are required (asbookmarks for instance).   The  design  idea  is  to keep the main code-base small and tosimplify  the  development of new features, by not requiring a PIprogramer to know the whole internals of dillo.   This  way,  extensions  can  be easily coded, and there's alsoplace  for  developing unrelated programs that can use dillo as aGUI! --------Overview--------   Technically, the plugin program is a coprocess (an independentprogram  spawned  by  the  browser)  that  communicates  with thebrowser through e certain channel.   The current implementation uses unix domain sockets:    ---------    ------------->    ------   | Browser |     | socket |     |  PI  |    ---------    <-------------    ------   Note:  actually,  there's a daemon manager process involved inthe  communication between dillo and its PIs. It is documented in"doc/Dpid.txt",  but in the end, things work as the above diagramshows. PI developers should read both this spec and Dpid.txt.   The  main idea is to have a bidirectional channel on which thebrowser  and  the  plugin  can  communicate.  The  data  flow  isencapsulated in a small but flexible protocol (dpi protocol) thatresembles HTML tags.   For instance, for sending an status message from the plugin tothe browser:   <dpi cmd='send_status_message' msg='Hello browser!'>---------------------------Note that with this scheme:---------------------------   *  PI-programs  can  be  one-demand/one-response,  and there'sspace  for  multiple  negotiations  between  the  browser and theplugin too.   *  Status/progress messages can be passed back to the browser.   *  Browser and plugin can request information from each other,      and they can also set data!   *  It  is possible to handle persistent dpi1 processes, thoughnot a requirement.   *  It's  easy  to extend the protocol by adding new operationswithout affecting backward compatibility.   *  There's plenty of choice for PI languages.   *  Plugin  programs  are  managed with an intermediate daemon.This makes possible having several dillo instances using the sameplugin.-----------------How does it work?-----------------   In general terms, the browser starts (or contacts) the PI whenit finds something that belongs to it (as an URI), and handles itto  the PI. The PI receives the request, processes it and answersback  in  the  form  of an HTML page, or status message, or a dpicommand.   A simple session (one-demand/one-response) can look like this:   1.- The browser finds an URI that belongs to a certain PI.   2.- The browser prepares to send the URI to the PI.   3.- The PI starts.   4.- The browser handles the URI to the PI with a dpi command.   5.- The PI  makes/composes its answer (an HTML page) and sendsit back within dpi protocol datagrams.   6.- The PI acknowledges the browser when done,  and optionallyexits.-------------Some examples-------------   Before getting into the protocol itself, it may be ilustrativeto  look  at  some functionality that could be handled using thiskind  of  PIs.  For  those willing to participate in the protocoldeveloping process, it's certainly adviced to keep these examplesin mind for testing design ideas.   1) Downloads      * current cache allocates them in memory, so handling them      to an external program (PI) has several advantages.      *  the PI chooses which agent (wget, curl, ...) to use, and      sends feedback to dillo's download interface (HTML).      * Downloads don't stop if dillo crashes.   2) FTP      * to browse FTP directories (PI outputs HTML).      * Downloads using 1).      * progress messages (as loggin in, etc) can go to the      status bar.   3) Bookmarks      * The BM PI maintains the BM database (a single file!).      * The PI builds an HTML page for dillo.      *  Needs  coomunication  with the browser (URI, titles, add      bookmark, etc).      * Some of the functions can be handled with FORMS.      *  Browser  may  clean  BM-pages  out  of the stack, for      them not to show in history.      * Exporting BM becomes a matter of Save-page-as!      NOTE: All of this has been already implemented (BM).   4) Preferences (AKA configuration)      * PI requests information from the browser (current prefs).      * PI commands the browser to change its "state"      * PI maintains dillorc      * As 3), may require stack-cleanning of page.   5) Info reader.      * Just a handy info to html filter (very simple).      * one-demand/one-response.   6) Doc viewer (?)      * uses an external program to convert doc to html   7) Help browser      * A PI that knows of local documentation, presents an index      and probably a search feature!   8) A handler PI for "mailto:".   (...)  Well, that's the idea of simple PI. A way of extending dillo tosuit end-user needs, without requiring extensive knowledge of itsinternals.  Note  also  that these extensions don't make into thecore of the browser.                     -------------------------                     Developing a dillo plugin                     -------------------------   I  think  the  best  way to get started will be to explain thebasics  of  "dillo <=> PI" communication, then review the alreadyimplemented tags of the dpi protocol, give some tips, and finallylet you examine the code of the already developed plugins.   * Don't forget to read: "doc/Dpid.txt" (inside the tarball).-----------------Dpi communication-----------------   ------------   dillo => PI:   ------------   There're three ways:      1.- dpi tag      2.- static URL      3.- dynamic URL (with form data)   Note  that  all  of them are in fact dpi tags, the division ismade to separate some concepts.   1.-  The  first  one is for any specific dpi command sent fromthe browser to a PI. For example:   <dpi cmd='add_bookmark' url='' title='Here!'>   2.-  The  second  one  usually refers an URL "inside" the PI'sscope. It commands the PI to do something (usually to send back apage). For intance:   <dpi cmd='open_url' url='dpi:/bm/'>   In  this  case the PI must react sending the main page for theuser's bookmarks.   3.- The third one is the most versatile. It sends back (to thePI)  all  the  data  associated with a FORM that was most usuallysent by the PI itself before. For instance:   <dpi cmd='open_url'   url='dpi:/bm/modify?operation=modify&submit=submit.&s1=on'>   (all in a single line)  That's  how  the  PI  knows we want to modify the first sectionname of the bookmarks file.   ------------   PI => dillo:   ------------   There're two ways:      1.- dpi tag      2.- HTML page   1.-  The first one is the generic mechanism and usually servesto command the browser. For instance:   <dpi cmd='send_status_message' msg='Bookmark added!'>   2.- The second one tells the browser we'll send an HTML page.   <dpi cmd='start_send_page' url='dpi:/bm/'>                            ------------                            The protocol                            ------------   Amazingly  (if you've already tried the bookmarks PI), all itsfunctionallity is carried on with these few dpi tags:   dillo => PI:                                     <dpi cmd='open_url' url='%s'>    <dpi cmd='add_bookmark' url='%s' title='%s'>    <dpi cmd='chat' msg='%s'>       PI => dillo:                   <dpi cmd='start_send_page' url='%s'>    <dpi cmd='send_status_message' msg='%s'>    <dpi cmd='reload_request' url='%s'>    <dpi cmd='chat' msg='%s'>      So it is NOT that hard!----------------------------Tips for developing a new PI----------------------------   First,  examine  the  above mentioned protocol tags, then readthe  code  of hello.c ("Hello world" PI) and figure out its work.You should also have read Dpid.txt at this point.   After that, it'll be clear what the basic skeleton is, and howto send back a page to dillo.   The  next step is to make a FORM, get some data from the user,and  to send back a new page that manipulates the input data. Forthis:   1.- Design an HTML page with the form   2.- Design an HTML page with the desired answer skeleton   3.- Make your PI send the form as the main page       (as an answer to "dpi:/mydpi/", for instance)   Once you have that, you'll need to process the input data:   4.- Make your PI print what it receives so it becomes clear       what you're dealing with.   5.- Either try to process it right away, or study how it's       done in the 'Bmsrv_parse_buf' function of the bookmarks       PI.   Now you have the parsed data:   5.- Manipulate it somehow   6.- Build the answer page and send it back   If  you  made  this  far,  the  rest will be clear enough fromstudying the commented bookmarks' code!   If you don't want to make a form-based PI, but one that spawnsanother process you may start looking in the ftp PI.-------------------------------------What's missing in this implementation-------------------------------------   1.- Some dpi commands       (This is good in the sense of a protocol being finished       when the last uneeded commands are removed! :)   2.- PI start/initialization.   3.- All the menu handling (from PI to browser) is to be       defined   (there're some ideas about these near the end)   1.- Dpi commands   ----------------   From the set of dpi commands, I want to comment two:   start_send_page:     <dpi cmd='start_send_page' length='%d'>   This  was  meant to encapsulate HTML chunks of certain length.The  final  chunk  should  be marked with length='0' or somethingakin.  The main purpose of it is to allow communication after theHTML  page  is sent. That way, for instance, the bookmarks plugincould  send  a  staus  message  adding  more  information about aperformed operation (as "Deleted 3 bookmarks").   The  only reason I didn't code it from the start was to get toa working framework as fast a s possible. It's not hard though.   Currently   the  dpi  framework  uses  'start_send_page'  wichassumes  that  all  the following data, up to EOF, is part of theHTML page.   send_data:   This  will  be a generic way of exchanging information. Usefulfor sending/setting preferences for instance. Very similar to theprevious command, and much like the current 'chat' command.   BTW,  the 'chat' command is not necessary (just gave a look tothe  funny  and irrelevant talk between dillo and the bm PI!). Itwas just a way of testing bidirectional data exchange.   If you need to send/request some data, feel free to use 'chat'until 'send_data' is implemented.   2.- PI start/initialization   ---------------------------   There's  a  lot  more  info about it in the following pages. Apolished interface for registering/launching plugins is yet to bedeveloped.   We  want  dillo and dpid to be able to add, remove and upgradeplugins in a clear and transparent way. We're working on it...   3.- Menu handling   -----------------   Just read the following section!.                               ----------                        ------------------------                               SECTION II                        (this is the draft part)                        ------------------------   This  section  is mainly a gathering of ideas about what's notyet implemented:   1.- Dpi commands   2.- Menus   3.- PI initialization   4.- Plugins options   5.- Plan   6.- Future extensions   7.- Final notes------------Dpi commands------------  NOTE:  this  commands  list  is just a set of ideas. It doesn'tmean  they  will be implemented. We gather them here just to keeptrack of the interesting ideas they bring.dillo -> PI-----------CMD   : DpiAbort::Sent  by dillo to abort an "in-progress" plugin. Dillo might senda "kill -9" command to the PI if it doesn't answer.(most probably dillo will ask the dpid to kill a PI, and the dpiditself will send a kill signal to the PI).CMD   : DpiBye::The plugin is commanded to quit (after it's done).It  may  be  used  to  end  a "resident" PI that's doing nothing.Plugins can end and exit when finished or keep sleeping, but theyMUST acknowledge dillo when they're done on a per client basis.CMD   : DpiError::Sent by dillo when a command is not understood.PI -> dillo-----------CMD   : DpiRequestInfo::Sent to the browser to request information. For instance:  - Dillo version  - title of current page  - URL of current page  - preferences (the whole preferences as text)  - All active colors (background, text, link, visited link)CMD   : DpiSetData::Sent to the browser to change its "state". Example: preferences, menus.CMD   : DpiDone::Sent by the PI when finished servicing a client.-----Menus-----   Menus  need to be defined thinking on the needs of current andfuture PIs. The may not require menu entries (as the FTP PI), mayrequire a single menu entry, or a whole menu, or a button.   The key point here is to provide simple support for menus, andnot to complicate the protocol with seldom required features.<< simplified menu registration protocol --EG  Main goals:  -  a  plugin  may or may not want to be registered in the menus  when initializing: menus are optionnal.  -  a plugin may want to change some or all its menus entries at  runtime.  - the plugins menu looks like this: there's a "Plugins" menu in   dillo's menubar, where all plugins are shown, provided they   asked  for it during the init stage. Then it's up to the plugin  to ask this entry to be an url or a submenu.  Commands:CMD   : DpiMenuMenu commands:- insert: add an entry named <string> just after the entry ID. Dillo must answer with the new entry ID.- delete: delete an entry ID and all dependent sub-menus.- modify: modity the <string> of an entry.- submenu: make an entry a submenu.- url: give <string> as the url to be called when the menu entry is clicked.>>-----------------PI initialization-----------------  The main problem with running each PI everytime dillo starts isthe  added  startup  time. If, for instance, there's a PERL and aPython PI, the time increase is too much.  We're  working on a scheme that avoids running each plugin whendillo  starts. It has a "pluginsrc" kind of file with informationabout  each PI. The whole work is done by dpid, but there'll be acontrol program for the user interface. Something like:  dpidc [register | upgrade | stop | ...]  dpidc  communicates  with  dpid  using the same service requestsocket  that  dillo  currently  uses.  All the operations will bedefined with dpip tags.  What  I like from this approach is that it is very flexible andcan  be  implemented incrementally ("dpidc register" is enough tostart).       It  also  avoids the burden of having to periodically check thedpis directory structure for changes or updates.      It  also  lets shell scripts an easy way to do the "dirty" workof  installing  dpis;  this may help building a dillo package forcertain distros.  For instance, these commands may be added to the protocol:dillo -> dpi:                                     CMD   : DpiInit                      ::                                   It should include some challenge data to assert it's a dpi.                                     and from dpi->dillo:                                                      CMD   : DpiInitAnswer                ::Data is composed of a list of name=value pairs separated with ';'just like in an URL (it must answer the challenge).Example:  Name="bookmarks";prefix="bm";MenuEntry="View Bookmarks"---------------Plugins options--------------- There're  two  different  things  here, the PI registration data(managed  by  dpid in the 'pluginsrc' file for every PI), and thespecific  options  a  specific PI-program may provide for tunningits  behaviour. The later one should be managed by the PI programitself.----Plan----   Currently  the  dpi framework has working dpis for: bookmarks,ftp, downloads and hello.  The  plan  is  to  polish&stabilize  both,  the  dpis  and  theframwork, with a view to 0.8.0 release.-----------------Future extensions (needs a lot more thought)-----------------1.- Postfixed call to a Dillo-plugin   Dillo plugins  registered as such will be called if the loadedpage's  <extension> in "<url>.<extension>" matches a Dillo-pluginregistration,  or  if the PI has registered the same MIME type ofthat content, and thus the PI acts as a file-processing filter.   Dillo  will  call  the  Dillo-plugin  sending  to it the wholeloaded  file, and expecting some data back. This can be used as afile-to-html filter.   Example:   ""  willrender  the  rtf  file  as  html if a Dillo-plugin has registereditself as dealing with "rtf" files. -----------Final notes-----------   The draft part of this material is not "written on stone", butprovided  as a structural basis on which to develop the simple PIprotocol. Please take it with a pinch of salt, in the best spiritof a RFC.   As   you  can  see,  there's  plenty  of  material  here,  anddiscussing  threads can get very large if posts are not carefullythought  of.  Please take your time to study and fully understandthe material, and only submit well backed/tested ideas.   Thanks  a  lot to all of those that have contributed in makingdpi1 possible!