DocPad Database Architecture Vision

[balupton]

In relation to #445 (abstract all the things) and my comment here about a possible abstraction architecture #702 (comment)

I'd like to use this issue as way to implement the goal of abstracting out the DocPad architecture, while at the same time addressing our performance and memory issues, and allowing us to evolve to the demands of the future.

I've outlined the following architecture already, that being:

• Core Experience
  • DocPad CLI Module
  • DocPad Core Module
    • DocPad Generate Module
    • DocPad Plugin Module
    • DocPad File Model & Collection Module
    • DocPad Block Model & Collection Module
  • DocPad Watching Plugin
  • DocPad Growl Plugin
• Extra Experience
  • Renderer Plugins
  • Helper Plugins
  • Source Plugins (formerly importers)
  • Interface Plugins

Tying this together, it would look something like this:

The parts on the right being source plugins, that serve pulling in different sources of content (formerly called importers), and keeping them up to date. The decision for the rename is that the name "importers" reflects only a one way data transfer, whereas with this new architecture, the source plugins would serve as bidirectional data transfer. Import into DocPad, and export into the original source.

The parts on the left being renderer plugins, which serve as modifying content during the generation phase.

I feel we can have a very basic file model, which event system serves as a mediator that DocPad then listens to, to perform generations and saves.

We should be able to abstract this in such a way that other static site generators, content management systems, and generic node modules can use this. As well as plugins would be written against the content database, rather than against DocPad, but with DocPad providing extra features and conventions etc to make things easier.

[geertbrit]

How about an api-endpoint as part of the Core experience to push CUD
documents to docpad? This could nicely sit beside the Watch Plugin and
share the same processing flow. Advantage being the option to connect
arbitrary external systems as producers.

On Tue, Nov 12, 2013 at 7:51 AM, Benjamin Arthur Lupton <
notifications@github.com> wrote:

In relation to #445 #445 all the things) and my comment here about a possible abstraction
architecture #702 (comment)#702 (comment)

I'd like to use this issue as way to implement the goal of abstracting out
the DocPad architecture, while at the same time addressing our performance
and memory issues, and allowing us to evolve to the demands of the future.

I've outlined the following architecture already, that being:

• Core Experience
  • DocPad CLI Module
  • DocPad Core Module
    • DocPad Generate Module
    • DocPad Plugin Module
    • DocPad File Model & Collection Module
    • DocPad Block Model & Collection Module
      • DocPad Watching Plugin
  • DocPad Growl Plugin
    • Extra Experience
  • Renderer Plugins
  • Helper Plugins
  • Source Plugins (formerly importers)
  • Interface Plugins

Tying this together, it would look something like this:

[image: Sketch of DocPad Architecture Vision]https://github-camo.global.ssl.fastly.net/5f8d78ade567f611688a8f6b4f721d49287ff3d8/687474703a2f2f642e70722f662f514c74752b

The parts on the right being source plugins, that serve pulling in
different sources of content (formerly called importers), and keeping them
up to date. The decision for the rename is that the name "importers"
reflects only a one way data transfer, whereas with this new architecture,
the source plugins would serve as bidirectional data transfer. Import into
DocPad, and export into the original source.

The parts on the left being renderer plugins, which serve as modifying
content during the generation phase.

I feel we can have a very basic file model, which event system serves as a
mediator that DocPad then listens to, to perform generations and saves.

We should be able to abstract this in such a way that other static site
generators, content management systems, and generic node modules can use
this. As well as plugins would be written against the content database,
rather than against DocPad, but with DocPad providing extra features and
conventions etc to make things easier.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/705
.

[balupton]

@gebrits good idea, can you provide more of an example?

Would this be like the restapi plugin or like #543

[geertbrit]

@balupton: the restapi plugin and #543 (the stuff on the the top in moss
green) seem to be overlapping at least the push part of it (i.e: from
express, geddy, etc TO docpad, and not the other way around), correct? But
indeed that's essentially what I'm thinking about. It would allow all sorts
of editor/Ide frontends to push documents to Docpad and have Docpad produce
the static html from it.

probably superfluous but: to do this efficiently I guess there needs to be
some notion of a dependency graph between documents (and other assets) .
I.e: which document changes (may) influence changes in other docs. This
could be used to calculate the smallest subset of documents that would have
to be reprocessed when a create/update/delete comes in. I could elaborate
if I'm not making sense.

On Wed, Nov 13, 2013 at 12:24 AM, Benjamin Arthur Lupton <
notifications@github.com> wrote:

@gebrits https://github.com/gebrits good idea, can you provide more of
an example?

Would this be like the restapi plugin http://docpad.org/p/restapi or
like #543 #543

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/705#issuecomment-28344407
.

[pflannery]

Continuing from #709

What's your thoughts on applying an interception technique to the data model?

[pflannery]

Thought I would add a quick representation of how interception could be used
This isn't anything real just a pretend object graph example

DocumentModel

    # Optional cached layer on a per model basis
    CachedModel (optional caching interceptor)

        # Optional logging layer
        ModelLogging (optional logging interceptor)

        # servicer's interceptor layer
            FileModel (Uses the CRUD Interface)

            GitModel (Uses the CRUD Interface)

            TumbrlModel (Uses the CRUD Interface)

            StockMarketTickerModel (Uses the CRUD Interface)

            Storage Model

                # storage servicer's
                LocalDBModel (Uses the CRUD Interface)

                MirrorApiModel (Uses the CRUD Interface)

    # well you get the picture

Typical CRUD Interface
    create
    read
    update
    delete

The graph above would be quite powerful allowing data to be dynamic, stored locally, stored externally, mirrored, read in from one and then write to another etc..etc..

Another thing goes with kind of pattern and would help create optimised V8 code

[balupton]

@gebrits

probably superfluous but: to do this efficiently I guess there needs to be
some notion of a dependency graph between documents (and other assets) .
I.e: which document changes (may) influence changes in other docs. This
could be used to calculate the smallest subset of documents that would have
to be reprocessed when a create/update/delete comes in. I could elaborate
if I'm not making sense.

Is this what you mean? #336

[geertbrit]

@balupton: yes indeed: Having a map of doc --refs --> [doc] instead of the current referenceOthers: true|false .

Once that's in place an intelligent (and optimal) algo for #359 'trivially' becomes a Toplogical Sort http://en.wikipedia.org/wiki/Topological_sorting

On Tue, Nov 19, 2013 at 5:33 PM, Benjamin Arthur Lupton <
notifications@github.com> wrote:

@gebrits https://github.com/gebrits

probably superfluous but: to do this efficiently I guess there needs to be
some notion of a dependency graph between documents (and other assets) .
I.e: which document changes (may) influence changes in other docs. This
could be used to calculate the smallest subset of documents that would have
to be reprocessed when a create/update/delete comes in. I could elaborate
if I'm not making sense.

Is this what you mean? #336 #336

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/705#issuecomment-28805340
.

[balupton]

For anyone keen, let's try for a Google Hangout on Air tomorrow. Sometime from 11am to 5pm Berlin Time works for me. Let me know what time in there works for each of you, and we'll lock in the most common time by 10am.

Windows of availability:

• @balupton: 10am to 5pm
• @greduan: 2pm to 11pm
• everyone: 2pm to 5pm

[greduan]

Tomorrow anything after 7:00 and before 16:00 is good for me. Mexico City time.

In Berlin Time it would be 14:00 to 23:00.

According to http://www.timebie.com/timezone/berlinmexicocity.php

[geertbrit]

14:00 - 17:00 Berlin Time would works for me.

On Tue, Nov 19, 2013 at 7:15 PM, Eduán Lávaque notifications@github.comwrote:

Tomorrow anything after 7:00 and before 16:00 is good for me. Mexico City
time.

In Berlin Time it would be 14:00 to 23:00.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/705#issuecomment-28816866
.

[greduan]

@balupton Add one more hour of availability. I am now available till 17:00 which is 24:00 in Berlin time.

[balupton]

Cool. Let's do 3pm/15:00 then :)

[balupton]

Agenda for the meeting.

The importing process is slow, we need to fix this.

This is the process:

1. DocPad emits populateCollections
   1. Tumblr plugin requests latest data from Tumblr API (with caching)
   2. Tumblr plugin gets results back
   3. Tumblr plugin injects results into DocPad models
   4. Tumblr plugin fires event completion callback
2. DocPad initial generate starts
   1. DocPad parses+contextualizes tumblr models
   2. DocPad renders tumblr documents
   3. DocPad writes tumblr documents

The slowness comes from every step besides 2.3. However the majority of the slowness comes from 1.3 until 2.3, as the 1.1 and 1.2 is actually really fast due to the caching of data.

We can speed up time to generation by {having data pulled in in the background}:

• A. Calling step 1.4 right away, and having steps 1.1-1.3 occur in the background, calling a generate action once completed
  • This needs to be turned off when doing one-off compiles, such as docpad generate (as it would cause one-off compiles to have incomplete data), and should only be enabled when doing docpad run

We can speed up injection time by {caching tumblr documents as physical files}:

• B. Having the tumblr plugin write the documents to the source directory by using the writeSource: once header.
  • This suggestion requires A to be implemented, otherwise it will be just as slow.
  • This suggestion requires that the tumblr plugin updates the existing models, rather than adding duplicates.

We can speed up rendering time by {going directly to out files at the start}:

• C. Turning off DocPad's HTTP request delay when we are doing the initial generation to have the ability to serve the out directory via the static middleware
  • This needs to be opt-in, as it can cause havoc on dynamic websites

We can speed up the entire process by {caching the entire database}:

• D. Injecting a cache step between 2.2 and 2.3. That caches the rendered results to an external database, and loads up that database on initial load.
  • This means that rather than doing initial generations each time, we do one initial generation, cache the result, then load up the cache each time.
  • This would have significant performance improvements for all DocPad sites.
  • A question here, is whether or not this caching should be in the core, or whether it should be a plugin.

We can speed up generating and rendering by only rendering documents as we need them:

• E. When doing a long-running DocPad instance, documents are only rendered when they are requested.

[geertbrit]

Sorry about this, but something came up so I can't make it.
Some quick thoughts:

• D. seems like low-hanging fruit. All for it. (Although I'm really not up
  to par with you guys on the internals of Docpad obviously)
• In between 2.1 and 2.2 there's probably the stuff I was talking about on
  keeping a cache of docs referencing eachother. Doing this together with D.
  would enable to only do 2.2 and 2.3 on the needed subset, i.e the
  (transitively) changed docs.

On Wed, Nov 20, 2013 at 1:02 PM, Benjamin Arthur Lupton <
notifications@github.com> wrote:

Agenda for the meeting.

The importing process is slow, we need to fix this.

This is the process:

1. DocPad emits populateCollections
   1. Tumblr plugin requests latest data from Tumblr API (with caching)
   2. Tumblr plugin gets results back
   3. Tumblr plugin injects results into DocPad models
   4. Tumblr plugin fires event completion callback
      1. DocPad initial generate starts
   5. DocPad parses+contextualizes tumblr models
   6. DocPad renders tumblr documents
   7. DocPad writes tumblr documents

The slowness comes from every step besides 2.3.

We can speed up time to generation by:

• A. Calling step 1.4 right away, and having steps 1.1-1.3 occur in
  the background, calling a generate action once completed

We can speed up injection time by:

• B. Having the tumblr plugin write the documents to the source
  directory by using the writeSource: once header.
  • This suggestion requires A to be implemented, otherwise it will
    be just as slow.
  • This suggestion requires that the tumblr plugin updates the
    existing models, rather than adding duplicates.

We can speed up rendering time by:

• C. Turning off DocPad's HTTP request delay when we are doing the
  initial generation to have the ability to serve the out directory via
  the static middleware
  • This needs to be opt-in, as it can cause havoc on dynamic websites

We can speed up the entire process by:

• D. Injecting a cache step between 2.2 and 2.3. That caches the
  rendered results to an external database, and loads up that database on
  initial load.
  • This means that rather than doing initial generations each time,
    we do one initial generation, cache the result, then load up the cache each
    time.
  • This would have significant performance improvements for all
    DocPad sites.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/705#issuecomment-28884118
.

[pflannery]

wow is google hangouts that bad or was the "join hangout" disabled? I sat here for over 30 mins trying to get involved with no luck.