Laurence Rowe: Layering web applications on web services with JSON Linked Data

This talk is about the ENCODE portal, an encyclopedia of DNA elements.

I want to talk here about a pattern, not really about the specific technologies involved.

I work at the data coordination center, generating and organizing data. We get metadata submissions, store it in a metadata database. It is really a knowledge management system. It could have been built in Plone, but it would not be a great fit. I started from scratch based on Pyramid and ReactJS. Nowadays services more and more have a Javascript UI, where Javascript talks to the backend.

Embrace Javascript. There has been progressive enhancement. Single page web apps really need it. For building a portal, I have been looking at isomorphic web applications. Originally from the Render framework. It is important that pages are loading quickly. The exit rate for visitors goes just as quickly up with the loading time.

Json is the lowest common denominator for data. xml is more flexible, but more complex. In Python it is much easier to use json.

JSON-LD: json link data. Adopted recently by the w3c. It is partly about semantic data, but we are not using that yet.

At first we needed to duplicate the routing information on the server and the client side. JSON-LD allows us to express type information, which avoids the duplication.

You can have JSON-LD like this [in "pseudo json" here, just for the idea, Maurits]:

{
  @context: context/jsonld
  @id: ...
  @type: [biosample, item]
  @submitted_by: {
    @id: /users/lrowe
    @type: [user, item]
    name: Laurence Rowe
    }
}

Defined using JSON Schema, which is an IETF draft. Schema version, field definitions, types, validators. It is an extensible format.

All our normalized data is stored in Postgres as json. JSON-LD has the concept of framing, which objects should be embedded (mapping to a view in Plone terms), possibly doing some elasticsearch. Above it is server rendering (with NodeJs) that is creating html, and sends it to the browser. After the first page load, the browser uses Javascript to talk the the JSON-LD backend directly, instead of via the NodeJS server, letting the ReactJS Javascript do its own rendering.

The browser opens a page. Gets html back. Then it does a query for json data and gets that back and shows it on the page.

Indexing linked data. Updating one item may need a reindex of other items that link to it. We have written code for that. Using elasticsearch as a cache.

Come work with us, we are hiring.

See code at https://github.com/ENCODE-DCC

Watch the video and slides of this talk.