HTTP API

This documentation is part of Pod.

HTTP API is the interface that Pod provides to store and access user data. This document explains the data types that Pod can store, and current API provided for that.

Accessing your Pod requires authentication. This is typically the owner_key and auth keys in the request, see below.

The owner_key tells the Pod who you claim to be. It is the full client’s public ED25519 key (see also wikipedia). It should be encoded as 64-character hex string. Pods configured with “ANY” as their owner will accept requests from any owner_key. For debugging, you can fake owner_key by providing any 64 hex characters, e.g. all zeroes. Note that for now, we don’t really use any cryptography for the owner_key, but in the future we probably will.

The auth value is used by Pod to validate that this is you, and to temporarily decrypt parts of your data required to serve your request. This key is immediately lost by Pod when the HTTP request is closed.

There are two types of auth keys,

• a plugin key (that plugins get from Pod and will use to authorize in Pod),
• and a client key (that Memri clients will use, e.g. a phone or desktop app).

If you’re a memri client (mobile, desktop), you keep the secret locally, in an encrypted/protected form.

Your auth key should be of the form:

{
  "type": "ClientAuth",
  "databaseKey": "*********"
}

The databaseKey is a 64-character hex string to be used for validation and encryption. You can use any cryptographically secure PRNG to generate the key. It should be generated once and stored securely, e.g. in a hardware-backed keystore, or secure enclave, etc. The key will internally be used with sqlcipher.

For testing (and for testing only!) you can use any 64 hex characters, e.g. all zeroes.

Plugins receive authentication information when they are started by the Pod.

Their authentication key should be:

{
  "type": "PluginAuth",
  "data": $POD_AUTH_JSON
}

Where $POD_AUTH_JSON is the JSON set by Pod, see the link above again.

As a plugin, you only have access to a subset of Items and their properties, as configured by the user.

For testing/debugging, if you run your own Pod locally, you can use client auth (below) by just making up a fake user and a fake database key.

Schema defines what types can be stored in Pod. You can read more about the Schema itself here.

In order to make changes to the schema, insert items of a specific structure in the Pod. The following properties are expected:

{
  "type": "ItemPropertySchema", /* exactly this, and nothing else */
  "itemType": "YourType", /* e.g. "Person" */
  "propertyName": "yourProperty", /* e.g. "age" */
  "valueType": "Integer", /* one of valid Schema types, see Schema docs*/
}

If a Schema with the same propertyName but different valueType already exists, the request will fail (even if the itemType is different). (Explanation: this is done to support heterogeneous queries in the Pod.)

If the new Schema item creates (valid) item properties that have not yet been defined, the new Schema will be added permanently.

If the new Schema item has an id specified, the new item will be created and may potentially duplicate already existing Schema definition items. (Note that it is recommended to avoid specifying an id for Schema items except for Memri Webapp/Mobile Clients.)

If no id has been specified and the new Schema item duplicates an already existing Schema item, no new items will be created on the server and the id of the old item will be returned. Note that this slightly violates the semantic meaning of item creation, however, this is the explicit choice due to the special meaning of the Schema items (and lack of a need for logic duplicates).

⚠️ UNSTABLE: We might require more properties to be defined here in the future, e.g. to what Plugin does the Schema addition belong to.

The edge schema is defined through ItemEdgeSchema items. It is not possible to add edges to the Pod without the matching ItemEdgeSchema present. The following format is expected:

{
  "type": "ItemEdgeSchema",
  "sourceType": "Message",
  "targetType": "Account",
  "edgeName": "sentBy"
}

The above json would define a directed edge Message --sentBy--> Account. Edges are defined separately for each sourceType, which means that a different sentBy schema could be defined for a sourceType that is not a Message.

The Pod allows edges with multiple target types (type unions). These unions can be defined in the pod by simply creating multiple ItemEdgeSchema items with the same edgeName and sourceType. For example, an edge with target types Message --sentBy--> [Account | Person] should be saved in the pod by two ItemEdgeSchemas, where targetType is Account for the first item, and Person for the second item.

Get version of the Pod: the git commit and cargo version that it was built from.

{
  "auth": $auth_json,
  "payload": "$id"
}

Get a single item by its id.

Returns an empty array if an item is not found, or an array with 1 item if item exists.

{
  "auth": $auth_json,
  "payload": { "type": "...", ... }
}

Create a single item.

• id if set, will be taken as new item’s id; otherwise, a new ID will be generated by Pod. It’s recommended not to set this property unless you’ll also need to reference this item for e.g. edges to/from this item. If you need to create a random unique ID, it’s recommended to use 32 random hex characters. Example of generating that in shell: hexdump --no-squeezing -e '/1 "%x"' -n 16 /dev/urandom. If you want to incorporate an external ID, you can use e.g. "myPluginName-12342" where 12342 is an example of an external ID.
• type sets the type of the item, cannot ever be changed
• dateCreated if not present, will be set by the backend
• dateModified if not present, will be set by the backend
• dateServerModified will be created by the backend
• deleted and all other properties will be added to the item’s properties

Returns an error if such id already exist in the DB. Returns an error if the new item doesn’t conform to the Schema. Returns id of the created item if the operation is successful.

{
  "auth": $auth_json,
  "payload": { "id": "$id", ... }
}

Update a single item.

• id from the input json will be taken as the item’s id
• type from the input json will be ignored
• dateCreated from the input json will be ignored
• dateModified if not present, will be updated by the backend
• dateServerModified will be created by the backend
• properties with a value of null will be erased from the item
• any other properties will be updated

Returns an empty JSON object if the operation is successful.

{
  "auth": $auth_json,
  "payload": {
    "item": "$id",
    "direction": "Outgoing", // Either "Outgoing" (default) or "Incoming" edges
    "expandItems": true // Whether to expand the target/source items for each edge
  }
}

Get all edges for a single item.

Example output:

[
  {
    "name": "friend",
    "item": {
      "id": "00000000",
      // all other fields if "expandItems" is requested
    }
  },
  {
    "name": "friend",
    "item": {
      "id": ".........."
    }
  },
  ...
]

Returns an array empty array if either the element does not exist or if it has no outgoing edges.

⚠ WARNING: this endpoint is unstable, and it might be deprecated and removed in next releases of Pod.️

{
  "auth": $auth_json,
  "payload": {
    "_source": "$source_id", /* Source item ID */
    "_target": "$target_id", /* Target item ID */
    "_name": "$edge_name",  /* Text name. For example: "entry" (in a list), "friend" (for a Person), etc */
    "_self": "$self_id", /* Optional field to specify the "self" ID to link to, see below */
    // Edge properties will be supported in the future
  }
}

Create a single directed edge item from an already existing item to another already existing item. ⚠️ UNSTABLE: As of the current implementation, an additional type="Edge" item will be created holding the additional properties of the edge. This implementation detail is unstable, in the future we might either keep “Edge” as the item type name, or potentially introduce type-specific type for the item, such as “EdgePersonAddress” or “_EdgePersonAddress” etc.

If the _self field is specified in the request, the edge will be wired to an already existing item. For Plugins this functionality is currently undocumented and advised not to be used. For Clients, see “edges” table definition.

An error will be returned if the edge already exists. An error will be returned if source item or target item do not exist. Returns id of the created edge if the operation is successful (for now, nothing is possible to do with the ID, but in the future there will be more options).

Returns an error if such id already exist in the DB. Returns an error if the new item doesn’t conform to the Schema. Returns id of the created item if the operation is successful.

{
  "auth": $auth_json,
  "payload": {
    "_source": "$source_id", /* Source item ID */
    "_target": "$target_id", /* Target item ID */
  }
}

Unlink the edge between source item ID and target item ID, and mark the underlying edge item as deleted.

Returns an error if the edge does not exist.

Returns an empty JSON object if the operation is successful.

{
  "auth": $auth_json,
  "payload": "$id"
}

Mark an item as deleted:

• Set deleted flag to true
• Update dateModified (server’s time is taken)
• Update dateServerModified

{
  "auth": $auth_json,
  "payload": {
    "id": "1d22ba3458147f73b8aa3d68744e3f2f", // ID filter
    "type": "Label", // type filter
    "dateServerModified>=": 1234567890, // date filter
    "dateServerModified<": 1234567890,
    "deleted": false, // deleted filter
    "_sortOrder": "Asc", // sort by server modification date, either "Asc" (by default) or "Desc"
    "_limit": 100, // minimum number of items to return, see below
    "_returnProperties": ["propertyA", "propertyB", ...],
    "[[edges]]": {}, // include all forward edges in the response
    "~[[edges]]": {}, // include all backward edges in the response
  }
}

Search items by their properties, all fields above are optional.

The endpoint will return an array of items which have all the properties above.

As a first step of the 2021-03 Pod rewrite, only the above properties are supported. In the future, any item properties will be available.

If _limit is specified, response will include the first _limit number of items from the database, plus also all other items with exactly the same dateServerModified as the last one.

If _returnProperties is specified, then only the specified properties and the relevant edges information will be returned.

The properties of "[[edges]]" and "~[[edges]]" are “magic constants” for now. This will change in the future when we’ll continuously expand our API to make it more intuitive and easier to use, especially around edges and recursive data fetching. For now only literally those two properties are supported, and all edges are returned without filtering.

{
  "auth": $auth_json,
  "payload": {
    query {
      Message {
        subject
        service
        sender {
          displayName
        },
        receiver: {
          displayName
        }
      }
      count
    }
  }
}

Fetch custom data using GraphQL standards. Returns data in the query structure provided, in form {"data":[]}.

⚠️ UNSTABLE: This api is currently still under development and only supports a subset of the full graphQL specification. For further information see its separate documentation.

{
  "auth": $auth_json,
  "payload": {
    "createItems": [
      { "id": "something-12345", "type": "Person", /* ... */ },
      {}, // same structure as create_item endpoint above
      // ...
    ],
    "updateItems": [
      { "id": "something-67899", /* ... */ },
      {}, // same structure as update_item endpoint above
      // ...
    ],
    "deleteItems": [ "$id", "$id", "$id", /* ... */ ],
    "createEdges": [
      {}, // same structure as create_edge endpoint above
      {},
      // ...
    ],
    "search": [
      { /* same structure as in /search API */ }
    ]
  }
}

Perform a bulk of operations in one request. The endpoint is “atomic”, meaning that either all the operations succeed, or the database won’t be changed at all.

Returns an empty JSON object if the operation is successful.

Plugins help getting data into your Pod and enriching it. Plugins must be authorized and started by the user.

In the future we plan to have configurable permissions and automatic rules to start plugins. For now, plugins are started when items of a particular types are inserted into Pod.

Items of the following structure need to be inserted to Pod to start a plugin:

{
  "type": "PluginRun", /* exactly this, and nothing else */
  "containerImage": "your_docker_image:optional_version", /* any locally available docker container if you run Pod locally */
  "targetItemId": "the item id this plugin needs to run against, NOT this item's id",
  /* Any other optional fields that your plugin might need... */
}

See Plugins on how plugins are started exactly.

⚠️ UNSTABLE: We might require more properties for Plugins to start in the future, e.g. permission limitation.

{
  "auth": $auth_json,
  "payload": "$id"
}

Get the logs of a pluginrun container, specified by the pluginrun id

Returns

{
  "logs": "<log_string>"
}

Email API is a temporary functionality of Pod that allows it to send emails.

To send an item to a particular user via the Pod, make the following API request:

{
  "auth": $auth_json,
  "payload": {
    "to": "address@example.com",
    "subject": "Example subject",
    "body": "Example message body."
  }
}

This will make Pod send that single email and return a success if the email was accepted by the configured SMTP server.

No retries will be made if SMTP connection did not succeed.

No changes will be made to the database.

⚠️ UNSTABLE: the API is unstable and might be removed without prior notice.

RAW-FILE-BINARY

Upload a file into Pod and verify it’s sha256. owner_key, database key and sha256 are all hex-encoded.

If a file with a given sha256 has already been uploaded, the request will fail. If the provided sha256 doesn’t match the hash of the contents, the request will fail. If no item with this sha256 exists in the database, Pod wouldn’t be able to store cryptographic information about the file, and the request will also fail.

If sha256 matches, the file has not yet been uploaded to Pod and if an item with such sha256 already exists in DB, Pod will accept the file and store it. The properties nonce and key will be updated for this item.

RAW-FILE-BINARY

The auth should be a URL-encoded Authorization JSON (see above).

Otherwise, this API endpoint behaves exactly like upload_file above.

{
  "auth": $auth_json,
  "payload": {
    "sha256": "$sha256"
  }
}

Get a file by its sha256 hash.

Returns 404 NOT FOUND if the item describing the file does not exist. Returns 404 NOT FOUND if the file has not been uploaded yet.

Returns the contents of the file if the item describing the file exists and the file has already been uploaded to the Pod.

POD is responsible for managing oauth2 tokens for platforms (like gitlab.memri.io) used by clients, either frontend, or plugins. Client shall get access token by calling oauth2/access_token And use it as long as possible. Upon request failure that was caused by token revocation or expiration, client should call oauth2/access_token again in order to retrieve new one.

{
  "auth": $auth_json,
  "payload": {
      "scopes": list, platform specific,
      "redirectUri": URL to which browser will redirect after user action, string,
      "platform": currently "gitlab" only supported
  }
}

Creates an authorization url that should be presented to the user. Most likely used by frontend, or for testing purpouses. Example response:

https://gitlab.memri.io/oauth/authorize?response_type=code&client_id=fd94965d76207c614a2798feaf78666e38364078a256a8f31e20e4773fabb018&state=EfqI60oIcFxhi8IDaJnTHw&redirect_uri=http%3A%2F%2Flocalhost%3A8080&scope=read_user

{
  "auth": $auth_json,
  "payload": {
      "authCode": code from auth_url response,
      "redirectUri": must be the same as for auth_url request,
      "platform": "gitlab"
    }
}

Creates new access_token from given authCode Example response:

{
  "accessToken": "cdc18d98dfd93bb4d8ea05054359538cc62f0d440012a69eda16cf1bd0c9d5d7",
  "tokenType": "bearer"
}

{
  "auth": $auth_json,
  "payload": {
      "platform": "gitlab"
    }
}

Returns currently valid access_token for given platform

{
  "accessToken": "cdc18d98dfd93bb4d8ea05054359538cc62f0d440012a69eda16cf1bd0c9d5d7",
  "tokenType": "bearer"
}