User manual

This document will provide the necessary information to use D1 Storage and introduce essential concepts and terminology. For a detailed description of the gRPC API, see the specification.

Configuration

By default D1 Storage reads its configuration from the TOML file config.toml. This behaviour can be modified by pointing the environment variable D1_CONFIGFILE at the path to the desired config file. The supported file formats are TOML, YAML, and JSON.

All configuration options are documented in the following example configuration:

# Key Provider configuration
[keys]
  # Static Key Provider configuration with cryptographic keys. Must be 64 hex digits (256 bits).
  [keys.static]
  kek = "0000000000000000000000000000000000000000000000000000000000000000"
  aek = "0000000000000000000000000000000000000000000000000000000000000001"
  tek = "0000000000000000000000000000000000000000000000000000000000000002"
  iek = "0000000000000000000000000000000000000000000000000000000000000003"

  # # K1 Key Provider configuration.
  # [keys.k1]
  # # The endpoint where the K1 server can be reached.
  # endpoint = "k1:50051"
  # # Enable or disable TLS.
  # tls = false
  # # Path to any additional CA certificates.
  # tlscacert = ""
  # # Path to a client certificate.
  # tlscert = ""
  # # Path to a client private key.
  # tlskey = ""
  # # The KIK ID and base64 encoded key generated by the K1 server.
  # kikid = "EXAMPLE5-53f2-40be-a837-6afc5c63b453"
  # kik = "EXAMPLEbq+HfGIHQTfhZiINRE5Jh4dozazZhMEVZ87Q="

# IO Provider configuration
[io]
  # # Redis configuration
  # [io.redis]
  # # Address of the Redis server.
  # address = "localhost:6379"
  # # Enable or disable TLS.
  # tls = false
  # # Path to any additional CA certificates.
  # tlscacert = ""
  # # Path to a client certificate.
  # tlscert = ""
  # # Path to a client private key.
  # tlskey = ""

  # # TiKV configuration
  # [io.tikv]
  # # Address of the PD.
  # address = "localhost:2379"
  # # Path to any additional CA certificates.
  # tlscacert = ""
  # # Path to a client certificate.
  # tlscert = ""
  # # Path to a client private key.
  # tlskey = ""

  # S3 configuration
  [io.s3]
  # Hostname for an S3 compatible endpoint
  address = "http://localhost:7000"
  # Name of the bucket.
  bucket = "objects"
  # S3 region
  region = "europe-west-4"
  # Key ID and secret key
  id = "storageid"
  key = "storagekey"
  # Path to any additional CA certificates.
  tlscacert = ""
  # Path to a client certificate.
  tlscert = ""
  # Path to a client private key.
  tlskey = ""

  # # SQL configuration
  # [io.sql]
  # # The name of the SQL driver. Must be one of: 'mysql', 'postgres', 'sqlite'.
  # driver = "sqlite"
  # # The connection string for the specific database.
  # source = "file:data.db"

  # # MongoDB configuration
  # [io.mongodb]
  # # Address of the MongoDB database.
  # address = "mongodb://localhost:2707"
  # # Name of the database.
  # database = "database"

  # # Etcd configuration
  # [io.etcd]
  # # Addresses of the etcd key-value store.
  # addresses = [ "localhost:4001", "localhost:4002", "localhost:4003" ]
  # # Path to any additional CA certificates.
  # tlscacert = ""
  # # Path to a client certificate.
  # tlscert = ""
  # # Path to a client private key.
  # tlskey = ""

  # # Azure Blob configuration
  # [io.azureblob]
  # # Address of the Azure Blob storage.
  # address = "http://localhost:10000"
  # # Account name
  # accountname = ""
  # # Account key
  # accountkey = ""
  # # Container name
  # container = ""

# ID Provider configuration
[id]
  # Standalone configuration
  [id.standalone]
  uek = "0000000000000000000000000000000000000000000000000000000000000004"
  gek = "0000000000000000000000000000000000000000000000000000000000000005"
  tek = "0000000000000000000000000000000000000000000000000000000000000006"

  # # OIDC configuration
  # [id.oidc]
  # # Address of the OIDC issuer.
  # issuer = "https://login.example.com"
  # # Path to any additional CA certificates.
  # tlscacert = ""
  # # Path to a client certificate.
  # tlscert = ""
  # # Path to a client private key.
  # tlskey = ""
  # # OIDC client ID
  # clientid = "client-id"
  # # Signing algorithm used by the issuer.
  # signingalg = "ES512"
  # # Groups and scopes are defined by mapping claims in the ID token. The mapping is done using
  # # JSONPath expressions.
  # claimtranslation = [
  #   {jsonpath = '.groups[?(@=="admin")]', target = "admin", groupid = "admin" , scopes = "rcudgmi"},
  #   {jsonpath = '.groups[?(@=="dev")]', target = "dev", groupid = "dev" , scopes = "rc"},
  # ]

All configuration options can be overwritten by a corresponding environment variable. For example, the name of the S3 Bucket can be overwritten by setting D1_IO_S3_BUCKET.

The configuration is divided into 3 sections. Each section is briefly described below.

Keys configs

Keys are used by D1 Storage to secure confidentiality and integrity of the data. Therefore make sure that these are generated securely and randomly. The data cannot be accessed without the keys, so make sure to have a proper backup.

IO config

The IO config determines which IO Provider is used to store operational data such as encrypted access lists. Only one IO Provider can be configured at at time. Currently the following IO Providers are supported:

ID Config

The ID config determines which ID Provider is used to convert tokens into identities. Two ID providers are currently supported:

Standalone: When enabled D1 Storage exposes a simple built-in IAM system through a gRPC service. See the section on Standalone User Management for more details.
OIDC: When enabled D1 Storage will refer to an external OIDC provider to authenticate and authorize users. See the section on OIDC User Management for more details.

Authentication

All authentication on D1 Storage is done via an authorization pair in gRPC metadata. It should contain the user access token and be in the form bearer <user access token>. See examples of how to pass the user access token when using gRPCurl in the Users and Groups section.

How to obtain an access token depends on which ID Provider D1 Storage is configured to use. For the Standalone provider the token is obtained upon user login as described in the Standalone User Management section. When using the OIDC provider you will have to obtain an OIDC ID Token in the usual way.

Users and Groups

A user is an authentication entity in D1 Storage and is only represented by a user ID. The user ID is used to identify a single user and how it is defined depends on the ID Provider.

A group is an authorization entity in D1 Storage and is only represented by a group ID. The group ID is used to identify one or more users who are members of the group. A group has a set of scopes which determine what endpoints its users are authorized to access. How groups are defined depends on the ID Provider.

An object is a cryptographically signed package containing the stored ciphertext and the associated data. A user can share stored data with other users/groups by modifying the permissions on an object. Instructions on how to share data between users/groups can be found in the section on Permissions. The user that creates an object automatically has permission to acccess that object, and any user/group that can access an object can modify the permissions.

It should be noted that while the access tokens should be kept secret, user IDs can be considered public information and safely shared between parties.

Standalone User Management

This section describes how to manage users when using the built-in Standalone ID Provider.

Bootstrapping users

To bootstrap D1 Storage with an initial user, execute ./d1-service-storage create-user <scopes>. Here, <scopes> is a string describing the scopes the user's group should have. Each scope is mapped to a character as described in the table below. The scopes are described in further detail in the API documentation. E.g., to create a user with READ and CREATE scopes, call ./d1-service-storage create-user rc.

Scope	Character
`READ`	`r`
`CREATE`	`c`
`UPDATE`	`u`
`DELETE`	`d`
`GETACCESS`	`g`
`MODIFYACCESS`	`m`
`INDEX`	`i`

Note that users are only valid for other D1 Storage instances that use the same key material. Information on bootstrapping in Docker environment is provided in the following section.

Docker example

If using docker run:

docker exec <CONTAINER ID> /d1-service-storage create-user <scopes>

Creating users through the API

To create a user through the API, you need to call the d1.authn.Authn.CreateUser endpoint. The request should contain an attribute named scopes which enumerates all the scopes the user's initial group should have.

Once a user has been created, a new user_id and password will be returned from the call. Note that a group with the requested scopes and an ID equal to the user_id is automatically created.

Throughout the next couple of sections, some small examples will be shown. The examples will illustrate how to call the endpoints using gRPCurl, a command-line tool that can be used to interact with gRPC servers.

Creating a new user:

grpcurl -plaintext -H "authorization: bearer <access token>" \
-d '{
      "scopes": ["READ", "CREATE", "GETACCESS", "MODIFYACCESS"]
}' \
localhost:9000 d1.authn.Authn.CreateUser

Output:

{
  "userId": "44c8fa82-f8ed-46b0-94d1-8921a19c0d62",
  "password": "Vju86gvJTEKK9zBIZAHloa2K0y2Vw_eJC7icmmCP-jc"
}

Note how the access token is passed as a header in the input above.

When a user is created, you get the User ID and the password. You can use this information to obtain the access token, which is necessary for authentication towards the API. Note that the access token is short lived (1 hour). You can login by calling the d1.authn.Authn.LoginUser endpoint. Provide the User ID and the password in your request object.

Logging in:

grpcurl -plaintext \
-d '{
      "user_id": "44c8fa82-f8ed-46b0-94d1-8921a19c0d62",
      "password": "Vju86gvJTEKK9zBIZAHloa2K0y2Vw_eJC7icmmCP-jc"
}' \
localhost:9000 d1.authn.Authn.LoginUser

Output:

{
  "accessToken": "<access token>",
  "expiryTime": "1652793412"
}

Remove user

To remove a user, you need to call the d1.authn.Authn.RemoveUser endpoint. The request must contain the user_id of the user to be removed. If the request was successful, you will receive an empty response.

Creating groups

To create a group through the API, you need to call the d1.authn.Authn.CreateGroup endpoint. The request should contain an attribute named scopes which enumerates all the scopes the user's initial group should have. Once a group has been created, a new group_id will be returned from the call.

Creating a new group:

grpcurl -plaintext -H "authorization: bearer <access token>" \
-d '{
      "scopes": ["READ", "CREATE", "GETACCESS", "MODIFYACCESS"]
}' \
localhost:9000 d1.authn.Authn.CreateGroup

Output:

{
  "groupId": "7b2d0dc5-3021-47ee-a30d-0adda2ae13d6"
}

Adding and removing users

In order to modify the members of a group, you need to call the d1.authn.Authn.AddUserToGroup and d1.authn.Authn.RemoveUserFromGroup endpoints. In both cases the request should contain the group_id of the group in question and the user_id of the user to be added/removed.

OIDC User Management

When authenticating and authorizing via an OIDC provider D1 Generic will translate claims in the ID Token to a specified group with some fixed scopes. This translation is controlled by the following configuration (see also the Configuration section):

[id.oidc]
# Groups and scopes are defined by mapping claims in the ID token. The mapping is done using
# JSONPath expressions.
claimtranslation = [
 {jsonpath = '.groups[?(@=="admin")]', target = "admin", groupid = "admin" , scopes = "rcudgmi"},
 {jsonpath = '.groups[?(@=="dev")]', target = "dev", groupid = "dev" , scopes = "rc"},
]

The claim translation is an array of individual translations (one for each group) consisting of four parts each:

jsonpath: A JSONPath template which should extract one particular value from the ID Token. In the above example, we expect an array claim called groups, and we look for the value admin in the array.
target: If the JSONPath template evaluates to the target value, then the translation was successful and user is part of the specified group.
groupid: The name of the group.
scopes: The scopes of the group. See Bootstrapping Users for a reference.

Encrypted Storage

Store

You can store an object by using the d1.storage.Storage.Store endpoint. The caller needs the CREATE scope in order to use this endpoint. You need to provide the plaintext and the associatedData. The response of this call will contain the objectId. Note that the associatedData is authenticated, but not encrypted.

Storing data:

grpcurl -plaintext -H "authorization: bearer <access token>" \
-d '{
      "plaintext": "1234",
      "associatedData": "5678"
}' \
localhost:9000 d1.storage.Storage.Store

Output:

{
  "objectId": "d7190dfa-778b-4b57-a1f4-3464ced21696"
}

Retrieve

To retrieve an object, you need to call the d1.storage.Storage.Retrieve endpoint and provide the objectId in the request. This endpoint requires the READ scope. If you are authenticated towards the API and authorized to read the object, the response will contain the plaintext and the associatedData.

Retrieving data:

grpcurl -plaintext -H "authorization: bearer <access token>" \
-d '{
      "objectId": "d7190dfa-778b-4b57-a1f4-3464ced21696"
}' \
localhost:9000 d1.storage.Storage.Retrieve

Output:

{
  "plaintext": "1234",
  "associatedData": "5678"
}

Update

You can update the data stored in an existing object by using the d1.storage.Storage.Update endpoint. The caller needs the UPDATE scope in order to use this endpoint. You need to provide the plaintext and the associatedData. If you are authenticated towards the API and authorized to read the object, the response will be empty.

Note that it is not safe to call the Update endpoint concurrently with Retrieve or Delete.

Updating stored data:

grpcurl -plaintext -H "authorization: bearer <access token>" \
-d '{
      "plaintext": "1234",
      "associatedData": "5678",
      "objectId": "d7190dfa-778b-4b57-a1f4-3464ced21696"
}' \
localhost:9000 d1.storage.Storage.Update

Output:

{}

Delete

To delete an existing object, you need to call the d1.storage.Storage.Delete endpoint and provide the objectId in the request. This endpoint requires the DELETE scope. If you are authenticated towards the API and authorized to read the object, the response will be empty.

Note that it is not safe to call the Delete endpoint concurrently with Retrieve or Update.

Deleting stored data:

grpcurl -plaintext -H "authorization: bearer <access token>" \
-d '{
      "objectId": "d7190dfa-778b-4b57-a1f4-3464ced21696"
}' \
localhost:9000 d1.storage.Storage.Delete

Output:

{}

Secure index API

You can use a secure index to search in encrypted data. For more information about how it works, see the "Searchable encrypted data" section in the CYBERCRYPT D1 Library explainer.

Add

To add keywords/identifier pairs to the secure index, you need to call the d1.index.Index.Add endpoint and provide the keywords and the identifier in the request. Note that multiple keywords can be added to the same identifier at the same time. This endpoint requires the INDEX scope.

Add keywords/identifier pairs:

grpcurl -plaintext -H "authorization: bearer <access token>" \
-d '{
      "keywords": ["keyword1", "keyword2", "keyword3"],
      "identifier": "id1"
}' \
localhost:9000 d1.index.Index.Add

Output:

{}

Search

To search for a keyword, you need to call the d1.index.Index.Search endpoint and provide the keyword in the request. If you are authenticated towards the API and authorized to use the secure index, the response will contain the identifiers that contain the keyword. This endpoint requires the INDEX scope.

Search for a keyword:

grpcurl -plaintext -H "authorization: bearer <access token>" \
-d '{
      "keyword": "keyword1"
}' \
localhost:9000 d1.index.Index.Search

Output:

{
  "identifiers": ["id1"]
}

Delete

To delete keywords/identifier pairs from the secure index, you need to call the d1.index.Index.Delete endpoint and provide the keywords and the identifier in the request. Note that multiple keywords can be deleted from the same identifier at the same time. This endpoint requires the INDEX scope.

Delete keywords/identifier pairs:

grpcurl -plaintext -H "authorization: bearer <access token>" \
-d '{
      "keywords": ["keyword1", "keyword2", "keyword3"],
      "identifier": "id1"
}' \
localhost:9000 d1.index.Index.Delete

Output:

{}

Permissions

Access to an object is shared through the concept of object permissions.

Every object has a list associated with it with the group IDs of the groups who are able to access and modify the object. In order to modify the permission list, the user must be in a group that has access to the object (i.e. is in the permission list of the object).

Get permissions of an object

To get the permission list of an object, you need to call the d1.authz.Authz.GetPermissions endpoint. To access this endpoint the GETACCESS scope is required. The operation will return a list of group_ids that have access to the object.

Getting an object's permission list:

grpcurl -plaintext -H "authorization: bearer <access token>" \
-d '{
      "object_id": "d7190dfa-778b-4b57-a1f4-3464ced21696"
}' \
localhost:9000 d1.authz.Authz.GetPermissions

Output:

{
  "groupIds": [
    "44c8fa82-f8ed-46b0-94d1-8921a19c0d62"
  ]
}

Add permissions to an object

To add a group to the permission list of an object, you need to call the d1.authz.Authz.AddPermission endpoint. To access this endpoint the MODIFYACCESS scope is required.

Adding permissions to an object: Input:

grpcurl -plaintext -H "authorization: bearer <access token>" \
-d '{
      "object_id": "d7190dfa-778b-4b57-a1f4-3464ced21696",
      "group_id": "7b2d0dc5-3021-47ee-a30d-0adda2ae13d6"
}' \
localhost:9000 d1.authz.Authz.AddPermission

Output:

{}

Remove permissions from an object

To remove a group's permission from an object, you need to call the d1.authz.Authz.RemovePermission endpoint. To access this endpoint the MODIFYACCESS scope is required.

Removing permissions from an object:

grpcurl -plaintext -H "authorization: bearer <access token>" \
-d '{
      "object_id": "d7190dfa-778b-4b57-a1f4-3464ced21696",
      "group_id": "7b2d0dc5-3021-47ee-a30d-0adda2ae13d6"
}' \
localhost:9000 d1.authz.Authz.RemovePermission

Output:

{}

Version

To get version information about the running encryption service, you need to call the d1.version.Version.Version endpoint. Currently, the endpoint returns the git commit hash and an optional git tag. This endpoint does not need any scopes but requires the user to be authenticated by presenting a valid access token.

Getting the version of the service:

grpcurl -plaintext -H "authorization: bearer <access token>" localhost:9000 d1.version.Version.Version

Output:

{
  "commit": "be023bbdc2aa3d7e8b7648692477f93cef688d43",
  "tag": "v0.1.13"
}

User manual

Configuration​

Keys configs​

IO config​

ID Config​

Authentication​

Users and Groups​

Standalone User Management​

Bootstrapping users​

Docker example​

Creating users through the API​

User login​

Remove user​

Creating groups​

Adding and removing users​

OIDC User Management​

Encrypted Storage​

Store​

Retrieve​

Update​

Delete​

Secure index API​

Add​

Search​

Delete​

Permissions​

Get permissions of an object​

Add permissions to an object​

Remove permissions from an object​

Version​

Configuration

Keys configs

IO config

ID Config

Authentication

Users and Groups

Standalone User Management

Bootstrapping users

Docker example

Creating users through the API

User login

Remove user

Creating groups

Adding and removing users

OIDC User Management

Encrypted Storage

Store

Retrieve

Update

Delete

Secure index API

Add

Search

Delete

Permissions

Get permissions of an object

Add permissions to an object

Remove permissions from an object

Version