DynamoDB

DynamoDB is the obvious choice for AppSync as they together provide a truly serverless setup. DynamoDB tables can scale up and down easily and AWS also offers per-request pricing (called on-demand capacity). This makes it possible for the API to scale to any load and also to go back to zero.

Also, GraphQL's concept of many individual resolvers usually maps to DynamoDB tables easily. As it is a NoSQL database that does not support complex queries but has no problems handling tons of simple ones, it can work well with AppSync.

In this chapter we'll look into how to write resolvers that fetch and store data in DynamoDB and what are the best practices to adapt the data model for efficient GraphQL queries.

Configuring the data source

When you create a DynamoDB data source, you need to define a table and a role. The formes defines what table to send the requests (when it's missing, such as for transactions, as we'll see later), and the latter gives AppSync permissions to do the operations.

DynamoDB data source configured with a table and an IAM role

With the IAM role setting, you can give fine-grained permissions for each data source. For example, if you don't add write permissions then AppSync won't be able to change data in the table.

One role or multiple?

With a data source per table, you have the choice to use a separate role for each of them, in contrast to using a single role with all permissions AppSync needs.

So, which approach is better?

I found that using separate roles adds too much verbosity and brings very little benefits. Since it brings no security benefits, I usually go with a single role per AppSync API.

AppSync APIDynamoDB TableIAM RoleService:appsync.amazonaws.comAction:sts:AssumeRoleAction:dynamodb:Query
AppSync permission model for DynamoDB

Operations

DynamoDB resolvers implement the same operations (GetItem, Query, Scan, DeleteItem, UpdateItem, etc) as you'd find anywhere else and while the structure is different they work the same. This means all usual constraints apply: you can only get an item with the full key, you can query with the partition key, and scanning is expensive. Moreover, you can use indices (local and global) the same way as in other languages.

To find out what structure each operation expects, the best resource is the official documentation. It contains examples for each section.

Note that you can't copy-paste DynamoDB code from other languages as the resolvers require a different structure. For example, this PutItemCommand uses the Javascript v3 SDK:

{
  TableName: COUNTS_TABLE,
  Item: {
    type: {S: "users"},
    count: {N: "0"},
  },
  ConditionExpression: "attribute_not_exists(#pk)",
  ExpressionAttributeNames: {"#pk": "type"},
}

Converted to an AppSync resolver:

{
  "version": "2018-05-29",
  "operation": "PutItem",
  "key": {
    "type": {"S": "users"},
    "count": {"N": "0"}
  },
  "condition": {
    "expression": "attribute_not_exists(id)",
    "expressionNames": {
      "#pk": "type"
    }
  }
}
Expression names and values

If you read the documentation you'll see that the expressionNames and the expressionValues are present in multiple places:

  • "update"
  • "query"
  • "filter"
  • "condition"

While it's not obvious, these collapse into a single value. This means if you have both an update and a condition block with the same expression names, one will overwrite the other. Make sure you use different names in different blocks.

Error handling is a common theme in all DynamoDB operations and it follows the usual AppSync way: when an operation fails, the response mapping template will get the error in the $ctx.error value. By default, the resolver should throw an error when the operation fails:

#if($ctx.error)
  $util.error($ctx.error.message, $ctx.error.type)
#end
...
Getting elements

(Official docs)

The GetItem operation retrieves an element defined by its full key. For example, to implement a resolver for the field groupById(id: String!): Group, use:

{
  "version": "2018-05-29",
  "operation": "GetItem",
  "key": {
    "id": {"S": $util.toJson($ctx.args.id)}
  },
  "consistentRead": true
}

The resolver gets the id argument as $ctx.args.id and constructs the input for the data source. The table has only the id defined as a key, so specifying a value for that is enough.

«AppSync»AppSync resolver[]«DynamoDB»DynamoDB Table[]idGetItemkey:{"id": {"S": id}}itemitem
GetItem operation

The $util.toJson structure is important even though things usually work without it. It converts the argument value to a JSON string, making sure that every special character is safely escaped. VTL is string concatenation, so any user input that goes to the result directly opens a way for injection attacks.

Then notice the DynamoDB type system: "id": {"S": "123"}. The data source follows the same structure as other utilities for types, so it's always "<property>": {"type": "value"}, even for numbers: {"N": "15"}.

AppSync provides a few utility functions that convert these types into native JSON types, but I find myself opting out of them. A similar helpes exists for filter expressions too.

Finally, the "consistentRead": true sends a strongly consistent read, meaning the result will always be the most up-to-date version of the item. While it is more expensive, it provides an opt-out of DynamoDB's eventual consistency model.

Item keys

As DynamoDB is a key-value store, you need to specify the full ID of the item.

The result is the a JSON with the properties of the item converted to native JSON types. This means, apart from error handling, if it matches the GraphQL schema then it can be returned as-is.

#if($ctx.error)
  $util.error($ctx.error.message, $ctx.error.type)
#end
$util.toJson($ctx.result)
Running queries

(Official docs)

To get a list of items from DynamoDB, you can use a Query. This is when you specify the partition key and get an ordered list of results. As is common in DynamoDB data modeling, you need to plan in advance: if your API needs to send a query that gets elements for a given partition key, a table or an index needs to be in place to support that. We'll cover this aspect a bit more in the Data modeling chapter.

The data source supports queries on indices (local and global) as well as the table itself. In the query expression you need to define the partition key, and optionally a range for the range key.

For example, if users belong to groups and there is a global secondary index (GSI) with the partition key as the group ID, this resolver mapping template returns users belonging to a given group:

{
  "version": "2018-05-29",
  "operation": "Query",
  "index": "groupId",
  "query": {
    "expression": "#groupId = :groupId",
    "expressionNames": {
      "#groupId": "groupId"
    },
    "expressionValues": {
      ":groupId": {"S": $util.toJson($ctx.source.id)}
    }
  }
}
«AppSync»AppSync resolver[]«DynamoDB»DynamoDB Table[]idQuery"#groupId = :groupId"":groupId": {"S": id}itemsnextTokenusersnextToken
Query operation

The consistentRead is also a valid argument for queries, but only for the table itself or a local secondary index (LSI). This is again a limitation of DynamoDB.

Also, the scanIndexForward defines whether the items are read in increasing or decreasing order. Finally, the limit defines the maximum number of items the query can return.

Limit

Limit sets the maximum number of items, there are no guarantees that the result will return that many elements even if the table has that many items.

The result is an object with an items and a nextToken field. The former contains the result items while the latter is important for pagination, which we'll see in the Pagination chapter. Here, you can return this structure without changing, or you can map the result properties to another names:

#if($ctx.error)
  $util.error($ctx.error.message, $ctx.error.type)
#end
{
  "users": $utils.toJson($ctx.result.items),
  "nextToken": $util.toJson($ctx.result.nextToken)
}
Query vs Scan

The other operation that returns a list is the Scan. The difference is that Scan evaluates all items in the table, while Query reads only those with a given partition key. While it's not a problem for small tables, it becomes extremely costly and slow to run scans when the table has many items.

Try to store your data in a way that you don't need to use scans only queries.

Pagination

Pagination is an important topic in DynamoDB as every operation that returns a list of items return pages. This means the AppSync resolver might not get all the results of a query just some, along with a continuation token called nextToken. If that token is null then there are guaranteed that a query returns no more results. If it is non-null then sending it along with the same query might return more items. To get all the elements for a query or a scan, send the same operation over and over again passing the previous token until the nextToken becomes null.

«AppSync»AppSync resolver[]«DynamoDB»DynamoDB Table[]idQuery"#groupId = :groupId"":groupId": iditem1, item2nextToken: <token1>item1, item2nextToken: <token1>idnextToken: <token1>Query"#groupId = :groupId"":groupId": idnextToken: <token1>item3item3
Paginating DynamoDB queries with nextToken

This is called cursor-based pagination as a cursor value (the nextToken) is used to get the next page of items. In the SQL world the so-called offset-based pagination is more prevalent, where you define an OFFSET that skips the first n items. Both have advantages and disadvantages, and the main reason why DynamoDB opted for cursors is that all operations run fast no matter which page you fetch.

DynamoDB pagination has some strange properties that might feel overly restrictive at first but are the direct consequences of one fundamental idea: all operations must be fast. Because of this, the database guarantees only a few things in terms of pagination.

If you use limit, the number of the returned items will never be more than the limit specified. On the other hand, it can have fewer items than the specified limit, even if the table has more results. In an extreme case, it can happen that a query returns zero items but a non-null nextToken and only the next page will start returning data. Moreover, it can happen that client needs to fetch several pages but all of them containg zero elements. The only guarantee here is that if there are more items then the nextToken will be non-null and that eventually it becomes null.

These limitations are inherent to DynamoDB and have nothing to do with AppSync. But as an API developer, it's good to know about them so that consumers of the API can have the right expectations.

Limit and nextToken

The nextToken defines where DynamoDB starts searching for elements. Think of it as a range query that means "start from here" and that it has no concept of previously sent queries.

Because of this, the limit only affects the current operation. If you want to get 10 items and the first query returns 3, set the limit to 7 for the next page.

The AppSync resolver can only send one query, which means it can fetch only a single page. Purely with the DynamoDB data source you can't implement a "fetch all items"-type functionality. While you could use a Lambda function between AppSync and the database that implements this, I don't recommend it. AppSync resolvers are meant to return a small amount of data and should run fast. Writing them in a way that might require an unbounded amount of time (for paging through potentially a large number of items) will reach a threshold eventually.

It is a best practice to expose DynamoDB pagination through the GraphQL schema and not hide it behind some abstraction. This way the clients can decide how to fetch pages and when to stop.

To implement this, whenever the API returns a list based on a DynamoDB query or a scan, add a nextToken argument and an indirection that exposes the result's nextToken.

type User {
  id: ID!
  name: String!
}

type PaginatedUsers {
  users: [User!]!
  nextToken: String
}

type Group {
  id: ID!
  name: String!
  users(count: Int, nextToken: String): PaginatedUsers!
}

The resolver can then use the argument, potentially also adding a count:

{
  "version": "2018-05-29",
  "operation": "Query",
  "index": "groupId",
  "query": {
    "expression": "#groupId = :groupId",
    "expressionNames": {
      "#groupId": "groupId"
    },
    "expressionValues": {
      ":groupId": {"S": $util.toJson($ctx.source.id)}
    }
  },
  "limit": $util.toJson($ctx.args.count),
  "nextToken": $util.toJson($ctx.args.nextToken)
}

Then the response mapping template can extract the items and the token from the data source's response:

#if($ctx.error)
  $util.error($ctx.error.message, $ctx.error.type)
#end
{
  "users": $utils.toJson($ctx.result.items),
  "nextToken": $util.toJson($ctx.result.nextToken)
}
«AppSync»Query.users resolver[]«DynamoDB»DynamoDB Table[]users {name}Query{items: [user1, user2],nextToken:"<token1>"}user1, user2,nextTokenusers(nextToken: "<token1>"){name}QuerynextToken: "<token1>"{items: [user3],nextToken: null}user3
DynamoDB pagination in GraphQL
Pagination guarantees

Note that by directly exposing the DynamoDB pagination to the clients, the GraphQL API also inherits all its problems. Consumers of the API need to handle things like zero-result pages.

Changing data

(Official docs)

Storing single items into DynamoDB tables is straightforward: you need to define the keys and the attributes. For example, this resolver adds an item with a generated ID and the provided name:

{
  "version": "2018-05-29",
  "operation": "PutItem",
  "key": {
    "id": {"S": "$util.autoId()"}
  },
  "attributeValues": {
    "name": {"S": $util.toJson($ctx.args.name)}
  }
}

The $util.autoId() is an AppSync-provided function that generates a random UUID string. Using a random ID is a best practice for databases that don't provide auto-incrementing values as it's unlikely that they will collide.

The data source returns the inserted item without the DynamoDB type markup, so if the structure matches the GraphQL schema it is enough to check for errors and return the result:

#if($ctx.error)
  $util.error($ctx.error.message, $ctx.error.type)
#end
$util.toJson($ctx.result)

When you need to use the generated ID in multiple places, such as when it is also present in an attribute, you can save it to a variable:

#set($id=$util.autoId())

And to use it in a pipeline resolver, store it in the stash:

$util.qr($ctx.stash.put("id", $util.autoId()))

Deleting items works the same, just without the attribute values (Official docs):

{
  "version": "2018-05-29",
  "operation": "DeleteItem",
  "key": {
    "id": {"S": $util.toJson($ctx.args.id)}
  }
}

And for updating, use the UpdateItem operation with an update expression (Official docs):

{
  "version": "2018-05-29",
  "operation": "UpdateItem",
  "key": {
    "id": {"S": $util.toJson($ctx.args.id)}
  },
  "update": {
    "expression": "SET #groupId = :groupId"
    "expressionNames": {
      "#groupId": "groupId"
    },
    "expressionValues": {
      ":groupId": {"S": $util.toJson($ctx.args.groupId)}
    }
  },
}
Conditions

All operations that change data supports conditions. A condition expression is evaluated atomically during the write and will fail the operation if it evaluates to false. This is called optimistic concurrency control as it relies on the assumption that concurrent modifications of the same data is rare. Combined with transactions, you can implement strong data consistency measures across several tables in DynamoDB.

{
  "version": "2018-05-29",
  "operation": "UpdateItem",
  "key": {
    "id": {"S": $util.toJson($ctx.args.id)}
  },
  "update": {
    "expression": "SET #groupId = :groupId",
    "expressionNames": {
      "#groupId": "groupId"
    },
    "expressionValues": {
      ":groupId": {"S": $util.toJson($ctx.args.groupId)}
    }
  },
  "condition": {
    "expression": "attribute_not_exists(#groupId)"
  }
}

Retrying

DynamoDB can return errors for all sorts of reasons. The service might be degraded or the capacity might not be enough, even for on-demand capacity mode. Moreover, conditions can fail if multiple operations change the same data, and transactions can easily produce transaction conflict errors. Failures are expected, so plan for them.

AppSync does not offer automatic retrying. If an operation fails, the resolver response will include the error that the mapping template can return to the caller, but there is no easy built-in way of running the same query again.

As a best practice, throw the error to the caller and let them retry the operation. This way, they can define a retry strategy that suits the application and the overall solution will be more resilient.

Retrying algorithm

A good way to implement retrying is to use an exponential backoff with added jitter. See here for more info.

Transactions

(Official docs)

The data source supports DynamoDB transactions too. With it you can implement atomic updates to multiple objects even across different tables. This is an extremely useful tool that opens a lot of possibilities that were not available in DynamoDB before: unique constraints, foreign keys, accurate counts.

Transactions feel like an afterthought for the DynamoDB data source as you need to define a table for the data source then a table for each individual transaction item. Then the table configured for the data source will be simply ignored, but on the other hand the resolver itself will need to know the tabl names which can be a problem when they are generated by a tool.

This transaction inserts a user into the users table but also checks if the referenced group exists:

{
  "version": "2018-05-29",
  "operation": "TransactWriteItems",
  "transactItems": [
    {
      "table": "user-13b359215bd4d5fb",
      "operation": "PutItem",
      "key": {
        "id" : {"S": "$util.autoId()"}
      },
      "attributeValues": {
        "name": {"S": $util.toJson($ctx.args.name)},
        "groupId": {"S": $util.toJson($ctx.args.groupId)}
      }
    },
    {
      "table": "group-13b359215bd4d5fb",
      "operation": "ConditionCheck",
      "key":{
        "id": {"S": $util.toJson($ctx.args.groupId)}
      },
      "condition":{
        "expression": "attribute_exists(#pk)",
        "expressionNames": {
          "#pk": "id"
        }
      }
    }
  ]
}

The role that is configured for the data source needs to have access to all the tables that the transaction touches. This makes it easier to use a single role for all AppSync operations.

The usual DynamoDB constraints apply here: maximum 25 items can be included in a transaction, and each item can only be included once. Moreover, as DynamoDB locks the items as part of the operation, concurrent reads and writes can fail with a transaction conflict error.

The result of the data source for the TransactWriteItems operation is a list with the keys for each transaction item. For example, to check for errors then return the first item's id:

#if($ctx.error)
  $util.error($ctx.error.message, $ctx.error.type)
#end
$util.toJson($ctx.result.keys[0].id)
Transactions and batches

DynamoDB supports another construct that operates on multiple items: batches. The main difference between the two is that transactions are atomic (either all or none of the operations are effective) while batches can partially fail.

Batches are good when you can easily retry failed operations, which is not easy to implement in AppSync. I almost never use them with resolvers because of this.

Data modeling

Choosing how to store data in DynamoDB is all about access patterns. Since the only performant operations are the item-level ones, where you know the key, and the Query, where you know the partition key, you need to think about what data the API will need and store it accordingly.

In this chapter, we'll see a few basic cases that happens most of the time when using DynamoDB with AppSync.

Mapping references

When an item references another item then make sure that it stores the target's key. For example, if a User references a Group, such as in this schema:

type User {
  id: ID!
  group: Group!
}

type Group {
  id: ID!
}

The entry for the user should contain the group's id.

Users tableid (PK)namegroup_iduser1User 1group1user2User 2group2Groups tableid (PK)namegroup1base_usersgroup2administrators
Storing a reference to another object

The resolver can use the id from the source object (the User) to do a GetItem operation to fetch the target (the Group).

An implementation for the User.group resolver:

{
  "version": "2018-05-29",
  "operation": "GetItem",
  "key": {
    "id": {"S": $util.toJson($ctx.source.group_id)}
  },
  "consistentRead": true
}
2-way mapping

When objects have one-to-one relations, such as a User can reference another User and the other User can reference back, you need to store the id of the other object on both ends even if it is redundant.

For example, let's say Users can pair-program and each participant references the other one. This relation is mapped with this template:

type User {
  id: ID!
  paired_with: User
}
Users tableidnamepaired_withuser1User 1user3user2User 2user3User 3user1
Storing references on both ends

With this structure, both participants can move to the other end with a simple GetItem:

{
  "version": "2018-05-29",
  "operation": "GetItem",
  "key": {
    "id": {"S": $util.toJson($ctx.source.paired_with)}
  },
  "consistentRead": true
}

Since this introduces redundancy (the same data is stored twice), updating needs some extra care. Use a transaction to update both ends atomically.

Mapping lists

Efficient queries in DynamoDB relies on a composite key: one part, the partition key, is known, and the result is an ordered list of the other part. Usually, this naturally maps to GraphQL: when you want to get the users in a group, the resolver knows the group's id. To get employees under a manager, the manager's id is known. Or appointments for a doctor, then the doctor's id is in the source object. The only thing needed is a table or an index with a partition key as the known value. Then you can optionally define the sort key for ordering.

Usually, the target of the query will be a GSI (global secondary index) as it gives the most versatility.

For example, let's implement a data structure that allows efficient querying of users in a group:

type User {
  id: ID!
}

type Group {
  id: ID!
  users: [User!]!
}
Users tableid (PK)namegroup_iduser1User 1group1user2User 2group2user3User 2group1Groups tableid (PK)namegroup1base_usersgroup2administratorsUsers GSIgroup_id (PK)id (SK)group1user1group2user2group1user3
Storing data in a GSI for efficient querying

The resolver can then run a Query operation on the GSI using the group_id as the partition key:

{
  "version": "2018-05-29",
  "operation": "Query",
  "index": "groupId",
  "query": {
    "expression": "#groupId = :groupId",
    "expressionNames": {
      "#groupId": "group_id"
    },
    "expressionValues": {
      ":groupId": {"S": $util.toJson($ctx.source.id)}
    }
  }
}
Eventual consistency

Queries on GSIs are eventually consistent which means the newly changed data might not be immediately available in the results. This is usually not a problem, but since DynamoDB does not guarantee an upper limit on how long this inconsistency can last, it's good to know about it.

Ordering results

Adding a sort key enables ordering and efficient range queries. For example, if users have a last_active field:

type User {
  id: ID!
  last_active: AWSDate!
}

type Group {
  id: ID!
  users: [User!]!
}

Then the GSI can include that as the sort key.

Users tableid (PK)namegroup_idlast_activeuser1User 1group12022-06-15user2User 2group22022-06-16user3User 2group12022-01-01Users GSIgroup_id (PK)last_active (SK)idgroup12022-06-15user1group22022-06-16user2group12022-01-01user3Groups tableid (PK)namegroup1base_usersgroup2administrators
Using a sort key with a GSI

Here, the last_active determines the ordering of the items, so querying group3 will return user3 then user1, in that order (or in reverse, depending on the scanIndexForward attribute).

Efficient filtering

DynamoDB supports filter expressions that you can use to define what items you need in the result. While this seems like a solution so many filtering-related problems, it is woefully inefficient in some cases.

The filter expression works by first reading the table, then dropping the non-conforming items. Because of this, the performance will be similar to not filtering at all, that can be a problem when there are many dropped items.

Let's say a GraphQL field can return user items that have a specific status field:

type User {
  id: ID!
  status: String!
}

type Group {
  id: ID!
  users(status: String!): [User!]!
}

If the table has many users and the requested status is rare then clients will receive many empty or almost empty pages.

Users tableid (PK)namegroup_idstatususer1User 1group1INACTIVEuser2User 2group1INACTIVE............user1000User 1000group1ACTIVEUsers GSIgroup_id (PK)id (SK)statusgroup1user1INACTIVEgroup1user2INACTIVEgroup1...INACTIVEgroup1user1000ACTIVEGroups tableid (PK)namegroup1base_users
Inefficient filtering is similar to performance to no filtering at all

Efficient filtering works by adding the status to the partition key so that the query returns only the matching entities and nothing is dropped from the result. This is done by string concatenation in a single key, usually with the # symbol.

Users tableid (PK)namegroup_idstatusgroup_id#statususer1User 1group1INACTIVEgroup1#INACTIVEuser2User 2group2INACTIVEgroup2#INACTIVEuser3User 3group1ACTIVEgroup1#ACTIVEUsers GSIgroup_id#status (PK)id (SK)group1#INACTIVEuser1group2#INACTIVEuser2group1#ACTIVEuser3Groups tableid (PK)namegroup1base_usersgroup2administrators
Adding the filtered fields to the partition key allows efficient filtering

Since the resolver knows the group_id (from the source object) as well as the status (from the resolver argument), it can construct the partition key for the query.

#set($groupIdStatus =
  $util.escapeJavaScript($ctx.source.id) +
  "#" +
  $util.escapeJavaScript($ctx.args.status)
)
{
  "version": "2018-05-29",
  "operation": "Query",
  "index": "groupIdStatus",
  "query": {
    "expression": "#groupIdStatus = :groupIdStatus",
    "expressionNames": {
      "#groupIdStatus": "group_id#status"
    },
    "expressionValues": {
      ":groupIdStatus" : {"S": "$groupIdStatus"}
    }
  }
}
How to make filtering optional

In the above example, there is no way to query users in a group regardless of their status. To also support that case, add a separate GSI with only the group_id as the partition key and check in the resolver whether the value is supplied or not.

Filter expression or composite partition key?

Filter expressions are good when:

  • The table is small (not many elements)
  • The number of dropped items for most queries is small and
  • Clients usually don't want to retrieve all matching items

The composite partition key is good when:

  • The table is large (many elements)
  • The number of dropped items can be large
Master AppSync and GraphQL
Support this book and get all future updates and extra chapters in ebook format.