KV Store Service

Summary

The M2_KVStoreService is an C++ base class that provides a basic API for connecting an external KV Store to your project. We have added a BP implementation using that: BP_M2_KVStoreService, which communicates with the Morpheus Platform KV Store (Key/Value Store).

The service also includes subscribe and unsubscribe events; in our example, these communicate with the Morpheus Platform Realtime Service (Realtime)

Since all the communication with the Morpheus Platform occurs at the Blueprint level, users have the flexibility to see how it all works, modify it, or replace the implementation with their own custom logic, even replacing the use of the Morpheus Platform KV Store with their own service.

API & Usage

The base API for the M2_KVStoreService is as follows:

Read(LocalUserIndex, Keys, Completion):
- Given a LocalUserIndex, a list of Keys and a Completion callback, it will attempt to request values for that list of keys. It will trigger the callback when done.
- The Callback will contain a list of results, one for each of the provided Keys. Each will have a Success boolean, on whether it successfully found a value. If it did, the value will be passed in.
Store(LocalUserIndex, Values, Completion):
- Given a LocalUserIndex, a map of keys to values (Values) and a Completion callback, it will attempt to update the keys in the KV store with your provided values. It will trigger the callback when done, informing on whether the operation was a success or not.
Subscribe(LocalUserIndex, Key) & ListenToSubscriptionUpdates(LocalUserIndex, Callback):
- Subscribe will register interest in the given Key, for the provided LocalUserIndex. Any time that KV Store value is updated, the service will be notified.
- ListenToSubscriptionUpdates takes a Callback for the provided LocalUserIndex. Any time a key that has been subscribed to updates (for that provided LocalUserIndex), the callback will be called.
- The callback contains the Key that was updated, and its latest Value
- In our example implementation, if you are listening to subscription updates, the Subscribe operation will itself trigger a subscription update with the key's initial value, if it is already there. That is a more opinionated behavior though, and may not be the case in all KV Store implementations.
Unsubscribe(LocalUserIndex, Key) & RemoveSubscriptionUpdateListener(LocalUserIndex, Callback):
- Same as the above, but in reverse.
- If you remove a Callback using RemoveSubscriptionUpdateListener, that was previously added using ListenToSubscriptionUpdates, then the callback won't be called, even if the KV store service is being notified of subscription updates.
- If you unsubscribe from a Key for a given LocalUserIndex, the service will not be notified of subscription updates for that specific key any more.

Regarding Local User Index

Our API is filled with references to a LocalUserIndex. What is it, and why is it needed?

This is an integer that gives a unique identifier for different clients on the same machine. In most cases this is irrelevant, and can be left at 0. However, this is required for bots: we can have multiple bots running on the same client, so we need a way of each being able to use the same service, but access their own unique KV store. This way, bot 1 can access LocalUserIndex = 0, and not interfere at all with bot 2, which uses the same functionality, but with LocalUserIndex = 1

Morpheus Platform-specific Usage

The BP_M2_KVStoreService is our example implementation of a KV Store Service, communicating with the Morpheus Platform KV Store. It has some added quirks for communicating specifically to this system, and some additional relevant helpers.

Scope

The biggest difference between our specific KV Store ( Key/Value Store), and a generic one, is that we expect our keys passed in to be of a specific format: <scope>|<store>|<userid>|<key>. This is so that we ensure all the information is contained within the passed in "key" string.

Scope indicates the "scope" of the KV store you're accessing, i.e. being either "world" (the data persists and is accessible only for the given world), "project" (the data is accessible in any world within the specific project), or "organization" (the data is accessible across any world, and across multiple projects within the same organization).
Store indicates whether we are using the client or server data store. In our Unreal implementation, this can be set automatically, since the server will only ever interact with the server data store, and vice versa.
UserId indicates which user's store we are accessing. On the server, we use a hard-coded known id, being server
Key is then the key itself.

This format is quite heavyweight to use, so we have some wrappers to make this easier.

MakeScopedKey(Key, Scope, LocalUserIndex): makes a key in the format <scope>|<store>|<userid>|<key> from the data provided.
SetDefaultScope(Scope): means that you can provide "unscoped" keys, and it'll automatically scope them with the provided scope
- This function returns Success if it successfully updated the scope.
- Since it is possible to set the default scope to different values in different places, which could lead to hard-to-track issues, it is best to avoid setting the default scope in multiple places.
ClearDefaultScope(): Removes the default scope set using the above.
SplitScopedKey: splits a scoped key into its respective parts, including the scope (w (world), p (project), or o (organization)), the store, id and the unscoped key.

Using these helper functions means that we can still expose the different scopes to downstream users to control, but without muddying the API too much

A note on the output "keys" from the KV Store requests

Since the KV store operates using scoped keys, both Read results, and the callback from subscriptions (ListenToSubscriptionUpdates) return the scoped key, not the unscoped one. (This is needed to differentiate between different scopes - if the subscription delegate returned the unscoped key, there would be no way of distinguishing between simultaneous subscriptions to e.g. the Test key in the world and organization scopes)

If you need to compare keys at the point of one of these callbacks, please use the MakeScopedKey or SplitScopedKey helpers accordingly, to compare with an unscoped key.

Startup considerations

Since the BP_M2_KVStoreService depends on the Realtime Service, we need to wait for them to be ready before fully initializing the KV Store Service. If you call Subscribe or Unsubscribe too early, the request could fail due to this not being ready.

We therefore make use of the The "Wait For Condition" System, making a custom KVStoreReady condition that users should wait for before using the KV Store service.

How to get and use the service

You can get the KV Store service using the GetWorldService helper function (see World Services).

As mentioned above in Startup considerations, if you are using the BP_M2_KVStoreService, you should first use WaitForCondition(KVStoreReady).

You can subclass to e.g. the BP_M2_KVStoreService if you need the specific functionality that that adds.

A note on CreateIfMissing

It's good practice here to have CreateIfMissing = false here, since we expect the KV Store to have been created and initialized for you. If it has not been created, that's a sign you're using the wrong KV store, rather than it being a service you should create live if missing (it could mean you end up with multiple KV Store Services, which would make finding individual ones more difficult)

How to change which KV Store Service is being used

In your map's World Settings, you can select which KV Store Service is being used, by selecting a class in the KV Store Service Class dropdown. If you create your own KV Store Service, extending the M2_KVStoreService base class, it will show up here.

NOTE: If you don't want to use our KV Store Service base class, you can also interact with an external KV store service without it, either by making a different type of World Service, or using completely different classes. Our base class just provides a base API, and does some logic for you in C++, that is difficult to do entirely in blueprints. (Tracking different delegates for e.g. individual read requests, or subscription updates for different users)

The functions to override

To make your own KV Store service from M2_KVStoreService, you will need to implement the following functions:

BeginRead - triggered when a read is started, with a RequestId for tracking. You can run your own arbitrary logic here, as long as you ultimately call EndRead with the right RequestId once the read is done.
BeginStore same as above, but call EndStore when done
Subscribe - use this to mark that we want to listen to updates to the KV store value. You should call NotifySubscriptionUpdate whenever the Key is updated for the provided LocalUserIndex
Unsubscribe - if this is called for a Key and LocalUserIndex, you should no longer call NotifySubscriptionUpdate for it.

My project is using the now-deprecated Persistence Subsystem. How can I migrate across?

Since the old J_PersistenceSubsystem has been deprecated, we advise any users of said system to migrate over when appropriate. Any uses of the old system will not break, but will trigger compiler warnings, and you will not be able to add further uses of the persistence subsystem.

The KV Store Service works very similarly to the old approach, so for the majority of use cases, we expect migration to be fairly simple, largely involving replacing calls with the new equivalents. The more complex edge cases will be called out.

The old AddKeyQuery and AddKeyQueries calls can be moved wholesale to Read
The old SetSessionValueInt/String/Struct values can largely be moved to Store
- Under the hood, the int and struct versions are converted to strings anyway. This will need to be done manually here (there are helpers to e.g. convert structs to Json strings and vice versa)
- As well as writing to a remote KV store, the old persistence subsystem also cached the data locally. This functionality is not present in our example. For more details, see Caching local "Session Data"
The old AddLiveKey and AddLiveKeys can be moved wholesale to Subscribe
Same goes for RemoveLiveKey & RemoveLiveKeys -> Unsubscribe
The old OnPersistentLiveValueChanged event is equivalent to ListenToSubscriptionUpdates & RemoveSubscriptionUpdateListener
The old OnPersistentQueryComplete event can largely be translated to the Completion callback of the Read call - instead of having the one callback for all queries, the API has changed so that each individual read call triggers its own callback.

Remember to consider scope! See Scope

Will persistent data currently stored via the persistence subsystem migrate to the KV Store Service?

Not by default:

Beware: Switching the server's UserId

As mentioned in Scope, the UserId used for the server is currently hard-coded to be server in our example. This is the required name to support communicating with the Realtime service.

However, in the old persistence subsystem, the value was instead hard-coded to unrealServer.

This means that if you switch to the new service, any existing persistent data would not transfer across, since it is effectively stored under a different user.

If this is a dealbreaker for projects meaning to migrate across, they can update the MakeScopedKey in their own implementation to use unrealServer. This would mean that the old data would be accessible, but that Subscribe related behavior would not work.

(Intermediate migratory logic could be added to transfer across, e.g. falling back to reading from the old store id if the value is not yet present in the new id, but then writing to the new store's id.)

Bootflow differences

The persistence subsystem added a hardcoded bootflow step, being HandleInitialValuesReceived

We don't have an equivalent out the box, but similar functionality could be achieved downstream by making use of custom conditions (see Adding custom steps to the bootflow)

Caching local "Session Data"

The main difference between the old "Set Session Value" functions and the new system's "Store" operation, is that the persistence subsystem achieved 2 things: It optionally sent details to a remote KV store, but also cached the values locally. Since the system was a GameInstanceSubsystem, data stored locally would persist across world travel, and so could be used as "per-session data". This functionality is no longer possible with the new system; due to our interoperable setup, users are unable change the game instance, so can't make custom logic in BP that persists across world travel.

MSquared has accepted this as a feature regression, with the understanding that no current projects depend on this. It is being tracked as upcoming work. If this is needed for your project, please speak to a support engineer.

For most use cases, similar functionality can be achieved without "session data". For local cached data, this can be achieved by adding it in your BPs as a wrapper around the existing KV Store Service functions. If you want data that persists across world travel (between worlds in your project or organization), this can be achieved by storing it in a Project or Organization-scoped KV store, and reading the value again after world travel.

"Deleted keys"

The Morpheus Platform KV Store does not currently have a complete API for deleting keys, since it has not been a requirement thus far. This is being tracked as upcoming work. If this is needed for your project, please speak to a support engineer.

In most cases, if you want to "clear the value" of a specific key, setting it to the empty string will suffice.

"Soft Persistence"

The old persistence subsystem added some optional suffixes to keys, to differentiate between different contexts and so achieve a "soft persistence":

Based on the config flag M2.UserData.AppendChangelistToDataStoreKeys, the current changelist was appended to the key, making keys unique per build, and so effectively refreshing persistence whenever the build updates.
Based on the config flag M2.UserData.AppendInEditorWhenInEditorToDataStoreKeys, a suffix was added to the key if in editor, making keys stored whilst in editor not be present in live builds, and vice versa.

These are not present in our example KV Store Service, but could be added as extensions in a downstream project if desired.

PreviousOnline Services NextWeb Browser

Last updated 1 month ago

Was this helpful?