# KV Store Service ## Summary The `M2_KVStoreService` is an C++ base class that provides a basic API for connecting an external KV Store to your project. We have added a BP implementation using that: `BP_M2_KVStoreService`, which communicates with the Morpheus Platform KV Store ([Broken link](https://docs.msquared.io/creation/unreal-development/features-and-tutorials/online-services/broken-reference "mention")). The service also includes subscribe and unsubscribe events; in our example, these communicate with the Morpheus Platform Realtime Service ([realtime](https://docs.msquared.io/apis-and-tooling/api-reference/realtime "mention")) Since all the communication with the Morpheus Platform occurs at the Blueprint level, users have the flexibility to see how it all works, modify it, or replace the implementation with their own custom logic, even replacing the use of the Morpheus Platform KV Store with their own service.

The the-m2-example-plugin's Example map has a KV Store checker, which uses the KV Store Service, to visualise how it works

## API & Usage The base API for the `M2_KVStoreService` is as follows: * `Read(LocalUserIndex, Keys, Completion)`: * Given a `LocalUserIndex`, a list of `Keys` and a `Completion` callback, it will attempt to request values for that list of keys. It will trigger the callback when done. * The Callback will contain a list of results, one for each of the provided `Keys`. Each will have a `Success` boolean, on whether it successfully found a value. If it did, the value will be passed in.
* `Store(LocalUserIndex, Values, Completion)`: * Given a `LocalUserIndex`, a map of keys to values (`Values`) and a `Completion` callback, it will attempt to update the keys in the KV store with your provided values. It will trigger the callback when done, informing on whether the operation was a success or not.
* `Subscribe(LocalUserIndex, Key)` & `ListenToSubscriptionUpdates(LocalUserIndex, Callback)`: * `Subscribe` will register interest in the given `Key`, for the provided `LocalUserIndex`. Any time that KV Store value is updated, the service will be notified. * `ListenToSubscriptionUpdates` takes a `Callback` for the provided `LocalUserIndex`. Any time a key that has been subscribed to updates (for that provided `LocalUserIndex`), the callback will be called. * The callback contains the `Key` that was updated, and its latest `Value` * In our example implementation, if you are listening to subscription updates, the `Subscribe` operation will itself trigger a subscription update with the key's initial value, if it is already there. That is a more opinionated behavior though, and may not be the case in all KV Store implementations.
* `Unsubscribe(LocalUserIndex, Key)` & `RemoveSubscriptionUpdateListener(LocalUserIndex, Callback)`: * Same as the above, but in reverse. * If you remove a `Callback` using `RemoveSubscriptionUpdateListener`, that was previously added using `ListenToSubscriptionUpdates`, then the callback won't be called, even if the KV store service is being notified of subscription updates. * If you unsubscribe from a `Key` for a given `LocalUserIndex`, the service will not be notified of subscription updates for that specific key any more.
{% hint style="info" %} **Regarding Local User Index** Our API is filled with references to a `LocalUserIndex`. What is it, and why is it needed? This is an integer that gives a unique identifier for different clients on the same machine. In most cases this is irrelevant, and can be left at 0. However, this is required for bots: we can have multiple bots running on the same client, so we need a way of each being able to use the same service, but access their own unique KV store. This way, bot 1 can access `LocalUserIndex = 0`, and not interfere at all with bot 2, which uses the same functionality, but with `LocalUserIndex = 1` {% endhint %} ### Morpheus Platform-specific Usage The `BP_M2_KVStoreService` is our example implementation of a KV Store Service, communicating with the Morpheus Platform KV Store. It has some added quirks for communicating specifically to this system, and some additional relevant helpers. #### Scope The biggest difference between our specific KV Store ( [Broken link](https://docs.msquared.io/creation/unreal-development/features-and-tutorials/online-services/broken-reference "mention")), and a generic one, is that we expect our keys passed in to be of a specific format: `|||`. This is so that we ensure all the information is contained within the passed in "key" string. * `Scope` indicates the "scope" of the KV store you're accessing, i.e. being either "world" (the data persists and is accessible only for the given world), "project" (the data is accessible in any world within the specific project), or "organization" (the data is accessible across any world, and across multiple projects within the same organization). * `Store` indicates whether we are using the client or server data store. In our Unreal implementation, this can be set automatically, since the server will only ever interact with the server data store, and vice versa. * `UserId` indicates which user's store we are accessing. On the server, we use a hard-coded known id, being `server` * `Key` is then the key itself. This format is quite heavyweight to use, so we have some wrappers to make this easier. * `MakeScopedKey(Key, Scope, LocalUserIndex)`: makes a key in the format `|||` from the data provided. * `SetDefaultScope(Scope)`: means that you can provide "unscoped" keys, and it'll automatically scope them with the provided scope * This function returns `Success` if it successfully updated the scope. * Since it is possible to set the default scope to different values in different places, which could lead to hard-to-track issues, it is best to avoid setting the default scope in multiple places. * `ClearDefaultScope()`: Removes the default scope set using the above. * `SplitScopedKey`: splits a scoped key into its respective parts, including the scope (`w` (world), `p` (project), or `o` (organization)), the store, id and the unscoped key.![](https://1456550285-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoWTlPaoHd1McSakqMigu%2Fuploads%2Fgit-blob-758c1ade507e032b57e86e111dd6f7abb88839d8%2Fimage.png?alt=media) Using these helper functions means that we can still expose the different scopes to downstream users to control, but without muddying the API too much

An example using some of the different helpers. Since we have used SetDefaultScope(World) at the start, the subsequent Read, Store and Subscribe operation will all use the World scoped KV store (and the right store, UserId etc.)
However, since the final Subscribe operation uses MakeScopedKey(KeyName, Organization, 0), it will instead subscribe to the Organization scoped KV store.

{% hint style="info" %} **A note on the output "keys" from the KV Store requests** Since the KV store operates using scoped keys, both `Read` results, and the callback from subscriptions (`ListenToSubscriptionUpdates`) return the scoped key, not the unscoped one. (This is needed to differentiate between different scopes - if the subscription delegate returned the unscoped key, there would be no way of distinguishing between simultaneous subscriptions to e.g. the `Test` key in the world and organization scopes) If you need to compare keys at the point of one of these callbacks, please use the `MakeScopedKey` or `SplitScopedKey` helpers accordingly, to compare with an unscoped key. {% endhint %} #### Startup considerations Since the `BP_M2_KVStoreService` depends on the Realtime Service, we need to wait for them to be ready before fully initializing the KV Store Service. If you call `Subscribe` or `Unsubscribe` too early, the request could fail due to this not being ready. We therefore make use of the [the-wait-for-condition-system](https://docs.msquared.io/creation/unreal-development/features-and-tutorials/helpers-and-extras/the-wait-for-condition-system "mention"), making a custom `KVStoreReady` condition that users should wait for before using the KV Store service.
### How to get and use the service You can get the KV Store service using the `GetWorldService` helper function (see [world-services](https://docs.msquared.io/creation/unreal-development/features-and-tutorials/helpers-and-extras/world-services "mention")). As mentioned above in [#startup-considerations](#startup-considerations "mention"), if you are using the `BP_M2_KVStoreService`, you should first use `WaitForCondition(KVStoreReady)`.
You can subclass to e.g. the `BP_M2_KVStoreService` if you need the specific functionality that that adds. {% hint style="info" %} **A note on CreateIfMissing** It's good practice here to have `CreateIfMissing = false` here, since we expect the KV Store to have been created and initialized for you. If it has not been created, that's a sign you're using the wrong KV store, rather than it being a service you should create live if missing (it could mean you end up with multiple KV Store Services, which would make finding individual ones more difficult) {% endhint %} ### How to change which KV Store Service is being used In your map's `World Settings`, you can select which KV Store Service is being used, by selecting a class in the `KV Store Service Class` dropdown. If you create your own KV Store Service, extending the `M2_KVStoreService` base class, it will show up here.
{% hint style="info" %} NOTE: If you don't want to use our KV Store Service base class, you can also interact with an external KV store service without it, either by making a different type of World Service, or using completely different classes. Our base class just provides a base API, and does some logic for you in C++, that is difficult to do entirely in blueprints. (Tracking different delegates for e.g. individual read requests, or subscription updates for different users) {% endhint %} #### The functions to override To make your own KV Store service from `M2_KVStoreService`, you will need to implement the following functions: * `BeginRead` - triggered when a read is started, with a `RequestId` for tracking. You can run your own arbitrary logic here, as long as you ultimately call `EndRead` with the right `RequestId` once the read is done. * `BeginStore` same as above, but call `EndStore` when done * `Subscribe` - use this to mark that we want to listen to updates to the KV store value. You should call `NotifySubscriptionUpdate` whenever the `Key` is updated for the provided `LocalUserIndex` * `Unsubscribe` - if this is called for a `Key` and `LocalUserIndex`, you should no longer call `NotifySubscriptionUpdate` for it. ## My project is using the now-deprecated Persistence Subsystem. How can I migrate across? Since the old `J_PersistenceSubsystem` has been deprecated, we advise any users of said system to migrate over when appropriate. Any uses of the old system will not break, but will trigger compiler warnings, and you will not be able to add further uses of the persistence subsystem. The KV Store Service works very similarly to the old approach, so for the majority of use cases, we expect migration to be fairly simple, largely involving replacing calls with the new equivalents. The more complex edge cases will be called out. * The old `AddKeyQuery` and `AddKeyQueries` calls can be moved wholesale to `Read` * The old `SetSessionValueInt/String/Struct` values can largely be moved to `Store` * Under the hood, the int and struct versions are converted to strings anyway. This will need to be done manually here (there are helpers to e.g. convert structs to Json strings and vice versa)
* As well as writing to a remote KV store, the old persistence subsystem also cached the data locally. This functionality is not present in our example. For more details, see [#caching-local-session-data](#caching-local-session-data "mention") * The old `AddLiveKey` and `AddLiveKeys` can be moved wholesale to `Subscribe` * Same goes for `RemoveLiveKey` & `RemoveLiveKeys` -> `Unsubscribe` * The old `OnPersistentLiveValueChanged` event is equivalent to `ListenToSubscriptionUpdates` & `RemoveSubscriptionUpdateListener` * The old `OnPersistentQueryComplete` event can largely be translated to the `Completion` callback of the `Read` call - instead of having the one callback for all queries, the API has changed so that each individual read call triggers its own callback. {% hint style="info" %} Remember to consider scope! See [#scope](#scope "mention") {% endhint %}

An example migration of our BPM_PersistenceExample - store and subscribe being replaced with the appropriate usage in the KV Store Service

### Will persistent data currently stored via the persistence subsystem migrate to the KV Store Service? Not by default: {% hint style="danger" %} **Beware: Switching the server's UserId** As mentioned in [#scope](#scope "mention"), the `UserId` used for the server is currently hard-coded to be `server` in our example. This is the required name to support communicating with the [realtime](https://docs.msquared.io/apis-and-tooling/api-reference/realtime "mention") service. However, in the old persistence subsystem, the value was instead hard-coded to `unrealServer`. This means that if you switch to the new service, any existing persistent data would not transfer across, since it is effectively stored under a different user. If this is a dealbreaker for projects meaning to migrate across, they can update the `MakeScopedKey` in their own implementation to use `unrealServer`. This would mean that the old data would be accessible, but that `Subscribe` related behavior would not work. (Intermediate migratory logic could be added to transfer across, e.g. falling back to reading from the old store id if the value is not yet present in the new id, but then writing to the new store's id.) {% endhint %} ### Bootflow differences The persistence subsystem added a hardcoded bootflow step, being `HandleInitialValuesReceived`
We don't have an equivalent out the box, but similar functionality could be achieved downstream by making use of custom conditions (see [#adding-custom-steps-to-the-bootflow](https://docs.msquared.io/creation/unreal-development/helpers-and-extras/the-wait-for-condition-system#adding-custom-steps-to-the-bootflow "mention"))
### Caching local "Session Data" The main difference between the old "Set Session Value" functions and the new system's "Store" operation, is that the persistence subsystem achieved 2 things: It optionally sent details to a remote KV store, but also cached the values locally. Since the system was a `GameInstanceSubsystem`, data stored locally would persist across world travel, and so could be used as "per-session data". This functionality is no longer possible with the new system; due to our interoperable setup, users are unable change the game instance, so can't make custom logic in BP that persists across world travel. MSquared has accepted this as a feature regression, with the understanding that no current projects depend on this. It is being tracked as upcoming work. If this is needed for your project, please speak to a support engineer. For most use cases, similar functionality can be achieved without "session data". For local cached data, this can be achieved by adding it in your BPs as a wrapper around the existing KV Store Service functions. If you want data that persists across world travel (between worlds in your project or organization), this can be achieved by storing it in a `Project` or `Organization`-scoped KV store, and reading the value again after world travel. ### "Deleted keys" The Morpheus Platform KV Store does not currently have a complete API for deleting keys, since it has not been a requirement thus far. This is being tracked as upcoming work. If this is needed for your project, please speak to a support engineer. In most cases, if you want to "clear the value" of a specific key, setting it to the empty string will suffice. ### "Soft Persistence" The old persistence subsystem added some optional suffixes to keys, to differentiate between different contexts and so achieve a "soft persistence": * Based on the config flag `M2.UserData.AppendChangelistToDataStoreKeys`, the current changelist was appended to the key, making keys unique per build, and so effectively refreshing persistence whenever the build updates. * Based on the config flag `M2.UserData.AppendInEditorWhenInEditorToDataStoreKeys`, a suffix was added to the key if in editor, making keys stored whilst in editor not be present in live builds, and vice versa. These are not present in our example KV Store Service, but could be added as extensions in a downstream project if desired.