Unique ID to Stable ID migration
1. What
UUniqueID is a legacy UCLASS UObject type that wraps a 64-bit integer.
FStableId is a new USTRUCT value type, still wrapping a 64-bit integer. In
Blueprint, these are called “Unique ID” and “Stable ID”.
- All internal use of the
UUniqueIDtype is going to be replaced byFStableId. - The storage format in
game.db(etc.) is going to be changed. - Existing
UUniqueIDBlueprint API will be deprecated and replaced with similarFStableIdvariants. - Both types will co-exist for an extended period of time (but not forever) to give modders (and the core game) time to migrate.
This change will take place over several releases to give mod maintainers time to update their content before any breaking changes.
No changes are necessary for CEE Version 2.0.
2. Why
UUniqueID is extremely inefficient for what it provides, and its API is easy
to use incorrectly.
- As an object-type, it interacts with garbage collection and contributes to the overall object count.
- Its API leads to a propagation of many temporary instances over time, and has been a contributing factor to a number of bugs due to incorrect use.
- Its replication is optimized based on legacy Unreal networking, and is interfering with other work in this area.
- Its contribution to the object count is part of the reason that some admins
have had to increase the object limit on their servers, even though most of
the
UUniqueIDinstances are temporary and destroyed on the next garbage collection.
(On some official servers used for testing, over 3 million instances were created while restoring the world persistence; after the world is restored and garbage collection cleans up, around 300 thousand instances remain. Around 90% of the instances created at startup are wasted overhead on those tested servers.)
3. When
- Communicate about the change: now (shortly after CEE 2.0 was released).
- Migrate the core game: ongoing already.
- Change how
UUniqueIDis stored and release the new Blueprint API: expected in 2.1. - Support mod migration: at least one release.
- Deprecate
UUniqueID: expected in 2.2. - Remove deprecated API: no sooner than 2.3.
“Deprecated” means that mods will generate cook warnings, but will still function if they use the old API until that API is eventually removed.
4. How
Convert Storage Format
This section is mostly relevant for people using custom scripts that interact with the game database directly.
UUniqueID is currently (in legacy and up to CEE 2.0) stored in the game
databases in two ways:
- some classes use the ID value as a key in various tables. This is not
changing; these are already explicitly extracted and stored using SQL. These
are (for example) the
BIGINTcolumns in thecharactersandpropertiestables. - most classes serialize the
UUniqueIDobject into a regular archive using customized Unreal archive serialization, which wraps the value in multiple layers of metadata, which is then stored in the database as a blob of bytes. This also applies to containers ofUUniqueIDand to structs that haveUUniqueIDmembers, however the examples in this documentation are primarily referring to singular values inside an archive.
The serialization for UUniqueID inside archives (database blobs) is changing
to make UUniqueID archive storage directly compatible with FStableId archive
storage.
This is a critical first step, since the current serialization is not compatible
with archives that contain both UUniqueID and FStableId values, but we
want both types to coexist for a migration period. Using the current UUniqueID
storage format while supporting both types in persisted data creates a
significant risk of data corruption. To avoid this, we are changing how
UUniqueID is serialized into an archive so that it is compatible with
FStableId before it is possible for any persisted archives to contain
FStableId values. This means that, when reading an archive from the database,
if both types exist in the same archive, the game can interpret both values the
same way and not care which type wrote the underlying data into the archive.
This table summarizes the storage layout; more detail is at the end of this page.
| Version | Format |
|---|---|
UUniqueID v2 | [archive_header][Object_version][UniqueID_version][Object_metadata][int64] |
UUniqueID v3 | [archive_header][UniqueID_version][int64] |
FStableId v0 | [archive_header][StableId_version][int64] |
UUniqueID versions are identified by the GUID
0xdeadbeef_bafeca0f_abcddbca_1234abcdfollowed by the version (generally 2 for legacy and 3 for the new version; older versions might exist in older databases, but are not expected to be common).
FStableId versions are identified by the GUID
0x1879cf47_330a4a93_9008b587_0bcc9475followed by the version (the only valid version is 0).
It is possible for an archive to contain versions for both types, if it (for
example) contains a struct with properties of both types. In this scenario, the
struct data will be serialized into a single archive containing both properties
and both version identifiers. This is not expected to exist for game data, but
could exist for mod data, and can only exist with UUniqueID version 3.
Add New Blueprint API for FStableId
Initially, we will create parallel FStableId API for Blueprint exposed
functionality, but also maintain the existing UUniqueID API to allow migration
over time. Internally, most systems will be entirely converted to use
FStableId; there are some exceptions to this to support existing blueprint
exposed UUniqueID properties, functions that return UUniqueID objects, and
delegates with UUniqueID parameters.
- There is a new function library (
StableIdFunctionLibrary) that has similar functionality provided by various existing function libraries. - Functions that take
UUniqueIDparameters will be duplicated forFStableId. - Functions that return
UUniqueIDobjects will be duplicated forFStableId. - Delegates that have
UUniqueIDparameters will be duplicated forFStableId. Both versions will fire during the migration period. SaveGameUUniqueIDproperties that are not exposed to Blueprint will be converted toFStableId.SaveGameUUniqueIDproperties that are exposed to Blueprint will remainUUniqueIDuntil after official deprecation, butFStableIdfunctions will be added as a replacement.
Migrate Core Game and Mods
We have already started migrating the core game. Mod authors should plan to update their mods “reasonably soon” so that we can support the migration, especially if we’ve missed something you depend on.
Starting in CEE version 2.1, Existing UUniqueID-based API will have comments
pointing to the new FStableId-based API. At that time, it should be feasible
to port most existing mods to FStableId.
Some core properties (specifically SaveGame properties exposed to blueprints)
will not be migrated yet, as that would cause a breaking change. We will provide
getters (and setters if appropriate) using FStableId as a replacement for those.
Mods should replace any UUniqueID variables they have defined with FStableId
versions. If the variables are SaveGame, or used in a SaveGame struct, then they
must keep the same name.
Officially Deprecate UUniqueID
After FStableId and the new API have existed for some time (at least one
release), the old UUniqueID API will be officially deprecated using the
engine’s deprecation markup. After this is released, mods will generate warnings
when cooking (pointing to any remaining functionality that needs to be migrated)
but should still function.
Breaking Change: Remove UUniqueID from Blueprints
After UUniqueID has been officially deprecated for some time (at least one
release), it will be removed from Blueprints.
Some additional properties in the core game will also be migrated at this time.
Specifically, any properties that are exposed to blueprints and also written to
the database need to be changed in-place to FStableId without changing the
property name, and that can’t happen except as a breaking change. We believe
that the number of properties impacted by this should be relatively small.
Appendix: Storage Format Details
Note that the database stores each value little-endian; if you are inspecting the blobs in the database then the raw bytes will read in the opposite order from the way that same value is written in hex in these diagrams.
In most cases, for an archive containing either version of a single UUniqueID
or a single FStableId, the value is stored in the trailing 8 bytes (except for
nullptr UUniqueID properties, which have a different representation
described below).
UUniqueID v2
nullptr UUniqueID data will have bIsNull = 1 with no additional bytes
0 1
0 1 2 3 4 5 6 7 8 9 A B C D E F 0 1 2 3 4 5 6 7 8 9 A B C D E F
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Package Version FileVersionUE4 |
| FileVersionUE5 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| CustomVersionCount = 2 |
+===============================================================+
| Object Version GUID.A = 0xDEADBEEF |
| GUID.B = 0xBAFECA0F |
| GUID.C = 0xABCFFBCA |
| GUID.D = 0x4321ABCD |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Version = 0 |
+===============================================================+
| UniqueID Version GUID.A = 0xDEADBEEF |
| GUID.B = 0xBAFECA0F |
| GUID.C = 0xABCDDBCA |
| GUID.D = 0x1234ABCD |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Version = 2 |
+===============================================================+
| bIsNull = 0 (false) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ClassPath.Length = 31 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
. ClassPath.Value (variable, not to scale) .
. "/Script/UE4Dreamworld.UniqueID\0" .
. .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ObjectName.Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
. ObjectName.Value (variable, not to scale) .
. Example: "UniqueID_2147417576\0" .
. .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| UUniqueID Value high 32 bits |
| low 32 bits |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+UUniqueID v3
nullptr UUniqueID Value will be MIN_int64: 0x80000000_00000000
0 1
0 1 2 3 4 5 6 7 8 9 A B C D E F 0 1 2 3 4 5 6 7 8 9 A B C D E F
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Package Version FileVersionUE4 |
| FileVersionUE5 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| CustomVersionCount = 1 |
+===============================================================+
| UniqueID Version GUID.A = 0xDEADBEEF |
| GUID.B = 0xBAFECA0F |
| GUID.C = 0xABCDDBCA |
| GUID.D = 0x1234ABCD |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Version = 3 |
+===============================================================+
| UUniqueID Value high 32 bits |
| low 32 bits |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+FStableId v0
nullptr is no longer meaningful, Value will be 0
0 1
0 1 2 3 4 5 6 7 8 9 A B C D E F 0 1 2 3 4 5 6 7 8 9 A B C D E F
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Package Version FileVersionUE4 |
| FileVersionUE5 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| CustomVersionCount = 1 |
+===============================================================+
| StableId Version GUID.A = 0x1879CF47 |
| GUID.B = 0x330A4A93 |
| GUID.C = 0x9008B587 |
| GUID.D = 0x0BCC9475 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Version = 0 |
+===============================================================+
| FStableId Value high 32 bits |
| low 32 bits |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+