diff --git a/Documentation/command-archive.md b/Documentation/command-archive.md index 90964dc..8e6b9f5 100644 --- a/Documentation/command-archive.md +++ b/Documentation/command-archive.md @@ -1,6 +1,6 @@ # archive Command -The `archive` command provides utilities for working with [Unity Archives](unity-content-format.md#unity-archive) — container files used for AssetBundles and web platform `.data` files. Archives hold one or more files (typically SerializedFiles) and may apply compression to the content. +The `archive` command provides utilities for working with [Unity Archives](unity-content-format.md#unity-archive) — container files used for AssetBundles, compressed Player builds (`data.unity3d`), and web platform `.data` files. Archives hold one or more files (typically SerializedFiles) and may apply compression to the content. See [Player Build Format](playerbuild-format.md) for how these containers are used in Player builds. To inspect the serialized objects *inside* an archive, use the [`dump`](command-dump.md#archive-support) command, which can open archives directly without extracting first. diff --git a/Documentation/comparing-builds.md b/Documentation/comparing-builds.md index 85a70a5..6bd3c28 100644 --- a/Documentation/comparing-builds.md +++ b/Documentation/comparing-builds.md @@ -29,7 +29,7 @@ When comparing builds there may be a huge number of individual differences. The ## AssetBundles and Player Builds -This topic focuses on AssetBundles, because most questions about changes in a build come up around AssetBundles. However the same techniques can also be used to analyze player builds (so long as the player is built with [TypeTrees enabled](./unity-content-format.md#enabling-typetrees-in-the-player)). +This topic focuses on AssetBundles, because most questions about changes in a build come up around AssetBundles. However the same techniques can also be used to analyze player builds (so long as the player is built with [TypeTrees enabled](./playerbuild-format.md#typetrees-in-the-player)). # Example 1 - Changes to serialized values in an Addressables build diff --git a/Documentation/contentdirectory-format.md b/Documentation/contentdirectory-format.md new file mode 100644 index 0000000..fe1ed7c --- /dev/null +++ b/Documentation/contentdirectory-format.md @@ -0,0 +1,259 @@ +# Content Directory Format + +Content directories are a build pipeline introduced in Unity 6.6 for shipping a project's assets as +separate content builds that load alongside a Player build. They are designed as a newer alternative to +[AssetBundles](assetbundle-format.md), with automatic de-duplication of shared content and per-asset +dependency tracking. For the full picture — what +they are, how to build them, and the APIs for loading content — see Unity's Manual topic +[Use content directories to load assets at runtime](https://docs.unity3d.com/6000.6/Documentation/Manual/content-directories.html). + +This page focuses on the parts that matter when inspecting a content directory build with +UnityDataTool. It complements the higher-level [Overview of Unity Content](unity-content-format.md), +which introduces SerializedFiles and Unity Archives. + +## What a content directory build produces + +A content directory build produces the same kinds of data as any other Unity build — SerializedFiles +holding the serialized objects, plus companion `.resS` (texture/mesh) and `.resource` (audio/video) +files — together with a build manifest that records what is needed to load the content at runtime. + +The output can be written as loose files or packed into a Unity Archive, so UnityDataTool opens it the +same way it opens Player and AssetBundle content. + +An example build, used for testing, can be found in `TestCommon/Data/LeadingEdgeBuilds/ContentDirectory`. + +The files use technical, content-hash-based names, rather than the familiar `level0` / `sharedassets` +names of a Player build or the user-specified names of AssetBundles. You do not need to understand the +internal file layout for typical usage — defining what to build, running builds, and loading content +at runtime. This page goes into that detail to help interpret UnityDataTool output in the cases where a +closer look is useful. + +> [!IMPORTANT] +> The details of what is *inside* a content directory are subject to change. The stable, supported +> surface is the build and load API and the observable loading behavior. The on-disk data +> representation described here continues to be optimized for size, performance, and incremental-build +> efficiency, so treat the specifics below as informational — do not build tooling that hard-codes +> them. + +## Content Files + +The SerializedFiles in a content directory are referred to as **Content Files**. They use the same +underlying SerializedFile format as Player builds and AssetBundles — UnityDataTool reads them the same +way — but follow different conventions for naming and for how references between files are recorded. +Content Files were introduced for building subscenes with the Entities package, and content +directories build on the same mechanism. + +There is a low-level public API for Content Files +([`Unity.Loading.ContentFile`](https://docs.unity3d.com/ScriptReference/Unity.Loading.ContentFile.html)), +but the content directory loading API encapsulates it and is the recommended way to access content at +runtime. + +## File naming + +Every build artifact is named by the hash of its own content: + +- **Content Files** use the `.cf` extension (for example `a77f98db89b6aa1aeaaad01d857e5115.cf`). +- **`.resS`** files hold streamed texture and mesh data. +- **`.resource`** files hold audio and video data. + +Because the name is a hash of the content, two artifacts with identical content collapse to the same +file — this is how the pipeline de-duplicates shared content automatically. The `.resS` and +`.resource` data files are granular in the same way as Content Files: a single texture, mesh, or audio +clip per file, each named by its content hash. + +A Unity object references data inside a `.resS`/`.resource` file by a content-addressable path of the +form `cah:/` (a single slash; `cah` stands for "content-addressable hash"). For example an +AudioClip's `m_Source` is a string like `cah:/4226b5c16a50dab6eff0f08dd1253d4b`, which resolves to the +`.resource` file of that hash. + +The extensions are informational. The loading system identifies content by its hash (through the +`cah:/` scheme), not by file extension, so the extension is not required to resolve content. Extensions +are present both for loose files and for entries packed inside an archive. + +### The build manifest + +The build manifest is a JSON file, also named by its content hash (for example +`15d5df98d98434e67e06716cfabfad1b.json`). It records everything needed to load the content: the list of +Content Files, their dependencies, and the loadable objects and scenes. Its schema is internal and may +change substantially, so this page does not document it. Instead, use +[`ContentLayout.json`](contentlayout.md), which presents the same information (plus source-asset +mapping) in a stable, tool-friendly form. + +A small `BuildManifestHash.txt` file records the hash of the manifest, so a tool can find the current +manifest by reading that pointer. When the build is packed into an archive, `BuildManifestHash.txt` +sits outside the archive while the manifest itself is inside it. + +### Archives + +When the content is packed rather than written loose, it goes into a Unity Archive named +`content0.archive`. The loader will mount additional archives if they are present, provided they follow +sequential names — `content1.archive`, `content2.archive`, and so on — stopping at the first gap. +`BuildPipeline.BuildContentDirectory()` itself only emits the single `content0.archive` containing all +the content. The Addressables package, however, supports splitting the output into multiple archives +based on a maximum-size heuristic. Custom tooling could also perform a loose build and then package the +content into one or more archives as a post-processing "packaging" step, using +[`ContentBuildInterface.ArchiveAndCompress`](https://docs.unity3d.com/ScriptReference/Build.Content.ContentBuildInterface.ArchiveAndCompress.html). + +Once all the archives are mounted, references between Content Files, and from Content Files to the +`.resS`/`.resource` data files, resolve through the `cah:/` scheme exactly as they do for loose files. +Archives serve two purposes: reducing the file-system overhead of many loose files in a folder, and +applying compression. + +## Build layout granularity + +The build layout is highly granular. Typically each source asset becomes its own Content File, and +many files contain just a single Unity object (a Material, Shader, Texture, AudioClip, and so on). A +few grouping rules shape the result: + +- **Scenes and Prefabs** store their whole GameObject/Component hierarchy together in a single Content + File, just as they are represented in the Editor. +- **Assets with sub-assets** are split across multiple Content Files. For example an FBX is separated + so that the component hierarchy and the meshes land in different files. +- **Scripts** are grouped: the `MonoScript` objects for several source scripts can share one Content + File. +- **Circular direct references are best avoided.** Currently a cycle forces all the affected Unity + objects to be clustered together into a single Content File, and the build prints a warning. The way + to break a cycle is to convert one of the references into a `Loadable` (an on-demand reference) + instead of a direct reference. + +Built-in resources are handled specially. The manifest carries an entry for `unity default resources` +flagged as built-in (with no content hash); references to it resolve through the runtime's built-in +resource mechanism and the `PersistentManager`, rather than through the `ContentLoadManager` that +manages Content Files. + +## Build history + +Every Player and content directory build in Unity 6.6 and later records a **build history** — a set +of files, in a per-build directory, that describe how the build ran and what it produced. By default +these live under `Library/BuildHistory`. The build history is separate from the shipped content and +is not distributed with your application. For the full picture, see the Manual topic +[Analyze builds](https://docs.unity3d.com/6000.6/Documentation/Manual/build-analyze-builds.html). + +Most of the files in a build history directory (the Trace Event Profile, `BuildLog.jsonl`, +`ScriptsOnlyCache.yaml`, `ContentSizeSummary.txt`, `BuildReportSummary.json`, and others) are outside +the scope of UnityDataTool and are documented in the Manual's +[Build history file reference](https://docs.unity3d.com/6000.6/Documentation/Manual/build-history-file-reference.html). +Two files in the build history are directly relevant here: + +* **[`ContentLayout.json`](contentlayout.md)** — maps the built content back to the source assets in + the project and describes the dependencies between the produced files. It is the key file for + understanding a content directory build. UnityDataTool support for `ContentLayout.json` is a work in + progress; for now, see the dedicated [ContentLayout.json](contentlayout.md) page for its structure. +* **The BuildReport file** — the build report for the build, in the same SerializedFile format that + UnityDataTool reads for Player and AssetBundle builds. In the build history it is named after the + build session GUID (rather than the fixed `LastBuild.buildreport` name), so reports from multiple + builds can sit side by side and be analyzed together. See [BuildReport Support](buildreport.md). + +## References between Content Files + +References between Content Files work differently from AssetBundles or Player builds, so they are worth +a closer look. In a content directory the reference information is stored **externally to the Content +File**, in the manifest, rather than in the file's own external-reference table. The manifest data is +reproduced in [`ContentLayout.json`](contentlayout.md), which is what the examples below use. + +Consider `ContentDirectoryRoot.asset`, which directly references several ScriptableObjects. Its entry +in `ContentLayout.json` names the source asset, the file's own content hash, and — crucially — an +ordered list of the files it depends on: + +```json +{ + "Index": 3, + "ID": "52b43dad178849b42ac753005736e7bb.cfid", + "SourceAssets": [ "Assets/ScriptableObjects/ContentDirectoryRoot.asset" ], + "SerializedFileDependencies": [ 4, 2, 6, 7 ], + "ContentHash": "a77f98db89b6aa1aeaaad01d857e5115" +} +``` + +The `ContentHash` tells us this asset lives in `a77f98db89b6aa1aeaaad01d857e5115.cf`. Dumping that file +shows the object's references and the file's external-reference table: + +``` +UnityDataTool dump --stdout a77f98db89b6aa1aeaaad01d857e5115.cf +``` +``` +External References +path(1): "a2a42d71dddef12e8889849faf59bdd7.cfid" GUID: 00000000000000000000000000000000 Type: 0 +path(2): "4038ff673d390134d924b57fcbed0432.cfid" GUID: 00000000000000000000000000000000 Type: 0 +path(3): "21679be819d6e9146a63bb02a7e51f2f.cfid" GUID: 00000000000000000000000000000000 Type: 0 +path(4): "78532141fd7679a458405eb16bdb75fd.cfid" GUID: 00000000000000000000000000000000 Type: 0 + +ID: -775554941117088049 (ClassID: 114) MonoBehaviour + ... + data[1] (SerializedKeyValue`2) + key (string) SingleAudioClipLoadableReference + value (PPtr<$ScriptableObject>) + m_FileID (int) 3 + m_PathID (SInt64) -2313013086301746513 +``` + +The external table lists symbolic names ending in `.cfid`, **not** the content-hash filenames. +These `.cfid` strings are placeholders and never exist as files on disk. This is intentional: if the +referencing file embedded the target's content hash, then a single property change could cascade — the +change would alter the target's hash, which would alter every file that referenced it, and so on up +through the build. A key goal of the content directory design is to reduce the amount of content +"churn" when doing updates, so references between Content Files now use an extra level of abstraction. + +Because of that, the content of the external table is effectively ignored by the loading system for +Content File references. What counts is the ordered dependency list in the manifest. The +`"SerializedFileDependencies": [4, 2, 6, 7]` array corresponds, in exact size and order, to the four +entries of the external table. + +So resolving the reference to `SingleAudioClipLoadableReference` above: + +1. The `PPtr` has `m_FileID` 3. A non-zero `m_FileID` is a 1-based index into the external table (0 + would mean "this same file"). +2. Index 3 maps to the 3rd entry of `SerializedFileDependencies`, which is `6`. +3. `ContentLayout.json` entry with `"Index": 6` is `SingleAudioClipLoadableReference.asset`, whose + `ContentHash` is `5c43454a3823f172a2a326410a36ba6b`. +4. The referenced file is therefore `5c43454a3823f172a2a326410a36ba6b.cf`. + +The full mapping for this file, showing how each `m_FileID` in the external table resolves through the +dependency list `[4, 2, 6, 7]` to a content-hash filename: + +```mermaid +flowchart TD + Root["ContentDirectoryRoot
cfid 52b43dad…
file a77f98db….cf
deps [4, 2, 6, 7]"] + Loadable["LoadableAudioClipReference
Index 2 · cfid 4038ff67…
file bfcf18a2….cf"] + Single["SingleAudioClipLoadableReference
Index 6 · cfid 21679be8…
file 5c43454a….cf"] + + Root -->|"m_FileID 2 → dep 2 → Index 2"| Loadable + Root -->|"m_FileID 3 → dep 6 → Index 6"| Single +``` + +> [!NOTE] +> The external table's GUID column is all zeros for Content File references (as above). It is used only +> for built-in resource references, which resolve through the runtime's built-in resource mechanism +> instead of the manifest dependency list. The GUIDs in the external table are also used for binary +> SerializedFiles in the Editor (for example in AssetDatabase artifacts). + +## Inspecting content directory output with UnityDataTool + +When you run [`analyze`](command-analyze.md) on a content directory build, analyze the **build output +folder and its matching build history folder together**, in a single `analyze` call, by passing both +paths: + +```bash +UnityDataTool analyze /path/to/ContentDirectory /path/to/Library/BuildHistory/ +``` + +Analyzing the build output alone records the objects, but not where they came from. The build report +in the build history adds the source-asset mapping (the PackedAssets data), so analyzing the two +together gives a database that ties each built object back to its source asset. See +[BuildReport Support](buildreport.md) for how the build report data is stored and queried. + +Currently the references between objects are not recorded properly for a content directory build, +because those references live in the manifest rather than in the Content Files themselves (see +[References between Content Files](#references-between-content-files) above). This will be resolved +using the information from the `ContentLayout.json` file. + +## Related documentation + +| Topic | Description | +|-------|-------------| +| [Use content directories to load assets at runtime](https://docs.unity3d.com/6000.6/Documentation/Manual/content-directories.html) | Unity Manual: what content directories are, and the APIs to build and load them. | +| [Analyze builds](https://docs.unity3d.com/6000.6/Documentation/Manual/build-analyze-builds.html) | Unity Manual: the build history and the build report. | +| [ContentLayout.json](contentlayout.md) | The build layout file that maps content directory output back to source assets. | +| [BuildReport Support](buildreport.md) | Analyzing Unity build report files with UnityDataTool. | +| [Overview of Unity Content](unity-content-format.md) | SerializedFiles, Unity Archives, and TypeTrees. | +| [AssetBundle Format](assetbundle-format.md) | The earlier system that content directories are a newer alternative to. | diff --git a/Documentation/contentlayout.md b/Documentation/contentlayout.md index 74f4f76..996338a 100644 --- a/Documentation/contentlayout.md +++ b/Documentation/contentlayout.md @@ -61,12 +61,13 @@ The file uses a few terms consistently: The schema is subject to change. The `Version` field records the schema version of the file, independently of the Unity version that produced it. When the schema changes, the version number increments. -[`ContentLayout.cs`](../UnityDataModels/ContentLayout.cs) always represents the latest schema version (currently version 1). +[`ContentLayout.cs`](../UnityDataModels/ContentLayout.cs) always represents the latest schema version (currently version 2). ## Related documentation | Topic | Description | |-------|-------------| +| [Content Directory Format](contentdirectory-format.md) | Content directory builds and inspecting them with UnityDataTool. | | [Build report and build history](https://docs.unity3d.com/6000.6/Documentation/Manual/build-reporting.html) | The build report directory and the files in it (Unity Manual). | | [BuildReport Support](buildreport.md) | Analyzing Unity build report files with UnityDataTool. | | [Unity Content Format](unity-content-format.md) | SerializedFiles, Unity Archives, and how build output maps back to source assets. | diff --git a/Documentation/playerbuild-format.md b/Documentation/playerbuild-format.md new file mode 100644 index 0000000..56e574f --- /dev/null +++ b/Documentation/playerbuild-format.md @@ -0,0 +1,331 @@ +# Player Build Format + +This topic digs into the content produced by a Unity Player build, with a focus on the parts that +are useful when inspecting that content with UnityDataTool. It complements the higher-level +[Overview of Unity Content](unity-content-format.md), which introduces SerializedFiles and Unity +Archives. + +For Unity's official reference on the output files, see +[Content output of a build](https://docs.unity3d.com/Manual/build-content-output.html). This page +builds on that information with extended detail and practical tips for viewing Player data with +UnityDataTool. + +## What UnityDataTool sees + +A Player build produces content, compiled code (assemblies, executables), and various configuration +files. UnityDataTool only concerns itself with the **content** portion of that output — the +SerializedFiles (and their companion binary files) that make up the scenes, assets, and settings of +your application. It does not interpret the assemblies, executables, or the small JSON/config files +that accompany the content (see [Files UnityDataTool does not parse](#files-unitydatatool-does-not-parse)). + +The content comprises: + +- the scenes in the **Scene List**, +- the contents of any `Resources` folders, +- the global project settings (the "Global Game Managers"), and +- every asset referenced from those root inputs. + +Because the same SerializedFile and Unity Archive formats are used for both Player builds and +AssetBundles, UnityDataTool can open Player content directly. The one important caveat is TypeTrees, +covered in [TypeTrees in the Player](#typetrees-in-the-player). + +## File naming and layout + +The SerializedFiles in a Player build are named by a stable convention. You do not create these +files directly, but they show up throughout UnityDataTool output: + +| File | Contents | +| --- | --- | +| `level0`, `level1`, … | One file per scene in the **Scene List**, in list order. Holds that scene's GameObject/Component hierarchy. | +| `sharedassets0.assets`, `sharedassets1.assets`, … | Assets referenced from scenes. The number matches the scene that first references the asset. | +| `resources.assets` | Everything found in `Resources` folders, whether or not a scene references it. Loadable at runtime with `Resources.Load`. | +| `globalgamemanagers` | Core engine data and global project settings (Quality, Graphics, Physics, Tags, Layers, and so on). | +| `globalgamemanagers.assets` | Assets referenced from `globalgamemanagers` (for example a render-pipeline settings ScriptableObject). | +| `Resources/unity_builtin_extra` | Built-in shaders and resources, when referenced by the build. | +| `Resources/unity default resources` | Built-in assets (default materials, fonts, meshes) shipped with the Editor. | + +Each SerializedFile can be accompanied by binary companion files: + +- `.resS` — texture and mesh data, stored so it can be streamed efficiently to the GPU (for example + `sharedassets0.assets.resS`). +- `.resource` — audio and video data (for example `sharedassets1.resource`; note the `.assets` + portion of the name is dropped). + +Unity keeps this binary data in separate files to reduce memory use at load time. + +> [!IMPORTANT] +> Treat the layout and naming above as informational only. Unity may change the exact file +> layout and naming between versions, so don't build tooling that hard-codes it. UnityDataTool +> discovers files rather than assuming specific names. + +### Shared-asset grouping + +Scenes are processed in Scene List order, and each asset is written into the `sharedassets` file for +the scene that references it **first**. As a result, a scene file only ever references +`sharedassets` files at its own number or lower — `level2` can reference `sharedassets2`, +`sharedassets1`, and `sharedassets0`, but never `sharedassets3`. This "stack-based" ordering ensures +each object is built once instead of being duplicated across scenes, and it deliberately favours the +first scenes in the list (which load first) at the expense of later ones. The +[Content output of a build](https://docs.unity3d.com/Manual/build-content-output.html#shared-asset-grouping) +manual page describes the grouping algorithm in more detail. + +To avoid duplication, assets are shared across all the roots: content already written to +`resources.assets` or `globalgamemanagers.assets` is reused rather than copied into a `sharedassets` +file, and vice versa. + +The **first scene gets special priority**. Because `level0` is loaded immediately when the player +starts, the layout algorithm tries to keep its dependencies as small as possible. To that end, the +processing order is: gather the Global Game Manager dependencies, build scene 0, then build the +`Resources` content, and only then build the remaining scenes. Placing `resources.assets` *after* +scene 0 but *before* the other scenes puts it higher in the stack than those later scenes: it can +reference objects already assigned to `sharedassets0` (and `globalgamemanagers.assets`), but never +objects in `sharedassets1` or later. The remaining scenes, in turn, can reference `resources.assets`. + +## Global Game Managers + +`globalgamemanagers` holds the engine's global settings singletons — Player Settings, Quality +Settings, Graphics Settings, Physics, the Tag and Layer definitions, and so on. Assets that those +managers reference are written into `globalgamemanagers.assets`. + +One of the managers is the **ResourceManager**, which tracks the contents of the `Resources` folders. +It is serialized inside `globalgamemanagers`, so there is no standalone "resources index" file — but +the assets it points at live in `resources.assets`, not `globalgamemanagers.assets`. + +Only the managers relevant to the build are included. Editor-only managers are excluded, and a +manager may be dropped if its feature is not enabled in the project. + +## PreloadData + +Every `sharedassets` file — and `globalgamemanagers.assets` — contains a **PreloadData** object +(ClassID 150) at local file id 1. Its `m_Assets` vector lists the objects that must be loaded when +that file's content is loaded, including objects that live in *other* files. At runtime Unity walks +this list to pull in the whole dependency set up front, instead of discovering dependencies one +reference at a time. + +`resources.assets` is the exception — it does not carry a PreloadData object, because its contents +are loaded individually on demand through the Resources API rather than as a preloaded set. The +`level` and `globalgamemanagers` files also have none. + +Dumping the PreloadData from a shared-assets file shows the list of `PPtr`s: + +``` +UnityDataTool dump --stdout sharedassets0.assets --type PreloadData +``` +``` +ID: 1 (ClassID: 150) PreloadData + m_Name (string) + m_Assets (vector) + Array>[3] + data[0] (PPtr) + m_FileID (int) 2 + m_PathID (SInt64) 10001 + data[1] (PPtr) + m_FileID (int) 1 + m_PathID (SInt64) 27 + data[2] (PPtr) + m_FileID (int) 1 + m_PathID (SInt64) 50 + m_Dependencies (vector) + Array[0] +``` + +Each entry's `m_FileID` says which file the object lives in, resolved through the file's external +reference table: `0` is this file itself, and higher numbers are entries in that table. Listing the +references for the same file explains the ids above: + +``` +UnityDataTool sf externalrefs sharedassets0.assets +``` +``` +Index: 1, Path: globalgamemanagers.assets +Index: 2, Path: Library/unity default resources +``` + +So `data[0]` (`m_FileID 2`) points at a built-in object in `unity default resources`, while +`data[1]` and `data[2]` (`m_FileID 1`) point into `globalgamemanagers.assets`. + +Because PreloadData enumerates the full dependency graph reachable from a file, its size grows with +the number of dependencies. Projects with very large hard-reference graphs (for example a scene +object that directly references thousands of assets, or a scene that references large prefabs from MonoBehaviours) +can end up with huge PreloadData tables that add +real metadata overhead to the build. The usual remedy is to load some of that content on demand +(via Addressables, AssetBundles, or `Resources.Load`) instead of through direct references. + +> [!NOTE] +> The same PreloadData object and loading mechanism are used when scenes are built into +> AssetBundles — Player builds and `BuildPipeline.BuildAssetBundles` share the scene-processing code +> path. The [AssetBundle Format](assetbundle-format.md#preloaddata) topic covers PreloadData from the +> scene-bundle angle. + +## Compression: `data.unity3d` + +By default a Player build writes the SerializedFiles as loose files in the `Data` folder. If you +enable [`BuildOptions.CompressWithLz4`](https://docs.unity3d.com/ScriptReference/BuildOptions.CompressWithLz4.html) +or [`CompressWithLz4HC`](https://docs.unity3d.com/ScriptReference/BuildOptions.CompressWithLz4HC.html), +Unity instead packs the content into a single Unity Archive called `data.unity3d` — the same archive +format used for AssetBundles. (`Lz4HC` compresses more aggressively than `Lz4` but takes longer to +build; both decompress at the same speed.) + +`data.unity3d` contains `globalgamemanagers`, the `level`/`sharedassets` files, `resources.assets`, +and `Resources/unity_builtin_extra`. It does **not** contain `Resources/unity default resources` (the +content build doesn't generate that file) nor the `.resource` audio/video files, which stay outside +the archive. + +Whether compression is on by default is platform-specific. Desktop standalone builds are uncompressed +(loose files) unless you opt in, whereas Android and WebGL builds compress the content by default and +so always produce a `data.unity3d`. + +Because `data.unity3d` is a Unity Archive, you can open it directly: + +``` +UnityDataTool archive list Data/data.unity3d +UnityDataTool archive extract Data/data.unity3d -o extracted +``` + +> [!NOTE] +> The references inside the SerializedFiles are identical whether or not compression is used — a +> reference is always written as, for example, `sharedassets0.assets`, never as a path into +> `data.unity3d`. At runtime the player mounts `data.unity3d` into the `Data` directory via the +> Virtual File System, so those plain names resolve correctly. + +## Built-in resource files + +Two files come from the Editor installation rather than from your project content, and they behave +differently from the rest of the build output: + +- **`Resources/unity default resources`** — default materials, fonts, and meshes that are always + available. This file is pre-built and shipped with the Editor for each target platform, and simply + copied into the build. It is **not** rebuilt during your build, so it never has TypeTrees. Expect + errors when analyzing it, even in a build made with TypeTrees enabled (see + [TypeTrees in the Player](#typetrees-in-the-player)). +- **`Resources/unity_builtin_extra`** — built-in shaders (for the Built-in Render Pipeline), stored + in the editor as uncompiled shader source. Only the built-in shaders actually referenced by your content are + compiled and written into the build output. The file is small or absent when you use a scriptable + render pipeline such as URP or HDRP. The project's "Always Included Shaders" list forces specific shaders in even when nothing + references them directly — useful for shaders you locate at runtime with `Shader.Find()`. + +## TypeTrees in the Player + +By default, Player builds **do not include TypeTrees**. The expectation for Player data is that you +rebuild all content with each build of the player, so the assemblies and serialized objects always +share matching type definitions and no TypeTree is needed to reconcile them. This keeps the +SerializedFile headers small and makes sure that the fast "streamed" load path is used. +(See the [TypeTrees](unity-content-format.md#typetrees) section of the +overview for the full explanation.) + +UnityDataTool's `analyze` and `dump` commands rely on TypeTrees to interpret object contents, so out +of the box they cannot read Player data. To make Player output usable with those commands, enable +the **ForceAlwaysWriteTypeTrees** Diagnostic Switch (Editor **Preferences → Diagnostics → Editor**) +before building. This is supported starting in Unity 2021.2. + +![ForceAlwaysWriteTypeTrees preference](./TypeTreeForPlayer.png) + +> [!IMPORTANT] +> `ForceAlwaysWriteTypeTrees` is a diagnostic aid, not something to ship. Use it for builds you +> intend to analyze, not for production. + +Even with the switch enabled, `Resources/unity default resources` still has no TypeTrees (it is +copied from the Editor, not rebuilt), so analyzing a Player build will report an error for that one +file. This is expected: + +``` +Error processing file: C:\TestProject\CompressedPlayer\TestProject_Data\Resources\unity default resources +System.ArgumentException: Invalid object id. +``` + +The [`serialized-file`](command-serialized-file.md) and [`archive`](command-archive.md) commands do +not need TypeTrees, so they work on any Player build. + +## Inspecting Player content with UnityDataTool + +How you reach the SerializedFiles depends on how the build stored them: + +- **Loose files** in the `Data` folder — read them in place. +- **A `data.unity3d` archive** — [`analyze`](command-analyze.md) and [`dump`](command-dump.md) open + Unity Archives directly, so you can run them on `data.unity3d` without extracting it first. +- **A platform container** (a web `.data` file, an Android `.apk`/`.obb`) — unwrap the outer + container first (see [Platform details](#platform-details)), then work with the loose files or the + `data.unity3d` inside. + +Once you have the SerializedFiles, [`dump`](command-dump.md) prints the contents of individual +objects, [`serialized-file`](command-serialized-file.md) reports a file's header, object list, and +external references, and [`analyze`](command-analyze.md) builds a queryable SQLite database across the +whole build. + +> [!TIP] +> You don't have to complete a full build to inspect the content. After a build, the staged content +> is left in `Library/PlayerDataCache//Data` inside your project (Unity keeps it there as a +> cache for the next build). That folder holds the same SerializedFiles that end up in the final +> build's `Data` folder, before any platform-specific packaging, which makes it a convenient place to +> point UnityDataTool — particularly on platforms where the shipped output is wrapped in a container +> file. + +### Files UnityDataTool does not parse + +The `Data` folder also contains small non-content files that are not SerializedFiles and that +UnityDataTool ignores, including `boot.config` (startup settings), `ScriptingAssemblies.json` (the +list of assemblies), and `RuntimeInitializeOnLoads.json` (methods tagged with +`[RuntimeInitializeOnLoadMethod]`). These are plain config/JSON files; they are mentioned here only +so their presence alongside the content is not surprising. + +## StreamingAssets and nested content builds + +A Player build can carry additional content-only build output inside its `StreamingAssets` folder, +alongside the main Player content. This is separate content, produced by other build pipelines, that +Unity copies into the build verbatim. Common examples include: + +- **AssetBundles**, including the AssetBundles produced by the Addressables package. +- **Content-directory builds.** +- **Entities (DOTS) content** — for example subscene content under `StreamingAssets/ContentArchives`. + +Each of these is made up of the same Unity Archives and SerializedFiles as the rest of the build, so +UnityDataTool can open them too. + +This matters when you run [`analyze`](command-analyze.md) from the `Data` folder. By default `analyze` +recurses into every subdirectory, discovering archives, SerializedFiles, and other supported files +wherever they are — so it will find the nested content under `StreamingAssets` and analyze it into the +*same* database as the main Player content. That may be what you want (a single database covering +everything shipped), but if you want to analyze only the Player content, pass `--no-recurse` to stop +`analyze` from descending into subdirectories. + +## Platform details + +The content stage of a Player build is essentially the same on every platform, but the final +packaging differs, so how you reach the SerializedFiles varies. + +### Android + +On Android the `Data` directory lives inside a container file — an `.apk`, and for larger titles an +`.obb` or Play Asset Delivery asset pack (`.aab` distribution). The content is under +`assets/bin/Data`, and by default it is itself a compressed `data.unity3d` archive, so the +SerializedFiles sit behind two nested container layers. + +- You can browse an `.apk` with a standard zip utility (for example 7-Zip; renaming to `.7z` or + `.zip` can help). Extract the inner `data.unity3d` and then run UnityDataTool on it. +- With the **split binary** option, only the first scene and its dependencies go into the base + module; later scenes and StreamingAssets content go into separate asset packs installed alongside. + +Android output can also depart from the standard layout in ways that affect how you inspect it. The +exact triggers depend on the build configuration (compression, split binary, and related Android +settings), but the two variations worth knowing about are: + +- **`resources.assets` written as separate per-asset files.** In some uncompressed configurations + Android does not combine the `Resources` content into a single `resources.assets`; instead each + asset is written to its own file named by its GUID. This lets the `.apk`'s zip compression + decompress individual assets rather than one large combined file. When compression packs everything + into `data.unity3d` this split does not apply. UnityDataTool can still open each file, but the GUID + names make it harder to recognise what you are looking at. +- **Large files segmented into `.split` parts.** For **uncompressed** builds, files larger than 1 MB + are segmented into `...split0`, `...split1`, … parts (for example + `sharedassets0.assets.resS.split0`), an optimisation for seeking within large files. Unity's file + system presents these as a single file at runtime, but UnityDataTool and `binary2text` do **not** + understand the split convention, so this can make uncompressed Android output awkward to analyze + directly. Compressed Android builds are not split. + +### Web (WebGL) + +Web builds place the content in a `.data` file (sometimes called a web bundle), which UnityDataTool +opens directly — the [`archive list`](command-archive.md#list) and +[`archive extract`](command-archive.md#extract) commands support it. Inside, you'll find the +compressed `data.unity3d` archive plus the `.resource` audio/video files that are kept outside it. +Once extracted, run any other UnityDataTool command on the output. diff --git a/Documentation/unity-content-format.md b/Documentation/unity-content-format.md index efac1e2..1c76918 100644 --- a/Documentation/unity-content-format.md +++ b/Documentation/unity-content-format.md @@ -6,7 +6,7 @@ This section gives an overview of the core Unity file types and how they are use ### SerializedFile -A SerializedFile is Unity's binary file format for serializing objects. It is made up of a file header, then each object serialized one after another. This binary format is also available in the Editor, but typically Editor content uses the Unity YAML format instead. +A SerializedFile is Unity's binary file format for serializing objects. It is made up of a file header, then each object serialized one after another. This binary format is the primary format loaded by the runtime, so when content is built for the Player this is the format that is produced. The binary format is also available in the Editor, but typically user-visible project content uses the Unity YAML format instead of the binary format because the YAML format is more friendly for manual edits and source control operations. The SerializedFiles in build output represent the project content, but optimized for the target platform. Unity will combine objects from multiple source assets together into files, exclude certain objects (for example editor-only objects), and potentially split or duplicate objects across multiple output files. This arrangement of objects is called the `build layout`. Because of all this transformation, there is not a one-to-one mapping between the source assets and the SerializedFiles in the build output. @@ -30,49 +30,36 @@ For a more technical, hands-on look at the internals of AssetBundles - the Asset ## Player Builds -A player build produces content as well as compiled code (assemblies, executables) and various configuration files. UnityDataTool only concerns itself with the content portion of that output. +A Player build produces content alongside compiled code (assemblies, executables) and various configuration files. UnityDataTool only concerns itself with the content portion of that output. -The content comprises the scenes in the Scene List, the contents of Resources folders, content from the Project Preferences (the "GlobalGameManagers"), and all Assets referenced from those root inputs. This translates into SerializedFiles in the build output. +That content is made up of the same SerializedFiles and Unity Archives described above. In brief: -The SerializedFiles are named in a predictable way. This is a very quick summary: +* Each scene in the Scene List becomes a `level` file (`level0`, `level1`, …). +* Assets referenced from scenes are written to `sharedassets` files (`sharedassets0.assets`, …). +* The contents of `Resources` folders become `resources.assets`. +* Global project settings become `globalgamemanagers`, with their referenced assets in `globalgamemanagers.assets`. +* With compression enabled, all of these are packed into a single Unity Archive named `data.unity3d`. -* Each scene in the SceneList becomes a "level" file, e.g. "level0", "level1". -* Assets referenced from Scenes becomes "sharedAssets" files, e.g. "sharedAssets0.assets", "sharedAssets1.assets". Scenes are processed in order of the scene list and assets are stored in the sharedasset file corresponding to the scene where they are first encountered. This means that a level file may reference multiple sharedasset files, but only the ones at the same number and lower. For example the 3rd scene, level2 can reference "sharedAssets2.assets", "sharedAssets1.assets" and "sharedAssets0.assets" but never "sharedAssets3.assets". -* The contents of the Resources folder becomes "resources.assets". -* The Preferences become "globalgamemanager". Assets referenced from "globalgamemanager" are saved in "globalgamemanager.assets". +By default Player builds omit TypeTrees, which limits what `analyze` and `dump` can do with them unless you rebuild with TypeTrees enabled (see [TypeTrees](#typetrees) below). -If [compression](https://docs.unity3d.com/6000.2/Documentation/ScriptReference/BuildOptions.CompressWithLz4HC.html) is enabled, the Player build will compress all the serialized files into a single Unity Archive file, called `data.unity3d`. +For a hands-on look at the Player build layout, the built-in resource files, compression, and how to inspect Player content (including platform-specific containers on Android and Web) with UnityDataTool, see [Player Build Format](playerbuild-format.md). For Unity's official reference, see the Manual topic [Content output of a build](https://docs.unity3d.com/Manual/build-content-output.html). -### Enabling TypeTrees in the Player +## Content Directories -UnityDataTools supports Player build output, because that uses the same SerializedFiles and Archives that AssetBundles use. But often its output is not very useful. That is because, by default, Player builds do not include TypeTrees. +Content directories are a build pipeline introduced in Unity 6.6 for shipping a project's assets as separate content builds that load alongside a Player build. They are designed as a replacement for AssetBundles, and are built from the same SerializedFiles and (optionally) Unity Archives described above. ->[!IMPORTANT] ->It is possible to generate TypeTrees for the Player data, starting in Unity 2021.2. ->This makes that output compatible with UnityDataTool, but it is not a recommended flag to enable for your production builds. - -To do so, the **ForceAlwaysWriteTypeTrees** Diagnostic Switch must be enabled in the Editor Preferences (Diagnostics->Editor section). - -![](./TypeTreeForPlayer.png) - - -Note: The `Resources\unity default resources` file is shipped with the Unity Editor and is not rebuilt when doing a Player Build. It does not have TypeTrees. Hence it is normal that this file emits errors when analyzing a player build, even after rebuilding with TypeTrees enabled. For example: - -``` -Error processing file: C:\TestProject\CompressedPlayer\TestProject_Data\Resources\unity default resources -System.ArgumentException: Invalid object id. -``` - -For more information about TypeTrees see the following section. +For how content directory output is laid out, its build history, and how to inspect it with UnityDataTool, see [Content Directory Format](contentdirectory-format.md). ## TypeTrees The TypeTree is a data structure describing how objects have been serialized, i.e. the name, type, and -size of their properties. It is used by Unity when loading a SerializedFile that was built by a -different Unity version. When Unity deserializes an object it checks whether the current type +size of their properties. It is used by Unity when loading objects out of a SerializedFile that was built by a +different version of the Unity editor, or from different serialization layout for MonoBehaviours and ScriptableObjects in a project. + +When Unity deserializes an object it checks whether the current type definition exactly matches the type definition used when the object was serialized. If they do not match, Unity will attempt to match up the properties as best it can, based on the property names and structure -of the data. This process is called a "Safe Binary Read" and is somewhat slower than the regular fast binary read path. +of the data. This process is called a "Safe Binary Read". This is significantly slower than the regular fast binary read path. TypeTrees are important in the case of AssetBundles, to avoid rebuilding and redistributing all AssetBundles after each minor upgrade of Unity or after doing minor changes to your MonoBehaviour and ScriptableObject serialization. However there can be a noticeable overhead to storing the TypeTrees in each AssetBundle, e.g. the header size of each SerializedFile is bigger. @@ -84,11 +71,13 @@ TypeTrees also make it possible to load an AssetBundle in the Editor, when testi For Player Data the expectation is that you always rebuild all content together with each new build of the player. So the assemblies and serialized objects will all have matching type definitions. That is why, by default, TypeTrees are not included. -UnityDataTools relies on TypeTrees to understand the content of serialized objects. This approach means it does +UnityDataTools relies on TypeTrees to understand the content of serialized objects for the `analyze` and `dump` command. This approach means it does not need to hard-code knowledge about the types and properties of each built-in Unity type (for example Materials and Transforms). It can also interpret serialized C# classes (e.g. MonoBehaviours, ScriptableObjects, and objects serialized through the SerializeReference attribute). This also means that UnityDataTools cannot understand -Player-built content unless the Player was built with TypeTrees enabled. +Player-built content, unless the Player was built with TypeTrees enabled. + +Note: the `serialized-file` and `archive` command do not require TypeTrees. >[!TIP] >The `binary2text` tool supports an optional argument `-typeinfo` to enable dumping out the TypeTrees in a SerializedFile header. That is a useful way to learn more about TypeTrees and to see exactly how Unity data is represented in the binary format. @@ -101,16 +90,6 @@ For details see [Addressable AssetBundle memory considerations](https://docs.uni UnityDataTools commands that open serialized files using the UnityFileSystem API require access to the TypeTrees. For example `dump` and `analyze`. Use the `--typetree-data` option to specify the `.typetreedata` file when examining a build that has extracted TypeTrees. -### Platform details for using UnityDataTool with Player Data - -The output structure and file formats for a Unity Player build are quite platform specific. - -On some platforms the content is packaged into platform-specific container files, for example Android builds use .apk and .obb files. So accessing the actual SerializedFiles may involve mounting or extracting the content of those files, and possibly also opening a data.unity3d file inside them. - -UnityDataTools directly supports opening the `.data` container file format used in Player builds that target Web platforms (e.g. WebGL). Specifically the [`archive list`](command-archive.md#list) and [`archive extract`](command-archive.md#extract) commands work with that format. Once extracted, you can run other UnityDataTool commands on the output. - -Android APK files are not difficult to open and expand using freely available utilities. For example on Windows they can be opened using 7-zip. Once the content is extracted you can run UnityDataTool commands on the output. - ## Mapping back to Source Assets Because Unity rearranges objects in the build into a build layout there is no 1-1 mapping between the output files and the original source assets. Only Scene files have a pretty direct mapping into the build output. @@ -121,10 +100,10 @@ However in cases where you want to understand what contributes to the size of yo Often the source of content can be easily inferred, based on your own knowledge of your project, and the names of objects. For example the name of a Shader should be unique, and typically has a filename that closely matches the Shader name. -You can include a Unity BuildReport file when running `UnityDataTools analyze`. This will import the PackedAsset information, tracking the source asset information for each object in the build output. See [Build Reports](./build-reports.md) for more information, including alternative ways to view the build report. +You can include a Unity BuildReport file when running `UnityDataTool analyze`. This will import the PackedAsset information, tracking the source asset information for each object in the build output. See [Build Reports](./buildreport.md) for more information, including alternative ways to view the build report. -`UnityDataTools analyze` can also import Addressables build layout files, which include source asset information. See [Addressable Build Reports](./addressables-build-reports.md). +`UnityDataTool analyze` can also import Addressables build layout files, which include source asset information. See [Addressable Build Reports](./addressables-build-reports.md). For AssetBundles built by [BuildPipeline.BuildAssetBundles()](https://docs.unity3d.com/ScriptReference/BuildPipeline.BuildAssetBundles.html), Unity creates a .manifest file for each AssetBundle that has source information. This is a text-based format. -For content directory builds, the [ContentLayout.json](contentlayout.md) file maps the build output back to the source assets in the project. +For [content directory builds](contentdirectory-format.md), the [ContentLayout.json](contentlayout.md) file maps the build output back to the source assets in the project. diff --git a/README.md b/README.md index f2bde65..6d2981d 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,9 @@ New to Unity's data files or to UnityDataTool? These topics are a good place to | --- | --- | | [Overview of Unity Content](./Documentation/unity-content-format.md) | The core file types (SerializedFiles and Unity Archives), TypeTrees, and how Player and AssetBundle builds are laid out. | | [AssetBundle Format](./Documentation/assetbundle-format.md) | A hands-on look inside AssetBundles: the AssetBundle object, preload tables, and how scenes are stored. | -| [ContentLayout.json](./Documentation/contentlayout.md) | The build layout file that maps content-directory build output back to source assets. | +| [Player Build Format](./Documentation/playerbuild-format.md) | The Player build layout, compression, built-in resource files, and tips for inspecting Player content (including Android and Web). | +| [Content Directory Format](./Documentation/contentdirectory-format.md) | Content directory builds (Unity 6.6+): what they produce, the build history, and inspecting them with UnityDataTool. | +| [ContentLayout.json](./Documentation/contentlayout.md) | The build layout file that maps content directory build output back to source assets. | **Using UnityDataTool** diff --git a/UnityDataModels/ContentLayout.cs b/UnityDataModels/ContentLayout.cs index 87be4e0..5b19dfb 100644 --- a/UnityDataModels/ContentLayout.cs +++ b/UnityDataModels/ContentLayout.cs @@ -3,8 +3,8 @@ // // See Documentation/contentlayout.md for further details. // -// ContentLayout always represents the latest schema version (currently 1). If the schema changes -// in future, older versions can be preserved under a version-specific namespace while this type +// ContentLayout always represents the latest schema version (currently 2). If the schema changes +// significantly in future, older versions can be preserved under a version-specific namespace while this type // continues to track the latest version. namespace UnityDataTools.Models { @@ -91,6 +91,10 @@ public class LoadableObjectIdLayout /// Index into for the file that contains this /// loadable, or -1 if it was dropped (e.g. server build shader references). public int SerializedFile = -1; + + /// Local file id of the object within its output Content File (the one identified by the + /// index). + public long OutputLFID; } /// @@ -168,7 +172,8 @@ public string FileExtension public class ContentLayout { /// The schema version this type represents. - public const int CurrentVersion = 1; + // v1 -> v2: added OutputLFID to LoadableObjectIds entries. + public const int CurrentVersion = 2; /// Schema version of the ContentLayout.json file. public int Version;