![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
\n",
+ "\n",
+ "# Plant-data reconciliation and a Bayesian digital twin with NeqSim\n",
+ "\n",
+ "This long-form tutorial turns noisy process measurements into an auditable, uncertainty-aware\n",
+ "digital twin. A source-built NeqSim model supplies the thermodynamic and process calculations;\n",
+ "NeqSim's native reconciliation and calibration APIs supply the engineering estimators; and an\n",
+ "independent Python calculation checks the algebra and adds a transparent Bayesian layer.\n",
+ "\n",
+ "The data are deliberately synthetic and reproducible. They represent a gas-condensate inlet\n",
+ "separator and an export compressor, not a named asset. Stored outputs are evidence from a clean,\n",
+ "top-to-bottom execution."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "markdown-002",
+ "metadata": {},
+ "source": [
+ "## Learning outcomes\n",
+ "\n",
+ "After completing the notebook, you can:\n",
+ "\n",
+ "1. explain why a steady-state gate must precede steady-state data reconciliation;\n",
+ "2. configure NeqSim's `SteadyStateDetector` from historian-style tag samples;\n",
+ "3. reconcile redundant mass-flow measurements with uncertainty-weighted least squares;\n",
+ "4. verify the native NeqSim result against the closed-form NumPy solution;\n",
+ "5. detect, rank, and isolate a gross sensor error without silently changing raw data;\n",
+ "6. calibrate compressor polytropic efficiency with `BatchParameterEstimator`;\n",
+ "7. form a Bayesian posterior from a validated NeqSim response surface;\n",
+ "8. test the calibrated twin on held-out operating points;\n",
+ "9. propagate parameter uncertainty to compressor power and a teaching constraint; and\n",
+ "10. publish a machine-readable evidence contract with explicit limitations."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "markdown-003",
+ "metadata": {},
+ "source": [
+ "## Why this topic belongs in the NeqSim-Colab collection\n",
+ "\n",
+ "The collection already contains excellent notebooks on online process simulation, IoT telemetry,\n",
+ "condition monitoring, machine learning, and model-versus-measurement calibration. The missing link\n",
+ "is a dedicated treatment of **measurement trust before model tuning**:\n",
+ "\n",
+ "`historian window -> steady-state qualification -> reconciliation -> gross-error isolation ->`\n",
+ "`parameter calibration -> posterior uncertainty -> engineering decision`\n",
+ "\n",
+ "This notebook also fulfils the maintenance-ledger follow-up that replaced the retired, corrupted\n",
+ "`syntheticdatageneration.ipynb`: create a current-master data-reconciliation and Bayesian\n",
+ "digital-twin tutorial.\n",
+ "\n",
+ "### Engineering boundary\n",
+ "\n",
+ "- The example is educational and uses synthetic, non-asset data.\n",
+ "- Pressure is absolute and reported in bara.\n",
+ "- Flow is mass flow in kg/h unless explicitly stated otherwise.\n",
+ "- Measurement uncertainties are one-standard-deviation values.\n",
+ "- The reconciliation constraints are linear, steady-state total-mass balances.\n",
+ "- The Bayesian result quantifies one parameter only: compressor polytropic efficiency.\n",
+ "- Nothing here is a certified meter-validation, custody-transfer, alarm, or equipment-design study."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "markdown-004",
+ "metadata": {},
+ "source": [
+ "## 1. Mathematical foundation\n",
+ "\n",
+ "### 1.1 Steady-state qualification\n",
+ "\n",
+ "The Cao-Rhinehart statistic compares variance in successive differences with ordinary sample\n",
+ "variance. For a window of $n$ measurements $x_i$,\n",
+ "\n",
+ "$$\\begin{aligned}\\sigma_f^2&=\\frac{1}{2(n-1)}\\sum_{i=2}^{n}(x_i-x_{i-1})^2,\\\\\\sigma_u^2&=\\frac{1}{n-1}\\sum_{i=1}^{n}(x_i-\\bar{x})^2,\\\\R&=\\frac{\\sigma_f^2}{\\sigma_u^2}.\\end{aligned}$$\n",
+ "\n",
+ "White-noise-like variation gives $R$ near one. A trend or step spreads the ordinary variance while\n",
+ "successive changes remain structured, so $R$ is usually smaller. In this tutorial all monitored\n",
+ "tags must have $R\\geq0.5$ over a full 20-sample window before reconciliation is allowed.\n",
+ "\n",
+ "### 1.2 Weighted least-squares reconciliation\n",
+ "\n",
+ "Let $\\mathbf{y}$ be measurements, $\\mathbf{V}=\\mathrm{diag}(\\sigma_i^2)$ their covariance matrix,\n",
+ "and $\\mathbf{A}\\mathbf{x}=\\mathbf{0}$ the balance constraints. NeqSim uses\n",
+ "\n",
+ "$$\\hat{\\mathbf{x}}=\\mathbf{y}-\\mathbf{V}\\mathbf{A}^{\\mathsf{T}}(\\mathbf{A}\\mathbf{V}\\mathbf{A}^{\\mathsf{T}})^{-1}\\mathbf{A}\\mathbf{y}.$$\n",
+ "\n",
+ "The objective $J=(\\hat{\\mathbf{x}}-\\mathbf{y})^{\\mathsf{T}}\\mathbf{V}^{-1}(\\hat{\\mathbf{x}}-\\mathbf{y})$\n",
+ "supports a global $\\chi^2$ consistency test. Per-variable normalized residuals help rank suspect\n",
+ "measurements, but conservation alone does not prove which instrument is faulty.\n",
+ "\n",
+ "### 1.3 Bayesian parameter update\n",
+ "\n",
+ "For compressor efficiency $\\eta$, observations $\\mathbf{z}$, and NeqSim predictions\n",
+ "$\\mathbf{g}(\\eta)$, Bayes' rule is\n",
+ "\n",
+ "$$p(\\eta\\mid\\mathbf{z})\\propto p(\\mathbf{z}\\mid\\eta)p(\\eta).$$\n",
+ "\n",
+ "With independent Gaussian temperature errors $\\sigma_T$,\n",
+ "\n",
+ "$$\\log p(\\mathbf{z}\\mid\\eta)=-\\frac{1}{2}\\sum_j\\left(\\frac{z_j-g_j(\\eta)}{\\sigma_T}\\right)^2+C.$$\n",
+ "\n",
+ "The posterior is computed on a dense one-dimensional grid. A shape-preserving interpolator makes\n",
+ "that grid inexpensive, but every response surface is generated by NeqSim and independently checked\n",
+ "at off-grid efficiencies before it is trusted."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "markdown-005",
+ "metadata": {},
+ "source": [
+ "## 2. Clean Colab setup\n",
+ "\n",
+ "The released `neqsim` wheel is pinned as the Python/JPype bridge. Advanced calculations use a Java\n",
+ "JAR built from the selected `equinor/neqsim` source ref. A validation runner may supply an exact\n",
+ "checkout and JAR with `NEQSIM_SOURCE_ROOT` and `NEQSIM_SOURCE_JAR`; otherwise this notebook clones\n",
+ "and builds current `master` itself."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "code-006",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Required Python packages are available.\n"
+ ]
+ }
+ ],
+ "source": [
+ "import importlib.metadata\n",
+ "import importlib.util\n",
+ "import os\n",
+ "from pathlib import Path\n",
+ "import subprocess\n",
+ "import sys\n",
+ "\n",
+ "REQUIRED_PACKAGES = {\n",
+ " \"neqsim\": \"3.18.0\",\n",
+ "}\n",
+ "\n",
+ "install_requirements = []\n",
+ "for package_name, required_version in REQUIRED_PACKAGES.items():\n",
+ " try:\n",
+ " installed_version = importlib.metadata.version(package_name)\n",
+ " except importlib.metadata.PackageNotFoundError:\n",
+ " installed_version = None\n",
+ " if installed_version != required_version:\n",
+ " install_requirements.append(f\"{package_name}=={required_version}\")\n",
+ "\n",
+ "for module_name in [\"matplotlib\", \"numpy\", \"pandas\", \"scipy\"]:\n",
+ " if importlib.util.find_spec(module_name) is None:\n",
+ " install_requirements.append(module_name)\n",
+ "\n",
+ "if install_requirements:\n",
+ " subprocess.run(\n",
+ " [\n",
+ " sys.executable,\n",
+ " \"-m\",\n",
+ " \"pip\",\n",
+ " \"install\",\n",
+ " \"--quiet\",\n",
+ " *install_requirements,\n",
+ " ],\n",
+ " check=True,\n",
+ " timeout=1200,\n",
+ " )\n",
+ "\n",
+ "print(\"Required Python packages are available.\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "code-007",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "NeqSim source ref: master\n",
+ "NeqSim resolved commit: fdf6b227b4240589ebbfb900527b37e667f3efc8\n",
+ "NeqSim JAR SHA-256: 070494e0b9488e1c2d1f2ba75c212466346ee6c07eed950001fa5360daecfbd0\n",
+ "Loaded reconciliation class from: file:/workspace/scratch/c5d511e6ac7e/neqsim-source/target/neqsim-3.18.0.jar\n"
+ ]
+ }
+ ],
+ "source": [
+ "import hashlib\n",
+ "import jpype\n",
+ "\n",
+ "\n",
+ "def run_command(command, *, cwd=None, timeout=1800, environment=None):\n",
+ " result = subprocess.run(\n",
+ " command,\n",
+ " cwd=cwd,\n",
+ " env=environment,\n",
+ " text=True,\n",
+ " stdout=subprocess.PIPE,\n",
+ " stderr=subprocess.STDOUT,\n",
+ " timeout=timeout,\n",
+ " )\n",
+ " if result.returncode != 0:\n",
+ " output_tail = \"\\n\".join(result.stdout.splitlines()[-80:])\n",
+ " raise RuntimeError(\n",
+ " f\"Command failed ({result.returncode}): {command}\\n{output_tail}\"\n",
+ " )\n",
+ " return result.stdout.strip()\n",
+ "\n",
+ "\n",
+ "NEQSIM_SOURCE_REF = os.environ.get(\"NEQSIM_SOURCE_REF\", \"master\")\n",
+ "supplied_source_root = os.environ.get(\"NEQSIM_SOURCE_ROOT\", \"\").strip()\n",
+ "supplied_source_jar = os.environ.get(\"NEQSIM_SOURCE_JAR\", \"\").strip()\n",
+ "\n",
+ "if supplied_source_root and supplied_source_jar:\n",
+ " neqsim_source_root = Path(supplied_source_root).resolve()\n",
+ " neqsim_source_jar = Path(supplied_source_jar).resolve()\n",
+ "else:\n",
+ " runtime_root = Path(\"/content\")\n",
+ " if not runtime_root.exists():\n",
+ " runtime_root = Path(os.environ.get(\"RUNNER_TEMP\", \"/tmp\")).resolve()\n",
+ " neqsim_source_root = runtime_root / \"neqsim-java-master\"\n",
+ " if not neqsim_source_root.exists():\n",
+ " run_command(\n",
+ " [\n",
+ " \"git\",\n",
+ " \"clone\",\n",
+ " \"--depth\",\n",
+ " \"1\",\n",
+ " \"--branch\",\n",
+ " NEQSIM_SOURCE_REF,\n",
+ " \"https://github.com/equinor/neqsim.git\",\n",
+ " str(neqsim_source_root),\n",
+ " ],\n",
+ " timeout=600,\n",
+ " )\n",
+ " else:\n",
+ " run_command(\n",
+ " [\"git\", \"fetch\", \"--depth\", \"1\", \"origin\", NEQSIM_SOURCE_REF],\n",
+ " cwd=neqsim_source_root,\n",
+ " timeout=600,\n",
+ " )\n",
+ " run_command(\n",
+ " [\"git\", \"checkout\", \"--detach\", \"FETCH_HEAD\"],\n",
+ " cwd=neqsim_source_root,\n",
+ " )\n",
+ "\n",
+ " maven_settings = runtime_root / \"neqsim-maven-settings.xml\"\n",
+ " maven_settings.write_text(\n",
+ " \"| \n", + " | Dependency | \n", + "Resolved version or identity | \n", + "
|---|---|---|
| 0 | \n", + "NeqSim Python bridge | \n", + "3.18.0 | \n", + "
| 1 | \n", + "NeqSim Java commit | \n", + "fdf6b227b4240589ebbfb900527b37e667f3efc8 | \n", + "
| 2 | \n", + "Java runtime | \n", + "17.0.20 | \n", + "
| 3 | \n", + "Python | \n", + "3.12.13 | \n", + "
| 4 | \n", + "NumPy | \n", + "2.5.2 | \n", + "
| 5 | \n", + "pandas | \n", + "3.0.5 | \n", + "
| 6 | \n", + "Matplotlib | \n", + "3.10.8 | \n", + "
| 7 | \n", + "SciPy | \n", + "1.17.0 | \n", + "
| \n", + " | Component | \n", + "Mole fraction | \n", + "Mole percent | \n", + "
|---|---|---|---|
| 0 | \n", + "nitrogen | \n", + "0.010101 | \n", + "1.010101 | \n", + "
| 1 | \n", + "CO2 | \n", + "0.020202 | \n", + "2.020202 | \n", + "
| 2 | \n", + "methane | \n", + "0.686869 | \n", + "68.686869 | \n", + "
| 3 | \n", + "ethane | \n", + "0.101010 | \n", + "10.101010 | \n", + "
| 4 | \n", + "propane | \n", + "0.070707 | \n", + "7.070707 | \n", + "
| 5 | \n", + "i-butane | \n", + "0.020202 | \n", + "2.020202 | \n", + "
| 6 | \n", + "n-butane | \n", + "0.030303 | \n", + "3.030303 | \n", + "
| 7 | \n", + "i-pentane | \n", + "0.015152 | \n", + "1.515152 | \n", + "
| 8 | \n", + "n-pentane | \n", + "0.015152 | \n", + "1.515152 | \n", + "
| 9 | \n", + "n-hexane | \n", + "0.015152 | \n", + "1.515152 | \n", + "
| 10 | \n", + "n-heptane | \n", + "0.015152 | \n", + "1.515152 | \n", + "
| \n", + " | Stream | \n", + "Mass flow [kg/h] | \n", + "Pressure [bara] | \n", + "Temperature [°C] | \n", + "
|---|---|---|---|---|
| 0 | \n", + "Feed | \n", + "60000.000000 | \n", + "85.0 | \n", + "45.000000 | \n", + "
| 1 | \n", + "Separator gas | \n", + "40717.078499 | \n", + "45.0 | \n", + "20.000000 | \n", + "
| 2 | \n", + "Separator liquid | \n", + "19282.921501 | \n", + "45.0 | \n", + "20.000000 | \n", + "
| 3 | \n", + "Export gas | \n", + "40717.078499 | \n", + "100.0 | \n", + "87.379134 | \n", + "
| \n", + " | Sample | \n", + "feed [kg/h] | \n", + "separator_gas [kg/h] | \n", + "separator_liquid [kg/h] | \n", + "All tags steady | \n", + "
|---|---|---|---|---|---|
| 110 | \n", + "110 | \n", + "60061.402609 | \n", + "40537.157356 | \n", + "19325.368047 | \n", + "True | \n", + "
| 111 | \n", + "111 | \n", + "60020.529403 | \n", + "40523.803541 | \n", + "19304.719790 | \n", + "True | \n", + "
| 112 | \n", + "112 | \n", + "60178.357174 | \n", + "40854.851800 | \n", + "19146.978293 | \n", + "True | \n", + "
| 113 | \n", + "113 | \n", + "60022.855989 | \n", + "40780.074353 | \n", + "19333.004595 | \n", + "True | \n", + "
| 114 | \n", + "114 | \n", + "59813.453092 | \n", + "40532.920145 | \n", + "19283.813315 | \n", + "True | \n", + "
| 115 | \n", + "115 | \n", + "60007.358207 | \n", + "40696.993438 | \n", + "19227.809911 | \n", + "True | \n", + "
| 116 | \n", + "116 | \n", + "59767.123474 | \n", + "40798.006768 | \n", + "19178.644432 | \n", + "True | \n", + "
| 117 | \n", + "117 | \n", + "59796.585249 | \n", + "40537.956409 | \n", + "19249.131511 | \n", + "True | \n", + "
| 118 | \n", + "118 | \n", + "60224.213512 | \n", + "40838.263150 | \n", + "19239.175334 | \n", + "True | \n", + "
| 119 | \n", + "119 | \n", + "59903.538563 | \n", + "40871.316934 | \n", + "19186.228207 | \n", + "True | \n", + "
| \n", + " | Tag | \n", + "True [kg/h] | \n", + "Measured [kg/h] | \n", + "Sigma [kg/h] | \n", + "Reconciled [kg/h] | \n", + "Adjustment [kg/h] | \n", + "Normalized residual [-] | \n", + "
|---|---|---|---|---|---|---|---|
| 0 | \n", + "feed | \n", + "60000.0000 | \n", + "60045.0000 | \n", + "180.0 | \n", + "60013.6110 | \n", + "-31.3890 | \n", + "-0.2062 | \n", + "
| 1 | \n", + "separator_gas | \n", + "40717.0785 | \n", + "40682.0785 | \n", + "140.0 | \n", + "40725.7745 | \n", + "43.6960 | \n", + "0.3933 | \n", + "
| 2 | \n", + "separator_liquid | \n", + "19282.9215 | \n", + "19307.9215 | \n", + "90.0 | \n", + "19287.8364 | \n", + "-20.0851 | \n", + "-0.2960 | \n", + "
| 3 | \n", + "export_gas | \n", + "40717.0785 | \n", + "40747.0785 | \n", + "130.0 | \n", + "40725.7745 | \n", + "-21.3040 | \n", + "-0.2169 | \n", + "
| 4 | \n", + "export_liquid | \n", + "19282.9215 | \n", + "19262.9215 | \n", + "85.0 | \n", + "19287.8364 | \n", + "24.9149 | \n", + "0.4080 | \n", + "
| \n", + " | Tag | \n", + "True [kg/h] | \n", + "Biased measurement [kg/h] | \n", + "Reconciled [kg/h] | \n", + "Normalized residual [-] | \n", + "Flagged | \n", + "
|---|---|---|---|---|---|---|
| 0 | \n", + "feed | \n", + "60000.0000 | \n", + "60045.0000 | \n", + "60212.3353 | \n", + "1.0992 | \n", + "False | \n", + "
| 1 | \n", + "separator_gas | \n", + "40717.0785 | \n", + "41282.0785 | \n", + "40947.9211 | \n", + "-3.0077 | \n", + "True | \n", + "
| 2 | \n", + "separator_liquid | \n", + "19282.9215 | \n", + "19307.9215 | \n", + "19264.4142 | \n", + "-0.6413 | \n", + "False | \n", + "
| 3 | \n", + "export_gas | \n", + "40717.0785 | \n", + "40747.0785 | \n", + "40947.9211 | \n", + "2.0452 | \n", + "True | \n", + "
| 4 | \n", + "export_liquid | \n", + "19282.9215 | \n", + "19262.9215 | \n", + "19264.4142 | \n", + "0.0244 | \n", + "False | \n", + "
| \n", + " | Estimate | \n", + "Gas flow [kg/h] | \n", + "Error vs synthetic truth [kg/h] | \n", + "
|---|---|---|---|
| 0 | \n", + "Raw FI-102 | \n", + "41282.0785 | \n", + "565.0000 | \n", + "
| 1 | \n", + "Reconciled virtual FI-102 from FI-104 | \n", + "40751.4642 | \n", + "34.3857 | \n", + "
| \n", + " | Role | \n", + "Discharge pressure [bara] | \n", + "True outlet temperature [°C] | \n", + "Measured outlet temperature [°C] | \n", + "Measurement error [K] | \n", + "True compressor power [MW] | \n", + "
|---|---|---|---|---|---|---|
| 0 | \n", + "Training | \n", + "75.0 | \n", + "62.4098 | \n", + "62.2877 | \n", + "-0.1221 | \n", + "0.7774 | \n", + "
| 1 | \n", + "Training | \n", + "82.0 | \n", + "70.1376 | \n", + "70.1364 | \n", + "-0.0012 | \n", + "0.9266 | \n", + "
| 2 | \n", + "Training | \n", + "89.0 | \n", + "77.3117 | \n", + "77.4954 | \n", + "0.1836 | \n", + "1.0676 | \n", + "
| 3 | \n", + "Training | \n", + "96.0 | \n", + "84.0096 | \n", + "83.6522 | \n", + "-0.3574 | \n", + "1.2016 | \n", + "
| 4 | \n", + "Training | \n", + "103.0 | \n", + "90.2931 | \n", + "90.0397 | \n", + "-0.2534 | \n", + "1.3294 | \n", + "
| 5 | \n", + "Training | \n", + "110.0 | \n", + "96.2129 | \n", + "96.1467 | \n", + "-0.0662 | \n", + "1.4519 | \n", + "
| 6 | \n", + "Training | \n", + "117.0 | \n", + "101.8109 | \n", + "102.1382 | \n", + "0.3273 | \n", + "1.5695 | \n", + "
| 7 | \n", + "Training | \n", + "124.0 | \n", + "107.1220 | \n", + "107.0363 | \n", + "-0.0857 | \n", + "1.6830 | \n", + "
| 8 | \n", + "Holdout | \n", + "78.5 | \n", + "66.3484 | \n", + "65.8513 | \n", + "-0.4971 | \n", + "0.8531 | \n", + "
| 9 | \n", + "Holdout | \n", + "92.5 | \n", + "80.7160 | \n", + "80.2870 | \n", + "-0.4290 | \n", + "1.1354 | \n", + "
| 10 | \n", + "Holdout | \n", + "106.5 | \n", + "93.2956 | \n", + "93.1380 | \n", + "-0.1577 | \n", + "1.3913 | \n", + "
| 11 | \n", + "Holdout | \n", + "125.0 | \n", + "107.8591 | \n", + "107.4254 | \n", + "-0.4337 | \n", + "1.6988 | \n", + "
| \n", + " | Metric | \n", + "Value | \n", + "Reported uncertainty | \n", + "
|---|---|---|---|
| 0 | \n", + "Synthetic true efficiency | \n", + "0.780000 | \n", + "- | \n", + "
| 1 | \n", + "Native batch estimate | \n", + "0.780616 | \n", + "0.001435 | \n", + "
| 2 | \n", + "Training RMSE [K] | \n", + "0.208532 | \n", + "- | \n", + "
| 3 | \n", + "Holdout RMSE [K] | \n", + "0.367367 | \n", + "- | \n", + "
| 4 | \n", + "Batch chi-square | \n", + "5.566167 | \n", + "- | \n", + "
| 5 | \n", + "Batch R² | \n", + "0.999797 | \n", + "- | \n", + "
| \n", + " | Response | \n", + "Maximum off-grid absolute error | \n", + "Unit | \n", + "
|---|---|---|---|
| 0 | \n", + "Discharge temperature | \n", + "9.690029e-08 | \n", + "K | \n", + "
| 1 | \n", + "Compressor power | \n", + "9.194516e-10 | \n", + "MW | \n", + "
| \n", + " | mean | \n", + "map | \n", + "lower_95 | \n", + "median | \n", + "upper_95 | \n", + "observations | \n", + "
|---|---|---|---|---|---|---|
| 0 | \n", + "0.782841 | \n", + "0.78264 | \n", + "0.769832 | \n", + "0.782740 | \n", + "0.796117 | \n", + "1 | \n", + "
| 1 | \n", + "0.781207 | \n", + "0.78112 | \n", + "0.772764 | \n", + "0.781141 | \n", + "0.789719 | \n", + "2 | \n", + "
| 2 | \n", + "0.779112 | \n", + "0.77904 | \n", + "0.772784 | \n", + "0.779057 | \n", + "0.785443 | \n", + "3 | \n", + "
| 3 | \n", + "0.781648 | \n", + "0.78160 | \n", + "0.776522 | \n", + "0.781599 | \n", + "0.786750 | \n", + "4 | \n", + "
| 4 | \n", + "0.782363 | \n", + "0.78232 | \n", + "0.778069 | \n", + "0.782316 | \n", + "0.786615 | \n", + "5 | \n", + "
| 5 | \n", + "0.781985 | \n", + "0.78200 | \n", + "0.778305 | \n", + "0.781940 | \n", + "0.785613 | \n", + "6 | \n", + "
| 6 | \n", + "0.780472 | \n", + "0.78048 | \n", + "0.777269 | \n", + "0.780428 | \n", + "0.783616 | \n", + "7 | \n", + "
| 7 | \n", + "0.780598 | \n", + "0.78056 | \n", + "0.777757 | \n", + "0.780555 | \n", + "0.783377 | \n", + "8 | \n", + "
| \n", + " | Statistic | \n", + "Efficiency [-] | \n", + "
|---|---|---|
| 0 | \n", + "Prior mean | \n", + "0.760000 | \n", + "
| 1 | \n", + "Native batch estimate | \n", + "0.780616 | \n", + "
| 2 | \n", + "Posterior mean | \n", + "0.780598 | \n", + "
| 3 | \n", + "Posterior MAP | \n", + "0.780560 | \n", + "
| 4 | \n", + "Posterior median | \n", + "0.780555 | \n", + "
| 5 | \n", + "95% lower | \n", + "0.777757 | \n", + "
| 6 | \n", + "95% upper | \n", + "0.783377 | \n", + "
| 7 | \n", + "Synthetic truth | \n", + "0.780000 | \n", + "
| \n", + " | Pressure [bara] | \n", + "Synthetic truth [°C] | \n", + "Held-out measurement [°C] | \n", + "Posterior predictive mean [°C] | \n", + "Predictive 2.5% [°C] | \n", + "Predictive 97.5% [°C] | \n", + "Measurement covered | \n", + "
|---|---|---|---|---|---|---|---|
| 0 | \n", + "78.5 | \n", + "66.3484 | \n", + "65.8513 | \n", + "66.3257 | \n", + "65.8237 | \n", + "66.8281 | \n", + "True | \n", + "
| 1 | \n", + "92.5 | \n", + "80.7160 | \n", + "80.2870 | \n", + "80.6817 | \n", + "80.1688 | \n", + "81.1808 | \n", + "True | \n", + "
| 2 | \n", + "106.5 | \n", + "93.2956 | \n", + "93.1380 | \n", + "93.2527 | \n", + "92.7319 | \n", + "93.7692 | \n", + "True | \n", + "
| 3 | \n", + "125.0 | \n", + "107.8591 | \n", + "107.4254 | \n", + "107.8105 | \n", + "107.2717 | \n", + "108.3580 | \n", + "True | \n", + "
| \n", + " | Pressure [bara] | \n", + "Posterior mean power [MW] | \n", + "Power 2.5% [MW] | \n", + "Power 97.5% [MW] | \n", + "
|---|---|---|---|---|
| 0 | \n", + "75.0 | \n", + "0.77677 | \n", + "0.77371 | \n", + "0.77974 | \n", + "
| 1 | \n", + "78.5 | \n", + "0.85241 | \n", + "0.84903 | \n", + "0.85570 | \n", + "
| 2 | \n", + "82.0 | \n", + "0.92585 | \n", + "0.92215 | \n", + "0.92944 | \n", + "
| 3 | \n", + "89.0 | \n", + "1.06673 | \n", + "1.06241 | \n", + "1.07092 | \n", + "
| 4 | \n", + "92.5 | \n", + "1.13447 | \n", + "1.12984 | \n", + "1.13895 | \n", + "
| 5 | \n", + "96.0 | \n", + "1.20056 | \n", + "1.19564 | \n", + "1.20534 | \n", + "
| 6 | \n", + "103.0 | \n", + "1.32827 | \n", + "1.32276 | \n", + "1.33362 | \n", + "
| 7 | \n", + "106.5 | \n", + "1.39006 | \n", + "1.38427 | \n", + "1.39569 | \n", + "
| 8 | \n", + "110.0 | \n", + "1.45059 | \n", + "1.44451 | \n", + "1.45650 | \n", + "
| 9 | \n", + "117.0 | \n", + "1.56815 | \n", + "1.56151 | \n", + "1.57460 | \n", + "
| 10 | \n", + "124.0 | \n", + "1.68146 | \n", + "1.67427 | \n", + "1.68845 | \n", + "
| 11 | \n", + "125.0 | \n", + "1.69733 | \n", + "1.69006 | \n", + "1.70439 | \n", + "
| \n", + " | Mass flow [kg/h] | \n", + "Discharge temperature [°C] | \n", + "Power [MW] | \n", + "Specific power [MW/(kg/h)] | \n", + "
|---|---|---|---|---|
| 0 | \n", + "38000.0 | \n", + "87.610883 | \n", + "1.210520 | \n", + "0.000032 | \n", + "
| 1 | \n", + "40000.0 | \n", + "87.610883 | \n", + "1.274232 | \n", + "0.000032 | \n", + "
| 2 | \n", + "42000.0 | \n", + "87.610883 | \n", + "1.337943 | \n", + "0.000032 | \n", + "
| \n", + " | Validation check | \n", + "Passed | \n", + "
|---|---|---|
| 0 | \n", + "source_built_reconciliation_class_loaded | \n", + "True | \n", + "
| 1 | \n", + "composition_normalized | \n", + "True | \n", + "
| 2 | \n", + "separator_gas_and_liquid_are_positive | \n", + "True | \n", + "
| 3 | \n", + "native_process_mass_balance | \n", + "True | \n", + "
| 4 | \n", + "steady_state_gate_opens_after_full_window | \n", + "True | \n", + "
| 5 | \n", + "final_historian_window_is_steady | \n", + "True | \n", + "
| 6 | \n", + "normal_reconciliation_converged | \n", + "True | \n", + "
| 7 | \n", + "normal_global_test_passed | \n", + "True | \n", + "
| 8 | \n", + "normal_constraints_close | \n", + "True | \n", + "
| 9 | \n", + "native_matches_independent_wls | \n", + "True | \n", + "
| 10 | \n", + "biased_global_test_fails | \n", + "True | \n", + "
| 11 | \n", + "largest_residual_localizes_separator_gas | \n", + "True | \n", + "
| 12 | \n", + "diagnostic_flags_separator_gas | \n", + "True | \n", + "
| 13 | \n", + "reduced_problem_passes | \n", + "True | \n", + "
| 14 | \n", + "virtual_meter_improves_gas_estimate | \n", + "True | \n", + "
| 15 | \n", + "native_batch_estimator_converged | \n", + "True | \n", + "
| 16 | \n", + "native_batch_recovers_efficiency | \n", + "True | \n", + "
| 17 | \n", + "native_batch_holdout_rmse_is_small | \n", + "True | \n", + "
| 18 | \n", + "temperature_surrogate_is_accurate | \n", + "True | \n", + "
| 19 | \n", + "power_surrogate_is_accurate | \n", + "True | \n", + "
| 20 | \n", + "posterior_contains_synthetic_truth | \n", + "True | \n", + "
| 21 | \n", + "posterior_mean_recovers_efficiency | \n", + "True | \n", + "
| 22 | \n", + "posterior_contracts_from_prior | \n", + "True | \n", + "
| 23 | \n", + "posterior_holdout_rmse_is_small | \n", + "True | \n", + "
| 24 | \n", + "power_increases_with_pressure | \n", + "True | \n", + "
| 25 | \n", + "flow_perturbation_preserves_temperature | \n", + "True | \n", + "
| 26 | \n", + "power_scales_with_flow | \n", + "True | \n", + "
+
+# Plant-data reconciliation and a Bayesian digital twin with NeqSim
+
+This long-form tutorial turns noisy process measurements into an auditable, uncertainty-aware
+digital twin. A source-built NeqSim model supplies the thermodynamic and process calculations;
+NeqSim's native reconciliation and calibration APIs supply the engineering estimators; and an
+independent Python calculation checks the algebra and adds a transparent Bayesian layer.
+
+The data are deliberately synthetic and reproducible. They represent a gas-condensate inlet
+separator and an export compressor, not a named asset. Stored outputs are evidence from a clean,
+top-to-bottom execution.
+"""))
+
+cells.append(md(r"""
+## Learning outcomes
+
+After completing the notebook, you can:
+
+1. explain why a steady-state gate must precede steady-state data reconciliation;
+2. configure NeqSim's `SteadyStateDetector` from historian-style tag samples;
+3. reconcile redundant mass-flow measurements with uncertainty-weighted least squares;
+4. verify the native NeqSim result against the closed-form NumPy solution;
+5. detect, rank, and isolate a gross sensor error without silently changing raw data;
+6. calibrate compressor polytropic efficiency with `BatchParameterEstimator`;
+7. form a Bayesian posterior from a validated NeqSim response surface;
+8. test the calibrated twin on held-out operating points;
+9. propagate parameter uncertainty to compressor power and a teaching constraint; and
+10. publish a machine-readable evidence contract with explicit limitations.
+"""))
+
+cells.append(md(r"""
+## Why this topic belongs in the NeqSim-Colab collection
+
+The collection already contains excellent notebooks on online process simulation, IoT telemetry,
+condition monitoring, machine learning, and model-versus-measurement calibration. The missing link
+is a dedicated treatment of **measurement trust before model tuning**:
+
+`historian window -> steady-state qualification -> reconciliation -> gross-error isolation ->`
+`parameter calibration -> posterior uncertainty -> engineering decision`
+
+This notebook also fulfils the maintenance-ledger follow-up that replaced the retired, corrupted
+`syntheticdatageneration.ipynb`: create a current-master data-reconciliation and Bayesian
+digital-twin tutorial.
+
+### Engineering boundary
+
+- The example is educational and uses synthetic, non-asset data.
+- Pressure is absolute and reported in bara.
+- Flow is mass flow in kg/h unless explicitly stated otherwise.
+- Measurement uncertainties are one-standard-deviation values.
+- The reconciliation constraints are linear, steady-state total-mass balances.
+- The Bayesian result quantifies one parameter only: compressor polytropic efficiency.
+- Nothing here is a certified meter-validation, custody-transfer, alarm, or equipment-design study.
+"""))
+
+cells.append(md(r"""
+## 1. Mathematical foundation
+
+### 1.1 Steady-state qualification
+
+The Cao-Rhinehart statistic compares variance in successive differences with ordinary sample
+variance. For a window of $n$ measurements $x_i$,
+
+$$\begin{aligned}\sigma_f^2&=\frac{1}{2(n-1)}\sum_{i=2}^{n}(x_i-x_{i-1})^2,\\\sigma_u^2&=\frac{1}{n-1}\sum_{i=1}^{n}(x_i-\bar{x})^2,\\R&=\frac{\sigma_f^2}{\sigma_u^2}.\end{aligned}$$
+
+White-noise-like variation gives $R$ near one. A trend or step spreads the ordinary variance while
+successive changes remain structured, so $R$ is usually smaller. In this tutorial all monitored
+tags must have $R\geq0.5$ over a full 20-sample window before reconciliation is allowed.
+
+### 1.2 Weighted least-squares reconciliation
+
+Let $\mathbf{y}$ be measurements, $\mathbf{V}=\mathrm{diag}(\sigma_i^2)$ their covariance matrix,
+and $\mathbf{A}\mathbf{x}=\mathbf{0}$ the balance constraints. NeqSim uses
+
+$$\hat{\mathbf{x}}=\mathbf{y}-\mathbf{V}\mathbf{A}^{\mathsf{T}}(\mathbf{A}\mathbf{V}\mathbf{A}^{\mathsf{T}})^{-1}\mathbf{A}\mathbf{y}.$$
+
+The objective $J=(\hat{\mathbf{x}}-\mathbf{y})^{\mathsf{T}}\mathbf{V}^{-1}(\hat{\mathbf{x}}-\mathbf{y})$
+supports a global $\chi^2$ consistency test. Per-variable normalized residuals help rank suspect
+measurements, but conservation alone does not prove which instrument is faulty.
+
+### 1.3 Bayesian parameter update
+
+For compressor efficiency $\eta$, observations $\mathbf{z}$, and NeqSim predictions
+$\mathbf{g}(\eta)$, Bayes' rule is
+
+$$p(\eta\mid\mathbf{z})\propto p(\mathbf{z}\mid\eta)p(\eta).$$
+
+With independent Gaussian temperature errors $\sigma_T$,
+
+$$\log p(\mathbf{z}\mid\eta)=-\frac{1}{2}\sum_j\left(\frac{z_j-g_j(\eta)}{\sigma_T}\right)^2+C.$$
+
+The posterior is computed on a dense one-dimensional grid. A shape-preserving interpolator makes
+that grid inexpensive, but every response surface is generated by NeqSim and independently checked
+at off-grid efficiencies before it is trusted.
+"""))
+
+cells.append(md(r"""
+## 2. Clean Colab setup
+
+The released `neqsim` wheel is pinned as the Python/JPype bridge. Advanced calculations use a Java
+JAR built from the selected `equinor/neqsim` source ref. A validation runner may supply an exact
+checkout and JAR with `NEQSIM_SOURCE_ROOT` and `NEQSIM_SOURCE_JAR`; otherwise this notebook clones
+and builds current `master` itself.
+"""))
+
+cells.append(code(r"""
+import importlib.metadata
+import importlib.util
+import os
+from pathlib import Path
+import subprocess
+import sys
+
+REQUIRED_PACKAGES = {
+ "neqsim": "3.18.0",
+}
+
+install_requirements = []
+for package_name, required_version in REQUIRED_PACKAGES.items():
+ try:
+ installed_version = importlib.metadata.version(package_name)
+ except importlib.metadata.PackageNotFoundError:
+ installed_version = None
+ if installed_version != required_version:
+ install_requirements.append(f"{package_name}=={required_version}")
+
+for module_name in ["matplotlib", "numpy", "pandas", "scipy"]:
+ if importlib.util.find_spec(module_name) is None:
+ install_requirements.append(module_name)
+
+if install_requirements:
+ subprocess.run(
+ [
+ sys.executable,
+ "-m",
+ "pip",
+ "install",
+ "--quiet",
+ *install_requirements,
+ ],
+ check=True,
+ timeout=1200,
+ )
+
+print("Required Python packages are available.")
+"""))
+
+cells.append(code(r"""
+import hashlib
+import jpype
+
+
+def run_command(command, *, cwd=None, timeout=1800, environment=None):
+ result = subprocess.run(
+ command,
+ cwd=cwd,
+ env=environment,
+ text=True,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ timeout=timeout,
+ )
+ if result.returncode != 0:
+ output_tail = "\n".join(result.stdout.splitlines()[-80:])
+ raise RuntimeError(
+ f"Command failed ({result.returncode}): {command}\n{output_tail}"
+ )
+ return result.stdout.strip()
+
+
+NEQSIM_SOURCE_REF = os.environ.get("NEQSIM_SOURCE_REF", "master")
+supplied_source_root = os.environ.get("NEQSIM_SOURCE_ROOT", "").strip()
+supplied_source_jar = os.environ.get("NEQSIM_SOURCE_JAR", "").strip()
+
+if supplied_source_root and supplied_source_jar:
+ neqsim_source_root = Path(supplied_source_root).resolve()
+ neqsim_source_jar = Path(supplied_source_jar).resolve()
+else:
+ runtime_root = Path("/content")
+ if not runtime_root.exists():
+ runtime_root = Path(os.environ.get("RUNNER_TEMP", "/tmp")).resolve()
+ neqsim_source_root = runtime_root / "neqsim-java-master"
+ if not neqsim_source_root.exists():
+ run_command(
+ [
+ "git",
+ "clone",
+ "--depth",
+ "1",
+ "--branch",
+ NEQSIM_SOURCE_REF,
+ "https://github.com/equinor/neqsim.git",
+ str(neqsim_source_root),
+ ],
+ timeout=600,
+ )
+ else:
+ run_command(
+ ["git", "fetch", "--depth", "1", "origin", NEQSIM_SOURCE_REF],
+ cwd=neqsim_source_root,
+ timeout=600,
+ )
+ run_command(
+ ["git", "checkout", "--detach", "FETCH_HEAD"],
+ cwd=neqsim_source_root,
+ )
+
+ maven_settings = runtime_root / "neqsim-maven-settings.xml"
+ maven_settings.write_text(
+ "