# Partner sync

Source: https://developers.novilabs.com/forecasts/guides/partner-sync/

> Keep another tool in step with your saved plans and Legacy Forecasts, pulling only what changed.

Keep another tool in step with your saved plans and Legacy Forecasts[^legacy]. Each scheduled run lists the plans
changed since the last run and reads their curves. You turn the cumulative points into whatever the tool loads.

## What stays and what changes

| Item | What can change |
| --- | --- |
| A done plan made with `POST /plans` | Only its name and whether it is saved. Its forecasts don't change |
| Curve points past `model_months` | A later release can move them. [Catch moved points](#catch-moved-points) with `ETag` |
| An unsaved plan | Deleted 90 days after it is made. Saved plans stay until someone deletes them |
| A Legacy Forecast | Its numbers, keeping its id. It can't be changed or deleted through the API |
| A deleted plan | Drops out of the lists and can no longer be read. No record is left |

## One sync run

1. **Note the run's start time** for the next run's `updated_from`.

2. **List what changed in each model set** from `GET /model-sets`:

   ```text
   GET /plans?origin=all&kept=true&status=done&updated_from=<last run start minus 15 minutes>&include=storage&limit=100&model_set=<id>
   ```

   - `kept=true`: saved plans and every Legacy Forecast. `status=done` leaves out running plans.
   - `updated_from`: every plan made, finished, renamed, saved, unsaved or shared since then. The 15 minutes catch
     changes your last run's list missed.
   - Every set's list holds every Legacy Forecast, so send `origin=v2` for each set after the first.
   - Page by `next_cursor` until it is `null`, with the same parameters.
   - `include=storage` adds `storage.updated_at`, the last stored change that `updated_from` reads.

3. **Decide what to read.** A renamed plan you hold has the same forecasts. Re-read a Legacy Forecast
   (`origin.engine` `fe`) when its `origin.indexed_at` is newer than your copy.

4. **Read the curves.** For each plan and design, page through `GET /plans/{id}/curves` (up to 200 wells a page). Key
   wells by `well_uid`. `start_date=til` adds volumes by calendar month.

5. **Save the start time.**

```python
import os
from datetime import datetime, timedelta, timezone

import requests

BASE = os.environ["NOVI_API_BASE"]
STATE = "last_sync_start.txt"
session = requests.Session()
session.headers["Authorization"] = f"Bearer {os.environ['NOVI_API_TOKEN']}"

def get(path, **params):
    r = session.get(f"{BASE}{path}", params=params, timeout=120)
    r.raise_for_status()
    return r.json()

def pages(path, params):
    while True:
        page = get(path, **params)
        yield page
        if page["next_cursor"] is None:
            return
        params = {**params, "cursor": page["next_cursor"]}

def changed_plans(since):
    for i, model_set in enumerate(get("/model-sets")["model_sets"]):
        params = {
            "origin": "all" if i == 0 else "v2",
            "kept": "true",
            "status": "done",
            "include": "storage",
            "limit": 100,
            "model_set": model_set["id"],
        }
        if since is not None:
            params["updated_from"] = (since - timedelta(minutes=15)).strftime("%Y-%m-%dT%H:%M:%SZ")
        for page in pages("/plans", params):
            if page.get("warnings"):
                print("warnings:", page["warnings"])
            yield from page["plans"]

def store(plan, design, well):
    print(plan["id"], design, well["well_uid"], sorted(well.get("curves", {})))

try:
    with open(STATE) as f:
        last_sync_start = datetime.fromisoformat(f.read().strip())
except FileNotFoundError:
    last_sync_start = None

sync_start = datetime.now(timezone.utc)
seen = set()
for plan in changed_plans(last_sync_start):
    if plan["id"] in seen:
        continue
    seen.add(plan["id"])
    for design in range(len(plan["designs"])):
        params = {"design": design, "limit": 200, "start_date": "til"}
        for page in pages(f"/plans/{plan['id']}/curves", params):
            for well in page["wells"]:
                store(plan, design, well)

with open(STATE, "w") as f:
    f.write(sync_start.isoformat())
```

Replace `store` with your load step. The first run lists every kept plan.

## Catch moved points

Keep each curve page's `ETag`. To check a plan, read its pages again with the `ETag` as `If-None-Match`: a Not
Modified answer, with no body, means nothing moved.

## Removals

Lists leave out unsaved, deleted and expired plans without saying so. Now and then, send the lists without
`updated_from` and `status`, and drop the ids you hold that they don't list.

> **Pending Legacy Forecasts:**
> A `fe_runs_pending` warning counts Legacy Forecasts not ready to list yet; a later run lists them. A
> `fe_runs_unavailable` warning means Legacy Forecasts could not be listed at all. After a list with either warning,
> drop no Legacy Forecast: the list may leave out one that still exists.

## Files instead of curves

When the tool loads files, [export](https://developers.novilabs.com/forecasts/guides/exports/) with the `combocurve` or `valnav` preset and
`identity.well_id: "uid"`, so Well IDs match the `well_uid` you key on.

## Coming later

| Primitive | Status |
| --- | --- |
| Deletion records, so a sync learns what was removed without a full list | [Planned] |
| Arps segments as JSON, outside an export zip | [Planned] |
| Read-only service tokens for partners, not tied to one person | [Planned] |
| Callbacks to your receiver when a plan finishes | [Planned] |

[^legacy]: Legacy Forecasts are dynamic imports from Forecast Engine v1.
