Skip to content
Developers
Get an API token

Measure a model's accuracy on held-out wells

Accuracy of the pre-refit train fit on wells it never saw (split=holdout, the default): its stored prediction against each well's actual cumulative at each ip_day.

Measure a model's accuracy on held-out wells

GET/models/{model_num}/evaluation

Accuracy of the pre-refit train fit on wells it never saw (split=holdout, the default): its stored prediction against each well's actual cumulative at each ip_day.

Send a Novi API token as a bearer token on every request. Authentication
The fit measured

The all fit that serves forecasts was refit on these wells, so the figures describe the model before that refit. split=train measures it on its own training wells: in-sample, so the error reads better; label it so. On the sample stack, sample numbers.

Metrics

Each row covers the wells that reached the ip_day with an actual above 0 and a stored prediction (wells); under min_wells (5) the metrics are null. Long horizons rest on the earliest holdout wells.

  • mdape: median |predicted − actual| / actual. median_ratio: median predicted / actual.
  • within_10, within_25: the share with |predicted / actual − 1| at most 10% or 25%. bias: median predicted / actual − 1.
  • correlation: Pearson's r of predicted and actual.
Options

group_by splits by formation or first-production year; ip_day keeps one row, and with it include=wells adds that row's wells with their predicted and actual cumulative and APE. near (with radius_mi) or bbox keeps the split's wells in that area, by lateral midpoint, and scope echoes it with its wells' count; an area without any answers groups: [] with scope.wells 0.

Errors

503 when this dataset has no stored predictions (GET /models/{model_num} evaluation.reason says why).

Parameters

Path parameters

Query parameters

  • splitstringOptional

    Default "holdout"

    holdout: wells the train fit never saw; train: its own training wells (in-sample).

    "holdout""train"
  • group_bystringOptional

    Default "none"

    none: one group of every well; formation or first_year: one group each, formations in GET /formations order, years oldest first.

    "none""formation""first_year"
  • ip_dayintegerOptional

    ≥ 1

    Only this ip_day's row in each group: one of the model's ip_days (400 otherwise, with them in details.allowed). Without it, every ip_day.

  • includestringOptional

    wells adds the wells behind that ip_day's metrics, each with its predicted and actual cumulative and APE, for a predicted-against-actual scatter or a residual plot. It needs ip_day (400 without it).

    • Exactly the metrics' population: the split's wells with a stored prediction and an actual above 0 at that ip_day, by formation (name order, none last) then key; with group_by, each with its group.
    • At most 10,000 wells: a larger population is a 400 with details.max and details.count, never a cut list. The sample's oil model at 360 days has 4,353 holdout wells, about 540 KB (73 KB gzipped).
    "wells"
  • nearstringOptional

    Only the split's wells within radius_mi of this point: lng,lat in degrees, matched as GET /wells near matches wells, by great-circle miles to each lateral's midpoint. Every row and include=wells cover those wells only, and scope echoes the area with its wells. A small area has few wells, so its cells may have null metrics (under min_wells). Send near or bbox, not both (400).

  • radius_minumberOptional

    Default 5> 0, ≤ 100

    With near (400 without it): how far from it, in miles, a decimal greater than 0 and at most 100 written as digits (2 or 2.5); default 5.

  • bboxstringOptional

    Only the split's wells whose lateral midpoint is inside west,south,east,north (degrees), as GET /wells bbox filters wells. Every row and include=wells cover those wells only, and scope echoes the box with its wells. Send near or bbox, not both (400).

  • model_setstringOptional

    pattern ^[a-z0-9][a-z0-9-]{0,62}$

    The model set to answer from, an id from GET /model-sets. Absent: the set GET /model-sets names as default, the base dataset unless the server answers from a published set.

    • model_set=default reaches the base dataset; once the server retires it, a 400 naming the set answering instead.
    • An unknown id, or a set scoped to other tenants, is a 400 listing the ids the caller may use in details.allowed (published sets only once default is retired); a set that can't load, a 503 with the reason.
    • Plans and exports stay in their set: a call naming them (by path, plan, ids, source.plan, a scenario's plan.id, or a comparison's a.plan and b.plan, also under source.comparison) reaches it without model_set.
    • An explicit model_set wins; naming resources of two sets in one request is a 400.

Responses

200 OK

The evaluation.

  • model_numstringRequired

    Model number.

  • sample_databooleanRequired

    True when the figures come from synthetic sample data.

  • fitstringRequired

    The fit measured: the pre-refit train fit.

    "train"
  • splitstringRequired

    The wells measured.

    "holdout""train"
  • rulestringRequired

    How holdout wells were chosen, in words.

  • as_ofstring or nullRequired

    Format date

    The production date actuals run to.

  • min_wellsintegerRequired

    ≥ 1

    Fewest wells for a cell's metrics.

  • group_bystringRequired

    How groups split the wells.

    "none""formation""first_year"
  • groupsarray of objectsRequired

    One group, or one per formation or year; none ([]) when near or bbox holds none of the split's wells (scope.wells 0).

    ShowHide 2 child fields in each item
    • groupstring or integer or nullRequired

      The formation or first-production year; null with group_by=none, and for wells without one (listed last).

    • rowsarray of objectsRequired

      By ip_day, ascending.

      ShowHide 8 child fields in each item
      • ip_dayintegerRequired

        Days since first production.

      • wellsintegerRequired

        ≥ 0

        Wells that had reached it with an actual above 0 and a stored prediction: the wells every metric is over (a blank prediction is left out).

      • mdapenumber or nullRequired

        Median of |predicted − actual| / actual, over wells with actual > 0.

      • median_rationumber or nullRequired

        Median of predicted / actual.

      • within_10number or nullRequired

        Share of wells with |predicted / actual − 1| at most 0.10.

      • within_25number or nullRequired

        Share of wells with |predicted / actual − 1| at most 0.25.

      • biasnumber or nullRequired

        Median of predicted / actual − 1: above 0 over-predicts.

      • correlationnumber or nullRequired

        Pearson's r of predicted and actual.

  • scopeobjectOptional

    With near or bbox: the applied area and its count of split wells with a stored prediction at any ip_day.

    ShowHide 2 variants

    One of

    • A radius around a point object

      A radius around a point.

      • neararray of numbersRequired

        2 items

        Center [lng, lat].

      • radius_minumberRequired

        > 0, ≤ 100

        The radius applied, in miles: radius_mi, or 5 when it wasn't sent.

      • wellsintegerRequired

        ≥ 0

        Distinct wells with a stored prediction in the area at any ip_day.

    • A bounding box object

      A bounding box.

      • bboxarray of numbersRequired

        4 items

        [west, south, east, north].

      • wellsintegerRequired

        ≥ 0

        Distinct wells with a stored prediction in the area at any ip_day.

  • wellsarray of objectsOptional

    With include=wells: each well behind the ip_day's metrics, by formation then key.

    ShowHide 7 child fields in each item
    • wellbore_keystringRequired

      Wellbore key.

    • formationstring or nullRequired

      Formation.

    • first_yearinteger or nullRequired

      First-production year.

    • predictednumberRequired

      The train fit's stored prediction of the cumulative at the ip_day, in the stream's unit, to 0.1.

    • actualnumberRequired

      The well's actual cumulative there, above 0, to 0.1.

    • apenumberRequired

      ≥ 0

      |predicted − actual| / actual, before rounding, to 4 decimals.

    • groupstring or integer or nullOptional

      With group_by: the well's group, as in groups (its formation or first-production year).

Conditional requests 304 Unchanged since the ETag in If-None-Match.

Errors

StatusWhen
404

No such well, grid, plan or route.

Shared errors400401403500503What they mean