Measure a model's accuracy on held-out wells
Accuracy of the pre-refit train fit on wells it never saw (split=holdout, the default): its stored prediction against each well's actual cumulative at each ip_day.
Measure a model's accuracy on held-out wells
/models/{model_num}/evaluationAccuracy of the pre-refit train fit on wells it never saw (split=holdout, the default): its stored prediction against each well's actual cumulative at each ip_day.
The fit measured
The all fit that serves forecasts was refit on these wells, so the figures describe the model before that refit. split=train measures it on its own training wells: in-sample, so the error reads better; label it so. On the sample stack, sample numbers.
Metrics
Each row covers the wells that reached the ip_day with an actual above 0 and a stored prediction (wells); under min_wells (5) the metrics are null. Long horizons rest on the earliest holdout wells.
mdape: median |predicted − actual| / actual.median_ratio: median predicted / actual.within_10,within_25: the share with |predicted / actual − 1| at most 10% or 25%.bias: median predicted / actual − 1.correlation: Pearson's r of predicted and actual.
Options
group_by splits by formation or first-production year; ip_day keeps one row, and with it include=wells adds that row's wells with their predicted and actual cumulative and APE. near (with radius_mi) or bbox keeps the split's wells in that area, by lateral midpoint, and scope echoes it with its wells' count; an area without any answers groups: [] with scope.wells 0.
Errors
503 when this dataset has no stored predictions (GET /models/{model_num} evaluation.reason says why).
Parameters
Path parameters
model_numstringRequiredModel number, from GET /models or GET /dataset
models[].model_num.
Query parameters
splitstringOptionalDefault
"holdout"holdout: wells the train fit never saw; train: its own training wells (in-sample).
"holdout""train"group_bystringOptionalDefault
"none"none: one group of every well; formation or first_year: one group each, formations in GET /formations order, years oldest first.
"none""formation""first_year"ip_dayintegerOptional≥ 1
Only this ip_day's row in each group: one of the model's ip_days (400 otherwise, with them in
details.allowed). Without it, every ip_day.includestringOptionalwellsadds the wells behind that ip_day's metrics, each with its predicted and actual cumulative and APE, for a predicted-against-actual scatter or a residual plot. It needsip_day(400 without it).- Exactly the metrics' population: the split's wells with a stored prediction and an actual above 0 at that ip_day, by formation (name order, none last) then key; with
group_by, each with itsgroup. - At most 10,000 wells: a larger population is a 400 with
details.maxanddetails.count, never a cut list. The sample's oil model at 360 days has 4,353 holdout wells, about 540 KB (73 KB gzipped).
"wells"- Exactly the metrics' population: the split's wells with a stored prediction and an actual above 0 at that ip_day, by formation (name order, none last) then key; with
nearstringOptionalOnly the split's wells within
radius_miof this point:lng,latin degrees, matched as GET /wellsnearmatches wells, by great-circle miles to each lateral's midpoint. Every row andinclude=wellscover those wells only, andscopeechoes the area with itswells. A small area has few wells, so its cells may have null metrics (undermin_wells). Sendnearorbbox, not both (400).radius_minumberOptionalDefault
5> 0, ≤ 100With
near(400 without it): how far from it, in miles, a decimal greater than 0 and at most 100 written as digits (2 or 2.5); default 5.bboxstringOptionalOnly the split's wells whose lateral midpoint is inside
west,south,east,north(degrees), as GET /wellsbboxfilters wells. Every row andinclude=wellscover those wells only, andscopeechoes the box with itswells. Sendnearorbbox, not both (400).model_setstringOptionalpattern ^[a-z0-9][a-z0-9-]{0,62}$
The model set to answer from, an
idfrom GET /model-sets. Absent: the set GET /model-sets names asdefault, the base dataset unless the server answers from a published set.model_set=defaultreaches the base dataset; once the server retires it, a 400 naming the set answering instead.- An unknown id, or a set scoped to other tenants, is a 400 listing the ids the caller may use in
details.allowed(published sets only oncedefaultis retired); a set that can't load, a 503 with the reason. - Plans and exports stay in their set: a call naming them (by path,
plan,ids,source.plan, a scenario'splan.id, or a comparison'sa.planandb.plan, also undersource.comparison) reaches it withoutmodel_set. - An explicit
model_setwins; naming resources of two sets in one request is a 400.
Responses
200 OK
The evaluation.
model_numstringRequiredModel number.
sample_databooleanRequiredTrue when the figures come from synthetic sample data.
fitstringRequiredThe fit measured: the pre-refit train fit.
"train"splitstringRequiredThe wells measured.
"holdout""train"rulestringRequiredHow holdout wells were chosen, in words.
as_ofstring or nullRequiredFormat
dateThe production date actuals run to.
min_wellsintegerRequired≥ 1
Fewest wells for a cell's metrics.
group_bystringRequiredHow groups split the wells.
"none""formation""first_year"groupsarray of objectsRequiredOne group, or one per formation or year; none (
[]) when near or bbox holds none of the split's wells (scope.wells0).ShowHide 2 child fields in each item
groupstring or integer or nullRequiredThe formation or first-production year; null with group_by=none, and for wells without one (listed last).
rowsarray of objectsRequiredBy ip_day, ascending.
ShowHide 8 child fields in each item
ip_dayintegerRequiredDays since first production.
wellsintegerRequired≥ 0
Wells that had reached it with an actual above 0 and a stored prediction: the wells every metric is over (a blank prediction is left out).
mdapenumber or nullRequiredMedian of |predicted − actual| / actual, over wells with actual > 0.
median_rationumber or nullRequiredMedian of predicted / actual.
within_10number or nullRequiredShare of wells with |predicted / actual − 1| at most 0.10.
within_25number or nullRequiredShare of wells with |predicted / actual − 1| at most 0.25.
biasnumber or nullRequiredMedian of predicted / actual − 1: above 0 over-predicts.
correlationnumber or nullRequiredPearson's r of predicted and actual.
scopeobjectOptionalWith near or bbox: the applied area and its count of split wells with a stored prediction at any ip_day.
ShowHide 2 variants
One of
A radius around a point object
A radius around a point.
neararray of numbersRequired2 items
Center [lng, lat].
radius_minumberRequired> 0, ≤ 100
The radius applied, in miles: radius_mi, or 5 when it wasn't sent.
wellsintegerRequired≥ 0
Distinct wells with a stored prediction in the area at any ip_day.
A bounding box object
A bounding box.
bboxarray of numbersRequired4 items
[west, south, east, north].
wellsintegerRequired≥ 0
Distinct wells with a stored prediction in the area at any ip_day.
wellsarray of objectsOptionalWith include=wells: each well behind the ip_day's metrics, by formation then key.
ShowHide 7 child fields in each item
wellbore_keystringRequiredWellbore key.
formationstring or nullRequiredFormation.
first_yearinteger or nullRequiredFirst-production year.
predictednumberRequiredThe train fit's stored prediction of the cumulative at the ip_day, in the stream's unit, to 0.1.
actualnumberRequiredThe well's actual cumulative there, above 0, to 0.1.
apenumberRequired≥ 0
|predicted − actual| / actual, before rounding, to 4 decimals.
groupstring or integer or nullOptionalWith group_by: the well's group, as in
groups(its formation or first-production year).
304 Unchanged since the ETag in If-None-Match.Errors
| Status | When |
|---|---|
| 404 | No such well, grid, plan or route. |
Shared errors400401403500503What they mean