Compute C-index and Brier score

Performs model assessment by computing the C-index and Brier score at time x_hor. There is the option to calculate their standard errors using bootstraping.

get_model_assessment(
  data,
  individual_id,
  event_prediction,
  event_status,
  event_time,
  x_hor,
  b
)

Arguments

data	Data frame containing survival outcomes and the event predictions from the model, there should be one row for each individual
individual_id	Character string specifying the column name in data which contains the individual identifiers
event_prediction	Character string specifying the column name in data containing the predicted probability of the event of interest
event_status	Character string specifying the column name in data which contains the event status (where 0=censoring, 1=event of interest, if there are competing events these are labelled 2 or above). Events at time x_hor should be labelled censored.
event_time	Character string specifying the column name in data which contains the event time.
x_hor	Numeric specifying the horizon time(s)
b	Integer specifying the number of bootstrap samples to take when calculating standard error of c-index and Brier score

Value

List containing C-index, Brier score and their standard errors

Details

There are two factors in assessing the performance of a prediction model; its discrimination and its calibration. The c-index is a commonly used metric which assesses discrimination, this refers to the ability of the model to separate individuals into those that will have an event and those that will not. The c-index at a horizon time x_hor looks at the pairs of individuals where one individual has the event at a time T and the other has not had the event at time T. It is calculated as the proportion of these pairs where their relative risk prediction agrees with the actual outcomes for the two individuals. This is extended to the competing risks case by comparing individuals where one had the event of interest at time T and the other individual either did not experience the event before this time T or experienced a competing event.

The Brier score gives an indication of the calibration of a model (and its discrimination to an extent), this refers to the agreement between the risk prediction and the outcome. The Brier score is calculated as the average mean squared error of the predicted risk and the event outcome (where an event is 1 and not experiencing the event is 0). This is extended to the competing risks case by including the competing risk events as not experiencing the event.

For both the c-index and Brier score calculations, inverse probability censoring weighting (IPCW) is used to create weights which account for the occurrence of censoring. The censoring model assumes for this function is the Kaplan Meier model, i.e. censoring occurs independently of covariates.

The c-index is calculated using the cindex function in package pec. The Brier score is calculated using pec function in package pec.

Author

Isobel Barrott isobel.barrott@gmail.com

Examples

if (FALSE) {
library(Landmarking)
data(data_repeat_outcomes)
data_model_landmark_LOCF <-
  fit_LOCF_landmark(
    data_long = data_repeat_outcomes,
    x_L = c(60, 61),
    x_hor = c(65, 66),
    covariates =
      c("ethnicity", "smoking", "diabetes", "sbp_stnd", "tchdl_stnd"),
    covariates_time =
      c(rep("response_time_sbp_stnd", 4), "response_time_tchdl_stnd"),
    k = 10,
    individual_id = "id",
    event_time = "event_time",
    event_status = "event_status",
    survival_submodel = "cause_specific"
  )
get_model_assessment(data = data_model_landmark_LOCF[["60"]]$data,
  individual_id = "id",
  event_prediction = "event_prediction",
  event_status = "event_status",
  event_time = "event_time",
  x_hor = 65,
  b = 100)}