Dataset#

Datasets in Pharmpy are represented using the pd.DataFrame class and a separate pharmpy.model.DataInfo class that provides additional information about the dataset. This could contain for example a description of how the columns are used in the model or the units used for the data.

Retrieving the dataset from a model#

The dataset connected to a model can be retrieved from the dataset attribute.

from pharmpy.modeling import read_model

model = read_model(path / "pheno_real.mod")
df = model.dataset
df

model <- read_model(path / "pheno_real$mod")
df <- model$dataset
df

	ID	TIME	AMT	WGT	APGR	DV	FA1	FA2
1	1	0.0	25.0	1.4	7.0	0.0	1.0	1.0
2	1	2.0	0.0	1.4	7.0	17.3	0.0	0.0
3	1	12.5	3.5	1.4	7.0	0.0	1.0	1.0
4	1	24.5	3.5	1.4	7.0	0.0	1.0	1.0
5	1	37.0	3.5	1.4	7.0	0.0	1.0	1.0
...	...	...	...	...	...	...	...	...
740	59	108.3	3.0	1.1	6.0	0.0	1.0	1.0
741	59	120.5	3.0	1.1	6.0	0.0	1.0	1.0
742	59	132.3	3.0	1.1	6.0	0.0	1.0	1.0
743	59	144.8	3.0	1.1	6.0	0.0	1.0	1.0
744	59	146.8	0.0	1.1	6.0	40.2	0.0	0.0

744 rows × 8 columns

This is the dataset after applying any model specific filtering and handling of special values.

The raw dataset can also be accessed

raw = model.read_raw_dataset()
raw

raw <- model$read_raw_dataset()
raw

	ID	TIME	AMT	WGT	APGR	DV	FA1	FA2
1	1	0.	25.0	1.4	7	0	1	1
2	1	2.0	0	1.4	7	17.3	0	0
3	1	12.5	3.5	1.4	7	0	1	1
4	1	24.5	3.5	1.4	7	0	1	1
5	1	37.0	3.5	1.4	7	0	1	1
...	...	...	...	...	...	...	...	...
740	59	108.3	3.0	1.1	6	0	1	1
741	59	120.5	3.0	1.1	6	0	1	1
742	59	132.3	3.0	1.1	6	0	1	1
743	59	144.8	3.0	1.1	6	0	1	1
744	59	146.8	0	1.1	6	40.2	0	0

744 rows × 8 columns

Note that all values here are strings

raw.dtypes

raw$dtypes

ID      object
TIME    object
AMT     object
WGT     object
APGR    object
DV      object
FA1     object
FA2     object
dtype: object

Update the dataset of a model#

Since the Pharmpy dataset is a pandas dataframe, it can be manipulated as such. A new or updated dataset can be set to a model like this:

import numpy as np

df['DV'] = np.log(df['DV'], where=(df['DV'] != 0).values)
model = model.replace(dataset=df)

df['DV'] <- np$log(df['DV'], where=(df['DV'] != 0).values)
model <- model$replace(dataset=df)

The pharmpy.modeling module has several functions to examine and modify the dataset, see the user guide for dataset modeling.

DataInfo#

Every model has a DataInfo object that describes the dataset.

Note

A datainfo file can be created for .csv-files here.

di = model.datainfo
di

di <- model$datainfo
di

name variable      type   scale  count  drop datatype                                           properties
  ID       ID        id nominal  False False    int32                 {'descriptor': 'subject identifier'}
TIME     TIME       idv   ratio  False False  float64                                          {'unit': h}
 AMT      AMT      dose   ratio  False False  float64                                         {'unit': mg}
 WGT      WGT covariate   ratio  False False  float64            {'unit': kg, 'descriptor': 'body weight'}
APGR     APGR covariate ordinal  False False  float64   {'categories': (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10)}
  DV       DV        dv   ratio  False False  float64 {'unit': mg/L, 'descriptor': 'plasma concentration'}
 FA1      FA1   unknown nominal  False False  float64                                                   {}
 FA2      FA2   unknown nominal  False False  float64                                                   {}

The path to the dataset file if one exists.

di.path

di$path

The path to the dataset is None if the original read in dataset is no longer possible to use, e.g. if a column has been added or changed.

Separator character for the dataset file.

di.separator

di$separator

'\\s+'

The provenance describes which operations have been made to a dataset. In other words: which steps are needed to go from the read in dataset to the current dataset.

di.provenance

di$provenance

Provenance(ReadDataset(path=/home/runner/work/pharmpy/pharmpy/pharmpy/tests/testdata/nonmem/pheno.dta))

ColumnInfo#

Each column of the dataset can here be given some additional information.

model.datainfo['AMT']

model$datainfo['AMT']

drop             False
datatype       float64
variable_id       None
variables          AMT
Name: AMT

type#

Column type is the role a data column has in the model. Some basic examples of types are id for the subject identification column, idv for the independent variable (mostly time), dv for the dependent variable and dose for the dose amount column. Columns that not have been given any particular type will get the type value unknown. See pharmpy.ColumnInfo.type for a list of all supported types.

scale#

The scale of a column is the statistical scale of measurement of its data using “Stevens’ typology” (see https://en.wikipedia.org/wiki/Level_of_measurement). The scale can be one of nominal for non-ordered categorical data, ordinal for ordered categorical data, interval for numeric data were ratios cannot be taken and ratio for general numeric data. Note that nominal and ordinal data is always discrete, but interval and ratio data can be both discrete and continuous.

continuous#

If this is True the data is continuous and if it is False it is discrete. Note that ratio data can be seen as discrete for example if it has been rounded to whole numbers and cannot take on any real number.

categories#

A list of all values that the data column could have. Not all values have to be present in the dataset. Instead categories creates a possibility to annotate all possible values. It is also possible to name the categories by using a dict from the name to its numerical encoding.

unit#

The physical unit of the column data. Units can be input as a string, e.g. “kg” or “mg/L.”

drop#

A boolean that is set to True if the column is not going to be used by the model or False otherwise.

datatype#

The datatype of the column data. This describes the low level encoding of the data. See pharmpy.ColumnInfo.datatype for a list of all supported datatypes.

descriptor#

The descriptor can provide a high level understanding of the data in a machine readable way. See pharmpy.ColumnInfo.descriptor for a list of all supported descriptors.

datainfo file#

If a dataset file has an accompanying file with the same name and the extension .datainfo this will be read in when handling the dataset in Pharmpy. This file is a representation (a serialization) of a DataInfo object and its content can be created manually, with an external tool or by Pharmpy. Here is an example of the content:

di.to_json()

di$to_json()

'{"columns": [{"name": "ID", "drop": false, "datatype": "int32", "variable_id": null, "variable_mapping": {"name": "ID", "type": "id", "scale": "nominal", "count": false, "properties": {"descriptor": "subject identifier"}}}, {"name": "TIME", "drop": false, "datatype": "float64", "variable_id": null, "variable_mapping": {"name": "TIME", "type": "idv", "scale": "ratio", "count": false, "properties": {"unit": "h"}}}, {"name": "AMT", "drop": false, "datatype": "float64", "variable_id": null, "variable_mapping": {"name": "AMT", "type": "dose", "scale": "ratio", "count": false, "properties": {"unit": "mg"}}}, {"name": "WGT", "drop": false, "datatype": "float64", "variable_id": null, "variable_mapping": {"name": "WGT", "type": "covariate", "scale": "ratio", "count": false, "properties": {"unit": "kg", "descriptor": "body weight"}}}, {"name": "APGR", "drop": false, "datatype": "float64", "variable_id": null, "variable_mapping": {"name": "APGR", "type": "covariate", "scale": "ordinal", "count": false, "properties": {"categories": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]}}}, {"name": "DV", "drop": false, "datatype": "float64", "variable_id": null, "variable_mapping": {"name": "DV", "type": "dv", "scale": "ratio", "count": false, "properties": {"unit": "mg/L", "descriptor": "plasma concentration"}}}, {"name": "FA1", "drop": false, "datatype": "float64", "variable_id": null, "variable_mapping": {"name": "FA1", "type": "unknown", "scale": "nominal", "count": false, "properties": {}}}, {"name": "FA2", "drop": false, "datatype": "float64", "variable_id": null, "variable_mapping": {"name": "FA2", "type": "unknown", "scale": "nominal", "count": false, "properties": {}}}], "path": null, "separator": "\\\\s+", "missing_data_token": "-99", "provenance": {"operations": [{"class": "ReadDataset", "path": "/home/runner/work/pharmpy/pharmpy/pharmpy/tests/testdata/nonmem/pheno.dta"}]}, "__version__": 1}'

It is a json file with the following top level structure:

Name	Type
`columns`	array of columns
`path`	string
`separator`	string

And the columns structure:

Name	Type
`type`	string
`scale`	string
`continuous`	boolean
`categories`	array of numbers or string-number map
`unit`	string
`drop`	boolean
`datatype`	string
`descriptor`	string

NONMEM dataset#

Since Pharmpy 2.2.0, Pharmpy supports using the original dataset if possible when estimating etc.

Parsing#

Pharmpy does the following filter when parsing a NONMEM dataset:

Filter individuals without observations
Filter all expression specified by IGNORE/ACCEPT statements in $DATA

These filters will be added in DataInfo.provenance.

As an example, if we have the following model which filters out records with DVID == 2:

$PROBLEM PHENOBARB SIMPLE MODEL
$DATA pheno_pd.csv IGNORE=@ IGNORE=(DVID.NEN.2)
$INPUT ID TIME AMT WGT APGR DV DVID
$SUBROUTINE ADVAN1 TRANS2

$PK
CL = THETA(1)*EXP(ETA(1))
VC = THETA(2)*EXP(ETA(2))
V=VC

$ERROR
CONC = A(1)/VC
Y = CONC + CONC*EPS(1)

$THETA (0,0.00469307) ; TVCL
$THETA (0,1.00916) ; TVV
$OMEGA 0.0309626  ; IVCL
$OMEGA 0.031128  ; IVV
$SIGMA 0.013241

$ESTIMATION METHOD=1 INTERACTION NOHABORT
$TABLE ID TIME DV CIPREDI CWRES FILE=pheno_pd_mytab NOAPPEND NOPRINT

We can examine the provenance of that dataset:

model.datainfo.provenance

model$datainfo$provenance

Provenance(ReadDataset(path=/home/runner/work/pharmpy/pharmpy/pharmpy/tests/testdata/nonmem/pheno_pd.csv), Ignore(Ne(DVID, 2)))

Here we can see the reading operation and then a filtering operation.

Updating#

If a change is made to the dataset which allows us to use the original dataset (e.g. filtering rows or removing columns), DROPs and IGNOREs will be added to the NONMEM code as needed. As an example, if we filter out some more records from the previous model:

from pharmpy.modeling import filter_dataset
model = filter_dataset(model, 'APGR >= 8')

model <- filter_dataset(model, 'APGR >= 8')

The provenance will add an ignore operation:

model.datainfo.provenance

model$datainfo$provenance

Provenance(ReadDataset(path=/home/runner/work/pharmpy/pharmpy/pharmpy/tests/testdata/nonmem/pheno_pd.csv), Ignore(Ne(DVID, 2)), Ignore(APGR < 8))

And $DATA and $INPUT will be updated accordingly:

print_model_code(model)

print_model_code(model)

$PROBLEM PHENOBARB SIMPLE MODEL
$DATA pheno_pd.csv IGNORE=@ IGNORE=(DVID.NEN.2) IGNORE=(APGR.LT.8)
$INPUT ID TIME AMT WGT APGR DV DVID
$SUBROUTINE ADVAN1 TRANS2

$PK
CL = THETA(1)*EXP(ETA(1))
VC = THETA(2)*EXP(ETA(2))
V=VC

$ERROR
CONC = A(1)/VC
Y = CONC + CONC*EPS(1)

$THETA (0,0.00469307) ; TVCL
$THETA (0,1.00916) ; TVV
$OMEGA 0.0309626  ; IVCL
$OMEGA 0.031128  ; IVV
$SIGMA 0.013241

$ESTIMATION METHOD=1 INTERACTION NOHABORT
$TABLE ID TIME DV CIPREDI CWRES FILE=pheno_pd_mytab NOAPPEND NOPRINT

In some cases, the original dataset can no longer be used. For example if a new column is added:

from pharmpy.modeling import add_time_after_dose
model = add_time_after_dose(model)

model <- add_time_after_dose(model)

The provenance will still have a new add column operation:

model.datainfo.provenance

model$datainfo$provenance

Provenance(ReadDataset(path=/home/runner/work/pharmpy/pharmpy/pharmpy/tests/testdata/nonmem/pheno_pd.csv), Ignore(Ne(DVID, 2)), Ignore(APGR < 8), AddColumn(TAD))

But the NONMEM code will now have a dummy path as the path and the IGNORE statements will be removed (since the Pharmpy dataset has removed those rows):

print_model_code(model)

print_model_code(model)

$PROBLEM PHENOBARB SIMPLE MODEL
$DATA DUMMYPATH IGNORE=@  
$INPUT ID TIME AMT WGT APGR DV DVID TAD
$SUBROUTINE ADVAN1 TRANS2

$PK
CL = THETA(1)*EXP(ETA(1))
VC = THETA(2)*EXP(ETA(2))
V=VC

$ERROR
CONC = A(1)/VC
Y = CONC + CONC*EPS(1)

$THETA (0,0.00469307) ; TVCL
$THETA (0,1.00916) ; TVV
$OMEGA 0.0309626  ; IVCL
$OMEGA 0.031128  ; IVV
$SIGMA 0.013241

$ESTIMATION METHOD=1 INTERACTION NOHABORT
$TABLE ID TIME DV CIPREDI CWRES FILE=pheno_pd_mytab NOAPPEND NOPRINT

If writing the model (e.g. when writing the files or estimating the model), DUMMYPATH will be replaced with a real path.