Back to the Top
The following message was posted to: PharmPK
Hans,
Hans Proost wrote:
> > Another interesting finding was the relative good performance of the
Standard Two-Stage (STS) procedure; in this case of
> > two measurements and two parameters, STS does not require any fitting
procedure, and can be performed in a simple spreadsheet.
The claim in the last sentence is not true. The first stage of the two
stage method has to have some kind of 'fitting procedure' i.e. estimation
of individual parameters. With just two measurements and two parameters a
naive 'fitting procedure' might involve only a simple spreadsheet. More
robust methods would require some kind of Bayesian procedure to account at
the very least for the residual error which cannot be estimated from two
observations with two unknown parameters.
> > STS performed less
> > well than PEM, NPAG and MW\Pharm (although parameter estimates were
still rather good), and markedly better than IT2B. Also, the
efficiency of STS was close to that of these programs. This would
imply that STS performs also better than NONMEM. This finding throws a
new light on the question with respect to the begin (STS, not NONMEM)
and end of population modeling.
Without details of the design of the experiments and the results it is
hard to evaluate your claim that STS performs better than NONMEM. However,
one can show quite easily that the STS method must be inferior to a mixed
effects modelling method that correctly distinguishes between parameter
variability (e.g. between subject) and parameter uncertainty (due to the
design/estimation procedure).
I have no difficulty accepting that NONMEM does not reliably distinguish
these factors but at least it makes an effort to do so -- unlike STS.
There are more sophisticated two stage methods that use the approximate
variance-covariance matrix of the estimate to try to do a better job. Is
that what IT2B does?
Nick
--
Nick Holford, Dept Pharmacology & Clinical Pharmacology
University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New
Zealand email:n.holford.at.auckland.ac.nz tel:+64(9)373-7599x86730
fax:373-7556 http://www.health.auckland.ac.nz/pharmacology/staff/nholford/
Back to the Top
The following message was posted to: PharmPK
Dear Nick,
Thank you for your comments. In reply to my statement:
>> in this case of two measurements and two parameters,
>> STS does not require any fitting procedure, and can
>> be performed in a simple spreadsheet.
you wrote:
> The claim in the last sentence is not true. The first
> stage of the two stage method has to have some kind
> of 'fitting procedure' i.e. estimation of
> individual parameters. With just two measurements
> and two parameters a naive 'fitting procedure' might
> involve only a simple spreadsheet. More
> robust methods would require some kind of Bayesian
> procedure to account at the very least for the
> residual error which cannot be estimated from two
> observations with two unknown parameters.
Yes and no. I fully agree that estimating two parameters
(V and k in this example of a single iv dose) from two
measurements is not the most sophisticated method of data
analysis. But in my opinion this is what the first stage
in 'standard STS' does; fitting the model parameters of
one individual without any further information. Indeed,
residual error cannot be estimated since the 'fitted'
curve passes exactly through the data points. This is
indeed
statistically not really sound. The interesting point was
that this method turns out to be rather good in this case!
Of course, less good than PEM and ITSB, but, looking at
the efficiency results reported by Roger Jelliffe, better
than NONMEM.
> Without details of the design of the experiments and
> the results it is hard to evaluate your claim that
> STS performs better than NONMEM.
I fully agree. Please note that it was not my intention to
claim that STS was better than NONMEM. I observed that
ITSB performed equally well as PEM and NPAG, and that STS
performed less well, although not really bad. The
comparison with NONMEM stems from the efficiency values
reported by Roger Jelliffe only. IMHO, the claims of Roger
Jelliffe are hard to evaluate without details of the
methods and procedures (I should have stated that
explicitly in my previous mail). I trust you agree.
> However, one can show quite easily that the STS method
> must be inferior to a mixed effects modelling method
> that correctly distinguishes between parameter
> variability (e.g. between subject) and parameter
> uncertainty (due to the design/estimation procedure).
I agree. Yes, I also expect that NONMEM would perform
better than STS, for the simple reason that STS is
statistically unsound. However, since STS appears to do
the job not that bad in this example, one should be
cautious with general statements.
> There are more sophisticated two stage methods that use
> the approximate variance-covariance matrix of the
> estimate to try to do a better job. Is that what IT2B
> does?
Yes, this is what the ITSB procedure, as described by
Mentre and Gomeni, and by Bennett and Wakefield, does. As
far as I understand this is also implemented in the IT2B
program of USC*PACK. This method does not perform as well
as my implementation of ITSB in MW\Pharm. Again, this
claim is based on the results reported by Roger Jelliffe
and Bob Leary.
Best regards,
Hans Proost
Johannes H. Proost
Dept. of Pharmacokinetics and Drug Delivery
University of Groningen
Antonius Deusinglaan 1
9713 AV Groningen
tel. ++31 50 363 3292
fax ++31 50 363 3247
e-mail: j.h.proost.aaa.rug.nl
Back to the Top
The following message was posted to: PharmPK
Hans,
"J.H.Proost" wrote:
> Thank you for your comments. In reply to my statement:
>
> >> in this case of two measurements and two parameters,
> >> STS does not require any fitting procedure, and can
> >> be performed in a simple spreadsheet.
>
> you wrote:
>
> > The claim in the last sentence is not true. The first
> > stage of the two stage method has to have some kind
> > of 'fitting procedure' i.e. estimation of
> > individual parameters. With just two measurements
> > and two parameters a naive 'fitting procedure' might
> > involve only a simple spreadsheet. More
> > robust methods would require some kind of Bayesian
> > procedure to account at the very least for the
> > residual error which cannot be estimated from two
> > observations with two unknown parameters.
>
> Yes and no.
My comment was not ambiguous. The two stage procedure is
Stage 1: Estimate parameters.
Stage 2: Calculate average and variance
Stage 1 is a 'fitting procedure'. Full stop. Period. Nothing more to say.
>
> > There are more sophisticated two stage methods that use
> > the approximate variance-covariance matrix of the
> > estimate to try to do a better job. Is that what IT2B
> > does?
>
> Yes, this is what the ITSB procedure, as described by
> Mentre and Gomeni, and by Bennett and Wakefield, does. As
> far as I understand this is also implemented in the IT2B
> program of USC*PACK. This method does not perform as well
> as my implementation of ITSB in MW\Pharm. Again, this
> claim is based on the results reported by Roger Jelliffe
> and Bob Leary.
I think you make a good point that the actual implementation of an
algorithm may differ depending on how it it is coded. But until both you
and Jelliffe/Leary report your claims in a peer reviewed journal with all
the relevant details I have to say that I consider them as a description
of 'vapourware' provided by people with good intentions but conflicts of
interest.
The population analysis software 'shoot out' held in Lyon last year will
be reported at PAGE in Pamplona next month.
http://www.page-meeting.org/default.asp?id=26&keuze=abstract-view&goto=abstracts&orderby=author&abstract_id=834
It will be interesting to see the observations made by 'blinded' Pascal
Girard and France MentrE but until the methodology of the algorithms and
the testing procedures are reviewed by critical peers with open eyes I
don't expect to be making any major changes in the way I try to model data
(I recognize it is hard to make a leopard change its spots or for an old
dog to change its tricks etc).
Nick
--
Nick Holford, Dept Pharmacology & Clinical Pharmacology
University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New
Zealand email:n.holford.aaa.auckland.ac.nz tel:+64(9)373-7599x86730
fax:373-7556 http://www.health.auckland.ac.nz/pharmacology/staff/nholford/
Back to the Top
The following message was posted to: PharmPK
Dear Nick,
I fully agree with your opinion on 'vapourware', a nice term that deserves
more extensive use!
You wrote:
> provided by people with good intentions but conflicts of interest.
IMHO, conflict of interest is not the major problem (but perhaps I am
rather naive). It is my impression that developpers of methods and
software have indeed good intentions, but are not always aware of all
problems that may occur when their methods and software are applied to
data sets different from that used as test examples during development and
testing. In the extreme case, a method or program works only for that
specific example. At the other extreme, a method or program works to any
data set. I don't think that any developper can claim this property. In
real life all programs and methods are somewhere between. Also, one may be
focussed on some specific properties, without considering other important
properties as well. E.g., consistency is a desirable property, but if bias
is small compared to RMSE, it is not a major topic, and no reason to
reject a method. Finally, personal experience and a good knowledge for the
selection of the appropriate settings for a specific method (which may be
crucial), are important also. Therefore the old dog should not change his
tricks (but he should be critical also to his own tricks). Personally I
think that there will be many different approaches that are appropriate in
population analysis of different data sets. STS is definitely not among
the best, and statistically inferior to 'real' population approaches.
Indeed, it will be interesting to see the observations made by 'blinded'
Pascal Girard and France MentrE. This year a new blinded comparison has
been sent out; the abstract at the PAGE-website refers to this new
comparison. I have sent my results obtained with the Iterative Two-Stage
Bayesian approach.
Best regards,
Hans Proost
Johannes H. Proost
Dept. of Pharmacokinetics and Drug Delivery
University Centre for Pharmacy
Antonius Deusinglaan 1
9713 AV Groningen, The Netherlands
tel. 31-50 363 3292
fax 31-50 363 3247
Email: j.h.proost.-at-.rug.nl
Back to the Top
Dear Nick:
You said below to Hans Proost :
"I think you make a good point that the actual
implementation of an
algorithm may differ depending on how it is coded. But until both you
and Jelliffe/Leary report your claims in a peer reviewed journal with
all
the relevant details, I have to say that I consider them as a
description
of 'vapourware' provided by people with good intentions but conflicts of
interest.
The population analysis software 'shoot out' held in Lyon
last year will
be reported at PAGE in Pamplona next month.
http://www.page-meeting.org/default.asp?id=26&keuze=abstract-
view&goto=abstracts&orderby=author&abstract_id=834
It will be interesting to see the observations made by 'blinded' Pascal
Girard and France Mentre but until the methodology of the algorithms and
the testing procedures are reviewed by critical peers with open eyes I
don't expect to be making any major changes in the way I try to model
data
(I recognize it is hard to make a leopard change its spots or for an old
dog to change its tricks etc)."
Now the results of the shootout have been presented at PAGE.
What do you think of them, Nick? In general, methods that use exact or
accurate likelihoods do better than those using approximations to
compute
the likelihood, such as FO or FOCE, whether implemented in IT2B, NONMEM,
or whatever. Those such as PEM, NPEM, NPML of Mallet, and NPAG.
do better because they compute the likelihood exactly or accurately.
Now, thanks for describing us as having good intentions, but
we are
confused about the statement about vaporware" and "conflict of
interest".
Can you clarify these terms for us and make them more explicit? Who
among us has a conflict of interest, and specifically what is it?
If you want to see a paper, a thesis, and a book about the
lack of consistency of NONMEM, try
1. Spieler G and Schumitzky A: Asymptotic Properties of Extended
Least Squares Estimators with Approximate Models, Proceedings of the
Biopharmaceutical Section of the American Statistical Association, 1993.
pages 177-182, Example 4.
2. Also, there is Spieler G: Parameter Estimation by System
Theoretic
Methods, PhD dissertation, Department of Mathematics, University of
Southern California, Los Angeles, 1989.
3. Further, you might look at Vonesh and Chinchilli, Linear and
Nonlinear
Models for the Analysis of Repeated Measurements, Marcel Dekker, New
York,
Basel, 1997, pages 354 - 357.
Alan Schumitzky adds "The relevant example in Vonesh-
Chinchilli ((VC)
is on pages 354-357. (Starts on last paragraph of page 354 and goes
through
page 357.). The model is given by (7.4.62). The true mean and var are
given by
(7.4.63). So a consistent method would converge to these results.
The NONMEM FO mean and var (called "first order PA" in VC)
are given
by (7.4.64). This means that with an infinite amount of data, the FO
method would
converge to the results given by (7.4.64), not (7.4.63). Therefore,
NONMEM FO
is not consistent.
Take a look, Nick. I don't think you are an old dog yet. Any
conflict we
have with you is about the science, not "interest" as far as I know.
Look at these
references. Your problem with "vaporware" perhaps was that you wanted
to use it,
I sent it to you, but at the time you did not want it, as it would
not run in
XP on your machine.
Best regards,
Roger Jelliffe
Roger W. Jelliffe, M.D. Professor of Medicine,
Division of Geriatric Medicine,
Laboratory of Applied Pharmacokinetics,
USC Keck School of Medicine
2250 Alcazar St, Los Angeles CA 90033, USA
Phone (323)442-1300, fax (323)442-1302, email= jelliffe.at.usc.edu
Our web site= http://www.lapk.org
PharmPK Discussion List Archive Index page
Copyright 1995-2010 David W. A. Bourne (david@boomer.org)