Rigorous evaluation of five models for e-diary-only seizure forecasting: Retrospective and prospective datasets do not outperform the Napkin method.

objectiveSeizure forecasting using e-diaries may help patients with seizures to organize their daily life. Until now, most methods were not rigorously tested against a strict standard. This study aims to assess whether the performance of various models for seizure forecasting using e-diaries is better than the performance of a moving window average (a.k.a. the Napkin method, due to simplicity of calculation).

methodsWe analyzed three cohorts from Seizure Tracker: a retrospective study and two prospective studies. E-diaries and the type of seizures were extracted from the datasets. We implemented five machine learning models (Perceptron, 1D-convolution, Multilayer Perceptron, Cycle, point-process generalized linear model) and compared their performance at seizure forecasting against the Napkin forecast. The models predicted the probability of having at least one seizure in the next 24-h period based on a 90-day historical window. Model performances were evaluated by commonly used metrics (area under the precision-recall curve, area under the receiver operating characteristic curve, and Brier score). We considered a model to be clinically ineffective if it did not outperform the Napkin method across metrics and seizure frequencies.

resultsA total of 5501 retrospective patients (3300 training, 1100 validation, and 1101 testing) and 36 prospective patients (21 from one cohort, 15 from the other) were included in the analysis. No model achieved significantly better performance than the Napkin method across metrics and frequencies. SIGNIFICANCE: Clinically effective seizure forecasting (i.e., beyond the Napkin method) for 24-h risk using e-diaries alone may be infeasible with currently available techniques.