Skip to contents

Power analysis by simulation

Source: vignettes/power-analysis.Rmd
power-analysis.Rmd

Rationale

We analysed sample and effect sizes a priori using a simulation approach, where power is defined as the proportion of simulations where a model detects an existing effect that we simulated. Instead of choosing a fixed sample size, this approach allowed us to have a full picture of the power of our model across a range of sample sizes and effect sizes. It replaced the question “What sample size do we need to detect an effect of a given (arbitrary) size?” with “Given the sample we managed to recruit, what is the smallest effect size we could detect with good power?” which allows much more flexibility in data collection. This was especially useful for us because aphantasia and hyperphantasia are rare phenomena (see Wright et al., 2024), and we had no way of knowing beforehand how many participants we would be able to recruit in each group.

Building the generative model

Structure

We designed our model a priori based on our hypotheses and variables. This is the model that will be used as the data-generating process for the simulations. The model we used is a generalised linear mixed-effects model with the following structure, in R syntax:

dependent_variable ~ grouping * category + (category | id) + (grouping | problem)

Where the dependent variable can be accuracy or response times (RTs).

  • The grouping can be our basic VVIQ groups (of varying precision depending on the definition, with 2, 3 or 4 groups), or OSIVQ cognitive style groups as proposed by Delem et al. (2025) (in aphantasia literature) or Gazzo et al. (2013) (in reasoning literature).

  • The category is the problem category (visual, spatial or control).

  • The interaction between grouping and category is a third fixed factor (implied by the *).

  • The random effects are the intercepts and slopes by category for each participant (category | id) and each problem by group (grouping | problem).

The parameters for each variable can be made explicit by writing the model formally as such:

DV=β0+τ0+(βgrouping+τproblem)×grouping+(βcategory+τcategory)×category+(βgrouping:category+τgrouping:category)×grouping×category+ϵ\begin{align*} DV = \ & \beta_0 + \tau_0 + \\ & (\beta_{grouping} + \tau_{problem}) \times grouping \ + \\ & (\beta_{category} + \tau_{category}) \times category \ + \\ & (\beta_{grouping:category} + \tau_{grouping:category}) \times grouping \times category \ + \\ & \epsilon \end{align*}

Where:

  • DVDV is the dependent variable (RTs or accuracy).
  • β0\beta_0 is the global intercept (the mean of the dependent variable).
  • τ0\tau_0 is the random intercept for each participant (the variation of the global intercept per participant).
  • βgrouping\beta_{grouping} is the fixed effect of the grouping variable (the difference in the dependent variable between the groups).
  • τproblem\tau_{problem} is the random effect of the problem (the variation of the grouping effect per problem).
  • βcategory\beta_{category} is the fixed effect of the category variable (the difference in the dependent variable between the categories).
  • τcategory\tau_{category} is the random effect of the category (the variation of the category effect per participant).
  • βgrouping:category\beta_{grouping:category} is the fixed effect of the interaction between the grouping and category variables (the difference in the dependent variable between the groups for each category).
  • τgrouping:category\tau_{grouping:category} is the random effect of the interaction between the grouping and category variables (the variation of the interaction effect per participant).
  • ϵ\epsilon is the residual error (the variation of the dependent variable that is not explained by the model).

There are actually two βcategory\beta_{category}, one for each category (minus one, for the reference level), and as many βgrouping:category\beta_{grouping:category} as there are combinations of categories and groups. In practice, for our purposes, we are only interested in the difference between the visual category and the others, that we called the βvis\beta_{vis} coefficient, and the difference in this difference (the interaction) for a specific group (aphantasia), that we called the βaph−vis\beta_{aph-vis} coefficient. Thus, for our simulations, we set βgrouping\beta_{grouping}, τproblem\tau_{problem}, βspatial\beta_{spatial} (the effect of the spatial category) and τspatial\tau_{spatial} (the associated slope) and all other interaction coefficients to 0, respectively because we had no hypothesis or previous data on the group effect, the difference between the problems, the spatial effect, or the variation in spatial performance.

Given this theoretical model and our assumptions, we tried to choose constant values for the remaining parameters that were not analysed (β0\beta_{0}, τ0\tau_{0} and τvis\tau_{vis}) and ranges of values for the parameters of interest in our power analysis (βvis\beta_{vis} and βaph−vis\beta_{aph-vis}) based on previous literature.

Literature reference for parameter values

We based the power analysis on the Visual Imagery Impedance Effect (VIIE) on RTs, which were the most robust and well-documented we could find. More precisely, we used the very well-described data and model from Tse et al. (2017). They had a study and paradigm very similar to ours, so their statistics were fairly simple to adapt to determine good simulation parameters. The main difference is that they used continuous and discontinuous problems, while we used only semi-continuous problems, so we chose to average the values of the two types of problems from their study.

Their statistics were:

  • Control category mean RT: 14.641 seconds

  • Visual category mean RT: 16.388 seconds

  • Spatial category mean RT: 14.197 seconds

Their Generalized Estimating Equations also showed:

  • An intercept of 12.051 seconds, 95% CI = [10.739, 13.363]

To which we can add the β\beta coefficient of the 4/5-terms (they are the same):

  • β\beta 4T: 2.482 seconds, 95% CI = [1.607, 3.356]

  • Finally, the visual category β\beta: 2.633 seconds, 95% CI = [1.121, 4.146]

Thus, for our simulations, we aimed for the following parameters:

  • A global mean β0\beta_0 around 14.5 seconds.

  • Variations between 12.5 and 16.5 seconds, which we translated as varying intercepts per subject τ0\tau_0 ranging between -2 and 2 seconds.

  • A fixed effect of the visual category βvis\beta_{vis} around 2.6 seconds.

  • Variations in this fixed effect between 1.1 and 4.1 seconds, which we translated as varying slopes per subject τvis\tau_{vis} ranging between -1.5 and 1.5 seconds.

Finding the simulation parameters

We hypothesised that the aphantasia group would not show a visual effect while the typical group would, a pattern that would result in an interaction between the visual category and the aphantasia group. The visual effect in the typical group would result in a positive βvis\beta_{vis} coefficient, while the aforementioned interaction would result in a negative βaph−vis\beta_{aph-vis} interaction coefficient that nullifies the visual effect for the aphantasia group only.

We chose to simulate the response times with a shifted log-normal distribution (generated with the package brms). We searched manually for the parameters of this distribution and our model coefficients that allowed to reach the desired statistics. The data simulation using the generative model is wrapped in the simulate_rt_data() function. The function has arguments to fine-tune all the distributional parameters and model coefficients discussed above. We searched for the appropriate values by trial-and-error using the simple code below:

library(aphantasiaReasoningViie)
#> Welcome to aphantasiaReasoningViie.
df_test <- 
  simulate_rt_data(
    n_subj_per_group = 100, 
    # Parameters of the shifted log-normal distribution (from brms)
    meanlog = 2.1,
    sdlog   = 0.55, 
    shift   = 5,
    # Varying intercept by-subject
    tau_0 = 0.9,
    # Visual category effect
    beta_vis = 2.35,
    # Varying visual effect by-subject
    tau_vis  = 0.75,
    # Aphantasia group x visual interaction
    beta_aph_vis = -2.35
  )

df_test |> 
  dplyr::group_by(category, group) |>  
  dplyr::reframe(
    mean   = mean(rt_total),     
    median = median(rt_total),  
    min    = min(rt_total), # To test the shift     
    max    = max(rt_total)  # To test the sdlog dispersion effect
  ) 
#> # A tibble: 6 × 6
#>   category group       mean median   min   max
#>   <fct>    <fct>      <dbl>  <dbl> <dbl> <dbl>
#> 1 Control  Aphantasia  14.3   13.3  5.53  48.9
#> 2 Control  Typical     14.8   13.5  5.11  60.4
#> 3 Spatial  Aphantasia  14.3   13.2  5.22  41.6
#> 4 Spatial  Typical     14.6   13.3  5.85  50.1
#> 5 Visual   Aphantasia  14.7   13.4  4.42  51.7
#> 6 Visual   Typical     17.1   15.8  8.21  64.6

Using the argument values above, we see that we managed to reach statistics close to the ones we wanted, while simulating an interaction effect between the groups in the visual category. We found that:

  • A mean of 2.1, SD of 0.55 seconds and a shift (non-decision time) of 5 seconds on the shifted log-normal distribution allowed to reach the desired β0\beta_0 of 14.5 seconds reliably (testing with no other source of variation, i.e., all other parameters at 0).

  • A varying intercept τ0\tau_0 with a SD of 0.9 allowed to obtain distributions of the varying RTs ranging between -2 and 2 seconds on average, as expected.

  • A visual effect βvis\beta_{vis} of 2.35 allowed to reproduce the visual category means observed in the study reliably.

  • A varying visual effect τvis\tau_{vis} with a SD of 0.75 allowed to obtain distributions of the varying visual RTs ranging between -1.5 and 1.5 seconds on average, as expected.

  • An interaction parameter βaph−vis\beta_{aph-vis} set as minus the visual effect allowed to nullify the visual effect in the aphantasia group. If the effect size gets small, a slight multiplier (e.g., 1.5) might be necessary to nullify the effect.

#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.5.1 (2025-06-13)
#>  os       Ubuntu 24.04.3 LTS
#>  system   x86_64, linux-gnu
#>  ui       X11
#>  language en
#>  collate  C.UTF-8
#>  ctype    C.UTF-8
#>  tz       UTC
#>  date     2025-09-04
#>  pandoc   3.1.11 @ /opt/hostedtoolcache/pandoc/3.1.11/x64/ (via rmarkdown)
#>  quarto   1.7.34 @ /usr/local/bin/quarto
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  ! package                 * version    date (UTC) lib source
#>    abind                     1.4-8      2024-09-12 [1] RSPM
#>    aphantasiaReasoningViie * 0.0.0.9000 2025-09-04 [1] local
#>    backports                 1.5.0      2024-05-23 [1] RSPM
#>    bayesplot                 1.14.0     2025-08-31 [1] RSPM
#>    bridgesampling            1.1-2      2021-04-16 [1] RSPM
#>    brms                      2.22.0     2024-09-23 [1] RSPM
#>    Brobdingnag               1.2-9      2022-10-19 [1] RSPM
#>    bslib                     0.9.0      2025-01-30 [1] RSPM
#>    cachem                    1.1.0      2024-05-16 [1] RSPM
#>    checkmate                 2.3.3      2025-08-18 [1] RSPM
#>    cli                       3.6.5      2025-04-23 [1] RSPM
#>    coda                      0.19-4.1   2024-01-31 [1] RSPM
#>    crayon                    1.5.3      2024-06-20 [1] RSPM
#>    desc                      1.4.3      2023-12-10 [1] RSPM
#>  P devtools                * 2.4.5      2022-10-11 [?] RSPM
#>    digest                    0.6.37     2024-08-19 [1] RSPM
#>    distributional            0.5.0      2024-09-17 [1] RSPM
#>    dplyr                     1.1.4      2023-11-17 [1] RSPM
#>  P ellipsis                  0.3.2      2021-04-29 [?] RSPM
#>    emmeans                   1.11.2-8   2025-08-27 [1] RSPM
#>    estimability              1.5.1      2024-05-12 [1] RSPM
#>    evaluate                  1.0.5      2025-08-27 [1] RSPM
#>    farver                    2.1.2      2024-05-13 [1] RSPM
#>    fastmap                   1.2.0      2024-05-15 [1] RSPM
#>    faux                      1.2.2      2025-01-15 [1] RSPM
#>    fs                        1.6.6      2025-04-12 [1] RSPM
#>    generics                  0.1.4      2025-05-09 [1] RSPM
#>    ggplot2                   3.5.2      2025-04-09 [1] RSPM
#>    glue                      1.8.0      2024-09-30 [1] RSPM
#>    gtable                    0.3.6      2024-10-25 [1] RSPM
#>    htmltools                 0.5.8.1    2024-04-04 [1] RSPM
#>    htmlwidgets               1.6.4      2023-12-06 [1] RSPM
#>    httpuv                    1.6.16     2025-04-16 [1] RSPM
#>    jquerylib                 0.1.4      2021-04-26 [1] RSPM
#>    jsonlite                  2.0.0      2025-03-27 [1] RSPM
#>    knitr                     1.50       2025-03-16 [1] RSPM
#>    later                     1.4.4      2025-08-27 [1] RSPM
#>  P lattice                   0.22-7     2025-04-02 [?] CRAN (R 4.5.1)
#>    lifecycle                 1.0.4      2023-11-07 [1] RSPM
#>    loo                       2.8.0      2024-07-03 [1] RSPM
#>    magrittr                  2.0.3      2022-03-30 [1] RSPM
#>  P Matrix                    1.7-3      2025-03-11 [?] CRAN (R 4.5.1)
#>    matrixStats               1.5.0      2025-01-07 [1] RSPM
#>    memoise                   2.0.1      2021-11-26 [1] RSPM
#>    mime                      0.13       2025-03-17 [1] RSPM
#>    miniUI                    0.1.2      2025-04-17 [1] RSPM
#>    mvtnorm                   1.3-3      2025-01-10 [1] RSPM
#>  P nlme                      3.1-168    2025-03-31 [?] CRAN (R 4.5.1)
#>    pillar                    1.11.0     2025-07-04 [1] RSPM
#>    pkgbuild                  1.4.8      2025-05-26 [1] RSPM
#>    pkgconfig                 2.0.3      2019-09-22 [1] RSPM
#>    pkgdown                   2.1.3      2025-05-25 [1] any (@2.1.3)
#>    pkgload                   1.4.0      2024-06-28 [1] RSPM
#>    posterior                 1.6.1      2025-02-27 [1] RSPM
#>  P profvis                   0.4.0      2024-09-20 [?] RSPM
#>    promises                  1.3.3      2025-05-29 [1] RSPM
#>    purrr                     1.1.0      2025-07-10 [1] RSPM
#>    R6                        2.6.1      2025-02-15 [1] RSPM
#>    ragg                      1.5.0      2025-09-02 [1] RSPM
#>    RColorBrewer              1.1-3      2022-04-03 [1] RSPM
#>    Rcpp                      1.1.0      2025-07-02 [1] RSPM
#>    RcppParallel              5.1.11-1   2025-08-27 [1] RSPM
#>  P remotes                   2.5.0      2024-03-17 [?] RSPM
#>    renv                      1.1.4      2025-03-20 [1] RSPM (R 4.5.0)
#>    rlang                     1.1.6      2025-04-11 [1] RSPM
#>    rmarkdown                 2.29       2024-11-04 [1] RSPM
#>    rstantools                2.5.0      2025-09-01 [1] RSPM
#>    sass                      0.4.10     2025-04-11 [1] RSPM
#>    scales                    1.4.0      2025-04-24 [1] RSPM
#>    sessioninfo               1.2.3      2025-02-05 [1] RSPM
#>    shiny                     1.11.1     2025-07-03 [1] RSPM
#>    stringi                   1.8.7      2025-03-27 [1] RSPM
#>    stringr                   1.5.1      2023-11-14 [1] RSPM
#>    systemfonts               1.2.3      2025-04-30 [1] RSPM
#>    tensorA                   0.36.2.1   2023-12-13 [1] RSPM
#>    textshaping               1.0.3      2025-09-02 [1] RSPM
#>    tibble                    3.3.0      2025-06-08 [1] RSPM
#>    tidyr                     1.3.1      2024-01-24 [1] RSPM
#>    tidyselect                1.2.1      2024-03-11 [1] RSPM
#>  P urlchecker                1.0.1      2021-11-30 [?] RSPM
#>  P usethis                 * 3.2.0      2025-08-28 [?] RSPM
#>    utf8                      1.2.6      2025-06-08 [1] RSPM
#>    vctrs                     0.6.5      2023-12-01 [1] RSPM
#>    withr                     3.0.2      2024-10-28 [1] RSPM
#>    xfun                      0.53       2025-08-19 [1] RSPM
#>    xtable                    1.8-4      2019-04-21 [1] RSPM
#>    yaml                      2.3.10     2024-07-26 [1] RSPM
#> 
#>  [1] /home/runner/.cache/R/renv/library/aphantasiaReasoningViie-b75da44b/linux-ubuntu-noble/R-4.5/x86_64-pc-linux-gnu
#>  [2] /home/runner/.cache/R/renv/sandbox/linux-ubuntu-noble/R-4.5/x86_64-pc-linux-gnu/179fe56a
#> 
#>  * ── Packages attached to the search path.
#>  P ── Loaded and on-disk path mismatch.
#> 
#> ──────────────────────────────────────────────────────────────────────────────

References

Delem, M., Turkben, S., Cousineau, D., Cavalli, E., & Plancher, G. (2025). Unsupervised clustering reveals spatial and verbal cognitive profiles in aphantasia and typical imagery.
Gazzo, E., Knauff, M., & Knauff, M. (2013). Individual differences, imagery and the visual impedance effect.
Tse, A. P. P., Ragni, M., & Lösch, J. (2017). Beyond the visual impedance effect. 15th International Conference on Cognitive Modeling, 115.
Wright, D. J., Scott, M. W., Kraeutner, S. N., Barhoun, P., Bertollo, M., Campbell, M. J., Waltzing, B. M., Dahm, S. F., Esselaar, M., Frank, C., Hardwick, R. M., Fuelscher, I., Marshall, B., Hodges, N. J., Hyde, C., & Holmes, P. S. (2024). An international estimate of the prevalence of differing visual imagery abilities. Frontiers in Psychology, 15, 1454107. https://doi.org/10.3389/fpsyg.2024.1454107