When environments lack compelling goals, humans often let their minds wander to thoughts with greater personal relevance; however, we currently do not understand how this context-dependent prioritisation process operates. Dorsolateral prefrontal cortex (DLPFC) maintains goal representations in a context-dependent manner. Here, we show this region is involved in prioritising off-task thought in an analogous way. In a whole brain analysis we established that neural activity in DLPFC is high both when 'on-task' under demanding conditions and 'off-task' in a non-demanding task. Furthermore, individuals who increase off-task thought when external demands decrease, show lower correlation between neural signals linked to external tasks and lateral regions of the DMN within DLPFC, as well as less cortical grey matter in regions sensitive to these external task relevant signals. We conclude humans prioritise daydreaming when environmental demands decrease by aligning cognition with their personal goals using DLPFC.