Abstract: Abstract—Recently, deep learning based models have been extensively applied to image restoration and enhancement tasks. However, existing approaches neglect the potential correlation among these tasks, and do not fully consider the different intensity factors of degradation which affect image quality. To this end, we propose Task-Adaptive Operation Network (TAO-Net) that stacks a series of operation blocks, guided by the supervised attention mechanism to adaptively enable the model to assign corresponding intensity weights for different degradation factors depending on the input signals. To better recover and preserve the accurate details during the repeated operations, we make full use of the image prior to intruduce structure refinement block as auxiliary information for reconstructing high-quality results. Furthermore, feature aggregation block is also adopted to avoid feature interference caused by the direct concatenation between the backbone operation block and the auxiliary refinement block together. Extensive experiments on multiple benchmark datasets demonstrate that our proposed method outperforms previous baselines in image deraining and low-light image enhancement.