Self-Supervised Pre-training Vision Transformer with Masked Autoencoders for Building Subsurface Model

Building subsurface models is a very important but challenging task in hydrocarbon exploration and development. The subsurface elastic properties are usually sourced from seismic data and well logs. Thus, we design a deep learning (DL) framework using Vision Transformer (ViT) as the backbone architecture to build the subsurface model using well log information as we apply full waveform inversion (FWI) on the seismic data. However, training a ViT network from scratch with limited well log data can be difficult to achieve good generalization. To overcome this, we implement an efficient self-supervised pre-training process using a masked autoencoder (MAE) architecture to learn important feature representations in seismic volumes. The seismic volumes required by the pre-training are randomly extracted from a seismic inversion, such as an FWI result. We can also incorporate reverse time migration (RTM) image into the seismic volumes to provide additional structure information. The pre-training task of MAE is to reconstruct the original image from the masked image with a masking ratio of 75%. This pre-training task enables the network to learn the high-level latent representations. After the pre-training process, we then fine-tune the ViT network to build the optimal mapping relationship between 2D seismic volumes and 1D well segments. Once the fine-tuning process is finished, we apply the trained ViT network to the whole seismic inversion domain to predict the subsurface model. At last, we use one synthetic data set and two field data sets to test the performance of the proposed method. The test results demonstrate that the proposed method effectively integrates seismic and well information to improve the resolution and accuracy of the velocity model.