Early detection of brain diseases is necessary to deliver further and suitable treatment to save the patient life. Automated brain disease diagnosis is possible to be carried out with the availability of imaging techniques and the Deep Learning method. In recent years, many researchers have been interested in medical image classification problems, including brain disease detection based on Magnetic Resonance Images (MRI) using the Convolutional Neural Network (CNN) algorithm of Deep Learning. CNN has a unique advantage compared with traditional Machine Learning to do automated image feature extraction. However, CNN will perform better if numerous datasets are provided. Unfortunately, the lack of data due to privacy is still a problem in the medical image analysis topic. In order to solve that problem, many researchers have implemented a transfer learning technique to train the CNN models with small data. This study has proposed bilinear models based on CNN to distinguish brain MR images into five classes. In this study, MobileNetV1 and MobileNetV2 are employed as backbone networks to extract features via transfer learning, and the bilinear method is implemented to integrate the features from both networks. The proposed method improved the classification performance of the CNN model with a testing accuracy of 98.03%.