Due to the various species of fish, and the influence of different light and background environments, the recognition accuracy of some traditional fish recognition algorithms bases on color texture or feature point extraction is reduced, and good classification results cannot be achieved. To solve the problem, based on the existing AlexNet convolutional neural network, this paper proposes one method to speed up model training by reducing redundant convolutional layers. An item-based soft attention algorithm is applied to the improved AlexNet convolutional neural network model, which consists of four convolutional layers, one item-based soft attention layer, and two fully-connected layers. Meanwhile, a fish recognition model is established using transfer learning methods.The test results show that the average recognition accuracy of the proposed algorithm achieves 97.43%, which is 4.08% higher than the original AlexNet model, and the average recognition rate of some fishes achieves 99.31%, and time consumption of fish recognition is reduced by 35%. In conclusion, compared with state-of-the-art fish recognition algorithms, the fish recognition algorithm proposed in this study achieve higher accuracy, lower model complexity, and stronger robustness.