To improve the speed and accuracy of hierarchical convolutional features for visual tracking method, a real-time and robust object tracking algorithm based on adaptive model update and single-layer convolutional features was proposed. This method firstly extracted the multi-channel convolutional features of the Pool4 layer to adjust the label function of the training samples,which improved the speed of the algorithm while ensuring the tracking accuracy.Meanwhile, the average peak-to-correlation energy was introduced in the proposed algorithm, which feedbacked the tracking results. Combined with the sparse model update strategy, the tracker was adaptively updated to improve the robustness of the algorithm to occlusion and similar object interference. For the problem of rapid scale variation, the scale pyramid was adopted to evaluate the scale to further improve the generalization ability of our tracker. Finally, the algorithm was verified on OTB2013 and OTB2015 benchmark datasets. The experimental results showed that the average distance precision was 91.0% and 86.8%, and the average speed was 43 frames per second, showing outperforming robustness and real-time performances.