This work presents a novel approach for monitoring 'beach scenes', utilizing existing beach webcams and computer vision algorithms based on deep neural networks to analyze the behavior of water bodies, humans, and the interactions between them. We used surf condition analysis as a study case for our method. The proposed approach detects and tracks both people and waves simultaneously, distinguishing between waiting surfers and riding surfers. The system uses a Faster-RCNN object detector for detection and classification and SORT, a fast multi-object tracking algorithm based on the Kalman filter, for tracking. Our method can provide useful information to surfers to compare surfing conditions at different surfing spots.