TrackNet: Tennis Ball Tracking from Broadcast Video by Deep Learning Networks
First, you have to install cuda, cudnn and tensorflow, tutorial:
https://medium.com/@zhanwenchen/install-cuda-and-cudnn-for-tensorflow-gpu-on-ubuntu-79306e4ac04e
Second, install some python library with pip:
- sudo pip install numpy
- sudo pip install matplotlib
- sudo pip install pillow
- sudo pip install keras
- sudo pip install opencv-python
- sudo pip install pydot
- sudo pip install h5py
- sudo apt-get install graphviz
How to train a new TrackNet I weight?
- Create heatmap as Ground Truth, and save heatmap as JPG file Code be save in TrackNet_Python.ipynb (first part), you may need to change the folder path in python code
- The training file name and testing file name of TrackNet must be output as csv file Code be save in TrackNet_Python.ipynb (second part), you also need to change the folder path in code
- Copy the training_model1.csv file and testing_model1.csv file to TrackNet_One_Frames_Input folder
-
After have training images and ground truth, we can start to train the TrackNet model I
- Open command line
- Change directory to TrackNet_One_Frames_Input folder
-
Using following command as example, you may need to change the command:
python train.py --save_weights_path=weights/model --training_images_name="training_model1.csv" --epochs=500 --n_classes=256 --input_height=360 --input_width=640 --load_weights=1 --step_per_epochs=200 --batch_size=2
Trained model weight will be save in weights/model.0
* Detailed explanation
--save_weights_path: Save the weight path
--training_images_name: Training images csv file path, training_model1.csv
--epochs: Epochs be set as 500 in this work
--n_classes: Last layer classes, since the output value of TrackNet is between 0-255, the last layer depth be set as 256
--input_height: Input height be resize as 360 in this work
--input_width: Input width be resize as 640 in this work
--load_weights: If you want to retrain the weights from previous weight, give the number of weight in weights/model. If not, delete it.
--step_per_epochs: Step per Epochs be set as 200 in this work
-batch_size: Batch size be set as 2 in this work
How to train a new TrackNet II weight?
- Create heatmap as Ground Truth, and save heatmap as JPG file Code be saved in TrackNet_Python.ipynb (first part), you may need to change the folder path in python code
- The training file name and testing file name of TrackNet must be output as csv file Code be save in TrackNet_Python.ipynb (third part), you also need to change the folder path in code
- Copy the training_model2.csv file and testing_model2.csv file to TrackNet_Three_Frames_Input folder
-
After have training images and ground truth, we can start to train the TrackNet model II
- Open command line
- Change directory to TrackNet_Three_Frames_Input folder
-
Using following command as example, you may need to change the command:
python train.py --save_weights_path=weights/model --training_images_name="training_model2.csv" --epochs=500 --n_classes=256 --input_height=360 --input_width=640 --load_weights=2 --step_per_epochs=200 --batch_size=2
Trained model weight will be save in weights/model.0
* Detailed explanation
--save_weights_path: Save the weight path
--training_images_name: Training images csv file path, training_model2.csv
--epochs: Epochs be set as 500 in this work
--n_classes: Last layer classes, since the output value of TrackNet is between 0-255, the last layer depth be set as 256
--input_height: Input height be resize as 360 in this work
--input_width: Input width be resize as 640 in this work
--load_weights: If you want to retrain the weights from previous weight, give the number of weight in weights/model. If not, delete it.
--step_per_epochs: Step per Epochs be set as 200 in this work
-batch_size: Batch size be set as 2 in this work
How to train a new TrackNet II' weight?
- Create heatmap as Ground Truth, and save heatmap as JPG file Code be save in TrackNet_Python.ipynb (first part), you may need to change the folder path in python code
- The training file name and testing file name of TrackNet must be output as csv file Code be save in TrackNet_Python.ipynb (third part, fourth part and fifth part), you also need to change the folder path in code
- Copy the training_model3.csv file to TrackNet_Three_Frames_Input folder
-
After have training images and ground truth, we can start to train the TrackNet model II'
- Open command line
- Change directory to TrackNet_Three_Frames_Input folder
-
Using following command as example, you may need to change the command:
python train.py --save_weights_path=weights/model --training_images_name="training_model3.csv" --epochs=500 --n_classes=256 --input_height=360 --input_width=640 --load_weights=2 --step_per_epochs=200 --batch_size=2
Trained model weight will be save in weights/model.0
* Detailed explanation
--save_weights_path: Save the weight path
--training_images_name: Training images csv file path, training_model3.csv
--epochs: Epochs be set as 500 in this work
--n_classes: Last layer classes, since the output value of TrackNet is between 0-255, the last layer depth be set as 256
--input_height: Input height be resize as 360 in this work
--input_width: Input width be resize as 640 in this work
--load_weights: If you want to retrain the weights from previous weight, give the number of weight in weights/model. If not, delete it.
--step_per_epochs: Step per Epochs be set as 200 in this work
-batch_size: Batch size be set as 2 in this work
How to output all of heatmap predictions?
- Open command line
- Change directory to TrackNet folder (TrackNet_Three_Frames_Input or TrackNet_One_Frames_Input)
-
Using following command as example, you may need to change the command:
python predict.py --save_weights_path=weights/model.2 --test_images="/media/andersen/D/Thesis/Dataset/Clip" --output_path="/media/andersen/D/Thesis/Prediction/Model2/Clip" --n_classes=256 --input_height=360 --input_width=640 --output_height=720 --output_width=1280
* Detailed explanation
--save_weights_path: which model weight need to be loaded
--test_images: testing images path
--output_path: output heatmap path
--n_classes: in this work depth be set as 256
--input_height: Input height be resize as 360 in this work
--input_width: Input width be resize as 360 in this work
--output_height: resize the heatmap height, output height be set as 720 in this work
-output_width: resize the heatmap width,output width be set as 1280 in this work
How to use TrackNet predict video?
- Open command line
- Change directory to TrackNet folder (TrackNet_Three_Frames_Input or TrackNet_One_Frames_Input)
-
using following command as example, you may need to change the command:
python predict_video.py --save_weights_path=weights/model.3 --input_video_path="/media/andersen/D/Test/Clip1.mp4" --output_video_path="/media/andersen/D/Test/Clip1_TrackNet.mp4" --n_classes=256
* Detailed explanation
--save_weights_path: which model weight need to be loaded
--input_video_path: Input video path
--output_video_path: Output video path, if not, the video will be save in the same path of input video
--n_classes: In this work depth be set as 256
TrackNet trained weights:
* TrackNet model I >> TrackNet_One_Frames_Input/weights.model.1
* TrackNet model II >> TrackNet_Three_Frames_Input/weights.model.2
* TrackNet model II' >> TrackNet_Three_Frames_Input/weights.model.3
Dataset
A dataset is available via the link: https://drive.google.com/file/d/1Dq2ag6a7ESHJm3ZHSJrYcu9_hWNyNkx1/view?usp=sharing
Labeling Tool: Make your own dataset
See the readme file in the LabelingTool directory
Reference:
If you use TrackNet in your work, please cite the following papers!
- Yu-Chuan Huang, "TrackNet: Tennis Ball Tracking from Broadcast Video by Deep Learning Networks," Master Thesis, advised by Tsì-Uí İk and Guan-Hua Huang, National Chiao Tung University, Taiwan, April 2018.
- Yu-Chuan Huang, I-No Liao, Ching-Hsuan Chen, Tsì-Uí İk, and Wen-Chih Peng, "TrackNet: A Deep Learning Network for Tracking High-speed and Tiny Objects in Sports Applications," in the IEEE International Workshop of Content-Aware Video Analysis (CAVA 2019) in conjunction with the 16th IEEE International Conference on Advanced Video and Signal-based Surveillance (AVSS 2019), 18-21 September 2019, Taipei, Taiwan.