Python影像辨識筆記(十三)：計算YOLOv4 Weights的mAP

2020 / 06 /11更新重要觀念：

Training set、Validation set、Test set的差別

Ripley, B.D（1996）, Pattern Recognition and Neural Networks
Training set : A set of examples used for learning , which is to fit the parameters [i.e., weights] of the classifier.
Validation set : A set of examples used to tune the parameters [i.e., architecture, not weights] of a classifier, for example to choose the number of hidden units in a neural network.
Test set : A set of examples used only to assess the performance [generalization] of a fully specified classifier.
簡單來說：
Training set做訓練。
Validation set在訓練時調整超參數。
Test set跟Training set完全不一樣，只拿來評估Model的mAP。
切勿將Val set拿去測試mAP，出來結果是錯誤的

能在local測試mAP嗎？

不行，因為官方沒有提供test set的label，只能透過COCO的Server去評估mAP

參考資料：

https://github.com/AlexeyAB/darknet/wiki

How to evaluate AP of YOLOv4

How to evaluate AP of YOLOv4 on the MS COCO evaluation server

1. 設定 width=608 height=608 (or 512x512, or 416x416) in the [net] section of cfg/yolov4.cfg file

https://github.com/AlexeyAB/darknet/blob/6f718c257815a984253346bba8fb7aa756c55090/cfg/yolov4.cfg#L8-L9

2. 下載並解壓縮test-dev2017 dataset:

test集圖片中的txt和jpg要擺在同一個資料夾裡面。
http://images.cocodataset.org/zips/test2017.zip

3. 下載testdev2017.txt清單檔案，並改成自己的路徑。

https://raw.githubusercontent.com/AlexeyAB/darknet/master/scripts/testdev2017.txt

4. 下載yolov4.weights

yolov4.weights

5. 建立cfg/coco-mAP.data

cfg/coco.data 的內容應該如下所示

#將train、valid改成屬於自己路徑的資料夾
classes= 80
train  = /yanwei.liu/darknet/coco/trainvalno5k.txt
valid  = yanwei.liu/darknet/coco/testdev2017.txt    #要用test set
names = /yanwei.liu/darknet/data/coco.names
backup = /yanwei.liu/darknet/backup/
eval=coco
trainvalno5k.txt的範例內容(只顯示一筆)
/yanwei.liu/darknet/coco/images/train2014/COCO_train2014_XXXXXX.jpg
testdev2017.txt的範例內容(只顯示一筆)
/yanwei.liu/darknet/coco/images/test2017/XXXXXX.jpg

6. 在darknet目錄底下建立/results/ 資料夾

mkdir results

7. 執行驗證:

./darknet detector valid cfg/coco-mAP.data cfg/yolov4.cfg yolov4.weights

8. 第七步驟完成後，將 /results/coco_results.json 改名成 detections_test-dev2017_yolov4_results.json 並且壓縮成 detections_test-dev2017_yolov4_results.zip

$ cd results
$ mv coco_results.json detections_test-dev2017_yolov4_results.json
$ zip detections_test-dev2017_yolov4_results.zip detections_test-dev2017_yolov4_results.json

9. 提交 detections_test-dev2017_yolov4_results.zip 到MS COCO 驗證伺服器，選擇（ test-dev2019 (bbox) ）項目

該網站需要註冊帳號才能使用
https://competitions.codalab.org/competitions/20794#participate
取得以下結果 (AP=0.435 and AP50=0.657) View scoring output log
overall performance
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.435
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.657
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.473
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.267
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.467
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.533
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | 
maxDets=  1 ] = 0.342
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | 
maxDets= 10 ] = 0.549
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.580
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.403
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.617
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.713
Done (t=334.58s)