Asg 5: OpenCV face detection and tracking

Assignment

Implement face detection and tracking in OpenCV, complete with facial features, i.e., nose, mouth, and left and right eye Areas Of Interest, or AOIs.

Face detection, by itself, e.g., 

face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_alt2.xml')
face_cascade.detectMultiScale(gray, 1.1, 5)
will return a list of rectangles for all features in the frame that the detector considers faces. There could be many of them, and there could be many false positives.

In this assignment, implement a face object that exhibits frame coherence. In other words, each face is labeled, e.g., face_0, face_1, and so on, and your detector tracks where the face goes over a number of frames. If the given face disappears from view, the detector can fade (out of existence) over a given number of frames.

For each detected and tracked face, display in the output identified eyes, nose, and mouth within each face.

Suggestions (e.g., for Python implementation)

  1. A good strategy would be to implement a Face object, which contains the eye, nose, and mouth AOIs. Then, whenever the face needs to be rendered (drawn), this object would draw all the required rectangles.
  2. For detection of individual features, these Haar cascades seem to work fairly well: 
        self.leyeCascadeName = './haarcascades/haarcascade_lefteye_2splits.xml'
    self.reyeCascadeName = './haarcascades/haarcascade_righteye_2splits.xml'

    if cv2.__version__ == '4.0.0' or cv2.__version__ == '4.2.0':
      self.noseCascadeName = './haarcascades/haarcascade_mcs_nose.xml'
    elif cv2.__version__ == '3.4.1':
      self.noseCascadeName = './haarcascades/Nariz.xml'

    if cv2.__version__ == '4.0.0' or cv2.__version__ == '4.2.0':
      self.mouthCascadeName = './haarcascades/haarcascade_mcs_mouth.xml'
    elif cv2.__version__ == '3.4.1':
      self.mouthCascadeName = './haarcascades/Mouth.xml'
    You can test various other Haar cascade filters here: Haar Cascades
  3. There are many similar solutions out on the web, e.g., It would of course be best for you to implement your own solution rather than just grabbing and using someone else's.

Further Suggestions

  1. You can try implementing the artistic heuristics described in this short paper:
    Duchowski, A. T., Gehrer, N. A., Schönenberg, M., and Krejtz, K., Art Facing Science: Artistic Heuristics for Face Detection, in Proceedings of the 4th Workshop on Eye Tracking and Visualization (ETVIS), June 25-28, 2019, Denver, CO.
  2. Additional interesting options could be to implement face recognition or to write an interactive face AOI selector (e.g., using the mouse)

Example Program Input

None. Find a short video that you like.

Example Program Output

Ideally an .mp4 video could be made, although that may need to be encoded post-facto by ffmpeg as OpenCV might only output raw video.

An example video with three faces is given here: pl3_exports-640x360.mp4

Turn in

Turn in all of your code, in one tar.gz archive of your asg##/ directory, including:
  1. A README file containing
    1. Course id--section no
    2. Name
    3. Brief solution description (e.g., program design, description of algorithm, etc., however appropriate).
    4. Lessons learned, identified interesting features of your program
    5. Any special usage instructions
  2. Makefile
  3. source code
  4. short video

How to hand in

See handin notes

Grading scheme