I want to share here my experience with using OpenCV and ffmpeg to capture a webcam output.
Setup:
- Jupyter notebook running in jupyter-lab
- Ubuntu 20.04
- USB web camera
Goal:
- Capture and display frames from the webcam
OpenCV: Video I/O with OpenCV Overview says that OpenCV: cv::VideoCapture Class calls video I/O backends (APIs) depending on which one is available.
To find out what backends (VideoCaptureAPIs) are available we can use the following code:
import cv2
# cv2a.videoio_registry.getBackends() returns list of all available backends.
availableBackends = [cv2.videoio_registry.getBackendName(b) for b in cv2.videoio_registry.getBackends()]
print(availableBackends)
# Returns list of available backends which works via cv::VideoCapture(int index)
availableCameraBackends = [cv2.videoio_registry.getBackendName(b) for b in cv2.videoio_registry.getCameraBackends()]
print(availableBackends)
The output in my case was:
['FFMPEG', 'GSTREAMER', 'CV_IMAGES', 'CV_MJPEG']
['FFMPEG', 'GSTREAMER', 'CV_IMAGES', 'CV_MJPEG']
Let's see what is each of these backends:
• FFMPEG is a multimedia framework which can record, convert and stream audio and video.
It contains libavcodec, libavutil, libavformat, libavfilter, libavdevice, libswscale and libswresample which can be used by applications. As well as ffmpeg, ffplay and ffprobe which can be used by end users for transcoding and playing.
• GSTREAMER is a pipeline-based multimedia framework with similar capabilities as ffmpeg.
• CV_IMAGES - OpenCV Image Sequence (e.g. img_%02d.jpg). Matches cv2.CAP_IMAGES API ID.
• CV_MJPEG - Built-in OpenCV MotionJPEG codec (used for reading video files). Matches cv2.CAP_OPENCV_MJPEG video capture API.
I was surprised to see GSTREAMER listed above as VideoCaptureAPIs documentation says
Backends are available only if they have been built with your OpenCV binaries.
...and OpenCV package installed in my environment was built only with FFMPEG support:
>>> import cv2
>>> cv2.getBuildInformation()
...
Video I/O:\n DC1394: NO\n FFMPEG: YES\n avcodec: YES (58.35.100)\n avformat: YES (58.20.100)\n avutil: YES (56.22.100)\n swscale: YES (5.3.100)\n avresample: YES (4.0.0)\n\n
...
...which can also be verifed by looking the cmake config in the repository (opencv-feedstock/build.sh at master · conda-forge/opencv-feedstock):
-DWITH_FFMPEG=1 \
-DWITH_GSTREAMER=0 \
Although my conda environment contained all relevant packages:
(my-env) $ conda list | grep 'opencv\|ffmpeg\|gstreamer'
ffmpeg 4.1.3 h167e202_0 conda-forge
gstreamer 1.14.5 h36ae1b5_2 conda-forge
opencv 4.1.0 py36h79d2e43_1 conda-forge
...it is important to know that having ffmpeg and gstreamer packages installed means only that we have their binaries installed (executables and .so libraries) but not Python bindings (modules) or their OpenCV plugins. We are able to launch these applications from terminal but can't import them in Python code.
I tried to force using FFMPEG:
import cv2
deviceId = "/dev/video0"
# videoCaptureApi = cv2.CAP_ANY # autodetect default API
videoCaptureApi = cv2.CAP_FFMPEG
# videoCaptureApi = cv2.CAP_GSTREAMER
cap = cv2.VideoCapture("/dev/video2", videoCaptureApi)
cap = cv2.VideoCapture(deviceId)
cap.open(deviceId)
if not cap.isOpened():
raise RuntimeError("ERROR! Unable to open camera")
try:
while True:
ret, frame = cap.read()
cv2.imshow('frame', frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
finally:
cap.release()
cv2.destroyAllWindows()
...but cell execution would fail with:
RuntimeError: ERROR! Unable to open camera
I checked ($ v4l2-ctl --list-devices) - my webcam was indeed with index 2. As this was failing at the very beginning I decided to open python interpreter console and debug there only the isolated code snippet which opens the camera:
(my-env) $ export OPENCV_LOG_LEVEL=DEBUG; export OPENCV_VIDEOIO_DEBUG=1
(my-env) $ python
Python 3.6.6 | packaged by conda-forge | (default, Oct 12 2018, 14:43:46)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cap = cv2.VideoCapture("/dev/video2", cv2.CAP_FFMPEG)
[ WARN:0] VIDEOIO(FFMPEG): trying capture filename='/dev/video2' ...
[ WARN:0] VIDEOIO(FFMPEG): can't create capture
>>> cap = cv2.VideoCapture("/dev/video2", cv2.CAP_GSTREAMER)
[ WARN:0] VIDEOIO(GSTREAMER): trying capture filename='/dev/video2' ...
[ INFO:0] VideoIO pluigin (GSTREAMER): glob is 'libopencv_videoio_gstreamer*.so', 1 location(s)
[ INFO:0] - /home/bojan/anaconda3/envs/my-env/lib/python3.6/site-packages/../..: 0
[ INFO:0] Found 0 plugin(s) for GSTREAMER
[ WARN:0] VIDEOIO(GSTREAMER): backend is not available (plugin is missing, or can't be loaded due dependencies or it is not compatible)
Indeed ~/anaconda3/envs/my-env/lib did not contain ffmpeg plugin (libopencv_videoio_ffmpeg*.so files) or Gstreamer plugin (libopencv_videoio_gstreamer*.so files).
These plugins are installed only if OpenCV is build with following CMake options:
- DWITH_FFMPEG=1 \
-DVIDEOIO_PLUGIN_LIST=ffmpeg
...or (for Gstreamer):
-DWITH_GSTREAMER=1 \
-DVIDEOIO_PLUGIN_LIST=gstreamer \
...and apart from WITH_FFMPEG no other were used in the cmake config that was used to build OpenCV package installed in my environment.
As I didn't want to compile OpenCV myself but to achieve my goal with what I have I decided to see if I can run ffmpg process to stream camera output into a pipe and then read the binary information from it and convert it into frames:
import os
import tempfile
import subprocess
import cv2
import numpy as np
# To get this path execute:
# $ which ffmpeg
FFMPEG_BIN = '/home/bojan/anaconda3/envs/my-env/bin/ffmpeg'
# To find allowed formats for the specific camera:
# $ ffmpeg -f v4l2 -list_formats all -i /dev/video3
# ...
# [video4linux2,v4l2 @ 0x5608ac90af40] Raw: yuyv422: YUYV 4:2:2: 640x480 1280x720 960x544 800x448 640x360 424x240 352x288 320x240 800x600 176x144 160x120 1280x800
# ...
def run_ffmpeg(fifo_path):
ffmpg_cmd = [
FFMPEG_BIN,
'-i', '/dev/video2',
'-video_size', '640x480',
'-pix_fmt', 'bgr24', # opencv requires bgr24 pixel format
'-vcodec', 'rawvideo',
'-an','-sn', # disable audio processing
'-f', 'image2pipe',
'-', # output to go to stdout
]
return subprocess.Popen(ffmpg_cmd, stdout = subprocess.PIPE, bufsize=10**8)
def run_cv_window(process):
while True:
# read frame-by-frame
raw_image = process.stdout.read(640*480*3)
if raw_image == b'':
raise RuntimeError("Empty pipe")
# transform the bytes read into a numpy array
frame = np.frombuffer(raw_image, dtype='uint8')
frame = frame.reshape((480,640,3)) # height, width, channels
if frame is not None:
cv2.imshow('Video', frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
process.stdout.flush()
cv2.destroyAllWindows()
process.terminate()
print(process.poll())
def run():
ffmpeg_process = run_ffmpeg()
run_cv_window(ffmpeg_process)
run()
Et voila! I got the camera capture from Python notebook thanks to ffmpeg and OpenCV.
