Thursday, 17 December 2020

Webcam capture with ffmpeg and OpenCV from Jupyter Notebook

I want to share here my experience with using OpenCV and ffmpeg to capture a webcam output.


Setup:
  • Jupyter notebook running in jupyter-lab
  • Ubuntu 20.04
  • USB web camera
Goal:
  • Capture and display frames from the webcam

OpenCV: Video I/O with OpenCV Overview says that OpenCV: cv::VideoCapture Class calls video I/O backends (APIs) depending on which one is available.

To find out what backends (VideoCaptureAPIs) are available we can use the following code:

import cv2

# cv2a.videoio_registry.getBackends() returns list of all available backends.
availableBackends = [cv2.videoio_registry.getBackendName(b) for b in cv2.videoio_registry.getBackends()]
print(availableBackends)

# Returns list of available backends which works via cv::VideoCapture(int index)
availableCameraBackends = [cv2.videoio_registry.getBackendName(b) for b in cv2.videoio_registry.getCameraBackends()]
print(availableBackends)

The output in my case was: 

['FFMPEG', 'GSTREAMER', 'CV_IMAGES', 'CV_MJPEG']
['FFMPEG', 'GSTREAMER', 'CV_IMAGES', 'CV_MJPEG']

Let's see what is each of these backends:

• FFMPEG is a multimedia framework which can record, convert and stream audio and video.

It contains libavcodec, libavutil, libavformat, libavfilter, libavdevice, libswscale and libswresample which can be used by applications. As well as ffmpeg, ffplay and ffprobe which can be used by end users for transcoding and playing.

• GSTREAMER is a pipeline-based multimedia framework with similar capabilities as ffmpeg.

• CV_IMAGES -  OpenCV Image Sequence (e.g. img_%02d.jpg). Matches cv2.CAP_IMAGES API ID.

• CV_MJPEG - Built-in OpenCV MotionJPEG codec (used for reading video files). Matches cv2.CAP_OPENCV_MJPEG video capture API.

I was surprised to see GSTREAMER listed above as VideoCaptureAPIs documentation says

Backends are available only if they have been built with your OpenCV binaries. 

...and OpenCV package installed in my environment was built only with FFMPEG support:

>>> import cv2
>>> cv2.getBuildInformation()
...
Video I/O:\n    DC1394:                      NO\n    FFMPEG:                      YES\n      avcodec:                   YES (58.35.100)\n      avformat:                  YES (58.20.100)\n      avutil:                    YES (56.22.100)\n      swscale:                   YES (5.3.100)\n      avresample:                YES (4.0.0)\n\n  
...

...which can also be verifed by looking the cmake config in the repository (opencv-feedstock/build.sh at master · conda-forge/opencv-feedstock):

-DWITH_FFMPEG=1     \
-DWITH_GSTREAMER=0  \

Although my conda environment contained all relevant packages:

(my-env) $ conda list | grep 'opencv\|ffmpeg\|gstreamer'
ffmpeg                    4.1.3                h167e202_0    conda-forge
gstreamer                 1.14.5               h36ae1b5_2    conda-forge
opencv                    4.1.0            py36h79d2e43_1    conda-forge

...it is important to know that having ffmpeg and gstreamer packages installed means only that we have their binaries installed (executables and .so libraries) but not Python bindings (modules) or their OpenCV plugins. We are able to launch these applications from terminal but can't import them in Python code.

I tried to force using FFMPEG:

import cv2

deviceId = "/dev/video0"

# videoCaptureApi = cv2.CAP_ANY       # autodetect default API
videoCaptureApi = cv2.CAP_FFMPEG
# videoCaptureApi = cv2.CAP_GSTREAMER 
cap = cv2.VideoCapture("/dev/video2", videoCaptureApi)

cap = cv2.VideoCapture(deviceId)
cap.open(deviceId)
if not cap.isOpened():
    raise RuntimeError("ERROR! Unable to open camera")

try:
    while True:
        ret, frame = cap.read()
        cv2.imshow('frame', frame)
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
finally:        
    cap.release()
    cv2.destroyAllWindows()

...but cell execution would fail with:

RuntimeError: ERROR! Unable to open camera

I checked ($ v4l2-ctl --list-devices) - my webcam was indeed with index 2. As this was failing at the very beginning I decided to open python interpreter console and debug there only the isolated code snippet which opens the camera:

(my-env) $ export OPENCV_LOG_LEVEL=DEBUG; export OPENCV_VIDEOIO_DEBUG=1

(my-env) $ python 
Python 3.6.6 | packaged by conda-forge | (default, Oct 12 2018, 14:43:46) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cap = cv2.VideoCapture("/dev/video2", cv2.CAP_FFMPEG)
[ WARN:0] VIDEOIO(FFMPEG): trying capture filename='/dev/video2' ...
[ WARN:0] VIDEOIO(FFMPEG): can't create capture

I also tried to force using Gstreamer to no avail (which was expected):

>>> cap = cv2.VideoCapture("/dev/video2", cv2.CAP_GSTREAMER)
[ WARN:0] VIDEOIO(GSTREAMER): trying capture filename='/dev/video2' ...
[ INFO:0] VideoIO pluigin (GSTREAMER): glob is 'libopencv_videoio_gstreamer*.so', 1 location(s)
[ INFO:0]     - /home/bojan/anaconda3/envs/my-env/lib/python3.6/site-packages/../..: 0
[ INFO:0] Found 0 plugin(s) for GSTREAMER
[ WARN:0] VIDEOIO(GSTREAMER): backend is not available (plugin is missing, or can't be loaded due dependencies or it is not compatible)

Indeed ~/anaconda3/envs/my-env/lib did not contain ffmpeg plugin (libopencv_videoio_ffmpeg*.so files) or Gstreamer plugin (libopencv_videoio_gstreamer*.so files).

These plugins are installed only if OpenCV is build with following CMake options:

- DWITH_FFMPEG=1     \
-DVIDEOIO_PLUGIN_LIST=ffmpeg

...or (for Gstreamer):

-DWITH_GSTREAMER=1 \
-DVIDEOIO_PLUGIN_LIST=gstreamer \

...and apart from WITH_FFMPEG no other were used in the cmake config that was used to build OpenCV package installed in my environment.

As I didn't want to compile OpenCV myself but to achieve my goal with what I have I decided to see if I can run ffmpg process to stream camera output into a pipe and then read the binary information from it and convert it into frames:

import os
import tempfile
import subprocess
import cv2
import numpy as np

# To get this path execute:
#    $ which ffmpeg
FFMPEG_BIN = '/home/bojan/anaconda3/envs/my-env/bin/ffmpeg'


# To find allowed formats for the specific camera:
#    $ ffmpeg -f v4l2 -list_formats all -i /dev/video3
#    ...
#    [video4linux2,v4l2 @ 0x5608ac90af40] Raw: yuyv422: YUYV 4:2:2: 640x480 1280x720 960x544 800x448 640x360 424x240 352x288 320x240 800x600 176x144 160x120 1280x800
#    ...

def run_ffmpeg(fifo_path):
    ffmpg_cmd = [
        FFMPEG_BIN,
        '-i', '/dev/video2',
        '-video_size', '640x480',
        '-pix_fmt', 'bgr24',        # opencv requires bgr24 pixel format
        '-vcodec', 'rawvideo',
        '-an','-sn',                # disable audio processing
        '-f', 'image2pipe',
        '-',                        # output to go to stdout
    ]
    return subprocess.Popen(ffmpg_cmd, stdout = subprocess.PIPE, bufsize=10**8)

def run_cv_window(process):
    while True:
        # read frame-by-frame
        raw_image = process.stdout.read(640*480*3)
        if raw_image == b'':
            raise RuntimeError("Empty pipe")
        
        # transform the bytes read into a numpy array
        frame =  np.frombuffer(raw_image, dtype='uint8')
        frame = frame.reshape((480,640,3)) # height, width, channels
        if frame is not None:
            cv2.imshow('Video', frame)
        
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
        process.stdout.flush()
    
    cv2.destroyAllWindows()
    process.terminate()
    print(process.poll())

def run():
    ffmpeg_process = run_ffmpeg()
    run_cv_window(ffmpeg_process)

run()

Et voila! I got the camera capture from Python notebook thanks to ffmpeg and OpenCV.



5 comments:

Bernard Ryan said...

Need The To Hire A Hacker❓ Then contact PYTHONAX✅

The really amazing deal about contacting PYTHONAX is that the Hack done by us can’t get traced to you, as every Hacking job we do is strongly protected by our Firewall. It’s like saying if anyone tries to trace the Hack, it will lead them to us and we block whatever actions they are doing.

We have been Invisible to Authorities for almost a decade now and if you google PYTHONAX, not really about us comes out, you can only see comments made by us or about us.

Another Amazing thing to you benefit from Hiring our Hackers is that you get a Legit and the best Hacking service, As we provide you with Professional Hackers who have their Hacking Areas of specialization.
We perform every Hack there is, using special Hacking tools we get from the dark web.

Some list of Hacking Services we provide are-:
▪️Phone Hacking & Cloning ✅
▪️Computer Hacking ✅
▪️Emails & Social Media Account Hacking✅
▪️Recovering Deleted Files✅
▪️Tracking & Finding People ✅
▪️Hunting Down Scammers✅
▪️Hack detecting ✅
▪️Stealing/Copying Files & Documents From Restricted Networks and Servers ✅
▪️Credit Score Manipulation ✅
▪️ Deleting Criminal Records✅
▪️Bitcoin Multiplication✅
▪️Binary Option Money Recovery ✅
▪️Scam Money Recovery✅ And lots more......

✳️ SPECIAL HACKING SERVICES-: we also specialize in Scam Bounty, as we chase down SCAMMERS and help individuals RECOVER Money stolen from them by this online SCAMMERS. Please be watchful about this SCAMMERS. They post ❌ENTICING TESTIMONIES and it quite Convincing.


Whatever Hacking service you require, just give us an Email to the Emails Address provided below.
pythonaxhacks@gmail.com
pythonaxservices@gmail.com

PYTHONAX.
2020 © All Right Reserved.

Tech Institute said...

Fantastic article with valuable information found very useful looking forward for next blog thank you.
Data Science Course in Bangalore

khadi Oman said...

Tried a lot of hackers but i strongly recommend thehackerspro.com where i eventually hired the dreaded darkhat hacker vladimir kolarov
although he was much more expensive than other hackers i contacted but he did a good job and their customer support on the website is friendly and very helpful.
These are ethical hackers that’ll solve all your problems like hack emails,Facebook, Twitter ,Instagram , change grades ,
erase criminal records, credit and debit top up, insurance paper,
access or recover lost files, background checks on individuals and organizations
or monitor cheating spouses’ phone or social media activities, contact at h4ckerspro@gmail.com or call/text 1 646 948 8125.
for faster and direct access just add him on telegram @h4ckerspro

doron said...

i was lost with no hope for my wife was cheating and had always got away with it because i did not know how or

always too scared to pin anything on her. with the help a friend who recommended me to who help hack her phone,

email, chat, sms and expose her for a cheater she is. I just want to say a big thank you to

SUPERIOR.HACK@GMAIL.COM . am sure someone out there is looking for how to solve his relationship problems, you can also contact him for all sorts of hacking job..he is fast and reliable. you could also text +1 213-295-1376(whatsapp) contact and thank me later

Ink Drop said...

Buy Real Driver's License Online

Email Us:- inkdrop121@gmail.com
We produce Unique Real Registered Documents with Best Quality Novelty.
All our documents are undetectable and will pass all Ultraviolet light Test
such documents includes:
-Passport
-Drivers License
-National Identification Card
-Resident Permit
-Visa/Invitation
-Diploma Certificate
-Bank Statement and many more.....

Email Us:- inkdrop121@gmail.com


What's App Number +44 7520660907

https://www.kemiio.com/shop/buyrealdriverslicenseonline