Code, Computer Vision, Technology And Science

Multi-camera Capture using OpenCV (Multi-threaded)

Previously, I have managed to create a simple program to capture images from multiple cameras (click here). However, it is not quite good approach to use just a single thread to handle all of the capturing processes. Thus, I need to improve it by using multi-thread approach. Each thread will capture images from a single camera, so the number of threads will be determined by the number of cameras. In addition, to make it even better for further application (e.g.: motion detection), I made it object oriented.

First, let’s make a header file which will contain all class members (attributes and functions) declaration. I call it, “CameraStreamer.hpp”:

#pragma once
#include <iostream>
#include <string>
#include <thread>
#include <vector>
#include <concurrent_queue.h>
#include "opencv2\videoio.hpp"

using namespace std;
using namespace cv;
using namespace concurrency;

class CameraStreamer{
public:
	//this holds camera stream urls
	vector<string> camera_source;
	//this holds usb camera indices
	vector<int> camera_index;
	//this holds OpenCV VideoCapture pointers
	vector<VideoCapture*> camera_capture;
	//this holds queue(s) which hold images from each camera
	vector<concurrent_queue<Mat>*> frame_queue;
	//this holds thread(s) which run the camera capture process
	vector<thread*>
 camera_thread;

	//Constructor for IP Camera capture
	CameraStreamer(vector<string> source);
	//Constructor for USB Camera capture
	CameraStreamer(vector<int> index);
	//Destructor for releasing resource(s)
	~CameraStreamer();

private:
	bool isUSBCamera;
	int camera_count;
	//initialize and start the camera capturing process(es)
	void startMultiCapture();
	//release all camera capture resource(s)
	void stopMultiCapture();
	//main camera capturing process which will be done by the thread(s)
	void captureFrame(int index);
};

Note: I am using std:thread as a camera capture worker and concurrency:concurrent_queue as a frame container.

After that, let’s make the class implementation in “CameraStreamer.cpp”:

#include "CameraStreamer.hpp"

CameraStreamer::CameraStreamer(vector<string> stream_source)
{
	camera_source = stream_source;
	camera_count = camera_source.size();
	isUSBCamera = false;

	startMultiCapture();
}

CameraStreamer::CameraStreamer(vector<int> capture_index)
{
	camera_index = capture_index;
	camera_count = capture_index.size();
	isUSBCamera = true;

	startMultiCapture();
}

CameraStreamer::~CameraStreamer()
{
	stopMultiCapture();
}

void CameraStreamer::captureFrame(int index)
{
	VideoCapture *capture = camera_capture[index];
	while (true)
	{
		Mat frame;
		//Grab frame from camera capture
		(*capture) >> frame;
		//Put frame to the queue
		frame_queue[index]->push(frame);
		//relase frame resource
		frame.release();
	}
}

void CameraStreamer::startMultiCapture()
{
	VideoCapture *capture;
	thread *t;
	concurrent_queue<Mat> *q;
	for (int i = 0; i < camera_count; i++)
	{
		//Make VideoCapture instance
		if (!isUSBCamera){
			string url = camera_source[i];
			capture = new VideoCapture(url);
			cout << "Camera Setup: " << url << endl;
		}
		else{
			int idx = camera_index[i];
			capture = new VideoCapture(idx);
			cout << "Camera Setup: " << to_string(idx) << endl;
		}

		//Put VideoCapture to the vector
		camera_capture.push_back(capture);

		//Make thread instance
		t = new thread(&CameraStreamer::captureFrame, this, i);

		//Put thread to the vector
		camera_thread.push_back(t);

		//Make a queue instance
		q = new concurrent_queue<Mat>;

		//Put queue to the vector
		frame_queue.push_back(q);
	}
}

void CameraStreamer::stopMultiCapture()
{
	VideoCapture *cap;
	for (int i = 0; i < camera_count; i++) 	{ 		cap = camera_capture[i]; 		if (cap->isOpened()){
			//Relase VideoCapture resource
			cap->release();
			cout << "Capture " << i << " released" << endl;
		}
	}
}

Finally, we can use the CameraStreamer to run multi-camera capture. So, let’s make another cpp file “MultiCamera.cpp” as the main program:

#include "CameraStreamer.hpp"
#include "opencv2\highgui.hpp"

void main()
{

	//IP camera URLs
	vector<string> capture_source = {
		"rtsp://192.168.2.100/profile2/media.smp",
		"rtsp://192.168.0.100/profile2/media.smp"
	};

	//USB Camera indices
	vector<int> capture_index = { 0, 1 };

	//Highgui window titles
	vector<string> label;
	for (int i = 0; i < capture_source.size(); i++)
	{
		string title = "CCTV " + to_string(i);
		label.push_back(title);
	}

	//Make an instance of CameraStreamer
	CameraStreamer cam(capture_source);

	while (waitKey(20) != 27)
	{
		//Retrieve frames from each camera capture thread
		for (int i = 0; i < capture_source.size(); i++) 		{ 			Mat frame; 			//Pop frame from queue and check if the frame is valid 			if (cam.frame_queue[i]->try_pop(frame)){
				//Show frame on Highgui window
				imshow(label[i], frame);
			}
		}
	}
}

You can choose whether you want to capture from IP cameras (capture_source) or USB cameras (capture_index). Please make sure you  have already specified the IP camera URLs properly.

13 thoughts on “Multi-camera Capture using OpenCV (Multi-threaded)

  1. This is not compiling on my ubuntu😦. Any helps ?
    Here is the log : http://pastebin.com/E9NHTL5Z
    I am pretty sure that all the libraries are properly installed, but in particular,
    concurrent_queue thing is giving the error here, moreover its not even recognizing the namespace.
    Any help would be appreciated! Thanks!

    1. Concurrency data structure is not built in C++ namespace. Lucky me, I use Ms.Visual Studio and MS has included that for me. Concurrency namespace is based on Intel’s implementation in Threading Building Blocks (TBB). Please visit TBB’s website (https://www.threadingbuildingblocks.org/). Once you already have installed TBB on your ubuntu, you can use concurrent queue by including tbb/concurrent_queue.h to your project.

    2. I have succesfully run this wonderfull code in ubuntu.

      This is little change i made at CameraStreamer.hpp

      #include
      #include
      #include
      #include
      #include
      #include
      #include
      #include
      //#include “opencv2/videoio.hpp”

      using namespace std;
      using namespace cv;
      using namespace tbb;

      You have to install tbb and add it to compiler and linker.

      Terima kasih

  2. Hi Puto, a question. In line 64, when you call the captureFrame function, I’m confused how the code is reading the next line since the captureFrame function contains an infinite while loop. I would assume that the code will only keep reading from one camera and would never proceed further. I’m asking you because I was having trouble with the code I wrote. I haven’t used your yet. Thank you.

    1. Line 64 is a thread constructor. I assign the function which will be done by the thread. The function captureFrame will not be executed by the main thread of our application. It will be executed by another thread. Thus, the next line will be executed by the main thread.
      I suggest you to read other simple multi-threading examples first, before using this code.

      1. Thanks so much for your response. I’m fairly new to using the multithreading library. I have been reviewing some other simple examples on how to use thread as you suggested and there is one more thing that has been confusing me. All these examples refer to using join() function to wait for the thread to finish execution before exiting the program. I wonder why you haven’t used it your program? Will exiting the main program without joining the threads result in run time error?

      2. If you read the documentation of join(), it is used for blocking the parent thread, and wait until the child thread finishes.
        I don’t use join() because I don’t want the main thread to be blocked by the camera capture thread.
        And if you read my code thoroughly, I call stopMultiCapture() on the CameraStreamer destructor for releasing the resource that the thread uses to prevent error when closing the application.

  3. Thanks for an amazing article! Can any one share link about how to compile tbb for ubuntu? I can’t get it working. Single threaded code is working perfectly fine.

  4. Who ever is trying to run the code in Ubuntu this is how i made it to work

    1. sudo apt-get install libtbb-dev
    2. add these lines to find the include files /usr/include/tbb (default installation directory)
    3. for libraries add /usr/lib and -ltbb to the linkers.

    Everything worked for me. Thank you for this tutorial.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s