Skeletonizer3D is a plugin that computes the 3D skeleton of a human body from a depth map. More...

Inheritance diagram for Skeletonizer3D:

Collaboration diagram for Skeletonizer3D:

[legend]

Public Member Functions
void	setup_OpenPoseModel ()

void	setup_Pipeline ()

void	setup_VideoCapture ()

return_type	acquire_frame (bool dummy=false)
	Acquire a frame from a camera device. Camera ID is defined in the parameters list. More...

return_type	skeleton_from_depth_compute (bool debug=false)
	Compute the skeleton from the depth map. More...

return_type	point_cloud_filter (bool debug=false)
	Remove unnecessary points from the point cloud. More...

pcl::visualization::PCLVisualizer::Ptr	simpleVis (pcl::PointCloud< pcl::PointXYZ >::ConstPtr cloud)

return_type	coordinate_transfrom (bool debug=false)
	Transform the 3D skeleton coordinates in the global reference frame. More...

return_type	skeleton_from_rgb_compute (bool debug=false)
	Compute the skeleton from RGB images only. More...

return_type	hessian_compute (bool debug=false)
	Compute the hessians for joints. More...

return_type	cov3D_compute (bool debug=false)
	Compute the 3D covariance matrix. More...

return_type	consistency_check (bool debug=false)
	Consistency check of the 3D skeleton according to human physiology. More...

void	set_params (void *params) override
	Set the parameters of the plugin. More...

return_type	get_output (json out, vector< unsigned char > blob=nullptr) override
	Get the output of the plugin. More...

map< string, string >	info () override
	Provide further info to Miroscic agent. More...

string	kind () override
	The plugin identifier. More...

Public Member Functions inherited from Source< json >
virtual return_type	get_output (json out, std::vector< unsigned char > blob=nullptr)=0

std::string	error ()

std::string	blob_format ()

Protected Attributes
Mat	_rgbd

Mat	_rgb

map< string, vector< unsigned char > >	_skeleton2D

map< string, vector< unsigned char > >	_skeleton3D

vector< Mat >	_heatmaps

Mat	_point_cloud

Mat	_cov3D

Mat	_cov3D_adj

json	_params

uint32_t	_tsize = 0

data_t	_threshold = 0.1

string	_layout = ""

string	_inference_device = "CPU"

uint32_t	_nireq = 0

string	_nstreams = ""

uint32_t	_nthreads = 0

uint32_t	_frames_processed = 0

int64_t	_frame_num = 0

bool	_dummy = false

int	_camera_device = 0

data_t	_fps = 25

string	_resolution_rgb = ""

int	_rgb_height

int	_rgb_width

vector< cv::Point2i >	_keypoints_list

vector< cv::Point3f >	_keypoints_cov

string	_model_file

string	_agent_id

VideoCapture	_cap

chrono::steady_clock::time_point	_start_time

ov::Core	_core

unique_ptr< ResultBase >	_result

OutputTransform	_output_transform

unique_ptr< ModelBase >	_model

AsyncPipeline *	_pipeline

vector< HumanPose >	_poses

Protected Attributes inherited from Source< json >
std::string	_blob_format

std::string	_error

Additional Inherited Members
Static Public Member Functions inherited from Source< json >
static const std::string	server_name ()

Public Attributes inherited from Source< json >
bool	dummy

Static Public Attributes inherited from Source< json >
static const int	version

Detailed Description

Skeletonizer3D is a plugin that computes the 3D skeleton of a human body from a depth map.

Member Function Documentation

◆ acquire_frame()

return_type Skeletonizer3D::acquire_frame ( bool dummy = false )

inline

Acquire a frame from a camera device. Camera ID is defined in the parameters list.

The acquired frame is stored in the #_k4a_rgbd, _rgbd and _rgb attributes.

See also: set_params

Author: Nicola

Returns: result status ad defined in return_type

◆ consistency_check()

return_type Skeletonizer3D::consistency_check ( bool debug = false )

inline

Consistency check of the 3D skeleton according to human physiology.

Authors: Marco, Matteo

Returns: result status ad defined in return_type

◆ coordinate_transfrom()

return_type Skeletonizer3D::coordinate_transfrom ( bool debug = false )

inline

Transform the 3D skeleton coordinates in the global reference frame.

Use the extrinsic camera parameters to transorm the 3D skeleton coordinates just before sending them as plugin output.

Returns: return_type

◆ cov3D_compute()

return_type Skeletonizer3D::cov3D_compute ( bool debug = false )

inline

Compute the 3D covariance matrix.

Compute the 3D covariance matrix. Two possible cases:

one Azure camera: use the 3D to uncertainty in the view axis, use the 2D image to uncertainty in the projection plane
one RGB camera: calculates a 3D ellipsoid based on the 2D covariance plus the "reasonable" depth range as a third azis (direction of view)

Author: Alessandro

Returns: result status ad defined in return_type

◆ get_output()

return_type Skeletonizer3D::get_output	(	json *	out,
		vector< unsigned char > *	blob = `nullptr`
	)

inlineoverride

Get the output of the plugin.

This method acquires a new image and computes the skeleton from it.

Author: Paolo

Parameters

out	The output of the plugin as JSON
blob	Possible additional binary data

Returns: return_type

◆ hessian_compute()

return_type Skeletonizer3D::hessian_compute ( bool debug = false )

inline

Compute the hessians for joints.

Compute the hessians for joints on the RGB frame based on the _heatmaps field.

Author: Alessandro

Returns: result status ad defined in return_type

◆ info()

map<string, string> Skeletonizer3D::info ( )

inlineoverridevirtual

Provide further info to Miroscic agent.

Provide the Miroscic agent loading this plugin with further info to be printed after initialization

Returns: a map with the information of the plugin

Implements Source< json >.

◆ kind()

string Skeletonizer3D::kind ( )

inlineoverridevirtual

The plugin identifier.

Author: Paolo

Returns: a string with plugin kind

Implements Source< json >.

◆ point_cloud_filter()

return_type Skeletonizer3D::point_cloud_filter ( bool debug = false )

inline

Remove unnecessary points from the point cloud.

Make the point cloud lighter by removing unnecessary points, so that it can be sent to the database via network

Author: Nicola

Returns: result status ad defined in return_type

◆ set_params()

void Skeletonizer3D::set_params ( void * params )

inlineoverridevirtual

Set the parameters of the plugin.

The parameters are stored in the _params attribute. This method shall be called imediately after the plugin is instantiated

Author: Paolo

Parameters

params

Reimplemented from Source< json >.

◆ skeleton_from_depth_compute()

return_type Skeletonizer3D::skeleton_from_depth_compute ( bool debug = false )

inline

Compute the skeleton from the depth map.

Compute the skeleton from the depth map. The resulting skeleton is stored in _skeleton3D attribute as a map of 3D points.

Author: Nicola

Returns: result status ad defined in return_type

◆ skeleton_from_rgb_compute()

return_type Skeletonizer3D::skeleton_from_rgb_compute ( bool debug = false )

inline

Compute the skeleton from RGB images only.

Compute the skeleton from RGB images only. On success, the field _skeleton2D is updated (as a map of 2D points). Also, the field _heatmaps is updated with the joints heatmaps (one per joint).

There is a configuration flag for optionally skipping this branch on Azure agents.

Author: Alessandro

Returns: result status ad defined in return_type

Member Data Documentation

◆ _cov3D

Mat Skeletonizer3D::_cov3D

protected

the 3D covariance matrix

◆ _cov3D_adj

Mat Skeletonizer3D::_cov3D_adj

protected

the adjusted 3D covariance matrix

◆ _heatmaps

vector<Mat> Skeletonizer3D::_heatmaps

protected

the joints heatmaps

◆ _inference_device

string Skeletonizer3D::_inference_device = "CPU"

protected

computation device

◆ _layout

string Skeletonizer3D::_layout = ""

protected

inputs layouts (NCHW, NHWC)

◆ _nireq

uint32_t Skeletonizer3D::_nireq = 0

protected

number of infer requests

◆ _nstreams

string Skeletonizer3D::_nstreams = ""

protected

number of streams

◆ _nthreads

uint32_t Skeletonizer3D::_nthreads = 0

protected

number of CPU threads

◆ _params

json Skeletonizer3D::_params

protected

the parameters of the plugin

◆ _point_cloud

Mat Skeletonizer3D::_point_cloud

protected

the filtered body point cloud

◆ _poses

vector<HumanPose> Skeletonizer3D::_poses

protected

contains all the keypoints of all identified people

◆ _rgb

Mat Skeletonizer3D::_rgb

protected

the last RGB frame

◆ _rgb_height

int Skeletonizer3D::_rgb_height

protected

image size rows

◆ _rgb_width

int Skeletonizer3D::_rgb_width

protected

image size cols

◆ _rgbd

Mat Skeletonizer3D::_rgbd

protected

the last RGBD frame

◆ _skeleton2D

map<string, vector<unsigned char> > Skeletonizer3D::_skeleton2D

protected

the skeleton from 2D cameras only

◆ _skeleton3D

map<string, vector<unsigned char> > Skeletonizer3D::_skeleton3D

protected

the skeleton from 3D cameras only

◆ _threshold

data_t Skeletonizer3D::_threshold = 0.1

protected

probability threshold

◆ _tsize

uint32_t Skeletonizer3D::_tsize = 0

protected

target size

The documentation for this class was generated from the following file:

src/plugin/skeletonizer3D.cpp

Public Member Functions

Protected Attributes

Additional Inherited Members

Detailed Description

Member Function Documentation

◆ acquire_frame()

◆ consistency_check()

◆ coordinate_transfrom()

◆ cov3D_compute()

◆ get_output()

◆ hessian_compute()

◆ info()

◆ kind()

◆ point_cloud_filter()

◆ set_params()

◆ skeleton_from_depth_compute()

◆ skeleton_from_rgb_compute()

Member Data Documentation

◆ _cov3D

◆ _cov3D_adj

◆ _heatmaps

◆ _inference_device

◆ _layout

◆ _nireq

◆ _nstreams

◆ _nthreads

◆ _params

◆ _point_cloud

◆ _poses

◆ _rgb

◆ _rgb_height

◆ _rgb_width

◆ _rgbd

◆ _skeleton2D

◆ _skeleton3D

◆ _threshold

◆ _tsize