neurapy_ai.clients package

neurapy_ai.clients.audio_output_client module

Client to play audio files.

class neurapy_ai.clients.audio_output_client.AudioOutputClient

Bases: BaseAiClient

Client to play audio files.

Audio can pe played on the chosen output device for sound feedback or user interaction.

beep_negative()

Play negative beep audio on the robot.

beep_neutral()

Play neutral beep audio on the robot.

beep_neutral_double()

Play neutral double beep audio on the robot.

beep_positive()

Play positive beep audio on the robot.

play_audio()

Play an audio file locally or in the robot system.

stop_all_audio()

Stop playing all audio in the robot system.

test_audio()

Play test audio on the robot.

beep_negative(blocking: bool = True) → ReturnCode

Play negative beep audio on the robot.

Parameters:

blocking (bool, optional) – Blocks while audio is playing. Default is True, to block execution until audio playing ends

Returns:

Return code with message

Return type:

ReturnCode

beep_neutral(blocking: bool = True) → ReturnCode

Play neutral beep audio on the robot.

Parameters:

blocking (bool, optional) – Blocks while audio is playing. Default is True, to block execution until audio playing ends

Returns:

Return code with message

Return type:

ReturnCode

beep_neutral_double(blocking: bool = True) → ReturnCode

Play neutral double beep audio on the robot.

Parameters:

blocking (bool, optional) – Blocks while audio is playing. Default is True, to block execution until audio playing ends

Returns:

Return code with message

Return type:

ReturnCode

beep_positive(blocking: bool = True) → ReturnCode

Play positive beep audio on the robot.

Parameters:

blocking (bool, optional) – Blocks while audio is playing. Default is True, to block execution until audio playing ends

Returns:

Return code with message

Return type:

ReturnCode

play_audio(audio_file: str, blocking: bool = True, target: str = 'robot') → ReturnCode

Play an audio file based on system choice.

Parameters:

  • audio_file (str) – Path to an audio file

  • blocking (bool, optional) – Blocks while audio is playing. Default is True, to block execution until audio playing ends

  • target (str, optional) – ‘local’ to play audio locally or ‘robot’ to play audio on the robot, by default robot

Returns:

Return code with message

Return type:

ReturnCode

stop_all_audio() → ReturnCode

Stop playing all audio in the robot system.

test_audio(blocking: bool = True) → ReturnCode

Play test audio on the robot.

Parameters:

blocking (bool, optional) – Blocks while audio is playing. Default is True, to block execution until audio playing ends

Returns:

Return code with message

Return type:

ReturnCode

neurapy_ai.clients.autonomous_pick_client module

Client to generate grasps on any or known objects.

class neurapy_ai.clients.autonomous_pick_client.AutonomousPickClient

Bases: BaseAiClient

Client to generate grasps on any or known objects.

This does not require pre-recording of grasp poses and only optionally requires prior training an object segmentation model, which is when you are selective with the object that needs to be picked.

reset_GGC_parameters()

Reset GraspGeneratorClient parameters to defaults.

start_detection()

Start grasp detection.

get_picks()

Wait for the detection to complete and get a list of grasps.

get_picks() → Tuple[ReturnCode, List[Pick]]

Wait for the detection to complete and get a list of grasps.

Please call start_detection before get_picks to start the detection thread.

Returns:

  • ReturnCode – Return code

  • List[Pick] – Detected grasps as Pick objects

reset_GGC_parameters()

Reset GraspGeneratorClient parameters to defaults.

start_detection(object_names: List[str], workspace_name: str, gripper_name: str, bin_name: str = '') → ReturnCode

Start grasp detection.

Parameters:

  • object_names (List[str]) – Objects that should be detected. If empty, grasps will be generated based on the point cloud in the region of interest

  • workspace_name (str) – Name of the workspace that should be used for detection

  • gripper_name (str) – Name of the gripper that should be used

  • bin_name (str, optional) – Name of the bin that should be used. If empty, bin detection will not be used, by default “”

Returns:

Return code

Return type:

ReturnCode

Raises:

ValueError – If gripper_name is empty

neurapy_ai.clients.base_ai_client module

Interface class for all AI clients.

class neurapy_ai.clients.base_ai_client.BaseAiClient(node_name: str, service_proxy: List[ServiceProxy], action_clients: List[SimpleActionClient], has_parameters: bool = True, log_level: int = 20)

Bases: object

Interface class for all AI clients.

reset_parameters() → None

Reset this node’s parameters to its defaults.

Raises:

SystemError – If the node has no parameters to reset

set_parameter(param_name: str, value: Any) → None

Set a single parameter of this node.

Parameters:

  • param_name (str) – Name of the parameter

  • value (Any) – New value for the parameter

Raises:

  • SystemError – If the node has no parameters

  • KeyError – If param_name is not a parameter of this node

set_parameters(params_dict: Dict) → None

Set multiple parameters of this node from a dictionary.

Parameters:

params_dict (Dict) – Dictionary with parameter names and values to set.

Raises:

  • SystemError – If the node has no parameters

  • KeyError – If one of the parameter names in params_dict is not a parameter of this node

neurapy_ai.clients.data_based_pick_client module

Client to generate grasps on known objects.

class neurapy_ai.clients.data_based_pick_client.DataBasedPickClient

Bases: BaseAiClient

Client to generate grasps on known objects.

Generate grasp poses for known objects based on their pre-recorded grasp poses.

Requires training a segmentation model and pose estimation model for an object, and also grasp pose recording.

reset_GGC_parameters()

Reset GraspGeneratorClient parameters to defaults.

start_detection()

Start grasp detection.

get_picks()

Wait for the detection to complete and get a list of grasps.

get_picks() → Tuple[ReturnCode, List[Pick]]

Wait for the detection to complete and get a list of grasps.

Please call start_detection before get_picks to start the detection thread.

Returns:

  • ReturnCode – Return code

  • List[Pick] – Detected grasps as Pick objects.

reset_GGC_parameters()

Reset GraspGeneratorClient parameters to defaults.

start_detection(object_names: List[str], workspace_name: str, gripper_name: str, bin_name: str | None = '') → ReturnCode

Start grasp detection.

Parameters:

  • object_names (List[str]) – Objects that should be detected. If empty, grasps will be generated based on the point cloud in the region of interest

  • workspace_name (str) – Name of the workspace that should be used for detection

  • gripper_name (str) – Name of the gripper that should be used

  • bin_name (str, optional) – Name of the bin that should be used. If empty, bin detection will not be used, by default “”

Returns:

Return code

Return type:

ReturnCode

Raises:

ValueError – If gripper_name is empty

neurapy_ai.clients.database_client module

Client to read the robot database.

class neurapy_ai.clients.database_client.DatabaseClient

Bases: BaseAiClient

Client to read the robot database.

The database contains robot and workspace related information that are saved by the user via the robot HMI.

get_pose()

Read robot position (point in database) as a robot pose.

get_joint_position()

Read robot position (point in database) as joint states.

get_workspace()

Read workspace stored in the database.

get_end_effector()

Read end effector stored in the database.

update_database()

Update information on the database server memory.

get_end_effector(end_effector_name: str = '') → Tuple[ReturnCode, EndEffector]

Read end effector stored in the database.

Parameters:

end_effector_name (str) – The name of a registered end effector. Defaults to “”, to get the currently selected end effector.

Returns:

  • ReturnCode – Return code

  • EndEffector – The end effector

get_joint_positions(point_name: str) → Tuple[ReturnCode, List[float]]

Read robot position (point in database) as joint states.

Parameters:

point_name (str) – The name of a registered point

Returns:

  • ReturnCode – Return code

  • List[float] – Joint states

get_pose(point_name: str) → Tuple[ReturnCode, Pose]

Read robot position (point in database) as a robot pose.

Parameters:

point_name (str) – The name of a registered point

Returns:

  • ReturnCode – Return code

  • Pose – Pose in robot coordinate frame

get_workspace(workspace_name: str) → Tuple[ReturnCode, Workspace]

Read workspace stored in the database.

Parameters:

workspace_name (str) – The name of a registered workspace

Returns:

  • ReturnCode – Return code

  • Workspace – The workspace

update_database() → ReturnCode

Update information on the database server memory.

Returns:

Return code

Return type:

ReturnCode

neurapy_ai.clients.instance_segmentation_client module

neurapy_ai.clients.marker_detection_client module

neurapy_ai.clients.pointing_pick_client module

Client to generate grasps for objects that a human operator points to.

class neurapy_ai.clients.pointing_pick_client.PointingPickClient

Bases: BaseAiClient

Client to generate grasps for objects that a human operator points to.

The operator would be prompted to point their forefinger to the object within the chosen workspace for which grasps are to be generated.

start_detection()

Start grasp detection.

get_picks()

Wait for the detection to complete and get a list of grasps.

stop()

Stop detection.

get_picks() → Tuple[ReturnCode, List[Pick]]

Wait for the detection to complete and get a list of grasps.

Please call start_detection before get_picks to start the detection process.

Returns:

  • ReturnCode – Return code

  • List[Pick] – Detected grasps as Pick objects

start_detection(workspace_name: str, gripper_name: str, object_name: str | None = '') → ReturnCode

Start grasp detection.

Parameters:

  • workspace_name (str) – Name of the workspace that should be used for detection.

  • gripper_name (str) – Name of the gripper that should be used.

  • object_name (str) – The name of the object that is selected. If empty, a random object will be assumed, by default “”.

Returns:

Return code

Return type:

ReturnCode

Raises:

ValueError – If gripper_name is empty

stop() → None

Stop detection.

neurapy_ai.clients.pointing_point_detection_client module

Client to detect the location that a human operator points to.

class neurapy_ai.clients.pointing_point_detection_client.PointingPointDetectionClient

Bases: BaseAiClient

Client to detect the location that a human operator points to.

The operator is prompted to point with their forefinger to a desired location on a workspace surface. Their hand should be in the camera view and the projected pointed location on the surface is detected.

start_detection()

Start detecting the pointed location.

get_point()

Wait for the detection to complete and get the desired location.

stop()

Stop detection.

get_point() → Tuple[ReturnCode, Pose]

Wait for the detection to complete and get the desired location.

Please call start_detection before get_point to start the detection thread.

Returns:

  • ReturnCode – Return code

  • Pose – The detected point relative to the robot

start_detection(workspace_name: str) → ReturnCode

Start detecting the pointed location.

Th ehuman operator will be prompted to point their forefinger towards the desired location on the chosen workspace surface. Their hand should be within the camera view.

Parameters:

workspace_name (str) – The name of a registered workspace

Returns:

Return code

Return type:

ReturnCode

stop() → None

Stop detection.

neurapy_ai.clients.pose_estimation_client module

Client to estimate object poses or train a new pose estimation model.

class neurapy_ai.clients.pose_estimation_client.PoseEstimationClient(model_name: str | None = '', model_version: str | None = 'newest', refinement: bool | None = True)

Bases: BaseAiClient

Client to estimate object poses or train a new pose estimation model.

Object poses could be estimation using a trained pose estimation model or a new pose estimation model can be trained on a neura-style dataset.

set_model()

Load a new pose estimation model.

get_model()

Get the currently loaded pose estimation model.

set_method()

Load a new pose estimation method.

get_method()

Get the currently used pose estimation method.

get_poses()

Get pose estimates of chosen objects.

multiview_pose_estimation_register_view()

Register the current camera view for multiview pose estimation.

multiview_pose_estimation_clear_views()

Clear the stored camera views for multiview pose estimation.

multiview_pose_estimation_get_poses()

Get pose estimates of chosen objects from the registered scene views.

Notes

  • See neurapy_ai.clients.DataGenerationClient to generate your own

neura-style dataset.

get_method() → Tuple[str, ReturnCode]

Get the current pose estimation method.

Returns:

  • str – Name of the current pose estimation method

  • ReturnCode – Numerical return code with error message

get_model() → Tuple[str, str, ReturnCode]

Get the currently loaded pose estimation model.

Returns:

  • str – Name of the loaded model

  • str – Version of the loaded model

  • ReturnCode – Numerical return code with error message

get_poses(class_names: Sequence[str] | None = (), target_frame: str | None = '') → Tuple[ReturnCode, List[DetectedObject]]

Get pose estimates of chosen objects.

Get all object pose estimates or a filtered result based on given object names that match.

Parameters:

  • class_names (Sequence[str], optional) – A list of object class names, by default () (all classes).

  • target_frame (str, optional) – Name of a coordinate frame in which the estimated poses should be returned, by default ‘’ (the frame of the camera).

Returns:

  • ReturnCode – Numerical return code with error message

  • List[DetectedObject] – A list of detected objects

get_poses_from_image(color_image: ndarray[Any, dtype[uint8]], depth_image: ndarray[Any, dtype[uint16]], camera_intrinsics: ndarray[Any, dtype[float32]], class_names: List[str] | None = [], target_frame: str | None = '', target_to_camera: ndarray[Any, dtype[float32]] = array([[1., 0., 0., 0.], [0., 1., 0., 0.], [0., 0., 1., 0.], [0., 0., 0., 1.]]), timestamp: float | None = None) → Tuple[ReturnCode, List[DetectedObject]]

Get pose estimates of chosen objects in the given scene.

Get all object pose estimates or a filtered result based on given object names that match.

Parameters:

  • color_image (np.array (shape [height, width, 3], dtype uint8)) – The input color image in rgb channel order and 0-255 value range

  • depth_image (np.array (shape [height, width, 1], dtype uint16)) – The input depth image with depth measurements in meter

  • camera_intrinsics (np.array (shape [3, 3], dtype=float32)) – The camera intrinsics as 3 x 3 matrix

  • class_names (List[str], optional) – A list of object class names, by default [] (all classes).

  • target_frame (str, optional) – Name of a coordinate frame in which the estimated poses should be returned, by default ‘’ (the frame of the camera).

  • target_to_camera (np.array (shape [4, 4], dtype float32)) – Transformation between target frame and camera frame, by default np.eye(4,4).

  • timestamp (float, optional) – Timestamp (in seconds) for the given scene, by default None (current time)

Returns:

  • ReturnCode – Numerical return code with error message

  • List[DetectedObject] – A list of detected objects

multiview_pose_estimation_clear_views() → ReturnCode

Clear the stored camera views for multiview pose estimation.

Returns:

Numerical return code with error message

Return type:

ReturnCode

multiview_pose_estimation_get_poses(class_names: Sequence[str] | None = (), target_frame: str | None = '') → Tuple[ReturnCode, List[DetectedObject]]

Get pose estimates of chosen objects from the registered scene views.

Get pose estimates of all objects or a filtered result based on given object names that match from the registered views of the scene.

Parameters:

  • class_names (Sequence[str], optional) – A list of object class names, by default () (all classes).

  • target_frame (str, optional) – Name of a coordinate frame in which the estimated poses should be returned, by default ‘’ (the frame of the camera).

Returns:

  • ReturnCode – Numerical return code with error message

  • List[DetectedObject] – List of detected objects

multiview_pose_estimation_register_view(class_names: Sequence[str] | None = ()) → Tuple[ReturnCode, List[DetectedObject]]

Register the current camera view for multiview pose estimation.

Parameters:

class_names (Sequence[str], optional) – A list of object class names, by default () (all classes).

Returns:

  • ReturnCode – Numerical return code with error message

  • List[DetectedObject] – List of detected objects

set_method(method: str) → ReturnCode

Load a new pose estimation method.

This will clear all registered views for multiview pose estimation and reset the currently loaded model.

Parameters:

method (str) – Name of the pose estimation method to load

Returns:

Numerical return code with error message

Return type:

ReturnCode

set_model(model_name: str, model_version: str | None = 'newest') → ReturnCode

Load a new pose estimation model.

Parameters:

  • model_name (str) – Name of the pose estimation model to load

  • model_version (str, optional) – Version of the pose estimation model that should be loadad, by default “newest”.

Returns:

Numerical return code with error message

Return type:

ReturnCode

neurapy_ai.clients.robot_scan_client module

Client to perform a scan of the robot’s environment.

class neurapy_ai.clients.robot_scan_client.RobotScanClient

Bases: BaseAiClient

Client to perform a scan of the robot’s environment.

start_scanning()

Start scanning process with set of input.

stop()

Stop scanning process and get scene as a mesh.

resume()

Resume the scanning process after pausing.

pause()

Pause the scanning process.

hard_stop()

Force stop the scanning process even during robot motion.

hard_stop() → ReturnCode

Force stop the scanning process even during robot motion.

Returns:

Numerical return code with error message

Return type:

ReturnCode

pause() → ReturnCode

Pause the scanning process.

Returns:

Numerical return code with error message

Return type:

ReturnCode

resume() → ReturnCode

Resume the scanning process after pausing.

Returns:

Numerical return code with error message

Return type:

ReturnCode

start_scanning(workspace_name: str, file_name: str, cam_pose_type: int, scan_type: int, scene_id: int | None = 0, do_texture_mapping: bool | None = False, data_path: str | None = '') → ReturnCode

Start scanning process with set of input.

Parameters:

  • workspace_name (str) – The name of available workspaces in the database

  • file_name (str) – The file name of output mesh from scan process

  • cam_pose_type (int) – type of method for estimating camera pose [0: use robot, 1: use marker, 2: use internal SLAM algorithm]

  • scan_type (int) – type of scan [0: object scanning, 1: workspace scanning, 2: environment scanning]

  • scene_id (int, optional) – id of the scene that robot scans, by default 0

  • do_texture_mapping (bool, optional) – boolean option to run texture mapping process after scanning, by default False

  • data_path (str, optional) – the directory to save scan output, by default “”

Returns:

Numerical return code with error message

Return type:

ReturnCode

stop() → Tuple[ReturnCode, Mesh]

Stop scanning process and get scene as a mesh.

User should call the stop function when the motion of robot stops. After stop, the client publish the scene pointcloud to octomap server

Returns:

  • ReturnCode – Numerical return code with error message

  • neura_vision_tools_msgs.msg.Mesh – The mesh of the scene with respect to robot link

neurapy_ai.clients.voice_control_client module

Client for voice-based interaction.

class neurapy_ai.clients.voice_control_client.VoiceControlClient

Bases: BaseAiClient

Client for voice-based interaction.

get_command()

Get the voice command. Command has to be triggered by “Hey Maira”.

get_command_with_trigger()

Get the results of command defined as an argument to this function.

set_command()

Set detecting command and trigger with “Hey Maira” (non-blocking).

get_last_command()

Get last command.

get_return_command()

Start continuous detection of the voice commands (blocking).

finish()

Check if a command has executed.

finish() → bool

Check if a command has been executed.

Returns:

True if command has been executed, else False

Return type:

bool

get_command(timeout: float | None = None) → Tuple[ReturnCode, str]

Get the voice command. Command has to be triggered by “Hey Maira”.

Parameters:

timeout (float, optional) – if specified funtion will terminate after that time even if no command was triggered, by default None - no timeout will be used

Returns:

  • ReturnCode – Return code

  • str – Resulting command (without actual command)

get_command_with_trigger(command: str) → Tuple[ReturnCode, str]

Get the results of command defined as an argument to this function

Parameters:

  • command (str) – command that should be started. If command is not defined and then return code is returned

  • similar (bool) – <description>

Returns:

  • ReturnCode – Return code

  • str – resulting command

get_last_command() → Tuple[ReturnCode, str]

Get last command.

If there were no command since the last call to this function, returns empty string. get_return_command() has to be started before, otherwise FUNCTION_NOT_INITIALIZED error code is returned.

Returns:

  • ReturnCode – Return code

  • str – The last command

get_return_command() → str

Start continuous detection of the voice commands (blocking).

Commands have to be started by “Hey Maira”. Commands can be read by calling get_last_command function.

Returns:

The last command

Return type:

str

set_command(command: str, similar: bool) → Tuple[bool, str]

Set detecting command and trigger with “Hey Maira” (non-blocking).

Parameters:

  • command (str) – Command that should be started.

  • similar (bool) – <description>

Returns:

  • bool – The return value, true if command setting success, false for otherwise.

  • str – Return message.

neurapy_ai.clients.data_generation_client module

Client for neura-style dataset generation.

class neurapy_ai.clients.data_generation_client.DataGenerationClient

Bases: BaseAiClient

Client for neura-style dataset generation.

Features include synthetic dataset generation which builds a scene with user-configured objects, with random background textures and distractor objects.

generate_synthetic_dataset()

Generate a synthetic dataset of images that is auto-annotated.

generate_synthetic_dataset(dataset_name: str, object_names: List[str], images_generated_count: int, camera_distance_range: Tuple[float, float], light_energy_range: Tuple[int, int]) → ReturnCode

Generate a synthetic dataset of images that is auto-annotated.

This dataset generaties requires that the objects of interest are available in a neura-style folder structure.

Parameters:

  • dataset_name (str) – Name of the dataset

  • object_names (List[str]) – Names of the user objects to include in the dataset

  • images_generated_count (int) – Number of images to generate and annotate.

  • camera_distance_range (Tuple[float, float]) – (Min,Max) camera distance in ‘m’, to the closest object of interest

  • light_energy_range (Tuple[int, int]) – (Min,Max) light energy in ‘Wm-2’ if the light type is SUN, otherwise ‘W’. Light type is randomly chosen between [POINT, SUN, SPOT, AREA]

Returns:

Numerical return code with error message

Return type:

ReturnCode

Raises:

KeyboardInterrupt – Received SIGINT signal during data generation, which stops the process