API Documentation

The Python API is a AI powered Python library to communicate with the robot. Basic tasks like planning motions and moving the robot are offered as well as more complex AI powered functionalities like picking, scanning, object segmentation and a lot more. Moreover utility functions for ROS message conversions, transformations, and return codes are provided.

The library includes two python packages:

neurapy_ai
neurapy_ai_utils

In the neurapy_ai package, different python clients and some utility functions can be found. The neurapy_ai_utils package contains the robot kinematics, the gripper interface and robot actions.

Robot Kinematics

Within the neurapy_ai_utils the robot kinematics are provided to handle kinematics-related operations like planning and executing motions and manage different end-effectors.

There are three types of robot kinematics:

DummyKinematics: This mode will print the movement command and its parameters.
MoveitKinematics: This mode will move the robot using Moveit. To use this, you need to start Moveit with the command: roslaunch maira_picker_moveit_config demo.launch
MairaKinematics: This mode will command either the real or simulated robot.

For a detailed class overview, click on the links provided below:

neurapy_ai_utils.robot.maira_kinematics
neurapy_ai_utils.robot.moveit_kinematics
neurapy_ai_utils.robot.dummy_kinematics

Move the robot using MairaKinematics

First import the he MairaKinematics class from the maira_kinematics module, which is part of the neurapy_ai_utils.robot package. Create an instance of the MairaKinematics class and assign it to the variable robot.

from neurapy_ai_utils.robot.maira_kinematics import MairaKinematics
robot = MairaKinematics()

To move the robot joints to a cartesian goal pose, use the move_joint_to_cartesian method and set a pose as goal.

goal_pose = [0.0,1.0,0.5,-math.pi,0.0,-math.pi]

print("Moving to cartesian goal...")
robot.move_joint_to_cartesian(goal_pose)

The robot can also be moved to a joint position.

joint_pose=[100*math.pi/180,25*math.pi/180,160*math.pi/180,-75*math.pi/180,9*math.pi/180,-78*math.pi/180,-96*math.pi/180]

print("Moving to joint goal...")
robot.move_joint_to_joint(joint_pose)

To get the current cartesian tcp pose or the current joint states, simply call the respective methods:

tcp_pose = robot.get_current_cartesian_tcp_pose()
print(f"Current tcp pose: {tcp_pose}")
joint_state = robot.get_current_joint_state()
print(f"Current joint state: {joint_state}")

Plan a motion using the various planning methods. Each planned motion is saved with an ID and can be executed by referencing this ID. To clear a planned motion, call its ID. Multiple IDs can be executed or deleted by inputing a list of IDs.

print("Plan a motion to cartesian goal ...")
success, id_motion1, joint_states = robot.plan_motion_joint_to_cartesian(goal_pose, speed=1, acc=1, reusable=True)
print("Execute a planned motion ...")
robot.execute([id_motion1], True)
print("Plan a motion to joint goal ...")
success, id_motion2, joint_states = robot.plan_motion_joint_to_joint(joint_pose, speed=10, reusable=True)
print("Execute a planned motion ...")
robot.execute([id_motion2], True)
print("Plan a linear movement ...")
success, id_motion3, joint_states = robot.plan_motion_linear(goal_pose, reusable=True)
print("Execute a planned motion ...")
robot.execute([id_motion3], True)
print("Clearing ids...")
robot.clear_ids([id_motion1, id_motion2, id_motion3])

Gripper Interface

Several gripper interfaces are provided within the neurapy_ai_utils.grippers package. Create an instance of the gripper class and call open or close to control the gripper.

from neurapy_ai_utils.grippers import Robotiq2F140Gripper

gripper = Robotiq2F140Gripper()

gripper.open()
gripper.close()

For more information, see the class documentation: neurapy_ai_utils.grippers.gripper_interface.

Clients

Audio Output Client

The Audio Output client enables users to play audio files either on the robot or a local speaker. Additionally, this client provides functionality for playing various types of audio feedback including beeps and custom audio files.

Input Parameters

audio_file_path: Path to your audio file to play (supports various audio formats like .mp3, .wav, etc.)
blocking_flag: Boolean flag to block robot motion during playing the audio (default: True)
target_device: String specifying the output device - either “robot” to use robot speaker, or “local” to use PC speaker (default: “robot”)

Class Definition

neurapy_ai.clients.audio_output_client

Methods

Method	Description
`beep_negative(blocking: bool = True) -> ReturnCode`	Play a negative feedback beep sound on the robot.
`beep_neutral(blocking: bool = True) -> ReturnCode`	Play a neutral feedback beep sound on the robot.
`beep_neutral_double(blocking: bool = True) -> ReturnCode`	Play a double neutral feedback beep sound on the robot.
`beep_positive(blocking: bool = True) -> ReturnCode`	Play a positive feedback beep sound on the robot.
`play_audio(audio_file: str, blocking: bool = True, target: str = "robot") -> ReturnCode`	Play an audio file on either the robot or local system.
`stop_all_audio() -> ReturnCode`	Stop playing all audio in the robot system (publishes stop command).
`test_audio(blocking: bool = True) -> ReturnCode`	Play a test audio sound on the robot.

Example

from neurapy_ai.clients.audio_output_client import AudioOutputClient
from neurapy_ai.utils.return_codes import ReturnCodes

# Initialize Audio Client
aoc_ = AudioOutputClient()

# Example 1: Play audio on robot
audio_file_path = "/usr/share/sounds/alsa/Front_Right.wav"
blocking_ = True
target_ = "robot"
try:
    ret_ = aoc_.play_audio(audio_file_path, blocking_, target_)
    if ret_.value != ReturnCodes.SUCCESS:
        print(ret_.message)
except Exception as e:
    print(e)

# Example 2: Play feedback beep
try:
    # Play positive feedback beep
    ret_ = aoc_.beep_positive(blocking=True)
    if ret_.value != ReturnCodes.SUCCESS:
        print(ret_.message)
except Exception as e:
    print(e)

Notes

When playing audio locally, ensure the audio file exists on the local machine
The blocking parameter determines whether the execution should wait for audio playback to complete
For robot audio playback, the file should be accessible from the robot’s file system
The client uses ROS services for robot audio playback and sounddevice for local playback

Autonomous Pick Client

The autonomous pick client generates grasps for known and unknown CAD models. This client does not require pre-recording of grasp poses and only optionally requires prior training of an object segmentation model when selective object picking is needed.

Parameters

object_names: List of object CAD models to be detected. If empty, grasps will be generated based on the point cloud in the region of interest.
workspace_name: Name of the recorded workspace to be used for detection.
gripper_name: Name of the gripper to be used (must match the name as shown in GUI: Settings –> Tools).
bin_name: Optional name of the bin to be used. If empty, bin detection will not be used.

Class Definition

neurapy_ai.clients.autonomous_pick_client

Methods

Method	Description
`reset_GGC_parameters()`	Reset GraspGeneratorClient parameters to defaults.
`start_detection(object_names: List[str], workspace_name: str, gripper_name: str, bin_name: str = "") -> ReturnCode`	Start grasp detection (acquires vision data and triggers grasp generation).
`get_picks() -> Tuple[ReturnCode, List[Pick]]`	Wait for the detection to complete and return the generated grasps.

Default Parameters

The client uses the following default parameters for grasp generation:

Maximum pick attempts per capture: 1
Collision checking with workspace: Enabled
Pre-grasp distance: 0.1
Post-grasp distance: 0.15
Maximum number of grasps: 50
Collision space padding: 0.005
End effector type: Two-fingered (0)

Example

from neurapy_ai.clients.autonomous_pick_client import AutonomousPickClient
from neurapy_ai.utils.return_codes import ReturnCodes

# Initialize Autonomous Pick Client
apc = AutonomousPickClient()

# Define detection parameters
object_names = []  # Empty list for general grasp detection
workspace_name = "workspace_3"  # Name of recorded workspace
gripper_name = "RobotiQ_140"  # Must match GUI settings
bin_name = ""  # Optional bin name

try:
    # Start grasp detection
    ret_detection = apc.start_detection(
        object_names,
        workspace_name,
        gripper_name,
        bin_name,
    )

    # Check detection result
    if ret_detection.value != ReturnCodes.SUCCESS:
        print(ret_detection.message)
    else:
        # Get generated grasps
        ret_picks, picks = apc.get_picks()
        if ret_picks.value != ReturnCodes.SUCCESS:
            print(ret_picks.message)
        else:
            # Process generated grasps
            for pick in picks:
                print(pick)

except Exception as e:
    print(e)

Notes

The client uses vision data (point cloud, RGB, depth) for grasp generation
Bin detection is optional and can be enabled through configuration
Object segmentation is only performed when object_names is not empty
The client provides audio feedback for detection failures
Grasp generation parameters can be customized using reset_GGC_parameters()

Base AI Client

The Base AI Client serves as the interface class for all AI clients in the system. It provides common functionality for parameter management, service and action client handling, and logging configuration.

Parameters

node_name: Name of the ROS node to connect to
service_proxy: List of service proxies that the node publishes
action_clients: List of action clients that the node publishes
has_parameters: Flag indicating whether the node has reconfiguration parameters (default: True)
log_level: Logging level for the client (default: 20/INFO). Options: * 50 (CRITICAL) * 40 (ERROR) * 30 (WARNING) * 20 (INFO) * 10 (DEBUG)

Class Definition

neurapy_ai.clients.base_ai_client

Methods

Method	Description
`reset_parameters() -> None`	Reset the node’s dynamic reconfigure parameters to their default values.
`set_parameters(params_dict: Dict) -> None`	Set multiple dynamic reconfigure parameters from a dictionary.
`set_parameter(param_name: str, value: Any) -> None`	Set a single dynamic reconfigure parameter value.

Example

from neurapy_ai.clients.base_ai_client import BaseAiClient
import rospy
from actionlib import SimpleActionClient

# Initialize base client
node_name = "example_node"
service_proxies = []  # List of service proxies
action_clients = []   # List of action clients

try:
    # Create base client with default parameters
    client = BaseAiClient(
        node_name=node_name,
        service_proxy=service_proxies,
        action_clients=action_clients,
        has_parameters=True,
        log_level=20  # INFO level
    )

    # Set a single parameter
    client.set_parameter("example_param", 42)

    # Set multiple parameters
    params = {
        "param1": 100,
        "param2": "value"
    }
    client.set_parameters(params)

    # Reset to default parameters
    client.reset_parameters()

except Exception as e:
    print(f"Error: {e}")

Notes

The client automatically checks for node existence and service/action availability
Logging is configured with a custom formatter (NeuraLogFormatter)
Dynamic parameter configuration is optional (controlled by has_parameters)
The client will create a unique ROS node if none exists
Service and action client connections are verified with timeouts
All parameter operations include error checking and logging

Bin Detection Client

The Bin Detection Client provides functionality to detect and track the pose of bins in the workspace. It supports both pre-recorded workspace-based detection and instance segmentation-based approaches.

Parameters

bin_name: Name of the bin to detect
workspace_name: Name of the workspace where the bin is located
bin_mesh: Optional triangle mesh message of the bin defined by NEURA Robotics. If not provided, the mesh file will be read based on the bin name
pointcloud_topic: Name of the pointcloud topic from 3D camera (default: “/camera/depth_registered/points”)
method: Detection method type (default: WITHOUT_PREPROCESSING): * WITHOUT_PREPROCESSING: Uses pre-recorded workspace * WITH_PREPROCESSING: Uses instance segmentation based approach

Class Definition

neurapy_ai.clients.bin_detection_client

Methods

Method	Description
`detect_bin_pose(bin_name: str, workspace_name: str, bin_mesh: Mesh = None, pointcloud_topic: Optional[str] = "/camera/depth_registered/points", method: Optional[int] = MethodType.WITHOUT_PREPROCESSING) -> Tuple[ReturnCode, ObjectWithPose]`	Detect bin pose from a point cloud topic (return code + detected bin pose).
`get_bin_pose(bin_name: str) -> Tuple[ReturnCode, ObjectWithPose]`	Retrieve the most recently detected pose for a bin (or the initial recorded pose if no detection was performed).

Example

from neurapy_ai.clients.bin_detection_client import BinDetectionClient
from neurapy_ai.utils.return_codes import ReturnCodes
from neura_bin_detection_msgs.msg import MethodType

# Initialize Bin Detection Client
bdc = BinDetectionClient()

# Define detection parameters
bin_name = "example_bin"
workspace_name = "example_workspace"
pointcloud_topic = '/camera/depth_registered/points'
method = MethodType.WITHOUT_PREPROCESSING

try:
    # Detect bin pose
    ret_detection, detected_bin_pose = bdc.detect_bin_pose(
        bin_name=bin_name,
        workspace_name=workspace_name,
        bin_mesh=None,  # Use default mesh
        pointcloud_topic=pointcloud_topic,
        method=method
    )

    if ret_detection.value != ReturnCodes.SUCCESS:
        print(ret_detection.message)
    else:
        print("Detected Pose:", detected_bin_pose)

    # Get previously detected pose
    ret_pose, bin_pose = bdc.get_bin_pose(bin_name=bin_name)
    if ret_pose.value != ReturnCodes.SUCCESS:
        print(ret_pose.message)
    else:
        print("Bin Pose:", bin_pose)

except Exception as e:
    print(e)

Notes

The client requires a running ROS node and access to point cloud data
Point cloud messages are expected to be in PointCloud2 format
Service calls have a 3-second timeout for point cloud messages
The client automatically handles mesh file loading if not provided
Detection results are cached internally for subsequent queries
The client provides detailed logging for service calls and errors
Both detection methods (WITH/WITHOUT_PREPROCESSING) are supported
The client inherits error handling and logging from BaseAiClient

Data Based Pick Client

The Data Based Pick Client generates grasps for known CAD models using pre-recorded grasp poses. This client requires training a segmentation model and pose estimation model for objects, along with grasp pose recording.

Parameters

object_names: List of object CAD models to be detected
workspace_name: Name of the recorded workspace to be used for detection
gripper_name: Name of the gripper to be used (must match the name as shown in GUI: Settings –> Tools)
bin_name: Optional name of the bin to be used. If empty, bin detection will not be used

Class Definition

neurapy_ai.clients.data_based_pick_client

Methods

Method	Description
`reset_GGC_parameters()`	Reset GraspGeneratorClient parameters to defaults.
`start_detection(object_names: List[str], workspace_name: str, gripper_name: str, bin_name: Optional[str] = "") -> ReturnCode`	Start grasp detection (acquires vision data, runs segmentation + pose estimation, then generates grasps).
`get_picks() -> Tuple[ReturnCode, List[Pick]]`	Wait for the detection to complete and return the generated grasps.

Default Parameters

The client uses the following default parameters for grasp generation:

Maximum pick attempts per capture: 1
Collision checking with workspace: Enabled
Pre-grasp distance: 0.1
Post-grasp distance: 0.15
Maximum number of grasps: 50
Collision space padding: 0.005
End effector type: Two-fingered (0)

Example

from neurapy_ai.clients.data_based_pick_client import DataBasedPickClient
from neurapy_ai.utils.return_codes import ReturnCodes

# Initialize Data Based Pick Client
dbpc = DataBasedPickClient()

# Define detection parameters
object_names = ["puzzle_half_trapezoid", "puzzle_cuboid_red"]
workspace_name = "workspace_3"  # Name of recorded workspace
gripper_name = "RobotiQ_140"   # Must match GUI settings
bin_name = ""                  # Optional bin name

try:
    # Start grasp detection
    ret_detection = dbpc.start_detection(
        object_names=object_names,
        workspace_name=workspace_name,
        gripper_name=gripper_name,
        bin_name=bin_name
    )

    if ret_detection.value != ReturnCodes.SUCCESS:
        print(ret_detection.message)
    else:
        # Get generated grasps
        ret_picks, picks = dbpc.get_picks()
        if ret_picks.value != ReturnCodes.SUCCESS:
            print(ret_picks.message)
        else:
            # Process generated grasps
            for pick in picks:
                print(pick)

except Exception as e:
    print(e)

Notes

The client requires trained segmentation and pose estimation models
Vision data (point cloud, RGB, depth) is used for object detection
Bin detection is optional and can be enabled through configuration
The client provides audio feedback for detection failures
Grasp generation parameters can be customized using reset_GGC_parameters()
The client uses TF for coordinate transformations
Instance segmentation and pose estimation are performed sequentially
Collision checking can be enabled/disabled through parameters
The client supports both two-fingered and suction end effectors

Data Generation Client

The Data Generation Client provides functionality to generate synthetic datasets using CAD models with configurable parameters for camera distance and lighting conditions.

Input Parameters

dataset_name: Name of the output dataset
object_names: List of CAD model names to be used in dataset generation. Models should be stored in HOME/data/object_perception/objects/user_objects
images_generated_count: Number of images to be generated
camera_distance_range: Tuple of (min, max) distance in meters between camera and objects
light_energy_range: Tuple of (min, max) light energy in Wm-2 (for SUN light type) or W (for other light types)

Class Definition

neurapy_ai.clients.data_generation_client

Methods

Method	Description
`generate_synthetic_dataset(dataset_name: str, object_names: List[str], images_generated_count: int, camera_distance_range: Tuple[float, float], light_energy_range: Tuple[int, int]) -> ReturnCode`	Generate a synthetic dataset with specified parameters.

Example

from neurapy_ai.clients.data_generation_client import DataGenerationClient
from neurapy_ai.utils.return_codes import ReturnCodes
import os

# Initialize Data Generation Client
dgc = DataGenerationClient()

# Configure dataset generation parameters
output_dataset_name = "synthetic_dataset_1"
object_names = ["puzzle_cuboid", "binzel_objects"]  # Objects as shown in AI Hub GUI
num_images = 10

# Camera distance range in meters
camera_distance_range = (0.2, 2.5)

# Light energy range in W.m^-2 (SUN) or W (other types)
light_energy_range = (100, 400)

try:
    # Generate the synthetic dataset
    ret_code = dgc.generate_synthetic_dataset(
        dataset_name=output_dataset_name,
        object_names=object_names,
        images_generated_count=num_images,
        camera_distance_range=camera_distance_range,
        light_energy_range=light_energy_range
    )

    if ret_code.value != ReturnCodes.SUCCESS:
        print(ret_code.message)
    else:
        print(f"Dataset {output_dataset_name} was generated successfully!")
        print(f"Output Dataset Directory: {os.getenv('HOME')}/data/object_perception/training_data/{output_dataset_name}")

except Exception as e:
    print(e)

Notes

The client uses ROS action client for dataset generation
Progress updates are provided during generation
Supports user interruption with SIGINT (Ctrl+C)
Light type is randomly chosen between POINT, SUN, SPOT, and AREA
Objects must be available in the neura-style folder structure
Generated datasets are stored in the object perception training data directory

Data Management Client

The Data Management Client provides functionality to query and retrieve information about known objects, segmentation models, pose estimation models, and datasets in the system.

Input Parameters

model_name: Name of the selected model
object_name: Name of the selected object
dataset_name: Name of the selected dataset

Class Definition

neurapy_ai.clients.data_management_client

Methods

Method	Description
`get_segmentation_model_info(model_name: str, method: str) -> Tuple[ReturnCode, Dict]`	Retrieve information about a given segmentation model (return code + model info dict).
`get_pose_estimation_model_info(model_name: str, method: str) -> Tuple[ReturnCode, Dict]`	Retrieve information about a given pose estimation model (return code + model info dict).
`get_object_info(object_name: str) -> Tuple[ReturnCode, Dict]`	Retrieve information about a given object including symmetry and size details.
`get_dataset_info(dataset_name: str) -> Tuple[ReturnCode, Dict]`	Retrieve information about a given dataset including size statistics.
`get_all_objects() -> Tuple[ReturnCode, List[str]]`	Return the names of all known objects.
`get_all_segmentation_models() -> Tuple[ReturnCode, List[str]]`	Return the names of all known segmentation models.
`get_all_pose_estimation_models() -> Tuple[ReturnCode, List[str]]`	Return the names of all known pose estimation models.
`get_all_datasets() -> Tuple[ReturnCode, List[str]]`	Return the names of all known datasets.
`get_segmentation_models_for_object(object_name: str) -> Tuple[ReturnCode, List[str]]`	Return the names of segmentation models that were trained to recognize the given object.
`get_pose_estimation_models_for_object(object_name: str) -> Tuple[ReturnCode, List[str]]`	Return the names of pose estimation models that were trained for localizing the given object.

Return Values

All methods return a tuple containing: * return_code: A ReturnCode object indicating success or failure * data: The requested information in dictionary format (for info methods) or list format (for listing methods)

Example

from neurapy_ai.clients.data_management_client import DataManagementClient
from neurapy_ai.utils.return_codes import ReturnCodes

# Initialize Data Management Client
dmc = DataManagementClient()

try:
    # List all known segmentation models
    ret_code, models = dmc.get_all_segmentation_models()
    if ret_code.value != ReturnCodes.SUCCESS:
        print(ret_code.message)
    print(f"There are {len(models)} segmentation models available:")
    for model in models:
        print(f"{model}")

    # Get information about a segmentation model
    model_name = "segmentation_model"
    method = "detectron2"
    ret_code, data = dmc.get_segmentation_model_info(model_name, method)
    if ret_code.value == ReturnCodes.SUCCESS:
        print(f"Model info: {data}")

    # List all known objects
    ret_code, objects = dmc.get_all_objects()
    if ret_code.value != ReturnCodes.SUCCESS:
        print(ret_code.message)
    print(f"There are {len(objects)} objects available.")

    # Get all segmentation models for an object
    object_name = "mouse"
    ret_code, models = dmc.get_segmentation_models_for_object(object_name)
    if ret_code.value != ReturnCodes.SUCCESS:
        print(ret_code.message)
    print(f"The object {object_name} is recognized by the following models:")
    for model in models:
        print(f"{model}")

except Exception as e:
    print(e)

Notes

The client uses ROS service proxies for communication
All methods handle service call failures gracefully
Object information includes symmetry details and size measurements
Dataset information includes statistics about training, validation, and test sets
Model information includes version history and associated datasets

Database Client

The Database Client provides functionality to read robot and workspace related information that are saved by the user via the robot HMI.

Input Parameters

workspace_name: Name of the registered workspace
end_effector_name: Name of the registered end effector (defaults to currently selected end effector)
point_name: Name of the registered point to get pose/joint states

Class Definition

neurapy_ai.clients.database_client

Methods

Method	Description
`get_workspace(workspace_name: str) -> Tuple[ReturnCode, Workspace]`	Read a workspace stored in the database.
`get_end_effector(end_effector_name: str = "") -> Tuple[ReturnCode, EndEffector]`	Read an end effector stored in the database (empty name returns the currently selected end effector).
`get_pose(point_name: str) -> Tuple[ReturnCode, Pose]`	Read a database point as a TCP pose in the robot coordinate frame.
`get_joint_positions(point_name: str) -> Tuple[ReturnCode, List[float]]`	Read a database point as joint positions.
`update_database() -> ReturnCode`	Update information on the database server memory.

Return Values

Most read operations return Tuple[ReturnCode, <data>].

ReturnCode: Numerical code and message indicating the operation result
Returned data (varies by method):
- Workspace: Workspace pose, frame, dimensions, lookat poses, type, mesh model, etc.
- EndEffector: Name, Neura-supported type name, and TCP pose
- Pose: TCP pose in robot coordinate frame
- List[float]: Joint positions

update_database() returns a single ReturnCode.

Example

from neurapy_ai.clients.database_client import DatabaseClient
from neurapy_ai.utils.return_codes import ReturnCodes

# Initialize Database Client
dbc = DatabaseClient()

try:
    # 1. Get workspace information
    workspace_name = "workspace_test"
    ret_code, workspace = dbc.get_workspace(workspace_name)
    if ret_code.value != ReturnCodes.SUCCESS:
        print(ret_code.message)
    print(f"Workspace {workspace.name} in {workspace.frame} frame has dimensions:")
    print(f"x: {workspace.len_x}, y: {workspace.len_y}, z: {workspace.len_z}")
    print(f"Workspace pose: {workspace.pose}")
    print(f"Lookat pose: {workspace.lookat_poses[0].pose}")
    print(f"Lookat joint states: {workspace.lookat_poses[0].joint_state}")

    # 2. Get end effector information
    gripper_name = "RobotiQ_140"  # Name from GUI: Settings --> Tools
    ret_code, end_effector = dbc.get_end_effector(gripper_name)
    if ret_code.value != ReturnCodes.SUCCESS:
        print(ret_code.message)
    print(f"Gripper {end_effector.name} has TCP pose: {end_effector.tcp_pose}")

    # 3. Get recorded point pose
    point_name = "P20"
    ret_code, tcp_pose = dbc.get_pose(point_name)
    if ret_code.value != ReturnCodes.SUCCESS:
        print(ret_code.message)
    print(f"Point {point_name} has pose: {tcp_pose}")

    # 4. Get recorded point joint positions
    ret_code, joint_positions = dbc.get_joint_positions(point_name)
    if ret_code.value != ReturnCodes.SUCCESS:
        print(ret_code.message)
    print(f"Point {point_name} has joint states: {joint_positions}")

except Exception as e:
    print(e)

Notes

Workspace types can be either “tabletop” or “bin”
End effector information includes Neura-supported type names
TCP poses are returned in robot coordinate frame
Database is automatically updated on client initialization
All methods handle failures gracefully

Instance Segmentation Client

The Instance Segmentation Client provides functionality to segment object instances or train instance segmentation models. It supports two main operation modes:

Training: Train an instance segmentation model using images from real or synthetic datasets
Inference: Use a trained model to segment images and localize instances of objects the model is trained to detect

Input Parameters

The parameters vary depending on the operation mode:

Training Mode

model_name: Name of the instance segmentation model to save
dataset_names: List of dataset names to use for training
dataset_types: List of dataset types (“real” or “synthetic”)
method: Training method to use (default: “detectron2”)
num_iterations: Number of training iterations (default: 1)
warmup_iterations: Steps to linearly increase learning rate (default: 0)
learning_rate: Base learning rate (default: 0.0003)
batch_size: Samples per batch (default: 1)
checkpoint_iterations: Iterations between checkpoints (default: 1)
filter_empty_images: Skip images without annotations (default: True)
pretrained_model: Name of model to use as base (default: “”)

Inference Mode

model_name: Name of the instance segmentation model to use
model_version: Version of the model (default: “newest”)
class_names: Optional list of object class names to detect

Class Definition

neurapy_ai.clients.instance_segmentation_client

Methods

Method	Description
`set_model(model_name: str, model_version: Optional[str] = "newest") -> ReturnCode`	Load a new instance segmentation model.
`get_model() -> Tuple[str, str, ReturnCode]`	Get the currently loaded model name + version (and return code).
`set_method(method: str) -> ReturnCode`	Load a new instance segmentation method.
`get_method() -> Tuple[str, ReturnCode]`	Get the current instance segmentation method (and return code).
`get_segmented_instances(class_names: Optional[Sequence[str]] = ()) -> Tuple[ReturnCode, List[SegmentedInstance], npt.NDArray[np.uint8], npt.NDArray[np.uint8]]`	Segment instances from the current scene (return code + instances + combined mask + input image).
`get_segmented_instances_from_image(color_image: npt.NDArray[np.uint8], class_names: Optional[Sequence[str]] = ()) -> Tuple[ReturnCode, List[SegmentedInstance], npt.NDArray[np.uint8]]`	Segment instances from a provided RGB image (return code + instances + combined mask).
`visualize_segmentation_result(input_image: npt.NDArray[np.uint8], segmented_instances: List[SegmentedInstance], segmentation_mask: npt.NDArray[np.uint8]) -> npt.NDArray[np.uint8]`	Visualize segmentation results with bounding boxes, masks, and class labels.
`train_segmentation_model(model_name: str, dataset_names: List[str], dataset_types: List[str], method: str = "detectron2", num_iterations: Optional[int] = 1, warmup_iterations: Optional[int] = 0, learning_rate: Optional[float] = 0.0003, batch_size: Optional[int] = 1, checkpoint_iterations: Optional[int] = 1, filter_empty_images: bool = True, pretrained_model: Optional[str] = "") -> ReturnCode`	Train a new segmentation model.

Return Values

Methods return different types of values depending on their purpose:

Model management methods return: - ReturnCode: Success/failure status with message - Model name and version (where applicable)
Segmentation methods return: - ReturnCode: Success/failure status - List of SegmentedInstance objects - Segmentation mask as numpy array - Input image (for scene segmentation)
Training method returns: - ReturnCode: Success/failure status with message

Example

Training a Model

from neurapy_ai.clients.instance_segmentation_client import InstanceSegmentationClient
from neurapy_ai.utils.return_codes import ReturnCodes

# Initialize client with training enabled
isc = InstanceSegmentationClient(training=True)

try:
    # Train a new model
    ret_code = isc.train_segmentation_model(
        model_name="my_model",
        dataset_names=["dataset1", "dataset2"],
        dataset_types=["real", "synthetic"],
        num_iterations=40,
        warmup_iterations=5,
        learning_rate=0.001,
        batch_size=2,
        checkpoint_iterations=10
    )

    if ret_code.value != ReturnCodes.SUCCESS:
        print(ret_code.message)
    else:
        print("Training completed successfully!")

except Exception as e:
    print(e)

Performing Inference

import cv2
from neurapy_ai.clients.instance_segmentation_client import InstanceSegmentationClient
from neurapy_ai.utils.return_codes import ReturnCodes

# Initialize client
isc = InstanceSegmentationClient("my_model")

try:
    # Load an image
    image = cv2.imread("test_image.jpg")

    # Get segmentation results
    ret_code, instances, mask = isc.get_segmented_instances_from_image(
        image,
        class_names=["object1", "object2"]
    )

    if ret_code.value != ReturnCodes.SUCCESS:
        print(ret_code.message)
    else:
        # Visualize results
        result_image = isc.visualize_segmentation_result(
            image, instances, mask
        )
        cv2.imshow("Segmentation Result", result_image)
        cv2.waitKey(0)

except Exception as e:
    print(e)

Notes

The client uses ROS service proxies for communication
Training requires the client to be initialized with training=True
Training can be interrupted with Ctrl+C
Instance segmentation results include: - Bounding boxes - Segmentation masks - Class names - Detection scores - Instance indices
The client supports both real and synthetic datasets
Visualization includes bounding boxes, masks, and class labels

Marker Detection Client

The Marker Detection Client provides functionality to detect and interpret various types of markers including ArUco, ChArUco, Chessboard, AprilTag, and QR codes.

Parameters

marker_type: Type of the marker to detect: * 0: ArUco * 1: ChArUco * 2: Chessboard
vertical_squares: Number of squares in the vertical direction
horizontal_squares: Number of squares in the horizontal direction
square_length: Length of a single square in meters
marker_dictionary: Dictionary of markers to use. Supported options: * DICT_4X4_250 * DICT_5X5_250 * DICT_6X6_250 * DICT_7X7_250 * DICT_ARUCO_ORIGINAL
marker_length: Length of the marker square including white borders (default: 0.03 meters)
marker_frame: Target parent coordinate frame for the detected marker pose. (default:”camera_color_optical_frame”)
timeout: Timeout in seconds to stop marker detection if no marker is found (default: 5)

Note

For ArUco markers, parameters vertical_squares and horizontal_squares are always set to 1
The Detected Marker Pose is with respect to the camera frame

Class Definition

neurapy_ai.clients.marker_detection_client

Methods

Method	Description
`get_detected_marker(marker_type: int, vertical_squares: int, horizontal_squares: int, square_length: float, marker_dictionary: str, marker_length: float, timeout: int = 5, marker_frame: str = "camera_color_optical_frame") -> Tuple[ReturnCode, Union[None, Marker]]`	Detect marker(s) and return their pose(s) in the requested frame.
`get_detected_markers(marker_dictionary: int, marker_length: float, marker_type: int = 1, vertical_squares: int = 9, horizontal_squares: int = 14, square_length: float = 0.04, timeout: int = 5, marker_frame: str = "camera_color_optical_frame") -> Tuple[ReturnCode, List[Marker]]`	Backwards-compatible wrapper around `get_detected_marker`.
`get_calibration_grid()`	Get pose of a standard calibration board (14x9). Returns ReturnCode and Pose object.
`activate_detection()`	Activate marker detection stream. Returns ReturnCode indicating success or failure.
`deactivate_detection()`	Deactivate marker detection stream. Returns ReturnCode indicating success or failure.

Example

from neurapy_ai.clients.marker_detection_client import MarkerDetectionClient
from neurapy_ai.utils.return_codes import ReturnCodes

# Initialize Marker Detection Client
mdc = MarkerDetectionClient()

# Define detection parameters
marker_type = 1  # ChArUco marker
vertical_squares = 14
horizontal_squares = 9
square_length = 0.04
marker_dictionary = "DICT_5X5_250"
marker_length = 0.03

try:
    # Detect marker
    ret_code, markers = mdc.get_detected_marker(
        marker_type=marker_type,
        vertical_squares=vertical_squares,
        horizontal_squares=horizontal_squares,
        square_length=square_length,
        marker_dictionary=marker_dictionary,
        marker_length=marker_length,
        timeout=5
    )

    # Deactivate detection stream
    mdc.deactivate_detection()

    if ret_code.value != ReturnCodes.SUCCESS:
        print(ret_code.message)
    elif markers is None:
        print("No marker was detected!")
    else:
        for marker in markers:
            print(f"Marker ID: {marker.id}")
            print(f"Marker Pose: {marker.pose}")

except Exception as e:
    print(e)

Notes

Camera calibration is required for accurate marker pose estimation
Service calls have a configurable timeout for marker detection
The client automatically manages detection streams
Detection results include marker IDs and poses in camera frame
The client supports multiple marker types and dictionaries
The client inherits error handling and logging from BaseAiClient
Marker poses are returned in the camera’s optical frame

Pointing Pick Client

The Pointing Pick Client provides functionality to generate grasps for objects that a human operator points to. The operator is prompted to point their forefinger to the object within the chosen workspace for which grasps are to be generated.

Parameters

workspace_name: Name of the recorded workspace to use for detection
gripper_name: Name of the gripper to use for grasping (must not be empty)
object_name: Name of the object to pick (if empty, a random object will be assumed)

Class Definition

neurapy_ai.clients.pointing_pick_client

Methods

Method	Description
`start_detection(workspace_name: str, gripper_name: str, object_name: Optional[str] = "") -> ReturnCode`	Start grasp detection process.
`get_picks() -> Tuple[ReturnCode, List[Pick]]`	Wait for detection to complete and return the generated grasp candidates.
`stop() -> None`	Stop the detection process (cancels active goals).

Example

from neurapy_ai.clients.pointing_pick_client import PointingPickClient
from neurapy_ai.utils.return_codes import ReturnCodes

# Initialize Pointing Pick Client
ppc = PointingPickClient()

# Define detection parameters
workspace_name = "workspace_3"  # Name of recorded workspace
gripper_name = "RobotiQ_140"   # Name of gripper from GUI settings
object_name = ""               # Empty for random object selection

try:
    # Start grasp detection
    ret_code = ppc.start_detection(
        workspace_name=workspace_name,
        gripper_name=gripper_name,
        object_name=object_name
    )

    if ret_code.value != ReturnCodes.SUCCESS:
        print(ret_code.message)
    else:
        # Get detected grasps
        ret_picks, picks = ppc.get_picks()
        if ret_picks.value != ReturnCodes.SUCCESS:
            print(ret_picks.message)
        else:
            for pick in picks:
                print(f"Grasp ID: {pick.grasp_id}")
                print(f"Grasp Quality: {pick.quality}")
                print(f"Object: {pick.object_with_pose.name}")
                print(f"Grasp Pose: {pick.approach_sequence.pick_pose.pose}")

    # Stop detection
    ppc.stop()

except Exception as e:
    print(e)

Notes

The operator must point with their index finger for a few seconds
The client provides audio feedback when the object is detected
Grasp detection has a 30-second timeout
The client automatically handles vision data acquisition
Grasp poses include pre-grasp, grasp, and post-grasp positions
The client supports multiple gripper types
Detection can be stopped at any time using the stop() method

Pointing Point Detection Client

The Pointing Point Detection Client provides functionality to detect the location that a human operator points to. The operator is prompted to point with their forefinger to a desired location on a workspace surface. Their hand should be in the camera view and the projected pointed location on the surface is detected.

Parameters

workspace_name: Name of the registered workspace to use for detection

Class Definition

neurapy_ai.clients.pointing_point_detection_client

Methods

Method	Description
`start_detection(workspace_name: str) -> ReturnCode`	Start detecting the pointed location.
`get_point() -> Tuple[ReturnCode, Pose]`	Wait for detection to complete and return the detected point as a pose.
`stop() -> None`	Stop the detection process (cancels active goals).

Example

from neurapy_ai.clients.pointing_point_detection_client import PointingPointDetectionClient
from neurapy_ai.utils.return_codes import ReturnCodes

# Initialize Pointing Point Detection Client
ppdc = PointingPointDetectionClient()

# Define detection parameters
workspace_name = "workspace_3"  # Name of registered workspace

try:
    # Start point detection
    ret_code = ppdc.start_detection(workspace_name)

    if ret_code.value != ReturnCodes.SUCCESS:
        print(ret_code.message)
    else:
        # Get detected point
        ret_point, pose = ppdc.get_point()
        if ret_point.value != ReturnCodes.SUCCESS:
            print(ret_point.message)
        else:
            print("Detected Point:")
            print(f"Translation: {pose.translation}")
            print(f"Orientation: {pose.orientation}")

    # Stop detection
    ppdc.stop()

except Exception as e:
    print(e)

Notes

The operator must point with their index finger within camera view
Detection has a 30-second timeout
The detected point is returned as a pose relative to the robot
The client automatically handles detection process
Detection can be stopped at any time using the stop() method
The client uses ROS action client for communication
The detected point is projected onto the workspace surface

Pose Estimation Client

The Pose Estimation Client provides functionality to estimate 6D poses of objects in the workspace. Object poses can be estimated using a trained pose estimation model or a new pose estimation model can be trained on a neura-style dataset.

Parameters

model_name: Name of the pose estimation model to use
model_version: Version of the pose estimation model (optional, defaults to newest)
class_names: List of object names to be detected
method: Pose estimation method to use. Currently supported methods: * neura_DLPE1 * neura_nonDLPE1 * neura_nonDLPE2
target_frame: Reference frame where the pose will be calculated

Class Definition

neurapy_ai.clients.pose_estimation_client

Methods

Method	Description
`set_model(model_name: str, model_version: Optional[str] = "newest") -> ReturnCode`	Load a new pose estimation model.
`get_model() -> Tuple[str, str, ReturnCode]`	Get the currently loaded model name + version (and return code).
`set_method(method: str) -> ReturnCode`	Load a new pose estimation method.
`get_method() -> Tuple[str, ReturnCode]`	Get the current pose estimation method (and return code).
`get_poses(class_names: Optional[Sequence[str]] = (), target_frame: Optional[str] = "") -> Tuple[ReturnCode, List[DetectedObject]]`	Get pose estimates for all objects or a filtered subset by class name.
`get_poses_from_image(color_image: npt.NDArray[np.uint8], depth_image: npt.NDArray[np.uint16], camera_intrinsics: npt.NDArray[np.float32], class_names: Optional[List[str]] = [], target_frame: Optional[str] = "", target_to_camera: npt.NDArray[np.float32] = np.eye(4, 4), timestamp: Optional[float] = None) -> Tuple[ReturnCode, List[DetectedObject]]`	Get pose estimates from a provided RGB-D image + intrinsics.
`multiview_pose_estimation_register_view(class_names: Optional[Sequence[str]] = ()) -> Tuple[ReturnCode, List[DetectedObject]]`	Register the current camera view for multiview pose estimation.
`multiview_pose_estimation_clear_views() -> ReturnCode`	Clear stored camera views for multiview pose estimation.
`multiview_pose_estimation_get_poses(class_names: Optional[Sequence[str]] = (), target_frame: Optional[str] = "") -> Tuple[ReturnCode, List[DetectedObject]]`	Get pose estimates from the registered multiview scene.

Example

from neurapy_ai.clients.pose_estimation_client import PoseEstimationClient
from neurapy_ai.utils.return_codes import ReturnCodes

# Initialize Pose Estimation Client
pec = PoseEstimationClient()

# Define detection parameters
model_name = "tower_building"  # Your custom model name
method = 'neura_DLPE1'  # or 'neura_nonDLPE1', 'neura_nonDLPE2'
class_names = ["puzzle_cuboid", "puzzle_cuboid_red", "puzzle_half_trapezoid"]
target_frame = "root_link"

try:
    # Set model (optional: define version number)
    ret = pec.set_model(model_name)

    if ret.value != ReturnCodes.SUCCESS:
        print(ret.message)
    else:
        # Get current model info
        model_name_loaded, model_version_loaded, ret_get_model = pec.get_model()
        print(f"Pose Estimation Model: {model_name_loaded} ({model_version_loaded})")

        # Set and verify method
        ret_set = pec.set_method(method)
        if ret_set.value != ReturnCodes.SUCCESS:
            print(ret_set.message)
        else:
            method_used, ret_get_m = pec.get_method()
            if ret_get_m.value != ReturnCodes.SUCCESS:
                print(ret_get_m.message)
            else:
                print(f"Used method: {method_used}")

                # Get pose estimates
                ret_detect, poses_list = pec.get_poses(
                    class_names=class_names,
                    target_frame=target_frame
                )
                if ret_detect.value != ReturnCodes.SUCCESS:
                    print(ret_detect.message)
                else:
                    for pose in poses_list:
                        print(f"Pose {pose.object_with_pose.name} value: {pose.object_with_pose.pose}")
                        print(f"Pose characteristics:")
                        print(f"  Score: {pose.detection_score}")
                        print(f"  Confidence: {pose.pose_confidence}")
                        print(f"  Mesh: {pose.mesh_folder_path}")
                        print(f"  Segmentation index: {pose.segmentation_index}")

except Exception as e:
    print(e)

Notes

Multiple pose estimation methods are supported (neura_DLPE1, neura_nonDLPE1, neura_nonDLPE2)
Models can be trained on neura-style datasets (see DataGenerationClient)
The client supports both single-view and multiview pose estimation
Pose estimates include detection scores and confidence values
The client can handle different coordinate frame transformations
Camera intrinsics are required for image-based pose estimation
The client provides detailed error messages for debugging
Pose estimates include mesh information for visualization

Real Data Collection Client

The Real Data Collection Client provides functionality to capture new images and save them as a dataset. It supports different dataset types for various annotation requirements.

Parameters

dataset_type: Type of dataset to create. Available types: * TO_LABEL: Images that must be annotated with objects of interest * ENVIRONMENT: Images containing no objects of interest, intended for capturing environmental context
dataset_name: Name of the dataset that will be created

Class Definition

neurapy_ai.clients.real_data_collection_client

Methods

Method	Description
`save_image() -> ReturnCode`	Save an image from the robot camera.
`save_dataset(dataset_type: DatasetTypes, dataset_name: str) -> ReturnCode`	Create a dataset from the collected images.
`clear_data() -> ReturnCode`	Clear all images that haven’t been saved as a dataset yet.

Example

from neurapy_ai.clients.real_data_collection_client import RealDataCollectionClient
from neurapy_ai.utils.return_codes import ReturnCodes

# Initialize Real Data Collection Client
rdcc = RealDataCollectionClient()

# Define dataset parameters
dataset_name = "example_dataset"
dataset_type = RealDataCollectionClient.DatasetTypes.TO_LABEL  # or ENVIRONMENT

try:
    # Save an image
    ret_image = rdcc.save_image()
    if ret_image.value != ReturnCodes.SUCCESS:
        print(ret_image.message)
    else:
        print("Image saved successfully")

        # Save dataset
        ret_dataset = rdcc.save_dataset(dataset_type, dataset_name)
        if ret_dataset.value != ReturnCodes.SUCCESS:
            print(ret_dataset.message)
        else:
            print(f"Dataset '{dataset_name}' created successfully")

            # Clear unsaved images
            ret_clear = rdcc.clear_data()
            if ret_clear.value != ReturnCodes.SUCCESS:
                print(ret_clear.message)
            else:
                print("Unsaved images cleared successfully")

except Exception as e:
    print(e)

Notes

Two dataset types are supported: TO_LABEL and ENVIRONMENT
Images must be saved before creating a dataset
The client automatically handles ROS service communication
Error messages provide detailed information about failures
The client maintains a clean state by allowing data clearing
Dataset names must be unique
The client supports both labeled and unlabeled data collection

Robot Scan Client

The Robot Scan Client provides functionality to perform a scan of the robot’s environment and generate a mesh file of the scanned scene. It supports different scanning types and camera pose estimation methods.

Parameters

workspace_name: Name of the available workspace in the database
file_name: Name of the output mesh file from scan process
cam_pose_type: Method for estimating camera pose: * 0: Use robot pose
scan_type: Type of scanning to perform: * 1: Workspace scanning * 2: Environment scanning
scene_id: ID of the scene that robot scans (optional, defaults to 0)
data_path: Directory to save scan output (optional, defaults to “”)

Class Definition

neurapy_ai.clients.robot_scan_client

Methods

Method	Description
`start_scanning(workspace_name: str, file_name: str, cam_pose_type: int, scan_type: int, scene_id: Optional[int] = 0, data_path: Optional[str] = "") -> ReturnCode`	Start the scanning process with the given parameters.
`stop() -> Tuple[ReturnCode, Mesh]`	Stop scanning and return the reconstructed scene mesh.
`resume() -> ReturnCode`	Resume scanning after pausing.
`pause() -> ReturnCode`	Pause the scanning process.
`hard_stop() -> ReturnCode`	Force stop the scanning process even during robot motion.

Example

from neurapy_ai.clients.robot_scan_client import RobotScanClient
from neurapy_ai.utils.return_codes import ReturnCodes

# Initialize Robot Scan Client
rsc = RobotScanClient()

# Define scanning parameters
workspace_name = "workspace_1"
file_name = "my_scan"
cam_pose_type = 0  # Use robot pose
scan_type = 1  # Workspace scanning

try:
    # Start scanning process
    ret = rsc.start_scanning(
        workspace_name=workspace_name,
        file_name=file_name,
        cam_pose_type=cam_pose_type,
        scan_type=scan_type
    )

    if ret.value != ReturnCodes.SUCCESS:
        print(ret.message)
    else:
        print("Scanning started successfully")

        # Optional: Pause scanning
        ret_pause = rsc.pause()
        if ret_pause.value == ReturnCodes.SUCCESS:
            print("Scanning paused")

            # Resume scanning
            ret_resume = rsc.resume()
            if ret_resume.value == ReturnCodes.SUCCESS:
                print("Scanning resumed")

        # Stop scanning and get mesh
        ret_stop, mesh = rsc.stop()
        if ret_stop.value == ReturnCodes.SUCCESS:
            print(f"Scanning completed. Mesh saved as {file_name}")
        else:
            print(ret_stop.message)

except Exception as e:
    print(e)
    # Force stop if needed
    rsc.hard_stop()

Notes

Three camera pose estimation methods are supported: * 0: Use robot pose
Three scanning types are available: * 1: Workspace scanning * 2: Environment scanning
The client supports texture mapping for better visualization
Scanning can be paused and resumed during operation
A hard stop is available for emergency situations
The client automatically handles ROS service communication
Mesh files are saved in the specified data path
The client provides detailed feedback during scanning
Scene pointcloud is published to octomap server after stopping

Voice Control Client

The Voice Control Client provides functionality to interact with the voice pipeline for getting and setting commands. It supports both blocking and non-blocking command detection with voice trigger “Hey Maira”.

Parameters

timeout: Time to wait for the command in seconds (optional, defaults to None - no timeout)
command: Command to be detected or set. Examples: * "open gripper" * "move axis two five degrees" * "move axis five" * "inner_command: continue with beep"
similar: Flag to enable similar command detection (optional, defaults to False)

Class Definition

neurapy_ai.clients.voice_control_client

Methods

Method	Description
`get_command_with_trigger(command: str) -> Tuple[ReturnCode, str]`	Get the result of a specific triggered voice command.
`get_command(timeout: Optional[float] = None) -> Tuple[ReturnCode, str]`	Blocking call to wait for a voice command (optionally with timeout).
`set_command(command: str, similar: bool) -> Tuple[bool, str]`	Configure which command to detect (non-blocking).
`get_last_command() -> Tuple[ReturnCode, str]`	Get the last detected command (requires continuous detection to be active).
`get_return_command() -> str`	Start continuous detection (blocking) and return the last command.
`finish() -> bool`	Check whether a command finished executing.

Example

from neurapy_ai.clients.voice_control_client import VoiceControlClient
from neurapy_ai.utils.return_codes import ReturnCodes

# Initialize Voice Control Client
vcc = VoiceControlClient()

try:
    # Blocking call with timeout
    ret_code, command = vcc.get_command(timeout=5)
    if ret_code.value == ReturnCodes.SUCCESS:
        print(f"Detected command: {command}")
    else:
        print(ret_code.message)

    # Get command with trigger
    command = "open gripper"
    ret_trigger, returned_command = vcc.get_command_with_trigger(command)
    if ret_trigger.value == ReturnCodes.SUCCESS:
        print(f"Command result: {returned_command}")
    else:
        print(ret_trigger.message)

    # Handle confirmation command
    ret_code, voice_return = vcc.get_command_with_trigger(
        "inner_command: continue with beep"
    )
    if ret_code.value == ReturnCodes.SUCCESS:
        if voice_return == "continue":
            print("Command confirmed")
        elif voice_return == "terminate":
            print("Command terminated")

    # Non-blocking command setting
    success, msg = vcc.set_command("move axis five", similar=False)
    if success:
        print("Command set successfully")
    else:
        print(f"Setting command failed: {msg}")

    # Check command execution
    if vcc.finish():
        print("Command executed successfully")

except Exception as e:
    print(e)

Notes

Commands must be triggered by saying “Hey Maira”
Both blocking and non-blocking operations are supported
Command detection can be configured with timeout
The client maintains command history
Similar command detection is available
The client supports command confirmation flow
Voice commands are processed in real-time
The client provides detailed error messages
Command execution status can be monitored

Experimental Clients

Audio Output Client

The Audio Output client is an implementation for playing audio files on the robot.

Class Definition

neurapy_ai.experimental.clients.audio_output_client

Methods

Method	Inputs / Outputs
`play_audio(audio_file: str, blocking: bool = True) -> ReturnCode`	Play an audio file on the robot (ReturnCode indicates success/failure).
`stop_all_audio() -> None`	Stop playing all audio in the robot system (publishes stop command).

Bin Detection Client

The Bin Detection client provides methods to detect and retrieve the pose of bins within a workspace. It allows you to estimate a bin’s pose from point cloud data and access the most recently computed pose for downstream processing or motion planning tasks.

Class Definition

neurapy_ai.experimental.clients.bin_detection_client

Methods

Method	Inputs / Outputs
`detect_bin_pose_ros(bin_name: str, workspace_name: str, pointcloud_msg: PointCloud2, z_ws_bottom_offset: float = 0.0, detection_method_type: int = 1) -> Tuple[ReturnCode, BinPose]`	Detect bin pose from a provided `PointCloud2` (ReturnCode + `BinPose`).
`get_bin_pose_ros(bin_name: str) -> Tuple[ReturnCode, BinPose]`	Return the most recently detected bin pose (ReturnCode + `BinPose`).

Cartesian Path Generator Clients

These clients generate Cartesian waypoint paths for downstream motion planning.

Class Definition

neurapy_ai.experimental.clients.cartesian_path_generator_client

Methods

Client / Method	Inputs / Outputs
`CartesianPathGeneratorClient.generate_cartesian_path_from_file(filename: str = "") -> Tuple[ReturnCode, Tuple[CartesianPath]]`	Load a previously saved cartesian path (ReturnCode + tuple of `CartesianPath` segments).
`PlanarPathGeneratorClient.generate_cartesian_path_from_service(object_name: str, use_robot: bool = False, mesh_file: str = "", save_filename: str = "") -> Tuple[ReturnCode, Tuple[CartesianPath]]`	Generate a planar path from a service call (optionally save to file).
`TrackingPathGeneratorClient.generate_cartesian_path_from_service(save_filename: str = "") -> Tuple[ReturnCode, Tuple[CartesianPath]]`	Generate a path by tracking poses (action-based; optionally save to file).
`ModelBasedPathGeneratorClient.generate_cartesian_path_from_service(object_name: str, waypoints_file: str, new_path: bool, new_pose: bool, use_robot: bool = False, mesh_file: str = "", save_filename: str = "") -> Tuple[ReturnCode, Tuple[CartesianPath]]`	Generate a path based on a model + waypoint file (optionally save to file).

Collision Detection Scene Client

The Collision Detection Client loads a collision scene.

Class Definition

neurapy_ai.experimental.clients.collision_detection_client

Methods

Method	Inputs / Outputs
`__init__(scene_id: str = "") -> None`	Initialize (or load) a collision scene by ID.
`attach_object(grasped_object: ObjectWithPose) -> ReturnCode`	Attach an object to the active gripper (placeholder).
`clear_all_temporary_objects() -> None`	Remove temporary objects/point clouds (placeholder).
`add_mesh(mesh_object: CollisionObject) -> ReturnCode`	Add a mesh collision object (placeholder).
`add_primitive(primitive_object: CollisionObject) -> ReturnCode`	Add a primitive collision object (placeholder).
`capture_collision_environment(keep_previous: bool = False) -> ReturnCode`	Capture current environment and add to scene (placeholder).

Grasp Generator Client

The Grasp Generator Client initiates grasp generation and provides access to the resulting grasp candidates as Pick objects. It enables you to supply perception data and task context, trigger grasp computation, and retrieve structured pick results for execution or further planning.

Class Definition

neurapy_ai.experimental.clients.grasp_generator_client

Methods

Method	Inputs / Outputs
`generate_grasps(...) -> ReturnCode`	Generate grasp candidates. Key inputs include: `object_names`, `workspace_name`, `gripper_name`, raw RGB-D + point cloud, optional segmentation/pose estimates, optional bin pose/bbox, and `pick_type`.
`wait_until_motion_allowed(timeout: float = 10) -> Tuple[ReturnCode, bool]`	Block until motion is allowed; returns ReturnCode + `is_restart_detection_required`.
`get_picks() -> Tuple[ReturnCode, List[Pick]]`	Blocking call that returns the generated picks.
`get_instance_picks() -> Tuple[ReturnCode, List[List[Pick]], List[str]]`	Return picks grouped by detected instance ID.
`stop() -> None`	Cancel all active goals.

Note

generate_grasps has a long signature; refer to the Python API for the full parameter list.

Hand Detection Client

The Hand Detection client tracks the index finger while it is moving and returns a time-ordered series of poses. These poses represent consecutive waypoints along the finger’s motion trajectory and can be used for gesture recording, path teaching, or real-time robot guidance.

Class Definition

neurapy_ai.experimental.clients.hand_detection_client

Methods

Method	Inputs / Outputs
`detect_hand_poses(num_poses: int) -> Tuple[ReturnCode, List[Pose]]`	Detect a series of pointed poses (ReturnCode + list of waypoint poses).

Note

The current implementation’s type hint says -> ReturnCode, but it returns (ReturnCode, waypoints).

IK Solver Client

The IK Solver client computes inverse kinematics solutions for a given target pose.

Class Definition

neurapy_ai.experimental.clients.ik_solver_client

Methods

Method	Inputs / Outputs
`compute_ik_solution(goal_pose: PoseStamped, ik_solver_type: int, planning_frame: Optional[str] = None, seed_ik_state: Optional[List[float]] = None, joint_constraint: Optional[List[JointConstraint]] = None, use_axis7_camera: Optional[bool] = False) -> Tuple[ReturnCode, JointState]`	Compute IK (ReturnCode + joint state solution).

Instance Segmentation Client

The Instance Segmentation Client extends the stable instance segmentation client.

Class Definition

neurapy_ai.experimental.clients.instance_segmentation_client

Methods

Method	Inputs / Outputs
`get_segmented_instances_from_image_ros(color_image: Image, class_names: Optional[List[str]] = []) -> Tuple[ReturnCode, List[SegmentedInstanceMsg], Image]`	Segment instances from a ROS `sensor_msgs/Image` (ReturnCode + instances + segmentation mask image).

Motion Execution Client (Experimental)

The Motion Execution Client executes given trajectories.

Class Definition

neurapy_ai.experimental.clients.motion_execution_client

Methods

Method	Inputs / Outputs
`execute_motion(trajectory: Trajectory) -> ReturnCode`	Execute a trajectory (ReturnCode indicates success/failure).
`execute_motion_from_file(filepath: str) -> ReturnCode`	Load trajectory JSON from file and execute all contained segments.

Motion Planning Client (Experimental)

The experimental motion planning client plans joint trajectories for joint goals, pose goals, or waypoint paths.

Class Definition

neurapy_ai.experimental.clients.motion_planning_client

Methods

Method	Inputs / Outputs
`plan_motion_to_joint_goal(goal_joint: JointState, start_joint_states: Optional[JointState] = None, seed_ik_states: Optional[JointState] = None) -> Tuple[ReturnCode, Trajectory]`	Plan motion to a joint-space goal.
`plan_motion_to_pose_goal(goal_pose: Pose, start_joint_states: Optional[JointState] = None, seed_ik_states: Optional[JointState] = None, planning_frame: Optional[str] = None) -> Tuple[ReturnCode, Trajectory]`	Plan motion to a Cartesian pose goal.
`plan_motion_via_cartesian_path(goal_cartesian_path: List[Pose], start_joint_states: Optional[JointState] = None, seed_ik_states: Optional[JointState] = None, filename: Optional[str] = "", ignore_axis7: Optional[bool] = False) -> Tuple[ReturnCode, Trajectory]`	Plan motion through Cartesian waypoints (optionally save trajectory).
`get_saved_motion(filename: str) -> Tuple[ReturnCode, Trajectory]`	Load a previously saved trajectory by filename.
`merge_trajectory_sequence(traj_sequence: Tuple[Trajectory]) -> Trajectory`	Merge sequential trajectory segments into a single trajectory.
`get_current_joint_states() -> List[float]`	Read current joint positions from `/joint_states`.
`update_planning_params(planner_param: PlannerParam) -> None`	Update motion planner parameters via dynamic reconfigure.

Note

Some debug visualization helper methods in the implementation are deprecated and raise DeprecationWarning.

Object Manipulation Client

The Object ManipulationClient queries a manipulation/regrasp sequence between start and end picks.

Class Definition

neurapy_ai.experimental.clients.object_manipulation_client

Methods

Method	Inputs / Outputs
`get_manipulation_sequence(start_picks: List[Pick], end_picks: List[Pick], valid_picks: List[List[Pick]], prefered_orientation: np.ndarray) -> Tuple[ReturnCode, ManipulationSequence]`	Compute a manipulation sequence (ReturnCode + `ManipulationSequence`).

Pick Offset Calculation Client (Experimental)

The experimental pick offset calculation client computes a Z-offset for the next place operation based on a previously executed pick and a workspace.

Class Definition

neurapy_ai.experimental.clients.offset_calculation_client

Methods

Method	Inputs / Outputs
`start_offset_calculation(pick: Pick, workspace_name: str) -> ReturnCode`	Start the offset calculation in a worker thread.
`get_offset() -> Tuple[ReturnCode, float]`	Block until the calculation finishes and return the offset.

Pick & Place Motion Planning Client

The Pick and Place Motion Planning Client plans approach/grasp/retreat motions for a single pick (or place) candidate and can optionally use a planning scene for collision checking.

Class Definition

neurapy_ai.experimental.clients.pickplace_motion_planning_client

Methods

Method	Inputs / Outputs
`set_collision_check(enable: bool = True) -> None`	Enable/disable planning scene collision checking.
`plan_motion_to_pick(pick_candidate: Pick, start_joint_states: Optional[JointState] = None, use_cartesian: Optional[bool] = False, is_place: Optional[bool] = False) -> Tuple[ReturnCode, PickTrajectory]`	Plan the motion to execute a pick (ReturnCode + `PickTrajectory`).
`plan_motion_to_picks(pick_candidates: List[Pick], start_joint_states: JointState = None) -> Tuple[ReturnCode, List[PickTrajectory]]`	Plan motion for multiple pick candidates.
`plan_motion_to_place(place_candidate: Pick, use_cartesian: bool = True) -> Tuple[ReturnCode, PickTrajectory]`	Plan motion to execute a place candidate.

Place Planner Client

The experimental place planner client generates place approach sequences for grid-based, point-based, or gesture-based place groups.

Class Definition

neurapy_ai.experimental.clients.place_planner_client

Methods

Method	Inputs / Outputs
`initialize_place_groups(place_groups: List[PlaceGroup]) -> ReturnCode`	Initialize multiple place groups.
`add_place_group(place_group: PlaceGroup) -> ReturnCode`	Add a place group.
`update_grid_origin_pose(place_group_id: int, updated_grid_origin_pose: Pose) -> ReturnCode`	Update the grid origin pose for a group.
`get_current_place_sequence(place_group_id: int) -> Tuple[ReturnCode, int, ApproachSequence]`	Get the next place sequence (ReturnCode + index + approach sequence).
`get_free_place_sequences(place_group_id: int) -> Tuple[ReturnCode, List[ApproachSequence], List[int]]`	Get all free place sequences (ReturnCode + sequences + free IDs).
`set_place_occupaccy(place_group_id: int, place_ids: List[int]) -> ReturnCode`	Mark place IDs as occupied.
`reset_place_occupaccy(place_group_id: int, reset_all: bool = True, place_ids: List[int] = None) -> ReturnCode`	Reset occupancy (all or selected IDs).
`get_all_used_group_ids() -> Tuple[ReturnCode, List[int]]`	Return all used group IDs.

Planning Scene Client

The Planning Scene Client updates the robot planning scene and octomap using point clouds or depth images, and can add/attach/remove collision objects.

Class Definition

neurapy_ai.experimental.clients.planning_scene_client

Methods

Method	Inputs / Outputs
`start_stop_voxel_update(start_stop_voxel: bool = False) -> ReturnCode`	Start/stop voxel updates.
`publish_octomap_from_topic(topic_name: Optional[str] = "/camera/depth_registered/points") -> ReturnCode`	Publish octomap from a point cloud topic.
`publish_octomap_from_depth_image_topic(topic_name: Optional[str] = "/camera/aligned_depth_to_color/image_raw", image_info_topic: Optional[str] = "/camera/aligned_depth_to_color/camera_info") -> ReturnCode`	Publish octomap from depth image + camera info topics.
`publish_octomap_from_depth_image(depth_image: Image, image_info_topic: Optional[str] = "/camera/aligned_depth_to_color/camera_info") -> ReturnCode`	Publish octomap from a provided depth image.
`publish_octomap_from_pcd(point_cloud: PointCloud2) -> ReturnCode`	Publish octomap from a provided point cloud message.
`clear_octomap() -> ReturnCode`	Clear the octomap.
`add_boxes(boxes: List[SolidPrimitive.Box], id: str) -> ReturnCode`	Add boxes to the planning scene.
`add_spheres(spheres: List[SolidPrimitive.Sphere], id: str) -> ReturnCode`	Add spheres to the planning scene.
`add_mesh(mesh_paths: List[str], mesh_poses: List[Pose] = [], id: str = "dummy_mesh") -> ReturnCode`	Add meshes to the planning scene.
`attach_boxes(boxes: List[SolidPrimitive.Box], id: str, start_state: Optional[JointState] = None, touch_links: Optional[List[str]] = [], link_name: Optional[str] = "gripper_tcp") -> ReturnCode`	Attach boxes to a robot link.
`attach_spheres(spheres: List[SolidPrimitive.Sphere], id: str, start_state: Optional[JointState] = None, link_name: Optional[str] = "gripper_tcp") -> ReturnCode`	Attach spheres to a robot link.
`attach_mesh(mesh_paths: List[str], mesh_poses: List[Pose], id: str, start_state: Optional[JointState] = None, touch_links: Optional[List[str]] = [], link_name: Optional[str] = "gripper_tcp") -> ReturnCode`	Attach meshes to a robot link.
`detach_solid(id: str, solids: Optional[List[SolidPrimitive]] = []) -> ReturnCode`	Detach an attached solid.
`detach_mesh(mesh_paths: List[str] = [], mesh_poses: List[Pose] = [], id: Optional[str] = "") -> ReturnCode`	Detach an attached mesh.
`remove(mesh_paths: Optional[List[str]] = [], mesh_poses: Optional[List[Pose]] = [], id: Optional[str] = "") -> ReturnCode`	Remove objects from the planning scene.

Note

The implementation contains additional helpers such as add_cage, add_bin, and reset_octomap.

Pose Estimation Client

The Pose Estimation Client extends the stable client with additional helpers for:

estimating poses from a full scene definition
estimating poses from instance segmentation results

Class Definition

neurapy_ai.experimental.clients.pose_estimation_client

Methods

Method	Inputs / Outputs
`get_poses_from_scene(color_image: npt.NDArray[np.uint8], depth_image: npt.NDArray[np.uint16], camera_intrinsics: npt.NDArray[np.float32], target_to_camera: npt.NDArray[np.float32], class_names: Optional[List[str]] = [], target_frame: Optional[str] = "", timestamp: Optional[float] = None) -> Tuple[ReturnCode, List[DetectedObject]]`	Pose estimation for a provided RGB-D scene.
get_poses_from_segmentation_result(color_image: npt.NDArray[np.uint8], depth_image: npt.NDArray[np.uint16], segmented_instances: List[SegmentedInstance], segmentation_mask: npt.NDArray[np.uint8], camera_intrinsics: npt.NDArray[np.float32], target_to_camera: npt.NDArray[np.float32], class_names: Optional[List[str]] = [], target_frame: Optional[str] = "", timestamp: Optional[float] = None, include_mask_ids: Optional[List[int]] = [], exclude_mask_ids: Optional[List[int]] = []) -> Tuple[ReturnCode, List[DetectedObject]]	Pose estimation constrained by instance segmentation results.
`get_poses_from_segmentation_result_ros(color_image: Image, depth_image: Image, segmented_instances: List[SegmentedInstanceMsg], segmentation_mask: Image, camera_intrinsics: CameraInfo, target_to_camera: npt.NDArray[np.float32], class_names: Optional[List[str]] = [], target_frame: Optional[str] = "", timestamp: Optional[float] = None, include_mask_ids: Optional[List[int]] = [], exclude_mask_ids: Optional[List[int]] = []) -> Tuple[ReturnCode, List[DetectedObjectMsg]]`	ROS-message friendly version of segmentation-constrained pose estimation.
`get_scene(target_frame: Optional[str] = "") -> Tuple[npt.NDArray[np.uint8], npt.NDArray[np.uint16], npt.NDArray[np.float32], npt.NDArray[np.float32]]`	Fetch RGB image, depth image, intrinsics, and target-to-camera transform.

Stable Pose Generator Client

The Stable Pose Generator Client computes stable poses for an object given a contact pose.

Class Definition

neurapy_ai.experimental.clients.stable_poses_generator_client

Methods

Method	Inputs / Outputs
`start_calculation(object_name: str, contact_pose: Pose, rotation_steps: int) -> ReturnCode`	Start stable pose generation in a worker thread.
`get_poses() -> Tuple[ReturnCode, List[ObjectWithPose]]`	Block until finished and return stable poses as `ObjectWithPose` items.

Robot Actions

Robot actions provide advanced robot movements like pick, place and moving with collision avoidance.

The robot actions class can be imported from the neurapy_ai_utils.functions.robot_actions package and initiated providing robot information like kinematics and gripper.

from neurapy_ai_utils.functions.robot_actions import RobotActions
from neurapy_ai_utils.robot.moveit_kinematics import MairaKinematics
from neurapy_ai_utils.robot.robot import Robot

robot = Robot(MairaKinematics(), DummyGripper())
robot_actions = RobotActions(robot)

Below is an example of how to pick objects using the MarkerDetectionClient to detect the object’s pose and execute the robot’s pick actions.

import rospy
from neurapy_ai.clients.marker_detection_client import MarkerDetectionClient
from neurapy_ai.return_codes import ReturnCodes

from neurapy_ai_utils.grippers.gripper_interface import DummyGripper
from neurapy_ai_utils.robot.kinematics_interface import DummyKinematics
from neurapy_ai_utils.robot.moveit_kinematics import MoveitKinematics
from neurapy_ai_utils.robot.robot import Robot
from neurapy_ai_utils.functions.robot_actions import RobotActions

if __name__ == "__main__":
    marker_detection_client = MarkerDetectionClient()
    rospy.init_node("robot_action_pick")

    robot = Robot(MairaKinmatics(), DummyGripper())
    robot_actions = RobotActions(robot)
    # define a oberserve postion to record the workspace
    observe_position = [0, 0, 0, 1.5, -1.5, 0, 0]
    drop_position = observe_position
    # move to robot to the observe position
    robot.move_joint_to_joint(observe_position)
    return_code, tags, ids = marker_detection_client.getMarkers(10)

    if return_code != ReturnCodes.SUCCESS or len(tags) == 0:
        print("Something went bad")
        exit(0)

    print("---- Picking the object ---- ")

    robot.actions.pick(tags[0], 0.1, -0.1)

    print("---- Droping the object ---- ")

    robot.move_joint_to_joint(drop_position)
    robot.gripper.open()

Collision Avoidance

The neurapy_ai_utils.functions.robot_actions_with_motion_planning_impl package provides robot actions with collison-aware motion planning.

Further Code Examples

Pick with Voice Handover

There are multiple identical objects in the workspace. The robot picks one object after the other and hands it to the operator. The operator coordinates the handover using their voice.

import logging
from typing import List
from neurapy_ai.clients.autonomous_pick_client import AutonomousPickClient
from neurapy_ai.clients.database_client import DatabaseClient
from neurapy_ai.utils.return_codes import ReturnCodes

from neurapy_ai_utils.robot.robot import Robot
from neurapy_ai_utils.robot.maira_kinematics import MairaKinematics
from neurapy_ai_utils.functions.robot_actions import RobotActions
import rospy


class DemoAutonomousPick:
    def __init__(self, id_manager=None):
        self._APC = AutonomousPickClient()

        # Initialize a robot and actions
        self._robot = Robot(MairaKinematics(id_manager=id_manager))
        self._robot_actions = RobotActions(self._robot)

    def set_motion_param(self, speed_mj, speed_ml, acc_mj, acc_ml):
        self._robot.set_motion_param(speed_mj, speed_ml, acc_mj, acc_ml)

    def set_parameter(self, param_name, value):
        self._APC.set_parameter(param_name=param_name, value=value)

    def move_to_look_at_point(self, look_at_point: List[float]):
        self._robot.move_joint_to_joint(look_at_point)

    def execute(
        self,
        workspace_name: str,
        gripper_name: str,
        object_names: List[str] = [],
        bin_name: str = "",
        use_path_planning=False,
    ):
        self._robot.change_gripper_to(gripper_name)
        # Detect picks
        picks = []
        return_code = self._APC.start_detection(
            object_names=object_names,
            workspace_name=workspace_name,
            gripper_name=gripper_name,
            bin_name=bin_name,
        )
        if return_code.value != ReturnCodes.SUCCESS:
            logging.warning("Fail to start detection!")
            logging.warning(
                return_code.message + "with value: " + str(return_code.value)
            )
            return picks

        return_code, picks = self._APC.get_picks()
        if return_code.value != ReturnCodes.SUCCESS:
            logging.warning("No valid sequence find!")
            logging.warning(
                return_code.message + "with value: " + str(return_code.value)
            )
            return

        # Execute picks
        logging.info("Start picking!")
        if len(picks) != 0:
            if use_path_planning:
                self._robot_actions.pick_with_collision_avoidance(picks[0])
            else:
                self._robot_actions.pick(picks[0])
        logging.info("Finish!")
        return


if __name__ == "__main__":
    rospy.init_node("demo_autonomous_pick")
    demo = DemoAutonomousPick()
    db_client = DatabaseClient()

    workspace_name = "test_table"    # Change values according to your setup
    gripper_name = "RobotiQ"         # Change values according to your setup

    object_names = [] # Leave empty for general pick
    bin_name = ""

    speed_mj = 20
    speed_ml = 0.1
    acc_mj = 20
    acc_ml = 0.1

    use_path_planning = False

    demo.set_parameter("enable_collision_checking_with_pointcloud", 1)
    demo.set_parameter("max_aperture", 0.12)
    demo.set_parameter("thresh_rad", 20)
    demo.set_parameter("default_opening", 0.12)
    demo.set_parameter("ranking_method", 0)
    demo.set_parameter("collision_space_padding", 0.005)
    demo.set_motion_param(speed_mj, speed_ml, acc_mj, acc_ml)

    # Save points where the robot observes the workspace and where it hands over the picked object
    # and name them "observe" and "handover" respectfully
    return_code, observe_point = db_client.get_joint_positions("observe")
    if return_code.value < 0:
        raise Exception(return_code.message)
    return_code, handover_point = db_client.get_joint_positions("handover")
    if return_code.value < 0:
        raise Exception(return_code.message)

    for i in range(3):
        demo._robot.move_joint_to_joint(observe_point)
        demo._robot.gripper.open()
        demo.execute(
            workspace_name, gripper_name, object_names, bin_name, use_path_planning
        )
        demo._robot.move_joint_to_joint(handover_point)
        # use voice control: Say Yes/No when asked whether top open gripper
        return_code, voice_return = voice_control_client.get_command_with_trigger(
                                        "Gripper open"
                                    )
        if voice_return == "Gripper open":
            demo._robot.gripper.open()
            return_code, voice_return = voice_control_client.get_command_with_trigger(
                                            "inner_command: continue"
                                        )
            if voice_return == "continue":
                continue
            else:
                return

Scan with Audio Feedback

Execute a scan of a workspace and notifying the user of the status using audio by combining multiple clients.

from neurapy_ai_utils.robot.robot import Robot
from neurapy_ai_utils.robot.maira_kinematics import MairaKinematics

from neurapy_ai.clients.robot_scan_client import RobotScanClient
from neurapy_ai.clients.database_client import DatabaseClient
from neura_ai_robot_api.clients.audio_output_client import AudioOutputClient

import pathlib
import rospy

rospy.init_node("robot_scan_demo")

RS = RobotScanClient()
db_client = DatabaseClient()
robot = Robot(MairaKinematics())
AOC = AudioOutputClient()
audio_dir = "{}/demo_scan/audio/".format(
    str(pathlib.Path(__file__).resolve().parents[3])
)

scan_params = {
    "simplified_factor_env": 0.8,
    "simplified_factor_obj": 0.7,
    "voxel_size_env": 0.035,
    "voxel_size_obj": 0.002
}
RS.set_parameters(scan_params)

# the observe point has to be defined in GUI
return_code, observe_point = db_client.get_joint_positions("observe")
observe_point = []

# scan points named "scan_point_{0,1,2}" should be defined
scan_points = []
for i in range(3):
    ret, point = db_client.get_joint_positions(f"scan_point_{i}")
    scan_points.append(point)

robot.move_joint_to_joint(self._scan_point_list[0], 30.0)
AOC.play_audio(self._audio_dir + "start-scanning.wav", False)

# A workspace named "ws_1" needs to be created on the robot
# from the Wizards > Workspace page
ret = self._RS.start_scanning(
    workspace_name="ws_1",
    file_name="my_workspace",
    cam_pose_type=0,
    scan_type=1,
)
robot.move_joint_via_points(scan_points, 30.0, 30.0)
AOC.play_audio(
    self._audio_dir + "The-scanning-is-finishedI-am-r.wav", False
)
ret, cloud = self._RS.stop()

Move to Marker Location

Use an AruCo marker as target position for the robot. The marker can be moved after each motion execution.

from neurapy_ai_utils.robot.robot import Robot
from neurapy_ai_utils.robot.maira_kinematics import MairaKinematics

from neurapy_ai.clients.marker_detection_client import MarkerDetectionClient
from neurapy_ai.clients.database_client import DatabaseClient

import rospy

if __name__ == "__main__":

    # Initialize a robot and actions
    robot = Robot(MairaKinematics(id_manager=id_manager))

    # --- INITIAL JOINT STATE & CARTESIAN POSE
    previous_joint_angles = robot_status.getRobotStatus("jointAngles")
    previous_cartesian_poses = robot_handler.ik_fk(
        "fk", target_angle=previous_joint_angles
    )

    # --- RUN MARKER DETECTION

    rospy.init_node("test_marker_detection", anonymous=True)
    detect_marker_client = MarkerDetectionClient()
    db_client = DatabaseClient()

    # Move to observe position
    res, observe_point = db_client.get_joint_positions(
        "observe"
    )  # observe point has to be defined in GUI

    if res.value < 0:
        raise RuntimeError("Could not find point in the database")

    robot.move_joint_to_joint(observe_point)

    for i in range(5):
        # Detect 4x4 Aruco markers
        return_code, markers = detect_marker_client.get_detected_markers(1)

        # --- MOVE ROBOT TO DETECTED MARKER

        if len(markers) > 0:
            # Move to the position of the first detected marker
            target_point = robot_handler.ik_fk(
                "ik",
                target_pose=markers[0].pose.to_list(),
                current_joint=previous_joint_angles,
            )

            robot.move_joint_to_joint(target_point)
        else:
            logging.warn("No marker detected. Trying again")

        rospy.sleep(10)

Error & Warning Codes

Code	Name	Description
0	`SUCCESS`	Success.
-1	`INVALID_ARGUMENT`	Invalid argument passed to function. Check network connection!
-2	`DATA_NOT_AVAILABLE`	Requested data is not available in the database. Check the type hint in the API for the function!
-3	`FUNCTION_NOT_INITIALIZED`	Unimplemented abstract method.
-4	`SERVICE_NOT_AVAILABLE`	Server process is not up. Restart process on supervisor page!
-5	`SERVICE_CALL_FAILED`	Service called failed due to invalid input.
-6	`SERVICE_CALL_RETURN_ERROR`	Service failed to be processed.