Augmented Reality interfaces provide intuitive and efficient human-robot interaction without the costly time delay of testing, stopping, and programming separately. Microsoft HoloLens is one such useful AR device, and with it, a whole range of interactive communication is possible.
In order to integrate the HoloLens into robotics projects, a robust communication system is necessary. Since many robotics projects use the ROS framework, incorporating the ROS communication format directly on the HoloLens would be ideal. The ROS-sharp library developed by Dr. Martin Bischoff uses JSON format communication to emulate ROS communication. The rosbridge package provides the conversion between JSON and ROS messages, and ROS-sharp connects to a ROS node running a rosbridge server using WebSockets.
Setup involves starting the WebSocket server on the ROS side and creating a WebSocket client on the HoloLens side.
Setting up the ROS server
Overview of the necessary steps:
1. Install the rosbridge_suite package
2. Launch the WebSocket server
3. Determine the server IP address
The rosbridge_suite package provides a convenient interface for creating ROS messages from JSON. Non-ROS programs such as the HoloLens application can connect to the rosbridge server and use JSON commands to create ROS functionality. This way, HoloLens projects can seamlessly communicate with any ROS node.
Install the rosbridge_suite package by following the site instructions. This just uses apt-get to install the package into your currently installed ROS distribution.
Launch the WebSocket server
After sourcing your ~/.bashrc file, start the rosbridge node that creates a WebSocket server using the command
roslaunch rosbridge_server rosbridge_websocket.launch
Determine the server IP address
Now that the server is running, check the server IP address using ifconfig. This will be used later to configure the HoloLens application so that it can find the WebSocket server.
Setting up the HoloLens project
Overview of the necessary steps:
1. Install Unity2017
2. Download and open the ROS-sharp Unity package
3. Modify the project configuration settings
4. Import and configure the HoloLens toolkit
5. Create a ROS communication link
Hololens projects are most conveniently created through Unity. The Unity Manual provides a good overview of Unity project architecture and how objects and scripts are used to create functionality. The ROS-sharp package provides a deafult Unity project that has the libraries necessary for ROS communication over WebSocket. Once configured properly, this project will allow seamless ROS communication for publishing and subscribing to topics.
The 2017 version is currently required (September 2018) for the HoloLens development toolkit, which provides many convenient scripts. Look for it in the Older Versions section.
Download and open ROS-sharp Unity package
The HoloLens runs on the Universal Windows Platform, so the ROS-sharp library version specific to UWP is necessary. This has been generously provided by David Whitney. Find it here, and download the repository to the Unity workspace. Open the project using the Unity interface:
This project provides the ROS-sharp library preloaded and ready to use.
Modify the project configuration settings
You might notice some build errors in Unity, and this is because the ROS-sharp library uses the .NET Framework 4.6, which is not activated by default in Unity 2017.
Edit > Project Settings > Player > UWP Settings modify the scripting and API settings to use .NET 4.6:
Import and Configure the HoloLens toolkit
The Microsoft Mixed Reality Toolkit provides a collection of scripts and components to make Unity development for the HoloLens as easy as possible. Follow the installation instructions in the link, making sure to download the 2017 release to match up with Unity2017. Import the toolkit as a custom package, and enable the project and scene configuration settings as described in the link’s instructions.
Create a ROS communication link
The HoloLens has some settings that controls what “capabilities” the application is allowed to have. In this case, enable the networking capabilities in
Edit > Project Settings > Player > Publishing Settings
Now that the project is all configured, the last thing to do is to create a WebSocket client to connect to the rosbridge server. Create a RosConnector component, and make sure is uses the WebSocketUWP Protocol. Input the WebSocket server IP address determined earlier during the ROS setup, with the default port of 9090.
Everything is ready! After building and deploying the application using Visual Studio, you will see the HoloLens successfully connect to the ROS WebSocket server.
With the WebSocket connection, the Hololens is connected to the ROS network and capable of publishing and subscribing to topics just like a regular node. The ROS-sharp project already provides some very useful publishers and subscribers as scripts:
as well as some standard ROS message types. It is very easy to create new messages and their accompanying publishers and subscribers. Just follow the directions on the ROS-sharp wiki to create custom messages.