January 30, 2024

Systems Engineering Your Robotics Project

Systems Engineering is a broad and potentially unhelpful term. For this blog, let’s assume it is the practice of drawing boxes and lines on a whiteboard that represent the various components of a system to solve a particular problem. The goal of systems engineering is to create a high-level technical structure that makes building the project easier. This is hard and often done poorly because it requires the person who is doing the design to have a good grasp of the technologies they are designing with. Doing this poorly or lazily often results in an architecture that mimics the organizational structure instead of helping you build a better project–see Conway’s Law. Here, we will give a brief overview of technologies that will help you engineer a robotics project. Specifically, you are going to learn about:

• ROS and how it helps you compose a robot software system.
• How ROS uses a many-to-many network topology.
• Why many-to-many is good for high-performance messaging needed inside your robot.
• How firewalls keep that sort of messaging inside your system and to keep bad actors on the outside from getting in.
• If firewalls kept everything out, the modern internet would not be helpful.
• Two different ways the web has standardized sending messages through firewalls.
• Why websockets are fantastic and have standardized how responsive parts of the web work.
• Why web apps using WebSockets are lovely, and new robotics software is being built.
• How you can bridge the gap between ROS messaging and a websocket using the rosbridge server and roslib libraries.
• How PickNik has used this technology to build MoveIt Pro.

The Blood of Your Robot

Before we can piece together a robotics system, I first need to explain some of the pieces of a robotics system and how they communicate with each other. ROS (Robot Operating System) is a collection of software libraries geared towards building robot applications. Notably, it contains a message passing standard, software packaging, configuration system, logging, and many other things you need. MoveIt is the motion planning framework in ROS you’ll use to calculate and execute motions with a robot arm. One of the key ways MoveIt fits with the other ROS packages is by sending and receiving ROS messages.

The topology of the message passing bus in ROS is many-to-many. Each network member (often a separate process) is called a node or participant. Direct communication in the graph is enabled by the publishing node advertising a unique key called a topic that can be subscribed to. When another node subscribes to a topic, it creates a direct link between the node sending the message and the node receiving it. This minimizes latency, allows you to build robust, redundant software components, and, most importantly, allows you to reuse nodes other people have written that have standardized on a set of topics as their interface. This is a microservices architecture for your robot.

To allow each node to be robust and have the highest communication rate to other nodes, they each use many different ports. Ports are the unique key for the transport layer of networking. When you use a separate port, the operating system’s network stack puts those messages in a unique queue for just that port. All that redundancy in the system creates a lot of discovery traffic. Each node has to notify and discover every other node whenever anything in the graph changes such as:

• New nodes being added to the graph
• New topics being published
• New subscribers on a topic

This network traffic is usually acceptable within a single machine as it is just queued in the operating system.

The problem is that as you scale this network up, the amount of traffic exponentially increases with the number of participants. This is the Brooks’ law of networking. Secondly, as the internet is not all that nice of a place, we use applications at our network borders called firewalls to keep most ports closed. Firewalls also do not generally propagate discovery messages.

The ROS message bus inside your robot is like the veins of your robot. It conveys crucial information from one part of the robot to another efficiently, but it would be really bad if the outside world could put wrong commands in the blood or get the blood out. That is where firewalls help. They separate the local and external networks to function without harming each other.

A Hole in the Wall

If firewalls were sealed entirely and did not pass messages between the inside of your system and the outside of the network, the internet would not be useful. As you design your robot system, it is often constructive to use network messaging to communicate with other computer systems. Some examples are:

• A Factory Automation System that coordinates many different robots and responds to safety events.
• Networks of emergency stops that need to trigger the shutdown of robots.
• A customer ordering website that triggers robots to retrieve or manufacture items.
• Human operators on separate networks that assist the robot in performing its tasks.
• Telemetry systems for collecting information about the performance of your robots so they can be analyzed and debugged.
• A UI for operators to give your robot high-level commands.
• Other robots that need to be on separate networks to allow them each to have the full performance of their local networks, but need to coordinate at a high level.

How does the internet work if all networks are separated and machines aren’t allowed to send messages to each other? The answer is that before sending messages, machines have to negotiate a connection, so we call a session to poke a hole in the firewall. This is where the concept of client and server machines comes from.

Servers are programs that can receive connections on a port, and clients are the ones that can initiate connections. This helps solve the problem of traffic quickly overwhelming the resources of the network because clients and servers have to know about each other beforehand. Firewalls can then do a better job of stopping rogue machines from sending messages that overwhelm the network. Instead of blasting messages to all participants every time something changes, discovery is done through another protocol called the Domain Name System (DNS). Clients send a request to the DNS server to get the IP address of the server associated with a key, called a URL or Unique Resource Locator, they want to access. This centralized database for the discovery of servers also dramatically reduces the traffic required versus a many-to-many discovery system like what ROS uses.

There are two types of communication used on the web: RPC and streaming. RPC stands for remote procedure call and is just what it sounds like. You send a message, and a server runs a procedure and responds with a single response. This is typically how your browser loads a website; it sends a GET request with a URL, and the server responds with the code to render that website in your browser.

The second kind of client-server communication is streaming. That is, the client establishes a connection and leaves it open for two-way message passing. The most common way to do this is called a websocket. This technology underpins much of the modern internet, from notifications on your phone to collaborative websites.

At this point, we are at an impasse. We have two separate networking systems that are awesome in their own way and are helpful at different layers of your system. First, we have ROS, which is good for communications inside your robot and allows you to build a robust robot system from reusable components.Second, we have websockets, which are good for sending and receiving messages over the internet. A natural step would be to bridge these two messaging systems with a bridge server.

Robot System Design

Bridging these two networks is precisely what Robot Web Tools did with their rosbridge_server node. By running this server as part of your ROS system, the outside world can now communicate with your robot. It bridges the two networks so you can send and receive messages across the buses.

At this point you are probably thinking, “I know about the limitations of scaling a many-to-many network like ROS, and multiple solutions for making ROS talk over larger networks.” This is true, those are good options if the problem you need to solve is to connect to is more ROS networks. The superpower of rosbridge is not just connecting over larger networks, but connecting webapp technologies to a ROS network.

If you don’t need the complexity of another ROS network on the other side of the connection, consider the lightweight and humble websocket. One of the many advantages of using it is software on the other side of the connection is not required to have the complexity of a ROS system if it does not need it.

To help you write software that talks the rosbridge protocol there are a set of libraries for various modern languages that work natively in those languages’ ecosystem.

• roslibjs - JavaScript
• jrosbridge - Java
• roslibpy - Python
• roslibrust - Rust

Many awesome projects have been created to enable you to interact with your robot through the web, including our software, MoveIt Pro.

MoveIt Pro runs the rosbridge on port 3201. So, assuming your robot is located at the IP address 192.168.0.56, here is a script that will command it to close the gripper:

You can even use this system to remotely introspect the messaging of a ROS system the same way you would with the ros2 command line interface if you were logged into the machine with the ROS network. Here are the docs with examples of querying.

To read more about how to interact with MoveIt Pro through the rosbridge protocol read our docs.