Remote (tele-op) connection from mobile

The remote platform has been taking some shape.

Here is a UR10e performing maneuvers remotely in the virtual Isaac sim environment. It also works with Lite6 and the mecharm in the background.

This is input streamed from a mobile phone to a simulation running in the Nebius cloud in another country. The live video feed of the movements is being returned.

The project is not intended for mobile-phone robot control, but this demonstrates the offloading of GPU to the cloud, allowing the operator to not need a powerful machine.

The 2nd part of the video is from a PC with an Xbox game-pad controller to show that it can be operated a little more smoothly.

---

Soft surface pickup with environment co-optimization

As detailed in The fine art of picking up on a soft surface — featuring Nebius Serverless AI

This is the extended work of soft-surface pickups.

Most of the surface optimization was done in the previous work. This chapter looked at what happens when you introduce a lot more environmental noise. It starts with trying to integrate the existing models into the remote-platform and part-sorting behavior trees.

And the results were mediocre, so there is not a lot to show. Nonetheless, here is a preview of some changes to the Winkler-Pasternak soft-surface. There have been some experiments with tilting the heads of the individual bed tiles.

In any case, I also couldn’t find much work done in this space: using a soft surface to help with object pickup. Maybe it would make a good white paper? (yeah… I’m not going to make one)

---

Ramp pickup and tool generation

This looks at the same problem in a different way. It’s inspired by a visit to another robotics startup that is building dexterity models to achieve all goals with only the grippers.

With the latest news on one-shot-learning or few-shot-learning, it is of course now considered “doable”.

But even then… an actual human professional doing the job would start considering tools after a while.

So, this project just asks AI to generate the tools it needs to optimize the task’s success rate. It’s then trivial to print them out.

In this video we see the result of the behavioral tree that:

• orients and picks up the sweeper tool
• moves the target to an appropriate starting position
• puts a ramp in an appropriate position to perform the maneuver
• pushes the target up the ramp
• picks up the target off the end of the ramp

Well almost - I selected this video to show what can go wrong, and when it can recover, and when a model can break down.

---

Press box demo simulation

This is a simple machine tending demo in its very early stages. It’s modeled first in Blender to get the dimensions right, then brought into Isaac sim for the physics and programming. It’s still in a stage of getting the collisions sorted out.

The robot loads 4 parts into a press and then takes out the finished item. The goal is 1 part every 2 minutes. The sub assembly is stored in a magazine with dispensers.

In this case the entire unit is portable in order to be able to bring the real hardware to exhibitions and conferences.

For the arms, we started with the AgileX PiPER, but are now considering the I2RT YAM.