Bin picking & computer vision

What is bin picking?

It is a complex process in which randomly placed parts must be picked from a container, without disturbing the environment.

In summary:

• Locate the easiest part to pick (usually on top), and compute its position and rotation in space.
• Plan the path of the robot to pick it up, and move it to the target position. The path must account for the position of the container as well as the other parts, in order to avoid collisions.
• Execute the planned path.

Bin picking types & challenges

There are several types of bin picking: Structured, semi-structured, and unstructured.

Structured bin picking involves parts that are well-organized within the bin. The parts are often arranged in a predictable manner, making it easier for robots to locate and pick them up.

Semi-Structured bin picking involves parts that are somewhat organized but not as neatly as in structured setups. There might be some level of disorder or overlap, but it’s manageable enough.

Unstructured bin picking is the most complex type, as parts are randomly placed in the bin with no specific order or orientation, often jumbled together.

In theory, this is seems really simple.

Structured bin picking (in organized bins) isn’t quite so complex to pull off. But semi-structured and unstructured bin picking are particularly challenging, and there are quite a few hurdles to jump over. Grasping parts randomly placed in a bin to then manipulate and reposition in a different pose is challenging, because it involves 3D-object recognition, grasping strategies, and path planning. These are three high-level tech challenges already.

One of the problem is the variations. Semi and unstructured bin picking means it’s almost impossible to predict the way in which the part can appear. It has an almost infinite amount of positions, from being fully visible to be only being partly visible. If it’s partly visible, the bin picking system either needs to be able to see through the other part (think x-ray vision), or it needs to be able to imagine the rest of the position of the part that isn’t visible, to decide where to pick it up from. It also needs to be able to decide whether it wants to pick up that half-hidden part first, or the part on top.

So the algorithm leading the bin picking system needs to be extremely intelligent, as it cannot simply use the available information; it also needs to be able to extrapolate in order to decide on the best course of action.

Essentially, it requires a human brain.

Or does it?

How 3D bin picking works

A 3D bin picking system is composed of two elements: robotics, and 3D computer vision. The vision system allows parts to be located, positioned and oriented, allowing the camera to recognize the scene and the position of the parts. The robotic arm and its end-of-arm-tooling manipulate the parts, and therefore depend on the task presented.

In an ideal world, the 3D vision system will be able to determine which part is the most suitable to extract in each iteration. To achieve this, the 3D vision system typically uses a combination of sensors and cameras that generate a detailed three-dimensional image of the contents of the bin. This imaging can be accomplished using various technologies (structured light, laser triangulation, stereoscopic cameras) which can help in creating a precise depth map of the bin’s contents, distinguishing each part even when they are entangled.

Once the parts are identified and their positions determined, algorithms analyze the optimal picking sequence, taking into consideration the accessibility of parts, the likelihood of causing disturbances to other parts, and the best approach path for the robotic arm.

The robotic arm, equipped with end-of-arm tooling (EOAT), is then guided to execute the pick. The EOAT varies depending on the application—from simple grippers to advanced tools with sensors that adjust grip based on the part’s material and geometry, often including force sensors to adjust the grip and prevent damage to the parts.

More recently, integration of machine learning is becoming prevalent, where the system learns from each pick, adapting and improving its predictions and handling strategies over time.

But one of the most crucial aspect is the communication between the vision system and the robotic arm, involving real-time data transmission and processing. This allows the vision system to adapt to any changes in the bin layout, and to update the strategy as needed.

The failure variables

Because robotic bin picking involves the picking of overlapping parts with an undefined position, several types of failures can occur:

• Object recognition failures: the computer vision algorithm is not able to recognize one or more parts inside the bin. This can happen with reflective, shiny, or transparent parts—and also with changes in lighting conditions, or if the parts are too entangled.
• Pose estimation failures: the part inside the bin is recognized, but its calculated 3D pose is incorrect, and therefore the grasp fails, often due to an inaccurate pose estimation algorithm.
• Constraints robot motion failures: The calculated part pose cannot be reached without collisions of the end effector with the bin edges/corners, or with other parts inside the bin.
• Grasping failures: Grasping can fail for a number of reasons—insufficient friction, grasp interference, entangled objects, multiple grasping, etc.

In many of these cases, the task execution failure will require the assembly line to be paused (if the fault is unrecoverable) so that a human can intervene by entering the robotic cell, clearing the fault, and restarting the line. But these failures negatively influence the average cycle time, which is an important factor in the line performance—and as such, a factor companies are always trying to improve on.

The grippers

Bin picking implies picking things from a bin. But one cannot pick something without a proper grip—which is exactly the role of the EOAT.

To start with: The robots are usually sold without an end effector, as each application has its own set of constraints, geometries and materials. As the situation can vary from one challenge to another, it is not unusual for the EOAT to be custom made. Grippers of all kinds allow robots to manipulate a part, like fingers and hands do for the human body. They are primarily used to hold parts, move them, and release them accurately at a desired location. There are virtually no limits to the style and size of gripper that can be used, and depending on the action, certain technologies might make more sense than others: suction, parallel-jaw, multi-finger (which allows the gripper to reach narrow spaces and pick up objects with a single point of contact), etc.

End effector cost varies widely based on its type and complexity. A small pneumatic gripper or small suction cup system could be only a few hundred euros, while more specialized grippers or large systems can be several thousand euros. High up still, electromagnets or welding systems can be in the tens of thousands of euros.

3D vision systems for bin picking

3D vision robotic systems have properties that make them particularly suited to manufacturing environments. They are flexible, adaptive, precise, need very minimal supervision, are rather safe (the cobots’ sensors allow them to coexist with human workers), and allow for a reduction of security costs (no need for cages or fancy security equipment).

Here are a few systems already on the market. Most of them cannot offer a fully fledged solution, but all these companies have started to come up with options, offering all or part of a solution (semi-structured bin picking, ordered bins, etc) to the hefty price tag usually linked to bin picking solutions:

• Inbolt
• Mech-Viz robot programming software
• Photoneo’s Bin Picking Studio
• Pickit 3D
• Apera

The evolution of 3D vision and machine learning technologies has already started to challenge the traditional bin picking methods used up until now. The variability and complexity of this task are being addressed through more sophisticated systems that enhance accuracy, efficiency, and adaptability in robotic operations.

With ongoing innovations in 3D vision systems (such as AI) and smarter, more intuitive algorithms, robots are not just replicating human capabilities but are starting to surpass them in speed and reliability. This progression will eventually lead to the reshaping of the manufacturing standard, with limitations being pushed and redefined as companies of all size start to look for (and find) solutions to unstructured bin picking problems.

‍