Skip to content
Petiole Pro
Agritech Blog8 min read

Strawberry Blossom Tracking and Fruit Tracking with Smartphone

Focusing on strawberry blossom tracking, in this post, we’ll discuss how to track many strawberry flowers on a video with Petiole Pro module.

By Maryna Kuzmenko

Strawberry Blossom Tracking with Petiole Pro

Nowadays we have been able to solve many plant phenotyping problems with astonishing speed and accuracy with Computer Vision. Yet, multiple object tracking remains a challenging task not only for horticulture and crop science. Only a few of the current methods provide a stable tracking of plant traits at reasonable speed. Focusing on strawberry blossom tracking, in this post, we’ll discuss how to track many strawberry blossoms or berries on a video – and we’ll use a combination of Neural Networks for this. We’re not sharing any code for implementing a strawberry flowers tracker or berry tracker but we provide the technical pieces one to put together a good tracker, the challenges, and a few applications.

Some Applications of Blossom Tracking and Fruit Tracking

Tracking is applied in a lot of real-life use cases, particularly, in the greenhouse, field and orchard.

Strawberry Blossom Tracking is not an easy task but it is possible to get accurate count with Computer Vision and smartphone
Strawberry Blossom Tracking is not an easy task but it is possible to get accurate count with Computer Vision and smartphone

Imagine you are responsible for yield prediction of your strawberry production system, and you would like to understand when is the most reasonable time for producing high yields of quality fruit with sufficient flexibility to meet market demands and labour availability. First of all, accurate estimation of the number of strawberry flowers and their distribution in a strawberry field is, therefore, imperative for predicting the coming strawberry yield. Secondly, predicting the timing and yield of strawberries is important for the industry since accurate forecasts may help with more efficient labour scheduling and reduced crop waste through better crop utilisation. Strawberry yield prediction significantly improves sales, boosts promotions and improves supply chain.

Moreover, you may want to analyze varietal specific berry data to assess the strawberry cultivation system. For all these tasks, you’ll need to detect and track the performance of strawberry plants and analyze how each specific strawberry plant grows and develops.

Usecase for Tracking and Computer Vision in Horticulture

Another usecase is automated crop yield estimation for apple orchards with a smartphone: to optimize the way smartphone camera recongises blossom and yield for apple trees, you may want to build the track for every tree and analyze them.

Strawberry fruit counter with smartphone by Petiole Pro
Strawberry fruit counter with smartphone by Petiole Pro

Finally, modern smartphones allow to detect the flower, fruit or recognise the plant disease on most of the commercially grown fruits and vegetables. For example, there are scientific evidence of successful application of Computer Vision for yield forecast on the following crops:

Cocoa fruit counter helps to count number of cocoa pods on a tree with a smartphone
Cocoa fruit counter helps to count number of cocoa pods on a tree with a smartphone

The approach for Strawberry Blossom Tracking

For now, let’s only focus on strawberry flowers tracking and counting. However, it’s possible to apply the same techniques to a variety of other objects.

A common way to solve the multi-object tracking is to use tracking by detection paradigm. To understand it, we’ll need to be familiar with two other Computer Vision algorithms: strawberry blossom detection and strawberry flower re-identification.

Strawberry Blossom / Fruit Detection

Strawberry Blossom Detection is based on the one of the most popular Computer Vision algorithms out there. Its goal is to find all the objects of interest – strawberry flowers, on the image and output their bounding boxes. This method is applied to a really wide range of objects – all the way from people and cars to bacteria. In our case, however, our focus is on the strawberry flowers – so we’ll do blossom detection.

Strawberry Blossom Detection is based on the one of the most popular Computer Vision algorithms.
Strawberry Blossom Detection is based on the one of the most popular Computer Vision algorithms.

For many years now, blossom detection for fruits and vegetables is almost exclusively solved by Deep Learning algorithms. And for a good reason – even despite this problem is a tough one, Neural Nets are great at it. They significantly advanced the state-of-the-art in detection and thus enabled so many real-world applications – robotic strawberry harvesters, for example, this one, where there is absolutely no error margin in strawberry detection.

Even despite this algorithm family is very advanced, we cannot say that blossom / fruit detection is a solved problem. Typically, detectors still have errors – like false positive detections for objects that are not flowers or strawberries, occlusions, which result in inaccurate localization of fruits, or missed detections for flowers and strawberries. Later, we will discuss how much these errors affect the tracking. Spoiler: a lot.

Strawberry Flower and Fruit Re-identification

Re-identification algorithms, or re-id, need to be able to recognize the same object – flower (or strawberry fruit), on different images. The task of this network is to build a vector of numbers that somehow describes the strawberry blossom visually. For different photos of the same flower, these vectors should be similar, and as different as possible compared to the vectors describing the appearances of leaves, fruits or other objects.

Recognition of strawberry flower with algorithms of Petiole Pro
Recognition of strawberry flower with algorithms of Petiole Pro

Full re-id uses a full image of the strawberry flower. This way has more information and can recognise strawberry blossom even if there is a partial occlusion. A downside though is that it only can be used for specific crop – it is not possible to apply the same flower counter for other berries, for example, raspberries or blueberries. The reason for this links to the algorithm design, which heavily relies to the details of specific flower and does not work when one has different colour or parts.

The vectors Re-ID produces for each image can be treated as points in a multi-dimensional space. For a good Re-ID network, the points corresponding to the different photos of each flower would form a separate cluster.

For us it’s important that with Re-ID, we can quantitatively compare how similar do the detections look. In multiple object tracking, we need to track the strawberry flowers within one specific location in the selected time slot. Because of this, we’ll use full re-id: we’ll get more information, and we don’t want to track other plant traits of strawberries like leaves or fruits.

Tracking by detection

Now let’s look into the tracking – and understand how we combine detection and re-id there.

We’ll treat a video from the strawberry greenhouse as a collection of consecutive frames.

Imagine we have two consecutive frames of a video. On every frame, we’ll first detect strawberry flowers using an object detection neural network. This way we’ll get the bounding boxes for the strawberry flowers on each frame.

Then we’ll match the detections from the second frame to the ones on the first.

Now imagine that we followed the same process for many frames of a video. This way, we’ll have a track corresponding to every strawberry flower we saw.

Matching tracks to detections

In fact, matching of the objects is the tricky part here. For two detections from consecutive frames we need to decide whether they correspond to the same strawberry flower. As usual, to do that, we’ll use three pieces of information about both boxes:

  • How close their appearance is. Here we’ll compare the vectors produced by the re-id network for both images.
  • How close their centres are on the consecutive frames. Given a decent frame rate of the video, we can assume that the strawberry flower cannot suddenly move from one corner of the image to another – which means that the centres of the detections of the same strawberry flower on consecutive frames must be close to each other.
  • The sizes of the boxes. Again, the sizes should be consistent for consecutive frames.

Strawberry Blossom Tracking: Detection in more details

In fact, many tracking algorithms use an internal movement prediction model. It remembers how the image of strawberry flower moved previously and predicts the next location based on a movement model. In video strawberry flowers typically are not captured randomly but rather filmed in a consistent direction. As a result, these models really help match the detections to the right track.

Now we’ll combine these three numbers into a measure of how likely is it that two boxes represent a strawberry flower. If we do this for every possible combination of detections on the first and the second frames, we’ll get a matrix of combined distances between the boxes:

Now we need to assign new detections to old tracks in the best possible way. This is essentially an optimization problem – and to solve it, researchers typically use Hungarian algorithm. It finds the combination of the assignments in this matrix that would be an optimal solution in terms of our combined metric.

Tracking by Detection approach works well in a wide range of tasks, and is pretty fast. Its performance applies the limits of the speed of the detector and re-id nets.

Of course, there are other methods for multiple object tracking out there. The most prominent group is graph-based approaches. These can be accurate but are slow – and don’t allow real-time inference. Moreover, they typically need to look into the “future” to build the correspondences, which again does not allow tracking on-the-fly.

Petiole Pro Strawberry Flowers Tracking Engine

At Petiole Pro, we have created a state-of-the-art engine for strawberry blossom tracking and counting.

To do this, we engineered an optimized neural net that uses 280x less computations than commodity ones. Because of this, our tracking works on smartphones. It is fast, accurate and stable – and thus allows a huge variety of applications for strawberry growers.

If you would like to learn more about it, please write us at support@petioleapp.com.

https://youtu.be/3R7V9e-X04k

Why Strawberry Blossom Tracking is Hard

You may have noticed that the process we described is not exactly bulletprooof. There are so many things that can go wrong:

  • The detector sometimes can miss flowers or create false positives. On the one hand, we need a fast detector to be able to work in realtime. On the other hand, the detector needs to be very accurate to be able to track every flower and not to create excess false positive tracks. To create a nice tracker, we need a tradeoff between the accuracy and speed – as it often happens in Computer Vision.

  • Strawberry flowers may have occlusions for some time and then get visible again on later videos. Tracking needs to “remember” this flower from the past and continue tracking them in the same track. To solve this one, we need to introduce a memory to the tracking. But we cannot remember every flower forever – not only because of the memory consumption, but also because it will make the matching part more complex.
    Again, we need a tradeoff here! For example, if we have a couple of videos from the same strawberry greenhouse we can remember a strawberry flower for several times, and if they return later, the tracking will not think it’s a new blossom.

  • Strawberry flowers in a bunch are a tough case for tracking. Indeed, there often are many strawberry flowers in a bunch or, for example, just a few of strawberry flowers next to each other. Re-identification will only be able to rightfully say that they look similarly. In this case, we’ll have to only rely to the positions and the sizes of the boxes.

https://youtu.be/7iM_7AcI7-0

Conclusions

Typically multiple object tracking algorithms are built on tradeoffs like these. Because of this, they are complex system with tens or hundreds of parameters. On the one hand, this allows customization for specific usecases – but on the other hand, it makes tracking systems complex and hard to build.

More from the blog

All posts →