Search for an image in an image

Using OpenCV should certainly be more successful. I experimented with AForge

It turned out pretty good too

Here is the class of working with the library, in fact, the use itself takes 2 lines, I repent, I took this part from the Internet, I wrote the rest myself

public class AforgeService
{
    //найденные совпадения
    private TemplateMatch[] _matchings;

    /// <summary>
    /// Количество найденных совпадений
    /// </summary>
    public int CountMatchings
    {
        get => _matchings != null ?  _matchings.Length : 0;
    }


    //ctor
    public AforgeService()
    {

    }

    /// <summary>
    /// Содержит ли исходное изображение представленый образец
    /// </summary>
    /// <param name="pathOriginalImage">путь к файлу исходного изображения</param>
    /// <param name="pathSampleImage">путь к файлу образца</param>
    /// <returns>true если содержит</returns>
    public async Task<bool> IsContains(string pathOriginalImage, string pathSampleImage)
    {
        if (String.IsNullOrEmpty(pathOriginalImage)) throw new ArgumentNullException(nameof(pathOriginalImage));
        if (String.IsNullOrEmpty(pathSampleImage)) throw new ArgumentNullException(nameof(pathSampleImage));

        var sample = new Bitmap(pathSampleImage);
        var orig = new Bitmap(pathOriginalImage);

        //пользуемся библиотекой
        ExhaustiveTemplateMatching tm = new ExhaustiveTemplateMatching(0.921f);
        _matchings = await Task.Run(() => tm.ProcessImage(orig, sample));

        return _matchings.Any();
    }


    /// <summary>
    /// Получение коллекции найденных мест где находится образец
    /// </summary>
    /// <returns>коллекция найденных мест</returns>
    public List<FoundPlace> GetPlaces()
    {
        List<FoundPlace> result = new List<FoundPlace>();
        if (CountMatchings == 0) return result;

        int id = 0;
        foreach (var match in _matchings)
        {
            FoundPlace place = new FoundPlace
            {
                Id = ++id,
                Similarity = match.Similarity,
                Top = match.Rectangle.Top,
                Left = match.Rectangle.Left,
                Height = match.Rectangle.Height,
                Width = match.Rectangle.Width
            };

            result.Add(place);
        }

        return result;
    }

}

This class is for saving the found location

public class FoundPlace
{
    public int Id { get; set; }
    public double Left { get; set; }
    public double Top { get; set; }
    public double Width { get; set; }
    public double Height { get; set; }
    public double Similarity { get; set; }
}

This is the viewmodel class

public class MainViewModel : INotifyPropertyChanged, IDisposable
{
    public event PropertyChangedEventHandler PropertyChanged;

    private readonly IMainWindow _mainWindow;
    private string _pathOriginalImage;

    //ctor
    public MainViewModel(IMainWindow mainWindow)
    {
        _mainWindow = mainWindow;
    }

    /// <summary>
    /// Флаг запуска поиска, для выкл./вкл. кнопок
    /// </summary>
    private bool _IsSearching;
    public bool IsSearching
    {
        get => _IsSearching;
        set
        {
            _IsSearching = value;
            SelectSampleCommand.RaiseCanExecuteChanged();
            SearchCommand.RaiseCanExecuteChanged();
        }
    }

    /// <summary>
    /// Исходное изображение (поле игры)
    /// </summary>
    public string OriginalImage
    {
        get => @"~\..\Assets\Original.jpg";
    }

    /// <summary>
    /// Образец для поиска
    /// </summary>
    private string _SampleImage;
    public string SampleImage
    {
        get => _SampleImage;
        set
        {
            _SampleImage = value;
            PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(nameof(SampleImage)));
            SearchCommand.RaiseCanExecuteChanged();
        }
    }

    /// <summary>
    /// Текстовое сообщение о процессе
    /// </summary>
    private string _Message;
    public string Message
    {
        get => _Message;
        set
        {
            _Message = value;
            PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(nameof(Message)));
        }
    }

    /// <summary>
    /// Список найденных мест для ListBox
    /// </summary>
    private List<FoundPlace> _Places;
    public List<FoundPlace> Places
    {
        get { return _Places; }
        set
        {
            _Places = value;
            PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(nameof(Places)));
        }
    }

    private FoundPlace _SelectedPlace;
    public FoundPlace SelectedPlace
    {
        get => _SelectedPlace;
        set
        {
            _SelectedPlace = value;
            PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(nameof(SelectedPlace)));
            _mainWindow.DrawPlace(_SelectedPlace);
        }
    }



    /// <summary>
    /// Кнопка Выбрать
    /// </summary>
    private RelayCommand _SelectSampleCommand;
    public RelayCommand SelectSampleCommand
    {
        get => _SelectSampleCommand = _SelectSampleCommand ?? new RelayCommand(OnSelectSample, CanSelectSample);
    }
    private bool CanSelectSample()
    {
        if (IsSearching)
        {
            return false;
        }
        return true;
    }
    private void OnSelectSample()
    {
        string file = _mainWindow.SelectSample();
        if (String.IsNullOrEmpty(file)) return;

        SampleImage = file;
    }

    /// <summary>
    /// Кнопка Искать
    /// </summary>
    private RelayCommand _SearchCommand;
    public RelayCommand SearchCommand
    {
        get => _SearchCommand = _SearchCommand ?? new RelayCommand(OnSearch, CanSearch);
    }
    private bool CanSearch()
    {
        if (String.IsNullOrEmpty(SampleImage) || IsSearching)
        {
            return false;
        }
        return true;
    }
    private async void OnSearch()
    {
        Message = "Ждите...";
        IsSearching = true;
        AforgeService service = new AforgeService();

        try
        {
            using (OverrideCursor cursor = OverrideCursor.GetWaitOverrideCursor())
            {
                string pathOrigin = GetOriginalImage();
                bool isContains = await service.IsContains(pathOrigin, SampleImage);
                if (isContains)
                {
                    Places = service.GetPlaces();
                }
                else
                {
                    Places = new List<FoundPlace>();
                }
            }
        }
        catch (Exception ex)
        {
            var message = $"Возникла ошибка: {ex.Message}";
            _mainWindow.ShowMessage(message, "Ошибка");
        }
        finally
        {
            Message = $"Найдено мест: {service.CountMatchings}";
            IsSearching = false;
        }
    }

    /// <summary>
    /// Получение пути к оригинальному изображению (полю игры)
    /// </summary>
    /// <returns></returns>
    private string GetOriginalImage()
    {
        if (!String.IsNullOrEmpty(_pathOriginalImage) && File.Exists(_pathOriginalImage))
        {
            return _pathOriginalImage;
        }

        _pathOriginalImage = Path.Combine(Path.GetTempPath(), "Original.jpg");


        Uri imgUri = new Uri("pack://application:,,,/Assets/Original.jpg");
        StreamResourceInfo imgStream = Application.GetResourceStream(imgUri);

        using (Stream imgs = imgStream.Stream)
        using (FileStream fs = File.Create(_pathOriginalImage))
        {
            byte[] ar = new byte[imgs.Length];
            imgs.Read(ar, 0, ar.Length);

            fs.Write(ar, 0, ar.Length);
        }

        return _pathOriginalImage;
    }

    /// <summary>
    /// IDisposable
    /// </summary>
    public void Dispose()
    {
        if (!String.IsNullOrEmpty(_pathOriginalImage) && File.Exists(_pathOriginalImage))
        {
            File.Delete(_pathOriginalImage);
        }
    }
}

The entire example can be downloaded here

I think you will have enough functionality implemented in OpenCV

OpenCV (Open Source Computer Vision Library, open source computer Vision Library) is an open source library of computer vision algorithms, image processing, and general - purpose numerical algorithms.

(c) Wikipedia.

Here are some good lessons on it, and here is what you might need in theory.

There is a large, complex, and popular area of technology called pattern recognition. For the most part, they use deep neural networks, but not only. And it is unlikely that such complexity is needed in your case (although it may inspire the right vector of further actions).

A simple way is to break the playing field into parts (it seems that this is especially true in this game) and check each part for similarities with known objects. If the items can be placed in random places you need a more advanced analysis: you can first check against the grid, then against the same grid with an offset of half (or even a third and two-thirds). You can also scale, take a certain hash from a part of the image...

By the way, if you can not achieve 99% accuracy (which is possible in principle, because all objects will be static, and not rotate under different lighting, as in the real world), then you can compensate for the accuracy of thoughtful clicks: save positions that have already been clicked unsuccessfully, so that the next time you do not blunt, do not waste time. Then the main thing is that the search algorithm produces as few false negative positives as possible, and false positive ones will be quickly cut off in practice.

Useful links:

• Closed question on ruSO, but still with some interesting ideas.

• Article on image recognition (Habr)

• Method Viola-Jones for face search and recognition (Habr)

If you need to search an image for a fragment of this image, and not something remotely resembling a fragment, there is nothing better than the correlation method. Fast and reliable. And you don't need any recognition.