Cloud Detection Challenge

CHALLENGE

CLOUD DETECTION CHALLENGE

maps

ORGANIZED BY

maps

Introduction

In this competition, researchers worldwide are invited to tackle the task of binary classification focused on cloud detection using a novel dataset exclusively released for this challenge. The dataset has been meticulously constructed by transforming raw data collected by a measuring instrument, specifically a lidar-based ceilometer, into images. These images are not conventional photographs of the sky or satellite images but are generated using backscatter profile measurements that depict the evolution of the sky's state above the instrument. Images were generated daily and subsequently divided into hourly segments.

The objective is to develop innovative solutions or architectures that surpass the proposed baseline, achieving higher accuracy in cloud detection. Participants will be provided with a dataset of images, each labelled with binary annotations indicating the presence or absence of clouds. The challenge is to design and implement a classification model that accurately identifies cloud presence in these images. The dataset is diverse, encompassing various environmental conditions, perspectives, and challenges commonly encountered in real-world scenarios.

The performance of the models will be evaluated based on accuracy, F1 score, precision, and recall, emphasizing achieving superior results compared to the baseline model provided. All metrics will be considered, and the winner will be the team with the best overall performance across all metrics. In case of a tie, precision and recall values will be prioritized. If a further tie occurs, the solution with the highest accuracy value will be selected. The evaluation order is as follows: (1) F1 score, (2) precision and recall, (3) accuracy.

The baseline model will serve as a carefully crafted reference point for participants to surpass. The goal is to showcase advancements in cloud detection through cutting-edge techniques, novel architectures, or innovative pre-processing methods.

Significance of the challenge

The Cloud Detection Challenge holds paramount importance in artificial vision applications and image analysis. The ability to accurately identify the presence of clouds in satellite imagery or landscape photographs carries profound implications across various sectors, including meteorology, environmental monitoring, agriculture, and satellite imaging. By utilizing a novel dataset, based on new types of data and, therefore, new types of images, this challenge provides an opportunity to advance research on this topic by integrating new data sources with those already known.

Precise cloud classification is essential for understanding climate changes, predicting weather phenomena, and optimizing agricultural operations. The challenge not only calls for innovation in developing advanced models for binary classification but also offers an opportunity to make substantial contributions to scientific and technological progress in key sectors reliant on accurate image analysis.

By participating in this challenge, researchers and developers can showcase their expertise in computer vision, contributing to the creation of solutions that push beyond current technological boundaries. The results could have a tangible impact on real-world applications, enhancing our understanding of the environment and supporting informed decision-making across various industries.

The Competition Website

The competition will be hosted on a dedicated server pointed by the following subdomain, on which rules, dataset, registration and submission forms will be available:

https://iplab.dmi.unict.it/cloud-detection-challenge

The website is now open.

Competition promotion plan

The competition will be promoted using dedicated social media groups and channels. Also, mails will be sent to the most known research groups/researchers on the field. In particular, the following media channels will be considered for promotion:

The Data

Competitions Rules (registration, submission, presentation)

Participants should start preparing a descriptive text for their final submission. Indeed, top-ranked teams will be invited to present their works at the MetroXRAINE 2024 conference during a special session for the challenge. More details will be available on the website dedicated to the challenge.

No later than 31/07/2024 (extended) participants must register at the following website (https://iplab.dmi.unict.it/cloud-detection-challenge/) where they must specify the team members (first name, last name, affiliation, email contact) and the name of their team. It will not be allowed to unify teams after the beginning of the challenge even if they belong to the same university. Training data will be released on 15/07/2024 (extended), while test data will be released on 05/08/2024 (extended).

The technical documentation that participants must submit no later than 31/08/2024 (extended) must be written in English and must contain all the details of the proposed approach (for example, if you use a deep neural network algorithm then you must describe the architecture used, the parameters, etc.) and the results obtained, as well as the comparison with the methods reported at the following link: (https://iplab.dmi.unict.it/cloud-detection-challenge/research).

For this purpose, templates will be provided to compile the report containing the details of the proposed method and the csv file containing the results. All templates will be available at the following link: (https://iplab.dmi.unict.it/cloud-detection-challenge/templates).

Criteria of judging a submission

  • Classification Accuracy: Precision in correctly distinguishing between the two classes is crucial. Accuracy will be used as a starting point to assess the overall model performance.
  • Precision and Recall: Precision indicates the proportion of true positive predictions among all positive predictions. Recall measures the proportion of true positive predictions among all actual positive instances. A good balance between precision and recall is desirable, but their importance may vary depending on the context of the problem.
  • F1 Score: The F1 score is the harmonic mean of precision and recall. This metric provides a balance between the two and can be particularly useful in cases where minimizing both false positives and false negatives is important.

The Competition Schedule

The challenge will be organized on the following time-schedule:

  • 21/06/2024: Competition website online
  • 22/06/2024 – 31/07/2024: Registration
  • 15/07/2024: Release of training set with baseline solutions
  • 05/08/2024: Release of test set
  • 31/08/2024: Deadline for final submission
  • 09/09/2024: Winners announcement
  • 23/10/2024: Presentation and/or Award at the MetroXRAINE conference

Dataset to be used

The dataset to be used consists of a new dataset created and provided by the challenge organizers. The dataset consists of images obtained by plotting the backscatter values measured by the ceilometer, a special measuring instrument used in meteorology, capable of detecting the height of a cloud base by emitting a modulated beam of light directed towards the sky. The height of the clouds is calculated by measuring the time of flight of the emitted light beam.

The ceilometer takes measurements every 15 seconds, quantifying the concentration of particles in the atmosphere. By taking advantage of the reflection, it is possible to determine the cloud layer coverage.

Once raw data were collected, the variables of interest were normalized using a normalization factor of the lidar- based ceilometer. The variable containing the backscatter coefficient is also used in addition to the time variable. All the variables are put together to be used to generate backscatter profiles. Specifically, we plot the time on the x-axis and the height of the measured particles (contained in the backscatter coefficient) on the y-axis. We generated a backscatter profile for each day of measurements and each of these were further divided for every hour. The produced backscatter profiles were labeled with the support of a Weather Research and Forecasting (WRF) Model.

Publication plan

The best-selected solutions will be described in a challenge report paper. The paper will include the most significant works and their findings.

In addition to the IEEE MetroXRAINE 2024 challenge presentation, the authors of the best-selected works will be invited to submit their contribution to a special issue of a valuable Journal.

Presentation at IEEE MetroXRAINE 2024

The authors of the best-selected works will be invited to present their contribution at MetroXRAINE 2024 in a Poster Session. In this case, the Registration fee will be 180€ per person in case of 1 Day Registration or free of charge for Conference Registered Attendees.

Cloud Detection Challenge Best Works - Click here to register

Organizers and Contacts

WITH THE PATRONAGE OF

University_of_Hertfordshire
bath
unina
polimi
unisalento
stiima
unisi
HSMW
POLIBA
UPV
eureca
puc
univpm
cei
cei
cirmis
dieti
arhemlab
res4net
ageit
mics
carmelo
imeko
GMEE

SPONSORED BY

gtec
unitree
ABB
neuroconcise
brain
animateyourscience
sensors
instruments